Is there an AI that generates malware already?
No sarcasm, I really wonder it there is already a model that finds exploits and makes rootkits or something.
It could make red teams obsolete.

I saw one of those way back when the AI hype was going. I didn't save the link

there's jailbroken darker AIs though, yes. or were at the tail-end of the AI hype

I just saw the screenshots and claims and it didn't seem fake to me

they hosted their own models

Yes. We already have server side polymorphs for a long time now.
The solution was dynamic behavior analysis - which will solve the "whatever AI can spew".
There is also a cool approch to defend against malware I encountered - based on "unique code" vs "old code pattern". Malware tend to the first, while non malware tend to the second. The reason is simple: what self respecting malware auther use glibc? answer: noobs.
Also - Checkout the "grand cyber challlenge" from 2016.

Github Copilot

I can imagine known exploits, fuzzing, generating fake content for payload delivery, etc...
Can't currently imagine it would "discover" new malware/approach, which wouldn't be based on existence within its training data