ChatGPT Custom Instructions: Persistent Data Exfiltration Demo
Brief
ChatGPT is vulnerable to data exfiltration via image markdown injections. This. is. pretty well known .
As more features are added to ChatGPT the exfiltration angle becomes more likely to be abused.
Recently OpenAI added Custom Instructions , which allow to have ChatGPT always automatically append instructions to every message exchange.
An adversary can abuse this feature to install a data exfiltration backdoor that depends on, and only works because of the image markdown injection vulnerability. The TTP is a similar to other post exploitation techniques adversaries are using, like enabling email forwarding rules .