ChatGPT Prompt Attack

Overview

Researchers have disclosed a vulnerability in ChatGPT that demonstrated how a malicious instruction hidden inside a conversation could cause ChatGPT to access and exfiltrate data from a user's connected applications without the user's knowledge. In a proof of concept, researchers used the technique to retrieve information from a victim's connected Gmail account and transfer it to another ChatGPT account controlled by the attacker.

The attack combines prompt injection with an unintended communication channel between ChatGPT's isolated code-execution environments. Prompt injections occur when an attacker places instructions into content that an AI processes, causing the AI to perform actions that were not requested by the legitimate user. OpenAI identifies prompt injection as a significant security challenge for AI systems that interact with external data and applications.

Importantly, the attack did not require the victim to deliberately provide their Gmail contents to the attacker. The malicious instruction could be introduced through a pasted prompt, a shared ChatGPT conversation or a custom GPT containing the instruction in its configuration.

Key Highlights

• Researchers demonstrated data moving from one ChatGPT account to another without direct communication between the two accounts.

• The proof-of-concept accessed data from the victim's connected Gmail account and could also target ChatGPT conversation history and files.

• The attack abused the access already available to the victim's ChatGPT session.

• ChatGPT could answer the user's legitimate question while simultaneously performing the attacker's hidden task.

• Depending on the connected-app permission configuration, ChatGPT could read information without requiring additional approval for every read operation.

Recommendations

1. Review AI connections: Identify AI assistants connected to Gmail, Microsoft 365, cloud storage and other sensitive systems.

2. Apply least privilege: Grant AI tools only the permissions required and enable approval for sensitive actions where available.

3. Treat AI content as untrusted: Do not blindly follow instructions in shared chats, documents, emails or custom GPTs.

4. Monitor AI activity: Watch for unusual access to email, files and other connected services.

5. Assess AI integrations: Include AI assistants and their integrations in existing third-party and application security reviews.

Threat Intelligence Assessment

The incident demonstrates an important shift in AI security where the risk is no longer limited to an AI generating an incorrect answer. An AI agent with access to enterprise data can potentially become part of the attack chain.

The immediate vulnerability has been addressed by OpenAI, but the broader security lesson remains: prompt injection should be treated as an access-control and data-exfiltration risk when AI systems are connected to sensitive business applications.