What Is AI Data Leakage?
AI data leakage occurs when sensitive, confidential, proprietary, or personal information is exposed through an AI model, application, prompt, training pipeline, retrieval system, integration, API, logging system, or AI-enabled workflow without appropriate authorization or controls. It's a broader concept than a single bad prompt: leakage can happen before data ever reaches a model (in an upload or integration), while the model is processing it (in a prompt or retrieval step), or after the model responds (through its output, logs, or a connected AI agent taking further action).
Because there are so many stages where this can go wrong, "AI data leakage" isn't one failure mode — it's a category that includes employee oversharing, misconfigured retrieval-augmented generation (RAG) systems, over-permissioned AI agents, exposed APIs, unmonitored logs, unmanaged "Shadow AI" tools, and vulnerabilities in the AI vendors an organization relies on.
AI Data Leakage vs. AI Data Breach
These terms get used interchangeably, but they aren't the same thing, and the distinction matters for how an organization responds.
AI data leakage describes an outcome: sensitive information has been exposed, disclosed, transferred, or made accessible outside its intended control boundary through an AI system. An employee pasting a client contract into an unapproved chatbot is a leak the moment that data leaves the organization's control — regardless of what happens afterward.
AI data breach is a more specific, often legally defined term: a security incident involving unauthorized access, disclosure, or acquisition of protected information, as defined by the applicable law, regulation, or contract. Not every AI data leak automatically meets the legal threshold for a "breach" — that depends on factors like the type of information involved, the jurisdiction, whether access was actually unauthorized, how many individuals were affected, and any contractual notification obligations.
In practice: a leak is the technical/operational event. Whether it also triggers breach-notification obligations is a legal question that depends on the specifics — this article isn't a substitute for that legal assessment, and organizations should involve legal and compliance teams when a leak involves regulated data.
AI Data Leaks vs. AI Model Leaks
This distinction is one of the most commonly confused — and most commonly searched — topics in this space, so it's worth being precise.
An AI data leak involves exposure of sensitive business, personal, financial, or technical information — the kind of data an organization feeds into or generates through an AI system. Source code pasted into a chatbot, customer records surfaced by a misconfigured RAG pipeline, or PII returned in an AI agent's output are all data leaks.
An AI model leak involves exposure of the model or its restricted components themselves — not the data flowing through it. This includes:
- Unauthorized model access — someone gaining access to a model or environment they weren't authorized to use.
- Model extraction or theft — an attacker reconstructing a model's behavior, or in extreme cases obtaining model weights, through repeated querying or a security failure.
- Model-weight or artifact exposure — proprietary model files, checkpoints, or fine-tuning artifacts becoming accessible outside their intended boundary.
- System-prompt leakage — the hidden instructions that shape an AI application's behavior being extracted and exposed, which can reveal proprietary logic or create a roadmap for bypassing safeguards.
- Training-data extraction — a model reproducing fragments of its training data when prompted in specific ways, which sits at the intersection of a data leak and a model-behavior issue.
These aren't interchangeable. A company can suffer a serious data leak (an employee overshares customer data) without any model leak occurring, and vice versa (a system prompt gets extracted with no customer data involved). Enterprise AI risk programs need controls for both categories, not a single generic "AI leak" policy.
Types of AI Data and Model Leaks
Prompt Leakage
Sensitive information entered directly into a prompt — source code, financial figures, health data, strategic plans — becomes part of that AI interaction and, depending on the tool's data-handling terms, may be retained or used for training. Mitigation: enforce data classification and prompt-inspection controls before submission to any AI tool.
Training-Data Leakage
A model memorizes fragments of the data it was trained or fine-tuned on and later reproduces them, verbatim or near-verbatim, in response to certain prompts. This is a known risk with large language models and is one reason organizations fine-tuning models on proprietary or customer data need to think carefully about what goes into that training set. Mitigation: minimize sensitive data in training/fine-tuning sets and test models for memorization before deployment.
RAG Leakage
A retrieval-augmented generation system pulls content from a document store or vector database to answer a query. If the retrieval layer doesn't preserve the original access permissions on those documents, a user can end up seeing content they were never authorized to see. Mitigation: enforce document-level, identity-aware authorization at the retrieval layer itself, not just at the application's front door.
AI Output Leakage
A model's response reveals information it shouldn't — because it retrieved something out of scope, memorized something from training, or was manipulated into disclosing it. Mitigation: apply output filtering and monitoring, particularly for AI applications with broad data access.
AI Agent Leakage
An AI agent with permissions to read files, query databases, call APIs, or take actions can move sensitive data across systems on its own initiative, sometimes as the result of a prompt injection or simply a task it misinterprets. Mitigation: least-privilege, identity-scoped permissions per agent, with human approval gates for sensitive actions.
API and Integration Leakage
AI tools are often wired into other business systems through APIs. A misconfigured integration, an overly broad API scope, or an exposed endpoint can leak data that never even touched a chat interface. Mitigation: apply the same API security discipline (authentication, scoping, rate limiting, monitoring) to AI integrations as to any other system integration.
Logging and Telemetry Leakage
Prompts, responses, and system logs from AI applications often contain the same sensitive data that triggered the original concern — and logs are frequently under-protected relative to production data stores. Mitigation: classify and access-control AI logs the same way you would the underlying data.
Shadow AI Leakage
Employees adopting AI tools without IT or security review means the organization has no visibility into what data is going where, under what retention terms, or under whose vendor agreement.
Mitigation: AI discovery tooling combined with a clear list of approved tools and a fast path for employees to request new ones.
Third-Party/Supply-Chain Leakage
Data shared with an AI vendor, or with that vendor's own subprocessors and model providers, inherits all of their security and retention practices — which the enterprise typically doesn't control directly. Mitigation: vendor due diligence covering data retention, training use, subprocessors, and security certifications (see the vendor-risk section below).
Model Access Leakage
Credentials, API keys, or accounts granting access to a restricted or proprietary model fall into the wrong hands, whether through credential theft, misconfiguration, or an exposed third-party environment. Mitigation: strong identity and access management around model endpoints, with the same rigor applied to production infrastructure.
System Prompt Leakage
An AI application's underlying instructions — which often encode business logic, guardrails, or proprietary prompting techniques — get extracted through adversarial prompting and exposed publicly. Mitigation: treat system prompts as sensitive configuration, not as content that's safe to expose, and test for extraction before launch.
Credential/API-Key Leakage
AI coding assistants and agents with broad filesystem or repository access can surface secrets — API keys, tokens, credentials — that were sitting in code or configuration files never meant to be shared. Mitigation: secrets scanning integrated into any workflow where an AI tool has repository or filesystem access.
Real AI Data Leak and AI Model Exposure Examples
The incidents below are documented in contemporaneous reporting or official regulatory decisions. Where facts were disputed or later clarified, that's noted.
Samsung — Source Code and Meeting Notes Entered Into ChatGPT (2023)
What happened?
In the weeks after Samsung lifted an internal restriction on generative AI tools in early 2023, Korean media reported at least three separate incidents in which Samsung engineers entered sensitive company information into ChatGPT.
What was exposed?
According to the reporting, one engineer pasted source code from a semiconductor facility's measurement database into ChatGPT to troubleshoot a fault. A second engineer submitted code related to defective equipment for optimization. A third converted an internal meeting recording into a transcript and asked ChatGPT to generate meeting minutes from it.
How did it happen?
Employees used a public, consumer-facing version of ChatGPT for legitimate work tasks — debugging code, summarizing a meeting — without an enterprise data-handling agreement in place.
Why it matters: ChatGPT's consumer interface can retain submitted data for model improvement, meaning Samsung had no straightforward way to "undo" the exposure once the data was submitted.
Enterprise lesson: Within about a month of lifting the restriction, Samsung banned employee use of ChatGPT and other generative AI tools and began developing internal alternatives. The lesson generalizes well beyond Samsung: well-intentioned, technically skilled employees will use whatever tool helps them finish a task faster, and policy alone doesn't stop that — technical controls and approved alternatives do.
Amazon — Internal Warning Over Confidential Data and ChatGPT (2023)
What happened?
Around the same period, internal Amazon Slack communications showed a company lawyer warning employees not to share Amazon confidential information — explicitly including code — with ChatGPT.
What was exposed?
This was a preventive warning rather than a confirmed leak of specific data. The lawyer's stated concern was that employee inputs could be used as training data for future versions of ChatGPT, and separately noted that some ChatGPT outputs had appeared to closely resemble existing internal material.
How did it happen?
Employees were experimenting with ChatGPT for coding assistance and other tasks, prompting internal legal concern before any confirmed incident.
Why it matters: This example is useful precisely because it's a near-miss rather than a confirmed breach — it shows how quickly organizations recognized the risk even without a headline-grabbing incident, and it illustrates that "no confirmed leak yet" isn't the same as "no risk."
Enterprise lesson: Waiting for a confirmed incident before acting on AI data risk means acting too late. Amazon's warning is a reasonable model for a first step — but a memo alone doesn't stop data from being pasted into an unapproved tool; it has to be paired with technical controls.
Italy's Garante and OpenAI — A Regulatory Data-Protection Action, Not a "Data Breach" Label
What happened?
On March 30, 2023, Italy's data protection authority (the Garante) ordered OpenAI to temporarily stop processing the personal data of individuals in Italy through ChatGPT, pending investigation. Access was restored about a month later after OpenAI made changes the Garante required.
What triggered it?
The Garante's action followed a data breach report involving ChatGPT users' chat titles and payment information being exposed, combined with separate concerns about the lack of a clear legal basis for processing personal data to train the model and the absence of age verification.
How did it happen?
The underlying breach was traced to an internal software bug rather than an external attack, and exposed chat histories and payment details for ChatGPT Plus subscribers active during a specific nine-hour window.
Why it matters — accurately stated: This wasn't simply "ChatGPT got hacked." It was a regulatory data-protection action built on multiple findings: an unreported security bug that exposed personal data, and separate, broader concerns about the legal basis for training on personal data at scale. In December 2024, Italy fined OpenAI €15 million for GDPR violations related to how it processed personal data to train ChatGPT, a decision OpenAI said it would appeal.
Enterprise lesson: Regulators are willing to treat "how we built and trained the model" as a data-protection question, not just "was there a hack." Organizations building or fine-tuning models on data that includes personal information need a defensible legal basis for that processing — this isn't only a vendor's problem.
EchoLeak — Zero-Click Data Exfiltration From Microsoft 365 Copilot (2025)
What happened?
Security researchers disclosed EchoLeak in June 2025 as a prompt-injection attack that achieved concrete data exfiltration from a production AI system — Microsoft 365 Copilot — without any user action.
What was exposed?
When Copilot ingested a maliciously crafted email through its normal retrieval process, it autonomously accessed internal files — including Word documents, PowerPoint files, and Outlook emails — and transmitted their contents to a server controlled by the attacker.
How did it happen?
The attack chained together multiple bypasses: it evaded Microsoft's cross-prompt injection classifier, got around link redaction using reference-style Markdown, and exploited auto-fetched images to exfiltrate data with no user interaction at all. Microsoft assigned it CVE-2025-32711 and shipped a server-side fix.
Why it matters: This is one of the first well-documented cases of prompt injection being weaponized for real-world data exfiltration against a mainstream enterprise AI product, not a research demo.
Enterprise lesson: Any AI system that ingests untrusted content — an email, a web page, a shared document — is a potential attack surface, even when the user never directly interacts with the malicious content.
ForcedLeak — Indirect Prompt Injection Against Salesforce Agentforce (2025)
What happened?
In September 2025, researchers disclosed ForcedLeak, an indirect prompt injection attack against Salesforce's Agentforce autonomous agents, in which agents were coerced into leaking sensitive data — including personal information, corporate secrets, and location data.
Why it matters: Unlike EchoLeak, this targeted a customer-facing, CRM-connected AI agent — showing that the risk extends well beyond productivity copilots into any AI agent connected to a system holding customer data.
Enterprise lesson: Any AI agent that processes external, untrusted input (a web form, a customer submission) and has access to internal systems needs its own threat model — treat that input as adversarial by default.
Meta — Internal AI Agent Data Exposure (March 2026)
What happened?
In mid-March 2026, Meta experienced an internal data exposure after an AI agent gave an engineer faulty guidance in response to a technical question, and implementing that guidance inadvertently exposed sensitive company and user data to employees who weren't authorized to see it. The exposure lasted roughly two hours before it was contained and was classified internally as a "Sev 1" incident.
What was confirmed vs. what remains limited detail: Meta stated that no user data was mishandled externally. The AI agent itself did not directly execute the change — it provided incorrect instructions that led the engineer to trigger the exposure — and the agent had operated with valid credentials, passing all identity checks.
Why it matters: This incident didn't involve an attacker at all — it was a well-intentioned employee following AI-generated guidance. It also exposes a specific gap: an agent authenticated correctly and still contributed to a serious exposure, which is a blind spot that identity checks alone don't close.
Enterprise lesson: AI-generated guidance for sensitive technical or access-control changes needs a review step before implementation, especially when the outcome touches broad internal data access — an agent passing identity checks doesn't mean its guidance is safe to act on.
How AI Data Leaks Happen
The incidents above map to a set of recurring paths:
- Employees entering sensitive information into public AI tools — the most common and most preventable path (Samsung, Amazon).
- Uploading confidential files to an AI tool for summarization, analysis, or troubleshooting.
- Shadow AI — unapproved tools used without IT visibility or a data-processing agreement.
- Misconfigured RAG systems that don't preserve document-level permissions.
- Excessive AI-agent permissions that let an agent read or move data beyond what a task actually requires.
- Third-party AI APIs with data-retention or training-use terms the organization didn't review.
- Poorly protected logs that retain the same sensitive prompts and outputs the organization was trying to control.
- Training or fine-tuning data that includes sensitive information later reproduced by the model.
- AI outputs that reveal information out of scope for the requesting user.
- Compromised or exploited integrations (EchoLeak, ForcedLeak).
- Model or infrastructure access falling into unauthorized hands.
- Credentials and API keys surfaced by AI coding assistants with broad repository access.
How RAG Systems Can Cause AI Data Leaks
Retrieval-augmented generation is one of the highest-value and least understood sources of enterprise AI leakage, so it's worth walking through the mechanics.
In a typical RAG pipeline: documents are ingested, converted into vector embeddings, and stored in a vector database. When a user submits a query, the system retrieves the most relevant embedded content and passes it to the model to generate an answer grounded in that content.
The problem: if the retrieval layer doesn't preserve the permissions attached to the original documents, a user can retrieve — and receive an AI-generated answer based on — information they were never authorized to see. A document restricted to a legal team, once embedded into a shared vector store without permission metadata, can end up informing an answer given to anyone in the company who asks the right question.
Reducing this risk requires:
- Document-level permissions carried through into the embedding and retrieval layer, not dropped during ingestion.
- Identity-aware retrieval, where the system checks the requesting user's authorization before surfacing retrieved content, not just before displaying the final answer.
- Metadata and access control tied to each document chunk, not just the source document as a whole.
- Audit logs of what was retrieved, for whom, and in response to what query.
- Data segmentation, keeping highly sensitive content in separate, more tightly controlled retrieval indexes rather than one shared store.
- Retrieval monitoring to catch unusual patterns — a user retrieving an unusually broad or sensitive set of documents.
How AI Agents Can Cause Data Leaks
AI agents can read files, query databases, call APIs, send messages, access other applications, execute multi-step workflows, and in some architectures communicate with other agents. That changes the risk model fundamentally: the data boundary is no longer just the prompt — an agent can move information between systems entirely on its own, as EchoLeak, ForcedLeak, and the Meta incident all illustrate in different ways.
Controls that matter most for agent-related leakage:
- Least privilege — an agent should have access only to what a specific task requires, not standing access to everything it might ever need.
- Identity-based access, so every agent action is tied to a verifiable identity and can be audited, not treated as an anonymous background process.
- Scoped tool permissions, limiting which tools and systems an agent can invoke for a given task.
- Explicit data boundaries between systems an agent can read from and systems it can write to.
- Approval gates for sensitive or irreversible actions, particularly anything touching regulated data or production systems.
- Sandboxing for agents processing untrusted input (emails, web content, customer submissions).
- Monitoring and logging of agent actions in real time, not just after-the-fact review.
- Output controls that check what an agent is about to send or expose before it does.
- Agent-to-agent restrictions, since multi-agent systems can propagate a leak or a bad decision faster than a human-in-the-loop process would catch it.
What Types of Data Are Most at Risk From AI Leaks?
Risk isn't evenly distributed — it depends on the combination of data sensitivity, where the data is going, who or what has access, how long it's retained, and how the AI output is subsequently used. Categories that show up repeatedly in real incidents:
- Source code and technical documentation
- API keys, credentials, and authentication tokens
- Customer personally identifiable information (PII)
- Financial data and forecasts
- Health information
- Contracts and legal documents
- Intellectual property and trade secrets
- Product roadmaps and unreleased plans
- M&A and strategic information
- Employee data
- Confidential internal communications
The common thread: data that's valuable to the business is also valuable to whoever might obtain it improperly, and AI systems create new, often under-monitored paths for that data to move.
How to Detect AI Data Leaks
No single tool detects every path a leak can take, but a layered detection approach covers most of them:
AI-aware DLP — data loss prevention tooling that understands AI-specific channels (prompts, uploads, API calls to AI providers), not just email and file transfers.
AI application discovery — network and endpoint visibility into which AI tools are actually in use across the organization, approved or not.
Prompt inspection — scanning outbound prompts for sensitive data before submission.
Data classification — knowing what's sensitive in the first place, so downstream tools have something to act on.
Endpoint and browser telemetry — visibility into what's being pasted or uploaded through browser-based AI tools.
API monitoring — tracking what data flows through API-based AI integrations.
RAG monitoring — auditing retrieval activity for unusual or unauthorized access patterns.
Agent activity logging — a full record of what an agent accessed, called, and returned.
User behavior analytics — flagging anomalous data-access patterns that might indicate misuse or compromise.
Shadow AI discovery — specifically surfacing unapproved AI tool usage that other controls might miss.
How to Prevent AI Data Leakage in the Enterprise
Effective prevention combines process and technical controls across the full data lifecycle:
- Maintain an AI inventory of every approved (and discovered unapproved) AI tool in use.
- Apply data classification and data minimization before data ever reaches an AI system.
- Deploy AI-aware DLP and prompt inspection at the point data would leave the organization's boundary.
- Use redaction and anonymization for sensitive fields before they're submitted to external AI tools.
- Enforce access controls and least privilege for every AI application, agent, and integration.
- Build RAG authorization and agent permission controls into the architecture, not bolted on afterward.
- Conduct vendor risk assessments before adopting any AI tool that touches sensitive data.
- Apply encryption where appropriate, in transit and at rest.
- Set retention controls so AI logs and outputs aren't kept longer than necessary.
- Maintain logging and monitoring across the AI stack.
- Run employee training specific to AI data risk, not generic security awareness.
- Have an incident-response process that specifically covers AI-related exposure.