Top 5 AI System Breaches for 2025 to date & Lessons learnt
In this article, we will step through five real security attacks on production AI systems in 2025 that collectively resulted in significant damages and business impact. We'll see how these AI systems were compromised, exploring the attack vectors used, the vulnerabilities exploited and the potential security guardrails that could have prevented these incidents.
Every day, millions of people are choosing to trust the rapidly evolving AI systems with their most sensitive information, sometimes by choice other times through indirect interaction with businesses adopting AI systems for competitive edge. Examples would include, use of voice authentication to access bank accounts, share sensitive information with with AI-powered customer service or HR assistants
This trust, built on the promise of AI's emerging capabilities, is being systematically exploited by attackers who understand something many organisations are still learning: AI systems require fundamentally different security approaches than traditional software.
The first half of 2025 delivered a harsh reality check for this maturing technology. Real AI systems in production suffered breaches resulting in monetary losses and exposure of personal data belonging to tens of millions of users. These weren't sophisticated state sponsored attacks requiring advanced persistent threats. Many succeeded through surprisingly simple techniques that exploited the evolving security landscape around AI deployments.
What makes these incidents particularly valuable as learning opportunities is how preventable they were. As AI technology rapidly advances, the cybersecurity community has been racing to document AI specific vulnerabilities, most notably through initiatives like the OWASP Top 10 for LLM Applications (referred as OWASP LLM Top-10 from here):
This collaborative effort brings together security researchers, AI practitioners, and industry experts to identify the most critical risks facing these emerging AI systems. The OWASP Top-10 serves as an evolving roadmap for organisations navigating the security challenges of this transformative technology. Yet as these five incidents demonstrate, the learning curve is steep.
Unfortunately, organisations continue to deploy AI systems with the same security assumptions they use for traditional applications, creating vulnerabilities that attackers are increasingly ready to exploit.
Let's dive into these breaches in detail:
1. Deepfake Voice Banking Scam - Your voice isn't yours anymore
Attack Vector: AI-Generated Deepfake Voice Impersonation
In March 2025, multiple banks across Hong Kong began detecting unauthorised transactions totaling approximately $25 million. Bank records showed customers had called to authorise transfers, and voice authentication systems had verified the callers' identities. However, the actual account holders reported they had never made these calls or authorised any transfers.
How the Attack Was Carried Out:
Scammers collected voice samples from public sources and used AI tools to generate highly realistic voice clones. These deepfakes were then employed to bypass voice recognition systems used by banks for authentication, successfully impersonating legitimate customers to authorise fraudulent transactions.
Business Impact:
$25 million in unauthorized transactions across multiple financial institutions. The incident demonstrated the fundamental weakness of single-factor voice authentication and highlighted the inadequacy of voice authentication against sophisticated AI-generated impersonations. Luckily, the authorities stepped in just-in-time and banks were able to recover the funds in this instance.
Weaknesses Exploited:
- LLM09: Overreliance on AI-Generated Content
- LLM07: Insecure Plugin Design
- LLM03: Training Data Poisoning
Guardrails / Mitigation:
- Implement multi-factor authentication combining voice with other verification methods.
- Deploy deepfake detection algorithms in authentication systems.
- Limit the use of voice authentication for high-risk transactions.
- Regularly update and train staff on emerging AI threats.
Source Reference:
- Singapore and Hong kong police force_recover over $670000 in a scam
- OWASP Gen AI Incident & Exploit Round-up, Q2’25
2. McDonald’s AI Hiring Bot Breach - The Job Application That Never Forgot
Attack Vector: Credential guessing on AI system
On June 30, 2025, security researchers discovered they could access McDonald's AI hiring chatbot "Olivia" administrative panel. After gaining access, they found they could retrieve personal records of up to 64 million job applicants dating back years. McDonald's and vendor Paradox.ai were notified of the breach and responded swiftly to secure the system.
How the Attack Was Carried Out:
Researchers found an unprotected staff login endpoint on McHire.com and used credential guessing to gain access. By trying the simple password "123456", they successfully accessed the admin console, enumerated applicant IDs, and retrieved millions of user records within 30 minutes of discovering the vulnerability.
Business Impact:
64 million personal records exposed including names, emails, and phone numbers across decades. The incident prompted Paradox.ai to launch an emergency bug bounty programme, whilst McDonald's faced regulatory scrutiny and significant damage to trust and compliance metrics.
Weaknesses Exploited:
- LLM06: Sensitive Data Exposure - trivial password allowed full access
- LLM07: Insecure Plugin Design – vendor system lacked standard security controls
Guardrails / Mitigation:
- Enforce strong credential policies, multi-factor authentication, and automated log monitoring.
- Conduct vendor risk assessments and contractually mandated security audits.
- Train developers on secure AI deployment and credential hygiene.
- Audit authentication on all AI‑powered endpoints and conduct third‑party vendor reviews.
Source Reference:
3. Microsoft 365 Copilot Zero-Click Data Leak - When Your Email Assistant Turned Spy
Attack Vector: Indirect prompt injection (scope violation)
In June 2025, Aim Security researchers discovered and reported a zero-click vulnerability in Microsoft 365 Copilot. The vulnerability could enable stealth exfiltration of sensitive corporate information across enterprise organisations worldwide. Microsoft acknowledged the issue and released a patch during their June Patch Tuesday update cycle.
How the Attack Was Carried Out:
Attackers embedded malicious prompts within email content. When Copilot automatically processed these emails as part of its normal operations, it executed the hidden instructions and leaked internal data without any user interaction or awareness—a technique described as "scope violation" that bypassed traditional security filters.
Business Impact:
Potential for silent data theft across enterprise environments worldwide. Microsoft patched the vulnerability in their June 2025 Patch Tuesday update, but the incident highlighted how AI assistants could become insider threats and the risks of AI systems with excessive automated permissions.
Weaknesses Exploited:
- LLM02: Indirect Prompt Injection – commands hidden in benign-looking content
- LLM09: Overreliance on AI Content Approvals – automated actions without verification
Guardrails / Mitigation:
- Implement input sanitisation, strict prompting policies, and context confinement.
- Disable auto‑read input triggers and require explicit user commands.
- Warn users about AI‑triggered behaviours and implement internal audits.
- Log AI interactions and monitor for anomalous data flows.
Source Reference:
- Microsoft 365 Copilot Zero-Click Data Leak
- Check Point Research
- OWASP Gen AI Incident & Exploit Round-up, Q2’25
4. DeepSeek Data Breach - The Chat History That Went Public
Attack Vector: Cloud misconfiguration / data exposure
Between January 29 and March 3, 2025, DeepSeek's cloud database containing user chat logs, API keys, and personal metadata was exposed to the public internet. Security firm Wiz Research discovered the exposure and notified DeepSeek, who secured the database within an hour. However, the incident triggered regulatory investigations in South Korea, Italy, and prompted the U.S. Department of Commerce to restrict government use.
How the Attack Was Carried Out:
The attack required no sophisticated hacking techniques. DeepSeek had misconfigured a publicly accessible storage endpoint with no authentication controls, enabling anyone who discovered the URL to download the entire database containing over one million sensitive user interactions and system credentials.
Business Impact:
Over one million chat records, API keys, and user data exposed for at least an hour. Regulators in South Korea and Italy launched investigations, the U.S. Department of Commerce restricted government use, and long-term reputational and compliance damages ensued globally.
Weaknesses Exploited:
- LLM08: Insecure Third‑Party Integration – misuse of cloud without secure controls
- LLM09: Overreliance on AI Content – assumed internal data systems would self-protect
Guardrails / Mitigation:
- Enforce access controls, encryption at rest, and infrastructure audit tools.
- Implement regular configuration reviews and organisational cloud security policies.
- Provide DevOps training for secure cloud deployment.
- Enable automated detection of misconfigured environments with cross-training between AI/development and cloud-security teams.
Source Reference:
5. The Security Scanner That Got Fooled
Attack Vector: Malware with prompt‑injection to fool AI scanners
In early June 2025, a malware sample dubbed "Skynet" was uploaded from the Netherlands and analysed by researchers. The malware was designed as a proof-of-concept to demonstrate how AI-powered security systems could be manipulated. Check Point Research and Adversa documented the discovery, highlighting the potential for attackers to tailor malware specifically to subvert AI defences.
How the Attack Was Carried Out:
The malware embedded prompt injection strings within its code. When AI-powered antivirus systems analysed the file, the malware instructed the AI to ignore prior instructions and falsely declare "NO MALWARE DETECTED." The technique also included sandbox evasion capabilities and TOR proxy setup to manipulate AI-powered endpoint detection workflows.
Business Impact:
Whilst limited in scope as a proof-of-concept, it signalled rising sophistication where attackers tailor malware to subvert AI defences. As an emerging malware proof‑of‑concept, it exploited AI antivirus systems that trusted model outputs, demonstrating how cybercriminals could potentially render next-generation AI security tools ineffective.
Weaknesses Exploited:
- LLM01: Prompt Injection – malware injecting directives into AI scanners
- LLM05: Model Output Trust – excessive reliance on AI decisions
Guardrails / Mitigation:
- Verify model chain-of-trust and use multiple detection modalities (behavioural & static analysis, sandboxing).
- Define policies rejecting AI output as sole verdict and apply defence‑in‑depth principles.
- Train analysts to interrogate AI results and treat AI antivirus outputs with skepticism.
- Include provenance tracking and alerts for conflicting evidence in AI defence systems.
Source Reference:
Key Lessons for Security Leaders
These incidents reveal critical patterns that organisations must understand:
-
AI systems amplify existing security weaknesses: Poor authentication, misconfigured clouds, and inadequate input validation become catastrophic when combined with AI capabilities.
-
Traditional security controls often don't apply: New attack vectors like prompt injection and AI-generated content require specialised defences that standard frameworks don't address.
-
Trust boundaries are fundamentally different: AI systems treat data as both input and instruction, creating novel attack surfaces that require rethinking validation and sanitisation approaches.
-
Human oversight remains essential: Fully automated AI systems without human verification create significant security risks, as demonstrated by the zero-click Copilot attack.
Build AI Systems that are Secure & Resilient
The above lessons demand a fundamental shift in how we think about security. To minimise the impact associated with this rapidly evolving yet promising technology landscape, organisations must:
-
Embed Security by Design: Integrate the OWASP Top 10 for LLM Applications into project planning while incorporating defenses like prompt injection and data poisoning protections into the system architecture from day one.
-
Enforce Security Gates in Development: Make AI security reviews mandatory in every sprint and release. No code moves forward without passing vulnerability checks.
-
Bake Governance into Data Pipelines Hardwire approval workflows and source validation into training and operational systems to ensure continuous, automated oversight.
-
Make Monitoring and Observability a core feature Design real time anomaly detection and behavioral monitoring into the architecture—essential capabilities, not afterthoughts.
-
Build Integrated Security Teams Place security experts inside AI teams from the start so every architectural choice is shaped by security, not patched later.
Conclusion
The question isn't whether your organisation will adopt AI. It's whether you'll implement proper security guardrails before or after your first incident.
In this rapidly evolving landscape, assume your AI systems can be compromised and design security from the ground up. The cost of prevention is always less than the cost of a breach.
The OWASP Gen AI Incident & Exploit Round-up Q2 2025 report documents 14 significant incidents that occurred in the first half of 2025, with additional cases emerging continuously. As AI adoption accelerates, so do the associated risks. Organisations must move beyond securing AI Systems with traditional security tools and recognise it as a fundamentally different technology requiring specialised security approaches that need to be baked into the code, the infrastructure and the runtime applications.
Need help securing your AI systems?