How AI Changed the Attacker's Economics

The shift from 2023 to 2026 is stark. ENISA's 2025 Threat Landscape report, analysing 4,875 incidents, found that AI-supported phishing now represents more than 80% of all observed social engineering activity worldwide. A Harvard study presented at a top-tier cybersecurity conference found AI-generated phishing achieves 50–57% higher success rates than traditional campaigns. Three independent studies — academic, controlled lab, and a real-world 9,000-person deployment — all converge on the same conclusion: AI phishing is at least as effective as expert human phishing and dramatically more accessible to novice attackers.

What changed? The economics. Traditional phishing was a labour problem — you needed skilled social engineers writing convincing emails. Generative AI turned it into an engineering problem. A single attacker with a $20/month LLM subscription can now generate thousands of hyper-personalised, grammatically flawless phishing emails in minutes, each referencing real projects, real colleagues, and real communication patterns scraped from LinkedIn and company websites. Malicious code payloads now mutate in real time to evade antivirus. Attack campaigns adapt dynamically to bypass security filters.

IBM's 2025 Cost of a Data Breach Report pegs the average financial loss from AI-powered social engineering at $4.4 million per incident. Organisations using security AI save an average of $1.9 million and detect or contain incidents roughly 100 days faster.

Deepfake Social Engineering: When Seeing Is No Longer Believing

The most alarming development for 2026 is real-time deepfake fraud. The barrier to entry collapsed — less than ten seconds of audio is now sufficient to clone a voice convincingly. Face-swapping technology operates in real time during live video calls.

Real-world cases that should terrify every CFO:

• $25 million video call fraud (Hong Kong, 2024): Attackers used real-time deepfake face-swapping during a multi-participant video conference, impersonating the CFO and other executives to authorise a wire transfer. The victim was the only real person on the call.
• $499,000 deepfake Zoom heist: An employee joined what appeared to be a routine Zoom meeting with colleagues and authorised a transfer — every face on screen was synthetic.
• $220,000 voice clone (UK energy firm): An AI-cloned voice of the CEO called a subordinate and demanded an urgent payment to a supplier. The employee recognised the voice, the accent, and the speech patterns.

Doppel's 2026 Social Engineering Predictions Report reveals that 45% of all social engineering campaigns are now multi-channel — spanning email, SMS, social media, ads, and even physical mail simultaneously. Entrust reports deepfake attempts occurring every five minutes. Sumsub tracked a 180% surge in advanced deepfake fraud. The FBI recorded $262 million in account takeover losses in early 2025 alone.

The attack pattern is consistent: infiltration (gather voice samples, build trust) → setup (create the synthetic identity across channels) → switch (flip from benign interaction to urgent financial request) → extraction (move funds before detection).

What Your Business Should Do Today

• Mandate out-of-band verification for any financial instruction over $5,000 — a known phone number, a code word, or a physical confirmation
• Treat all remote video calls requesting payments or sensitive data as potentially synthetic until independently verified
• Train finance teams specifically on deepfake scenarios, not generic phishing awareness
• Deploy AI-assisted threat detection that can identify synthetic media patterns in real-time communications

Prompt Injection: The Hidden Crisis in Your AI Agents

If your business deploys AI agents that read email, access databases, execute code, or trigger financial transactions, you have a prompt injection exposure — whether you know it or not. Prompt injection surged 340% in 2026 and is now ranked the #1 risk to AI systems by the OWASP Top 10 for LLM Applications.

The attack is devastatingly simple. An attacker crafts input — an email, a shared document, a web page — containing hidden instructions that override the AI agent's system prompt. When the agent processes that content, it executes the malicious instruction instead of its intended function.

The "Lethal Trifecta" that makes agent compromise catastrophic: the agent has access to private data (emails, documents, databases), the agent processes untrusted external inputs (incoming emails, shared files, web content), and the agent can take autonomous action (send emails, access APIs, execute transactions).

Microsoft Copilot exposure demonstrates the risk. An attacker sends a single poisoned email containing invisible prompt-injection text. When any employee's Copilot searches the mailbox, the agent retrieves the poisoned email, executes the embedded instructions, and exfiltrates sensitive data via an image URL — all without a single click. The same attack chain works across Gmail, Google Calendar, and Google Docs environments using agentic integrations.

In a recent fintech security assessment, penetration testers extracted an AI agent's full system prompt in 20 minutes using a basic role-swap injection attack.

Defending AI Agents

Defence requires multiple layers:

• Input sanitisation: Strip or neutralise prompt-like syntax from all external content before it reaches the LLM
• Least-privilege tool access: AI agents should never have credentials or access scopes broader than absolutely required for their function
• Output filtering: Validate and sanitise all agent outputs before they trigger downstream actions
• Human-in-the-loop gates: Financial transactions, data exports, and configuration changes must require human approval — even when initiated by an AI agent
• Canary tokens and tripwires: Plant detectable markers in system prompts that would be disclosed if prompt extraction succeeds, providing an early warning

Model Theft: When Your AI Is the Target

Beyond attacking through AI, attackers are increasingly targeting the AI models themselves. Model theft — extracting a proprietary model through API querying, weight exfiltration, or side-channel attacks — represents a new frontier of IP loss that most governance frameworks have not yet addressed.

Attack vectors include:

• Model extraction via systematic API queries to reconstruct a functionally equivalent model
• Weight theft through compromised model registries (Hugging Face, private repositories)
• Fine-tuning data exposure through training data extraction attacks
• Supply chain compromise via poisoned or backdoored model dependencies

For businesses that have invested heavily in fine-tuning or custom model development, the loss of a proprietary model can represent millions in R&D value gone in minutes. Defences include rate limiting API access, differential privacy during training, watermarking model outputs, and treating model weights with the same access controls as source code.

The Governance Gap and What to Do About It

Most Australian businesses have not updated their cybersecurity governance to account for AI-specific threats. Traditional frameworks — Essential Eight, ISO 27001 — do not directly address prompt injection, deepfake verification protocols, or model supply chain risk.

The NIST AI Risk Management Framework (AI RMF 1.0) provides the most mature starting point, structured around four functions: Govern (establish AI risk culture), Map (understand AI context and impacts), Measure (assess AI risks quantitatively), and Manage (treat AI risks through controls). ISO 42001 offers a certifiable AI management system standard.

Concrete governance steps for mid-market businesses:

1. Add AI-specific threats to your risk register and business continuity plan
2. Require AI supply chain due diligence for any vendor providing AI-powered services
3. Update acceptable use policies to address employee use of public LLMs with company data
4. Implement AI-specific incident response playbooks — deepfake payment fraud requires different triage than ransomware
5. Conduct tabletop exercises simulating AI-powered attacks at least quarterly

FAQ

Conclusion

AI has not just accelerated existing threats — it has created fundamentally new attack surfaces that bypass traditional controls. The three urgent priorities for business leaders are: implement verification protocols that assume synthetic media is the default, add AI agent security and prompt injection defences to your security program, and update governance to treat your AI supply chain — and your own models — as critical assets.

The businesses that fare best in this new threat landscape are not those with the biggest security budgets, but those that adapt their processes fastest to acknowledge that seeing and hearing are no longer grounds for trust.

Visit consult.lil.business for a free cybersecurity assessment tailored to your organisation's AI risk profile.

References

1. ENISA Threat Landscape 2025
2. OWASP Top 10 for LLM Applications
3. NIST AI Risk Management Framework 1.0
4. IBM Cost of a Data Breach Report 2025
5. Doppel 2026 Social Engineering Predictions Report