Deepfake Social Engineering: How AI Voice and Video Fraud Is Costing Businesses Millions

---

The Deepfake Threat Is Already Here — and the Losses Are Real

AI voice cloning reached a tipping point in 2024. Tools like ElevenLabs and Resemble AI can clone a voice from as little as three seconds of audio — a snippet available from any LinkedIn video, conference talk, or voicemail. The quality is convincing enough to fool finance staff into authorising wire transfers.

The most publicised case occurred in January 2024, when the Hong Kong branch of Arup, a global engineering firm, was targeted in a sophisticated deepfake video call scam. Fraudsters used AI-generated likenesses of the company's CFO and other executives in a live video conference. The victim, a finance employee, was deceived into authorising 15 separate transfers totalling approximately HK$200 million (US$25.6 million). The employee only realised the fraud when they followed up with the corporate office afterward — the money was already gone.

Earlier cases predate this but follow the same pattern. In 2019, scammers used AI voice-cloning to impersonate the CEO of a UK energy firm's German parent company, convincing a UK director to transfer €220,000 (US$243,000) to a Hungarian supplier. In 2024, Ferrari NV's chairman was impersonated in a voice call attempting to extract funds. These are no longer isolated incidents — they represent a repeatable attack template.

Practical recommendations:

• Implement callback verification for any fund transfer request above a defined threshold, regardless of the apparent source. The callback must go to a pre-registered number, not a number provided in the request itself.
• Train finance staff specifically on deepfake awareness — not generic phishing, but the mechanics of how voice and video can be synthesised.
• Restrict the availability of executive audio and video on public platforms. Every conference talk and LinkedIn video is training data for an attacker.

AI-Powered Phishing: Why Traditional Filters Are Falling Behind

Generative AI has transformed phishing from a volume game into a precision game. Previous phishing campaigns could be filtered by spotting grammatical errors, generic greetings, and mismatched domains. AI-generated phishing emails are grammatically flawless, contextually aware, and can be personalised at scale using data from LinkedIn, annual reports, and social media.

The ASD ACSC has explicitly addressed this in its 2025 publication "Using AI to strengthen cyber defence," which outlines how threat actors are leveraging AI for reconnaissance, social engineering, and automated exploitation — while also recommending how defenders can use AI to strengthen detection and response. Their companion publication "Frontier AI models and their impact on cyber security" examines what frontier models can and cannot do in terms of offensive capability, providing a sober assessment that rejects both hype and complacency.

The cost differential is stark: traditional spear-phishing campaigns required skilled operators and significant time per target. AI reduces the per-target cost to near zero while increasing the volume of personalised lures by orders of magnitude.

Practical recommendations:

• Deploy AI-aware email security gateways that analyse behavioural anomalies, not just signature-based detection. Products like Proofpoint Threat Response and Mimecast use ML models to flag contextually suspicious patterns.
• Implement DMARC enforcement (p=reject) to prevent domain spoofing — many organisations still run in monitoring-only mode.
• Red-team your own staff with AI-generated phishing campaigns quarterly. If you're not testing, you're guessing.

Prompt Injection and AI Agent Security: The New Attack Surface

As businesses deploy AI agents that can read emails, manage documents, and execute transactions, a new class of vulnerability has emerged: prompt injection. Unlike traditional software vulnerabilities, prompt injection targets the AI model's instruction layer itself.

A prompt injection attack works by embedding malicious instructions in content the AI processes — an email body, a document, a web page the agent reads. When the AI agent ingests that content, it may follow the embedded instructions instead of its original system prompt. For example, an agent reading a "meeting agenda" attachment could be instructed to forward all contacts to an external address, and comply without the user ever seeing the instruction.

In 2023, Samsung engineers pasted proprietary source code into ChatGPT for debugging assistance, inadvertently exposing trade secrets to OpenAI's training pipeline. This wasn't a traditional breach — it was a new category of risk where employees voluntarily feed sensitive data to third-party AI services. The cost: Samsung banned ChatGPT company-wide, but the data was already exposed.

Practical recommendations:

• Treat every AI agent as an untrusted boundary. Apply the principle of least privilege: agents should never have write access to financial systems, email forwarding, or external APIs without human-in-the-loop confirmation.
• Use system prompt isolation techniques — separate user content from instructions at the architecture level. Frameworks like Llama Guard and NVIDIA NeMo Guardrails provide middleware enforcement.
• Never allow employee use of consumer AI tools (ChatGPT, Claude, Gemini) for processing confidential data without an enterprise agreement that contractually prohibits training on your data.

Model Theft and Data Exfiltration

AI models themselves are high-value targets. A fine-tuned model trained on proprietary data represents months of engineering effort and terabytes of business-critical information. Model theft can occur through API extraction (systematic querying to reconstruct model weights), insider exfiltration, or supply-chain compromise of model registries.

The risk extends beyond the model weights to the training data itself. If an attacker can extract a model's training data through membership inference attacks, they may recover sensitive customer information, proprietary algorithms, or business intelligence embedded in the training corpus.

Practical recommendations:

• Rate-limit model API access to detect extraction patterns. Unusual query volumes from a single source should trigger alerts.
• Encrypt model artefacts at rest and in transit. Use hardware security modules (HSMs) for production model serving in high-value environments.
• Monitor for model exfiltration the same way you monitor for data exfiltration — DLP tools should be configured to recognise model file formats (.pt, .safetensors, .gguf, .bin).

Governance: The Framework Businesses Need

Technical controls alone are insufficient. The NIST AI Risk Management Framework (AI RMF 1.0), published in January 2023, provides the most widely adopted governance structure. It organises AI risk into four functions: Govern, Map, Measure, and Manage. For Australian businesses, the ASD ACSC publications complement NIST with locally relevant threat intelligence and mitigation guidance.

The EU AI Act, which entered full enforcement in 2026, imposes obligations on organisations deploying high-risk AI systems, including mandatory risk assessments, logging, and human oversight. Even Australian businesses without EU operations should align with these standards — they represent emerging best practice and will likely influence future Australian regulation.

Governance isn't a checkbox exercise. It requires a designated AI security owner, documented risk assessments for every AI deployment, and regular review cycles. For SMBs, this doesn't require a dedicated team — it requires someone accountable, a documented process, and quarterly review.

FAQ

Conclusion

AI has fundamentally changed the threat landscape — not in a theoretical future sense, but right now, with documented losses exceeding US$25 million from a single deepfake call. The defensive playbook requires three things: process controls that don't trust any single communication channel, technical controls that treat AI systems themselves as attack surfaces, and governance that assigns clear accountability. Start with a callback protocol for fund transfers today. It costs nothing and would have stopped every case mentioned in this article.

Visit consult.lil.business for a free cybersecurity assessment tailored to your organisation's AI risk exposure.

References

1. ASD ACSC — Using AI to strengthen cyber defence
2. NIST AI Risk Management Framework (AI RMF 1.0)
3. ASD ACSC — Frontier AI models and their impact on cyber security
4. CISA — Joint Guidance on Mitigating Deepfake Threats

Verifier warning: verifier could not run (PluginLlmTrustError).