The $25 Million Video Call: Deepfake Social Engineering Has Arrived

In February 2024, a finance employee at the multinational engineering firm Arup's Hong Kong office joined what appeared to be a routine video conference with the company's CFO and other senior colleagues. The faces and voices were convincing. The instructions were clear: authorize a series of wire transfers. The employee complied, transferring approximately HK$200 million (US$25.6 million) across 15 transactions before anyone realized every person on that call—except the victim—was a deepfake generated from publicly available corporate videos and audio recordings.

This was not an isolated event. In 2019, the CEO of a UK energy company received a phone call from what he believed was his parent company's German chief executive, instructing him to transfer €220,000 (US$243,000) to a Hungarian supplier. The voice was a synthetic clone. The CEO later described the deepfake voice as remarkably authentic—it matched the accent, the tone, and the rhythm of the real person perfectly.

The cost of entry for attackers has collapsed. Open-source voice cloning tools like XTTS and Coqui TTS can replicate someone's voice from as little as three seconds of audio scraped from a LinkedIn video, a podcast, or a conference recording. Video deepfake generation that once required Hollywood-grade budgets can now be produced on a consumer GPU in minutes. The question for business leaders is no longer whether your organization will be targeted—it is when, and whether your controls will hold.

Practical recommendation: Implement mandatory out-of-band verification for any financial transaction or credential change above a defined threshold. The instruction should come through a video call? Call the requester back on a number from your internal directory—not caller ID. This single control would have stopped both incidents above.

Beyond Voice Cloning: The Full Spectrum of AI-Enabled Fraud

Deepfake voice fraud dominates headlines, but the threat surface extends further:

Deepfake recruitment operations. In June 2024, the FBI warned that criminals were using deepfake video and stolen identities to apply for remote IT and programming positions. The goal: gain access to company networks, source code, and customer data through legitimate employment channels. The deepfakes were convincing enough to pass multiple rounds of video interviews, with candidates lip-syncing to AI-generated responses while someone else fed answers in real time.

Deepfake extortion. Attackers generate compromising synthetic images or video of executives, employees, or their family members, then threaten to release the material unless payment is made. The material never existed—it is entirely fabricated—but the social and reputational pressure is real, and victims have paid.

AI-enhanced phishing at scale. Large language models enable attackers to generate thousands of highly personalized, grammatically perfect phishing emails tailored to specific employees, scraped from their LinkedIn profiles, recent posts, and company announcements. The telltale signs of phishing—awkward phrasing, generic greetings, spelling errors—are gone. A 2024 IBM Security study found that AI-assisted phishing campaigns achieved significantly higher click-through rates than human-authored ones, while requiring a fraction of the time to produce.

Practical recommendation: Layer detection tools alongside human awareness training. Solutions from providers like Pindrop (voice authentication and deepfake detection for call centres), Reality Defender (real-time deepfake detection for video and audio), and Hive Moderation (AI-generated content classification) can flag synthetic media before it reaches decision-makers. Budget approximately AUD $15,000–$50,000 annually for mid-market deployments depending on integration scope.

Prompt Injection and AI Agent Security: The Threat Inside Your Tools

As businesses deploy AI assistants, autonomous agents, and LLM-powered customer service chatbots, a new attack vector has emerged that most organizations have never heard of: prompt injection.

Prompt injection is the AI equivalent of SQL injection. An attacker embeds hidden instructions in content that an AI system processes—a webpage, a PDF, an email body, or even an image caption. When the AI ingests that content, it follows the attacker's instructions instead of the user's. In a business context, this could mean: an AI assistant with access to internal documents leaking confidential data after reading a malicious email, an autonomous procurement agent ordering equipment from an attacker-controlled supplier after processing a poisoned catalogue entry, or a customer service chatbot being manipulated into offering unauthorized refunds or exposing account details.

The OWASP Top 10 for Large Language Model Applications ranks prompt injection as the number one security risk for LLM systems—above data poisoning, above supply chain vulnerabilities, above model theft. Yet most organizations deploying AI tools have zero controls in place for it.

Practical recommendation: Treat any AI system that processes untrusted input (emails, web content, customer messages, uploaded files) as a high-risk integration. Implement strict permission boundaries so that even if an AI agent is compromised via prompt injection, it cannot execute financial transactions, access systems beyond its task scope, or exfiltrate data. Never grant AI agents standing administrative privileges. Segment their access as aggressively as you would a contractor's.

Model Theft and Intellectual Property Risk

When you feed proprietary data, customer records, or trade secrets into a third-party AI platform, where does that data go? Model theft and data leakage represent an underappreciated category of AI risk. Attackers can extract training data from models through carefully crafted queries—a technique called model inversion. They can clone proprietary models by observing their outputs. And in some cases, the AI provider itself may retain and reuse your inputs.

Samsung learned this the hard way in 2023 when engineers pasted proprietary source code and internal meeting notes into ChatGPT, inadvertently exposing confidential intellectual property to OpenAI's systems. The company subsequently banned the use of consumer generative AI tools across all divisions.

Practical recommendation: Establish a data classification policy for AI tools before deployment. Proprietary source code, customer PII, financial data, and strategic plans should never be processed through consumer-grade AI services. Negotiate enterprise agreements with zero data retention clauses, or deploy models on infrastructure you control.

The Governance Framework: What Business Leaders Actually Need

Technology controls alone will not prevent AI-enabled attacks. Organizations need governance frameworks that address AI risk at the board level. The NIST AI Risk Management Framework (AI RMF) provides a practical foundation organized around four functions: Govern, Map, Measure, and Manage. The Australian Cyber Security Centre (ACSC) has also published guidance on the secure deployment of AI systems specifically for Australian organizations.

A minimum viable AI governance programme should include:

1. An AI usage policy defining what tools are approved, what data may be processed, and who is accountable.
2. Out-of-band verification protocols for financial transactions, credential changes, and executive communications—designed specifically to counter deepfake social engineering.
3. Incident response playbooks that account for deepfake extortion, AI-assisted phishing, and prompt injection—not just traditional malware and ransomware.
4. Employee awareness training updated to include AI-specific threats: voice cloning, synthetic video, and the reality that "I saw them on a video call" is no longer proof of identity.
5. Vendor risk assessments for any AI tool that processes company data, evaluating data retention, model training practices, and breach notification terms.

The cost of implementing these controls is modest compared to the alternative. A single deepfake CEO fraud incident can exceed AUD $500,000 in losses; the 2024 IBM Cost of a Data Breach Report pegged the average breach at AUD $5.9 million globally.

FAQ

Conclusion

AI has fundamentally changed the economics of social engineering. What once required a skilled human operator crafting bespoke attacks can now be automated, personalized, and scaled with tools that are free or nearly free. The $25 million Arup incident is not a worst-case scenario—it is an early signal of what is coming as the technology improves and proliferates.

The organizations that weather this shift will be the ones that treat AI-enabled fraud as an operational risk requiring specific controls, not a hypothetical concern for the IT team to monitor. Start with out-of-band verification for financial transactions. Add deepfake awareness to your security training. Establish an AI usage policy before your employees adopt tools on their own. And build prompt injection testing into your security assessments.

Visit consult.lil.business for a free cybersecurity assessment. We will evaluate your organization's exposure to AI-enabled social engineering, review your AI tool usage and governance posture, and provide a prioritized roadmap of controls—backed by real-world threat intelligence and tailored to your industry.

References

1. NIST AI Risk Management Framework (AI RMF 1.0) — The US National Institute of Standards and Technology framework for managing AI-related risks across the Govern, Map, Measure, and Manage functions.
2. OWASP Top 10 for Large Language Model Applications — The Open Worldwide Application Security Project's ranked list of critical security risks for LLM-powered systems, with prompt injection ranked as the number one threat.
3. IBM Cost of a Data Breach Report — Annual benchmark study quantifying the financial impact of data breaches across industries, including the rising cost of AI-assisted attacks.
4. Sumsub Fraud Report 2024 — Industry analysis of fraud trends including the surge in deepfake-enabled identity fraud and synthetic media abuse across regulated industries.
5. Mandiant: Deepfake Disinformation Research — Threat intelligence analysis of state-sponsored deepfake campaigns and the operational techniques used to create and distribute synthetic media at scale.

Verifier warning: verifier could not run (PluginLlmTrustError).