CTF: Your IT Provider Got Hacked — And So Did You"

Difficulty: Hard | Time: 25–35 min | Linked product: IRP Template ($47)​‌‌‌​​‌‌‍​‌‌‌​‌​‌‍​‌‌‌​​​​‍​‌‌‌​​​​‍​‌‌​‌‌​​‍​‌‌‌‌​​‌‍​​‌​‌‌​‌‍​‌‌​​​‌‌‍​‌‌​‌​​​‍​‌‌​​​​‌‍​‌‌​‌​​‌‍​‌‌​‌‌‌​‍​​‌​‌‌​‌‍​‌‌​​​‌‌‍​‌‌‌​‌​​‍​‌‌​​‌‌​


The Setup

It's Friday at 4:45 PM. Your managed service provider (MSP) — the company that manages your servers, backups, and workstations for your 30-person logistics company in Townsville — calls you.

"We need to let you know we've had a security incident. We're still investigating, but our RMM platform may have been used to access some client environments between Tuesday and Friday. We're asking all clients to check their systems for unusual activity."​‌‌‌​​‌‌‍​‌‌‌​‌​‌‍​‌‌‌​​​​‍​‌‌‌​​​​‍​‌‌​‌‌​​‍​‌‌‌‌​​‌‍​​‌​‌‌​‌‍​‌‌​​​‌‌‍​‌‌​‌​​​‍​‌‌​​​​‌‍​‌‌​‌​​‌‍​‌‌​‌‌‌​‍​​‌​‌‌​‌‍​‌‌​​​‌‌‍​‌‌‌​‌​​‍​‌‌​​‌‌​

They hang up quickly — they've clearly got 50 of these calls to make.

You open Task Manager on your main server. Two processes you don't recognise: svchost_update.exe and nvmupdate.exe. Both running as SYSTEM. Both have outbound TCP connections on port 443 to IPs in Eastern Europe. Your backup software shows the last successful backup was Monday. Today is Friday.

This is a supply chain / MSP compromise. Your trusted IT provider's remote management tool became the attacker's entry point. You effectively had someone with admin access to your environment for three days.

What do you do when the people you trusted with your systems are the reason you've been compromised?


The Challenge


Question 1 — Severing the MSP connection

Your first instinct is correct: the MSP's RMM agent is still installed on your systems and potentially still under attacker control. But your MSP also manages your backups, your server patching, and your firewall. Cutting them off has operational consequences.

  • What's the correct sequence for severing the RMM connection without losing your operational backup and monitoring capabilities?
  • How do you determine which of your systems have the RMM agent installed?
  • Who in your organisation has the authority to make this call? (Your MSP contract likely has clauses about unilateral changes. Does that matter in a crisis?)

Question 2 — Scoping the damage

The attacke

r had RMM-level access (essentially admin) to your environment for approximately 72 hours. What's the scope of what they could have done?

List five categories of malicious activity that an attacker with admin-level RMM access could have performed during a 72-hour dwell time. For each, describe the forensic indicator that would confirm whether it happened.


Question 3 — The backup problem, revisited

Your last good backup was Monday. It's Friday. You have a four-day gap. Your business runs on a logistics management system — four days of shipment records, driver logs, and client invoicing data.

  • If you were ransomwared right now (entirely possible given what's on your systems), what's the recovery time and recovery point for your critical logistics system?
  • What's the difference between an RTO and an RPO, and what are yours right now?
  • Is there any source of partial data recovery for the four-day gap that doesn't rely on your backup system?

Question 4 — Your MSP's liability

Your MSP's RMM platform was compromised through a vulnerability in their tool. This wasn't your misconfiguration — it was theirs. You want to understand your options.

  • What does your MSP contract likely say about liability for security incidents caused by the MSP's own systems?
  • In Australia, what legal avenues exist for recovering damages from an MSP that negligently exposed your environment?
  • What should you do right now in terms of evidence preservation related to the MSP relationship?

Question 5 — Third-party risk in future: What should your contract have said?

If your MSP contract had been written properly from a security perspective, what three clauses would now be giving you much stronger legal and operational protection?


Hints

Hint 1 (Q1): The RMM agent runs as a service. You can disable the service without uninstalling it — this severs the attacker's access while preserving the installed software for forensic examination. services.msc or sc stop [service name] followed by sc config [service name] start= disabled. Identify the MSP's specific RMM tool (Kaseya, ConnectWise Automate, NinjaRMM, etc.) — each has a documented service name. Your MSP contract almost certainly does not give you a right to damages for their security failures — these contracts are almost universally written with broad liability caps and indemnity protections for the MSP.

Hint 2 (Q2): Five categories: credential harvesting (dump LSASS, pull SAM/NTDS.dit), persistence installation (scheduled tasks, registry run keys, new service creation), data exfiltration (bulk file copy to external destination), lateral movement staging (enumeration of other hosts in the environment), and ransomware pre-staging (disabling shadow copies, deleting backups, mapping drives). Each has specific forensic artefacts: Windows Event IDs, scheduled task logs, network connection logs, VSS deletion events.

Hint 3 (Q3): RTO (Recovery Time Objective) is how long you can tolerate being down. RPO (Recovery Point Objective) is how much data loss you can accept. Right now, your RPO is four days and your RTO is unknown (you haven't tested restoration). Partial data recovery sources for the four-day gap: transaction logs in your logistics software's database (if not encrypted), email records of invoices and dispatch notifications, mobile devices used by drivers (GPS logs, app-level records), and client portals if your logistics system has them.

Hint 4 (Q4): Australian MSP contracts typically cap liability at the value of the annual contract. If your MSP charges you $3,000/year, their maximum liability is often $3,000 regardless of the damage. Claims in negligence (Australian Consumer Law, tort of negligence) can go beyond contract caps, but are harder to establish. Evidence preservation: keep all communications with the MSP, get their incident report in writing, do not sign any "we're sorry" document they send that includes liability waivers.

Hint 5 (Q5): The three clauses: (1) Minimum security standard for the MSP's own infrastructure (requiring MFA, vulnerability management, incident response procedures); (2) Right to audit or receive security certification from the MSP annually; (3) Liability uncap for incidents caused by the MSP's own security failures. Almost no MSP will sign the third clause — but asking for it tells you something about their security posture.


Reveal: Full Answer to Question 2

Five categories of malicious activity with forensic indicators:

1. Credential harvesting

What happened: Attacker dumps credentials from memory or the registry to obtain all usernames and passwords on the compromised host — enabling further access even after the RMM is blocked.

Forensic indicator: Windows Security Event ID 4624 (successful logon) with unexpected logon types (Type 3 = network, Type 10 = remote interactive) from unusual source IPs. Also check for processes accessing LSASS memory: Security Event ID 10 in Sysmon logs (lsass.exe as target process). On domain-joined systems, check for NTDS.dit access: Event ID 4799, or the presence of Volume Shadow Copy deletion events.

2. Persistence installation

What happened: Attacker installs a backdoor that survives the RMM being disconnected — a scheduled task, a new Windows service, or a registry Run key.

Forensic indicator: Check HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Run and the equivalent CurrentVersion\RunOnce keys for entries created in the Tuesday–Friday window. Check C:\Windows\System32\Tasks\ for scheduled tasks with recent creation timestamps. Use schtasks /query /fo LIST /v and sort by "Last Run Time." Check Windows Event ID 7045 (new service installed).

3. Data exfiltration

What happened: Attacker copies files to an external destination — either via the RMM channel itself or via a second outbound connection.

Forensic indicator: Check Windows Event ID 5145 (network share access) and 4663 (file access auditing, if enabled). Check your firewall logs for unusual data volumes outbound from your server IPs during Tuesday–Friday — a large file exfiltration shows up as a sustained high-bandwidth outbound connection. The nvmupdate.exe process already flagged with an Eastern European IP is your primary indicator.

4. Shadow copy and backup deletion

What happened: Pre-ransomware staging step — attacker deletes Volume Shadow Copies and backup jobs to eliminate your recovery options.

Forensic indicator: Check vssadmin list shadows — if no shadows exist and your server has been running for more than a day, they may have been deleted. Windows Event ID 7036 (service state change) for backup services. Check whether your backup software's scheduled jobs are still active. The fact that your last backup was Monday (four days ago) while backups were presumably scheduled nightly is itself a strong indicator this step may have occurred.

5. Lateral movement staging

What happened: Attacker uses admin access on one system to enumerate and prepare access to other systems in your network — building a map of your environment.

Forensic indicator: Windows Event ID 4648 (logon with explicit credentials — i.e., using a specific username/password rather than current session). Network logon events (Type 3) from the compromised server to other internal hosts. Check the %TEMP% and %APPDATA% folders on the compromised host for network scanning output files — many attackers use tools like net view, nmap, or AD enumeration tools whose output gets saved to temp.


Get the Full Answer Key

You've seen the damage scoping answer in full. The remaining questions — on severing the RMM connection safely, RTO/RPO assessment, MSP liability under Australian law, and the contract clauses you need — are covered in the Incident Response Plan Template for SMBs.

The template includes:

  • MSP/third-party IR playbook with RMM disconnection steps
  • Windows forensic indicator checklist (Event IDs, artefact locations)
  • RTO/RPO worksheet
  • MSP contract security clause reference (what to ask for)
  • Third-party breach notification template

Get the IRP Template for $47 → lil.business/products/incident-response-plan-template

Or buy via Polar: https://buy.polar.sh/polar_cl_G95ZMX6xnZpa7JuXj1AROgffKr1aL0JDmJ2KU1rHJ84


The MSP RMM compromise scenario reflects documented attack patterns including the 2021 Kaseya VSA incident and subsequent copycat attacks. Windows Event IDs are accurate for Windows Server 2019/2022.

TL;DR

  • Bad actors snuck harmful code into a popular AI tool called LiteLLM that thousands of businesses use [1].
  • The attack stole passwords, secret keys, and digital wallets from anyone who installed the poisoned version [1].
  • They did it by first compromising a security tool that LiteLLM trusted — like poisoning the water at the treatment plant [2].
  • Here is what it means for your business and how to stay safe.

What Is LiteLLM?

Imagine you run a restaurant and instead of ordering from one food supplier, you want to compare prices from ten different ones. LiteLLM is like a universal ordering app that lets businesses talk to different AI services — ChatGPT, Claude, Gemini — all through one simple connection.

Thousands of companies use it to build AI features into their products [1].

What Went Wrong?

A group of hackers called TeamPCP figured out something clever. Instead of breaking into LiteLLM directly, they first broke into a security scanner called Trivy — a tool that LiteLLM used to check itself for bugs [2].

Think of it this way: imagine a locksmith who checks all the locks in your building gets compromised. Now the attacker does not need to pick any locks — they have the locksmith's master key.

Once inside, TeamPCP published two fake versions of LiteLLM (versions 1.82.7 and 1.82.8) to PyPI, the online store where developers download software [1]. Anyone who downloaded these versions unknowingly installed malware that:

  • Collected passwords and secret keys stored on their computers [1]
  • Spread to other computers on the same network [1]
  • Set up a hidden door that let the hackers come back anytime they wanted [1]

Why Should You Care?

You might not use LiteLLM directly, but your business probably relies on software that works the same way — built from dozens of smaller pieces, each one downloaded from the internet.

According to security research firm Sonatype, attacks on these software building blocks increased by 156% in just one year [3]. And IBM found that when hackers steal login credentials this way, the average cleanup cost is $4.81 million [4].

The Australian Cyber Security Centre has flagged these kinds of attacks as one of the top threats businesses face today [5].

What Can You Do?

Ask your IT team or provider three questions:

  1. "Do we pin our software to specific versions so updates do not happen automatically?" — This stops poisoned updates from sneaking in.

  2. "Do we have tools that scan our software for known threats?" — Free and paid tools exist that check every package you download against a database of known attacks [6].

  3. "If a tool we depend on gets compromised, how quickly would we know?" — The answer tells you whether your business would catch something like this in hours or months.

If you do not have an IT team: Start by keeping an inventory of the software your business uses. Know what you depend on. That awareness alone puts you ahead of most small businesses.

The Simple Takeaway

Every AI tool and every piece of software your business uses is built from smaller parts. If any of those parts gets poisoned, the whole thing becomes dangerous. The best protection is knowing what you depend on and having someone who watches for these threats.

It is like food safety — you trust your suppliers, but smart restaurants still check what arrives at the loading dock.

FAQ

Instead of attacking your business directly, hackers attack the tools or software your business depends on. When you update or install that trusted software, you unknowingly install the attacker's code too. It is like someone tampering with ingredients at a factory — every product made with those ingredients gets affected.

If anyone in your organisation uses Python and has LiteLLM installed, check the version number. Versions 1.82.7 and 1.82.8 were the compromised ones. Run pip list | grep litellm to check. If you see those versions, contact an IT professional immediately.

Very common and growing fast. Sonatype tracked a 156% increase in software supply chain attacks in 2025 [3]. The LiteLLM incident is the fifth software ecosystem TeamPCP has targeted, showing these attackers are becoming more ambitious [2].

No. AI tools can genuinely help your business work smarter and save money. The key is using them with proper safeguards — verified versions, dependency scanning, and regular security reviews. Think of it like driving: cars are useful, but you still wear a seatbelt.

References

[1] Endor Labs, "TeamPCP Isn't Done — LiteLLM Supply Chain Attack Analysis," Endor Labs Research, Mar. 24, 2026. [Online]. Available: https://www.endorlabs.com/learn/teampcp-isnt-done

[2] R. Lakshmanan, "TeamPCP Backdoors LiteLLM Versions 1.82.7–1.82.8 Likely via Trivy CI/CD Compromise," The Hacker News, Mar. 24, 2026. [Online]. Available: https://thehackernews.com/2026/03/teampcp-backdoors-litellm-versions.html

[3] Sonatype, "2025 State of the Software Supply Chain Report," Sonatype, 2025. [Online]. Available: https://www.sonatype.com/state-of-the-software-supply-chain

[4] IBM Security, "Cost of a Data Breach Report 2025," IBM, 2025. [Online]. Available: https://www.ibm.com/reports/data-breach

[5] Australian Cyber Security Centre, "Annual Cyber Threat Report 2024-2025," Australian Signals Directorate, 2025. [Online]. Available: https://www.cyber.gov.au/about-us/reports-and-statistics/annual-cyber-threat-report

[6] Socket Security, "TeamPCP Targeting Security Tools Across OSS Ecosystem," Socket Blog, Mar. 2026. [Online]. Available: https://socket.dev/blog/teampcp-targeting-security-tools-across-oss-ecosystem

[7] JFrog, "LiteLLM Compromised by TeamPCP — Supply Chain Attack Analysis," JFrog Security Research, Mar. 24, 2026. [Online]. Available: https://research.jfrog.com/post/litellm-compromised-teampcp/

[8] McKinsey & Company, "The State of AI in 2025," McKinsey Global Institute, 2025. [Online]. Available: https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai


Wondering if your business software is safe? Talk to lilMONSTER — we help businesses understand their technology risks in plain language.

Ready to strengthen your security?

Talk to lilMONSTER. We assess your risks, build the tools, and stay with you after the engagement ends. No clipboard-and-leave consulting.

Get a Free Consultation