Terabytes of credentials leaked in massive supply-chain

What happened

Terabytes worth of credentials, many belonging to the world’s biggest and most sensitive organizations, have been exposed in a supply-chain attack on LiteLLM, an open source tool that streamlines AI-driven software development. Microsoft, Amazon, Cisco, Samsung, and Salesforce are only a handful of the entities whose access secrets were exposed. The revelation was posted on Tuesday and Wednesday by security firms CloudSEK and Hudson Rock.

” Both firms are urging all organizations that used the compromised versions of LiteLLM—particularly those listed in the high-confidence section of the list—to thoroughly rotate all credentials in their pipelines. 8 of LiteLLM, the two compromised versions of the software.

The firm advised all those affected to perform “aggressive credential revocation,” assume any secret accessible to the LiteLLM environment is compromised, invalidate and rotate all cloud keys, Kubernetes service account tokens, and GitLab/GitHub PATs, and audit logging and egress filtering. As a cautionary tale, CloudSEK said that Trivy developers rotated, but failed to fully revoke an automation token over a 20-day window.

The lapse gave the attackers a nearly three-week period to force-push malicious code to third-party builds that used the vulnerability scanner. As Beaumont observed, organizations’ rush to integrate AI into their software delivery systems has also greatly contributed to the scale of the damage. Update:There are already signs that some of the affected organizations aren’t taking the disclosure with the seriousness warranted.

Why it matters

CloudSEK said it found cloud keys, repository tokens, SSH keys, Kubernetes secrets, package publishing credentials, environment variables, and AI provider keys that could allow attackers to gain access to more than 2,500 organizations. The credentials were extracted during a 40-minute window in March while the victims used compromised versions of LiteLLM downloaded from the package’s official location in the Python Package Index repository.

Hudson Rock said it made the discovery after analyzing a 195TB file that it obtained. Neither firm identified the source of the information. The LiteLLM compromise was the result of a previous supply-chain attack that infected the widely used vulnerability scanner Trivy. Other software infected in the campaign includes KICS and the Telnyx Python SDK.

TeamPCP, a ramshackle but extremely capable gang largely made up of teenagers, took credit for the attack, and researchers have largely corroborated the claim. “I’ve confirmed the data is legit by the way, multiple victim orgs,” independent security researcher Kevin Beaumont said. “It contains a significant volume of sensitive content at orgs.

” The compromised versions of all four software packages contained code that accessed the memory of infected machines, scraped its contents, and exfiltrated it through an attacker-controlled channel. The data is filled with an assortment of information. Interspersed in the wall of data are credentials to software pipelines maintained by the tens of thousands of organizations that ran LiteLLM during the 40-minute span that the supply-chain attack remained active.

In all, both security firms said some 434,000 CI/CD (continuous integration/continuous delivery) software pipelines had credentials exposed after running the compromised LiteLLM versions. In many cases, researchers at CloudSEK and Hudson Rock had trouble identifying the organizations the credentials belonged to. com ultimately didn’t indicate a breach at the satellite broadcaster, but rather one within the infrastructure of SiriusXM subsidiary AdsWizz. A full list of organizations is here.

The researchers had high confidence that these organizations had their credentials exposed: “Many CI/CD pipelines are configured generically,” Hudson Rock said. “The dumped variables contain active database passwords, third-party API keys, and cloud credentials without any identifiable company email, custom domain string, or internal server name.

What to watch

After this post went live, Beaumont reported: Ultimately, the new revelations concerning the LiteLLM supply-chain attack underscore the growing threat of such campaigns and hence the importance of maintaining vigilance around the use of open source software that, when infected, can spread rapidly across the Internet.

“The key takeaway is how supply chains have evolved to make a single upstream breach affect thousands of companies simultaneously,” Alon Gal, co-founder and chief technology officer of Hudson Rock, wrote in an email. “A window of roughly 40 minutes in which the LiteLLM dependency was hacked led to over 430,000 instances in which millions of secrets were harvested. ”