Security agencies from several nations warn that attackers were able to deceive the integrity checking tools provided by Ivanti in response to the recent attacks exploiting zero-day vulnerabilities in its Connect Secure and Policy Secure gateways. The agency also identified a technique in a lab setting that could be used to achieve malware persistence on Ivanti devices despite factory resets.
“The authoring organizations strongly urge all organizations to consider the significant risk of adversary access to, and persistence on, Ivanti Connect Secure and Ivanti Policy Secure gateways when determining whether to continue operating these devices in an enterprise environment,” the US Cybersecurity and Infrastructure Security Agency (CISA) said in an advisory co-authored with the US Federal Bureau of Investigation (FBI), the Australian Signals Directorate, the UK’s National Cyber Security Centre, Canada’s Communications Security Establishment (CSE), and New Zealand’s National Cyber Security Centre.
Ivanti responded by releasing an enhanced version of its external integrity checking tool (ICT) and said it believes the persistence technique devised by CISA in its lab would not work in a live customer environment because attackers would lose their connection to the device.
Integrity checker failed to detect compromises in some cases
CISA identified during multiple incident response engagements that both the internal and external integrity checking tools provided by Ivanti failed to detect the existing compromises. These are tools that check important areas of the file system for modifications and known signs that could indicate an attack.
However, since these tools execute periodically and not continuously — the internal one checks every two hours — malware authors could attempt to evade detection by activating their malware in between the scans. This is exactly what incident response firm Mandiant has observed in limited attacks perpetrated by a China-based APT group that it tracks as UNC5325. This group started exploiting the CVE-2024-21893 vulnerability hours after Ivanti publicly disclosed it on January 31 and displayed a high level of knowledge and familiarity with the internal workings of Ivanti SSL VPN gateways, suggesting it has reversed-engineered these devices.
“Notably, Mandiant has identified UNC5325 using a combination of living-off-the-land (LotL) techniques to better evade detection, while deploying novel malware such as LITTLELAMB.WOOLTEA in an attempt to persist across system upgrades, patches, and factory resets,” the company said in a report this week.
One of the implants deployed by UNC5325 is a web shell — a web-based remote access backdoor — dubbed BUSHWALK that’s written in Perl and embedded into a legitimate Ivanti Connect Secure component called querymanifest.cgi. In the most recent attacks, the group used a new variant of this shell and a technique that allowed them to enable and disable it based on the user-agent string specified in requests sent to the shell.
This allowed them to keep an encrypted copy of the web shell in an area of the file system that is not checked by Ivanti’s ICT and only decrypt it and add it to querymanifest.cgi when needed using one of the Ivanti built-in tools. Then, when deactivating it, the original querymanifest.cgi is restored, its timestamp is changed to hide the recent modification and the encrypted copy of the web shell is kept in the location that’s not scanned by Ivanti’s ICT.
“Ivanti, Mandiant and CISA recommended using the updated external ICT to help detect known attack vectors and detect additional files or changed files,” Ivanti said in a blog post. “As Ivanti has emphasized, this is a useful and informative security tool in your arsenal, to complement other security and monitoring tools. Our recommendation remains that you should use the updated ICT in concert with continuous monitoring.”
Additional continuous monitoring tools are needed on the devices because ICT provides only a snapshot in time of the file system and cannot detect changes made by the attackers in the past and later reverted before the tool was run.
Attempted persistence across system patches and factory resets
The UNC5325 group also devised methods to deploy a backdoor dubbed by Mandiant as LITTLELAMB.WOOLTEA that would persist even after system upgrades, patches, or factory resets. The backdoor is deployed as a shared object called libchilkat.so as part of a rogue plugin for SparkGateway, a legitimate component of the Ivanti Connect Secure appliance that enables remote access over a browser using protocols like RDP or SSH.
The attackers devised two methods to make this backdoor persist across patches. First, it appends its malicious components like the modified SparkGateway configuration and the rogue plugin to an archive called /data/pkg/data-backup.tgz. This is a backup copy of the data directory from the device that is restored during the system upgrade process.
The second method involves constantly checking for the presence of a filesystem path called /tmp/data/root/dev that only gets created during a system upgrade event. If this path is detected, the backdoor proceeds to copy itself and the other modified files to /tmp/data/root/samba_upgrade.tar, an archive that is used for data migration during the upgrade process.
To survive factory resets the attackers devised a complicated technique that checks the type of device then mounts the factory reset root partition to a temporary directory and attempts to make modifications to deploy a trojanized version of the tar archiving tool inside. This version of tar would automatically unpack the maliciously modified samba_upgrade.tar during a factory reset procedure basically restoring the backdoor.
However, this persistence technique failed because the factory reset partition is encrypted with an encryption key that’s hard-coded in the running version’s kernel at build compilation and is unique for each appliance version. “If the current running version and the factory reset deployment versions differ (i.e., the appliance or VM has been updated at least once), then /bin/losetup will fail to decrypt the factory reset partition due to the encryption key mismatch and thus the malware will not persist after factory reset,” Mandiant explained. Neither Mandiant nor Ivanti have seen any incident where this technique seems to have worked.
This is not the same persistence technique that CISA claims to have found during its own internal laboratory testing and which Ivanti doesn’t think would be practical in real attacks. No details have been publicly released about that technique, but UNC5325’s attempts show attackers are interested in developing such methods.
Mitigation advice for undetected Ivanti VPN compromise
“The authoring organizations encourage network defenders to (1) assume that user and service account credentials stored within the affected Ivanti VPN appliances are likely compromised, (2) hunt for malicious activity on their networks using the detection methods and indicators of compromise (IOCs) within this advisory, (3) run Ivantiâs most recent external ICT, and (4) apply available patching guidance provided by Ivanti as version updates become available,” CISA said in its advisory. “If a potential compromise is detected, organizations should collect and analyze logs and artifacts for malicious activity and apply the incident response recommendations within this advisory.”
The authoring agencies also stressed that given their observations and findings the “safest course of action” for network defenders is to assume sophisticated threat actors might be able to deploy persistent rootkits on devices that have been factory reset and then potentially lay dormant for an arbitrary amount of time in order to evade detection. This is why the agencies advise organizations to consider the risks of keeping Ivanti devices on their networks at all.
Meanwhile Ivanti advises customers who have already completed a factory reset on their hardware appliances or deployed a new build on their virtual appliances — deploying new build is recommended vs factory reset on virtual appliances — to continue to run the internal and the updated external ICT and perform continuous monitoring using other tools that take into account the IoCs and TTPs released by Mandiant and CISA.
Customers who haven’t yet deployed the patches for all the previously announced and exploited vulnerabilities should do so urgently by following the instructions in Ivanti’s knowledgebase article.