(Re)check your patched NetScaler ADC and Gateway appliances for signs of compromise

Administrators of Citrix NetScaler ADC and Gateway appliances should check for evidence of installed webshells even if they implemented fixes for CVE-2023-3519 quickly: A recent internet scan by Fox-IT researchers has revealed over 1,800 backdoored NetScaler devices, 69% of which have been patched for the flaw.

“This indicates that while most administrators were aware of the vulnerability and have since patched their NetScalers to a non-vulnerable version, they have not been (properly) checked for signs of successful exploitation,” the researchers noted.

CVE-2023-3519 exploited to drop webshells on NetScaler devices

CVE-2023-3519 is an unauthenticated remote code execution (RCE) vulnerability that has been patched by Citrix on July 18, but not before having been exploited as a zero-day in targeted attacks against a critical infrastructure organization in the US.

As various organizations that were involved in incident response (IR) engagements at compromised enterprises have since discovered, the vulnerability was later exploited on a large scale in an automated fashion.

In collaboration with the Dutch Institute of Vulnerability Disclosure, Fox-IT researchers have been scanning the internet to pinpoint devices running the specific webshells used in these attacks, and found them on 1,828 instances. 1,248 of those appliances have been patched for CVE-2023-3519.

“We initially only scanned systems that were not patched on July 21st, as the exploitation was believed to be between July 20th and July 21st. Later, we decided to also scan the systems that were already patched on July 21st. The results exceeded our expectations. Based on the internet wide scan, approximately 2000 unique IP addresses seem to have been backdoored with a webshell as of August 9th,” they explained.

The interesting thing about this mass automated attack is that the attackers did not compromise all (31,000 or so) vulnerable NetScaler devices on July 21, but just 1,952 of them – and most of those devices are located in Europe.

Top 20 countries with backdoored Citrix NetScaler devices as of August 14th 2023 (Source: Fox-IT)

“While Canada, Russia and the United States of America all had thousands of vulnerable NetScalers on July 21st, virtually none of these NetScalers were found to have a webshell on them. As of now, we have no clear explanation for these differences, nor do we have a confident hypothesis to explain which NetScalers were targeted by the adversary and which ones were not. Moreover, we do not see a particular targeting in terms of victim industry,” the researchers added.

Tools to use to search for IOCs

Whether they’ve fixed CVE-2023-3519 on the devices they manage or not, Fox-IT is urging enterprise admins to check again for evidence of compromise by using:

• A Python script created by Fox-IT to perform triage on forensic images of NetScaler devices, and
• Mandiant’s bash-script that surfaces associated indicators of compromise (IOCs) on live systems

“If traces of compromise are discovered, secure forensic data,” they advised.

“If a webshell is found, investigate whether it has been used to perform activities. Usage of the webshell should be visible in the NetScaler access logs. If there are indications that the webshell has been used to perform unauthorised activities, it is essential to perform a larger investigation, to identify whether the adversary has successfully taken steps to move laterally from the NetScaler, towards another system in your infrastructure.”