Security flaw could turn load balancers into beachheads for cyber attacks

Cyber security provider F-Secure is advising organizations using F5 Networks’ BIG-IP load balancer, which is popular amongst governments, banks, and other large corporations, to address security issues in some common configurations of the product.

Adversaries can exploit these insecurely configured load balancers to penetrate networks and perform a wide variety of attacks against organizations, or individuals using web services managed by a compromised device.

Exploitation potential

The security issue is present in the Tcl programming language that BIG-IP’s iRules (i.e., Tcl scripts) are written in. Certain coding practices allow attackers to inject arbitrary Tcl commands, which could be executed in the security context of the target Tcl script.

Adversaries that successfully exploit such insecurely configured iRules can use the compromised BIG-IP device as a beachhead to launch further attacks, resulting in a potentially severe breach for an organization. They could also intercept and manipulate web traffic, leading to the exposure of sensitive information, including authentication credentials and application secrets, as well as allowing the users of an organization’s web services to be targeted and attacked.

In some cases, exploiting a vulnerable system can be as simple as submitting a command or piece of code as part of a web request, that the technology will execute for the attacker. To make matters worse, there are cases where the compromised device will not record the adversaries’ actions, meaning there would be no evidence that an attack took place. In other cases, an attacker could delete logs that contain evidence of their post-exploit activities – severely hindering any incident investigations.

“This configuration issue is really quite severe because it’s stealthy enough for an attacker to get in, achieve a wide variety of objectives, and then cover their tracks. Plus, many organizations aren’t prepared to find or fix issues that are buried deep in software supply chains, which adds up to a potentially big security problem,” explains F-Secure Senior Security Consultant Christoffer Jerkeby. “Unless you know what to look for, it’s tough to foresee this problem occurring, and even harder to deal with in an actual attack.”

Jerkeby discovered over 300,000 active BIG-IP implementations on the internet during the course of his research, but due to methodological limitations, suspects the real number could be much higher. Approximately 60 percent of the BIG-IP instances he found were in the United States.

The coding flaw and class of vulnerability is not novel and has been known, along with other command injection vulnerabilities in other popular languages, for some time. And while not everyone using BIG-IP will be affected, the load balancer’s popularity amongst banks, governments, and other entities that provide online services to large numbers of people, combined with the relative obscurity of the underlying security issues with Tcl, means any organization using BIG-IP needs to investigate and assess their exposure.

“Unless an organization has done an in-depth investigation of this technology, there’s a strong chance they’ve got this problem,” says Jerkeby. “Even someone incredibly knowledgeable about security that works at a well-resourced company can make this mistake. So, spreading awareness about the issue is really important if we want to help organizations better protect themselves from a potential breach scenario.”

Recommendations for organizations

F5 has been notified of the research several months ago, and has already posted a public advisory detailing the affected products, Tcl statements, and more.

Because it is possible to mass scan the internet to identify and exploit vulnerable instances of the technology, and in some cases, automate this process, the issue is likely to attract attention from bug bounty hunters and attackers.

Furthermore, free trial versions of the technology can be obtained from the vendor, and cloud instances can be accessed from the AWS store for a minimal cost. For these reasons, as well as the potentially severe impact of attacks using this flaw, F-Secure is advising organizations to proactively investigate whether or not they’re affected.

Jerkeby has helped develop two free, open-source tools (1, 2) that organizations can use to identify insecure configurations in their BIG-IP implementations. But according to him, there’s no quick fix for security issues like these, so it’s up to organizations to tackle the issue.

“The upside of this kind of security problem is that not everyone using the product will be affected. But the downside is that the problem can’t be fixed with a patch or software update from the vendor, so it’s up to organizations to do the work to check to see if they have this issue, and fix it if they find it,” explains Jerkeby. “That’s why it’s important for anyone using BIG-IP to be proactive about this.”

He recommends consulting these web pages to learn how to modify vulnerable Tcl scripts.

Jerkeby has presented his findings at Black Hat USA 2019. More technical information can be found here.

UPDATE: August 11, 7 AM PT – F5 Networks sent us the following statement:

This is not a vulnerability in Tcl, nor F5 products, but rather an issue relating to coding practices used in creating the scripts. As with most programming or scripting languages, it is possible to write code in a way that creates vulnerabilities. We have been working with the researcher on documentation and notification to ensure customers can evaluate their exposure and take necessary steps to mitigate. The best practice for Tcl scripting is to escape all expressions, ensuring they are not substituted or evaluated unexpectedly. Customers are advised to evaluate Tcl scripts and make all changes they deem appropriate under this guidance.