Building security into cloud native apps with NGINX
Companies like Airbnb, Uber and DoorDash, which have a cloud-based software infrastructure as one of their main enablers, are disrupting the hospitality, transportation and food delivery sector. Why do all these new companies use the cloud and what advantages does it give them?
Unlike legacy competitors, innovators with new infrastructure can:
1. Quickly scale and grow their customer base.
2. Support their business in different geographies and ensure availability.
3. Ensure convenience (users are accessing the services from their mobile phones).
4. Respond quickly when the market or technical environment changes.
One of the issues that is top of mind for these innovators is making sure that their infrastructure – the heart and soul of their business – is protected from attackers. Smaller companies that are just starting on the journey to new online services also find security to be important for their everyday business as well as competitive differentiation.
While there is a well-developed market of commercial security solutions, there are also budget-friendly alternatives that may be suitable for startups or companies developing cloud-native applications for internal use.
If technical resources are available, there are many open source tools that can help protect those assets against OWASP Top10 attacks, some bots, application abuse and even select data leakage issues. One of those tools is NGINX.
What does cloud-native mean and how does NGINX fit in?
NGINX is one of the “engines” that powers cloud infrastructure. This open source tool has a thriving multi-million user community and has been popular for years now, displacing many LAMP stack solutions. NGINX tends to be used as a core of solutions as diverse as web servers, load balancers and API gateways. With Kubernetes rapidly becoming the most popular platform for dockerized infrastructures, a lot of companies are connecting their container-based web applications to the outside world with a Kubernetes Ingress Controller, sometimes without even knowing that it is based on NGINX.
Not many know that a tool like NGINX can also be used for security. In most architectures, traffic passes through NGINX reverse-proxy instances. NGINX is usually used as a load balancer and HTTP requests are parsed by NGINX. Add a few modules for attack detection, and you get what you need right on the load-balancer layer, without adding any additional hops by installing other types of software or hardware.
In a cloud-native environment, there are always options. For instance, in a Kubernetes environment you can add individual security protections to every microservice by installing NGINX as a side-car. Alternatively, you can ensure security for their Kubernetes cluster by adding detection facilities to the Kubernetes Ingress Controller, where every request that comes into the Kubernetes cluster is analyzed.
Adding basic web security to cloud applications
Adding an open source Web Application Firewall (WAF) for your microservice-powered application is not hard. For example, mod_security is a popular open source WAF for Apache and NGINX and it is already a part of Google Kubernetes Ingress Controller. To use it, you need to enable the corresponding flag in the configuration file `ConfigMap`:
apiVersion: v1
kind: ConfigMap
metadata:
name: nginx-configuration-external
namespace: ingress-nginx
data:
enable-modsecurity: "true"
enable-owasp-modsecurity-crs: "true"
...
ModSecurity is not the easiest WAF for NGINX to manage nor is it the most accurate (it generates a fair number of false positives). But it has been around for a long time, has a sizeable user community and can be reasonably effective with sufficient technical resources available for on-going configuration and to interpret the results that it generates. Another open source WAF alternative is NAXI — but this too requires a good deal of effort and manual tuning in order to run.
Alternatively, there are a number of commercial WAF offerings for NGINX and Kubernetes Ingress Controller. While these products are not free, in many cases the cost can be recovered, as you need fewer people to maintain the tool and you can enjoy advance capabilities like API protection and a lower rate of false positives.
Knowing and monitoring your cloud-native application
One of the most important aspects of maintaining security is visibility and monitoring. The most logical and straightforward way to get that visibility is by analyzing the logs. You can get data from NGINX and mod_security logs and make an informative security dashboard using a combination of ElasticSearch, Kibana and Fluentd.
WAF logs are located in /var/log/modsec/audit.log. In addition, there is Fluentd, a special tool that parses those logs and sends them to ElasticSearch. This in itself is already a powerful tool for event analysis and forensic research. For even richer graphing capabilities, logs in ElasticSearch can be visualized with Kibana.
One way to make this process more effective is to run Fluentd as a sidecar in the ingress-nginx pod. This option works best if the mod_security Fluentd plugin is enabled in Kubernetes.
Share a volume mount between ingress-nginx and Fluentd so that Fluentd can access and index mod_security logs.
This is all that is needed to have ElasticSearch and Kibana create beautiful custom dashboards that unify the log data and WAF events across the applications and APIs for operations and security teams. (They also create impressive presentations for company leadership.)
Alerting
Dashboards are nice but if hackers are attacking your services now, the reaction needs to be near instantaneous or the company risks losing revenue, reputation, confidential data and customers. To enable this kind of incident response, real-time alerts and actionable notifications are a must.
There is a perfect open source tool for generating this type of alert from the data in ElasticSearch. The tool is called ElastAlert. ElastAlert was developed by Yelp. It’s a very powerful framework for alerting on anomalies, spikes or other patterns of interest from data in ElasticSearch.
For example, you can send a notification if you see more than N requests of SQL injections and applications response with 5xx errors. You can set different triggers with complex logic and get alerts as SMS messages, in your incident response tools like OpsGenie or PagerDuty, or via any messenger (Slack, Hipchat, Telegram, etc.).
Building a fence against hackers
In addition to all of this, you can restrict traffic from certain countries with the GeoIP module (ngx_http_geoip_module). For example, this practice could be useful if your apps are not intended for use by customers outside of the US. This, however, isn’t always the best practice, since GeoIP data isn’t completely accurate.
Most malicious tools and proxies are hosted somewhere, such as public clouds. You can try to get lists of IP addresses from cloud providers to block traffic that comes from suspicious data centers and data providers. Here are where the big three cloud providers publish their lists: GCP, AWS, Azure.
Google also provides IP ranges for others.
What else to block? If you don’t want traffic from streaming tools like Tor, you can block Tor exit nodes. You can also block malicious IPs from Project HoneyPot, and anything else that is typically suspicious for your apps.
Blocking bots
Bots are automated agents that probe online applications in an attempt to penetrate the application defenses, steal data or cause denial of service. Behavior analytics tools are needed to detect most bots, but many bots can still be caught with straightforward checks. If a client supports cookies, performs HTTP Redirects or supports JavaScript, these checks can be done right on the NGINX server.
To get this function working, use the NGINX module called testcookie-nginx. The module works as a filter between the bots and the backend during L7 attacks, allowing you to screen out junk requests.
Conclusion
Cloud security doesn’t need to be expensive if it’s architected correctly.
For startups and larger companies that trust NGINX to power their cloud-native or Kubernetes infrastructure, there are many alternatives to ensure their applications are protected from hackers. We have shown several practical examples on how to detect simple attacks, achieve visibility and enable alerting without breaking the bank. This might be especially interesting for folks who are die-hard proponents of open source and use the open source version of NGINX and the Kubernetes Ingress Controller.