Using Big Data for intelligent enterprise security

No industry remains untouched by the potential of Big Data – and the cybersecurity sector is certainly no exception. A recent MarketsandMarkets study predicted the cybersecurity analytics market would grow from $2.83 Billion US in 2016 to $9.38 Billion in 2021, a Compound Annual Growth Rate of 27.1%.

This reason for this vast growth is relatively straightforward to understand. With an ever-more sophisticated and better-funded attack environment, many of the most insidious and potentially dangerous threats demand very deep insight into an organisation’s data, networks, and usage for detection.

For years now, we’ve heard a number of security researchers lamenting traditional approaches to cybersecurity as doing little for organisations hoping to take a strong, complete defence against the complex omnipresent cyber threats. While some organisations may have systems that alert them to events that deviate from “business as usual” patterns, many hackers are now ahead of the game.

Data overload for traditional solutions

Some traditional solutions, for instance, will generate an alert of potentially “suspicious activity” if it identifies a number of failed login attempts to the same account and from the same IP address. Alternatively, it may send an alert if it identifies excessive data access outside of normal working hours from a certain account. But hackers, conscious that such patterns are tracked by these solutions, are adapting their approach to avoid relying on such methods.

Moreover – and this is a key point with organisations’ growing networks and web perimeters – too often traditional security solutions just cannot handle the sheer quantity of data being created. Unstructured data, which doesn’t fit a pre-defined data model and is not organised in a pre-defined way, is by far the fastest growing data type that is accessed and used, and too much of it ingested at once can also choke the system.

Enter security analytics

Big Data solutions can now be used for security analytics to capture, filter, and analyse millions of network events – no matter how discrete – every second. These solutions work with data from a wide variety of data sources, such as audit and log files, as well as unstructured data, including email, social media, images, video, news feeds, and more.

With the International Institute of Analytics predicting that security analytics will become the “first line of defence,” combining text mining, machine learning, and ontology modelling to sniff our security threats before they strike, there’s no question that demand for the holistic, integrated security approach will increase in the coming years. Indeed, it is no small wonder that the Big Data analytics market topped 125 billion dollars in 2015.

But in order for a successful deployment of security analytics to meet the requirements of today’s threat landscape, it is essential that organisations retain the colossal amounts of data needed for large-scale analytics. This is crucial to get optimal visibility into all the activity across the infrastructure and network – enabling many organisations to get the long-coveted “knowledge of the unknown” through the use of automated, actionable intelligence to spot potentially malicious, behavioural anomalies.

Getting started

To an organisation that has long depended on off-the-shelf traditional security solutions, there is no doubt that incorporating Big Data for security analytics into your organisation’s defences will seem complicated.

But to get started, here are a few recommended first steps:

1. Make sure that the data platform you deploy has a strong focus on authorisation, authentication, and data protection capabilities to secure all of your collected data. Auditing is also crucial to track data access and for tracing it back to any breaches.

2. With traditional data platforms are simply incapable of handling unstructured data in a clean and efficient manner, it is essential that organisations looking to deploy Big Data for security analytics opt for a platform that enables them to apply a wide variety of analytical tools in a scalable manner.

Organisations should look to solutions that deploy the latest Big Data tools, to help them get a more comprehensive picture of what potential threats may be lurking in their environment.

3. Organising internal resources can also greatly facilitate Big Data for security analytics. Companies should ensure that their IT security analysts and data scientists are talking and meeting regularly to ensure they are synced on the best approach to evolve the security platform against emerging threats.

Businesses can no longer rely on a “detect and respond” approach to cybersecurity. In today’s threat environment, predicting attacks before they even hit and identifying weaknesses in the system before hackers do, is essential. Indeed, it is naïve and could have potentially disastrous consequences to presume that an off-the shelf solution could offer the comprehensive protection that an organisation needs.

With the value that organisations across all industries are generating from using Big Data to identify new avenues for growth, it is important that they also consider the massive potential of using this technology to defend their current customers and assets against the growing cyber threat.

Organisations who are serious about protecting their networks must look beyond traditional security solutions, and deploy large-scale, advanced analytics on converged data platforms, incorporating anomaly detection and machine learning to give them a fighting chance in the battle against cybercrime.