Q&A: Splunk

In this interview the Splunk team discusses Splunk in detail.

What is Splunk?
Splunk was born from its founders’ frustration in running some of the worlds largest IT infrastructures. Using state-of-the-art IT management, security and compliance tools, they found it nearly impossible to locate the root cause of problems, investigate security attacks and assimilate all the data required for audits. Their conclusion was the silo approach to managing IT, with separate tools for every technology and IT function, was cumbersome, costly and didn’t scale.

So, they founded Splunk to develop a new approach. The concept was simple. If Google could index and let users search across billions of pages of Web content in seconds, why not do the same for the datacenter? The result of thinking differently, Splunk is software that indexes the data generated by any application, server or network device running across technical, functional and geographic IT silos and lets you instantly search, alert and report on it.

Splunk does this without having to rely on inflexible and brittle databases, costly custom data connectors/ parsers, or force users to learn a new interface and vendor semantics. Using Splunk organizations can now troubleshoot application outages, investigate security incidents, and demonstrate compliance in minutes, instead of hours or days.

Splunk is a free software download (up to 500mb of uncompressed data indexed per day), works with all the leading operating systems, and is easy to install and use. Most users are up and running in less than an hour.

Recently you released Splunk 4. What are the most notable improvements and changes in this version?
Splunk 4 replaces a very successful product Splunk 3.x and builds on an innovative and tremendously successful approach Splunk introduced in 2005, Search for IT data. In Splunk 4 there are over 50 new features that impact four major areas of product functionality:

Speed and scale – Splunk 4 introduces 10x improvements in search speed and 2x improvements in indexing speed over the previous release. No other technology for managing time-series, unstructured IT data delivers this level of performance and the product has tested 100x faster than competitive security and event management solutions. Splunk’s MapReduce implementation delivers dramatic scale improvements enabling the indexing of terabytes of data per day on commodity hardware. These improvements help customers run Splunk 4 with fewer resources that the previous release.

Usability – Splunk 4 enables both technical and non-technical users to leverage the power of IT Search. This enterprise-wide usability expands the universe of potential Splunk users, well beyond the “traditional” Splunk user who is more technical and hands on. Users can create personalized dashboards and views specific to their preferences and which integrate visual charts, search results, data from external applications (such as Tivoli, SAP, security consoles, etc.) and more. This opens the door to new users and business-level use cases leveraging the same underlying IT data for multiple critical use cases, such as application troubleshooting, infrastructure and operations management, security investigations and compliance reporting.

Splunk as the IT app engine – Splunk 4 introduces the Splunk App Framework which enables Customers, Partners and 3rd party technology vendors to further innovate on top of Splunk and build apps to meet critical operations, security and compliance requirements. Examples of technology partner apps for Splunk 4 include F5, Blue Coat, Cisco, VMware and Microsoft . The Splunk App Framework provides the ability to develop and package apps through a single user interface, apply role-based access controls for the app, and deploy apps with an app-specific installation experience. Once deployed, users can easily switch context between apps using the new App Launcher user interface.

Enterprise manageability – Splunk 4 includes comprehensive enhancements that significantly improve the ability to administer and manage Splunk installations.

First is a completely new interface called the Splunk Manager for both administrators and users. Administrators can now manage their Splunk installation centrally and delegate management of Splunk to different departments and users. Individual users can now see and manage their own saved searches, reports, event types and dashboards and only view Splunk resources they have/need access to.

Second is a new Jobs Management user interface; Splunk 4 introduces the notion of jobs, such as individual searches, and the new interface enables administrators to see which jobs are draining resources, assign priorities to jobs, allocate job quotas to different users and pause, finalize and resume jobs.

Thirdly, a new Splunk Monitor user interface provides systems administrators visibility across their entire Splunk deployment, such as license usage per indexer, Splunk CPU usage metrics, indexer performance etc.

Finally, the Splunk Deployment Server has been enhanced to improve how distributed Splunk deployments are managed, including support for managing the deployment of entire apps, managing configurations hierarchically, and support for automatic load balancing and failover.

What are the features you see your customers use the most?
The customers who report the most dramatic impact (reducing MTTI/MTTR 65%, escalations reduction of 92%, passing audits the first time after repeatedly failing) use Splunk in a circle of continuous improvement. There are five steps:

1. This starts with indexing all their IT data
2. Then users search and investigate through the index to find the needles in the haystack, detect errors/attacks, find root causes
3. Over time users add knowledge to Splunk such as saving/sharing valuable searches, tagging fields and definitions of data (events, hosts, sources, transactions, etc.)
4. Monitoring and alerting lets you save any search as a proactive alert to run on a schedule, these results trigger notifications and automate actions (email, RSS, post to other systems/consoles via SNMP, trigger scripts to open tickets, restart a server, etc.)
5. Report and analyze the data to review trends and other findings to become more proactive in a cycle of improved IT operations.

Our customers typically start by using Splunk to solve a specific problem area. Often it’s application management and troubleshooting, or security monitoring and incident investigation, or compliance. After quickly making their initial use case an internal success, Splunk is typically deployed across other areas of IT.

How demanding is it to deploy Splunk in a massive network?
One of the reasons for Splunk’s success is its ability to be installed quickly and efficiently – typically in less than an hour. For complex architectures with distributed global topologies this is still very easy, although a little more complex. Unlike other enterprise technology companies who must employ legions of consultants and professional services personnel to install their products, Splunk’s services team is a handful of people. Complex engagements rarely take longer than a few days versus weeks or months.

With an easy to install free download model, Splunk is typically already deployed in the organization in a test environment, moving to production is typically very easy to accommodate.

Can Splunk help with compliance?
Splunk helps organizations find the valuable information needed for compliance buried in all the logs and IT data their infrastructures spit out everyday. e-Discovery, FFIEC, FISMA, HIPAA, IT Governance, PCI, SOX and other mandates require regular review of logs and IT data. But most solutions only work with a small number of data sources, require constant maintenance and are too rigid to be used for other applications. Splunk provides the ability to achieve sustainable compliance and leverages the same investment for other IT use cases (e.g., security, application management, change management, operations management, and more).

Splunk meets auditor requirements for log review, audit trail collection, reporting and file integrity monitoring. In addition, Splunk will empower operations staff and developers too, through access to production data logs without requiring logging into production systems.

With the strong exponential growth of the IT infrastructure within the enterprise, what technological challenges do you face?
Splunk faces the same challenges as other vendors in this space, but we feel we are better positioned than the traditional IT security and management solutions to address them. Delivering customers speed and scale within this complexity are probably two of the most significant challenges.

Today’s IT infrastructures feature a mix of old and new technologies. Common examples of this are organizations adopting virtualization and maintaining custom applications – both strategic and both adding to IT complexity. Virtualization represents great efficiencies but it comes with a price, a dramatic increase in the IT data generated (metrics, logs, events etc.) and a steep learning curve (transient sessions, persisted data, storage implications, security and compliance). Custom applications are often strategic differentiators but introduce massive custom logs that are difficult for automating troubleshooting and compliance. Traditional siloed technologies for managing and securing IT weren’t designed for either of these common elements in IT.

Splunk’s ability to universally index, without custom connectors or a database on the backend, time-series IT data (including virtualized environments and custom applications) leads to faster problem resolution and incident investigations. Implemented on MapReduce, Splunk’s massive IT data indexing capabilities provide the industry’s fastest, most scalable solution for time series-based, unstructured IT data. Because of this distributed footprint, Splunk can scale to terabytes of data per day running on commodity hardware.

Splunk also offers apps that enable users to do more with it. What are some of your favorites?
Splunk for Splunk for UNIX (free App), Splunk for Windows (free App), Splunk Technology Partner Apps (free Apps), and Enterprise Security (premium App) are the most popular Splunk Apps.

The ability to easily create new Apps using the Splunk 4 App Framework should dramatically expand this list and broaden the content developed (new Apps) to run on the Splunk engine. Splunk for *NIX, and Splunk for Windows are Apps designed to help their administrators (and teams) manage and monitor their environments. They feature pre-packaged reports, dashboards, searches and alerts designed to get them running and productive faster than solely using the Splunk Search interface.

Splunk Technology Partner Apps are free Apps purpose-built by Partners (e.g., F5, BlueCoat, Cisco) to extend their product’s capabilities (performance, troubleshooting, monitoring, reporting) by correlating external IT data with their data using the Splunk engine.

Splunk Premium Apps: Enterprise Security provides pre-built capabilities for specific security use cases, accelerating the learning curve and delivering domain-specific capabilities.

Splunk Enterprise Security App is an integrated set of security capabilities consisting of packaged correlations, searches, reports, alerts, dashboards and workflow actions integrating a wide variety of security use cases including:

• Security event management
• Security information management
• Incident response
• Forensics
• Governance controls
• Compliance reporting
• Log management