Log Management – Lifeblood of Information Security

The responsibility to protect sensitive private information is now legally mandated and has become a key focus for many regulations within multiple industries. Information security is vital to the success of an organisation’s day-to-day operations; and must be managed as a proactive and strategic business process throughout the entire enterprise – not an intermittent or point-in-time event for technology staff alone.

Love them or loathe them, log files play a central role in this. Logs are the lifeblood. They tell us the Who, the What, the Where, and the When. They give is insight. They give us answers. Very occasionally they might even make us laugh when the computer jargon points out the very obvious or make a simple fault sound incredibly serious.

Because of the widespread deployment of networked servers, workstations, and other computing devices, and the ever-increasing number of threats against networks and systems, the number, volume, and variety of computer security logs has increased greatly. This has created the need for computer security log management, which is the process for generating, transmitting, storing, analysing, and disposing of computer security log data.

Log files are critical to the successful investigation and prosecution of security incidents, therefore best practices recommend logging all events. However, enforcing such a policy can often overwhelm already overworked system administrators. The last thing you want is information overload. But it is true to say that logging only subsets is a risk. There are emerging solutions that do indeed gather a log for every event that takes place on the network, and provide an easy way to retrieve specific information if and when required.

Log files generally fall into one of three categories. Security software logs primarily contain computer security-related information, while operating system logs and application logs typically contain a variety of information, including computer security-related data.

Security Software

• Anti-Virus Software
• Intrusion Detection & Protection
• Remote Access Software
• Web Proxies
• Vulnerability Management Software
• Authentication Servers
• Routers
• Firewalls
• Network Devices

Operating Systems
Operating systems (OS) for servers, workstations, and networking devices (e.g., routers, switches) usually log a variety of information related to security. The most common types of security-related OS data are:

System Events. System events are operational actions performed by OS components, such as shutting down the system or starting a service. Typically, failed events and the most significant successful events are logged. The details logged for each event also vary widely; each event is usually timestamped, and other supporting information could include event, status, and error codes; service name; and user or system account associated with an event.

Audit Records. Audit records contain security event information such as successful and failed authentication attempts, file accesses, security policy changes, account changes (e.g., account creation and deletion, account privilege assignment), and use of privileges.

Operating systems and security software provide the foundation and protection for applications, which are used to store, access, and manipulate the data used for the organization’s business processes.

Applications
Some applications generate their own log files, while others use the logging capabilities of the OS on which they are installed. Applications vary significantly in the types of information that they log.

Account information such as successful and failed authentication attempts, account changes (e.g., account creation and deletion, account privilege assignment), and use of privileges. In addition to identifying security events such as brute force password guessing and escalation of privileges, it can be used to identify who has used the application and when each person has used it.

Usage information such as the number of transactions occurring in a certain period (e.g., minute, hour) and the size of transactions (e.g., e-mail message size, file transfer size). This can be useful for certain types of security monitoring (e.g., a ten-fold increase in e-mail activity might indicate a new e-mail-borne malware threat; an unusually large outbound e-mail message might indicate inappropriate release of information).

In determining which data is sufficient and appropriate to collect, organisations should implement processes that:

• Identify components and events that warrant logging.
• Establish the amount of data to be logged.
• Identify and establish mandated log retention timeframes.
• Implement polices for securely handling and analysing log files.

The issue of retention has become a difficult one for many organisations. Satisfying the reporting demands of government regulations and corporate security policies requires the retention of vast amounts of security data. Not only must you collect log and event data from security products like firewalls and identity management systems, auditors must also be able to go back several years to trace security violations. One effect of government regulations is that security information, including event logs and transaction logs, has now become legal records that must be produced when requested by legal authorities. This could potentially stretch data retention periods to the duration of the litigation process.

Penalties for non-compliance include monetary fines, civil liability and executive accountability. In some cases, such as with Sarbanes-Oxley, the statutes allow for fines that may reach into the millions of dollars. However, the largest penalties for non-compliance are likely to be the market-driven costs of having the company name associated with a security breach, and not being able to demonstrate reasonable security precautions with an acceptable compliance statement. The damaged trust relationship effects customer satisfaction, consumer confidence, and the organization’s ability to compete in the marketplace.

On top of retention requirements, log files must be secured and access restricted and monitored. In an attempt to conceal unauthorised access or attempted access, intruders will try to edit or delete log files. Efforts to secure log files should include:

• Encryption of data residing on database and in transit where necessary.
• Segregation of logged data to an independent server.
• Collection of data on Write Once Read Many (WORM) disks or drives.
• Secure storage of backup and destruction of log files.

Secure log files also assist in effective and timely identification and response to security incidents and to monitoring and enforcement policy compliance.

A good log management solution should provide a scalable and centralized process that can collect, normalise, aggregate, compress and encrypt log data from disparate sources such as routers, switches, firewalls, IDS/IPS, AV, SPAM/spyware, Windows, UNIX, and Linux systems to identify security breaches, hacker intrusion and or any other activity that could potentially be crippling valuable corporate assets. A good log management solution should also automate the process of producing reports, with relevant information that will indicate an anomaly or glitch. Having the system email these reports to your inbox at set intervals can save trouble and most importantly time.

A solution that automatically mines and manages that data can provide immediate insight into network activity, helping IT departments respond rapidly to security events and other network availability problems. Additionally, with stricter requirements imposed by best practices frameworks and regulatory legislation, companies must find more reliable ways of managing and securely archiving complete log data for compliance purposes and legal protection. Reporting requirements for security information are going to increase. Regulations are sure to call for log data from additional sources. Plan now for performance to handle streams of security information without impacting application performance and storage capacity that offers efficient growth paths as the enterprise storage requirements grow.

Log files may not be pretty, but they make fantastic partners, working tirelessly in the background, never complaining, always on top! Sometimes, they can be difficult to make sense of. A centralised log management system will undoubtedly help.