Hypervisor-based, hardware-assisted system monitoring

In the last few years, many different techniques have been introduced to analyse malicious binary executables. Most of these techniques take advantage of Virtual Machine Introspection (VMI), the process of analysing the state of a virtual machine from the outside.

On the one hand, many approaches are based on system emulators which enable a tight control over the program execution. Unfortunately, such approaches typically induce a huge performance overhead. On the other hand, there are approaches based on hypervisors.

Early implementations were hampered by the missing virtualizability of the x86 instruction set architecture: since the memory management unit (MMU) itself was not virtualized, memory separation needed to be enforced in software with the help of so-called ‘shadow page’ tables, an approach that again induced performance overhead. However, processor vendors have recently added hardware support for MMU virtualization, and modern CPUs offer so-called ‘two-dimensional paging’ to overcome such performance bottlenecks.

This presentation from Virus Bulletin 2013 demonstrates how this processor feature can be utilized to implement a binary analysis framework. More specifically, the authors introduce an approach to monitor code execution based on the concept of Currently eXecutable Pages (CXP), i.e. they precisely control which memory pages are currently executable to enable the interception of intermodular function calls and their corresponding returns.

When an interception occurs, they apply VMI to deduce runtime information such as function parameters. To demonstrate the practical feasibility of the proposed approach, they implemented ‘VMMInspector’, a framework for binary analysis on 64-bit machines and Windows 7. In several case studies they present different usage scenarios for that framework. Amongst other applications, they demonstrate how the kernel rootkit TDSS/TDL4 can be analysed in an automated way.