Types of rootkits

A rootkit is simply a set of tools that can maintain root privileged access to an operating system. A malware rootkit will usually carry a malicious code/software that is deployed secretly into the target system. 

To maintain backdoor access for the malware, rootkits can exploit background system processes at various privilege levels. In a Windows system, this usually means attacking at either the user mode or kernel mode.

The importance of understanding the attack vector

Contrary to popular perception, a rootkit does not compromise the system security on its own. The penetration of an operating system's defenses is handled by the method employed to infect the system. There are many ways to accomplish this today:

• Sending an infected file/Trojans as email attachments
• Creating malware apps masquerading as harmless banners/pop-ups on websites
• Using phishing attacks and other malware like keyloggers give hackers root access, which can then be used to inject the rootkit

A rootkit starts its dirty work once the vector has successfully compromised the system, giving it a chance to get into root privileged modes inside the OS. Once it gains access, depending on the loci of its attack, a rootkit can change installed software or even parts of the OS itself. This can make a rootkit infection difficult to detect, or even virtually invisible.

Understanding a rootkit's position in the OS

Windows has two modes for executing code — user mode and kernel mode. Of these, the kernel mode is the most important, as it acts as the direct link between the hardware and software code. This is the highest privileged mode, or "root" in a Windows system. 

But not all programs and processes require access to the full range of processing power and system hardware. These lower-level functions are handled at a mode with lower privileges — the user mode. 

In kernel mode, the OS runs codes directly to execute all the high-level tasks of managing the system. In contrast, user mode uses an Application Programming Interface (API) to indirectly execute codes. During the normal operation of the system, the API performs function calls to the kernel and receives information in return. 

A rootkit usually hijacks these function calls, either at the user mode or the kernel mode, depending on its design. 

User mode rootkits and DLL injection

In Windows, API hooking is a technique that is commonly employed to modify or change the flow of API calls. A user mode rootkit exploits this technique to inject malicious code into a program's memory while remaining concealed. 

For this to work, the rootkit has to insert its code into a commonly called process. In Windows, Dynamic Link Libraries (DLLs) are perfect for this, since they are called by many different programs installed in the OS. 

DLL injection is the method used by the rootkit user (hacker) to insert malicious code into various programs performing API calls in the operating system. There are several ways to achieve this:

• Modifying the registry values to add a new malicious DLL to the Windows system
• Using global hooking in Windows to affect all program API calls
• By creating a remote thread for the target process to add infected DLL to it

IAT hooking versus inline hooking

The main weakness of a user-level rootkit is that it lacks direct access to the OS kernel. So they use API hooking an exploit. There are two different techniques often employed here — IAT hooking and Inline hooking. 

In Windows, an Import Address Table (IAT) is used to streamline access to DLLs with their variable function addresses. The IAT hosts the function pointers of DLLs as they are loaded by the Windows loader during an API call. 

Using IAT hooking, a rootkit can make changes to the DLL function calls list, replacing existing functions with its own address. When an application makes an API call for that function, the rootkit code is loaded instead into the victim program's memory space.

In contrast, inline hooking directly makes modifications to function codes of specific programs. This is a more targeted and narrow approach than IAT hooking. But like IAT hooking, the aim here is to make the program API calls load the malicious code from the rootkit address. 

Kernel mode rootkits and hooking techniques

The use of DLL injection makes it relatively easier to detect a user mode rootkit infection. But kernel mode rootkits are a different beast altogether. It is located on the same level as most malware detection software and can evade detection by altering parts of the kernel code itself. 

A kernel mode rootkit can make changes to the system using several different hooking techniques. These include: 

SSDT hooking

In Windows, system calls to the kernel are managed according to the System Service Dispatch Table, or SSDT. This table contains the important memory addresses of the main functions that are called. If a rootkit gains kernel access, it can make changes to the SSDT, with similar results as in IAT hooking. 

IDT hooking

Windows kernel-mode also contains an Interrupt Descriptor Table (IDT) to handle any interrupts caused by a software or hardware. A simple example would be a hardware interrupt caused by the keyboard. A rootkit can use IDT hooking to collect important information, including keyboard data. 

Inline function hooking

Kernel rootkits can also use hooks not related to system tables. In inline hooking, the malicious code attacks specific parts of a target function in the kernel, forcing it to jump to a specific area in the system memory containing the rootkit code. 

Become a certified reverse engineer!

Get live, hands-on malware analysis training from anywhere, and become a Certified Reverse Engineering Analyst.

Start Learning

Key takeaways

• Rootkits use many different attack vectors and techniques to compromise a system's security and infect it
• Rootkits work by hijacking or hooking API function calls in an OS
• In Windows, they can do it at Ring-3 (user-level) and Ring-0 (kernel-level)
• User-level rootkits often use DLL injection to add malicious code to applications
• At kernel-level, they abuse SSDT and IDT function calls to make changes

Sources

• Rootkits, ENISA
• Sungkwan Kim, Junyoung Park, Kyungroul Lee, Ilsun You, Kangbin Yim, A Brief Survey on Rootkit Techniques in Malicious Codes