Hacking

How to build a hook syscall detector

Windows API calls are often hooked by AV and EDR systems by using inline patching approaches to find strange behaviors or malicious artifacts. 

What should you learn next?

From SOC Analyst to Secure Coder to Security Manager — our team of experts has 12 free training plans to help you hit your goals. Get your free copy now.

Get Your Plan

Windows API hooking

Windows API hooking is a popular technique used to instrument and modifies the behavior and execution flow of syscalls. For instance, this technique is largely used by AV and EDR systems to determine if a piece of code is suspicious.

More technically, a hook can be compared to a proxy — where all or a specific group of syscalls such as CreateFile(), ReadFile(), OpenProcess(), VirtualAlloc() and so on, could be intercepted and inspected to validate if the intent of the behavior is suspicious or not. [CLICK IMAGES TO ENLARGE]

Figure 1: High-level diagram of a hooked Win API call (source).

In detail, AV and EDR vendors are taking advantage of userland APIs by hijacking the definitions of the functions in Windows DLLs, such as kernel32/kernel base and ntdll (source). The process is quite simple: a jmp instruction is added to the execution flow when a specific and predefined syscall is invoked, such as CreateFileA(). 

The workflow is then changed and redirected to the proxy DLL (EDR.DLL) that will perform validation tasks and return the flow to the original DLL (KERNEL32.DLL). With this technique in place, the proxy DLL can validate the program behavior and the input parameters and thus monitor all the execution. The redirection process we mentioned before is also called a detour or trampoline within the security landscape.

The art of unhooking

The reverse process of hooking a syscall is unhooking it. The jmp must be restored, and we need to know the initial state before the hook process. This process is typically simple, and we just need to check the untouched DLL present on the disk: c:\windows\system32\kernel32.dll. The Relative Virtual Address (RVA) can be observed on the Export Address Table (EAT), in this case: 00022080.

Figure 2: RVA addresses of the CreateFileA() call fom kernel32.dl.

If we convert the RVA to the raw address (the physical one) before it is executed in memory, we will get the original address. If the DLL in memory doesn’t match that value, it is hooked.

How to build the hook syscall detector

The overall process of detecting a hooked function is composed of several steps, namely:

1. Running the exported functions of the NTDLL.DLL.
2. Reading the first 4 bytes and checking if it starts with 4c 8b d1 b8.
1. If it matches, the syscall is not hooked.
2. If it does not match, the function is probably hooked with some exceptions mentioned below.

False positives to consider in this experiment:

Figure 3: Hook functions false positives.

The source code to implement a simple hook syscall detector can be observed below (credits to ired.team blog).

Figure 4: Simple implementation of a hook syscall detector (source).

After compiling the source code and executing the binary on a Windows machine with an AV/EDR installed, we can observe all the hooked functions, with the exception of the NtGetTickCount() call present on the exception list from Figure 3.

Figure 5: Hooked functions by EDR system and false positive.

Become a Certified Ethical Hacker, guaranteed!

Get training from anywhere to earn your Certified Ethical Hacker (CEH) Certification — backed with an Exam Pass Guarantee.

Learn More

Learn about Windows API and hook syscall

We learned how EDR/AV systems hook windows API calls in runtime. This process can be easily reversed by comparing the RVA addresses with the original ones on the disk.

With this simple premise in place, building a hook syscall detector is possible, but always keep in mind that some of the detected changes in memory can be false positives from Windows OS. This process can be applied when malicious software is running and detecting changes on the Windows DLLs — a process very similar to that executed by AV/EDR systems.

 

Sources:

• Hook detector, Ired.team
• Hook syscalls, GitBook Seguranca-Informática