How to build a hook syscall detector

Pedro Tavares
May 17, 2022 by
Pedro Tavares

Windows API calls are often hooked by AV and EDR systems by using inline patching approaches to find strange behaviors or malicious artifacts

FREE role-guided training plans

FREE role-guided training plans

Get 12 cybersecurity training plans — one for each of the most common roles requested by employers.

Windows API hooking

Windows API hooking is a popular technique used to instrument and modifies the behavior and execution flow of syscalls. For instance, this technique is largely used by AV and EDR systems to determine if a piece of code is suspicious.

More technically, a hook can be compared to a proxy — where all or a specific group of syscalls such as CreateFile(), ReadFile(), OpenProcess(), VirtualAlloc() and so on, could be intercepted and inspected to validate if the intent of the behavior is suspicious or not. [CLICK IMAGES TO ENLARGE]

Figure 1: High-level diagram of a hooked Win API call (source).

In detail, AV and EDR vendors are taking advantage of userland APIs by hijacking the definitions of the functions in Windows DLLs, such as kernel32/kernel base and ntdll (source). The process is quite simple: a jmp instruction is added to the execution flow when a specific and predefined syscall is invoked, such as CreateFileA(). 

The workflow is then changed and redirected to the proxy DLL (EDR.DLL) that will perform validation tasks and return the flow to the original DLL (KERNEL32.DLL). With this technique in place, the proxy DLL can validate the program behavior and the input parameters and thus monitor all the execution. The redirection process we mentioned before is also called a detour or trampoline within the security landscape.

The art of unhooking

The reverse process of hooking a syscall is unhooking it. The jmp must be restored, and we need to know the initial state before the hook process. This process is typically simple, and we just need to check the untouched DLL present on the disk: c:\windows\system32\kernel32.dll. The Relative Virtual Address (RVA) can be observed on the Export Address Table (EAT), in this case: 00022080.

Figure 2: RVA addresses of the CreateFileA() call fom kernel32.dl.

If we convert the RVA to the raw address (the physical one) before it is executed in memory, we will get the original address. If the DLL in memory doesn’t match that value, it is hooked.

How to build the hook syscall detector

The overall process of detecting a hooked function is composed of several steps, namely:

  1. Running the exported functions of the NTDLL.DLL.
  2. Reading the first 4 bytes and checking if it starts with 4c 8b d1 b8.
    1. If it matches, the syscall is not hooked.
    2. If it does not match, the function is probably hooked with some exceptions mentioned below.

False positives to consider in this experiment:

Figure 3: Hook functions false positives.

The source code to implement a simple hook syscall detector can be observed below (credits to ired.team blog).

Figure 4: Simple implementation of a hook syscall detector (source).

After compiling the source code and executing the binary on a Windows machine with an AV/EDR installed, we can observe all the hooked functions, with the exception of the NtGetTickCount() call present on the exception list from Figure 3.

Figure 5: Hooked functions by EDR system and false positive.

What should you learn next?

What should you learn next?

From SOC Analyst to Secure Coder to Security Manager — our team of experts has 12 free training plans to help you hit your goals. Get your free copy now.

Learn about Windows API and hook syscall

We learned how EDR/AV systems hook windows API calls in runtime. This process can be easily reversed by comparing the RVA addresses with the original ones on the disk.

With this simple premise in place, building a hook syscall detector is possible, but always keep in mind that some of the detected changes in memory can be false positives from Windows OS. This process can be applied when malicious software is running and detecting changes on the Windows DLLs — a process very similar to that executed by AV/EDR systems.



Pedro Tavares
Pedro Tavares

Pedro Tavares is a professional in the field of information security working as an Ethical Hacker, Malware Analyst and a Security Evangelist. He is also Editor-in-Chief of the security computer blog seguranca-informatica.pt.

In recent years, he has invested in the field of information security, exploring and analyzing a wide range of topics, such as malware, reverse engineering, pentesting (Kali Linux), hacking/red teaming, mobile, cryptography, IoT, and security in computer networks. He is also a Freelance Writer.