Reverse engineering

Using SetWindowsHookEx for DLL injection on windows

Dejan Lukan
June 3, 2013 by
Dejan Lukan

Note: all the code examples can be found on my Github profile under visual-studio-projects accessible here: https://github.com/proteansec .

Become a certified reverse engineer!

Become a certified reverse engineer!

Get live, hands-on malware analysis training from anywhere, and become a Certified Reverse Engineering Analyst.

In this tutorial, we'll take a look at the DLL injections by using the SetWindowsHookEx method. I haven't found a good blog post on the subject, which is the reason I decided to write one that will cover that and serve as a reference for future security researchers trying to understand this method. First, I'll explain what we'll actually be doing throughout the article, so you may understand the method before looking at an actual example.

Let's take a look at the picture below, which clarifies what we'll be doing next. At first, the picture might seem daunting, but we'll explain it in a bit. In the middle of the picture we can see the victim.exe process (in purple color), where we want to inject a malicious DLL. When the DLL is injected into the process it's DllMain function will be called where we can do whatever we may want. Usually, we want to change the IAT import table of the process, so whenever the process calls some function, it will end up calling some other function: our malicious function.

We've figured that if we're able to inject our DLL into the victim.exe process, we would practically be able to do anything in the process' address space. The first thing that we need is malicious DLL file, which we must code ourselves. We must write the code that does exactly what we want the DLL to do when it's injected into the process's address space. To inject the DLL into the process's address space, we'll use the SetWindowsHookEx function. To be able to use that function, we first need another program program.exe, which we can see on the picture below. That program needs the action we'll be hooking and the inject.dll, which is our malicious DLL we'll be injecting into the victim.exe's process space. The program.exe must first get a handle to our DLL and then get address of one of the exported functions inside that DLL. This means that we need to code our DLL is such a way it exports at least one function. After that, the program.exe will be calling SetWindowsHookEx function passing it the action to be hooked as well as the address of the exported function inside inject.dll.

The SetWindowsHookEx function will install the hook routine into the hook chain of the victim.exe process, which will be invoked whenever certain event is triggered. In our case, the event that needs to be triggered is the action that we've inputted into the SetWindowsHookEx function. Later on, we'll be using the WH_KEYBOARD action, which means that whenever we'll press some key inside the victim.exe process, the previously obtained exported function in the injected DLL will be called. When the event occurs, the OS first checks whether the required DLL is already loaded in to the process's address space. If it isn't, then the OS must load (read inject) the DLL into the process's address space upon which the DllMain function of the DLL is called. After that, the exported function we passed to the SetWindowsHookEx is also called to handle the triggered event—in our case a key press. On all subsequent key presses, the DLL need not be reloaded, because it's already loaded in the process's address space. This effectively enables us to do whatever we want in the hooked address space of the program.

The SetWindowsHookEx Method

The SetWindowsHookEx installs a hook routine into the hook chain, which is then invoked whenever certain events are triggered. Let's take a look at the function syntax (note that the picture was taken from [4]):

  • The parameters to the SetWindowsHooEx functions are as follows (summarized after [4]):
  • idHook: the type of hook to be installed, which can hold one of the following values:
    • WH_CALLWNDPROC
    • WH_CALLWNDPROCRET
    • WH_CBT
    • WH_DEBUG
    • WH_FOREGROUNDIDLE
    • WH_GETMESSAGE
    • WH_JOURNALPLAYBACK
    • WH_JOURNALRECORD
    • WH_KEYBOARD
    • WH_KEYBOARD_LL
    • WH_MOUSE
    • WH_MOUSE_LL
    • WH_MSGFILTER
    • WH_SHELL
    • WH_SYSMSGFILTER
  • If you would like to understand every single hook type, reference the documentation accessible at [4].
  • lpfn: a pointer to the hook function.
  • hMod: a handle to the DLL that contains the hook function.
  • dwThreadId: the identifier of the thread, which calls the hook function. If the parameter is 0, the hook will be called by all threads, so we don't have to restrict it to particular thread ID.

If the function succeeds, it returns a HHOOK handle to the hook function. Otherwise the function returns NULL. All of the above constants are written in the winuser.h header file, which is part of the Windows driver development kit, and can be seen on the picture below:

In our case, we'll be hooking the WH_KEYBOARD type of event, which will allow us to monitor keystroke messages.

Creating the inject.dll

The first step when injecting the DLL into some process's address space is creating the DLL itself. We won't go into the details on how to do that, since it's pretty much self-explanatory. We need to start a new project inside Visual Studio and select DLL when creating it. After that, we can change the dllmain.c source code into something that looks like the following:

[cpp]

#include "stdafx.h"

#include <stdio.h>

#include <windows.h>

INT APIENTRY DllMain(HMODULE hDLL, DWORD Reason, LPVOID Reserved) {