Debugging Fundamentals for Exploit Development

Stephen Bradshaw
March 1, 2011 by
Stephen Bradshaw


This is a basic exploit writers tutorial for OllyDbg, a 32 bit assembler level analyzing user mode debugger for Windows. Version 1.10 of OllyDbg is used, but the majority of the techniques discussed should also be applicable to other versions of OllyDbg (including version 2) as well as to the Immunity Debugger, which is based on OllyDbg.

What should you learn next?

What should you learn next?

From SOC Analyst to Secure Coder to Security Manager — our team of experts has 12 free training plans to help you hit your goals. Get your free copy now.

What should you learn next?

What should you learn next?

From SOC Analyst to Secure Coder to Security Manager — our team of experts has 12 free training plans to help you hit your goals. Get your free copy now.

My intention with this tutorial is to provide a reference to those who want to learn how to use the OllyDbg debugger to facilitate the writing of basic to intermediate level software exploits. In the past, my tendency has been to intersperse debugger usage tips into tutorials designed to teach exploitation skills. I found however, that this approach tended to take attention away from the main purpose of the article – to teach how to exploit software. This tutorial is intended to build a foundation of the basic skills necessary before the more complex skills of exploit writing can be effectively taught. No previous debugger knowledge is required to follow along, but if you already have some skills with a debugger you can skip any areas with which you are already familiar.

This tutorial uses the deliberately vulnerable program Vulnserver as its debugging target. You should obtain a copy of Vulnserver from the following link and extract the archive to your hard disk in order to follow along with the steps in this tutorial. When running this program, make sure that your firewall allows the necessary traffic, but ensure that you don't grant access from untrusted networks like the Internet. Read the details provided at the download page for more information. Download Vulnserver from here: http://grey-corner.blogspot.com/2010/12/introducing-vulnserver.html

In addition, you should also install Perl on your Windows system, as this guide will make use of a number of Perl scripts in order to trigger certain actions within the debugger. ActiveState has a free Perl distribution called ActivePerl you can use for this purpose. All of the commands in this tutorial referencing Perl scripts will be provided under the assumption that they are being run from the same system that is running OllyDbg. You can run the scripts from another location if you wish, but you will need to modify some of the command line options accordingly. ActivePerl can be downloaded from here: http://www.activestate.com/activeperl .

You will also, obviously, need a copy of the OllyDbg debugger, version 1.10. This can be downloaded from here:


Before beginning this tutorial, we will be referring to a vulnerability we discovered in an earlier exercise. You may want to familiarize yourself with how the vulnerability was discovered here:



This tutorial is broken up into two articles and the first article will cover the following subjects:

We cover the following subjects in the second article:

Starting the Debugger

Before starting OllyDbg, you need to ensure that you are using an account that has the appropriate privileges, generally local Administrator equivalent.

To start OllyDbg on Windows XP/2003 or below, simply double click on the OllyDbg executable or shortcut, just as you would any other Windows application. If you are running OllyDbg on Windows Vista or above, you will need to ensure that you run it using the "Run as Administrator" option, available when you right click on the executable or shortcut. If you don't run the program with Administrative privileges, you won't be able to properly perform your debugging tasks.

Opening and Attaching to the debugging target application

Once OllyDbg has been opened, the first thing you will want to do is to access the target application you want to analyze within the debugger.

There are two main primary ways to achieve this:

  • By opening the target executable from disk using the File->Open menu option, or
  • By attaching to an already running program using the File->Attach menu option.

What's the difference between the two methods? By choosing to open the program directly from disk you can control execution of the program from the very start, whereas when you attach to an already running program you can only take control from the point at which you perform the attach operation. For exploit writing this distinction may not actually matter, as long as you actually start the process sufficiently early before the actual crash you want to attempt to exploit.

Generally, if I have a choice, I usually "open" executables instead of attaching to their running processes, because this allows me to catch all of the program's operation, and to restart it easily from within the debugger interface. Sometimes, however this may not be a convenient option, like in cases where you are trying to exploit an executable running as a Windows service, and in this case attaching may work sufficiently well. Just be aware that if you do this you may not be able to restart the program from within the debugger — you may need to use something like the Services control panel if the executable runs as a service.

You may also find that in some cases, programs don't like being "attached" to when they are running, so it is wise to perform some testing to ensure that the program operates normally after you attempt this. To do this, you should let the program run and confirm it still behaves as it usually does after you have "attached" the debugger.

Now, open OllyDbg, if you haven't already, and try using the File->Open menu option to open vulnserver.exe from where it is stored on your hard disk.

You should see something like the following.

Now use the Debug->Close menu option to close this debugging session, and hit Yes if the "Process still active" warning message appears. This is essentially just telling you that you are about to terminate an active process — and in this case this is exactly what we want to do. You can turn this warning off under the Options->Debugging Options menu option, by selecting the Security tab and unticking the Warn when terminating active process option. I highly recommend you do this, as this warning can get tiresome very quickly.

Now, navigate to the location on disk where vulnserver.exe is stored using Windows Explorer and double click on it to run it. Now switch back to OllyDbg, and select the File->Attach menu option. A list of running processes will appear. Select vulnserver from the list (it might help you find it if you sort the list by name first) and hit the Attach button.

You should now see the following.

Now use the Debug->Close menu option again to close the debugging session.

If you compare the two methods of creating the debug session by checking the bottom left hand corner of the screen in the debugger, you will notice that "opening" the program seems to enter the program at the "Program entry point", and "attaching" to it enters by having the "Attached program paused at ntdll.DbgBreakpoint." We can see right there that there is a difference in the way that the session begins.

In both cases the program was automatically "Paused" after you started the debugging session (see the Paused text in the bottom right hand corner). We will go into more detail about what this actually means later on in this tutorial.

The OllyDbg CPU view

Use the File->Open menu option to open up vulnserver.exe.

You should now be greeted by the OllyDbg CPU view, which is the default OllyDbg view and the one we will be spending the majority of our time in as exploit writers. The view is broken into four main panes, as shown in the screenshot below.

The pane in the top left corner of the screen shows the actual instructions of the program we are about to run. I will be referring to this as the CPU instruction or disassembler pane (since it shows instructions disassembled from their binary format).

From left to right, the columns in this pane show:

  • the memory address of each instruction,
  • the hexadecimal representation of each byte that comprises that instruction (or if you prefer, the "opcode" of that instruction),
  • the instruction itself in X86 assembly language, shown (by default) in MASM syntax, and finally
  • an information/comment column which shows string values, higher level function names, user defined comments, etc.

I'll delve deeper into what this assembly stuff is and how to interpret it in the next section, but for now just realize that this top left hand pane is where the assembly instructions are shown.

The pane in the top right hand corner of the screen shows the value of various registers and flags in the CPU. I will be referring to this as the register pane. These registers are small storage areas within the CPU itself, and they are used to facilitate various operations that are performed within the X86 assembly language. I will cover the purpose of these registers in the following section in this tutorial.

The pane in the bottom left hand corner shows a section of the programs memory. I will be referring to this as the memory dump pane. Within this pane you can view memory in a variety of different formats, as well as copy and even change the contents of that memory.

The pane in the bottom right hand corner shows the stack. I will be referring to this as the stack pane. The left hand column in this pane contains memory addresses of stack entries, the second column contain the values of those stack entries, and the right hand column contains information such as the purpose of particular entries or additional detail about their contents.

There is also an optional third column in the stack pane that will display an ASCII or Unicode dump of the stack value — this can be enabled by right clicking on the stack pane and selecting either "Show ASCII dump" or "Show UNICODE dump." The next section contains some more detail on the purpose of the stack.

The 20 second guide to X86 assembly language for exploit writers

If you want to write software exploits, you need to understand assembly.

Now, what I refer to as assembly is actually a generic term for low level programming language that operates only one step above basic machine code (1s and 0s) that is natively executed by a CPU. Because assembly is so closely related to the code a CPU directly executes, it is actually CPU specific. That means different families of CPU have different assembly languages. You can't, for example, run assembly code written for the Sun SPARC CPU architecture on a X86 processor, a IA-64 bit processor or a MIPS processor. These CPUs all have different assembly languages. OllyDbg, and this tutorial, will focus specifically on the 32 X86 architecture used on the vast majority of "common use" 32 bit systems in the world.

This section is only intended to provide a very brief guide to this language — hopefully just enough to get you started on your exploitation journey, but almost definitely not enough to cover all possible scenarios you may face in this area. You should expect to have to do some research of your own on this topic to get really comfortable.

I also will not be covering how to write assembly, just how to interpret it in OllyDbg, and I am going to be simplifying things in some cases for the sake of brevity. I would suggest that that anything written in this section not be taken as gospel, but more as a general set of guidelines to help you gain a basic level of understanding of assembly so you can get going with exploit writing.

This section is a reasonably long one, and I have broken it up into a number of sub-sections as follows:

  • Syntax and other boring stuff
  • Registers and flags
  • The stack
  • Assembly Instructions

Syntax and other boring stuff

Before we get into the nitty gritty of things here, I will just cover off on some detail about the syntax of the assembly instructions you will see in Ollydbg, and also discuss an important point about the endian order on the X86 processor.

OllyDbg, by default, uses the MASM syntax when it disassembles the raw machine code instructions of an executable into the more digestible assembly code you can see in the CPU view. MASM syntax, when there are two operands to an instruction, places the destination operand first and the source operand second. As an example, the following command will copy the contents of the register EAX to the register ECX

mov ECX, EAX

I will go into more detail about some of the particular instructions available later on in this guide, but for now just remember that in MASM syntax the destination for an instruction comes first and the source second. You can choose a different syntax for OllyDbg to use in the Options->Debugging options menu, under the Disasm tab. I will be using MASM syntax in this tutorial.

One other thing to be aware of here is the endian order of the X86 processor — little endian. This essentially means that certain values are represented in the CPU, left to right, from least to most significant bytes.

Bytes are shown in OllyDbg as two digit hexadecimal numbers with possible values of 0 to F (0123456789ABCDEF) for each digit, with the decimal equivalent of the digits A-F being 10-16. The highest possible single byte value is FF (sometimes preceded by "0x" and written as 0xFF to denote that hexadecimal numbering is being used) which is equivalent to 255 in decimal.

If you are at all unfamiliar with hexadecimal numbering I suggest you do some reading on the subject so you get a good understanding of how it works, because this is a critical subject for exploit writing work (Google "hexadecimal numbers" for some good references). You can easily convert between hexadecimal and decimal formats using a good computer based calculator, like the Windows Calculator or something like kcalc on Linux. Just switch the calculators view modes until you see the letter keys appear, and then use the appropriate controls, which will probably be labeled something like "Hex" and "Dec" to switch as needed.

As an example of how the little endian order works, if we want to represent a hexadecimal number such as 12ABCDEF in little endian order, we would actually write the number as EFCDAB12. What we have done is break the number into its individual component bytes:




And then we reverse the order of those bytes and put them back together.




Remember here that we are only reversing the order of the bytes, not the digits that the bytes are comprised of. You would not reverse the order of the 1 and 2 within the first byte of 12 for example. Have a look at this a few times to make sure you understand it. This reversing of the bytes is something that you need to understand when you come to actually writing buffer overflow exploits, as having a firm grasp of this will allow you to be sure that the values you insert into the code via your exploits are interpreted correctly by the CPU.

Registers and Flags

There are nine different 32 bit registers shown in OllyDbg (in the registers pane — top right hand pane of the CPU view). As mentioned earlier, these registers are storage areas inside the CPU that can each hold four bytes (32 bits) of data. While these registers all have nominal purposes (being used for particular things by most programs) the majority of these registers can be used to store any old value that fits. For practical purposes, as an exploit writer you can generally think of most of the registers only as very small storage areas. There are two important exceptions to this however, and these are the EIP and ESP registers, which have very specific purposes that you do need to be aware of.

The EIP register is known as the instruction pointer, and its purpose is to "point" to the memory address that contains the next instruction that the CPU is to execute. Assuming you have OllyDbg open with vulnserver.exe being debugged, looking at the EIP register should show a value that matches the memory address of the selected entry in the top left hand pane of the OllyDbg CPU view. When the debugged program is allowed to continue, this is the first instruction that the CPU will run. All of the assembly focused exploitation techniques focus on various different ways of making this EIP register point to somewhere of the attacker's choosing, so their own code can be run.

The ESP register is known as the stack pointer, and this contains a memory address that "points" to the current location on the stack. Looking at OllyDbg again, the value in ESP should correspond with the address of the highlighted value in the stack pane in the bottom right hand corner of the CPU view. See the next section for more details on the operation of the stack.

The flags register is a collection of single bit values that are used to indicate the outcome of various operations. You can see the values of the flags just below the EIP register in the top right hand pane of OllyDbg, the C, P, A, Z, S, T, D, and O designators and the numbers (0 or 1) next to them show whether each particular flag is on or off. The flag values are mostly used to control the outcomes for conditional jumps, which will be discussed a bit later on.

Operations to set the values of the registers will replace any existing values currently being held. It is however, possible to set (or access) only part of a value of a register by the use of subregisters.

If you want more information on these registers and flags, as well as subregisters you can check here: http://msdn.microsoft.com/en-us/library/ff561502%28v=vs.85%29.aspx

The values of the registers and flags we have discussed are located in the top right hand pane of the CPU view in OllyDbg. In the screenshot above, the topmost red rectangle is surrounding the registers, and the red rectangle immediately below it is surrounding the flags.

The Stack

The stack is a special memory structure that is used to make the operation of functions more efficient. Functions, for those unfamiliar with programming terms, are sections of code that perform a specific purpose that can be called from within other code. When the operation of a function is complete, it hands control back to the calling code, which should continue executing from where it left off. I say "should continue executing from where it left off" because stack overflows, one of the most common and simple to exploit code execution vulnerabilities around, actually subvert this process to gain control of a program.

The stack is comprised of a number of 32 bit values, piled on top of each other like a stack of plates. It is a LIFO (Last In First Out) structure, which means that only the top most entry can be accessed (or taken off the stack), and any new entries added must be added on top of existing ones. As an example, if you want to access the third entry on the stack using stack management processes, you can't just go straight at it; you have to remove the two entries above it first. The process of reading an entry on the stack also usually involves removing it from the stack.

The stack takes up a defined section of memory space and grows downwards to smaller addresses in memory from its base address. As an example, if the base address of the stack was 2222FFFF, the bottom stack entry would be at address 2222FFFC, the next entry would be at 2222FFF8, the next at 2222FFF4 and so on. The top entry on the stack, the one that can be accessed, is pointed to by the stack pointer register, ESP. As stack operations are performed, such as adding or removing entries, the ESP register will be automatically updated to reflect these changes with the value of the register getting smaller if new entries are added, or larger if entries are removed (remember that the stack grows downward — to smaller addresses.) By the same token, changing the value of the ESP register by other means (such as by directly setting it) will also change the current position on the stack.

The stack is used to hold a lot of interesting things, including the local variables of functions, the return addresses that functions should return to once they are complete as well as certain exception handling addresses.

In OllyDbg, a representation of the data in the stack is shown in the bottom right hand pane of the CPU view, inside the area enclosed in red in the screenshot above. Each line in the section of the screenshot in red represents an entry on the stack.

Assembly Instructions

Now we have taken the 10,000 meter view of the registers and the stack, we can examine some of the instructions in the assembly language that can be used to manipulate them. There are a large number of these instructions, but I am only going to focus on just a few, the ones that exploit writers absolutely must know. These particular instructions may not suffice for all of your exploiting activities, but I have found that they are the most commonly used for exploitation, so they are a great place to start.

Something to note about these instructions is that each instruction has an equivalent opcode, and where there are different variants of an instruction, it will have a different opcode to allow the CPU to differentiate between them. If you look in the second column of the disassembler pane of the CPU view in OllyDbg, you will see the opcodes for each instruction being executed.

I will cover the following instructions in this section:

  • JMP
  • CALL
  • POP
  • PUSH
  • RETN
  • INT3
  • NOP


By default, the CPU will execute its instructions one after the other, starting with the first instruction, then continuing with the second, and so on. A JMP is one of a number of instructions that tells the CPU to move to another location with the code, and to continue execution from there. There are a number of different types of JMP instructions, some that "jump" a distance backwards or forwards (relative to the current location in memory), and others that jump to absolute locations in memory (irrespective of where the code currently being executed is located). There are also conditional jump instructions (referred to in many instruction references as Jcc instructions). These jump to another location only if a certain condition is met, usually determined based on the value of one or more flags which are set by various other instructions. These conditional jump instructions all begin with the letter J, and some examples are JE (Jump if equal), JNZ (Jump if not zero) and JB (Jump if below). All conditional jumps are relative jumps and these are usually used to perform branching within programs e.g. the code executes one way if a certain condition is true, or a different way if it is not.

One of the most common JMP instructions used in exploit development is the short jump. The assembly instruction for this type of JMP is normally written with an absolute memory address following the "JMP SHORT"; however the opcode value is not specified in absolute terms, but rather as a value relative to the address of the next instruction in memory. (Please note I am preceding all opcode values in this tutorial with "x" to denote them as hexadecimal).

xEBx08 JMP SHORT [Address of the Next Instruction + 8]

The opcode for the short JMP is "xEB" followed by a single byte value that controls the distance of the jump, which is made relative to the start of the next instruction in memory. The example above will jump forward 8 bytes. We can substitute any hexadecimal value to "x7F" to allow jumping up to 127 bytes relative to the address of the next instruction.

We can also use the short jump instruction to jump backwards from our current position, by using the values above "x7F". We essentially count down from "xFF" to "x80" in order to jump up to 128 bytes backwards, relative to the address of the next instruction. Using a value of "xFF" jumps backwards one byte, "xFE" jumps back two bytes, "xFD" three bytes and so on. Remember that the jump is relative to the address of the next instruction in memory, so using an instruction of "xEBxFE" will jump back to the start of its own instruction, creating a loop.

Jumps can also be performed directly to the locations held by various registers, such as the ones shown below









If any of these jump instructions are used, execution will jump to the memory address specified by the value of the given register. So, if you run an instruction of JMP EAX, and the EAX register holds the value 00401130, execution will jump to the instructions located in memory at the address 00401130.

There are a number of other jump instructions you can use, which you can find out by referring to an Instruction Set Reference, such as the one linked to at the end of this section. The main thing you should remember about the JMP instruction is that it allows you to redirect code execution.


A CALL instruction is used to call a procedure. When a CALL is made execution jumps to a given address in memory, the code beginning at that location is executed until a RETN instruction is reached, and then execution should return to the instruction in memory immediately following the initial CALL instruction. The ability to return back to the location of the initial CALL statement is achieved by using the stack to store the address where code execution needs to return once the RETN instruction is reached. Given that this "return address" is stored on the stack it is in an ideal location to be overwritten by a stack overflow (but this is a topic for another time). For now, you can forget about what happens when you RETN from the CALL instruction, and just concentrate on the fact that that like JMP, CALL is another instruction we can use to redirect the CPU's code execution path.

As with the JMP instruction, there are a number of different types of CALL instructions, however the most interesting from our perspective are the ones that redirect execution to a location specified by a CPU register. Here are some examples:









As with the JMP instructions discussed previously, a CALL EAX instruction will redirect code execution to the memory address held by the EAX register. Use of CALL can also be a clever way to find out your current position in memory when writing shellcode, as the address of the next instruction will automatically be added to the top of the stack when the CALL instruction is executed.


The POP instruction works with the stack and allows you to "pop" the top entry from the stack and place it into the location specified by the provided operand, which will either be a register or a memory location. The POP instruction, along with its companion instruction PUSH are incredibly useful to us as exploit writers, as they allow us to manipulate the stack and to shift various values around in memory.

Some useful example POP commands are:









Running an instruction such as POP EAX will remove the value from the top of the stack and place it into the EAX register. The current position on the stack will be adjusted via adding 4 bytes to the value of the ESP register. (As discussed earlier in the section on the stack, the stack grows downward, making the value of ESP larger makes the stack smaller)


The companion instruction to POP is PUSH, which adds new entries to the stack. As with POP, the first operand can be either a memory address or a register, and the value from that location will be pushed onto the stack. Some useful POP commands are.









POP EAX will take the value of EAX and put it onto the stack, automatically shrinking the value of the ESP register by four bytes in the process.


Sometimes written as RET, this instruction takes the top value on the stack, interprets it as a memory address and redirect code execution to that location. Simple. An optional operand can be used to release additional bytes from the stack by modifying the value of the ESP register when the instruction is executed. RETN is often used in combination with the CALL command.


Imagine what you could do if you could modify the return address on the stack before the RETN instruction was executed in a program...


An INT3 instruction found within the code causes a pause of execution when run in the debugger — it's the assembly instruction equivalent of setting a breakpoint in the debugger (see the next section for more details about what a breakpoint is).


Using these characters in the data you send to an application is a great way to pause execution during the actual process of crafting an exploit. If placed at a location where you anticipate execution to be redirected, it will allow you to confirm that redirection is occurring as expected, or if placed immediately before shellcode, it will give you an opportunity to examine that the shellcode has not been mangled before it gets executed. Think of it as providing you an extra level of control which can assist in identifying and resolving problems during the exploit development process. Once any such problems are ironed out, the breakpoints can of course be removed.

You can see an example of a use of the INT3 instruction in the upcoming section on editing code within the debugger.


NOP stands for No Operation, and is basically an instruction that does... nothing.

x90 NOP

So what is the point of an instruction that does nothing? Well, as an exploit writer, we can make use of the fact that this instruction takes up space and won't change register values or the stack in a few ways.

The most common use of the NOP instruction is in a NOP slide (or sometimes NOPsled). When you have an opportunity to redirect code execution, but you only know the general area in which you will end up, you can stick a NOP slide in that area on which to land your code execution. Each consecutive NOP instruction from the point of landing will be executed until the end of the slide is reached. At this point, any shellcode placed at the end of the slide will end up being executed.

Another use of NOP instructions is to provide a buffer zone in memory for certain types of shellcode. Some shellcode, especially encoded shellcode, can require some working space in memory in order to operate properly. A sufficient number of NOP instructions, placed in the correct location in the data sent from your exploit, can provide this working space.

One important thing to keep in mind here is that certain other instructions can be used to substitute for the traditional NOP when the NOP instruction itself is not suitable. For example when character filtering prevents the "x90" byte from being sent, or for when the opcodes used for the instruction must also be able to be interpreted as a memory address within a certain proscribed range.

In this case, any instruction which fits the appropriate criteria for the exploit and does not change a register value or the stack in a way that might break future shellcode can be used. Finding these types of instructions will be a matter of identifying which particular bytes can be used given the requirements of the exploit, and then determining what instructions the given bytes can be used to create, while considering the effect those instructions may have on the stack and important registers. This is where an instruction set reference and the ability of OllyDbg to disassemble your own code (see the section on editing code, memory and registers) can come in handy.

This completes the subsection on assembly instructions. For more information on the instructions available in X86 assembly, you can visit the following page to obtain an instruction set reference manual: http://www.intel.com/design/pentiumii/manuals/243191.htm

Become a Certified Ethical Hacker, guaranteed!

Become a Certified Ethical Hacker, guaranteed!

Get training from anywhere to earn your Certified Ethical Hacker (CEH) Certification — backed with an Exam Pass Guarantee.

Become a Certified Ethical Hacker, guaranteed!

Become a Certified Ethical Hacker, guaranteed!

Get training from anywhere to earn your Certified Ethical Hacker (CEH) Certification — backed with an Exam Pass Guarantee.

Once you read the guide in the second article: on how to run code in a debugger, you will also have an opportunity to directly observe how these, and other instructions, actually work. I encourage you to do this so you can obtain a better understanding of their operation.

Stephen Bradshaw
Stephen Bradshaw

Stephen Bradshaw is security researcher for InfoSec Institute and an IT Security Specialist in Australia, with a focus on the areas of penetration testing and incident detection and response.