Debugging in the Cloud

In the world we live in, there are different kinds of professions where debugging has been a vital piece of knowledge we must have in order to do our jobs successfully or more efficiently. There are different professions where debugging knowledge is important and are outlined below:

• Programmers: every programmer knows that debugging the program when it's not working properly is the only right way to determine why the program is misbehaving and is not producing the intended results. There are still those programmers out there who use different kinds of print statements in order to display the debugging information in a way that makes sense to them only and for a limited amount of time. After a certain period of time, the debugging comments displayed in the stdout (or anywhere else where the output could easily be inspected, like in a file) don't make any sense anymore, which is the primary reason why we should stop debugging like that – if we can even call it debugging at this point. In any case, whenever we put any kind of print statements into the code, the whole code needs to be recompiled when written in a low-level programming language.
  
• System Administrators: various system administrators are tasked with setting up and maintaining a whole infrastructure of the company, which is not an easy task to deal with. There will be different times when the programs of the programmers will malfunction and the system administrator will be the only one having access to the production environment where the program malfunctions. At times like these, the administrator should use a debugger in order to determine what the problem is and report it back to the programmers in order to fix it as soon as possible.
  
• Security Researchers: debugger skills can be lethal in the hands of a security specialist, because he can use it to make a program do unexpected things. Debuggers are indispensable when used to analyze how the program works in order to gain deeper understanding about the program internals. There are various fields of a security domain where knowledge about debuggers is a must have skill and the people not having mastered it already will have a hard time completing their jobs; such fields include reverse engineering, malware analysis, exploit writing, etc. The skills will come in handy even when dealing with web applications, for example, when we've reversed engineered a web application written in ASP.NET, which used custom encryption/decryption functions in order to pass data between the client and a server in an encrypted form. Having the reverse engineering skills, we quickly put together an algorithm, which was able to decrypt the encrypted data in order for us to modify it and then re-encrypt the data back to its encrypted form to have it sent to the server for processing, which revealed interesting XSS, URL redirection and other kinds of bugs.

Despite our job profession outlined above, we should invest the time and learn how to debug properly, which will enable us to find problems sooner and with ease; no more print statements need be introduced into the code. When debugging properly, we have to choose a debugger of our choice and run the program in a debugger, and set appropriate breakpoints so the execution will stop at the time of the program misbehavior, after which we can inspect the program state. Inspecting the program state doesn't include only a few of the items we had output to the stdout when doing it the wrong way, but the whole program state – we no longer have to put additional print statements into the code, recompile and rerun the program in order to get more information about the program state. Instead we can get all that information for free out of a program stopped in a debugger without many problems.

Presenting different kinds of debuggers

There are many debuggers that we can use for debugging and are separated into two groups at the highest level, which are presented below. Note that most operating systems are constituted from two parts: the user-mode applications in ring 3, where all of the applications run from and have only limited access via the system calls to the kernel-mode operating system code in ring 0. Therefore, depending on whether we're debugging a user-mode application in ring 3 or an operating system function/structure in ring 0, the debuggers are divided between two groups presented below.

• Kernel-Mode debuggers: the debuggers running in kernel-mode, which are able to debug the kernel operating system internals as well as the user-mode applications. An example of debuggers supporting kernel-mode debugging are the following: SoftICE, Syser, HyperDbg, WinDbg, Gdb, VirtDbg.
  
• User-Mode debuggers: the debuggers running in user-mode, which are able to debug only the user-mode applications, but don't have access to the kernel. User-mode debuggers are the following: OllyDbg, Hopper, Hiew, Ida Pro, Dbg, x64dbg, VDB, Radare, etc.

All of the debuggers have support for debugging local programs or systems, but only some of them have remote debugging possibilities that allow us to use debuggers in the cloud. The following debuggers have a possibility of a remote debugging session, which we can use in a cloud-based session and debug the problem remotely:

• WinDbg
• Gdb
• VirtDbg
• Ida Pro
• Radare
• Hopper

Remote debugging

In this example, we'll take a look at how we can debug an application running in the cloud remotely by using gdb, which can be downloaded and installed by running the following commands:

[plain]

# wget ftp://sourceware.org/pub/gdb/releases/gdb-7.5.tar.bz2

# tar xvjf gdb-7.5.tar.bz2

# cd gdb-7.5/gdb/gdbserver/

# ./configure && make && make install

[/plain]

Let's first display all the parameters that we can pass to gdbserver program.

[plain]

# gdbserver

Usage: gdbserver [OPTIONS] COMM PROG [ARGS …]

gdbserver [OPTIONS] --attach COMM PID

gdbserver [OPTIONS] --multi COMM

COMM may either be a TTY device (for serial debugging), or HOST:PORT to listen for a TCP connection.

Options:

--debug Enable general debugging output.

--remote-debug Enable remote protocol debugging output.

--version Display version information and exit.

--wrapper WRAPPER -- Run WRAPPER to start new programs.

--once Exit after the first connection has closed.

[/plain]

Let's now present a simple program, that accepts exactly one argument, which must be set to the "secretarg" string in order for the program to return the secret key "KeepingHiddenSecrets". Otherwise, the program exists with a notification that incorrect input string was passed to the program as the first argument. The program can be seen below.

[c]
#include <stdio.h>

int main(int argc, char **argv) {

if(argc != 2) {

printf("The wrong number of parameters passed into the program; quitting.n");

exit(1);

}

if(strcmp(argv[1], "secretarg") == 0) {

printf("The secret password is: KeepingHiddenSecrets.n");

}

else {

printf("The secret password is not revealed to you, because you didn't supply the right secret argument.n");

}

return 0;

}

[/c]

We can compile the program into the main executable by simply running the "gcc main.c -o main" command, after which we can run the program by running "./main secretarg", which will print the secret key to the standard output. Imagine that we're a system administrator or a security researcher and only have access to the main executable, but we don't have the code, neither we know the secret argument we have to pass to the program in order to reveal the secret key. To complicate matters somehow, let's also imagine that the program is running on a server on the cloud and can't be easily recompiled and used on our local computer; nevertheless, we have access to the server and we're able to run and debug the program. Note that the compiled program should also be copied to the host system where we'll be inputting the gdb commands in order to be sent to the gdbserver, so the gdb will be able to load and use program symbols.

In such cases, it's best to run the program remotely in the cloud in gdbserver in order to debug it. We can use the command line below to bind to the 0.0.0.0:8080 host and port combination where the remote debugging session will be accessible.

We have to run the following commands on the remote host in the cloud where the program will be debugged. Note that the two processes are created during the debugging session because we've invoked the program two times, once with the wrong input parameter and another time with the right input parameter. The first invocation of the program revealed that the input argument was not correct, while the second invocation received the secret password, because we've passed the correct input argument to the program invocation.

[plain]

# gdbserver --multi 0.0.0.0:8080

Listening on port 8080

Remote debugging from host 4.3.2.1

Process /srv/main created; pid = 20272

The secret password is not revealed to you, because you didn't supply the right secret argument.

Child exited with status 0

Process /srv/main created; pid = 20274

The secret password is: KeepingHiddenSecrets.

Child exited with status 0

[/plain]

Then we can use the netstat command to confirm whether the gdbserver has actually been started, which can be seen below.

[plain]

# netstat -luntp | grep LISTEN

tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 20223/gdbserver

[/plain]

After starting the remote session, we can connect to it by executing the following commands on the client, which connects to the remote session and starts debugging the remote process.

[plain]

# gdb

(gdb) target extended-remote 1.2.3.4:8080

Remote debugging using 1.2.3.4:8080

(gdb) set remote exec-file /srv/main

(gdb) file /tmp/main

Reading symbols from /tmp/main...done.

(gdb) set architecture i386:x86-64:intel

The target architecture is assumed to be i386:x86-64:intel

(gdb) run test

Starting program: /tmp/main test

[Inferior 1 (process 20272) exited normally]

(gdb) run secretarg

Starting program: /tmp/main secretarg

[Inferior 1 (process 20274) exited normally]

[/plain]

At this point we can run any command supported by the gdb debugger right on the remote session in the cloud, which enables us to do anything we would have done with a local process.

Conclusion

Debugging skills are a vital and very important piece of knowledge we have to gain in order to complete our job faster and more efficiently. The hardest thing to do in the process is grasping the idea that such a knowledge will actually benefit us all. After we've convinced ourselves that the debugging knowledge will come in handy, we have to choose an appropriate debugger and learn as much as we can about it. Usually, there are different articles and tutorials, even books written on the subject, but we must not despair. We can start slow with a simple tutorial and work our way from there. Whenever a new bug arises, we should take some extra time to find the problem with a debugger rather than using print statements. At first, it will seem like a waste of time, but sooner or later, it will become extremely easy and the first benefits of the newly acquired knowledge will be visible.

We've seen how easy it is to debug applications in the cloud by using one of the remote capabilities of various debuggers that support it. By using remote debugging, we can easily start a program in the cloud and debug it remotely, not having to setup our own environment when trying to determine what the problem was. If the client wishes to debug a software, which requires various pieces to work together, we can easily use remote debugging capabilities to remotely identify the problem they have been facing. This gets more and more important when debugging SCADA applications, which require certain kinds of hardware that we normally don't have access to in our every day lives, like a nuclear plant, an air conditioning, etc. In such circumstances, we would have to fly to the client's location in order to identify the problem at hand, but by using remote debugging capabilities we can do it from our own office from an entirely different country, which reduces costs considerably.

We should all invest the time to learn and obtain debugging knowledge, which will save us time and money when trying to determine the cause of the problem. It is only by practicing that we become better and better at what we do and it's the same with debugging: keep practicing and enjoy using your newly obtained knowledge.