Reverse engineering

Environment for Malware Analysis

Dejan Lukan
November 12, 2012 by
Dejan Lukan


Before trying to analyze any kind of malware, we need to have a proper environment set up, so we can be most efficient and collect the most information from the malicious executable without getting ourselves infected. The best thing to do in such cases is have a virtual machine image ready for testing purposes. We can have multiple physical computers if we would like to test our malicious software, but it's easier to work with virtual machines. We can use any virtual machine software to set-up different virtual machines on a hardware virtualization technology system. But often, we don't have the means for an expensive hardware based solution, so we're stuck with our daily working computer, either PC or a laptop. This is why we'll take a look at what we can do with our laptop computer to set-up the environment for analyzing malware.

We've already mentioned that we need some virtualization software, which we can use to set-up different virtual machines that can be easily added/deleted/duplicated, etc. It's best to choose from either a VMWare or Virtualbox virtualization software.

The first thing we need to understand when analyzing malware is that the malicious software programs usually spread through the Internet. This is why we don't actually want to give a malicious program the means to spread out onto the other machines in our LAN. To do that we must configure our virtual machine with limited access to the Internet. Every virtual machine has the options listed below under the Network configuration settings that we can use to give/take the Internet from s virtual machine:

- NAT: The virtual machine uses the host's IP address to connect to other machines on the LAN and on the entire Internet. Other machines on the LAN cannot connect to the virtual machine that is behind NAT; the virtual machine must initiate the connection. In Virtualbox NAT, the virtual machines will have the addresses in range, which are inaccessible from the host, but the virtual machines themselves can contact the Internet.

- Bridged Adapter: The virtual machine has its own IP even though it uses the network card of the host's system (both the host and the guest use the same network card, but they have different IP addresses). Other machines on LAN have direct access to the virtual machine.

- Host-only Adapter: The virtual machine uses a private network shared between the host and a set of virtual machines. The virtual machines in this private network cannot communicate with the Internet, but only between themselves. In Virtualbox the host-only adapter is named vboxnet0 and can be thought of as a router with the IP address This adapter serves the IP addresses to the virtual machines, which creates an internal LAN between them. The VMWare has two virtual adapters called vmnet1 and vmnet8, which are used to give IP addresses to virtual machines in NAT and host-only network modes. The virtual machines in host-only network mode won't have Internet access, since the vboxnet0 does not have a default gateway. Host-only network mode is useful when we need a private network where the virtual machines need to access one another but can't access the outside world, the Internet.

- Internal Network: Very similar to the host-only network with one difference: in the internal network even the host itself cannot access the virtual machines, which are connected into an internal network.

We can see that only if the virtual machine is in bridged mode, it can be accessed by other machines on the current LAN, and no other mode exposes the virtual machines to the current network. The internal network mode can be used to create a truly internal network, which cannot even be accessed by the host itself. We also need to be aware of the fact that only the NAT and bridged mode allow the virtual machines to connect to the network services outside the internal network restricted by host.

Virtualbox and VMWare

Whenever we're setting up a new virtual machine, we need to install the operating system (usually Windows) without any personal information. We should choose some username and other information that isn't relevant to the information about ourself. We should also set up a read-only shared folder, so that any malware operating in the Windows guest virtual machine won't be able to compromise with the contents of that shared folder. There is a Virtualbox command VBoxManage which is used for the management of the virtual machines. Here we'll take a closer look at it, because we can use the same commands as those that will be presented as the basis for our automation script whenever we would like to automate a constantly repeating task. If we run the VboxManage command only, it will display all subcommands we can execute with that command.

Let's take a new snapshot of an already configured Linux virtual machine named Ubuntu. In the picture below we can see that the presented virtual machine doesn't have any shared folder configured just yet:

We can of course configure a new shared folder with a graphical user interface, but we want to automate it, which is why we'll rather use command line. First we need to display the names of our virtual machines. This can be done with the command below:


# VBoxManage list vms

"Ubuntu" {a21b53f5-5252-42a1-837b-39bc3ef47e42}


We can see that a virtual machine with a name Ubuntu exists. This is why we can enter the command below to create a new shared folder named Home that provides a read only access to the /home/eleanor/ directory:


# VBoxManage sharedfolder add "Ubuntu" --name "Home" --hostpath "/home/eleanor/" --readonly


Now the graphical user interface will also list the previously created shared folder as can be seen in the picture below:

When booting up the Ubuntu virtual machine, we need to mount the created shared folder with the command presented below:


# mount -t vboxsf Home /mnt/


To take a snapshot with a command line tool, we would need to execute the command below:


# VBoxManage snapshot "Ubuntu" take "myimage"



There are various things we can do with the VBoxManage tool and there are quite a lot of other tools that we can use to mount the NTFS filesystem within the Windows .vdi VirtualBox image. But since we want to be using Virtualbox as well as VMWare in our testing, we don't want to develop different techniques to analyze .vdi (VirtualBox images) vs. .vmdk (VMWare images). We want to use a single technique that works with both products. We can do that by saving the virtual machines in VirtualBox as .vmdk images instead of .vdi images. After that we can use the VMWare tools to analyze the .vmdk images that are now being produces by VMWare as well as VirtualBox.

The picture below shows the step where we can choose between .vdi/.vmdk images when creating a new virtual machine:

With .vmdk images we can use VMWare VIX API to control the virtual machines. We can also use the vmrun command that provides a wrapper around the VIX API and provides basic and advanced commands which we can use to automate the needed functions when working with virtual machines.

If we run vmrun by itself, we receive the help page of the tool notifying us about authentication flags as well as all supported commands (note that only a few commands are listed):

# vmrun

vmrun version 1.11.4 build-744019




These must appear before the command and any command parameters.

-h <hostName> (not needed for Workstation)

-P <hostPort> (not needed for Workstation)

-T <hostType> (ws|server|server1|fusion|esx|vc|player)

for example, use '-T server' for VMware Server 2.0

use '-T server1' for VMware Server 1.0

use '-T ws' for VMware Workstation

use '-T ws-shared' for VMware Workstation (shared mode)

use '-T esx' for VMware ESX

use '-T vc' for VMware vCenter Server

-u <userName in host OS> (not needed for Workstation)

-p <password in host OS> (not needed for Workstation)

-vp <password for encrypted virtual machine>

-gu <userName in guest OS>

-gp <password in guest OS>


-------------- ---------- -----------

start Path to vmx file Start a VM or Team


stop Path to vmx file Stop a VM or Team

reset Path to vmx file Reset a VM or Team

suspend Path to vmx file Suspend a VM or Team

pause Path to vmx file Pause a VM

unpause Path to vmx file Unpause a VM


----------------- ---------- -----------

listSnapshots Path to vmx file List all snapshots in a VM


snapshot Path to vmx file Create a snapshot of a VM
Snapshot name

deleteSnapshot Path to vmx file Remove a snapshot from a VM

Snapshot name


revertToSnapshot Path to vmx file Set VM state to a snapshot

There are multiple things we can do with vmrun, but we should remember that with vmrun we can do practically everything regarding our virtual machine: we can start and stop the virtual machine, we can boot off the snapshot of the virtual machine, we can transfer a file from our host operating system to the guest operating system, we can start a program in the guest operating system, etc. The most important thing is to monitor for the changes a malware does when being started.

We need to monitor a malware program for the following things when we start it:

  • registry changes/accesses: does the malware software try to access any of the registry keys or add/delete some of them.
  • network activity: what web pages malware communicates with and which protocol it uses.
  • filesystem activity: which files malware reads/writes to the filesystem.

These are just basic things we need to know about every malware. And these are also things that can be automated. If we run a honepot network, it's reasonable to automate the first reconnaissance of the malware software program. This can give us basic information of what the malware is doing and how it is doing it. After that we need to start with the reverse engineering, which can tell us more about the details of how the malware does what it does.


We've seen how we can use the VirtualBox and VMWare environment to run the malware samples. We should first think about whether to allow the virtual machine to use the Internet or configure just the internal network, which we can do easily with the VirtualBox networking options. Afterwards we need to install the right operating system for the malware to be in a native environment; if we're trying to analyze Windows executable malware programs, we need to install a Windows operating system and run malware in there.

We can also use different Linux distributions that use Wine to run the malware executables (like ZeroWine Tryouts Linux distribution), but by running the malware in non-native environment anything can happen. Most of the time, we will be able to analyze the Windows malware programs even in a Linux environment, but we won't be able to analyze kernel rootkits and more advanced malware samples.

We've also seen that we can automate most of the first steps when analyzing malware: capturing the network traffic, determining the created/deleted files on the file system, determining the created registry keys, etc. These are also the first steps we need to take when analyzing malware, because they give us a basic knowledge about what the malware does and how it does it.


[1] Chapter 6. Virtual networking,

[2] 5 Steps to Building a Malware Analysis Toolkit Using Free Tools,

[3] Using vmrun to Control Virtual Machines,

[4] Controlling VMware Virtual Machines from the Command Line with vmrun,

Dejan Lukan
Dejan Lukan

Dejan Lukan is a security researcher for InfoSec Institute and penetration tester from Slovenia. He is very interested in finding new bugs in real world software products with source code analysis, fuzzing and reverse engineering. He also has a great passion for developing his own simple scripts for security related problems and learning about new hacking techniques. He knows a great deal about programming languages, as he can write in couple of dozen of them. His passion is also Antivirus bypassing techniques, malware research and operating systems, mainly Linux, Windows and BSD. He also has his own blog available here: