Reverse engineering

CrackMe Challenge Part 3: The Logic Behind the First Challenge

If we take our predicate that we've seen in the end of part 2 into account and input at least 64 bytes (0x40) into the Key 1 field and leave the Name field at a value AAAAAAAA, a second message box is displayed as can be seen in the picture below:

Become a certified reverse engineer!

Get live, hands-on malware analysis training from anywhere, and become a Certified Reverse Engineering Analyst.

Start Learning

Become a certified reverse engineer!

Get live, hands-on malware analysis training from anywhere, and become a Certified Reverse Engineering Analyst.

Start Learning

The message box says that we entered an Incorrect Name/Key pair and we must try again. Again, we must first look at the referenced strings and determine where the string comes from. This can be seen in the picture below:

The picture deliberately contains two functions that display certain values in a message box. The first one start at address 0x00401720 and displays Congratulations !!! Go On !!! and the second function starts at address 0x004017240 and displays Incorrect Name/Key pair. Try again.

We can gather two things from this. First, we must backtrace our program from the address 0x004017240, which displays exactly the message box that's being shown. Secondly, we must tell the program to execute the function at address 0x00401720, which clearly displays the congratulations message box. This is a very good proof that when the program execution will be directed at address 0x00401720 rather than 0x00401740, our Stage 1 challenge will be complete and we will be able to begin solving the Stage 2 challenge.

Again, we must backtrace our program. The message box function was called from an address 0x00401967. It's always a good idea to try to deal with the whole function frame when trying to determine what the function does. The whole function in our case is presented here - again the code is quite daunting, but let's not despair:

00401760 /$ 55 push ebp

00401761 |. 8BEC mov ebp,esp

00401763 |. 83E4 F8 and esp,FFFFFFF8

00401766 |. 81EC FC000000 sub esp,0FC

0040176C |. A1 D0E45500 mov eax,dword ptr ds:[55E4D0]

00401771 |. 33C4 xor eax,esp

00401773 |. 898424 F800000>mov dword ptr ss:[esp+F8],eax

0040177A |. 53 push ebx

0040177B |. 56 push esi

0040177C |. 57 push edi

0040177D |. 6A 4C push 4C

0040177F |. 8D4424 14 lea eax,dword ptr ss:[esp+14]

00401783 |. 6A 00 push 0

00401785 |. 50 push eax

00401786 |. 8BD9 mov ebx,ecx

00401788 |. E8 135B0F00 call main.004F72A0

0040178D |. 83C4 0C add esp,0C

00401790 |. 837D 10 40 cmp dword ptr ss:[ebp+10],40

00401794 |. C74424 5C 4017>mov dword ptr ss:[esp+5C],main.00401740

0040179C |. 0F8E C5010000 jle main.00401967

004017A2 |. 8B45 08 mov eax,dword ptr ss:[ebp+8]

004017A5 |. B9 14000000 mov ecx,14

004017AA |. BE B8195400 mov esi,main.005419B8

004017AF |. 8D7C24 70 lea edi,dword ptr ss:[esp+70]

004017B3 |. F3:A5 rep movs dword ptr es:[edi],dword ptr ds:[esi]

004017B5 |. 83F8 50 cmp eax,50

004017B8 |. 72 05 jb short main.004017BF

004017BA |. B8 50000000 mov eax,50

004017BF |> 50 push eax

004017C0 |. 8D4C24 74 lea ecx,dword ptr ss:[esp+74]

004017C4 |. 53 push ebx

004017C5 |. 51 push ecx

004017C6 |. E8 455D0F00 call main.004F7510

004017CB |. 83C4 0C add esp,0C

004017CE |. 8D9424 C000000>lea edx,dword ptr ss:[esp+C0]

004017D5 |. 52 push edx

004017D6 |. 8D4424 74 lea eax,dword ptr ss:[esp+74]

004017DA |. 50 push eax

004017DB |. B9 50000000 mov ecx,50

004017E0 |. E8 3B180000 call main.00403020

004017E5 |. B8 3F000000 mov eax,3F

004017EA |. 8D4C24 70 lea ecx,dword ptr ss:[esp+70]

004017EE |. 8BFF mov edi,edi

004017F0 |> 8B55 0C /mov edx,dword ptr ss:[ebp+C]

004017F3 |. 8A1410 |mov dl,byte ptr ds:[eax+edx]

004017F6 |. 8811 |mov byte ptr ds:[ecx],dl

004017F8 |. 41 |inc ecx

004017F9 |. 48 |dec eax

004017FA |.^79 F4 jns short main.004017F0

004017FC |. B8 40000000 mov eax,40

00401801 |. 33C9 xor ecx,ecx

00401803 |> 8B940C C000000>/mov edx,dword ptr ss:[esp+ecx+C0]

0040180A |. 3B540C 70 |cmp edx,dword ptr ss:[esp+ecx+70]

0040180E |. 0F85 53010000 |jnz main.00401967

00401814 |. 83E8 04 |sub eax,4

00401817 |. 83C1 04 |add ecx,4

0040181A |. 83F8 04 |cmp eax,4

0040181D |.^73 E4 jnb short main.00401803

0040181F |. B9 10000000 mov ecx,10

00401824 |. 8DB424 C000000>lea esi,dword ptr ss:[esp+C0]

0040182B |. 8D7C24 10 lea edi,dword ptr ss:[esp+10]

0040182F |. F3:A5 rep movs dword ptr es:[edi],dword ptr ds>

00401831 |. 8B75 10 mov esi,dword ptr ss:[ebp+10]

00401834 |. 33C0 xor eax,eax

00401836 |. 85F6 test esi,esi

00401838 |. 7E 1E jle short main.00401858

0040183A |. 8D9B 00000000 lea ebx,dword ptr ds:[ebx]

00401840 |> 8B55 0C /mov edx,dword ptr ss:[ebp+C]

00401843 |. 8A1410 |mov dl,byte ptr ds:[eax+edx]

00401846 |. 8BC8 |mov ecx,eax

00401848 |. 83E1 3F |and ecx,3F

0040184B |. 30540C 10 |xor byte ptr ss:[esp+ecx+10],dl

0040184F |. 8D4C0C 10 |lea ecx,dword ptr ss:[esp+ecx+10]

00401853 |. 40 |inc eax

00401854 |. 3BC6 |cmp eax,esi

00401856 |.^7C E8 jl short main.00401840

00401858 |> 33F6 xor esi,esi

0040185A |. 33C9 xor ecx,ecx

0040185C |. 8D6424 00 lea esp,dword ptr ss:[esp]

00401860 |> 8A5C34 10 /mov bl,byte ptr ss:[esp+esi+10]

00401864 |. 0FB6C3 |movzx eax,bl

00401867 |. 99 |cdq

00401868 |. BF 50000000 |mov edi,50

0040186D |. F7FF |idiv edi

0040186F |. 83C6 04 |add esi,4

00401872 |. 0FB64414 10 |movzx eax,byte ptr ss:[esp+edx+10]

00401877 |. 03C1 |add eax,ecx

00401879 |. 99 |cdq

0040187A |. 8BCF |mov ecx,edi

0040187C |. F7F9 |idiv ecx

0040187E |. 0FB6CA |movzx ecx,dl

00401881 |. 8A440C 10 |mov al,byte ptr ss:[esp+ecx+10]

00401885 |. 885C0C 10 |mov byte ptr ss:[esp+ecx+10],bl

00401889 |. 8A5C34 0D |mov bl,byte ptr ss:[esp+esi+D]

0040188D |. 884434 0C |mov byte ptr ss:[esp+esi+C],al

00401891 |. 0FB6C3 |movzx eax,bl

00401894 |. 99 |cdq

00401895 |. F7FF |idiv edi

00401897 |. 0FB64414 10 |movzx eax,byte ptr ss:[esp+edx+10]

0040189C |. 03C1 |add eax,ecx

0040189E |. 99 |cdq

0040189F |. 8BCF |mov ecx,edi

004018A1 |. F7F9 |idiv ecx

004018A3 |. 0FB6CA |movzx ecx,dl

004018A6 |. 8A440C 10 |mov al,byte ptr ss:[esp+ecx+10]

004018AA |. 885C0C 10 |mov byte ptr ss:[esp+ecx+10],bl

004018AE |. 8A5C34 0E |mov bl,byte ptr ss:[esp+esi+E]

004018B2 |. 884434 0D |mov byte ptr ss:[esp+esi+D],al

004018B6 |. 0FB6C3 |movzx eax,bl

004018B9 |. 99 |cdq

004018BA |. F7FF |idiv edi

004018BC |. 0FB64414 10 |movzx eax,byte ptr ss:[esp+edx+10]

004018C1 |. 03C1 |add eax,ecx

004018C3 |. 99 |cdq

004018C4 |. 8BCF |mov ecx,edi

004018C6 |. F7F9 |idiv ecx

004018C8 |. 0FB6CA |movzx ecx,dl

004018CB |. 8A440C 10 |mov al,byte ptr ss:[esp+ecx+10]

004018CF |. 885C0C 10 |mov byte ptr ss:[esp+ecx+10],bl

004018D3 |. 8A5C34 0F |mov bl,byte ptr ss:[esp+esi+F]

004018D7 |. 884434 0E |mov byte ptr ss:[esp+esi+E],al

004018DB |. 0FB6C3 |movzx eax,bl

004018DE |. 99 |cdq

004018DF |. F7FF |idiv edi

004018E1 |. 0FB64414 10 |movzx eax,byte ptr ss:[esp+edx+10]

004018E6 |. 03C1 |add eax,ecx

004018E8 |. 99 |cdq

004018E9 |. 8BCF |mov ecx,edi

004018EB |. F7F9 |idiv ecx

004018ED |. 0FB6CA |movzx ecx,dl

004018F0 |. 8A440C 10 |mov al,byte ptr ss:[esp+ecx+10]

004018F4 |. 885C0C 10 |mov byte ptr ss:[esp+ecx+10],bl

004018F8 |. 884434 0F |mov byte ptr ss:[esp+esi+F],al

004018FC |. 83FE 40 |cmp esi,40

004018FF |.^0F8C 5BFFFFFF jl main.00401860

00401905 |. 817C24 54 0A0B>cmp dword ptr ss:[esp+54],0D0C0B0A

0040190D |. 75 50 jnz short main.0040195F

0040190F |. 8D5424 70 lea edx,dword ptr ss:[esp+70]

00401913 |. 52 push edx

00401914 |. 8D4424 14 lea eax,dword ptr ss:[esp+14]

00401918 |. 50 push eax

00401919 |. 8D4F FC lea ecx,dword ptr ds:[edi-4]

0040191C |. C74424 6C CA8A>mov dword ptr ss:[esp+6C],12578ACA

00401924 |. C74424 70 78B6>mov dword ptr ss:[esp+70],EFCAB678

0040192C |. C74424 74 7856>mov dword ptr ss:[esp+74],12345678

00401934 |. E8 E7160000 call main.00403020

00401939 |. 8D47 BC lea eax,dword ptr ds:[edi-44]

0040193C |. 33C9 xor ecx,ecx

0040193E |. 8BFF mov edi,edi

00401940 |> 8B540C 70 /mov edx,dword ptr ss:[esp+ecx+70]

00401944 |. 3B540C 64 |cmp edx,dword ptr ss:[esp+ecx+64]

00401948 |. 75 1D |jnz short main.00401967

0040194A |. 83E8 04 |sub eax,4

0040194D |. 83C1 04 |add ecx,4

00401950 |. 83F8 04 |cmp eax,4

00401953 |.^73 EB jnb short main.00401940

00401955 |. C74424 5C 2017>mov dword ptr ss:[esp+5C],main.00401720

0040195D |. EB 08 jmp short main.00401967

0040195F |> C74424 5C 4017>mov dword ptr ss:[esp+5C],main.00401740

00401967 |> FF5424 5C call dword ptr ss:[esp+5C]

0040196B |. 8B8C24 0401000>mov ecx,dword ptr ss:[esp+104]

00401972 |. 5F pop edi

00401973 |. 5E pop esi

00401974 |. 5B pop ebx

00401975 |. 33CC xor ecx,esp

00401977 |. E8 3D410F00 call main.004F5AB9

0040197C |. 8BE5 mov esp,ebp

0040197E |. 5D pop ebp

0040197F . C2 0C00 retn 0C

Our message box function is called at the address 0x00401967, which defines the following instruction (from the above function code):

00401967 |> FF5424 5C call dword ptr ss:[esp+5C]

From the code around this function call, we can see instructions that are used to call either a function at address 0x00401740, which is our error message box that we're trying to avoid, or a function at address 0x00401720 which is the success message box that we're trying to call. The following instructions are the ones that determine which function is called:

00401955 |. C74424 5C 2017>mov dword ptr ss:[esp+5C],main.00401720

0040195D |. EB 08 jmp short main.00401967

0040195F |> C74424 5C 4017>mov dword ptr ss:[esp+5C],main.00401740

00401967 |> FF5424 5C call dword ptr ss:[esp+5C]

These instructions save a pointer to the address of either 0x00401720 or 0x00401740 function on the stack at address [esp+0x5C], which is then called by the call instruction. This effectively calls either the first or the second function. We can also see that if the program flow gets to the first instruction presented in the previous code output, the success message box is called. The failure message box must be called explicitly by jumping directly to the call assembler instruction.

Now we must decipher the above function to determine what the function does and how to convince it to execute the instruction at address 0x00401955, which would in turn execute the successful message box, letting us proceed to the next level.

If we look at the function, we can quickly determine that it uses various checks before displaying the success message box. This is why it's best to split the function down into logical segments and determine what each segment does.

The code of the first code segment is presented here:

00401760 /$ 55 push ebp

00401761 |. 8BEC mov ebp,esp

00401763 |. 83E4 F8 and esp,FFFFFFF8

00401766 |. 81EC FC000000 sub esp,0FC

0040176C |. A1 D0E45500 mov eax,dword ptr ds:[55E4D0]

00401771 |. 33C4 xor eax,esp

00401773 |. 898424 F800000>mov dword ptr ss:[esp+F8],eax

0040177A |. 53 push ebx

0040177B |. 56 push esi

0040177C |. 57 push edi

0040177D |. 6A 4C push 4C

0040177F |. 8D4424 14 lea eax,dword ptr ss:[esp+14]

00401783 |. 6A 00 push 0

00401785 |. 50 push eax

00401786 |. 8BD9 mov ebx,ecx

00401788 |. E8 135B0F00 call main.004F72A0

0040178D |. 83C4 0C add esp,0C

00401790 |. 837D 10 40 cmp dword ptr ss:[ebp+10],40

00401794 |. C74424 5C 4017>mov dword ptr ss:[esp+5C],main.00401740

0040179C |. 0F8E C5010000 jle main.00401967

004017A2 |. 8B45 08 mov eax,dword ptr ss:[ebp+8]

004017A5 |. B9 14000000 mov ecx,14

004017AA |. BE B8195400 mov esi,main.005419B8

This piece of code can be assumed to be an initialization code that doesn't really matter at this point. We can see that the function takes care of its own stack frame. It uses 0xFC local arguments, which is quite something, so we can immediately infer that the function's logic will be complex, challenging and interesting.

The code above doesn't change the stack memory we're interested in, so we don't need to follow it in complete details. The only thing the code introduces and we've already looked at is the following instructions:

00401790 |. 837D 10 40 cmp dword ptr ss:[ebp+10],40

00401794 |. C74424 5C 4017>mov dword ptr ss:[esp+5C],main.00401740

0040179C |. 0F8E C5010000 jle main.00401967

The [ebp+10] input argument contains the length computed as 6/8*(length of Key 1), which is compared to a number 0x40. If the length of the input value is less than the specified number of bytes, the error message is returned.

Conclusion
In the next part we'll take a look at the other logical code segments that will eventually describe the whole function that is used for checking the correct Name and Key 1 input fields.