Emulating decryption function with radare2

This is the first part of our three-part series on code emulation for malware analysis:

• Part 1 (this post) explains how to use radare2’s function emulation feature, featuring a password-cracking exercise using radare2’s Python scripting plugin, r2pipe.
• Part 2 demonstrates how to decode the configuration of the Mirai IoT botnet by writing an automation script using radare2’s Python scripting capabilities.
• Part 3 builds on the previous script by adding support for searching encrypted string addresses and generating function signatures to locate the decryption routine dynamically.

radare2 is an exceptionally powerful reverse engineering framework supporting a wide variety of CPU architectures. Among its many capabilities, one feature that consistently stands out is partial code emulation. While I was initially skeptical about its practical use cases, experimenting with it quickly revealed its massive potential. It is simply a game-changer for static analysis.

Let’s consider a common scenario: a malware author encrypts strings inside a binary and decrypts them on-the-fly right before they are needed. For example, if you inspect such an executable in PEView, the Import Address Table (IAT) might only show basic APIs like LoadLibrary and GetProcAddress. The malware resolves actual APIs dynamically at runtime to stay stealthy. Since function names are decrypted right before loading them, you are left with no immediate clues—just basic string obfuscation. Important endpoints like attack IPs or C2 URLs might be hidden the same way.

Usually, you would have to reverse-engineer the custom decryption routine and rewrite the algorithm in a Python script to recover these strings. This involves the tedious chore of tracing string offsets, flipping back and forth between your disassembler and shell, and manually managing the output.

We can bypass this static reversing process entirely if we let the binary do the work. By running just the decryption routine inside a lightweight emulator, we can feed it arbitrary encrypted pointers and read the decrypted strings straight from virtual memory.

Beyond string decryption, code emulation is invaluable for unpacking self-modifying code or analyzing shellcode. While radare2’s ESIL (Evaluable Strings Instruction Language) engine cannot emulate a full operating system (meaning OS system calls or external library calls won’t work out of the box), malware decryption routines are almost always self-sufficient and dependency-free. This makes them perfect candidates for emulation. Let’s look at how to set this up.

#The Challenge: A Sample Decryption Function

For the sake of simplicity, I have written a basic C program. It accepts an input string, obfuscates it using an index-based key mapping, and compares the result with a hardcoded goal string. If the strings match, we have solved the challenge. Here is the C source code:

#include <stdio.h>
#include <stdbool.h>

bool is_valid(char *user_input){
    char *key_stub = "isrveawhobpnutfg";
    char *solution = "gaints";
    int index ;
    bool is_same = true;

    for(int i=0; i<6; i++){
        index = user_input[i] & 0x0f;
        user_input[i] = key_stub[index];
        if(key_stub[index] != solution[i]){
            is_same = false;
        }
    }
    return is_same;
}

int main(){
    char decrypted_stub[6] = "oe0kma";
    bool ans = is_valid(decrypted_stub);
    if(ans){
        printf("Passed\n");
    }else{
        printf("Failed\n");
    }
    return 0;
}

To understand how parameters are passed to this function, let’s briefly review calling conventions on x86 32-bit machines:

1. Arguments are pushed onto the stack. Pointers contain the memory address of the actual variable.
2. When the function returns, the result is stored in the eax (or rax on 64-bit) register.
3. Within the function, arguments are referenced via offsets from the frame pointer (e.g., ebp + 0x8 for the first argument, ebp + 0xc for the second).

To emulate this call, we can manually set up the stack frame inside the VM, configure the instruction pointer to target the function, let the emulator run, and then inspect the eax register.

#Initializing the Virtual Machine

Before we can use radare2’s emulator, we need to initialize its state by specifying the architecture, byte ordering, and memory constraints. Use the following commands:

1. e asm.bits=32: Configure a 32-bit address space.
2. e asm.arch=x86: Set the target architecture to x86.
3. aei: Initialize the ESIL VM.
4. aeim: Allocate virtual stack memory. By default, this places a stack of size 0xf0000 at 0x100000 (this can be customized by passing parameters).
5. s sym.is_valid: Seek to the entry point of our is_valid function.
6. aeip: Sync the VM’s instruction pointer (eip) with the current seek address.
7. pxw @ esp: Inspect the stack layout in 32-bit word format.

#Configuring the Virtual Stack

We will write our input string into a higher segment of our virtual memory (outside the active stack frame region) and push a pointer referencing this address onto the stack. This satisfies the function’s requirements.

1. w o0ekma @ 0x001780f0: Write the test string "o0ekma" at memory address 0x001780f0 inside our virtual allocation space.

2. wx 0xf0801700 @ ebp+0x4: Write the address 0x001780f0 in little-endian format to the stack. Here is why we use this syntax:

   • We write our argument to ebp + 0x4 instead of ebp + 0x8 because the function prologue (push ebp) has not yet executed. Once the prologue runs, the offsets shift naturally, and the code inside will correctly resolve the first argument at ebp + 0x8.
   • The address must be written backwards (little-endian) as 0xf0801700 to be correctly parsed by x86 instructions.

   Stack Data	Offset/Register
   func param n	EBP + 0xc
   func param 2	EBP + 0x8
   func param 1	EBP + 0x4
   callee EIP	<= ESP (when VM is initialized)
   callee EBP
   caller local vars

3. pxw @ esp: Visually verify that our pointer is correctly structured on the stack.

4. Now that our stack is configured, we can step through the function’s assembly.

5. aes; aer; pxW 50 @ esp; pd 10 @ eip: This combined command executes a single instruction, dumps all register states, inspects the top 50 bytes of the stack, and disassembles the next 10 instructions from eip.

6. aecu 0x5f1: Stepping through code instruction-by-instruction is great for deep-dives, but if we just want the final result, we can run the function to completion. aecu <address> executes instructions up to the target address (e.g., the function return or epilogue) and pauses. We can then read the return value directly with aer eax.

#Automating the Search with Python

Using Python and r2pipe, we can automate this entire sequence. We can systematically try different inputs, run the emulation, and check the eax return value. If eax == 1, we found our winning password.

Here is the complete solution script:

import r2pipe

def check_val(user_input):
     r = r2pipe.open("./decode")
     
     # Initialize the ESIL environment
     r.cmd('aaa')
     r.cmd('e asm.emu=true')
     r.cmd('e asm.bits=32')
     r.cmd('e asm.arch=x86')
     
     # Zero out general registers
     r.cmd('ar0')
     r.cmd('s sym.is_valid')
     r.cmd('aei')
     r.cmd('aeim')
     r.cmd('aeip')
     r.cmd('aer')
     
     # Set up the stack argument (pointer to the input string)
     r.cmd('w {} @ 0x001780f0'.format(user_input))
     r.cmd('wx 0xf0801700 @ ebp+0x4')
     
     # Emulate up to the return instruction
     r.cmd('aecu 0x000005f1')

     # Retrieve the return value from EAX
     sol = int(r.cmd('aer eax'), 0)
     if sol == 1:
         print('Found solution: ', user_input)
     else:
         print('Invalid user input: ', user_input)

u_inputs = ['oe0kma', 'ashi', 'adsh', 'aasdf']
for u_i in u_inputs:
    check_val(u_i)

Inside this script:

1. r2pipe.open("./decode") handles loading the binary into radare2.
2. r.cmd() executes commands exactly as you would type them in the console.
3. Standard commands can be suffixed with j (e.g., aerj) to output clean JSON arrays or dictionaries directly, allowing for easy integration with Python’s built-in parsing libraries.

#Conclusion

We have explored how to use partial code emulation to treat a compiled assembly function as a black-box routine, allowing us to send inputs and read outputs directly from virtual memory. This bypasses the need to manually replicate complex algorithms in Python.

In our next post, we will apply these techniques to de-obfuscate a real-world malware configuration from the Mirai IoT botnet. See you there!