Module 3: Vectored Exception Handling Deep Dive

The process-wide exception handler that intercepts every breakpoint before anyone else sees it.

1. VEH vs SEH: Why VEH?

Windows provides two exception handling mechanisms that user-mode code can use. Understanding the difference is critical to understanding why ShellGhost uses VEH.

2. The Exception Dispatch Order

When an exception occurs, Windows dispatches it through a defined sequence. Understanding this order explains why VEH guarantees ShellGhost's handler runs first:

1. Vectored Exception Handlers (VEH): Called first. If any handler returns EXCEPTION_CONTINUE_EXECUTION, dispatch stops and execution resumes with the (potentially modified) context.
2. Structured Exception Handlers (SEH): Only reached if no VEH handler handled the exception. Walks the per-thread stack-based handler chain.
3. Vectored Continue Handlers (VCH): Called after an SEH handler returns EXCEPTION_CONTINUE_EXECUTION. Rarely used.
4. Unhandled Exception Filter: The last-resort handler registered via SetUnhandledExceptionFilter.
5. Process Termination: If nothing handles the exception, the process crashes.

3. Registering a VEH Handler

The AddVectoredExceptionHandler API registers a new VEH handler. Its First parameter controls whether the handler is inserted at the beginning or end of the VEH list:

C// Function signature (from winnt.h / errhandlingapi.h)
PVOID AddVectoredExceptionHandler(
    ULONG First,                          // 1 = first in list, 0 = last
    PVECTORED_EXCEPTION_HANDLER Handler   // pointer to handler function
);

// Handler function signature
LONG CALLBACK VectoredHandler(
    PEXCEPTION_POINTERS ExceptionInfo     // exception + context
);

// Return values from the handler:
#define EXCEPTION_CONTINUE_EXECUTION  (-1)  // resume execution
#define EXCEPTION_CONTINUE_SEARCH      (0)  // pass to next handler

C// ShellGhost's VEH registration
PVOID hVeh = AddVectoredExceptionHandler(
    1,              // First = 1: highest priority
    ShellGhostHandler
);

if (hVeh == NULL) {
    // Registration failed
    return -1;
}

// Later, when done:
RemoveVectoredExceptionHandler(hVeh);

4. The EXCEPTION_POINTERS Structure

The VEH handler receives a single argument: a pointer to an EXCEPTION_POINTERS structure. This structure contains everything the handler needs to understand the exception and modify the execution state:

Ctypedef struct _EXCEPTION_POINTERS {
    PEXCEPTION_RECORD ExceptionRecord;  // What happened
    PCONTEXT          ContextRecord;    // CPU state when it happened
} EXCEPTION_POINTERS, *PEXCEPTION_POINTERS;

Ctypedef struct _EXCEPTION_RECORD {
    DWORD ExceptionCode;          // e.g., 0x80000003 (BREAKPOINT)
    DWORD ExceptionFlags;         // 0 = first-chance, 1 = non-continuable
    struct _EXCEPTION_RECORD *ExceptionRecord;  // nested exception
    PVOID ExceptionAddress;       // instruction that caused the exception
    DWORD NumberParameters;       // number of params in array below
    ULONG_PTR ExceptionInformation[EXCEPTION_MAXIMUM_PARAMETERS];
} EXCEPTION_RECORD;

5. The CONTEXT Structure

The CONTEXT structure represents the complete CPU register state at the moment of the exception. This is the mechanism that allows the VEH handler to change execution flow:

C// Key fields of the x64 CONTEXT structure (simplified)
typedef struct _CONTEXT {
    // Control registers
    DWORD64 Rip;        // Instruction pointer - where to resume
    DWORD64 Rsp;        // Stack pointer
    DWORD   EFlags;     // Flags register (includes Trap Flag at bit 8)
    WORD    SegCs;      // Code segment

    // General-purpose registers
    DWORD64 Rax, Rcx, Rdx, Rbx;
    DWORD64 Rbp, Rsi, Rdi;
    DWORD64 R8, R9, R10, R11, R12, R13, R14, R15;

    // ... SSE registers, debug registers, etc.
} CONTEXT;

6. Handler Return Values

A VEH handler must return one of two values:

C// ShellGhost's handler logic (simplified)
LONG CALLBACK ShellGhostHandler(PEXCEPTION_POINTERS ep) {
    DWORD code = ep->ExceptionRecord->ExceptionCode;

    if (code == EXCEPTION_BREAKPOINT) {
        // Re-encrypt previous instruction (if any),
        // decrypt current instruction, toggle RW/RX
        // ... (covered in Module 6)
        return EXCEPTION_CONTINUE_EXECUTION;
    }

    // Not our exception - let someone else handle it
    return EXCEPTION_CONTINUE_SEARCH;
}

7. VEH Internals: The Handler List

Internally, Windows maintains VEH handlers in a doubly-linked list protected by a critical section. Understanding this structure helps explain registration and removal behavior:

C// Internal structure (ntdll, not officially documented)
typedef struct _VECTORED_HANDLER_ENTRY {
    LIST_ENTRY                    List;       // Doubly-linked list pointers
    ULONG                         Refs;       // Reference count
    PVECTORED_EXCEPTION_HANDLER   Handler;    // The handler function pointer
    // Note: In modern Windows, the handler pointer is
    // encoded with RtlEncodePointer for security
} VECTORED_HANDLER_ENTRY;

// The global list head lives in ntdll:
// ntdll!LdrpVectorHandlerList (exception handlers)
// ntdll!LdrpVectorHandlerList + sizeof(LIST_ENTRY) (continue handlers)

8. VEH in the ShellGhost Architecture

Here is how VEH fits into the overall ShellGhost execution model:

C// Complete VEH setup for ShellGhost (simplified)
#include <windows.h>
#include <stdio.h>

// Forward declaration of the handler
LONG CALLBACK GhostHandler(PEXCEPTION_POINTERS ep);

int main() {
    // Step 1: Prepare the per-instruction encrypted shellcode data
    // (generated by ShellGhost_mapping.py preprocessing script)
    unsigned char encrypted_sc[] = { /* per-instruction encrypted data */ };
    SIZE_T sc_size = sizeof(encrypted_sc);

    // Step 2: Allocate execution buffer filled with 0xCC (RW, not RWX)
    LPVOID exec_buf = VirtualAlloc(NULL, sc_size,
        MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE);
    memset(exec_buf, 0xCC, sc_size);

    // Step 3: Register VEH handler with highest priority
    PVOID hVeh = AddVectoredExceptionHandler(1, GhostHandler);

    // Step 4: Create a thread whose entry point is at the end of .text
    // ResolveEndofTextSegment() finds null bytes in .text to use as entry
    // The first 0xCC triggers EXCEPTION_BREAKPOINT
    LPVOID entry = ResolveEndofTextSegment();
    HANDLE hThread = CreateThread(NULL, 0,
        (LPTHREAD_START_ROUTINE)entry, NULL, 0, NULL);
    WaitForSingleObject(hThread, INFINITE);

    // Step 5: Cleanup (if shellcode returns)
    RemoveVectoredExceptionHandler(hVeh);
    VirtualFree(exec_buf, 0, MEM_RELEASE);
    return 0;
}