Module 3: Windows Exception Handling & VEH

The mechanism that lets LayeredSyscall intercept and redirect CPU execution.

1. Structured Exception Handling (SEH)

SEH is the traditional Windows exception handling mechanism. It is frame-based and per-function, meaning handlers are tied to specific stack frames and function scopes.

C// SEH example - handler is scoped to this function
__try {
    // Code that might fault
    int *p = NULL;
    int x = *p;  // ACCESS_VIOLATION
}
__except (EXCEPTION_EXECUTE_HANDLER) {
    // Only catches exceptions from within this __try block
    printf("Caught exception in this function\n");
}

SEH handlers are registered on the stack and automatically cleaned up when the function returns. They only handle exceptions that occur within their __try scope. This makes SEH unsuitable for LayeredSyscall, which needs to intercept exceptions that occur deep inside ntdll — far from any __try block in its own code.

2. Vectored Exception Handling (VEH)

VEH is a process-wide exception handling mechanism introduced in Windows XP. Unlike SEH, VEH handlers are not tied to any stack frame. They are stored in an internal linked list and are invoked for every exception in the process, regardless of where it occurs.

C++// VEH handler - process-wide, catches everything
LONG WINAPI MyVehHandler(PEXCEPTION_POINTERS ExceptionInfo) {
    if (ExceptionInfo->ExceptionRecord->ExceptionCode == EXCEPTION_ACCESS_VIOLATION) {
        // Handle the exception...
        return EXCEPTION_CONTINUE_EXECUTION;  // Resume execution
    }
    return EXCEPTION_CONTINUE_SEARCH;  // Not ours, pass to next handler
}

// Registration - CALL_FIRST (1) means this handler goes to the front of the list
PVOID hVeh = AddVectoredExceptionHandler(1, MyVehHandler);

The Order parameter controls placement in the handler chain:

LayeredSyscall registers its handlers with CALL_FIRST to ensure they execute before any other exception handlers in the process.

3. Exception Dispatch Flow

When a CPU exception occurs (access violation, breakpoint, single-step, etc.), Windows follows a well-defined dispatch sequence:

The critical insight for LayeredSyscall: VEH handlers run before everything else. This means LayeredSyscall's handler gets first crack at every exception in the process, including hardware breakpoint traps fired inside ntdll.

4. The EXCEPTION_POINTERS Structure

Every VEH handler receives a pointer to EXCEPTION_POINTERS, which bundles two critical pieces of information:

C++typedef struct _EXCEPTION_POINTERS {
    PEXCEPTION_RECORD ExceptionRecord;  // What happened
    PCONTEXT          ContextRecord;     // CPU state when it happened
} EXCEPTION_POINTERS, *PEXCEPTION_POINTERS;

ExceptionRecord — What Happened

ContextRecord — Full CPU State

The CONTEXT structure contains the complete register state of the CPU at the time of the exception. This is the key to LayeredSyscall's power:

C++// Key fields in CONTEXT (x64) relevant to LayeredSyscall
typedef struct _CONTEXT {
    // Control registers
    DWORD64 Rip;        // Instruction pointer (where to resume)
    DWORD64 Rsp;        // Stack pointer
    DWORD   EFlags;     // CPU flags (includes Trap Flag at bit 8)

    // Integer registers (function arguments in x64 calling convention)
    DWORD64 Rcx;        // 1st argument
    DWORD64 Rdx;        // 2nd argument
    DWORD64 R8;         // 3rd argument
    DWORD64 R9;         // 4th argument
    DWORD64 Rax;        // Return value / SSN for syscall
    DWORD64 R10;        // Saved 1st arg (after mov r10, rcx)

    // Additional integer registers
    DWORD64 Rbx, Rbp, Rsi, Rdi;
    DWORD64 R11, R12, R13, R14, R15;

    // Debug registers (hardware breakpoints!)
    DWORD64 Dr0;        // Breakpoint address 0
    DWORD64 Dr1;        // Breakpoint address 1
    DWORD64 Dr2;        // Breakpoint address 2
    DWORD64 Dr3;        // Breakpoint address 3
    DWORD64 Dr6;        // Debug status
    DWORD64 Dr7;        // Debug control (enable/disable)
    // ... floating point, SSE, etc.
} CONTEXT;

5. Handler Return Values

A VEH handler must return one of two values, which tells the exception dispatcher what to do next:

C++LONG WINAPI ExampleHandler(PEXCEPTION_POINTERS ExceptionInfo) {
    DWORD code = ExceptionInfo->ExceptionRecord->ExceptionCode;

    if (code == EXCEPTION_SINGLE_STEP) {
        // This is our hardware breakpoint!
        // Modify context...
        ExceptionInfo->ContextRecord->Rax = newSsn;
        return EXCEPTION_CONTINUE_EXECUTION;  // Resume with our changes
    }

    // Not our exception, let someone else handle it
    return EXCEPTION_CONTINUE_SEARCH;
}

6. Triggering Exceptions Intentionally

LayeredSyscall doesn't wait for exceptions to happen naturally. It deliberately triggers an EXCEPTION_ACCESS_VIOLATION (code 0xC0000005) to enter its VEH handler on demand:

C++// From LayeredSyscall's code - intentional null pointer dereference
#define TRIGGER_ACCESS_VIOLOATION_EXCEPTION  int *a = 0; int b = *a;

// Usage in a wrapped API function:
NTSTATUS LayeredNtAllocateVirtualMemory(...) {
    // Set up arguments in global state
    SetupSyscallArgs(SSN_NtAllocateVirtualMemory, args...);

    // Deliberately crash to enter VEH handler
    TRIGGER_ACCESS_VIOLOATION_EXCEPTION

    // Execution never reaches here normally -
    // the VEH handler redirects RIP
}

When the null dereference occurs:

1. CPU raises an access violation exception
2. Windows dispatches it to VEH handlers
3. LayeredSyscall's AddHwBp handler catches it
4. Handler reads the desired syscall info from global state
5. Handler sets hardware breakpoints on the target syscall stub
6. Handler advances RIP past the faulting instruction to avoid re-triggering
7. Handler calls a legitimate API (e.g., WriteFile) that will eventually hit the breakpoint

C++// Skip past the faulting instruction
ExceptionInfo->ContextRecord->Rip += INSTRUCTION_SIZE;
return EXCEPTION_CONTINUE_EXECUTION;

7. LayeredSyscall's VEH Setup

LayeredSyscall registers two VEH handlers during initialization, each with a different role:

C++// From InitializeHooks() in LayeredSyscall
PVOID h1 = AddVectoredExceptionHandler(CALL_FIRST, AddHwBp);
PVOID h2 = AddVectoredExceptionHandler(CALL_FIRST, HandlerHwBp);

This dual-handler architecture is explored in detail in Module 5. For now, the key takeaway is that VEH provides the mechanism for intercepting CPU exceptions process-wide and modifying the execution context — which is the foundation of everything LayeredSyscall does.

Common Exception Codes Reference