Difficulty: Beginner

Module 2: Windows Syscall Architecture

The gateway between user mode and the kernel -- one instruction to rule them all.

The Big Picture

Hell's Gate works by invoking the syscall instruction directly. To understand why this is powerful, you must understand the entire mechanism Windows uses to transition from user mode (Ring 3) to kernel mode (Ring 0). This module covers the hardware and software architecture that makes system calls possible on x64 Windows.

Protection Rings & Privilege Levels

x86-64 processors implement a privilege ring model. While the architecture supports four rings (0-3), Windows uses only two:

RingNameWhat Runs HereAccess
Ring 0Kernel Modentoskrnl.exe, drivers, kernel objectsFull hardware access, all memory, all instructions
Ring 3User ModeApplications, DLLs (including ntdll.dll)Restricted: cannot execute privileged instructions or access kernel memory

Your application code runs in Ring 3. It cannot directly access hardware, modify page tables, or touch kernel data structures. To perform any privileged operation (allocating memory, creating threads, opening files), it must ask the kernel via a system call.

Ring Transition via syscall

Ring 3 (User)
mov eax, SSN
→
syscall
instruction
→
Ring 0 (Kernel)
KiSystemCall64
→
SSDT Dispatch
Nt* kernel func

The syscall Instruction

On x86-64, the syscall instruction is the mechanism for Ring 3 to Ring 0 transitions. When the CPU executes syscall, the following happens atomically (in hardware):

ASM; What the CPU does when it executes 'syscall':
;
; 1. Save current RIP (return address) into RCX
;    RCX = address of instruction after syscall
;
; 2. Save current RFLAGS into R11
;    R11 = RFLAGS (processor flags)
;
; 3. Load RIP from IA32_LSTAR MSR
;    RIP = MSR[0xC0000082]  (this points to KiSystemCall64)
;
; 4. Load CS and SS from IA32_STAR MSR
;    CS = kernel code segment selector
;    SS = kernel stack segment selector
;
; 5. Mask RFLAGS with IA32_FMASK MSR
;    RFLAGS &= ~MSR[0xC0000084]  (clears IF, disables interrupts)
;
; 6. CPL changes from 3 to 0 (Ring 3 -> Ring 0)

The key Model-Specific Register (MSR) is IA32_LSTAR (address 0xC0000082). Windows sets this to point to nt!KiSystemCall64 during boot. Every syscall instruction jumps to this single kernel entry point.

Why RCX is Clobbered

The syscall instruction saves the return address in RCX. This destroys whatever was in RCX. Since the Windows x64 calling convention passes the first parameter in RCX, the ntdll stub must save RCX before executing syscall. That is why every stub starts with mov r10, rcx -- it moves the first parameter to R10 where the kernel expects it. If you build a direct syscall stub and forget this step, your first parameter will be the return address instead of your intended value.

KiSystemCall64: The Kernel Entry Point

When syscall transfers control to KiSystemCall64, the kernel performs several operations:

ASM; Simplified KiSystemCall64 flow:
;
; 1. Swap to kernel stack (via KPCR->Prcb->RspBase)
;    mov rsp, gs:[0x1A8]     ; load kernel stack pointer
;
; 2. Build a KTRAP_FRAME on the kernel stack
;    push saved registers (RCX, R11, RBP, etc.)
;
; 3. Validate the syscall number (EAX)
;    cmp eax, [KeServiceDescriptorTable.Limit]
;    jae KiSystemServiceExit  ; invalid SSN -> return error
;
; 4. Index into the SSDT to find the target function
;    lea r10, [KeServiceDescriptorTable.Base]
;    movsxd r11, [r10 + rax*4]  ; load encoded offset
;    sar r11, 4                  ; decode (shift right 4)
;    add r10, r11                ; r10 = kernel function address
;
; 5. Copy user-mode arguments from user stack to kernel stack
;
; 6. Call the kernel function (e.g., NtAllocateVirtualMemory)
;
; 7. Return via sysret (or iretq) back to Ring 3

The System Service Descriptor Table (SSDT)

The SSDT (also known as KeServiceDescriptorTable) is a kernel-mode structure that maps SSNs to kernel function addresses. It contains:

FieldPurpose
ServiceTableBaseArray of encoded offsets to Nt* kernel functions
ServiceCounterTableBasePerformance counters (unused on retail builds)
NumberOfServicesTotal number of valid syscall numbers
ParamTableBaseArray of byte-sized parameter counts for stack argument copying

On x64 Windows, the SSDT does not store raw pointers. Instead, it stores 4-byte signed offsets relative to the table base. The lower 4 bits encode the number of parameters (for stack argument copying). The kernel decodes each entry by shifting right by 4 and adding to the table base address.

C// How the kernel resolves a syscall number to a function pointer:
typedef struct _KSERVICE_TABLE_DESCRIPTOR {
    PLONG   ServiceTableBase;      // Array of encoded offsets
    PULONG  ServiceCounterTableBase;
    ULONG   NumberOfServices;      // e.g., ~470 on Win10 21H2
    PUCHAR  ParamTableBase;        // Bytes: num params per service
} KSERVICE_TABLE_DESCRIPTOR;

// Decoding: given SSN in EAX
LONG encodedOffset = ServiceTableBase[SSN];
PVOID kernelFunc = (PBYTE)ServiceTableBase + (encodedOffset >> 4);

Why SSN Is The Key

The entire dispatch mechanism hinges on the value in EAX at the time the syscall instruction executes. EAX holds the System Service Number (SSN). The kernel uses it as an index into the SSDT to find the function to call. If you know the SSN for NtAllocateVirtualMemory and you set EAX to that value before executing syscall, the kernel will allocate memory for you -- regardless of whether you called through ntdll or not. This is the entire foundation of Hell's Gate.

The Return Path: sysret

After the kernel function completes, the kernel returns to user mode via the sysret instruction (or iretq in some edge cases). The sysret instruction reverses the syscall process:

ASM; What the CPU does when it executes 'sysret':
;
; 1. Load RIP from RCX (return to user code)
; 2. Load RFLAGS from R11
; 3. Load CS and SS from IA32_STAR MSR (user segments)
; 4. CPL changes from 0 to 3 (Ring 0 -> Ring 3)
;
; The function's return value is in RAX (NTSTATUS)

After sysret, execution resumes at the instruction after the syscall in user mode. RAX contains the NTSTATUS return code indicating success or failure.

Complete Syscall Round-Trip

1. User code: mov r10, rcx (save 1st param)
2. User code: mov eax, SSN (load syscall number)
3. User code: syscall (Ring 3 → Ring 0)
4. Kernel: KiSystemCall64 → SSDT[EAX] → Nt* function
5. Kernel: Execute operation, set NTSTATUS in RAX
6. Kernel: sysret (Ring 0 → Ring 3)
7. User code: ret (return to caller with NTSTATUS)

Legacy: int 0x2e and the WoW64 Path

On older x86 Windows (32-bit), the transition used int 0x2e (interrupt) or sysenter instead of syscall. On x64 Windows running 32-bit (WoW64) processes, the syscall goes through wow64cpu.dll which transitions to 64-bit mode and then issues the 64-bit syscall. Hell's Gate targets native x64 processes and uses the syscall instruction directly.

Pop Quiz: Syscall Architecture

Q1: What MSR does the CPU read to find the kernel entry point when syscall executes?

IA32_LSTAR (Long STAR) at MSR 0xC0000082 contains the RIP value loaded when syscall executes. On Windows, this points to nt!KiSystemCall64.

Q2: Why does the ntdll stub execute mov r10, rcx before syscall?

The syscall instruction stores RIP (the return address) in RCX, destroying the first parameter. The stub saves it to R10 first. The kernel's KiSystemCall64 knows to read the first parameter from R10 instead of RCX.

Q3: The SSDT on x64 Windows stores entries as 4-byte signed values. How are they decoded to function pointers?

Each SSDT entry encodes a relative offset in the upper 28 bits and a parameter count in the lower 4 bits. The kernel shifts right by 4 (discarding the param count) and adds the result to the ServiceTableBase to get the kernel function address.
← Previous: EDR Hooking Next: System Service Numbers →