Difficulty: Intermediate

Module 4: ntdll.dll Stub Anatomy

The 20-byte pattern that every direct syscall technique is built on.

From Theory to Bytes

Now we move from abstract concepts to concrete bytes. Every Nt*/Zw* syscall function in ntdll.dll follows a predictable machine code pattern. Hell's Gate reads these bytes directly from memory to extract the SSN. In this module, you will learn the exact byte layout of a syscall stub, how to find these stubs via the Export Address Table (EAT), and the data structures Hell's Gate uses to represent them.

The Syscall Stub Pattern

On x64 Windows 10 and later, every Nt* syscall stub in ntdll.dll follows this byte pattern:

ASM; Standard ntdll syscall stub layout (x64, Windows 10+)
; Total: approximately 20 bytes
;
; Offset  Bytes              Instruction
; ------  -----------------  -----------
; 0x00    4c 8b d1           mov r10, rcx        ; save 1st param (RCX clobbered by syscall)
; 0x03    b8 XX XX 00 00     mov eax, <SSN>      ; load System Service Number
; 0x08    f6 04 25 08 03 fe  test byte ptr [7FFE0308h], 1  ; check SharedUserData
; 0x0F    7f 01                                   ; (VBS/HVCI check)
; 0x10    75 03              jnz  short +3        ; if set, use int 2e path
; 0x12    0f 05              syscall              ; Ring 3 -> Ring 0
; 0x14    c3                 ret                  ; return to caller
; 0x15    cd 2e              int 0x2e             ; alternative syscall path
; 0x17    c3                 ret

Stub Layout Varies by Windows Version

The exact byte layout has changed across Windows versions. On Windows 7, the stub is simpler (no hypervisor check). On early Windows 10 builds, the test/jnz sequence differs. The critical constant that Hell's Gate relies on is the first 8 bytes: the mov r10, rcx (3 bytes) followed by mov eax, SSN (5 bytes). These are stable across all x64 Windows versions. Hell's Gate checks for the byte pattern 4c 8b d1 b8 at the start to confirm a clean, unhooked stub.

Byte-Level Breakdown

Let us examine the bytes that Hell's Gate reads and validates:

OffsetBytesMeaningHell's Gate Check
0x004c 8b d1mov r10, rcxByte 0 must be 0x4c, byte 1 must be 0x8b, byte 2 must be 0xd1
0x03b8mov eax, (opcode)Byte 3 must be 0xb8
0x04XX XX 00 00SSN value (little-endian DWORD)Extract as *(WORD*)(stub + 4) -- the SSN
0x06-0x0700 00High 2 bytes of SSN (always zero, SSNs < 0x10000)Used as secondary validation

Hell's Gate reads the SSN from bytes at offset 4 and 5 of the stub. Since SSNs are always less than ~500, bytes 6 and 7 are always 0x00, which provides an additional integrity check.

C// How Hell's Gate reads the SSN from a stub pointer
// 'pFunctionAddress' points to the start of the Nt* function
BYTE* stub = (BYTE*)pFunctionAddress;

// Validate the stub pattern
if (stub[0] == 0x4c &&    // mov r10, rcx
    stub[1] == 0x8b &&
    stub[2] == 0xd1 &&
    stub[3] == 0xb8) {    // mov eax, <imm32>

    // Extract SSN from bytes 4-5 (little-endian 16-bit)
    WORD ssn = *(WORD*)(stub + 4);
    // ssn now holds the System Service Number
}

Finding Stubs via the Export Address Table

To find the syscall stubs, Hell's Gate walks the Export Address Table (EAT) of ntdll.dll. The EAT is a PE data structure that maps function names to their addresses. Here is the process:

EAT Walking Process

1. Find ntdll.dll base address (PEB → InMemoryOrderModuleList)
2. Parse PE headers: DOS Header → NT Headers → Optional Header
3. Locate Export Directory: DataDirectory[IMAGE_DIRECTORY_ENTRY_EXPORT]
4. Walk AddressOfNames[] array to find target function name
5. Use AddressOfNameOrdinals[] to get the ordinal index
6. Use AddressOfFunctions[ordinal] to get the function RVA
7. Add ntdll base + RVA = absolute address of syscall stub

Hell's Gate Data Structures

The original Hell's Gate code defines two key structures to represent a resolved syscall:

C// From HellsGate main.c - the VX_TABLE_ENTRY structure
typedef struct _VX_TABLE_ENTRY {
    PVOID   pAddress;        // Address of the Nt* function in ntdll
    DWORD64 dwHash;          // DJB2 hash of the function name
    WORD    wSystemCall;     // The resolved SSN (System Service Number)
} VX_TABLE_ENTRY, *PVX_TABLE_ENTRY;

// The VX_TABLE holds entries for all syscalls we need
typedef struct _VX_TABLE {
    VX_TABLE_ENTRY NtAllocateVirtualMemory;
    VX_TABLE_ENTRY NtProtectVirtualMemory;
    VX_TABLE_ENTRY NtCreateThreadEx;
    VX_TABLE_ENTRY NtWaitForSingleObject;
} VX_TABLE, *PVX_TABLE;

Each VX_TABLE_ENTRY stores three pieces of information: the address of the function in ntdll (for reading the stub bytes), a hash of the function name (for locating it in the EAT without storing the plaintext name), and the resolved SSN (the value that will be placed in EAX before syscall).

PEB Walking to Find ntdll Base

Before parsing the EAT, Hell's Gate needs the base address of ntdll.dll. It finds this by walking the Process Environment Block (PEB):

C// Hell's Gate finds ntdll.dll through the PEB
// The PEB is accessible via the TEB (Thread Environment Block)
// which is at gs:[0x60] on x64 Windows

// PEB->Ldr->InMemoryOrderModuleList contains loaded modules
// Module order: [0] = process exe, [1] = ntdll.dll

PPEB pPeb = (PPEB)__readgsqword(0x60);  // Read PEB from TEB

PLDR_DATA_TABLE_ENTRY pLdrEntry =
    (PLDR_DATA_TABLE_ENTRY)
    ((PBYTE)pPeb->Ldr->InMemoryOrderModuleList.Flink->Flink  // 2nd entry = ntdll
     - 0x10);  // adjust for LIST_ENTRY offset

PVOID ntdllBase = pLdrEntry->DllBase;  // ntdll.dll base address

Why ntdll Is Always Second

In the InMemoryOrderModuleList, the first entry is always the process executable itself. The second entry is always ntdll.dll, because it is the first DLL loaded by the kernel (it is mapped before the process entry point runs). This is reliable across all Windows versions and is a standard technique used by many offensive tools.

API Hashing: Avoiding Plaintext Strings

Hell's Gate does not store function names as plaintext strings (which would be trivially detectable by static analysis). Instead, it uses DJB2 hashing to compare function names:

C// DJB2 hash function used by Hell's Gate
DWORD64 djb2(PBYTE str) {
    DWORD64 dwHash = 0x7734773477347734;  // Custom seed
    CHAR c;

    while (c = *str++)
        dwHash = ((dwHash << 0x5) + dwHash) + c;  // hash * 33 + c

    return dwHash;
}

// Pre-computed hashes for target functions:
// djb2("NtAllocateVirtualMemory") = 0xF5BD373480A6B89B
// djb2("NtProtectVirtualMemory")  = 0x858BCB1046FB6A37
// djb2("NtCreateThreadEx")        = 0x64DC7DB288C5015F
// djb2("NtWaitForSingleObject")   = 0xC6A2FA174E551BCB

When walking the EAT, Hell's Gate computes the DJB2 hash of each exported function name and compares it against pre-computed target hashes. This avoids embedding suspicious strings like "NtAllocateVirtualMemory" in the binary.

The Hashing Trade-Off

API hashing prevents simple string-based detection, but it introduces a different signature: the hash values and the hashing algorithm themselves become IOCs (Indicators of Compromise). Threat intelligence teams maintain databases of known API hash values. Using a custom seed (as Hell's Gate does with 0x7734773477347734) helps, but the DJB2 algorithm structure is still recognizable.

Pop Quiz: Stub Anatomy

Q1: In the ntdll syscall stub, at what byte offset is the SSN located?

The first 3 bytes are mov r10, rcx (4c 8b d1), byte 3 is the mov eax opcode (b8), and bytes 4-7 are the SSN as a 32-bit little-endian immediate. The SSN occupies bytes 4 and 5 (with bytes 6-7 being 0x00 since SSNs are small).

Q2: How does Hell's Gate detect that a stub has NOT been hooked?

Hell's Gate validates that the stub starts with the expected byte sequence: 0x4c 0x8b 0xd1 (mov r10, rcx) followed by 0xb8 (mov eax, imm32). If these bytes are different (e.g., replaced by E9 for a JMP hook), the stub is hooked.

Q3: Why does Hell's Gate use DJB2 hashing instead of string comparison for function names?

Storing "NtAllocateVirtualMemory" as a string in the binary is an obvious detection signature. Hashing the target names and comparing hash values at runtime hides the intent from static string scanning, though the hash values themselves can become known IOCs.
← Previous: SSN Basics Next: Hell's Gate Algorithm →