Difficulty: Beginner

Module 1: EDR Hooking & The Problem

Why calling NtAllocateVirtualMemory normally is like walking through a metal detector.

Why This Module?

Before understanding Hell's Gate (by am0nsec & RtlMateusz), you need to understand the problem it solves. Modern Endpoint Detection and Response (EDR) products hook critical Windows API functions in userland to inspect every sensitive operation your code performs. This module explains exactly how that hooking works, why it catches attackers, and why direct syscalls became necessary.

The Windows API Call Chain

When your program calls a Windows API function, the call travels through multiple layers before reaching the kernel. Understanding this chain is critical because EDRs insert themselves at a strategic point in the middle.

Normal API Call Flow

Your Code
VirtualAlloc()
→
kernel32.dll
VirtualAlloc
→
ntdll.dll
NtAllocateVirtualMemory
→
Kernel
syscall

The Win32 API (kernel32.dll, kernelbase.dll) provides the documented, user-friendly interface. These functions perform parameter validation and then call down to ntdll.dll, the lowest user-mode layer. The Nt* functions in ntdll.dll contain tiny stubs that execute the syscall instruction to transition into the kernel. The kernel then dispatches the request to the appropriate kernel function via the System Service Descriptor Table (SSDT).

How EDR Hooking Works

EDR products inject a DLL into every process at startup (often via a kernel driver callback on process creation). This DLL modifies the first bytes of critical ntdll.dll functions in memory, replacing them with a JMP instruction that redirects execution into the EDR's own inspection code.

ASM; BEFORE hooking - clean NtAllocateVirtualMemory stub
NtAllocateVirtualMemory:
  4c 8b d1          mov r10, rcx        ; save 1st param
  b8 18 00 00 00    mov eax, 0x18       ; SSN = 0x18 (Win10 1903)
  0f 05             syscall             ; transition to kernel
  c3                ret

; AFTER EDR hooks - first bytes overwritten
NtAllocateVirtualMemory:
  e9 xx xx xx xx    jmp EDR_Hook_Func   ; 5-byte relative JMP
  b8 18 00 00 00    mov eax, 0x18       ; (partially clobbered)
  0f 05             syscall
  c3                ret

When your code calls NtAllocateVirtualMemory, instead of executing the syscall stub, execution jumps to the EDR's hook function. The EDR inspects the parameters (what memory are you allocating? what permissions? which process?), makes a decision (allow, block, or alert), and then either calls the original function or denies the request.

What EDRs Actually Inspect

EDR hooks are not just logging calls. They perform deep inspection: checking if the target process is a different process (injection), whether the memory permissions include RWX (shellcode staging), whether the caller is a known legitimate module, and correlating multiple calls into behavioral chains. A single NtAllocateVirtualMemory + NtWriteVirtualMemory + NtCreateThreadEx sequence in a remote process is a classic injection pattern that will immediately trigger an alert.

The Inline Hook Mechanism

The most common hooking technique used by EDRs is the inline hook (also called a trampoline hook). Here is how it works step by step:

StepActionDetail
1Save original bytesEDR copies the first 5+ bytes of the target function to a trampoline buffer
2Write JMPEDR overwrites those bytes with JMP rel32 (E9 + 4-byte offset) pointing to its handler
3Intercept callWhen the function is called, execution jumps to the EDR handler
4Inspect & decideEDR analyzes parameters, context, and caller. Decides allow/block/alert
5Call trampolineIf allowed, EDR calls the trampoline (saved original bytes + JMP back) to execute the real function

Hooked API Call Flow

Your Code
NtAllocateVirtualMemory()
→
JMP to
EDR Hook
→
EDR Inspection
(params, caller, behavior)
→
Trampoline
→ real syscall

Which Functions Get Hooked?

EDRs do not hook every function in ntdll.dll -- that would destroy performance. They focus on security-sensitive Nt* functions. Common targets include:

C// Memory operations (shellcode staging)
NtAllocateVirtualMemory    // Allocate memory in a process
NtWriteVirtualMemory       // Write data to process memory
NtProtectVirtualMemory     // Change memory permissions (e.g., RW -> RX)
NtMapViewOfSection         // Map shared memory (alternative injection)

// Thread/process creation (code execution)
NtCreateThreadEx           // Create a new thread (remote thread injection)
NtQueueApcThread           // Queue APC to a thread (APC injection)
NtResumeThread             // Resume a suspended thread
NtCreateProcess/Ex         // Create a new process

// Handle operations
NtOpenProcess              // Open a handle to another process
NtDuplicateObject          // Duplicate handles across processes

// File and registry (persistence, payload drop)
NtCreateFile               // Create or open files
NtWriteFile                // Write to files

The Key Insight

All EDR userland hooks exist in user-mode memory inside your own process. The EDR DLL and its JMP patches are in your address space. This means your code has full read access to the hooked bytes -- and theoretically, write access too. This is the fundamental weakness that Hell's Gate and other direct syscall techniques exploit: if you can figure out the System Service Number (SSN) yourself, you can issue the syscall instruction directly without ever touching the hooked ntdll stub.

Why Not Just Unhook ntdll?

One common evasion technique is to simply restore the original bytes of ntdll.dll functions by reading a clean copy from disk. While this works, it has several drawbacks:

ApproachProsCons
Read ntdll from diskFull clean copy of all stubsFile I/O is logged; re-mapping ntdll is detectable via ETW or kernel callbacks
Read ntdll from KnownDllsAvoids disk I/OStill requires NtOpenSection + NtMapViewOfSection which may be hooked
Overwrite hooked bytesSimple implementationEDR may monitor for writes to ntdll pages; periodic integrity checks catch this
Direct syscalls (Hell's Gate)Never touches hooked code at allNeed to resolve SSN; syscall from non-ntdll region is detectable by kernel instrumentation

Hell's Gate takes the approach of reading the SSN from the ntdll stub bytes (or neighboring stubs if hooked) and then executing the syscall instruction from its own assembly stub. The hooked function is never called at all -- the entire ntdll trampoline is bypassed.

The Hell's Gate Philosophy

Instead of fighting the hook (unhooking, re-mapping), Hell's Gate reads through the hook. The EDR overwrites the first bytes of the stub, but the SSN (in the mov eax, SSN instruction) is often still intact at an offset beyond the JMP patch. Even when the SSN itself is clobbered, neighboring clean stubs can be used to calculate the target SSN. Once you have the number, you invoke the syscall instruction yourself. The hook is irrelevant because you never execute through it.

Pop Quiz: EDR Hooking Fundamentals

Q1: Where in the API call chain do EDR inline hooks typically reside?

EDRs patch the first bytes of ntdll.dll Nt* functions with JMP instructions that redirect to their inspection code. This is in user-mode memory, making it both effective and bypassable.

Q2: What is the primary advantage of Hell's Gate over ntdll unhooking?

Hell's Gate reads the SSN from the stub bytes (or calculates it from neighbors) and executes syscall directly. It never writes to ntdll memory or re-maps the DLL, so EDR integrity checks for unhooking activity see nothing suspicious.

Q3: An EDR hook replaces the first 5 bytes of NtCreateThreadEx with E9 xx xx xx xx. What instruction is this?

The opcode E9 is a 5-byte relative JMP instruction (1-byte opcode + 4-byte signed offset). This is the most common inline hook: it redirects execution to the EDR's interception function at a calculated relative address.
Next: Syscall Architecture →