Module 1: EDR Hooks & The Detection Problem

Why calling NtAllocateVirtualMemory isn't as simple as it used to be.

1. What Is an EDR?

Endpoint Detection and Response (EDR) is the evolution of traditional antivirus. While AV relies on static signature matching, EDR monitors process behavior in real-time. It watches what your code does, not just what it looks like.

Every modern EDR deploys a user-mode component — typically a DLL that gets loaded into every process. This DLL patches (hooks) critical API functions so the EDR can inspect arguments and return values before they reach the kernel.

The user-mode hook is the most relevant to LayeredSyscall. It intercepts calls at the ntdll.dll level — the lowest user-mode layer before the kernel transition.

2. How EDR Hooks Work

Every Nt* function in ntdll.dll is a thin wrapper called a syscall stub. In its clean (unhooked) state, the stub looks like this:

Unhooked ntdll Syscall Stub

x86-64 ASMNtAllocateVirtualMemory:
  4C 8B D1          mov r10, rcx        ; save 1st arg (RCX clobbered by syscall)
  B8 18 00 00 00    mov eax, 0x18       ; SSN (System Service Number)
  0F 05             syscall             ; transition to Ring 0
  C3                ret                 ; return to caller

The stub is only 12 bytes. It loads the syscall number into EAX, executes the syscall instruction, and returns. No logic, no validation — just a gateway to the kernel.

Hooked ntdll Syscall Stub

An EDR replaces the first 5 bytes with a relative jump (E9 opcode) to its own inspection function:

x86-64 ASMNtAllocateVirtualMemory (HOOKED):
  E9 XX XX XX XX    jmp EDR_Hook        ; 5-byte relative jump to EDR DLL
  00 00 00          ; remaining bytes (corrupted/overwritten)
  0F 05             syscall             ; still present but never reached via normal flow
  C3                ret

The E9 jump replaces mov r10, rcx (3 bytes) and the first 2 bytes of mov eax, SSN. When your code calls NtAllocateVirtualMemory, execution is redirected to the EDR's hook handler, which:

3. Three Layers of Detection

Over time, EDRs developed multiple detection layers. Each one catches a different class of evasion technique. Understanding all three is critical to appreciating why LayeredSyscall exists.

Detection Logic// InstrumentationCallback pseudo-code
if (returnAddress < ntdll_base || returnAddress > ntdll_end) {
    // syscall did NOT originate from ntdll
    // Flag as DIRECT SYSCALL → alert/block
}

Legitimate Stackntdll!NtAllocateVirtualMemory      ; syscall stub
ntdll!RtlpAllocateHeapInternal      ; internal ntdll logic
KERNELBASE!VirtualAlloc             ; Win32 wrapper
kernel32!VirtualAllocStub           ; kernel32 thunk
myapp.exe!main                      ; application code

Anomalous Stackntdll!NtAllocateVirtualMemory      ; syscall stub
myapp.exe!some_function             ; DIRECT jump from EXE to ntdll!
                                     ; Where's kernel32? KERNELBASE?

4. Why Indirect Syscalls Aren't Enough

Indirect syscalls (used by SysWhispers2, HWSyscalls, etc.) solve Layer 2 by jumping to the syscall instruction inside ntdll's own memory. The return address now correctly points to ntdll, so InstrumentationCallback sees nothing wrong.

Some tools attempted to solve Layer 3 by spoofing call stack frames — manually pushing fake return addresses onto the stack before the syscall. However, advanced EDRs can validate these frames by checking if the return addresses actually correspond to valid CALL instruction sites in the supposed caller modules.

5. What LayeredSyscall Solves

6. Real-World EDR Hooking Examples

Different EDR vendors implement their hooks in slightly different ways, but the core technique remains the same. Here are common patterns observed in the wild:

Detecting Hooks Programmatically

If you wanted to check whether a function is hooked, you could compare the first bytes against the expected clean stub pattern. This is how tools like Hell's Gate determine if they can read the SSN directly:

C++BOOL IsStubHooked(PVOID funcAddr) {
    BYTE* p = (BYTE*)funcAddr;

    // Clean stub starts with: 4C 8B D1 B8 (mov r10, rcx; mov eax, ...)
    if (p[0] == 0x4C && p[1] == 0x8B && p[2] == 0xD1 && p[3] == 0xB8) {
        return FALSE;  // Unhooked
    }

    // Check for common hook patterns
    if (p[0] == 0xE9)         return TRUE;  // JMP rel32
    if (p[0] == 0xFF)         return TRUE;  // JMP [rip+disp32]
    if (p[0] == 0xEB)         return TRUE;  // JMP short

    return TRUE;  // Unknown pattern, assume hooked
}

LayeredSyscall doesn't need this check because its SSN resolution method (Module 2) is completely independent of stub bytes. However, understanding hook detection helps appreciate why earlier techniques were fragile.

Key Terminology