Module 8: Full Chain, Output Formats & Detection

End-to-end execution walkthrough, PIC output structure, COFF/PE wrapping mechanics, entropy analysis, emulation-based detection, and comparison with other encoder frameworks.

1. Full Execution Chain Walkthrough

Let us trace what happens from the moment a Shoggoth-encrypted payload begins executing in memory. This walkthrough assumes both encryption stages are active (the default):

Step-by-Step Execution

1. Junk Preamble (variable ~50-500 bytes): The first bytes executed are garbage instructions — NOPs, self-canceling pairs, jump-over blocks. They execute harmlessly and fall through to the block cipher decoder.
2. Block Cipher Decoder: Uses lea reg, [rip + offset] to locate the doubly-encrypted payload. Iterates over 8-byte blocks, applying the inverse operation chain (e.g., ROR → SUB → XOR). After this loop completes, the data is now only RC4-encrypted.
3. Junk Interlude (variable): More garbage instructions separate the two decoder stubs. This prevents a pattern like “block loop immediately followed by RC4 setup” from becoming a signature.
4. RC4 Decoder: Allocates 256 bytes on the stack for the S-box. Runs the KSA using the embedded key to initialize the permutation. Then runs the PRGA, generating keystream bytes and XORing them with the remaining encrypted data. After completion, restores the stack.
5. Control Transfer: Execution falls through (or jumps) to the now-decrypted payload. In raw mode, this is the original shellcode. In PE/COFF mode, this is the PIC loader followed by the PE/COFF file.
6. PE/COFF Loader (if applicable): The PIC loader resolves API addresses by walking PEB → Ldr → InMemoryOrderModuleList, maps PE sections or COFF symbols, applies relocations, and transfers control to the original payload’s entry point.

2. PIC Output Memory Layout

The flat PIC blob that Shoggoth produces has no headers, no sections, no metadata — just raw executable machine code followed by encrypted data. This is intentional: any structural metadata would become a signature.

Layout+--------------------------------------------------+
| Offset 0x0000: Junk Preamble                     |  Variable size
|   - Side-effect-free instructions                 |  (~50-500 bytes)
|   - Jump-over blocks, fake calls                  |
+--------------------------------------------------+
| Block Cipher Decoder Stub                         |  Variable size
|   - Register setup (random regs)                  |  (~100-400 bytes)
|   - LEA to locate encrypted data                  |
|   - Decryption loop (inverse ops + junk)          |
|   - Keys embedded as immediates                   |
+--------------------------------------------------+
| Junk Interlude                                    |  Variable size
+--------------------------------------------------+
| RC4 Decoder Stub                                  |  Variable size
|   - SUB RSP, 256 (S-box allocation)              |  (~200-600 bytes)
|   - KSA loop (with junk)                          |
|   - PRGA loop (with junk)                         |
|   - ADD RSP, 256 (stack restore)                  |
+--------------------------------------------------+
| RC4 Key Data                                      |  8-32 bytes
+--------------------------------------------------+
| Encrypted Payload                                 |  Same size as
|   (doubly encrypted: RC4 then block cipher)       |  original input
+--------------------------------------------------+

3. PE Mode: Reflective Loading

When Shoggoth wraps a PE file, the decrypted output is not a standalone executable — it is the PIC PE loader followed by the PE file bytes. The loader acts as a minimal reflective loader:

All API addresses (VirtualAlloc, VirtualProtect, LoadLibraryA, GetProcAddress) are resolved at runtime by the loader through PEB walking — no import table in the PIC blob itself.

4. COFF Mode: BOF Loading

COFF/BOF (Beacon Object File) mode handles .o object files, commonly used with Cobalt Strike’s inline-execute or similar BOF runners. The COFF loader performs a simplified version of what a linker does:

• Parse COFF headers — read section table, symbol table, string table
• Map sections — allocate memory and copy section data
• Process relocations — apply COFF relocations (IMAGE_REL_AMD64_ADDR64, IMAGE_REL_AMD64_REL32, etc.)
• Resolve external symbols — BOF external functions (like BeaconOutput, Windows APIs) are resolved through a function table or PEB walking
• Call entry point — execute the go function (standard BOF entry point) with optional arguments

The --coff-arg flag allows passing arguments to the BOF in the format expected by Cobalt Strike’s BeaconDataParse API, generated using the included beacon_generate.py script.

5. Detection: Entropy Analysis

One of the most effective detection techniques against encrypted payloads is entropy analysis. Shannon entropy measures the randomness of data on a scale from 0 (completely uniform) to 8 (maximum randomness for byte data). Encrypted data has characteristically high entropy:

Defenders can use sliding-window entropy analysis to flag regions of a binary that transition sharply from normal code entropy (~6.0) to encrypted data entropy (~7.9). This does not reveal what the payload is, but it identifies that encryption is present — which is suspicious in shellcode.

6. Detection: Emulation & Sandbox Analysis

Despite Shoggoth’s junk code slowing emulation, advanced detection still relies on emulation-based unpacking:

The fundamental limitation of any encryption-based evasion (including Shoggoth) is that the payload must be decrypted in memory before it can execute. At that moment, it is vulnerable to memory scanning. Polymorphism protects the file on disk and the decoder stub from signature matching, but the decrypted payload is the original unmodified shellcode or PE file.

7. Detection: Behavioral Heuristics

Beyond entropy and emulation, behavioral heuristics target the execution patterns common to polymorphic decoders:

• Loop-then-jump pattern: A tight loop that modifies a memory region followed by a jump into that region is strongly indicative of a decoder stub
• Stack-allocated S-box: Allocating exactly 256 bytes on the stack, filling with an identity permutation, then permuting — this is the RC4 KSA fingerprint
• RIP-relative self-reference: Code that computes its own address and then reads/writes data relative to itself suggests position-independent self-modifying code
• High junk-to-real instruction ratio: Code with an unusually high proportion of side-effect-free instructions suggests automated junk insertion
• Entropy transition in code flow: Execution transitioning from normal code into a high-entropy region that it just wrote to

8. Comparison with Other Frameworks

How does Shoggoth compare with other commonly used payload encoding tools?

9. Limitations & Operational Considerations

Understanding Shoggoth’s limitations is as important as understanding its capabilities:

Shoggoth is best understood as one layer in a defense-in-depth offensive strategy. It protects the payload at rest (on disk, in transit) and provides initial evasion when the payload first begins executing. For ongoing runtime protection, it should be combined with sleep obfuscation, in-memory encryption, and behavioral evasion techniques.

10. Course Summary