Module 2: Polymorphic Engines — Concepts

What polymorphism means in the malware context, how it differs from metamorphism, and the historical evolution of self-mutating code.

1. Defining Polymorphism

In biology, polymorphism refers to organisms of the same species existing in multiple forms. In the context of executable code, polymorphism means that a program can change its own binary representation while preserving its original functionality. Each instance of a polymorphic program looks different at the byte level, but they all perform the same computation when executed.

More precisely, a polymorphic engine takes an input payload P and produces an output E(P) such that:

• Functional equivalence: Executing E(P) eventually produces the same behavior as executing P
• Structural variance: Given two outputs E1(P) and E2(P) from different runs, the byte content of E1 and E2 differs substantially
• Self-contained decryption: E(P) includes all code necessary to reverse the transformation and recover P at runtime

The output typically consists of two parts: a decoder stub (variable code that performs decryption) and an encrypted payload (the original code, transformed to be unreadable without the decoder). The polymorphic property applies primarily to the decoder stub — it is the part that must change between generations to avoid signature detection.

2. Polymorphic vs Metamorphic

These two terms are often confused. While both involve code mutation, they operate at fundamentally different levels:

3. Historical Evolution

Polymorphic engines have a four-decade lineage in the virus world. Understanding this history contextualizes Shoggoth as a modern implementation of well-established principles.

3.1 The Early Era (1990–1992)

The concept of self-encrypting code appeared in the late 1980s, but the first true polymorphic engine was the Mutation Engine (MtE) released in 1992 by the virus author known as Dark Avenger. MtE was distributed as a linkable object file — any virus author could incorporate it. It generated random decryption loops using different registers, instruction orderings, and interspersed junk instructions.

MtE demonstrated that polymorphism could be packaged as a reusable library rather than being tightly coupled to a specific virus. This modular approach directly parallels how Shoggoth functions today: a standalone tool that takes any payload and wraps it in polymorphic packaging.

3.2 The Arms Race (1993–2000)

After MtE, virus authors and AV vendors entered an escalating arms race. Engines like SMEG (Simulated Metamorphic Encryption Generator) and TPE (Trident Polymorphic Engine) pushed the boundaries with more registers, more junk instruction types, and more complex control flow in the decoder. AV vendors responded with generic decryption (GD) — running the code in an emulator until the payload was decrypted, then scanning the cleartext result.

3.3 Metamorphic Leap (2000–2010)

Some virus authors moved beyond polymorphism to full metamorphism. Win32.Simile (2002) could rewrite its own code using instruction substitution, register reassignment, and code transposition — all without encrypting anything. Win32.ZMist (by the author Z0mbie) took this further with code integration, embedding itself within the host program’s control flow graph.

3.4 Modern Offensive Tools (2010–Present)

The red team and offensive security community adopted polymorphic techniques for payload generation. Tools like msfvenom (Metasploit) offer encoder chains (shikata_ga_nai, xor_dynamic), Veil generates polymorphic payloads in various languages, and Shoggoth brings true asmjit-based x86-64 polymorphic encoding with multi-stage encryption and sophisticated junk code generation.

4. Core Components of a Polymorphic Engine

Every polymorphic engine, from MtE to Shoggoth, must implement a set of core components. The sophistication of each component determines the engine’s evasion capability:

5. The Decoder Stub Anatomy

The decoder stub is the heart of any polymorphic output. It must accomplish a fixed set of tasks, but the way it accomplishes them varies. Here is a conceptual pseudocode for a polymorphic decoder:

Pseudocode// Conceptual decoder stub structure (varies each generation)
//
// 1. Locate encrypted payload (RIP-relative addressing)
// 2. Set up decryption loop counter
// 3. For each block/byte of encrypted data:
//    a. Apply inverse operations (reverse order of encryption)
//    b. Advance pointer
// 4. Transfer control to decrypted payload
//
// What changes between generations:
//   - Which registers hold the pointer, counter, key
//   - Whether the loop counts up or down
//   - Which instructions implement each operation
//   - Where junk instructions are inserted
//   - Whether operations are inlined or use subroutines

The invariant is the algorithm (decrypt the payload, then jump to it). Everything else — the registers, the instruction encodings, the ordering of setup operations, the junk code — is variable. This separation of algorithm from implementation is the fundamental principle of polymorphism.

6. Equivalent Instruction Substitution

A key technique in polymorphic engines is replacing instructions with functionally equivalent alternatives. The x86-64 instruction set is rich with redundancies that engines can exploit:

Each substitution changes the binary encoding (different opcodes, different lengths) while preserving the computational result. When combined with register randomization and junk code insertion, the resulting machine code looks completely different between generations.

7. Why Shoggoth Chose asmjit

Historical polymorphic engines often assembled machine code manually — writing raw bytes into a buffer, computing jump offsets by hand, and managing relocations manually. This is error-prone and difficult to maintain. Shoggoth uses asmjit, a mature C++ library for runtime x86/x64 code generation, which provides:

• Assembler-level abstraction: Write a.xor_(rax, rbx) instead of computing opcode bytes manually
• Automatic encoding: asmjit handles REX prefixes, ModR/M bytes, SIB bytes, and displacement encoding
• Label management: Forward references and branch target resolution are handled automatically
• CodeHolder: A container that accumulates generated code, manages sections, and handles relocations
• JitRuntime: Allocates executable memory and copies the generated code for execution (used in testing)

This allows Shoggoth to express its polymorphic transformations at a high level while asmjit handles the low-level encoding details. We will explore asmjit in depth in Module 3.