Module 4: Function Registration & X86RetModPass

The LLVM MachineFunctionPass that transforms ordinary functions into self-masking ones.

1. Pass Registration in the X86 Backend

The X86RetModPass is registered as a PreEmit pass in the X86 target machine configuration. In LLVM, each target defines its pass pipeline through a TargetPassConfig subclass:

C++// X86TargetMachine.cpp - Pass pipeline configuration
void X86PassConfig::addPreEmitPass() {
  // Standard pre-emit passes run first
  addPass(new X86IndirectBranchTrackingPass());  // CET IBT support

  // FunctionPeekaboo's pass runs last in PreEmit
  // This ensures all other transformations are complete
  addPass(new X86RetModPass());
}

Running last in PreEmit is critical: it means all other passes (register allocation, frame layout, branch optimization) have finished. The function’s machine code is in its final form, and the pass can calculate exact byte sizes and offsets.

2. Function Identification

The pass’s runOnMachineFunction() method is called for every function in the compilation unit. The first thing it does is check whether this function should be instrumented:

C++bool X86RetModPass::runOnMachineFunction(MachineFunction &MF) {
  // Get the underlying LLVM IR function
  const Function &F = MF.getFunction();

  // Check for the "peekaboo" annotation attribute
  if (!F.hasFnAttribute("peekaboo"))
    return false;  // Skip this function - no changes

  // Also skip very small functions (< minimum stub size)
  // The prologue stub is 0x46 bytes; function must be larger
  if (estimateFunctionSize(MF) < MIN_FUNCTION_SIZE)
    return false;

  // This function should be instrumented
  instrumentFunction(MF);
  return true;  // We modified the function
}

3. Function Size Calculation

Before injecting stubs, the pass needs to know the exact byte size of the function’s body. At the PreEmit stage, instructions are MachineInstr objects, not bytes, so the pass estimates the encoded size:

C++unsigned X86RetModPass::estimateFunctionSize(MachineFunction &MF) {
  unsigned Size = 0;
  for (MachineBasicBlock &MBB : MF) {
    for (MachineInstr &MI : MBB) {
      // Each instruction has a known encoding size
      // The MCCodeEmitter can compute exact sizes
      Size += getInstructionSize(MI);
    }
  }
  return Size;
}

The function size is recorded in the .funcmeta entry so the handler knows exactly how many bytes to XOR at runtime. Accuracy is essential — encrypting too few bytes leaves code exposed, while encrypting too many corrupts adjacent code.

4. The Instrumentation Process

Once a function is identified for instrumentation, X86RetModPass performs three transformations:

C++void X86RetModPass::instrumentFunction(MachineFunction &MF) {
  // Step 1: Generate a random XOR key for this function
  uint8_t xorKey = generateRandomKey();

  // Step 2: Calculate the function body size (excluding stubs)
  unsigned bodySize = estimateFunctionSize(MF);

  // Step 3: Inject prologue stub at function entry
  MachineBasicBlock &EntryBB = MF.front();
  injectPrologue(EntryBB, xorKey);

  // Step 4: Find and replace all RET instructions
  for (MachineBasicBlock &MBB : MF) {
    for (auto MI = MBB.begin(); MI != MBB.end(); ) {
      if (MI->isReturn()) {
        MI = replaceReturn(MBB, MI);  // Replace RET with epilogue stub
      } else {
        ++MI;
      }
    }
  }

  // Step 5: Emit .funcmeta entry
  emitFuncMetaEntry(MF, bodySize, xorKey);
}

5. Handling Multiple Return Points

C/C++ functions can have multiple return paths (early returns, conditional returns, switch/case exits). Every single RET instruction in the function must be replaced with an epilogue stub. Missing even one return point means the function could return without re-encrypting itself, leaving it in cleartext.

C++// Example: function with multiple returns
int process_command(int cmd) {
    if (cmd == 0) return -1;      // Early return (RET #1)

    if (cmd == 1) {
        do_thing_a();
        return 0;                  // Normal return (RET #2)
    }

    if (cmd == 2) {
        do_thing_b();
        return 1;                  // Another return (RET #3)
    }

    return -2;                     // Default return (RET #4)
}

// X86RetModPass replaces ALL FOUR RET instructions with epilogue stubs
// The compiler guarantees it sees every return point at the MachineInstr level

6. .funcmeta Entry Emission

For each instrumented function, the pass emits a structured entry into the .funcmeta section. This is done through LLVM’s MC (Machine Code) layer, which handles section management and object file emission:

C++void X86RetModPass::emitFuncMetaEntry(
    MachineFunction &MF, unsigned bodySize, uint8_t xorKey) {

  // Get or create the .funcmeta section
  MCContext &Ctx = MF.getContext();
  MCSection *MetaSection = Ctx.getELFSection(
      ".funcmeta", ELF::SHT_PROGBITS,
      ELF::SHF_ALLOC | ELF::SHF_WRITE);  // RW, no execute

  // Switch to the .funcmeta section
  MCStreamer &Streamer = ...;
  Streamer.switchSection(MetaSection);

  // Emit the entry fields
  MCSymbol *FuncSym = MF.getJTISymbol(0, Ctx);
  Streamer.emitSymbolValue(FuncSym, 4);   // Function RVA (4 bytes)
  Streamer.emitIntValue(bodySize, 4);      // Body size (4 bytes)
  Streamer.emitIntValue(xorKey, 1);        // XOR key (1 byte)
  Streamer.emitIntValue(0, 1);             // IsEncrypted = 0 initially
  Streamer.emitIntValue(0, 2);             // Padding (2 bytes)
}

7. XOR Key Generation

Each function gets its own unique single-byte XOR key. A single byte (256 possible values, excluding 0) is used because:

8. Register Preservation

The injected prologue and epilogue stubs must not corrupt any registers that the original function uses. At the PreEmit stage, register allocation is complete, so the pass knows exactly which registers are live:

C++void X86RetModPass::injectPrologue(MachineBasicBlock &EntryBB, uint8_t key) {
  // The prologue stub must:
  // 1. Save ALL registers it uses (push to stack)
  // 2. Save flags (pushfq)
  // 3. Call the handler
  // 4. Restore flags (popfq)
  // 5. Restore ALL registers (pop from stack)
  // 6. Fall through to the (now decrypted) function body

  // The stub uses a CALL/POP trick for PIC (position-independent)
  // addressing, which means it uses one register for the return
  // address. All other registers are preserved via push/pop.
}

9. Interaction with Optimizations

Because X86RetModPass runs after all optimization passes, it does not interfere with any standard compiler optimizations:

10. Pass Output Summary

After X86RetModPass processes a function, the output consists of:

Text+----------------------------------+
| Prologue Stub (0x46 bytes)       |  ← Injected: saves regs, calls handler to decrypt
+----------------------------------+
| Original Function Body           |  ← Unchanged machine code (will be XOR'd at runtime)
|   ... instructions ...           |
|   ... multiple basic blocks ...  |
+----------------------------------+
| Epilogue Stub (at each RET)      |  ← Injected: calls handler to re-encrypt, then RET
+----------------------------------+

.funcmeta entry:
  FunctionRVA  = RVA of function body start (after prologue)
  FunctionSize = byte size of original body (excluding stubs)
  XorKey       = random byte (1-255)
  IsEncrypted  = 0 (set to 1 by .stub initialization)