Module 6: Encryption & Obfuscation
Hide your payload in plain sight — four encryption algorithms, polymorphic encoding, string obfuscation, and binary signing.
Why Encrypt Shellcode?
Static analysis engines scan binary files for known byte patterns, signatures, and YARA rules. Raw shellcode — especially well-known payloads like Cobalt Strike beacons or Meterpreter — contains recognizable sequences that trigger immediate detection. Encryption transforms the shellcode into random-looking ciphertext that matches no known signature. Decryption happens at runtime, in memory, just before injection. The defender must now either break the encryption or rely entirely on behavioral analysis.
Encryption in the Loader Lifecycle
Understanding when encryption and decryption occur is critical. The shellcode is never stored or transmitted in plaintext in the final binary:
Encryption Lifecycle
Read shellcode
AES/3DES/RC4/XOR
Ciphertext in binary
Decrypt in memory
Plaintext shellcode
Key Management
The encryption key is generated randomly at build time and embedded in the generated Go source code alongside the ciphertext. Both key and ciphertext are stored as byte arrays in the binary. While this means the key is technically recoverable by a skilled analyst who reverse-engineers the binary, the primary goal is to defeat automated static scanning — not to resist manual analysis. The key is never transmitted externally.
Hooka's Four Encryption Algorithms
Hooka supports four encryption methods via the --enc CLI flag. Each offers different trade-offs between speed, strength, and detection resistance.
1. AES — Advanced Encryption Standard
--enc aes
AES is the most widely recommended option and the default for serious loaders. Hooka uses AES in CBC mode with a randomly generated 256-bit key. A random Initialization Vector (IV) is prepended to the ciphertext so each encryption produces unique output even with the same key and plaintext.
- Key size: 256 bits (32 bytes)
- Block size: 128 bits (16 bytes)
- Mode: CBC (Cipher Block Chaining)
- IV: Random, prepended to ciphertext
AES produces high-entropy output that is indistinguishable from random data. Go's crypto/aes and crypto/cipher packages provide the implementation.
2. 3DES — Triple DES
--enc 3des
Triple DES applies the legacy DES algorithm three times with different keys (encrypt-decrypt-encrypt). While considered obsolete for protecting sensitive data, it is still effective for obfuscating shellcode against pattern matching. The triple-round application makes it significantly stronger than single DES.
- Key size: 168 bits (three 56-bit keys)
- Block size: 64 bits (8 bytes)
- Rounds: 3 (EDE — encrypt, decrypt, encrypt)
3DES is slower than AES but adds variety — defenders tuning detection for AES patterns may miss 3DES-encrypted payloads.
3. RC4 — Rivest Cipher 4
--enc rc4
RC4 is a stream cipher that generates a pseudorandom keystream from a variable-length key and XORs it with the plaintext byte by byte. It is extremely fast and simple to implement, with minimal code footprint in the generated loader.
- Key size: Variable (typically 128 bits)
- Type: Stream cipher (no block padding needed)
- Speed: Very fast — single-pass XOR operation
RC4 has known cryptographic weaknesses (key scheduling biases), but these are irrelevant for shellcode obfuscation where the goal is static evasion, not long-term secrecy.
4. XOR — Exclusive OR
--enc xor
The simplest encryption: each byte of shellcode is XORed with the corresponding byte of a repeating key. XOR is its own inverse, so the same operation encrypts and decrypts. This produces the smallest decryption stub in the generated code.
- Key size: Variable (typically 16–32 bytes)
- Type: Substitution cipher
- Speed: Fastest possible — single XOR per byte
XOR encryption is trivially breakable by frequency analysis or known-plaintext attacks, but it still defeats automated signature matching. Best combined with other obfuscation layers.
Encryption Comparison
| Algorithm | CLI Flag | Key Size | Speed | Strength | Detection Resistance |
|---|---|---|---|---|---|
| AES-256-CBC | --enc aes | 256 bits | Fast | Strong | High — indistinguishable from random |
| 3DES-EDE | --enc 3des | 168 bits | Slow | Moderate | High — uncommon choice adds variety |
| RC4 | --enc rc4 | Variable | Very Fast | Weak | Moderate — known patterns in keystream |
| XOR | --enc xor | Variable | Fastest | Minimal | Low — breakable with known plaintext |
Recommendation
For most scenarios, AES is the best choice. It produces the highest-entropy output, is well-supported in Go's standard library, and is the industry standard. Use XOR only when you need the smallest possible decryption stub or are layering multiple obfuscation techniques.
Shikata Ga Nai (SGN) Encoding
Shikata Ga Nai ("it can't be helped" in Japanese) is a polymorphic XOR encoder that has been used in exploit development for over a decade. Each time it processes shellcode, it produces a completely different encoded output — even with the same input.
--sgn Flag
When the --sgn flag is specified, Hooka passes the encrypted shellcode through the external sgn tool after encryption. This adds a second encoding layer on top of the encryption. The SGN encoder uses:
- Polymorphic XOR: each encoding uses different XOR keys and decoder stubs
- Instruction substitution: replaces instructions with semantically equivalent alternatives
- Dead code insertion: adds non-functional instructions (NOP sleds, redundant operations) to change the binary signature
- Register reassignment: varies which CPU registers the decoder uses
The result is that two encodings of the same shellcode produce completely different byte sequences, defeating signature-based detection even if the original payload's encrypted form becomes signatured.
Bash# Encrypt with AES then apply SGN encoding
hooka -i shellcode.bin -o loader.exe --enc aes --sgn
SGN Requires External Tool
The sgn binary must be installed separately and available in your PATH. Hooka calls it as a subprocess. Install from: github.com/EgeBalci/sgn. Without it, the --sgn flag will produce an error.
Caesar Cipher String Obfuscation
Beyond encrypting the shellcode itself, the generated Go source code contains string literals that can reveal intent: function names like "VirtualAlloc", DLL names like "ntdll.dll", and error messages. Static analysis of the compiled binary can extract these strings.
--strings Flag
The --strings flag applies a Caesar cipher shift to all string literals in the generated source code. Each character is shifted by N positions in the ASCII table. At runtime, the strings are decoded by shifting back before use.
Go// Without --strings:
dll, _ := windows.LoadDLL("ntdll.dll")
// With --strings (shift by 3):
dll, _ := windows.LoadDLL(decryptStr("qwgoo1goo")) // Each char shifted +3
// 'n'+3='q', 't'+3='w', 'd'+3='g', etc.
This prevents tools like strings, FLOSS, or YARA string rules from finding meaningful API names in the binary.
Random Variable & Function Naming
The generated Go source code uses descriptive variable names by default (e.g., shellcode, encryptedPayload, decryptionKey). While this is only visible in the source, some information leaks into the compiled binary through Go's symbol table and DWARF debug info.
-r / --rand Flag
The random naming flag replaces all variable and function names in the generated Go code with random strings before compilation. This defeats:
- Source-level pattern matching (if the source is recovered)
- Symbol table analysis of the compiled binary
- YARA rules targeting specific variable name patterns in Go binaries
Go// Without -r:
func decryptShellcode(key []byte, encrypted []byte) []byte { ... }
shellcode := decryptShellcode(encKey, encPayload)
// With -r:
func aK7mNxQ2(xR9pLw []byte, bT4vHn []byte) []byte { ... }
jM3cYs := aK7mNxQ2(wE8dFq, nU6gRk)
UPX Compression
Go binaries are notoriously large — a minimal loader can be 5–10 MB due to the Go runtime. UPX (Ultimate Packer for eXecutables) compresses the binary significantly while keeping it directly executable.
--compress Flag
When --compress is specified, Hooka runs UPX on the compiled binary after the Go build completes. Combined with Go's linker flags (-s -w to strip debug info and symbols), this typically reduces binary size by 60–70%.
| Stage | Approximate Size |
|---|---|
| Default Go build | 8–12 MB |
With -s -w linker flags | 5–7 MB |
| With UPX compression | 2–3 MB |
UPX also adds an obfuscation layer: the compressed binary has a different structure than the original, which can confuse some static analysis tools. However, UPX-packed binaries have a recognizable header that some security tools specifically flag.
Code Signing
Signed executables receive less scrutiny from some security tools. Windows SmartScreen, for example, is less likely to block a signed binary. Hooka supports two signing approaches.
PFX Certificate Signing (-c)
If you have a valid code signing certificate in PFX format, the -c flag applies it to the generated binary using osslsigncode. This produces a legitimately signed binary that passes Windows signature verification.
Bash# Sign with a real PFX certificate
hooka -i shellcode.bin -o loader.exe -c /path/to/cert.pfx
Fake Signature Domain (-d)
The -d flag generates a self-signed certificate with a specified domain name (e.g., www.microsoft.com) and signs the binary with it. The signature will not pass rigorous verification, but it adds metadata that can fool some automated tools that only check for the presence of a signature rather than its validity chain.
Bash# Sign with a fake Microsoft signature
hooka -i shellcode.bin -o loader.exe -d www.microsoft.com
Why Code Signing Matters
Some AV products treat unsigned binaries with higher suspicion. Adding any signature — even an invalid one — can reduce the initial suspicion score enough to allow execution. However, EDR products with deep verification will flag self-signed or invalid certificates. Real PFX certificates from compromised or purchased signing identities are far more effective but carry legal and ethical implications.
The maldev Companion Library
D3Ext also maintains github.com/D3Ext/maldev, a general-purpose Go library for malware development that provides additional cryptographic and encoding functions beyond what Hooka includes natively:
Additional Crypto Functions
| Category | Functions |
|---|---|
| Symmetric Encryption | ChaCha20 |
| Hashing | SHA512, MD5, SHA1, SHA256 |
| Encoding | Base32, Base64, ROT13, ROT47 |
| Password Hashing | Bcrypt |
| Asymmetric | ECC (Elliptic Curve Cryptography) |
These functions are primarily useful when building custom loaders via the Go library (Module 7) rather than through the CLI. They enable advanced encryption chains like ChaCha20 + Base64 encoding or multi-layer encryption with different algorithms.
Detection Considerations
Encryption is not a silver bullet. Defenders have adapted with techniques that can identify encrypted payloads without decrypting them.
What Defenders Look For
- Entropy analysis: Encrypted data has near-maximum Shannon entropy (~7.9 out of 8.0 for a byte). Sections of a binary with unusually high entropy are flagged as potentially encrypted payloads.
- Decryption function patterns: Even with encrypted shellcode, the decryption routine itself must be in plaintext. AES decryption functions, XOR loops, and crypto library imports are detectable patterns.
- Import table analysis: A binary importing
crypto/aesbut performing no legitimate cryptographic operations is suspicious. - Behavioral analysis: Sandboxes observe the decryption at runtime. The shellcode is encrypted on disk but plaintext in memory after decryption.
- Memory scanning: EDR products can scan process memory after decryption occurs, finding the plaintext shellcode regardless of on-disk encryption.
Layered Approach
The most effective strategy combines multiple obfuscation layers. Hooka enables this by stacking techniques:
Bash# Maximum obfuscation: AES encryption + SGN encoding + string obfuscation
# + random naming + UPX compression + fake signature
hooka -i shellcode.bin -o loader.exe \
--enc aes --sgn --strings -r --compress \
-d www.microsoft.com
Each layer addresses a different detection vector: encryption defeats byte pattern matching, SGN defeats re-signaturing, string obfuscation defeats string-based rules, random naming defeats symbol analysis, UPX changes binary structure, and signing reduces initial suspicion.
Module 6 Quiz: Encryption & Obfuscation
Q1: When is the shellcode decrypted in Hooka's loader lifecycle?
Q2: What makes Shikata Ga Nai (SGN) "polymorphic"?
Q3: Why might a defender still detect encrypted shellcode despite strong encryption?