Difficulty: Intermediate

Module 6: Encryption & Obfuscation

Hide your payload in plain sight — four encryption algorithms, polymorphic encoding, string obfuscation, and binary signing.

Why Encrypt Shellcode?

Static analysis engines scan binary files for known byte patterns, signatures, and YARA rules. Raw shellcode — especially well-known payloads like Cobalt Strike beacons or Meterpreter — contains recognizable sequences that trigger immediate detection. Encryption transforms the shellcode into random-looking ciphertext that matches no known signature. Decryption happens at runtime, in memory, just before injection. The defender must now either break the encryption or rely entirely on behavioral analysis.

Encryption in the Loader Lifecycle

Understanding when encryption and decryption occur is critical. The shellcode is never stored or transmitted in plaintext in the final binary:

Encryption Lifecycle

Build Time
Read shellcode
Encrypt
AES/3DES/RC4/XOR
Embed
Ciphertext in binary
Runtime
Decrypt in memory
Inject
Plaintext shellcode

Key Management

The encryption key is generated randomly at build time and embedded in the generated Go source code alongside the ciphertext. Both key and ciphertext are stored as byte arrays in the binary. While this means the key is technically recoverable by a skilled analyst who reverse-engineers the binary, the primary goal is to defeat automated static scanning — not to resist manual analysis. The key is never transmitted externally.

Hooka's Four Encryption Algorithms

Hooka supports four encryption methods via the --enc CLI flag. Each offers different trade-offs between speed, strength, and detection resistance.

1. AES — Advanced Encryption Standard

--enc aes

AES is the most widely recommended option and the default for serious loaders. Hooka uses AES in CBC mode with a randomly generated 256-bit key. A random Initialization Vector (IV) is prepended to the ciphertext so each encryption produces unique output even with the same key and plaintext.

AES produces high-entropy output that is indistinguishable from random data. Go's crypto/aes and crypto/cipher packages provide the implementation.

2. 3DES — Triple DES

--enc 3des

Triple DES applies the legacy DES algorithm three times with different keys (encrypt-decrypt-encrypt). While considered obsolete for protecting sensitive data, it is still effective for obfuscating shellcode against pattern matching. The triple-round application makes it significantly stronger than single DES.

3DES is slower than AES but adds variety — defenders tuning detection for AES patterns may miss 3DES-encrypted payloads.

3. RC4 — Rivest Cipher 4

--enc rc4

RC4 is a stream cipher that generates a pseudorandom keystream from a variable-length key and XORs it with the plaintext byte by byte. It is extremely fast and simple to implement, with minimal code footprint in the generated loader.

RC4 has known cryptographic weaknesses (key scheduling biases), but these are irrelevant for shellcode obfuscation where the goal is static evasion, not long-term secrecy.

4. XOR — Exclusive OR

--enc xor

The simplest encryption: each byte of shellcode is XORed with the corresponding byte of a repeating key. XOR is its own inverse, so the same operation encrypts and decrypts. This produces the smallest decryption stub in the generated code.

XOR encryption is trivially breakable by frequency analysis or known-plaintext attacks, but it still defeats automated signature matching. Best combined with other obfuscation layers.

Encryption Comparison

AlgorithmCLI FlagKey SizeSpeedStrengthDetection Resistance
AES-256-CBC--enc aes256 bitsFastStrongHigh — indistinguishable from random
3DES-EDE--enc 3des168 bitsSlowModerateHigh — uncommon choice adds variety
RC4--enc rc4VariableVery FastWeakModerate — known patterns in keystream
XOR--enc xorVariableFastestMinimalLow — breakable with known plaintext

Recommendation

For most scenarios, AES is the best choice. It produces the highest-entropy output, is well-supported in Go's standard library, and is the industry standard. Use XOR only when you need the smallest possible decryption stub or are layering multiple obfuscation techniques.

Shikata Ga Nai (SGN) Encoding

Shikata Ga Nai ("it can't be helped" in Japanese) is a polymorphic XOR encoder that has been used in exploit development for over a decade. Each time it processes shellcode, it produces a completely different encoded output — even with the same input.

--sgn Flag

When the --sgn flag is specified, Hooka passes the encrypted shellcode through the external sgn tool after encryption. This adds a second encoding layer on top of the encryption. The SGN encoder uses:

The result is that two encodings of the same shellcode produce completely different byte sequences, defeating signature-based detection even if the original payload's encrypted form becomes signatured.

Bash# Encrypt with AES then apply SGN encoding
hooka -i shellcode.bin -o loader.exe --enc aes --sgn

SGN Requires External Tool

The sgn binary must be installed separately and available in your PATH. Hooka calls it as a subprocess. Install from: github.com/EgeBalci/sgn. Without it, the --sgn flag will produce an error.

Caesar Cipher String Obfuscation

Beyond encrypting the shellcode itself, the generated Go source code contains string literals that can reveal intent: function names like "VirtualAlloc", DLL names like "ntdll.dll", and error messages. Static analysis of the compiled binary can extract these strings.

--strings Flag

The --strings flag applies a Caesar cipher shift to all string literals in the generated source code. Each character is shifted by N positions in the ASCII table. At runtime, the strings are decoded by shifting back before use.

Go// Without --strings:
dll, _ := windows.LoadDLL("ntdll.dll")

// With --strings (shift by 3):
dll, _ := windows.LoadDLL(decryptStr("qwgoo1goo"))  // Each char shifted +3
// 'n'+3='q', 't'+3='w', 'd'+3='g', etc.

This prevents tools like strings, FLOSS, or YARA string rules from finding meaningful API names in the binary.

Random Variable & Function Naming

The generated Go source code uses descriptive variable names by default (e.g., shellcode, encryptedPayload, decryptionKey). While this is only visible in the source, some information leaks into the compiled binary through Go's symbol table and DWARF debug info.

-r / --rand Flag

The random naming flag replaces all variable and function names in the generated Go code with random strings before compilation. This defeats:

Go// Without -r:
func decryptShellcode(key []byte, encrypted []byte) []byte { ... }
shellcode := decryptShellcode(encKey, encPayload)

// With -r:
func aK7mNxQ2(xR9pLw []byte, bT4vHn []byte) []byte { ... }
jM3cYs := aK7mNxQ2(wE8dFq, nU6gRk)

UPX Compression

Go binaries are notoriously large — a minimal loader can be 5–10 MB due to the Go runtime. UPX (Ultimate Packer for eXecutables) compresses the binary significantly while keeping it directly executable.

--compress Flag

When --compress is specified, Hooka runs UPX on the compiled binary after the Go build completes. Combined with Go's linker flags (-s -w to strip debug info and symbols), this typically reduces binary size by 60–70%.

StageApproximate Size
Default Go build8–12 MB
With -s -w linker flags5–7 MB
With UPX compression2–3 MB

UPX also adds an obfuscation layer: the compressed binary has a different structure than the original, which can confuse some static analysis tools. However, UPX-packed binaries have a recognizable header that some security tools specifically flag.

Code Signing

Signed executables receive less scrutiny from some security tools. Windows SmartScreen, for example, is less likely to block a signed binary. Hooka supports two signing approaches.

PFX Certificate Signing (-c)

If you have a valid code signing certificate in PFX format, the -c flag applies it to the generated binary using osslsigncode. This produces a legitimately signed binary that passes Windows signature verification.

Bash# Sign with a real PFX certificate
hooka -i shellcode.bin -o loader.exe -c /path/to/cert.pfx

Fake Signature Domain (-d)

The -d flag generates a self-signed certificate with a specified domain name (e.g., www.microsoft.com) and signs the binary with it. The signature will not pass rigorous verification, but it adds metadata that can fool some automated tools that only check for the presence of a signature rather than its validity chain.

Bash# Sign with a fake Microsoft signature
hooka -i shellcode.bin -o loader.exe -d www.microsoft.com

Why Code Signing Matters

Some AV products treat unsigned binaries with higher suspicion. Adding any signature — even an invalid one — can reduce the initial suspicion score enough to allow execution. However, EDR products with deep verification will flag self-signed or invalid certificates. Real PFX certificates from compromised or purchased signing identities are far more effective but carry legal and ethical implications.

The maldev Companion Library

D3Ext also maintains github.com/D3Ext/maldev, a general-purpose Go library for malware development that provides additional cryptographic and encoding functions beyond what Hooka includes natively:

Additional Crypto Functions

CategoryFunctions
Symmetric EncryptionChaCha20
HashingSHA512, MD5, SHA1, SHA256
EncodingBase32, Base64, ROT13, ROT47
Password HashingBcrypt
AsymmetricECC (Elliptic Curve Cryptography)

These functions are primarily useful when building custom loaders via the Go library (Module 7) rather than through the CLI. They enable advanced encryption chains like ChaCha20 + Base64 encoding or multi-layer encryption with different algorithms.

Detection Considerations

Encryption is not a silver bullet. Defenders have adapted with techniques that can identify encrypted payloads without decrypting them.

What Defenders Look For

Layered Approach

The most effective strategy combines multiple obfuscation layers. Hooka enables this by stacking techniques:

Bash# Maximum obfuscation: AES encryption + SGN encoding + string obfuscation
# + random naming + UPX compression + fake signature
hooka -i shellcode.bin -o loader.exe \
  --enc aes --sgn --strings -r --compress \
  -d www.microsoft.com

Each layer addresses a different detection vector: encryption defeats byte pattern matching, SGN defeats re-signaturing, string obfuscation defeats string-based rules, random naming defeats symbol analysis, UPX changes binary structure, and signing reduces initial suspicion.

Module 6 Quiz: Encryption & Obfuscation

Q1: When is the shellcode decrypted in Hooka's loader lifecycle?

The shellcode is encrypted at build time and embedded as ciphertext. At runtime, the generated loader decrypts it in memory immediately before injection. The plaintext shellcode only exists in process memory, never on disk.

Q2: What makes Shikata Ga Nai (SGN) "polymorphic"?

Polymorphic means "many forms." SGN uses randomized XOR keys, varied decoder stubs, instruction substitution, dead code insertion, and register reassignment so that each encoding of the same shellcode produces a unique byte sequence. This defeats signature-based detection even after a specific encoded form is signatured.

Q3: Why might a defender still detect encrypted shellcode despite strong encryption?

Defenders don't need to break the encryption. Entropy analysis identifies sections of the binary with near-maximum randomness (characteristic of encrypted data). Additionally, sandbox behavioral analysis and EDR memory scanning can observe or capture the plaintext shellcode after it is decrypted at runtime in process memory.