← back to writeups
rev 2026-03-28 · 15 min read

Unpacking a custom packer by hand with Ghidra + x64dbg

The binary decrypts itself in memory at launch. Raw static analysis gives nothing—you have to let it unpack, then capture the result.

First reflex: open the binary in Ghidra and you see… almost nothing. The entry point jumps into a small loop that processes a large section, then jumps into the void. A classic sign of a packer: the real code is encrypted on disk and reconstructed at runtime.

Spot the static clues

Two signals confirm the hypothesis:

  • The entropy of the .text section is close to 8 bits/byte—typical of encrypted or compressed data.
  • An unusual section marked RWX (read + write + execute). Nobody writes code there by accident: it's the unpacking buffer.
$ python3 -c "import math,sys; \
d=open('packed.bin','rb').read()[0x1000:0x9000]; \
from collections import Counter; \
c=Counter(d); n=len(d); \
print(round(-sum((v/n)*math.log2(v/n) for v in c.values()),2))"
7.98   # near maximal -> encrypted

The decryption routine

In Ghidra, the loop at the entry point applies an unrolled XOR with a derived key, then a jmp to an address computed on the fly. That target is the Original Entry Point (OEP)—where the real program begins once unpacked.

// strategy We don't try to reimplement the decryption (slow and fragile). We let the binary do the work, set a breakpoint just before the jump to the OEP, and capture memory at that exact moment.

Break at the right moment (x64dbg)

The final jmp rax is our landmark. We break on it. When RAX points outside the unpacking loop, into a region that looks like a function prologue (push rbp ; mov rbp, rsp), we're there.

; the stub's last jump
00401337  ff e0            jmp  rax        ; <- breakpoint here
; RAX = 0000000000403A20  -> that's the OEP

Dump and reconstruct

With the Scylla plugin (or Scylla-x64dbg), we dump the process from the OEP and repair the import table (IAT), which the packer often wipes to complicate analysis.

  1. Dump memory from the current OEP.
  2. IAT Autosearch then Get Imports—Scylla reconstructs the dynamically resolved API calls.
  3. Fix Dump to write out a statically analyzable PE.
! anti-analysis This sample checked IsDebuggerPresent and the PEB's BeingDebugged flag. We patch the return to 0 once and for all, otherwise unpacking branches into a decoy path that prints a fake flag. Always be suspicious of a flag that's "too easy".

Analyzing the unpacked binary

Reloaded into Ghidra, the repaired dump finally reveals the business logic: a license check comparing a machine fingerprint against a hard-coded value. The flag was the string passed to the comparison function.

flag captured flag{unp4ck_th3_l4y3rs_0n3_by_0n3}

Takeaways

  • A packer doesn't "hide" code—it delays it. At some instant, the real code must exist in the clear in memory.
  • The static (Ghidra) + dynamic (x64dbg) combination almost always beats either one alone.
  • Anti-debug checks are common: identify them early and neutralize them before wasting an hour on a false lead.