5152d0c9f7
Foundation for replacing fold's dead/abstract binary encoder with a real
SSA P-Code frontend. Verified root-cause gap: serializeBinaryIR() emits only
data-discovery nodes (sections/strings/imports/exports); the control-flow
BinaryExprKind values (BasicBlock/CallSite/...) are never constructed, so the
Z3-BB encoder's encodable filter is always empty and encodeBinaryBBs never
runs — hence 0 binary Z3 findings. The P-Code pipeline supersedes that path.
Phase 0 — docs/internal/ghidra-pcode-integration-research.md: P-Code opcode
core, the decompiler C++ embedding API (LoadImage/SleighArchitecture/Funcdata),
the CORE+DECCORE+SLEIGH source sets, no-bison finding, and pypcode as build
reference.
Phase 1 — vendored Ghidra decompiler at pinned commit (Ghidra_12.1_build,
7e89d94e34...) under third_party/ghidra-decompiler/ (cpp + AArch64 specs +
Apache-2.0 LICENSE + VENDOR.md + .gitignore for build artifacts). Validated by
hand on this clang-21/macOS toolchain:
- full SSA engine (CORE+DECCORE+SLEIGH, 80 files incl. heritage/funcdata/
action) compiles clean, -std=c++11 -O2, NO BFD, NO bison (generated
parsers are committed upstream; ruleparse isn't in the lib set);
- sleigh compiler builds; AArch64 .slaspec -> .sla compiles;
- tools/pcode_dump.cpp (custom LoadImage, no BFD) lifts the real arm64
sym._div_by_zero_direct to correct P-Code: `mov w9,#0 -> COPY(const 0)`,
`sdiv -> INT_SDIV(w8,w9)` — divisor zero-ness explicit in the data flow,
so the Phase-3 encoder proves div-by-zero trivially, multi-arch.
Not yet meson-wired (next step) and no fold-side IR/encoder yet (Phase 2-3);
this commit is the validated vendor + smoke tool.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
142 lines
9.4 KiB
Markdown
142 lines
9.4 KiB
Markdown
# Ghidra Decompiler → SSA P-Code Integration Research (Phase 0)
|
|
|
|
> Generated 2026-05-31. Phase 0 of the "vendor Ghidra decompiler" plan. Docs-first:
|
|
> understand the engine's API, source layout, and build before writing any glue. Sources are
|
|
> the Ghidra repo (NationalSecurityAgency/ghidra) and angr/pypcode (a lift-only vendor we
|
|
> mirror for the build approach).
|
|
|
|
## Decision recap
|
|
|
|
- **Vendor** Ghidra's **full decompiler** C++ engine (Apache-2.0) at a **pinned commit** —
|
|
we control the version, so no ABI skew (the failure mode of the r2ghidra plugin). Consume
|
|
its **SSA high P-Code** (Funcdata). Ghidra does data-flow/SSA/variable recovery; fold does
|
|
Z3 path-reasoning + findings via the existing `SafetyObserver::*Ir` hooks.
|
|
- **Pin:** tag `Ghidra_12.1_build` (Ghidra 12.1, released 2026-05-13). Record the exact
|
|
40-char SHA in `third_party/ghidra-decompiler/VENDOR.md` at clone time.
|
|
- **License:** Apache-2.0 — vendorable; carry upstream `LICENSE` + a `NOTICE` listing the
|
|
vendored tree, commit SHA, and modifications (build glue only).
|
|
|
|
## 1. P-Code semantics (what we consume)
|
|
|
|
P-Code is a register-transfer IL; **every architecture lowers to the same op set**, so the
|
|
encoder is written once. The opcode enum is `OpCode` in `opcodes.hh` (`CPUI_*`). Two levels:
|
|
|
|
- **Raw P-Code** — straight from SLEIGH lifting; per-instruction, no data-flow. (This is all
|
|
pypcode exposes.)
|
|
- **High P-Code (SSA)** — after the decompiler's analysis actions run on a `Funcdata`:
|
|
SSA form (each Varnode written once), dead code removed, variables/types recovered,
|
|
def-use explicit. **This is what we want** — it adds `CPUI_MULTIEQUAL` (SSA phi),
|
|
`CPUI_INDIRECT`, `CPUI_PTRADD`/`CPUI_PTRSUB`, `CPUI_CAST` over the raw set.
|
|
|
|
**Varnode** = (address space, offset, size). Spaces we care about: `const` (immediates →
|
|
Z3 `bv_val`), `register`, `unique` (temporaries), `ram` (memory), plus the stack space.
|
|
A Varnode in `const` space with offset 0 is the literal zero our current const-hazard pass
|
|
hunts for — but here it falls out of SSA naturally.
|
|
|
|
**Core ops fold maps to `SafetyObserver` hooks (the "~25-op core"):**
|
|
|
|
| P-Code op | fold hook | Finding |
|
|
|-----------|-----------|---------|
|
|
| `CPUI_INT_SDIV` / `INT_UDIV` / `INT_SREM` / `INT_REM` | `onDivision` / `onBinaryArith(Div)` | div-by-zero (CWE-369) when divisor SAT==0 under path |
|
|
| `CPUI_LOAD` / `CPUI_STORE` | `onPointerDerefIr` | null-deref (CWE-476) when addr Varnode SAT==0 |
|
|
| `CPUI_INT_ADD` / `INT_SUB` / `INT_MULT` | `onBinaryArith` | int-overflow (CWE-190); reuse source guard heuristics to avoid every-op FP |
|
|
| `CPUI_CALL` / `CPUI_CALLIND` | `onCallExprIr` / `onLibraryCall` | free/malloc/strcpy models → UAF/buffer via existing observers |
|
|
| `CPUI_CBRANCH` | (path predicate) | real path-sensitivity → covers `div_by_zero_variable` / `null_deref_conditional` |
|
|
| `CPUI_COPY` / `INT_ZEXT` / `INT_SEXT` / `SUBPIECE` / `PIECE` | value propagation in encoder | — |
|
|
| `CPUI_MULTIEQUAL` (phi) | Z3 ite/merge at block joins | SSA join |
|
|
| `CPUI_INT_EQUAL` / `INT_SLESS` / `INT_LESS` … | branch conditions | path constraints |
|
|
|
|
Reference: `Ghidra/Features/Decompiler/src/decompile/cpp/opcodes.hh` and the rendered P-Code
|
|
reference `Ghidra/Features/Decompiler/src/main/doc/pcoderef.xml`.
|
|
|
|
## 2. Decompiler C++ embedding API (the glue surface)
|
|
|
|
Confine all of this to `src/binary/ghidra_lifter.cpp` (Ghidra types never leak into
|
|
`fold::ir`, per the M30 `*_clang.cpp` convention).
|
|
|
|
- **`LoadImage`** (`loadimage.hh`) — abstract byte source. Subclass it to serve a function's
|
|
bytes from r2 (r2 gives us `arch/bits/endian`, the function vaddr+size, and raw bytes via
|
|
`p8`/`prj`). Implement `loadFill(uint1*, int4, const Address&)` + `getArchType`/`adjustVma`.
|
|
- **`SleighArchitecture` / `Architecture`** (`sleigh_arch.hh`, `architecture.hh`) — owns the
|
|
translator, spec databases, and the action pipeline. Built from a spec id; specs are loaded
|
|
via **`DocumentStorage`** (`xml.hh`) reading the `.ldefs`/`.pspec`/`.cspec` and the compiled
|
|
`.sla`. Call `arch->init(store)` to bring up sleigh + types + the default action set.
|
|
- **Decompile a function:** obtain a `Funcdata*` for the entry address from the architecture's
|
|
scope, then run the analysis pipeline:
|
|
`arch->allacts.getCurrent()->reset(*fd); arch->allacts.getCurrent()->perform(*fd);`
|
|
After `perform`, `fd` holds **high (SSA) P-Code**. Iterate ops via `fd->beginOpAll()` /
|
|
`fd->endOpAll()` (each `PcodeOp` has `getOpcode()`, `getOut()`, `getIn(i)` Varnodes) and
|
|
basic blocks via `fd->getBasicBlocks()`. The canonical example is the interactive console
|
|
`consolemain.cc` (the `decomp_opt` binary) and the doxygen "Building a Project" page.
|
|
- **Thread-safety:** the decompiler uses process/global state (it was designed to run as a
|
|
one-function-at-a-time child process, `ghidra_process`). Treat an `Architecture` as
|
|
**not thread-safe** → one instance per fold worker thread, or serialize lifting.
|
|
|
|
## 3. Source layout + what to vendor
|
|
|
|
From `Ghidra/Features/Decompiler/src/decompile/cpp/Makefile` (authoritative file lists):
|
|
|
|
- **`CORE`** (lifting/serialization base): `xml marshal space float address pcoderaw translate
|
|
opcodes globalcontext`
|
|
- **`SLEIGH`** (the lifter): `sleigh pcodeparse pcodecompile sleighbase slghsymbol
|
|
slghpatexpress slghpattern semantics context slaformat compression filemanage`
|
|
- **`DECCORE`** (the full decompiler / SSA engine — this is the delta over pypcode's lift-only
|
|
vendor): `capability architecture options graph cover block cast typeop database cpool
|
|
comment stringmanage modelrules fspec action loadimage grammar varnode op type variable
|
|
varmap jumptable emulate emulateutil flow userop expression multiprecision funcdata
|
|
funcdata_block funcdata_op funcdata_varnode unionresolve pcodeinject heritage prefersplit
|
|
rangeutil ruleaction subflow blockaction merge double transform constseq bitfield coreaction
|
|
condexe override dynamic crc32 prettyprint printlanguage printc printjava memstate opbehavior
|
|
paramid signature`
|
|
- SSA-relevant: **`heritage`** (SSA construction), **`funcdata*`** (the SSA container),
|
|
`action`/`ruleaction`/`coreaction` (the simplification rules), `merge`, `flow`,
|
|
`blockaction`, `jumptable`.
|
|
- **Targets:** vendor builds **`libdecomp.a`** (= CORE + DECCORE + SLEIGH) and links it into
|
|
fold. (`libsla.a` = CORE + SLEIGH only is the lift-only library; not enough for SSA.)
|
|
`sleigh` (the spec compiler) is also built to produce `.sla`.
|
|
- **bison/flex generated files:** `grammar.cc`←`grammar.y`, `xml.cc`←`xml.y`,
|
|
`pcodeparse.cc`←`pcodeparse.y`, `slghparse.cc`/`.hh`←`slghparse.y`, `slghscan.cc`←`slghscan.l`,
|
|
`ruleparse.cc`/`.hh`←`ruleparse.y`.
|
|
→ **Pre-generate these at vendor time and check the `.cc`/`.hh` in**, so fold's build needs
|
|
no bison/flex (matches the "trade runtime-debugging for up-front work" goal).
|
|
- **Processor specs:** `Ghidra/Processors/<arch>/data/languages/*.{slaspec,ldefs,pspec,cspec,sinc}`.
|
|
Compile `.slaspec`→`.sla` once with the vendored `sleigh` binary and **check the `.sla` +
|
|
`.ldefs`/`.pspec`/`.cspec` in** under `third_party/ghidra-decompiler/specs/<arch>/`. Vendor
|
|
only the arches we target (below) to keep the tree small.
|
|
|
|
## 4. Target architectures (match corpus + bug-bounty reality)
|
|
|
|
`x86` (32) + `x86-64`, `AARCH64` (corpus is arm64 Mach-O), `ARM` (LE/BE), `MIPS` (LE/BE).
|
|
These cover the CTF corpus and the common closed-source targets. More arches are just more
|
|
checked-in `.sla` files later — no code change.
|
|
|
|
## 5. Build approach (meson)
|
|
|
|
- `third_party/ghidra-decompiler/` holds the pinned `cpp/` sources (with pre-generated
|
|
parser/lexer `.cc`/`.hh`) + `specs/<arch>/*.sla` + `LICENSE` + `NOTICE` + `VENDOR.md`
|
|
(commit SHA + the exact upstream paths + the pre-generation/compile commands used).
|
|
- meson: a static-lib target `libghidra_decompiler.a` over the CORE+DECCORE+SLEIGH `.cc` set
|
|
(a `files()` list mirroring the Makefile vars); C++17, `-DLOCAL_ZLIB`/bundled zlib as
|
|
pypcode does, define `-DFOLD_HAVE_GHIDRA=1`. Gate behind a `-Dghidra=auto|enabled|disabled`
|
|
option mirroring the existing `radare2` option (`meson.build:35`), so builds without it fall
|
|
back to the shipped r2-disasm passes.
|
|
- Smoke: `tools/pcode_dump.cpp` links the lib, lifts one function from a corpus dylib, prints
|
|
SSA P-Code ops — validates vendor + spec loading before any fold integration.
|
|
|
|
## 6. Reference: pypcode (mirror its build mechanics, not its scope)
|
|
|
|
`angr/pypcode` `CMakeLists.txt` compiles `SLEIGH_COMMON` (= our CORE+SLEIGH, lift-only:
|
|
`address compression context float globalcontext marshal opcodes pcodecompile pcodeparse
|
|
pcoderaw semantics slaformat sleigh sleighbase slghpatexpress slghpattern slghsymbol space
|
|
translate xml`), builds the `sleigh` compiler from `+ filemanage slgh_compile slghparse
|
|
slghscan`, and runs `sleigh -a <processors-dir>` at build time to compile `.sla`. We extend
|
|
the file set with `DECCORE` (for SSA) and **pre-compile `.sla` at vendor time** instead of at
|
|
fold build time. pypcode is Apache-2.0/BSD and a good build reference.
|
|
|
|
## 7. Open items resolved in later phases
|
|
|
|
- Exact pinned SHA for `Ghidra_12.1_build` → recorded in `VENDOR.md` at clone (Phase 1).
|
|
- Whether to also expose decompiled **C** (printc) for a future decompile→Clang→source-detector
|
|
path → out of scope now; `printc` is in DECCORE so it's available for free later.
|
|
- Per-function decompile cost vs the `binary_backend` test budget → measured in Phase 1 with
|
|
`pcode_dump`; cap/deadline designed in Phase 4.
|