277 lines
14 KiB
Markdown
277 lines
14 KiB
Markdown
# Fold — Component & CLI Architecture Map
|
||
|
||
A reference map of fold's layered components, analysis backends, CLI surface, and
|
||
detector taxonomy, plus an audit of the profiling and testing infrastructure.
|
||
|
||
Fold is a **bounded-symbolic SAST + binary-analysis** tool. It analyzes C/C++ (clang),
|
||
Rust and Swift (external frontends), and compiled binaries/firmware (radare2 + Ghidra
|
||
P-Code), encodes program semantics into Z3, and emits findings classified by how strongly
|
||
they were established (a *confidence* model, not a severity scale).
|
||
|
||
---
|
||
|
||
## 1. Layered components
|
||
|
||
Fold enforces a strict no-upward-includes layering (see `CLAUDE.md`). Lower layers never
|
||
include higher ones.
|
||
|
||
```mermaid
|
||
flowchart TD
|
||
ir["fold::ir<br/>frontend-agnostic vocabulary<br/>Expr · Symbol · SourceLoc · Call · CFG · summaries"]
|
||
report["fold::report<br/>Finding · Confidence · triage · SARIF/MD/JSON"]
|
||
symbolic["fold::symbolic<br/>encoder family (Z3) · SymbolicEnv · SafetyObserver<br/>heap/interval/octagon · stdlib models · loop invariants"]
|
||
detect["fold::detect<br/>detectors subclassing the observer surface<br/>+ security_walker composite engine"]
|
||
cli["fold::cli<br/>scan router/runner · detector plan · presets · worker fanout"]
|
||
|
||
ir --> report --> symbolic --> detect --> cli
|
||
|
||
subgraph backends [side modules]
|
||
binary["fold::binary<br/>r2_backend · ghidra_lifter · pcode_router<br/>rop_gadgets · gadget_semantics · rop_chain"]
|
||
frontend["fold::frontend<br/>clang_backend · rust_backend · swift_backend<br/>ast_visitor · cfg_builder · comp_db"]
|
||
confirm["fold::confirm<br/>FindingConfirmationTask (deeper bounded re-proof)"]
|
||
fuzz["fold::fuzz<br/>harness_emitter · corpus · asan_parser · runner"]
|
||
mcp["fold::mcp<br/>JSON-RPC stdio server · ToolRegistry · ScanCache"]
|
||
lsp["fold::lsp<br/>Content-Length framed JSON-RPC diagnostics"]
|
||
end
|
||
|
||
frontend --> symbolic
|
||
binary --> symbolic
|
||
detect --> confirm
|
||
detect --> fuzz
|
||
cli --> mcp
|
||
cli --> lsp
|
||
```
|
||
|
||
All of `src/binary`, `src/frontend`, `src/ir`, `src/symbolic`, `src/detect`, `src/index`,
|
||
`src/report`, `src/confirm`, `src/witness`, `src/fuzz`, `src/mcp`, `src/lsp` compile into
|
||
the **`fold_core`** static library (top-level `meson.build:217-228`); the `fold`
|
||
executable (`src/cli`, `meson.build:292`) links it. The optional `fold_ghidra` lib is
|
||
gated behind `ghidra_enabled`.
|
||
|
||
**Frontend-agnostic invariant (M30):** `fold::ir` headers name no `clang::*` types. Clang
|
||
bridges live in `*_clang.cpp` (`src/symbolic/ir_expr_clang.cpp`,
|
||
`clang_location_mapper.cpp`). A Rust/Swift frontend builds `ir::Expr` directly. Detectors
|
||
and observer hooks must not break this.
|
||
|
||
---
|
||
|
||
## 2. Analysis flow
|
||
|
||
```mermaid
|
||
flowchart LR
|
||
subgraph inputs [Inputs]
|
||
cdb[compile_commands.json]
|
||
bin[binary / firmware]
|
||
rs[".rs (HIR JSON)"]
|
||
sil["Swift SIL (SWAN)"]
|
||
end
|
||
|
||
cdb --> clang[clang_backend]
|
||
rs --> rustfe[rust_backend → fold-rust]
|
||
sil --> swiftfe[swift_backend → fold-swift]
|
||
bin --> r2[r2_backend + ghidra_lifter]
|
||
|
||
clang --> irc[fold::ir]
|
||
rustfe --> irc
|
||
swiftfe --> irc
|
||
r2 --> pcode[P-Code]
|
||
|
||
irc --> enc[encoder.cpp — Z3 encoding]
|
||
pcode --> penc[pcode_encoder.cpp — Z3]
|
||
|
||
enc --> det[detectors via SafetyObserver +<br/>security_walker engine modules]
|
||
penc --> det
|
||
|
||
det --> triage[report::triage<br/>score · suppress · baseline]
|
||
triage --> out{--format}
|
||
out --> sarif[SARIF 2.1.0]
|
||
out --> native[Versioned findings JSON]
|
||
out --> md[Markdown]
|
||
out -. deprecated json alias .-> sarif
|
||
|
||
det -. confirm/fuzz .-> promote[Proved → ConcreteWitness]
|
||
promote --> triage
|
||
```
|
||
|
||
**Mode resolution** (`src/cli/scan_router.cpp` `resolveMode`): `auto` infers
|
||
source/binary/hybrid/rust/swift from inputs; explicit `--mode` overrides. **Hybrid** runs
|
||
source + binary lanes and merges/corroborates. Source mode fans out across `fold worker`
|
||
subprocesses when `--processes > 1` (`src/cli/subprocess_fanout.cpp`).
|
||
|
||
---
|
||
|
||
## 3. CLI command tree
|
||
|
||
Entry: `src/cli/main.cpp` (CLI11). Exactly one subcommand required. Global `--quiet/-q`.
|
||
Exit codes `0` (no actionable findings) / `1` (findings) / `2` (error).
|
||
|
||
```mermaid
|
||
flowchart TD
|
||
fold[fold]
|
||
fold --> scan
|
||
fold --> grep
|
||
fold --> confirm
|
||
fold --> fuzz
|
||
fold --> mcp
|
||
fold --> lsp
|
||
fold --> skill
|
||
fold -. hidden .-> worker
|
||
|
||
fuzz --> fprep[prep]
|
||
fuzz --> frun[run]
|
||
|
||
scan --> sg["flag groups:<br/>Input · Output · Detectors ·<br/>Analysis · Hybrid ·<br/>Performance/Misc · Suppression"]
|
||
```
|
||
|
||
| Subcommand | Purpose |
|
||
|---|---|
|
||
| `scan` | Source/binary/hybrid/Rust/Swift analysis → SARIF/findings-json/Markdown |
|
||
| `grep` | Query existing Fold SARIF/findings-json reports with shared typed filters and pagination |
|
||
| `confirm` | Re-run a finding with deeper bounded symbolic analysis; optional GraphML witnesses |
|
||
| `fuzz prep` | Emit libFuzzer harnesses + Z3-derived seed corpora |
|
||
| `fuzz run` | Compile + run harnesses; ingest crashes as `ConcreteWitness` findings |
|
||
| `mcp` | JSON-RPC 2.0 stdio MCP server (canonical tools include `fold.scan*`, `fold.query_findings`, and `fold.explain`) |
|
||
| `lsp` | LSP diagnostics server |
|
||
| `skill` | Print the embedded `SKILL.md` for agents (`-o` to file) |
|
||
| `worker` | **Internal** — process a shard (stdin file list → NDJSON), for subprocess fanout |
|
||
|
||
**Key `scan` flags** (selection): `--mode`, `--compile-db`, `--binary`, paths
|
||
(Input); `--output/-o`, `--format/-f sarif\|findings-json\|markdown\|json`, `--fix-output` (Output; `json` is a deprecated SARIF alias);
|
||
`--detectors` / `--scope` (Detectors); `--budget-seconds`, `--loop-unroll`,
|
||
`--profile-solver` (Analysis); `--processes/-j` (Performance); `--suppressions`,
|
||
`--baseline` (Suppression).
|
||
|
||
**Compartmentalizing scans:** `--scope security|quality|redundancy|binary|all` is the
|
||
friendly front-end that expands to a detector category; it composes with `--detectors`.
|
||
`--profile` is metrics-only; modeling presets live on `--preset <name>` (currently
|
||
`freestanding`). A value passed to `--profile` is a deprecated alias for `--preset`.
|
||
|
||
---
|
||
|
||
## 4. Detector taxonomy
|
||
|
||
Categories live in `src/cli/detector_plan.cpp` (`kSourceDetectorDescriptors`) as
|
||
`SourceDetectorCategory{Redundancy, Security, Quality}`. Binary detectors are named
|
||
separately (`src/cli/detector_registry.cpp` `kBinaryLeaves`).
|
||
|
||
```mermaid
|
||
flowchart TD
|
||
root[detectors]
|
||
root --> sec[Security]
|
||
root --> qual[Quality]
|
||
root --> red[Redundancy]
|
||
root --> binr[Binary]
|
||
|
||
sec --> slist["div-by-zero · null-deref · int-overflow<br/>int-truncation · sign-conversion · oob · taint-track<br/>sql-injection · uninit-read · format-string<br/>use-after-free · double-free · memory-leak<br/>hardcoded-credential · weak-crypto · deprecated-banned"]
|
||
qual --> qlist["dead-statement · unreachable · unused-function<br/>complexity · deprecated-api · use-after-move<br/>static-ptr-asymmetry · iterator-invalidation"]
|
||
red --> rlist["clone · equiv · class-dup · dead<br/>always-condition · redundant-store"]
|
||
binr --> blist["binary-security · binary-null-deref<br/>dangerous-import · hardcoded-string<br/>+ ROP/gadget exploit synthesis"]
|
||
```
|
||
|
||
**Confidence model** (`include/fold/report/finding.hpp`), strongest → weakest:
|
||
`ConcreteWitness` (fuzzer-reproduced crash) > `Proved` (Z3 UNSAT of negation) > `Latent`
|
||
(proved modulo caller precondition) > `Heuristic` (over-approximated) > `Syntactic`
|
||
(AST/CFG-shape match). There is **no `Severity` enum** — `report::triage::scoreFinding`
|
||
derives a 0–1 score from confidence + summary backing + cross-TU corroboration.
|
||
|
||
**Supported frontends:** C/C++ (clang, full suite), Rust (`fold-rust` HIR→IR; kinds
|
||
`rust-unsafe-api`, `rust-unchecked-index`, `rust-path-traversal`, `rust-command-injection`,
|
||
`rust-crypto-misuse`), Swift (`fold-swift` SWAN SIL, reuses the Rust IR/kinds), and
|
||
binaries (radare2 + Ghidra P-Code, heuristic hazards + ROP primitives).
|
||
|
||
---
|
||
|
||
## 5. Audit — testing infrastructure
|
||
|
||
**Framework:** Catch2, gated behind `-Dbuild-tests`.
|
||
|
||
| Suite | Wiring | Count |
|
||
|---|---|---|
|
||
| Unit | `tests/unit/meson.build` | 89 base targets plus feature-gated tests |
|
||
| Integration | `tests/integration/meson.build` | 13 base targets plus feature-gated tests |
|
||
| Benchmark / architecture | `tests/benchmarks/meson.build` | evaluator, source-structure, source-metamorphic, UBSan/ASan/Clang-SA differential, and organic-pair gates |
|
||
| Confirmation / KLEE smoke | `tests/confirmation/` | — |
|
||
| Fixtures | `tests/fixtures/` | by category |
|
||
|
||
> **Doc correction:** fixed test-count claims become stale as feature-gated suites grow.
|
||
> The validated release configuration on 2026-07-13 registered 113 Meson tests; use
|
||
> `meson test -C build/release --list` as the source of truth.
|
||
|
||
**Product-level test features:**
|
||
|
||
- `fold fuzz prep`/`run` — libFuzzer + ASan/UBSan harness generation and crash ingestion
|
||
(promotes `Proved` → `ConcreteWitness`).
|
||
- **Precision/recall harness** in `bench/`: `run_fold_on_juliet.py`, `run_svcomp.py`,
|
||
`run_curl_ab.py`, `run_security_targets.py`, `run_quality_gate.py`,
|
||
`compare_literature.py`, plus baselines `fp-baseline*.json` and `literature_baselines.json`.
|
||
This is the real precision guard the unit suite cannot provide.
|
||
|
||
**What "green" means** (per `CLAUDE.md`): build clean, all tests pass, **and** finding-count
|
||
parity on a known OSS target (curl is canonical) for any encoder/observer change.
|
||
|
||
---
|
||
|
||
## 6. Audit — profiling infrastructure
|
||
|
||
Fold is **self-instrumented**; no external profiler (Tracy/perf/Instruments) is wired.
|
||
|
||
- **`ScanProfile` / `DetectorProfile`** (`src/cli/profile.cpp`,
|
||
`include/fold/cli/profile.hpp`): ~45 phase-level counters + `peak_rss_kb`, serialized via
|
||
`ScanProfile::to_json`. Emitted by bare `--profile` (consolidated metrics JSON), written
|
||
to `--profile-output`.
|
||
- **`SolverStats`** — per-detector Z3 query counts/time, surfaced by `--profile-solver`;
|
||
encoder reachability/bail metrics via `--profile-reach`.
|
||
- **`bench/perf_audit_run.py`** — multi-run perf harness; dated baseline snapshots under
|
||
`docs/internal/perf-audit/` (e.g. `baseline-2026-05-23.json`,
|
||
`baseline-phases-2026-05-23.json`).
|
||
|
||
**Gap / opportunity:** profiling measures *fold's own* runtime. It does **not** measure
|
||
*target-code* complexity. Complexity/hotspot analysis of the analyzed program is a separate
|
||
capability (a Quality-category detector — see `complexity`), complementing the existing
|
||
dead-code detectors (`dead`, `dead-statement`, `unreachable`, `unused-function`).
|
||
|
||
For the structural-quality detector map, Yams rollout tiers, and the benchmark
|
||
contract that joins target-code findings to Fold runtime/reliability metrics,
|
||
see [Code quality and Yams benchmark](code-quality-and-yams-benchmark.md).
|
||
|
||
---
|
||
|
||
## 7. Key files index
|
||
|
||
| Concern | Path |
|
||
|---|---|
|
||
| CLI entry / flag wiring | `src/cli/main.cpp` |
|
||
| Scan routing / mode resolve | `src/cli/scan_router.cpp` |
|
||
| Source scan coordination | `src/cli/scan_runner.cpp` |
|
||
| Backend shard enrichment / dispatch | `src/cli/backend_shard_runner.cpp` |
|
||
| Incremental function fingerprints | `src/cli/incremental_fingerprint.cpp` |
|
||
| Source call graph and conservative dispatch resolution | `src/cli/source_call_graph.cpp` |
|
||
| Symbolic-summary SCC/fixpoint pipeline | `src/cli/source_summary_pipeline.cpp` |
|
||
| Per-TU detector execution and solver-stat ownership | `src/cli/source_tu_runner.cpp` |
|
||
| Finding proof/evidence annotation | `src/cli/finding_evidence.cpp` |
|
||
| TU quarantine persistence | `src/cli/tu_quarantine.cpp` |
|
||
| Detector taxonomy & plan | `src/cli/detector_plan.{hpp,cpp}` |
|
||
| Detector name registry | `src/cli/detector_registry.cpp` |
|
||
| Modeling presets | `src/cli/scan_preset.cpp` |
|
||
| Scan options struct | `include/fold/cli/scan_options.hpp` |
|
||
| Profiling | `src/cli/profile.cpp`, `include/fold/cli/profile.hpp` |
|
||
| Finding / confidence model | `include/fold/report/finding.hpp` |
|
||
| Triage / scoring | `src/report/triage.cpp` |
|
||
| Output formats | `src/report/{sarif,markdown,finding_json,findings_json_report}.cpp` |
|
||
| Z3 encoder dispatch / foundations / operators / expression arms | `src/symbolic/encoder{,_foundation,_operators,_expr,_expr_arms}.cpp` |
|
||
| Composite security-walker facade / analysis / contracts / refinement / control | `src/detect/security_walker{,_analysis,_contracts,_refinement,_engine_core,_engine_control}.cpp` |
|
||
| Div-by-zero guard facts and guard context | `src/detect/div_by_zero{,_guard_facts,_guard_context}.cpp` |
|
||
| Null, overflow, and OOB candidate analysis | `src/detect/{null_deref,int_overflow,oob_access}_analysis.cpp` |
|
||
| radare2 orchestration / session / serialization / disassembly state / extraction | `src/binary/r2_{backend,session,serialize,disasm_state,extract}.cpp` |
|
||
| Benchmark case/run schemas and artifact hashes | `scripts/benchmarks/fold_eval/schema.py` |
|
||
| Classification, sliced micro/macro, and paired statistics | `scripts/benchmarks/fold_eval/metrics.py`, `performance.py` |
|
||
| Canonical SARIF and metamorphic finding comparison | `scripts/benchmarks/fold_eval/sarif.py`, `metamorphic.py` |
|
||
| Runtime confirmation state model | `scripts/benchmarks/fold_eval/confirmation.py` |
|
||
| Source metamorphic benchmark runner | `scripts/benchmarks/run_source_metamorphic.py` |
|
||
| UBSan/ASan vulnerable/fixed differential runner | `scripts/benchmarks/run_sanitizer_differential.py` |
|
||
| Clang Static Analyzer differential runner | `scripts/benchmarks/run_clang_sa_differential.py` |
|
||
| Pinned organic vulnerable/fixed runner | `scripts/benchmarks/run_organic_pairs.py` |
|
||
| Shared Fold source comparator lane | `scripts/benchmarks/fold_eval/source_lane.py` |
|
||
| Reviewed disposition and change-time evaluator | `scripts/benchmarks/fold_eval/dispositions.py` |
|
||
| Yams performance, reliability, and change-time runner | `scripts/benchmarks/run_code_quality.py` |
|
||
| Production source-size ratchet | `scripts/check_source_structure.py`, `tests/architecture/source_budgets.json` |
|