fold/docs/research/tool-maturity-plan-2026-06-03.md

# Fold Tool Maturity Plan

> Date: 2026-06-03
> Scope: post linked-r2/P-Code FP hardening. This is a maturity plan, not a blocker list for shipped functionality.

## Objective

Move Fold from feature-complete prototypes toward an analyzer with measurable maturity:

- fewer false positives on real binaries/source trees;
- stable report contracts and provenance;
- bounded runtime/memory behavior;
- repeatable benchmark evidence before and after detector/backend changes;
- tests that expose poor architectural shortcuts, not just local detector bugs.

## Guiding principle

Before adding more detectors, define and guard the analyzer behaviors we already rely on:

1. Findings must have physical locations and backend provenance.
2. Source, binary, Rust, Swift, and confirmation lanes must remain separately measurable.
3. Heuristic binary findings must not be upgraded silently to proved without a sound model.
4. Encoder/lifter failures degrade locally and visibly, not by aborting a scan.
5. Benchmarks should include safe code that looks dangerous, not only vulnerable code.

## Maturity scorecard

Track this per release or major backend change.

| Metric | Gate | Why it matters |
|---|---:|---|
| Full test suite | 100% pass | Baseline correctness |
| Focused binary suite | 100% pass | r2/Ghidra/Capstone regressions |
| Locationless findings | 0 emitted | Report quality / agent usability |
| Backend provenance | 100% findings tagged | Lane separation |
| Known FP corpus regressions | 0 new must-not-report hits | Precision discipline |
| Known TP corpus regressions | 0 missing must-report hits | Recall discipline |
| Scan crash rate | 0 on benchmark corpus | Operational stability |
| Encoder/lifter fatal aborts | 0 | Degrade, do not abort |
| Runtime regression | <= 20% unless justified | Usability on real codebases |
| Peak RSS regression | <= 20% unless justified | Large-binary/codebase stability |
| Schema compatibility | golden JSON/SARIF keys stable | Downstream tooling |

## Benchmark layers

### 1. Unit regression fixtures

Fast tests that run on every local validation.

- `pcode_encoder`: P-Code semantics and sink modeling.
- `pcode_summary`: interprocedural write-loop summaries.
- report filtering/dedup tests.
- Rust/Swift IR encoder round-trips and degradation behavior.

Required additions:

- read-only scanners are never write sinks: `memchr`, `strchr`, `strstr`;
- fortified wrappers are not unchecked copies: `__strncpy_chk`, `__memcpy_chk`;
- guarded arithmetic/copy patterns suppress hazards;
- machine-instruction gating for raw P-Code div/rem;
- malformed child IR degrades per-node.

### 2. Golden source corpus

Small C/C++ fixtures with paired vulnerable/safe cases.

For each rule:

- true positive: minimal exploitable/reachable shape;
- safe lookalike: same syntax, but guarded by range/null/bounds checks;
- unsupported construct: must degrade without false proof;
- generated/vendor case: should be suppressible/excludable.

Rules to cover first:

- null-deref;
- div-by-zero;
- OOB;
- signed overflow;
- taint to sink;
- UAF/double-free;
- uninitialized read.

Each fixture should assert:

- rule ID;
- confidence;
- physical location;
- backend;
- no extra findings beyond budget.

### 3. Golden binary corpus

Small compiled binaries plus optional real-system smoke scans.

Must include:

- import-stub adjacency (`memchr` next to `memcpy`);
- fortified libc wrappers;
- stripped binaries;
- PIE and non-PIE variants;
- AArch64 guarded branch idioms (`b.ls`, `b.hi`, `cbz`, `cbnz`);
- x86/x64 division and guard idioms;
- safe read-only scanners;
- real unguarded copy overflow fixture;
- large binary runtime smoke with capped functions.

Golden checks:

- must-report findings for real unsafe copy/div cases;
- must-not-report findings for guarded-safe cases;
- backend provenance remains `binary`, `binary-ir`, `binary-pcode-raw`, or `binary-pcode-ssa` as expected;
- no locationless findings.

### 4. Cross-lane/hybrid corpus

Purpose: catch router and provenance regressions.

Cases:

- source-only finding;
- binary-only finding;
- hybrid run with both inputs;
- same conceptual bug observed in source and binary with distinct provenance;
- confirmation task generated from source/Rust findings.

Gates:

- hybrid does not hide source coverage;
- binary heuristics do not masquerade as source proofs;
- confirmation tasks preserve lane/backend and target identity.

### 5. Poor-design pressure tests

These are intentionally hostile to bad analyzer architecture.

| Pressure test | Bad design exposed |
|---|---|
| Generated/vendor code with many scary imports | no suppression strategy / noisy defaults |
| Huge function with many branches | path explosion / no budget discipline |
| Stripped binary with dense import stubs | fragile target-name resolution |
| Malformed Rust/Swift NDJSON | frontend crash instead of local degradation |
| Empty/missing source locations | report artifacts leaking to users |
| Unsupported P-Code ops near real hazards | proof over unconstrained nonsense |
| Large executable section | memory blowup in gadget scan |
| Same finding through multiple lanes | broken dedup/provenance |

A maturity benchmark should include safe pressure tests. If Fold reports many high-confidence findings on intentionally safe pressure tests, the model is overclaiming.

## Harness plan

Create `scripts/benchmarks/` with:

```text
scripts/benchmarks/
├── run_source_goldens.py       # done: source fixture goldens
├── run_binary_goldens.py       # done: binary fixture/smoke goldens
├── run_hybrid_goldens.py       # done: merge-only hybrid goldens
├── run_rust_swift_goldens.py   # done: optional frontend goldens
├── runner_common.py            # done: resource measurement/budget helper
├── compare_findings.py         # done: report/spec comparator
├── summarize_maturity.py       # done: umbrella runner
└── baselines/                  # next gate: required compare inputs
    ├── source.json
    ├── binary.json
    ├── hybrid.json
    └── performance.json
```

Required comparator behavior:

- fail if a `must_report` finding is missing;
- fail if a `must_not_report` finding appears;
- fail if a finding has no physical location;
- fail if a finding lacks backend/lane provenance;
- warn or fail on runtime/RSS regression over configured threshold;
- emit markdown and JSON summaries.

## CI / local validation tiers

### Fast local gate

```sh
ninja -C build/release
meson test -C build/release \
  fold:pcode_encoder fold:pcode_summary fold:scan_e2e \
  fold:binary_backend fold:cli_paths --print-errorlogs
git diff --check
```

### Binary maturity gate

```sh
scripts/benchmarks/run_binary_goldens.py --build build/release --compare scripts/benchmarks/baselines/binary.json
```

### Full maturity gate

```sh
meson test -C build/release --print-errorlogs
scripts/benchmarks/summarize_maturity.py --build build/release --all --compare scripts/benchmarks/baselines/
```

The next maturity increment is to make those baseline paths first-class: missing optional toolchains may produce structured skips, but malformed or missing required baseline files should fail closed rather than silently degrading to an un-compared smoke run.

## Assertion/invariant maturity

See also `docs/research/assertion-invariant-maturity-audit-2026-06-03.md`.

Add or audit always-on invariants where violation corrupts analysis:

- every emitted finding has a physical location unless explicitly internal-only;
- every emitted finding has backend provenance;
- scan router mode resolves to exactly the intended backend set;
- Rust/Swift frontend failures are attached to the node/function, not whole scan aborts;
- P-Code call target annotation does not overwrite a raw call-site name with a synthesized SSA target;
- confirmation task/result JSON keys remain centralized and round-trippable;
- benchmark comparator fails closed when a baseline file is malformed.

Avoid assertions for recoverable environment issues:

- missing KLEE;
- missing optional radare2 dev libs when CLI fallback is allowed;
- unsupported architecture;
- absent compile database in binary-only mode.

## Task backlog

Completed in the current working tree:

1. Define Fold maturity scorecard.
2. Build binary FP regression corpus.
3. Build source precision benchmark corpus.
4. Add benchmark harness and umbrella summary.
5. Audit assertion and invariant policy.
6. Create initial poor-design pressure tests.

Open next increments:

1. Promote benchmark baselines into a required maturity gate with fail-closed malformed-baseline handling.
2. Add baselines for source, binary, hybrid, and performance/RSS summaries.
3. Keep expanding safe pressure tests for generated/vendor code, unsupported P-Code near hazards, and same-finding multi-lane provenance.
4. Require the maturity summary in release/checkpoint validation once baselines stabilize across supported hosts.

## Done criteria for this maturity phase

Current status:

- A one-command benchmark summary exists: `scripts/benchmarks/summarize_maturity.py --all`.
- Source, binary, hybrid, Rust, and Swift golden suites exist.
- Known curl-derived binary FPs are locked by unit tests and a binary smoke baseline.
- Portable generated binary fixtures now cover guarded division, scanner-vs-writer P-Code pressure, and a hardcoded secret positive control.
- Report location/provenance checks are automated in `compare_findings.py`, including aggregate backend presence checks for hybrid/frontends.
- The benchmark corpus includes vulnerable/safe lookalike pairs and positive controls.
- Runtime, CPU, and RSS metrics are recorded where `/usr/bin/time` exposes them; loose optional budgets are enabled on deterministic small fixtures.