fold/docs/research/tool-maturity-plan-2026-06-03.md

Fold Tool Maturity Plan

Date: 2026-06-03 Scope: post linked-r2/P-Code FP hardening. This is a maturity plan, not a blocker list for shipped functionality.

Objective

Move Fold from feature-complete prototypes toward an analyzer with measurable maturity:

• fewer false positives on real binaries/source trees;
• stable report contracts and provenance;
• bounded runtime/memory behavior;
• repeatable benchmark evidence before and after detector/backend changes;
• tests that expose poor architectural shortcuts, not just local detector bugs.

Guiding principle

Before adding more detectors, define and guard the analyzer behaviors we already rely on:

1. Findings must have physical locations and backend provenance.
2. Source, binary, Rust, Swift, and confirmation lanes must remain separately measurable.
3. Heuristic binary findings must not be upgraded silently to proved without a sound model.
4. Encoder/lifter failures degrade locally and visibly, not by aborting a scan.
5. Benchmarks should include safe code that looks dangerous, not only vulnerable code.

Maturity scorecard

Track this per release or major backend change.

Metric	Gate	Why it matters
Full test suite	100% pass	Baseline correctness
Focused binary suite	100% pass	r2/Ghidra/Capstone regressions
Locationless findings	0 emitted	Report quality / agent usability
Backend provenance	100% findings tagged	Lane separation
Known FP corpus regressions	0 new must-not-report hits	Precision discipline
Known TP corpus regressions	0 missing must-report hits	Recall discipline
Scan crash rate	0 on benchmark corpus	Operational stability
Encoder/lifter fatal aborts	0	Degrade, do not abort
Runtime regression	<= 20% unless justified	Usability on real codebases
Peak RSS regression	<= 20% unless justified	Large-binary/codebase stability
Schema compatibility	golden JSON/SARIF keys stable	Downstream tooling

Benchmark layers

1. Unit regression fixtures

Fast tests that run on every local validation.

• pcode_encoder: P-Code semantics and sink modeling.
• pcode_summary: interprocedural write-loop summaries.
• report filtering/dedup tests.
• Rust/Swift IR encoder round-trips and degradation behavior.

Required additions:

• read-only scanners are never write sinks: memchr, strchr, strstr;
• fortified wrappers are not unchecked copies: __strncpy_chk, __memcpy_chk;
• guarded arithmetic/copy patterns suppress hazards;
• machine-instruction gating for raw P-Code div/rem;
• malformed child IR degrades per-node.

2. Golden source corpus

Small C/C++ fixtures with paired vulnerable/safe cases.

For each rule:

• true positive: minimal exploitable/reachable shape;
• safe lookalike: same syntax, but guarded by range/null/bounds checks;
• unsupported construct: must degrade without false proof;
• generated/vendor case: should be suppressible/excludable.

Rules to cover first:

• null-deref;
• div-by-zero;
• OOB;
• signed overflow;
• taint to sink;
• UAF/double-free;
• uninitialized read.

Each fixture should assert:

• rule ID;
• confidence;
• physical location;
• backend;
• no extra findings beyond budget.

3. Golden binary corpus

Small compiled binaries plus optional real-system smoke scans.

Must include:

• import-stub adjacency (memchr next to memcpy);
• fortified libc wrappers;
• stripped binaries;
• PIE and non-PIE variants;
• AArch64 guarded branch idioms (b.ls, b.hi, cbz, cbnz);
• x86/x64 division and guard idioms;
• safe read-only scanners;
• real unguarded copy overflow fixture;
• large binary runtime smoke with capped functions.

Golden checks:

• must-report findings for real unsafe copy/div cases;
• must-not-report findings for guarded-safe cases;
• backend provenance remains binary, binary-ir, binary-pcode-raw, or binary-pcode-ssa as expected;
• no locationless findings.

4. Cross-lane/hybrid corpus

Purpose: catch router and provenance regressions.

Cases:

• source-only finding;
• binary-only finding;
• hybrid run with both inputs;
• same conceptual bug observed in source and binary with distinct provenance;
• confirmation task generated from source/Rust findings.

Gates:

• hybrid does not hide source coverage;
• binary heuristics do not masquerade as source proofs;
• confirmation tasks preserve lane/backend and target identity.

5. Poor-design pressure tests

These are intentionally hostile to bad analyzer architecture.

Pressure test	Bad design exposed
Generated/vendor code with many scary imports	no suppression strategy / noisy defaults
Huge function with many branches	path explosion / no budget discipline
Stripped binary with dense import stubs	fragile target-name resolution
Malformed Rust/Swift NDJSON	frontend crash instead of local degradation
Empty/missing source locations	report artifacts leaking to users
Unsupported P-Code ops near real hazards	proof over unconstrained nonsense
Large executable section	memory blowup in gadget scan
Same finding through multiple lanes	broken dedup/provenance

A maturity benchmark should include safe pressure tests. If Fold reports many high-confidence findings on intentionally safe pressure tests, the model is overclaiming.

Harness plan

Create scripts/benchmarks/ with:

scripts/benchmarks/
├── run_source_goldens.py       # done: source fixture goldens
├── run_binary_goldens.py       # done: binary fixture/smoke goldens
├── run_hybrid_goldens.py       # done: merge-only hybrid goldens
├── run_rust_swift_goldens.py   # done: optional frontend goldens
├── runner_common.py            # done: resource measurement/budget helper
├── compare_findings.py         # done: report/spec comparator
├── summarize_maturity.py       # done: umbrella runner
└── baselines/                  # next gate: required compare inputs
    ├── source.json
    ├── binary.json
    ├── hybrid.json
    └── performance.json

Required comparator behavior:

• fail if a must_report finding is missing;
• fail if a must_not_report finding appears;
• fail if a finding has no physical location;
• fail if a finding lacks backend/lane provenance;
• warn or fail on runtime/RSS regression over configured threshold;
• emit markdown and JSON summaries.

CI / local validation tiers

Fast local gate

ninja -C build/release
meson test -C build/release \
  fold:pcode_encoder fold:pcode_summary fold:scan_e2e \
  fold:binary_backend fold:cli_paths --print-errorlogs
git diff --check

Binary maturity gate

scripts/benchmarks/run_binary_goldens.py --build build/release --compare scripts/benchmarks/baselines/binary.json

Full maturity gate

meson test -C build/release --print-errorlogs
scripts/benchmarks/summarize_maturity.py --build build/release --all --compare scripts/benchmarks/baselines/

The next maturity increment is to make those baseline paths first-class: missing optional toolchains may produce structured skips, but malformed or missing required baseline files should fail closed rather than silently degrading to an un-compared smoke run.

Assertion/invariant maturity

See also docs/research/assertion-invariant-maturity-audit-2026-06-03.md.

Add or audit always-on invariants where violation corrupts analysis:

• every emitted finding has a physical location unless explicitly internal-only;
• every emitted finding has backend provenance;
• scan router mode resolves to exactly the intended backend set;
• Rust/Swift frontend failures are attached to the node/function, not whole scan aborts;
• P-Code call target annotation does not overwrite a raw call-site name with a synthesized SSA target;
• confirmation task/result JSON keys remain centralized and round-trippable;
• benchmark comparator fails closed when a baseline file is malformed.

Avoid assertions for recoverable environment issues:

• missing KLEE;
• missing optional radare2 dev libs when CLI fallback is allowed;
• unsupported architecture;
• absent compile database in binary-only mode.

Task backlog

Completed in the current working tree:

1. Define Fold maturity scorecard.
2. Build binary FP regression corpus.
3. Build source precision benchmark corpus.
4. Add benchmark harness and umbrella summary.
5. Audit assertion and invariant policy.
6. Create initial poor-design pressure tests.

Open next increments:

1. Promote benchmark baselines into a required maturity gate with fail-closed malformed-baseline handling.
2. Add baselines for source, binary, hybrid, and performance/RSS summaries.
3. Keep expanding safe pressure tests for generated/vendor code, unsupported P-Code near hazards, and same-finding multi-lane provenance.
4. Require the maturity summary in release/checkpoint validation once baselines stabilize across supported hosts.

Done criteria for this maturity phase

Current status:

• A one-command benchmark summary exists: scripts/benchmarks/summarize_maturity.py --all.
• Source, binary, hybrid, Rust, and Swift golden suites exist.
• Known curl-derived binary FPs are locked by unit tests and a binary smoke baseline.
• Portable generated binary fixtures now cover guarded division, scanner-vs-writer P-Code pressure, and a hardcoded secret positive control.
• Report location/provenance checks are automated in compare_findings.py, including aggregate backend presence checks for hybrid/frontends.
• The benchmark corpus includes vulnerable/safe lookalike pairs and positive controls.
• Runtime, CPU, and RSS metrics are recorded where /usr/bin/time exposes them; loose optional budgets are enabled on deterministic small fixtures.