Round 3, iteration 1 of the fold optimization loop
(docs/research/optimization-loop-2026-07-30.md).
The foundation audit found gold scoring machinery (26 oracle files in
bench/gold/, fold_eval/scoring.py, gold_threshold_failures gates)
fully implemented but wired into zero project manifests — the eval
measured finding drift, never precision/recall.
New gold standards anchored on the reviewed dispositions:
- bench/gold/fold_self_gold.json: 9 cases (5 SafetyObserver stub
clones, 1 detector-internal clone, 1 loop_inventory clone, 2
sql-injection sites reviewed false-positive-but-intentionally-
reported)
- bench/gold/yams_self_gold.json: 7 cases (3 clone, 4 high-complexity)
Both manifests now enforce gold_thresholds precision 1.0 / recall 1.0:
any detector drift that loses a reviewed finding (recall drop) or
emits a new unmatched finding (precision drop) fails the evaluation.
Proof runs: fold TP=9/FP=0/FN=0, yams TP=7/FP=0/FN=0, precision 1.0,
recall 1.0, fingerprints exactly preserved (0fec78e5.../247523c5...),
full suite 137/137. Gold files live in bench/gold/ alongside the
existing oracles (the tests/benchmarks schema-test glob correctly does
not sweep that directory).
- checkAdaptive: cap escalation at 4 min (8x base) instead of 1 hour. The
1-hour ceiling let a genuinely-hard query pin a worker for an hour with
zero file completion, stalling the whole scan (observed: 5 workers at 100%
CPU, log stopped advancing). 4 min still resolves "just barely over budget"
queries while bounding pathological ones.
- score_fp.py: guard the last bare .read_text(), drop an unused `elapsed`,
log the fold --version failure instead of bare pass, and suppress the
runtime sys.path import (reportMissingImports/E402).
Replace bare json.loads/int/float calls with small guarded helpers so a
missing or malformed gold/SARIF/compile-DB file exits with a clear message
instead of a raw traceback, and numeric fields degrade to defaults.
Add negative_cases_from_universe to expand an explicit file-by-rule universe
into labeled negatives (enabling TP/FP/TN/FN, not just TP/FP), plus project-eval
harness updates in run_project_eval and the score/score_fp/quality-gate helpers.
- skills/fold-sast: rename to skill-fold-sast and expand the scan/precision
guidance for C++/Rust/Swift lanes
- docs/architecture: record call-graph phase decomposition and the
optimization-loop TP/FP/TN/FN evidence policy
- docs/mcp-tools + README: update budget/per-query defaults (1800s/6000ms)
and note the OpenGrep pre-push hook
Extend ir::Loop classification: a monotone single-induction-variable for-loop
with a symbolic bound (e.g. `for (i=0; i<n; ++i)`) is now Affine (was General).
The walker summarizes Affine loops like Counted ones — bind the induction
variable to a fresh symbolic, assert its start bound plus the loop condition
(encoded to a Z3 bool) as path constraints, walk the body once — exposing
high-index OOB that bounded unroll misses. Findings are over-approximations
(LoopHavoc/heuristic), never proved.
Net effect on the gate: oob recall 0.531 -> 0.559, F1 0.637 -> 0.652 (precision
dips 0.796 -> 0.782) — a contract-sanctioned over-approximation tradeoff.
Also regenerate bench/fp-baseline.json deterministically from a clean run: the
prior committed oob numbers (0.574) were not reproducible (transient warm-state
artifact); fresh runs are now bit-identical, so the gate is reliable.
Tests: test_ir_loop (affine classification), test_loop_summary (symbolic-bound
OOB exposed, not proved). Suite green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
For a counted loop whose trip count exceeds the unroll bound, bind the
induction variable to its full closed-form range (asserted as a Z3 path
constraint, not just interval metadata) and walk the body once, instead of
unrolling loop_unroll times. This fires each in-body sink against all
iterations at once: it exposes high-index OOB that bounded unroll misses
(oob recall 0.531 -> 0.574, F1 0.637 -> 0.658 on the gate) and collapses
per-iteration queries to one. Findings from the summarized walk are
over-approximations and are tagged LoopHavoc (heuristic) confidence, never
proved. General/while(1) loops keep the bounded-unroll fallback.
Per the precision contract (loop-summarization-research.md), this is a
recall-improving over-approximation that may add bounded, tagged FPs. The
drift gate is updated to match: it now fails on recall- or F1-drop beyond
tolerance (the regressions it was built for) and allows precision dips that
come with recall gains; the committed baseline is regenerated to the B2
numbers.
Tests: test_loop_summary (high-index OOB found + not proved; in-bounds loop
clean). Suite 68/0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Enumeration: replace the hand-rolled top-level `tu->decls()` walks (TU
security prefilter + every interprocedural pre-pass/cross-TU primer in
div-by-zero/int-overflow/null-deref/oob) with a shared RAV-based
`fold::detect::forEachFunctionWithBody`, so functions nested in namespaces
and classes are analyzed. Fixes 0-findings on namespaced C++ (dbench CTF).
TU prefilter: extracted to fold/cli/tu_security_prefilter; flags unsigned
-/* wraps so int-overflow candidates in size arithmetic are not skipped.
Timeout robustness: cooperative per-TU walk deadline (WalkOptions) checked
in the walk/unrollLoop/CFG-block checkpoints and the BatchSafetyObserver
flush loop, plus a non-progress unroll fixpoint — slow TUs degrade with
partial findings instead of being SIGALRM-quarantined.
OOB precheck: treat `*(p+i)` and indirect callees as candidates.
null-deref drift fix: remove the unsound `priorFunctionTextProvesNonNull`
source-text heuristic (added in 331a68467) that treated a textual `p!=0`
before a deref as proof of non-null, silently suppressing real null-derefs
(CWE-476 F1 0.485 -> 0.583, recall 0.379 -> 0.517).
Tests: test_function_enumeration, test_tu_security_prefilter, namespaced
cross-TU int-overflow regression. Full suite green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds bench/perf_audit_run.py which runs fold N times against a
fixed compile DB and reports median + IQR + min/max for every
metric in --profile JSON (scan-level + per-detector). Output
snapshots land under docs/perf-audit/runs/<label>.json so
before/after diffs are git-tracked.
Motivation: the audit's writtenvars-cache commit (5221847)
showed sub-15% perf moves can't be reliably attributed from a
single run — cross-baseline variance was ~15% just from system
load. Future perf PRs need a statistical baseline. This harness
is the single-machine version of that discipline.
Usage:
bench/perf_audit_run.py --label baseline --runs 5
bench/perf_audit_run.py --label after-fix --runs 5 \\
--diff docs/perf-audit/runs/baseline.json
The --diff mode prints a side-by-side with significance flagged
("yes"/"no") when the delta exceeds 1.5×IQR. That's a rough rule
of thumb; for high-confidence claims, run 10+ samples.
Smoke-tested with --runs 2 (15min wall) on the curl baseline:
2-run IQR = 13s on a ~440s scan (~3% band). 5 runs should
tighten that.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
