Add negative_cases_from_universe to expand an explicit file-by-rule universe
into labeled negatives (enabling TP/FP/TN/FN, not just TP/FP), plus project-eval
harness updates in run_project_eval and the score/score_fp/quality-gate helpers.
Adds bench/perf_audit_run.py which runs fold N times against a
fixed compile DB and reports median + IQR + min/max for every
metric in --profile JSON (scan-level + per-detector). Output
snapshots land under docs/perf-audit/runs/<label>.json so
before/after diffs are git-tracked.
Motivation: the audit's writtenvars-cache commit (5221847)
showed sub-15% perf moves can't be reliably attributed from a
single run — cross-baseline variance was ~15% just from system
load. Future perf PRs need a statistical baseline. This harness
is the single-machine version of that discipline.
Usage:
bench/perf_audit_run.py --label baseline --runs 5
bench/perf_audit_run.py --label after-fix --runs 5 \\
--diff docs/perf-audit/runs/baseline.json
The --diff mode prints a side-by-side with significance flagged
("yes"/"no") when the delta exceeds 1.5×IQR. That's a rough rule
of thumb; for high-confidence claims, run 10+ samples.
Smoke-tested with --runs 2 (15min wall) on the curl baseline:
2-run IQR = 13s on a ~440s scan (~3% band). 5 runs should
tighten that.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
