Round 4, iteration 2 of the fold optimization loop
(docs/research/optimization-loop-2026-07-30.md).
The yams wall-regression gate at 20% fired on a 113% concurrent-load
swing (8.1s -> 17.3s on identical code) and passed quiet at -11% —
with measured identical-code walls of 7.2-11.4s (quiet) and 17.3s
(loaded), a 20% threshold on a ~9s workload is noise-dominated and
flaky. A flaky gate is worthless as a regression oracle.
Calibrated to 50%: fires above 12.15s sustained (50% over the 8.10s
baseline) — catches true 2x regressions while tolerating the measured
40-113% machine swings. Validated: yams eval ok (7.37s vs 8.104s
baseline, -9%), gold precision/recall 1.0, full suite 137/137.
Fingerprints exactly preserved.
Round 3, iteration 2 of the fold optimization loop
(docs/research/optimization-loop-2026-07-30.md).
The foundation audit found both manifests declaring
wall_regression_pct/rss_regression_pct with no baseline artifacts —
the regression gates never engaged in the entire campaign.
New: tests/benchmarks/projects/baselines/{fold_source,yams_source}.json
(full prior-eval summaries with matching tu-set/config fingerprints
and exact finding fingerprints). Both manifests now reference their
baselines, activating the eval's baseline-compare path
(compare_performance_lanes + rss_regression).
Engage proof:
- fold: baseline_comparison engaged (paired-by-repetition, baseline
103.65s vs candidate 82.7s, -20.2%), status ok
- yams: gate FIRED on a 113% concurrent-load swing (proving it
discriminates), then passed quiet (7.181s vs 8.104s)
Also extends the schema-test prune to {runs, baselines} — third
instance of the same glob-sweep fragility the foundation audit
predicted. Full suite 137/137; fingerprints exactly preserved
(0fec78e5.../247523c5...); gold precision/recall 1.0 both.
Round 3, iteration 1 of the fold optimization loop
(docs/research/optimization-loop-2026-07-30.md).
The foundation audit found gold scoring machinery (26 oracle files in
bench/gold/, fold_eval/scoring.py, gold_threshold_failures gates)
fully implemented but wired into zero project manifests — the eval
measured finding drift, never precision/recall.
New gold standards anchored on the reviewed dispositions:
- bench/gold/fold_self_gold.json: 9 cases (5 SafetyObserver stub
clones, 1 detector-internal clone, 1 loop_inventory clone, 2
sql-injection sites reviewed false-positive-but-intentionally-
reported)
- bench/gold/yams_self_gold.json: 7 cases (3 clone, 4 high-complexity)
Both manifests now enforce gold_thresholds precision 1.0 / recall 1.0:
any detector drift that loses a reviewed finding (recall drop) or
emits a new unmatched finding (precision drop) fails the evaluation.
Proof runs: fold TP=9/FP=0/FN=0, yams TP=7/FP=0/FN=0, precision 1.0,
recall 1.0, fingerprints exactly preserved (0fec78e5.../247523c5...),
full suite 137/137. Gold files live in bench/gold/ alongside the
existing oracles (the tests/benchmarks schema-test glob correctly does
not sweep that directory).
