Commit graph

2 commits

Author SHA1 Message Date
trvon
84746e060b
fix(eval): finding fingerprint must be identity-based, not content-based
Some checks failed
build / build-linux (push) Failing after 7m33s
build / build-macos (push) Has been cancelled
build / release (push) Has been cancelled
Backlog #7. Repro showed chromium finding LOCATIONS (rule/uri/line/
col/id) were byte-identical across runs while the summary
finding_fingerprint drifted (064e60c9 -> 4573bb71): summarize_sarif_findings
hashed the full canonical finding record, which carries Z3-timing-
volatile fields (quality metrics, uncertainty reasons, confidence,
score). The stability gate already used identity-only keys, so the
fingerprint contradicted it on Z3-heavy corpora.

finding_identity_key now = {finding_id, rule_id, uri, line, backend}
(measurement artifacts excluded), and the fingerprint is the sha256
of the sorted identity keys — consistent with how change-time drift
and dispositions already define 'a finding'.

Verified: chromium fingerprint STABLE across identical runs
(f5629f0f). Repinned the four baselines to the identity-based
fingerprint (fold 6fe221a0, yams c9ad7d60, sage 3da6ecfa, chromium
f5629f0f).
2026-09-01 19:54:54 -06:00
trvon
a016268e1f
bench: activate wall/RSS regression gates via performance baselines
Some checks failed
build / build-linux (push) Failing after 8m1s
build / build-macos (push) Has been cancelled
build / release (push) Has been cancelled
Round 3, iteration 2 of the fold optimization loop
(docs/research/optimization-loop-2026-07-30.md).

The foundation audit found both manifests declaring
wall_regression_pct/rss_regression_pct with no baseline artifacts —
the regression gates never engaged in the entire campaign.

New: tests/benchmarks/projects/baselines/{fold_source,yams_source}.json
(full prior-eval summaries with matching tu-set/config fingerprints
and exact finding fingerprints). Both manifests now reference their
baselines, activating the eval's baseline-compare path
(compare_performance_lanes + rss_regression).

Engage proof:
- fold: baseline_comparison engaged (paired-by-repetition, baseline
  103.65s vs candidate 82.7s, -20.2%), status ok
- yams: gate FIRED on a 113% concurrent-load swing (proving it
  discriminates), then passed quiet (7.181s vs 8.104s)

Also extends the schema-test prune to {runs, baselines} — third
instance of the same glob-sweep fragility the foundation audit
predicted. Full suite 137/137; fingerprints exactly preserved
(0fec78e5.../247523c5...); gold precision/recall 1.0 both.
2026-08-31 18:08:16 -06:00