fold/scripts/benchmarks/fold_eval
trvon 84746e060b
Some checks failed
build / build-linux (push) Failing after 7m33s
build / build-macos (push) Has been cancelled
build / release (push) Has been cancelled
fix(eval): finding fingerprint must be identity-based, not content-based
Backlog #7. Repro showed chromium finding LOCATIONS (rule/uri/line/
col/id) were byte-identical across runs while the summary
finding_fingerprint drifted (064e60c9 -> 4573bb71): summarize_sarif_findings
hashed the full canonical finding record, which carries Z3-timing-
volatile fields (quality metrics, uncertainty reasons, confidence,
score). The stability gate already used identity-only keys, so the
fingerprint contradicted it on Z3-heavy corpora.

finding_identity_key now = {finding_id, rule_id, uri, line, backend}
(measurement artifacts excluded), and the fingerprint is the sha256
of the sorted identity keys — consistent with how change-time drift
and dispositions already define 'a finding'.

Verified: chromium fingerprint STABLE across identical runs
(f5629f0f). Repinned the four baselines to the identity-based
fingerprint (fold 6fe221a0, yams c9ad7d60, sage 3da6ecfa, chromium
f5629f0f).
2026-09-01 19:54:54 -06:00
..
__init__.py bench: extend fold_eval scoring with labeled negative cases 2026-08-16 16:43:51 -06:00
confirmation.py feat: harden Fold quality guard 2026-07-13 15:05:35 -04:00
differential.py feat: harden Fold quality guard 2026-07-13 15:05:35 -04:00
dispositions.py feat: harden Fold quality guard 2026-07-13 15:05:35 -04:00
flaky_evidence.py feat: converge quality and scan interfaces 2026-07-29 00:57:33 -04:00
metamorphic.py feat: harden Fold quality guard 2026-07-13 15:05:35 -04:00
metrics.py bench: extend fold_eval scoring with labeled negative cases 2026-08-16 16:43:51 -06:00
performance.py bench: extend fold_eval scoring with labeled negative cases 2026-08-16 16:43:51 -06:00
project.py bench: extend fold_eval scoring with labeled negative cases 2026-08-16 16:43:51 -06:00
sarif.py fix(eval): finding fingerprint must be identity-based, not content-based 2026-09-01 19:54:54 -06:00
schema.py test(quality): strengthen evaluation evidence 2026-07-29 20:47:04 -04:00
scoring.py bench: extend fold_eval scoring with labeled negative cases 2026-08-16 16:43:51 -06:00
source_lane.py test(quality): strengthen evaluation evidence 2026-07-29 20:47:04 -04:00
stability.py feat: harden Fold quality guard 2026-07-13 15:05:35 -04:00
test_selection.py feat: converge quality and scan interfaces 2026-07-29 00:57:33 -04:00
token_efficiency.py feat: converge quality and scan interfaces 2026-07-29 00:57:33 -04:00
toolchain.py feat: harden Fold quality guard 2026-07-13 15:05:35 -04:00