fold/scripts/benchmarks
trvon 84746e060b
Some checks failed
build / build-linux (push) Failing after 7m33s
build / build-macos (push) Has been cancelled
build / release (push) Has been cancelled
fix(eval): finding fingerprint must be identity-based, not content-based
Backlog #7. Repro showed chromium finding LOCATIONS (rule/uri/line/
col/id) were byte-identical across runs while the summary
finding_fingerprint drifted (064e60c9 -> 4573bb71): summarize_sarif_findings
hashed the full canonical finding record, which carries Z3-timing-
volatile fields (quality metrics, uncertainty reasons, confidence,
score). The stability gate already used identity-only keys, so the
fingerprint contradicted it on Z3-heavy corpora.

finding_identity_key now = {finding_id, rule_id, uri, line, backend}
(measurement artifacts excluded), and the fingerprint is the sha256
of the sorted identity keys — consistent with how change-time drift
and dispositions already define 'a finding'.

Verified: chromium fingerprint STABLE across identical runs
(f5629f0f). Repinned the four baselines to the identity-based
fingerprint (fold 6fe221a0, yams c9ad7d60, sage 3da6ecfa, chromium
f5629f0f).
2026-09-01 19:54:54 -06:00
..
fold_eval fix(eval): finding fingerprint must be identity-based, not content-based 2026-09-01 19:54:54 -06:00
build_svcomp_gold.py bench: extend fold_eval scoring with labeled negative cases 2026-08-16 16:43:51 -06:00
compare_findings.py feat: converge quality and scan interfaces 2026-07-29 00:57:33 -04:00
run_ab_evaluation.py test(quality): strengthen evaluation evidence 2026-07-29 20:47:04 -04:00
run_binary_goldens.py checkpoint 2026-06-04 20:20:29 -04:00
run_clang_sa_differential.py feat: harden Fold quality guard 2026-07-13 15:05:35 -04:00
run_code_quality.py test(quality): strengthen evaluation evidence 2026-07-29 20:47:04 -04:00
run_hybrid_goldens.py checkpoint 2026-06-04 20:20:29 -04:00
run_optimization_harness.py feat: converge quality and scan interfaces 2026-07-29 00:57:33 -04:00
run_organic_pairs.py feat: harden Fold quality guard 2026-07-13 15:05:35 -04:00
run_project_eval.py bench: wire chromium + sage corpora as stress/diversity manifests 2026-09-01 19:07:46 -06:00
run_reachability_evaluation.py feat(analysis): harden project-wide scans 2026-07-30 18:18:04 -04:00
run_rust_swift_goldens.py feat: converge quality and scan interfaces 2026-07-29 00:57:33 -04:00
run_sanitizer_differential.py feat: harden Fold quality guard 2026-07-13 15:05:35 -04:00
run_source_goldens.py feat: harden Fold quality guard 2026-07-13 15:05:35 -04:00
run_source_metamorphic.py test(quality): strengthen evaluation evidence 2026-07-29 20:47:04 -04:00
runner_common.py feat: converge quality and scan interfaces 2026-07-29 00:57:33 -04:00
summarize_maturity.py checkpoint 2026-06-04 20:20:29 -04:00