Commit graph

7 commits

Author SHA1 Message Date
trvon
af66058f3a
bench: wire chromium + sage corpora as stress/diversity manifests
Some checks failed
build / build-linux (push) Failing after 7m32s
build / build-macos (push) Has been cancelled
build / release (push) Has been cancelled
Backlog #5 (operator-confirmed). chromium_source.json and
sage_rust.json were validated but unwired. This adds:

- reviewed dispositions (5 parallel review subagents): chromium 11
  (10 FP/1 accepted-risk — deliberate modular arithmetic in
  base/file_path.cc); sage 2714 (9 actionable uncontained path-join
  reads in skill install/daemon library/model-downloader; 890 FP;
  1815 accepted-risk quality bulk)
- wall/RSS baselines for both corpora

Measured both corpora produce finding drift run-to-run (chromium Z3
fingerprint drifts; sage +5 findings) and chromium's wall swings up
to ~2x under load — so they are wired as STRESS/DIVERSITY corpora:
a new additive evaluation.max_new_findings override lets a manifest
declare its finding-drift tolerance (CLI default unchanged for
fold/yams), allow_finding_drift is on, and chromium's wall/RSS
thresholds are relaxed. Final: both evals pass.

Also fixes a pre-existing /tmp-literal default (now platform tempdir)
in run_project_eval.py.
2026-09-01 19:07:46 -06:00
trvon
e30764fb19
bench: extend fold_eval scoring with labeled negative cases
Add negative_cases_from_universe to expand an explicit file-by-rule universe
into labeled negatives (enabling TP/FP/TN/FN, not just TP/FP), plus project-eval
harness updates in run_project_eval and the score/score_fp/quality-gate helpers.
2026-08-16 16:43:51 -06:00
trvon
bd29aab697
test(quality): strengthen evaluation evidence 2026-07-29 20:47:04 -04:00
trvon
b0b51f1e71
feat: converge quality and scan interfaces
- fail closed on MCP input and compile database policy
- add shared scan catalogs, finding queries, and tiered gates
2026-07-29 00:57:33 -04:00
trvon
ab262b2220
feat(quality): enforce evaluator policies 2026-07-28 14:09:12 -04:00
trvon
f37222d4fb
fix(scan): harden frontend analysis 2026-07-28 11:57:38 -04:00
trvon
f0ece0794a
feat(eval): add manifest project runner 2026-07-27 23:18:03 -04:00