a72016c546
Round 3, iteration 1 of the fold optimization loop (docs/research/optimization-loop-2026-07-30.md). The foundation audit found gold scoring machinery (26 oracle files in bench/gold/, fold_eval/scoring.py, gold_threshold_failures gates) fully implemented but wired into zero project manifests — the eval measured finding drift, never precision/recall. New gold standards anchored on the reviewed dispositions: - bench/gold/fold_self_gold.json: 9 cases (5 SafetyObserver stub clones, 1 detector-internal clone, 1 loop_inventory clone, 2 sql-injection sites reviewed false-positive-but-intentionally- reported) - bench/gold/yams_self_gold.json: 7 cases (3 clone, 4 high-complexity) Both manifests now enforce gold_thresholds precision 1.0 / recall 1.0: any detector drift that loses a reviewed finding (recall drop) or emits a new unmatched finding (precision drop) fails the evaluation. Proof runs: fold TP=9/FP=0/FN=0, yams TP=7/FP=0/FN=0, precision 1.0, recall 1.0, fingerprints exactly preserved (0fec78e5.../247523c5...), full suite 137/137. Gold files live in bench/gold/ alongside the existing oracles (the tests/benchmarks schema-test glob correctly does not sweep that directory).
51 lines
1.9 KiB
JSON
51 lines
1.9 KiB
JSON
{
|
|
"name": "fold-self",
|
|
"description": "Gold standard for the fold self-scan manifest (tests/benchmarks/projects/fold_source.json). Anchored on the reviewed dispositions (tests/benchmarks/fold/source_dispositions.json, re-baselined 2026-07-31): 5 SafetyObserver virtual-stub clones, 1 detector-internal clone in clone.cpp, 1 in loop_inventory.cpp, and 2 sql-injection reports at src/index/store.cpp:22/:251 (reviewed false-positives that the detector intentionally reports; see dispositions rationale). One gold case per emitted finding group; the eval's one-to-one matching plus precision/recall 1.0 thresholds make any drift (new, lost, or re-classed findings) fail the gate.",
|
|
"cases": [
|
|
{
|
|
"id": "gold-fold-clone-clone-cpp",
|
|
"kind": "clone",
|
|
"files": ["src/detect/clone.cpp"]
|
|
},
|
|
{
|
|
"id": "gold-fold-clone-loop-inventory",
|
|
"kind": "clone",
|
|
"files": ["src/symbolic/loop_inventory.cpp"]
|
|
},
|
|
{
|
|
"id": "gold-fold-clone-safety-observer-1",
|
|
"kind": "clone",
|
|
"files": ["include/fold/symbolic/safety_observer.hpp"]
|
|
},
|
|
{
|
|
"id": "gold-fold-clone-safety-observer-2",
|
|
"kind": "clone",
|
|
"files": ["include/fold/symbolic/safety_observer.hpp"]
|
|
},
|
|
{
|
|
"id": "gold-fold-clone-safety-observer-3",
|
|
"kind": "clone",
|
|
"files": ["include/fold/symbolic/safety_observer.hpp"]
|
|
},
|
|
{
|
|
"id": "gold-fold-clone-safety-observer-4",
|
|
"kind": "clone",
|
|
"files": ["include/fold/symbolic/safety_observer.hpp"]
|
|
},
|
|
{
|
|
"id": "gold-fold-clone-safety-observer-5",
|
|
"kind": "clone",
|
|
"files": ["include/fold/symbolic/safety_observer.hpp"]
|
|
},
|
|
{
|
|
"id": "gold-fold-sqlinjection-store-prepare",
|
|
"kind": "sql-injection",
|
|
"files": ["src/index/store.cpp"]
|
|
},
|
|
{
|
|
"id": "gold-fold-sqlinjection-store-execute",
|
|
"kind": "sql-injection",
|
|
"files": ["src/index/store.cpp"]
|
|
}
|
|
]
|
|
}
|