a72016c546
Round 3, iteration 1 of the fold optimization loop (docs/research/optimization-loop-2026-07-30.md). The foundation audit found gold scoring machinery (26 oracle files in bench/gold/, fold_eval/scoring.py, gold_threshold_failures gates) fully implemented but wired into zero project manifests — the eval measured finding drift, never precision/recall. New gold standards anchored on the reviewed dispositions: - bench/gold/fold_self_gold.json: 9 cases (5 SafetyObserver stub clones, 1 detector-internal clone, 1 loop_inventory clone, 2 sql-injection sites reviewed false-positive-but-intentionally- reported) - bench/gold/yams_self_gold.json: 7 cases (3 clone, 4 high-complexity) Both manifests now enforce gold_thresholds precision 1.0 / recall 1.0: any detector drift that loses a reviewed finding (recall drop) or emits a new unmatched finding (precision drop) fails the evaluation. Proof runs: fold TP=9/FP=0/FN=0, yams TP=7/FP=0/FN=0, precision 1.0, recall 1.0, fingerprints exactly preserved (0fec78e5.../247523c5...), full suite 137/137. Gold files live in bench/gold/ alongside the existing oracles (the tests/benchmarks schema-test glob correctly does not sweep that directory).
41 lines
1.6 KiB
JSON
41 lines
1.6 KiB
JSON
{
|
|
"name": "yams-self",
|
|
"description": "Gold standard for the yams self-scan manifest (tests/benchmarks/projects/yams_source.json). Anchored on the reviewed dispositions (tests/benchmarks/yams/metadata_dispositions.json, re-baselined 2026-07-31): 3 clone findings (MetadataValueTypeUtils helpers x2 in document_metadata.h, connection-pool trace helpers x1 in connection_pool.cpp) and 4 high-complexity findings (types.h, metadata_repository.h, connection_pool.cpp x2). One gold case per emitted finding group; precision/recall 1.0 thresholds make any drift fail the gate.",
|
|
"cases": [
|
|
{
|
|
"id": "gold-yams-clone-document-metadata-tostring",
|
|
"kind": "clone",
|
|
"files": ["include/yams/metadata/document_metadata.h"]
|
|
},
|
|
{
|
|
"id": "gold-yams-clone-document-metadata-fromstring",
|
|
"kind": "clone",
|
|
"files": ["include/yams/metadata/document_metadata.h"]
|
|
},
|
|
{
|
|
"id": "gold-yams-clone-connection-pool",
|
|
"kind": "clone",
|
|
"files": ["src/metadata/connection_pool.cpp"]
|
|
},
|
|
{
|
|
"id": "gold-yams-high-complexity-types",
|
|
"kind": "high-complexity",
|
|
"files": ["include/yams/core/types.h"]
|
|
},
|
|
{
|
|
"id": "gold-yams-high-complexity-metadata-repository",
|
|
"kind": "high-complexity",
|
|
"files": ["include/yams/metadata/metadata_repository.h"]
|
|
},
|
|
{
|
|
"id": "gold-yams-high-complexity-connection-pool-1",
|
|
"kind": "high-complexity",
|
|
"files": ["src/metadata/connection_pool.cpp"]
|
|
},
|
|
{
|
|
"id": "gold-yams-high-complexity-connection-pool-2",
|
|
"kind": "high-complexity",
|
|
"files": ["src/metadata/connection_pool.cpp"]
|
|
}
|
|
]
|
|
}
|