121 lines
4.5 KiB
Markdown
121 lines
4.5 KiB
Markdown
# Project Evaluation Scaffolding — Decision Record
|
|
|
|
> Date: 2026-07-27
|
|
> Status: accepted (implementation in progress)
|
|
> Literature grounding: `bench/literature_baselines.json` group
|
|
> `profiling-eval-2026-07` (paperbridge pass of 2026-07-27).
|
|
|
|
## Decision
|
|
|
|
Fold's "evaluate my own codebase" workflow becomes a **manifest-driven,
|
|
lane-generic runner** instead of a set of per-lane ad-hoc scripts:
|
|
|
|
```text
|
|
project manifest (versioned JSON)
|
|
-> scripts/benchmarks/run_project_eval.py
|
|
-> lane adapters: source | rust | binary
|
|
-> immutable RunArtifact records (fold_eval.schema)
|
|
-> finding fingerprint stability (fold_eval.stability)
|
|
-> optional gold scoring (fold_eval.scoring)
|
|
-> optional disposition compare (fold_eval.dispositions)
|
|
-> optional baseline perf compare (fold_eval.performance)
|
|
-> summary.json / summary.md
|
|
```
|
|
|
|
`run_code_quality.py` remains the pinned Yams source runner and is not
|
|
refactored; the new runner reuses the same `fold_eval` modules so metrics,
|
|
matching, and gate semantics cannot drift between the two.
|
|
|
|
## Why a manifest
|
|
|
|
The 2026-07-13 evaluation-foundation review established that runners must not
|
|
invent matching, metrics, or gate policy. The missing piece was a versioned
|
|
declaration of *what to run*: today the Yams invocation lives in a doc code
|
|
block, the Rust invocation in `bench/run_rust_oss.py` flags, and there is no
|
|
binary-lane project runner at all. A manifest:
|
|
|
|
- pins corpus, lane, detector scope, budgets, and fingerprints
|
|
(`mordahl-2023-configurable-sast-benchmarking`,
|
|
`braga-2017-crypto-sast-benchmark`);
|
|
- gives gates a stable input to diff against (`johns-2011-scanstud`);
|
|
- makes Rust and binary lanes first-class with the same disposition-review
|
|
workflow the source lane already has (`bae-2021-rudra`,
|
|
`cui-2023-safedrop`).
|
|
|
|
## Lane parity contract
|
|
|
|
Every lane emits the same envelope: per-run SARIF, profile JSON, logs, and a
|
|
hash-addressed RunArtifact. The C++ backend work that makes this true:
|
|
|
|
- the Rust and binary router routes previously returned a minimal
|
|
`ScanProfile` (no `detectors[]`, no Z3 counters, no phase fields); they now
|
|
populate per-detector wall time, findings emitted, functions analyzed, and a
|
|
combined `frontend_extraction_ms` phase covering helper execution plus
|
|
frontend ingest;
|
|
- Rust frontend/IR-encoder and clone counts are emitted as C++ shard profile
|
|
rows rather than scraped from stdout;
|
|
- binary shards export heuristic, BinaryIR, and P-Code wall-time rows; the
|
|
P-Code row records solver query/sat/unsat/unknown outcomes.
|
|
|
|
Profile JSON changes are additive-only; consumers pinned to the older schema
|
|
(`score_fp.py`, `compare_literature.py`) keep working.
|
|
|
|
## Performance gates are statistical
|
|
|
|
Single-median comparisons are not a regression oracle
|
|
(`hewson-2015-perf-regression-oracles`). Project runs carry repetition
|
|
samples; baseline comparison uses `fold_eval.metrics.paired_comparison` when
|
|
paired samples exist and is explicitly labelled median-only otherwise.
|
|
|
|
## Comparator policy
|
|
|
|
Inter-tool agreement is low (`lenarduzzi-2021-six-sast`), so comparator lanes
|
|
(Clang SA, sanitizers, other SAST) remain differential evidence. They never
|
|
become ground truth for project evaluation; ground truth is gold files plus
|
|
reviewed dispositions.
|
|
|
|
## Rust gold oracles
|
|
|
|
Rust bug classes differ from C (`zhuang-2025-dont-panic`). Rust manifests
|
|
score against Rust-native expectations (unsafe API surfaces, unchecked index,
|
|
panic-side-effect sites, taint sinks) rather than transplanted C CWE slices.
|
|
Juliet remains the C/C++ synthetic oracle only.
|
|
|
|
## Manifest schema v1
|
|
|
|
```json
|
|
{
|
|
"schema_version": 1,
|
|
"project": "yams",
|
|
"lane": "source | rust | binary",
|
|
"corpus": {
|
|
"path": "../yams",
|
|
"compile_db": "../yams/build/release/compile_commands.json",
|
|
"binaries": ["../yams/build/release/yamsd"],
|
|
"missing_corpus": "fail | skip"
|
|
},
|
|
"analysis": {
|
|
"detectors": ["complexity", "dead-statement"],
|
|
"scopes": ["quality"],
|
|
"path_regex": "/src/metadata/",
|
|
"exclude_regex": "",
|
|
"max_tus": 12,
|
|
"processes": 4,
|
|
"budget_seconds": 0,
|
|
"per_query_ms": 0,
|
|
"extra_args": []
|
|
},
|
|
"evaluation": {
|
|
"repeats": 3,
|
|
"gold": "tests/benchmarks/gold/<project>.json",
|
|
"dispositions": "tests/benchmarks/<project>/dispositions.json",
|
|
"baseline": "tests/benchmarks/projects/baselines/<project>.json",
|
|
"wall_regression_pct": 10.0,
|
|
"rss_regression_pct": 10.0,
|
|
"z3_timeout_budget": 0
|
|
}
|
|
}
|
|
```
|
|
|
|
Checked-in manifests live under `tests/benchmarks/projects/`; reviewed
|
|
baselines under `tests/benchmarks/projects/baselines/`.
|