# Project Evaluation Scaffolding — Decision Record

> Date: 2026-07-27
> Status: accepted (implementation in progress)
> Literature grounding: `bench/literature_baselines.json` group
> `profiling-eval-2026-07` (paperbridge pass of 2026-07-27).

## Decision

Fold's "evaluate my own codebase" workflow becomes a **manifest-driven,
lane-generic runner** instead of a set of per-lane ad-hoc scripts:

```text
project manifest (versioned JSON)
    -> scripts/benchmarks/run_project_eval.py
        -> lane adapters: source | rust | binary
        -> immutable RunArtifact records (fold_eval.schema)
        -> finding fingerprint stability (fold_eval.stability)
        -> optional gold scoring (fold_eval.scoring)
        -> optional disposition compare (fold_eval.dispositions)
        -> optional baseline perf compare (fold_eval.performance)
    -> summary.json / summary.md
```

`run_code_quality.py` remains the pinned Yams source runner and is not
refactored; the new runner reuses the same `fold_eval` modules so metrics,
matching, and gate semantics cannot drift between the two.

## Why a manifest

The 2026-07-13 evaluation-foundation review established that runners must not
invent matching, metrics, or gate policy. The missing piece was a versioned
declaration of *what to run*: today the Yams invocation lives in a doc code
block, the Rust invocation in `bench/run_rust_oss.py` flags, and there is no
binary-lane project runner at all. A manifest:

- pins corpus, lane, detector scope, budgets, and fingerprints
  (`mordahl-2023-configurable-sast-benchmarking`,
  `braga-2017-crypto-sast-benchmark`);
- gives gates a stable input to diff against (`johns-2011-scanstud`);
- makes Rust and binary lanes first-class with the same disposition-review
  workflow the source lane already has (`bae-2021-rudra`,
  `cui-2023-safedrop`).

## Lane parity contract

Every lane emits the same envelope: per-run SARIF, profile JSON, logs, and a
hash-addressed RunArtifact. The C++ backend work that makes this true:

- the Rust and binary router routes previously returned a minimal
  `ScanProfile` (no `detectors[]`, no Z3 counters, no phase fields); they now
  populate per-detector wall time, findings emitted, functions analyzed, and a
  combined `frontend_extraction_ms` phase covering helper execution plus
  frontend ingest;
- Rust frontend/IR-encoder and clone counts are emitted as C++ shard profile
  rows rather than scraped from stdout;
- binary shards export heuristic, BinaryIR, and P-Code wall-time rows; the
  P-Code row records solver query/sat/unsat/unknown outcomes.

Profile JSON changes are additive-only; consumers pinned to the older schema
(`score_fp.py`, `compare_literature.py`) keep working.

## Performance gates are statistical

Single-median comparisons are not a regression oracle
(`hewson-2015-perf-regression-oracles`). Project runs carry repetition
samples; baseline comparison uses `fold_eval.metrics.paired_comparison` when
paired samples exist and is explicitly labelled median-only otherwise.

## Comparator policy

Inter-tool agreement is low (`lenarduzzi-2021-six-sast`), so comparator lanes
(Clang SA, sanitizers, other SAST) remain differential evidence. They never
become ground truth for project evaluation; ground truth is gold files plus
reviewed dispositions.

## Rust gold oracles

Rust bug classes differ from C (`zhuang-2025-dont-panic`). Rust manifests
score against Rust-native expectations (unsafe API surfaces, unchecked index,
panic-side-effect sites, taint sinks) rather than transplanted C CWE slices.
Juliet remains the C/C++ synthetic oracle only.

## Manifest schema v1

```json
{
  "schema_version": 1,
  "project": "yams",
  "lane": "source | rust | binary",
  "corpus": {
    "path": "../yams",
    "compile_db": "../yams/build/release/compile_commands.json",
    "binaries": ["../yams/build/release/yamsd"],
    "missing_corpus": "fail | skip"
  },
  "analysis": {
    "detectors": ["complexity", "dead-statement"],
    "scopes": ["quality"],
    "path_regex": "/src/metadata/",
    "exclude_regex": "",
    "max_tus": 12,
    "processes": 4,
    "budget_seconds": 0,
    "per_query_ms": 0,
    "extra_args": []
  },
  "evaluation": {
    "repeats": 3,
    "gold": "tests/benchmarks/gold/<project>.json",
    "dispositions": "tests/benchmarks/<project>/dispositions.json",
    "baseline": "tests/benchmarks/projects/baselines/<project>.json",
    "wall_regression_pct": 10.0,
    "rss_regression_pct": 10.0,
    "z3_timeout_budget": 0
  }
}
```

Checked-in manifests live under `tests/benchmarks/projects/`; reviewed
baselines under `tests/benchmarks/projects/baselines/`.