Project Evaluation Scaffolding — Decision Record

Date: 2026-07-27 Status: accepted (implementation in progress) Literature grounding: bench/literature_baselines.json group profiling-eval-2026-07 (paperbridge pass of 2026-07-27).

Decision

Fold's "evaluate my own codebase" workflow becomes a manifest-driven, lane-generic runner instead of a set of per-lane ad-hoc scripts:

project manifest (versioned JSON)
    -> scripts/benchmarks/run_project_eval.py
        -> lane adapters: source | rust | binary
        -> immutable RunArtifact records (fold_eval.schema)
        -> finding fingerprint stability (fold_eval.stability)
        -> optional gold scoring (fold_eval.scoring)
        -> optional disposition compare (fold_eval.dispositions)
        -> optional baseline perf compare (fold_eval.performance)
    -> summary.json / summary.md

run_code_quality.py remains the pinned Yams source runner and is not refactored; the new runner reuses the same fold_eval modules so metrics, matching, and gate semantics cannot drift between the two.

Why a manifest

The 2026-07-13 evaluation-foundation review established that runners must not invent matching, metrics, or gate policy. The missing piece was a versioned declaration of what to run: today the Yams invocation lives in a doc code block, the Rust invocation in bench/run_rust_oss.py flags, and there is no binary-lane project runner at all. A manifest:

pins corpus, lane, detector scope, budgets, and fingerprints (mordahl-2023-configurable-sast-benchmarking, braga-2017-crypto-sast-benchmark);
gives gates a stable input to diff against (johns-2011-scanstud);
makes Rust and binary lanes first-class with the same disposition-review workflow the source lane already has (bae-2021-rudra, cui-2023-safedrop).

Lane parity contract

Every lane emits the same envelope: per-run SARIF, profile JSON, logs, and a hash-addressed RunArtifact. The C++ backend work that makes this true:

the Rust and binary router routes previously returned a minimal ScanProfile (no detectors[], no Z3 counters, no phase fields); they now populate per-detector wall time, findings emitted, functions analyzed, and a combined frontend_extraction_ms phase covering helper execution plus frontend ingest;
Rust frontend/IR-encoder and clone counts are emitted as C++ shard profile rows rather than scraped from stdout;
binary shards export heuristic, BinaryIR, and P-Code wall-time rows; the P-Code row records solver query/sat/unsat/unknown outcomes.

Profile JSON changes are additive-only; consumers pinned to the older schema (score_fp.py, compare_literature.py) keep working.

Performance gates are statistical

Single-median comparisons are not a regression oracle (hewson-2015-perf-regression-oracles). Project runs carry repetition samples; baseline comparison uses fold_eval.metrics.paired_comparison when paired samples exist and is explicitly labelled median-only otherwise.

Comparator policy

Inter-tool agreement is low (lenarduzzi-2021-six-sast), so comparator lanes (Clang SA, sanitizers, other SAST) remain differential evidence. They never become ground truth for project evaluation; ground truth is gold files plus reviewed dispositions.

Rust gold oracles

Rust bug classes differ from C (zhuang-2025-dont-panic). Rust manifests score against Rust-native expectations (unsafe API surfaces, unchecked index, panic-side-effect sites, taint sinks) rather than transplanted C CWE slices. Juliet remains the C/C++ synthetic oracle only.

Manifest schema v1

{
  "schema_version": 1,
  "project": "yams",
  "lane": "source | rust | binary",
  "corpus": {
    "path": "../yams",
    "compile_db": "../yams/build/release/compile_commands.json",
    "binaries": ["../yams/build/release/yamsd"],
    "missing_corpus": "fail | skip"
  },
  "analysis": {
    "detectors": ["complexity", "dead-statement"],
    "scopes": ["quality"],
    "path_regex": "/src/metadata/",
    "exclude_regex": "",
    "max_tus": 12,
    "processes": 4,
    "budget_seconds": 0,
    "per_query_ms": 0,
    "extra_args": []
  },
  "evaluation": {
    "repeats": 3,
    "gold": "tests/benchmarks/gold/<project>.json",
    "dispositions": "tests/benchmarks/<project>/dispositions.json",
    "baseline": "tests/benchmarks/projects/baselines/<project>.json",
    "wall_regression_pct": 10.0,
    "rss_regression_pct": 10.0,
    "z3_timeout_budget": 0
  }
}

Checked-in manifests live under tests/benchmarks/projects/; reviewed baselines under tests/benchmarks/projects/baselines/.

4.5 KiB Raw Permalink Blame History