fold/tests/benchmarks/README.md

# Fold Benchmark Goldens

This directory contains benchmark specs used by `scripts/benchmarks/*`.

## Schema and artifacts

Committed manifests use `schema_version: 1` and declare a stable `case_id`,
`case_type`, `origin`, `backend`, and oracle data appropriate to that case type.
The canonical case types are positive, negative, vulnerable/fixed,
metamorphic, differential, dynamic-confirmation, and real-project advisory.
Differential cases name at least two comparators and state whether agreement is
expected. `scripts/benchmarks/fold_eval/schema.py` validates these contracts;
legacy input remains an explicit compatibility boundary rather than the write
format for new cases.

Runners write immutable `RunArtifact` JSON records containing the experiment
and case identity, exact command, timestamps, exit status, environment/tool
facts, measurements, and SHA-256-addressed inputs and outputs. Logs and raw
SARIF remain beside the record. Scoring consumes those artifacts; runners do
not invent tool-specific definitions of precision, recall, or finding identity.

## Spec checks

Specs can require:

- physical locations on every finding;
- backend provenance on every finding;
- aggregate backend presence via `require_backends`;
- maximum counts by rule;
- findings that must be present;
- findings that must not be present;
- optional loose resource budgets under `expect.budgets`:
  - `max_wall_seconds`
  - `max_cpu_seconds`
  - `max_rss_mb`

When `max_wall_seconds` is set, runners also apply a subprocess timeout with
slack so hung scans fail the benchmark instead of hanging CI forever.

## Binary goldens

`binary/*.json` specs run `fold scan --mode binary` and compare the output with
`compare_findings.py`.

Comparison rules may match `ruleId`, `backend`, `messageRegex`, `uriRegex`,
and `functionRegex`. Multiple fields on one rule are conjunctive. The URI and
function matchers are useful for findings whose human message deliberately
does not repeat its source location.

The binary suite includes:

- `binary/curl_pcode_fp_smoke.json`, an optional real-binary smoke for the P-Code
  false positives found during linked-r2 testing. It is `skip_if_missing` because
  `/usr/bin/curl` is host-specific.
- generated portable C fixtures built from `binary/fixtures/*.c` for guarded
  division, scanner-vs-writer P-Code false-positive pressure, and a hardcoded
  secret positive control. The guarded case prevents high P-Code branch
  restructuring from overriding raw instruction evidence. The secret case also
  requires broad `security` to emit one finding, not duplicate the artifact via
  both binary-security and the explicit hardcoded-string classifier.

Specs can point at a prebuilt binary with `binary` or request an on-the-fly build
with `build_source`, `compiler`, and `cflags`.

## Source goldens

`source/*.json` specs generate a temporary `compile_commands.json`, run
`fold scan --mode source`, and compare the output with `compare_findings.py`.

The source goldens include `source/div_zero_guarded.json`, which requires a
constant-zero division finding while rejecting the guarded divisor lookalike,
and `source/code_quality_structural.json`, which pins structural debt and clean
near misses.

## Source metamorphic benchmark

`metamorphic/division_metamorphic.json` applies alpha-renaming plus harmless
statement insertion, an equivalent early-return guard rewrite, translation-unit
ordering, and process-count changes. The runner compares duplicate-sensitive
canonical finding multisets after the declared source mapping. It also rejects
missing location/backend provenance and prevents local results from being
silently promoted to summary-backed or cross-TU-corroborated evidence.

Run it with:

```sh
scripts/benchmarks/run_source_metamorphic.py --build build/release
```

## Guarded production reachability

`run_reachability_evaluation.py` validates the whole-program graph lane with
positive, negative, cross-TU callback, test-only, default-inactive,
all-configuration-infeasible, unknown-guard, unresolved-dispatch, metamorphic
rename, and binary-symbol corroboration cases. Its explicit v1 variant manifest
bounds the fixture build universe; the production classifier does not infer
`compile-variant-absent` from a single compile database.

The runner requires at least five cold/warm/profile-change pairs and retains
commands, logs, native reports, profiles, and descriptive median/IQR summaries.
The permanent Meson test checks all 15 fixture cells plus cold, warm, and
profile-change invalidation counts. By default each invocation uses a unique
system temporary directory, so concurrent runs and same-second retries cannot
share artifacts; an explicit `--artifacts` path must be new or empty. Fixture
compilation databases use argument arrays, preserving paths containing spaces.
It sets no performance threshold. Optional Fold and YAMS compile databases run
the same paired protocol over a deterministic 24-TU production `src/` slice
(the established calibration TUs first, then `main`, then canonical order).
Use `--project-max-tus 0` for an exhaustive
run; large exhaustive attempts are retained even when stopped or timed out.
Binary symbol presence is corroboration; absence is recorded but never treated
as source unreachability proof.

```sh
scripts/benchmarks/run_reachability_evaluation.py \
  --fold build/release/src/cli/fold --repeats 5 \
  --fold-compile-db build/release/compile_commands.json \
  --yams-compile-db ../yams/build/release/compile_commands.json
```

## Sanitizer differential benchmarks

`differential/sanitizer_pairs.json` compares Fold source findings with real
UBSan execution for vulnerable/fixed division and signed-overflow pairs. The
vulnerable program must fail with the expected sanitizer class twice; the fixed
program and Fold result must remain clean. Results retain separate
reached/triggered/detected/replayed states and report per-property plus micro
and macro classification metrics. An unavailable compiler/sanitizer is a
structured Meson skip (exit 77), not a fabricated pass.

`differential/asan_pairs.json` uses the same runner for stack OOB and heap UAF
pairs. The ASan workload pins `-O0` because optimizing a deliberately constant
invalid access can remove the runtime oracle before instrumentation. Fixed
programs are classified only by explicit sanitizer diagnostics; an ordinary
non-zero application exit is not treated as a crash.

Run it with:

```sh
scripts/benchmarks/run_sanitizer_differential.py --build build/release
scripts/benchmarks/run_sanitizer_differential.py --build build/release \
  --spec tests/benchmarks/differential/asan_pairs.json
```

## Clang Static Analyzer differential benchmark

`differential/clang_sa_pairs.json` compares Fold with Clang SA `core` checkers
on vulnerable/fixed division and null-dereference programs. Each comparator
retains its own command, log, timing, and classification; disagreements fail
the controlled suite but remain inspectable artifacts.

```sh
scripts/benchmarks/run_clang_sa_differential.py --build build/release
```

## Organic vulnerable/fixed benchmark

`organic/cve_pairs.json` pins adapted semantic slices of GStreamer
CVE-2024-47835, libssh2 CVE-2019-17498, and OpenSSL CVE-2015-0288. Every entry
records repository and patch references, vulnerable/fixed revisions, upstream
path, license, adaptation statement, and hashes of both local fixtures. The
fixtures are explicitly dependency-minimized adaptations, not pristine project
snapshots. Results are reported separately from synthetic metrics. The current
reviewed baseline catches libssh2, reports no fixed cases, and retains the two
null-contract misses as named known gaps.

```sh
scripts/benchmarks/run_organic_pairs.py --build build/release
```

## Hybrid goldens

`hybrid/*.json` specs generate a source compile DB, resolve or build a binary
fixture, run `fold scan --mode hybrid --hybrid-strategy merge-only`, and compare
the merged report.

The first hybrid golden is `hybrid/source_div_binary_secret.json`, which requires
both source and binary backend provenance in one report.

## Rust/Swift frontend goldens

`rust/*.json` and `swift/*.json` specs run `fold scan --mode rust` or
`fold scan --mode swift`. Missing frontend tools are structured skips when
`skip_if_missing_tools` is true.

The first Rust golden checks unsafe API findings in the Rust security crate. The
first Swift golden is a no-crash/report-shape guard for the current Swift fixture,
whose user-facing findings are filtered to zero.

Run:

```sh
scripts/benchmarks/run_binary_goldens.py --build build/release
scripts/benchmarks/run_hybrid_goldens.py --build build/release
scripts/benchmarks/run_rust_swift_goldens.py --build build/release
scripts/benchmarks/run_source_goldens.py --build build/release
scripts/benchmarks/run_source_metamorphic.py --build build/release
scripts/benchmarks/run_sanitizer_differential.py --build build/release
scripts/benchmarks/run_clang_sa_differential.py --build build/release
scripts/benchmarks/run_organic_pairs.py --build build/release
scripts/benchmarks/run_code_quality.py \
  --compile-db ../yams/build/release/compile_commands.json \
  --project-root ../yams --path-regex '/src/metadata/' --max-tus 12 \
  --dispositions tests/benchmarks/yams/metadata_dispositions.json
scripts/benchmarks/summarize_maturity.py --build build/release --all
```

`run_code_quality.py` is the real-project structural-quality performance and
reliability surface. It creates a deterministic filtered compile database,
runs repeated profiled-serial and parallel scans, and fails on crashes,
timeouts, missing provenance/location data, Z3 timeout budget violations, or
finding-fingerprint drift. Its schema also reports set intersection, union,
churn, and pairwise Jaccard so partial drift is diagnosable. Pass a prior
`summary.json` with `--baseline` to also gate wall-time regressions.
Current summaries retain raw wall-time repetitions and use paired comparison,
effect, and an exact sign test when the baseline has matching samples. Older
median-only baselines remain readable but are clearly labeled as an unpaired
fallback. On macOS, the portable external timer does not provide RSS; the Yams
lane uses Fold's internal `peak_rss_kb` profile instead of inventing a value.
The reviewed disposition baseline adds stable-ID change-time reporting:
new/persisting/resolved findings, counts by disposition and rule, optional
changed-file scoping, review coverage, and current/new alert density per KLOC.
With dispositions enabled, new unreviewed findings fail by default; use
`--max-new-findings` only for an explicit rollout allowance. `fold quality`
defaults to `projects/fold_source.json`, an eight-TU calibrated Fold self-policy
with reviewed findings in `fold/source_dispositions.json`. The slice covers
production frontend/subprocess lifetime handling, visitor callbacks, clone
identity/ranking, credential matching, caller-preconditioned SQL taint, project
indexing, and symbolic observer hooks. It runs the explicitly reviewed
`clone`, `unused-function`, `uninit-read`, `hardcoded-credential`,
`use-after-move`, and `sql-injection` detector set. It requires exact repeat
stability, complete locations/backend provenance, zero solver timeouts, and
zero new unreviewed finding IDs. This is a change-time and operational-coverage
gate, not a performance claim: no wall-time, memory, finding-count, or token
threshold was inferred from the calibration run. Expensive control-flow and
OOB/division proof behavior remains gated by focused positive/negative and
metamorphic tests rather than making the default project policy operationally
unbounded. Yams remains the separate external benchmark in
`projects/yams_source.json`.

To review a Fold baseline update, run the manifest into a fresh output
directory, inspect every report and `summary.json`, record a rationale for every
finding in `fold/source_dispositions.json`, then update its revision and target
fingerprints. Never copy new IDs into the baseline merely to make the gate pass.

Density uses the explicit changed-file KLOC when `--changed-files` is supplied;
otherwise it uses the union of selected TUs and finding-bearing project files,
so header findings are not divided only by main-file size.

For reviewed optimization claims, use the immutable paired A/B harness rather
than comparing ad hoc terminal timings:

```sh
scripts/benchmarks/run_ab_evaluation.py \
  --baseline-fold /absolute/path/to/baseline/fold \
  --current-fold build/release/src/cli/fold \
  --out-dir /tmp/fold-ab-$(date +%Y%m%d-%H%M%S) \
  --repeats 3 --fanout-processes 2
```

The harness defaults to both Fold and YAMS manifests, alternates baseline/current
order by repetition, retains exact commands, binary and manifest hashes, SARIF,
profiles, logs, and run artifacts, and refuses a non-empty output directory.
Serial cold/warm cells measure fresh versus reused Fold project-index state.
Incremental scans intentionally remain serial; fanout warm measurements therefore
use a separately retained and excluded preflight process and are labeled as such
instead of pretending the host page cache was portably flushed. Baseline/current
and within-variant finding fingerprints must agree unless an explicit reviewed
`--allow-finding-drift` run is requested. The YAMS manifest excludes absolute
`third_party`, `subprojects`, and generated paths plus `proto_serializer.cpp`,
whose generated protobuf header is not available in the release tree; these are
harness coverage exclusions, not analyzer findings.

Project-evaluation run artifacts and summaries classify operational coverage
separately from findings: missing generated dependencies, per-TU worker
timeouts, worker crashes, and ordinary frontend failures are retained with
bounded log evidence and fail the run even if Fold continued and returned 0/1.
Source profiles also expose selected, excluded, attempted, completed, timed-out,
and frontend-failed TU counters. Project-relative `--exclude-glob` patterns are
matched against absolute compile-database paths at path-component boundaries;
for example, `third_party/**` excludes any nested `third_party` tree without
requiring an artificial leading `**/`.

Benchmark runners share evaluator semantics through
`scripts/benchmarks/fold_eval/`: schema/artifact validation, classification
metrics, micro/macro aggregation, paired comparison, one-to-one gold matching,
canonical SARIF identities, metamorphic multiset comparison, confirmation
states, and repeated-run stability belong there, not in individual runners.
`meson test -C build/release fold:benchmark_evaluation` pins those semantics;
`fold:source_metamorphic`, `fold:sanitizer_differential`,
`fold:asan_differential`, `fold:clang_sa_differential`, and
`fold:organic_vulnerable_fixed` pin the executable oracles.

Production file-size budgets live in `tests/architecture/source_budgets.json`.
Run `python3 scripts/check_source_structure.py --json` to see the ranked
decomposition surface. The tracked-exception list is currently empty. The
`source_structure` Meson test rejects new production C++ files above the
1,000-line default and would prevent any future temporary exception from
growing. The checked-in architecture policy also protects the approved
subprocess-boundary list. Compiled behavior tests protect report backend
provenance and canonical frontend NDJSON keys without brittle source substrings.

Project manifests can set aggregate `gold_thresholds`, per-rule
`per_rule_gold_thresholds`, `stability` budgets (`max_churn` and
`min_pairwise_jaccard`), explicitly exempt `experimental_rules`, and absolute
resource `budgets` (`max_wall_seconds`, `max_peak_rss_mb`,
`max_frontend_extraction_ms`, and `max_output_bytes`).

The rationale and complete v2 evaluation protocol are in
`docs/research/evaluation-foundation-literature-review-2026-07-13.md`.

To run the release/checkpoint maturity gate with explicit baselines, provide the
baseline directory and require every selected suite to have a valid JSON baseline:

```sh
scripts/benchmarks/summarize_maturity.py \
  --build build/release --all \
  --require-baselines \
  --baseline-dir scripts/benchmarks/baselines
```

In required-baseline mode, missing or malformed baseline files fail closed before
that suite runs. Optional frontend tools may still report structured skips from
their suite runners; baseline shape problems are never treated as skips.