fold/tests/benchmarks

Fold Benchmark Goldens

This directory contains benchmark specs used by scripts/benchmarks/*.

Schema and artifacts

Committed manifests use schema_version: 1 and declare a stable case_id, case_type, origin, backend, and oracle data appropriate to that case type. The canonical case types are positive, negative, vulnerable/fixed, metamorphic, differential, dynamic-confirmation, and real-project advisory. Differential cases name at least two comparators and state whether agreement is expected. scripts/benchmarks/fold_eval/schema.py validates these contracts; legacy input remains an explicit compatibility boundary rather than the write format for new cases.

Runners write immutable RunArtifact JSON records containing the experiment and case identity, exact command, timestamps, exit status, environment/tool facts, measurements, and SHA-256-addressed inputs and outputs. Logs and raw SARIF remain beside the record. Scoring consumes those artifacts; runners do not invent tool-specific definitions of precision, recall, or finding identity.

Spec checks

Specs can require:

• physical locations on every finding;
• backend provenance on every finding;
• aggregate backend presence via require_backends;
• maximum counts by rule;
• findings that must be present;
• findings that must not be present;
• optional loose resource budgets under expect.budgets:
  • max_wall_seconds
  • max_cpu_seconds
  • max_rss_mb

When max_wall_seconds is set, runners also apply a subprocess timeout with slack so hung scans fail the benchmark instead of hanging CI forever.

Binary goldens

binary/*.json specs run fold scan --mode binary and compare the output with compare_findings.py.

Comparison rules may match ruleId, backend, messageRegex, uriRegex, and functionRegex. Multiple fields on one rule are conjunctive. The URI and function matchers are useful for findings whose human message deliberately does not repeat its source location.

The binary suite includes:

• binary/curl_pcode_fp_smoke.json, an optional real-binary smoke for the P-Code false positives found during linked-r2 testing. It is skip_if_missing because /usr/bin/curl is host-specific.
• generated portable C fixtures built from binary/fixtures/*.c for guarded division, scanner-vs-writer P-Code false-positive pressure, and a hardcoded secret positive control. The guarded case prevents high P-Code branch restructuring from overriding raw instruction evidence. The secret case also requires broad security to emit one finding, not duplicate the artifact via both binary-security and the explicit hardcoded-string classifier.

Specs can point at a prebuilt binary with binary or request an on-the-fly build with build_source, compiler, and cflags.

Source goldens

source/*.json specs generate a temporary compile_commands.json, run fold scan --mode source, and compare the output with compare_findings.py.

The source goldens include source/div_zero_guarded.json, which requires a constant-zero division finding while rejecting the guarded divisor lookalike, and source/code_quality_structural.json, which pins structural debt and clean near misses.

Source metamorphic benchmark

metamorphic/division_metamorphic.json applies alpha-renaming plus harmless statement insertion, an equivalent early-return guard rewrite, translation-unit ordering, and process-count changes. The runner compares duplicate-sensitive canonical finding multisets after the declared source mapping. It also rejects missing location/backend provenance and prevents local results from being silently promoted to summary-backed or cross-TU-corroborated evidence.

Run it with:

scripts/benchmarks/run_source_metamorphic.py --build build/release

Guarded production reachability

run_reachability_evaluation.py validates the whole-program graph lane with positive, negative, cross-TU callback, test-only, default-inactive, all-configuration-infeasible, unknown-guard, unresolved-dispatch, metamorphic rename, and binary-symbol corroboration cases. Its explicit v1 variant manifest bounds the fixture build universe; the production classifier does not infer compile-variant-absent from a single compile database.

The runner requires at least five cold/warm/profile-change pairs and retains commands, logs, native reports, profiles, and descriptive median/IQR summaries. The permanent Meson test checks all 15 fixture cells plus cold, warm, and profile-change invalidation counts. By default each invocation uses a unique system temporary directory, so concurrent runs and same-second retries cannot share artifacts; an explicit --artifacts path must be new or empty. Fixture compilation databases use argument arrays, preserving paths containing spaces. It sets no performance threshold. Optional Fold and YAMS compile databases run the same paired protocol over a deterministic 24-TU production src/ slice (the established calibration TUs first, then main, then canonical order). Use --project-max-tus 0 for an exhaustive run; large exhaustive attempts are retained even when stopped or timed out. Binary symbol presence is corroboration; absence is recorded but never treated as source unreachability proof.

scripts/benchmarks/run_reachability_evaluation.py \
  --fold build/release/src/cli/fold --repeats 5 \
  --fold-compile-db build/release/compile_commands.json \
  --yams-compile-db ../yams/build/release/compile_commands.json

Sanitizer differential benchmarks

differential/sanitizer_pairs.json compares Fold source findings with real UBSan execution for vulnerable/fixed division and signed-overflow pairs. The vulnerable program must fail with the expected sanitizer class twice; the fixed program and Fold result must remain clean. Results retain separate reached/triggered/detected/replayed states and report per-property plus micro and macro classification metrics. An unavailable compiler/sanitizer is a structured Meson skip (exit 77), not a fabricated pass.

differential/asan_pairs.json uses the same runner for stack OOB and heap UAF pairs. The ASan workload pins -O0 because optimizing a deliberately constant invalid access can remove the runtime oracle before instrumentation. Fixed programs are classified only by explicit sanitizer diagnostics; an ordinary non-zero application exit is not treated as a crash.

Run it with:

scripts/benchmarks/run_sanitizer_differential.py --build build/release
scripts/benchmarks/run_sanitizer_differential.py --build build/release \
  --spec tests/benchmarks/differential/asan_pairs.json

Clang Static Analyzer differential benchmark

differential/clang_sa_pairs.json compares Fold with Clang SA core checkers on vulnerable/fixed division and null-dereference programs. Each comparator retains its own command, log, timing, and classification; disagreements fail the controlled suite but remain inspectable artifacts.

scripts/benchmarks/run_clang_sa_differential.py --build build/release

Organic vulnerable/fixed benchmark

organic/cve_pairs.json pins adapted semantic slices of GStreamer CVE-2024-47835, libssh2 CVE-2019-17498, and OpenSSL CVE-2015-0288. Every entry records repository and patch references, vulnerable/fixed revisions, upstream path, license, adaptation statement, and hashes of both local fixtures. The fixtures are explicitly dependency-minimized adaptations, not pristine project snapshots. Results are reported separately from synthetic metrics. The current reviewed baseline catches libssh2, reports no fixed cases, and retains the two null-contract misses as named known gaps.

scripts/benchmarks/run_organic_pairs.py --build build/release

Hybrid goldens

hybrid/*.json specs generate a source compile DB, resolve or build a binary fixture, run fold scan --mode hybrid --hybrid-strategy merge-only, and compare the merged report.

The first hybrid golden is hybrid/source_div_binary_secret.json, which requires both source and binary backend provenance in one report.

Rust/Swift frontend goldens

rust/*.json and swift/*.json specs run fold scan --mode rust or fold scan --mode swift. Missing frontend tools are structured skips when skip_if_missing_tools is true.

The first Rust golden checks unsafe API findings in the Rust security crate. The first Swift golden is a no-crash/report-shape guard for the current Swift fixture, whose user-facing findings are filtered to zero.

Run:

scripts/benchmarks/run_binary_goldens.py --build build/release
scripts/benchmarks/run_hybrid_goldens.py --build build/release
scripts/benchmarks/run_rust_swift_goldens.py --build build/release
scripts/benchmarks/run_source_goldens.py --build build/release
scripts/benchmarks/run_source_metamorphic.py --build build/release
scripts/benchmarks/run_sanitizer_differential.py --build build/release
scripts/benchmarks/run_clang_sa_differential.py --build build/release
scripts/benchmarks/run_organic_pairs.py --build build/release
scripts/benchmarks/run_code_quality.py \
  --compile-db ../yams/build/release/compile_commands.json \
  --project-root ../yams --path-regex '/src/metadata/' --max-tus 12 \
  --dispositions tests/benchmarks/yams/metadata_dispositions.json
scripts/benchmarks/summarize_maturity.py --build build/release --all

run_code_quality.py is the real-project structural-quality performance and reliability surface. It creates a deterministic filtered compile database, runs repeated profiled-serial and parallel scans, and fails on crashes, timeouts, missing provenance/location data, Z3 timeout budget violations, or finding-fingerprint drift. Its schema also reports set intersection, union, churn, and pairwise Jaccard so partial drift is diagnosable. Pass a prior summary.json with --baseline to also gate wall-time regressions. Current summaries retain raw wall-time repetitions and use paired comparison, effect, and an exact sign test when the baseline has matching samples. Older median-only baselines remain readable but are clearly labeled as an unpaired fallback. On macOS, the portable external timer does not provide RSS; the Yams lane uses Fold's internal peak_rss_kb profile instead of inventing a value. The reviewed disposition baseline adds stable-ID change-time reporting: new/persisting/resolved findings, counts by disposition and rule, optional changed-file scoping, review coverage, and current/new alert density per KLOC. With dispositions enabled, new unreviewed findings fail by default; use --max-new-findings only for an explicit rollout allowance. fold quality defaults to projects/fold_source.json, an eight-TU calibrated Fold self-policy with reviewed findings in fold/source_dispositions.json. The slice covers production frontend/subprocess lifetime handling, visitor callbacks, clone identity/ranking, credential matching, caller-preconditioned SQL taint, project indexing, and symbolic observer hooks. It runs the explicitly reviewed clone, unused-function, uninit-read, hardcoded-credential, use-after-move, and sql-injection detector set. It requires exact repeat stability, complete locations/backend provenance, zero solver timeouts, and zero new unreviewed finding IDs. This is a change-time and operational-coverage gate, not a performance claim: no wall-time, memory, finding-count, or token threshold was inferred from the calibration run. Expensive control-flow and OOB/division proof behavior remains gated by focused positive/negative and metamorphic tests rather than making the default project policy operationally unbounded. Yams remains the separate external benchmark in projects/yams_source.json.

To review a Fold baseline update, run the manifest into a fresh output directory, inspect every report and summary.json, record a rationale for every finding in fold/source_dispositions.json, then update its revision and target fingerprints. Never copy new IDs into the baseline merely to make the gate pass.

Density uses the explicit changed-file KLOC when --changed-files is supplied; otherwise it uses the union of selected TUs and finding-bearing project files, so header findings are not divided only by main-file size.

For reviewed optimization claims, use the immutable paired A/B harness rather than comparing ad hoc terminal timings:

scripts/benchmarks/run_ab_evaluation.py \
  --baseline-fold /absolute/path/to/baseline/fold \
  --current-fold build/release/src/cli/fold \
  --out-dir /tmp/fold-ab-$(date +%Y%m%d-%H%M%S) \
  --repeats 3 --fanout-processes 2

The harness defaults to both Fold and YAMS manifests, alternates baseline/current order by repetition, retains exact commands, binary and manifest hashes, SARIF, profiles, logs, and run artifacts, and refuses a non-empty output directory. Serial cold/warm cells measure fresh versus reused Fold project-index state. Incremental scans intentionally remain serial; fanout warm measurements therefore use a separately retained and excluded preflight process and are labeled as such instead of pretending the host page cache was portably flushed. Baseline/current and within-variant finding fingerprints must agree unless an explicit reviewed --allow-finding-drift run is requested. The YAMS manifest excludes absolute third_party, subprojects, and generated paths plus proto_serializer.cpp, whose generated protobuf header is not available in the release tree; these are harness coverage exclusions, not analyzer findings.

Project-evaluation run artifacts and summaries classify operational coverage separately from findings: missing generated dependencies, per-TU worker timeouts, worker crashes, and ordinary frontend failures are retained with bounded log evidence and fail the run even if Fold continued and returned 0/1. Source profiles also expose selected, excluded, attempted, completed, timed-out, and frontend-failed TU counters. Project-relative --exclude-glob patterns are matched against absolute compile-database paths at path-component boundaries; for example, third_party/** excludes any nested third_party tree without requiring an artificial leading **/.

Benchmark runners share evaluator semantics through scripts/benchmarks/fold_eval/: schema/artifact validation, classification metrics, micro/macro aggregation, paired comparison, one-to-one gold matching, canonical SARIF identities, metamorphic multiset comparison, confirmation states, and repeated-run stability belong there, not in individual runners. meson test -C build/release fold:benchmark_evaluation pins those semantics; fold:source_metamorphic, fold:sanitizer_differential, fold:asan_differential, fold:clang_sa_differential, and fold:organic_vulnerable_fixed pin the executable oracles.

Production file-size budgets live in tests/architecture/source_budgets.json. Run python3 scripts/check_source_structure.py --json to see the ranked decomposition surface. The tracked-exception list is currently empty. The source_structure Meson test rejects new production C++ files above the 1,000-line default and would prevent any future temporary exception from growing. The checked-in architecture policy also protects the approved subprocess-boundary list. Compiled behavior tests protect report backend provenance and canonical frontend NDJSON keys without brittle source substrings.

Project manifests can set aggregate gold_thresholds, per-rule per_rule_gold_thresholds, stability budgets (max_churn and min_pairwise_jaccard), explicitly exempt experimental_rules, and absolute resource budgets (max_wall_seconds, max_peak_rss_mb, max_frontend_extraction_ms, and max_output_bytes).

The rationale and complete v2 evaluation protocol are in docs/research/evaluation-foundation-literature-review-2026-07-13.md.

To run the release/checkpoint maturity gate with explicit baselines, provide the baseline directory and require every selected suite to have a valid JSON baseline:

scripts/benchmarks/summarize_maturity.py \
  --build build/release --all \
  --require-baselines \
  --baseline-dir scripts/benchmarks/baselines

In required-baseline mode, missing or malformed baseline files fail closed before that suite runs. Optional frontend tools may still report structured skips from their suite runners; baseline shape problems are never treated as skips.