fold/docs/research/benchmark-next-increments-audit-2026-06-03.md

Benchmark Next Increments Audit

Date: 2026-06-03 Scope: what it takes to add hybrid goldens, Rust/Swift goldens, performance/RSS budget gates, and portable binary fixtures to Fold's maturity benchmark harness.

Implementation status

This audit has been implemented in the current working tree:

• portable binary fixtures and build_source binary specs;
• merge-only hybrid golden runner;
• Rust/Swift frontend golden runner with structured skips;
• wall/CPU/RSS measurement and optional loose budget gates;
• umbrella summary wiring for all implemented suites.

Current baseline

Implemented benchmark pieces:

• scripts/benchmarks/compare_findings.py
  • validates report shape;
  • checks physical locations;
  • checks backend provenance;
  • supports must_report, must_not_report, min_total, max_total, and max_by_rule.
• scripts/benchmarks/run_binary_goldens.py
  • runs fold scan --mode binary --binary ...;
  • supports optional host-specific smokes via skip_if_missing;
  • supports generated fixtures via build_source.
• scripts/benchmarks/run_source_goldens.py
  • generates temporary compile_commands.json;
  • runs fold scan --mode source.
• scripts/benchmarks/run_hybrid_goldens.py
  • generates source compile DBs, resolves/builds binary fixtures, and runs merge-only hybrid scans.
• scripts/benchmarks/run_rust_swift_goldens.py
  • runs Rust/Swift frontend scans with structured missing-tool skips.
• scripts/benchmarks/runner_common.py
  • records wall/CPU/RSS metrics and enforces optional budget gates.
• scripts/benchmarks/summarize_maturity.py
  • one-command JSON umbrella over all implemented suites.
• Existing shipped specs include binary, source, hybrid, Rust, and Swift goldens.

The comparator is intentionally dependency-free and has a SARIF/JSON report view only; runners handle scan invocation.

1. Hybrid golden runner

What it needs

A new runner should mirror run_source_goldens.py plus run_binary_goldens.py:

• spec directory: tests/benchmarks/hybrid/;
• runner: scripts/benchmarks/run_hybrid_goldens.py;
• build or resolve a source compile DB;
• resolve a binary path;
• invoke:

fold scan --mode hybrid \
  --compile-db <tmp compile_commands.json> \
  --binary <fixture binary> \
  --source-detectors <...> \
  --binary-detectors <...> \
  --hybrid-strategy merge-only \
  --format json --output <report>

Initial target should be merge-only. Hint-exchange strategies remain future work.

Fixture strategy

Use a tiny source fixture and matching tiny compiled binary fixture. The first hybrid golden should not require exact source/binary correlation yet; it should prove that both backends can contribute findings and provenance survives merge.

Candidate first fixture:

• source side: tests/fixtures/security/div_const_zero.cpp / div_guarded.cpp;
• binary side: existing tests/fixtures/binary/bof_* or a new compiled small binary with a hardcoded string / simple null-deref heuristic;
• expected:
  • at least one backend=source finding;
  • at least one backend=binary or backend=binary-* finding;
  • guarded source lookalike must not report;
  • no locationless findings.

Comparator gap

compare_findings.py can already match backend and rule IDs, but it cannot yet assert aggregate backend presence such as require_backends: ["source", "binary"]. Options:

1. encode with must_report entries that match known source and binary rules;
2. extend comparator with expect.require_backends.

Recommendation: add require_backends because hybrid provenance is the point of this suite.

Risks

• Exact binary findings may vary by architecture/compiler.
• Existing bof_arm64/bof_x64 are committed binaries, but a generated fixture would be more portable if built during the runner.
• Hybrid returns 1 if either backend emits actionable findings; runner already accepts 0/1 patterns from source/binary runners.

Acceptance criteria

• scripts/benchmarks/run_hybrid_goldens.py --build build/release runs at least one spec.
• summarize_maturity.py --all includes hybrid_goldens.
• First spec proves source + binary backend provenance in one report.
• Spec fails closed on missing required inputs unless explicitly marked skippable.

2. Rust/Swift golden runner

What it needs

A new language-frontend runner should cover fold scan --mode rust and fold scan --mode swift without assuming both toolchains exist everywhere.

Proposed runner:

scripts/benchmarks/run_frontend_goldens.py

or split runners:

scripts/benchmarks/run_rust_swift_goldens.py

Spec directories:

tests/benchmarks/rust/
tests/benchmarks/swift/

Spec fields:

• mode: rust or swift;
• paths: source file or crate/package directory;
• detectors;
• env: optional environment overrides such as FOLD_RUST_BIN;
• skip_if_missing_tools: true by default for Rust/Swift until CI images are pinned.

Existing fixture surface

Rust:

• tests/fixtures/rust/security_crate/ exists;
• integration tests already locate fold-rust via FOLD_RUST_BIN or tools/fold-rust/target/release/fold-rust;
• detectors include rust-unsafe-api, rust-unchecked-index, rust-path-traversal, rust-command-injection, and rust-crypto-misuse.

Swift:

• tests/fixtures/swift/Package.swift and Sources/fuzz_me.swift exist;
• fold-swift is optional through Meson -Dswift-frontend=true/false;
• Swift mode is already routed in scan_router and forces serial execution.

Comparator gaps

compare_findings.py is enough for initial Rust/Swift findings if specs use ruleId, backend, and message regexes.

Possible extension later:

• require_confidence;
• require_correlationKind;
• per-frontend degradation counters if profile/report properties expose them.

Risks

• Rust frontend depends on nightly/toolchain and fold-rust build output.
• Swift frontend depends on swiftc and platform availability.
• These suites should skip when tools are missing, but a skipped suite should be visible in the maturity summary.

Acceptance criteria

• Runner handles missing frontend binaries/toolchains as structured skips, not false passes or hard crashes.
• At least one Rust spec checks a must-report unsafe API and a safe/missing rule shape.
• At least one Swift spec checks a stable finding if current Swift detectors emit one; otherwise the first Swift spec should be a report-shape/no-crash golden with explicit max_total and provenance/location requirements for any emitted findings.
• summarize_maturity.py --all includes Rust/Swift suite status.

3. Performance/RSS budget gates

What it needs

Current runners record wall-clock seconds only. A maturity budget gate needs structured resource measurement:

• wall time;
• CPU time when available;
• max RSS;
• optional Fold --profile JSON fields.

On macOS, /usr/bin/time -l gives maximum resident set size. On Linux, /usr/bin/time -v gives maximum resident set size. Python resource.getrusage can capture child maxrss but units differ by platform and it may aggregate over children. Recommendation: implement a small cross-platform Python measurement wrapper using resource.getrusage(RUSAGE_CHILDREN) plus optional platform-specific /usr/bin/time fallback only if needed.

Spec model

Add optional budget fields under expect:

"budgets": {
  "max_wall_seconds": 10.0,
  "max_cpu_seconds": 20.0,
  "max_rss_mb": 512,
  "profile_max_total_wall_ms": 10000
}

Budget gates should start loose and informational for local work. CI can tighten specific small fixtures once stable.

Integration options

1. Embed measurement in each runner.
2. Create shared helper module used by all benchmark runners.

Recommendation: shared helper to avoid drift:

scripts/benchmarks/runner_common.py

Potential contents:

• run_fold_measured(cmd, cwd);
• load_json, resolve_path, find_fold_binary;
• resource result schema.

Risks

• RSS is noisy across Apple Silicon/Linux and linked-r2 vs CLI-r2 modes.
• Cold caches make first run slower.
• Profile mode forces serial execution for source/Rust/Swift paths, so profile timing should not be compared directly against non-profile timing unless the spec opts in.

Acceptance criteria

• Benchmark summaries include wall_seconds, cpu_seconds, and rss_mb when available.
• Specs can set loose budgets and fail closed when exceeded.
• Initial budgets are applied only to small, deterministic source/binary fixtures; host /usr/bin/curl smoke remains informational or skip-budgeted.
• Docs explain platform noise and how to update budgets.

4. More portable binary fixtures

What it needs

The current real-binary curl smoke is useful but host-specific. Portable binary fixtures should be compiled from checked-in C/C++ source during setup or runner execution, or committed in controlled small forms for multiple architectures.

Recommended layout:

tests/benchmarks/binary/fixtures/
  guarded_division.c
  fortified_wrappers.c
  scanner_vs_writer.c
  hardcoded_secret.c
  build_binary_fixtures.py

Generated outputs should live under the benchmark output directory, not be checked in unless there is a strong cross-toolchain reason.

First fixture set

1. guarded_division.c

   • contains actual machine division but guarded denominator;
   • expected: no fold.div-by-zero.

2. scanner_vs_writer.c

   • contains read-only scanners: memchr, strstr, strchr;
   • contains controlled writer: memcpy or strncpy in safe shape;
   • expected: no P-Code OOB false positives from scanner/import-stub adjacency.

3. fortified_wrappers.c

   • compiled with _FORTIFY_SOURCE=2 where supported;
   • includes __*_chk-lowered patterns;
   • expected: fortified wrappers are not normalized to unchecked sinks.

4. hardcoded_secret.c

   • stable positive control for binary string/credential finding;
   • expected: at least one fold.hardcoded-credential backend binary finding.

Runner support

Extend run_binary_goldens.py or add helper support for specs with build_source:

{
  "name": "binary_guarded_division",
  "build_source": "fixtures/guarded_division.c",
  "compiler": "cc",
  "cflags": ["-O1"],
  "detectors": "security",
  "expect": { ... }
}

If compiler is missing, fail closed for portable fixtures; unlike curl, these are core benchmark assets.

Risks

• _FORTIFY_SOURCE behavior varies by libc/platform; on macOS it may not lower exactly like glibc.
• Division instruction selection varies under optimization.
• r2/lifter behavior can vary by architecture.

Mitigations:

• Use separate specs for portable generic fixtures and platform-specific fortified behavior.
• Include relaxed exact-count expectations; prefer must_not_report for known FP signatures and stable positive controls for one finding.
• Use --binary-max-functions small caps only when function ordering is stable; otherwise scan all tiny fixture functions.

Acceptance criteria

• At least two generated binary fixtures are built and scanned in the binary golden suite.
• One fixture is a stable positive control.
• One fixture locks a P-Code false-positive shape independent of /usr/bin/curl.
• Curl smoke remains optional and complementary.

Recommended sequencing

1. Portable binary fixtures first: they also feed hybrid and perf suites.
2. Hybrid runner using one portable binary fixture plus existing source fixtures.
3. Performance/RSS budget infrastructure around existing source/binary runners.
4. Rust/Swift runner after deciding skip policy and stable Swift expectations.

This sequencing gives hybrid/perf deterministic inputs before adding toolchain- optional frontend complexity.