fold/docs/research/benchmark-next-increments-audit-2026-06-03.md

# Benchmark Next Increments Audit

> Date: 2026-06-03
> Scope: what it takes to add hybrid goldens, Rust/Swift goldens,
> performance/RSS budget gates, and portable binary fixtures to Fold's maturity
> benchmark harness.

## Implementation status

This audit has been implemented in the current working tree:

- portable binary fixtures and `build_source` binary specs;
- merge-only hybrid golden runner;
- Rust/Swift frontend golden runner with structured skips;
- wall/CPU/RSS measurement and optional loose budget gates;
- umbrella summary wiring for all implemented suites.

## Current baseline

Implemented benchmark pieces:

- `scripts/benchmarks/compare_findings.py`
  - validates report shape;
  - checks physical locations;
  - checks backend provenance;
  - supports `must_report`, `must_not_report`, `min_total`, `max_total`, and
    `max_by_rule`.
- `scripts/benchmarks/run_binary_goldens.py`
  - runs `fold scan --mode binary --binary ...`;
  - supports optional host-specific smokes via `skip_if_missing`;
  - supports generated fixtures via `build_source`.
- `scripts/benchmarks/run_source_goldens.py`
  - generates temporary `compile_commands.json`;
  - runs `fold scan --mode source`.
- `scripts/benchmarks/run_hybrid_goldens.py`
  - generates source compile DBs, resolves/builds binary fixtures, and runs
    merge-only hybrid scans.
- `scripts/benchmarks/run_rust_swift_goldens.py`
  - runs Rust/Swift frontend scans with structured missing-tool skips.
- `scripts/benchmarks/runner_common.py`
  - records wall/CPU/RSS metrics and enforces optional budget gates.
- `scripts/benchmarks/summarize_maturity.py`
  - one-command JSON umbrella over all implemented suites.
- Existing shipped specs include binary, source, hybrid, Rust, and Swift goldens.

The comparator is intentionally dependency-free and has a SARIF/JSON report view
only; runners handle scan invocation.

## 1. Hybrid golden runner

### What it needs

A new runner should mirror `run_source_goldens.py` plus `run_binary_goldens.py`:

- spec directory: `tests/benchmarks/hybrid/`;
- runner: `scripts/benchmarks/run_hybrid_goldens.py`;
- build or resolve a source compile DB;
- resolve a binary path;
- invoke:

```sh
fold scan --mode hybrid \
  --compile-db <tmp compile_commands.json> \
  --binary <fixture binary> \
  --source-detectors <...> \
  --binary-detectors <...> \
  --hybrid-strategy merge-only \
  --format json --output <report>
```

Initial target should be merge-only. Hint-exchange strategies remain future work.

### Fixture strategy

Use a tiny source fixture and matching tiny compiled binary fixture. The first
hybrid golden should not require exact source/binary correlation yet; it should
prove that both backends can contribute findings and provenance survives merge.

Candidate first fixture:

- source side: `tests/fixtures/security/div_const_zero.cpp` / `div_guarded.cpp`;
- binary side: existing `tests/fixtures/binary/bof_*` or a new compiled small
  binary with a hardcoded string / simple null-deref heuristic;
- expected:
  - at least one `backend=source` finding;
  - at least one `backend=binary` or `backend=binary-*` finding;
  - guarded source lookalike must not report;
  - no locationless findings.

### Comparator gap

`compare_findings.py` can already match backend and rule IDs, but it cannot yet
assert aggregate backend presence such as `require_backends: ["source", "binary"]`.
Options:

1. encode with `must_report` entries that match known source and binary rules;
2. extend comparator with `expect.require_backends`.

Recommendation: add `require_backends` because hybrid provenance is the point of
this suite.

### Risks

- Exact binary findings may vary by architecture/compiler.
- Existing `bof_arm64`/`bof_x64` are committed binaries, but a generated fixture
  would be more portable if built during the runner.
- Hybrid returns 1 if either backend emits actionable findings; runner already
  accepts 0/1 patterns from source/binary runners.

### Acceptance criteria

- `scripts/benchmarks/run_hybrid_goldens.py --build build/release` runs at least
  one spec.
- `summarize_maturity.py --all` includes `hybrid_goldens`.
- First spec proves source + binary backend provenance in one report.
- Spec fails closed on missing required inputs unless explicitly marked skippable.

## 2. Rust/Swift golden runner

### What it needs

A new language-frontend runner should cover `fold scan --mode rust` and
`fold scan --mode swift` without assuming both toolchains exist everywhere.

Proposed runner:

```text
scripts/benchmarks/run_frontend_goldens.py
```

or split runners:

```text
scripts/benchmarks/run_rust_swift_goldens.py
```

Spec directories:

```text
tests/benchmarks/rust/
tests/benchmarks/swift/
```

Spec fields:

- `mode`: `rust` or `swift`;
- `paths`: source file or crate/package directory;
- `detectors`;
- `env`: optional environment overrides such as `FOLD_RUST_BIN`;
- `skip_if_missing_tools`: true by default for Rust/Swift until CI images are
  pinned.

### Existing fixture surface

Rust:

- `tests/fixtures/rust/security_crate/` exists;
- integration tests already locate `fold-rust` via `FOLD_RUST_BIN` or
  `tools/fold-rust/target/release/fold-rust`;
- detectors include `rust-unsafe-api`, `rust-unchecked-index`,
  `rust-path-traversal`, `rust-command-injection`, and `rust-crypto-misuse`.

Swift:

- `tests/fixtures/swift/Package.swift` and `Sources/fuzz_me.swift` exist;
- `fold-swift` is optional through Meson `-Dswift-frontend=true/false`;
- Swift mode is already routed in `scan_router` and forces serial execution.

### Comparator gaps

`compare_findings.py` is enough for initial Rust/Swift findings if specs use
`ruleId`, `backend`, and message regexes.

Possible extension later:

- `require_confidence`;
- `require_correlationKind`;
- per-frontend degradation counters if profile/report properties expose them.

### Risks

- Rust frontend depends on nightly/toolchain and `fold-rust` build output.
- Swift frontend depends on `swiftc` and platform availability.
- These suites should skip when tools are missing, but a skipped suite should be
  visible in the maturity summary.

### Acceptance criteria

- Runner handles missing frontend binaries/toolchains as structured skips, not
  false passes or hard crashes.
- At least one Rust spec checks a must-report unsafe API and a safe/missing rule
  shape.
- At least one Swift spec checks a stable finding if current Swift detectors emit
  one; otherwise the first Swift spec should be a report-shape/no-crash golden
  with explicit `max_total` and provenance/location requirements for any emitted
  findings.
- `summarize_maturity.py --all` includes Rust/Swift suite status.

## 3. Performance/RSS budget gates

### What it needs

Current runners record wall-clock seconds only. A maturity budget gate needs
structured resource measurement:

- wall time;
- CPU time when available;
- max RSS;
- optional Fold `--profile` JSON fields.

On macOS, `/usr/bin/time -l` gives maximum resident set size. On Linux,
`/usr/bin/time -v` gives maximum resident set size. Python `resource.getrusage`
can capture child maxrss but units differ by platform and it may aggregate over
children. Recommendation: implement a small cross-platform Python measurement
wrapper using `resource.getrusage(RUSAGE_CHILDREN)` plus optional
platform-specific `/usr/bin/time` fallback only if needed.

### Spec model

Add optional budget fields under `expect`:

```json
"budgets": {
  "max_wall_seconds": 10.0,
  "max_cpu_seconds": 20.0,
  "max_rss_mb": 512,
  "profile_max_total_wall_ms": 10000
}
```

Budget gates should start loose and informational for local work. CI can tighten
specific small fixtures once stable.

### Integration options

1. Embed measurement in each runner.
2. Create shared helper module used by all benchmark runners.

Recommendation: shared helper to avoid drift:

```text
scripts/benchmarks/runner_common.py
```

Potential contents:

- `run_fold_measured(cmd, cwd)`;
- `load_json`, `resolve_path`, `find_fold_binary`;
- resource result schema.

### Risks

- RSS is noisy across Apple Silicon/Linux and linked-r2 vs CLI-r2 modes.
- Cold caches make first run slower.
- Profile mode forces serial execution for source/Rust/Swift paths, so profile
  timing should not be compared directly against non-profile timing unless the
  spec opts in.

### Acceptance criteria

- Benchmark summaries include `wall_seconds`, `cpu_seconds`, and `rss_mb` when
  available.
- Specs can set loose budgets and fail closed when exceeded.
- Initial budgets are applied only to small, deterministic source/binary
  fixtures; host `/usr/bin/curl` smoke remains informational or skip-budgeted.
- Docs explain platform noise and how to update budgets.

## 4. More portable binary fixtures

### What it needs

The current real-binary curl smoke is useful but host-specific. Portable binary
fixtures should be compiled from checked-in C/C++ source during setup or runner
execution, or committed in controlled small forms for multiple architectures.

Recommended layout:

```text
tests/benchmarks/binary/fixtures/
  guarded_division.c
  fortified_wrappers.c
  scanner_vs_writer.c
  hardcoded_secret.c
  build_binary_fixtures.py
```

Generated outputs should live under the benchmark output directory, not be
checked in unless there is a strong cross-toolchain reason.

### First fixture set

1. `guarded_division.c`
   - contains actual machine division but guarded denominator;
   - expected: no `fold.div-by-zero`.

2. `scanner_vs_writer.c`
   - contains read-only scanners: `memchr`, `strstr`, `strchr`;
   - contains controlled writer: `memcpy` or `strncpy` in safe shape;
   - expected: no P-Code OOB false positives from scanner/import-stub adjacency.

3. `fortified_wrappers.c`
   - compiled with `_FORTIFY_SOURCE=2` where supported;
   - includes `__*_chk`-lowered patterns;
   - expected: fortified wrappers are not normalized to unchecked sinks.

4. `hardcoded_secret.c`
   - stable positive control for binary string/credential finding;
   - expected: at least one `fold.hardcoded-credential` backend binary finding.

### Runner support

Extend `run_binary_goldens.py` or add helper support for specs with
`build_source`:

```json
{
  "name": "binary_guarded_division",
  "build_source": "fixtures/guarded_division.c",
  "compiler": "cc",
  "cflags": ["-O1"],
  "detectors": "security",
  "expect": { ... }
}
```

If compiler is missing, fail closed for portable fixtures; unlike curl, these are
core benchmark assets.

### Risks

- `_FORTIFY_SOURCE` behavior varies by libc/platform; on macOS it may not lower
  exactly like glibc.
- Division instruction selection varies under optimization.
- r2/lifter behavior can vary by architecture.

Mitigations:

- Use separate specs for portable generic fixtures and platform-specific
  fortified behavior.
- Include relaxed exact-count expectations; prefer `must_not_report` for known
  FP signatures and stable positive controls for one finding.
- Use `--binary-max-functions` small caps only when function ordering is stable;
  otherwise scan all tiny fixture functions.

### Acceptance criteria

- At least two generated binary fixtures are built and scanned in the binary
  golden suite.
- One fixture is a stable positive control.
- One fixture locks a P-Code false-positive shape independent of `/usr/bin/curl`.
- Curl smoke remains optional and complementary.

## Recommended sequencing

1. Portable binary fixtures first: they also feed hybrid and perf suites.
2. Hybrid runner using one portable binary fixture plus existing source fixtures.
3. Performance/RSS budget infrastructure around existing source/binary runners.
4. Rust/Swift runner after deciding skip policy and stable Swift expectations.

This sequencing gives hybrid/perf deterministic inputs before adding toolchain-
optional frontend complexity.