fold/bench/README.md

fold benchmark harness

Drives fold and comparator SAST tools across a corpus, writes raw findings per tool, scores them against a gold standard, and produces a precision / recall / F1 table you can paste into a PR or write up.

Layout

bench/
├── run_bench.py       # subprocess driver: corpus → per-tool JSON
├── run_summary_ab.py  # fold summary off/on benchmark with profile deltas
├── run_curl_ab.py     # curl source-vs-binary backend A/B harness
├── compare_literature.py # Fold runs → literature/ceiling report
├── literature_baselines.json # paperbridge-grounded external baselines
├── score.py           # JSON + gold → summary.md
├── corpora/           # corpus fetchers + cached source trees
│   ├── fold-fixtures/ # symlink → ../tests/fixtures/clones (zero-download)
│   ├── poj-104/       # fetch.sh stub; see "Adding POJ-104" below
│   └── sard-cwe-561/  # fetch.sh stub; see "Adding SARD" below
├── gold/              # gold-standard answer files
│   └── fold-fixtures.json
└── results/           # one dated subdirectory per run

Smoke run (no downloads)

cd bench
python3 run_bench.py \
    --corpus corpora/fold-fixtures \
    --name fold-fixtures \
    --tools fold,clang-tidy \
    --processes 1
python3 score.py \
    --run-dir results/<latest> \
    --gold gold/fold-fixtures.json

Optimization micro-benchmarks

Use run_micro_opt.py for synthetic before/after validation of fold speed and false-positive optimization patches. It generates focused C++ projects, runs fold scan --profile, and writes summary.json / summary.md plus raw SARIF and profile artifacts. See docs/internal/perf-audit/micro-benchmarks.md for the case matrix.

For a complete real-project or precision iteration, follow docs/architecture/optimization-loop.md. It pins the workload through existing manifests, requires profile attribution and repeated measurements, and keeps source/binary provenance and explicit precision labels separate.

python3 bench/run_micro_opt.py --case all --repeats 3
python3 bench/run_micro_opt.py --case branch-merge --size 200 --repeats 5

Quality gate

Use run_quality_gate.py to collect the focused micro-benchmarks plus correctness smokes into one manifest. It wraps run_micro_opt.py, run_summary_ab.py, run_bench.py + score.py, and optional curl/Juliet runs.

Project manifests

scripts/benchmarks/run_project_eval.py is the generic real-project runner for C++ source, Rust workspaces, and binaries. It uses versioned manifests under tests/benchmarks/projects/ and writes SARIF, profiles, immutable run artifacts, finding-stability summaries, and optional disposition/baseline comparisons.

python3 scripts/benchmarks/run_project_eval.py \
  --manifest tests/benchmarks/projects/yams_source.json \
  --out-dir /tmp/fold-yams-eval

python3 scripts/benchmarks/run_project_eval.py \
  --manifest tests/benchmarks/projects/sage_rust.json \
  --out-dir /tmp/fold-sage-eval

Chromium is an opt-in production-scale canary. The committed manifest selects eight deterministic translation units across base, net, services/network, and url; it does not attempt an unbounded monorepo scan. From a sibling ../chromium/src checkout, generate the compilation database using Chromium's official helper, then run the shared evaluator:

cd ../chromium/src
tools/clang/scripts/generate_compdb.py -p out/Default > compile_commands.json
cd ../../fold
python3 scripts/benchmarks/run_project_eval.py \
  --manifest tests/benchmarks/projects/chromium_source.json \
  --out-dir /tmp/fold-chromium-eval

Build the selected Chromium targets first when generated headers are absent. The manifest skips cleanly when no checkout exists, so Chromium remains a local/periodic benchmark rather than a required CI download.

Include a project run in the local quality gate only when its reviewed baseline is stable:

python3 bench/run_quality_gate.py --skip-curl --skip-juliet \
  --project-manifest tests/benchmarks/projects/yams_source.json

python3 bench/run_quality_gate.py --skip-curl --skip-juliet
python3 bench/run_quality_gate.py --juliet-cwes 476 369 --juliet-timeout 300

Summary A/B run

Use run_summary_ab.py to compare fold scan with summaries disabled against --build-summaries on the same compile database. By default it uses bench/corpora/interproc, runs null-deref, div-by-zero, oob, and int-overflow, enables profile output, and scores against <corpus>/gold.json when present.

python3 bench/run_summary_ab.py
python3 bench/run_summary_ab.py --repeats 3
python3 bench/run_summary_ab.py \
    --corpus bench/corpora/interproc \
    --detectors null-deref,div-by-zero,oob,int-overflow \
    --extra-arg=--trace-z3

Each run writes summary.json, summary.md, per-variant SARIF, profile JSON, and stdout/stderr logs under bench/results/<timestamp>_summary-ab/. The Markdown report includes finding-count deltas, Z3 query and detector wall-time deltas, exact added/removed finding keys, optional precision/recall/F1, and an optimization_signals payload that separates useful TP/recall gain from pure implementation overhead. Set LLVM_BIN=/path/to/llvm/bin if the default Homebrew LLVM path is not available.

curl Source-vs-Binary A/B

Use run_curl_ab.py to exercise the source backend and the radare2 binary backend on curl in one run. The harness writes source.sarif, binary.sarif, logs, summary.json, and summary.md under bench/results/<timestamp>_curl-ab/.

By default it looks for curl source at ../curl-source/curl, configures a temporary CMake build to obtain compile_commands.json, scans the source build, and compares it against /usr/bin/curl with the binary backend.

Fast smoke run against one source file and only a few binary functions:

python3 bench/run_curl_ab.py \
    --source-detectors clone \
    --source-max-files 8 \
    --source-path-regex '/src/tool_main\.c$' \
    --binary-max-functions 8 \
    --processes 1

Source security detectors are intentionally more expensive on production C code. For source-security A/Bs, start with a small file cap and explicit budgets:

python3 bench/run_curl_ab.py \
    --source-detectors null-deref,div-by-zero,int-overflow,oob \
    --source-max-files 2 \
    --source-path-regex '/(lib|src)/' \
    --source-budget-seconds 10 \
    --source-per-query-ms 250 \
    --source-per-tu-timeout 15 \
    --binary-max-functions 16

If curl is already configured, point directly at its compile database:

python3 bench/run_curl_ab.py \
    --compile-db /path/to/curl-build/compile_commands.json \
    --curl-binary /usr/bin/curl \
    --binary-max-functions 64

For a stricter same-source comparison, let the harness build curl and scan the freshly built binary instead of the Apple system binary:

python3 bench/run_curl_ab.py \
    --build-curl-binary \
    --source-max-files 0 \
    --binary-max-functions 0

Additional CMake options can be passed through with repeated --cmake-arg, for example --cmake-arg=-DBUILD_TESTING=OFF. The binary side uses fold's --input-type binary path; linked libr_core is used when fold was configured with radare2 support, otherwise the backend falls back to the r2 CLI.

For Juliet slices, use file-level scoring because each gold entry names a testcase file rather than an exact finding location. Juliet C++ files are kept on c++17 to match the existing Juliet runner.

python3 bench/corpora/juliet/filter_cwes.py
python3 bench/run_summary_ab.py \
    --name juliet-476-summary-ab \
    --corpus bench/corpora/juliet/filtered/476 \
    --gold bench/gold/juliet-476.json \
    --detectors null-deref \
    --score-mode file \
    --cxx-standard c++17
python3 bench/run_summary_ab.py \
    --name juliet-121-summary-ab \
    --corpus bench/corpora/juliet/filtered/121 \
    --gold bench/gold/juliet-121.json \
    --detectors oob \
    --score-mode file \
    --cxx-standard c++17 \
    --max-delta-details 25

Gold files are currently checked in for these Juliet slices:

121, 122, 124, 126, 127, 190, 191, 194, 195, 196, 197, 369, 476, 561

run_fold_on_juliet.py defaults to that full set and maps each CWE to the Fold detector/kind expected by its gold file:

python3 bench/run_fold_on_juliet.py --processes 1
python3 bench/run_fold_on_juliet.py --cwes 476 369 190 121 --processes 4

Long Juliet runs can be resumed without rerunning completed variants:

python3 bench/run_summary_ab.py \
    --out-dir bench/results/<timestamp>_juliet-190-summary-ab \
    --reuse-existing \
    --corpus bench/corpora/juliet/filtered/190 \
    --gold bench/gold/juliet-190.json \
    --detectors int-overflow \
    --score-mode file \
    --cxx-standard c++17

Literature and Ceiling Comparison

literature_baselines.json records paperbridge-resolved references and broad external baseline metrics for Juliet scanner comparisons, NIST SATE/SARD, SV-COMP, production-code SAST studies, CVEfixes, and DiverseVul. Use it for context and expansion planning, not as a direct head-to-head score unless a baseline row explicitly uses the same corpus, scoring unit, and detector scope.

compare_literature.py consumes Fold benchmark outputs and writes a Markdown and JSON report with:

• all scored Fold rows from run_summary_ab.py, run_fold_on_juliet.py, or comparator payloads,
• best Fold variant per CWE/kind,
• F1 ceilings for perfect precision, full recall at current FP count, alert-budget oracle triage, and perfect oracle scoring,
• optimization signals that rank summary variants by TP delta, FP delta, elapsed cost, and safe next action,
• literature baseline and benchmark-expansion tables.

python3 bench/compare_literature.py \
    --fold-run bench/results/2026-05-17T20-06-05Z_juliet-476-summary-ab/summary.json \
    --fold-run bench/results/2026-05-17T20-09-41Z_juliet-369-summary-ab/summary.json \
    --fold-run bench/results/2026-05-17T20-19-02Z_juliet-190-summary-ab/summary.json \
    --fold-run bench/results/2026-05-17T21-16-12Z_juliet-121-summary-ab/summary.json

Without --fold-run, it picks the latest *summary-ab/summary.json result per run name. Outputs land in bench/results/<timestamp>_literature-comparison/.

Adding a corpus

A corpus is just a directory of .cpp files. The runner generates a flat compile_commands.json over *.cpp recursively. For corpora that ship with their own build systems, supply a corpora/<name>/fetch.sh that downloads and (optionally) configures them.

POJ-104

bench/corpora/poj-104/fetch.sh is already wired up — it sparse-clones just the POJ-104 subtree from Microsoft's CodeXGLUE repo and renames the .txt files (CodeXGLUE's archive convention) to .cpp. The resulting tree is ~50,000 submissions (~700 MB of source), so plan disk budget accordingly.

bench/corpora/poj-104/fetch.sh
python3 bench/gold/_generate_poj104.py        # → bench/gold/poj-104.json
python3 bench/run_bench.py \
    --corpus bench/corpora/poj-104/dataset \
    --name poj-104 \
    --tools fold,clang-tidy \
    --detectors equiv \
    --processes 4
python3 bench/score.py \
    --run-dir bench/results/<latest> \
    --gold bench/gold/poj-104.json

Gold entries are one-per-algorithm-folder with the rule "at least one cross-pair in this cluster must be flagged equivalent" — generous to clustering detectors (fold) without requiring O(n²) redundant findings per cluster.

NIST SARD CWE-561 (Dead Code)

# corpora/sard-cwe-561/fetch.sh
curl -L https://samate.nist.gov/SARD/test-suite/...-CWE-561.zip -o sard.zip
unzip sard.zip -d sard-src

Gold standard: each annotated dead block becomes a {"kind": "dead", "files": ["..."], "id": "..."} entry.

Adding a tool

Drop a run_<tool>(corpus_dir, out_dir) -> dict helper into run_bench.py. The dict must have tool, version, elapsed_s, exit_code, and findings: list[{file, line, kind, id}]. Wire it into main()'s tool dispatch.

Currently supported:

• fold — uses --format sarif, parses results out of the SARIF doc.
• clang-tidy — runs with a focused -checks set covering redundancy and branch-clone diagnostics.

Planned: pmd-cpd (Java jar), codeql (DB build + query bundle).

Determinism

Two consecutive runs against the same corpus must produce byte-identical fold SARIF (see src/report/sarif.cpp::writeSarif sort step). The bench runner asserts this via diff in .github/workflows/bench.yml so an accidental nondeterminism regression breaks CI before it muddles the precision numbers.

Precision drift gate (the gate is local, not CI)

bench/fp-baseline.json is the committed per-detector precision/recall/F1 reference (regenerate it intentionally when a change is meant to move the numbers). The drift gate compares a fresh run to it and fails on regression — the enforcement that was previously missing (drift went unnoticed because nothing compared to a baseline).

Run the gate locally (≈2 min; deterministic):

python3 bench/score_fp.py --gate --run --security-bugs --juliet \
  --juliet-cwes 476 369 190 191 121 122 124 126 127 194 195 196 197 561 \
  --max-files-per-cwe 120 --fold-bin build/release/src/cli/fold
# exit 0 = no per-detector precision/recall/F1 drop beyond --tolerance (default 0.02)
# exit 1 = regression (prints which detector and the delta)

--max-files-per-cwe N deterministically stride-samples each Juliet CWE so huge CWEs (121/122 are ~8–10k files) stay tractable; the gold set is automatically restricted to the scanned subset so recall stays meaningful. The same flags must be used for the gate as for the committed baseline (they are, above).

Regenerate the committed baseline (only when the change should move numbers):

python3 bench/score_fp.py --run --security-bugs --juliet \
  --juliet-cwes 476 369 190 191 121 122 124 126 127 194 195 196 197 561 \
  --max-files-per-cwe 120 --summary --out bench/fp-baseline.json

This is a local gate (developer / pre-push), not a CI requirement.