# Brick benchmark loop

The runner is an artifact-first, bounded experiment loop inspired by YAMS
xplan and Fold's optimization harness.

    python3 tests/benchmarks/runner.py self-test
    python3 tests/benchmarks/runner.py run lexical_smoke
    python3 tests/benchmarks/runner.py run source_languages
    python3 tests/benchmarks/runner.py run rust_imports
    python3 tests/benchmarks/runner.py run p4_syntax
    python3 tests/benchmarks/runner.py run p4_metrics
    python3 tests/benchmarks/runner.py run p4c_evidence
    python3 tests/benchmarks/runner.py run tree_sitter_p4_pilot
    python3 tests/benchmarks/diff_runner.py run
    python3 tests/benchmarks/policy_runner.py run
    python3 tests/benchmarks/mcp_runner.py run
    python3 tests/benchmarks/runner.py run yams_architecture --corpus /path/to/yams

Each run writes a resolved plan, per-repeat graph/HTML artifacts, summary JSON,
and REPORT.md under build/benchmarks. Canonical full-model JSON (including
attributes, evidence, diagnostics, and metrics) must remain identical across
repeats. The seed corpus has explicit positive cycle and relationship
expectations; it is not representative enough for performance or language-wide
precision claims.

The `source_languages` plan scans authored C++, JS, TS, TSX, Dart, and Rust
positive/clean-negative fixtures three times. Its labels enumerate every symbol
(including owner) and dependency. Missing facts, unexpected facts, and duplicate
facts fail the gate; clean-negative files participate in the same comparison.
The summary includes exactness and precision/recall by language stratum, full
JSON stability, latency, RSS, and Git provenance. The quality gate runs this plan.

The `p4_syntax` plan runs three deterministic scans of labeled P4_16 positive
and clean-negative fixtures. It guards symbol/dependency coverage and a clean
cycle count; it is not a claim of P4 language-wide precision or p4c parity.
The `p4_metrics` plan asserts every nested P4 metric across parser-cycle,
unreachable-state, terminal/select, reuse, repeated-apply, and unresolved
fixtures. The `p4c_evidence` plan repeats the qualified v1.2.5.16 diagnostic import and
retains emitted JSON/HTML plus its dirty-tree manifest; stable identities here
cover the Imported finding and Flags edge as well as syntax structure.

The `tree_sitter_p4_pilot` plan builds the pinned optional feature once, then
runs the Brick binary directly three times. It compares labeled table-action
relationships for precision/recall and records elapsed time plus per-process
maximum RSS from `wait4`, alongside raw graphs and command logs. Its two
positive relationships and comment/string clean negatives are a seed check,
not a language-wide accuracy claim.

The `structural_diff` plan rebuilds positive and identical clean-negative
baseline/current fixtures at one canonical root, then runs `brick check` three
times per case. It retains complete models/diffs, exact count assertions,
latency, per-process maximum RSS, command logs, and provenance. It measures the
diff/check path, not scan or render time.

The `policy` plan rebuilds one labeled project and runs a violating and a
clean-negative repository-relative policy three times each. It asserts exit
status, exact match counts, evidence retention, deterministic result JSON,
latency, per-process maximum RSS, logs, and provenance.

The `mcp_queries` plan starts the read-only stdio server three times for
positive cycle/bridge/impact and isolated clean-negative projects. It retains
JSONL requests/responses and asserts protocol/tool/query schema identities,
complete evidence, text/structured result equality, stable result JSON,
latency, per-process maximum RSS, logs, and provenance.

The `yams_architecture` plan is a real-corpus integration benchmark. It records
three stable-identity scans and excludes YAMS's checked-in `site` output with
Brick's explicit `--exclude` option. Validate each resulting graph with YAMS's
`tests/benchmarks/brick/contract.py`; the plan's broad count bounds are coverage
guards, not precision or architecture grades.

`run_manifest.json` records Git provenance for both Brick and the corpus. Dirty
trees include a SHA-256 fingerprint over status, tracked diffs, and untracked
file contents so retained results identify the exact measured working state.

Before changing an extractor or score, add the expected positive and
clean-negative cases, name one owner, and state the expected metric/identity
delta. Use at least three repeats for performance claims.