brick/tests/benchmarks

Brick benchmark loop

The runner is an artifact-first, bounded experiment loop inspired by YAMS xplan and Fold's optimization harness.

python3 tests/benchmarks/runner.py self-test
python3 tests/benchmarks/runner.py run lexical_smoke
python3 tests/benchmarks/runner.py run source_languages
python3 tests/benchmarks/runner.py run rust_imports
python3 tests/benchmarks/runner.py run p4_syntax
python3 tests/benchmarks/runner.py run p4_metrics
python3 tests/benchmarks/runner.py run p4c_evidence
python3 tests/benchmarks/runner.py run tree_sitter_p4_pilot
python3 tests/benchmarks/diff_runner.py run
python3 tests/benchmarks/policy_runner.py run
python3 tests/benchmarks/mcp_runner.py run
python3 tests/benchmarks/runner.py run yams_architecture --corpus /path/to/yams

Each run writes a resolved plan, per-repeat graph/HTML artifacts, summary JSON, and REPORT.md under build/benchmarks. Canonical full-model JSON (including attributes, evidence, diagnostics, and metrics) must remain identical across repeats. The seed corpus has explicit positive cycle and relationship expectations; it is not representative enough for performance or language-wide precision claims.

The source_languages plan scans authored C++, JS, TS, TSX, Dart, and Rust positive/clean-negative fixtures three times. Its labels enumerate every symbol (including owner) and dependency. Missing facts, unexpected facts, and duplicate facts fail the gate; clean-negative files participate in the same comparison. The summary includes exactness and precision/recall by language stratum, full JSON stability, latency, RSS, and Git provenance. The quality gate runs this plan.

The p4_syntax plan runs three deterministic scans of labeled P4_16 positive and clean-negative fixtures. It guards symbol/dependency coverage and a clean cycle count; it is not a claim of P4 language-wide precision or p4c parity. The p4_metrics plan asserts every nested P4 metric across parser-cycle, unreachable-state, terminal/select, reuse, repeated-apply, and unresolved fixtures. The p4c_evidence plan repeats the qualified v1.2.5.16 diagnostic import and retains emitted JSON/HTML plus its dirty-tree manifest; stable identities here cover the Imported finding and Flags edge as well as syntax structure.

The tree_sitter_p4_pilot plan builds the pinned optional feature once, then runs the Brick binary directly three times. It compares labeled table-action relationships for precision/recall and records elapsed time plus per-process maximum RSS from wait4, alongside raw graphs and command logs. Its two positive relationships and comment/string clean negatives are a seed check, not a language-wide accuracy claim.

The structural_diff plan rebuilds positive and identical clean-negative baseline/current fixtures at one canonical root, then runs brick check three times per case. It retains complete models/diffs, exact count assertions, latency, per-process maximum RSS, command logs, and provenance. It measures the diff/check path, not scan or render time.

The policy plan rebuilds one labeled project and runs a violating and a clean-negative repository-relative policy three times each. It asserts exit status, exact match counts, evidence retention, deterministic result JSON, latency, per-process maximum RSS, logs, and provenance.

The mcp_queries plan starts the read-only stdio server three times for positive cycle/bridge/impact and isolated clean-negative projects. It retains JSONL requests/responses and asserts protocol/tool/query schema identities, complete evidence, text/structured result equality, stable result JSON, latency, per-process maximum RSS, logs, and provenance.

The yams_architecture plan is a real-corpus integration benchmark. It records three stable-identity scans and excludes YAMS's checked-in site output with Brick's explicit --exclude option. Validate each resulting graph with YAMS's tests/benchmarks/brick/contract.py; the plan's broad count bounds are coverage guards, not precision or architecture grades.

run_manifest.json records Git provenance for both Brick and the corpus. Dirty trees include a SHA-256 fingerprint over status, tracked diffs, and untracked file contents so retained results identify the exact measured working state.

Before changing an extractor or score, add the expected positive and clean-negative cases, name one owner, and state the expected metric/identity delta. Use at least three repeats for performance claims.