01836ba3f5
Expand module-scoped use trees into exact import paths and bindings. Resolve renamed public symbols and module aliases without global-name fallback; keep unknown and block-local bindings conservative. Add positive and negative integration tests and an exact three-repeat import benchmark to the quality gate.
78 lines
4.4 KiB
Markdown
78 lines
4.4 KiB
Markdown
# Brick benchmark loop
|
|
|
|
The runner is an artifact-first, bounded experiment loop inspired by YAMS
|
|
xplan and Fold's optimization harness.
|
|
|
|
python3 tests/benchmarks/runner.py self-test
|
|
python3 tests/benchmarks/runner.py run lexical_smoke
|
|
python3 tests/benchmarks/runner.py run source_languages
|
|
python3 tests/benchmarks/runner.py run rust_imports
|
|
python3 tests/benchmarks/runner.py run p4_syntax
|
|
python3 tests/benchmarks/runner.py run p4_metrics
|
|
python3 tests/benchmarks/runner.py run p4c_evidence
|
|
python3 tests/benchmarks/runner.py run tree_sitter_p4_pilot
|
|
python3 tests/benchmarks/diff_runner.py run
|
|
python3 tests/benchmarks/policy_runner.py run
|
|
python3 tests/benchmarks/mcp_runner.py run
|
|
python3 tests/benchmarks/runner.py run yams_architecture --corpus /path/to/yams
|
|
|
|
Each run writes a resolved plan, per-repeat graph/HTML artifacts, summary JSON,
|
|
and REPORT.md under build/benchmarks. Canonical full-model JSON (including
|
|
attributes, evidence, diagnostics, and metrics) must remain identical across
|
|
repeats. The seed corpus has explicit positive cycle and relationship
|
|
expectations; it is not representative enough for performance or language-wide
|
|
precision claims.
|
|
|
|
The `source_languages` plan scans authored C++, JS, TS, TSX, Dart, and Rust
|
|
positive/clean-negative fixtures three times. Its labels enumerate every symbol
|
|
(including owner) and dependency. Missing facts, unexpected facts, and duplicate
|
|
facts fail the gate; clean-negative files participate in the same comparison.
|
|
The summary includes exactness and precision/recall by language stratum, full
|
|
JSON stability, latency, RSS, and Git provenance. The quality gate runs this plan.
|
|
|
|
The `p4_syntax` plan runs three deterministic scans of labeled P4_16 positive
|
|
and clean-negative fixtures. It guards symbol/dependency coverage and a clean
|
|
cycle count; it is not a claim of P4 language-wide precision or p4c parity.
|
|
The `p4_metrics` plan asserts every nested P4 metric across parser-cycle,
|
|
unreachable-state, terminal/select, reuse, repeated-apply, and unresolved
|
|
fixtures. The `p4c_evidence` plan repeats the qualified v1.2.5.16 diagnostic import and
|
|
retains emitted JSON/HTML plus its dirty-tree manifest; stable identities here
|
|
cover the Imported finding and Flags edge as well as syntax structure.
|
|
|
|
The `tree_sitter_p4_pilot` plan builds the pinned optional feature once, then
|
|
runs the Brick binary directly three times. It compares labeled table-action
|
|
relationships for precision/recall and records elapsed time plus per-process
|
|
maximum RSS from `wait4`, alongside raw graphs and command logs. Its two
|
|
positive relationships and comment/string clean negatives are a seed check,
|
|
not a language-wide accuracy claim.
|
|
|
|
The `structural_diff` plan rebuilds positive and identical clean-negative
|
|
baseline/current fixtures at one canonical root, then runs `brick check` three
|
|
times per case. It retains complete models/diffs, exact count assertions,
|
|
latency, per-process maximum RSS, command logs, and provenance. It measures the
|
|
diff/check path, not scan or render time.
|
|
|
|
The `policy` plan rebuilds one labeled project and runs a violating and a
|
|
clean-negative repository-relative policy three times each. It asserts exit
|
|
status, exact match counts, evidence retention, deterministic result JSON,
|
|
latency, per-process maximum RSS, logs, and provenance.
|
|
|
|
The `mcp_queries` plan starts the read-only stdio server three times for
|
|
positive cycle/bridge/impact and isolated clean-negative projects. It retains
|
|
JSONL requests/responses and asserts protocol/tool/query schema identities,
|
|
complete evidence, text/structured result equality, stable result JSON,
|
|
latency, per-process maximum RSS, logs, and provenance.
|
|
|
|
The `yams_architecture` plan is a real-corpus integration benchmark. It records
|
|
three stable-identity scans and excludes YAMS's checked-in `site` output with
|
|
Brick's explicit `--exclude` option. Validate each resulting graph with YAMS's
|
|
`tests/benchmarks/brick/contract.py`; the plan's broad count bounds are coverage
|
|
guards, not precision or architecture grades.
|
|
|
|
`run_manifest.json` records Git provenance for both Brick and the corpus. Dirty
|
|
trees include a SHA-256 fingerprint over status, tracked diffs, and untracked
|
|
file contents so retained results identify the exact measured working state.
|
|
|
|
Before changing an extractor or score, add the expected positive and
|
|
clean-negative cases, name one owner, and state the expected metric/identity
|
|
delta. Use at least three repeats for performance claims.
|