brick/docs/research/tree-sitter-p4-pilot.md

Tree-sitter P4 pilot decision

Candidate and pin

Brick's optional pilot uses duskmoon314/tree-sitter-p4 at commit 19b9dbdc2a2fb1d6c1b5346a75537434892a94b1. The repository is MIT licensed. Its Rust binding declares Tree-sitter 0.26.8; the resolved Brick build uses the compatible 0.26 API. The upstream README reports 119 of 121 official Tofino examples parsing, but that is upstream's claim, not a Brick accuracy result.

The dependency is default-disabled behind Cargo feature tree-sitter-p4-pilot. A graph records both scope.p4_frontend = "tree-sitter-p4-pilot" and evidence backend tree-sitter-p4:<commit>:brick-p4-pilot-v1. Grammar repository, exact commit, and Brick query version are also retained in provenance.

Setup probe

Run brick setup --json to obtain the versioned capability report before selecting the pilot. It records the Cargo feature, grammar repository and commit, query version, availability, and any initialization failure. Setup and CI can require a usable parser explicitly:

cargo run --features tree-sitter-p4-pilot -- \
  setup --require-tree-sitter-p4-pilot

The requirement fails closed if the feature was omitted or the pinned grammar cannot initialize. Detection does not silently select a frontend; callers must still pass --p4-frontend tree-sitter-pilot when building.

Evidence boundary

The lexical P4 pass always runs and remains the default. Selecting the pilot augments rather than replaces its graph:

• Tree-sitter declaration matches add parallel Syntactic evidence to the lexical Defines edge.
• Tree-sitter table action-list matches create Calls edges with p4.relation = "table-action".
• The grammar match is Syntactic. Resolving the captured action name through Brick's control-scoped index is separate Heuristic evidence from brick-p4-tree-sitter-name-index:v1.
• Nothing from the pilot is labeled Symbolic or Imported, and imported p4c diagnostics retain their original evidence unchanged.
• Any Tree-sitter error or missing node rejects all pilot facts for that file and emits a diagnostic. Lexical facts remain available; the pilot does not partially bless a malformed tree.

Unsupported languages are unchanged. Because the selected frontend changes observable extraction, graphs built with lexical-only and pilot scopes are intentionally incompatible for comparison.

Fixture result

The labeled corpus is tests/fixtures/p4_tree_sitter/. It contains multiline/nested action lists, duplicate table/action names in separate controls, and clean negatives in comments and annotation strings. Precision/recall labels include repository-relative path and owner names so scoped relationships cannot collapse by endpoint name. Accepted runs are retained under build/benchmarks/tree_sitter_p4_pilot/<run-stamp>/; the workflow completion evidence names the exact run. On the recorded Linux host and direct debug Brick binary, the accepted run must report:

• relationship precision: 1.000 (4 true positives, 0 false positives)
• relationship recall: 1.000 (4 true positives, 0 false negatives)
• three elapsed-time and per-process maximum-RSS samples
• stable node and edge identities across all three runs

The artifact's summary.json and REPORT.md are authoritative for the host-specific latency and RSS values.

These numbers characterize only this small seed corpus. They are not a claim about general P4 correctness or production performance. The run manifest records the dirty-worktree fingerprint, host, build command, raw graphs, command logs, and per-run RSS records.

Limitations

The grammar skips preprocessor directives rather than expanding them. Macro-generated declarations, token pasting, and include-dependent syntax therefore remain outside the pilot's demonstrated coverage. The Brick query currently covers declarations and nested table action-list references only. It does not replace lexical transitions, apply calls, includes, imported compiler diagnostics, or any verifier.

Decision

Retain as an opt-in pilot; reject promotion to the default frontend. The pilot improves nested table action-list extraction on the labeled fixture while preserving evidence lanes and identities, but the corpus is too small and preprocessing remains unresolved. Promotion requires a larger labeled P4_16 corpus with macro/include cases, explicit malformed-source expectations, repeated precision/recall and resource measurements, and no regression in lexical or imported-evidence contracts.