merge/audit-and-state-map #7

Merged
trevon merged 15 commits from merge/audit-and-state-map into main 2026-09-25 00:07:04 +00:00
6 changed files with 853 additions and 1 deletions
Showing only changes of commit 4b499c1e21 - Show all commits

feat(cpp): experimental C++ front end for scoped-enum state

tool/cpp_front.py is a second front end written against the state model,
not the rules: it reads a compile database with libclang (Homebrew LLVM)
and writes the JSON that `pipe check --model` takes. Every rule,
baseline and gate works on it unchanged.

Mapping, per project `enum class` (identity from clang's resolved AST,
so two enums both named QueryIntent are never confused):
- a reference that produces a value (assignment, atomic store, argument,
  return, initialiser) is an emission; a cast from an int is an indirect
  one;
- a `case`, `==` or `!=` is an observation;
- a busy enumerator written to the method's OWN state, followed by a
  top-level try whose catch neither writes state nor rethrows, is a
  swallowed failure;
- sources the build does not compile are recorded as unresolved, so
  findings naming them drop a confidence step.

One rule-side change, optional and Dart-neutral: an EmissionSite may name
its stateType. rendered-not-emitted attributed indirect emissions by owner
(a bloc owns one state type), so a cast to one enum made "never emitted"
unprovable for every other enum in the same C++ class. With stateType
set, the suppression is scoped to that enum. Dart output is still byte
identical on the fixture and all four dogfood apps.

Run on ~/work/tools/yams (375 translation units from build/debug, 0 clang
errors, about two minutes): 118 state enums, 82 findings. A grep-based
hand check of 13 found 12 factually correct - e.g. the IPC connection FSM
never enters ImplState::ReadingHeader, the downloader never reports
ProgressStage::Verifying though three CLI commands render it,
ExtractScopeType qualifiers are parsed but never acted on - and one false
positive (a nested JSON-parse fallback taken for a stuck busy state),
now excluded by requiring the method's own state and a top-level try.

Bugs found on the way and fixed: ccache/compiler wrappers passed as
inputs, relative -I paths resolved against the wrong directory (7 TUs
silently lost third_party headers), a symlinked path putting every TU
"outside" the project, and an empty compile database producing an empty
"clean" model with exit 0 - all now fail loudly or are handled.

test/cpp_front_test.dart runs the front end on test/fixtures/cpp_state
with positive and negative controls for each rule; skipped without LLVM.
138 tests pass.
trvon 2026-09-22 19:16:53 -06:00
Signed by: trevon
GPG key ID: 38686EA797D7A8A7

View file

@ -411,6 +411,41 @@ rule, `--baseline` ratcheting and `--fail-on` gating without touching Dart.
`check --model` refuses `--write-baseline`, because a model does not record
whether its front end read every file.
### Experimental: C++ state enums
`tool/cpp_front.py` is a second front end, written against the model rather than
the rules. It reads a compile database with libclang (Homebrew LLVM) and maps
each project `enum class` to a union: a reference that produces a value
(assignment, atomic `store`, argument, return, initialiser) is an emission; a
`case`, `==` or `!=` is an observation; a busy enumerator set on the method's
own state followed by a top-level `catch` that neither writes state nor
rethrows is a swallowed failure. Identity comes from clang's resolved AST, so
two enums both named `QueryIntent` are never confused.
```bash
tool/cpp_front.py ~/work/tools/yams --build-dir ~/work/tools/yams/build/debug -o yams.json
pipe check --model yams.json
```
Read the rules as: `rendered-not-emitted` - an enumerator something branches on
that nothing ever produces (a dead branch, or a state an FSM can never enter);
`emitted-not-rendered` - a value produced that nothing ever branches on
(computed and ignored, or reaching output only through a name table). Sources
the build does not compile are recorded as unresolved, so findings naming them
drop a confidence step. `test/cpp_front_test.dart` holds the positive and
negative controls (`test/fixtures/cpp_state`); it is skipped without LLVM.
On yams (375 translation units, 0 clang errors, about two minutes) the first
run produced 82 findings. A grep-based hand check of 13 found 12 factually
correct and 1 false positive (a nested JSON-parse fallback taken for a stuck
busy state), which the front end now excludes. Factual is not the same as a
bug: of the 12, `TransportState::Error` is covered by `!= Connected` checks and
`InboundSessionStage` is fuzzing instrumentation. The substantive ones include
the IPC connection FSM never entering `ImplState::ReadingHeader`, the downloader
never reporting `ProgressStage::Verifying` although three CLI commands render
it, and `ExtractScopeType` query qualifiers being parsed but never acted on.
n=13 from one repo bounds the false-positive rate; it does not measure it.
Built on `package:analyzer` with whole-project resolution, not a
`custom_lint`/analyzer plugin: plugins have no project-wide index, `flutter
analyze` does not report new-system plugin diagnostics, and a CLI can pin its own

View file

@ -523,8 +523,14 @@ FindingReport runChecks({
if (enabled.contains(ruleRenderedNotEmitted)) {
for (final coverage in flow.coverages) {
if (coverage.renderedNeverEmitted.isEmpty) continue;
// An emission that names its state type belongs to that union only; one
// that does not (a Dart `emit(_helper())`) is attributed by owner, since a
// bloc owns one state type. A C++ class holds several enums, and a cast
// to one of them must not make every other one unprovable.
final ownerEmissions = flow.emissions.where(
(emission) => emission.owner == coverage.union.owner,
(emission) => emission.stateType == null
? emission.owner == coverage.union.owner
: emission.stateType == coverage.union.name,
);
final uncertain = ownerEmissions.any(
(emission) =>

View file

@ -104,6 +104,7 @@ class EmissionSite {
required this.confidence,
required this.variants,
required this.snippet,
this.stateType,
});
final String path;
@ -111,6 +112,11 @@ class EmissionSite {
final String owner;
final String member;
/// The state type this emission produces, when the front end knows it even
/// though the variant is unknown (a C++ `static_cast<State>(n)`). Null means
/// "attribute by owner", which is right when an owner has one state type.
final String? stateType;
/// `emit`, `assign` or `notifyListeners`.
final String kind;
@ -128,6 +134,7 @@ class EmissionSite {
'confidence': confidence,
if (variants.isNotEmpty) 'variants': variants,
'snippet': snippet,
if (stateType != null) 'stateType': stateType,
};
}
@ -2039,6 +2046,7 @@ StateFlow stateFlowFromJson(Map<String, Object?> json) {
confidence: e['confidence']! as String,
variants: strings(e['variants']),
snippet: e['snippet']! as String,
stateType: e['stateType'] as String?,
),
],
renders: [

95
test/cpp_front_test.dart Normal file
View file

@ -0,0 +1,95 @@
import 'dart:convert';
import 'dart:io';
import 'package:pipe/pipe.dart';
import 'package:test/test.dart';
/// Contract for the experimental C++ front end (tool/cpp_front.py): it writes a
/// state model, and the unchanged rules must fire on each positive control in
/// test/fixtures/cpp_state and stay silent on each negative one.
///
/// Skipped, not passed, when Homebrew LLVM (libclang + its Python bindings) or
/// python3 is unavailable.
void main() {
final llvm = Platform.environment['PIPE_LLVM'] ?? '/opt/homebrew/opt/llvm';
final available =
File('$llvm/lib/libclang.dylib').existsSync() &&
Process.runSync('python3', ['--version']).exitCode == 0;
late FindingReport report;
setUpAll(() async {
if (!available) return;
final root = Directory.systemTemp.createTempSync('pipe-cpp');
Directory('${root.path}/src').createSync();
File(
'test/fixtures/cpp_state/src/engine.cpp',
).copySync('${root.path}/src/engine.cpp');
File('${root.path}/compile_commands.json').writeAsStringSync(
jsonEncode([
{
'directory': root.path,
'file': 'src/engine.cpp',
'arguments': ['c++', '-std=c++20', '-c', 'src/engine.cpp'],
},
]),
);
final model = '${root.path}/model.json';
final result = Process.runSync('python3', [
'tool/cpp_front.py',
root.path,
'--include',
'src',
'-o',
model,
]);
expect(result.exitCode, 0, reason: '${result.stderr}');
expect('${result.stderr}', contains('0 clang error(s)'));
report = runChecks(
flow: stateFlowFromJson(
jsonDecode(File(model).readAsStringSync()) as Map<String, Object?>,
),
);
});
Iterable<Finding> hits(String rule) =>
report.findings.where((finding) => finding.rule == rule);
test('an enum produced but never examined is reported', () {
expect(
hits(ruleEmittedNotRendered).map((f) => f.data['union']),
contains('fx::Unread'),
);
}, skip: available ? false : 'Homebrew LLVM / python3 not available');
test('an enumerator examined but never produced is a dead branch', () {
final ghost = hits(
ruleRenderedNotEmitted,
).where((f) => f.data['variant'] == 'Ghost');
expect(ghost, hasLength(1));
expect(ghost.single.file, 'src/engine.cpp');
expect(ghost.single.line, isNotNull);
}, skip: available ? false : 'Homebrew LLVM / python3 not available');
test('a cast to one enum does not mute "never emitted" for another', () {
// Casted::Y is produced only through static_cast, so it is unprovable
// and suppressed. Phase lives in the same class and must still be judged.
expect(
hits(ruleRenderedNotEmitted).map((f) => f.data['union']),
isNot(contains('fx::Casted')),
);
expect(report.suppressed.where((s) => s.contains('fx::Phase')), isEmpty);
}, skip: available ? false : 'Homebrew LLVM / python3 not available');
test(
'a swallowed failure on the method\'s own busy state fires, and only it',
() {
// startGuarded resolves the state, startNested's try is a nested local
// fallback, and Reporter::run sets a field of a local event object.
expect(hits(ruleLoadingNotReset).map((f) => f.data['member']).toList(), [
'startSwallowed',
]);
},
skip: available ? false : 'Homebrew LLVM / python3 not available',
);
}

104
test/fixtures/cpp_state/src/engine.cpp vendored Normal file
View file

@ -0,0 +1,104 @@
// Fixture for tool/cpp_front.py: each case is a positive or negative control
// for a rule run on the C++ state model (see test/cpp_front_test.dart).
namespace fx {
// Ghost is compared against but never produced: rendered-not-emitted.
enum class Phase { Idle, Starting, Running, Failed, Ghost };
// Produced, never compared or switched on: emitted-not-rendered.
enum class Unread { A, B };
// Y is compared against and produced only through a cast from an int, so the
// rule must suppress itself rather than call the branch dead.
enum class Casted { X, Y };
void mayThrow();
class Engine {
public:
// Negative: the catch resolves the busy state.
void startGuarded() {
phase_ = Phase::Starting;
try {
mayThrow();
phase_ = Phase::Running;
} catch (...) {
phase_ = Phase::Failed;
}
}
// Positive: busy state set, then a top-level try whose catch swallows the
// failure, so Starting can outlive it (loading-not-reset).
void startSwallowed() {
phase_ = Phase::Starting;
try {
mayThrow();
phase_ = Phase::Running;
} catch (...) {
}
}
// Negative: a nested try is a local fallback, not the method's failure path.
void startNested() {
phase_ = Phase::Starting;
if (ready_) {
try {
mayThrow();
} catch (...) {
}
}
phase_ = Phase::Running;
}
bool ghost() const { return phase_ == Phase::Ghost; }
bool idle() const {
switch (phase_) {
case Phase::Idle:
return true;
case Phase::Starting:
case Phase::Running:
case Phase::Failed:
return false;
default:
return false;
}
}
void setUnread() {
unread_ = Unread::A;
unread_ = Unread::B;
}
void load(int n) { casted_ = static_cast<Casted>(n); }
bool isY() const { return casted_ == Casted::Y; }
private:
bool ready_ = false;
Phase phase_ = Phase::Idle;
Unread unread_ = Unread::A;
Casted casted_ = Casted::X;
};
// Negative: `ev.stage` is a field of a local event object handed to a callback,
// not state Reporter holds, so its try cannot leave Reporter stuck.
struct Event {
Phase stage;
};
void notify(const Event&);
class Reporter {
public:
void run() {
Event ev;
ev.stage = Phase::Starting;
notify(ev);
try {
mayThrow();
} catch (...) {
}
}
};
} // namespace fx

604
tool/cpp_front.py Normal file
View file

@ -0,0 +1,604 @@
#!/usr/bin/env python3
"""Experimental C++ front end: writes a pipe state model from clang's AST.
Usage:
tool/cpp_front.py <project-root> [--build-dir DIR] [--include src,include]
[--limit N] [--jobs N] [-o model.json]
pipe check --model model.json
pipe's rules read only the state model, so this front end never touches them.
It uses libclang's *resolved* AST: every enumerator reference is linked to its
EnumConstantDecl, so identity is exact rather than matched by name.
What it maps (scoped enums only, `enum class`, declared in the project):
union an `enum class`; variants are its enumerators
emission a reference that PRODUCES the value: assigned to a field or
variable, stored into an atomic, passed, returned, initialised
observation a reference that EXAMINES the value: a `case` label, `==`, `!=`
(these fill pipe's "rendered" set: something branches on it)
swallowed a member writes a busy enumerator (Starting, Loading, ...) to a
failure field, then has a try whose catch writes no state and does not
rethrow (loading-not-reset)
Consequences worth knowing before reading the findings:
- `emitted-not-rendered` here means "produced, but nothing ever branches on or
compares it". A value that reaches the user through a lookup table or
reflection (magic_enum) is invisible to this, so treat those as leads.
- `rendered-not-emitted` means "compared against, but never produced anywhere":
a dead branch, unless the value comes from a cast (`static_cast<E>(n)`), which
is recorded as an indirect emission so the rule suppresses itself.
- The rule messages still speak of "UI"; for C++ read "nothing observes it".
Requires Homebrew LLVM's libclang and its Python bindings (found automatically).
"""
from __future__ import annotations
import argparse
import glob
import json
import os
import re
import shlex
import subprocess
import sys
from collections import Counter, defaultdict
from concurrent.futures import ProcessPoolExecutor, as_completed
from pathlib import Path
LLVM = Path(os.environ.get("PIPE_LLVM", "/opt/homebrew/opt/llvm"))
def _load_cindex():
for candidate in sorted(glob.glob(str(LLVM / "lib/python3.*/site-packages")), reverse=True):
if candidate not in sys.path:
sys.path.insert(0, candidate)
import clang.cindex as cindex # noqa: E402
# Pool workers are reused, and libclang may be configured only once.
if not cindex.Config.loaded:
cindex.Config.set_library_file(str(LLVM / "lib/libclang.dylib"))
return cindex
_SDK = None
BUSY = re.compile(
r"^(Loading|Starting|StartingUp|Initializing|Initialising|Connecting|Stopping|"
r"ShuttingDown|Busy|Pending|InProgress|Processing|Saving|Updating|Creating|"
r"Syncing|Indexing|Warming|WarmingUp|Draining|Migrating|Recovering)$"
)
# Kinds that only wrap an expression; the classifier looks through them.
_TRANSPARENT = {
"UNEXPOSED_EXPR",
"PAREN_EXPR",
"CSTYLE_CAST_EXPR",
"CXX_FUNCTIONAL_CAST_EXPR",
"CONDITIONAL_OPERATOR",
"INIT_LIST_EXPR",
}
def _resource_dir() -> str:
return subprocess.check_output([str(LLVM / "bin/clang"), "-print-resource-dir"], text=True).strip()
def _sdk_root() -> str | None:
"""Apple's compiler finds the macOS SDK itself; libclang needs it spelled out,
or every standard header (<chrono>, sqlite3.h) is missing."""
try:
return subprocess.check_output(["xcrun", "--show-sdk-path"], text=True).strip()
except (OSError, subprocess.CalledProcessError):
return None
def _args_for(entry: dict, resource_dir: str) -> list[str]:
"""Compiler flags only: drop wrappers (ccache), the compiler, the source,
and output/dependency options that make no sense for a syntax-only parse."""
args = entry.get("arguments") or shlex.split(entry["command"])
while args and (os.path.basename(args[0]) in ("ccache", "sccache", "distcc")
or not args[0].startswith("-") and args[0] != entry["file"]
and os.path.basename(args[0]) in ("c++", "g++", "clang++", "cc", "gcc", "clang")):
args = args[1:]
out: list[str] = []
skip = False
directory = entry["directory"]
# Include paths are relative to the build directory, but libclang resolves
# them against this process's working directory: made absolute, or headers
# like third_party/... silently go missing.
path_flags = ("-I", "-isystem", "-iquote", "-idirafter", "-include")
absolute = []
pending = None
for arg in args:
if pending is not None:
absolute.append(arg if os.path.isabs(arg) else os.path.normpath(os.path.join(directory, arg)))
pending = None
continue
if arg in path_flags:
absolute.append(arg)
pending = arg
continue
for flag in path_flags:
if arg.startswith(flag) and len(arg) > len(flag) and flag != "-include":
value = arg[len(flag):]
if not os.path.isabs(value):
arg = flag + os.path.normpath(os.path.join(directory, value))
break
absolute.append(arg)
args = absolute
for arg in args:
if skip:
skip = False
continue
if arg in ("-o", "-MF", "-MT", "-MQ"):
skip = True
continue
if arg in ("-c", "-MD", "-MMD", "-fdiagnostics-color=always") or \
arg.startswith(("-o", "-MF", "-MT", "-MQ")):
continue
if arg == entry["file"] or not arg.startswith("-") and arg.endswith((".cpp", ".cc", ".cxx")):
continue
out.append(arg)
out += ["-resource-dir", resource_dir, "-Wno-everything", "-fsyntax-only"]
sdk = _SDK
if sdk and "-isysroot" not in out:
out += ["-isysroot", sdk]
return out
def _qualified(cursor) -> str:
parts = []
c = cursor
while c is not None and c.kind.name != "TRANSLATION_UNIT":
if c.spelling:
parts.append(c.spelling)
c = c.semantic_parent
return "::".join(reversed(parts))
def _enclosing(stack, kinds):
for c in reversed(stack):
if c.kind.name in kinds:
return c
return None
def _operator(cursor) -> str | None:
try:
op = cursor.binary_operator
return op.name if op is not None else None
except AttributeError:
return None
def _top_level_try(stack) -> bool:
"""True when the try sits directly in its function's body, not nested in a
loop, branch, block or another try."""
for i in range(len(stack) - 1, -1, -1):
kind = stack[i].kind.name
if kind in ("CXX_METHOD", "FUNCTION_DECL", "CONSTRUCTOR", "LAMBDA_EXPR"):
# stack[i+1] is the body; the try must be its direct child.
return len(stack) - 1 == i + 1 and stack[i + 1].kind.name == "COMPOUND_STMT"
if kind == "CXX_TRY_STMT":
return False
return False
def _field_owner(expr, cindex):
"""For `x.state_` / `state_` / `this->state_`: the class declaring the field."""
for node in expr.walk_preorder():
if node.kind.name == "MEMBER_REF_EXPR":
ref = node.referenced
if ref is not None and ref.kind.name == "FIELD_DECL":
return ref.semantic_parent, ref.spelling
return None, None
def analyse_tu(task):
"""Parse one translation unit; return the facts located in project files."""
file, directory, args, root, includes = task
cindex = _load_cindex()
index = cindex.Index.create()
try:
tu = index.parse(
os.path.normpath(os.path.join(directory, file)),
args=args,
options=cindex.TranslationUnit.PARSE_INCOMPLETE,
)
except cindex.TranslationUnitLoadError as error:
return {"file": file, "error": str(error)}
root = os.path.realpath(root)
prefixes = tuple(os.path.join(root, p) + os.sep for p in includes)
def in_project(cursor) -> str | None:
loc = cursor.location
if loc.file is None:
return None
path = os.path.realpath(loc.file.name)
if not path.startswith(prefixes):
return None
return os.path.relpath(path, root)
enums = {}
refs = []
busy_tries = []
def classify(stack, ref):
"""Return (role, owner, field, kind) for an enumerator reference."""
child = ref
for parent in reversed(stack):
kind = parent.kind.name
if kind in _TRANSPARENT:
child = parent
continue
if kind == "CASE_STMT":
first = next(iter(parent.get_children()), None)
if first is not None and first.hash == child.hash:
return ("observe", None, None, "case")
return ("produce", None, None, "other")
if kind == "BINARY_OPERATOR":
op = _operator(parent)
children = list(parent.get_children())
if op in ("EQ", "NE"):
return ("observe", None, None, "compare")
if op == "Assign" and len(children) == 2 and children[1].hash == child.hash:
owner, field = _field_owner(children[0], cindex)
return ("produce", owner, field, "assign")
return ("produce", None, None, "other")
if kind == "CALL_EXPR" and parent.spelling in ("store", "exchange"):
owner, field = _field_owner(parent, cindex)
return ("produce", owner, field, "assign")
if kind == "MEMBER_REF":
ref_decl = parent.referenced
if ref_decl is not None and ref_decl.kind.name == "FIELD_DECL":
return ("produce", ref_decl.semantic_parent, ref_decl.spelling, "initial")
if kind == "FIELD_DECL":
return ("produce", parent.semantic_parent, parent.spelling, "initial")
if kind in ("CALL_EXPR", "RETURN_STMT", "VAR_DECL", "COMPOUND_STMT",
"IF_STMT", "CXX_METHOD", "FUNCTION_DECL", "CONSTRUCTOR"):
return ("produce", None, None, "other")
child = parent
return ("produce", None, None, "other")
def member_name(stack):
fn = _enclosing(stack, {"CXX_METHOD", "FUNCTION_DECL", "CONSTRUCTOR",
"DESTRUCTOR", "FUNCTION_TEMPLATE", "LAMBDA_EXPR"})
if fn is None:
return "<top>", None
if fn.kind.name == "LAMBDA_EXPR":
outer = _enclosing(stack[: stack.index(fn)], {"CXX_METHOD", "FUNCTION_DECL", "CONSTRUCTOR"})
name = outer.spelling if outer is not None else "<lambda>"
return f"{name}::<lambda>", outer
return fn.spelling, fn
def scan_try(stack, try_cursor, rel):
"""A try inside a member that wrote a busy value earlier."""
body = list(try_cursor.get_children())
handlers = [c for c in body if c.kind.name == "CXX_CATCH_STMT"]
swallowed = []
for handler in handlers:
writes_state = False
rethrows = False
for node in handler.walk_preorder():
if node.kind.name == "CXX_THROW_EXPR":
rethrows = True
if node.kind.name == "DECL_REF_EXPR":
ref = node.referenced
if ref is not None and ref.kind.name == "ENUM_CONSTANT_DECL":
writes_state = True
if node.kind.name in ("CALL_EXPR",) and re.search(
r"(set|mark|fail|error|transition|reset|clear)", node.spelling or "", re.I
):
writes_state = True
if not writes_state and not rethrows:
tokens = " ".join(t.spelling for t in handler.get_tokens())
swallowed.append((handler.location.line, tokens[:117] + ("..." if len(tokens) > 117 else "")))
return swallowed
def visit(cursor, stack, busy_seen):
rel = in_project(cursor)
if rel is None and cursor.kind.name != "TRANSLATION_UNIT":
return
kind = cursor.kind.name
if kind == "ENUM_DECL" and cursor.is_scoped_enum() and cursor.is_definition():
name = _qualified(cursor)
enums[name] = {
"name": name,
"declaredIn": rel,
"line": cursor.location.line,
"variants": [c.spelling for c in cursor.get_children()
if c.kind.name == "ENUM_CONSTANT_DECL"],
}
elif kind == "DECL_REF_EXPR":
ref = cursor.referenced
if ref is not None and ref.kind.name == "ENUM_CONSTANT_DECL":
parent_enum = ref.semantic_parent
if parent_enum is not None and parent_enum.is_scoped_enum():
role, owner, field, how = classify(stack, cursor)
member, fn = member_name(stack)
klass = _enclosing(stack, {"CLASS_DECL", "STRUCT_DECL", "CLASS_TEMPLATE"})
if fn is not None and fn.semantic_parent is not None and \
fn.semantic_parent.kind.name in ("CLASS_DECL", "STRUCT_DECL", "CLASS_TEMPLATE"):
klass = fn.semantic_parent
record = {
"enum": _qualified(parent_enum),
"variant": ref.spelling,
"role": role,
"how": how,
"owner": _qualified(owner) if owner is not None else None,
"field": field,
"member": member,
"memberOwner": _qualified(klass) if klass is not None else None,
"path": rel,
"line": cursor.location.line,
"col": cursor.location.column,
}
refs.append(record)
# Only the method's OWN state counts: `ev.stage = Connecting` on a
# local event object handed to a callback is not state this
# class can leave stuck.
if role == "produce" and how == "assign" and BUSY.match(ref.spelling) \
and owner is not None and klass is not None \
and _qualified(owner) == _qualified(klass):
busy_seen.append(record)
elif kind == "CXX_STATIC_CAST_EXPR" or kind == "CSTYLE_CAST_EXPR":
target = cursor.type.get_canonical()
decl = target.get_declaration()
if decl is not None and decl.kind.name == "ENUM_DECL" and decl.is_scoped_enum():
children = list(cursor.get_children())
if children and children[-1].type.get_canonical().kind.name != "ENUM":
refs.append({
"enum": _qualified(decl), "variant": None, "role": "cast", "how": "cast",
"owner": None, "field": None, "member": member_name(stack)[0],
"memberOwner": None, "path": rel, "line": cursor.location.line,
"col": cursor.location.column,
})
elif kind == "CXX_TRY_STMT" and busy_seen and _top_level_try(stack):
# (a top-level try of the method: nested tries are local fallbacks,
# like a JSON parse guarded with `catch (...) { skip = false; }`)
member, fn = member_name(stack)
for busy in busy_seen:
if busy["member"] != member:
continue
for line, handler in scan_try(stack, cursor, rel):
busy_tries.append({
"owner": busy["owner"], "member": member, "path": rel,
"line": line, "handler": handler,
})
break
new_busy = busy_seen
if kind in ("CXX_METHOD", "FUNCTION_DECL", "CONSTRUCTOR", "LAMBDA_EXPR"):
new_busy = []
stack.append(cursor)
for child in cursor.get_children():
visit(child, stack, new_busy)
stack.pop()
visit(tu.cursor, [], [])
errors = [f"{d.location.file}:{d.location.line}: {d.spelling}"
for d in tu.diagnostics if d.severity >= 3]
return {"file": file, "enums": list(enums.values()), "refs": refs, "tries": busy_tries,
"errors": len(errors), "errorSamples": errors[:3]}
def build_model(root: Path, results: list[dict], includes: list[str]) -> dict:
enums: dict[str, dict] = {}
seen_refs = set()
refs = []
tries = {}
failed = []
for result in results:
if "error" in result:
failed.append(result["file"])
continue
for enum in result["enums"]:
enums.setdefault(enum["name"], enum)
for ref in result["refs"]:
key = (ref["path"], ref["line"], ref["col"], ref["enum"], ref["variant"])
if key in seen_refs:
continue
seen_refs.add(key)
refs.append(ref)
for t in result["tries"]:
tries.setdefault((t["path"], t["line"]), t)
by_enum = defaultdict(list)
for ref in refs:
by_enum[ref["enum"]].append(ref)
unions, coverage, emissions, renders = [], [], [], []
for name, enum in sorted(enums.items()):
if len(enum["variants"]) < 2:
continue
erefs = by_enum.get(name, [])
owners = Counter(r["owner"] for r in erefs if r["role"] == "produce" and r["owner"])
# The class whose field holds this state; without one it is a plain value
# type (a mode flag, a return code), not state anyone could leave stale.
if not owners:
continue
owner = owners.most_common(1)[0][0]
variants = enum["variants"]
emitted, initial, observed = set(), set(), set()
observe_sites = 0
emit_sites = 0
for r in erefs:
if r["role"] == "observe":
observed.add(r["variant"])
observe_sites += 1
renders.append({
"path": r["path"], "line": r["line"], "widget": r["how"],
"stateType": name,
"branches": [{"variant": r["variant"], "line": r["line"], "trivial": False}],
})
elif r["role"] in ("produce", "cast"):
emit_sites += 1
if r["variant"]:
emitted.add(r["variant"])
if r["how"] == "initial":
initial.add(r["variant"])
emissions.append({
"path": r["path"], "line": r["line"], "owner": owner,
"member": r["member"], "kind": "assign" if r["how"] == "assign" else "emit",
"confidence": "exact" if r["role"] == "produce" else "indirect",
"variants": [r["variant"]] if r["variant"] else [],
"snippet": f"{r['field'] or ''} = {name.split('::')[-1]}::{r['variant']}"
if r["variant"] else f"static_cast<{name.split('::')[-1]}>(...)",
# A class holds several enums: name the union so a cast to one
# does not make "never emitted" unprovable for the others.
"stateType": name,
})
union = {
"name": name, "declaredIn": enum["declaredIn"], "style": "enumUnion",
"owner": owner, "ownerKind": "Cubit",
"variants": [{"name": v, "declaredIn": enum["declaredIn"]} for v in variants],
}
unions.append(union)
coverage.append({
"union": name, "owner": owner, "style": "enumUnion",
"declared": sorted(variants), "emitted": sorted(emitted), "initial": sorted(initial),
"rendered": sorted(observed), "renderedInBuilder": [], "trivialInBuilder": [],
"emissionSites": emit_sites, "renderSites": observe_sites,
})
owners = {u["owner"] for u in unions}
swallowed = []
ordinals = Counter()
for (path, line), t in sorted(tries.items()):
if t["owner"] not in owners:
continue
ordinals[(t["owner"], t["member"])] += 1
swallowed.append({
"owner": t["owner"], "member": t["member"], "path": path, "line": line,
"ordinal": ordinals[(t["owner"], t["member"])], "handler": t["handler"],
})
# Files outside an owner's own declaration that mention it: pipe uses these
# to tell a dormant (unwired) union from a partly observed one.
references = defaultdict(set)
for r in refs:
if r["role"] == "observe" and r["enum"] in {u["name"] for u in unions}:
owner = next(u["owner"] for u in unions if u["name"] == r["enum"])
references[owner].add(r["path"])
return {
"root": str(root),
"directories": includes,
"frontEnd": {"name": "pipe-cpp-front (experimental)", "failedTranslationUnits": failed},
"unions": unions,
"eventUnions": [],
"emissions": emissions,
"renders": renders,
"handlers": [],
"dispatches": [],
"coverage": coverage,
"ownerReferences": {k: sorted(v) for k, v in references.items()},
"flagStates": [],
"riverpod": {"providers": [], "sites": []},
"constructedByFile": {},
"swallowedFailures": swallowed,
"unsettledFlags": [],
}
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
parser.add_argument("root")
parser.add_argument("--build-dir", help="directory holding compile_commands.json")
parser.add_argument("--include", default="src,include",
help="project subdirectories whose code is analysed")
parser.add_argument("--limit", type=int)
parser.add_argument("--match", help="only translation units whose path contains this")
parser.add_argument("--jobs", type=int, default=os.cpu_count())
parser.add_argument("-o", "--output", default="-")
args = parser.parse_args()
root = Path(args.root).resolve()
db_dir = Path(args.build_dir) if args.build_dir else root
db_path = db_dir / "compile_commands.json"
if not db_path.is_file():
print(f"cpp-front: no compile database at {db_path}; generate one "
f"(meson/CMake write it to the build dir) and pass --build-dir",
file=sys.stderr)
return 2
database = json.loads(db_path.read_text())
includes = [p for p in args.include.split(",") if p]
resource_dir = _resource_dir()
global _SDK
_SDK = _sdk_root()
tasks = []
seen = set()
for entry in database:
# Resolved on both sides: a symlinked path (macOS /var -> /private/var)
# otherwise lands "outside" the resolved root and the TU is skipped.
path = os.path.realpath(os.path.join(entry["directory"], entry["file"]))
if path in seen or not path.endswith((".cpp", ".cc", ".cxx")):
continue
rel = os.path.relpath(path, root)
if not any(rel.startswith(p + os.sep) for p in includes):
continue
if args.match and args.match not in rel:
continue
seen.add(path)
tasks.append((entry["file"], entry["directory"], _args_for(entry, resource_dir),
str(root), includes))
if args.limit:
tasks = tasks[: args.limit]
if not tasks:
# An empty model reads as a clean project; refuse instead.
print(f"cpp-front: no translation unit under {', '.join(includes)} in "
f"{db_dir / 'compile_commands.json'}; nothing was analysed", file=sys.stderr)
return 2
results = []
with ProcessPoolExecutor(max_workers=args.jobs) as pool:
futures = [pool.submit(analyse_tu, t) for t in tasks]
for i, future in enumerate(as_completed(futures), 1):
results.append(future.result())
if i % 25 == 0 or i == len(tasks):
print(f"cpp-front: {i}/{len(tasks)} translation units", file=sys.stderr)
model = build_model(root, results, includes)
# Sources the build does not compile (optional backends, benchmarks) were
# never analysed, so evidence in them is missing. Recorded as unresolved with
# the names they mention, pipe lowers confidence for any finding naming them.
unanalysed = []
names: set[str] = set()
for include in includes:
for path in sorted((root / include).rglob("*")):
if path.suffix not in (".cpp", ".cc", ".cxx") or os.path.realpath(path) in seen:
continue
unanalysed.append(os.path.relpath(path, root))
names.update(re.findall(r"[A-Za-z_][A-Za-z0-9_]*", path.read_text(errors="replace")))
failed_files = [os.path.normpath(f) for f in model["frontEnd"]["failedTranslationUnits"]]
if unanalysed or failed_files:
files = unanalysed + failed_files
model["unresolved"] = {
"files": files,
"analysed": len(tasks) - len(failed_files),
"first": f"{files[0]}: not compiled by this build, so not analysed",
"names": sorted(names),
}
failed = model["frontEnd"]["failedTranslationUnits"]
errors = sum(r.get("errors", 0) for r in results)
for r in results:
for sample in r.get("errorSamples", [])[:1]:
print(f"cpp-front: clang error in {r['file']}: {sample}", file=sys.stderr)
print(f"cpp-front: {len(model['unions'])} state enums, {len(model['emissions'])} "
f"emissions, {len(model['renders'])} observations, "
f"{len(model['swallowedFailures'])} swallowed failures; "
f"{len(failed)} TU(s) failed, {errors} clang error(s)", file=sys.stderr)
text = json.dumps(model, indent=2)
if args.output == "-":
print(text)
else:
Path(args.output).write_text(text)
return 0
if __name__ == "__main__":
raise SystemExit(main())