merge/audit-and-state-map #7
6 changed files with 853 additions and 1 deletions
feat(cpp): experimental C++ front end for scoped-enum state
tool/cpp_front.py is a second front end written against the state model, not the rules: it reads a compile database with libclang (Homebrew LLVM) and writes the JSON that `pipe check --model` takes. Every rule, baseline and gate works on it unchanged. Mapping, per project `enum class` (identity from clang's resolved AST, so two enums both named QueryIntent are never confused): - a reference that produces a value (assignment, atomic store, argument, return, initialiser) is an emission; a cast from an int is an indirect one; - a `case`, `==` or `!=` is an observation; - a busy enumerator written to the method's OWN state, followed by a top-level try whose catch neither writes state nor rethrows, is a swallowed failure; - sources the build does not compile are recorded as unresolved, so findings naming them drop a confidence step. One rule-side change, optional and Dart-neutral: an EmissionSite may name its stateType. rendered-not-emitted attributed indirect emissions by owner (a bloc owns one state type), so a cast to one enum made "never emitted" unprovable for every other enum in the same C++ class. With stateType set, the suppression is scoped to that enum. Dart output is still byte identical on the fixture and all four dogfood apps. Run on ~/work/tools/yams (375 translation units from build/debug, 0 clang errors, about two minutes): 118 state enums, 82 findings. A grep-based hand check of 13 found 12 factually correct - e.g. the IPC connection FSM never enters ImplState::ReadingHeader, the downloader never reports ProgressStage::Verifying though three CLI commands render it, ExtractScopeType qualifiers are parsed but never acted on - and one false positive (a nested JSON-parse fallback taken for a stuck busy state), now excluded by requiring the method's own state and a top-level try. Bugs found on the way and fixed: ccache/compiler wrappers passed as inputs, relative -I paths resolved against the wrong directory (7 TUs silently lost third_party headers), a symlinked path putting every TU "outside" the project, and an empty compile database producing an empty "clean" model with exit 0 - all now fail loudly or are handled. test/cpp_front_test.dart runs the front end on test/fixtures/cpp_state with positive and negative controls for each rule; skipped without LLVM. 138 tests pass.
commit
4b499c1e21
35
README.md
35
README.md
|
|
@ -411,6 +411,41 @@ rule, `--baseline` ratcheting and `--fail-on` gating without touching Dart.
|
|||
`check --model` refuses `--write-baseline`, because a model does not record
|
||||
whether its front end read every file.
|
||||
|
||||
### Experimental: C++ state enums
|
||||
|
||||
`tool/cpp_front.py` is a second front end, written against the model rather than
|
||||
the rules. It reads a compile database with libclang (Homebrew LLVM) and maps
|
||||
each project `enum class` to a union: a reference that produces a value
|
||||
(assignment, atomic `store`, argument, return, initialiser) is an emission; a
|
||||
`case`, `==` or `!=` is an observation; a busy enumerator set on the method's
|
||||
own state followed by a top-level `catch` that neither writes state nor
|
||||
rethrows is a swallowed failure. Identity comes from clang's resolved AST, so
|
||||
two enums both named `QueryIntent` are never confused.
|
||||
|
||||
```bash
|
||||
tool/cpp_front.py ~/work/tools/yams --build-dir ~/work/tools/yams/build/debug -o yams.json
|
||||
pipe check --model yams.json
|
||||
```
|
||||
|
||||
Read the rules as: `rendered-not-emitted` - an enumerator something branches on
|
||||
that nothing ever produces (a dead branch, or a state an FSM can never enter);
|
||||
`emitted-not-rendered` - a value produced that nothing ever branches on
|
||||
(computed and ignored, or reaching output only through a name table). Sources
|
||||
the build does not compile are recorded as unresolved, so findings naming them
|
||||
drop a confidence step. `test/cpp_front_test.dart` holds the positive and
|
||||
negative controls (`test/fixtures/cpp_state`); it is skipped without LLVM.
|
||||
|
||||
On yams (375 translation units, 0 clang errors, about two minutes) the first
|
||||
run produced 82 findings. A grep-based hand check of 13 found 12 factually
|
||||
correct and 1 false positive (a nested JSON-parse fallback taken for a stuck
|
||||
busy state), which the front end now excludes. Factual is not the same as a
|
||||
bug: of the 12, `TransportState::Error` is covered by `!= Connected` checks and
|
||||
`InboundSessionStage` is fuzzing instrumentation. The substantive ones include
|
||||
the IPC connection FSM never entering `ImplState::ReadingHeader`, the downloader
|
||||
never reporting `ProgressStage::Verifying` although three CLI commands render
|
||||
it, and `ExtractScopeType` query qualifiers being parsed but never acted on.
|
||||
n=13 from one repo bounds the false-positive rate; it does not measure it.
|
||||
|
||||
Built on `package:analyzer` with whole-project resolution, not a
|
||||
`custom_lint`/analyzer plugin: plugins have no project-wide index, `flutter
|
||||
analyze` does not report new-system plugin diagnostics, and a CLI can pin its own
|
||||
|
|
|
|||
|
|
@ -523,8 +523,14 @@ FindingReport runChecks({
|
|||
if (enabled.contains(ruleRenderedNotEmitted)) {
|
||||
for (final coverage in flow.coverages) {
|
||||
if (coverage.renderedNeverEmitted.isEmpty) continue;
|
||||
// An emission that names its state type belongs to that union only; one
|
||||
// that does not (a Dart `emit(_helper())`) is attributed by owner, since a
|
||||
// bloc owns one state type. A C++ class holds several enums, and a cast
|
||||
// to one of them must not make every other one unprovable.
|
||||
final ownerEmissions = flow.emissions.where(
|
||||
(emission) => emission.owner == coverage.union.owner,
|
||||
(emission) => emission.stateType == null
|
||||
? emission.owner == coverage.union.owner
|
||||
: emission.stateType == coverage.union.name,
|
||||
);
|
||||
final uncertain = ownerEmissions.any(
|
||||
(emission) =>
|
||||
|
|
|
|||
|
|
@ -104,6 +104,7 @@ class EmissionSite {
|
|||
required this.confidence,
|
||||
required this.variants,
|
||||
required this.snippet,
|
||||
this.stateType,
|
||||
});
|
||||
|
||||
final String path;
|
||||
|
|
@ -111,6 +112,11 @@ class EmissionSite {
|
|||
final String owner;
|
||||
final String member;
|
||||
|
||||
/// The state type this emission produces, when the front end knows it even
|
||||
/// though the variant is unknown (a C++ `static_cast<State>(n)`). Null means
|
||||
/// "attribute by owner", which is right when an owner has one state type.
|
||||
final String? stateType;
|
||||
|
||||
/// `emit`, `assign` or `notifyListeners`.
|
||||
final String kind;
|
||||
|
||||
|
|
@ -128,6 +134,7 @@ class EmissionSite {
|
|||
'confidence': confidence,
|
||||
if (variants.isNotEmpty) 'variants': variants,
|
||||
'snippet': snippet,
|
||||
if (stateType != null) 'stateType': stateType,
|
||||
};
|
||||
}
|
||||
|
||||
|
|
@ -2039,6 +2046,7 @@ StateFlow stateFlowFromJson(Map<String, Object?> json) {
|
|||
confidence: e['confidence']! as String,
|
||||
variants: strings(e['variants']),
|
||||
snippet: e['snippet']! as String,
|
||||
stateType: e['stateType'] as String?,
|
||||
),
|
||||
],
|
||||
renders: [
|
||||
|
|
|
|||
95
test/cpp_front_test.dart
Normal file
95
test/cpp_front_test.dart
Normal file
|
|
@ -0,0 +1,95 @@
|
|||
import 'dart:convert';
|
||||
import 'dart:io';
|
||||
|
||||
import 'package:pipe/pipe.dart';
|
||||
import 'package:test/test.dart';
|
||||
|
||||
/// Contract for the experimental C++ front end (tool/cpp_front.py): it writes a
|
||||
/// state model, and the unchanged rules must fire on each positive control in
|
||||
/// test/fixtures/cpp_state and stay silent on each negative one.
|
||||
///
|
||||
/// Skipped, not passed, when Homebrew LLVM (libclang + its Python bindings) or
|
||||
/// python3 is unavailable.
|
||||
void main() {
|
||||
final llvm = Platform.environment['PIPE_LLVM'] ?? '/opt/homebrew/opt/llvm';
|
||||
final available =
|
||||
File('$llvm/lib/libclang.dylib').existsSync() &&
|
||||
Process.runSync('python3', ['--version']).exitCode == 0;
|
||||
|
||||
late FindingReport report;
|
||||
|
||||
setUpAll(() async {
|
||||
if (!available) return;
|
||||
final root = Directory.systemTemp.createTempSync('pipe-cpp');
|
||||
Directory('${root.path}/src').createSync();
|
||||
File(
|
||||
'test/fixtures/cpp_state/src/engine.cpp',
|
||||
).copySync('${root.path}/src/engine.cpp');
|
||||
File('${root.path}/compile_commands.json').writeAsStringSync(
|
||||
jsonEncode([
|
||||
{
|
||||
'directory': root.path,
|
||||
'file': 'src/engine.cpp',
|
||||
'arguments': ['c++', '-std=c++20', '-c', 'src/engine.cpp'],
|
||||
},
|
||||
]),
|
||||
);
|
||||
final model = '${root.path}/model.json';
|
||||
final result = Process.runSync('python3', [
|
||||
'tool/cpp_front.py',
|
||||
root.path,
|
||||
'--include',
|
||||
'src',
|
||||
'-o',
|
||||
model,
|
||||
]);
|
||||
expect(result.exitCode, 0, reason: '${result.stderr}');
|
||||
expect('${result.stderr}', contains('0 clang error(s)'));
|
||||
report = runChecks(
|
||||
flow: stateFlowFromJson(
|
||||
jsonDecode(File(model).readAsStringSync()) as Map<String, Object?>,
|
||||
),
|
||||
);
|
||||
});
|
||||
|
||||
Iterable<Finding> hits(String rule) =>
|
||||
report.findings.where((finding) => finding.rule == rule);
|
||||
|
||||
test('an enum produced but never examined is reported', () {
|
||||
expect(
|
||||
hits(ruleEmittedNotRendered).map((f) => f.data['union']),
|
||||
contains('fx::Unread'),
|
||||
);
|
||||
}, skip: available ? false : 'Homebrew LLVM / python3 not available');
|
||||
|
||||
test('an enumerator examined but never produced is a dead branch', () {
|
||||
final ghost = hits(
|
||||
ruleRenderedNotEmitted,
|
||||
).where((f) => f.data['variant'] == 'Ghost');
|
||||
expect(ghost, hasLength(1));
|
||||
expect(ghost.single.file, 'src/engine.cpp');
|
||||
expect(ghost.single.line, isNotNull);
|
||||
}, skip: available ? false : 'Homebrew LLVM / python3 not available');
|
||||
|
||||
test('a cast to one enum does not mute "never emitted" for another', () {
|
||||
// Casted::Y is produced only through static_cast, so it is unprovable
|
||||
// and suppressed. Phase lives in the same class and must still be judged.
|
||||
expect(
|
||||
hits(ruleRenderedNotEmitted).map((f) => f.data['union']),
|
||||
isNot(contains('fx::Casted')),
|
||||
);
|
||||
expect(report.suppressed.where((s) => s.contains('fx::Phase')), isEmpty);
|
||||
}, skip: available ? false : 'Homebrew LLVM / python3 not available');
|
||||
|
||||
test(
|
||||
'a swallowed failure on the method\'s own busy state fires, and only it',
|
||||
() {
|
||||
// startGuarded resolves the state, startNested's try is a nested local
|
||||
// fallback, and Reporter::run sets a field of a local event object.
|
||||
expect(hits(ruleLoadingNotReset).map((f) => f.data['member']).toList(), [
|
||||
'startSwallowed',
|
||||
]);
|
||||
},
|
||||
skip: available ? false : 'Homebrew LLVM / python3 not available',
|
||||
);
|
||||
}
|
||||
104
test/fixtures/cpp_state/src/engine.cpp
vendored
Normal file
104
test/fixtures/cpp_state/src/engine.cpp
vendored
Normal file
|
|
@ -0,0 +1,104 @@
|
|||
// Fixture for tool/cpp_front.py: each case is a positive or negative control
|
||||
// for a rule run on the C++ state model (see test/cpp_front_test.dart).
|
||||
|
||||
namespace fx {
|
||||
|
||||
// Ghost is compared against but never produced: rendered-not-emitted.
|
||||
enum class Phase { Idle, Starting, Running, Failed, Ghost };
|
||||
|
||||
// Produced, never compared or switched on: emitted-not-rendered.
|
||||
enum class Unread { A, B };
|
||||
|
||||
// Y is compared against and produced only through a cast from an int, so the
|
||||
// rule must suppress itself rather than call the branch dead.
|
||||
enum class Casted { X, Y };
|
||||
|
||||
void mayThrow();
|
||||
|
||||
class Engine {
|
||||
public:
|
||||
// Negative: the catch resolves the busy state.
|
||||
void startGuarded() {
|
||||
phase_ = Phase::Starting;
|
||||
try {
|
||||
mayThrow();
|
||||
phase_ = Phase::Running;
|
||||
} catch (...) {
|
||||
phase_ = Phase::Failed;
|
||||
}
|
||||
}
|
||||
|
||||
// Positive: busy state set, then a top-level try whose catch swallows the
|
||||
// failure, so Starting can outlive it (loading-not-reset).
|
||||
void startSwallowed() {
|
||||
phase_ = Phase::Starting;
|
||||
try {
|
||||
mayThrow();
|
||||
phase_ = Phase::Running;
|
||||
} catch (...) {
|
||||
}
|
||||
}
|
||||
|
||||
// Negative: a nested try is a local fallback, not the method's failure path.
|
||||
void startNested() {
|
||||
phase_ = Phase::Starting;
|
||||
if (ready_) {
|
||||
try {
|
||||
mayThrow();
|
||||
} catch (...) {
|
||||
}
|
||||
}
|
||||
phase_ = Phase::Running;
|
||||
}
|
||||
|
||||
bool ghost() const { return phase_ == Phase::Ghost; }
|
||||
|
||||
bool idle() const {
|
||||
switch (phase_) {
|
||||
case Phase::Idle:
|
||||
return true;
|
||||
case Phase::Starting:
|
||||
case Phase::Running:
|
||||
case Phase::Failed:
|
||||
return false;
|
||||
default:
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
void setUnread() {
|
||||
unread_ = Unread::A;
|
||||
unread_ = Unread::B;
|
||||
}
|
||||
|
||||
void load(int n) { casted_ = static_cast<Casted>(n); }
|
||||
bool isY() const { return casted_ == Casted::Y; }
|
||||
|
||||
private:
|
||||
bool ready_ = false;
|
||||
Phase phase_ = Phase::Idle;
|
||||
Unread unread_ = Unread::A;
|
||||
Casted casted_ = Casted::X;
|
||||
};
|
||||
|
||||
// Negative: `ev.stage` is a field of a local event object handed to a callback,
|
||||
// not state Reporter holds, so its try cannot leave Reporter stuck.
|
||||
struct Event {
|
||||
Phase stage;
|
||||
};
|
||||
void notify(const Event&);
|
||||
|
||||
class Reporter {
|
||||
public:
|
||||
void run() {
|
||||
Event ev;
|
||||
ev.stage = Phase::Starting;
|
||||
notify(ev);
|
||||
try {
|
||||
mayThrow();
|
||||
} catch (...) {
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
} // namespace fx
|
||||
604
tool/cpp_front.py
Normal file
604
tool/cpp_front.py
Normal file
|
|
@ -0,0 +1,604 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Experimental C++ front end: writes a pipe state model from clang's AST.
|
||||
|
||||
Usage:
|
||||
tool/cpp_front.py <project-root> [--build-dir DIR] [--include src,include]
|
||||
[--limit N] [--jobs N] [-o model.json]
|
||||
pipe check --model model.json
|
||||
|
||||
pipe's rules read only the state model, so this front end never touches them.
|
||||
It uses libclang's *resolved* AST: every enumerator reference is linked to its
|
||||
EnumConstantDecl, so identity is exact rather than matched by name.
|
||||
|
||||
What it maps (scoped enums only, `enum class`, declared in the project):
|
||||
|
||||
union an `enum class`; variants are its enumerators
|
||||
emission a reference that PRODUCES the value: assigned to a field or
|
||||
variable, stored into an atomic, passed, returned, initialised
|
||||
observation a reference that EXAMINES the value: a `case` label, `==`, `!=`
|
||||
(these fill pipe's "rendered" set: something branches on it)
|
||||
swallowed a member writes a busy enumerator (Starting, Loading, ...) to a
|
||||
failure field, then has a try whose catch writes no state and does not
|
||||
rethrow (loading-not-reset)
|
||||
|
||||
Consequences worth knowing before reading the findings:
|
||||
- `emitted-not-rendered` here means "produced, but nothing ever branches on or
|
||||
compares it". A value that reaches the user through a lookup table or
|
||||
reflection (magic_enum) is invisible to this, so treat those as leads.
|
||||
- `rendered-not-emitted` means "compared against, but never produced anywhere":
|
||||
a dead branch, unless the value comes from a cast (`static_cast<E>(n)`), which
|
||||
is recorded as an indirect emission so the rule suppresses itself.
|
||||
- The rule messages still speak of "UI"; for C++ read "nothing observes it".
|
||||
|
||||
Requires Homebrew LLVM's libclang and its Python bindings (found automatically).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import glob
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shlex
|
||||
import subprocess
|
||||
import sys
|
||||
from collections import Counter, defaultdict
|
||||
from concurrent.futures import ProcessPoolExecutor, as_completed
|
||||
from pathlib import Path
|
||||
|
||||
LLVM = Path(os.environ.get("PIPE_LLVM", "/opt/homebrew/opt/llvm"))
|
||||
|
||||
|
||||
def _load_cindex():
|
||||
for candidate in sorted(glob.glob(str(LLVM / "lib/python3.*/site-packages")), reverse=True):
|
||||
if candidate not in sys.path:
|
||||
sys.path.insert(0, candidate)
|
||||
import clang.cindex as cindex # noqa: E402
|
||||
|
||||
# Pool workers are reused, and libclang may be configured only once.
|
||||
if not cindex.Config.loaded:
|
||||
cindex.Config.set_library_file(str(LLVM / "lib/libclang.dylib"))
|
||||
return cindex
|
||||
|
||||
|
||||
_SDK = None
|
||||
|
||||
BUSY = re.compile(
|
||||
r"^(Loading|Starting|StartingUp|Initializing|Initialising|Connecting|Stopping|"
|
||||
r"ShuttingDown|Busy|Pending|InProgress|Processing|Saving|Updating|Creating|"
|
||||
r"Syncing|Indexing|Warming|WarmingUp|Draining|Migrating|Recovering)$"
|
||||
)
|
||||
|
||||
# Kinds that only wrap an expression; the classifier looks through them.
|
||||
_TRANSPARENT = {
|
||||
"UNEXPOSED_EXPR",
|
||||
"PAREN_EXPR",
|
||||
"CSTYLE_CAST_EXPR",
|
||||
"CXX_FUNCTIONAL_CAST_EXPR",
|
||||
"CONDITIONAL_OPERATOR",
|
||||
"INIT_LIST_EXPR",
|
||||
}
|
||||
|
||||
|
||||
def _resource_dir() -> str:
|
||||
return subprocess.check_output([str(LLVM / "bin/clang"), "-print-resource-dir"], text=True).strip()
|
||||
|
||||
|
||||
def _sdk_root() -> str | None:
|
||||
"""Apple's compiler finds the macOS SDK itself; libclang needs it spelled out,
|
||||
or every standard header (<chrono>, sqlite3.h) is missing."""
|
||||
try:
|
||||
return subprocess.check_output(["xcrun", "--show-sdk-path"], text=True).strip()
|
||||
except (OSError, subprocess.CalledProcessError):
|
||||
return None
|
||||
|
||||
|
||||
def _args_for(entry: dict, resource_dir: str) -> list[str]:
|
||||
"""Compiler flags only: drop wrappers (ccache), the compiler, the source,
|
||||
and output/dependency options that make no sense for a syntax-only parse."""
|
||||
args = entry.get("arguments") or shlex.split(entry["command"])
|
||||
while args and (os.path.basename(args[0]) in ("ccache", "sccache", "distcc")
|
||||
or not args[0].startswith("-") and args[0] != entry["file"]
|
||||
and os.path.basename(args[0]) in ("c++", "g++", "clang++", "cc", "gcc", "clang")):
|
||||
args = args[1:]
|
||||
out: list[str] = []
|
||||
skip = False
|
||||
directory = entry["directory"]
|
||||
# Include paths are relative to the build directory, but libclang resolves
|
||||
# them against this process's working directory: made absolute, or headers
|
||||
# like third_party/... silently go missing.
|
||||
path_flags = ("-I", "-isystem", "-iquote", "-idirafter", "-include")
|
||||
absolute = []
|
||||
pending = None
|
||||
for arg in args:
|
||||
if pending is not None:
|
||||
absolute.append(arg if os.path.isabs(arg) else os.path.normpath(os.path.join(directory, arg)))
|
||||
pending = None
|
||||
continue
|
||||
if arg in path_flags:
|
||||
absolute.append(arg)
|
||||
pending = arg
|
||||
continue
|
||||
for flag in path_flags:
|
||||
if arg.startswith(flag) and len(arg) > len(flag) and flag != "-include":
|
||||
value = arg[len(flag):]
|
||||
if not os.path.isabs(value):
|
||||
arg = flag + os.path.normpath(os.path.join(directory, value))
|
||||
break
|
||||
absolute.append(arg)
|
||||
args = absolute
|
||||
for arg in args:
|
||||
if skip:
|
||||
skip = False
|
||||
continue
|
||||
if arg in ("-o", "-MF", "-MT", "-MQ"):
|
||||
skip = True
|
||||
continue
|
||||
if arg in ("-c", "-MD", "-MMD", "-fdiagnostics-color=always") or \
|
||||
arg.startswith(("-o", "-MF", "-MT", "-MQ")):
|
||||
continue
|
||||
if arg == entry["file"] or not arg.startswith("-") and arg.endswith((".cpp", ".cc", ".cxx")):
|
||||
continue
|
||||
out.append(arg)
|
||||
out += ["-resource-dir", resource_dir, "-Wno-everything", "-fsyntax-only"]
|
||||
sdk = _SDK
|
||||
if sdk and "-isysroot" not in out:
|
||||
out += ["-isysroot", sdk]
|
||||
return out
|
||||
|
||||
|
||||
def _qualified(cursor) -> str:
|
||||
parts = []
|
||||
c = cursor
|
||||
while c is not None and c.kind.name != "TRANSLATION_UNIT":
|
||||
if c.spelling:
|
||||
parts.append(c.spelling)
|
||||
c = c.semantic_parent
|
||||
return "::".join(reversed(parts))
|
||||
|
||||
|
||||
def _enclosing(stack, kinds):
|
||||
for c in reversed(stack):
|
||||
if c.kind.name in kinds:
|
||||
return c
|
||||
return None
|
||||
|
||||
|
||||
def _operator(cursor) -> str | None:
|
||||
try:
|
||||
op = cursor.binary_operator
|
||||
return op.name if op is not None else None
|
||||
except AttributeError:
|
||||
return None
|
||||
|
||||
|
||||
def _top_level_try(stack) -> bool:
|
||||
"""True when the try sits directly in its function's body, not nested in a
|
||||
loop, branch, block or another try."""
|
||||
for i in range(len(stack) - 1, -1, -1):
|
||||
kind = stack[i].kind.name
|
||||
if kind in ("CXX_METHOD", "FUNCTION_DECL", "CONSTRUCTOR", "LAMBDA_EXPR"):
|
||||
# stack[i+1] is the body; the try must be its direct child.
|
||||
return len(stack) - 1 == i + 1 and stack[i + 1].kind.name == "COMPOUND_STMT"
|
||||
if kind == "CXX_TRY_STMT":
|
||||
return False
|
||||
return False
|
||||
|
||||
|
||||
def _field_owner(expr, cindex):
|
||||
"""For `x.state_` / `state_` / `this->state_`: the class declaring the field."""
|
||||
for node in expr.walk_preorder():
|
||||
if node.kind.name == "MEMBER_REF_EXPR":
|
||||
ref = node.referenced
|
||||
if ref is not None and ref.kind.name == "FIELD_DECL":
|
||||
return ref.semantic_parent, ref.spelling
|
||||
return None, None
|
||||
|
||||
|
||||
def analyse_tu(task):
|
||||
"""Parse one translation unit; return the facts located in project files."""
|
||||
file, directory, args, root, includes = task
|
||||
cindex = _load_cindex()
|
||||
index = cindex.Index.create()
|
||||
try:
|
||||
tu = index.parse(
|
||||
os.path.normpath(os.path.join(directory, file)),
|
||||
args=args,
|
||||
options=cindex.TranslationUnit.PARSE_INCOMPLETE,
|
||||
)
|
||||
except cindex.TranslationUnitLoadError as error:
|
||||
return {"file": file, "error": str(error)}
|
||||
|
||||
root = os.path.realpath(root)
|
||||
prefixes = tuple(os.path.join(root, p) + os.sep for p in includes)
|
||||
|
||||
def in_project(cursor) -> str | None:
|
||||
loc = cursor.location
|
||||
if loc.file is None:
|
||||
return None
|
||||
path = os.path.realpath(loc.file.name)
|
||||
if not path.startswith(prefixes):
|
||||
return None
|
||||
return os.path.relpath(path, root)
|
||||
|
||||
enums = {}
|
||||
refs = []
|
||||
busy_tries = []
|
||||
|
||||
def classify(stack, ref):
|
||||
"""Return (role, owner, field, kind) for an enumerator reference."""
|
||||
child = ref
|
||||
for parent in reversed(stack):
|
||||
kind = parent.kind.name
|
||||
if kind in _TRANSPARENT:
|
||||
child = parent
|
||||
continue
|
||||
if kind == "CASE_STMT":
|
||||
first = next(iter(parent.get_children()), None)
|
||||
if first is not None and first.hash == child.hash:
|
||||
return ("observe", None, None, "case")
|
||||
return ("produce", None, None, "other")
|
||||
if kind == "BINARY_OPERATOR":
|
||||
op = _operator(parent)
|
||||
children = list(parent.get_children())
|
||||
if op in ("EQ", "NE"):
|
||||
return ("observe", None, None, "compare")
|
||||
if op == "Assign" and len(children) == 2 and children[1].hash == child.hash:
|
||||
owner, field = _field_owner(children[0], cindex)
|
||||
return ("produce", owner, field, "assign")
|
||||
return ("produce", None, None, "other")
|
||||
if kind == "CALL_EXPR" and parent.spelling in ("store", "exchange"):
|
||||
owner, field = _field_owner(parent, cindex)
|
||||
return ("produce", owner, field, "assign")
|
||||
if kind == "MEMBER_REF":
|
||||
ref_decl = parent.referenced
|
||||
if ref_decl is not None and ref_decl.kind.name == "FIELD_DECL":
|
||||
return ("produce", ref_decl.semantic_parent, ref_decl.spelling, "initial")
|
||||
if kind == "FIELD_DECL":
|
||||
return ("produce", parent.semantic_parent, parent.spelling, "initial")
|
||||
if kind in ("CALL_EXPR", "RETURN_STMT", "VAR_DECL", "COMPOUND_STMT",
|
||||
"IF_STMT", "CXX_METHOD", "FUNCTION_DECL", "CONSTRUCTOR"):
|
||||
return ("produce", None, None, "other")
|
||||
child = parent
|
||||
return ("produce", None, None, "other")
|
||||
|
||||
def member_name(stack):
|
||||
fn = _enclosing(stack, {"CXX_METHOD", "FUNCTION_DECL", "CONSTRUCTOR",
|
||||
"DESTRUCTOR", "FUNCTION_TEMPLATE", "LAMBDA_EXPR"})
|
||||
if fn is None:
|
||||
return "<top>", None
|
||||
if fn.kind.name == "LAMBDA_EXPR":
|
||||
outer = _enclosing(stack[: stack.index(fn)], {"CXX_METHOD", "FUNCTION_DECL", "CONSTRUCTOR"})
|
||||
name = outer.spelling if outer is not None else "<lambda>"
|
||||
return f"{name}::<lambda>", outer
|
||||
return fn.spelling, fn
|
||||
|
||||
def scan_try(stack, try_cursor, rel):
|
||||
"""A try inside a member that wrote a busy value earlier."""
|
||||
body = list(try_cursor.get_children())
|
||||
handlers = [c for c in body if c.kind.name == "CXX_CATCH_STMT"]
|
||||
swallowed = []
|
||||
for handler in handlers:
|
||||
writes_state = False
|
||||
rethrows = False
|
||||
for node in handler.walk_preorder():
|
||||
if node.kind.name == "CXX_THROW_EXPR":
|
||||
rethrows = True
|
||||
if node.kind.name == "DECL_REF_EXPR":
|
||||
ref = node.referenced
|
||||
if ref is not None and ref.kind.name == "ENUM_CONSTANT_DECL":
|
||||
writes_state = True
|
||||
if node.kind.name in ("CALL_EXPR",) and re.search(
|
||||
r"(set|mark|fail|error|transition|reset|clear)", node.spelling or "", re.I
|
||||
):
|
||||
writes_state = True
|
||||
if not writes_state and not rethrows:
|
||||
tokens = " ".join(t.spelling for t in handler.get_tokens())
|
||||
swallowed.append((handler.location.line, tokens[:117] + ("..." if len(tokens) > 117 else "")))
|
||||
return swallowed
|
||||
|
||||
def visit(cursor, stack, busy_seen):
|
||||
rel = in_project(cursor)
|
||||
if rel is None and cursor.kind.name != "TRANSLATION_UNIT":
|
||||
return
|
||||
kind = cursor.kind.name
|
||||
if kind == "ENUM_DECL" and cursor.is_scoped_enum() and cursor.is_definition():
|
||||
name = _qualified(cursor)
|
||||
enums[name] = {
|
||||
"name": name,
|
||||
"declaredIn": rel,
|
||||
"line": cursor.location.line,
|
||||
"variants": [c.spelling for c in cursor.get_children()
|
||||
if c.kind.name == "ENUM_CONSTANT_DECL"],
|
||||
}
|
||||
elif kind == "DECL_REF_EXPR":
|
||||
ref = cursor.referenced
|
||||
if ref is not None and ref.kind.name == "ENUM_CONSTANT_DECL":
|
||||
parent_enum = ref.semantic_parent
|
||||
if parent_enum is not None and parent_enum.is_scoped_enum():
|
||||
role, owner, field, how = classify(stack, cursor)
|
||||
member, fn = member_name(stack)
|
||||
klass = _enclosing(stack, {"CLASS_DECL", "STRUCT_DECL", "CLASS_TEMPLATE"})
|
||||
if fn is not None and fn.semantic_parent is not None and \
|
||||
fn.semantic_parent.kind.name in ("CLASS_DECL", "STRUCT_DECL", "CLASS_TEMPLATE"):
|
||||
klass = fn.semantic_parent
|
||||
record = {
|
||||
"enum": _qualified(parent_enum),
|
||||
"variant": ref.spelling,
|
||||
"role": role,
|
||||
"how": how,
|
||||
"owner": _qualified(owner) if owner is not None else None,
|
||||
"field": field,
|
||||
"member": member,
|
||||
"memberOwner": _qualified(klass) if klass is not None else None,
|
||||
"path": rel,
|
||||
"line": cursor.location.line,
|
||||
"col": cursor.location.column,
|
||||
}
|
||||
refs.append(record)
|
||||
# Only the method's OWN state counts: `ev.stage = Connecting` on a
|
||||
# local event object handed to a callback is not state this
|
||||
# class can leave stuck.
|
||||
if role == "produce" and how == "assign" and BUSY.match(ref.spelling) \
|
||||
and owner is not None and klass is not None \
|
||||
and _qualified(owner) == _qualified(klass):
|
||||
busy_seen.append(record)
|
||||
elif kind == "CXX_STATIC_CAST_EXPR" or kind == "CSTYLE_CAST_EXPR":
|
||||
target = cursor.type.get_canonical()
|
||||
decl = target.get_declaration()
|
||||
if decl is not None and decl.kind.name == "ENUM_DECL" and decl.is_scoped_enum():
|
||||
children = list(cursor.get_children())
|
||||
if children and children[-1].type.get_canonical().kind.name != "ENUM":
|
||||
refs.append({
|
||||
"enum": _qualified(decl), "variant": None, "role": "cast", "how": "cast",
|
||||
"owner": None, "field": None, "member": member_name(stack)[0],
|
||||
"memberOwner": None, "path": rel, "line": cursor.location.line,
|
||||
"col": cursor.location.column,
|
||||
})
|
||||
elif kind == "CXX_TRY_STMT" and busy_seen and _top_level_try(stack):
|
||||
# (a top-level try of the method: nested tries are local fallbacks,
|
||||
# like a JSON parse guarded with `catch (...) { skip = false; }`)
|
||||
member, fn = member_name(stack)
|
||||
for busy in busy_seen:
|
||||
if busy["member"] != member:
|
||||
continue
|
||||
for line, handler in scan_try(stack, cursor, rel):
|
||||
busy_tries.append({
|
||||
"owner": busy["owner"], "member": member, "path": rel,
|
||||
"line": line, "handler": handler,
|
||||
})
|
||||
break
|
||||
new_busy = busy_seen
|
||||
if kind in ("CXX_METHOD", "FUNCTION_DECL", "CONSTRUCTOR", "LAMBDA_EXPR"):
|
||||
new_busy = []
|
||||
stack.append(cursor)
|
||||
for child in cursor.get_children():
|
||||
visit(child, stack, new_busy)
|
||||
stack.pop()
|
||||
|
||||
visit(tu.cursor, [], [])
|
||||
errors = [f"{d.location.file}:{d.location.line}: {d.spelling}"
|
||||
for d in tu.diagnostics if d.severity >= 3]
|
||||
return {"file": file, "enums": list(enums.values()), "refs": refs, "tries": busy_tries,
|
||||
"errors": len(errors), "errorSamples": errors[:3]}
|
||||
|
||||
|
||||
def build_model(root: Path, results: list[dict], includes: list[str]) -> dict:
|
||||
enums: dict[str, dict] = {}
|
||||
seen_refs = set()
|
||||
refs = []
|
||||
tries = {}
|
||||
failed = []
|
||||
for result in results:
|
||||
if "error" in result:
|
||||
failed.append(result["file"])
|
||||
continue
|
||||
for enum in result["enums"]:
|
||||
enums.setdefault(enum["name"], enum)
|
||||
for ref in result["refs"]:
|
||||
key = (ref["path"], ref["line"], ref["col"], ref["enum"], ref["variant"])
|
||||
if key in seen_refs:
|
||||
continue
|
||||
seen_refs.add(key)
|
||||
refs.append(ref)
|
||||
for t in result["tries"]:
|
||||
tries.setdefault((t["path"], t["line"]), t)
|
||||
|
||||
by_enum = defaultdict(list)
|
||||
for ref in refs:
|
||||
by_enum[ref["enum"]].append(ref)
|
||||
|
||||
unions, coverage, emissions, renders = [], [], [], []
|
||||
for name, enum in sorted(enums.items()):
|
||||
if len(enum["variants"]) < 2:
|
||||
continue
|
||||
erefs = by_enum.get(name, [])
|
||||
owners = Counter(r["owner"] for r in erefs if r["role"] == "produce" and r["owner"])
|
||||
# The class whose field holds this state; without one it is a plain value
|
||||
# type (a mode flag, a return code), not state anyone could leave stale.
|
||||
if not owners:
|
||||
continue
|
||||
owner = owners.most_common(1)[0][0]
|
||||
variants = enum["variants"]
|
||||
emitted, initial, observed = set(), set(), set()
|
||||
observe_sites = 0
|
||||
emit_sites = 0
|
||||
for r in erefs:
|
||||
if r["role"] == "observe":
|
||||
observed.add(r["variant"])
|
||||
observe_sites += 1
|
||||
renders.append({
|
||||
"path": r["path"], "line": r["line"], "widget": r["how"],
|
||||
"stateType": name,
|
||||
"branches": [{"variant": r["variant"], "line": r["line"], "trivial": False}],
|
||||
})
|
||||
elif r["role"] in ("produce", "cast"):
|
||||
emit_sites += 1
|
||||
if r["variant"]:
|
||||
emitted.add(r["variant"])
|
||||
if r["how"] == "initial":
|
||||
initial.add(r["variant"])
|
||||
emissions.append({
|
||||
"path": r["path"], "line": r["line"], "owner": owner,
|
||||
"member": r["member"], "kind": "assign" if r["how"] == "assign" else "emit",
|
||||
"confidence": "exact" if r["role"] == "produce" else "indirect",
|
||||
"variants": [r["variant"]] if r["variant"] else [],
|
||||
"snippet": f"{r['field'] or ''} = {name.split('::')[-1]}::{r['variant']}"
|
||||
if r["variant"] else f"static_cast<{name.split('::')[-1]}>(...)",
|
||||
# A class holds several enums: name the union so a cast to one
|
||||
# does not make "never emitted" unprovable for the others.
|
||||
"stateType": name,
|
||||
})
|
||||
union = {
|
||||
"name": name, "declaredIn": enum["declaredIn"], "style": "enumUnion",
|
||||
"owner": owner, "ownerKind": "Cubit",
|
||||
"variants": [{"name": v, "declaredIn": enum["declaredIn"]} for v in variants],
|
||||
}
|
||||
unions.append(union)
|
||||
coverage.append({
|
||||
"union": name, "owner": owner, "style": "enumUnion",
|
||||
"declared": sorted(variants), "emitted": sorted(emitted), "initial": sorted(initial),
|
||||
"rendered": sorted(observed), "renderedInBuilder": [], "trivialInBuilder": [],
|
||||
"emissionSites": emit_sites, "renderSites": observe_sites,
|
||||
})
|
||||
|
||||
owners = {u["owner"] for u in unions}
|
||||
swallowed = []
|
||||
ordinals = Counter()
|
||||
for (path, line), t in sorted(tries.items()):
|
||||
if t["owner"] not in owners:
|
||||
continue
|
||||
ordinals[(t["owner"], t["member"])] += 1
|
||||
swallowed.append({
|
||||
"owner": t["owner"], "member": t["member"], "path": path, "line": line,
|
||||
"ordinal": ordinals[(t["owner"], t["member"])], "handler": t["handler"],
|
||||
})
|
||||
|
||||
# Files outside an owner's own declaration that mention it: pipe uses these
|
||||
# to tell a dormant (unwired) union from a partly observed one.
|
||||
references = defaultdict(set)
|
||||
for r in refs:
|
||||
if r["role"] == "observe" and r["enum"] in {u["name"] for u in unions}:
|
||||
owner = next(u["owner"] for u in unions if u["name"] == r["enum"])
|
||||
references[owner].add(r["path"])
|
||||
|
||||
return {
|
||||
"root": str(root),
|
||||
"directories": includes,
|
||||
"frontEnd": {"name": "pipe-cpp-front (experimental)", "failedTranslationUnits": failed},
|
||||
"unions": unions,
|
||||
"eventUnions": [],
|
||||
"emissions": emissions,
|
||||
"renders": renders,
|
||||
"handlers": [],
|
||||
"dispatches": [],
|
||||
"coverage": coverage,
|
||||
"ownerReferences": {k: sorted(v) for k, v in references.items()},
|
||||
"flagStates": [],
|
||||
"riverpod": {"providers": [], "sites": []},
|
||||
"constructedByFile": {},
|
||||
"swallowedFailures": swallowed,
|
||||
"unsettledFlags": [],
|
||||
}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
parser.add_argument("root")
|
||||
parser.add_argument("--build-dir", help="directory holding compile_commands.json")
|
||||
parser.add_argument("--include", default="src,include",
|
||||
help="project subdirectories whose code is analysed")
|
||||
parser.add_argument("--limit", type=int)
|
||||
parser.add_argument("--match", help="only translation units whose path contains this")
|
||||
parser.add_argument("--jobs", type=int, default=os.cpu_count())
|
||||
parser.add_argument("-o", "--output", default="-")
|
||||
args = parser.parse_args()
|
||||
|
||||
root = Path(args.root).resolve()
|
||||
db_dir = Path(args.build_dir) if args.build_dir else root
|
||||
db_path = db_dir / "compile_commands.json"
|
||||
if not db_path.is_file():
|
||||
print(f"cpp-front: no compile database at {db_path}; generate one "
|
||||
f"(meson/CMake write it to the build dir) and pass --build-dir",
|
||||
file=sys.stderr)
|
||||
return 2
|
||||
database = json.loads(db_path.read_text())
|
||||
includes = [p for p in args.include.split(",") if p]
|
||||
resource_dir = _resource_dir()
|
||||
global _SDK
|
||||
_SDK = _sdk_root()
|
||||
|
||||
tasks = []
|
||||
seen = set()
|
||||
for entry in database:
|
||||
# Resolved on both sides: a symlinked path (macOS /var -> /private/var)
|
||||
# otherwise lands "outside" the resolved root and the TU is skipped.
|
||||
path = os.path.realpath(os.path.join(entry["directory"], entry["file"]))
|
||||
if path in seen or not path.endswith((".cpp", ".cc", ".cxx")):
|
||||
continue
|
||||
rel = os.path.relpath(path, root)
|
||||
if not any(rel.startswith(p + os.sep) for p in includes):
|
||||
continue
|
||||
if args.match and args.match not in rel:
|
||||
continue
|
||||
seen.add(path)
|
||||
tasks.append((entry["file"], entry["directory"], _args_for(entry, resource_dir),
|
||||
str(root), includes))
|
||||
if args.limit:
|
||||
tasks = tasks[: args.limit]
|
||||
if not tasks:
|
||||
# An empty model reads as a clean project; refuse instead.
|
||||
print(f"cpp-front: no translation unit under {', '.join(includes)} in "
|
||||
f"{db_dir / 'compile_commands.json'}; nothing was analysed", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
results = []
|
||||
with ProcessPoolExecutor(max_workers=args.jobs) as pool:
|
||||
futures = [pool.submit(analyse_tu, t) for t in tasks]
|
||||
for i, future in enumerate(as_completed(futures), 1):
|
||||
results.append(future.result())
|
||||
if i % 25 == 0 or i == len(tasks):
|
||||
print(f"cpp-front: {i}/{len(tasks)} translation units", file=sys.stderr)
|
||||
|
||||
model = build_model(root, results, includes)
|
||||
|
||||
# Sources the build does not compile (optional backends, benchmarks) were
|
||||
# never analysed, so evidence in them is missing. Recorded as unresolved with
|
||||
# the names they mention, pipe lowers confidence for any finding naming them.
|
||||
unanalysed = []
|
||||
names: set[str] = set()
|
||||
for include in includes:
|
||||
for path in sorted((root / include).rglob("*")):
|
||||
if path.suffix not in (".cpp", ".cc", ".cxx") or os.path.realpath(path) in seen:
|
||||
continue
|
||||
unanalysed.append(os.path.relpath(path, root))
|
||||
names.update(re.findall(r"[A-Za-z_][A-Za-z0-9_]*", path.read_text(errors="replace")))
|
||||
failed_files = [os.path.normpath(f) for f in model["frontEnd"]["failedTranslationUnits"]]
|
||||
if unanalysed or failed_files:
|
||||
files = unanalysed + failed_files
|
||||
model["unresolved"] = {
|
||||
"files": files,
|
||||
"analysed": len(tasks) - len(failed_files),
|
||||
"first": f"{files[0]}: not compiled by this build, so not analysed",
|
||||
"names": sorted(names),
|
||||
}
|
||||
failed = model["frontEnd"]["failedTranslationUnits"]
|
||||
errors = sum(r.get("errors", 0) for r in results)
|
||||
for r in results:
|
||||
for sample in r.get("errorSamples", [])[:1]:
|
||||
print(f"cpp-front: clang error in {r['file']}: {sample}", file=sys.stderr)
|
||||
print(f"cpp-front: {len(model['unions'])} state enums, {len(model['emissions'])} "
|
||||
f"emissions, {len(model['renders'])} observations, "
|
||||
f"{len(model['swallowedFailures'])} swallowed failures; "
|
||||
f"{len(failed)} TU(s) failed, {errors} clang error(s)", file=sys.stderr)
|
||||
text = json.dumps(model, indent=2)
|
||||
if args.output == "-":
|
||||
print(text)
|
||||
else:
|
||||
Path(args.output).write_text(text)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
Loading…
Add table
Add a link
Reference in a new issue