SessionService reads the current session from XDG_STATE_HOME. On a build
host whose user had an auto-created session ("auto: /data") with watch
enabled, every test that booted a ServiceManager inherited it, the session
watcher walked /data during init, and six daemon tests timed out. With the
state dir isolated the same six pass.
The shared test main now points XDG_STATE_HOME at a per-process temp dir
when the caller did not set one, and removes it at exit. A small test checks
the process sees no host session.
Signed-off-by: trvon <me@trevon.dev>
* feat(profiling): Tracy zones on p2p and memory-sync background paths
Zone the memory-sync cycle (publish, sync, reconcile, apply, backfill,
delete-outbox drain), the p2p reconnect cycle, accept loop, per-session
worker and frame read, and the delta send/apply/bootstrap stages. Plot the
retained session count and pending erase intents. No behavior change; the
macros are no-ops without Tracy.
Signed-off-by: trvon <me@trevon.dev>
* test(infra): per-test-case CPU and peak RSS accounting for Catch2 binaries
A Catch2 listener in the shared test main records getrusage (GetProcessTimes
on Windows) CPU per case and prints the five heaviest cases plus the run's
peak RSS to stderr at run end. YAMS_TEST_RUSAGE=1 adds one line per case.
The measurement helpers are header-only and unit tested.
Signed-off-by: trvon <me@trevon.dev>
* fix(vector): make vector_database.cpp compile with Tracy enabled
The Profiling lane did not build: eight YAMS_PLOT sites in
vector_database.cpp used -> on a Result<T>, and two passed an int that is
ambiguous between Tracy's PlotData overloads. Use .value() and int64_t
casts. No behavior change outside TRACY_ENABLE builds.
Signed-off-by: trvon <me@trevon.dev>
* build: enable Tracy for every yams library in the Profiling lane
Define yams_tracy_args/yams_tracy_deps once in the root meson.build
(Tracy found, optional rocprofiler hooks, link check) and apply them to
every library and executable under src/, replacing the per-library copies
in search, topology, vector, daemon and tests/benchmarks. Empty when
enable-profiling is off.
Make YAMS_PLOT accept any numeric type and fix a Result deref in
search_service so the newly instrumented TUs compile with Tracy.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
With 5 x 10 MB rotation the daemon log covered about three hours on the
audited machine, mostly one of a few lines repeated:
- "Failed to submit extraction ...: entity_graph_jobs channel full"
(11k lines) from GraphComponent, echoed per batch by PostIngestQueue's
"KG batch failed", plus "Entity channel full, dropping job";
- the per-document "Title+NL extraction for <hash> in N ms" info line;
- session-watcher scan/indexing warnings retried every pass.
Add yams::common::LogRateLimiter, a small lock-free helper that admits
one message per interval and reports how many it held back ("(N similar
messages suppressed)"), and use it at those sites (60 s for the queue
paths, 10 min for the session watcher, including its first-failure
warning per path). The Title+NL timing line is per-document progress
and moves to debug. The drop counters are unchanged, so status still
shows the full numbers.
Signed-off-by: trvon <me@trevon.dev>
sanitizeUtf8 and ensureValidUtf8 only checked continuation-byte bits, so
overlong encodings (E0 80 80), UTF-16 surrogates (ED A0 80) and values
above U+10FFFF (F4 90 80 80) passed through. They exist to satisfy
proto3 string fields (proto_serializer.cpp calls them for every string),
which reject exactly those sequences, so a document containing them could
make a peer fail to parse the message. Share one well-formedness check
with sanitizeUtf8Strict; the lenient variant still replaces bytes with
'?'.
* refactor(daemon): one table for repair operation wire codes
The code published under repair_current_operation_code was defined in
three places: a name->code chain in RepairService.cpp, an enum->name
switch beside it, and a code->label switch in the CLI. The two daemon
tables agreed with each other but the CLI copy had already drifted in
wording, and nothing tied the three together.
metric_keys.h now carries the single code<->name table next to the
metric key it describes, with constexpr lookups in both directions.
RepairService's name->code chain delegates to it; the enum->name
switch stays because it is keyed by the private enum. The CLI copy
goes in the next commit with the status renderer.
Signed-off-by: trvon <me@trevon.dev>
* refactor(cli): extract the daemon status renderer
showStatus() in daemon_command.cpp was 1,762 lines: two RPCs, a
bootstrap-file fallback, and two ~400/1,100-line render bodies that
wrote straight to std::cout from inside the command class, reachable
only through a live daemon. The render bodies read nothing from the
command except the expected data directory, the explicit-socket flag,
and a memory-sync fetch they triggered mid-render.
The bodies now live in src/cli/commands/daemon_status_render.cpp as
free functions over StatusResponse plus a small DaemonStatusRenderContext
(configured data dir, explicit-socket flag, optional MemorySyncResponse)
writing to an ostream. The readiness/label helpers that only they used
(humanizeToken, classifyReadiness, isEphemeralDataDir,
daemonDataDirFromStatus, paintStatus, neutralText) move with them.
DaemonCommand keeps the RPCs and the bootstrap fallback; the memory-sync
fetch becomes fetchMemorySyncStatus() and runs just before rendering
instead of in the middle of it, so the section prints in the same place
with the same bounded 3 s timeout.
The move is verbatim: after the mechanical substitutions
(std::cout -> os, expectedDataDir() -> ctx.configuredDataDir, the
memory-sync call, the explicit-socket flag, and the repair label
function) the two bodies match the originals character for character
modulo whitespace. The CLI's private repair-operation switch is gone;
the label now comes from the shared table in metric_keys.h with
underscores rendered as spaces, which reproduces the previous wording
("stuck docs", "block refs", "path tree").
tests/unit/cli/daemon_status_render_catch2_test.cpp drives both
renderers and the memory-sync section from synthetic responses:
overview rows, blockers with init progress, last error, data-dir
mismatch (and its suppression under --socket), temporary data dir,
the active repair operation label, every detailed section, and the
started/not-started memory-sync gate.
daemon_command.cpp: 4,082 -> 2,341 lines; daemon_status_render.cpp:
1,642 lines.
Signed-off-by: trvon <me@trevon.dev>
* refactor(app): one home for hash predicates, path aliasing, and metadata qualifiers
Three small helpers had grown parallel copies with quietly different
rules:
- "does this look like a hash" existed four times: search_service
(8-64 hex), document_service (6-63), cat_command (6-64), plus a
never-defined utils::looksLikeHash declaration in services.hpp. The
8 vs 6 threshold is deliberate (D3): a bare query token must reach
8 hex characters before it stops being text, while an explicit hash
argument may be 6. yams/common/hash_predicates.h now names the two
predicates apart (looksLikeHashQueryToken, looksLikePartialHashArgument)
over one isHexDigits; document_service's 6..63 bound was deliberate
for the name resolver (a full 64-hex name must not resolve as a prefix
and delete by hash), so its two resolver sites now guard size() != 64
explicitly, as the partial-hash resolution site already did.
- macOS /private aliasing had three incompatible policies: search
appended the alias of /var only, grep canonicalized /var and /tmp
to the /private spelling, and the delete resolver displayed
/private/var as /var. yams/common/fs_utils.h now carries one alias
table ({/var,/tmp} <-> /private/...) with three lexical policies over
it (otherSpelling, canonicalSpelling, displaySpelling) and the
platform-gated wrappers callers use. Declared behavior change, macOS
only: search patterns under /tmp now also match their /private/tmp
spelling, and the resolver displays /private/tmp/... as /tmp/....
- Inline key=value metadata filters were parsed twice: SearchService's
extractStructuredMetadataQuery (exactly one '=', no quoting/grouping
characters, deduplicated by key) and SearchEngine's
parseMetadataFiltersFromQuery (any '=' token, duplicates kept). The
stricter, tested parser moves into yams/search/query_qualifiers.hpp
and both call it. Declared behavior change for the hybrid metadata
component: tokens like name="x" or a=b=c stay in the query instead
of becoming filters, and repeated keys collapse to the last value.
Tests: hash_predicates_catch2_test.cpp pins both thresholds and the
64-character bound; path_alias_catch2_test.cpp pins the alias table,
the three policies, and the platform gating; query_qualifiers_catch2_test.cpp
gains the structured-metadata cases (split, dedupe, malformed tokens,
filters-only). search_service, document_service, grep_service,
service_utilities, and resolve_submodule pass unchanged. Portability
allowlist anchors reflowed for the four files whose lines moved.
Signed-off-by: trvon <me@trevon.dev>
* fix(cli): flush status before optional collection
Render useful primary status before waiting for memory-sync details.
Keep both brief and detailed paths behind the tested ordering seam.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
Profiling running daemon: bit-by-bit CRC32 top CPU consumer (2289
samples/15s during grep). Same algorithm/polynomial, byte-identical
output (oracle test 0..512 len), stored checksums stay valid.
Applies slice-by-8 everywhere crc ran:
- yams::common::crc32 (+ crc32Update seeded variant for mid-stream)
- compression::calculateCRC32/updateCRC32 delegate to it
- storage-local duplicate copy removed (dead scope)
- WAL entry checksum copy removed
Also restructures sizeof arithmetic in WAL size math so opengrep does
not misread it as pointer arithmetic.
The same string helpers were copy-pasted across CLI and service code:
trim/trimCopy in 5 files, escapeRegex in 3, hasWildcard in 2, and the
regex meta-char table in 2. Move canonical versions to
yams::common::string_utils.h (trimCopy, hasWildcard, escapeRegex,
isRegexMetaChar) and wire all call sites to them; daemon_helpers'
trim_copy now delegates. Add a Catch2 test for the shared helpers.
Kept isTransientMetadataError and annotateResultRelations untouched:
they have semantically different implementations and need a separate
review, not a mechanical merge.
Honor root-anchored and slash-qualified ignore patterns during live fallback. Center hydrated snippets on query matches and preserve symbol ranking through the daemon boundary.
