`meson install` of a Release tree staged 2.4 GiB: all 28 internal static
libraries (1.3 GiB), the whole include/ tree (459 files), 22 pkg-config
files for libraries that were never meant to be linked from outside, and
the ONNX resource library in the system libdir. Packages hid this by
deleting include/, *.a and pkgconfig/ after the fact.
- Internal static libraries are `install: false` and their pkg.generate()
calls are gone, as is the top-level yams.pc (it required yams-api, which
is not installable). Benchmarks are no longer installed either.
- Headers are not installed: no installable C++ library exists to link
them against. The one SDK surface, the mobile C ABI, installs
mobile_bindings.h and its .pc under the 'devel' install tag.
- Runtime installs carry explicit install_tag: 'runtime' where Meson
cannot infer one (data, systemd unit/preset, the yams alias symlink,
Windows DLL copies, the libzpdf install script), so
`meson install --tags runtime` stages the same set as a plain install.
- libyams_onnx_resource moves from <libdir> to the private <libdir>/yams
on Linux/macOS, without a soname version since nothing outside this
build links it. yams-cli/yams-daemon RUNPATH becomes $ORIGIN/../lib/yams
(@executable_path on macOS) and the ONNX/Glint plugins use $ORIGIN/..
so they resolve it and the bundled ONNX Runtime, which also moves to
<libdir>/yams (the ONNX plugin's runtime loader already searches its
parent directory). Windows keeps its existing DLL layout.
- Homebrew templates and the macOS install smoke follow the new layout.
- scripts/prune-runtime-install.sh only strips now.
Measured on grim (Linux x86_64 Release, prefix /usr/local):
staged paths 522 -> 11, staged bytes 2.4 GiB -> 1.0 GiB (unstripped)
check_install_surface path violations: 6 -> 0
`--tags devel` stages nothing; ldd resolves libyams_onnx_resource from
bin/../lib/yams and the staged `yams --version` runs.
Signed-off-by: trvon <me@trevon.dev>
Packages have only looked small because scripts/prune-runtime-install.sh
deletes headers and static archives and strips after `meson install`. A
plain install of a Release tree stages 2.4 GiB: 28 internal static
libraries (1.3 GiB), 459 headers and 22 .pc files, unstripped binaries
(yams-cli 497 MiB), an S3 plugin exporting 11,154 symbols including
OpenSSL, libcurl, zlib, spdlog and nlohmann, and an ONNX resource library
in the system libdir.
scripts/ci/check_install_surface.py stages `meson install --destdir` for
a build dir (or reads a staged tree) and asserts it against
scripts/ci/install_surface_policy.json: an allowed runtime path set and a
forbidden devel set, required paths, per-binary size budgets, exported
symbol budgets and forbidden third-party export patterns, $ORIGIN-only
RUNPATH, ELF hardening (PIE, BIND_NOW, GNU_RELRO, NX; CET and FORTIFY
reported as warnings) and stripped binaries. ELF inspection uses
readelf/nm; Mach-O uses otool/nm. Standard library only.
The release packaging job runs it on the exact staged payload after the
existing runtime asset checks. On current experimental it is RED
(Linux x86_64 Release, grim):
raw meson install: 43 violations (paths, debug info, exports, sizes)
after prune script: 29 violations (exports, libdir placement)
Size budgets start at the measured stripped baseline plus ~10% and are
tightened by the follow-up build changes.
Signed-off-by: trvon <me@trevon.dev>
The deferred-drain check treated a missing value as zero, so a node
whose p2p status call failed or returned non-JSON satisfied it. Drain
now needs a numeric 0 from every node, and so does the quarantined
writer check once any node reports the field; an unreadable status is
reported as such instead of passing.
Signed-off-by: trvon <me@trevon.dev>
converge proves convergence with yams get once yams list shows every
hash. When those reads kept failing until the timeout, the listing gap
was empty, so converge.json and REPORT.md claimed every document was
readable next to converged=false. The final yams get failures are now
kept (unreadable_via_get) and used for the readable counts whenever the
read-path check ran.
Signed-off-by: trvon <me@trevon.dev>
seed floored --docs-per-node to whole rounds of topics: 10 documents
with 8 topics seeded 8, 0 still seeded a full round, and --topics 0
raised ZeroDivisionError. Both counts are now validated, and the last
round is truncated so every node gets exactly the requested number.
Signed-off-by: trvon <me@trevon.dev>
up --force only skipped the existing-state guard. Compose then recreated
the services on the old named volumes, so nodes came up with the stale
/node/ready marker, keys and config of the previous mesh, and services
of a larger previous mesh kept running as orphans. A forced up now runs
down (volumes, orphans, image) for the previous project first and starts
under a fresh project name.
Signed-off-by: trvon <me@trevon.dev>
A mesh of honest writers must never durably quarantine one of them, but
converge only checked document reachability and deferral drain, so a run
that quarantined a peer could still pass. converge now records
quarantined_writers per node and exits 4 when any node reports one.
Every up/run also gets its own compose project (yamsmesh-<random>, or
--project), so containers, network, volumes and image tag of concurrent
meshes on one host no longer collide.
Signed-off-by: trvon <me@trevon.dev>
Production meshes (transport=direct, full mesh, operator-pinned mutual TLS)
stalled at a fixed record count with failed_cycles=0 because a failed inbound
apply stage skipped the node's own publish. Reproducing that needs several
real daemons with enrolled pins and a corpus of embeddings and topology
edges authored on different nodes, so add tests/integration/p2p_mesh:
- Dockerfile and node entrypoint that run prebuilt yams/yams-daemon from a
host build dir or a released .deb, one isolated volume per node.
- mesh.py (stdlib only) with up/seed/converge/status/chaos/run/down. It
generates writer keys and manifests, enrolls every peer pin and connects
a full mesh the way an operator does, and records apply-health counters
when the build reports them.
Local/manual tooling only; not wired into any CI.
Signed-off-by: trvon <me@trevon.dev>
* feat(metadata): add keyset node/edge scans and exact edge lookup
The memory-sync topology backfill paged nodes with LIMIT/OFFSET and no
ORDER BY, so a topology rebuild that deleted and reinserted cluster
nodes between two pages shifted the offset and the sweep skipped live
nodes for good. Ids in kg_nodes and kg_edges are AUTOINCREMENT and never
reused, so a cursor on the last id returned is stable under concurrent
inserts and deletes and survives restarts.
getNodesAfterId/getEdgesAfterId scan in id order from such a cursor.
getEdgesBetween returns every edge between two nodes for one relation;
getEdgesFrom truncates at its page limit, which made "is this edge
present" checks on high-degree nodes unreliable.
Signed-off-by: trvon <me@trevon.dev>
* fix(memory-sync): publish topology edge endpoints and retract deleted ones
On a Docker mesh that converges on documents, topology deferrals never
drained: peers held member_of and semantic_neighbor edges whose endpoint
node had not replicated (43/35/25 per node at 3x40 docs, ~140 at 5x80).
Inspecting the kept 3x40 mesh after 20 minutes: every stale deferral
named a doc or cluster node that its origin still held but had not
published. The origin's sweep was still inside the "document" type: no
"path", "topology_cluster" or "topology_snapshot" node had been published
by any node, while the doc nodes' edges into them had been.
The backfill swept node types in sorted order and published each node's
edges right after the node, so an edge reached peers long before its
target node of a later type; at the backfill's rate that is tens of
minutes. Its paging also made some of those waits permanent:
- Nodes were paged per type with OFFSET and no ORDER BY. A topology
rebuild deletes every cluster node and inserts replacements, so the
offset skipped live nodes whose member_of edges were published.
- Per-type offsets were never reset, so a node was visited once: edges
added after its sweep (semantic neighbors, new memberships) and
in-place node updates were never published.
- Nothing published deletions. Worse, apply() wrote this node's own
winners back into its graph, resurrecting the cluster nodes and edges
its rebuild had just deleted, so the deletion could not be observed.
- Peers deferred an edge whose endpoint was tombstoned forever.
The topology backfill now runs repeating keyset passes: every node by
id, then every edge by id published together with both endpoint nodes
(endpoints first, so a peer never holds a writer's edge without them),
then this writer's committed topology winners by logical key, tombstoning
those the local graph no longer holds. Cursors are stable ids/keys, so
inserts and deletes between items cannot shift them, and a restart just
begins a new pass; unchanged records publish nothing.
apply() still validates every record but never applies the writer's own
ones: the local graph is authoritative for them. An edge whose endpoint
is known deleted (tombstoned, or an own record the local graph no longer
holds) is dropped instead of deferred, and applies again if the endpoint
is re-published. Edge lookups during apply use the exact endpoint query
instead of a page-limited adjacency scan.
Signed-off-by: trvon <me@trevon.dev>
* perf(memory-sync): publish documents before derived records in backfill
With the topology sweep now publishing every node and edge, a 3-node x
40-doc mesh took longer until every document was listed on every node
(603 s against 426 s before the topology fix). Debug logs showed why:
a publish takes most of the backfill's 100 ms per-cycle time budget, so
each cycle published about one item, and the fair rotation over
documents, vectors and topology handed two of every three cycles to
derived records while peers waited for the documents.
The backfill now serves domains in priority order: documents, then
vectors, then topology, each taking items until it has nothing left this
cycle. Vectors and topology defer on a peer until their document
arrives, so publishing the document first never adds a deferral. In the
same mesh, run concurrently with both other builds, every document is
listed on every node after 140 s (426 s before the topology fix, 603 s
with it alone), and deferrals still drain to 0.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* feat(p2p): log durable writer quarantines and outbound session failures
A node that durably quarantined a peer writer left no trace of why: the
three p2p quarantine sites returned an error to their session, and the
reconnect loop dropped every outbound session error without logging it,
so a mesh that stopped receiving one writer showed only a quarantined
record count. Outbound sessions are half of all sessions in a full mesh.
Each p2p quarantine site now logs the writer, the failed check and the
local and frozen counters, and a failed outbound session logs its peer
and error, matching the existing inbound failure log.
Signed-off-by: trvon <me@trevon.dev>
* fix(p2p): stop quarantining honest writers for concurrent-session races
Each pair of direct peers normally runs two sessions at once, one
initiated from each side. Both freeze a window of the peer's writer at
handshake, and whichever applies first advances the receiver's durable
prefix under the other. The second session then compared its frozen
window with that live prefix and treated the difference as forged
history: validateHistoryExtension demanded the window start right after
the live commitment ("history extension is not contiguous"), and an
empty window demanded the live counter equal the frozen frontier. Both
durably quarantined an honest writer, after which the node dropped its
documents and refused every later session with it. A 5-node mesh run
logged exactly this (local=30 frontier=32 staged=4: a window frozen at
28 arriving after another session had applied 29-30), and the mesh
never converged.
The receiver now validates a window against durable history instead of
a snapshot. Operations at or below the advanced prefix are replays that
applyDeltas leaves untouched (a different record under a known operation
id is still a fork). The rest must chain from the durable commitment
exactly to the authenticated frontier, and an empty window must match
the durable commitment at its frontier. validateHistoryExtension returns
a typed WriterHistoryViolation for writer faults (fork, forged frontier,
gap, unauthenticated record) and an Error for local or transient
failures, and only a violation quarantines; an Error aborts the session
for a retry on the next cycle.
The security property is unchanged: nothing is applied unless a hash
chain links durable history, itself verified against the writer's
authenticated commitments, to the frontier the writer authenticated in
this handshake. Overlapping replays change no state, and forks, forged
frontiers, gaps and unauthenticated records still quarantine, now also
when they overlap an advanced prefix (new tests cover each).
The pre- and post-exchange history checks get the same treatment. A
live prefix between the proven handshake prefix and the frozen frontier
is consistent, since delta exchange verifies the rest; one beyond the
frontier is compared with the durable commitment at the frontier rather
than rejected as a rollback.
The handshake also clamps the live local-writer window resolver to the
frozen snapshot, so concurrent local writes no longer fail the session
with "local writer-window resolver violated negotiated bounds".
Signed-off-by: trvon <me@trevon.dev>
* feat(p2p): report outbound sessions and durable writer quarantines
`yams p2p status` showed inbound session failures but not outbound
ones, and only a `quarantined` record count that mixes durable writer
quarantine with transient rejections (a record whose causal dependency
from a third writer has not arrived yet). An operator could not tell a
mesh that was retrying from one that had permanently cut off a writer.
Status now reports:
- outbound_sessions / outbound_failures and the last outbound failure
with its peer, counted for both explicit connects and reconnects.
Failed sessions in either direction are aborted and retried.
- quarantined_writers: writers durably quarantined by this node, read
from the replication checkpoint, so it survives restarts. Each
quarantine is also logged with its reason.
The fields travel through MemorySyncStatus, MemorySyncResponse and
protobuf fields 38-42 to the text and JSON status and to `yams daemon
status -d`, following the apply-health fields.
Signed-off-by: trvon <me@trevon.dev>
* fix(memory-sync): verify replayed writer counters against durable history
validateHistoryExtension skipped every staged operation at or below the
writer's durable prefix as a replay without looking at its record hash,
leaving fork detection to applyDeltas. applyDeltas recognizes a fork only
by an operation id it still remembers, and that memory is lost on restart
and bounded by maxTrackedIdentities. A different, internally valid record
for an already-applied counter therefore passed as a harmless replay
instead of being quarantined as the fork it is, whether the window only
replayed old counters or rode the fork along an honest extension.
Before skipping a replayed counter, compare its record hash with the
durable history entry committed at that counter (rebuilding the entry like
computeHistoryCommitmentAt when it is missing). A mismatch is a
WriterHistoryViolation, so the p2p session durably quarantines the writer;
an identical replay is still accepted as a no-op. A history entry that
cannot be read stays a local Error and only aborts the session.
Signed-off-by: trvon <me@trevon.dev>
* fix(memory-sync): log every durable writer quarantine where it happens
Durable writer quarantine was logged only by its two p2p callers. The
duplicate-operation fork path inside applyDeltas and the shared-store scan
path quarantined silently, so the documented guarantee that every durable
quarantine logs its reason and counters did not hold, and an operator saw
a quarantined_writers count with nothing in the log to explain it.
quarantineWriter now takes the reason and is the one place that logs a new
quarantine: writer, reason, source node, and the writer's local history and
version counters. It logs nothing when the writer was already quarantined,
so a repeated request is not reported twice. The p2p callers drop their own
warnings and fold their session context (frontier, staged count, prefix
match) into the reason instead. memory_sync now declares its spdlog
dependency explicitly.
Signed-off-by: trvon <me@trevon.dev>
* fix(cli): show outbound p2p sessions in daemon status
The daemon already reports outbound sessions, failures, and the last
outbound failure, and `yams p2p status` prints them, but `yams daemon
status -d` only gained the quarantined-writers row. An outbound direction
that keeps failing is aborted and retried, never quarantined, so the
detailed status was the place an operator would miss it.
The Memory Sync section now shows an Outbound sessions row (started and
failed counts, warning when any failed) for direct-backend nodes, and a
Last outbound failure row with the peer, the error, and its age.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
* fix(daemon): never gate memory-sync publish on inbound apply
applyMemorySyncWinners() ran the inbound stages content -> metadata ->
vector -> topology and returned on the first stage failure, before the
node's own outbound backfill. In a mesh where every node holds an inbound
record whose prerequisite only a peer can publish (an embedding whose
content blob has not replicated, an edge whose target node has not), every
node refused to publish, so the prerequisites never arrived: a stable
mutual deadlock with healthy transport and nothing converging.
The outbound publish now runs after the inbound stages whatever their
outcome; only a stop request ends a cycle early. Each inbound stage also
runs even when an earlier one failed. Cross-stage dependencies (metadata
and vectors need local content bytes, edges need their endpoint nodes) are
checked per record against the local stores, so a failed stage shows up
downstream as unapplied records rather than as skipped stages. Stages keep
their prerequisite order so a prerequisite landing in a cycle is visible to
its dependents in that same cycle.
A two-node test drives real ServiceManager apply cycles over a shared
store, each node holding an edge whose target only the other can publish;
it converges in a bounded number of rounds only with this change.
Signed-off-by: trvon <me@trevon.dev>
* fix(memory-sync): defer records whose prerequisite has not replicated
Vector, metadata, and topology apply treated a missing replicated
prerequisite as a hard error and rejected the whole batch: one embedding
whose content blob had not arrived, or one edge whose target node had not,
blocked every other winner in that stage on every cycle. These are
ordering races between independently published records, not failures.
Each adapter now defers such a record instead: it is skipped, the rest of
the batch applies, and its logical key is reported through deferredKeys().
Nothing marks it applied, so it remains a winner in the replicated index
and the next cycle retries it once the prerequisite lands; apply needs no
pending queue of its own and loses nothing across restarts. An edge whose
endpoint is being deleted in the same batch is deferred too rather than
inserted against a stale node id. Malformed records, probe failures, and
store errors still fail the stage.
Content blobs are independent, so the content stage now attempts every
blob and fails afterwards with the first genuine error; a corrupt blob no
longer starves the blobs after it. A blob whose bytes are not hydrated in
the sync cache yet is deferred like the other stages.
Signed-off-by: trvon <me@trevon.dev>
* feat(p2p): report memory-sync apply health in status
The mesh deadlock was invisible: every node reported failed_cycles=0 and a
climbing successful_cycles while nothing converged, because a cycle counts
as successful once reconciliation completes, whatever the apply stages did
afterwards. Deferred records would be just as invisible without a signal.
ServiceManager now keeps an ApplyHealth record of the apply path:
- deferred records per stage, and the age of the oldest one still pending,
with a warning once a deferral passes ten minutes (its prerequisite
likely never arrived). Ages are tracked for at most 4096 keys per stage,
keeping the oldest; counts stay exact.
- hard failures per stage, plus the last failure's stage, message, and age.
- apply_cycles and apply_failed_cycles. failed_cycles keeps meaning
reconciliation failures; a cycle whose apply stage hard-failed is counted
in apply_failed_cycles instead of passing as a plain success.
- publish_skipped_cycles (delete outbox preempted the cycle) and
publish_failed_cycles (an outbound backfill domain failed).
The fields travel through MemorySyncStatus, MemorySyncResponse, and
protobuf fields 22-37 to `yams p2p status` (new key=value tokens on the
first line, an `apply` line with per-stage counts, a last-apply-failure
line, and an `apply` object in --json) and to the memory-sync section of
`yams daemon status`. The p2p status formatter moves behind a header so it
can be tested without a daemon.
Signed-off-by: trvon <me@trevon.dev>
* fix(memory-sync): keep content deferrals when the same scan hits a failure
The content stage scans every blob and fails afterwards with the first
genuine error, but the apply cycle discarded the deferred keys that the
same scan collected whenever it failed. A corrupt blob therefore hid every
blob still waiting for its bytes: the deferred count stayed at its previous
value (0 on the first pass) and the stale-deferral warning never fired for
a prerequisite that was actually missing.
A stage pass now carries both outcomes. The content-blob scan moves into
ContentBlobSyncAdapter::applyWinners, which reports the applied count, the
deferred keys, and the first failure of one complete scan, and
ApplyHealth::recordPass records the failure and replaces the deferred set
together. A stage that returns an error still leaves the deferred set as
last observed, since an aborted scan says nothing about prerequisites.
The tracked-deferral cap already carried every still-deferred tracked key
before admitting new ones, but its carry loop had a cap check that read as
if it could stop early. Drop it (the tracked set never exceeds the cap) and
pin the behaviour with tests that place the oldest key after more than the
cap's worth of new keys and that refill a full tracked set.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
tests/integration/search/meson.build reassigned catch2_main_dep to Catch2's
own catch2-with-main. Meson variables are global, so every test executable
defined after that subdir (catch2_search_service among them) linked Catch2's
main instead of tests/support/catch2_main.cpp. The Windows hang stack showed
main coming from Catch2's catch_main.cpp, which is why the CRT report setup
in the shared main never ran there.
Use the shared catch2_main_dep for the hybrid search integration binaries
too.
Signed-off-by: trvon <me@trevon.dev>
supportedExtensions() returned its static set by value, so capabilities()
constructed the extension vector from begin() of one temporary and end()
of another. That range is undefined behavior. libstdc++ and libc++ release
iterators happened to walk the first copy to completion, but the MSVC debug
STL rejects it in _Verify_range with a modal "Microsoft Visual C++ Runtime
Library" assertion dialog. HandlerRegistry::selectBest calls capabilities()
for every candidate, so every lines: search and every applyExtractScope call
hung forever on Windows Debug builds.
Return the shared set by const reference and take begin()/end() from that
one instance. New text_basic_handler unit tests pin the shared instance,
the capability list, registry selection and line-range extraction; the old
code also fails them under -D_GLIBCXX_DEBUG with "function requires a valid
iterator range".
Signed-off-by: trvon <me@trevon.dev>
Three catch2_search_service cases hang on windows-latest with no output at
all, so the diagnose workflow could only name them. A silent hang on a
Debug MSVC build is often a modal CRT assert or abort dialog that nobody
can dismiss.
Before killing a hung case the workflow now prints its stdout/stderr tail
(run with -s -d yes), its top-level windows, any WER helper, and every
thread's stack plus !analyze -v -hang from cdb.exe. A test_filter input
limits the run to a Catch2 test spec for faster iterations.
The shared Catch2 main now routes CRT warnings, errors and assertions to
stderr and disables the Windows error-reporting dialogs, so a future
Debug assertion fails the case with its message instead of hanging.
Signed-off-by: trvon <me@trevon.dev>
* build: standardize the Debug build dir on build/debug
`docs/BUILD.md` and `AGENTS.md` document `./setup.sh Debug` producing
`build/debug`, but `setup.sh`/`setup.ps1` created `builddir` and `.clangd`
pointed there too, so the documented layout and the actual tree disagreed.
Align the Debug mapping with the documented layout:
- setup.sh / setup.ps1: Debug -> `build/debug`
- CMakeLists.txt: Debug `_meson_build_dir` -> `build/debug`
- .clangd: CompilationDatabase -> `build/debug`
- plugin test fallbacks, S3 smoke default, quality-gate candidate list, and
doc examples -> `build/debug`
Legacy `builddir` trees stay readable: the quality gate and local-ci scripts
still probe `builddir` as a fallback, and `plugin_trust` already treats
`/build/` paths as build artifacts.
* build: migrate remaining Debug consumers off builddir
Follow-up to the Debug build-dir standardization: tracked consumers still
hardcoded `builddir` and would break or silently skip against a fresh
`build/debug` tree. Migrate them, keeping tolerant fallbacks where a legacy
tree may still exist.
- CI: `.github/workflows/tests.yml` runs `./setup.sh Debug` and then compiled
`-C builddir`; switch BUILD_DIR/COMPILE_DB_DIR/compile paths to `build/debug`.
Same for windows-test-diagnose.yml.
- IDE: `.vscode/tasks.json` build/test/bench tasks target `build/debug`.
- Runtime: `object_storage_plugin_loader` also probes `build/debug/plugins`
(and the flat `build/debug`), retaining the `builddir` fallbacks.
- Tests/docs: ONNX tuning integration plugin dir, benchmarks, and plugin
READMEs/examples use `build/debug`.
`./build/*` is already excluded from the quality-gate find and `.dockerignore`,
so no exclusion lists change.
The build needs third_party/symspell (tests.yml initializes it); without it
meson stopped at 'Nonexistent build file third_party/symspell/meson.build'
before any test ran.
Signed-off-by: trvon <me@trevon.dev>
* fix(init): align fresh-install embedding dimensions and vector sentinel
A fresh `yams init` emitted dimension-conflict warnings and created a
vector store whose dimension did not match the configured embedding model:
Embedding policy: embedding dimension conflict:
embeddings.embedding_dim='1024' overrides compatibility key
vector_database.embedding_dim='384' (and vector_index.dimension='384')
Root causes:
- getLatestConfigDefaults() hardcoded vector_database.embedding_dim and
vector_index.dimension to 384 while [embeddings] used 1024 for the
default Simeon backend, so the generated config was self-inconsistent.
- init resolved the vector dimension only by scanning the optional ONNX
model list, leaving it at 0 for the default "simeon-default", and never
wrote vectors_sentinel.json.
- updateV2Config searched for [vector_database] keys (enabled/model/
model_path) that do not exist in the v3 schema and never updated the
dimension keys.
Fixes:
- Default vector_database.embedding_dim and vector_index.dimension to 1024.
- Add resolveModelDimension() and stamp vectors_sentinel.json from the
vector DB's effective dimension only when this init created the store,
so a forced re-init cannot overwrite a real stored dimension.
- Rewrite updateV2Config to set enable/embedding_dim for [vector_database],
enable/preferred_model/backend/embedding_dim for [embeddings], and
dimension for [vector_index] consistently; disable the enable flags when
semantic search is declined.
- Non-interactive init now defaults to semantic search enabled, matching
the interactive default.
- Keep compatibility dimension keys aligned during v1/v2 migration so an
existing 384-d install never receives a 1024-d sibling default.
* fix(init): prompt for the tuning profile once
`execute()` asked for the tuning profile before the idempotency check and
`handleAlreadyInitialized()` asked again for an already-initialized instance,
so `yams init` showed the tuning prompt twice. Collect the profile once on the
fresh path (after the idempotency check); the already-initialized path keeps
its single prompt.
Add interactive regression tests with a scripted stdin helper:
- already-initialized init prompts for tuning exactly once and persists the
chosen profile;
- declining semantic search disables both `[vector_database]` and
`[embeddings]`, writes no dimension sentinel, and still prompts for tuning
once (also covers the fresh interactive path).
---------
Co-authored-by: trvon <git@trevon.dev>
Full isolation still let get_runtime_dir() consume the ambient XDG_RUNTIME_DIR, so yams-daemon.status.json landed outside the temp root. Relocate XDG_RUNTIME_DIR (POSIX) and APPDATA/LOCALAPPDATA/USERPROFILE (Windows) under the run root, and scrub YAMS_EMBED_BACKEND, YAMS_PREFERRED_MODEL, YAMS_RERANKER_MODEL and YAMS_EMBED_DIM, which ConfigResolver ranks above TOML, so the generated config is deterministic. Apply the same to ingestion_e2e_bench and extend the isolation test with a decoy runtime dir and model overrides.
Signed-off-by: trvon <me@trevon.dev>
repair_ability_bench built its harness without any isolation, so the daemon
resolved config from ambient YAMS_CONFIG, ~/.config/yams and HOME and could
index real files. multi_client_ingestion_bench isolated state but not config,
HOME or the XDG cache/data dirs, and ingestion_e2e_bench left HOME untouched.
Use DaemonHarnessOptions::isolateEnvironment in repair_ability_bench (with an
explicit simeon embeddings config so the embed fault kind stays repairable)
and both multi_client_ingestion_bench option builders; an explicit
YAMS_BENCH_CONFIG still wins. In ingestion_e2e_bench set HOME and
XDG_DATA_HOME to the fixture root and unset YAMS_STORAGE and the daemon
socket aliases. Shift the affected portability allowlist entries.
Signed-off-by: trvon <me@trevon.dev>
DaemonHarness only relocated state when isolateState/isolateConfig were set,
and even then left HOME, XDG_DATA_HOME, XDG_CACHE_HOME and ambient
YAMS_DAEMON_SOCKET_PATH/YAMS_STORAGE/YAMS_DATA_DIR visible to the daemon, so a
run could still resolve a developer's config, data and cache.
Add DaemonHarnessOptions::isolateEnvironment (and DaemonHarness::isolatedOptions())
which implies isolateState and isolateConfig, points HOME and every XDG base
directory at the run's temp root, and scrubs the ambient config/data/socket
variables for the daemon's lifetime, restoring them through ScopedEnvVar RAII.
The new Catch2 case exports a decoy config.toml (distinctive data_dir marker) and
decoy XDG/HOME/YAMS_* values, then asserts the daemon's config path, data dir and
every ambient path resolver stay under the temp root, the decoy is never read or
populated, and the ambient environment is restored after stop.
Signed-off-by: trvon <me@trevon.dev>
* fix(windows): ship yams-daemon and yams-mcp-server in the MSI
The Windows release already builds both and meson install stages them next
to yams.exe, but yams.wxs only packaged yams.exe, so daemon-backed commands
and yams serve failed after an MSI install. Package both, and make
build-msi.ps1 fail when a required executable is missing from the stage or a
staged executable is not packaged.
Signed-off-by: trvon <me@trevon.dev>
* ci(release): fall back to the stable R2 and signing secrets for experimental
The experimental channel needs EXPERIMENTAL_* Cloudflare and GPG secrets that
were never created, so even a passing nightly could not publish and
repo.yamsmemory.ai/experimental/* has never existed. Fall back to the stable
token, account and key when the channel-specific secrets are unset; setting
them later still takes precedence.
Signed-off-by: trvon <me@trevon.dev>
* ci: add a manual Windows workflow that finds hanging test cases
search_service times out at 360 s on Windows in every Tests run, which keeps
experimental from going green and blocks every nightly. The meson log only
shows the timeout. This workflow builds once and runs each Catch2 case on its
own with a per-case timeout, then reports durations and the cases that hang.
It runs only on workflow_dispatch or a push to a diagnose/** branch.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
Picks up the Worker fix 525a0a4 (preserve plugin cache policy) merged
in yams-repo PR #1; the pointer had stayed at dca5bd6.
Signed-off-by: trvon <me@trevon.dev>
The #206 revert restored the wrangler upload path, which passes
--content-type, and removed the S3 "ContentType" field this step grepped
for. Act Sanity has failed on experimental since then.
Signed-off-by: trvon <me@trevon.dev>
The snippet wrote /etc/pacman.d/yams.conf but nothing includes that file,
so pacman never saw the repository. Append the [yams] section to
/etc/pacman.conf instead.
Signed-off-by: trvon <me@trevon.dev>
${{ cond && '' || 'experimental/' }} always yields 'experimental/' because
'' is falsy in Actions expressions, so the generated USAGE.txt for stable
releases pointed YUM and Arch users at experimental paths that return 404.
Invert the expression. The YUM instructions also claimed an unsigned
repository although repomd.xml is signed whenever a channel key is set;
enable repo_gpgcheck with the channel key in that case.
Signed-off-by: trvon <me@trevon.dev>
The #206 revert restored the pre-September upload code, which passed
--force, while the workflow still pins wrangler 4.69.0. That version's
'r2 object put' has no --force option (it overwrites without prompting)
and exits with 'Unknown argument: force', so the v0.20.3 release failed
its first R2 upload and no apt, yum or Arch repository was published.
Immutable objects are still checked with 'get' before any write.
Signed-off-by: trvon <me@trevon.dev>
Feature PRs merged with merge commits carry the PR title in the merge
commit body, which Release Please parsed as a second conventional commit.
Keep the entry that links the original commit.
Signed-off-by: trvon <me@trevon.dev>
WranglerObjectStore passes str(Path("/wrangler")) to the command, which
is \\wrangler on Windows, so the Windows lane failed the literal
"/wrangler" comparison.
Signed-off-by: trvon <me@trevon.dev>
CI runs the unit lanes with YAMS_SQLITE_VEC_SKIP_INIT=1, so the test's
VectorDatabase came up without vector tables and the post-VACUUM insert
failed on every platform. The case already clears the other vector
toggles; clear this one too.
Signed-off-by: trvon <me@trevon.dev>
When the daemon holds background work back for a busy host, a user
looking at a stale topology or an unfinished repair scan needs to see
why. Status had no host signal at all.
Carry the governor's host sample through the metrics snapshot and
StatusResponse: source (psi, loadavg), whether the host counts as busy,
the PSI cpu/io/memory "some avg10" values or the per-CPU load average,
and which deferrable work is currently held back. The proto transport
piggybacks the fields on request_counts like the other string status
fields; the native serializer appends them as optional tail fields.
`yams daemon status` (brief and detailed) shows a Host Load row, e.g.
"Busy cpu 72% · io 3% · mem 0% · deferring topology, repair", and
`yams status --json` gains a host_pressure object. Document the new
[tuning.resource] keys next to the other typed tuning sections.
Signed-off-by: trvon <me@trevon.dev>
requestTopologyRebuild() re-posted itself immediately whenever ingest,
post-ingest, embedding or KG work was still in flight. It released its
in-progress flag before re-posting, so every request that arrived while
the daemon was busy (one per embedding batch) could win the flag and
start its own re-post loop. Those loops never slept: they occupied the
worker pool until the ingest drained, and a slower drain on a loaded
host meant more of them for longer.
Per-thread sampling during a 1899-file ingest on a 2x oversubscribed
48-core host showed ~36 worker threads at 100% for the whole drain. For
the same ingest (120s window, background work only):
daemon CPU seconds before after
idle host 41, 41 23, 20, 19
loaded host 96, 95 28, 30, 30 (deferral off)
Arm a single timer instead: a busy daemon re-checks after 500ms, a
rebuild deferred for host pressure after 5s, and only one retry is
armed at a time. post_ingest_drain still requests a rebuild directly
when the pipeline empties, so topology freshness does not wait on the
timer in the common case (idle-host topology completion is unchanged at
~7s after ingest).
Add a ServiceManager test for the deferral path: with the host marked
busy the rebuild is held and the governor reports it deferred; once the
host calms down nothing is held back and, when a vector database is
available, the armed retry builds the artifacts without a new request.
Signed-off-by: trvon <me@trevon.dev>
Background work started as soon as it was scheduled, no matter what else
the machine was doing, so a topology rebuild or a repair scan competed
with the user's own jobs for CPU at the worst time.
Classify background work. Must-run work keeps the corpus consistent and
is never gated: crash recovery, database integrity checks and WAL
checkpoints. Deferrable work waits while host pressure is elevated:
- topology rebuilds and semantic topology maintenance (the dirty set
keeps accumulating; a deferred rebuild re-checks on a 5s timer)
- repair maintenance: the initial scan (which is what queues embedding
catch-up for documents missing vectors), the legacy snapshot backfill
and the vectors.db VACUUM
- semantic_neighbor backfill for older documents
Embedding of newly ingested documents is not deferred. Throttling it to
one job under host load was measured and dropped: on a 2x
oversubscribed 48-core host it stretched the embedding drain 1.5-4x and
sometimes triggered two extra vector index builds, without lowering the
daemon's CPU use.
Each kind goes through a DeferralGate in the resource governor. A
starvation guard bounds the wait: after background_max_deferral_s
(default 300s) one pass runs and the wait starts over, so sustained
load slows this work to a trickle but never stops it. With the guard at
45s, a loaded run held the topology rebuild for 45s and then built it
(48-53s after ingest instead of 22-33s with deferral off). Setting
[tuning.resource] background_max_deferral_s = 0 turns deferral off.
The governor logs when a kind starts deferring, when the guard forces
a pass, and when it resumes.
RepairService already gated maintenance on client activity and daemon
pressure; the host check sits last in each condition so the deferral
clock only measures time spent waiting on other processes.
Signed-off-by: trvon <me@trevon.dev>
The governor only measured the daemon itself (RSS, own CPU), so it could
not tell that the machine was busy. On a loaded host the daemon still
started background work at full width, and that work ran far less
efficiently: ingesting the same 1899-file corpus cost 41 daemon CPU
seconds on an idle 48-core host and 96 CPU seconds with the host 2x
oversubscribed (stress-ng), and took 4x longer to finish.
Sample a cheap host-wide signal on the existing governor tick, at most
once per second: Linux PSI "some avg10" for cpu, io and memory, falling
back to the one-minute load average per online CPU; on macOS the load
average plus the kernel VM pressure level. A busy host is a flag with a
release band (signals must drop to 75% of the threshold to clear) so
work does not flap around a threshold.
Host load stays a separate dimension from the governor's pressure
level. Folding it into the level would reuse the Warning response,
which blocks model loads and trims foreground concurrency; a busy build
on the same machine should not stop a search from loading its model.
Background scheduling consumes the flag in a follow-up change.
Thresholds are typed config under [tuning.resource]: host_pressure,
host_cpu_pressure_pct (40), host_io_pressure_pct (30),
host_memory_pressure_pct (10) and host_load_per_cpu (1.5). They show in
the runtime tuning status with their provenance.
Signed-off-by: trvon <me@trevon.dev>
libyams_mobile links YAMS, SQLite, Boost and the rest statically and
exported all of it (about 127k dynamic symbols). In a process that also
carries YAMS, such as the mobile test binary running a DaemonHarness, the
dynamic linker bound some of the daemon's metadata calls (for example
makeSqliteKnowledgeGraphStore) into the library's copy while the rest of
the daemon used its own. Two SQLite copies then opened the same yams.db.
POSIX advisory locks are per process, so neither copy saw the other's
WAL-index locks; one truncated the -shm file and the other faulted with
SIGBUS in walFindFrame while initialising the knowledge-graph store.
A linker version script (Linux) and exported-symbols list (macOS) keep
everything but yams_mobile_* local. The library now exports exactly the
51 API functions, and scripts/ci/check_mobile_abi.sh still passes.
This is a host-build fix; it does not claim device support.
Signed-off-by: trvon <me@trevon.dev>
libyams_mobile stopped compiling without anyone noticing because no CI
lane built it: every lane forced YAMS_ENABLE_MOBILE_BINDINGS=false, and
the "Check mobile ABI surface" step was commented out and pointed at
scripts/ci/check_mobile_abi.sh, which was never committed.
Add a linux-x86_64-mobile matrix entry that configures with
enable-mobile-bindings and enable-mobile-tests, compiles only the
yams_mobile and catch2_mobile_abi targets, runs the ABI check, and runs
the mobile meson suite. Other lanes keep mobile bindings off.
Add scripts/ci/check_mobile_abi.sh. It compiles the public header as C11
and C++20 and takes the address of every YAMS_MOBILE_API function, so a
declaration hidden from C (the extern "C" bug fixed in the previous
commit) fails. It then checks that the library exports exactly the
declared yams_mobile_* functions with unmangled names.
This lane proves the Linux x86_64 host build only. It does not show that
an Android .so or iOS XCFramework loads on a device.
Signed-off-by: trvon <me@trevon.dev>
With -Denable-mobile-bindings=true, libyams_mobile no longer compiled:
- diff/cat called IDocumentService::retrieveDocument, which was replaced
by retrieve(); they now fetch stored bytes through retrieve() with
acceptCompressed left false, so content comes back uncompressed, and
map service errors through map_error_code instead of NOT_FOUND.
- repair awaited DaemonClient::call() as a future and read response
fields that no longer exist; it now goes through daemon_call() and
folds RepairResponse::operationResults into the C result.
- YAMS_PROFILE_ZONE is not a defined macro; use YAMS_ZONE_SCOPED_N.
- doctor built its version with std::string(char); emit JSON instead.
- yams_mobile_string_view_destroy, _cat_result_destroy and
_restore_result_destroy were declared but never defined.
The repair/diff/cat/restore/model/doctor declarations sat outside the
extern "C" block and inside #ifdef __cplusplus, so C and FFI hosts could
not see them and C++ builds exported mangled names. Move them into the
C block.
Daemon-mode contexts handed their own worker_pool executor to the
process-wide connection pool keyed by socket path. The pool outlived
yams_mobile_context_destroy, so a second context reused sockets bound to
a freed thread pool (ASAN heap-use-after-free in AsioConnection) and the
daemon was left with connections that never drained, which then tripped
a use-after-free in daemon shutdown. Contexts now use single-use
connections and disconnect the client before joining the worker pool.
Tests: add an embedded cat/diff test (exact bytes, line diff, identical
docs, not-found mapping) and a daemon test that creates and destroys two
contexts and checks the server returns to its baseline connection count.
This proves the Linux x86_64 host build and host-side behavior only. It
does not show that an Android .so or iOS XCFramework loads on a device.
Signed-off-by: trvon <me@trevon.dev>
