Commit graph

52 commits

Author SHA1 Message Date
Trevon
60dd2c9afe
fix(daemon): admit cold reads and preserve sync deferral accounting (#331)
Dispatch explicit committed memory reads on admitted workers, checking the
governor before queueing and before I/O. Require the coordinator instead of
using the synchronous strand fallback; keep cached APIs and wire opcodes
unchanged and preserve task-record validation.

Use the paged content adapter, distinguish typed local admission deferral
from hard errors and retain known deferred totals/ages after incomplete
scans. Forward exact topology totals independently of diagnostic samples
and keep attempted cycles distinct from complete scans and successful apply.

Add real daemon IPC tests for authenticated off-resident reads, pressure
health, exact 70-edge status/retry accounting and authenticated cold document
retrieval through the existing MCP get handler, DaemonClient IPC and CAS.
Reanchor existing dispatcher environment-read entries without new exceptions.

Scan uncertainty remains C++-only: typed wire/status/MCP exposure, broader
fault/readiness/P2P convergence and patched resource measurements remain open.

Signed-off-by: trvon <git@trevon.dev>
2026-10-22 07:12:25 -06:00
Trevon
e48d9f8d1f
fix(test): use the daemon timeout variable for the harness isolation test (#273)
7e744805 (#241) registered daemon_harness_full_isolation with get_option('integration-daemon-timeout'). ca944de5 removed that option, so every Tests lane on experimental fails at meson setup: "Option integration-daemon-timeout does not exist". Use daemon_integration_timeout, as the other daemon_harness tests do. Configured on grim with the default, asan, ubsan, mobile and macOS option sets from tests.yml.

Signed-off-by: trvon <me@trevon.dev>
2026-10-07 09:51:52 -06:00
Trevon
3840a6495d
Merge branch 'experimental' into bench/isolate-kpi-daemons-v2 2026-10-05 07:35:36 -06:00
trvon
ca944de51b
build: trim Meson options and build only the Boost libraries YAMS links
meson_options.txt had 47 options. Several were read but never set by
any workflow, script or doc, several only fed test-lane timeouts, and
enable-{tsan,asan,ubsan} duplicated b_sanitize. 47 -> 33.

Removed (behavior kept at the previous default everywhere):
- require-fts5 (configure now always errors on a SQLite without FTS5,
  the old default), conan-generators-dir, cli11-cmake-path,
  onnxruntime-cmake-path (Conan/CMake discovery under the build dir is
  unchanged), split-debug.
- unit-timeout, integration-timeout, integration-ui-timeout,
  integration-smoke-timeout, integration-daemon-timeout and stress-iters:
  replaced by the same constants at their call sites (smoke keeps its
  test-timeout-scale multiplier). Only meson/profiles/ci-fast.ini and
  nightly-soak.ini set them, and nothing used those profiles; both go.
- enable-integration-services/-smoke and enable-daemon-integration:
  folded into enable-integration-tests. Debug builds still register every
  integration suite; nothing set the sub-toggles.

Kept, because third_party/symspell's meson.build reads them:
bench-timeout, enable-tsan and enable-asan. YAMS itself now derives all
sanitizer handling (extra flags, TSan runtime links, test timeouts) from
b_sanitize; setup.sh still mirrors its choice into enable-tsan/asan for
the submodule, and the CI ASan lane keeps -Denable-asan=true for the
same reason. enable-ubsan is a deprecated no-op (a warning if it
disagrees with b_sanitize) so build dirs configured with it still
regenerate; the UBSan lane passes only -Db_sanitize=undefined and still
gets -fno-sanitize-recover=all (standalone UBSan aborts; ASan+UBSan stays
recoverable, as before).

Conan: the without_* Boost list was applied only to iOS/Android. YAMS
uses Boost.Asio (header-only) and links only boost_system/boost_thread
on every platform (asio/ssl uses OpenSSL directly; no spawn/context), so
the list now applies everywhere. Debug Boost package on grim: 45 static
libraries / 229 MiB -> 10 / 6.8 MiB (Release: 45 / 30 MiB before).

Verified on grim: ./setup.sh Debug --no-tsan --with-tests reconfigures
an existing build/debug and builds; an existing build/release (configured
with -Denable-tsan/-Denable-asan) regenerates.

Signed-off-by: trvon <me@trevon.dev>
2026-10-01 18:52:34 -06:00
Trevon
7be908e5b7
fix(p2p): stop the memory-sync apply/publish deadlock and report apply health (#243)
* fix(daemon): never gate memory-sync publish on inbound apply

applyMemorySyncWinners() ran the inbound stages content -> metadata ->
vector -> topology and returned on the first stage failure, before the
node's own outbound backfill. In a mesh where every node holds an inbound
record whose prerequisite only a peer can publish (an embedding whose
content blob has not replicated, an edge whose target node has not), every
node refused to publish, so the prerequisites never arrived: a stable
mutual deadlock with healthy transport and nothing converging.

The outbound publish now runs after the inbound stages whatever their
outcome; only a stop request ends a cycle early. Each inbound stage also
runs even when an earlier one failed. Cross-stage dependencies (metadata
and vectors need local content bytes, edges need their endpoint nodes) are
checked per record against the local stores, so a failed stage shows up
downstream as unapplied records rather than as skipped stages. Stages keep
their prerequisite order so a prerequisite landing in a cycle is visible to
its dependents in that same cycle.

A two-node test drives real ServiceManager apply cycles over a shared
store, each node holding an edge whose target only the other can publish;
it converges in a bounded number of rounds only with this change.

Signed-off-by: trvon <me@trevon.dev>

* fix(memory-sync): defer records whose prerequisite has not replicated

Vector, metadata, and topology apply treated a missing replicated
prerequisite as a hard error and rejected the whole batch: one embedding
whose content blob had not arrived, or one edge whose target node had not,
blocked every other winner in that stage on every cycle. These are
ordering races between independently published records, not failures.

Each adapter now defers such a record instead: it is skipped, the rest of
the batch applies, and its logical key is reported through deferredKeys().
Nothing marks it applied, so it remains a winner in the replicated index
and the next cycle retries it once the prerequisite lands; apply needs no
pending queue of its own and loses nothing across restarts. An edge whose
endpoint is being deleted in the same batch is deferred too rather than
inserted against a stale node id. Malformed records, probe failures, and
store errors still fail the stage.

Content blobs are independent, so the content stage now attempts every
blob and fails afterwards with the first genuine error; a corrupt blob no
longer starves the blobs after it. A blob whose bytes are not hydrated in
the sync cache yet is deferred like the other stages.

Signed-off-by: trvon <me@trevon.dev>

* feat(p2p): report memory-sync apply health in status

The mesh deadlock was invisible: every node reported failed_cycles=0 and a
climbing successful_cycles while nothing converged, because a cycle counts
as successful once reconciliation completes, whatever the apply stages did
afterwards. Deferred records would be just as invisible without a signal.

ServiceManager now keeps an ApplyHealth record of the apply path:

- deferred records per stage, and the age of the oldest one still pending,
  with a warning once a deferral passes ten minutes (its prerequisite
  likely never arrived). Ages are tracked for at most 4096 keys per stage,
  keeping the oldest; counts stay exact.
- hard failures per stage, plus the last failure's stage, message, and age.
- apply_cycles and apply_failed_cycles. failed_cycles keeps meaning
  reconciliation failures; a cycle whose apply stage hard-failed is counted
  in apply_failed_cycles instead of passing as a plain success.
- publish_skipped_cycles (delete outbox preempted the cycle) and
  publish_failed_cycles (an outbound backfill domain failed).

The fields travel through MemorySyncStatus, MemorySyncResponse, and
protobuf fields 22-37 to `yams p2p status` (new key=value tokens on the
first line, an `apply` line with per-stage counts, a last-apply-failure
line, and an `apply` object in --json) and to the memory-sync section of
`yams daemon status`. The p2p status formatter moves behind a header so it
can be tested without a daemon.

Signed-off-by: trvon <me@trevon.dev>

* fix(memory-sync): keep content deferrals when the same scan hits a failure

The content stage scans every blob and fails afterwards with the first
genuine error, but the apply cycle discarded the deferred keys that the
same scan collected whenever it failed. A corrupt blob therefore hid every
blob still waiting for its bytes: the deferred count stayed at its previous
value (0 on the first pass) and the stale-deferral warning never fired for
a prerequisite that was actually missing.

A stage pass now carries both outcomes. The content-blob scan moves into
ContentBlobSyncAdapter::applyWinners, which reports the applied count, the
deferred keys, and the first failure of one complete scan, and
ApplyHealth::recordPass records the failure and replaces the deferred set
together. A stage that returns an error still leaves the deferred set as
last observed, since an aborted scan says nothing about prerequisites.

The tracked-deferral cap already carried every still-deferred tracked key
before admitting new ones, but its carry loop had a cap check that read as
if it could stop early. Drop it (the tracked set never exceeds the cap) and
pin the behaviour with tests that place the oldest key after more than the
cap's worth of new keys and that refill a full tracked set.

Signed-off-by: trvon <me@trevon.dev>

---------

Signed-off-by: trvon <me@trevon.dev>
2026-10-01 12:18:47 -06:00
trvon
7e744805d2
test(harness): add full environment isolation to DaemonHarness
DaemonHarness only relocated state when isolateState/isolateConfig were set,
and even then left HOME, XDG_DATA_HOME, XDG_CACHE_HOME and ambient
YAMS_DAEMON_SOCKET_PATH/YAMS_STORAGE/YAMS_DATA_DIR visible to the daemon, so a
run could still resolve a developer's config, data and cache.

Add DaemonHarnessOptions::isolateEnvironment (and DaemonHarness::isolatedOptions())
which implies isolateState and isolateConfig, points HOME and every XDG base
directory at the run's temp root, and scrubs the ambient config/data/socket
variables for the daemon's lifetime, restoring them through ScopedEnvVar RAII.

The new Catch2 case exports a decoy config.toml (distinctive data_dir marker) and
decoy XDG/HOME/YAMS_* values, then asserts the daemon's config path, data dir and
every ambient path resolver stay under the temp root, the decoy is never read or
populated, and the ambient environment is restored after stop.

Signed-off-by: trvon <me@trevon.dev>
2026-09-30 19:47:19 -06:00
trvon
f8fce1db18
fix(daemon): destroy the SocketServer only after its handlers are gone
YamsDaemon::stop() stopped the SocketServer and destroyed it while the
IOCoordinator threads were still running, and only then stopped and
destroyed the IOCoordinator. SocketServer::stop() waits at most ~2s for
connections to drain, and the handle_connection coroutine frames keep a
raw pointer to the server in their cleanup guard. A connection that
outlived the drain window either resumed on an I/O thread after the
server was freed, or had its frame destroyed together with the
io_context, and both paths decremented the connection counters of the
freed SocketServer (ASAN heap-use-after-free in CleanupGuard). The
ServiceManager-init failure path stopped the server the same way and
left the I/O threads running.

Tear the IPC layer down in one place, teardownSocketServer(): stop the
server (which still releases its acceptors and accept strand while the
io_context runs), stop and join the I/O threads so no handler can
resume, destroy the IOCoordinator so the io_context destroys the pending
handler frames while the server is valid, and only then destroy the
server. All daemon stop and startup-failure paths use it.

The unit test stops the io_context under the connection handlers so
they are deterministically pending when the teardown runs; the daemon
integration test parks the I/O threads past the drain window.

Signed-off-by: trvon <me@trevon.dev>
2026-09-27 20:11:43 -06:00
trvon
c39ed84bf7
refactor(plugins)!: remove the symbol_extractor_v1 plugin ABI
Migration 40 dropped the symbol tables and graph edges, but the daemon
could still adopt an external symbol_extractor_v1 plugin (for example a
0.19 plugin left in a user plugin directory) and rebuild them.

Remove the interface header, the ABI adapter, plugin adoption, the
EntityGraphService extract-and-write path, the post-ingest symbol
channel, the write-coordinator symbol ops, and the KG-store symbol
metadata and extraction-state API. The ABI loader and the external host
now refuse a plugin that declares the interface (or the legacy
symbol_extraction capability) with a one-time warning. Status no longer
reports symbol extractors, generated configs no longer write
[plugins.symbol_extraction] (an existing section still parses), and
the benchmark suite drops symbol_extraction_bench.

Tests that exercised the dropped tables or the v39 schema template are
removed; the rest run against the current schema.

BREAKING CHANGE: code-symbol extraction is gone. Removed: the
symbol_extractor_v1 plugin ABI, `yams config grammar` (list, download,
path, auto-enable, auto-disable), the grammar steps in `yams init`, the
meson option `plugin-symbols`, the conan option
`enable_symbol_extraction`, and the Windows installer SymbolPlugin
component. Databases migrate to schema 41; a v0.19 binary refuses to
open them, and symbol data dropped by migration 40 cannot be recovered.
2026-09-22 13:30:35 -06:00
trvon
dde68293d7
test: honor memory sync timeout scaling
Apply the existing configured timeout scale to the integration executable. Observe periodic backfill attempts while retaining all publication assertions; no forced backfill or production scheduling change. Full TSan memory-sync suite passed.

Signed-off-by: trvon <me@trevon.dev>
2026-09-13 13:19:00 -06:00
Trevon
e954c6da71
fix(p2p): serialize sync application (#73)
Signed-off-by: trvon <git@trevon.dev>
2026-08-30 11:27:10 -06:00
Trevon
70a4977249
fix(p2p): replicate content deletion (#69)
Signed-off-by: trvon <git@trevon.dev>
2026-08-30 11:27:09 -06:00
Trevon
5a53cdfa90
feat(p2p): add authenticated direct replication (#68)
* fix(memory-sync): keep replication responsive

Skip poisoned blobs without wedging backfill and publish immutable status snapshots so daemon IPC does not block on an active reconciliation cycle.

Reconstructed from ff62d3f8, 8e232721, and e60f513d.

Signed-off-by: trvon <git@trevon.dev>

* feat(p2p): add authenticated direct transport

Require TLS 1.3 mutual identity and bind the authenticated certificate, node, corpus, and epoch before replication state is exchanged.

Reconstructed from 96f9db88 and c46de77b.

Signed-off-by: trvon <git@trevon.dev>

* feat(p2p): exchange causal deltas

Persist peer pins and causal resume state before visibility so reconnects continue bounded authenticated replication without becoming remote query proxies.

Reconstructed from 1b8be146 and 41883338.

Signed-off-by: trvon <git@trevon.dev>

* feat(p2p): integrate direct sync services

Wire listener, reconnect, IPC, CLI, peer persistence, and daemon lifecycle around the authenticated direct transport.

Reconstructed from a19aaa44.

Signed-off-by: trvon <git@trevon.dev>

* fix(p2p): apply replicated document state

Index authenticated remote winners, propagate document tombstones, and keep service/test wiring complete across daemon paths.

Reconstructed from 4ca660c6, 443253f1, 3e9db24d, and 4b2ddcc5.

Signed-off-by: trvon <git@trevon.dev>

* test(p2p): register direct sync gates

Replace the superseded internal contract with operator-facing direct-sync guidance and register daemon integration coverage in the standard test layout.

Reconstructed from 360b9115 and 89416ccb.

Signed-off-by: trvon <git@trevon.dev>

* fix(p2p): require bounded peer enrollment

Bound aggregate sessions and worker lifetimes, create owner-only identity keys atomically, and reject unknown inbound peers until operators pin their mTLS identity.

Reconstructed from f91be528, 79de8e48, 4fb1e4b8, dc30a6c0, and b329edd3.

Signed-off-by: trvon <git@trevon.dev>

* fix(p2p): persist quarantine and deletes

Persist full-history writer commitments before invalidating forked state, and order local deletion through a durable authenticated outbox so tombstones survive crashes and reconnects.

Reconstructed from 74ce01e0 and 19888604.

Signed-off-by: trvon <git@trevon.dev>

* feat(p2p): add authenticated cold bootstrap

Install bounded zero-state snapshots only from previously operator-pinned peers, bind them to exact frontiers and full-history commitments, and recover staged journals before replication can resume.

Reconstructed from 8a9f5acf.

Signed-off-by: trvon <git@trevon.dev>

---------

Signed-off-by: trvon <git@trevon.dev>
2026-08-30 11:27:08 -06:00
trvon
910a1d7f9c
test(daemon): harden background wait, slow-lane hysteresis 2026-08-21 08:55:23 -06:00
trvon
780c602d50
feat(daemon): wire bounded sync lifecycle 2026-08-17 21:43:11 -06:00
trvon
9c9e19199e
refactor(test): consolidate daemon harness 2026-08-05 20:53:18 -04:00
Trevon
a3d065dd8d
feat(search): make topology-assisted retrieval the default (#57)
* fix(release): repair package repo publish

Arch repo metadata and R2 uploads need to be reproducible from CI, with retry hardening for transient Cloudflare failures.

* fix(daemon): stop shutdown log storm

Init waiter must treat ShuttingDown as terminal so daemon shutdown tests do not flood Meson logs while the daemon is stopping.

* refactor(cli): delegate serve to MCP binary

Keep yams serve UX while removing MCP server linkage from yams-cli. The standalone MCP binary owns stdio serving and exits on stdin EOF.

* refactor(cli): move daemon path helpers

Keep CLI path resolution out of YamsDaemon so later link cuts can separate CLI and daemon internals. Full yams_daemon removal still needs a socket-only daemon-client split.

* refactor(daemon): split embedded client

* perf(search): add lean-guided benchmark gates

Add Lean topology/search models, ingestion ablation docs, and focused regression tests for topology and compression threshold behavior.

Extract search observability, keep benchmark-only ablation controls, and make retrieval quality benchmarks default to Simeon while preserving explicit ONNX opt-in.

* refactor(config): move daemon policies

* refactor(daemon): extract plugin runtime

* refactor(daemon): untangle service coordinators

Untangle CLI/MCP from daemon service coordinators and the daemon static lib.

- MetadataRepository uses a daemon-neutral tree-diff callback. The
  GraphComponent wrapper moves to src/daemon/components/metadata_graph_bridge.cpp,
  owned by the daemon library.
- WriteCoordinator moves to a dedicated build-only yams_write_coordinator
  static lib. Drop ResourceGovernor coupling inside the writer loop; batch
  sizing still respects config.maxBatchSize.
- AppContext gains enqueuePostIngest callback. ServiceManager wires it to
  its own enqueuePostIngest; app services no longer call ServiceManager
  directly.
- Embedding repair uses BulkIngestLease/BeginBulkIngestCallback instead of
  concrete VectorIndexCoordinator::BulkScope. Daemon dispatchers provide a
  lease factory backed by the real coordinator.
- CLI daemon_command uses config helpers instead of YamsDaemon::getXDGRuntimeDir.
- Remove explicit dependency('yams_daemon') from src/cli and tools/yams-mcp.
- Declare missing yams_extraction/yams_ingest deps on yams_app_services so
  app services link cleanly without daemon.

Validated:
  meson compile -C build/release -j4 yams-daemon yams-cli yams-mcp-server
  ninja -C build/release -t commands tools/yams-cli/yams-cli tools/yams-mcp/yams-mcp-server | grep -c libyams_daemon.a -> 0
  yams-cli --version, yams-mcp-server init, yams serve --help smoke tests

Size drop on macOS release: yams-cli 41M -> 32M, yams-mcp-server 37M -> 22M.

* fix(cli): keep stdin add local fallback

No embedded daemon now. When yams add - buffers stdin and daemon IPC is missing, store via local app services instead of failing socket_missing.

Validated:
  meson compile -C build/debug -j4 yams-cli catch2_cli_commands
  MESON_BUILD_ROOT=/Users/trevon/work/tools/yams/build/debug build/debug/tests/catch2_cli_commands 'AddCommand - reads from piped stdin and stores content'
  MESON_BUILD_ROOT=/Users/trevon/work/tools/yams/build/debug build/debug/tests/catch2_cli_commands
  meson compile -C build/release -j4 yams-cli yams-mcp-server
  meson compile -C build/asan-api -j2 yams-cli catch2_cli_commands
  MESON_BUILD_ROOT=/Users/trevon/work/tools/yams/build/asan-api ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 build/asan-api/tests/catch2_cli_commands 'AddCommand - reads from piped stdin and stores content'
  ninja -C build/release -t commands tools/yams-cli/yams-cli tools/yams-mcp/yams-mcp-server | grep libyams_daemon.a -> no output

* feat(bench): add xplan multi-arm experiment harness

Unified plan runner, ablation axes, REPORT/ablation docs, and
KPI snapshot so optimization loops stop living in shell matrices.

* feat(formal): expand topology/search Lean contracts

Boundary models for search provenance, construction gates,
and cluster/route objectives so Lean guides correctness while
xplan ranks performance levers.

* feat(search,topology): construction purity + graph expansion path

Raise topology build defaults (min edge 0.25, max component 64) and
split oversized CC pieces so hybrid assist is quality-positive on
scifact. Add GraphNeighbors expansion with medoid-anchor fallback,
structured helpers instead of nested session logic, and BEIR-default
xplan plans. Search engine extraction (fanout/classic pipeline) rides
along as the integration surface for routing.

* feat(topology): add KMeans engine and SGC identity contracts

Register kmeans in the factory with deterministic spherical k-means
over embeddings, and enforce SGC document-identity invariants in
debug builds with matching unit coverage.

* feat(daemon): topology readiness gates and selectable reranker

Drive topologyReady from published epoch/freshness/rebuild/dirty
ingest state (not just KG drain), surface those fields in search
stats, and wire onnx|simeon|auto|none cross-reranker selection.

* chore(search): drop meta_path tuner knobs

Meta-path routing is gone; remove dead TunedParams fields and the
ConfigResolver test for [search.meta_path].

* feat(bench): add topology_purity_validate plan

Repeats=3 validation matrix for construction-purity winners vs
topo_off so MRR deltas can clear run-to-run noise.

* chore: format add_command and config include order

Mechanical clang-format / include reorder only; no behavior change.

* feat(bench): decision-grade xplan loop for search engine

Document search-engine measurement in AGENTS; harden xplan with
repeats=3 plans, quality×latency primaries, mechanism metrics from
debug.jsonl, multi-rep log promotion, and archive superseded topology
plans.

* feat(graph): denser product semantic_neighbor topK

Raise default YAMS_GRAPH_SEMANTIC_TOPK fallback from 4 to 8 so product
graphs leave fewer degree-0 docs for GraphNeighbors expansion.

* feat(search): extract topology assist and cross-rerank seams

Pull topology assist into search_topology_stage (seed merge, skip
reasons, optional vector seed probe default 0). Extract cross_rerank
compute from search_engine. Wire probe via typed config/ConfigResolver.

* feat(bench): product-path search engine ablation plans

Add topology-off component, vector-weight, and tiered-execution xplan
plans (repeats=3) and wire them into the AGENTS measurement loop table.

* fix(ci): repair pre-push fallbacks

* feat(bench): lock-free mid-run builds + multi-corpus gates

xplan skips meson rebuild when the quality binary exists (flock +
YAMS_BENCH_FORCE_BUILD override) to stop multi-arm link races. Add
graph_vector weight and nfcorpus transfer-gate plans; document in AGENTS.

* fix(daemon): keep SearchEngine alive via shared_ptr cache

getCachedEngine previously published a raw pointer that metrics polling
used through rebuild, racing destructor of SearchEngine::Impl (TSAN).
Store and return a shared_ptr snapshot so diagnostic readers pin the
engine for the duration of getConfig/health checks.

Also adds the cautious vectorWeight 0.30→0.20 multi-corpus xplan plan.

* feat(search): route vectors through cached topology

Build and reuse validated topology snapshots, rank routed members by query similarity, and narrow the vector leg to the accepted document set with explicit work diagnostics.

Remove the retired HDBSCAN engine and vendor dependency, preserve typed topology policy through search tuning, and cover the runtime/configuration contracts.

* feat(bench): consolidate search decision plans

Enforce deterministic experiment identity and declared baselines, expose routed/global vector work metrics, and register the focused benchmark contracts.

Promote the compact repeats=3 routing-budget gate, archive superseded topology mechanism screens, and add multi-corpus product calibration plans.

* feat(search): refine topology vector routing

* fix(search): propagate daemon readiness

* feat(bench): measure generalized memory health

* fix(cli): honor in-process fallbacks

* ci: harden smolvm debug builds

* perf(search): route topology through filtered ANN

* feat(bench): measure read-write pressure

* perf(metadata): index symbol lookups on read pool

* fix(daemon): isolate startup reads and maintenance

* feat(bench): replay isolated generalized corpora

* feat(search): add evidence routing pipeline

Replace the blocking smolvm pre-push path with host ASan/TSan validation, harden sanitizer-sensitive smoke tests, and add recovery cleanup for malformed SQLite WAL sidecars.

* refactor(search): unify topology evidence path

Remove classic and sidecar fusion lanes so topology augments the recall-preserving ANN and lexical candidate pool through one product path.

* refactor(bench): prune superseded search lanes

Measure the topology-plus-ANN product path with stable experiment identity and remove obsolete sidecar and tiered controls.

* docs(search): retire stale experiment notes

Keep the benchmark handoff focused on the topology-plus-ANN product lane and its active decision plans.

* refactor(search): prune dead search surfaces

Remove unused fusion, reranker, policy, and router abstractions.
Bound topology materialization and keep document trace payloads opt-in.

* fix(cli): clear build warnings

* fix(xplan): reject stale run identity

* refactor(search)!: remove legacy query stack

Replace the unused AST/ranker/post-processor surface with focused facet aggregation used by product search.

BREAKING CHANGE: removes query parser, result ranker, search filter, and SearchResults container APIs from yams_search.

* refactor(search): type value stage traces

Remove synthetic ComponentResult markers for embedding and concept stages. Preserve failure and timeout states instead of overwriting them with fake successful results.

* refactor(search): isolate benchmark env overlays

Move test-only search environment parsing behind an injectable internal adapter. Keep all existing overlay keys and tuner pin semantics while restoring SearchEngineBuilder to composition.

* refactor(search): unify topology routing options

Snapshot topology policy once at the assist boundary and pass it through routing sessions and diagnostics. Remove repeated scalar copying without changing routing behavior or metric keys.

* fix(search): unify tuner config ownership

Use one complete config-to-params overlay for runtime seeding and no-tuner policy resolution. Reseed attached tuners on setConfig so reported and executed configuration cannot diverge.

* fix(search): snapshot tuner state per query

Read tuned config, params, and state under one lock before routing. Add a TSAN correlation test so adaptive observation cannot produce a torn logical query snapshot.

* fix(search): enforce topology routing contract

Generated configs must preserve hybrid-assist shadow routing, while daemon-backed search must abstain when published topology artifacts are stale. Keep standalone validation and global ANN recall intact.

* fix(daemon): let SQLite reclaim its file

Directly resizing a live database from sampled page counts can truncate pages allocated by concurrent writers. VACUUM must remain the only file-reclamation mechanism.

* fix(metadata): tolerate missing symbol FTS

Readiness lookups can race an unavailable optional FTS migration. Retain the indexed fast path, but retry against the base table when prepare or cached-statement execution proves the capability absent.

* build(fuzz): drop removed query parser target

The optional AFL build still referenced the deleted legacy query parser API and could no longer compile.

* fix(bench): include writes in drain oracle

A zero post-ingest queue is not durable drain while coordinated writes remain queued or in flight.

* fix(daemon): vacuum padded WAL databases safely
2026-07-13 20:26:43 -04:00
trvon
c051423e27
fix(ci): stabilize remaining test lanes
Handle vector-disabled, all-skipped, UBSan, and benchmark-gated paths that CI exercises.
2026-05-18 20:25:19 -04:00
trvon
afe5074abb
fix(ci): stabilize test matrix
- Guard post-ingest callbacks before teardown.
- Skip unavailable vector/provider paths in CI.
- Fix Windows assumptions and trim ASAN benchmarks.
2026-05-18 16:36:21 -04:00
trvon
8b7bfe575d
feat: improved search fusion logic and tuning parameters. CI improvements, reverting to push with wrangler in CI. Experimenting with vector rebuild to address additional lost overhead in search 2026-05-05 08:58:20 -04:00
trvon
c3db49ffdd
test: test fixes for flaky tests and bug fix for single db writing for kg. Contemplating swapping backend 2026-04-29 22:41:45 -04:00
trvon
f825bc3cf5
feat: ci/test regressions updates, reinstated onnxruntime (simeon will still be default), expanding topology instrumentation in search, newsletter update with throughput comparison 2026-04-26 17:03:37 -04:00
trvon
df0c78cf29
feat: test fixes, metric expansion, build improvements for gliner 2026-04-23 08:48:33 -04:00
trvon
4cf1704b36
fix: addressing TSAN and empty catch blocks 2026-03-18 16:44:27 -04:00
trvon
4fe9ce6120
fix: bug fixes and test improvements for small correctness issues 2026-03-18 15:16:42 -04:00
trvon
0cdb3ab6c0
fix: testing improvements and bug fixes 2026-03-18 12:30:32 -04:00
trvon
3b1a3007ab
feat: sandbox testing expansion (sandbox relates to how the daemon communicates in sandbox). Graph optimizations for retrieval and UX improvements. Will continue improving graph system 2026-02-25 11:37:29 -05:00
trvon
3c09becde5
fix: repair improvements and socketserver hardening 2026-02-07 23:37:32 -05:00
trvon
ded16c11d4
improvements of parallel ingestion 2026-02-05 11:51:24 -05:00
trvon
2b0f5f4d89
imrpoving daemon internals, mcp fixes for metadata queries 2026-01-27 12:09:08 -05:00
trvon
47e6292261
thread safety fixes and metric seeding fixes 2026-01-26 20:23:42 -05:00
trvon
80ceef9a93
additional bug fixes and testing improvements 2026-01-24 14:18:50 -05:00
trvon
36c587e776
bug fixes and test improvements 2026-01-24 11:49:48 -05:00
trvon
d3b50e4dc4
bug fixes for onnx and improving support for larger models. tuning serach engine and expanding module communication via daemon 2026-01-10 01:32:48 -05:00
trvon
83bd1b7fc1
updating entity interface and adding new plugin for entity extraction 2026-01-08 15:06:58 -05:00
trvon
f32852f4b9
symspell fuzzy index optimization for search 2026-01-07 20:31:11 -05:00
trvon
0bce48a899
bug fixes, extended tree dedup logic for ingestion, dynamic search engine returning with internal benchmark 2026-01-06 18:57:35 -05:00
trvon
ca990c6bfa
removing flaky tests 2026-01-03 17:15:43 -05:00
trvon
c73b899b8f
bug fixes and daemon communication improvements 2026-01-02 22:37:19 -05:00
trvon
49e3f638bc
improving daemon startup logic, improve repair and doctor cli UX 2026-01-02 13:57:37 -05:00
trvon
94d89fe0d5
bug fixes, test updates and ci improvements for TSAN test runs 2026-01-01 13:52:52 -05:00
trevon
16d225599d
fixing plugin load logic for external plugin, test compilation issues and adding entity integration test 2025-12-20 11:19:37 -05:00
trevon
48dc0dfa89
removing gtest migrated tests and continuing with test migration 2025-12-16 10:50:18 -05:00
trevon
8c58e9d961
testing improvements 2025-12-08 20:52:24 -05:00
trvon
b70253587a
use after free fix and ci build fix 2025-11-28 20:30:19 -05:00
trvon
c4c42e0266
additional streaming fixes for idle daemons 2025-11-18 16:16:39 -05:00
trvon
34edf49dc1
adding test for idle bug 2025-11-17 20:00:32 -05:00
trvon
5b23afa99d
adding dedicate graph component for graph operations, updating socket stale logic and fixing CI issues in release. 2025-11-16 13:31:04 -05:00
trvon
ea9341303f
fixing stale connection bug 2025-11-14 16:44:43 -05:00
trvon
166d8c520d
staging v0.7.7 2025-11-01 16:02:12 -04:00
trvon
bbf23bb190
test migrations to catch2, service manager + socket server refactor for better startup performance. search bug fixes 2025-10-29 23:08:09 -04:00