Commit graph

29 commits

Author SHA1 Message Date
Trevon
d398556ee5
fix(tests): replace frozen expectations and harden release gates (#323)
* test(search,daemon,storage): assert behaviour instead of frozen literals

Several suites encoded exact internal values that legitimate tuning and
ranking changes are allowed to move: SCIENTIFIC-era profile weights and
rrfK values, corpus-statistic figures, and hardcoded timeout/cancellation
numbers. They failed on every platform after those changes landed, so the
real behaviour regressions they should catch were hidden behind noise.

Replace the frozen literals with contract assertions: weights are finite,
non-negative and normalize, state selection still holds, capability-gated
sources stay zeroed, and hybrid results keep matching canonical hash/path
sets. Deadlines are not widened and the daemon paths they cover are
unchanged.

Also add independent corpus and adapter oracles, deterministic collector
cancellation and late-result gates, and synchronize ServiceManager
shutdown observation, so the tests fail for the behaviour they name.

The portability allowlist keeps its two reviewed raw test-env reads; only
their line numbers moved.

Signed-off-by: trvon <git@trevon.dev>

* ci(tests): summarise Meson failures without flooding the log

A failing test run printed the whole Meson suite output, so the actual
failing test names and their assertion messages were buried and the
uploaded logs were dominated by passing-test chatter.

Add a bounded reporter that extracts the failing test names plus their
surrounding diagnostics, keeps an overall cap on emitted lines, and is
unit-tested against representative Meson output. Wire it into the Tests
workflow so failures stay readable.

Signed-off-by: trvon <git@trevon.dev>

* build(deb): validate candidate identity, mapping and upgrade ordering

The Debian packaging scripts took filenames and versions on trust, so a
misnamed candidate or a wrong package ordering could pass validation while
the real install path was never exercised.

Fail closed on candidate identity, version and architecture, verify the
package-to-source mapping, seed the legacy-only layout, wait on the actual
services, and assert install/upgrade/remove ordering for both Debian and
RPM in an offline container. Cases that cannot be proven on a host are
reported as skipped rather than silently passing.

Signed-off-by: trvon <git@trevon.dev>

---------

Signed-off-by: trvon <git@trevon.dev>
2026-10-19 19:16:57 -06:00
trvon
c4f0ce4338
ci(packaging): probe the bundled ONNX Runtime with a separate user
Switching the main test user's embedding backend to onnxruntime (no
model installed) and back left state behind that the later phases then
tripped over. Probe the runtime with a third user whose own config
enables plugin autoload with the auto backend (config_version = 3 so
the daemon does not migrate it to the simeon default), and keep the main
user on yams init's configuration throughout.

Signed-off-by: trvon <me@trevon.dev>
2026-10-02 10:12:02 -06:00
trvon
c1483e4478
ci(packaging): make the user-unit checks match real sessions
First runs of the user-unit validation on all four lanes found harness
gaps, fixed here:

- Arch's `su -` does not run pam_systemd, so the login shell had no
  XDG_RUNTIME_DIR and the CLI looked for /tmp/yams-daemon-<uid>.sock;
  the checks now provide the runtime dir a real login session gets and
  report whether su set it.
- Ubuntu 24.04's systemctl --user needs the user D-Bus bus: the
  Debian/Ubuntu lane image gains dbus-user-session (standard on real
  installs) and the checks pass the bus address when present.
- CLI commands fall back to an in-process daemon instead of spawning
  one, so the lifecycle check uses `yams daemon start` (which starts the
  unit) rather than expecting an ordinary command to start it.
- `yams init`'s default embedder is the built-in simeon backend, which
  skips the onnx model provider by design. The onnx phase now checks
  that init's config autoloads plugins, then switches [embeddings]
  backend to onnxruntime to prove the bundled runtime loads (from the
  user journal), and restores init's config so the later phases run the
  user's real configuration.

docs/install.md no longer claims ordinary CLI commands start the unit:
`yams daemon start` and on-demand starters do.

Signed-off-by: trvon <me@trevon.dev>
2026-10-02 09:18:38 -06:00
trvon
e0b0b575e2
ci(packaging): validate the systemd user unit in every lane
The package validation now exercises the per-user service:

- service: the user unit and user preset are installed, the unit is
  enabled for all users (systemctl --global), no system unit remains,
  and the unit does not hide the user's home or /tmp.
- user: a normal user with lingering gets a user manager; the unit is
  enabled and active without a manual start and `systemd-analyze --user
  verify` is clean; the socket is in $XDG_RUNTIME_DIR and owned by the
  user; through a login shell with no YAMS_* variables the CLI reports
  status, adds from $HOME and /tmp, searches, stores in
  ~/.local/share/yams and logs to ~/.local/state/yams, and the only
  daemon is the unit's MainPID; `yams daemon stop`, CLI auto-start and
  `yams daemon restart` go through systemd; `yams daemon install --user`
  enables the packaged unit without writing an override; a second user
  cannot reach the first one's daemon.
- onnx: after `yams init` the restarted user daemon logs "Using bundled
  ONNX Runtime" and `yams plugin health` lists the onnx plugin.
- persist: with lingering the user's daemon is back after a reboot with
  the data intact.
- upgrade from 0.20.3: the old system unit is stopped, removed and
  leaves no enablement links, /var/lib/yams is left alone, the user unit
  is enabled for all users and the user's daemon and CLI work.
- remove: running user daemons are stopped, global enablement is gone,
  no daemon process is left, user corpora survive (deb: also purge).
- minimal: bare base images still install with declared dependencies
  only; the account checks are gone with the yams account.

The Fedora lane image replaces the systemd-user PAM stack, which cannot
run in a container (unix_chkpwd, pam_selinux, pam_loginuid), with a
minimal one that still runs pam_systemd, so user@.service starts.

Signed-off-by: trvon <me@trevon.dev>
2026-10-02 08:30:55 -06:00
trvon
ca1710fdcc
fix(packaging): give the system daemon an /etc configuration file
The packaged service had no configuration file it would read: with
HOME=/var/lib/yams it looked for /var/lib/yams/.config/yams/config.toml,
which nothing creates. Plugins load only with [daemon]
auto_load_plugins = true, so the onnx and glint plugins and the bundled
ONNX Runtime shipped in the packages were never used by the service
("No plugins loaded"), and an administrator had no obvious place to turn
them on.

The unit now sets YAMS_CONFIG=/etc/yams/config.toml (the package ships
no file; ProtectSystem=strict leaves /etc readable). With
auto_load_plugins = true there, the sandboxed daemon loads all three
plugins and logs "Using bundled ONNX Runtime 1.23.0 from
/usr/lib/yams/onnxruntime/libonnxruntime.so.1 (no system copy found)",
ready=true. docs/install.md describes the file and how to check which
runtime was loaded; the validation's onnx phase writes it, restarts the
unit and checks the bundled runtime and plugin health.

Signed-off-by: trvon <me@trevon.dev>
2026-10-02 00:57:34 -06:00
trvon
2cd2ff68e3
ci(packaging): validate on bare base images and check the bundled ONNX Runtime
The lane images boot systemd and carry the tools the checks use, so
systemd-sysusers always existed and the deb postinst's broken `adduser`
fallback was never exercised.

Each lane now also installs the package in the raw distro image
(debian:trixie-slim, ubuntu:24.04, fedora:42, the Arch base) with no
systemd running and nothing preinstalled, so only declared dependencies
are present. It checks that the install finishes (deb: dpkg status
"install ok installed", not half-configured), that the yams account and
group exist, and that `yams add` + `yams search` work without the
service.

A new onnx phase checks, on the systemd lane, that the package ships the
private runtime in <libdir>/yams/onnxruntime and the onnx plugin, that
the system daemon logs "Using bundled ONNX Runtime", and that
`yams plugin health` lists the onnx plugin.

Signed-off-by: trvon <me@trevon.dev>
2026-10-02 00:01:55 -06:00
trvon
6e25a8b90f
ci(packaging): allow the bundled ONNX Runtime in package validation
Packages now carry the private fallback ONNX Runtime in
/usr/lib/yams/onnxruntime/ (libonnxruntime.so -> .so.1 -> .so.1.x.y and
libonnxruntime_providers_shared.so). Without an allowlist entry the
layout check rejects them as unreviewed paths. The pattern admits only
those library names under that directory.

Signed-off-by: trvon <me@trevon.dev>
2026-10-01 22:42:44 -06:00
trvon
4563f92d55
ci(packaging): allow private libraries under lib/yams in package validation
The install-surface cleanup moved libyams_onnx_resource from /usr/lib to
/usr/lib/yams, matching scripts/ci/install_surface_policy.json. The
package validation allowlist only accepted libyams_* directly in the
library directory and would have failed every lane; accept both, and
nested paths under /usr/share/doc/yams like the policy does.

Signed-off-by: trvon <me@trevon.dev>
2026-10-01 21:16:29 -06:00
trvon
7ef6cc9366
ci(packaging): make the per-user service and reboot checks host-robust
x86_64 runs on a loaded host showed three harness problems:

- The Debian/Ubuntu and Fedora lane images lacked pam_systemd, so
  user@.service could not start and `yams daemon install --user` looked
  broken. Install libpam-systemd / systemd-pam. Where a host still
  cannot start a user manager in a container, the check reports INFO
  instead of failing, since that says nothing about yams.
- `systemctl --user -M user@` needs machined; query the user manager as
  that user instead. The "CLI uses its unit" check now also requires that
  the only daemon running for the user is the unit's MainPID, so a
  CLI-spawned second daemon no longer passes.
- A container reboot can sit in 'starting' past 60 s under load; wait up
  to three minutes and print pending jobs if it still does not boot.

Signed-off-by: trvon <me@trevon.dev>
2026-10-01 21:11:34 -06:00
trvon
25114dadc7
ci(packaging): harden package validation from its first real runs
Running the gate against v0.20.3 and branch packages on Debian, Ubuntu,
Fedora and Arch showed harness gaps, fixed here:

- The driver ran only under bash 4+ (mapfile, empty arrays under set -u);
  it now works with macOS's bash 3.2 so it runs on Docker Desktop too.
- Paths dropped by the image's dpkg path-exclude (docs) or rpm nodocs
  no longer fail the permission check, and rpm's /usr/lib/.build-id links
  and the package doc/license directories are on the allowlist.
- The upgrade check pins the system socket, so it measures whether the
  corpus survived rather than whether an older CLI can discover it.
- Debian keeps enablement links until purge, so for deb the dangling-link
  check runs after purge.
- New per-user service check: `yams daemon install --user` beside the
  system unit starts a working user unit, and that user's CLI uses it.

The unit comment and install guide now describe the socket as group
read/write with no access for others (the daemon sets owner rwx, group
rw), rather than a literal 0660.

Signed-off-by: trvon <me@trevon.dev>
2026-10-01 21:11:27 -06:00
trvon
53f0061d31
ci(release): gate packages on a real systemd install, CLI and upgrade run
The old package-validate.sh only checked that the unit came up and that
`yams daemon status` answered with YAMS_DAEMON_SOCKET set by hand, so it
passed while nobody's CLI ever reached the system daemon, the socket was
world-writable and `yams add ~/file` failed.

package-validate.sh stays the single entry point (same --only/--deb/
--rpm/--arch flags the workflow uses) and now drives
package-validate-checks.sh inside Debian, Ubuntu, Fedora and Arch
containers booted with systemd as PID 1 (private cgroup namespace; the
host's cgroups are not shared). Per lane it checks: install with no
manual steps; the installed file list against a runtime allowlist with
no headers, static libs, pkg-config or CMake files; no world-writable,
setuid or non-root files; resolvable DT_NEEDED and $ORIGIN-or-/usr/lib
RPATH/RUNPATH; unit enabled and active as the yams account; socket group
yams and closed to others; `systemd-analyze verify` and a
`systemd-analyze security` exposure budget; root, a yams-group member
and a non-member using the CLI with no YAMS_* variables (add from $HOME
and /tmp, search, shared corpus, no private fallback store, clear
directory error, non-member denied, `daemon stop` deferring to systemd);
state after a container reboot; removal and deb purge. With
--upgrade-from-<fmt> it installs an older package, stores a document,
upgrades and checks the restart and the document. Every check reports
PASS/FAIL, a summary table follows, and any FAIL fails the run.

release.yml's existing validation steps (the gate before artifacts are
uploaded for publishing) pass the latest published release's packages as
the upgrade source. The unreferenced packaging/systemd/Dockerfile is
removed; the lane images take a BASE_IMAGE argument and add binutils
and util-linux for the checks.

Signed-off-by: trvon <me@trevon.dev>
2026-10-01 21:11:15 -06:00
trvon
e167498be4
fix(ci): build aarch64 Arch packages on an Arch Linux ARM base image
archlinux/archlinux only publishes linux/amd64, so the native aarch64 Arch
lane failed with 'exec format error' before provisioning. Both Arch
Dockerfiles now take ARCH_BASE_IMAGE; the aarch64 matrix entry pins
ghcr.io/menci/archlinuxarm by digest and x86_64 keeps the official image.
package-validate.sh forwards ARCH_BASE_IMAGE to the systemd substrate.
2026-09-24 15:00:32 -06:00
trvon
89bf7165b7
fix(ci): stop package-validate from exiting silently on a missing build dir
The release Arch job passes --arch explicitly, but the script still ran
deb/rpm discovery against the default build/release dir. find fails on the
missing dir, pipefail + set -e turn that into a silent exit 1. Only discover
packages for selected lanes and tolerate a missing build dir.
2026-09-24 14:52:18 -06:00
trvon
e1abefaf6f
ci(gate): distinguish transient infra from real failures
Retry setup.sh dependency resolution up to 3 times on Conan network
timeouts with backoff; report a distinct infra-retry (exit 77) when the
network failure persists, while real compile/test failures still fail
hard. Prevents a transient center2.conan.io timeout masquerading as a
code failure.
2026-08-12 20:09:40 -06:00
trvon
c9fbee0d50
ci(cross): add compile-only cross-compile smoke
Detects mingw-w64/linux-gnu cross compilers and compiles a portability
probe with -fsyntax-only to catch narrow-vs-wide API and std::jthread
issues before GitHub CI. Skips cleanly when no toolchain is present;
--self-test proves dispatch and failure propagation with a fake compiler.
2026-08-12 20:02:23 -06:00
trvon
be24d07c27
fix(ci): stabilize sanitizer validation
Rebuild macOS TSan outputs so compiler instrumentation and compiler-rt come from one Apple toolchain generation. Preserve daemon fixture dependencies until ServiceManager teardown so sanitizer lanes do not exercise a test-only use-after-free.
2026-08-08 11:42:33 -04:00
trvon
2ce2c3fb58
ci: cap push build parallelism at 4 cores
Push-triggered builds (local pre-push gate and GitHub Actions Tests)
ran meson compile without a job cap, saturating every host/runner core.
Default to -j4; keep the memory-sensitive linux-arm64 lane at 2.
Override the local lane with YAMS_MACOS_NINJA_JOBS.
2026-07-31 16:53:39 -04:00
Trevon
0dd3985db4
refactor: simplify runtime paths and harden retrieval (#61)
* refactor(tuning): prune dormant runtime knobs

Remove unused tuning controls and adaptive search budget state. Extract testable ONNX slot reconciliation and search telemetry hydration seams, and correct the Apple Silicon sysctl value width.

* refactor(vector): retire dormant backends

Remove unused FAISS, model-infrastructure, VectorIndexManager, and TurboQuant paths while keeping Simeon PQ and float-vector storage.

Migrate unused V2.2 quantized schemas transactionally to V2.3. Databases containing non-default quantized formats or packed-code blobs fail closed with recovery guidance and no mutation.

* refactor(plugins): remove legacy SDK tooling

Remove the Ghidra and SDK submodules, their bespoke tests, and stale scan exclusions. Retain generic PluginProcess coverage against the repository mock fixture.

* build(vector): remove stale FAISS overrides

* refactor(ipc): remove dormant FSM snapshots

* refactor(ingest): unify bounded KG enqueue

* fix(plugins): harden allocation contracts

Keep ABI outputs null or fully initialized when allocations fail, and represent empty object payloads without relying on malloc(0).

* fix(storage): define empty object reads

Return a null buffer with zero length for successful empty-object reads instead of treating malloc(0) as an allocation contract.

* refactor(storage): trim through string views

* test: prune dormant harness helpers

Remove unused readiness and fixture helpers, and keep benchmark client-count doubling in a wide integer domain.

* chore(cpp): initialize analyzer-facing locals

* build(simeon): advance analyzer cleanup

* refactor(config): drop dead daemon client keys

* refactor(ipc): inject daemon client transport

Socket and in-process execution now share DaemonClient request dispatch, so CLI fallbacks do not need alternate service paths.

* refactor(retrieval): route CLI through daemon

Search and grep now use the same daemon handlers over socket or in-process transport. Live grep remains an explicit freshness inspection.

* refactor(graph): move CLI scope into daemon

Graph CLI requests now share daemon handlers across socket and in-process transports. CWD filtering occurs before pagination so scoped totals and pages stay correct.

* build: isolate test-only runtime controls

* refactor(metadata): decouple lock telemetry

* fix(services): return on request timeout

* fix(retrieval): enforce workspace scope

* fix(graph): scope nodes before pagination

* fix(grep): drop binary result lines

* fix(daemon): bound checkpoint wait

* fix(repair): use KG health checks

* fix(storage): always verify compressed CRC

* perf(vector): skip unchanged PQ snapshots

* build(zyp): exclude bench from default build

* build: protect Meson files from formatter

* refactor: remove unreachable production APIs

* refactor(daemon): remove dormant runtime state

* fix(vector): harden daemon provider calls

Timed-out coroutines must own their client and requests. Provider discovery must also fail instead of returning synthetic embeddings in production.

* refactor(storage): remove unusable adapters

* fix(cli): unify in-process add requests

* fix(tsan): preserve async worker progress

* test(cli): isolate daemon-dependent suites

* fix(repair): expose queue backoff progress
2026-07-31 14:16:45 -04:00
trvon
75ceff2975
fix(ci): stabilize sanitizer test paths 2026-07-26 20:42:21 -04:00
Trevon
a3d065dd8d
feat(search): make topology-assisted retrieval the default (#57)
* fix(release): repair package repo publish

Arch repo metadata and R2 uploads need to be reproducible from CI, with retry hardening for transient Cloudflare failures.

* fix(daemon): stop shutdown log storm

Init waiter must treat ShuttingDown as terminal so daemon shutdown tests do not flood Meson logs while the daemon is stopping.

* refactor(cli): delegate serve to MCP binary

Keep yams serve UX while removing MCP server linkage from yams-cli. The standalone MCP binary owns stdio serving and exits on stdin EOF.

* refactor(cli): move daemon path helpers

Keep CLI path resolution out of YamsDaemon so later link cuts can separate CLI and daemon internals. Full yams_daemon removal still needs a socket-only daemon-client split.

* refactor(daemon): split embedded client

* perf(search): add lean-guided benchmark gates

Add Lean topology/search models, ingestion ablation docs, and focused regression tests for topology and compression threshold behavior.

Extract search observability, keep benchmark-only ablation controls, and make retrieval quality benchmarks default to Simeon while preserving explicit ONNX opt-in.

* refactor(config): move daemon policies

* refactor(daemon): extract plugin runtime

* refactor(daemon): untangle service coordinators

Untangle CLI/MCP from daemon service coordinators and the daemon static lib.

- MetadataRepository uses a daemon-neutral tree-diff callback. The
  GraphComponent wrapper moves to src/daemon/components/metadata_graph_bridge.cpp,
  owned by the daemon library.
- WriteCoordinator moves to a dedicated build-only yams_write_coordinator
  static lib. Drop ResourceGovernor coupling inside the writer loop; batch
  sizing still respects config.maxBatchSize.
- AppContext gains enqueuePostIngest callback. ServiceManager wires it to
  its own enqueuePostIngest; app services no longer call ServiceManager
  directly.
- Embedding repair uses BulkIngestLease/BeginBulkIngestCallback instead of
  concrete VectorIndexCoordinator::BulkScope. Daemon dispatchers provide a
  lease factory backed by the real coordinator.
- CLI daemon_command uses config helpers instead of YamsDaemon::getXDGRuntimeDir.
- Remove explicit dependency('yams_daemon') from src/cli and tools/yams-mcp.
- Declare missing yams_extraction/yams_ingest deps on yams_app_services so
  app services link cleanly without daemon.

Validated:
  meson compile -C build/release -j4 yams-daemon yams-cli yams-mcp-server
  ninja -C build/release -t commands tools/yams-cli/yams-cli tools/yams-mcp/yams-mcp-server | grep -c libyams_daemon.a -> 0
  yams-cli --version, yams-mcp-server init, yams serve --help smoke tests

Size drop on macOS release: yams-cli 41M -> 32M, yams-mcp-server 37M -> 22M.

* fix(cli): keep stdin add local fallback

No embedded daemon now. When yams add - buffers stdin and daemon IPC is missing, store via local app services instead of failing socket_missing.

Validated:
  meson compile -C build/debug -j4 yams-cli catch2_cli_commands
  MESON_BUILD_ROOT=/Users/trevon/work/tools/yams/build/debug build/debug/tests/catch2_cli_commands 'AddCommand - reads from piped stdin and stores content'
  MESON_BUILD_ROOT=/Users/trevon/work/tools/yams/build/debug build/debug/tests/catch2_cli_commands
  meson compile -C build/release -j4 yams-cli yams-mcp-server
  meson compile -C build/asan-api -j2 yams-cli catch2_cli_commands
  MESON_BUILD_ROOT=/Users/trevon/work/tools/yams/build/asan-api ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 build/asan-api/tests/catch2_cli_commands 'AddCommand - reads from piped stdin and stores content'
  ninja -C build/release -t commands tools/yams-cli/yams-cli tools/yams-mcp/yams-mcp-server | grep libyams_daemon.a -> no output

* feat(bench): add xplan multi-arm experiment harness

Unified plan runner, ablation axes, REPORT/ablation docs, and
KPI snapshot so optimization loops stop living in shell matrices.

* feat(formal): expand topology/search Lean contracts

Boundary models for search provenance, construction gates,
and cluster/route objectives so Lean guides correctness while
xplan ranks performance levers.

* feat(search,topology): construction purity + graph expansion path

Raise topology build defaults (min edge 0.25, max component 64) and
split oversized CC pieces so hybrid assist is quality-positive on
scifact. Add GraphNeighbors expansion with medoid-anchor fallback,
structured helpers instead of nested session logic, and BEIR-default
xplan plans. Search engine extraction (fanout/classic pipeline) rides
along as the integration surface for routing.

* feat(topology): add KMeans engine and SGC identity contracts

Register kmeans in the factory with deterministic spherical k-means
over embeddings, and enforce SGC document-identity invariants in
debug builds with matching unit coverage.

* feat(daemon): topology readiness gates and selectable reranker

Drive topologyReady from published epoch/freshness/rebuild/dirty
ingest state (not just KG drain), surface those fields in search
stats, and wire onnx|simeon|auto|none cross-reranker selection.

* chore(search): drop meta_path tuner knobs

Meta-path routing is gone; remove dead TunedParams fields and the
ConfigResolver test for [search.meta_path].

* feat(bench): add topology_purity_validate plan

Repeats=3 validation matrix for construction-purity winners vs
topo_off so MRR deltas can clear run-to-run noise.

* chore: format add_command and config include order

Mechanical clang-format / include reorder only; no behavior change.

* feat(bench): decision-grade xplan loop for search engine

Document search-engine measurement in AGENTS; harden xplan with
repeats=3 plans, quality×latency primaries, mechanism metrics from
debug.jsonl, multi-rep log promotion, and archive superseded topology
plans.

* feat(graph): denser product semantic_neighbor topK

Raise default YAMS_GRAPH_SEMANTIC_TOPK fallback from 4 to 8 so product
graphs leave fewer degree-0 docs for GraphNeighbors expansion.

* feat(search): extract topology assist and cross-rerank seams

Pull topology assist into search_topology_stage (seed merge, skip
reasons, optional vector seed probe default 0). Extract cross_rerank
compute from search_engine. Wire probe via typed config/ConfigResolver.

* feat(bench): product-path search engine ablation plans

Add topology-off component, vector-weight, and tiered-execution xplan
plans (repeats=3) and wire them into the AGENTS measurement loop table.

* fix(ci): repair pre-push fallbacks

* feat(bench): lock-free mid-run builds + multi-corpus gates

xplan skips meson rebuild when the quality binary exists (flock +
YAMS_BENCH_FORCE_BUILD override) to stop multi-arm link races. Add
graph_vector weight and nfcorpus transfer-gate plans; document in AGENTS.

* fix(daemon): keep SearchEngine alive via shared_ptr cache

getCachedEngine previously published a raw pointer that metrics polling
used through rebuild, racing destructor of SearchEngine::Impl (TSAN).
Store and return a shared_ptr snapshot so diagnostic readers pin the
engine for the duration of getConfig/health checks.

Also adds the cautious vectorWeight 0.30→0.20 multi-corpus xplan plan.

* feat(search): route vectors through cached topology

Build and reuse validated topology snapshots, rank routed members by query similarity, and narrow the vector leg to the accepted document set with explicit work diagnostics.

Remove the retired HDBSCAN engine and vendor dependency, preserve typed topology policy through search tuning, and cover the runtime/configuration contracts.

* feat(bench): consolidate search decision plans

Enforce deterministic experiment identity and declared baselines, expose routed/global vector work metrics, and register the focused benchmark contracts.

Promote the compact repeats=3 routing-budget gate, archive superseded topology mechanism screens, and add multi-corpus product calibration plans.

* feat(search): refine topology vector routing

* fix(search): propagate daemon readiness

* feat(bench): measure generalized memory health

* fix(cli): honor in-process fallbacks

* ci: harden smolvm debug builds

* perf(search): route topology through filtered ANN

* feat(bench): measure read-write pressure

* perf(metadata): index symbol lookups on read pool

* fix(daemon): isolate startup reads and maintenance

* feat(bench): replay isolated generalized corpora

* feat(search): add evidence routing pipeline

Replace the blocking smolvm pre-push path with host ASan/TSan validation, harden sanitizer-sensitive smoke tests, and add recovery cleanup for malformed SQLite WAL sidecars.

* refactor(search): unify topology evidence path

Remove classic and sidecar fusion lanes so topology augments the recall-preserving ANN and lexical candidate pool through one product path.

* refactor(bench): prune superseded search lanes

Measure the topology-plus-ANN product path with stable experiment identity and remove obsolete sidecar and tiered controls.

* docs(search): retire stale experiment notes

Keep the benchmark handoff focused on the topology-plus-ANN product lane and its active decision plans.

* refactor(search): prune dead search surfaces

Remove unused fusion, reranker, policy, and router abstractions.
Bound topology materialization and keep document trace payloads opt-in.

* fix(cli): clear build warnings

* fix(xplan): reject stale run identity

* refactor(search)!: remove legacy query stack

Replace the unused AST/ranker/post-processor surface with focused facet aggregation used by product search.

BREAKING CHANGE: removes query parser, result ranker, search filter, and SearchResults container APIs from yams_search.

* refactor(search): type value stage traces

Remove synthetic ComponentResult markers for embedding and concept stages. Preserve failure and timeout states instead of overwriting them with fake successful results.

* refactor(search): isolate benchmark env overlays

Move test-only search environment parsing behind an injectable internal adapter. Keep all existing overlay keys and tuner pin semantics while restoring SearchEngineBuilder to composition.

* refactor(search): unify topology routing options

Snapshot topology policy once at the assist boundary and pass it through routing sessions and diagnostics. Remove repeated scalar copying without changing routing behavior or metric keys.

* fix(search): unify tuner config ownership

Use one complete config-to-params overlay for runtime seeding and no-tuner policy resolution. Reseed attached tuners on setConfig so reported and executed configuration cannot diverge.

* fix(search): snapshot tuner state per query

Read tuned config, params, and state under one lock before routing. Add a TSAN correlation test so adaptive observation cannot produce a torn logical query snapshot.

* fix(search): enforce topology routing contract

Generated configs must preserve hybrid-assist shadow routing, while daemon-backed search must abstain when published topology artifacts are stale. Keep standalone validation and global ANN recall intact.

* fix(daemon): let SQLite reclaim its file

Directly resizing a live database from sampled page counts can truncate pages allocated by concurrent writers. VACUUM must remain the only file-reclamation mechanism.

* fix(metadata): tolerate missing symbol FTS

Readiness lookups can race an unavailable optional FTS migration. Retain the indexed fast path, but retry against the base table when prepare or cached-statement execution proves the capability absent.

* build(fuzz): drop removed query parser target

The optional AFL build still referenced the deleted legacy query parser API and could no longer compile.

* fix(bench): include writes in drain oracle

A zero post-ingest queue is not durable drain while coordinated writes remain queued or in flight.

* fix(daemon): vacuum padded WAL databases safely
2026-07-13 20:26:43 -04:00
trvon
43ebe26d36
fix: ci stabilizations for release 2026-07-04 09:28:59 -04:00
trvon
b2ede10e54
fix: stabilize CI release gates
Fix DB vacuum, compression gates, fuzzy search, temp dirs.
   Repair release packaging: Bash 3.2, Arch containers,
   package validation defaults, artifact names, matrix metadata.

   Local tests + release act lane pass.
2026-07-03 22:40:05 -04:00
trvon
bed8578857
fix(ci): stablize ci 2026-07-02 21:34:18 -04:00
trvon
e58aef6cf5
fix(ci): isolate arch package builds
- stop arch lanes from reusing build/release
   - fix local arch lane quoting
   - align release arch lane with isolated build dirs
2026-07-01 10:02:54 -04:00
trvon
5ce5e6a774
feat: Arch Linux packaging + CI
Add arch:
   - PKGBUILD + yams.install in packaging/arch/
   - build-arch-pkg.sh (pacman deps, makepkg, repo-add)
   - arch-lane.Dockerfile systemd validation substrate
   - arch-package.Dockerfile pre-baked CI builder
   - package-lane.sh --only arch (build + validate local lane)
   - package-validate.sh Arch lane (pacman -U/-R smoke-test)
   - release.yml: 2 new matrix entries, arch-repo metadata, R2 upload
   - publish-repos.yml: arch-repo download + R2 upload
   - yams-repo worker: /archrepo route, .pkg.tar.zst + .db/.files types
2026-07-01 05:50:59 +00:00
trvon
aea9b64404
feat(packaging): mature systemd service install + local validation 2026-06-24 12:57:16 -04:00
Trevon
b32e1e06e6
v0.2.5 2025-08-14 06:34:56 -04:00
Trevon
847fefb97a
CHANGELOG updated. v0.0.4 and test improvements 2025-08-11 18:40:05 -04:00
Trevon
917aecf514
yams 2025-08-10 23:41:49 -04:00