The ONNX plugin cached compiled CoreML models directly in the model
directory. With the plugin now able to run on either the system or the
bundled ONNX Runtime, a cache compiled by one version was reused by the
other: on macOS, after a run on Homebrew's 1.30 the bundled 1.23 failed
every embedding batch with 'Non-zero status code returned while running
..._CoreML_... node'. The default cache is now
<model dir>/coreml-ort-<runtime version>; YAMS_COREML_CACHE_DIR still
overrides it.
Verified on macOS arm64: a run on the system runtime creates
coreml-ort-1.30.0, a run with Homebrew's copy hidden uses the bundled
runtime, creates coreml-ort-1.23.0 and embeds new documents without
CoreML errors.
Signed-off-by: trvon <me@trevon.dev>
On Linux the Glint plugin always built in mock mode: meson found the ONNX
Runtime headers but its find_library('onnxruntime') looked in a libdir
taken from the already-normalized dependency, found nothing, and dropped
YAMS_USE_ONNX_RUNTIME, so GLiNER never ran (the v0.20.3 deb's glint has
no ONNX Runtime dependency either). On macOS it linked Homebrew's copy
directly.
Glint now uses the shared ONNX Runtime loader like the ONNX plugin: it
compiles against the headers only (ORT_API_MANUAL_INIT through
ort_cxx_api_wrapper.h), and before creating the session it resolves a
runtime with the same system-first, bundled-fallback policy. If none is
available the session reports "ONNX Runtime unavailable: ..." and entity
extraction is disabled instead of silently running the mock. Plugin
health gains an onnx_runtime object (available, source, version, path or
error), and [plugins.glint] runtime_library pins a copy.
Verified on grim (Linux, release, staged tree in a clean debian:testing
container): "Glint plugin: ONNX Runtime enabled (loaded at runtime)" at
configure, "[Glint] Loaded ONNX model with 6 inputs, 5 outputs" at run
time, and the daemon's title+NL stage extracts entities ("nl=5" for a
document naming people, companies, a place and a date).
Signed-off-by: trvon <me@trevon.dev>
The ONNX plugin dlopen()ed ONNX Runtime from directories next to the
plugin first and only then from a few system prefixes, with no notion of
which copy it got or why, and the shipped packages contained no copy at
all, so on most hosts the plugin could only report "runtime not found".
The runtime resolution moves into plugins/ort_runtime/, built as a
hidden-visibility static library both ONNX-using plugins link (their
export maps still expose only the plugin C ABI). The policy is split out
as pure code (ort_runtime_resolution.{h,cpp}) so it is testable without a
runtime. Candidates are tried in this order and the first compatible one
wins:
1. YAMS_ONNX_RUNTIME_LIB / YAMS_ONNX_RUNTIME_DIR (existing pin, kept),
then the plugin config key runtime_library ([plugins.onnx] /
[plugins.glint], a file or directory)
2. a system copy: the loader search by soname, then the usual library
directories (multiarch dir, /usr/lib64, /usr/lib, /usr/local/lib,
/opt/onnxruntime, Homebrew) for versioned names such as Debian's
libonnxruntime.so.1.23
3. the private copy in <libdir>/yams/onnxruntime/
4. legacy locations next to the plugin (build trees)
A candidate is accepted only if it exports OrtGetApiBase, GetApi() returns
an API table for the ORT_API_VERSION the plugins are compiled against, and
it meets the 1.23 minimum; otherwise it is skipped with a reason. A soname
lookup that returns a copy another plugin already loaded from the bundled
directory is reported as that copy. The choice ("Using system|bundled ONNX
Runtime <ver> from <path> (<why>)") and every skipped candidate go to the
plugin log and, through the libyams_onnx_resource host sink (new
OnnxConcurrencyRegistry::emitHostLog), to the daemon log; plugin health
reports runtime_source. With no compatible copy the plugin reports "ONNX
Runtime unavailable: ..." instead of failing.
Tests: tests/unit/plugins/ort_runtime_resolution_catch2_test.cpp covers
the order, the override pin, configured directories, system-over-bundled
preference, skipping an incompatible system copy, the not-found result
and the already-loaded classification.
Signed-off-by: trvon <me@trevon.dev>
Plugins and libyams_onnx_resource were linked with default visibility, so
their dynamic symbol tables exported everything they contained, including
statically linked third-party code. The S3 plugin exported 11,147
symbols (OpenSSL, libcurl, zlib, spdlog, nlohmann_json); the ONNX
resource library exported 895 (850 of them spdlog). Every loader uses
RTLD_LOCAL, so this was not a live interposition bug, but the export
table is the ABI and anything that binds RTLD_GLOBAL (or a future host)
could resolve into a plugin's private copies.
Plugins:
- plugins/exports/yams_plugin.map (ELF version script, global
yams_plugin_*, local *) and yams_plugin.exp (ld64
-exported_symbols_list) are applied to every plugin (s3, onnx, glint,
zyp) with gnu_symbol_visibility: 'hidden' and --exclude-libs,ALL.
- The S3 legacy factory (yams_plugin_create/destroy_object_storage) is
marked YAMS_PLUGIN_API so it stays exported under hidden visibility.
libyams_onnx_resource:
- Builds with hidden visibility; YAMS_ONNX_RESOURCE_API now expands to
visibility("default") on GCC/Clang, so only OnnxConcurrencyRegistry is
exported.
- It no longer links spdlog. Its messages reached the host log only
because the executable's spdlog interposed the library's copy; with the
copy hidden, the registry would have logged through a private default
logger to stdout. The registry now takes a host log sink
(OnnxConcurrencyRegistry::setLogSink); TuningManager and
AbiModelProviderAdapter install a spdlog forwarder from the header-only
onnx_registry_spdlog_sink.h. Without a sink, messages are dropped.
Tests (red first on grim, Debug):
- tests/plugins/plugin_export_surface_catch2_test.cpp dlopens the built
S3 and zyp plugins and asserts the entry points resolve while libcurl,
OpenSSL, zlib and zyp internals do not: 2/2 cases failed (11 of 31
assertions) before, pass after.
- resource_governance_test gains a case for the log sink routing (did
not compile before; passes after). `meson test --suite plugins`: 5/5.
Measured on grim (release, stripped staged payload):
file bytes before after exports before after
libyams_object_storage_s3.so 7,854,064 6,240,112 11,147 10
yams_zyp.so 80,112 63,728 103 8
libyams_onnx_resource.so 477,472 30,840 895 20
bin/yams-cli 33,712,056 33,859,512 80 39
bin/yams-daemon 28,389,784 28,537,304 98 39
The executables now carry their own spdlog code paths instead of
exporting them for the resource library to interpose (+0.4%), and export
41 fewer symbols each. check_install_surface turns GREEN; budgets are
reset to the measured payload plus ~20%.
Signed-off-by: trvon <me@trevon.dev>
`meson install` of a Release tree staged 2.4 GiB: all 28 internal static
libraries (1.3 GiB), the whole include/ tree (459 files), 22 pkg-config
files for libraries that were never meant to be linked from outside, and
the ONNX resource library in the system libdir. Packages hid this by
deleting include/, *.a and pkgconfig/ after the fact.
- Internal static libraries are `install: false` and their pkg.generate()
calls are gone, as is the top-level yams.pc (it required yams-api, which
is not installable). Benchmarks are no longer installed either.
- Headers are not installed: no installable C++ library exists to link
them against. The one SDK surface, the mobile C ABI, installs
mobile_bindings.h and its .pc under the 'devel' install tag.
- Runtime installs carry explicit install_tag: 'runtime' where Meson
cannot infer one (data, systemd unit/preset, the yams alias symlink,
Windows DLL copies, the libzpdf install script), so
`meson install --tags runtime` stages the same set as a plain install.
- libyams_onnx_resource moves from <libdir> to the private <libdir>/yams
on Linux/macOS, without a soname version since nothing outside this
build links it. yams-cli/yams-daemon RUNPATH becomes $ORIGIN/../lib/yams
(@executable_path on macOS) and the ONNX/Glint plugins use $ORIGIN/..
so they resolve it and the bundled ONNX Runtime, which also moves to
<libdir>/yams (the ONNX plugin's runtime loader already searches its
parent directory). Windows keeps its existing DLL layout.
- Homebrew templates and the macOS install smoke follow the new layout.
- scripts/prune-runtime-install.sh only strips now.
Measured on grim (Linux x86_64 Release, prefix /usr/local):
staged paths 522 -> 11, staged bytes 2.4 GiB -> 1.0 GiB (unstripped)
check_install_surface path violations: 6 -> 0
`--tags devel` stages nothing; ldd resolves libyams_onnx_resource from
bin/../lib/yams and the staged `yams --version` runs.
Signed-off-by: trvon <me@trevon.dev>
* build: standardize the Debug build dir on build/debug
`docs/BUILD.md` and `AGENTS.md` document `./setup.sh Debug` producing
`build/debug`, but `setup.sh`/`setup.ps1` created `builddir` and `.clangd`
pointed there too, so the documented layout and the actual tree disagreed.
Align the Debug mapping with the documented layout:
- setup.sh / setup.ps1: Debug -> `build/debug`
- CMakeLists.txt: Debug `_meson_build_dir` -> `build/debug`
- .clangd: CompilationDatabase -> `build/debug`
- plugin test fallbacks, S3 smoke default, quality-gate candidate list, and
doc examples -> `build/debug`
Legacy `builddir` trees stay readable: the quality gate and local-ci scripts
still probe `builddir` as a fallback, and `plugin_trust` already treats
`/build/` paths as build artifacts.
* build: migrate remaining Debug consumers off builddir
Follow-up to the Debug build-dir standardization: tracked consumers still
hardcoded `builddir` and would break or silently skip against a fresh
`build/debug` tree. Migrate them, keeping tolerant fallbacks where a legacy
tree may still exist.
- CI: `.github/workflows/tests.yml` runs `./setup.sh Debug` and then compiled
`-C builddir`; switch BUILD_DIR/COMPILE_DB_DIR/compile paths to `build/debug`.
Same for windows-test-diagnose.yml.
- IDE: `.vscode/tasks.json` build/test/bench tasks target `build/debug`.
- Runtime: `object_storage_plugin_loader` also probes `build/debug/plugins`
(and the flat `build/debug`), retaining the `builddir` fallbacks.
- Tests/docs: ONNX tuning integration plugin dir, benchmarks, and plugin
READMEs/examples use `build/debug`.
`./build/*` is already excluded from the quality-gate find and `.dockerignore`,
so no exclusion lists change.
Remove legacy Tree-sitter symbol extractor plugin (plugins/symbol_extractor_treesitter),
the unused plugin-symbols build option, and associated tests and benchmark targets:
- Drop plugins/symbol_extractor_treesitter/ (grammar_loader, symbol_extractor, plugin_api).
- Remove plugin-symbols Meson option and packaging references.
- Remove dedicated symbol plugin tests and benchmark targets (symbol_extraction_bench,
symbol_extractor_all_langs_catch2_test, symbol_extractor_edge_cases_catch2_test,
symbol_extractor_solidity_catch2_test, symbol_extractor_query_test,
symbol_extractor_validation_test).
- Clean up dead symbol extraction plugin helper and test from hnsw_entity_search_bench.
- Prune obsolete portability allowlist entries.
Signed-off-by: trvon <me@trevon.dev>
* test(memory-sync): give the CLI fixture the shared corpus scope it needs
94e948c6 made createMemorySyncService fail closed unless
memory_sync.corpus_scope is "shared" and updated the service-manager
test, but not the p2p CLI fixture, whose eight cases have failed at
construction ever since ("replication requires
memory_sync.corpus_scope=shared"). The fixture now sets the scope; the
cases run again and pass.
Signed-off-by: trvon <me@trevon.dev>
* refactor(memory_sync): move MemorySyncLoop bodies into a translation unit
include/yams/memory_sync/memory_sync.h was a 3,527-line header-only
class: 120 inline method definitions, 2,900 lines of implementation,
pulled into every translation unit that includes daemon.h or the p2p
protocol header. The memory-sync subsystem had no source directory at
all.
MemorySyncLoop's bodies now live in src/memory_sync/memory_sync.cpp,
compiled once into a new yams_memory_sync static library that follows
the yams_wal pattern. The header keeps the declarations, the nested
types, the constructor, one-line accessors, and the YAMS_TESTING hooks
(those must stay inline because the library is not built with the
testing define). The daemon library and its declared dependency link
the new library, as do the five memory-sync test binaries.
Zero semantic change: the move is mechanical (a scripted split that
copies each body verbatim, drops default arguments and static from the
definitions, and qualifies nested return types). All seven memory-sync
test binaries and the service-manager suite pass unchanged.
memory_sync.h: 3,527 -> 569 lines.
Signed-off-by: trvon <me@trevon.dev>
* refactor(daemon): move TuneAdvisor method bodies out of the header
TuneAdvisor.h was 3,345 lines, almost all of it the bodies of ~300
static accessors and setters, included by 60 translation units. Each
of them recompiled the whole policy surface on every touch, and any
change to one knob rebuilt most of the daemon.
The bodies now live in src/daemon/components/TuneAdvisor.cpp (which
previously held only the system-memory probe). The header keeps the
declarations, the static inline override slots (unchanged, so the
ODR story for process-wide tuning state is exactly what it was), the
YAMS_TESTING hooks, and the two methods whose bodies are
preprocessor-conditional on YAMS_TESTING; those must stay inline
because the library is not built with the testing define and moving
them would silently switch tests to the production branch.
Mechanical move by the same scripted split used for memory_sync.h:
bodies copied verbatim, static and default arguments dropped from the
definitions, nested return types qualified. The one template
(readConfiguredOverridesSnapshot) stays in the header because three
other translation units instantiate it.
The header is included well outside src/daemon (vector, repair, app
services, MCP, the CLI doctor, the ONNX plugin), and those targets do
not link yams_daemon. They used to get the bodies for free from the
header, so the move needs a home they can all link: the new
yams_tune_advisor static library (src/daemon/tune_advisor/meson.build)
builds TuneAdvisor.cpp alone, and yams_daemon, yams_vector,
yams_repair, yams_app_services, yams_mcp, yams_cli, the ONNX plugin,
and the standalone snapshot test depend on it.
Executables compiled with -DYAMS_TESTING link a second build of the
same file, yams_tune_advisor_testing, ahead of the production archive.
Under that define compatibilityEnvironment() reads the process
environment live instead of the startup snapshot, which every
env-driven tuning test relies on; before the split each such TU
instantiated the inline bodies itself with the define. Meson emits
dependencies in list order and the first archive defining a symbol
wins, so the order is load-bearing: test_deps, the benchmark
dependency lists, and the mobile ABI test all put the testing archive
first, and tests/meson.build says so. The testing archive is built
under the same predicate that enters tests/ (a plain debug configure
included), so `meson setup --buildtype=debug` keeps working. The
standalone snapshot test keeps the production build on purpose.
No behavior change; a full debug compile is clean and the tuning
suites pass unchanged.
TuneAdvisor.h: 3,345 -> 1,170 lines.
Signed-off-by: trvon <me@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
plugins/hound was a single README for an abandoned experiment, and
tools/hound_import.py imported a yams_sdk.hound module that no longer
exists anywhere in the tree. Removing them also removes the only
mention of createGraphAdapterV1, which let three headers go:
- include/yams/daemon/resource/graph_adapter.h: declared IGraphAdapter
and three factory functions with no definition and no includer.
- include/yams/daemon/resource/abi_content_extractor_adapter.h: a
self-described PBI-096 compatibility forwarder with no includer.
- include/yams/core/fsm_helpers.h: shared_ptr wrappers around
daemon/ipc/fsm_helpers.h with no includer; the one consumer already
includes the real header.
Proof of zero references (build dirs excluded):
rg -n 'graph_adapter\.h|abi_content_extractor_adapter|core/fsm_helpers\.h|hound' \
--glob '!build/**' --glob '!third_party/**'
→ only the deleted files themselves.
Signed-off-by: trvon <me@trevon.dev>
* chore: remove dead post-ingest helper
Remove shouldPrepareSequentially, the sole Fold unused-function finding
confirmed dead by clean ASan and TSan builds. All other 45 findings have
live callers or uses that Fold's bounded model missed and remain intact.
Fold provenance: build/fold/release-prep-79.sarif (2026-09-01 run).
Signed-off-by: trvon <me@trevon.dev>
* chore(plugins): document Fold ownership
Document 6 proved memory-leak findings as intentional ownership:
process-lifetime Ort::Env singletons and host-freed C-ABI result data.
Refresh portability allowlist anchors shifted by the comments.
Fold provenance: build/fold/release-prep-79.sarif (2026-09-01 run).
Signed-off-by: trvon <me@trevon.dev>
* fix(storage): treat reserve boundary as emergency
Classify available bytes equal to the configured emergency reserve as Emergency, matching the hard-minimum contract and operator message that the reserve has been reached. Add a regression assertion at the exact boundary.
Signed-off-by: trvon <git@trevon.dev>
---------
Signed-off-by: trvon <me@trevon.dev>
Signed-off-by: trvon <git@trevon.dev>
Rebuild macOS TSan outputs so compiler instrumentation and compiler-rt come from one Apple toolchain generation. Preserve daemon fixture dependencies until ServiceManager teardown so sanitizer lanes do not exercise a test-only use-after-free.
* refactor(tuning): prune dormant runtime knobs
Remove unused tuning controls and adaptive search budget state. Extract testable ONNX slot reconciliation and search telemetry hydration seams, and correct the Apple Silicon sysctl value width.
* refactor(vector): retire dormant backends
Remove unused FAISS, model-infrastructure, VectorIndexManager, and TurboQuant paths while keeping Simeon PQ and float-vector storage.
Migrate unused V2.2 quantized schemas transactionally to V2.3. Databases containing non-default quantized formats or packed-code blobs fail closed with recovery guidance and no mutation.
* refactor(plugins): remove legacy SDK tooling
Remove the Ghidra and SDK submodules, their bespoke tests, and stale scan exclusions. Retain generic PluginProcess coverage against the repository mock fixture.
* build(vector): remove stale FAISS overrides
* refactor(ipc): remove dormant FSM snapshots
* refactor(ingest): unify bounded KG enqueue
* fix(plugins): harden allocation contracts
Keep ABI outputs null or fully initialized when allocations fail, and represent empty object payloads without relying on malloc(0).
* fix(storage): define empty object reads
Return a null buffer with zero length for successful empty-object reads instead of treating malloc(0) as an allocation contract.
* refactor(storage): trim through string views
* test: prune dormant harness helpers
Remove unused readiness and fixture helpers, and keep benchmark client-count doubling in a wide integer domain.
* chore(cpp): initialize analyzer-facing locals
* build(simeon): advance analyzer cleanup
* refactor(config): drop dead daemon client keys
* refactor(ipc): inject daemon client transport
Socket and in-process execution now share DaemonClient request dispatch, so CLI fallbacks do not need alternate service paths.
* refactor(retrieval): route CLI through daemon
Search and grep now use the same daemon handlers over socket or in-process transport. Live grep remains an explicit freshness inspection.
* refactor(graph): move CLI scope into daemon
Graph CLI requests now share daemon handlers across socket and in-process transports. CWD filtering occurs before pagination so scoped totals and pages stay correct.
* build: isolate test-only runtime controls
* refactor(metadata): decouple lock telemetry
* fix(services): return on request timeout
* fix(retrieval): enforce workspace scope
* fix(graph): scope nodes before pagination
* fix(grep): drop binary result lines
* fix(daemon): bound checkpoint wait
* fix(repair): use KG health checks
* fix(storage): always verify compressed CRC
* perf(vector): skip unchanged PQ snapshots
* build(zyp): exclude bench from default build
* build: protect Meson files from formatter
* refactor: remove unreachable production APIs
* refactor(daemon): remove dormant runtime state
* fix(vector): harden daemon provider calls
Timed-out coroutines must own their client and requests. Provider discovery must also fail instead of returning synthetic embeddings in production.
* refactor(storage): remove unusable adapters
* fix(cli): unify in-process add requests
* fix(tsan): preserve async worker progress
* test(cli): isolate daemon-dependent suites
* fix(repair): expose queue backoff progress
Add focused API, CLI, doctor, downloader, search, and vector
infrastructure coverage.
Cover retry seams, path-scoped search fallback, renderer fail-soft
paths, router helpers, and model infrastructure without changing
production behavior.
Keep coverage gates stable by skipping Zyp on Darwin coverage builds
and ignoring SIGPIPE in daemon protocol tests.
- Implement Document::extractAllParallelized() using std::async
to distribute pages across threads with per-thread Document
instances, achieving true parallelism (not the old fake alias)
- zyp_plugin.cpp now uses extractAllParallelized() by default
for PDF extraction during ingestion
- Add zyp_extraction_bench standalone benchmark measuring
sequential vs 2/4/N-thread extraction throughput (pages/s, MB/s)
Build: ninja bench_zyp
Run: build/debug/plugins/zyp/zyp_extraction_bench <pdf>
- Add IContentExtractor::extractTextAndMetadata() to carry plugin metadata
- Add extractDocumentContent() in extraction_util for text+metadata
- PIQ uses plugin-provided title from PDF metadata, skipping GLiNER
entirely when a title already exists (e.g. from zyp PDF parsing)
- Fix misleading 'parallel' naming in zyp wrapper/comments
(zpdf_extract_all_parallel is actually sequential)
This removes the dominant slow-write source for paperbridge PDF
ingestion — title metadata batches that previously backed up the
WriteCoordinator with 34s queue waits.
