Commit graph

15 commits

Author SHA1 Message Date
trvon
93f2439471
build(plugins): export only the plugin C ABI; hide onnx_resource internals
Plugins and libyams_onnx_resource were linked with default visibility, so
their dynamic symbol tables exported everything they contained, including
statically linked third-party code. The S3 plugin exported 11,147
symbols (OpenSSL, libcurl, zlib, spdlog, nlohmann_json); the ONNX
resource library exported 895 (850 of them spdlog). Every loader uses
RTLD_LOCAL, so this was not a live interposition bug, but the export
table is the ABI and anything that binds RTLD_GLOBAL (or a future host)
could resolve into a plugin's private copies.

Plugins:
- plugins/exports/yams_plugin.map (ELF version script, global
  yams_plugin_*, local *) and yams_plugin.exp (ld64
  -exported_symbols_list) are applied to every plugin (s3, onnx, glint,
  zyp) with gnu_symbol_visibility: 'hidden' and --exclude-libs,ALL.
- The S3 legacy factory (yams_plugin_create/destroy_object_storage) is
  marked YAMS_PLUGIN_API so it stays exported under hidden visibility.

libyams_onnx_resource:
- Builds with hidden visibility; YAMS_ONNX_RESOURCE_API now expands to
  visibility("default") on GCC/Clang, so only OnnxConcurrencyRegistry is
  exported.
- It no longer links spdlog. Its messages reached the host log only
  because the executable's spdlog interposed the library's copy; with the
  copy hidden, the registry would have logged through a private default
  logger to stdout. The registry now takes a host log sink
  (OnnxConcurrencyRegistry::setLogSink); TuningManager and
  AbiModelProviderAdapter install a spdlog forwarder from the header-only
  onnx_registry_spdlog_sink.h. Without a sink, messages are dropped.

Tests (red first on grim, Debug):
- tests/plugins/plugin_export_surface_catch2_test.cpp dlopens the built
  S3 and zyp plugins and asserts the entry points resolve while libcurl,
  OpenSSL, zlib and zyp internals do not: 2/2 cases failed (11 of 31
  assertions) before, pass after.
- resource_governance_test gains a case for the log sink routing (did
  not compile before; passes after). `meson test --suite plugins`: 5/5.

Measured on grim (release, stripped staged payload):

  file                          bytes before  after       exports before  after
  libyams_object_storage_s3.so   7,854,064   6,240,112          11,147     10
  yams_zyp.so                       80,112      63,728             103      8
  libyams_onnx_resource.so         477,472      30,840             895     20
  bin/yams-cli                  33,712,056  33,859,512              80     39
  bin/yams-daemon               28,389,784  28,537,304              98     39

The executables now carry their own spdlog code paths instead of
exporting them for the resource library to interpose (+0.4%), and export
41 fewer symbols each. check_install_surface turns GREEN; budgets are
reset to the measured payload plus ~20%.

Signed-off-by: trvon <me@trevon.dev>
2026-10-01 18:51:34 -06:00
Trevon
577adf1e0f
build: standardize the Debug build dir on build/debug (#248)
* build: standardize the Debug build dir on build/debug

`docs/BUILD.md` and `AGENTS.md` document `./setup.sh Debug` producing
`build/debug`, but `setup.sh`/`setup.ps1` created `builddir` and `.clangd`
pointed there too, so the documented layout and the actual tree disagreed.

Align the Debug mapping with the documented layout:
- setup.sh / setup.ps1: Debug -> `build/debug`
- CMakeLists.txt: Debug `_meson_build_dir` -> `build/debug`
- .clangd: CompilationDatabase -> `build/debug`
- plugin test fallbacks, S3 smoke default, quality-gate candidate list, and
  doc examples -> `build/debug`

Legacy `builddir` trees stay readable: the quality gate and local-ci scripts
still probe `builddir` as a fallback, and `plugin_trust` already treats
`/build/` paths as build artifacts.

* build: migrate remaining Debug consumers off builddir

Follow-up to the Debug build-dir standardization: tracked consumers still
hardcoded `builddir` and would break or silently skip against a fresh
`build/debug` tree. Migrate them, keeping tolerant fallbacks where a legacy
tree may still exist.

- CI: `.github/workflows/tests.yml` runs `./setup.sh Debug` and then compiled
  `-C builddir`; switch BUILD_DIR/COMPILE_DB_DIR/compile paths to `build/debug`.
  Same for windows-test-diagnose.yml.
- IDE: `.vscode/tasks.json` build/test/bench tasks target `build/debug`.
- Runtime: `object_storage_plugin_loader` also probes `build/debug/plugins`
  (and the flat `build/debug`), retaining the `builddir` fallbacks.
- Tests/docs: ONNX tuning integration plugin dir, benchmarks, and plugin
  READMEs/examples use `build/debug`.

`./build/*` is already excluded from the quality-gate find and `.dockerignore`,
so no exclusion lists change.
2026-10-01 08:45:20 -06:00
trvon
2261019ee9
feat(storage): add sync backend primitives 2026-08-17 20:03:34 -06:00
Trevon
0dd3985db4
refactor: simplify runtime paths and harden retrieval (#61)
* refactor(tuning): prune dormant runtime knobs

Remove unused tuning controls and adaptive search budget state. Extract testable ONNX slot reconciliation and search telemetry hydration seams, and correct the Apple Silicon sysctl value width.

* refactor(vector): retire dormant backends

Remove unused FAISS, model-infrastructure, VectorIndexManager, and TurboQuant paths while keeping Simeon PQ and float-vector storage.

Migrate unused V2.2 quantized schemas transactionally to V2.3. Databases containing non-default quantized formats or packed-code blobs fail closed with recovery guidance and no mutation.

* refactor(plugins): remove legacy SDK tooling

Remove the Ghidra and SDK submodules, their bespoke tests, and stale scan exclusions. Retain generic PluginProcess coverage against the repository mock fixture.

* build(vector): remove stale FAISS overrides

* refactor(ipc): remove dormant FSM snapshots

* refactor(ingest): unify bounded KG enqueue

* fix(plugins): harden allocation contracts

Keep ABI outputs null or fully initialized when allocations fail, and represent empty object payloads without relying on malloc(0).

* fix(storage): define empty object reads

Return a null buffer with zero length for successful empty-object reads instead of treating malloc(0) as an allocation contract.

* refactor(storage): trim through string views

* test: prune dormant harness helpers

Remove unused readiness and fixture helpers, and keep benchmark client-count doubling in a wide integer domain.

* chore(cpp): initialize analyzer-facing locals

* build(simeon): advance analyzer cleanup

* refactor(config): drop dead daemon client keys

* refactor(ipc): inject daemon client transport

Socket and in-process execution now share DaemonClient request dispatch, so CLI fallbacks do not need alternate service paths.

* refactor(retrieval): route CLI through daemon

Search and grep now use the same daemon handlers over socket or in-process transport. Live grep remains an explicit freshness inspection.

* refactor(graph): move CLI scope into daemon

Graph CLI requests now share daemon handlers across socket and in-process transports. CWD filtering occurs before pagination so scoped totals and pages stay correct.

* build: isolate test-only runtime controls

* refactor(metadata): decouple lock telemetry

* fix(services): return on request timeout

* fix(retrieval): enforce workspace scope

* fix(graph): scope nodes before pagination

* fix(grep): drop binary result lines

* fix(daemon): bound checkpoint wait

* fix(repair): use KG health checks

* fix(storage): always verify compressed CRC

* perf(vector): skip unchanged PQ snapshots

* build(zyp): exclude bench from default build

* build: protect Meson files from formatter

* refactor: remove unreachable production APIs

* refactor(daemon): remove dormant runtime state

* fix(vector): harden daemon provider calls

Timed-out coroutines must own their client and requests. Provider discovery must also fail instead of returning synthetic embeddings in production.

* refactor(storage): remove unusable adapters

* fix(cli): unify in-process add requests

* fix(tsan): preserve async worker progress

* test(cli): isolate daemon-dependent suites

* fix(repair): expose queue backoff progress
2026-07-31 14:16:45 -04:00
Trevon
771deebc84
fix: harden CI and metadata stability (#48)
* feat: dead code removal, profling foudnation improvements, addressing code smells in reference db implementation

* fix: test fixes and optimmizations for benchmarks

* fix: harden audit findings

   Reject invalid tokenizer ids, fix move-order hazards, bound manifest
   and audio IO, and guard numeric/process edge cases found by Fold.

* fix: route SymSpell writes through coordinator

* test: fix batch content initializers

* fix(ci): restore cross-platform builds

* fix: address Copilot review findings
2026-06-11 16:42:54 -04:00
trvon
ef3d925797
feat: continued corectness improvements, ANN / vectordb improvements 2026-04-12 12:53:29 -04:00
trvon
41a2282691
feat: expanding storage system to work with r2 backends via object storage plugin. Updating benchmarks and documentation 2026-03-07 19:43:14 -05:00
trvon
92bb0c5bfd
fix: fixing s3 signing bug 2026-03-06 16:51:58 -05:00
trvon
91e212389c
feat: fixing TSAN issues in MCP. implementing cloudflare code mode https://blog.cloudflare.com/code-mode-mcp/ 2026-02-20 20:39:12 -05:00
trvon
aacd7af41f
running linter 2025-12-13 16:27:07 -05:00
trevon
86378b8cb6
testing improvements, bug fixes for plugin loading and logic. Search engine bug fix 2025-12-12 04:54:25 -05:00
Trevon
43699f99a3
v0.7.0-prerelease 2025-09-20 22:55:34 -04:00
Trevon
3fbb49ee96
v0.6.13 hotfixes 2025-09-11 12:55:18 -04:00
Trevon
956c43b80b
v0.6.12 experiemental 2025-09-11 11:28:42 -04:00
Trevon
91830ae23e
protobufs / boost for daemon upgrades. start of plugin system 2025-08-31 06:34:51 -04:00