Commit graph

20 commits

Author SHA1 Message Date
Trevon
211db15606
fix(daemon): bound semantic neighbor scores and report startup phase (#334)
Semantic-neighbor maintenance aborted the daemon during startup:
std::clamp<float> asserted '!(__hi < __lo)' because the adaptive cosine
threshold could exceed 1.0 for a legal pair of identical vectors. The same
work was also admitted while the daemon was still initializing, so the
300-second starvation guard forced a pass before readiness.

+ semanticInverseNorm/semanticCosineSimilarity/semanticExplicitThreshold in the
  existing embed policy header, shared by the streamed and corpus branches;
  non-finite vectors or scores produce no neighbor and legal scores are bounded
  before ranking and threshold selection
+ startupMaintenanceEligible() requires ServiceManagerState::Ready, not
  stopping and idle, and is evaluated before
  ResourceGovernor::admitDeferrable(), so an elapsed deferral deadline can no
  longer manufacture missing readiness
+ StartupPhaseTracker (two atomics, no mutex) plus a warn-level init-waiter
  line reporting state, phase and phase_elapsed_ms instead of a debug-level
  state number; a 53-minute startup previously logged nothing actionable
+ regression through the public backfill and corpus-rebuild APIs using a 64-D
  vector whose old float inverse norm rounds its self-cosine above 1
- removed the duplicated inverse-norm and cosine lambdas from EmbeddingService
- renamed __test_setModelProvider/__test_setKgStore to testing_* because they
  used reserved identifiers, across 11 files
- re-anchored 13 pre-existing raw-environment allowlist entries to their new
  line numbers; no new exception was added

Startup is still blocked by corpus-sized memory-sync scans; this change stops
the abort and names the blocking phase, it does not make startup fast.

Signed-off-by: trvon <git@trevon.dev>
2026-10-22 16:48:49 -06:00
Trevon
577adf1e0f
build: standardize the Debug build dir on build/debug (#248)
* build: standardize the Debug build dir on build/debug

`docs/BUILD.md` and `AGENTS.md` document `./setup.sh Debug` producing
`build/debug`, but `setup.sh`/`setup.ps1` created `builddir` and `.clangd`
pointed there too, so the documented layout and the actual tree disagreed.

Align the Debug mapping with the documented layout:
- setup.sh / setup.ps1: Debug -> `build/debug`
- CMakeLists.txt: Debug `_meson_build_dir` -> `build/debug`
- .clangd: CompilationDatabase -> `build/debug`
- plugin test fallbacks, S3 smoke default, quality-gate candidate list, and
  doc examples -> `build/debug`

Legacy `builddir` trees stay readable: the quality gate and local-ci scripts
still probe `builddir` as a fallback, and `plugin_trust` already treats
`/build/` paths as build artifacts.

* build: migrate remaining Debug consumers off builddir

Follow-up to the Debug build-dir standardization: tracked consumers still
hardcoded `builddir` and would break or silently skip against a fresh
`build/debug` tree. Migrate them, keeping tolerant fallbacks where a legacy
tree may still exist.

- CI: `.github/workflows/tests.yml` runs `./setup.sh Debug` and then compiled
  `-C builddir`; switch BUILD_DIR/COMPILE_DB_DIR/compile paths to `build/debug`.
  Same for windows-test-diagnose.yml.
- IDE: `.vscode/tasks.json` build/test/bench tasks target `build/debug`.
- Runtime: `object_storage_plugin_loader` also probes `build/debug/plugins`
  (and the flat `build/debug`), retaining the `builddir` fallbacks.
- Tests/docs: ONNX tuning integration plugin dir, benchmarks, and plugin
  READMEs/examples use `build/debug`.

`./build/*` is already excluded from the quality-gate find and `.dockerignore`,
so no exclusion lists change.
2026-10-01 08:45:20 -06:00
trvon
66689cd7a3
refactor(daemon): remove the symbol_concurrent tuning knob and stage metrics
The retired symbol stage never activates, so tuning.post_ingest.
symbol_concurrent and YAMS_POST_SYMBOL_CONCURRENT had no effect, and
status reported symbol_* queue metrics and post_symbol_limit that were
always zero. Remove the config key and environment override, the status
keys and CLI rows, the metrics snapshot fields, and the event-bus symbol
counters. The internal stage slot stays so budget and limiter indices
are unchanged; benchmark JSON keeps its symbol fields at zero.
2026-09-22 14:41:05 -06:00
trvon
30618eb113
refactor(config): consolidate runtime tuning 2026-08-05 20:53:18 -04:00
trvon
9c9e19199e
refactor(test): consolidate daemon harness 2026-08-05 20:53:18 -04:00
trvon
4d7f15dd5c
refactor(bench): isolate benchmark controls 2026-08-05 20:53:18 -04:00
trvon
5066fc0893
test: isolate process environment 2026-08-04 21:31:39 -04:00
Trevon
0dd3985db4
refactor: simplify runtime paths and harden retrieval (#61)
* refactor(tuning): prune dormant runtime knobs

Remove unused tuning controls and adaptive search budget state. Extract testable ONNX slot reconciliation and search telemetry hydration seams, and correct the Apple Silicon sysctl value width.

* refactor(vector): retire dormant backends

Remove unused FAISS, model-infrastructure, VectorIndexManager, and TurboQuant paths while keeping Simeon PQ and float-vector storage.

Migrate unused V2.2 quantized schemas transactionally to V2.3. Databases containing non-default quantized formats or packed-code blobs fail closed with recovery guidance and no mutation.

* refactor(plugins): remove legacy SDK tooling

Remove the Ghidra and SDK submodules, their bespoke tests, and stale scan exclusions. Retain generic PluginProcess coverage against the repository mock fixture.

* build(vector): remove stale FAISS overrides

* refactor(ipc): remove dormant FSM snapshots

* refactor(ingest): unify bounded KG enqueue

* fix(plugins): harden allocation contracts

Keep ABI outputs null or fully initialized when allocations fail, and represent empty object payloads without relying on malloc(0).

* fix(storage): define empty object reads

Return a null buffer with zero length for successful empty-object reads instead of treating malloc(0) as an allocation contract.

* refactor(storage): trim through string views

* test: prune dormant harness helpers

Remove unused readiness and fixture helpers, and keep benchmark client-count doubling in a wide integer domain.

* chore(cpp): initialize analyzer-facing locals

* build(simeon): advance analyzer cleanup

* refactor(config): drop dead daemon client keys

* refactor(ipc): inject daemon client transport

Socket and in-process execution now share DaemonClient request dispatch, so CLI fallbacks do not need alternate service paths.

* refactor(retrieval): route CLI through daemon

Search and grep now use the same daemon handlers over socket or in-process transport. Live grep remains an explicit freshness inspection.

* refactor(graph): move CLI scope into daemon

Graph CLI requests now share daemon handlers across socket and in-process transports. CWD filtering occurs before pagination so scoped totals and pages stay correct.

* build: isolate test-only runtime controls

* refactor(metadata): decouple lock telemetry

* fix(services): return on request timeout

* fix(retrieval): enforce workspace scope

* fix(graph): scope nodes before pagination

* fix(grep): drop binary result lines

* fix(daemon): bound checkpoint wait

* fix(repair): use KG health checks

* fix(storage): always verify compressed CRC

* perf(vector): skip unchanged PQ snapshots

* build(zyp): exclude bench from default build

* build: protect Meson files from formatter

* refactor: remove unreachable production APIs

* refactor(daemon): remove dormant runtime state

* fix(vector): harden daemon provider calls

Timed-out coroutines must own their client and requests. Provider discovery must also fail instead of returning synthetic embeddings in production.

* refactor(storage): remove unusable adapters

* fix(cli): unify in-process add requests

* fix(tsan): preserve async worker progress

* test(cli): isolate daemon-dependent suites

* fix(repair): expose queue backoff progress
2026-07-31 14:16:45 -04:00
trvon
4df44e0c2f
fix(tests): use portable getpid shim 2026-05-20 15:54:13 -04:00
trvon
b6b6ba9384
feat: additional test clean up and expansion for benchmarks 2026-04-14 11:39:53 -04:00
trvon
a930776655
feat: cleaning up test suite 2026-04-14 10:25:54 -04:00
trvon
f4fffd73a0
feat: instrumentation improvements for benchmarks, shell completition improvements, packaging updates for completions and docs updates 2026-04-13 21:34:18 -04:00
trvon
15980aa8fe
fix: TSAN fixes and sample updates 2026-02-24 23:06:32 -05:00
trvon
94a49472f1
feat: fixing bug with multiple processes access gpu, read/write pools now added in metadata repository, and updated plugins for opencode and vscode 2026-02-14 18:16:57 -05:00
trvon
cc28b8a343
feat: instrumented embedding for deeper insight into hotspots. Fixed CoreML scaling that resulted in high ram usage due to VRAM calculation. Expanded testing for embedding and draining of work 2026-02-11 12:07:08 -05:00
trvon
ff77c6b304
fix: improving socket and chunking reliability and robustness. Added better regression tests and errror logging logic for state connections 2026-02-10 12:25:30 -05:00
trvon
1d255d59ce
feat: address bugs with scaling, batching in daemon processing 2026-02-09 17:07:00 -05:00
trvon
cc1212bb1f
fix: fixing pipeline ingestion of kg and embeddings. Still finding stalls and will continue to investigate 2026-02-09 10:39:48 -05:00
trvon
0c8ffea321
feat: improving ingestion speed with kg batch and updating metrics to use constexpr 2026-02-09 01:33:24 -05:00
trvon
3101780fe2
feat: adding additional benchmark and validating onnxruntime model usage for embedding generations. Should be much faster 2026-02-08 21:59:14 -05:00