Commit graph

33 commits

Author SHA1 Message Date
Trevon
211db15606
fix(daemon): bound semantic neighbor scores and report startup phase (#334)
Semantic-neighbor maintenance aborted the daemon during startup:
std::clamp<float> asserted '!(__hi < __lo)' because the adaptive cosine
threshold could exceed 1.0 for a legal pair of identical vectors. The same
work was also admitted while the daemon was still initializing, so the
300-second starvation guard forced a pass before readiness.

+ semanticInverseNorm/semanticCosineSimilarity/semanticExplicitThreshold in the
  existing embed policy header, shared by the streamed and corpus branches;
  non-finite vectors or scores produce no neighbor and legal scores are bounded
  before ranking and threshold selection
+ startupMaintenanceEligible() requires ServiceManagerState::Ready, not
  stopping and idle, and is evaluated before
  ResourceGovernor::admitDeferrable(), so an elapsed deferral deadline can no
  longer manufacture missing readiness
+ StartupPhaseTracker (two atomics, no mutex) plus a warn-level init-waiter
  line reporting state, phase and phase_elapsed_ms instead of a debug-level
  state number; a 53-minute startup previously logged nothing actionable
+ regression through the public backfill and corpus-rebuild APIs using a 64-D
  vector whose old float inverse norm rounds its self-cosine above 1
- removed the duplicated inverse-norm and cosine lambdas from EmbeddingService
- renamed __test_setModelProvider/__test_setKgStore to testing_* because they
  used reserved identifiers, across 11 files
- re-anchored 13 pre-existing raw-environment allowlist entries to their new
  line numbers; no new exception was added

Startup is still blocked by corpus-sized memory-sync scans; this change stops
the abort and names the blocking phase, it does not make startup fast.

Signed-off-by: trvon <git@trevon.dev>
2026-10-22 16:48:49 -06:00
Trevon
4e97b31fef
chore(warnings): clean up compiler warnings on GCC, Clang and Apple Clang (#279)
Some checks failed
Fuzz Smoke / libFuzzer smoke (push) Has been cancelled
* chore(warnings): default-initialize aggregate members left out of brace-init

GCC -Wmissing-field-initializers fired about 1300 times where tests and
src build structs with a short positional brace list. Give the omitted
members an empty default member initializer. Value-initialization from
{} is what aggregate init already did for them, so behavior is
unchanged.

Signed-off-by: trvon <me@trevon.dev>

* chore(warnings): remove or annotate unused parameters, functions and variables

- init_command, s3_signer test, request_handler_lifetime test: comment out
  the unused parameter name.
- video_content_handler: path is only read under YAMS_HAVE_FFPROBE.
- vector_memory_scaling_bench: main() takes no arguments.
- daemon_status_render: delete unused neutralText helper.
- DaemonMetrics: mallocStackLoggingNoCompactEnabled is only called in the
  __APPLE__ block, so define it there.
- SocketServer: einval_streak only matters in the __APPLE__ EINVAL path.
- onnx_plugin_harness test, vector_backend_engine_compare bench,
  work_coordinator test: mark platform-conditional or busy-loop
  variables [[maybe_unused]].

Signed-off-by: trvon <git@trevon.dev>

* chore(warnings): fix sign-compare, range-loop, nodiscard and dangling-ref in tests

- search_admission_controller: count started workers in std::uint32_t so
  the comparison with the uint32 cap is same-signed.
- p2p_manager: iterate the invalid-host list by const reference.
- integrity_verifier: discard the nodiscard results the tests do not
  inspect with static_cast<void>.
- post_ingest_queue: findNode/findEdge take std::string_view so no
  temporary std::string is created for the call (GCC
  -Wdangling-reference false positive on the returned reference).
- topology_ablation_bench: the doc comment ended a line with a backslash,
  which continues the comment (-Wcomment); show the env export on its own
  line.

Signed-off-by: trvon <git@trevon.dev>

* chore(warnings): drop redundant std::move of a const captured json

The coroutine lambda is not mutable, so the captured optional<json> is
const and std::move(*progressToken) already selected the copy assignment.
Drop the std::move so the code states what actually happens.

Signed-off-by: trvon <git@trevon.dev>

* chore(warnings): mark __int128 use in db_integrity_stamp as an extension

-Wpedantic flags __int128. Use a __extension__ typedef, the GCC/Clang
idiom for an intentional extension, instead of the bare keyword.

Signed-off-by: trvon <git@trevon.dev>

* chore(warnings): remove always-true unsigned >= 0 comparisons

GCC -Wtype-limits flagged 51 test assertions of the form
CHECK(unsignedValue >= 0) that can never fail, plus a dead
`totalCount.value() < 0` clause in the graph listByType handler (the
count is std::size_t). In the tests keep the evaluation as a
static_cast<void> so the accessors are still exercised; in the handler
drop the dead clause and keep the range check.

Signed-off-by: trvon <me@trevon.dev>

---------

Signed-off-by: trvon <me@trevon.dev>
Signed-off-by: trvon <git@trevon.dev>
2026-10-08 14:30:28 -06:00
Trevon
291f8728f5
fix(graph): scope health checks to graph data (#63)
Signed-off-by: trvon <git@trevon.dev>
2026-08-23 23:12:38 -04:00
trvon
7485b482f8
fix(build): restore clean test and benchmark links 2026-08-06 09:59:07 -04:00
trvon
5066fc0893
test: isolate process environment 2026-08-04 21:31:39 -04:00
Trevon
3f37e8f564
Sync experimental into main: Search retrieval improvements (#60)
* fix(graph): handle qualified C++ symbols

Signed-off-by: trvon <me@trevon.dev>

* refactor(vector)!: remove dead transition APIs

Split Simeon vector and retrieval linkage so consumers inherit only the runtime surface they use.

BREAKING CHANGE: remove unused async/model/config helpers and the undefined vector-backend factory from public headers.

Signed-off-by: trvon <me@trevon.dev>

* fix(vector): harden PQ snapshot lifecycle

Validate and atomically install persisted generations, revalidate after races, and fall back to exact search when a snapshot is stale. Add the exact-scan reference lane and distinguish observed work counters from unavailable backend metrics.

Signed-off-by: trvon <me@trevon.dev>

* feat(topology): expose route coordinates

Persist chart distortion and local-dimension evidence, retain pre-scalarization route axes, and emit detailed trace rows without treating unavailable vector work as zero.

Signed-off-by: trvon <me@trevon.dev>

* feat(xplan): add vector decision gates

Track binary and corpus identity, preserve missing-work semantics, and add repeated PQ routing, crossover, and rerank plans with quality and cost summaries.

Signed-off-by: trvon <me@trevon.dev>

* feat(topology): formalize coordinate routing

Separate chart geometry from route-risk admission and prove candidate-stage recall and strict-focus boundaries. Record the measured atlas limitations and literature basis for the next construction experiment.

Signed-off-by: trvon <me@trevon.dev>

* fix(search): serialize SymSpell readers

SQLite-backed lookups reuse mutable prepared statements, so shared-reader locking is unsafe. Advance the owned SymSpell revision for batched delete insertion and cover concurrent root lookups under TSAN.

Signed-off-by: trvon <me@trevon.dev>

* fix(vector): make mutations atomic

Preserve stored vectors on replacement and commit failures, reject cross-thread writes during explicit transactions, and restore exact reranking for quantized-primary PQ rows.

* fix(search)!: surface SymSpell write errors

Propagate persistent dictionary failures through metadata ingestion, roll back failed batches, and make clear remove stored terms.

BREAKING CHANGE: SymSpellSearch mutations now return Result values.

* feat(topology): formalize candidate expansion

* feat(vector): add coordinate-aware embeddings

* feat(topology): preserve protected relation cover

* feat(search): add graph candidate rescue

* feat(daemon): expose retrieval hydration telemetry

* feat(xplan): add rescue and hydration gates

* refactor(content): dedupe media handlers

Reuse shared graph presenter helpers and centralize repeated audio/video/archive buffer processing, validation, lifecycle, and stats helper logic.

* feat(lifecycle): prune stale corpus data

Keep cleanup preview-first while retention stays opt-in and long prune work no longer holds IPC responses open.

* feat(search): add topology candidate rescue

Preserve high-value routed candidates across vector backends while keeping exact rerank diagnostics explicit.

* feat(search): expose evidence paths

* docs: define mobile host contract

* formal: model bounded candidate rescue

* fix(prune): inspect extensionless signatures

* fix(prune): tolerate stale extensions

---------

Signed-off-by: trvon <me@trevon.dev>
2026-07-24 19:00:22 -04:00
Trevon
b603738f44
Stabilize sanitizer lanes and daemon concurrency (#40)
* fix(repair): keep RPC progress alive

* fix(daemon): propagate DB open failure to FSM and add shutdown cancellation

* fix(db): classify FTS5 inverted-index errors as repairable, not transient locks

* fix(daemon): clean up stale corrupt files after salvage and suppress optional readiness keys

* test(db_recovery): update FTS5 classification test for new transient-lock exclusion

* test(gc): add garbage collector unit tests for collection, dry-run, scheduling

* test(repair): expand repair manager tests with hash-verified recovery and source ordering

* fix(gc): dry-run mode no longer increments blocksDeleted/bytesReclaimed

* fix(mock,mcp): add atomic bool for lock consistency, move field assignments

* style: fix unnecessary-copy, dead-store, and const-refable findings from Infer

* fix(daemon): increase default shutdown timeout from 15s to 120s

* fix(daemon): demote non-critical warnings to info (FTS5, VACUUM, WorkCoordinator timeout)

* fix(db): re-enable WAL auto-checkpoint to prevent unbounded WAL growth

* test(pool): add WAL auto-checkpoint and concurrency coverage for connection pool

* test(pool): add exhaustion, stats, lifecycle, and read-only connection pool tests

* test(pool): add tagging, WAL journal, reuse, and busy timeout tests

* test(metrics): relax session snippet assertion to tolerate stub behavior

* fix(test): stabilize sanitizer triage failures

* fix(build): wire version and zyp deps

* fix(daemon): close TSAN race clusters

* fix(test): guard optional zyp dep

* fix(test): address PR review findings
2026-06-01 12:14:43 -04:00
Trevon
a62a901e2e
Stabilize daemon recovery and vector reconciliation (#36)
* fix(daemon): avoid socket cancel on stop

Signal active sockets with shutdown only so pending I/O wakes without entering Boost.Asio's reactor cancel path during teardown.

* fix(metadata): open read pools read-only

* ci: fix coverage report generation

* fix(metadata): tag unscoped pool holders

* test(daemon): avoid ipc teardown takeover

* ci: skip plugin coverage objects

* fix(daemon): repair completed FTS5 ghosts

* build(vector): guard unusable FAISS headers

* ci: use clang gcov for coverage

* test(daemon): stabilize arm CI flakes

* Harden search and compression numeric config

* fix(daemon): reconcile orphan vector embeddings

* fix: address vector cleanup review comments
2026-05-23 21:21:55 -04:00
trvon
8b7bfe575d
feat: improved search fusion logic and tuning parameters. CI improvements, reverting to push with wrangler in CI. Experimenting with vector rebuild to address additional lost overhead in search 2026-05-05 08:58:20 -04:00
trvon
df0c78cf29
feat: test fixes, metric expansion, build improvements for gliner 2026-04-23 08:48:33 -04:00
trvon
d25cc39e12
fix: vector and embedding count fixes related to repair functionality. Improvements for search and tuning scope of candidates 2026-04-05 08:30:28 -04:00
trvon
86ec015eea
feat: massive commit. Download logic updates with expanding controls (stop, start, resume etc.) from daemon control. This is used in my archive openclaw agent. This commit also includes work to harden onnxruntime abi access. this should allow loading newer abi's when builds are pinned to 1.23.0. 2026-03-21 17:12:47 -04:00
trvon
943393af6d
fix: fixing search regression when fuzzy search triggered and testing improvements 2026-03-21 09:00:54 -04:00
trvon
554bf47366
fix: test coverage updates and address protobuf bug in search 2026-03-20 22:15:36 -04:00
trvon
727d5b38a1
fix: additional cli and search optimizations to reduce latency, test coverage improvements 2026-03-20 13:14:31 -04:00
trvon
6d1ec69e77
fix: regressions fixes in request handlers and correctness improvements in repair service 2026-03-19 09:47:30 -04:00
trvon
081fb740b4
fix: improving test coverage and addressign ASAN issues 2026-03-19 06:32:13 -04:00
trvon
4a26ce6ddd
fix: improvements based on TSAN and strict quality gates 2026-03-18 20:55:56 -04:00
trvon
4cf1704b36
fix: addressing TSAN and empty catch blocks 2026-03-18 16:44:27 -04:00
trvon
985d03abdd
fix: test improvements and correctness improvements using check-quality script 2026-03-18 15:55:43 -04:00
trvon
4fe9ce6120
fix: bug fixes and test improvements for small correctness issues 2026-03-18 15:16:42 -04:00
trvon
0cdb3ab6c0
fix: testing improvements and bug fixes 2026-03-18 12:30:32 -04:00
trvon
33821d93f1
feat: test hardening, added benchmark for multiclient and improve stability to support up to 16 clients before performance degrades, improved parallization and request handling in daemon ingestion pipeline 2026-02-16 15:46:16 -05:00
trvon
36c587e776
bug fixes and test improvements 2026-01-24 11:49:48 -05:00
trvon
80ea3ca869
testing improvements 2026-01-14 16:27:42 -05:00
trvon
c73b899b8f
bug fixes and daemon communication improvements 2026-01-02 22:37:19 -05:00
trvon
94d89fe0d5
bug fixes, test updates and ci improvements for TSAN test runs 2026-01-01 13:52:52 -05:00
trvon
c16939f482
linter 2025-11-28 20:30:41 -05:00
trevon
1de9e31a58
cleaning up tests and exposing windows packaging and builds in ci 2025-11-27 10:54:09 -05:00
trevon
3b58ac1743
test stabilizations for windows. search metadata/tags improvements 2025-11-26 21:45:50 -05:00
trvon
2913adcd20
pre-release v0.7.8 2025-11-14 11:58:19 -05:00
trvon
b06e9335b9
staging v0.7.8. running benchmarks on updated search engine and improving code for testing/benchmarking 2025-11-13 10:27:10 -05:00
trvon
bbf23bb190
test migrations to catch2, service manager + socket server refactor for better startup performance. search bug fixes 2025-10-29 23:08:09 -04:00