Commit graph

13 commits

Author SHA1 Message Date
Trevon
4e97b31fef
chore(warnings): clean up compiler warnings on GCC, Clang and Apple Clang (#279)
Some checks failed
Fuzz Smoke / libFuzzer smoke (push) Has been cancelled
* chore(warnings): default-initialize aggregate members left out of brace-init

GCC -Wmissing-field-initializers fired about 1300 times where tests and
src build structs with a short positional brace list. Give the omitted
members an empty default member initializer. Value-initialization from
{} is what aggregate init already did for them, so behavior is
unchanged.

Signed-off-by: trvon <me@trevon.dev>

* chore(warnings): remove or annotate unused parameters, functions and variables

- init_command, s3_signer test, request_handler_lifetime test: comment out
  the unused parameter name.
- video_content_handler: path is only read under YAMS_HAVE_FFPROBE.
- vector_memory_scaling_bench: main() takes no arguments.
- daemon_status_render: delete unused neutralText helper.
- DaemonMetrics: mallocStackLoggingNoCompactEnabled is only called in the
  __APPLE__ block, so define it there.
- SocketServer: einval_streak only matters in the __APPLE__ EINVAL path.
- onnx_plugin_harness test, vector_backend_engine_compare bench,
  work_coordinator test: mark platform-conditional or busy-loop
  variables [[maybe_unused]].

Signed-off-by: trvon <git@trevon.dev>

* chore(warnings): fix sign-compare, range-loop, nodiscard and dangling-ref in tests

- search_admission_controller: count started workers in std::uint32_t so
  the comparison with the uint32 cap is same-signed.
- p2p_manager: iterate the invalid-host list by const reference.
- integrity_verifier: discard the nodiscard results the tests do not
  inspect with static_cast<void>.
- post_ingest_queue: findNode/findEdge take std::string_view so no
  temporary std::string is created for the call (GCC
  -Wdangling-reference false positive on the returned reference).
- topology_ablation_bench: the doc comment ended a line with a backslash,
  which continues the comment (-Wcomment); show the env export on its own
  line.

Signed-off-by: trvon <git@trevon.dev>

* chore(warnings): drop redundant std::move of a const captured json

The coroutine lambda is not mutable, so the captured optional<json> is
const and std::move(*progressToken) already selected the copy assignment.
Drop the std::move so the code states what actually happens.

Signed-off-by: trvon <git@trevon.dev>

* chore(warnings): mark __int128 use in db_integrity_stamp as an extension

-Wpedantic flags __int128. Use a __extension__ typedef, the GCC/Clang
idiom for an intentional extension, instead of the bare keyword.

Signed-off-by: trvon <git@trevon.dev>

* chore(warnings): remove always-true unsigned >= 0 comparisons

GCC -Wtype-limits flagged 51 test assertions of the form
CHECK(unsignedValue >= 0) that can never fail, plus a dead
`totalCount.value() < 0` clause in the graph listByType handler (the
count is std::size_t). In the tests keep the evaluation as a
static_cast<void> so the accessors are still exercised; in the handler
drop the dead clause and keep the range check.

Signed-off-by: trvon <me@trevon.dev>

---------

Signed-off-by: trvon <me@trevon.dev>
Signed-off-by: trvon <git@trevon.dev>
2026-10-08 14:30:28 -06:00
Trevon
0effb84a5c
perf(vector): harden portable backends (#66)
* perf(vector): update portable backends

Keep generic release binaries on baseline ISA until runtime CPU dispatch can guard AVX and FMA paths.

Signed-off-by: trvon <git@trevon.dev>

* test(vector): isolate backend measurements

Signed-off-by: trvon <git@trevon.dev>

---------

Signed-off-by: trvon <git@trevon.dev>
2026-08-23 23:12:39 -04:00
Trevon
3f37e8f564
Sync experimental into main: Search retrieval improvements (#60)
* fix(graph): handle qualified C++ symbols

Signed-off-by: trvon <me@trevon.dev>

* refactor(vector)!: remove dead transition APIs

Split Simeon vector and retrieval linkage so consumers inherit only the runtime surface they use.

BREAKING CHANGE: remove unused async/model/config helpers and the undefined vector-backend factory from public headers.

Signed-off-by: trvon <me@trevon.dev>

* fix(vector): harden PQ snapshot lifecycle

Validate and atomically install persisted generations, revalidate after races, and fall back to exact search when a snapshot is stale. Add the exact-scan reference lane and distinguish observed work counters from unavailable backend metrics.

Signed-off-by: trvon <me@trevon.dev>

* feat(topology): expose route coordinates

Persist chart distortion and local-dimension evidence, retain pre-scalarization route axes, and emit detailed trace rows without treating unavailable vector work as zero.

Signed-off-by: trvon <me@trevon.dev>

* feat(xplan): add vector decision gates

Track binary and corpus identity, preserve missing-work semantics, and add repeated PQ routing, crossover, and rerank plans with quality and cost summaries.

Signed-off-by: trvon <me@trevon.dev>

* feat(topology): formalize coordinate routing

Separate chart geometry from route-risk admission and prove candidate-stage recall and strict-focus boundaries. Record the measured atlas limitations and literature basis for the next construction experiment.

Signed-off-by: trvon <me@trevon.dev>

* fix(search): serialize SymSpell readers

SQLite-backed lookups reuse mutable prepared statements, so shared-reader locking is unsafe. Advance the owned SymSpell revision for batched delete insertion and cover concurrent root lookups under TSAN.

Signed-off-by: trvon <me@trevon.dev>

* fix(vector): make mutations atomic

Preserve stored vectors on replacement and commit failures, reject cross-thread writes during explicit transactions, and restore exact reranking for quantized-primary PQ rows.

* fix(search)!: surface SymSpell write errors

Propagate persistent dictionary failures through metadata ingestion, roll back failed batches, and make clear remove stored terms.

BREAKING CHANGE: SymSpellSearch mutations now return Result values.

* feat(topology): formalize candidate expansion

* feat(vector): add coordinate-aware embeddings

* feat(topology): preserve protected relation cover

* feat(search): add graph candidate rescue

* feat(daemon): expose retrieval hydration telemetry

* feat(xplan): add rescue and hydration gates

* refactor(content): dedupe media handlers

Reuse shared graph presenter helpers and centralize repeated audio/video/archive buffer processing, validation, lifecycle, and stats helper logic.

* feat(lifecycle): prune stale corpus data

Keep cleanup preview-first while retention stays opt-in and long prune work no longer holds IPC responses open.

* feat(search): add topology candidate rescue

Preserve high-value routed candidates across vector backends while keeping exact rerank diagnostics explicit.

* feat(search): expose evidence paths

* docs: define mobile host contract

* formal: model bounded candidate rescue

* fix(prune): inspect extensionless signatures

* fix(prune): tolerate stale extensions

---------

Signed-off-by: trvon <me@trevon.dev>
2026-07-24 19:00:22 -04:00
Trevon
7df64b1161
perf(search): bound topology routing and harden runtime paths (#59)
* fix: address review build issues

* perf(search): bound topology routing

Cache route membership, shortlist centroids with static ANN, and route SPQ through explicit allowed-document budgets.

* test(xplan): measure routed vector work

Pin benchmark binary identity and export route, cache, and phase costs for repeated equal-budget comparisons.

* test(search): use portable jthread wrapper

* fix(daemon): serialize detached cancellation

* fix(cli): honor pinned daemon sockets

* fix(xplan): support Windows contract imports

* fix(list): apply metadata filters before paging

* docs(prompts): require memory hydration

Treat search results as candidates and hydrate selected artifacts before use. Quote argument hints for strict YAML loaders.

* test(cli): keep add on pinned daemon

* test(metadata): isolate performance timing

* fix(graph): tolerate stale snippet ranges

Source files may change before persisted symbol metadata is refreshed. Omit stale snippets instead of aborting graph exploration.

* test(daemon): await request readiness

Reuse the retrying list probe before asserting post-shutdown error quality under sanitizer load.

* ci(release): use hosted cmake

Resolve Conan CMake tool requirements from PATH so cold ARM64 caches do not depend on cmake.org. Default BUILD_DIR for failure cleanup.
2026-07-19 14:53:10 -04:00
Trevon
771deebc84
fix: harden CI and metadata stability (#48)
* feat: dead code removal, profling foudnation improvements, addressing code smells in reference db implementation

* fix: test fixes and optimmizations for benchmarks

* fix: harden audit findings

   Reject invalid tokenizer ids, fix move-order hazards, bound manifest
   and audio IO, and guard numeric/process edge cases found by Fold.

* fix: route SymSpell writes through coordinator

* test: fix batch content initializers

* fix(ci): restore cross-platform builds

* fix: address Copilot review findings
2026-06-11 16:42:54 -04:00
Trevon
9cb6421593
feat: daemon stability fixes, vector search optimizations, and docs refresh (#46)
* fix(daemon): resolve sign-conversion warnings

* fix(daemon): add IO thread startup barrier

* refactor: consolidate toLowerCopy clones

* fix(daemon): add atexit handler for static host map cleanup

* fix(build): add missing include, fix daemon_helpers syntax, use atomic barrier

* feat(docs): docs cleanup, and metadata refactor updates

* fix(daemon): resolve Windows-only PostIngestQueue shutdown SIGSEGV (#45)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

* feat: submodule bump, multiclient optimizations and benchmark optimizations

* feat(perf): reduced contention with update to placement of flushPathCacheBuffer()

* fix(metadata): invalidate path cache on document mutation

* fix(metadata): hydrate cached counters before first write

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-10 10:30:38 -04:00
trvon
d67d5d63ac
Remediate Fold SAST findings
Centralize duplicated helpers and harden numeric, timing, IPC, SQLite retry, config, plugin, metrics, and async content-store paths identified during Fold SAST triage.\n\nAdd targeted tests for shared utilities and preserve per-task YAMS checkpoints.
2026-05-27 22:41:44 -04:00
trvon
0b61c0818a
docs: clean up docs and ci fixes for tests 2026-04-25 00:08:18 -04:00
trvon
d8cb0764d8
feat: address test drift, removing hnsw remants for simeon backend implementation, continued refactor of vector backend 2026-04-22 15:03:36 -04:00
trvon
4617bf4d8f
feat: further removing hnsw references and centralizing on simeon pq implementation 2026-04-22 01:10:57 -04:00
trvon
d669c009d0
feat: removing experiments and unifying compression in embeddings storage. 2026-04-21 18:13:37 -04:00
trvon
bb0d33a436
feat: vector optimizations. Most performance gains in search felt on macos 2026-04-12 18:16:10 -04:00
trvon
ef3d925797
feat: continued corectness improvements, ANN / vectordb improvements 2026-04-12 12:53:29 -04:00