Commit graph

1,189 commits

Author SHA1 Message Date
trvon
342a3f8fae
chore: bump simeon, add extraction readiness stats
Simeon: 8bf225e -> 6320ab6 (adapter + QPP router framework).
CorpusStats: add contentExtracted*, ftsIndexed*, titleCoverage,
languageCoverage, and KG topology richness fields.
2026-05-07 20:14:02 -04:00
trvon
68a49e70dd
fix(metadata): deduplicate batch entries before UPSERT
Duplicate (document_id, key) pairs within a single batch cause SQLite
ON CONFLICT DO UPDATE to fail with constraint violation. Dedup keeps
the last write per key.
2026-05-07 16:22:00 -04:00
trvon
e971b56aa9
fix(metadata): retry constraint failures in executeQueryOnPool
FK race on Linux causes transient constraint failures when metadata
write races ahead of document commit. Previously treated as terminal.
Now retried with backoff like lock errors.
2026-05-07 14:23:57 -04:00
trvon
41276c0375
ci(bench): add faiss disable, ccache, and failure diagnostics to weekly benchmarks 2026-05-07 11:23:16 -04:00
trvon
db778ce72d
ci: disable faiss in warm and test jobs nightly-20260507-db778ce7 2026-05-06 21:27:48 -04:00
trvon
b463a29acc
ci: pin free-disk-space to v1.3.1 2026-05-06 20:23:03 -04:00
trvon
e7c520f41d
ci: add experimental branch to tests trigger 2026-05-06 20:23:03 -04:00
trvon
597aed4fc9
fix(docker): add wget for Zig, disable faiss
openblas arm64 build fails in Docker containers. YAMS_DISABLE_FAISS=1
env var skips faiss (and its openblas transitive dep). Zig download
now uses wget -c for resume on flaky ziglang.org connections.
2026-05-06 15:10:03 -04:00
trvon
ec1e0a6ac2
ci: fix free-disk-space action ref, sha didnt exist 2026-05-06 10:01:37 -04:00
trvon
8f5b0d0eb3
fix(docker): add gfortran for openblas arm64 build 2026-05-06 09:08:44 -04:00
trvon
b4b22e428c
ci(warm): add ccache dep, tolerate populate timeout 2026-05-05 21:09:19 -04:00
trvon
1f9bc767a0
feat(search): reduce graph noise, add edge-creation diagnostics
- PostIngestQueue: halve co-occurrence edge budgets (entities 12→8, edges 36→16),
  skip edges where neither entity is biomedical high-value, raise the
  confidence floor for biomedical pairs (0.10→0.20) and non-biomedical
  pairs (0.05→0.08), bump base multiplier (0.50→0.65). This cuts total
  edge volume by over 50% while retaining signal quality.

- search_tuner.h: raise graphRerankMinSignal to 0.08 (from default 0.01)
  to match the elevated edge-confidence floor.

- New atomic graph-construction counters (gs_processed_, gs_totalEntities_,
  gs_highValueEntities_, gs_totalEdges_, gs_totalPrimaryTopicEdges_) log
  entity → edge pipeline every 100 documents via [PIQ-graph].

- Periodic [graph_rerank] log (every 50 queries) reports reranker
  window size, matched candidates, whether boost was applied, concept
  count, and elapsed time — enabling end-to-end traceability from
  ingestion to retrieval.
2026-05-05 20:56:12 -04:00
trvon
bec8164d4c
fix(vector): guard faiss backend init behind YAMS_HAS_FAISS
Types only declared when YAMS_HAS_FAISS defined. Falls through
to SqliteVec PQ when faiss unavailable.
2026-05-05 20:32:44 -04:00
trvon
6704e0bb76
fix(search): wire GLiNER concept extractor, fix PQ dirty race, add KG NER awareness 2026-05-05 20:06:06 -04:00
trvon
18b10dc444
fix: addressing build regressions where fiass needed openmp installed 2026-05-05 19:50:16 -04:00
trvon
bc100efcdc
feat: updating zyp pdf extractor to use v0.16.0 compat in plugin build 2026-05-05 19:37:50 -04:00
trvon
8433d4f6fc
fix: disable faiss on Apple Clang macOS, remove hardcoded /opt/homebrew paths
faiss recipe validate() rejects apple-clang (no OpenMP). Disable faiss
in configure() when compiler=apple-clang on macOS. The list order is
swapped to [False,True] so False is the default; configure() enables
faiss on all non-apple-clang platforms.

Also removes hardcoded /opt/homebrew/{include,opt/libomp/include} paths
from src/vector/meson.build that broke Linux ARM / Docker builds when
the cmake faiss dep fallback was reached.
2026-05-05 19:11:47 -04:00
trvon
f14c7d64ab
fix: disable faiss on Apple Clang macOS, restore profile cache keys
faiss requires OpenMP which Apple Clang does not ship. Skip faiss
on macOS/apple-clang to prevent conan install failures during warm.

Also restore trailing whitespace in Linux gcc profiles to match
original cache key hash (hashFiles change invalidated all platform
caches, exposing the macOS issue).
2026-05-05 18:50:56 -04:00
trvon
c3127d55c9
fix(ci): restore cache-hit guard for macOS/Windows warm step
The conan install step fails on macOS when run unconditionally (previously
hidden by the cache-hit skip). Restore the guard for non-Linux platforms
while keeping Linux always-running to populate ccache.
2026-05-05 18:41:13 -04:00
trvon
07ad8667ea
fix(ci): revert Conan profile ccache wiring (breaks compiler_executables)
tools.build:compiler_executables requires bare executable names, not
'ccache clang' with a space. setup.sh already wraps CC/CXX with ccache
for Meson builds. The warm step's meson compile (Fix 3) populates
ccache for YAMS source. Conan dep ccache will be addressed separately.
2026-05-05 18:38:20 -04:00
trvon
76c7e9d60f
security,ci: security issues address from codeql and improving cache usage logic in ci 2026-05-05 18:27:43 -04:00
trvon
cbb86506d0
feat: add RM3 pseudo-relevance feedback + adopt research-tuned config 2026-05-05 16:13:18 -04:00
trvon
73fbe4c842
feat: add BM25 variant RRF fusion to lexical backend 2026-05-05 16:13:18 -04:00
trvon
102d5a1e50
feat: integrate simeon concept mining into lexical backend 2026-05-05 16:13:18 -04:00
trvon
2d84e44013
security: addressing codeql security finding and expanding opengrep rules 2026-05-05 15:38:59 -04:00
trvon
dc6aaa13ce
feat: add FAISS HNSW vector backend with ConfigResolver policy 2026-05-05 15:38:22 -04:00
trvon
9cf74ec420
ci: test fixes 2026-05-05 13:38:53 -04:00
trvon
8b7bfe575d
feat: improved search fusion logic and tuning parameters. CI improvements, reverting to push with wrangler in CI. Experimenting with vector rebuild to address additional lost overhead in search 2026-05-05 08:58:20 -04:00
trvon
ea6cd51da3
fix: removed reduant corpus adapter nightly-20260505-ea6cd51d 2026-05-04 16:33:57 -04:00
trvon
7ae52e8b8b
fix: test fixes and search fusion update to correctly expose simeontext fusion so that it does not drop ft5s candidates. Still explore fusion optimizations v0.14.2 2026-05-04 15:54:57 -04:00
trvon
77dad749af
docs: revert CHANGELOG.md to condensed version, update release to fix overwrite attempt in release asset stage 2026-05-04 15:32:02 -04:00
trvon
052258861f
merge 2026-05-04 15:28:29 -04:00
trvon
14e8a17c59
docs: updating changes logs archive 2026-05-04 15:27:58 -04:00
github-actions[bot]
591f664b82
chore(main): release 0.14.1 (#21)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-05-04 14:04:13 -04:00
trvon
f80013063d
build: windows packaging fixes 2026-05-04 14:02:43 -04:00
trvon
2d5c6625eb
fix: download logic improvements and ci test fixes. Stagging download fixes seperate from search features 2026-05-04 13:29:17 -04:00
github-actions[bot]
ef6c8ade12
chore(main): release 0.14.0 (#19) v0.14.0
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-05-04 12:29:28 -04:00
trvon
e0b288f4e0
ci: improve release please configuration and CHANGELOG 2026-05-04 11:11:45 -04:00
trvon
7d7f981fb4
ci: patching orphaned boost setting 2026-05-04 08:39:27 -04:00
trvon
a7ea3da82a
updating boost variable exposure 2026-05-03 17:03:34 -04:00
trvon
34e8ede523
build: updating meson.build for windows boost exposure improvements 2026-05-03 13:38:27 -04:00
trvon
75e7dcfa47
ci: windows ci improvements 2026-05-03 05:39:54 -04:00
trvon
ba9030bce8
fix: repair fix in db optimize 2026-05-03 01:31:40 -04:00
trvon
f3ffb92264
security: updates to opengrep rules, repair subsystem scanning logic and windows build logic 2026-05-03 01:06:25 -04:00
trvon
5adb8f153f
security: dev scripts and hooks update, perf update, security fixes for quality issues 2026-05-02 21:40:47 -04:00
Trevon
557f8c1da3
Merge branch 'main' of https://github.com/trvon/yams 2026-05-02 21:34:38 -04:00
Trevon
79415ff379
build: windows arm build update 2026-05-02 21:34:13 -04:00
Trevon
c2bc563157
Merge branch 'main' of https://github.com/trvon/yams 2026-05-02 21:34:12 -04:00
Trevon
d27278b315
Merge branch 'main' of https://github.com/trvon/yams 2026-05-02 21:34:12 -04:00
Trevon
447c810a68
Merge branch 'main' of https://github.com/trvon/yams 2026-05-02 21:34:12 -04:00