Commit graph

600 commits

Author SHA1 Message Date
trvon
8e4b0df477
feat: additional optimizations for faster responses and ASAN fixes 2026-02-20 16:31:47 -05:00
trvon
2c8664b327
feat: optimizing query logic to reduce hangs with large corpuses 2026-02-20 11:55:17 -05:00
trvon
677756b25a
fix: lifecycle regression fix and repair improvements 2026-02-19 22:21:03 -05:00
trvon
226cea7ca2
feat: additional improvements to reduce read and request contention 2026-02-19 20:06:57 -05:00
trvon
37b95bf6eb
feat: improving sql access under contention and expanding tuning logic with graph 2026-02-19 16:43:06 -05:00
trvon
fc284f3b37
fix: resolving TSAN issues and expanding graph building to use gliner data 2026-02-19 14:52:01 -05:00
trvon
8401db189f
feat: expanding graph usage in search 2026-02-19 12:41:23 -05:00
trvon
218499386b
fix: pipeline scaling fixes, post ingestion benchmark updates 2026-02-19 12:16:47 -05:00
trvon
52303afbe1
fix: refining ingestion logic 2026-02-19 10:53:49 -05:00
trvon
55b2dc517f
fix: additional merge fixes and search improvements 2026-02-19 10:24:23 -05:00
trvon
207532f46a
merge: integrate main into experimental (preserve PBI-081 simplification)
Merge main's 93 commits into experimental, preserving architectural updates
decisions:

- Keep ResourceGovernor as centralized pool registry (experimental's PBI-081 Phase 2)
- Translate all PoolManager API calls to ResourceGovernor equivalents
- Remove PoolManager.{h,cpp} from build (absorbed into ResourceGovernor)
- Preserve main's enhancements: RepairService, IOCoordinator, GradientLimiter,
  embed_preparer, GPU info, plugin trust, 60+ new test files
- Take main's version for 28 of 31 conflicts (more comprehensive rewrites)
- Keep experimental's ResourceGovernor.h (expanded pool API)
- Manual merge for .vscode/settings.json (use workspace-relative paths)
2026-02-19 10:01:01 -05:00
trvon
8b79d8afaf
fix: cleaning up postingest pipeline 2026-02-18 22:12:48 -05:00
trvon
dea5315ae3
feat: streamlining tuning and pool logic in daemon 2026-02-18 16:56:25 -05:00
trvon
a9bad18fa4
feat: adding additional tests, making the aggressive profile scaling better 2026-02-17 19:15:26 -05:00
trvon
014e12c975
fix: testing improvements and bug fixes in daemon logic for lifecycle management 2026-02-17 15:55:29 -05:00
trvon
6a90f8c934
feat: test improvements, ASAN fixes, and onnx model performance improvements 2026-02-17 08:59:53 -05:00
trvon
e39273a207
fix: fixing repair regression 2026-02-16 19:47:28 -05:00
trvon
be77f1f4d3
feat: additional test improvements, improved logic for ingestion queue to increase throughput and client stability 2026-02-16 18:13:54 -05:00
trvon
33821d93f1
feat: test hardening, added benchmark for multiclient and improve stability to support up to 16 clients before performance degrades, improved parallization and request handling in daemon ingestion pipeline 2026-02-16 15:46:16 -05:00
trvon
8516a1b779
feat: tuning improvements with expanded db metrics, package hardening and updated tests 2026-02-15 17:52:43 -05:00
trvon
c4ae922b7a
feat: mobile bindings expansion, update mcp for graph operations and download tool hardening 2026-02-15 17:50:14 -05:00
trvon
247f3374a2
fix: fixing mcp download bug, updating socket server configs for more client connections 2026-02-14 22:08:14 -05:00
trvon
94a49472f1
feat: fixing bug with multiple processes access gpu, read/write pools now added in metadata repository, and updated plugins for opencode and vscode 2026-02-14 18:16:57 -05:00
trvon
8f38ee76be
feat: updating metrics for search and retrieval for future learning system. Additional tuning for search performance 2026-02-13 21:09:26 -05:00
trvon
8256917e5e
feat: Hardening moodel logic for embedings. Adding trace-linked feedback event persistence checkpoint 2026-02-12 23:42:21 -05:00
trvon
11ba59f4bb
feat: updating embedding default checking to paragraph 2026-02-12 11:55:59 -05:00
trvon
a44c2fb4af
feat: implementing embedding chunking strategy and two staged ingestion that processes then distributes document data 2026-02-12 00:46:04 -05:00
trvon
44a94eabe1
feat: fuzzing and config expansion. Moving to optimizing ingestion. Adding tunable embedding (chunk size etc.) 2026-02-11 23:21:26 -05:00
trvon
4ecdbe7d73
fix: moved SocketServer read/write into dedicated IOCoordinator and fixed proxy socket paths resolution logic. Fixed TSAN race in SymSpell init 2026-02-11 14:40:23 -05:00
trvon
cc28b8a343
feat: instrumented embedding for deeper insight into hotspots. Fixed CoreML scaling that resulted in high ram usage due to VRAM calculation. Expanded testing for embedding and draining of work 2026-02-11 12:07:08 -05:00
trvon
ff77c6b304
fix: improving socket and chunking reliability and robustness. Added better regression tests and errror logging logic for state connections 2026-02-10 12:25:30 -05:00
trvon
a177027320
fix: house cleaning 2026-02-09 21:12:26 -05:00
trvon
925426426a
feat: expanding mcp UI support, improvements for slots for component usage and other socket server stabilizations 2026-02-09 20:02:26 -05:00
trvon
1d255d59ce
feat: address bugs with scaling, batching in daemon processing 2026-02-09 17:07:00 -05:00
trvon
cc1212bb1f
fix: fixing pipeline ingestion of kg and embeddings. Still finding stalls and will continue to investigate 2026-02-09 10:39:48 -05:00
trvon
4d6c163c9d
feat: adding drain to constrexpr metrics 2026-02-09 01:45:38 -05:00
trvon
0c8ffea321
feat: improving ingestion speed with kg batch and updating metrics to use constexpr 2026-02-09 01:33:24 -05:00
trvon
c64a0b239d
feat: adding fuzzing updates and fixes in ipc 2026-02-08 22:20:47 -05:00
trvon
f4f3c055fe
feat: benchmark expansion and normalization. Imrpoving metrics for repair and models 2026-02-08 20:35:39 -05:00
trvon
c83110319d
feat: benchmark, model testing / enhancements, insertion performance updates 2026-02-08 18:33:29 -05:00
trvon
a3522b8b22
fix: expose gpu usage, add command imrpovments and other bug fixes 2026-02-08 14:04:57 -05:00
trvon
430057f4bf
fix: SocketServer slots for proxy and cli requests, fixed tests and increase doctor request handling 2026-02-08 13:02:08 -05:00
trvon
3c09becde5
fix: repair improvements and socketserver hardening 2026-02-07 23:37:32 -05:00
trvon
448f5e47b4
fix: fixing repairmanager logic, windows build issues, linting, resource scaling 2026-02-07 22:07:56 -05:00
trvon
567ee32a99
fix: fixing macos coreml detection 2026-02-06 10:16:35 -05:00
trvon
3fd243f687
fix: repair command improvements and optimizations 2026-02-06 10:15:50 -05:00
trvon
90d28d722e
fix: docker build fixes addition clamping logic for onbound memory group 2026-02-06 09:27:34 -05:00
trvon
df1348f3c7
fix: tuning improvements and better tests. Now working address memory issue of huge queue when ingesting allot of files 2026-02-06 04:31:09 -05:00
trvon
ab06614505
fix(repair): fix SIGSEGV crash, embedding failures, and runaway repair loops
- Fix writer_drain() SIGSEGV by setting connection_closing_ on write failure
- Add InternalError/ResourceExhausted to embedding retry list
- Fix document-level embedding chunk_id (0→999999) and level mismatch
- Add circuit breaker to bail out after consecutive/rate-threshold failures
- Add semaphore concurrency limiter (max 2) for repair embed requests
- Replace silent catch-all with error-logging catch blocks
2026-02-05 21:15:26 -05:00
trvon
fc4b5ceffb
fix: fixing metrics system and removing deadcode 2026-02-05 19:41:50 -05:00