trevon/yams
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2
yams/CHANGELOG.md
trvon d9c7b14208
 bug fixes. v0.7.8
2025-11-14 20:26:09 -05:00

10 KiB
Raw Permalink Blame History

Changelog

All notable changes to YAMS (Yet Another Memory System) will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Archived Changelogs

  • v0.6.x archive: docs/changelogs/v0.6.md
  • v0.5.x archive: docs/changelogs/v0.5.md
  • v0.4.x archive: docs/changelogs/v0.4.md
  • v0.3.x archive: docs/changelogs/v0.3.md
  • v0.2.x archive: docs/changelogs/v0.2.md
  • v0.1.x archive: docs/changelogs/v0.1.md

[v0.7.8] - 2025-11-14

Added

  • Thread Pool Consolidation
    • WorkCoordinator Component: New centralized thread pool manager with Boost.Asio io_context
      • Replaces 3 separate thread pools (IngestService, PostIngestQueue, EmbeddingService)
      • Provides strand allocation for per-service ordering guarantees
      • Hardware-aware thread count (8-32 threads based on CPU cores)
    • Search Service Parallel Post-Processing
      • New ParallelPostProcessor class for concurrent search result processing
      • Parallelizes filtering, facet generation, and highlighting when result count ≥ 100
      • Uses std::async to run independent operations concurrently
      • Threshold-based activation (PARALLEL_THRESHOLD = 100) avoids overhead on small result sets
      • Performance Measured (100 iterations):
        • 100 results: 0.06ms (~1.66M ops/sec) - sequential path
        • 500 results: 0.23ms (~2.21M ops/sec) - parallel path
        • 1000 results: 0.43ms (~2.32M ops/sec) - parallel path
        • Speedup: ~3.4x faster at 1000 results vs linear scaling
      • Location: include/yams/search/parallel_post_processor.hpp, src/search/parallel_post_processor.cpp
      • Integration: search_executor.cpp now uses ParallelPostProcessor instead of sequential processing
      • Benchmarks: tests/benchmarks/search_benchmarks.cpp

Changed

Changed

  • Search Service: --fuzzy searches now merge BM25 keyword matches with fuzzy results so enabling typo tolerance never suppresses literal hits. (src/app/services/search_service.cpp)
  • Metadata Repository: Removed the default 50K fuzzy-index cap. The index now covers the full corpus by default and only enforces limits when YAMS_FUZZY_INDEX_LIMIT is set, adding a small safety buffer and explicit guard logging. (src/metadata/metadata_repository.cpp, include/yams/metadata/fuzzy_index_builder.h)
  • Service Architecture Refactor
    • IngestService: Converted from manual thread pool to strand-based channel polling
      • Removed kSyncThreshold heuristics and compat::jthread pool
      • New channelPoller() awaitable for document processing
    • PostIngestQueue: Converted from worker threads to strand-based pipeline
      • Removed Worker struct, thread pool, and token bucket scheduler (~200 lines)
      • Implemented awaitable pipeline: processMetadataStage → (processKnowledgeGraphStage || processEmbeddingStage)
      • Parallel KG and Embedding stages using make_parallel_group
    • EmbeddingService: Converted from worker threads to strand-based channel polling
      • Removed worker thread pool (~70 lines)
      • New channelPoller() awaitable with async timer
    • TuningManager: Converted from manual thread to strand-based periodic execution
      • Removed compat::jthread with stop_token
      • New tuningLoop() awaitable with boost::asio::steady_timer
      • Uses WorkCoordinator strand for pool size adjustments
      • Maintains TuneAdvisor::statusTickMs() polling interval
    • DaemonMetrics: Converted from manual thread to strand-based polling loop
      • Removed std::thread for CPU/memory metrics collection
      • New pollingLoop() awaitable with 250ms timer interval
      • Uses WorkCoordinator strand for metric updates
      • Thread-safe snapshot access via shared_mutex
    • BackgroundTaskManager: Migrated from GlobalIOContext to WorkCoordinator
      • Removed fallback to GlobalIOContext (proper architectural separation)
      • Now uses WorkCoordinator executor for all background tasks
      • Integrated with unified work-stealing thread pool
      • Fts5Job consumer polling delay reduced: 200ms → 10ms (20x throughput improvement)
      • Fixed orphan scan queue overflow (was causing hundreds of dropped batches)
    • ServiceManager: Refactored async operations
      • Eliminated all 5 uses of std::future/std::async
      • Converted database operations to use make_parallel_group with timeouts
  • SearchPool Removal
    • Deleted the unused SearchPool component and associated meson/build wiring
    • ServiceManager no longer constructs dead search infrastructure; HybridSearchEngine remains the sole search path
    • TuneAdvisor/TuningManager now derive concurrency targets directly from SearchExecutor load metrics instead of phantom pool sizes
  • Ingestion Pipeline Cleanup (PBI-004 Phase 1)
    • Removed deferExtraction Technical Debt: Eliminated bypass mechanism that skipped full production pipeline
      • Removed deferExtraction field from StoreDocumentRequest and AddDirectoryRequest structs
      • Removed conditional logic in DocumentService that skipped FTS5 extraction
      • All document ingestion now uses full pipeline: metadata storage → FTS5 extraction → PostIngestQueue → (KG extraction || Embedding generation)
      • Updated IngestService to always enqueue to PostIngestQueue (removed lines setting deferExtraction=true)
      • Updated CLI add_command fallback paths (3 locations) to use full pipeline
      • Updated mobile bindings to remove sync_now-based deferral
      • Removed --defer-extraction and --no-defer-extraction flags from ingestion_throughput_bench
      • Updated test helpers (tests/common/capability.h, integration test) to use full pipeline
    • Impact: Benchmarks and production code now exercise identical code paths, improving test coverage and eliminating hidden complexity
  • Grep Output Update
    • New default output format
    • Example output: 
      === Results for "TaskManager" in 3 files (5 regex, 2 semantic) ===

File: src/core/TaskManager.cpp (cpp)
   Matches: 3 (3 regex)

   Line   45: [Regex] class TaskManager {
   Line  102: [Regex] TaskManager::TaskManager() : initialized_(false) {
   Line  237: [Regex] void TaskManager::shutdown() {

[Total: 7 matches across 3 files]
    • Location: src/cli/commands/grep_command.cpp:531-645
  • Grep Service Optimizations
    • Literal Extraction from Regex Patterns
      • New LiteralExtractor utility extracts literal substrings from regex patterns
      • Enables two-phase matching: fast literal pre-filter → full regex only on candidates
      • Based on ripgrep's literal extraction strategy
    • Boyer-Moore-Horspool (BMH) String Search
      • Replaces std::string::find() with BMH algorithm for patterns ≥ 3 characters
    • SIMD Vectorized Newline Scanning
      • Platform-specific implementations: AVX2 (32 bytes), SSE2 (16 bytes), NEON (16 bytes)
      • Scalar fallback using optimized memchr for portability
      • Replaces byte-by-byte scanning in line boundary detection
      • Performance: 4-8x speedup on large files
    • Parallel Candidate Filtering
      • Pre-filters unsuitable files before worker distribution using std::async
      • Integrates magic_numbers.hpp for accurate binary detection (86 compile-time patterns)
      • Filters build artifacts (.o, .class, .pyc), libraries (.a, .so, .dll), executables, packages
      • Chunk-based parallel processing for large candidate sets (>100 files)
      • Performance: 2-4x speedup on large corpora

Fixed

  • Content-backed Fuzzy Hits: Content-derived fuzzy matches (_content entries) now map back to their owning documents, ensuring CLI searches show the expected files. (src/metadata/metadata_repository.cpp, tests/unit/metadata/metadata_repository_test.cpp)
  • Cold Start Vector Index Loading: Fixed issue where search and grep commands returned no results after daemon cold start despite having indexed documents.
  • Search Async Path: Fixed SearchCommand::executeAsync() not populating pathPatterns field in daemon request, causing server-side multi-pattern filtering to fail. The async code path (default execution) now correctly sends all include patterns to the daemon, matching the behavior of the sync path. (src/cli/commands/search_command.cpp:1360-1365)
  • Database Schema Compatibility: Fixed "constraint failed" errors during document insertion on databases with migration v12 (pre-path-indexing schema). The insertDocument() function now conditionally builds INSERT statements based on the hasPathIndexing_ flag, supporting both legacy (13-column) and modern (17-column with path indexing) schemas. This allows YAMS to work correctly regardless of whether migration v13 has been applied. (src/metadata/metadata_repository.cpp:318-380)
  • MCP Protocol Version Negotiation: Fixed "Unsupported protocol version requested by client" error (code -32901) by making protocol version negotiation permissive by default (strictProtocol_ = false). The server now gracefully accepts any protocol version requested by clients, falling back to the latest supported version (2025-03-26) if the requested version is not in the supported list. Also added intermediate MCP protocol versions (2024-12-05, 2025-01-15) to the supported list. This ensures maximum compatibility with MCP clients regardless of which spec version they implement. (src/mcp/mcp_server.cpp:560,1254-1260)
  • MCP Large Response Buffering: Fixed "Error: MPC -32602: Error: End of file" errors when MCP server sends large responses (list, search, grep with many results). Implemented chunked buffered output in StdioTransport::sendFramedSerialized() that breaks payloads >512KB into 64KB chunks with explicit flushes between chunks. This prevents stdout buffer overflow and ensures reliable delivery of large JSON-RPC responses over stdio transport. Also added threshold-based routing in MCPServer::sendResponse() to use buffered sending for payloads >256KB. (src/mcp/mcp_server.cpp:69-95,169-203)