# Changelog

All notable changes to YAMS (Yet Another Memory System) will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

- [SourceHut](https://sr.ht/~trvon/yams/): https://sr.ht/~trvon/yams/

## Archived Changelogs
- v0.6.x archive: docs/changelogs/v0.6.md
- v0.5.x archive: docs/changelogs/v0.5.md
- v0.4.x archive: docs/changelogs/v0.4.md
- v0.3.x archive: docs/changelogs/v0.3.md
- v0.2.x archive: docs/changelogs/v0.2.md
- v0.1.x archive: docs/changelogs/v0.1.md

## [v0.7.8] - 2025-11-14

### Added
- **Thread Pool Consolidation**
  - **WorkCoordinator Component**: New centralized thread pool manager with Boost.Asio io_context
    - Replaces 3 separate thread pools (IngestService, PostIngestQueue, EmbeddingService)
    - Provides strand allocation for per-service ordering guarantees
    - Hardware-aware thread count (8-32 threads based on CPU cores)
  - **Search Service Parallel Post-Processing** 
    - New `ParallelPostProcessor` class for concurrent search result processing
    - Parallelizes filtering, facet generation, and highlighting when result count ≥ 100
    - Uses `std::async` to run independent operations concurrently
    - Threshold-based activation (PARALLEL_THRESHOLD = 100) avoids overhead on small result sets
    - **Performance Measured** (100 iterations):
      - 100 results: 0.06ms (~1.66M ops/sec) - sequential path
      - 500 results: 0.23ms (~2.21M ops/sec) - parallel path
      - 1000 results: 0.43ms (~2.32M ops/sec) - parallel path
      - **Speedup**: ~3.4x faster at 1000 results vs linear scaling
    - Location: `include/yams/search/parallel_post_processor.hpp`, `src/search/parallel_post_processor.cpp`
    - Integration: `search_executor.cpp` now uses ParallelPostProcessor instead of sequential processing
    - Benchmarks: `tests/benchmarks/search_benchmarks.cpp`
### Changed
### Changed
- **Search Service**: `--fuzzy` searches now merge BM25 keyword matches with fuzzy results so enabling typo tolerance never suppresses literal hits. (`src/app/services/search_service.cpp`)
- **Metadata Repository**: Removed the default 50K fuzzy-index cap. The index now covers the full corpus by default and only enforces limits when `YAMS_FUZZY_INDEX_LIMIT` is set, adding a small safety buffer and explicit guard logging. (`src/metadata/metadata_repository.cpp`, `include/yams/metadata/fuzzy_index_builder.h`)
- **Service Architecture Refactor**
  - **IngestService**: Converted from manual thread pool to strand-based channel polling
    - Removed `kSyncThreshold` heuristics and `compat::jthread` pool
    - New `channelPoller()` awaitable for document processing
  - **PostIngestQueue**: Converted from worker threads to strand-based pipeline
    - Removed Worker struct, thread pool, and token bucket scheduler (~200 lines)
    - Implemented awaitable pipeline: `processMetadataStage → (processKnowledgeGraphStage || processEmbeddingStage)`
    - Parallel KG and Embedding stages using `make_parallel_group`
  - **EmbeddingService**: Converted from worker threads to strand-based channel polling
    - Removed worker thread pool (~70 lines)
    - New `channelPoller()` awaitable with async timer
  - **TuningManager**: Converted from manual thread to strand-based periodic execution
    - Removed `compat::jthread` with stop_token
    - New `tuningLoop()` awaitable with `boost::asio::steady_timer`
    - Uses WorkCoordinator strand for pool size adjustments
    - Maintains `TuneAdvisor::statusTickMs()` polling interval
  - **DaemonMetrics**: Converted from manual thread to strand-based polling loop
    - Removed `std::thread` for CPU/memory metrics collection
    - New `pollingLoop()` awaitable with 250ms timer interval
    - Uses WorkCoordinator strand for metric updates
    - Thread-safe snapshot access via `shared_mutex`
  - **BackgroundTaskManager**: Migrated from GlobalIOContext to WorkCoordinator
    - Removed fallback to GlobalIOContext (proper architectural separation)
    - Now uses WorkCoordinator executor for all background tasks
    - Integrated with unified work-stealing thread pool
    - Fts5Job consumer polling delay reduced: 200ms → 10ms (20x throughput improvement)
    - Fixed orphan scan queue overflow (was causing hundreds of dropped batches)
  - **ServiceManager**: Refactored async operations
    - Eliminated all 5 uses of `std::future`/`std::async`
    - Converted database operations to use `make_parallel_group` with timeouts
- **SearchPool Removal**
  - Deleted the unused `SearchPool` component and associated meson/build wiring
  - `ServiceManager` no longer constructs dead search infrastructure; `HybridSearchEngine`
    remains the sole search path
  - `TuneAdvisor`/`TuningManager` now derive concurrency targets directly from
    `SearchExecutor` load metrics instead of phantom pool sizes
- **Ingestion Pipeline Cleanup (PBI-004 Phase 1)**
  - **Removed `deferExtraction` Technical Debt**: Eliminated bypass mechanism that skipped full production pipeline
    - Removed `deferExtraction` field from `StoreDocumentRequest` and `AddDirectoryRequest` structs
    - Removed conditional logic in DocumentService that skipped FTS5 extraction
    - All document ingestion now uses full pipeline: metadata storage → FTS5 extraction → PostIngestQueue → (KG extraction || Embedding generation)
    - Updated IngestService to always enqueue to PostIngestQueue (removed lines setting `deferExtraction=true`)
    - Updated CLI add_command fallback paths (3 locations) to use full pipeline
    - Updated mobile bindings to remove `sync_now`-based deferral
    - Removed `--defer-extraction` and `--no-defer-extraction` flags from ingestion_throughput_bench
    - Updated test helpers (tests/common/capability.h, integration test) to use full pipeline
  - **Impact**: Benchmarks and production code now exercise identical code paths, improving test coverage and eliminating hidden complexity
- **Grep Output Update**
  - New default output format
  - Example output:
    ```
    === Results for "TaskManager" in 3 files (5 regex, 2 semantic) ===

    File: src/core/TaskManager.cpp (cpp)
       Matches: 3 (3 regex)

       Line   45: [Regex] class TaskManager {
       Line  102: [Regex] TaskManager::TaskManager() : initialized_(false) {
       Line  237: [Regex] void TaskManager::shutdown() {

    [Total: 7 matches across 3 files]
    ```
  - Location: `src/cli/commands/grep_command.cpp:531-645`
- **Grep Service Optimizations**
  - **Literal Extraction from Regex Patterns**
    - New `LiteralExtractor` utility extracts literal substrings from regex patterns
    - Enables two-phase matching: fast literal pre-filter → full regex only on candidates
    - Based on ripgrep's literal extraction strategy
  - **Boyer-Moore-Horspool (BMH) String Search**
    - Replaces `std::string::find()` with BMH algorithm for patterns ≥ 3 characters
  - **SIMD Vectorized Newline Scanning**
    - Platform-specific implementations: AVX2 (32 bytes), SSE2 (16 bytes), NEON (16 bytes)
    - Scalar fallback using optimized memchr for portability
    - Replaces byte-by-byte scanning in line boundary detection
    - Performance: 4-8x speedup on large files
  - **Parallel Candidate Filtering**
    - Pre-filters unsuitable files before worker distribution using `std::async`
    - Integrates `magic_numbers.hpp` for accurate binary detection (86 compile-time patterns)
    - Filters build artifacts (.o, .class, .pyc), libraries (.a, .so, .dll), executables, packages
    - Chunk-based parallel processing for large candidate sets (>100 files)
    - Performance: 2-4x speedup on large corpora

### Fixed
- **Content-backed Fuzzy Hits**: Content-derived fuzzy matches (`_content` entries) now map back to their owning documents, ensuring CLI searches show the expected files. (`src/metadata/metadata_repository.cpp`, `tests/unit/metadata/metadata_repository_test.cpp`)
- **Cold Start Vector Index Loading**: Fixed issue where search and grep commands returned no results after daemon cold start despite having indexed documents. 
- **Search Async Path**: Fixed `SearchCommand::executeAsync()` not populating `pathPatterns` field in daemon request, causing server-side multi-pattern filtering to fail. The async code path (default execution) now correctly sends all include patterns to the daemon, matching the behavior of the sync path. (`src/cli/commands/search_command.cpp:1360-1365`)
- **Database Schema Compatibility**: Fixed "constraint failed" errors during document insertion on databases with migration v12 (pre-path-indexing schema). The `insertDocument()` function now conditionally builds INSERT statements based on the `hasPathIndexing_` flag, supporting both legacy (13-column) and modern (17-column with path indexing) schemas. This allows YAMS to work correctly regardless of whether migration v13 has been applied. (`src/metadata/metadata_repository.cpp:318-380`)
- **MCP Protocol Version Negotiation**: Fixed "Unsupported protocol version requested by client" error (code -32901) by making protocol version negotiation permissive by default (`strictProtocol_ = false`). The server now gracefully accepts any protocol version requested by clients, falling back to the latest supported version (`2025-03-26`) if the requested version is not in the supported list. Also added intermediate MCP protocol versions (`2024-12-05`, `2025-01-15`) to the supported list. This ensures maximum compatibility with MCP clients regardless of which spec version they implement. (`src/mcp/mcp_server.cpp:560,1254-1260`)
- **MCP Large Response Buffering**: Fixed "Error: MPC -32602: Error: End of file" errors when MCP server sends large responses (list, search, grep with many results). Implemented chunked buffered output in `StdioTransport::sendFramedSerialized()` that breaks payloads >512KB into 64KB chunks with explicit flushes between chunks. This prevents stdout buffer overflow and ensures reliable delivery of large JSON-RPC responses over stdio transport. Also added threshold-based routing in `MCPServer::sendResponse()` to use buffered sending for payloads >256KB. (`src/mcp/mcp_server.cpp:69-95,169-203`)