trevon/yams
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2
yams/CHANGELOG.md

6.4 KiB
Raw Permalink Blame History

Changelog

All notable changes to YAMS (Yet Another Memory System) will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Archived Changelogs

  • v0.6.x archive: docs/changelogs/v0.6.md
  • v0.5.x archive: docs/changelogs/v0.5.md
  • v0.4.x archive: docs/changelogs/v0.4.md
  • v0.3.x archive: docs/changelogs/v0.3.md
  • v0.2.x archive: docs/changelogs/v0.2.md
  • v0.1.x archive: docs/changelogs/v0.1.md

[v0.7.8] - Unreleased

Added

  • Search Service Query Literal Extraction

    • New QueryLiteralExtractor class for extracting selective terms from search QueryAST
    • Recursively traverses AST (Term, Phrase, And, Or, Field nodes) to collect literals
    • Scores terms by selectivity: length, uppercase chars, special characters
    • Enables future two-phase search: literal pre-filter → full FTS5
    • Location: include/yams/search/query_literal_extractor.hpp, src/search/query_literal_extractor.cpp

  • Retrieval Service Optimizations

    • BMH Suffix Matching: Applied to getByNameSmart() post-SQL filtering
      • Threshold-based activation: ≥100 candidates → BMH, <100 → sequential rfind()
    • Parallel Session List Processing: Applied to basename matching
      • Threshold: ≥100 items → std::async parallel filter
    • Location: src/app/services/retrieval_service.cpp

  • Search Service Parallel Post-Processing

    • New ParallelPostProcessor class for concurrent search result processing
    • Parallelizes filtering, facet generation, and highlighting when result count ≥ 100
    • Uses std::async to run independent operations concurrently
    • Threshold-based activation (PARALLEL_THRESHOLD = 100) avoids overhead on small result sets
    • Performance Measured (100 iterations):
      • 100 results: 0.06ms (~1.66M ops/sec) - sequential path
      • 500 results: 0.23ms (~2.21M ops/sec) - parallel path
      • 1000 results: 0.43ms (~2.32M ops/sec) - parallel path
      • Speedup: ~3.4x faster at 1000 results vs linear scaling
    • Location: include/yams/search/parallel_post_processor.hpp, src/search/parallel_post_processor.cpp
    • Integration: search_executor.cpp now uses ParallelPostProcessor instead of sequential processing
    • Benchmarks: tests/benchmarks/search_benchmarks.cpp

  • Grep Service Optimizations

    • Literal Extraction from Regex Patterns
      • New LiteralExtractor utility extracts literal substrings from regex patterns
      • Enables two-phase matching: fast literal pre-filter → full regex only on candidates
      • Based on ripgrep's literal extraction strategy
      • Performance: 10-50x speedup on typical patterns with rare matches
    • Boyer-Moore-Horspool (BMH) String Search
      • Replaces std::string::find() with BMH algorithm for patterns ≥ 3 characters
      • Sublinear average-case complexity: O(n/m) vs O(n·m) for naive search
      • Preprocessing builds bad-character shift table for efficient skipping
      • Performance: 2-3x speedup over std::string::find on typical text
    • SIMD Vectorized Newline Scanning
      • Platform-specific implementations: AVX2 (32 bytes), SSE2 (16 bytes), NEON (16 bytes)
      • Scalar fallback using optimized memchr for portability
      • Replaces byte-by-byte scanning in line boundary detection
      • Performance: 4-8x speedup on large files
    • Parallel Candidate Filtering
      • Pre-filters unsuitable files before worker distribution using std::async
      • Integrates magic_numbers.hpp for accurate binary detection (86 compile-time patterns)
      • Filters build artifacts (.o, .class, .pyc), libraries (.a, .so, .dll), executables, packages
      • Chunk-based parallel processing for large candidate sets (>100 files)
      • Performance: 2-4x speedup on large corpora

Changed

  • Search Executor Post-Processing

    • Replaced sequential post-processing with ParallelPostProcessor::process()
    • Filters, facets, highlights, and snippets now run concurrently on large result sets
    • Maintains identical output to sequential processing

  • Grep Service Pattern Matching

    • Pure literal patterns ≥ 3 chars now use BMH instead of std::string::find
    • Regex patterns with extractable literals use two-phase matching
    • Binary file detection now uses magic_numbers.hpp prune categories
    • Line boundary detection uses SIMD vectorization where available
    • Maintains backward compatibility: all existing tests pass

Performance

  • Benchmark Suite: tests/benchmarks/grep_benchmarks.cpp
    • 8 benchmark scenarios: literal patterns, regex, large corpus (10K files), single large file (100MB), high match count, case-insensitive, word boundary, complex regex
    • Framework ready for before/after performance validation --include="*.md,*.cpp" correctly return all matching files regardless of their directory location.
  • Search Async Path: Fixed SearchCommand::executeAsync() not populating pathPatterns field in daemon request, causing server-side multi-pattern filtering to fail. The async code path (default execution) now correctly sends all include patterns to the daemon, matching the behavior of the sync path. (src/cli/commands/search_command.cpp:1360-1365)
  • Database Schema Compatibility: Fixed "constraint failed" errors during document insertion on databases with migration v12 (pre-path-indexing schema). The insertDocument() function now conditionally builds INSERT statements based on the hasPathIndexing_ flag, supporting both legacy (13-column) and modern (17-column with path indexing) schemas. This allows YAMS to work correctly regardless of whether migration v13 has been applied. (src/metadata/metadata_repository.cpp:318-380)
  • MCP Protocol Version Negotiation: Fixed "Unsupported protocol version requested by client" error (code -32901) by making protocol version negotiation permissive by default (strictProtocol_ = false). The server now gracefully accepts any protocol version requested by clients, falling back to the latest supported version (2025-03-26) if the requested version is not in the supported list. Also added intermediate MCP protocol versions (2024-12-05, 2025-01-15) to the supported list. This ensures maximum compatibility with MCP clients regardless of which spec version they implement. (src/mcp/mcp_server.cpp:560,1254-1260)