trevon/yams
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 2
yams/CHANGELOG.md

9.2 KiB
Raw Permalink Blame History

Changelog

All notable changes to YAMS (Yet Another Memory System) will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Archived Changelogs

  • v0.6.x archive: docs/changelogs/v0.6.md
  • v0.5.x archive: docs/changelogs/v0.5.md
  • v0.4.x archive: docs/changelogs/v0.4.md
  • v0.3.x archive: docs/changelogs/v0.3.md
  • v0.2.x archive: docs/changelogs/v0.2.md
  • v0.1.x archive: docs/changelogs/v0.1.md

[v0.7.1] - 2025-09-29

Changed

  • GrepService: expanded candidate discovery to preselect from req.paths using SQL LIKE prefix scans, aligning service behavior with CLI expectations for directory patterns.

  • Post‑ingest pipeline: refactored PostIngestQueue into true staged processing (Metadata → Knowledge Graph → Embeddings). Metadata extraction persists text and indexes FTS first; completion enqueues independent KG and Embedding tasks so slow extractors no longer block vector availability. Weighted‑fair scheduling across queues maintains fairness.

  • Embeddings integrated into the main pipeline: the new Embedding stage generates vectors immediately after extraction using the daemon’s configured model and inserts into the vector DB, marking embedding status in metadata. Initial embedding generation is no longer coupled to the RepairCoordinator’s idle window.

  • ServiceManager enqueue path: simplified enqueuePostIngest to a direct blocking enqueue (removed busy‑wait retry loop). This improves predictability and throughput under high load.

  • CLI Download UX: yams download now clearly displays the ingested content hash (labeled "Content Hash") rather than a transfer/download hash. The success tip also shows both retrieval options when a friendly name is present: yams get --name "<name>" or yams get <content-hash>. JSON output shape is unchanged and still returns hash, which now prefers the ingested hash when available.

Fixed

  • GrepService streaming: flushes the final partial line when scanning cold CAS streams so single-line files are matched reliably (e.g., hello.txt).

  • Reduced GrepService log verbosity to debug for internal counters and match traces.

  • Fixed IPC protocol regression where grep and list commands failed to properly communicate with the daemon after migration, causing incomplete results or timeouts in multi-service environments.

    • This issue impacted other tools result output

  • Guarded compression monitor global statistics with a dedicated mutex to stop concurrent tracker updates from crashing unit_shard5 (validated via meson test -C build/debug unit_shard5 --print-errorlogs).

  • Repaired the document_service metadata pipeline regression so fixture-driven search tests no longer observe missing extracted content.

  • MCP stdio transport: replaced unused static output mutex with an instance mutex to satisfy ODR/build on certain platforms.

Maintenance

  • RepairCoordinator refocus: on live DocumentAdded events, skip queuing when the post‑ingest pipeline is active (embeddings are now scheduled by PostIngestQueue). Coordinator continues to scan/repair missed or failed embeddings opportunistically.

[v0.7.0] - 2025-09-25

Highlights

  • These changes reduce CPU spikes observed in profiles for large greps and remove blocking storage scans from interactive status paths. Post-ingest work is intentionally bounded; processing may take longer, but overall system responsiveness improves.
  • Stability: resolved connection timeouts under multi-agent load by removing the hard 100-connection cap and deriving a dynamic accept limit. Defaults honor YAMS_MAX_ACTIVE_CONN or compute a safe cap from CPU cores and IO concurrency.
  • Throughput: added tuning profiles (efficient | balanced | aggressive). Profiles modulate pool growth, IO thresholds, and post-ingest workers. Default is balanced.
  • Indexing UX: Add/ingest returns fast; post‑ingest queue handles FTS/embeddings/KG in the background. Path‑series versioning (Phase 1) is on by default behind an env flag.

Added

  • Tuning profiles selectable via config or env:

    • Config: yams config set tuning.profile <efficient|balanced|aggressive>
    • Env: YAMS_TUNING_PROFILE=<profile>

  • Config defaults now include [tuning] profile = "balanced".

  • Docs: docs/admin/tuning_profiles.md covering profiles, envs, and observability.

  • Versioning (Phase 1): path‑series lineage with VersionOf edges and metadata flags version, is_latest, series_key. Duplicate (same hash) re‑ingest does not create a new version; alternate locations and timestamps are updated.

  • CLI Search: grouped multi‑version presentation (default on) with new controls.

    • Groups results by canonical path when multiple versions of the same file are returned.
    • New flags:
      • --no-group-versions — disable grouping and show the flat list.
      • --versions <latest|all> — choose best only (default: latest) or list versions per path.
      • --versions-topk <N> — cap versions shown per path when --versions=all (default: 3).
      • --versions-sort <score|path|title> — sort versions within a group (default: score).
      • --no-tools — hide per‑version tool hints.
      • --json-grouped — emit grouped JSON; plain --json remains flat and backward compatible.
    • Tool hints shown per version (when grouped): yams get --hash <hash> | yams cat --hash <hash> | yams restore --hash <hash>; if a local file path is resolved, a yams diff --hash <hash> <local-path> hint is added.
    • Environment toggles: YAMS_NO_GROUP_VERSIONS=1 and YAMS_NO_GROUP_TOOLS=1 to flip defaults.
    • Note: This is a presentation‑layer change; service/daemon APIs are unchanged.

Changed

  • Build System
    • The primary build system has been migrated from CMake to Meson. All build, test, and packaging scripts have been updated to use the new Meson-based workflow.
  • Status/Stats (CLI): use daemon metrics by default and never trigger local storage scans.
    • yams status and yams stats -v now render from the same non-detailed daemon snapshot; removed the "scanning storage..." spinner and filesystem walks.
    • Verbose output formats the JSON fields instead of performing extra scans.
  • Tools/Stats (yams-tools): tools/yams-tools/src/commands/stats_command.cpp refactored to prefer daemon-first metrics with a legacy local fallback only if daemon is unavailable.
  • MCP add_directory: switched to daemon-first ingestion with a brief readiness wait to avoid "Content store not available" races. Removes local store preflight; maps NotInitialized to a clear, retryable message from the daemon.
  • MCP search: path normalization + optional diff parity with CLI.
    • New request field include_diff adds a structured diff block to results when the path_pattern points to a local file; mirrors yams search diff behavior.
    • MCPSearch DTOs extended to round-trip include_diff, diff, and local_input_file.
  • Daemon accept scaling: removed fixed cap; now dynamically computes maxConnections from recommendedThreads * ioConnPerThread * 4 (min 256) unless YAMS_MAX_ACTIVE_CONN is set.
  • Backpressure: increased default read pause to 10ms to smooth heavy load.
  • Post‑ingest: preserves bounded capacity; de‑dupes inflight, indexes FTS, updates fuzzy index, and emits KG nodes/edges best‑effort.
  • Status/Stats: JSON correctness improvements; omit misleading savings when physical size unknown; surface post‑ingest bus usage and document counters.
  • CLI Search: grouping of multiple versions per path is enabled by default; paths‑only output and flat JSON remain unchanged unless --json-grouped is specified.

Fixed

  • Regression in metadata extraction and storage used in search and grep tools The async post-ingest pipeline never persisted extracted text into the metadata store. As a result, document_content stayed empty, so search, repairs, and semantic pipelines saw “Document content not found” despite vector insert logs.
  • Many tuning optimizations for daemon usage
  • Grep pipeline: staged KG → metadata → content with caps and budget.
    • Prefers "hot" text (metadata-extracted) and caps cold CAS reads; early path/include filters.
    • Added a global time budget (internal) to stop long content scans gracefully.
    • Capped grep worker threads to a small, background-friendly number by default (≤4).
  • Grep streaming optimization: replaced per-character streambuf overflow with bulk line splitting (memchr-based) to eliminate the per-byte hotspot in profiles during CAS streaming.
  • Post-ingest queue: bounded by configuration, not CPU heuristics.
    • Default worker threads set conservatively to 1 unless configured in [tuning] as post_ingest_threads. Queue capacity now honored from post_ingest_queue_max.
    • Added a tiny yield between tasks to reduce contention and smooth CPU.
  • Addressed intermittent CLI timeouts and “Broken pipe” logs observed when many agents connected concurrently. Accept loop backoff now respects the higher connection cap and IO pool growth from the tuning manager.
  • Minor unit test fixes (Result value handling) to unblock CI.