162 lines
5.2 KiB
Markdown
162 lines
5.2 KiB
Markdown
# LongMemEval_S Retrieval Quality Baseline
|
|
|
|
**Date:** 2026-04-08
|
|
**Dataset:** LongMemEval_S (ICLR 2025) — `xiaowu0162/longmemeval-cleaned`, split `longmemeval_s_cleaned`
|
|
**Conversion:** `scripts/prepare_longmemeval_s.py` (BEIR format)
|
|
|
|
## Dataset Statistics
|
|
|
|
| Stat | Value |
|
|
|------|-------|
|
|
| Sessions (corpus docs) | 940 |
|
|
| Queries evaluated | 470 (of 500; 30 abstention/false-premise filtered) |
|
|
| Qrels | 890 |
|
|
| Chunks (embedded) | 5045 |
|
|
| Avg chunks/doc | 5.37 |
|
|
| Vectors generated | 5985 |
|
|
| Embedding model | embeddinggemma-300m (768-dim) |
|
|
|
|
## Baseline Results
|
|
|
|
| Metric | Hybrid (text+vector) | Keyword-only (FTS5) | Delta |
|
|
|--------|---------------------|---------------------|-------|
|
|
| MRR | 0.4074 | 0.5026 | -0.0951 |
|
|
| Recall@10 | 0.6529 | 0.6202 | +0.0327 |
|
|
| NDCG@10 | 0.4181 | — | — |
|
|
| MAP | 0.3629 | 0.4893 | -0.1264 |
|
|
| Precision@10 | 0.1148 | 0.1481 | -0.0333 |
|
|
|
|
## Auto-Tuned Configuration
|
|
|
|
The benchmark auto-selected:
|
|
```
|
|
state=SCIENTIFIC
|
|
zoom=MAP
|
|
rrfK=12
|
|
textWeight=0.70
|
|
vectorWeight=0.25
|
|
fusion=WEIGHTED_RECIPROCAL
|
|
semanticRescueSlots=0
|
|
similarityThreshold=0.40
|
|
enableSubPhraseRescoring=true
|
|
rerankAnchoredMinRelativeScore=0.70
|
|
```
|
|
|
|
Reranker: `bge-reranker-base` (cross-encoder, CPU)
|
|
|
|
## Run Configuration
|
|
|
|
```bash
|
|
./setup.sh Debug --with-tests
|
|
meson compile -C build/debug retrieval_quality_bench
|
|
|
|
DYLD_INSERT_LIBRARIES=... \
|
|
YAMS_TEST_SAFE_SINGLE_INSTANCE=1 \
|
|
YAMS_BENCH_DATASET=longmemeval_s \
|
|
YAMS_BENCH_EMBED_MAX_WAIT=0 \
|
|
./build/debug/tests/benchmarks/retrieval_quality_bench
|
|
```
|
|
|
|
Total runtime: ~4 hours
|
|
|
|
## Root Cause Analysis
|
|
|
|
Vector search adds only +3.3% recall and MRR *regresses* by -0.095. Three compounding issues:
|
|
|
|
### 1. MAX-POOL Chunk Aggregation
|
|
|
|
Only the single highest-scoring chunk per document contributes to the vector score.
|
|
With 5.37 chunks/doc average, 80%+ of chunk-level
|
|
semantic signal is discarded. For multi-session conversational data where different
|
|
parts of a session may be relevant, this is the wrong trade-off.
|
|
|
|
### 2. ~5x Text Advantage in Fusion
|
|
|
|
- Component weights: text=0.70, vector=0.25 (2.8x ratio)
|
|
- Field-aware score scaling: text=1.00, vector=0.45 (2.2x additional ratio)
|
|
- Vector-only penalty: 0.8x multiplier for semantic-only matches
|
|
- Combined: text contributions are ~5x stronger than vector in fusion scoring
|
|
|
|
### 3. Semantic Rescue Disabled
|
|
|
|
SCIENTIFIC profile sets `semanticRescueSlots=0`. High-confidence vector matches
|
|
that fall below the fusion cutoff are never rescued into the final top-K.
|
|
|
|
## Reference Baselines
|
|
|
|
The LongMemEval paper (Table 3) reports on LongMemEval_M (larger variant):
|
|
- Best retrieval config (K=V+fact): Recall@10=0.784, NDCG@10=0.536
|
|
- Retrievers evaluated: BM25, Contriever, Stella V5 1.5B, GTE-Qwen2 7B-instruct
|
|
- No published retrieval baselines for LongMemEval_S specifically
|
|
|
|
## First Tuning Attempt (2026-04-08)
|
|
|
|
Changes applied:
|
|
- Chunk aggregation: MAX → SUM
|
|
- SCIENTIFIC profile: textWeight 0.70→0.60, vectorWeight 0.25→0.35
|
|
- Vector scoreScale: 0.45 → 0.75
|
|
- Semantic rescue: 2 slots, min score 0.65
|
|
- Rerank top-K: 5 → 12
|
|
|
|
| Metric | Baseline | Attempt 1 | Delta |
|
|
|--------|----------|-----------|-------|
|
|
| MRR | 0.407 | 0.414 | +0.007 |
|
|
| Recall@10 | 0.653 | 0.485 | **-0.168** |
|
|
| NDCG@10 | 0.418 | 0.387 | -0.031 |
|
|
| MAP | 0.363 | 0.397 | +0.034 |
|
|
|
|
**Result: Recall@10 regressed significantly.** Root causes:
|
|
|
|
### 4. Similarity Threshold Filters Most Vector Results
|
|
|
|
`similarityThreshold=0.40` but embeddinggemma-300m produces cosine similarities
|
|
of 0.25-0.55 on this dataset. Of 479 HNSW searches:
|
|
- 54 (11.3%) returned **zero** results (all 150 candidates filtered)
|
|
- Many others heavily filtered (e.g. returned=1/150, 7/150, 17/150)
|
|
|
|
Raising vector weight without lowering the threshold just weakened text scoring.
|
|
|
|
### 5. CorpusStats-Driven Graph Reranking Activated
|
|
|
|
The `473274de` commit's CorpusStats-driven tuning detected high "symbol density"
|
|
(9.05) in chat sessions (proper nouns = brand names, people, places) and enabled
|
|
graph reranking that was OFF in baseline:
|
|
- `kg_weight`: 0.00 → 0.14 (consumed weight from text/vector)
|
|
- `graph_rerank_weight`: 0.00 → 0.32
|
|
- `graph_community_weight`: 0.00 → 0.16
|
|
|
|
Auto-tuner also reduced textWeight from 0.60→0.51 and vectorWeight from 0.35→0.30
|
|
due to weight redistribution to graph features.
|
|
|
|
### Key Env-Vars for A/B Testing
|
|
|
|
```bash
|
|
YAMS_SEARCH_SIMILARITY_THRESHOLD=0.25 # let vector results through
|
|
YAMS_SEARCH_ENABLE_GRAPH_RERANK=false # disable confounding graph rerank
|
|
YAMS_SEARCH_TEXT_WEIGHT=0.60 # override auto-tuner adjustment
|
|
YAMS_SEARCH_VECTOR_WEIGHT=0.35 # override auto-tuner adjustment
|
|
YAMS_SEARCH_CHUNK_AGGREGATION=sum # SUM chunk aggregation
|
|
```
|
|
|
|
## Second Tuning Attempt — Graph Pipeline Activation (2026-04-09)
|
|
|
|
Fixes: entity extraction enabled (`PostIngestQueue.cpp:1102`), SCIENTIFIC zoom
|
|
`Map` → `Auto` (`search_tuner.h`), 7 graph fields added to `toJson()`.
|
|
|
|
### Pre-Fix Benchmark (v3 — entity extraction off, zoom=Map)
|
|
|
|
| Metric | Baseline | v3 | Delta |
|
|
|--------|----------|----|-------|
|
|
| MRR | 0.407 | 0.507 | +0.100 |
|
|
| Recall@10 | 0.653 | 0.554 | **-0.099** |
|
|
| NDCG@10 | 0.418 | — | — |
|
|
| graph_rerank_apply_rate | 0.0 | 0.0 | — |
|
|
|
|
### Pending Validation
|
|
|
|
Both fixes applied but not yet benchmarked together.
|
|
|
|
## Improvement Targets
|
|
|
|
- Recall@10: 0.653 → 0.75+
|
|
- MRR: 0.407 → 0.55+
|