MemTether Benchmarks
All results from our own harness. We do not self-report cross-system comparisons —
all evaluation scripts are in the repo and can be independently reproduced. See the
reproduce guide (中文版可用).
LongMemEval (full 500Q, 2026-10-02)
| Metric | Score | Denominator | Notes |
|---|
| Strict match (global) | 62.6% | 209/334 applicable | |
| LLM judge (global) | 55.7% | 264/474 | |
| Multi-session strict | 48.8% | 39/80 | Hardest type |
| Multi-session LLM judge | 60.0% | 72/120 | |
| EAF (Evidence-Anchored) | 75.7% | 253/334 | multi-session 32.5%→60.0% |
Per-type breakdown
| Type | n | Strict | LLM Judge |
|---|
| knowledge-update | 78 | 76.6% | 62.5% |
| multi-session | 133 | 48.8% | 60.0% |
| single-session-assistant | 56 | 40.4% | 41.1% |
| single-session-user | 70 | 86.4% | 85.7% |
| temporal-reasoning | 133 | 60.7% | 40.5% |
Test suite (as of a61, 2026-10-08)
| Test | Result |
|---|
| pytest | 466 passed, 4 skipped |
| check_packaging | ✅ exit 0 |
| hub_selfcheck | ✅ rc=0 |
Coverage (a61)
| Module | Coverage |
|---|
| memory_court | 98% |
| predicates | 95% |
| consolidation | 67% |
| governance | 57% |
| memsearch | ~57% |
| gateway | 54% |
| tether_connect | 49% |
| TOTAL | ~55% |
Results use our own harness with disclosed failure modes. Not directly comparable
with other systems' self-reported numbers without matching denominators and judge models.