Reproduce MemTether

100% reproducible Apache-2.0 Independent runs welcome

Don't trust the numbers. Run them yourself. Every claim on this page has a reproducible command behind it.

1. Environment

# Python 3.10 or 3.12
python --version

# Create clean venv
python -m venv .venv
source .venv/bin/activate  # Linux/macOS
# .venv\Scripts\activate   # Windows

# Install
pip install memtether

2. Smoke test

# Initialize + verify 23 adapters detected
memtether init

# Write a memory
memtether remember "Test content for reproduction" --source reproduce

# Search it back
memtether search "Test content" --list

# Expected: [fact/reproduce] Test content for reproduction

3. Run the test suite

git clone https://github.com/MemTether/MemTether.git
cd MemTether
pip install -e ".[server]"

# Run the full test suite
pytest tests/ -v --tb=short

# Expected: 464+ passed, 4 skipped (a63 口径; count grows each release)

4. Coverage

pip install pytest-cov
pytest tests/ --cov=. --cov-report=term

# Expected: TOTAL ~55% (a61+; 466 tests)
# Breakdown: memory_court 98% / predicates 95% / governance 57% / gateway 54% / tether_connect 49%

5. LongMemEval benchmark

MemTether reports strict-match 62.6% (209/334 applicable) on LongMemEval 500Q full run. EAF mode lifts multi-session strict from 32.5% to 60.0% (paired p=0.008, n=131).

Reproducibility status: the full EAF benchmark artifact bundle (per-question records, McNemar stats) is available for download from the v0.1.0a43 Release.. Until then, the numbers are declared, not independently reproduced.

Reference: LongMemEval paper · official repo

6. Claim-to-evidence mapping

Claim	Evidence	Status
23 adapters auto-connect	`memtether init` output + tether_connect.py 597 lines	✅ Reproducible
334 → 466 tests	`pytest tests/ -v`	✅ Reproducible
Coverage ~55%	`pytest --cov`	✅ Reproducible
WAL mode enabled	`sqlite3 memory.db "PRAGMA journal_mode"`	✅ Reproducible (a26 fix)
Tenant isolation	`tests/test_tenant.py` — cross-tenant 0 leak	✅ Reproducible
Rate limit 60/min	`tests/test_rate_limit.py`	✅ Reproducible
EAF +19.4pp (500Q)	`releases/eaf_artifact_bundle/` — 500 per-question records, McNemar p=7.84e-07	✅ Artifact bundle
strict 62.6% (500Q baseline)	`releases/eaf_artifact_bundle/eaf_500q_pq_final.json` — baseline strict per-question	✅ Same artifact bundle

7. File integrity

All releases publish wheel + tar.gz with SHA-256 digests on the GitHub Release page. Verify with:

sha256sum memtether-0.1.0a31-py3-none-any.whl

Compare against the digest in the release notes.

8. Independent reproduction

If you reproduce (or fail to reproduce) any claim, open an issue at MemTether issues with:

Your environment (OS / Python / versions)
Command you ran
Expected vs actual output
Raw artifact (JSON/text) if possible

Submissions are reviewed and attributed. Honest negative results are welcome.