Write Basic Memory provider for Supermemory's memorybench

[bm-clawd]

Context

Dhravya Shah (Supermemory founder, 52K followers) is pushing memorybench as the standard benchmark for the memory/retrieval space. His tweet calling everyone liars got 16K impressions: https://x.com/dhravyashah/status/2032626482053525851

Current providers: Supermemory, Mem0, Zep. Basic Memory is not included.

We already have LoCoMo results from our own benchmarking (Recall@5: 76.4%, Recall@10: 85.5%, MRR: 0.658). Showing up on their benchmark with real numbers on their framework is the strongest credibility move we can make.

What to build

A Basic Memory provider adapter for memorybench. The interface is straightforward — see src/providers/README.md in their repo.

The provider needs to implement:

1. Ingest — Load benchmark conversations into BM (write_note or batch import)
2. Index — Wait for BM indexing (bm watch / reindex)
3. Search — Query BM via MCP tools (search_notes / memory_search)
4. Answer — Pass retrieved context to an LLM for answer generation

Why this matters

• We show up in their comparison table alongside Mem0, Zep, Supermemory
• On THEIR benchmark, using THEIR framework — no "grading your own homework"
• If our numbers are competitive, it is free marketing with built-in credibility
• If Dhravya's pitch is "everyone's lying, use our benchmark" then real numbers on his benchmark is the strongest counter
• MIT licensed, so we can fork/contribute freely
• Could submit as a PR to their repo — gets us visibility in their community

Approach

1. Fork memorybench
2. Write src/providers/basic-memory.ts adapter
3. Run LoCoMo benchmark locally, verify results
4. Submit PR to supermemoryai/memorybench
5. Share results publicly (blog post, Twitter thread)

References

• memorybench: https://github.com/supermemoryai/memorybench
• Our LoCoMo results: #608
• Our benchmark framework branch: claw/benchmark-framework
• Retrieval improvements (will affect scores): Retrieval pipeline improvements: reranking, length normalization, noise filtering, adaptive recall #666

[claude]

Claude finished @bm-clawd's task in 1m 0s —— View job

---

Issue Triage

• Analyze issue
• Add labels
• Post triage summary

---

Analysis

Type: Feature Request — external benchmark adapter

Summary: This issue requests building a TypeScript provider adapter for memorybench so Basic Memory appears in their comparison table alongside Mem0, Zep, and Supermemory. The motivation is marketing credibility — running on a third-party benchmark framework with verifiable results.

Classification:

Dimension	Assessment
Type	Feature / Enhancement
Priority	Medium — visibility/marketing value, not core functionality
Component	Testing / external benchmarking
Complexity	Medium

Labels added: enhancement

---

Implementation Notes

The work lives outside this repo (fork of supermemoryai/memorybench), but the adapter needs to interface with Basic Memory's MCP tools. Key implementation steps:

1. Fork supermemoryai/memorybench and create src/providers/basic-memory.ts
2. Ingest — use write_note or batch import to load LoCoMo conversations into BM
3. Index — trigger/wait for BM indexing (bm watch / reindex)
4. Search — query via search_notes MCP tool
5. Answer — pass retrieved context to LLM for answer generation
6. Run LoCoMo benchmark locally, capture results
7. Submit PR to supermemoryai/memorybench

Dependencies to note:

• Issue Retrieval pipeline improvements: reranking, length normalization, noise filtering, adaptive recall #666 (retrieval improvements) should land first — it will improve benchmark scores
• Issue #608 (existing LoCoMo results) provides the baseline to beat/validate against
• The memorybench provider interface is documented in their src/providers/README.md

Considerations:

• This is TypeScript work (memorybench is a TS project) but the actual BM interaction happens via MCP/HTTP — could use the @modelcontextprotocol/sdk client or direct HTTP calls to BM's API
• The adapter should handle BM's async indexing (may need polling after ingest before searching)
• Worth waiting for Retrieval pipeline improvements: reranking, length normalization, noise filtering, adaptive recall #666 to merge before running the final benchmark numbers to maximize scores