diff --git a/content/9.blog/.navigation.yml b/content/9.blog/.navigation.yml new file mode 100644 index 0000000..3dd779a --- /dev/null +++ b/content/9.blog/.navigation.yml @@ -0,0 +1,2 @@ +title: Blog +icon: i-lucide-pen-line diff --git a/content/9.blog/1.v0-19-0-release.md b/content/9.blog/1.v0-19-0-release.md new file mode 100644 index 0000000..ce0e32d --- /dev/null +++ b/content/9.blog/1.v0-19-0-release.md @@ -0,0 +1,99 @@ +--- +title: "Basic Memory v0.19.0: Search That Understands What You Mean" +description: "Semantic search, schema validation, per-project cloud routing, and more — the biggest Basic Memory release yet." +--- + +We just shipped v0.19.0 — the biggest release since we launched. Sixty-six commits, three months of work, and a fundamental change in how your AI finds what it knows. + +Here's what matters. + +--- + +## Semantic Search, Everywhere + +Until now, Basic Memory searched your notes the way `grep` does: match the words, return the results. It works. It's fast. And it completely fails when you search for "login security improvements" but your notes say "authentication hardening." + +v0.19.0 adds semantic search — hybrid by default, combining keyword matching with meaning-based retrieval. It ships enabled out of the box. No configuration. No separate embedding service. No API keys. Install, and it works. + +``` +search_notes("how do I speed up the app") +``` + +That query now finds your note about "performance optimization strategies" even though you never used the word "speed." The embedding model runs locally, your data never leaves your machine, and the results are ranked by a fusion of exact matches and semantic similarity. + +Three modes if you want control: +- **hybrid** (default) — best of both worlds +- **text** — keyword-only, like before +- **vector** — pure semantic similarity + +For most people: just search normally. It got smarter. + +## Schemas: Your AI Checks Its Own Work + +Knowledge bases grow organically. You write thirty person notes and realize half say `[name]`, a quarter say `[full_name]`, and a few just put the name in the title. Relation types drift — `works_at`, `employed_by`, `employer` — all meaning the same thing. + +The new schema system lets you define what "consistent" means, then validate against it. But here's the key design decision: **schemas are just notes.** + +```yaml +--- +title: Person +type: schema +entity: person +schema: + name: string, full name + role?: string, job title + works_at?: Organization, employer +--- +``` + +That's a regular markdown file in your knowledge base. Your AI reads it, uses it as a guide when creating new notes, and can validate existing notes against it. You can edit it in any text editor. + +The workflow is conversational. Ask your AI to analyze your existing notes and infer a schema. It finds the patterns, proposes a definition, and you approve it. Then it checks the outliers and offers to fix them. No migration scripts. No database alterations. Just a conversation about what "good" looks like for your notes. + +We built `schema_infer`, `schema_validate`, and `schema_diff` as MCP tools — so any AI that connects to Basic Memory can use them. Infer from what exists. Validate against the definition. Detect drift over time. + +## Per-Project Cloud Routing + +This one's for people running both local and cloud projects. You can now route individual projects through Basic Memory Cloud while keeping others purely local. + +```bash +bm project set-cloud research # this project syncs to cloud +bm project set-local private-notes # this one stays on your machine +``` + +The routing is transparent to your AI — it uses the same tools regardless of where the data lives. You decide what's sensitive and what's portable. + +Three levels of control: global default, per-project override, per-command flag. Mix and match however makes sense for your setup. + +## The Small Things That Matter + +**Write protection.** `write_note` no longer silently overwrites existing notes. If a note exists, you get an error instead of data loss. Pass `overwrite=True` when you mean it. This one change will save someone's knowledge base — AI assistants are enthusiastic writers. + +**Metadata search.** Filter notes by any frontmatter field — status, priority, tags, custom fields. No more searching for "status: in-progress" as a text string and hoping it matches. + +``` +search_notes(metadata_filters={"status": "active", "priority": {"$in": ["high", "critical"]}}) +``` + +**Project dashboard.** `bm project info` now shows an htop-inspired dashboard with bar charts for note types, embedding coverage, and health status. It's the fastest way to see what's in your knowledge base. + +**JSON everywhere.** Every MCP tool supports `output_format="json"` for scripting and automation. Five more CLI commands got `--json` flags. If you're building on top of Basic Memory, this makes your life easier. + +--- + +## The Upgrade + +```bash +uv tool upgrade basic-memory +bm reindex --embeddings +``` + +The reindex generates vector embeddings for semantic search — a one-time operation. Cloud projects handle this automatically. + +Everything else is backward compatible. Existing notes, permalinks, and workflows keep working. New defaults are saner, and the old behavior is one config flag away if you need it. + +[Full release notes →](/whats-new/v0.19.0) + +--- + +*Basic Memory is local-first AI knowledge infrastructure. Plain text files, semantic knowledge graph, works with Claude, Codex, Cursor, and anything that speaks MCP. [Get started →](https://basicmemory.com)* diff --git a/content/9.blog/2.openclaw-plugin.md b/content/9.blog/2.openclaw-plugin.md new file mode 100644 index 0000000..6bba296 --- /dev/null +++ b/content/9.blog/2.openclaw-plugin.md @@ -0,0 +1,78 @@ +--- +title: "Giving Your AI Agent a Memory That Actually Works" +description: "The openclaw-basic-memory plugin gives OpenClaw agents persistent, searchable memory — two commands, zero configuration." +--- + +Most AI assistants wake up every morning with amnesia. You explain your project. Again. You remind them about that decision you made last week. Again. You re-establish context that took three conversations to build. Again. + +We built a plugin that fixes this for OpenClaw agents. Two commands, zero configuration, and your agent remembers everything — across sessions, across days, across context window resets. + +```bash +openclaw plugins install @basicmemory/openclaw-basic-memory +openclaw gateway restart +``` + +That's the whole setup. + +--- + +## What Your Agent Gets + +**Composited memory search.** When your agent searches its memory, the plugin queries three sources in parallel: your MEMORY.md file, the full knowledge graph, and any active tasks. One search, three data sources, merged results. The agent doesn't need to know where something is stored — it just finds it. + +**Auto-capture.** Every conversation turn gets recorded as a timestamped entry in a daily note. You're not maintaining a journal — it happens in the background. Weeks later, when you need to know "what did we decide about the database migration?", it's there. + +**Auto-recall.** Session starts, the agent immediately loads active tasks and recent notes. No "let me catch you up" ritual. No "read my files first" prompt engineering. It just knows what it was working on. + +**14 tools, 7 slash commands, 6 skills.** The full Basic Memory toolset — search, read, write, edit, schema validation, project management. Plus shortcuts like `/remember` (save something), `/recall` (find something), and `/reflect` (review and consolidate). + +## Why This Isn't Just Another Memory Plugin + +Here's what's different: everything your agent remembers is stored as plain markdown files that you can open and read. + +Not embeddings in a vector database you can't inspect. Not JSON blobs in a cloud API you don't control. Markdown files. In a folder. On your machine. + +Your agent writes a note about a meeting? Open it in VS Code. See exactly what it captured. Fix the part it got wrong. Add the context it missed. Save the file — your agent sees the change instantly. + +This is what "bidirectional" means in practice. The AI writes things down. You read them. You correct them. The AI reads your corrections. The knowledge base gets better because both sides are working with the same transparent format. + +It's the difference between an AI that claims to remember and an AI whose memory you can verify. + +## Multi-Project Access + +Because the plugin connects to the full Basic Memory stack, your agent isn't limited to a single workspace. It can search across every project in your knowledge base — personal notes, work projects, research, whatever you've organized. + +Some projects stay local. Some route through Basic Memory Cloud for cross-device access. The agent uses the same tools either way. You decide what lives where. + +## The Skills + +The plugin bundles six pre-built skills that teach your agent best practices: + +- **memory-notes** — How to write well-structured notes with proper frontmatter, observations, and relations +- **memory-tasks** — Structured task tracking that survives when the context window resets +- **memory-schema** — Keeping note structures consistent as your knowledge base grows +- **memory-reflect** — Reviewing recent work and consolidating insights (inspired by how humans review journals) +- **memory-defrag** — Splitting bloated files, merging duplicates, pruning stale info +- **memory-metadata-search** — Finding notes by status, priority, or any custom field + +These aren't documentation. They're instructions your agent reads and follows — like onboarding a new team member by handing them a playbook. + +--- + +## Getting Started + +```bash +openclaw plugins install @basicmemory/openclaw-basic-memory +openclaw gateway restart +``` + +The plugin auto-detects your Basic Memory installation and starts working. If you don't have Basic Memory installed yet, it handles that too. + +From there, just talk to your agent. Say "remember that we decided to use Postgres for the new service." Say "what were we working on yesterday?" Say "create a task for the API refactor." The plugin handles the rest. + +[Full setup guide →](/integrations/openclaw) +[GitHub →](https://github.com/basicmachines-co/openclaw-basic-memory) + +--- + +*Basic Memory gives AI assistants persistent knowledge stored as plain text you own. Works with Claude, Codex, Cursor, OpenClaw, and anything that supports MCP. [Try it →](https://basicmemory.com)* diff --git a/content/9.blog/3.agent-skills.md b/content/9.blog/3.agent-skills.md new file mode 100644 index 0000000..741a5c2 --- /dev/null +++ b/content/9.blog/3.agent-skills.md @@ -0,0 +1,91 @@ +--- +title: "Teaching Your AI How to Remember" +description: "Pre-built skills that teach AI agents when and how to use Basic Memory effectively — not just tool access, but good habits." +--- + +You gave your AI a knowledge base. It has the tools — write notes, search, build context. But does it actually use them well? + +Probably not. At least, not consistently. + +An AI with access to Basic Memory's MCP tools is like a new hire with access to the company wiki. They *can* use it. But without guidance, they'll write notes in five different formats, forget to link related concepts, and never think to review what they wrote last week. The tools are there. The habits aren't. + +That's what skills fix. + +--- + +## Skills Are Playbooks, Not Documentation + +Basic Memory Skills are markdown instruction files that teach AI agents *when* and *how* to use their memory tools. Not API docs. Not reference pages. Playbooks — "here's the workflow, here's when to use it, here are the patterns that work." + +Install them with a single command: + +```bash +npx skills add basicmachines-co/basic-memory-skills +``` + +Your agent reads the skill file, learns the pattern, and follows it. Claude Desktop, Claude Code, Cursor, Windsurf — anything that supports markdown-based skills. + +## The Nine Skills + +**The foundation:** + +**memory-notes** is the one you install first. It teaches proper note structure — frontmatter, semantic observation categories, relations with wiki-links, memory:// URL usage. This is the difference between an agent that dumps text into a file and an agent that builds a connected knowledge graph. + +**memory-tasks** handles structured task tracking. AI context windows reset. Sessions end. Your agent loses its train of thought. Task notes externalize multi-step work so it survives compaction. The agent writes down where it is in a process, and picks up from that checkpoint next session. + +**memory-schema** manages the lifecycle of structured note types. Infer a schema from what you've already written. Validate notes against it. Detect when patterns drift over time. This is how consistency scales — not by enforcing rules upfront, but by observing patterns and codifying them. + +**Knowledge maintenance:** + +**memory-reflect** is inspired by what sleep researchers call "memory consolidation." Your agent reviews recent conversations, extracts the important parts, and writes them into long-term storage. Run it daily. It's like journaling — the act of reviewing forces the agent to decide what matters. + +**memory-defrag** is spring cleaning. Split a 500-line note into focused pieces. Merge three notes that are basically saying the same thing. Remove information that's no longer true. Run it weekly. Knowledge bases that never get reorganized become knowledge graveyards. + +**memory-lifecycle** manages status transitions — moving notes from active to archive, tracking where things are in a workflow. The philosophy is "archive, never delete." Information is preserved through moves, not removal. + +**Advanced workflows:** + +**memory-metadata-search** teaches your agent to query by structured fields instead of guessing at text matches. Find all notes with `status: active` and `priority: high`. Filter by date ranges. Search by tags. Frontmatter becomes queryable. + +**memory-ingest** handles external input — meeting transcripts, pasted documents, conversation logs. The agent parses unstructured text into properly formatted notes with observations, relations, and metadata. Bulk import without bulk cleanup. + +**memory-research** sends the agent to the web and brings findings back as structured knowledge. Research a company, a technology, a person — the agent searches, synthesizes, and writes a note you can actually use later. + +## Pick What You Need + +You don't need all nine. Start with what matches your workflow: + +**Developers:** `memory-notes` + `memory-tasks` + `memory-schema` — core note writing, task tracking across sessions, consistent project documentation. + +**Researchers:** `memory-notes` + `memory-research` + `memory-ingest` — capture web research and process external documents into structured knowledge. + +**Knowledge maintenance:** `memory-notes` + `memory-defrag` + `memory-lifecycle` + `memory-reflect` — keep your knowledge base healthy over time. + +**Everything:** All nine skills for comprehensive memory management. Install the full collection and let your agent use what's relevant. + +## Why This Matters + +The gap between "AI has access to tools" and "AI uses tools effectively" is larger than most people realize. An agent without skills will use `write_note` to create flat, unstructured text dumps. An agent *with* the memory-notes skill creates notes with proper observations, semantic categories, and wiki-link relations that connect to the rest of your knowledge graph. + +Same tool. Dramatically different output. The skill is the difference. + +This is also how Basic Memory stays useful as your knowledge base grows. A hundred notes are easy to manage. A thousand notes need structure, consistency, and maintenance. Skills teach your agent the habits that keep a large knowledge base navigable instead of overwhelming. + +--- + +Install a single skill: +```bash +npx skills add basicmachines-co/basic-memory-skills --skill memory-tasks +``` + +Install everything: +```bash +npx skills add basicmachines-co/basic-memory-skills +``` + +[Full skill guide →](/integrations/skills) +[GitHub →](https://github.com/basicmachines-co/basic-memory-skills) + +--- + +*Basic Memory is local-first AI knowledge infrastructure. Plain text files you can read, edit, and keep forever. [Get started →](https://basicmemory.com)* diff --git a/content/9.blog/4.benchmarks.md b/content/9.blog/4.benchmarks.md new file mode 100644 index 0000000..2bb5a9e --- /dev/null +++ b/content/9.blog/4.benchmarks.md @@ -0,0 +1,137 @@ +--- +title: "We Benchmark in the Open" +description: "How Basic Memory performs on academic retrieval benchmarks — and why we publish everything, including the parts that aren't flattering." +--- + +Most AI memory products don't publish benchmarks. The ones that do tend to cherry-pick metrics, use proprietary evaluation setups, or — in at least one case — [get caught inflating their numbers](https://github.com/getzep/zep-papers/issues/5). + +We decided to do it differently. We built a standalone, reproducible benchmark suite, ran it against an academic dataset, and published everything: the code, the results, the methodology, and the gaps. + +Here's what we found. + +--- + +## The Dataset: LoCoMo + +[LoCoMo](https://snap-research.github.io/locomo/) is an academic benchmark from Snap Research designed to test long-conversation memory systems. It's 10 multi-session conversations with 1,982 questions across five categories: + +- **Single-hop** — straightforward fact recall ("Where does Alice work?") +- **Multi-hop** — connecting facts across conversations ("Who works at the same company as the person who likes hiking?") +- **Temporal** — time-sensitive reasoning ("What was Bob's job before he switched in March?") +- **Open-domain** — broad knowledge retrieval +- **Adversarial** — questions designed to trip up memory systems + +It's the same benchmark used by Mem0, Zep, MemMachine, and others. Common ground for comparison. + +## Our Results + +Basic Memory v0.18.5, running entirely local on SQLite with local embeddings. No cloud APIs. No external services. + +| Metric | Score | +|--------|-------| +| **Recall@5** | **76.4%** | +| **Recall@10** | **85.5%** | +| **MRR** | **0.658** | +| **Mean Latency** | **1,063ms** | + +By category (Recall@5): + +| Category | Score | +|----------|-------| +| Open-domain | 86.6% | +| Multi-hop | 84.1% | +| Adversarial | 67.0% | +| Temporal | 59.1% | +| Single-hop | 57.7% | + +We're strong on open-domain and multi-hop retrieval. The knowledge graph structure helps here — connecting facts across conversations is literally what a graph does. We're weaker on single-hop and temporal queries, and we know why. + +## What's Working + +**Multi-hop retrieval (84.1%)** is our standout. Questions that require connecting information across multiple conversations play to our architecture's strength. When you write a note about Alice's job and another about Alice's hiking trip, Basic Memory's relation graph links them. A query about "Alice's hobbies and career" traverses that graph. + +**Open-domain (86.6%)** is strong because hybrid search — combining keyword matching with semantic similarity — handles broad queries well. The keyword side catches exact terms; the vector side catches related concepts. + +**Fully local execution** matters more than it sounds. Every query runs against a local SQLite database with local embeddings. No network calls, no API keys, no per-query costs. The 1,063ms mean latency is end-to-end on consumer hardware. + +## What's Not Working (Yet) + +**Single-hop (57.7%)** is our biggest gap. This is basic fact recall — the kind of thing that should be easy. The issue is architectural: we store full conversations and rely on chunk matching, while competitors like Mem0 extract atomic facts before storing. Their approach creates precise, granular memory units. Ours preserves full context but makes pinpoint retrieval harder. + +**Temporal (59.1%)** suffers because we don't yet distinguish between "when a note was created" and "when the event it describes happened." If you write today about a meeting that happened last Tuesday, Basic Memory timestamps today's date. Temporal queries need the event date. This is a solvable problem — we just haven't solved it yet. + +## The Comparison Problem + +Here's where we need to be honest about something the industry mostly isn't. + +There are two ways to evaluate memory systems: + +1. **Retrieval metrics** — Did the system find the right document? (Recall@K, MRR) +2. **LLM-as-Judge** — Given what was retrieved, did an LLM produce the correct answer? (Binary score from GPT-4o) + +We currently measure (1). Most published competitor numbers use (2). **These are not directly comparable.** + +A system with perfect retrieval but bad prompting scores high on (1) and low on (2). A system with mediocre retrieval but excellent answer generation could score higher on (2) than a system with better retrieval. They're measuring different things. + +Published LLM-as-Judge scores from competitors: + +| System | Overall Score | +|--------|--------------| +| MemMachine | 84.9% | +| Mem0 (graph) | 68.5% | +| Mem0 | 66.9% | +| Zep (corrected) | 58.4% | +| LangMem | 58.1% | +| OpenAI Memory | 52.9% | + +We can't put our number in that table yet because we haven't run the LLM-as-Judge step. We're adding it. When we do, the results go in the same public repo alongside everything else. + +We could have estimated where we'd land, or presented our retrieval metrics alongside their answer metrics and hoped nobody noticed the difference. We chose not to. + +## The Zep Incident + +Speaking of benchmark honesty: Zep originally published an 84% LoCoMo score. Mem0's CTO found that Zep had included adversarial category answers in the numerator while excluding adversarial questions from the denominator — inflating their number significantly. The corrected score is 58.4%. + +This is why we publish the code. Every query. Every result. The exact commands to reproduce the run. If our methodology is wrong, you can find it and tell us. + +## What We're Improving + +Based on these results, we're working on: + +- **Better observation extraction** — more atomic facts per conversation, closing the single-hop gap +- **Temporal indexing** — separating document dates from event dates +- **LLM-as-Judge evaluation** — so we can compare apples-to-apples with published numbers +- **Cloud benchmarks with OpenAI embeddings** — local embeddings are good; cloud embeddings should be better + +Each improvement gets benchmarked before and after, on the same dataset, with the same methodology. No "trust us, it's better now." Numbers or it didn't happen. + +## Reproduce It Yourself + +The entire benchmark suite is open source: + +```bash +git clone https://github.com/basicmachines-co/basic-memory-benchmarks +cd basic-memory-benchmarks +uv sync --group dev +uv run bm-bench datasets fetch --dataset locomo +uv run bm-bench convert locomo +uv run bm-bench run retrieval --providers bm-local +``` + +Every run produces a manifest with provenance metadata: git SHA, BM version, dataset checksums, provider configurations. Full reproducibility, not "we ran it once on a good day." + +[Benchmark repo →](https://github.com/basicmachines-co/basic-memory-benchmarks) + +--- + +## Why This Matters + +Benchmarks are how you know if a product actually works, or if it just has good marketing. The AI memory space is full of big claims and few receipts. + +We'd rather show you a 76.4% that you can verify than an 85% that you can't. And when we improve that number — and we will — you'll be able to see exactly what changed and why. + +That's the same philosophy behind the product itself. Plain text you can read. Results you can reproduce. No black boxes. + +--- + +*Basic Memory is local-first AI knowledge infrastructure. Benchmarked, open source, plain text. [Get started →](https://basicmemory.com)* diff --git a/content/9.blog/5.metadata-search.md b/content/9.blog/5.metadata-search.md new file mode 100644 index 0000000..0943e98 --- /dev/null +++ b/content/9.blog/5.metadata-search.md @@ -0,0 +1,151 @@ +--- +title: "Stop Grepping Your Frontmatter" +description: "Search notes by status, priority, tags, or any custom field — structured queries instead of text-string guessing." +--- + +You've been searching for notes like this: + +``` +search_notes("status: active") +``` + +And sometimes it works. And sometimes it returns notes that mention the word "active" in a completely different context. And sometimes it misses notes where the frontmatter says `status: in-progress` because you searched for "active" and those are different strings. + +This is text search doing a job that structured queries should be doing. + +--- + +## Frontmatter Is Data. Search It Like Data. + +Every Basic Memory note has YAML frontmatter — the metadata block at the top of the file: + +```yaml +--- +title: API Refactor Plan +type: task +status: active +priority: high +tags: [backend, q1-2026] +assigned: paul +--- +``` + +With v0.19.0, you can search these fields directly: + +``` +search_notes(metadata_filters={"status": "active"}) +``` + +That returns every note where `status` is literally `active` in the frontmatter. Not notes that happen to contain the word "active" somewhere in their body. Not fuzzy matches. Exact structured queries on structured data. + +## Building Up + +Start simple — find all active tasks: + +``` +search_notes(metadata_filters={"status": "active"}) +``` + +Filter by multiple fields — active tasks that are high priority: + +``` +search_notes(metadata_filters={ + "status": "active", + "priority": "high" +}) +``` + +Use operators — find anything high or critical: + +``` +search_notes(metadata_filters={ + "priority": {"$in": ["high", "critical"]} +}) +``` + +Combine with text search — active tasks about authentication: + +``` +search_notes( + query="authentication", + metadata_filters={"status": "active"} +) +``` + +Search by tags using the shorthand: + +``` +search_notes(tags=["backend", "q1-2026"]) +``` + +Or the tag syntax in the query itself: + +``` +search_notes("tag:backend AND tag:security") +``` + +## The Operators + +Beyond simple equality, you get range and set operations: + +``` +# Notes created after a date +search_notes(metadata_filters={ + "created": {"$gte": "2026-02-01"} +}) + +# Priority in a specific set +search_notes(metadata_filters={ + "priority": {"$in": ["high", "critical"]} +}) + +# Sprint number between 10 and 15 +search_notes(metadata_filters={ + "sprint": {"$between": [10, 15]} +}) + +# Notes with a specific tag (array contains) +search_notes(metadata_filters={ + "tags": "security" +}) +``` + +Available operators: `$in`, `$gt`, `$gte`, `$lt`, `$lte`, `$between`. They work on strings, numbers, and dates. + +## Why This Matters + +Metadata search turns your knowledge base into something closer to a database — without giving up the plain-text format that makes it readable and editable. + +Your notes are still markdown files. You can open them in any editor. But when your AI agent needs to find "all active high-priority tasks tagged backend," it doesn't have to guess at text patterns. It queries the frontmatter directly and gets precise results. + +This is especially powerful combined with the [schema system](/blog/schema-workflows). Define what fields a task note should have, validate that they're consistent, then query them with confidence. The schema ensures the data is there. Metadata search makes it findable. + +## The Practical Pattern + +Here's how this changes daily workflow: + +**Morning standup:** "What tasks are active and high priority?" +``` +search_notes(metadata_filters={"status": "active", "priority": "high"}, tags=["sprint-current"]) +``` + +**Weekly review:** "What did we complete this week?" +``` +search_notes(metadata_filters={"status": "done", "completed": {"$gte": "2026-02-24"}}) +``` + +**Project scoping:** "Show me all notes tagged for the API refactor." +``` +search_notes(tags=["api-refactor"]) +``` + +No custom tooling. No project management SaaS. Just frontmatter in markdown files, queried through MCP tools that any AI assistant can call. + +--- + +[Metadata search guide →](/concepts/semantic-search) +[Schema system →](/concepts/schema-system) + +--- + +*Basic Memory is local-first AI knowledge infrastructure. Plain text files, structured when you need it, searchable always. [Get started →](https://basicmemory.com)* diff --git a/content/9.blog/6.schema-workflows.md b/content/9.blog/6.schema-workflows.md new file mode 100644 index 0000000..ab790a0 --- /dev/null +++ b/content/9.blog/6.schema-workflows.md @@ -0,0 +1,182 @@ +--- +title: "Your Knowledge Base Grew. Here's How to Keep It Honest." +description: "Using Basic Memory's schema system to bring consistency to organic knowledge bases — without migration scripts or upfront planning." +--- + +You started with a few notes. Then a few dozen. Now you have hundreds, and you've noticed something: your AI writes "person" notes three different ways. + +Some have a `[name]` observation. Some use `[full_name]`. A few skip the name entirely and put it in the title. Relations are worse — `works_at`, `employed_by`, `employer`, `company` — four ways to say the same thing. + +This is normal. Knowledge bases grow organically. You don't sit down on day one and design a schema for every note type you'll ever write. You write what you need, when you need it, and structure emerges from use. + +The problem is that inconsistency compounds. Your AI can't reliably query "all people and their employers" if the relation type varies across notes. Metadata search returns incomplete results when some notes use `status: active` and others use `state: in-progress`. + +The schema system fixes this — not by forcing structure upfront, but by observing what you've already written and codifying the patterns. + +--- + +## Step 1: Discover What You Have + +Start by asking your AI to look at what exists: + +``` +"I've been writing a lot of person notes. Can you figure out +what they have in common?" +``` + +Behind the scenes, the AI runs `schema_infer` on your person notes. It analyzes observation categories, relation types, and frontmatter fields across every note with `type: person`. Then it proposes a schema: + +``` +Done! I analyzed 45 person notes. Here's what I found: + +Required (in 90%+ of notes): + - name: string — full name + - role: string — job title or function + +Optional (in 25-89%): + - works_at: Organization — employer + - expertise: string[] — areas of knowledge + - email: string — contact email + +Common relations: + - works_at → Organization (37 notes) + - collaborates_with → Person (12 notes) + +Want me to create this as a schema? +``` + +## Step 2: Create the Schema + +Say yes. The AI creates a schema note — which is just a regular markdown file: + +```yaml +--- +title: Person +type: schema +entity: person +version: 1 +schema: + name: string, full name + role?: string, job title + works_at?: Organization, employer + expertise?(array): string, areas of knowledge + email?: string, contact email +settings: + validation: warn +--- +``` + +This lives in your knowledge base alongside everything else. Open it in VS Code, Obsidian, any editor. It's a note about how notes should look. + +The `?` marks optional fields. `validation: warn` means the schema advises but doesn't block. Your AI reads this schema when creating new person notes and follows it as a guide. + +## Step 3: Validate What Exists + +Now check your existing notes against the schema: + +``` +"How do my existing person notes look against that schema?" +``` + +The AI runs `schema_validate`: + +``` +42 of 45 person notes pass validation. + +3 issues found: + - "John Smith" — missing [name] observation (name is in title only) + - "Dr. Sarah Chen" — uses [full_name] instead of [name] + - "Bob (contractor)" — missing [role] observation + +Want me to fix these? +``` + +No migration script. No database alteration. The AI reads each note, identifies what's missing or inconsistent, and offers to edit them. You approve or adjust. + +## Step 4: Detect Drift Over Time + +A month later, run `schema_diff`: + +``` +"Has my person note format drifted from the schema?" +``` + +``` +Drift detected in Person schema: + +New fields found in notes (not in schema): + - linkedin_url (appeared in 8 notes) + - department (appeared in 15 notes) + +Declared fields with low usage: + - email (only 12% of notes — consider making optional or removing) + +Relation type variations: + - "employed_by" used 3 times (schema says "works_at") +``` + +Now you decide: add `department` to the schema because it's useful? Normalize `employed_by` back to `works_at`? Drop `email` because nobody uses it? The schema evolves with your knowledge base, not against it. + +--- + +## The Full Workflow Loop + +This is what schema-managed knowledge looks like in practice: + +1. **Write freely** — don't worry about structure upfront +2. **Infer** — let the AI discover patterns in what you've written +3. **Codify** — create a schema from those patterns +4. **Validate** — check existing notes and fix outliers +5. **Create** — new notes follow the schema automatically +6. **Drift** — periodically check if reality has diverged from the definition +7. **Evolve** — update the schema to match how your knowledge base actually works + +It's the same cycle that good database teams follow — observe, define, validate, evolve — but applied to a plain-text knowledge base managed through conversation. + +## Why Schemas Are Just Notes + +This was a deliberate design decision. Schemas could have been configuration files, database tables, or API-only constructs. We made them notes because: + +- **You can read them.** Open the file, see exactly what "Person" means in your knowledge base. +- **You can edit them.** Change a field name in your editor, save, done. +- **Your AI can read them.** When creating a new person note, the AI checks the schema and follows it. +- **They're versioned.** Git tracks changes. You can see how your schemas evolved over time. +- **They're searchable.** `search_notes(metadata_filters={"type": "schema"})` finds all your schemas. + +Schemas aren't a separate system bolted onto your knowledge base. They're part of it. + +## Real-World Example: Task Management + +Schemas shine for workflow note types. Here's a task schema: + +```yaml +--- +title: Task +type: schema +entity: task +version: 1 +schema: + description: string, what needs to be done + status: string, current status (active/blocked/done) + priority?: string, urgency level + assigned?: string, who owns this + current_step?: string, where we are in the process + context?: string, accumulated working state + blocked_by?: string, what's preventing progress +settings: + validation: warn +--- +``` + +Every task note your AI creates follows this structure. When you search for active tasks, you know `status` exists and is consistent. When your AI resumes work after a context reset, it reads `current_step` and `context` to pick up where it left off. + +The schema didn't require upfront planning. You wrote a few task notes, noticed they had common fields, inferred a schema, and now every future task is consistent. + +--- + +[Schema system guide →](/concepts/schema-system) +[Metadata search →](/blog/metadata-search) + +--- + +*Basic Memory is local-first AI knowledge infrastructure. Structure when you need it, plain text always. [Get started →](https://basicmemory.com)* diff --git a/content/9.blog/7.agent-knowledge-architecture.md b/content/9.blog/7.agent-knowledge-architecture.md new file mode 100644 index 0000000..76e97f8 --- /dev/null +++ b/content/9.blog/7.agent-knowledge-architecture.md @@ -0,0 +1,139 @@ +--- +title: "How to Give Your AI Agent a Knowledge Base That Spans Local and Cloud" +description: "Using Basic Memory's local/cloud routing to build agents that write locally and read globally — the architecture behind persistent AI assistants." +--- + +Your AI agent runs on a machine. Maybe it's your laptop. Maybe it's a server. Maybe it's a Mac Mini under your desk that never sleeps. + +It has a workspace — local files, daily notes, task tracking, conversation logs. That workspace is its memory. It reads from it on startup, writes to it throughout the day, and searches it when you ask a question about something that happened last week. + +But your agent also needs knowledge that doesn't live on that machine. Project specs maintained by your team. Business strategy documents. Customer research. Technical documentation that lives in a shared cloud project. + +The question is: how do you give one agent access to all of it? + +--- + +## The Architecture: Write Local, Read Global + +Basic Memory's routing model lets you point individual projects at different backends — some local, some cloud — while the agent uses the same tools for everything. + +```bash +# Agent's own workspace stays local +bm project set-local my-agent-memory + +# Shared team knowledge routes through cloud +bm project set-cloud specs +bm project set-cloud company-docs +bm project set-cloud customer-research +``` + +From the agent's perspective, nothing changes. It calls `search_notes`, `read_note`, `build_context` — the same MCP tools regardless of where the data lives. The routing happens underneath, transparent to both the agent and its instructions. + +This is how we run our own agent. Claw has a local workspace with daily notes, heartbeat state, drafts, and personal memory files. But it also reads from cloud projects: technical specs, business strategy, customer interviews, product documentation. One agent, multiple knowledge sources, no special configuration per project. + +## Why Not Just Put Everything in the Cloud? + +You could. Basic Memory Cloud works fine as a single source of truth. But there are reasons to keep some things local: + +**Speed.** Local SQLite queries are sub-millisecond. Cloud queries add network latency. For an agent that searches memory dozens of times per session, the difference is noticeable. + +**Volume.** An agent's daily conversation logs, heartbeat state, and working files generate a lot of writes. Keeping those local avoids unnecessary cloud traffic and storage costs. + +**Independence.** If the cloud is down, your agent still has its own memory. Local projects work offline. The agent degrades gracefully — it loses access to shared knowledge but retains its own context. + +**Separation of concerns.** The agent's working memory is ephemeral and personal. Shared knowledge is curated and collaborative. They have different lifecycles. Mixing them in one project creates noise. + +## Why Not Just Keep Everything Local? + +Also a valid choice for single-machine setups. But it falls apart when: + +**You need the same knowledge on multiple devices.** A laptop agent and a server agent both need access to project specs. Cloud projects sync automatically. + +**Team members need to contribute.** Your team writes specs and documentation. Those notes should be searchable by the agent without manual file syncing. + +**You want cross-device continuity.** Start a conversation on your laptop, continue on your phone through the cloud MCP endpoint. The knowledge base is the same. + +## Setting It Up + +### For OpenClaw Agents + +The [openclaw-basic-memory plugin](/integrations/openclaw) handles this automatically. Your agent's workspace (`~/.openclaw/workspace/memory/`) is a local Basic Memory project. Cloud projects are accessible through the same tools. + +```bash +# Install the plugin +openclaw plugins install @basicmemory/openclaw-basic-memory +openclaw gateway restart + +# The agent's local project is configured automatically +# Add cloud projects for shared knowledge +bm cloud api-key save bmc_... +bm project set-cloud specs +bm project set-cloud company-docs +``` + +Your agent can now: +- Write daily notes, tasks, and memory to its local project +- Search across all projects (local and cloud) in a single query +- Read specific notes from any project by name +- List all available projects with `list_memory_projects` + +### For Claude Code / Cursor / Other MCP Clients + +The same routing works for any MCP client connected to Basic Memory's stdio server: + +```bash +# Configure routing +bm project set-local my-workspace +bm project set-cloud shared-specs + +# The MCP server respects routing for all tool calls +basic-memory mcp +``` + +Tool calls include an optional `project` parameter. When specified, the request routes to that project's backend. When omitted, it uses the default project. + +## The Three-Level Override + +Routing has three levels of specificity: + +1. **Global default** — `bm cloud login` sets cloud as default for everything +2. **Per-project** — `bm project set-cloud specs` or `bm project set-local private` overrides for specific projects +3. **Per-command** — `--local` or `--cloud` flags on individual CLI commands + +Most setups only need per-project routing. Set it once and forget it. + +## What This Looks Like in Practice + +Here's a real session with an agent that has both local and cloud projects: + +**Agent starts up:** +- Auto-recall loads active tasks from local project +- Reads today's daily note and yesterday's for context +- Ready to work + +**You ask:** "What does the search ranking spec say about reranking?" +- Agent calls `search_notes(query="reranking", project="specs")` +- Routes to cloud, searches the specs project +- Returns relevant sections from SPEC-28 + +**You ask:** "Add a task to investigate that for our implementation." +- Agent calls `write_note(title="Investigate reranking", folder="tasks", project="my-agent-memory")` +- Routes to local, writes to the agent's own workspace +- Task is now in the local knowledge graph, searchable alongside the agent's other work + +**You ask:** "What have we learned about reranking from customer interviews?" +- Agent calls `search_notes(query="reranking customer feedback", project="customer-research")` +- Routes to cloud, searches the customer research project +- Combines findings with the spec knowledge from earlier + +One conversation. Three projects. Two backends. The agent doesn't know or care about the routing. It just searches and gets answers. + +--- + +[Cloud routing guide →](/cloud/routing) +[OpenClaw plugin →](/integrations/openclaw) +[API keys →](/cloud/api-keys) + +--- + +*Basic Memory is local-first AI knowledge infrastructure. Your agent's memory on your machine. Your team's knowledge in the cloud. Same tools, transparent routing. [Get started →](https://basicmemory.com)*