feat: Concept Substrate — link repos by shared concepts (Pillar 3, Spec 2a)#38
Open
New1Direction wants to merge 17 commits into
Open
feat: Concept Substrate — link repos by shared concepts (Pillar 3, Spec 2a)#38New1Direction wants to merge 17 commits into
New1Direction wants to merge 17 commits into
Conversation
…time
compatStorageKeys() did not emit `${id}EmbedModel`, so the embeddings-model
override (read + tested in embeddingsModelFor/providerSupportsEmbeddings) could
never load via PROVIDER_KEYS — the default model always won. Emit the slot for
embeddings-capable providers + assert it in the test.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Foundation of the Knowledge Graph (Pillar 3, Spec 2). Persists the deep-dive's per-repo atoms (today thrown away after the blueprint) and links repos by shared concepts — lexical everywhere, embeddings where the provider supports them. This is the substrate the corkboard concept-graph (2b) and anchor-to-library explanations (2c) will consume.
Spec-first → plan → subagent-driven execution. Spec:
docs/superpowers/specs/2026-06-16-concept-substrate-design.md; plan:docs/superpowers/plans/2026-06-16-concept-substrate.md(both with rendered HTML).What's in it
concepts.js(new, pure, fully unit-tested) —normalizeConcept,cosineSimilarity,conceptIndex(lexical),lexicalMatcher,bestEmbeddingMatch, andderiveConceptLinkswith a per-pair hybrid: when both repos have vectors it links by embedding similarity, else falls back to lexical for that pair.conceptsIDB store (DB v6→v7, additive) + CRUD (getConcepts/getAllConcepts/setConcepts), mirroring themastery/decisionsstores.runDeepDivewrites the repo's atoms to the store on completion (best-effort; never fails the dive).providers.jsgains embeddings capability (providerSupportsEmbeddings,compatEmbeddingsEndpoint,embeddingsModelFor,pickEmbeddingsProvider, pureembeddingsBody/parseEmbeddings);background.jsgainscallEmbeddings(OpenAI-compatible/embeddings, BYO-key, same timeout ascallAI, errors →null→ lexical). On a deep dive, atom text is embedded when a capable provider is configured.Two deliberate departures (flagged in the spec)
background.jsand adds a provider/embeddingscall — the price of the hybrid matcher, and the first AI-layer change since the MCP work. Still no hosted backend: the call goes to the BYO provider, vectors cache in IDB, all cosine/matching is local JS.Final-review fix folded in
The final reviewer caught that the embeddings-model override was read + tested but
compatStorageKeys()never emitted its slot, so it could never load at runtime (the default always won). Fixed: the slot is now emitted for embeddings-capable providers, with a test asserting it.Test plan
npx vitest run→ 892 passed (96 files; +concepts.test.js, +store-concepts.test.js, +concepts-embeddings.test.js)npx eslint .→ 0 errorsnpm run check:html→ 26 files cleannode --checkonconcepts.js,store.js,store/idb.js,providers.js,background.jsgetConcepts(repoId).vectorsnon-null); with Anthropic,vectorsis null and linking is lexicalOut of scope (per spec → next)
The UI (2b — concept links on the corkboard), anchor-to-library explanations (2c), and embeddings for the 5 first-class providers incl. Google (v1 covers OpenAI-protocol compat providers; Google is a clean follow-up via its separate call path).