Skip to content

feat: Concept Substrate — link repos by shared concepts (Pillar 3, Spec 2a)#38

Open
New1Direction wants to merge 17 commits into
mainfrom
feat/concept-substrate
Open

feat: Concept Substrate — link repos by shared concepts (Pillar 3, Spec 2a)#38
New1Direction wants to merge 17 commits into
mainfrom
feat/concept-substrate

Conversation

@New1Direction

Copy link
Copy Markdown
Owner

Summary

Foundation of the Knowledge Graph (Pillar 3, Spec 2). Persists the deep-dive's per-repo atoms (today thrown away after the blueprint) and links repos by shared conceptslexical everywhere, embeddings where the provider supports them. This is the substrate the corkboard concept-graph (2b) and anchor-to-library explanations (2c) will consume.

Spec-first → plan → subagent-driven execution. Spec: docs/superpowers/specs/2026-06-16-concept-substrate-design.md; plan: docs/superpowers/plans/2026-06-16-concept-substrate.md (both with rendered HTML).

What's in it

  • concepts.js (new, pure, fully unit-tested) — normalizeConcept, cosineSimilarity, conceptIndex (lexical), lexicalMatcher, bestEmbeddingMatch, and deriveConceptLinks with a per-pair hybrid: when both repos have vectors it links by embedding similarity, else falls back to lexical for that pair.
  • concepts IDB store (DB v6→v7, additive) + CRUD (getConcepts / getAllConcepts / setConcepts), mirroring the mastery/decisions stores.
  • Atom persistencerunDeepDive writes the repo's atoms to the store on completion (best-effort; never fails the dive).
  • Embeddings path (provider-gated)providers.js gains embeddings capability (providerSupportsEmbeddings, compatEmbeddingsEndpoint, embeddingsModelFor, pickEmbeddingsProvider, pure embeddingsBody/parseEmbeddings); background.js gains callEmbeddings (OpenAI-compatible /embeddings, BYO-key, same timeout as callAI, errors → null → lexical). On a deep dive, atom text is embedded when a capable provider is configured.

Two deliberate departures (flagged in the spec)

  1. This touches background.js and adds a provider /embeddings call — the price of the hybrid matcher, and the first AI-layer change since the MCP work. Still no hosted backend: the call goes to the BYO provider, vectors cache in IDB, all cosine/matching is local JS.
  2. No user-visible change in 2a — it's the data substrate. The concept links appear on the corkboard in 2b.

Final-review fix folded in

The final reviewer caught that the embeddings-model override was read + tested but compatStorageKeys() never emitted its slot, so it could never load at runtime (the default always won). Fixed: the slot is now emitted for embeddings-capable providers, with a test asserting it.

Test plan

  • npx vitest run892 passed (96 files; +concepts.test.js, +store-concepts.test.js, +concepts-embeddings.test.js)
  • npx eslint . → 0 errors
  • npm run check:html → 26 files clean
  • node --check on concepts.js, store.js, store/idb.js, providers.js, background.js
  • Final whole-branch review: readyToMerge, verified the per-pair hybrid, provider-gating + null→lexical fallback, no backend, no UI, additive IDB migration, unchanged scan/lens contracts
  • Manual smoke (no DOM/SW test env): with an OpenAI key, a deep dive caches vectors (getConcepts(repoId).vectors non-null); with Anthropic, vectors is null and linking is lexical

Out of scope (per spec → next)

The UI (2b — concept links on the corkboard), anchor-to-library explanations (2c), and embeddings for the 5 first-class providers incl. Google (v1 covers OpenAI-protocol compat providers; Google is a clean follow-up via its separate call path).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant