diff --git a/CLAUDE.md b/CLAUDE.md index 47268b6..edc6b38 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,6 +1,6 @@ # markdocs -Go CLI that compiles library documentation into Claude Code skill files (`.claude/skills//.md`). +Go CLI that helps you draft and create custom Claude Code skill files from existing documentation (`.claude/skills//.md`). ## Commands @@ -17,11 +17,12 @@ go run main.go # run locally - **No SQLite / no CGO** — metadata lives entirely in each skill file's YAML frontmatter (`SkillMeta`), not a database - **Single binary** — no runtime requirements for end users; distributed via npm and goreleaser - **Skills are plain markdown** — written to `.claude/skills//.md`; Claude Code picks them up automatically next session +- **Draft-review before write** — after LLM compilation, users review and optionally edit the skill in `$EDITOR` before it's written to disk; `--no-interactive` skips this ## Directory Structure ``` -cmd/ → cobra commands (root, add, scan, update, list, init, pipeline) +cmd/ → cobra commands (root, add, draft, scan, update, list, init, pipeline) internal/ config/ → read/write ~/.markdocs/config.json llm/ → LLMProvider interface + Anthropic / OpenAI / Ollama backends @@ -75,9 +76,26 @@ Followed by LLM-compiled markdown with sections: What This Is, Installation, Key 1. Create `cmd/.go` with a `cobra.Command` 2. Register it in `cmd/root.go` `init()` +See `cmd/draft.go` as a recent example. + ## Shared Pipeline -`cmd/pipeline.go:runAddPipeline` contains the reusable scrape → compile → write flow used by both `add` and `scan --add-all`. Keep command-specific logic in the command files; pipeline logic here. +`cmd/pipeline.go:runAddPipeline` contains the reusable scrape → compile → [review] → write flow used by `add`, `scan --add-all`, and `update`. + +Signature: `runAddPipeline(ctx, library, url, provider, cwd, noInteractive bool) (*PipelineResult, error)` + +- `noInteractive = false` (default for `add`): shows the draft review prompt (`ui.ReviewDraft`) after compile, before write. Returns `nil, nil` if the user discards. +- `noInteractive = true` (used by `scan --add-all` and `update`): skips review, writes immediately. + +### Draft Review (`ui.ReviewDraft`) + +Defined in `internal/ui/prompts.go`. Shows a bordered preview of the first 50 lines of the compiled skill, then prompts: Accept / Edit in $EDITOR / Discard. + +- **Edit**: writes content to a temp `.md` file, opens `$EDITOR` (fallback `nano`), re-reads on exit +- **Accept**: passes markdown through unchanged +- **Discard**: returns `ReviewDiscard`; caller returns `nil, nil` to signal cancellation + +`ReviewAction` constants: `ReviewAccept`, `ReviewEdit`, `ReviewDiscard` (in `internal/ui/prompts.go`). ## Code Style diff --git a/README.md b/README.md index 5f59c93..34eaa81 100644 --- a/README.md +++ b/README.md @@ -2,17 +2,25 @@ markdocs -CLI to compile library documentation into Claude Code skill files +[![Build](https://github.com/Nithin-Valiyaveedu/markdocs/actions/workflows/build.yml/badge.svg)](https://github.com/Nithin-Valiyaveedu/markdocs/actions/workflows/build.yml) +[![Test](https://github.com/Nithin-Valiyaveedu/markdocs/actions/workflows/test.yml/badge.svg)](https://github.com/Nithin-Valiyaveedu/markdocs/actions/workflows/test.yml) +[![npm](https://img.shields.io/npm/v/@val-nithin/markdocs)](https://www.npmjs.com/package/@val-nithin/markdocs) +[![Downloads](https://img.shields.io/npm/dm/@val-nithin/markdocs)](https://www.npmjs.com/package/@val-nithin/markdocs) +[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) +[![Go Report Card](https://goreportcard.com/badge/github.com/Nithin-Valiyaveedu/markdocs)](https://goreportcard.com/report/github.com/Nithin-Valiyaveedu/markdocs) + +Draft and create custom Claude Code skill files from existing documentation — with LLM assistance and a review step before anything is written to disk. ## How It Works -`markdocs add react` does five things: +`markdocs add react` does six things: 1. Asks the configured LLM for official documentation URLs 2. Lets you pick which URL(s) to scrape (interactive) 3. Scrapes the page with a built-in Go scraper (no external APIs) -4. Compiles the content into a structured skill file using the LLM -5. Writes `.claude/skills/frontend/react.md` — Claude Code picks it up next session +4. Compiles the content into a structured skill draft using the LLM +5. Shows you the draft in the terminal — edit it in `$EDITOR` if you want +6. Writes `.claude/skills/frontend/react.md` — Claude Code picks it up next session ## Installation @@ -41,15 +49,18 @@ brew install markdocs # 1. Configure your LLM provider (runs once) markdocs init -# 2. Add a skill for any library +# 2. Add a skill for any library (shows draft review before writing) markdocs add shadcn markdocs add stripe markdocs add drizzle-orm -# 3. Scan your project for missing skills +# 3. Create a skill from your own notes or docs +markdocs draft internal-api + +# 4. Scan your project for missing skills markdocs scan -# 4. Keep skills up to date +# 5. Keep skills up to date markdocs update --all ``` @@ -71,7 +82,7 @@ Config is saved to `~/.markdocs/config.json`. ### `markdocs add ` -Find, scrape, compile, and write a skill file for the given library. +Find, scrape, compile, and write a skill file for the given library. After compilation, a draft review prompt lets you inspect and optionally edit the skill before it's written to disk. ```bash markdocs add react @@ -83,12 +94,28 @@ markdocs add stripe --no-interactive | Flag | Description | |------|-------------| -| `--no-interactive` | Skip URL selection prompt, use first suggested URL | +| `--no-interactive` | Skip URL selection and draft review prompts; write immediately | The skill is written to `.claude/skills//.md` in the current directory. --- +### `markdocs draft ` + +Compile your own notes, internal docs, or any text content into a skill file. Useful for private APIs, internal tools, or any library not well-covered by public documentation. + +```bash +# Interactive — paste content in the terminal +markdocs draft internal-api + +# Pipe content from a file or command +cat docs/api.md | markdocs draft internal-api +``` + +After compilation, the same draft review prompt is shown — accept as-is, edit in `$EDITOR`, or discard. + +--- + ### `markdocs scan` Detect libraries in the current project that don't have skill files yet. Reads `package.json`, `go.mod`, and `requirements.txt`. diff --git a/cmd/add.go b/cmd/add.go index edabc90..7f74453 100644 --- a/cmd/add.go +++ b/cmd/add.go @@ -5,8 +5,6 @@ import ( "os" "strings" - "github.com/Nithin-Valiyaveedu/markdocs/internal/llm" - "github.com/Nithin-Valiyaveedu/markdocs/internal/skill" "github.com/Nithin-Valiyaveedu/markdocs/internal/ui" "github.com/spf13/cobra" ) @@ -16,32 +14,30 @@ var addNoInteractive bool var addCmd = &cobra.Command{ Use: "add ", Short: "Find, scrape, compile, and write a skill for a library.", - Long: `Discovers documentation URLs for the given library using the configured LLM, -presents them for interactive selection, scrapes the pages, compiles the content -into a structured skill file, and writes it to .claude/skills//.md.`, + Long: `Discovers documentation URLs for the given library, scrapes the pages, +compiles the content into a structured skill file, and writes it to +.claude/skills//.md. + +By default, structured extraction is used (no API key required). +Pass --llm to use an LLM provider for richer content (requires 'markdocs init').`, Args: cobra.MinimumNArgs(1), RunE: runAdd, } func init() { - addCmd.Flags().BoolVar(&addNoInteractive, "no-interactive", false, "Skip URL selection prompt, use first suggested URL") + addCmd.Flags().BoolVar(&addNoInteractive, "no-interactive", false, "Skip URL selection and draft review prompts") } func runAdd(cmd *cobra.Command, args []string) error { library := strings.Join(args, " ") ctx := cmd.Context() - provider, err := llm.NewProvider(appConfig) - if err != nil { - ui.Error(fmt.Sprintf("Failed to create provider: %s", err)) - os.Exit(2) - } + compiler, providerName, modelName := newCompiler() // Step 1: Discover URLs var urls []string - ui.Step(1, fmt.Sprintf("Asking LLM for %s documentation URLs...", library)) - err = ui.Spin("", func() error { - compiler := skill.NewLLMCompiler(provider) + ui.Step(1, fmt.Sprintf("Discovering %s documentation URLs...", library)) + err := ui.Spin("", func() error { var err error urls, err = compiler.SuggestURLs(ctx, library) return err @@ -70,11 +66,14 @@ func runAdd(cmd *cobra.Command, args []string) error { return fmt.Errorf("getting working directory: %w", err) } - result, err := runAddPipeline(ctx, library, selectedURL, provider, cwd) + result, err := runAddPipeline(ctx, library, selectedURL, compiler, providerName, modelName, cwd, addNoInteractive) if err != nil { ui.Error(fmt.Sprintf("Pipeline failed: %s", err)) os.Exit(2) } + if result == nil { + return nil // user discarded the draft + } ui.Blank() ui.Success(fmt.Sprintf("Skill ready: %s", result.SkillPath)) diff --git a/cmd/draft.go b/cmd/draft.go new file mode 100644 index 0000000..840eaf9 --- /dev/null +++ b/cmd/draft.go @@ -0,0 +1,132 @@ +package cmd + +import ( + "fmt" + "io" + "os" + "strings" + + "github.com/Nithin-Valiyaveedu/markdocs/internal/skill" + "github.com/Nithin-Valiyaveedu/markdocs/internal/ui" + "github.com/charmbracelet/huh" + "github.com/spf13/cobra" +) + +var draftCmd = &cobra.Command{ + Use: "draft ", + Short: "Compile your own notes or docs into a skill file.", + Long: `Prompts you to paste documentation, notes, or any text content. +The LLM compiles it into a structured skill file, shows a draft preview, +and lets you edit before writing to .claude/skills//.md. + +You can also pipe content directly: + + cat docs.md | markdocs draft mylib`, + Args: cobra.ExactArgs(1), + RunE: runDraft, +} + +func runDraft(cmd *cobra.Command, args []string) error { + library := args[0] + ctx := cmd.Context() + + compiler, providerName, modelName := newCompiler() + + // Step 1: Get content from stdin (pipe) or interactive textarea + var content string + if isStdinPipe() { + raw, err := io.ReadAll(os.Stdin) + if err != nil { + return fmt.Errorf("reading stdin: %w", err) + } + content = string(raw) + ui.Info(fmt.Sprintf("Read %d chars from stdin", len(content))) + } else { + ui.Info("Paste your documentation, notes, or any relevant content below.") + ui.Info("Press Ctrl+D or Esc to finish.") + ui.Blank() + err := huh.NewText(). + Title(fmt.Sprintf("Content for %q skill", library)). + Placeholder("Paste documentation, API references, examples..."). + Value(&content). + Run() + if err != nil { + return fmt.Errorf("content input: %w", err) + } + } + + if len(strings.TrimSpace(content)) < 50 { + ui.Warning("Content is too short — add more detail and try again.") + return nil + } + + // Step 2: Compile + cwd, err := os.Getwd() + if err != nil { + return fmt.Errorf("getting working directory: %w", err) + } + + var compiled *skill.CompileOutput + err = ui.Spin("Compiling skill...", func() error { + framework := skill.DetectFramework(cwd) + var err error + compiled, err = compiler.Compile(ctx, skill.CompileInput{ + Library: library, + URL: "user-provided notes", + ScrapedMarkdown: content, + ProjectFramework: framework, + Model: modelName, + }) + return err + }) + if err != nil { + return fmt.Errorf("compiling skill for %s: %w", library, err) + } + ui.StepDone(1, fmt.Sprintf("Compiled skill (category: %s)", compiled.Category)) + + // Step 3: Review draft + action, finalMarkdown, err := ui.ReviewDraft(compiled.Markdown) + if err != nil { + return fmt.Errorf("draft review: %w", err) + } + if action == ui.ReviewDiscard { + ui.Warning("Discarded — no skill written.") + return nil + } + if action == ui.ReviewEdit { + compiled.Markdown = finalMarkdown + } + + // Step 4: Write + framework := skill.DetectFramework(cwd) + meta := skill.NewSkillMeta( + library, + providerName, + modelName, + compiled.Category, + framework, + compiled.Description, + compiled.WhenToUse, + []string{"user-provided notes"}, + content, + ) + writer := skill.NewFSWriter(cwd) + path, err := writer.Write(library, compiled.Category, compiled.Markdown, meta) + if err != nil { + return fmt.Errorf("writing skill for %s: %w", library, err) + } + + ui.Blank() + ui.Success(fmt.Sprintf("Skill ready: %s", path)) + ui.Info("Claude Code will pick it up automatically next session.") + return nil +} + +// isStdinPipe reports whether stdin is a pipe (non-interactive). +func isStdinPipe() bool { + fi, err := os.Stdin.Stat() + if err != nil { + return false + } + return fi.Mode()&os.ModeCharDevice == 0 +} diff --git a/cmd/pipeline.go b/cmd/pipeline.go index 47271ef..aee47a5 100644 --- a/cmd/pipeline.go +++ b/cmd/pipeline.go @@ -24,15 +24,33 @@ func providerFromConfig() (llm.LLMProvider, error) { return provider, nil } +// newCompiler returns the appropriate Compiler based on the --llm flag. +// In LLM mode, it requires appConfig to be loaded. +// providerName and modelName are set to identify the backend in skill frontmatter. +func newCompiler() (compiler skill.Compiler, providerName, modelName string) { + if useLLM { + provider, err := providerFromConfig() + if err != nil { + ui.Error(err.Error()) + os.Exit(2) + } + return skill.NewLLMCompiler(provider), string(appConfig.Provider), provider.Model() + } + return skill.NewStructuredExtractor(), "structured", "" +} + // PipelineResult holds the output of a successful runAddPipeline call. type PipelineResult struct { SkillPath string Category string } -// runAddPipeline executes the full add pipeline: scrape → compile → write. -// It is called by both `add` and `scan --add-all`. -func runAddPipeline(ctx context.Context, library, url string, provider llm.LLMProvider, cwd string) (*PipelineResult, error) { +// runAddPipeline executes the full add pipeline: scrape → compile → [review] → write. +// It is called by `add`, `scan --add-all`, and `update`. +// providerName and modelName are stored in the skill file frontmatter for traceability. +// When noInteractive is false, a draft review prompt is shown before writing. +// Returns nil, nil if the user discards the draft. +func runAddPipeline(ctx context.Context, library, url string, compiler skill.Compiler, providerName, modelName string, cwd string, noInteractive bool) (*PipelineResult, error) { // Step 1: Scrape var scraped string err := ui.Spin(fmt.Sprintf("Scraping %s...", url), func() error { @@ -50,14 +68,13 @@ func runAddPipeline(ctx context.Context, library, url string, provider llm.LLMPr var compiled *skill.CompileOutput err = ui.Spin("Compiling skill...", func() error { framework := skill.DetectFramework(cwd) - compiler := skill.NewLLMCompiler(provider) var err error compiled, err = compiler.Compile(ctx, skill.CompileInput{ Library: library, URL: url, ScrapedMarkdown: scraped, ProjectFramework: framework, - Model: provider.Model(), + Model: modelName, }) return err }) @@ -66,12 +83,27 @@ func runAddPipeline(ctx context.Context, library, url string, provider llm.LLMPr } ui.StepDone(2, fmt.Sprintf("Compiled skill (category: %s)", compiled.Category)) - // Step 3: Write — metadata lives entirely in the skill file's frontmatter + // Step 3: Review draft (skipped in non-interactive mode) + if !noInteractive { + action, editedMarkdown, err := ui.ReviewDraft(compiled.Markdown) + if err != nil { + return nil, fmt.Errorf("draft review: %w", err) + } + if action == ui.ReviewDiscard { + ui.Warning("Discarded — no skill written.") + return nil, nil + } + if action == ui.ReviewEdit { + compiled.Markdown = editedMarkdown + } + } + + // Step 4: Write — metadata lives entirely in the skill file's frontmatter framework := skill.DetectFramework(cwd) meta := skill.NewSkillMeta( library, - string(appConfig.Provider), - provider.Model(), + providerName, + modelName, compiled.Category, framework, compiled.Description, diff --git a/cmd/root.go b/cmd/root.go index decefe9..c43e451 100644 --- a/cmd/root.go +++ b/cmd/root.go @@ -22,28 +22,38 @@ func resolveVersion() string { } // appConfig holds the loaded configuration, available to all subcommands. +// It is nil when running without --llm (structured extraction mode). var appConfig *config.Config +// useLLM controls whether commands use LLM compilation (requires a configured provider). +// When false (default), structured heading-based extraction is used and no API key is needed. +var useLLM bool + var rootCmd = &cobra.Command{ Use: "markdocs", Short: "Compile library documentation into Claude Code skill files.", Long: `markdocs scrapes library documentation and compiles it into structured Claude Code skill files (.claude/skills//.md). -Configure a provider with 'markdocs init', then add skills with 'markdocs add '.`, +By default, markdocs uses structured extraction — no API key required. +Pass --llm to use an LLM provider instead (configure with 'markdocs init').`, Run: func(cmd *cobra.Command, args []string) { _ = cmd.Help() }, PersistentPreRunE: func(cmd *cobra.Command, args []string) error { ui.PrintBanner(Version) ui.Blank() - // init command creates the config, so skip loading it + // init creates the config, so skip loading it if cmd.Name() == "init" { return nil } + // Config is only required when --llm is explicitly set + if !useLLM { + return nil + } cfg, err := config.Load() if err != nil { - return fmt.Errorf("run 'markdocs init' first to configure a provider: %w", err) + return fmt.Errorf("--llm requires a configured provider — run 'markdocs init' first: %w", err) } appConfig = cfg return nil @@ -59,8 +69,10 @@ func Execute() { } func init() { + rootCmd.PersistentFlags().BoolVar(&useLLM, "llm", false, "Use LLM compilation instead of structured extraction (requires 'markdocs init')") rootCmd.AddCommand(initCmd) rootCmd.AddCommand(addCmd) + rootCmd.AddCommand(draftCmd) rootCmd.AddCommand(scanCmd) rootCmd.AddCommand(updateCmd) rootCmd.AddCommand(listCmd) diff --git a/cmd/scan.go b/cmd/scan.go index caba47c..d2de8ad 100644 --- a/cmd/scan.go +++ b/cmd/scan.go @@ -79,13 +79,7 @@ func runScan(cmd *cobra.Command, args []string) error { } // --add-all: run pipeline for each missing library - if appConfig == nil { - return fmt.Errorf("no provider configured — run 'markdocs init' first") - } - provider, err := providerFromConfig() - if err != nil { - return err - } + compiler, providerName, modelName := newCompiler() ui.Blank() ui.Section(fmt.Sprintf("Adding %d missing skills", len(missing))) @@ -95,8 +89,6 @@ func runScan(cmd *cobra.Command, args []string) error { ui.Blank() ui.Section(lib.name) - // Use LLM to discover URL, pick first one automatically - compiler := skill.NewLLMCompiler(provider) urls, err := compiler.SuggestURLs(ctx, lib.name) if err != nil { ui.Error(fmt.Sprintf("skipping %s: %s", lib.name, err)) @@ -107,7 +99,7 @@ func runScan(cmd *cobra.Command, args []string) error { continue } - _, err = runAddPipeline(ctx, lib.name, urls[0], provider, cwd) + _, err = runAddPipeline(ctx, lib.name, urls[0], compiler, providerName, modelName, cwd, true) if err != nil { ui.Error(fmt.Sprintf("failed: %s", err)) continue diff --git a/cmd/update.go b/cmd/update.go index e927a2b..bac51bf 100644 --- a/cmd/update.go +++ b/cmd/update.go @@ -64,10 +64,7 @@ func runUpdate(cmd *cobra.Command, args []string) error { } } - provider, err := providerFromConfig() - if err != nil { - return err - } + compiler, providerName, modelName := newCompiler() sc := scraper.NewWaterfall() updated := 0 @@ -109,7 +106,7 @@ func runUpdate(cmd *cobra.Command, args []string) error { } ui.Recompiling("content changed — recompiling...") - _, err = runAddPipeline(ctx, name, primaryURL, provider, cwd) + _, err = runAddPipeline(ctx, name, primaryURL, compiler, providerName, modelName, cwd, true) if err != nil { ui.Error(fmt.Sprintf("recompile failed: %s", err)) skipped++ diff --git a/internal/resolver/crates.go b/internal/resolver/crates.go new file mode 100644 index 0000000..b6d32ab --- /dev/null +++ b/internal/resolver/crates.go @@ -0,0 +1,68 @@ +package resolver + +import ( + "context" + "encoding/json" + "fmt" + "net/http" +) + +type cratesResponse struct { + Crate struct { + Documentation string `json:"documentation"` + Homepage string `json:"homepage"` + Repository string `json:"repository"` + } `json:"crate"` +} + +// resolveCrates queries crates.io for the library's documentation URL. +func (r *Resolver) resolveCrates(ctx context.Context, library string) []string { + apiURL := fmt.Sprintf("%s/api/v1/crates/%s", r.CratesBaseURL, library) + + req, err := newRequest(ctx, apiURL) + if err != nil { + return nil + } + + resp, err := r.HTTPClient.Do(req) + if err != nil || resp.StatusCode != 200 { + if resp != nil { + resp.Body.Close() + } + return nil + } + defer resp.Body.Close() + + var pkg cratesResponse + if err := json.NewDecoder(resp.Body).Decode(&pkg); err != nil { + return nil + } + + var urls []string + if pkg.Crate.Documentation != "" { + urls = append(urls, pkg.Crate.Documentation) + } + if pkg.Crate.Homepage != "" { + urls = append(urls, pkg.Crate.Homepage) + } + if pkg.Crate.Repository != "" { + urls = append(urls, pkg.Crate.Repository) + } + return urls +} + +// resolveGo constructs the pkg.go.dev URL for Go modules directly from the module path. +func (r *Resolver) resolveGo(library string) []string { + return []string{fmt.Sprintf("https://pkg.go.dev/%s", library)} +} + +// newRequest creates an HTTP GET request with the markdocs User-Agent header. +func newRequest(ctx context.Context, url string) (*http.Request, error) { + req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil) + if err != nil { + return nil, err + } + req.Header.Set("User-Agent", userAgent) + req.Header.Set("Accept", "application/json") + return req, nil +} diff --git a/internal/resolver/npm.go b/internal/resolver/npm.go new file mode 100644 index 0000000..03171a8 --- /dev/null +++ b/internal/resolver/npm.go @@ -0,0 +1,71 @@ +package resolver + +import ( + "context" + "encoding/json" + "fmt" + "net/url" + "strings" +) + +type npmPackage struct { + Homepage string `json:"homepage"` + Repository struct { + URL string `json:"url"` + } `json:"repository"` + Bugs struct { + URL string `json:"url"` + } `json:"bugs"` +} + +// resolveNPM queries the npm registry for the library's homepage and repository URL. +func (r *Resolver) resolveNPM(ctx context.Context, library string) []string { + // Scoped packages like @scope/name need the slash URL-encoded + encodedName := strings.ReplaceAll(library, "/", "%2F") + apiURL := fmt.Sprintf("%s/%s", r.NPMBaseURL, encodedName) + + req, err := newRequest(ctx, apiURL) + if err != nil { + return nil + } + + resp, err := r.HTTPClient.Do(req) + if err != nil || resp.StatusCode != 200 { + if resp != nil { + resp.Body.Close() + } + return nil + } + defer resp.Body.Close() + + var pkg npmPackage + if err := json.NewDecoder(resp.Body).Decode(&pkg); err != nil { + return nil + } + + var urls []string + if pkg.Homepage != "" { + urls = append(urls, pkg.Homepage) + } + if repoURL := cleanGitURL(pkg.Repository.URL); repoURL != "" { + urls = append(urls, repoURL) + } + return urls +} + +// cleanGitURL converts a git remote URL to an HTTPS URL. +// e.g. "git+https://github.com/foo/bar.git" → "https://github.com/foo/bar" +func cleanGitURL(raw string) string { + raw = strings.TrimPrefix(raw, "git+") + raw = strings.TrimSuffix(raw, ".git") + raw = strings.TrimPrefix(raw, "git://") + if strings.HasPrefix(raw, "github.com/") { + raw = "https://" + raw + } + // Validate it's an HTTP URL + u, err := url.Parse(raw) + if err != nil || (u.Scheme != "https" && u.Scheme != "http") { + return "" + } + return raw +} diff --git a/internal/resolver/pypi.go b/internal/resolver/pypi.go new file mode 100644 index 0000000..956ff0a --- /dev/null +++ b/internal/resolver/pypi.go @@ -0,0 +1,57 @@ +package resolver + +import ( + "context" + "encoding/json" + "fmt" +) + +type pypiResponse struct { + Info struct { + HomePage string `json:"home_page"` + ProjectURLs map[string]string `json:"project_urls"` + } `json:"info"` +} + +// docURLKeys are the project_urls keys we consider official documentation, in priority order. +var docURLKeys = []string{ + "Documentation", "Docs", "documentation", "docs", + "Homepage", "Home", "homepage", +} + +// resolveP yPI queries PyPI for the library's documentation URL. +func (r *Resolver) resolvePyPI(ctx context.Context, library string) []string { + apiURL := fmt.Sprintf("%s/pypi/%s/json", r.PyPIBaseURL, library) + + req, err := newRequest(ctx, apiURL) + if err != nil { + return nil + } + + resp, err := r.HTTPClient.Do(req) + if err != nil || resp.StatusCode != 200 { + if resp != nil { + resp.Body.Close() + } + return nil + } + defer resp.Body.Close() + + var pkg pypiResponse + if err := json.NewDecoder(resp.Body).Decode(&pkg); err != nil { + return nil + } + + var urls []string + // Check project_urls in priority order + for _, key := range docURLKeys { + if u := pkg.Info.ProjectURLs[key]; u != "" { + urls = append(urls, u) + break + } + } + if pkg.Info.HomePage != "" { + urls = append(urls, pkg.Info.HomePage) + } + return urls +} diff --git a/internal/resolver/resolver.go b/internal/resolver/resolver.go new file mode 100644 index 0000000..e4c341c --- /dev/null +++ b/internal/resolver/resolver.go @@ -0,0 +1,102 @@ +// Package resolver discovers authoritative documentation URLs for libraries by +// querying package registries (npm, PyPI, crates.io, pkg.go.dev) before +// falling back to web search. This gives maintainer-declared URLs rather than +// whatever a search engine surfaces. +package resolver + +import ( + "context" + "net/http" + "strings" + "time" +) + +const userAgent = "markdocs/0.3.0 (https://github.com/Nithin-Valiyaveedu/markdocs)" + +// Resolver queries package registries to find official documentation URLs. +// Base URLs are configurable so tests can inject httptest servers. +type Resolver struct { + NPMBaseURL string + PyPIBaseURL string + CratesBaseURL string + HTTPClient *http.Client +} + +// New returns a Resolver configured for the real public registries. +func New() *Resolver { + return &Resolver{ + NPMBaseURL: "https://registry.npmjs.org", + PyPIBaseURL: "https://pypi.org", + CratesBaseURL: "https://crates.io", + HTTPClient: &http.Client{Timeout: 10 * time.Second}, + } +} + +// Resolve returns candidate documentation URLs for the given library, ordered +// from most to least authoritative. It never returns an error unless all +// registry lookups fail — in that case callers should fall back to web search. +func (r *Resolver) Resolve(ctx context.Context, library string) ([]string, error) { + var candidates []string + + switch { + case isGoModule(library): + candidates = append(candidates, r.resolveGo(library)...) + case strings.HasPrefix(library, "@"): + // Scoped npm package — only try npm + candidates = append(candidates, r.resolveNPM(ctx, library)...) + default: + // Unknown ecosystem: try npm first (largest registry), then PyPI, then crates + candidates = append(candidates, r.resolveNPM(ctx, library)...) + candidates = append(candidates, r.resolvePyPI(ctx, library)...) + candidates = append(candidates, r.resolveCrates(ctx, library)...) + } + + return dedupe(candidates), nil +} + +// isGoModule reports whether the library name looks like a Go module path. +func isGoModule(library string) bool { + goPrefixes := []string{ + "github.com/", "golang.org/", "gopkg.in/", + "k8s.io/", "sigs.k8s.io/", "go.uber.org/", + "go.opentelemetry.io/", "cloud.google.com/", + } + for _, p := range goPrefixes { + if strings.HasPrefix(library, p) { + return true + } + } + return false +} + +// isRepoURL reports whether a URL points to a source code host rather than docs. +// We still include these, but docs-like URLs are placed first. +func isRepoURL(u string) bool { + repoHosts := []string{"github.com", "gitlab.com", "bitbucket.org", "sr.ht"} + for _, h := range repoHosts { + if strings.Contains(u, h) { + return true + } + } + return false +} + +// dedupe removes duplicate and empty URLs, preserving order. +// Non-repo (docs) URLs are returned before repo URLs. +func dedupe(urls []string) []string { + seen := make(map[string]bool) + var docs, repos []string + for _, u := range urls { + u = strings.TrimSpace(u) + if u == "" || seen[u] { + continue + } + seen[u] = true + if isRepoURL(u) { + repos = append(repos, u) + } else { + docs = append(docs, u) + } + } + return append(docs, repos...) +} diff --git a/internal/resolver/resolver_test.go b/internal/resolver/resolver_test.go new file mode 100644 index 0000000..fe36d46 --- /dev/null +++ b/internal/resolver/resolver_test.go @@ -0,0 +1,225 @@ +package resolver + +import ( + "context" + "encoding/json" + "net/http" + "net/http/httptest" + "strings" + "testing" +) + +// fakeNPM returns a test server that responds to registry.npmjs.org-style requests. +func fakeNPM(t *testing.T, homepage, repoURL string) *httptest.Server { + t.Helper() + return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(map[string]any{ + "homepage": homepage, + "repository": map[string]string{ + "url": repoURL, + }, + }) + })) +} + +func fakePyPI(t *testing.T, docURL, homepage string) *httptest.Server { + t.Helper() + return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(map[string]any{ + "info": map[string]any{ + "project_urls": map[string]string{ + "Documentation": docURL, + }, + "home_page": homepage, + }, + }) + })) +} + +func fakeCrates(t *testing.T, docURL, homepage, repo string) *httptest.Server { + t.Helper() + return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(map[string]any{ + "crate": map[string]string{ + "documentation": docURL, + "homepage": homepage, + "repository": repo, + }, + }) + })) +} + +func fakeNotFound() *httptest.Server { + return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(http.StatusNotFound) + })) +} + +func TestResolveNPM_WithDocsHomepage(t *testing.T) { + srv := fakeNPM(t, "https://zustand.docs.pmnd.rs/", "git+https://github.com/pmndrs/zustand.git") + defer srv.Close() + + r := &Resolver{NPMBaseURL: srv.URL, PyPIBaseURL: "http://unused", CratesBaseURL: "http://unused", HTTPClient: &http.Client{}} + urls, err := r.Resolve(context.Background(), "zustand") + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if len(urls) == 0 { + t.Fatal("expected at least one URL") + } + // docs URL should come first (non-GitHub) + if urls[0] != "https://zustand.docs.pmnd.rs/" { + t.Errorf("expected docs URL first, got %q", urls[0]) + } + // GitHub repo should still be present but after docs + found := false + for _, u := range urls { + if strings.Contains(u, "github.com") { + found = true + } + } + if !found { + t.Error("expected GitHub repo URL in results") + } +} + +func TestResolveNPM_GitHubHomepageOnly(t *testing.T) { + // When homepage IS the GitHub repo, it should still be returned (many libs only have GH docs) + srv := fakeNPM(t, "https://github.com/lodash/lodash", "git+https://github.com/lodash/lodash.git") + defer srv.Close() + + r := &Resolver{NPMBaseURL: srv.URL, PyPIBaseURL: "http://unused", CratesBaseURL: "http://unused", HTTPClient: &http.Client{}} + urls, err := r.Resolve(context.Background(), "lodash") + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + // Should have the GitHub URL (deduplicated — homepage and repo are the same) + if len(urls) == 0 { + t.Fatal("expected at least one URL even when homepage is GitHub") + } + for _, u := range urls { + if !strings.Contains(u, "github.com") { + t.Errorf("unexpected non-GitHub URL %q when only GitHub is available", u) + } + } +} + +func TestResolveNPM_NotFound(t *testing.T) { + srv := fakeNotFound() + defer srv.Close() + + r := &Resolver{NPMBaseURL: srv.URL, PyPIBaseURL: srv.URL, CratesBaseURL: srv.URL, HTTPClient: &http.Client{}} + urls, err := r.Resolve(context.Background(), "nonexistent-package-xyz") + if err != nil { + t.Fatalf("Resolve should not return error on 404, got: %v", err) + } + // Empty is fine — caller falls back to web search + if urls == nil { + urls = []string{} + } + _ = urls +} + +func TestResolvePyPI(t *testing.T) { + srv := fakePyPI(t, "https://docs.python-requests.org/", "https://requests.readthedocs.io/") + defer srv.Close() + + r := &Resolver{NPMBaseURL: "http://unused", PyPIBaseURL: srv.URL, CratesBaseURL: "http://unused", HTTPClient: &http.Client{}} + // PyPI resolution is triggered for simple names (non-Go, non-scoped) + urls := r.resolvePyPI(context.Background(), "requests") + if len(urls) == 0 { + t.Fatal("expected URLs from PyPI") + } + if urls[0] != "https://docs.python-requests.org/" { + t.Errorf("expected Documentation URL first, got %q", urls[0]) + } +} + +func TestResolveCrates(t *testing.T) { + srv := fakeCrates(t, "https://docs.rs/tokio", "https://tokio.rs", "https://github.com/tokio-rs/tokio") + defer srv.Close() + + r := &Resolver{NPMBaseURL: "http://unused", PyPIBaseURL: "http://unused", CratesBaseURL: srv.URL, HTTPClient: &http.Client{}} + urls := r.resolveCrates(context.Background(), "tokio") + if len(urls) == 0 { + t.Fatal("expected URLs from crates.io") + } + if urls[0] != "https://docs.rs/tokio" { + t.Errorf("expected documentation URL first, got %q", urls[0]) + } +} + +func TestResolveGoModule(t *testing.T) { + r := New() + urls, err := r.Resolve(context.Background(), "github.com/gin-gonic/gin") + if err != nil { + t.Fatalf("unexpected error: %v", err) + } + if len(urls) == 0 { + t.Fatal("expected pkg.go.dev URL") + } + if urls[0] != "https://pkg.go.dev/github.com/gin-gonic/gin" { + t.Errorf("expected pkg.go.dev URL, got %q", urls[0]) + } +} + +func TestDeduplication(t *testing.T) { + // Same URL from homepage and repo should appear only once + srv := fakeNPM(t, "https://github.com/foo/bar", "git+https://github.com/foo/bar.git") + defer srv.Close() + + r := &Resolver{NPMBaseURL: srv.URL, PyPIBaseURL: "http://unused", CratesBaseURL: "http://unused", HTTPClient: &http.Client{}} + urls, _ := r.Resolve(context.Background(), "foo") + seen := make(map[string]int) + for _, u := range urls { + seen[u]++ + if seen[u] > 1 { + t.Errorf("duplicate URL returned: %q", u) + } + } +} + +func TestCleanGitURL(t *testing.T) { + cases := []struct { + input string + want string + }{ + {"git+https://github.com/foo/bar.git", "https://github.com/foo/bar"}, + {"https://github.com/foo/bar", "https://github.com/foo/bar"}, + {"git://github.com/foo/bar.git", "https://github.com/foo/bar"}, + {"", ""}, + {"not-a-url", ""}, + } + for _, tc := range cases { + got := cleanGitURL(tc.input) + if got != tc.want { + t.Errorf("cleanGitURL(%q) = %q, want %q", tc.input, got, tc.want) + } + } +} + +func TestIsGoModule(t *testing.T) { + goModules := []string{ + "github.com/gin-gonic/gin", + "golang.org/x/net", + "gopkg.in/yaml.v3", + "k8s.io/client-go", + "go.uber.org/zap", + } + notGoModules := []string{ + "react", "zustand", "@scope/pkg", "requests", "tokio", + } + for _, m := range goModules { + if !isGoModule(m) { + t.Errorf("isGoModule(%q) = false, want true", m) + } + } + for _, m := range notGoModules { + if isGoModule(m) { + t.Errorf("isGoModule(%q) = true, want false", m) + } + } +} diff --git a/internal/skill/compiler.go b/internal/skill/compiler.go index 1388687..2ea9740 100644 --- a/internal/skill/compiler.go +++ b/internal/skill/compiler.go @@ -9,6 +9,7 @@ import ( "text/template" "github.com/Nithin-Valiyaveedu/markdocs/internal/llm" + "github.com/Nithin-Valiyaveedu/markdocs/internal/resolver" "github.com/Nithin-Valiyaveedu/markdocs/internal/search" ) @@ -81,25 +82,42 @@ type LLMCompiler struct { // searchFn is the web search backend. Defaults to search.DocURLs. // Override in tests to avoid real network calls. searchFn func(library string, max int) ([]string, error) + // resolveFn is the registry resolver backend. Defaults to resolver.New().Resolve. + // Override in tests to avoid real network calls. + resolveFn func(ctx context.Context, library string) ([]string, error) } var _ Compiler = (*LLMCompiler)(nil) // NewLLMCompiler creates a new LLMCompiler using the given provider. func NewLLMCompiler(provider llm.LLMProvider) *LLMCompiler { - return &LLMCompiler{provider: provider, searchFn: search.DocURLs} + r := resolver.New() + return &LLMCompiler{ + provider: provider, + searchFn: search.DocURLs, + resolveFn: r.Resolve, + } } // SuggestURLs discovers documentation URLs for the given library. -// It searches the web first (DuckDuckGo) and validates reachability, falling -// back to asking the LLM if web search yields nothing usable. +// Resolution order: package registry APIs → DuckDuckGo → LLM. func (c *LLMCompiler) SuggestURLs(ctx context.Context, library string) ([]string, error) { - // Step 1: web search + // Step 1: Package registry — authoritative, maintainer-declared + if regURLs, err := c.resolveFn(ctx, library); err == nil && len(regURLs) > 0 { + validated := search.ValidateURLs(regURLs) + if len(validated) > 0 { + if len(validated) > 5 { + validated = validated[:5] + } + return validated, nil + } + } + + // Step 2: Web search webURLs, err := c.searchFn(library, 8) if err == nil && len(webURLs) > 0 { validated := search.ValidateURLs(webURLs) if len(validated) > 0 { - // Return top 5 validated results if len(validated) > 5 { validated = validated[:5] } @@ -107,7 +125,7 @@ func (c *LLMCompiler) SuggestURLs(ctx context.Context, library string) ([]string } } - // Step 2: fall back to LLM + // Step 3: LLM fallback prompt, err := renderTemplate(urlDiscoveryPrompt, map[string]string{"Library": library}) if err != nil { return nil, fmt.Errorf("rendering url discovery prompt: %w", err) @@ -126,7 +144,6 @@ func (c *LLMCompiler) SuggestURLs(ctx context.Context, library string) ([]string return nil, fmt.Errorf("no URLs returned for library %q", library) } - // Validate LLM-suggested URLs too validated := search.ValidateURLs(urls) if len(validated) > 0 { return validated, nil diff --git a/internal/skill/compiler_test.go b/internal/skill/compiler_test.go index 0d1f95b..2b3ab86 100644 --- a/internal/skill/compiler_test.go +++ b/internal/skill/compiler_test.go @@ -24,13 +24,18 @@ func (m *mockProvider) Complete(ctx context.Context, prompt string) (string, err func (m *mockProvider) Model() string { return m.model } -// noopSearch is a search function that always returns nothing, forcing the LLM fallback. +// noopSearch is a search function that always returns nothing. func noopSearch(_ string, _ int) ([]string, error) { return nil, nil } -// newTestCompiler returns an LLMCompiler with web search disabled. +// noopResolve is a resolver function that always returns nothing. +func noopResolve(_ context.Context, _ string) ([]string, error) { return nil, nil } + +// newTestCompiler returns an LLMCompiler with web search and registry resolution disabled, +// so tests exercise only the LLM fallback path. func newTestCompiler(provider *mockProvider) *LLMCompiler { c := NewLLMCompiler(provider) c.searchFn = noopSearch + c.resolveFn = noopResolve return c } diff --git a/internal/skill/extractor.go b/internal/skill/extractor.go new file mode 100644 index 0000000..e2b6e0e --- /dev/null +++ b/internal/skill/extractor.go @@ -0,0 +1,242 @@ +package skill + +import ( + "context" + "fmt" + "strings" + + "github.com/Nithin-Valiyaveedu/markdocs/internal/resolver" + "github.com/Nithin-Valiyaveedu/markdocs/internal/search" +) + +// sectionEntry pairs a skill section key with its heading keyword patterns. +type sectionEntry struct { + key string + keywords []string +} + +// sectionMappings is an ordered list of section → keyword mappings. +// Order matters: more specific sections are checked first so that broad keywords +// like "guide" or "overview" don't shadow more precise matches (e.g. "migration guide"). +var sectionMappings = []sectionEntry{ + {"versions", []string{"changelog", "migration", "upgrade", "breaking", "release note", "what's new", "history"}}, + {"installation", []string{"install", "getting started", "quick start", "quickstart", "setup", "prerequisites", "requirement"}}, + {"errors", []string{"error", "troubleshoot", "common issue", "faq", "problem", "debug", "fix"}}, + {"gotchas", []string{"gotcha", "pitfall", "caveat", "warning", "limitation", "caveats", "known issue"}}, + {"config", []string{"config", "configuration", "options", "settings", "environment", "customize", "parameter"}}, + {"api", []string{"api", "reference", "methods", "functions", "interface", "endpoints", "usage", "example", "hook"}}, + // concepts last — "guide", "overview", "introduction" are broad and should only match + // when no more specific section has matched. + {"concepts", []string{"concept", "overview", "architecture", "how it works", "introduction", "guide", "basics", "core", "fundamental"}}, +} + +// StructuredExtractor implements Compiler using deterministic heading-based +// extraction from scraped documentation — no LLM required. +type StructuredExtractor struct { + searchFn func(library string, max int) ([]string, error) + resolveFn func(ctx context.Context, library string) ([]string, error) +} + +var _ Compiler = (*StructuredExtractor)(nil) + +// NewStructuredExtractor creates a StructuredExtractor using DuckDuckGo web search +// and package registry resolution. +func NewStructuredExtractor() *StructuredExtractor { + r := resolver.New() + return &StructuredExtractor{ + searchFn: search.DocURLs, + resolveFn: r.Resolve, + } +} + +// SuggestURLs discovers documentation URLs using a three-layer approach: +// 1. Package registry APIs (npm, PyPI, crates.io, pkg.go.dev) — authoritative +// 2. markdocs curated registry DocHints — verified overrides +// 3. DuckDuckGo web search — last resort +func (e *StructuredExtractor) SuggestURLs(ctx context.Context, library string) ([]string, error) { + // Layer 1: Package registry — maintainer-declared docs URL + if regURLs, err := e.resolveFn(ctx, library); err == nil && len(regURLs) > 0 { + validated := search.ValidateURLs(regURLs) + if len(validated) > 0 { + if len(validated) > 5 { + validated = validated[:5] + } + return validated, nil + } + } + + // Layer 2: Curated DocHints registry + if lib := RegistryByName(sanitizeName(library)); lib != nil && len(lib.DocHints) > 0 { + return lib.DocHints, nil + } + + // Layer 3: DuckDuckGo web search + urls, searchErr := e.searchFn(library, 8) + if searchErr == nil && len(urls) > 0 { + validated := search.ValidateURLs(urls) + if len(validated) > 0 { + if len(validated) > 5 { + validated = validated[:5] + } + return validated, nil + } + } + + if searchErr != nil { + return nil, fmt.Errorf("web search for %q: %w", library, searchErr) + } + return nil, fmt.Errorf("no documentation URLs found for %q", library) +} + +// Compile extracts skill sections from scraped markdown using heading pattern matching. +// It never calls an LLM. If the library's category cannot be detected from the registry, +// CompileOutput.Category will be empty — callers should prompt the user for it. +func (e *StructuredExtractor) Compile(_ context.Context, input CompileInput) (*CompileOutput, error) { + sections := extractSections(input.ScrapedMarkdown) + + whatThisIs := firstParagraph(input.ScrapedMarkdown) + + // Category from registry (exact then prefix match); fall back to "general" + category := CategoryForLibrary(input.Library) + if category == "" { + category = "general" + } + + description := whatThisIs + if len(description) > 200 { + description = description[:197] + "..." + } + + whenToUse := fmt.Sprintf("user mentions %s, working with %s libraries", input.Library, input.Library) + if category != "" { + whenToUse = fmt.Sprintf("user mentions %s, working with %s libraries", input.Library, category) + } + + var sb strings.Builder + sb.WriteString(fmt.Sprintf("# %s — markdocs skill\n\n", input.Library)) + + sb.WriteString("## What This Is\n") + if whatThisIs != "" { + sb.WriteString(whatThisIs + "\n\n") + } else { + sb.WriteString(fmt.Sprintf("%s library documentation.\n\n", input.Library)) + } + + writeSkillSection(&sb, "## Installation (project-specific)", sections["installation"]) + writeSkillSection(&sb, "## Key Concepts", sections["concepts"]) + writeSkillSection(&sb, "## API / Usage Patterns", sections["api"]) + writeSkillSection(&sb, "## Your Project Config (detected)", sections["config"]) + writeSkillSection(&sb, "## Hidden Gotchas", sections["gotchas"]) + writeSkillSection(&sb, "## Common Errors", sections["errors"]) + writeSkillSection(&sb, "## Version Notes", sections["versions"]) + + return &CompileOutput{ + Markdown: strings.TrimSpace(sb.String()), + Category: category, + Description: description, + WhenToUse: whenToUse, + }, nil +} + +// extractSections parses scraped markdown into skill section buckets by heading matching. +// Content under unrecognised headings is discarded. +func extractSections(markdown string) map[string]string { + lines := strings.Split(markdown, "\n") + sections := make(map[string]string) + currentKey := "" + var currentLines []string + + flush := func() { + if currentKey == "" || len(currentLines) == 0 { + return + } + content := strings.TrimSpace(strings.Join(currentLines, "\n")) + if content == "" { + return + } + if existing := sections[currentKey]; existing != "" { + sections[currentKey] = existing + "\n\n" + content + } else { + sections[currentKey] = content + } + } + + for _, line := range lines { + if isHeading(line) { + newKey := matchSection(line) + if newKey != "" { + // Matched a skill section: flush previous and start new + flush() + currentLines = nil + currentKey = newKey + } else if currentKey != "" { + // Unmatched sub-heading: include as content in the current section + currentLines = append(currentLines, line) + } + } else if currentKey != "" { + currentLines = append(currentLines, line) + } + } + flush() + + return sections +} + +// matchSection returns the skill section key for the given heading line, or "" if none match. +// Sections are checked in priority order (most specific first). +func matchSection(heading string) string { + lower := strings.ToLower(strings.TrimLeft(heading, "# ")) + for _, entry := range sectionMappings { + for _, kw := range entry.keywords { + if strings.Contains(lower, kw) { + return entry.key + } + } + } + return "" +} + +func isHeading(line string) bool { + return strings.HasPrefix(line, "#") +} + +// firstParagraph returns the first non-heading, non-empty paragraph of the markdown. +func firstParagraph(markdown string) string { + lines := strings.Split(markdown, "\n") + var para []string + inPara := false + + for _, line := range lines { + if isHeading(line) { + if inPara { + break + } + continue + } + trimmed := strings.TrimSpace(line) + if trimmed == "" { + if inPara { + break + } + continue + } + para = append(para, trimmed) + inPara = true + } + + result := strings.Join(para, " ") + // Strip common markdown inline formatting that would look odd as a description + result = strings.ReplaceAll(result, "**", "") + result = strings.ReplaceAll(result, "__", "") + return strings.TrimSpace(result) +} + +// writeSkillSection writes a section heading and its content (or a placeholder). +func writeSkillSection(sb *strings.Builder, header, content string) { + sb.WriteString(header + "\n") + if content != "" { + sb.WriteString(content + "\n\n") + } else { + sb.WriteString("_No information found in documentation._\n\n") + } +} diff --git a/internal/skill/extractor_test.go b/internal/skill/extractor_test.go new file mode 100644 index 0000000..4166dac --- /dev/null +++ b/internal/skill/extractor_test.go @@ -0,0 +1,220 @@ +package skill + +import ( + "context" + "strings" + "testing" +) + +const sampleDocs = `# Zustand + +Zustand is a small, fast state management library for React. + +## Installation + +Install via npm: + +` + "```bash" + ` +npm install zustand +` + "```" + ` + +Or with yarn: + +` + "```bash" + ` +yarn add zustand +` + "```" + ` + +## Core Concepts + +Zustand uses a store-based approach where state is managed outside React components. +Stores are created with the create function and accessed via hooks. + +## API Reference + +### create + +` + "```ts" + ` +const useStore = create((set) => ({ + count: 0, + increment: () => set((state) => ({ count: state.count + 1 })), +})) +` + "```" + ` + +## Configuration Options + +You can configure middleware like devtools and immer: + +` + "```ts" + ` +const useStore = create(devtools(immer((set) => ({ ... })))) +` + "```" + ` + +## Common Errors + +### Stale closures + +If you read state inside an event handler without using getState(), you may get stale values. + +## Changelog + +### v5.0 + +Breaking: removed deprecated create() overload. +` + +func TestExtractSections(t *testing.T) { + sections := extractSections(sampleDocs) + + if sections["installation"] == "" { + t.Error("expected installation section to be populated") + } + if !strings.Contains(sections["installation"], "npm install zustand") { + t.Errorf("installation section missing npm command, got: %s", sections["installation"]) + } + + if sections["concepts"] == "" { + t.Error("expected concepts section to be populated") + } + if !strings.Contains(sections["concepts"], "store-based") { + t.Errorf("concepts section missing expected content, got: %s", sections["concepts"]) + } + + if sections["api"] == "" { + t.Error("expected api section to be populated") + } + if !strings.Contains(sections["api"], "create") { + t.Errorf("api section missing expected content, got: %s", sections["api"]) + } + + if sections["config"] == "" { + t.Error("expected config section to be populated") + } + + if sections["errors"] == "" { + t.Error("expected errors section to be populated") + } + if !strings.Contains(sections["errors"], "Stale closures") { + t.Errorf("errors section missing expected content, got: %s", sections["errors"]) + } + + if sections["versions"] == "" { + t.Error("expected versions section to be populated") + } + if !strings.Contains(sections["versions"], "v5.0") { + t.Errorf("versions section missing expected content, got: %s", sections["versions"]) + } +} + +func TestFirstParagraph(t *testing.T) { + result := firstParagraph(sampleDocs) + if result == "" { + t.Fatal("expected non-empty first paragraph") + } + if !strings.Contains(result, "Zustand") { + t.Errorf("expected first paragraph to mention Zustand, got: %s", result) + } + // Should not include heading text + if strings.HasPrefix(result, "#") { + t.Errorf("first paragraph should not start with #, got: %s", result) + } +} + +func TestMatchSection(t *testing.T) { + cases := []struct { + heading string + want string + }{ + {"## Installation", "installation"}, + {"## Getting Started", "installation"}, + {"## API Reference", "api"}, + {"## Core Concepts", "concepts"}, + {"## Configuration", "config"}, + {"## Common Errors", "errors"}, + {"## Changelog", "versions"}, + {"## Migration Guide", "versions"}, + {"## About This Library", ""}, + } + + for _, tc := range cases { + got := matchSection(tc.heading) + if got != tc.want { + t.Errorf("matchSection(%q) = %q, want %q", tc.heading, got, tc.want) + } + } +} + +func TestStructuredExtractorCompile(t *testing.T) { + e := &StructuredExtractor{ + searchFn: func(library string, max int) ([]string, error) { return nil, nil }, + resolveFn: func(_ context.Context, _ string) ([]string, error) { return nil, nil }, + } + + out, err := e.Compile(context.Background(), CompileInput{ + Library: "zustand", + URL: "https://zustand.docs.pmnd.rs/", + ScrapedMarkdown: sampleDocs, + }) + if err != nil { + t.Fatalf("Compile returned error: %v", err) + } + + if out.Category != "frontend" { + t.Errorf("expected category 'frontend' (zustand is in registry), got %q", out.Category) + } + if out.Description == "" { + t.Error("expected non-empty description") + } + if out.WhenToUse == "" { + t.Error("expected non-empty when_to_use") + } + if !strings.Contains(out.Markdown, "## What This Is") { + t.Error("expected markdown to contain '## What This Is'") + } + if !strings.Contains(out.Markdown, "## Installation") { + t.Error("expected markdown to contain '## Installation'") + } + if !strings.Contains(out.Markdown, "## API / Usage Patterns") { + t.Error("expected markdown to contain '## API / Usage Patterns'") + } +} + +func TestStructuredExtractorCompileUnknownLibrary(t *testing.T) { + e := &StructuredExtractor{ + searchFn: func(library string, max int) ([]string, error) { return nil, nil }, + resolveFn: func(_ context.Context, _ string) ([]string, error) { return nil, nil }, + } + + out, err := e.Compile(context.Background(), CompileInput{ + Library: "some-unknown-lib", + URL: "https://example.com/docs", + ScrapedMarkdown: "# some-unknown-lib\n\nA library for doing things.\n\n## Installation\n\nnpm install some-unknown-lib\n", + }) + if err != nil { + t.Fatalf("Compile returned error: %v", err) + } + // Should default to "general" when not in registry + if out.Category != "general" { + t.Errorf("expected category 'general' for unknown library, got %q", out.Category) + } +} + +func TestCategoryForLibrary(t *testing.T) { + cases := []struct { + name string + want string + }{ + {"react", "frontend"}, + {"react-query", "frontend"}, // prefix match: "react" prefix + {"zustand", "frontend"}, + {"prisma", "database"}, + {"jest", "testing"}, + {"stripe", "payments"}, + {"completely-unknown-lib", ""}, + } + + for _, tc := range cases { + got := CategoryForLibrary(tc.name) + if got != tc.want { + t.Errorf("CategoryForLibrary(%q) = %q, want %q", tc.name, got, tc.want) + } + } +} diff --git a/internal/skill/registry.go b/internal/skill/registry.go index fdd1e80..eb73577 100644 --- a/internal/skill/registry.go +++ b/internal/skill/registry.go @@ -1,5 +1,7 @@ package skill +import "strings" + // KnownLibrary describes a library in the markdocs registry. type KnownLibrary struct { Name string @@ -10,7 +12,7 @@ type KnownLibrary struct { // Registry is the built-in list of commonly-documented libraries. // Used by `markdocs scan` to cross-reference detected dependencies. var Registry = []KnownLibrary{ - // Frontend + // Frontend frameworks {Name: "react", Category: "frontend", DocHints: []string{"https://react.dev"}}, {Name: "vue", Category: "frontend", DocHints: []string{"https://vuejs.org/guide"}}, {Name: "next", Category: "frontend", DocHints: []string{"https://nextjs.org/docs"}}, @@ -19,31 +21,72 @@ var Registry = []KnownLibrary{ {Name: "astro", Category: "frontend", DocHints: []string{"https://docs.astro.build"}}, {Name: "remix", Category: "frontend", DocHints: []string{"https://remix.run/docs"}}, {Name: "@angular/core", Category: "frontend", DocHints: []string{"https://angular.io/docs"}}, + {Name: "solid-js", Category: "frontend", DocHints: []string{"https://docs.solidjs.com"}}, + {Name: "qwik", Category: "frontend", DocHints: []string{"https://qwik.dev/docs"}}, + // State management + {Name: "zustand", Category: "frontend", DocHints: []string{"https://zustand.docs.pmnd.rs"}}, + {Name: "jotai", Category: "frontend", DocHints: []string{"https://jotai.org/docs"}}, + {Name: "recoil", Category: "frontend", DocHints: []string{"https://recoiljs.org/docs"}}, + {Name: "redux", Category: "frontend", DocHints: []string{"https://redux.js.org/introduction"}}, + {Name: "@reduxjs/toolkit", Category: "frontend", DocHints: []string{"https://redux-toolkit.js.org/introduction"}}, + {Name: "mobx", Category: "frontend", DocHints: []string{"https://mobx.js.org/README.html"}}, // UI Libraries {Name: "shadcn-ui", Category: "ui", DocHints: []string{"https://ui.shadcn.com/docs"}}, {Name: "@radix-ui/react-dialog", Category: "ui", DocHints: []string{"https://www.radix-ui.com/docs"}}, {Name: "tailwindcss", Category: "ui", DocHints: []string{"https://tailwindcss.com/docs"}}, {Name: "@mui/material", Category: "ui", DocHints: []string{"https://mui.com/material-ui"}}, {Name: "chakra-ui", Category: "ui", DocHints: []string{"https://chakra-ui.com/docs"}}, - // Backend + {Name: "antd", Category: "ui", DocHints: []string{"https://ant.design/docs"}}, + {Name: "mantine", Category: "ui", DocHints: []string{"https://mantine.dev/docs"}}, + {Name: "daisyui", Category: "ui", DocHints: []string{"https://daisyui.com/docs"}}, + // Data fetching / async + {Name: "@tanstack/react-query", Category: "frontend", DocHints: []string{"https://tanstack.com/query/latest/docs"}}, + {Name: "swr", Category: "frontend", DocHints: []string{"https://swr.vercel.app/docs"}}, + {Name: "axios", Category: "backend", DocHints: []string{"https://axios-http.com/docs"}}, + // Backend (JS/TS) {Name: "express", Category: "backend", DocHints: []string{"https://expressjs.com"}}, {Name: "fastify", Category: "backend", DocHints: []string{"https://fastify.dev/docs"}}, + {Name: "koa", Category: "backend", DocHints: []string{"https://koajs.com"}}, + {Name: "hono", Category: "backend", DocHints: []string{"https://hono.dev/docs"}}, + {Name: "nestjs", Category: "backend", DocHints: []string{"https://docs.nestjs.com"}}, + {Name: "trpc", Category: "backend", DocHints: []string{"https://trpc.io/docs"}}, + // Backend (Go) + {Name: "gin", Category: "backend", DocHints: []string{"https://gin-gonic.com/docs"}}, + {Name: "fiber", Category: "backend", DocHints: []string{"https://docs.gofiber.io"}}, + {Name: "echo", Category: "backend", DocHints: []string{"https://echo.labstack.com/docs"}}, + {Name: "chi", Category: "backend", DocHints: []string{"https://go-chi.io/docs"}}, // Database {Name: "prisma", Category: "database", DocHints: []string{"https://www.prisma.io/docs"}}, {Name: "drizzle-orm", Category: "database", DocHints: []string{"https://orm.drizzle.team/docs"}}, {Name: "@supabase/supabase-js", Category: "database", DocHints: []string{"https://supabase.com/docs"}}, + {Name: "mongoose", Category: "database", DocHints: []string{"https://mongoosejs.com/docs"}}, + {Name: "typeorm", Category: "database", DocHints: []string{"https://typeorm.io"}}, + {Name: "kysely", Category: "database", DocHints: []string{"https://kysely.dev/docs"}}, + {Name: "sqlalchemy", Category: "database", DocHints: []string{"https://docs.sqlalchemy.org"}}, // Testing {Name: "vitest", Category: "testing", DocHints: []string{"https://vitest.dev/guide"}}, {Name: "jest", Category: "testing", DocHints: []string{"https://jestjs.io/docs"}}, {Name: "playwright", Category: "testing", DocHints: []string{"https://playwright.dev/docs"}}, {Name: "cypress", Category: "testing", DocHints: []string{"https://docs.cypress.io"}}, + {Name: "@testing-library/react", Category: "testing", DocHints: []string{"https://testing-library.com/docs"}}, + {Name: "pytest", Category: "testing", DocHints: []string{"https://docs.pytest.org"}}, // Auth {Name: "next-auth", Category: "auth", DocHints: []string{"https://next-auth.js.org/getting-started"}}, {Name: "@auth/core", Category: "auth", DocHints: []string{"https://authjs.dev/getting-started"}}, + {Name: "clerk", Category: "auth", DocHints: []string{"https://clerk.com/docs"}}, + {Name: "lucia", Category: "auth", DocHints: []string{"https://lucia-auth.com/docs"}}, // Payments {Name: "stripe", Category: "payments", DocHints: []string{"https://stripe.com/docs"}}, - // Infra + {Name: "@lemonsqueezy/lemonsqueezy.js", Category: "payments", DocHints: []string{"https://docs.lemonsqueezy.com"}}, + // Infra / DevOps {Name: "docker", Category: "infra", DocHints: []string{"https://docs.docker.com"}}, + {Name: "kubernetes", Category: "infra", DocHints: []string{"https://kubernetes.io/docs"}}, + // DevTools + {Name: "vite", Category: "devtools", DocHints: []string{"https://vitejs.dev/guide"}}, + {Name: "webpack", Category: "devtools", DocHints: []string{"https://webpack.js.org/concepts"}}, + {Name: "esbuild", Category: "devtools", DocHints: []string{"https://esbuild.github.io/api"}}, + {Name: "turbo", Category: "devtools", DocHints: []string{"https://turbo.build/repo/docs"}}, + {Name: "turborepo", Category: "devtools", DocHints: []string{"https://turbo.build/repo/docs"}}, } // RegistryByName returns a KnownLibrary for the given name, or nil if not found. @@ -55,3 +98,23 @@ func RegistryByName(name string) *KnownLibrary { } return nil } + +// CategoryForLibrary looks up the category for a library by exact match first, +// then by prefix match (e.g. "react" matches "react-query"). Returns "" if unknown. +func CategoryForLibrary(library string) string { + normalized := strings.ToLower(sanitizeName(library)) + // Exact match + for _, lib := range Registry { + if strings.ToLower(sanitizeName(lib.Name)) == normalized { + return lib.Category + } + } + // Prefix match: registry name is a prefix of the input (e.g. "react" → "react-query") + for _, lib := range Registry { + registryName := strings.ToLower(sanitizeName(lib.Name)) + if registryName != "" && strings.HasPrefix(normalized, registryName) { + return lib.Category + } + } + return "" +} diff --git a/internal/skill/writer.go b/internal/skill/writer.go index 74ce8f7..33069b2 100644 --- a/internal/skill/writer.go +++ b/internal/skill/writer.go @@ -102,7 +102,7 @@ func NewSkillMeta(library, provider, model, category, framework, description, wh Model: model, Provider: provider, ProjectFramework: framework, - MarkdocsVersion: "0.2.1", + MarkdocsVersion: "0.3.0", } } diff --git a/internal/ui/prompts.go b/internal/ui/prompts.go index 96ef416..3b4de05 100644 --- a/internal/ui/prompts.go +++ b/internal/ui/prompts.go @@ -2,10 +2,116 @@ package ui import ( "fmt" + "os" + "os/exec" + "strings" "github.com/charmbracelet/huh" + "github.com/charmbracelet/lipgloss" ) +// ReviewAction represents what the user chose during draft review. +type ReviewAction int + +const ( + ReviewAccept ReviewAction = iota + ReviewEdit + ReviewDiscard +) + +// ReviewDraft shows a preview of the compiled skill draft and prompts the user +// to Accept, Edit in $EDITOR, or Discard before the file is written to disk. +func ReviewDraft(draftMarkdown string) (ReviewAction, string, error) { + Section("Draft Preview") + fmt.Println(renderDraftPreview(draftMarkdown)) + + const ( + optAccept = "Accept as-is" + optEdit = "Edit in $EDITOR" + optDiscard = "Discard" + ) + + var chosen string + err := huh.NewSelect[string](). + Title("What would you like to do with this draft?"). + Options( + huh.NewOption(optAccept, optAccept), + huh.NewOption(optEdit, optEdit), + huh.NewOption(optDiscard, optDiscard), + ). + Value(&chosen). + Run() + if err != nil { + return ReviewDiscard, "", fmt.Errorf("review prompt: %w", err) + } + + switch chosen { + case optEdit: + edited, err := openInEditor(draftMarkdown) + if err != nil { + return ReviewDiscard, "", fmt.Errorf("editor: %w", err) + } + return ReviewEdit, edited, nil + case optDiscard: + return ReviewDiscard, "", nil + default: + return ReviewAccept, draftMarkdown, nil + } +} + +// renderDraftPreview returns a lipgloss-bordered preview of the first 50 lines. +func renderDraftPreview(markdown string) string { + lines := strings.Split(markdown, "\n") + const maxLines = 50 + var body string + if len(lines) > maxLines { + body = strings.Join(lines[:maxLines], "\n") + + "\n" + StyleMuted.Render(fmt.Sprintf(" … %d more lines", len(lines)-maxLines)) + } else { + body = markdown + } + return lipgloss.NewStyle(). + Border(lipgloss.RoundedBorder()). + BorderForeground(lipgloss.Color(ColorBorder)). + Padding(0, 1). + MaxWidth(100). + Render(body) +} + +// openInEditor writes content to a temp file, opens it in $EDITOR (fallback: nano), +// waits for the editor to exit, then returns the (possibly modified) content. +func openInEditor(content string) (string, error) { + tmp, err := os.CreateTemp("", "markdocs-draft-*.md") + if err != nil { + return "", fmt.Errorf("creating temp file: %w", err) + } + defer os.Remove(tmp.Name()) + + if _, err := tmp.WriteString(content); err != nil { + return "", fmt.Errorf("writing temp file: %w", err) + } + tmp.Close() + + editor := os.Getenv("EDITOR") + if editor == "" { + editor = "nano" + } + + editorCmd := exec.Command(editor, tmp.Name()) + editorCmd.Stdin = os.Stdin + editorCmd.Stdout = os.Stdout + editorCmd.Stderr = os.Stderr + if err := editorCmd.Run(); err != nil { + return "", fmt.Errorf("editor exited with error: %w", err) + } + + result, err := os.ReadFile(tmp.Name()) + if err != nil { + return "", fmt.Errorf("reading edited file: %w", err) + } + return string(result), nil +} + // Select shows an interactive single-select prompt and returns the chosen value. func Select(title string, options []string) (string, error) { opts := make([]huh.Option[string], len(options)) diff --git a/package.json b/package.json index 7fc9f55..340e761 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@val-nithin/markdocs", - "version": "0.2.2", + "version": "0.3.0", "description": "CLI to compile library documentation into Claude Code skill files.", "main": "index.js", "scripts": { diff --git a/skills/markdocs/SKILL.md b/skills/markdocs/SKILL.md index ee45afb..7efeeb8 100644 --- a/skills/markdocs/SKILL.md +++ b/skills/markdocs/SKILL.md @@ -1,11 +1,11 @@ --- name: markdocs description: Use this skill when you need to compile library documentation into Claude Code skill files, scan a project for missing documentation skills, or update existing skills when dependencies change. -compatibility: Requires go 1.22+ OR npm OR brew. Requires an LLM provider configured via `markdocs init` (supports Anthropic, OpenAI, OpenAI-compatible endpoints, and local Ollama). +compatibility: Requires go 1.22+ OR npm OR brew. No API key required by default. Use --llm flag with a configured provider (markdocs init) for LLM-compiled output. license: MIT metadata: author: Nithin-Valiyaveedu - version: "0.1.0" + version: "0.3.0" --- # `markdocs` Skill @@ -16,7 +16,8 @@ Compile library documentation into structured Claude Code skill files that Claud When this skill is invoked, confirm: - `markdocs` is installed (see Installation below) -- A provider is configured (`markdocs init`) +- For default structured mode: no API key needed +- For `--llm` mode: a provider is configured (`markdocs init`) ## Installation @@ -31,28 +32,23 @@ npm install -g @Nithin-Valiyaveedu/markdocs brew install Nithin-Valiyaveedu/markdocs/markdocs ``` -## Step 1: Configure a Provider - -```bash -markdocs init -``` - -Auto-detects from environment (`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, or local Ollama). Supports OpenAI-compatible endpoints (Groq, Together AI) via base URL override. - -## Step 2: Add Skills +## Step 1: Add Skills (no API key required) ```bash markdocs add ``` -Example: `markdocs add shadcn` → writes `.claude/skills/ui/shadcn.md` +Example: `markdocs add zustand` → writes `.claude/skills/frontend/zustand.md` + +By default, markdocs uses **structured extraction** — it scrapes the official docs and maps headings to skill sections deterministically. No LLM, no API key. Flags: | Flag | Description | |------|-------------| -| `--no-interactive` | Skip URL selection, use first suggested URL | +| `--llm` | Use LLM compilation instead of structured extraction (requires `markdocs init`) | +| `--no-interactive` | Skip URL selection prompt, use first suggested URL | -## Step 3: Manage Skills +## Step 2: Manage Skills | Command | Description | |---------|-------------| @@ -65,15 +61,19 @@ Flags: ## How It Works -1. LLM discovers official documentation URLs for the library -2. User selects which URL(s) to scrape (interactive) -3. Built-in Go scraper fetches and cleans the content (no external APIs) +### Default (structured extraction) +1. Web search finds official documentation URLs for the library +2. User selects which URL to scrape (interactive) +3. Built-in Go scraper fetches and cleans the content - Layer 1: `net/http` + `go-readability` (static sites) - Layer 2: `go-rod` headless browser (JS-heavy sites, fallback) -4. LLM compiles scraped content into a structured skill file +4. Heading-based extractor maps doc sections to skill sections 5. Skill is written to `.claude/skills//.md` 6. Claude Code picks it up automatically next session +### With `--llm` +Steps 1–3 are the same. Step 4 sends scraped content to the configured LLM provider (Anthropic, OpenAI, or Ollama) for richer compilation. + ## Skill File Format Skills are plain Markdown with YAML frontmatter. The frontmatter stores all metadata — no database required: @@ -81,22 +81,25 @@ Skills are plain Markdown with YAML frontmatter. The frontmatter stores all meta ```markdown --- name: react +description: React is a JavaScript library for building UIs... +when_to_use: user mentions react, building a component, JSX +user-invocable: false category: frontend sources: - https://react.dev/learn -compiled: 2026-04-12T10:23:00Z +compiled: 2026-04-19T10:23:00Z checksum: sha256:abc123def456 -model: llama3.2 -provider: ollama +model: "" +provider: structured project_framework: next.js -markdocs_version: 0.1.0 +markdocs_version: 0.2.1 --- # react — markdocs skill ... ``` -## Step 4: Report +## Step 3: Report After running any command, confirm to the user: - Which skills were compiled / updated / skipped