diff --git a/CHANGELOG.md b/CHANGELOG.md index 82299d3..9cb18a5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,30 @@ Accuracy figures inside a released entry are the numbers measured **at that rele left as written. The current numbers live on the [evidence page](https://usefixmap.vercel.app/evidence), which is generated from the recorded results rather than transcribed by hand. +## 0.9.0 - 2026-08-13 + +### Added + +- Every plan now contains a separate Impact Graph built from direct imports, reverse dependents, routed tests, and repeated Git co-change relationships. Each relationship carries its evidence, confidence, and inspection order; generated and backup artifacts stay excluded. +- `fixmap context` builds a deterministic Markdown or JSON source pack from primary and impact files. It selects line ranges within an estimated source-token budget and records roles, reasons, confidence, truncation, and omissions. +- `fixmap graph` exports the Impact Graph as portable Mermaid or versioned JSON while preserving relationship direction and evidence. +- `fixmap watch --report plan.json --repo .` emits verification and recalculated impact only when the local working tree changes, with Markdown or JSON Lines output. +- `fixmap benchmark --repo . --last 50` backtests BM25-over-code, FixMap context, and FixMap with Impact Graph on identical historical parent-snapshot corpora. It reports all, path-mentioned, and unmentioned cohorts, Wilson intervals, raw cases, skip counts, and secondary-file recall without executing repository code or scoring generated twins as primary answers. +- `fixmap plan --format agent` emits a compact, stable handoff organized as EDIT CANDIDATE, INSPECT, TEST, RISK, AVOID, and UNCERTAINTY. +- A frozen four-arm agent-study protocol and validator are checked in for future controlled measurements. No agent-effectiveness or time-saved claim is made without completed, auditable runs. +- A 32-second motion-first agent comparison is available on the README and website in animated-preview and 1080p H.264/AAC formats, with original no-vocals music and no unsupported savings claim. + +### Improved + +- Verify recalculates impact around the files actually changed and adds advisory findings for high-evidence related paths outside the original plan. +- Plan, Context, Graph, MCP, the GitHub Action, slash-command discovery, the live demo, package documentation, and the website expose the new evidence consistently. +- Git-history collection is bounded, cached, non-executing, and explicit about shallow, truncated, missing, or unreadable history. Import and test relationships continue to work when history is unavailable. + +### Evidence and release engineering + +- The original held-out benchmark and its honest BM25 comparison remain unchanged. The repository benchmark is a separate local backtest and does not rewrite frozen evidence. +- The release candidate must pass workspace typechecking, all automated tests, lint, dependency audits, production builds, metadata and generated-artifact checks, package and Action smoke tests, evaluation gates, the agent-study protocol check, scanner performance checks, and desktop/mobile browser verification before any release or deployment action. + ## 0.8.9 - 2026-08-11 ### Added diff --git a/README.md b/README.md index c8ac781..da69476 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ Know where to edit before the first edit. -Paste a GitHub issue URL, describe a task, or point at a diff. FixMap returns ranked context files, reachable test commands, risk notes, and explicit diagnostics—without an account, API key, or model call. +Paste a GitHub issue URL, describe a task, or point at a diff. FixMap returns ranked context, an evidence-backed Impact Graph, reachable test commands, risks, and explicit uncertainty—without an account, API key, or model call. [![CI](https://github.com/aryamthecodebreaker/FixMap/actions/workflows/ci.yml/badge.svg)](https://github.com/aryamthecodebreaker/FixMap/actions/workflows/ci.yml) [![npm](https://img.shields.io/npm/v/%40aryam%2Ffixmap)](https://www.npmjs.com/package/@aryam/fixmap) @@ -16,6 +16,10 @@ Paste a GitHub issue URL, describe a task, or point at a diff. FixMap returns ra +[![FixMap v0.9.0: two coding agents work the same issue, with FixMap supplying ranked context, an Impact Graph, Watch feedback, and Verify evidence](docs/assets/fixmap-v0.9.0-agent-comparison.gif)](https://usefixmap.vercel.app/fixmap-launch.mp4) + +

Watch the 32-second launch film with sound

+ ![A generated FixMap CLI report showing ranked context files, test routes, risks, analysis, and diagnostics](docs/assets/fixmap-cli-demo.svg) ## Install @@ -51,6 +55,38 @@ Save a plan before editing: fixmap plan --issue "password reset emails fail" --format json --output plan.json ``` +The plan separates primary context from likely impact: imports, reverse dependents, routed tests, and repeated Git co-change relationships. Impact files are places to inspect, not assumed edits. + +Measure BM25, FixMap, and Impact Graph on your own repository's history: + +```bash +fixmap benchmark --repo . --last 50 +``` + +Keep the saved plan beside an agent while it edits. Watch emits a new verification only when the working tree changes and recalculates impact around the actual diff: + +```bash +fixmap watch --report plan.json --repo . --include-untracked +``` + +Give an agent the relevant source ranges instead of only file names. Context draws from primary and impact files and stays within an estimated source-token budget: + +```bash +fixmap context --issue "password reset emails fail" --budget 10000 +``` + +Export the evidence graph for a pull request, issue, or design note: + +```bash +fixmap graph --issue "password reset emails fail" --format mermaid +``` + +Use compact headings in an agent context window: + +```bash +fixmap plan --issue "password reset emails fail" --format agent +``` + Ask why an expected path is missing: ```bash @@ -99,17 +135,33 @@ Use `--working-tree` for staged and unstaged tracked edits, `--include-untracked - Explains task grounding, ranking shape, unresolved or partially matched identifiers, exclusions, scan limits, unread content, skipped submodules, empty diffs, and Git failures. - Supports a strict decimal `--limit`, repeatable `--exclude`, and ordered `.fixmapignore` patterns with negation. Root-leading patterns are repository-relative, pasted absolute paths inside the repository are normalized, and patterns that match nothing produce a warning. Limits change only how many rows are shown, never confidence or ranking-shape analysis. +### Impact Graph and repository benchmark + +- Builds a separate likely-impact view from direct imports, reverse dependents, routed tests, and Git files that repeatedly changed with a primary ranked file. +- Reads at most 1,000 non-merge commits, excludes commits touching more than 30 files, requires at least two co-occurrences, and marks shallow or unavailable history instead of inventing evidence. +- Recalculates impact around files actually changed during Verify and identifies high-evidence dependents outside the original plan as inspection notes, never mandatory edits. +- `fixmap benchmark --repo . --last 50` evaluates BM25-over-code, ordinary FixMap context, and Impact Graph against historical parent snapshots. Every case's history stops before its target change, all arms see one scanned corpus, generated twins are not scored as primary answers, and mentioned/unmentioned tasks are reported separately. +- `fixmap watch --report plan.json --repo .` monitors a local working tree, re-runs Verify, and recalculates impact only when edits change. It never executes repository code; `--format json` produces one JSON object per update. +- Benchmark Markdown and versioned JSON include Wilson intervals, excluded-case counts, secondary-file recall, safeguards, and raw per-case outcomes. Historical commit messages are a repository-specific backtest, never proof of agent savings. + +### Context packs and graph export + +- `fixmap context` selects deterministic line ranges from primary and impact files, labels each snippet as primary or impact, and records its reason, confidence, line range, estimated token cost, source truncation, and omitted-file reason. +- The budget counts source using the stable estimate `ceil(UTF-8 bytes / 4)`; metadata is excluded. This is a reproducible planning estimate, not a tokenizer-specific exact count. +- Context may use FixMap's bounded scanner sample rather than an entire large file. `sourceTruncated` makes that boundary explicit in JSON and Markdown. +- `fixmap graph` exports the same Impact Graph as Mermaid or versioned JSON, preserving imports, imported-by, test-route, and co-change direction and evidence. + ### Exclusion pattern syntax `--exclude` and `.fixmapignore` use repository-relative gitignore-style patterns. `/docs/**` anchors at the repository root, `docs/**` matches the same root directory and nested occurrences, `!docs/keep.md` re-includes a path after an earlier exclusion, and trailing `/` targets a directory. `*`, `?`, and `**` are supported; brace groups such as `{src,test}` are literal text, not alternation. Pass repeated `--exclude` flags or put one pattern per `.fixmapignore` line so commas in literal names stay unambiguous. -- Produces Markdown for people or versioned JSON for tools, writes to `--output`, and gives one grounded next action. +- Produces Markdown for people, versioned JSON for tools, or `--format agent` for compact `EDIT CANDIDATE`/`INSPECT`/`TEST`/`RISK`/`AVOID`/`UNCERTAINTY` sections; writes any format with `--output`. ### Explain, Compare, Verify, Validate, and Doctor - **Explain** tells you whether a path ranked, fell below the cutoff, was excluded, resolves through a submodule, or was never scanned—and uses the same task and diff evidence as Plan. - **Compare** shows files that entered, left, moved, or changed confidence after the task was refined, plus changes in task grounding. -- **Verify** compares a saved JSON plan with a diff or working tree and flags generated edits, unmapped changes, an untouched leading file, source changes without tests, newly reached risk areas, and plan/repository mismatches. It fails on errors by default; `--fail-on warning` and the Action's `fail-on: warning` turn advisory findings into an opt-in CI gate. +- **Verify** compares a saved JSON plan with a diff or working tree, recalculates impact, and flags generated edits, unmapped changes, an untouched leading file, source changes without tests, newly reached risk areas, and plan/repository mismatches. It fails on errors by default; `--fail-on warning` and the Action's `fail-on: warning` turn advisory findings into an opt-in CI gate. - **Validate** checks any saved JSON report with the structural compatibility validator shared by Compare, Verify, the Action, and MCP. - **Doctor** prints the running version and executable path and diagnoses project, global, PATH, and npm-exec version shadows. - `FIXMAP_PROGRESS` controls remote clone/scan progress, and `FIXMAP_VERBOSE_USAGE` restores full usage text after argument errors. @@ -117,14 +169,14 @@ Use `--working-tree` for staged and unstaged tracked edits, `--include-untracked ### Agent and automation interfaces - `fixmap setup` installs `/fixmap` discovery for Claude Code, Cursor, GitHub Copilot prompt files, and the open Agent Skills layout; the no-argument command lists every FixMap workflow before making changes. -- The MCP server exposes `fixmap_plan`, `fixmap_explain`, `fixmap_compare`, `fixmap_verify`, and `fixmap_doctor` over local stdio and is published in the official MCP Registry. +- The MCP server exposes `fixmap_plan`, `fixmap_context`, `fixmap_graph`, `fixmap_explain`, `fixmap_compare`, `fixmap_verify`, and `fixmap_doctor` over local stdio and is published in the official MCP Registry. - The GitHub Action runs Plan or Verify on pull requests, appends within the job summary's remaining 1 MiB budget, bounds its report output and comment, and creates or updates one FixMap comment instead of posting duplicates. - The Action accepts explicit task input or pull-request context, uses the same report validator as the CLI and MCP server, and fails clearly when a requested diff cannot be resolved. - The browser demo runs the real core Plan, Explain, Compare, and Verify logic against a sample repository without uploading the task. ### TypeScript library -- `@aryam/fixmap-core` exposes repository scanning, exclusion resolution, ranking, task grounding, language and import-proximity analysis, test/risk routing, report validation, and Markdown/JSON rendering. +- `@aryam/fixmap-core` exposes repository scanning, exclusion resolution, ranking, Context Pack and Impact Graph construction, BM25 retrieval, task grounding, language/import analysis, test/risk routing, report validation, and Markdown/JSON/agent/Mermaid rendering. - Its public API also exposes Explain, Compare, and Verify builders and result types, so another tool can compose the same workflow without shelling out to the CLI. - The `@aryam/fixmap-core/browser` entry runs the filesystem-free report, comparison, explanation, verification, and rendering logic in a browser bundle. @@ -139,6 +191,7 @@ Use `--working-tree` for staged and unstaged tracked edits, `--include-untracked ## What the report contains - Ranked context files with scores, confidence, and evidence. +- Likely impact files with relationship-specific evidence, confidence, history coverage, and inspection order. - Test routes that correspond to commands the repository actually declares. - Six bounded risk areas: authentication, billing, automation, data, public API, and dependencies. - Diagnostics for uncertainty, unread content, scan boundaries, excluded matches, and unresolved diffs. @@ -148,7 +201,7 @@ FixMap is deterministic. It narrows investigation; it does not prove that a rank ## MCP server -Expose Plan, Explain, Compare, Verify, and Doctor over local stdio: +Expose Plan, Context, Graph, Explain, Compare, Verify, and Doctor over local stdio: ```bash fixmap mcp @@ -187,7 +240,7 @@ jobs: - uses: actions/checkout@v7 with: fetch-depth: 0 - - uses: aryamthecodebreaker/FixMap@v0.8.9 + - uses: aryamthecodebreaker/FixMap@v0.9.0 with: github-token: ${{ secrets.GITHUB_TOKEN }} ``` diff --git a/action.yml b/action.yml index 93228f2..cb3ff18 100644 --- a/action.yml +++ b/action.yml @@ -1,9 +1,9 @@ name: FixMap -description: Map pull request changes to context files, test routes, and review risks. +description: Map pull request changes to primary context, likely impact, test routes, and review risks. inputs: mode: description: >- - plan (default) maps the change to context files, test routes, and review risks. + plan (default) maps the change to context files, likely impact, test routes, and review risks. verify compares a saved plan against the diff that followed it and needs report-path. Explain and compare are intentionally CLI/MCP-only because Action comments operate on complete reports. required: false diff --git a/apps/web/app/_components/product-map.tsx b/apps/web/app/_components/product-map.tsx index 390cbaf..b3a2f68 100644 --- a/apps/web/app/_components/product-map.tsx +++ b/apps/web/app/_components/product-map.tsx @@ -13,8 +13,8 @@ const outputs = [ }, { icon: CheckCircle, - title: "Checks to run", - items: report.testRoutes.map((route) => route.command) + title: "Likely impact", + items: report.impact?.files.slice(0, 3).map((file) => `${file.path} · ${file.evidence[0]?.kind}`) ?? [] }, { icon: Warning, diff --git a/apps/web/app/_lib/site-data.ts b/apps/web/app/_lib/site-data.ts index 0b90268..f0ff1e4 100644 --- a/apps/web/app/_lib/site-data.ts +++ b/apps/web/app/_lib/site-data.ts @@ -101,6 +101,8 @@ export const commands = { publicIssue: "fixmap plan --issue https://github.com/chalk/chalk/issues/624", localTask: "fixmap plan --issue \"password reset emails fail\"", + context: "fixmap context --issue \"password reset emails fail\" --budget 10000", + graph: "fixmap graph --issue \"password reset emails fail\" --format mermaid", diff: "fixmap plan --diff main...HEAD", verify: "fixmap verify --report plan.json --diff main...HEAD", validate: "fixmap validate plan.json", diff --git a/apps/web/app/changelog/page.tsx b/apps/web/app/changelog/page.tsx index 483d984..50af597 100644 --- a/apps/web/app/changelog/page.tsx +++ b/apps/web/app/changelog/page.tsx @@ -43,10 +43,43 @@ type Release = { }; const releases: Release[] = [ + { + version: "0.9.0", + date: "August 13, 2026", + label: "Latest release", + summary: "Impact Graph, budgeted Context Packs, portable graph export, continuous Watch, compact agent output, and honest repository-local benchmarking form FixMap's first feature release.", + groups: [ + { + label: "Added", + items: [ + "Every plan now separates primary task-ranked context from likely impact backed by imports, reverse dependents, routed tests, and repeated Git co-change evidence.", + "fixmap context packages deterministic source ranges within an explicit estimated-token budget and reports truncation or omissions.", + "fixmap graph exports the Impact Graph as Mermaid or versioned JSON while preserving relationship direction and evidence.", + "fixmap benchmark compares BM25, FixMap, and Impact Graph on identical historical parent snapshots without executing repository code.", + "fixmap watch streams working-tree drift findings and recalculated impact as an agent edits, with Markdown or JSON Lines output.", + "Compact agent output provides a stable edit, inspect, test, risk, avoid, and uncertainty handoff.", + "A 32-second motion-first comparison shows two agents handling the same issue, with an original no-vocals soundtrack and no unsupported savings claim." + ] + }, + { + label: "Improved", + items: [ + "Verify recalculates impact around the actual diff and keeps newly related paths advisory.", + "CLI, MCP, GitHub Action, slash-command discovery, live demo, docs, and website expose the impact evidence consistently." + ] + }, + { + label: "Evidence", + items: [ + "The repository benchmark reports mentioned and unmentioned cohorts, Wilson intervals, raw cases, skip counts, and secondary-file recall.", + "A frozen agent-study protocol is published without claiming an effectiveness result before controlled runs exist." + ] + } + ] + }, { version: "0.8.9", date: "August 11, 2026", - label: "Latest release", summary: "A complete reliability sweep across ranking, verification, agent discovery, workflows, evidence, and the public product surface.", groups: [ { diff --git a/apps/web/app/demo.tsx b/apps/web/app/demo.tsx index f587dcd..cf58109 100644 --- a/apps/web/app/demo.tsx +++ b/apps/web/app/demo.tsx @@ -240,6 +240,18 @@ function PlanPanel({ report }: { report: ReturnType {report.testRoutes[0]!.command} ) : null} + + {report.impact?.files.length ? ( +
+

Likely impact · inspect, not assumed edits

+ {report.impact.files.slice(0, 5).map((file) => ( +

+ {file.confidence} + {file.path} — {file.evidence.map((entry) => entry.reason).join("; ")} +

+ ))} +
+ ) : null} {report.testRoutes[0]?.relatedFiles.length ? (

Nearest test: {report.testRoutes[0]!.relatedFiles[0]} diff --git a/apps/web/app/docs/page.tsx b/apps/web/app/docs/page.tsx index 5e88d15..be42fbe 100644 --- a/apps/web/app/docs/page.tsx +++ b/apps/web/app/docs/page.tsx @@ -14,7 +14,7 @@ import { commands, repoUrl } from "../_lib/site-data"; export const metadata: Metadata = { title: "Documentation", - description: "FixMap documentation for slash-command discovery, planning, explaining, comparing, verifying, validating, MCP, and GitHub Actions.", + description: "FixMap documentation for Impact Graph, repository benchmarking, compact agent output, planning, explaining, comparing, verifying, MCP, and GitHub Actions.", alternates: { canonical: "/docs" }, openGraph: { title: "FixMap documentation", @@ -43,28 +43,36 @@ export default function DocsPage() {

{docLinks.map(({ href, icon: Icon, title, body }) => {title}{body})}
-

Slash command

Open the complete FixMap menu.

fixmap setup installs project-level discovery for Claude Code, Cursor, GitHub Copilot, and Agent Skills. Invoking /fixmap with no task lists Plan, Explain, Compare, Verify, Validate, Doctor, MCP, focus controls, working-tree mapping, and fresh scans.

The installer is idempotent and will not overwrite a customized command unless you explicitly pass --force.

+

Slash command

Open the complete FixMap menu.

fixmap setup installs project-level discovery for Claude Code, Cursor, GitHub Copilot, and Agent Skills. Invoking /fixmap with no task lists Plan, Context, Graph, Explain, Compare, Verify, Watch, Benchmark, Validate, Doctor, MCP, focus controls, working-tree mapping, and fresh scans.

The installer is idempotent and will not overwrite a customized command unless you explicitly pass --force.

-

Plan

Find the right place to start.

Give FixMap one task source: plain issue text, a task file, stdin, a public GitHub issue URL, or a git diff.

Public issue URL

Working-tree or branch diff

Remote repository mode is issue-only. Clone the repository locally when you need --diff, --base, or --head.

+

Plan and Impact Graph

Find the right place to start—and what to inspect next.

Give FixMap one task source: plain issue text, a task file, stdin, a public GitHub issue URL, or a git diff. The primary context remains task-ranked. A separate Impact Graph maps imports, reverse dependents, routed tests, and files that repeatedly changed together in bounded Git history. Impact files are inspection candidates, not instructions to edit every path.

Public issue URL

Working-tree or branch diff

Remote repository mode is issue-only. Clone the repository locally when you need --diff, --base, --head, or complete local Git-history evidence. Shallow and unavailable history are reported explicitly while import and test relationships continue to work.

+ +

Context Pack

Package the relevant source ranges.

fixmap context uses the Plan and Impact Graph to select deterministic source ranges. The default 10,000-token budget is an estimate of one token per four UTF-8 bytes of source; metadata is not charged to that budget. It is reproducible, but it is not a model-specific tokenizer count.

Every snippet records its primary or impact role, reason, confidence, line range, estimated source tokens, and whether the scanner sampled only part of a large file. Files omitted for budget or availability remain listed instead of disappearing silently.

+ +

Graph export

Carry the evidence into a review.

fixmap graph exports imports, reverse dependents, routed tests, and co-change relationships as Mermaid or versioned JSON. Direction is preserved, and only relationships already supported by the Impact Graph are emitted.

Explain

Ask the missing-file question.

Use --explain when you expected a path and it did not appear. The response distinguishes five different situations:

  • The file ranked, just lower than expected.
  • It scored below the report cutoff.
  • It tied for a reported place but fell outside --limit.
  • It was excluded intentionally, such as a generated file whose source ranked instead.
  • The scanner never saw it, including a scan limit, a sparse checkout, or an unsupported extension.

Focus

Narrow the map to what matters.

Demo pages, marketing copy, and documentation often contain every symptom word a product documents, so they compete with the implementation. FixMap knows about conventions like examples/; it cannot know your repository’s own layout.

Patterns can also live in a .fixmapignore file at the repository root, one per line. The two combine, and --explain reports an excluded file as excluded, naming the pattern that matched.

Map what you are editing now

That means staged and unstaged tracked changes against HEAD. Untracked files stay out of the change set unless you add --include-untracked, so scratch metadata is not reported as an edit. They remain ranking candidates either way — a file an agent just wrote is usually the most relevant thing in the repository.

Measure a better task

Refine the wording, re-plan, and see whether the real file moved up:

-

Verify

Compare the plan with the change.

Save a JSON plan before editing, then compare it with the real diff afterwards.

Verify does not run tests or judge correctness. Errors fail by default. Add --fail-on warning when advisory findings, such as a source change without a test change, must also block CI.

+

Verify

Compare the plan with the change.

Save a JSON plan before editing, then compare it with the real diff afterwards. Verify recalculates impact around the files that actually changed and reports strong relationships outside the original plan as advisory inspection notes.

Verify does not run tests or judge correctness. Errors fail by default. Add --fail-on warning when advisory findings, such as a source change without a test change, must also block CI.

+ +

Watch

Keep the map beside the edit.

fixmap watch fingerprints a local Git working tree and emits a fresh verification only when its state changes. Each update recalculates impact around the real edits, so an agent can see unmapped files, missing tests, new risks, and related inspection candidates while it works.

JSON output is newline-delimited for streaming consumers. Use --once for a bounded automation check. Watch reads Git and source text, but never runs repository code or tests.

+ +

Repository benchmark

Measure FixMap where you work.

fixmap benchmark selects recent bounded non-merge commits, checks out each parent in an isolated temporary worktree, derives task text only from the commit metadata, and scores changed maintained source paths rather than generated twins. BM25-over-code, FixMap, and Impact Graph see the identical pre-change scanned corpus.

Results split all, path-mentioned, and unmentioned cohorts and include raw cases and Wilson intervals. Commits that are too large, have no usable task text, or contain no pre-existing target source are skipped with counts. FixMap never runs repository code during this process.

-

Output

Readable by people and tools.

Markdown is the default handoff. Add --format json for structured output and --output <path> to save it. The current issue, comparison, verification, and output files are kept out of ranking, change detection, and cache state, so a saved FixMap report cannot recommend itself.

New JSON plans include reportVersion: 1. Within a report version, fields may be added but existing fields will not be removed or change type; consumers should ignore unknown fields. A breaking output change requires a new report version. Compare and Verify still accept legacy plans without a marker, while rejecting marker values they do not understand.

Context files

Ranked paths, scores, confidence labels, and evidence.

Test routes

Workspace-aware commands and reachable related tests.

Risks

Sensitive areas inferred from paths, symbols, and changes.

Diagnostics

Vague tasks, unresolved identifiers, scan limits, and other uncertainty.

+

Output

Readable by people, agents, and tools.

Markdown is the default handoff. Add --format agent for a compact EDIT CANDIDATE / INSPECT / TEST / RISK / AVOID / UNCERTAINTY handoff, or --format json for structured output. Use --output <path> to save it. The current issue, comparison, verification, and output files are kept out of ranking, change detection, and cache state, so a saved FixMap report cannot recommend itself.

New JSON plans include reportVersion: 1. Within a report version, fields may be added but existing fields will not be removed or change type; consumers should ignore unknown fields. A breaking output change requires a new report version. Compare and Verify still accept legacy plans without a marker, while rejecting marker values they do not understand.

Context files

Task-ranked primary paths, scores, confidence labels, and evidence.

Impact Graph

Dependencies, dependents, routed tests, and repeated co-change relationships to inspect.

Test routes

Workspace-aware commands and reachable related tests.

Risks

Sensitive areas inferred from paths, symbols, and changes.

Diagnostics

Vague tasks, unresolved identifiers, scan limits, history coverage, and other uncertainty.

Validate

Check a saved report directly.

The CLI exposes the same additive structural validator used by Compare, Verify, the GitHub Action, and MCP. It accepts legacy unmarked reports, accepts version 1 with additive fields, and rejects unsupported report versions or malformed context entries.

-

MCP

Five tools for the agent workflow.

fixmap_plan maps tasks and working trees. fixmap_explain answers why a file is missing. fixmap_compare measures task refinement. fixmap_verify checks the later diff, and fixmap_doctor diagnoses install shadows. All five run locally over stdio.

MCP setup examples
+

MCP

Seven tools for the agent workflow.

fixmap_plan maps tasks and working trees. fixmap_context packages source ranges. fixmap_graph exports relationships. fixmap_explain answers why a file is missing. fixmap_compare measures task refinement. fixmap_verify checks the later diff, and fixmap_doctor diagnoses install shadows. All seven run locally over stdio.

MCP setup examples

Doctor

Check what actually started.

doctor reports the running version, resolved path, conflicting global, and Node version. Version 0.8.4 and newer also checks an exact npm-requested version when that newer Doctor starts.

An older project-local binary can win before newer Doctor code runs, so always check the printed running version. Use the isolated-prefix/direct-shim procedure in the README when the exact version matters.

diff --git a/apps/web/app/evidence/page.tsx b/apps/web/app/evidence/page.tsx index 1de4366..24c72c5 100644 --- a/apps/web/app/evidence/page.tsx +++ b/apps/web/app/evidence/page.tsx @@ -123,7 +123,30 @@ export default function EvidencePage() {
BM25 retrieval, code files only{rate(heldoutBaselines.bm25.top1HitRate)}{rate(heldoutBaselines.bm25.top3HitRate)}{rate(heldoutBaselines.bm25.top5HitRate)}
FixMap{rate(heldoutBaselines.fixmap.top1HitRate)}{rate(heldoutBaselines.fixmap.top3HitRate)}{rate(heldoutBaselines.fixmap.top5HitRate)}
-

FixMap does not beat BM25 over code files on repositories it was never tuned against. Top 1 and Top 3 are exact ties — a paired McNemar exact test puts both at p = 1.0, with two disagreements each way. At Top 5 the baseline wins three cases FixMap misses and FixMap wins none: BM25 has the fixing file in its top five for 9 of 9 of these cases, FixMap for 6 of 9. FixMap does lead on the regression suite (69% vs 39% Top 1), but that is the suite whose cases shaped the ranker, and even there the lead is not significant against this baseline. We publish this because it is what the measurement says; closing the Top-5 recall gap is the next piece of work.

+

FixMap does not beat BM25 over code files on repositories it was never tuned against. BM25 leads the Top 1 point estimate 4/9 to 3/9, while Top 3 ties at 5/9; both paired McNemar exact tests have p = 1.0. At Top 5 the baseline wins four cases FixMap misses and FixMap wins none: BM25 has the fixing file in its top five for 9 of 9 of these cases, FixMap for 5 of 9 (p = 0.125). FixMap does lead on the regression suite (69% vs 39% Top 1), but that is the suite whose cases shaped the ranker, and even there the lead is not significant against this baseline. We publish this because it is what the measurement says; closing the Top-5 recall gap is the next piece of work.

+
+ +
+
+

Your repository

Run the baseline comparison locally.

+

fixmap benchmark --repo . --last 50 backtests BM25-over-code, FixMap, and FixMap with Impact Graph on bounded recent Git history.

+
+
+
01

Parent snapshots

Every case runs before its target commit, so the target change and later co-change history are unavailable.

+
02

Identical corpus

All three arms see the same scanned files. The comparison never weakens a baseline with a noisier candidate set.

+
03

Separated cohorts

Tasks that name an expected path are reported separately from tasks that require retrieval.

+
04

Bounded and non-executing

Temporary worktrees, commit and file caps, and no repository code execution keep the run inspectable.

+
+

No universal performance claim is derived from one repository. Commit messages are imperfect task proxies, and historical changes reflect that project’s own maintenance patterns. The JSON output includes every eligible case, skip counts, cohort scores, and Wilson intervals so users can judge the evidence directly.

+
+ +
+
+

Controlled agent study

The protocol exists; the result does not yet.

+

FixMap publishes a frozen four-arm protocol for baseline, available, instructed, and Impact Graph-assisted runs. It requires the same agent model and version, pinned repository revision, complete transcripts, and one run per task and arm.

+
+

No fix-rate, turn-count, token, cost, or time-saved claim is made until real runs are completed and audited. The checked-in evaluator validates the protocol and rejects incomplete or mismatched run sets.

+ Read the frozen protocol
diff --git a/apps/web/app/get-started/page.tsx b/apps/web/app/get-started/page.tsx index 296b188..344e80f 100644 --- a/apps/web/app/get-started/page.tsx +++ b/apps/web/app/get-started/page.tsx @@ -28,7 +28,7 @@ export default function GetStartedPage() {
01

Slash command

Type /fixmap. See every workflow.

-

Install project-level discovery for Claude Code, Cursor, GitHub Copilot, and Agent Skills. Invoking /fixmap without a task opens the complete Plan, Explain, Compare, Verify, Validate, Doctor, MCP, focus, working-tree, and fresh-scan menu.

+

Install project-level discovery for Claude Code, Cursor, GitHub Copilot, and Agent Skills. Invoking /fixmap without a task opens the complete Plan, Context, Graph, Explain, Compare, Watch, Verify, Benchmark, Validate, Doctor, MCP, focus, working-tree, and fresh-scan menu.

The installer is idempotent and refuses to overwrite an existing customized command. Target one integration with --agent claude, cursor, copilot, or agents. Use --force only after reviewing the file.

See the same menu in a terminal

@@ -45,6 +45,18 @@ export default function GetStartedPage() {

Or work inside a local repository

+

Compact agent handoff

+ +

Budgeted source context

+ +

Portable Impact Graph

+ +

Watch an agent's edits

+

Save the JSON plan, then stream drift and recalculated-impact updates whenever the local working tree changes:

+ +

Benchmark this repository

+

Backtest BM25, FixMap, and Impact Graph on historical parent snapshots without running repository code:

+

Pin it to a project instead

Use a project dependency when everyone working on that repository should get the same version:

@@ -84,7 +96,7 @@ npx fixmap plan --issue "password reset emails fail"`} />
03

MCP

Let the agent ask.

-

FixMap exposes five local stdio tools: fixmap_plan before editing, fixmap_explain when a file is missing, fixmap_compare to measure a refined plan, fixmap_verify after the diff exists, and fixmap_doctor to diagnose install shadows.

+

FixMap exposes seven local stdio tools: fixmap_plan, fixmap_context, fixmap_graph, fixmap_explain, fixmap_compare, fixmap_verify, and fixmap_doctor.

Claude Code

Cursor, Windsurf, and other MCP clients

@@ -96,7 +108,7 @@ npx fixmap plan --issue "password reset emails fail"`} /> } } }`} -

Analysis runs locally over stdio. FixMap does not send repository source to a hosted model or service. Plan, Explain, and Verify accept noCache: true for an explicit fresh scan.

+

Analysis runs locally over stdio. FixMap does not send repository source to a hosted model or service. Plan, Context, Graph, Explain, and Verify accept noCache: true for an explicit fresh scan.

diff --git a/apps/web/app/globals.css b/apps/web/app/globals.css index f49f1d0..3a0328c 100644 --- a/apps/web/app/globals.css +++ b/apps/web/app/globals.css @@ -736,6 +736,10 @@ h3 { margin-bottom: 10px; font-size: 21px; line-height: 1.25; letter-spacing: -. .pro-trust-line svg { color: var(--green); } .pro-hero .interactive-map-stage { width: 100%; border-radius: 10px; box-shadow: 0 18px 50px rgba(10, 33, 59, .08); } +.pro-film { padding-block: 24px 104px; display: grid; grid-template-columns: minmax(250px, .5fr) minmax(0, 1.5fr); align-items: end; gap: clamp(48px, 7vw, 104px); } +.pro-film .pro-section-heading { align-self: center; } +.pro-film video { display: block; width: 100%; aspect-ratio: 16 / 9; border: 1px solid var(--line-strong); border-radius: 10px; background: var(--navy); box-shadow: 0 18px 50px rgba(10, 33, 59, .1); } + .pro-workflow, .pro-surfaces { padding-block: 96px; } .pro-workflow { display: grid; grid-template-columns: minmax(250px, .62fr) minmax(0, 1.38fr); gap: clamp(64px, 8vw, 120px); border-top: 1px solid var(--line); } .pro-section-heading { max-width: 520px; } @@ -783,7 +787,7 @@ h3 { margin-bottom: 10px; font-size: 21px; line-height: 1.25; letter-spacing: -. } @media (max-width: 820px) { - .pro-workflow, .pro-surfaces, .pro-proof-inner { grid-template-columns: 1fr; gap: 48px; } + .pro-film, .pro-workflow, .pro-surfaces, .pro-proof-inner { grid-template-columns: 1fr; gap: 48px; } .pro-proof-inner { align-items: start; } .pro-proof-metrics { max-width: 620px; } } @@ -792,6 +796,7 @@ h3 { margin-bottom: 10px; font-size: 21px; line-height: 1.25; letter-spacing: -. .pro-hero { padding-block: 54px 70px; gap: 48px; } .pro-hero-copy h1 { font-size: 50px; } .pro-hero-lede { font-size: 17px; } + .pro-film { padding-bottom: 72px; gap: 30px; } .pro-workflow, .pro-surfaces { padding-block: 72px; } .pro-workflow-list > a { min-height: 128px; grid-template-columns: 34px 1fr 22px; gap: 8px 12px; } .pro-workflow-list p { grid-column: 2 / -1; } diff --git a/apps/web/app/page.tsx b/apps/web/app/page.tsx index 8d030b1..6418c65 100644 --- a/apps/web/app/page.tsx +++ b/apps/web/app/page.tsx @@ -14,16 +14,19 @@ import { InteractiveMapStage } from "./_components/interactive-map-stage"; import { commands, repoUrl, siteStats } from "./_lib/site-data"; const workflow = [ - { number: "01", name: "Plan", detail: "Rank the files, tests, and risks that matter before the first edit.", href: "/product#plan" }, - { number: "02", name: "Explain", detail: "Ask why a file ranked, missed the cutoff, was excluded, or was never scanned.", href: "/product#explain" }, - { number: "03", name: "Compare", detail: "Measure whether a clearer task produced a better context map.", href: "/product#compare" }, - { number: "04", name: "Verify", detail: "Check the completed diff against the plan that guided the work.", href: "/product#verify" } + { number: "01", name: "Plan", detail: "Rank primary context, then map likely impact without pretending every related file must change.", href: "/product#plan" }, + { number: "02", name: "Context", detail: "Package relevant source ranges inside a visible estimated-token budget.", href: "/product#context" }, + { number: "03", name: "Graph", detail: "Export the impact relationships as Mermaid or structured JSON.", href: "/product#graph" }, + { number: "04", name: "Explain", detail: "Ask why a file ranked, missed the cutoff, was excluded, or was never scanned.", href: "/product#explain" }, + { number: "05", name: "Compare", detail: "Measure whether a clearer task produced a better context map.", href: "/product#compare" }, + { number: "06", name: "Watch", detail: "See drift and recalculated impact while the working tree changes.", href: "/product#watch" }, + { number: "07", name: "Verify", detail: "Check the completed diff against the plan that guided the work.", href: "/product#verify" } ]; const surfaces = [ { icon: Command, name: "Slash command", detail: "Install /fixmap in supported coding agents and open the complete workflow menu.", href: "/get-started#slash-command" }, { icon: Laptop, name: "CLI", detail: "Run locally in a terminal with no account or API key.", href: "/get-started#cli" }, - { icon: ShieldCheck, name: "MCP", detail: "Expose Plan, Explain, Compare, Verify, and Doctor to an agent.", href: "/get-started#mcp" }, + { icon: ShieldCheck, name: "MCP", detail: "Expose all seven Plan, Context, Graph, Explain, Compare, Verify, and Doctor tools.", href: "/get-started#mcp" }, { icon: GithubLogo, name: "GitHub Action", detail: "Post the map or verify a saved plan on every pull request.", href: "/get-started#action" } ]; @@ -35,8 +38,8 @@ export default function HomePage() {

FixMap v{siteStats.version} · open source repo intelligence

Start the change with evidence.

- Give FixMap a task, issue, or diff. It returns the files to inspect, the checks to run, - and the risks to review—with every recommendation tied to repository evidence. + Give FixMap a task, issue, or diff. It returns primary context, likely impact, reachable + checks, and risks—with every recommendation tied to repository evidence.

@@ -52,11 +55,23 @@ export default function HomePage() {
+
+
+

FixMap in 32 seconds

+

Same issue. Better first move.

+

Watch two coding agents approach the same task as ranked context, impact evidence, Watch, and Verify change the feedback loop.

+
+ +
+

One workflow

From task to verified diff.

-

Four focused steps, each inspectable on its own.

+

Seven focused steps, each inspectable on its own.

{workflow.map((item) => ( @@ -75,7 +90,7 @@ export default function HomePage() {

Measured in public

Useful without pretending to be certain.

-

FixMap publishes the misses, the cohort boundaries, and the baseline comparison. A map narrows the search; it does not replace reading, tests, or review.

+

FixMap publishes the misses, cohort boundaries, and baseline comparisons. You can also backtest BM25, FixMap, and Impact Graph against your own repository history.

Read the benchmark methodology
diff --git a/apps/web/app/product/page.tsx b/apps/web/app/product/page.tsx index 477ca43..86ef989 100644 --- a/apps/web/app/product/page.tsx +++ b/apps/web/app/product/page.tsx @@ -2,6 +2,7 @@ import type { Metadata } from "next"; import Link from "next/link"; import { ArrowRight, + ChartLineUp, CheckCircle, Eye, FileMagnifyingGlass, @@ -31,12 +32,30 @@ const stages = [ icon: FileMagnifyingGlass, eyebrow: "Before the edit", title: "Plan: find the few places that matter.", - body: "FixMap reads the task and the repository together. It ranks likely context files, attaches the evidence behind each score, and routes the nearest tests it can actually reach.", - details: ["Ranked source files with reasons", "Workspace-aware test commands", "Risk areas and scan diagnostics"] + body: "FixMap reads the task and repository together. It ranks primary context, then builds a separate Impact Graph from imports, reverse dependents, routed tests, and repeated Git co-change evidence.", + details: ["Ranked source files with reasons", "Impact files to inspect, not assumed edits", "Workspace-aware tests, risks, and diagnostics"] }, { - id: "explain", + id: "context", number: "02", + icon: FileMagnifyingGlass, + eyebrow: "Before the agent reads", + title: "Context: send the source, not just its address.", + body: "Context packages deterministic line ranges from primary and impact files inside an estimated source-token budget. Every snippet keeps its role, reason, confidence, range, and truncation state.", + details: ["Markdown or structured JSON", "Visible UTF-8 byte estimate", "Explicit omissions and scanner bounds"] + }, + { + id: "graph", + number: "03", + icon: Path, + eyebrow: "When relationships matter", + title: "Graph: make the evidence portable.", + body: "Graph exports the Impact Graph as Mermaid for review documents or versioned JSON for tools, preserving the direction and reason for every import, dependent, test, and co-change edge.", + details: ["Mermaid or versioned JSON", "Directional relationships", "No invented dependencies"] + }, + { + id: "explain", + number: "04", icon: Eye, eyebrow: "When the map surprises you", title: "Explain: ask why a file is missing.", @@ -45,7 +64,7 @@ const stages = [ }, { id: "compare", - number: "03", + number: "05", icon: Gauge, eyebrow: "When you refine the task", title: "Compare: check whether a better task moved the answer.", @@ -54,12 +73,12 @@ const stages = [ }, { id: "verify", - number: "04", + number: "07", icon: GitDiff, eyebrow: "After the edit", title: "Verify: compare the plan with the real change.", - body: "FixMap checks the saved plan against a git diff. It points out unplanned files, untouched leading context, missing tests, risky areas, and edits in generated or retired locations.", - details: ["Plan versus diff", "Advisory findings by default", "Non-zero only for discarded generated edits"] + body: "FixMap checks the saved plan against a git diff. It points out unplanned files, untouched leading context, missing tests, risky areas, and recalculated impact around the files that actually changed.", + details: ["Plan versus diff", "Recalculated impact", "Advisory findings by default"] } ]; @@ -68,7 +87,7 @@ export default function ProductPage() {

The product

-

One problem.{" "}
Three useful answers.

+

One problem.{" "}
A map that stays useful.

FixMap narrows the first step, explains its reasoning, and checks the work that followed. It is a map you can inspect—not a promise that the map is always right. @@ -79,6 +98,32 @@ export default function ProductPage() {

+
+
+

Measure it locally

Backtest the map on your own history.

+

fixmap benchmark --repo . --last 50 compares BM25-over-code, ordinary FixMap context, and FixMap with Impact Graph against historical parent snapshots.

+
+
+

One candidate corpus

Every arm sees the same scanned files, so a weaker baseline is never manufactured by changing the search space.

+

Pre-change cutoff

Each case is evaluated on its parent snapshot. The target change and later Git history cannot leak into its evidence.

+

Raw cases included

All, mentioned, and unmentioned cohorts plus Wilson intervals make misses and small samples visible.

+

No repository code runs

The benchmark reads Git and source text in temporary worktrees without installing dependencies, running hooks, or executing tests.

+
+
+ +
+
+

While the agent edits

Watch the implementation drift—or stay aligned.

+

fixmap watch --report plan.json --repo . emits only when the working tree changes, then re-runs Verify and recalculates the Impact Graph around the real diff.

+
+
+

Changed states only

A lightweight fingerprint avoids repeating full scans when nothing moved.

+

Drift made visible

Unmapped edits, untouched leading context, and new impact relationships appear as evidence, not verdicts.

+

Agent-ready stream

Markdown stays readable; JSON Lines gives automation one complete record per update.

+

Still local-only

Watch reads Git and source text without running repository code, installing dependencies, or calling a model.

+
+
+
diff --git a/apps/web/app/sample-repo.ts b/apps/web/app/sample-repo.ts index 1ee4ff3..ce60681 100644 --- a/apps/web/app/sample-repo.ts +++ b/apps/web/app/sample-repo.ts @@ -245,7 +245,23 @@ export const sampleRepo: RepoMap = { changedFiles: [], diffText: "", packageManager: "npm", - diagnostics: [] + diagnostics: [], + history: { + inspectedCommits: 8, + skippedLargeCommits: 1, + shallow: false, + truncated: false, + commits: [ + { hash: "1".repeat(40), committedAt: 8, files: ["src/auth/reset-password.ts", "src/auth/token-store.ts", "test/auth/reset-password.test.ts"] }, + { hash: "2".repeat(40), committedAt: 7, files: ["src/auth/reset-password.ts", "src/auth/token-store.ts", "src/http/routes.ts"] }, + { hash: "3".repeat(40), committedAt: 6, files: ["src/auth/reset-password.ts", "test/auth/reset-password.test.ts", "docs/configuration.md"] }, + { hash: "4".repeat(40), committedAt: 5, files: ["src/auth/reset-password.ts", "src/http/routes.ts", "test/auth/reset-password.test.ts"] }, + { hash: "5".repeat(40), committedAt: 4, files: ["src/email/transport.ts", "src/config.ts"] }, + { hash: "6".repeat(40), committedAt: 3, files: ["src/email/transport.ts", "src/config.ts", "test/auth/reset-password.test.ts"] }, + { hash: "7".repeat(40), committedAt: 2, files: ["src/billing/invoice.ts", "src/config.ts"] }, + { hash: "8".repeat(40), committedAt: 1, files: ["README.md"] } + ] + } }; export const samplePaths: string[] = files.map((file) => file.path); diff --git a/apps/web/package.json b/apps/web/package.json index 5bedc7d..d1b2c70 100644 --- a/apps/web/package.json +++ b/apps/web/package.json @@ -15,7 +15,7 @@ "typecheck": "tsc --noEmit" }, "dependencies": { - "@aryam/fixmap-core": "0.8.9", + "@aryam/fixmap-core": "0.9.0", "@phosphor-icons/react": "^2.1.10", "next": "16.2.11", "react": "19.2.7", diff --git a/apps/web/public/fixmap-launch-poster.jpg b/apps/web/public/fixmap-launch-poster.jpg index e4c46c2..fe9d1d8 100644 Binary files a/apps/web/public/fixmap-launch-poster.jpg and b/apps/web/public/fixmap-launch-poster.jpg differ diff --git a/apps/web/public/fixmap-launch.mp4 b/apps/web/public/fixmap-launch.mp4 index 6d3c0f0..190790e 100644 Binary files a/apps/web/public/fixmap-launch.mp4 and b/apps/web/public/fixmap-launch.mp4 differ diff --git a/benchmarks/adversarial/results.json b/benchmarks/adversarial/results.json index 073b1b3..e5c7102 100644 --- a/benchmarks/adversarial/results.json +++ b/benchmarks/adversarial/results.json @@ -13,6 +13,7 @@ "diagnostics": [ "submodules-skipped", "content-too-large", + "impact-history-shallow", "identifier-unverified", "flat-ranking" ], @@ -31,6 +32,7 @@ "grounding": "descriptive", "diagnostics": [ "content-too-large", + "impact-history-shallow", "identifier-unverified", "flat-ranking" ], @@ -49,6 +51,7 @@ "grounding": "descriptive", "diagnostics": [ "content-too-large", + "impact-history-shallow", "identifier-unverified" ], "contextFileCount": 7, @@ -66,6 +69,7 @@ "grounding": "vague", "diagnostics": [ "content-too-large", + "impact-history-shallow", "gated-test-skipped", "vague-task", "flat-ranking" @@ -84,6 +88,7 @@ "topConfidence": null, "grounding": "vague", "diagnostics": [ + "impact-history-shallow", "vague-task", "no-context-match" ], @@ -100,7 +105,9 @@ "maxConfidence": "medium", "topConfidence": "medium", "grounding": "descriptive", - "diagnostics": [], + "diagnostics": [ + "impact-history-shallow" + ], "contextFileCount": 7, "overconfident": false, "groundingOk": true, @@ -115,6 +122,7 @@ "topConfidence": "medium", "grounding": "descriptive", "diagnostics": [ + "impact-history-shallow", "flat-ranking" ], "contextFileCount": 8, @@ -134,6 +142,7 @@ "submodules-skipped", "content-not-utf8", "content-too-large", + "impact-history-shallow", "import-graph-truncated" ], "contextFileCount": 8, @@ -149,6 +158,7 @@ "topConfidence": null, "grounding": "descriptive", "diagnostics": [ + "impact-history-unavailable", "no-context-match" ], "contextFileCount": 0, diff --git a/benchmarks/agent-study/protocol.json b/benchmarks/agent-study/protocol.json new file mode 100644 index 0000000..988728a --- /dev/null +++ b/benchmarks/agent-study/protocol.json @@ -0,0 +1,35 @@ +{ + "protocolVersion": 1, + "status": "protocol-only", + "suite": "navigation-focused pinned tasks", + "arms": [ + "baseline", + "fixmap-available", + "fixmap-instructed", + "fixmap-impact" + ], + "requirements": { + "sameModelVersion": true, + "sameTaskText": true, + "sameRepositoryRevision": true, + "freshContextPerRun": true, + "randomizedArmOrder": true, + "fixedTimeoutAndBudget": true, + "noFixMapChangesMidStudy": true, + "rawTranscriptsRequired": true + }, + "metrics": [ + "taskResolved", + "correctFileInFirstThreeOpened", + "toolCallsToFirstRelevantFile", + "filesOpenedBeforeFirstEdit", + "incorrectFilesEdited", + "totalToolCalls", + "inputTokens", + "outputTokens", + "testsSelectedCorrectly", + "finalPatchAccepted", + "fixmapPlanUsed", + "verifyUsefulWarnings" + ] +} diff --git a/benchmarks/heldout/README.md b/benchmarks/heldout/README.md index 28d3ed9..47e902f 100644 --- a/benchmarks/heldout/README.md +++ b/benchmarks/heldout/README.md @@ -31,17 +31,17 @@ These 12 repositories were selected by the same frozen rule. When a case informs ## Results -Measured 2026-08-04 (Node v24, `rankContextFiles` with a top-5 window): +Verified 2026-08-13 (Node v24, `rankContextFiles` with a top-5 window): Three tasks name their expected fixing file in the issue text. They legitimately test FixMap's explicit-file-mention signal, but they do not test whether it can locate a file the task did not name. The evaluator therefore derives and reports both cohorts every run: | Cohort | Cases | Top-1 | Top-3 | Top-5 | | --- | ---: | ---: | ---: | ---: | -| Task did not name the file | 9 | **4/9 (44.4%)** | **5/9 (55.6%)** | **6/9 (66.7%)** | +| Task did not name the file | 9 | **3/9 (33.3%)** | **5/9 (55.6%)** | **5/9 (55.6%)** | | Task named the file | 3 | 3/3 (100%) | 3/3 (100%) | 3/3 (100%) | -| Pooled, previously published | 12 | 7/12 (58.3%) | 8/12 (66.7%) | 9/12 (75.0%) | +| Pooled | 12 | 6/12 (50.0%) | 8/12 (66.7%) | 8/12 (66.7%) | -**Plan around the unmentioned cohort.** At nine cases its Top-3 95% Wilson interval is 27–81%, so the point estimate is exploratory rather than a precise success probability. The three Top-5 misses — `socketio/socket.io`, `vitejs/vite`, and `vuejs/core` — remain recorded in [`results.json`](results.json) with their actual rankings. The Jest answer is fourth, so it also misses Top-3. +**Plan around the unmentioned cohort.** At nine cases its Top-3 95% Wilson interval is 27–81%, so the point estimate is exploratory rather than a precise success probability. The four Top-5 misses — `jestjs/jest`, `knex/knex`, `vitejs/vite`, and `vuejs/core` — remain recorded in [`results.json`](results.json) with their actual rankings. ## Baseline-relative result @@ -51,9 +51,9 @@ Three tasks name their expected fixing file in the issue text. They legitimately | --- | ---: | ---: | ---: | | Literal keyword retrieval, code files | 2/9 | 4/9 | 6/9 | | BM25 retrieval, code files | **4/9** | **5/9** | **9/9** | -| FixMap | 4/9 | 5/9 | 6/9 | +| FixMap | 3/9 | 5/9 | 5/9 | -FixMap does not beat BM25-over-code on this unseen cohort: Top-1 and Top-3 tie, while BM25 leads 9/9 to 6/9 at Top-5. With nine cases this is not a stable effect-size estimate, but the previously published advantage does not survive the baseline comparison. +FixMap does not beat BM25-over-code on this unseen cohort: BM25 leads Top-1 4/9 to 3/9, Top-3 ties at 5/9, and BM25 leads Top-5 9/9 to 5/9. With nine cases this is not a stable effect-size estimate, but an advantage over naive retrieval is not established. ## Confidence calibration @@ -61,9 +61,9 @@ Both suites record the confidence label on the top-ranked file, so the label can | Top result labeled | Correct | Accuracy | | --- | ---: | ---: | -| high | 7 / 13 | 54% | -| medium | 9 / 11 | 82% | -| low | 2 / 4 | 50% | +| high | 5 / 6 | 83% | +| medium | 11 / 19 | 58% | +| low | 1 / 3 | 33% | The ordering is not monotonic in this small sample, so the labels must not be read as calibrated probabilities. Counts are published so readers can weigh that limitation themselves. Per-suite figures are in each `results.json` under `calibration`. diff --git a/docs/AGENT_STUDY.md b/docs/AGENT_STUDY.md new file mode 100644 index 0000000..1c8084c --- /dev/null +++ b/docs/AGENT_STUDY.md @@ -0,0 +1,33 @@ +# FixMap differential agent study + +FixMap 0.9 includes a frozen, machine-checked four-arm protocol. It does **not** publish an +effectiveness percentage until complete raw runs exist for every task and arm. + +The arms are: + +1. Baseline agent with ordinary repository tools. +2. FixMap available, with no instruction requiring its use. +3. FixMap explicitly instructed before exploration and Verify after editing. +4. The same instructed workflow with the 0.9 Impact Graph and compact agent output. + +Every paired task must use the same model and model version, task text, repository revision, +timeout, and budget. Each run starts with fresh context, arm order is randomized, FixMap is +frozen during the study, and the raw transcript is retained. The primary navigation metric is +tool calls to the first relevant file, not turns to the first edit. + +Validate the protocol without claiming a result: + +```bash +npm run study:agent:check +``` + +When paid or externally metered runs have been authorized and collected, store one JSON object +per line outside the repository and evaluate it explicitly: + +```bash +node scripts/evaluate-agent-study.mjs --input path/to/runs.jsonl +``` + +Run data is deliberately not checked in by default: transcripts can contain source, prompts, +and account metadata. Any public study must use consented, reviewed, redacted artifacts and link +the exact model/version and frozen task-selection record. diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index c421be8..bb8b362 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -2,17 +2,17 @@ ## Cross-repository ranking and efficiency -![FixMap evidence audit: on nine held-out tasks that did not name the fixing file, FixMap and BM25 both ranked it in the top three for five cases, while BM25 led six to nine at Top-5.](assets/fixmap-benchmark.svg) +![FixMap evidence audit: on nine held-out tasks that did not name the fixing file, FixMap and BM25 both ranked it in the top three for five cases, while BM25 led five to nine at Top-5.](assets/fixmap-benchmark.svg) Two suites answer two different questions. The [regression suite](../benchmarks/external/README.md) uses 16 repositories whose cases have guided ranking work, so it measures fit rather than generalization. The [held-out suite](../benchmarks/heldout/README.md) uses 12 further repositories selected by the identical frozen rule and rotates any case that informs a ranking change, so it remains unseen evidence. Each case in both pins the repository state before the fix and freezes the fixing source paths before FixMap ranks anything. -Ranking outputs refreshed 2026-08-04 on Node v24.13.0, Windows 11 (10.0.26200), Intel Core i5-8350U; the scan-time measurement remains from 2026-07-26: +Ranking outputs verified 2026-08-13 on Node v24, Windows 11, Intel Core i5-8350U; the scan-time measurement remains from 2026-07-26: | Quantity | Held-out (12) | Regression (16) | Evidence type | | --- | ---: | ---: | --- | -| Expected fixing file in Top-1 | 7/12 (58%) | 11/16 (69%) | Measured, **pooled — see cohorts below** | +| Expected fixing file in Top-1 | 6/12 (50%) | 11/16 (69%) | Measured, **pooled — see cohorts below** | | Expected fixing file in Top-3 | 8/12 (67%) | 16/16 (100%) | Measured, **pooled — see cohorts below** | -| Expected fixing file in Top-5 | 9/12 (75%) | 16/16 (100%) | Measured, **pooled — see cohorts below** | +| Expected fixing file in Top-5 | 8/12 (67%) | 16/16 (100%) | Measured, **pooled — see cohorts below** | | Median scan + rank time | — | 1,747.7 ms | Measured, three warm runs per pinned repository | **The held-out, unmentioned cohort below is the one to plan around.** The pooled held-out column includes three tasks that name their fixing file, while the regression column describes performance on cases that shaped the ranker. @@ -35,9 +35,9 @@ named; a bare `index.ts` is ordinary prose in an issue and does not. | Suite | Cohort | Cases | Top-1 | Top-3 | Top-5 | | --- | --- | ---: | ---: | ---: | ---: | -| Held-out | Task did not name the file | 9 | **44.4%** (95% CI 19–73%) | **55.6%** (95% CI 27–81%) | 66.7% | +| Held-out | Task did not name the file | 9 | **33.3%** (95% CI 12–65%) | **55.6%** (95% CI 27–81%) | 55.6% | | Held-out | Task named the file | 3 | 100% | 100% | 100% | -| Held-out | Pooled | 12 | 58.3% | 66.7% | 75.0% | +| Held-out | Pooled | 12 | 50.0% | 66.7% | 66.7% | | Regression | Task did not name the file | 13 | 69.2% | 100% | 100% | | Regression | Task named the file | 3 | 66.7% | 100% | 100% | | Regression | Pooled | 16 | 68.8% | 100% | 100% | @@ -93,14 +93,14 @@ Both keyword arms are case-insensitive and expand camelCase, which favours the b | `bm25:raw` | 11.1% | 22.2% | 33.3% | | `bm25:source` | 11.1% | 22.2% | 44.4% | | **`bm25:code`** | **44.4%** | **55.6%** | **100%** | -| `fixmap` | 44.4% | 55.6% | 66.7% | +| `fixmap` | 33.3% | 55.6% | 55.6% | -**FixMap does not beat BM25-over-code on repositories it was never tuned against.** Top-1 and Top-3 -are exact ties (McNemar p = 1.0, two disagreements each way). At Top-5 the baseline wins three cases -FixMap misses and FixMap wins none — BM25 has the fixing file in its top five for 9 of 9 cases, -FixMap for 6 of 9. +**FixMap does not beat BM25-over-code on repositories it was never tuned against.** BM25 leads the +Top-1 point estimate 4/9 to 3/9, while Top-3 is tied at 5/9; both paired comparisons have McNemar +p = 1.0. At Top-5 the baseline wins four cases FixMap misses and FixMap wins none — BM25 has the +fixing file in its top five for 9 of 9 cases, FixMap for 5 of 9 (McNemar p = 0.125). -The three FixMap misses BM25 catches are `socketio/socket.io`, `vitejs/vite` and `vuejs/core`. +The four FixMap misses BM25 catches are `jestjs/jest`, `knex/knex`, `vitejs/vite`, and `vuejs/core`. ### Regression, tasks that did not name the file (13) diff --git a/docs/GROWTH_LOG.md b/docs/GROWTH_LOG.md index 840ea8f..d18d8b1 100644 --- a/docs/GROWTH_LOG.md +++ b/docs/GROWTH_LOG.md @@ -125,3 +125,7 @@ An audit found that 3 of the 12 held-out tasks name their expected fixing file i The same scanned corpora were then ranked with literal keyword retrieval and BM25. On the held-out unmentioned cohort, BM25-over-code ties FixMap at Top-1 and Top-3 and leads 9/9 to 6/9 at Top-5. The paired Top-1 and Top-3 comparisons are exact ties; at this sample size none establishes a stable effect size. **Decision gate:** FixMap does not currently beat the naive retrieval baseline on unseen repositories. Pause distribution built on a “better than search” claim and do not add workflow surface area on that premise. The next evidence work is to expand the mechanically selected unmentioned cohort and improve recall without tuning against cases that remain held out. + +### Recorded-results refresh — 2026-08-13 + +The frozen held-out artifacts now record FixMap at 3/9 Top-1, 5/9 Top-3, and 5/9 Top-5 on the unmentioned cohort. BM25-over-code records 4/9, 5/9, and 9/9 on the identical corpora. Current docs and launch copy use those values; the August 4 paragraphs above remain the historical snapshot from that audit day. diff --git a/docs/LAUNCH_KIT.md b/docs/LAUNCH_KIT.md index c704535..22c25d7 100644 --- a/docs/LAUNCH_KIT.md +++ b/docs/LAUNCH_KIT.md @@ -21,8 +21,8 @@ The report ranks likely files with reasons, suggests test routes, and names risk - CLI, MCP server, and GitHub Action share the same core ranker. - Public GitHub issue URLs supply both task context and the repository in one input; source is scanned in an isolated anonymous shallow checkout that is removed after analysis. - Two frozen evaluations use real fixed issues at pinned pre-fix commits, selected by a mechanical rule. -- **Held-out tasks that did not name the fixing file (9 repositories, never tuned against): top-1 `4/9` (44%), top-3 `5/9` (56%), top-5 `6/9` (67%).** Three further cases named their answer and are reported separately. -- On that same cohort and scanned corpus, BM25-over-code ties FixMap at Top-1 and Top-3 and leads `9/9` to `6/9` at Top-5. FixMap's advantage over naive retrieval is unproven. +- **Held-out tasks that did not name the fixing file (9 repositories, never tuned against): top-1 `3/9` (33%), top-3 `5/9` (56%), top-5 `5/9` (56%).** Three further cases named their answer and are reported separately. +- On that same cohort and scanned corpus, BM25-over-code leads `4/9` to `3/9` at Top-1, ties `5/9` at Top-3, and leads `9/9` to `5/9` at Top-5. FixMap's advantage over naive retrieval is unproven. - Regression (16 repositories, guided development): top-1 `11/16` (69%), top-3 and top-5 `16/16` (100%). - Confidence labels are directional heuristics, not calibrated probabilities; all per-band counts remain public. - An adversarial suite measures false confidence on fabricated identifiers, vague tasks, and absent features: `0.0` across 9 cases. @@ -125,7 +125,7 @@ Points for the maintainer to explain personally: 2. The one-sentence solution: deterministic repo context—ranked files, test routes, risks, and diagnostics. 3. The fastest trial: include the one-input public GitHub issue URL command. 4. The technical mechanism: path/content signals, real git diff signals, bounded static import proximity, file-kind priors, and workspace boundaries. -5. The honest evidence: on nine held-out tasks that did not name their fixing file, FixMap scores `4/9` Top-1 and `5/9` Top-3, exactly tied with BM25-over-code; BM25 leads at Top-5. Link every per-case ranking. +5. The honest evidence: on nine held-out tasks that did not name their fixing file, FixMap scores `3/9` Top-1 and `5/9` Top-3; BM25 scores `4/9` and `5/9`, then leads at Top-5. Link every per-case ranking. 6. The scope: JavaScript/TypeScript today; remote URLs are issue-only; suggested tests are not executed. 7. What the benchmarks did not catch: both suites passed while FixMap could not find chalk's own color-detection code, because a directory blocklist ran after git had already applied `.gitignore` and a frequency cutoff suppressed the word "color" in a library about color. Pointing it at a repository it had never been run on found that; the benchmark never would have. 8. Ask for technical criticism of the evaluation and useful next signals. @@ -146,7 +146,7 @@ These are angles and evidence, not identical copy to syndicate. - Problem: agents spend context and tokens discovering where to start. - Demo: run FixMap first on a public repository, then hand the report to the agent. -- Evidence: deterministic, zero model calls, and inspectable reasons; on nine held-out tasks that did not name the file, FixMap ties BM25-over-code at Top-1 and Top-3 and trails it at Top-5. +- Evidence: deterministic, zero model calls, and inspectable reasons; on nine held-out tasks that did not name the file, FixMap trails BM25-over-code at Top-1 and Top-5 and ties it at Top-3. - Honest caveat: it is a routing aid, not semantic code understanding or a correctness oracle. ### Claude Code and Cursor communities @@ -173,12 +173,14 @@ claude mcp add fixmap -- npx -y @aryam/fixmap@latest mcp 1. One pain sentence. 2. The public repository command. 3. A screenshot or short terminal video of the real output. -4. One evidence sentence: nine held-out pinned bugs whose tasks did not name the file, FixMap tied with BM25-over-code at `4/9` Top-1 and `5/9` Top-3, with every ranking public. +4. One evidence sentence: nine held-out pinned bugs whose tasks did not name the file, FixMap scored `3/9` Top-1 and `5/9` Top-3 versus BM25's `4/9` and `5/9`, with every ranking public. 5. Repository link and a specific feedback question. Avoid generic feature lists and unsupported superlatives. -## Ready-to-post LinkedIn update +## Historical LinkedIn draft (superseded) + +Do not publish the copy below. Use `docs/releases/v0.9.0-social-posts.md`, which contains the reviewed X, LinkedIn, and Show HN drafts for the current release candidate. Coding agents are fast once they know where to work. The expensive mistakes happen before the first edit: opening the wrong module, missing the owning test, or overlooking a risky change. @@ -197,8 +199,8 @@ The latest work includes: - a shared CLI, MCP server, and GitHub Action workflow - a GitHub Marketplace listing for the Action - a production dependency audit gate with no high or critical findings -- 234 automated tests, production builds, smoke tests, and frozen cross-repository and adversarial evaluation gates -- a new 24-second product film on the README and live site +- 727 automated tests, production builds, smoke tests, and frozen cross-repository and adversarial evaluation gates +- a new 32-second product film on the README and live site The evaluation is intentionally public and modest. Across 12 pinned real bugs in repositories the ranker was never tuned against, FixMap ranks an expected file in the top 1 for 7 and in the top 3 for 9. Every per-case ranking is published, including the three misses, and 12 cases are not a general accuracy claim. @@ -210,7 +212,7 @@ Marketplace: https://github.com/marketplace/actions/fixmap npm: https://www.npmjs.com/package/@aryam/fixmap -Release: https://github.com/aryamthecodebreaker/FixMap/releases/tag/v0.7.4 +Release: publish the v0.9.0 URL only after the release exists and is verified I would especially value feedback on the ranking explanations and which repository signals would make FixMap more useful before an agent starts editing. diff --git a/docs/assets/fixmap-cli-demo.svg b/docs/assets/fixmap-cli-demo.svg index ebbf4d0..d30fd1b 100644 --- a/docs/assets/fixmap-cli-demo.svg +++ b/docs/assets/fixmap-cli-demo.svg @@ -1,4 +1,4 @@ - + - + @@ -16,26 +16,31 @@ fixmap plan — examples/tiny-auth-app $ npx @aryam/fixmap plan --issue "password reset emails fail" - FixMap found 1 context file and generated 1 test route. + FixMap found 1 context file, 1 impact file, and generated 1 test route. ## Context Files - src/auth/reset-password.ts (medium confidence, score 28): path matches task terms: reset, password; multiple task terms converge in the file path; content matches task terms: reset, password, email; defines symbols matching task terms: ResetPasswordRequest, buildResetPasswordEmail; auth-related task signal - ## Test Routes - - npm run test: repository root script named test. Related: - test/auth/reset-password.test.ts. - ## Risk Map - - low authentication: ranked files touch authentication; review this area before - editing, but no diff evidence is available yet - ## Changed Files - - None found - ## Analysis - - Task grounding: **descriptive** - - Repository scan: **complete** - - Ranking shape: **separated** - - Next action: Inspect src/auth/reset-password.ts and its routed tests before editing. - ## Diagnostics - - **info** Repository scan caching was bypassed by --no-cache; this report used a fresh - scan. + ## Impact Graph + - test/auth/reset-password.test.ts (high confidence, impact 13): this file imports + src/auth/reset-password.ts; routed test for src/auth/reset-password.ts via npm run test + Inspection order: src/auth/reset-password.ts → test/auth/reset-password.test.ts. + History evidence: 28 eligible commits. + ## Test Routes + - npm run test: repository root script named test. Related: + test/auth/reset-password.test.ts. + ## Risk Map + - low authentication: ranked files touch authentication; review this area before + editing, but no diff evidence is available yet + ## Changed Files + - None found + ## Analysis + - Task grounding: **descriptive** + - Repository scan: **complete** + - Ranking shape: **separated** + - Next action: Inspect src/auth/reset-password.ts and its routed tests before editing. + ## Diagnostics + - **info** Repository scan caching was bypassed by --no-cache; this report used a fresh + scan. diff --git a/docs/assets/fixmap-v0.9.0-agent-comparison.gif b/docs/assets/fixmap-v0.9.0-agent-comparison.gif new file mode 100644 index 0000000..b83e24c Binary files /dev/null and b/docs/assets/fixmap-v0.9.0-agent-comparison.gif differ diff --git a/docs/releases/2026-08-04-benchmark-self-audit.md b/docs/releases/2026-08-04-benchmark-self-audit.md index 3aa268e..1ca533a 100644 --- a/docs/releases/2026-08-04-benchmark-self-audit.md +++ b/docs/releases/2026-08-04-benchmark-self-audit.md @@ -1,5 +1,7 @@ # I audited FixMap's benchmark and found it was leaking answers +> **Update, 2026-08-13:** this article preserves the numbers measured when the audit was published. The current frozen artifacts record the unmentioned cohort at 3/9 Top-1, 5/9 Top-3, and 5/9 Top-5 for FixMap versus 4/9, 5/9, and 9/9 for BM25-over-code. See [`docs/BENCHMARKS.md`](../BENCHMARKS.md) for the current evidence. + FixMap's held-out benchmark was meant to answer one question: when a task does not tell you where to look, does FixMap surface the file that later fixed it? Three of its twelve tasks did tell FixMap where to look. Mongoose named `lib/document.js` with a line number; the Svelte and yargs tasks included the expected path in their text. All three ranked Top-1. They remain valid tests of explicit-file-mention handling, but they cannot count as evidence that FixMap located an unnamed file. diff --git a/docs/releases/v0.9.0-post-release-checklist.md b/docs/releases/v0.9.0-post-release-checklist.md new file mode 100644 index 0000000..0d75d24 --- /dev/null +++ b/docs/releases/v0.9.0-post-release-checklist.md @@ -0,0 +1,14 @@ +# FixMap v0.9.0 post-release checklist + +Run only after the exact verified candidate receives explicit release authorization. + +- [ ] Merge the approved pull request and record the immutable main SHA. +- [ ] Tag `v0.9.0`; verify the tag and GitHub release resolve to that SHA. +- [ ] Verify `@aryam/fixmap-core@0.9.0` and `@aryam/fixmap@0.9.0` from npm, including provenance and the `latest` dist-tag. +- [ ] Install 0.9.0 into a new isolated npm prefix and run `fixmap --version`, `fixmap doctor`, Plan Markdown/JSON/agent output, Context Markdown/JSON, Graph Mermaid/JSON, Verify, Watch `--once`, and a small repository benchmark. +- [ ] Verify `io.github.aryamthecodebreaker/fixmap@0.9.0` in the MCP Registry and complete real public-package MCP Plan, Context, and Graph calls. +- [ ] Verify `aryamthecodebreaker/FixMap@v0.9.0` resolves to the release SHA and run the public Action smoke workflow. +- [ ] Verify the production website reports v0.9.0 and smoke every public route on desktop and mobile. +- [ ] Verify the production launch film returns H.264/AAC media, plays with audio, uses the matching poster, and the README animation resolves from GitHub. +- [ ] Confirm README, npm documentation, changelog, evidence page, Action example, release notes, and public package metadata agree. +- [ ] Publish launch material only after every public artifact is live and rechecked. diff --git a/docs/releases/v0.9.0-release-notes-draft.md b/docs/releases/v0.9.0-release-notes-draft.md new file mode 100644 index 0000000..b9de511 --- /dev/null +++ b/docs/releases/v0.9.0-release-notes-draft.md @@ -0,0 +1,31 @@ +# FixMap v0.9.0 release notes + +FixMap v0.9.0 adds a separate, evidence-backed view of what a change may affect—without turning related files into assumed edits. + +## What is new + +- **Impact Graph:** Plan maps direct imports, reverse dependents, routed tests, and repeated Git co-change relationships around its primary context. Every relationship includes a reason, confidence, and inspection order. +- **Context packs:** `fixmap context --issue "describe the change" --budget 10000` packages deterministic line ranges from primary and impact files as Markdown or JSON. The budget uses an explicit UTF-8 byte estimate, and truncated samples or omitted files stay visible. +- **Portable graph export:** `fixmap graph --issue "describe the change"` emits Mermaid for documents or versioned JSON for tools while preserving relationship direction and evidence. +- **Impact-aware Verify:** after the diff exists, Verify recalculates impact around the files actually changed and surfaces strong relationships outside the original plan as advisory review notes. +- **Repository benchmark:** `fixmap benchmark --repo . --last 50` compares BM25-over-code, FixMap, and FixMap with Impact Graph on identical historical parent snapshots. It scores maintained source rather than generated twins, reports path-mentioned and unmentioned cohorts separately, includes raw cases and Wilson intervals, and never executes repository code. +- **Continuous watch:** `fixmap watch --report plan.json --repo .` notices working-tree state changes, re-runs Verify, and recalculates impact around the real edits. It supports compact JSON Lines output for agents and never executes repository code. +- **Compact agent output:** `fixmap plan --format agent` emits a stable EDIT CANDIDATE / INSPECT / TEST / RISK / AVOID / UNCERTAINTY handoff. +- **Controlled-study harness:** a frozen four-arm protocol and validator are published for future with/without-FixMap agent testing. v0.9.0 makes no effectiveness or time-saved claim before real runs exist. +- **Motion-first launch film:** the README and website include a 32-second, 1080p agent comparison with an original no-vocals music bed. It illustrates less search churn without claiming measured time or token savings. + +The ordinary held-out benchmark is not replaced or rewritten by the new local backtest. Its existing result—including the cases where BM25 leads—remains public. + +## Install after release + +```bash +npm install --global @aryam/fixmap@0.9.0 +fixmap doctor +fixmap plan --issue "describe the change" --format agent +fixmap context --issue "describe the change" --budget 10000 +fixmap graph --issue "describe the change" --format mermaid +``` + +## Release boundary + +These notes are a draft until the exact candidate commit passes the full release matrix and the matching npm packages, MCP Registry entry, GitHub release, Action tag, and production deployment are independently verified. diff --git a/docs/releases/v0.9.0-social-posts.md b/docs/releases/v0.9.0-social-posts.md new file mode 100644 index 0000000..6ac71a6 --- /dev/null +++ b/docs/releases/v0.9.0-social-posts.md @@ -0,0 +1,66 @@ +# FixMap v0.9.0 launch posts + +Publish only after the release, npm packages, Action tag, MCP Registry entry, and production site all resolve to the same verified commit. + +## X + +Two coding agents get the same issue. + +One starts by searching. The other starts with a deterministic map: ranked source, likely impact, bounded context, reachable tests, Watch feedback, and Verify evidence. + +FixMap v0.9.0 is local-first, open source, and makes no model call. + +The benchmark is intentionally honest: on nine held-out tasks that did not name the fixing file, FixMap scores 3/9 Top-1 and 5/9 Top-3; BM25-over-code scores 4/9 and 5/9 on the same corpus. + +Try it: + +`npx -y @aryam/fixmap@latest plan --issue https://github.com/chalk/chalk/issues/624` + +https://github.com/aryamthecodebreaker/FixMap + +Attach: `apps/web/public/fixmap-launch.mp4` + +## LinkedIn + +Coding agents are fast after they find the right part of a repository. FixMap v0.9.0 focuses on the uncertain work before and around that first edit. + +The new workflow can: + +- rank primary context and build an evidence-backed Impact Graph; +- package deterministic source ranges inside a visible context budget; +- export the graph as Mermaid or JSON; +- watch a working tree and recalculate impact as edits change; +- verify the final diff against the saved plan; and +- backtest BM25, FixMap, and Impact Graph on identical historical snapshots. + +It remains local-first: no account, API key, source upload, or model call. + +The evidence is public, including misses. On nine held-out tasks that did not name the fixing file, FixMap scores 3/9 Top-1 and 5/9 Top-3 versus BM25-over-code at 4/9 and 5/9 on the same scanned corpus. That does not prove agent savings, so the release makes no time-saved or token-saved claim. + +Install after release: + +`npm install --global @aryam/fixmap@0.9.0` + +Repository: https://github.com/aryamthecodebreaker/FixMap + +Live demo: https://usefixmap.vercel.app/demo + +Evidence: https://usefixmap.vercel.app/evidence + +Attach: `apps/web/public/fixmap-launch.mp4` + +## Show HN + +Title: Show HN: FixMap 0.9 – deterministic context and impact maps for coding agents + +FixMap is a local-first CLI, MCP server, and GitHub Action that maps an issue, task, or diff to ranked source, likely impact, reachable tests, risks, and explicit uncertainty. + +Version 0.9 adds an evidence-backed Impact Graph, token-budgeted Context Packs, Mermaid/JSON graph export, continuous working-tree Watch, compact agent output, and a repository-local benchmark comparing BM25, FixMap, and Impact Graph on identical pre-change snapshots. + +There is no model call and local source is not uploaded. The held-out result is deliberately modest and fully published: on nine unmentioned-path tasks, FixMap scores 3/9 Top-1 and 5/9 Top-3; BM25-over-code scores 4/9 and 5/9 on the same corpora. I would especially value criticism of the benchmark and suggestions for deterministic repository signals. + +Repository: https://github.com/aryamthecodebreaker/FixMap + +Demo: https://usefixmap.vercel.app/demo + +Evidence: https://usefixmap.vercel.app/evidence diff --git a/docs/releases/v0.9.0-verification.md b/docs/releases/v0.9.0-verification.md new file mode 100644 index 0000000..6fcf320 --- /dev/null +++ b/docs/releases/v0.9.0-verification.md @@ -0,0 +1,54 @@ +# FixMap v0.9.0 verification ledger + +This ledger maps each v0.9.0 promise to implementation evidence and a release gate. It must be completed against the exact candidate commit before release or deployment. + +| Promise | Implementation evidence | Required verification | +| --- | --- | --- | +| Plan returns a separate Impact Graph. | `packages/core/src/impact.ts`, `packages/core/src/report.ts` | Core impact tests, report validation, Markdown/JSON smoke. | +| Direct imports and reverse dependents are directional and explained. | Impact evidence kinds `imports` and `imported-by`. | Synthetic graph unit tests. | +| Context packs stay deterministic and within budget. | `packages/core/src/context.ts`; CLI/MCP Context tools. | Core range/budget tests, CLI/MCP tests, packed-package smoke. | +| Context limitations remain visible. | Per-snippet `sourceTruncated` and pack-level omissions. | Large-sample, empty, and insufficient-budget tests. | +| Graph export preserves evidence and direction. | `packages/core/src/graph.ts`; CLI/MCP Graph tools. | Mermaid/JSON core, CLI, and MCP tests. | +| Routed tests appear as impact evidence. | Plan test routes feed the Impact Graph. | Core impact and workspace-routing tests. | +| Co-change uses only repeated bounded history. | `packages/core/src/repo-scan.ts` and `packages/core/src/impact.ts` | History parsing, single-occurrence rejection, shallow/truncated/unavailable diagnostics. | +| Verify recalculates around changed files. | `packages/core/src/verify.ts` | Verify unit tests plus CLI, MCP, and Action suites. | +| Compact agent output is stable and bounded. | `renderAgentReport` and `--format agent`. | Renderer and CLI/MCP contract tests. | +| Repository benchmark prevents future leakage. | `packages/cli/src/benchmark.ts` temporary parent worktrees. | Synthetic Git-history test, cleanup test, local smoke. | +| Watch detects implementation drift without busy rescans. | `packages/cli/src/watch.ts`; `fixmap watch`. | Fingerprint/change-loop tests, command validation tests, packed CLI `--once` smoke. | +| BM25 and FixMap see the same corpus. | Benchmark calls both arms from one scanned snapshot. | Raw-case assertions and code review. | +| Agent study does not fabricate evidence. | `benchmarks/agent-study/protocol.json`, evaluator, `docs/AGENT_STUDY.md`. | `npm run study:agent:check`; no result claim without complete runs. | +| All integrations expose compatible output. | CLI, seven-tool MCP server, Action, setup templates, browser exports, demo. | Workspace suites, Action bundle check, generated-render check, browser verification. | +| Existing evidence remains honest. | Frozen held-out/external/adversarial records unchanged. | Evaluation gates and diff review. | +| Launch film is truthful and accessible. | `apps/web/public/fixmap-launch.mp4`, poster, README GIF, and launch copy. | HyperFrames validation, codec/audio probe, website playback, and no unsupported savings claim. | +| Published artifacts all match one commit. | Release workflow plus post-release checklist. | Fresh npm install, MCP call, tag/Release/Action SHA, production HTTP and browser smoke. | + +## Candidate verification + +Local candidate: branch `codex/v0.9.0-impact-benchmark`, based on `c9af1e7701c5abf738ae39ec775c10059691e6f3`. + +- [x] Clean `npm ci`: 451 packages installed; zero vulnerabilities. +- [x] Typechecking across Action, CLI, core, and web. +- [x] All 727 workspace tests: 42 Action, 247 CLI, and 438 core. +- [x] Full and production dependency audits: zero vulnerabilities. +- [x] Lint across every workspace. +- [x] Core, CLI, Action, and optimized Next.js production website builds; all 12 static routes generated. +- [x] Action metadata, server manifest, Action bundle, and generated-render parity. +- [x] Packed-package installation into an isolated prefix, plus version, dependency-tree, feature-catalog, and compact Plan smokes. +- [x] CLI, Action, and workspace smoke tests. +- [x] Held-out gate: 12 total cases; 9 unmentioned at 33.3% Top-1, 55.6% Top-3, and 55.6% Top-5. +- [x] Held-out baseline comparison: BM25-over-code is 44.4% Top-1, 55.6% Top-3, and 100% Top-5 on the identical nine-case unmentioned corpora; current public prose matches the recorded artifacts. +- [x] External regression evaluation: 16 cases at 68.8% Top-1 and 93.8% Top-3/Top-5. +- [x] Self-evaluation: baseline 5/8 Top-1, 7/8 Top-3, 8/8 Top-5; issue cohort 8/23 Top-1, 18/23 Top-3, 20/23 Top-5. +- [x] Adversarial gate: 9/9 pass with zero false confidence. +- [x] Agent-study protocol validation, with no run data and no effectiveness claim. +- [x] Scanner benchmark: exactly 1,000 files, ignored directories skipped, no scan-limit diagnostic. +- [x] Repository benchmark smoke with parent cutoff and temporary-worktree cleanup confirmed; hostile required checkout-filter and generated-twin target regressions pass. +- [x] Watch unit and command tests cover changed-state emission, duplicate suppression, retained history, local-only input, and JSON Lines output. +- [x] Final HyperFrames comparison video approved and rendered: 32.0 seconds, five scenes, 1920x1080 at 30 fps, H.264 video, AAC stereo audio at 48 kHz, original no-vocals music, 17 UI cues, zero lint/runtime/layout/motion errors or warnings, and 158/158 sampled text contrast checks. +- [x] README GIF (768x432, 10 fps), website poster, and full 1080p MP4 generated from the approved final timeline; no narration, repeated captions, or measured-savings claim. +- [x] Desktop browser verification of `/`, `/demo`, `/product`, `/docs`, `/evidence`, `/get-started`, and `/changelog`: meaningful content, no framework overlay, no console errors, no horizontal overflow. +- [x] Mobile 390x844 verification of `/`, `/demo`, `/product`, `/docs`, and `/evidence`, plus working mobile-menu navigation. +- [x] Live-demo preset interaction changes the leading context to `src/email/transport.ts` while retaining the Impact Graph. +- [x] Original user checkout remains clean and untouched on branch `0.8.7`. +- [x] Hosted Linux and supported-platform CI against the exact committed SHA: Node 20.11 and 22 on Ubuntu, Node 24 on macOS and Windows, plus the full CI workflow. +- [ ] Explicit release authorization received for the exact CI-verified candidate. diff --git a/examples/reports/declines-fabricated-identifier.md b/examples/reports/declines-fabricated-identifier.md index 1802212..3281e4f 100644 --- a/examples/reports/declines-fabricated-identifier.md +++ b/examples/reports/declines-fabricated-identifier.md @@ -8,12 +8,19 @@ npx -y @aryam/fixmap@latest plan --repo examples/tiny-auth-app \ # FixMap Report -FixMap found 0 context files and generated 0 test routes. +FixMap found 0 context files, 0 impact files, and generated 0 test routes. ## Context Files - None found +## Impact Graph + +- None found + +Inspection order: None. +History evidence: 28 eligible commits. + ## Test Routes - None found diff --git a/examples/reports/declines-unmatched-terms.md b/examples/reports/declines-unmatched-terms.md index bc3bbc6..e3b68d6 100644 --- a/examples/reports/declines-unmatched-terms.md +++ b/examples/reports/declines-unmatched-terms.md @@ -8,12 +8,19 @@ npx -y @aryam/fixmap@latest plan --repo examples/tiny-auth-app \ # FixMap Report -FixMap found 0 context files and generated 0 test routes. +FixMap found 0 context files, 0 impact files, and generated 0 test routes. ## Context Files - None found +## Impact Graph + +- None found + +Inspection order: None. +History evidence: 28 eligible commits. + ## Test Routes - None found diff --git a/examples/reports/declines-vague-task.md b/examples/reports/declines-vague-task.md index d47753b..82247d0 100644 --- a/examples/reports/declines-vague-task.md +++ b/examples/reports/declines-vague-task.md @@ -8,12 +8,19 @@ npx -y @aryam/fixmap@latest plan --repo examples/tiny-auth-app \ # FixMap Report -FixMap found 0 context files and generated 0 test routes. +FixMap found 0 context files, 0 impact files, and generated 0 test routes. ## Context Files - None found +## Impact Graph + +- None found + +Inspection order: None. +History evidence: 28 eligible commits. + ## Test Routes - None found diff --git a/examples/reports/workspace-api-discount.md b/examples/reports/workspace-api-discount.md index 83b40f7..6183389 100644 --- a/examples/reports/workspace-api-discount.md +++ b/examples/reports/workspace-api-discount.md @@ -1,12 +1,20 @@ # FixMap Report -FixMap found 2 context files and generated 3 test routes. +FixMap found 2 context files, 2 impact files, and generated 3 test routes. ## Context Files - `apps/api/src/orders.ts` (high confidence, score 41): path matches task terms: order; content matches task terms: discount, order, total, code; defines task identifiers: orderTotal; task identifier is defined in maintained implementation source - `packages/utils/src/currency.ts` (medium confidence, score 14): content matches task terms: discount, total; defines symbols matching task terms: applyDiscount, discounted +## Impact Graph + +- `apps/api/test/orders.test.ts` (high confidence, impact 13): this file imports apps/api/src/orders.ts; routed test for apps/api/src/orders.ts via pnpm --dir apps/api run test +- `packages/utils/test/currency.test.ts` (high confidence, impact 13): this file imports packages/utils/src/currency.ts; routed test for packages/utils/src/currency.ts via pnpm --dir packages/utils run test + +Inspection order: `apps/api/src/orders.ts` → `packages/utils/src/currency.ts` → `apps/api/test/orders.test.ts` → `packages/utils/test/currency.test.ts`. +History evidence: 63 eligible commits. + ## Test Routes - `pnpm --dir apps/api run test`: nearest package (apps/api) script named test. Related: `apps/api/test/orders.test.ts`. diff --git a/examples/reports/workspace-utils-rounding.md b/examples/reports/workspace-utils-rounding.md index 18f8338..dc33470 100644 --- a/examples/reports/workspace-utils-rounding.md +++ b/examples/reports/workspace-utils-rounding.md @@ -1,11 +1,18 @@ # FixMap Report -FixMap found 1 context file and generated 2 test routes. +FixMap found 1 context file, 1 impact file, and generated 2 test routes. ## Context Files - `packages/utils/src/currency.ts` (high confidence, score 37): path matches task terms: currency; content matches task terms: round, cent; defines task identifiers: roundToCents; task identifier is defined in maintained implementation source +## Impact Graph + +- `packages/utils/test/currency.test.ts` (high confidence, impact 13): this file imports packages/utils/src/currency.ts; routed test for packages/utils/src/currency.ts via pnpm --dir packages/utils run test + +Inspection order: `packages/utils/src/currency.ts` → `packages/utils/test/currency.test.ts`. +History evidence: 63 eligible commits. + ## Test Routes - `pnpm --dir packages/utils run test`: nearest package (packages/utils) script named test. Related: `packages/utils/test/currency.test.ts`. diff --git a/package-lock.json b/package-lock.json index b762d57..e81f7e1 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "fixmap-workspace", - "version": "0.8.9", + "version": "0.9.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "fixmap-workspace", - "version": "0.8.9", + "version": "0.9.0", "license": "MIT", "workspaces": [ "packages/*", @@ -30,7 +30,7 @@ "name": "@fixmap/web", "version": "0.0.0", "dependencies": { - "@aryam/fixmap-core": "0.8.9", + "@aryam/fixmap-core": "0.9.0", "@phosphor-icons/react": "^2.1.10", "next": "16.2.11", "react": "19.2.7", @@ -8843,18 +8843,18 @@ }, "packages/action": { "name": "@fixmap/action", - "version": "0.8.9", + "version": "0.9.0", "license": "MIT", "dependencies": { - "@aryam/fixmap-core": "0.8.9" + "@aryam/fixmap-core": "0.9.0" } }, "packages/cli": { "name": "@aryam/fixmap", - "version": "0.8.9", + "version": "0.9.0", "license": "MIT", "dependencies": { - "@aryam/fixmap-core": "0.8.9", + "@aryam/fixmap-core": "0.9.0", "@modelcontextprotocol/sdk": "1.30.0" }, "bin": { @@ -8866,7 +8866,7 @@ }, "packages/core": { "name": "@aryam/fixmap-core", - "version": "0.8.9", + "version": "0.9.0", "license": "MIT", "devDependencies": {}, "engines": { diff --git a/package.json b/package.json index 6cab516..55b91ee 100644 --- a/package.json +++ b/package.json @@ -1,8 +1,8 @@ { "name": "fixmap-workspace", - "version": "0.8.9", + "version": "0.9.0", "private": true, - "description": "Local-first repo context for coding agents: paste a GitHub issue URL to get ranked files, test routes, and risks.", + "description": "Local-first repo intelligence for coding agents: ranked context, likely impact, test routes, risks, and repository benchmarks.", "license": "MIT", "type": "module", "repository": { @@ -51,7 +51,7 @@ "benchmark:check": "npm run build:core && node scripts/benchmark-scan.mjs --tier 1000 --check", "benchmark:savings": "npm run build:core && node scripts/benchmark-savings.mjs", "benchmark:savings:record": "npm run build:core && node scripts/benchmark-savings.mjs --record", - "ci": "npm run typecheck && npm test && npm run audit:all && npm run lint && npm run build:ci && npm run check:action-metadata && npm run check:server-manifest && npm run check:action-bundle && npm run check:rendered && npm run smoke && npm run evaluate && npm run evaluate:adversarial:gate && npm run benchmark:check", + "ci": "npm run typecheck && npm test && npm run audit:all && npm run lint && npm run build:ci && npm run check:action-metadata && npm run check:server-manifest && npm run check:action-bundle && npm run check:rendered && npm run smoke && npm run study:agent:check && npm run evaluate && npm run evaluate:adversarial:gate && npm run benchmark:check", "evaluate": "npm run build:core && node scripts/evaluate.mjs", "evaluate:external": "npm run build:core && node scripts/evaluate-external.mjs", "evaluate:external:record": "npm run build:core && node scripts/evaluate-external.mjs --record", @@ -67,7 +67,8 @@ "evaluate:adversarial:gate": "npm run build:core && node scripts/evaluate-adversarial.mjs --gate --check-recorded", "evaluate:adversarial:record": "npm run build:core && node scripts/evaluate-adversarial.mjs --record", "render:examples": "npm run build:core && node scripts/render-honest-examples.mjs", - "check:rendered": "node scripts/check-generated.mjs rendered" + "check:rendered": "node scripts/check-generated.mjs rendered", + "study:agent:check": "node scripts/evaluate-agent-study.mjs" }, "engines": { "node": ">=20.11" diff --git a/packages/action/action.yml b/packages/action/action.yml index e27b6e4..8b422e4 100644 --- a/packages/action/action.yml +++ b/packages/action/action.yml @@ -1,9 +1,9 @@ name: FixMap -description: Map pull request changes to context files, test routes, and review risks. +description: Map pull request changes to primary context, likely impact, test routes, and review risks. inputs: mode: description: >- - plan (default) maps the change to context files, test routes, and review risks. + plan (default) maps the change to context files, likely impact, test routes, and review risks. verify compares a saved plan against the diff that followed it and needs report-path. Explain and compare are intentionally CLI/MCP-only because Action comments operate on complete reports. required: false diff --git a/packages/action/dist/index.mjs b/packages/action/dist/index.mjs index 6fe4e76..8dc84e5 100644 --- a/packages/action/dist/index.mjs +++ b/packages/action/dist/index.mjs @@ -1180,6 +1180,116 @@ function addEdge(edges, from, to) { } } +// packages/core/dist/impact.js +var DEFAULT_IMPACT_LIMIT = 12; +var MAX_IMPACT_SEEDS = 3; +var MIN_CO_CHANGE_OCCURRENCES = 2; +function buildImpactMap(repo, requestedSeeds, testRoutes = [], limit = DEFAULT_IMPACT_LIMIT) { + const repositoryPaths = new Set(repo.files.map((file) => file.path)); + const seeds = [...new Set(requestedSeeds)].filter((path) => repositoryPaths.has(path)).slice(0, MAX_IMPACT_SEEDS); + const seedSet = new Set(seeds); + const candidates = /* @__PURE__ */ new Map(); + const addEvidence = (path, score, evidence) => { + if (seedSet.has(path) || !repositoryPaths.has(path) || isGeneratedPath(path) || isBackupPath(path)) + return; + const current = candidates.get(path) ?? { path, score: 0, evidence: [] }; + if (!current.evidence.some((entry) => entry.kind === evidence.kind && entry.seed === evidence.seed)) { + current.evidence.push(evidence); + current.score += score; + } + candidates.set(path, current); + }; + const graph = buildImportGraph(repo.files); + for (const seed of seeds) { + for (const imported of [...graph.imports.get(seed) ?? []].sort((a, b) => a.localeCompare(b))) { + addEvidence(imported, 4, { + kind: "imports", + seed, + reason: `${seed} imports this file` + }); + } + for (const importer of [...graph.importedBy.get(seed) ?? []].sort((a, b) => a.localeCompare(b))) { + addEvidence(importer, 6, { + kind: "imported-by", + seed, + reason: `this file imports ${seed}` + }); + } + } + for (const route of testRoutes.filter((entry) => entry.kind === "test")) { + for (const path of route.relatedFiles) { + const seed = nearestSeed(path, seeds) ?? seeds[0]; + if (!seed) + continue; + addEvidence(path, 7, { + kind: "test-route", + seed, + reason: `routed test for ${seed} via ${route.command}` + }); + } + } + const history = repo.history; + if (history) { + for (const seed of seeds) { + const seedCommits = history.commits.filter((commit) => commit.files.includes(seed)); + const coOccurrences = /* @__PURE__ */ new Map(); + for (const commit of seedCommits) { + for (const path of commit.files) { + if (path !== seed && repositoryPaths.has(path)) { + coOccurrences.set(path, (coOccurrences.get(path) ?? 0) + 1); + } + } + } + for (const [path, occurrences] of coOccurrences) { + if (occurrences < MIN_CO_CHANGE_OCCURRENCES) + continue; + const strength = occurrences / Math.max(seedCommits.length, 1); + const score = Math.min(8, 2 + Math.round(strength * 6)); + addEvidence(path, score, { + kind: "co-change", + seed, + reason: `changed alongside ${seed} in ${occurrences} of its ${seedCommits.length} eligible ${seedCommits.length === 1 ? "change" : "changes"}`, + occurrences, + seedChanges: seedCommits.length + }); + } + } + } + const files = [...candidates.values()].map(toImpactFile).sort((left, right) => right.score - left.score || left.path.localeCompare(right.path)).slice(0, Math.max(0, limit)); + return { + seeds, + files, + inspectionOrder: [...seeds, ...files.map((file) => file.path)], + history: { + available: Boolean(history), + eligibleCommits: history?.commits.length ?? 0, + shallow: history?.shallow ?? false, + truncated: history?.truncated ?? false + } + }; +} +function toImpactFile(candidate) { + const kinds = new Set(candidate.evidence.map((entry) => entry.kind)); + const strongestCoChange = candidate.evidence.filter((entry) => entry.kind === "co-change").reduce((best, entry) => Math.max(best, (entry.occurrences ?? 0) / Math.max(entry.seedChanges ?? 1, 1)), 0); + const confidence = kinds.has("test-route") || kinds.size >= 2 || strongestCoChange >= 0.6 ? "high" : kinds.has("imported-by") || kinds.has("imports") || strongestCoChange >= 0.3 ? "medium" : "low"; + return { + path: candidate.path, + score: candidate.score, + confidence, + evidence: candidate.evidence.sort((left, right) => left.kind.localeCompare(right.kind) || left.seed.localeCompare(right.seed)) + }; +} +function nearestSeed(path, seeds) { + const pathParts = path.split("/"); + return [...seeds].map((seed) => { + const seedParts = seed.split("/"); + let common = 0; + while (common < pathParts.length && common < seedParts.length && pathParts[common] === seedParts[common]) + common += 1; + return { seed, common }; + }).sort((left, right) => right.common - left.common || left.seed.localeCompare(right.seed))[0]?.seed; +} + // packages/core/dist/rank.js var DEPLOYMENT_TERMS = [ "deploy", @@ -1812,12 +1922,14 @@ function buildReportFromRepo(repo, input) { const contextPaths = contextFiles.map((file) => file.path); const testRoutes = buildTestRoutes(repo, contextPaths); const routedTestPaths = [...new Set(testRoutes.flatMap((route) => route.relatedFiles))]; + const impact = buildImpactMap(repo, contextPaths, testRoutes); return { reportVersion: 1, - summary: buildSummary(contextFiles.length, testRoutes.length), + summary: buildSummary(contextFiles.length, testRoutes.length, impact.files.length), contextFiles, testRoutes, risks: buildRiskNotes(contextPaths, repo.changedFiles), + impact, changedFiles: repo.changedFiles, diagnostics: [ ...repo.diagnostics, @@ -2123,10 +2235,11 @@ function findRelatedTests(repo, contextPaths) { }).filter((file) => file.score > 0).sort((a, b) => b.score - a.score || a.path.localeCompare(b.path)).map((file) => file.path); return [...changedTests, ...overlapping].slice(0, 8); } -function buildSummary(contextFileCount, testRouteCount) { +function buildSummary(contextFileCount, testRouteCount, impactFileCount = 0) { const files = contextFileCount === 1 ? "context file" : "context files"; const routes = testRouteCount === 1 ? "test route" : "test routes"; - return `FixMap found ${contextFileCount} ${files} and generated ${testRouteCount} ${routes}.`; + const impact = impactFileCount === 1 ? "impact file" : "impact files"; + return `FixMap found ${contextFileCount} ${files}, ${impactFileCount} ${impact}, and generated ${testRouteCount} ${routes}.`; } function renderMarkdownReport(report) { const lines = [ @@ -2138,6 +2251,15 @@ function renderMarkdownReport(report) { "", ...listOrEmpty(report.contextFiles.map((file) => `- ${markdownCode(file.path)} (${file.confidence} confidence, score ${file.score}): ${file.reasons.join("; ")}`)), "", + "## Impact Graph", + "", + ...listOrEmpty((report.impact?.files ?? []).map((file) => `- ${markdownCode(file.path)} (${file.confidence} confidence, impact ${file.score}): ${file.evidence.map((entry) => entry.reason).join("; ")}`)), + ...report.impact ? [ + "", + `Inspection order: ${report.impact.inspectionOrder.map(markdownCode).join(" \u2192 ") || "None"}.`, + `History evidence: ${report.impact.history.available ? `${report.impact.history.eligibleCommits.toLocaleString()} eligible commits${report.impact.history.shallow ? " (shallow)" : ""}${report.impact.history.truncated ? " (bounded)" : ""}` : "not available; import and test evidence only"}.` + ] : [], + "", "## Test Routes", "", ...listOrEmpty(report.testRoutes.map((route) => { @@ -2237,8 +2359,11 @@ var MAX_TEXT_SAMPLE_BYTES = 64e3; var MAX_DIFF_TEXT_CHARS = 2e5; var MAX_SCANNED_FILES = 25e3; var GIT_MAX_BUFFER = 10 * 1024 * 1024; +var GIT_HISTORY_MAX_BUFFER = 24 * 1024 * 1024; +var MAX_HISTORY_COMMITS = 1e3; +var MAX_HISTORY_FILES_PER_COMMIT = 30; var exec = promisify(execFile); -var SCAN_CACHE_VERSION = 3; +var SCAN_CACHE_VERSION = 4; var SCAN_CACHE_MAX_AGE_MS = 7 * 24 * 60 * 60 * 1e3; var SCAN_CACHE_MAX_FUTURE_SKEW_MS = 5 * 60 * 1e3; var SCAN_CACHE_FILE = /^[a-f0-9]{24}-[a-f0-9]{24}\.json$/; @@ -2264,7 +2389,7 @@ async function scanRepo(input) { const internalPaths = await resolveInternalPaths(repoRoot, input.internalExclude ?? []); const cacheRoot = configuredScanCacheRoot(); const internalCacheRoot = sameFilesystemPath(cacheRoot, repoRoot) || containedPath(repoRoot, cacheRoot) !== void 0 ? cacheRoot : void 0; - const cacheDecision = input.useCache === true ? await buildScanCacheLocation(repoRoot, cacheRoot, internalPaths) : void 0; + const cacheDecision = input.useCache === true ? await buildScanCacheLocation(repoRoot, cacheRoot, internalPaths, input.includeHistory === true) : void 0; const cacheLocation = cacheDecision?.location; if (input.useCache === false) { diagnostics.push({ @@ -2284,11 +2409,13 @@ async function scanRepo(input) { let trackedFiles; let packageScripts; let packageManager; + let history; if (cached) { files = cached.files; trackedFiles = cached.trackedFiles; packageScripts = cached.packageScripts; packageManager = cached.packageManager; + history = cached.history ?? void 0; diagnostics.push(...cached.diagnostics, { code: "cache-hit", severity: "info", @@ -2299,6 +2426,7 @@ async function scanRepo(input) { trackedFiles = await listTrackedPaths(repoRoot, internalPaths); packageScripts = await readPackageScripts(repoRoot, files, diagnostics); packageManager = detectPackageManager(files, diagnostics); + history = input.includeHistory === true ? await readRepositoryHistory(repoRoot, new Set(files.map((file) => file.path)), diagnostics) : void 0; if (cacheLocation) { await writeScanCache(cacheLocation, { version: SCAN_CACHE_VERSION, @@ -2308,12 +2436,17 @@ async function scanRepo(input) { trackedFiles, packageScripts, packageManager, - diagnostics: [...diagnostics] + diagnostics: [...diagnostics], + history: history ?? null }); } } const diffSpec = resolveDiffSpec(input); const diff = input.workingTree ? await readWorkingTree(repoRoot, input.includeUntracked === true, diagnostics, internalPaths) : await readDiff(repoRoot, diffSpec, diagnostics, internalPaths); + const orderedDiagnostics = [ + ...diagnostics.filter((entry) => !entry.code.startsWith("impact-history-")), + ...diagnostics.filter((entry) => entry.code.startsWith("impact-history-")) + ]; return { root: repoRoot, files, @@ -2322,7 +2455,8 @@ async function scanRepo(input) { changedFiles: diff.changedFiles, diffText: diff.diffText, packageManager, - diagnostics + diagnostics: orderedDiagnostics, + ...history ? { history } : {} }; } function configuredScanCacheRoot() { @@ -2353,7 +2487,7 @@ function hasInternalPath(paths, path) { function gitPathspec(internalPaths) { return ["--", ".", ...[...internalPaths].sort((a, b) => a.localeCompare(b)).map((path) => `:(exclude,literal)${path}`)]; } -async function buildScanCacheLocation(root, cacheRoot, internalPaths) { +async function buildScanCacheLocation(root, cacheRoot, internalPaths, includeHistory) { if (sameFilesystemPath(cacheRoot, root) || containedPath(root, cacheRoot) !== void 0) { return { skipReason: "Repository scan caching was skipped because FIXMAP_CACHE_DIR is inside the scanned repository. Move the cache outside the repository to enable exact-state reuse." @@ -2382,6 +2516,7 @@ async function buildScanCacheLocation(root, cacheRoot, internalPaths) { head.trim(), status, dirtyDiff, + includeHistory ? "history" : "no-history", ...[...internalPaths].sort((a, b) => a.localeCompare(b)) ].join("\0")); return { location: { @@ -2398,13 +2533,23 @@ async function readScanCache(location) { try { const cached = JSON.parse(await readFile(location.path, "utf8")); const createdAt = typeof cached.createdAt === "string" ? Date.parse(cached.createdAt) : Number.NaN; - if (cached.version !== SCAN_CACHE_VERSION || cached.stateKey !== location.stateKey || typeof cached.createdAt !== "string" || !Number.isFinite(createdAt) || Date.now() - createdAt > SCAN_CACHE_MAX_AGE_MS || createdAt - Date.now() > SCAN_CACHE_MAX_FUTURE_SKEW_MS || !Array.isArray(cached.files) || !cached.files.every(isCachedRepoFile) || !Array.isArray(cached.trackedFiles) || !cached.trackedFiles.every(isCachedRelativePath) || !Array.isArray(cached.packageScripts) || !cached.packageScripts.every(isCachedPackageScript) || !Array.isArray(cached.diagnostics) || !cached.diagnostics.every(isCachedDiagnostic) || !["npm", "pnpm", "yarn", "bun"].includes(cached.packageManager ?? "")) + if (cached.version !== SCAN_CACHE_VERSION || cached.stateKey !== location.stateKey || typeof cached.createdAt !== "string" || !Number.isFinite(createdAt) || Date.now() - createdAt > SCAN_CACHE_MAX_AGE_MS || createdAt - Date.now() > SCAN_CACHE_MAX_FUTURE_SKEW_MS || !Array.isArray(cached.files) || !cached.files.every(isCachedRepoFile) || !Array.isArray(cached.trackedFiles) || !cached.trackedFiles.every(isCachedRelativePath) || !Array.isArray(cached.packageScripts) || !cached.packageScripts.every(isCachedPackageScript) || !Array.isArray(cached.diagnostics) || !cached.diagnostics.every(isCachedDiagnostic) || !(cached.history === null || isCachedHistory(cached.history)) || !["npm", "pnpm", "yarn", "bun"].includes(cached.packageManager ?? "")) return void 0; return cached; } catch { return void 0; } } +function isCachedHistory(candidate) { + if (!isRecord(candidate) || !Array.isArray(candidate.commits) || typeof candidate.inspectedCommits !== "number" || !Number.isSafeInteger(candidate.inspectedCommits) || candidate.inspectedCommits < 0 || typeof candidate.skippedLargeCommits !== "number" || !Number.isSafeInteger(candidate.skippedLargeCommits) || candidate.skippedLargeCommits < 0 || typeof candidate.shallow !== "boolean" || typeof candidate.truncated !== "boolean") { + return false; + } + return candidate.commits.every((commit) => { + if (!isRecord(commit) || typeof commit.hash !== "string" || !/^[a-f0-9]{40}$/i.test(commit.hash) || typeof commit.committedAt !== "number" || !Number.isSafeInteger(commit.committedAt) || commit.committedAt < 0 || !Array.isArray(commit.files)) + return false; + return commit.files.every(isCachedRelativePath); + }); +} function isCachedRepoFile(candidate) { if (!isRecord(candidate)) return false; @@ -2923,6 +3068,89 @@ async function readWorkingTree(repoRoot, includeUntracked, diagnostics, internal } var NOT_A_GIT_CHECKOUT = "this directory is not a git checkout. Ranking still works from the task text; --diff, --base/--head and --working-tree need a repository with history."; var NO_GIT_HISTORY = "this repository has no commits yet, so there is nothing to diff against. Commit the initial work first, or run with --issue alone to rank from the task text."; +async function readRepositoryHistory(root, repositoryPaths, diagnostics) { + try { + const [{ stdout: shallowText }, { stdout: countText }, { stdout: logText }] = await Promise.all([ + exec("git", ["rev-parse", "--is-shallow-repository"], { cwd: root, maxBuffer: GIT_MAX_BUFFER }), + exec("git", ["rev-list", "--count", "--no-merges", "HEAD"], { cwd: root, maxBuffer: GIT_MAX_BUFFER }), + exec("git", [ + "-c", + "core.quotepath=false", + "log", + "--no-merges", + "-n", + String(MAX_HISTORY_COMMITS), + "--format=%x1e%H%x1f%ct", + "--name-only", + "-z", + "HEAD" + ], { cwd: root, maxBuffer: GIT_HISTORY_MAX_BUFFER }) + ]); + const parsed = parseHistoryLog(logText, repositoryPaths); + const totalCommits = Number.parseInt(countText.trim(), 10); + const shallow = shallowText.trim() === "true"; + const truncated = Number.isFinite(totalCommits) && totalCommits > parsed.inspectedCommits; + const history = { + commits: parsed.commits, + inspectedCommits: parsed.inspectedCommits, + skippedLargeCommits: parsed.skippedLargeCommits, + shallow, + truncated + }; + if (shallow) { + diagnostics.push({ + code: "impact-history-shallow", + severity: "info", + message: `Impact history is shallow (${parsed.inspectedCommits.toLocaleString()} visible non-merge ${parsed.inspectedCommits === 1 ? "commit" : "commits"}). Import and test relationships remain available, but co-change evidence may be incomplete.` + }); + } + if (truncated) { + diagnostics.push({ + code: "impact-history-truncated", + severity: "info", + message: `Impact history inspected the newest ${parsed.inspectedCommits.toLocaleString()} of ${totalCommits.toLocaleString()} non-merge commits. Commits touching more than ${MAX_HISTORY_FILES_PER_COMMIT} files were excluded from co-change evidence.` + }); + } + return history; + } catch (error) { + const checkoutState = isMissingGit(error) ? void 0 : await describeGitCheckout(root); + diagnostics.push({ + code: "impact-history-unavailable", + severity: "info", + message: checkoutState === "not-repository" ? "Impact history is unavailable because this directory is not a Git checkout; import and test relationships are still reported." : checkoutState === "no-history" ? "Impact history is unavailable because this repository has no commits; import and test relationships are still reported." : `Impact history could not be read (${truncateForDiagnostic(gitErrorDetail(error), DIAGNOSTIC_SPEC_LIMIT * 2)}); import and test relationships are still reported.` + }); + return void 0; + } +} +function parseHistoryLog(logText, repositoryPaths) { + const commits = []; + let inspectedCommits = 0; + let skippedLargeCommits = 0; + for (const record of logText.split("")) { + if (!record) + continue; + const fields = record.split("\0"); + const header = fields.shift()?.replace(/^\r?\n/, "") ?? ""; + const separator = header.indexOf(""); + if (separator === -1) + continue; + const hash = header.slice(0, separator).trim(); + const committedAt = Number.parseInt(header.slice(separator + 1).trim(), 10); + if (!/^[a-f0-9]{40}$/i.test(hash) || !Number.isSafeInteger(committedAt) || committedAt < 0) + continue; + inspectedCommits += 1; + const allFiles = [...new Set(fields.map((path) => path.replace(/^\r?\n/, "")).filter(Boolean).map(normalizePath))]; + if (allFiles.length > MAX_HISTORY_FILES_PER_COMMIT) { + skippedLargeCommits += 1; + continue; + } + const currentFiles = allFiles.filter((path) => repositoryPaths.has(path)); + if (currentFiles.length === 0) + continue; + commits.push({ hash, committedAt, files: currentFiles }); + } + return { commits, inspectedCommits, skippedLargeCommits }; +} async function describeGitCheckout(root) { try { const { stdout } = await exec("git", ["rev-parse", "--is-inside-work-tree"], { cwd: root, maxBuffer: GIT_MAX_BUFFER }); @@ -3043,7 +3271,10 @@ function normalizePath(path) { // packages/core/dist/plan.js async function buildFixMapReport(input) { - const repo = await scanRepo(input); + return (await buildFixMapAnalysis(input)).report; +} +async function buildFixMapAnalysis(input) { + const repo = await scanRepo({ ...input, includeHistory: input.includeHistory !== false }); const requestedExclude = await resolveExclusions(input.repoRoot, input.exclude ?? []); const internalExclude = buildPathExcluder((input.internalExclude ?? []).map((pattern) => normalizeAbsolutePattern(input.repoRoot, pattern))); const exclude = combineExclusions(requestedExclude, internalExclude); @@ -3072,7 +3303,7 @@ async function buildFixMapReport(input) { }); } } - return report; + return { report, repo }; } function combineExclusions(primary, internal) { if (internal.patterns.length === 0) @@ -3209,6 +3440,16 @@ function verifyPlan(report, repo) { message: suggested.length > 0 ? `Code changed but no test did. The plan routed ${suggested.length === 1 ? "this test" : "these tests"} as most related.` : report.testRoutes.length > 0 ? `Code changed but no test did. Run the routed ${report.testRoutes.length === 1 ? "command" : "commands"}: ${report.testRoutes.map((route) => route.command).join(", ")}.` : "Code changed but no test did, and the plan found no related test to point at." }); } + const impact = buildImpactMap(repo, changed, report.testRoutes); + const highImpactOutsidePlan = impact.files.filter((entry) => entry.confidence === "high" && !planned.has(entry.path) && !changed.includes(entry.path) && !isTest(entry.path)); + if (highImpactOutsidePlan.length > 0) { + findings.push({ + code: "impact-file-unreviewed", + severity: "info", + paths: highImpactOutsidePlan.slice(0, 8).map((entry) => entry.path), + message: `${highImpactOutsidePlan.length === 1 ? "One high-evidence impact file is" : `${highImpactOutsidePlan.length} high-evidence impact files are`} outside both the original plan and this diff. They are not required edits, but inspect the recorded import/history evidence before finishing.` + }); + } const plannedAreas = new Set(report.risks.map((risk) => risk.area)); const newRisks = buildRiskNotes(changed, changed).filter((risk) => !plannedAreas.has(risk.area)); for (const risk of newRisks) { @@ -3223,7 +3464,8 @@ function verifyPlan(report, repo) { summary: buildVerifySummary(changed.length, findings), changedFiles: changed, findings, - diagnostics: repo.diagnostics + diagnostics: repo.diagnostics, + impact }; } function buildVerifySummary(changedCount, findings) { @@ -3264,10 +3506,24 @@ function renderVerifyMarkdown(result) { } lines.push("", "## Changed Files", ""); lines.push(...result.changedFiles.length > 0 ? result.changedFiles.map((path) => `- ${markdownCode(path)}`) : ["- None found"]); + if (result.impact) { + lines.push("", "## Recalculated Impact", ""); + lines.push(...result.impact.files.length > 0 ? result.impact.files.map((file) => `- ${markdownCode(file.path)} (${file.confidence} confidence): ${file.evidence.map((entry) => entry.reason).join("; ")}`) : ["- None found"]); + } return `${lines.join("\n")} `; } +// packages/core/dist/retrieval.js +var STOPWORDS = new Set(`a about above after again against all am an and any are as at be because been before being +below between both but by can cannot could did do does doing down during each few for from further had has have having +he her here hers him his how i if in into is it its itself just me more most my no nor not of off on once only or other +ought our out over own same she should so some such than that the their them then there these they this those through +to too under until up very was we were what when where which while who whom why with would you your +bug issue issues error errors expected actual behavior behaviour reproduce reproduction steps version versions node npm +report repo repository description example code please thanks title type severity confidence location line lines +following above below see also would should could may might must will can also using used use uses`.split(/\s+/)); + // packages/core/dist/validate.js function validateFixMapReport(candidate, label) { if (typeof candidate !== "object" || candidate === null || Array.isArray(candidate) || !Array.isArray(candidate.contextFiles)) { @@ -3360,6 +3616,21 @@ function validateFixMapReport(candidate, label) { message: `${label} has an invalid risks entry at index ${invalidRisk}; each risk needs a non-empty string "area", and optional reason and severity fields must use their documented types.` }; } + if (record.impact !== void 0) { + const impact = record.impact; + const history = isRecord2(impact) ? impact.history : void 0; + if (!isRecord2(impact) || !isRepositoryRelativePathArray(impact.seeds) || !Array.isArray(impact.files) || !isRepositoryRelativePathArray(impact.inspectionOrder) || !isRecord2(history) || typeof history.available !== "boolean" || typeof history.eligibleCommits !== "number" || !Number.isSafeInteger(history.eligibleCommits) || history.eligibleCommits < 0 || typeof history.shallow !== "boolean" || typeof history.truncated !== "boolean") { + return { success: false, message: `${label} has an invalid impact graph envelope.` }; + } + const invalidImpact = impact.files.findIndex((file) => { + if (!isRecord2(file) || !isRepositoryRelativePath(file.path) || typeof file.score !== "number" || !Number.isFinite(file.score) || file.score < 0 || file.confidence !== "high" && file.confidence !== "medium" && file.confidence !== "low" || !Array.isArray(file.evidence)) + return true; + return file.evidence.some((evidence) => !isRecord2(evidence) || !["imports", "imported-by", "co-change", "test-route"].includes(String(evidence.kind)) || !isRepositoryRelativePath(evidence.seed) || typeof evidence.reason !== "string" || !evidence.reason.trim() || evidence.occurrences !== void 0 && (!Number.isSafeInteger(evidence.occurrences) || evidence.occurrences < 0) || evidence.seedChanges !== void 0 && (!Number.isSafeInteger(evidence.seedChanges) || evidence.seedChanges < 0)); + }); + if (invalidImpact !== -1) { + return { success: false, message: `${label} has an invalid impact.files entry at index ${invalidImpact}.` }; + } + } if (!isRepositoryRelativePathArray(record.changedFiles)) { return { success: false, message: `${label} has invalid changedFiles; every entry must be a safe repository-relative path.` }; } @@ -3794,6 +4065,7 @@ async function runVerifyMode(context) { workingTree: context.workingTree, includeUntracked: context.includeUntracked, useCache: !context.noCache, + includeHistory: true, internalExclude: [resolve3(repoRoot, reportPath)] }); const diffFailure = repo.diagnostics.find((diagnostic) => diagnostic.code === "diff-unavailable"); diff --git a/packages/action/package.json b/packages/action/package.json index a94c9bd..2079474 100644 --- a/packages/action/package.json +++ b/packages/action/package.json @@ -1,6 +1,6 @@ { "name": "@fixmap/action", - "version": "0.8.9", + "version": "0.9.0", "description": "GitHub Action wrapper for FixMap pull request reports.", "private": true, "license": "MIT", @@ -11,6 +11,6 @@ "typecheck": "tsc -p tsconfig.json --noEmit" }, "dependencies": { - "@aryam/fixmap-core": "0.8.9" + "@aryam/fixmap-core": "0.9.0" } } diff --git a/packages/action/src/runner.ts b/packages/action/src/runner.ts index da95288..e4e1b04 100644 --- a/packages/action/src/runner.ts +++ b/packages/action/src/runner.ts @@ -225,6 +225,7 @@ async function runVerifyMode(context: VerifyModeContext): Promise { workingTree: context.workingTree, includeUntracked: context.includeUntracked, useCache: !context.noCache, + includeHistory: true, internalExclude: [resolve(repoRoot, reportPath)] }); const diffFailure = repo.diagnostics.find((diagnostic) => diagnostic.code === "diff-unavailable"); diff --git a/packages/cli/README.md b/packages/cli/README.md index bdecdf1..d479af5 100644 --- a/packages/cli/README.md +++ b/packages/cli/README.md @@ -2,7 +2,7 @@ **Give your coding agent a map before it starts editing.** -FixMap turns an issue, prompt, or git diff into ranked context files, test routes, and risk notes — with no account or API key and no source upload. +FixMap turns an issue, prompt, or git diff into ranked primary context, an evidence-backed Impact Graph, test routes, and risk notes — with no account or API key and no source upload. ## Quick start @@ -48,30 +48,66 @@ Machine-readable output: fixmap plan --base main --head HEAD --format json --output fixmap-report.json ``` +Compact output for an agent context window: + +```bash +fixmap plan --issue "password reset emails fail" --format agent +``` + +Backtest BM25, FixMap, and Impact Graph against pre-change snapshots from your own repository: + +```bash +fixmap benchmark --repo . --last 50 +``` + +Build a deterministic source pack for an agent, bounded by an estimated source-token budget: + +```bash +fixmap context --issue "password reset emails fail" --budget 10000 +``` + +Export the Impact Graph for an issue as Mermaid or JSON: + +```bash +fixmap graph --issue "password reset emails fail" --format mermaid +``` + +Continuously compare agent edits with a saved plan and recalculate impact: + +```bash +fixmap watch --report plan.json --repo . --include-untracked +``` + Public GitHub issue, pull request, and repository URL modes are available in the CLI and MCP server for issue-only analysis. FixMap fetches task context anonymously, shallow-clones the default branch into an isolated temporary directory, disables credentials and repository execution surfaces, and removes the checkout before returning. Clone locally to use `--diff`, `--base`, `--head`, or working-tree inputs. For long task text, use `--issue-file task.md` or pipe text to `--issue -`. A leading `@` in `--issue` is ordinary task text; only the explicit file flag reads from disk. A one-off `npx -y @aryam/fixmap@latest ...` run is also available, but npm may choose an existing project-local FixMap first. Run `fixmap doctor`, treat its printed running version as authoritative, and update or remove a stale install. For a reproducible clean test, install the exact version into an isolated npm prefix and invoke that prefix's `fixmap` shim directly; the repository README includes complete PowerShell and POSIX commands. ## Complete feature catalog -- **Plan** — rank context files, related tests, test commands, risks, changed files, diagnostics, grounding, confidence, and a next action from task text, a task file, stdin, a public GitHub issue or pull request, a branch diff, or the working tree. +- **Plan** — rank primary context, then map likely impact from imports, reverse dependents, related tests, and repeated Git co-change relationships. Impact paths are inspection candidates, not assumed edits. +- **Context Pack** — use `fixmap context` to package deterministic line ranges from primary and impact files within an estimated source-token budget. Markdown is readable by people and agents; JSON preserves roles, reasons, line ranges, truncation, and omitted-file diagnostics. +- **Graph export** — use `fixmap graph` to export the evidence-backed Impact Graph as portable Mermaid or versioned JSON while preserving relationship direction. - **Explain** — use `--explain ` to distinguish ranked, below-cutoff, tie-truncated, excluded, and not-scanned paths. - **Compare** — use `--compare ` to measure how a refined task changed ranks, scores, confidence, and grounding. -- **Verify** — compare a saved plan with the completed diff or working tree; errors fail by default and `--fail-on warning` provides an opt-in strict CI gate without pretending FixMap ran tests or proved correctness. +- **Verify** — compare a saved plan with the completed diff or working tree and recalculate impact around the files actually changed; errors fail by default and `--fail-on warning` provides an opt-in strict CI gate without pretending FixMap ran tests or proved correctness. +- **Repository benchmark** — use `fixmap benchmark --repo . --last 50` to compare BM25, FixMap, and Impact Graph on identical parent-snapshot corpora. Primary hits use maintained source rather than generated twins, and repository code is never executed. +- **Watch** — use `fixmap watch --report plan.json --repo .` to emit drift findings and a recalculated Impact Graph whenever the working tree changes. JSON format is newline-delimited for agent consumers. - **Validate** — run `fixmap validate ` to check report compatibility without writing custom JavaScript. - **Focus controls** — cap output with `--limit`, repeat `--exclude`, or use ordered `.fixmapignore` patterns with negation. Pasted absolute paths inside the repository are normalized, and unmatched patterns produce a warning. - **Live changes** — `--working-tree` maps staged and unstaged tracked edits; `--include-untracked` opts new files into the changed-file set. - **Exact-state cache** — clean and tracked dirty git states are cached by repository, commit, status, and binary diff. Cache hits report age, entries expire after seven days, and `--no-cache` reports a fresh bypass. - **Artifact isolation** — current issue, comparison, verification, and output files are removed from ranking, change detection, and cache state, so a saved plan cannot recommend or invalidate itself. - **Doctor** — report the running version, resolved binary, global/PATH shadows, Node compatibility, and an optionally requested npm version. -- **MCP** — expose Plan, Explain, Compare, Verify, and Doctor as five local stdio tools. +- **MCP** — expose Plan, Context, Graph, Explain, Compare, Verify, and Doctor as seven local stdio tools. - **Slash-command discovery** — `fixmap setup` installs `/fixmap`; `fixmap features` prints the same complete catalog in Markdown or JSON. - **Safe repository handling** — public repositories use isolated temporary checkouts with credentials, hooks, filters, and submodule recursion disabled. Local source is read without installing dependencies or running repository scripts. -- **Human and machine output** — Markdown is the default; JSON reports carry `reportVersion: 1` and follow the documented additive compatibility policy. +- **Human, agent, and machine output** — Markdown is the default; `--format agent` is a compact handoff, and JSON reports carry `reportVersion: 1` with the documented additive compatibility policy. ## MCP server -FixMap ships five Model Context Protocol tools: `fixmap_plan`, `fixmap_explain`, `fixmap_compare`, `fixmap_verify`, and `fixmap_doctor`. Plan, Explain, and Verify accept `noCache: true` when an agent needs a fresh repository scan rather than an exact-state cache hit. +FixMap ships seven Model Context Protocol tools: `fixmap_plan`, `fixmap_context`, `fixmap_graph`, `fixmap_explain`, `fixmap_compare`, `fixmap_verify`, and `fixmap_doctor`. Plan, Context, Graph, Explain, and Verify accept `noCache: true` when an agent needs a fresh repository scan rather than an exact-state cache hit. + +Context budgets use a deterministic estimate of one token per four UTF-8 bytes of source. Metadata does not consume that source budget. Scanner sample limits are reported per snippet with `sourceTruncated`, so consumers can distinguish a complete file from a bounded sample. Claude Code: @@ -96,7 +132,11 @@ Cursor, Windsurf, or any MCP client: ```text fixmap plan Generate a FixMap report for a task or diff +fixmap context Build a budgeted Markdown or JSON source pack +fixmap graph Export the Impact Graph as Mermaid or JSON fixmap verify Compare a saved plan with the diff that followed +fixmap benchmark Backtest BM25, FixMap, and Impact Graph on local Git history +fixmap watch Recheck working-tree drift and impact whenever edits change fixmap doctor Report the resolved version and any shadowing install fixmap validate Validate a saved FixMap JSON report fixmap features List every FixMap capability in Markdown or JSON @@ -116,7 +156,8 @@ fixmap mcp Run FixMap as an MCP server over stdio --exclude Leave paths out of ranking; repeatable, gitignore-flavored --no-cache Bypass the exact-state repository scan cache --repo Local path, file:// URL, or public GitHub HTTPS/SSH URL ---format Output format: markdown (default) or json +--format Plan: markdown, agent, or json; context: markdown or json; graph: mermaid or json +--budget Context estimated source-token budget, 256 to 200000 (default 10000) --output Write the report to a file instead of stdout --fail-on Verify exit policy: error (default) or warning ``` @@ -127,6 +168,9 @@ fixmap mcp Run FixMap as an MCP server over stdio ## Context Files - src/auth/reset-password.ts (high confidence): path and content match +## Impact Graph +- src/session.ts (high confidence): imported by the primary edit candidate + ## Test Route - npm --prefix apps/api run test diff --git a/packages/cli/package.json b/packages/cli/package.json index 8418bf5..555374b 100644 --- a/packages/cli/package.json +++ b/packages/cli/package.json @@ -1,8 +1,8 @@ { "name": "@aryam/fixmap", - "version": "0.8.9", + "version": "0.9.0", "mcpName": "io.github.aryamthecodebreaker/fixmap", - "description": "Local-first CLI and MCP server mapping GitHub issue URLs, tasks, and diffs to ranked files, tests, and risks.", + "description": "Local-first CLI and MCP server mapping tasks and diffs to ranked context, likely impact, tests, and risks.", "license": "MIT", "repository": { "type": "git", @@ -46,7 +46,7 @@ "typecheck": "tsc -p tsconfig.json --noEmit" }, "dependencies": { - "@aryam/fixmap-core": "0.8.9", + "@aryam/fixmap-core": "0.9.0", "@modelcontextprotocol/sdk": "1.30.0" }, "engines": { diff --git a/packages/cli/src/agent-setup.ts b/packages/cli/src/agent-setup.ts index 0727695..f21915f 100644 --- a/packages/cli/src/agent-setup.ts +++ b/packages/cli/src/agent-setup.ts @@ -7,12 +7,15 @@ export type AgentTarget = "claude" | "cursor" | "copilot" | "agents"; export const FIXMAP_FEATURES = [ { name: "Plan", command: "fixmap plan", detail: "Rank context files, test routes, risks, changed files, and uncertainty from a task, issue URL, diff, or working tree." }, + { name: "Impact Graph", command: "fixmap plan", detail: "Map likely dependents, dependencies, tests, and repeated Git co-change relationships around the primary context, with explicit evidence." }, + { name: "Context Pack", command: "fixmap context --issue --budget 10000", detail: "Select task-aware source ranges from primary and impact files within a deterministic token budget." }, + { name: "Graph export", command: "fixmap graph --issue --format mermaid", detail: "Export imports, reverse dependents, routed tests, and co-change evidence as Mermaid or JSON." }, { name: "Explain", command: "fixmap plan --explain ", detail: "Show whether a path ranked, tied below the limit, was excluded, or was never scanned." }, { name: "Compare", command: "fixmap plan --compare ", detail: "Compare a refined task and current plan with an earlier JSON report." }, { name: "Verify", command: "fixmap verify --report ", detail: "Compare the completed diff or working tree with the saved plan; add --fail-on warning for a strict CI gate." }, { name: "Validate", command: "fixmap validate ", detail: "Check a saved report against FixMap's structural compatibility contract." }, { name: "Doctor", command: "fixmap doctor", detail: "Diagnose stale local, global, PATH, and npx install shadows." }, - { name: "MCP", command: "fixmap mcp", detail: "Expose Plan, Explain, Compare, Verify, and Doctor over local stdio." }, + { name: "MCP", command: "fixmap mcp", detail: "Expose Plan, Context, Graph, Explain, Compare, Verify, and Doctor over local stdio." }, { name: "Public tasks", command: "fixmap owner/repository#123", detail: "Fetch public GitHub issue or pull-request text anonymously and scan its repository in an isolated checkout." }, { name: "Repository sources", command: "--repo --ref ", detail: "Map a local checkout, file URL, directory archive, or a named branch or tag from a public GitHub repository." }, { name: "Task files", command: "--issue-file ", detail: "Read long task text from UTF-8, UTF-16, or stdin, including BOM-less UTF-16 from common Windows tools." }, @@ -20,6 +23,9 @@ export const FIXMAP_FEATURES = [ { name: "Live changes", command: "--working-tree --include-untracked", detail: "Map staged, unstaged, and optionally untracked work against HEAD." }, { name: "Fresh scan", command: "--no-cache", detail: "Bypass the exact git-state cache with CLI --no-cache, MCP noCache: true, or Action no-cache: true, and report that a fresh scan was used." }, { name: "Machine output", command: "--format json --output ", detail: "Emit a versioned JSON contract or readable Markdown without executing repository code." }, + { name: "Compact agent output", command: "--format agent", detail: "Emit EDIT CANDIDATE, INSPECT, TEST, RISK, AVOID, and UNCERTAINTY sections for a small context window." }, + { name: "Repository benchmark", command: "fixmap benchmark --repo . --last 50", detail: "Backtest BM25, FixMap, and Impact Graph on historical parent snapshots with history cut off before each target change." }, + { name: "Watch", command: "fixmap watch --report plan.json --repo .", detail: "Continuously verify working-tree drift and recalculate impact as an agent edits." }, { name: "Test routing", command: "fixmap plan", detail: "Detect package, workspace, and language test commands, related tests, and skipped or gated suites." }, { name: "Risk and diagnostics", command: "fixmap plan", detail: "Report bounded risk areas, grounding quality, unread content, scan limits, package-manager conflicts, and unresolved diffs." }, { name: "GitHub Action", command: "uses: aryamthecodebreaker/FixMap@", detail: "Plan or verify pull requests with bounded summaries, outputs, and one updated comment." }, @@ -48,9 +54,11 @@ When this command is invoked without a task, run \`fixmap features\` and present When the invocation includes a task, issue URL, diff, file path, or workflow name: 1. Run \`fixmap features\` if the requested capability is ambiguous. -2. Use the matching local command: Plan, Explain, Compare, Verify, Validate, Doctor, or MCP. -3. Preserve the user's repository and never imply that FixMap ran tests or proved correctness; it produces a starting map and verification findings. -4. Report the exact command used and summarize files, checks, risks, and diagnostics. +2. Use the matching local command: Plan, Context, Graph, Explain, Compare, Verify, Watch, Benchmark, Validate, Doctor, or MCP. +3. Read the Impact Graph as files to inspect, not a claim that each file must change. Preserve each relationship's evidence. +4. Prefer \`--format agent\` when context is constrained, \`fixmap watch\` while an agent is editing, and \`fixmap benchmark\` when the user asks whether FixMap works on this repository. +5. Preserve the user's repository and never imply that FixMap ran tests or proved correctness; it produces a starting map and verification findings. +6. Report the exact command used and summarize files, impact, checks, risks, and diagnostics. Prefer \`fixmap plan --issue "$ARGUMENTS" --repo .\` for task text. Use a canonical public GitHub issue URL directly when one is provided.`; diff --git a/packages/cli/src/analysis-commands.ts b/packages/cli/src/analysis-commands.ts new file mode 100644 index 0000000..d64645f --- /dev/null +++ b/packages/cli/src/analysis-commands.ts @@ -0,0 +1,184 @@ +import { writeFile } from "node:fs/promises"; +import { homedir } from "node:os"; +import { resolve } from "node:path"; +import { + buildFixMapGraph, + renderContextPackMarkdown, + renderFixMapGraphMermaid, + type ContextPack, + type FixMapGraph +} from "@aryam/fixmap-core"; +import { analyzeRepository, contextFromAnalysis, type AnalysisSourceInput, type AnalyzedRepository } from "./analysis-source.js"; +import { isSafeGitRefName, parseRepositorySource, tryParseGitHubIssueSource } from "./repository-source.js"; + +const CONTEXT_USAGE = `Usage: fixmap context --issue [--repo ] [--budget <256-200000>] [--format markdown|json] [--output ]\n fixmap context --working-tree [--include-untracked] [--repo ] [--budget ]\n\nBuilds a deterministic, task-aware source package from primary and impact files. The budget applies to estimated source tokens; FixMap never executes repository code or calls a model.\n`; +const GRAPH_USAGE = `Usage: fixmap graph --issue [--repo ] [--format mermaid|json] [--output ]\n fixmap graph --working-tree [--include-untracked] [--repo ]\n\nExports the evidence-backed Impact Graph as Mermaid or structured JSON.\n`; + +type CommandIo = { + stdout: (text: string) => void; + stderr: (text: string) => void; + writeOutput?: (path: string, contents: string) => Promise; + analyze?: (input: AnalysisSourceInput) => Promise; +}; + +type ParsedAnalysisArgs = AnalysisSourceInput & { + format: string; + output?: string | undefined; + budget: number; +}; + +export async function runContextCommand(args: string[], io: CommandIo): Promise { + if (args[0] === "--help" || args[0] === "-h") { io.stdout(CONTEXT_USAGE); return 0; } + const options = parseAnalysisArgs(args, "context", io.stderr); + if (!options) return 1; + try { + const analysis = await (io.analyze ?? analyzeRepository)({ ...options, internalExclude: options.output ? [options.output] : [] }); + const pack = contextFromAnalysis(analysis, options.budget); + const rendered = options.format === "json" ? `${JSON.stringify(pack, null, 2)}\n` : renderContextPackMarkdown(pack); + await emit(rendered, options.output, io); + return pack.snippets.length > 0 ? 0 : 1; + } catch (error) { + io.stderr(`${error instanceof Error ? error.message : String(error)}\n`); + return 1; + } +} + +export async function runGraphCommand(args: string[], io: CommandIo): Promise { + if (args[0] === "--help" || args[0] === "-h") { io.stdout(GRAPH_USAGE); return 0; } + const options = parseAnalysisArgs(args, "graph", io.stderr); + if (!options) return 1; + try { + const analysis = await (io.analyze ?? analyzeRepository)({ ...options, internalExclude: options.output ? [options.output] : [] }); + const graph = buildFixMapGraph(analysis.report); + const rendered = options.format === "json" ? `${JSON.stringify(graph, null, 2)}\n` : renderFixMapGraphMermaid(graph); + await emit(rendered, options.output, io); + return graph.nodes.length > 0 ? 0 : 1; + } catch (error) { + io.stderr(`${error instanceof Error ? error.message : String(error)}\n`); + return 1; + } +} + +function parseAnalysisArgs( + args: string[], + command: "context" | "graph", + stderr: (text: string) => void +): ParsedAnalysisArgs | undefined { + const usage = command === "context" ? CONTEXT_USAGE : GRAPH_USAGE; + const valueFlags = new Set([ + "--issue", "--repo", "--ref", "--format", "--output", "--limit", "--exclude", "--diff", "--base", "--head", + ...(command === "context" ? ["--budget"] : []) + ]); + const booleanFlags = new Set(["--working-tree", "--include-untracked", "--no-cache"]); + const seen = new Set(); + const options: ParsedAnalysisArgs = { + format: command === "context" ? "markdown" : "mermaid", + budget: 10_000, + useCache: true, + exclude: [] + }; + + for (let index = 0; index < args.length; index += 1) { + const raw = args[index]!; + const separator = raw.indexOf("="); + const flag = separator === -1 ? raw : raw.slice(0, separator); + const inline = separator === -1 ? undefined : raw.slice(separator + 1); + if (!valueFlags.has(flag) && !booleanFlags.has(flag)) { + stderr(`Unknown ${command} option: ${raw}\n\n${usage}`); + return undefined; + } + if (flag !== "--exclude" && seen.has(flag)) { + stderr(`Pass ${flag} only once.\n\n${usage}`); + return undefined; + } + seen.add(flag); + if (booleanFlags.has(flag)) { + if (inline !== undefined) { stderr(`${flag} does not take a value.\n\n${usage}`); return undefined; } + if (flag === "--working-tree") options.workingTree = true; + else if (flag === "--include-untracked") options.includeUntracked = true; + else options.useCache = false; + continue; + } + const following = args[index + 1]; + const value = inline ?? (following && !following.startsWith("-") ? following : undefined); + if (inline === undefined && value !== undefined) index += 1; + if (!value?.trim()) { stderr(`${flag} requires a value.\n\n${usage}`); return undefined; } + const normalized = value.trim(); + if (flag === "--issue") options.issueText = normalized; + else if (flag === "--repo") options.repo = expandHomePath(normalized); + else if (flag === "--ref") { + if (!isSafeGitRefName(normalized)) { + stderr(`--ref requires a safe branch or tag name.\n\n${usage}`); + return undefined; + } + options.checkoutRef = normalized; + } + else if (flag === "--output") options.output = expandHomePath(normalized); + else if (flag === "--diff") options.diffSpec = normalized; + else if (flag === "--base") options.baseRef = normalized; + else if (flag === "--head") options.headRef = normalized; + else if (flag === "--exclude") options.exclude!.push(normalized); + else if (flag === "--format") options.format = normalized.toLowerCase(); + else { + const parsed = Number(normalized); + const minimum = flag === "--budget" ? 256 : 1; + const maximum = flag === "--budget" ? 200_000 : 20; + if (!/^\d+$/.test(normalized) || !Number.isSafeInteger(parsed) || parsed < minimum || parsed > maximum) { + stderr(`${flag} must be a whole number from ${minimum} to ${maximum}.\n\n${usage}`); + return undefined; + } + if (flag === "--budget") options.budget = parsed; + else options.limit = parsed; + } + } + + const formats = command === "context" ? ["markdown", "json"] : ["mermaid", "json"]; + if (!formats.includes(options.format)) { + stderr(`--format must be ${formats.join(" or ")}.\n\n${usage}`); + return undefined; + } + if (!options.issueText && !options.diffSpec && !options.baseRef && !options.workingTree) { + stderr(`${command} needs --issue, --diff, --base/--head, or --working-tree.\n\n${usage}`); + return undefined; + } + if (options.includeUntracked && !options.workingTree) { + stderr(`--include-untracked only applies with --working-tree.\n\n${usage}`); + return undefined; + } + if (options.workingTree && (options.diffSpec || options.baseRef || options.headRef)) { + stderr(`Use either --working-tree or --diff/--base, not both.\n\n${usage}`); + return undefined; + } + if (options.diffSpec && (options.baseRef || options.headRef)) { + stderr(`Use either --diff or --base/--head, not both.\n\n${usage}`); + return undefined; + } + if (options.headRef && !options.baseRef) { + stderr(`--head requires --base.\n\n${usage}`); + return undefined; + } + if (options.checkoutRef) { + const inferredRepository = options.issueText + ? tryParseGitHubIssueSource(options.issueText)?.repositoryUrl + : undefined; + const repository = parseRepositorySource(options.repo ?? inferredRepository ?? process.cwd()); + if (repository.kind !== "github") { + stderr(`--ref only applies when --repo is a remote GitHub URL or the issue URL infers one.\n\n${usage}`); + return undefined; + } + } + return options; +} + +async function emit(contents: string, output: string | undefined, io: CommandIo): Promise { + if (output) await (io.writeOutput ?? ((path, text) => writeFile(path, text, "utf8")))(output, contents); + else io.stdout(contents); +} + +function expandHomePath(path: string): string { + if (path === "~") return homedir(); + if (path.startsWith("~/") || path.startsWith("~\\")) return resolve(homedir(), path.slice(2)); + return path; +} + +export type { ContextPack, FixMapGraph }; diff --git a/packages/cli/src/analysis-source.ts b/packages/cli/src/analysis-source.ts new file mode 100644 index 0000000..911f35c --- /dev/null +++ b/packages/cli/src/analysis-source.ts @@ -0,0 +1,121 @@ +import { + buildFixMapAnalysis, + buildContextPack, + type ContextPack, + type FixMapReport, + type RepoMap, + type ScanDiagnostic +} from "@aryam/fixmap-core"; +import { + fetchPublicGitHubIssue, + findLocalGitHubRepositoryUrl, + parseGitHubIssueSource, + parseRepositorySource, + RepositorySourceError, + withRepositorySource, + type RepositorySourceDependencies +} from "./repository-source.js"; + +const MAX_ISSUE_BODY_CHARS = 20_000; + +export type AnalysisSourceInput = { + repo?: string | undefined; + issueText?: string | undefined; + diffSpec?: string | undefined; + baseRef?: string | undefined; + headRef?: string | undefined; + checkoutRef?: string | undefined; + workingTree?: boolean | undefined; + includeUntracked?: boolean | undefined; + useCache?: boolean | undefined; + limit?: number | undefined; + exclude?: string[] | undefined; + internalExclude?: string[] | undefined; +}; + +export type AnalyzedRepository = { + task: string; + report: FixMapReport; + repo: RepoMap; +}; + +export async function analyzeRepository( + input: AnalysisSourceInput, + dependencies: RepositorySourceDependencies = {} +): Promise { + const issueSource = input.issueText ? parseGitHubIssueSource(input.issueText) : undefined; + const source = parseRepositorySource(input.repo ?? issueSource?.repositoryUrl ?? process.cwd()); + const localRepositoryUrl = issueSource && source.kind === "local" + ? await findLocalGitHubRepositoryUrl(source.repoRoot) + : undefined; + if ( + issueSource && + ((source.kind === "github" && source.displayUrl.toLowerCase() !== issueSource.repositoryUrl.toLowerCase()) || + (source.kind === "local" && localRepositoryUrl && localRepositoryUrl.toLowerCase() !== issueSource.repositoryUrl.toLowerCase())) + ) { + const actualRepository = source.kind === "github" ? source.displayUrl : localRepositoryUrl; + throw new RepositorySourceError( + `GitHub issue "${issueSource.displayUrl}" belongs to ${issueSource.repositoryUrl}, ` + + `but the scanned repository is ${actualRepository}. Remove --repo or use the matching repository.` + ); + } + if ( + source.kind === "github" && + (input.diffSpec !== undefined || input.baseRef !== undefined || input.headRef !== undefined || input.workingTree || input.includeUntracked) + ) { + throw new RepositorySourceError( + "Git diff and working-tree options need a local checkout. A GitHub URL is fetched as a " + + "single-commit shallow clone of the selected branch, so it has no history for a diff range " + + "to resolve against and no working tree to compare. Clone the repository and pass --repo " + + "with a local path, or use --issue alone." + ); + } + let task = input.issueText ?? ""; + let issueDiagnostic: ScanDiagnostic | undefined; + if (issueSource) { + const issue = await (dependencies.fetchPublicIssue ?? fetchPublicGitHubIssue)(issueSource); + const body = issue.body.slice(0, MAX_ISSUE_BODY_CHARS); + task = [issue.title, body].filter(Boolean).join("\n\n"); + issueDiagnostic = { + code: issueSource.isPullRequest ? "remote-pull-fetched" : "remote-issue-fetched", + severity: "info", + message: `Fetched ${issueSource.displayUrl} anonymously for context selection.` + }; + } + + return withRepositorySource(source, async (resolved) => { + const { report, repo } = await buildFixMapAnalysis({ + repoRoot: resolved.repoRoot, + issueText: task, + diffSpec: input.diffSpec, + baseRef: input.baseRef, + headRef: input.headRef, + workingTree: input.workingTree, + includeUntracked: input.includeUntracked, + useCache: input.useCache, + includeHistory: true, + limit: input.limit, + exclude: input.exclude, + internalExclude: input.internalExclude + }); + report.diagnostics.unshift(...[issueDiagnostic, resolved.diagnostic].filter( + (entry): entry is ScanDiagnostic => entry !== undefined + )); + return { task, report, repo }; + }, dependencies, (result, cleanupError, temporaryRoot) => { + result.report.diagnostics.push({ + code: "remote-checkout-cleanup-failed", + severity: "warning", + message: `Analysis completed, but temporary checkout "${temporaryRoot}" could not be removed: ${cleanupError instanceof Error ? cleanupError.message : String(cleanupError)}.` + }); + }, input.checkoutRef); +} + +export function contextFromAnalysis(analysis: AnalyzedRepository, budgetTokens: number): ContextPack { + return buildContextPack({ + report: analysis.report, + repo: analysis.repo, + task: analysis.task, + budgetTokens + }); +} diff --git a/packages/cli/src/benchmark.ts b/packages/cli/src/benchmark.ts new file mode 100644 index 0000000..094ce5f --- /dev/null +++ b/packages/cli/src/benchmark.ts @@ -0,0 +1,374 @@ +import { execFile } from "node:child_process"; +import { mkdir, mkdtemp, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { basename, join, resolve } from "node:path"; +import { promisify } from "node:util"; +import { + buildReportFromRepo, + isBackupPath, + isGeneratedPath, + moduleStem, + rankByBm25, + retrievalQueryTerms, + scanRepo, + taskMentionsExpectedPath +} from "@aryam/fixmap-core"; + +const exec = promisify(execFile); +const MAX_CHANGED_FILES = 30; +const MAX_BENCHMARK_CASES = 100; +const MAX_TASK_CHARS = 20_000; +const GIT_BUFFER = 24 * 1024 * 1024; + +export type BenchmarkArm = "bm25" | "fixmap" | "impact"; +export type BenchmarkCaseResult = { + commit: string; + task: string; + expected: string[]; + mentionsExpectedPath: boolean; + arms: Record; + impactSecondary: { hits: number; of: number } | null; +}; + +export type RepositoryBenchmark = { + benchmarkVersion: 1; + generatedAt: string; + repository: string; + requestedCommits: number; + eligibleCases: number; + skipped: Record; + safeguards: { + parentSnapshots: true; + historyCutoff: "target-parent"; + maxChangedFiles: number; + primaryTargets: "changed-maintained-non-test-code"; + sameScannedCorpus: true; + checkoutFiltersDisabled: true; + repositoryCodeExecuted: false; + }; + cohorts: Record<"all" | "mentioned" | "unmentioned", Record>; + impactSecondary: { hits: number; of: number; recall: number | null }; + cases: BenchmarkCaseResult[]; +}; + +export type BenchmarkScore = { + cases: number; + top1: Rate; + top3: Rate; + top5: Rate; +}; + +type Rate = { hits: number; of: number; rate: number | null; interval95: [number, number] | null }; +type CandidateCommit = { hash: string; parent: string; task: string; changedFiles: string[] }; + +export async function benchmarkRepository(input: { + repoRoot: string; + last?: number; + progress?: (message: string) => void; +}): Promise { + const repoRoot = resolve(input.repoRoot); + const requested = input.last ?? 20; + if (!Number.isSafeInteger(requested) || requested < 1 || requested > MAX_BENCHMARK_CASES) { + throw new Error(`--last must be a whole number from 1 to ${MAX_BENCHMARK_CASES}.`); + } + await assertGitRepository(repoRoot); + const checkoutConfig = await safeCheckoutConfig(repoRoot); + const candidates = await listCandidateCommits(repoRoot, Math.min(MAX_BENCHMARK_CASES * 5, requested * 5)); + const repository = await repositoryLabel(repoRoot); + const skipped: Record = {}; + const results: BenchmarkCaseResult[] = []; + + for (const candidate of candidates) { + if (results.length >= requested) break; + if (candidate.changedFiles.length === 0) { increment(skipped, "empty-change"); continue; } + if (candidate.changedFiles.length > MAX_CHANGED_FILES) { increment(skipped, "oversized-change"); continue; } + if (retrievalQueryTerms(candidate.task).length < 2) { increment(skipped, "insufficient-task-text"); continue; } + + input.progress?.(`Benchmarking ${results.length + 1}/${requested}: ${candidate.hash.slice(0, 8)}`); + const row = await benchmarkCommit(repoRoot, candidate, checkoutConfig); + if (!row) { increment(skipped, "no-preexisting-target"); continue; } + results.push(row); + } + if (results.length === 0) { + throw new Error("No eligible historical changes were found. Use a repository with descriptive commit messages and at least one prior revision."); + } + + const all = results; + const mentioned = results.filter((row) => row.mentionsExpectedPath); + const unmentioned = results.filter((row) => !row.mentionsExpectedPath); + const impactSecondaryRows = results.map((row) => row.impactSecondary).filter((row): row is { hits: number; of: number } => row !== null); + const secondaryHits = impactSecondaryRows.reduce((sum, row) => sum + row.hits, 0); + const secondaryOf = impactSecondaryRows.reduce((sum, row) => sum + row.of, 0); + + return { + benchmarkVersion: 1, + generatedAt: new Date().toISOString(), + repository, + requestedCommits: requested, + eligibleCases: results.length, + skipped, + safeguards: { + parentSnapshots: true, + historyCutoff: "target-parent", + maxChangedFiles: MAX_CHANGED_FILES, + primaryTargets: "changed-maintained-non-test-code", + sameScannedCorpus: true, + checkoutFiltersDisabled: true, + repositoryCodeExecuted: false + }, + cohorts: { + all: scoreArms(all), + mentioned: scoreArms(mentioned), + unmentioned: scoreArms(unmentioned) + }, + impactSecondary: { + hits: secondaryHits, + of: secondaryOf, + recall: secondaryOf === 0 ? null : round(secondaryHits / secondaryOf) + }, + cases: results + }; +} + +async function benchmarkCommit( + repoRoot: string, + candidate: CandidateCommit, + checkoutConfig: string[] +): Promise { + const temporaryRoot = await mkdtemp(join(tmpdir(), "fixmap-benchmark-")); + const snapshot = join(temporaryRoot, "snapshot"); + const emptyHooks = join(temporaryRoot, "hooks-disabled"); + await mkdir(emptyHooks); + let worktreeAdded = false; + try { + await runGit(repoRoot, [ + "-c", `core.hooksPath=${emptyHooks}`, + ...checkoutConfig, + "worktree", "add", "--detach", snapshot, candidate.parent + ]); + worktreeAdded = true; + const repo = await scanRepo({ repoRoot: snapshot, includeHistory: true, useCache: false }); + const fileByPath = new Map(repo.files.map((file) => [file.path, file])); + const changedExisting = candidate.changedFiles.filter((path) => fileByPath.has(path)); + const maintainedStems = new Set(repo.files + .filter((file) => !isGeneratedPath(file.path) && !isBackupPath(file.path)) + .map((file) => moduleStem(file.path))); + // Primary retrieval compares like with like: changed maintained implementation code, + // excluding tests. Generated twins cannot become impossible expected answers when all + // three retrieval arms deliberately prefer their maintained source. + const expected = changedExisting.filter((path) => { + const file = fileByPath.get(path); + return file?.kind === "code" && + !file.isTest && + !isBackupPath(path) && + !(isGeneratedPath(path) && maintainedStems.has(moduleStem(path))); + }); + if (expected.length === 0) return undefined; + + const report = buildReportFromRepo(repo, { issueText: candidate.task, limit: 5 }); + const fixmap = report.contextFiles.map((file) => file.path).slice(0, 5); + const bm25 = rankByBm25(repo.files, candidate.task, 5); + const impact = uniquePaths([ + ...(fixmap[0] ? [fixmap[0]] : []), + ...(report.impact?.files.map((file) => file.path) ?? []), + ...fixmap.slice(1) + ]).slice(0, 5); + const primary = fixmap.find((path) => expected.includes(path)); + const secondaryExpected = primary + ? changedExisting.filter((path) => path !== primary && fileByPath.get(path)?.kind === "code") + : []; + const impactSecondary = primary && secondaryExpected.length > 0 + ? { hits: secondaryExpected.filter((path) => report.impact?.files.some((file) => file.path === path)).length, of: secondaryExpected.length } + : null; + + return { + commit: candidate.hash, + task: candidate.task, + expected, + mentionsExpectedPath: taskMentionsExpectedPath(candidate.task, expected), + arms: { + bm25: scoreCase(bm25, expected), + fixmap: scoreCase(fixmap, expected), + impact: scoreCase(impact, expected) + }, + impactSecondary + }; + } finally { + if (worktreeAdded) { + try { + await runGit(repoRoot, ["-c", `core.hooksPath=${emptyHooks}`, ...checkoutConfig, "worktree", "remove", "--force", snapshot]); + } catch (error) { + throw new Error(`Could not remove temporary benchmark worktree "${snapshot}": ${error instanceof Error ? error.message : String(error)}`); + } + } + await rm(temporaryRoot, { recursive: true, force: true }); + } +} + +async function listCandidateCommits(repoRoot: string, limit: number): Promise { + const hashes = (await runGit(repoRoot, ["rev-list", "--no-merges", `--max-count=${limit}`, "HEAD"])) + .split(/\r?\n/).filter(Boolean); + const candidates: CandidateCommit[] = []; + for (const hash of hashes) { + const parents = (await runGit(repoRoot, ["show", "-s", "--format=%P", hash])).trim().split(/\s+/).filter(Boolean); + if (parents.length !== 1) continue; + const task = (await runGit(repoRoot, ["show", "-s", "--format=%B", hash])).slice(0, MAX_TASK_CHARS).trim(); + const changedFiles = (await runGit(repoRoot, [ + "-c", "core.quotepath=false", "diff-tree", "--no-ext-diff", "--no-commit-id", "--name-only", "-r", "-z", hash + ])).split("\0").filter(Boolean).map((path) => path.replace(/\\/g, "/")); + candidates.push({ hash, parent: parents[0]!, task, changedFiles: [...new Set(changedFiles)] }); + } + return candidates; +} + +async function safeCheckoutConfig(repoRoot: string): Promise { + let names: string[] = []; + try { + const output = await runGit(repoRoot, [ + "config", "--null", "--name-only", "--get-regexp", "^filter\\..*\\.(clean|smudge|process|required)$" + ]); + names = output.split("\0").filter(Boolean); + } catch { + // `git config --get-regexp` exits 1 when no matching filter is configured. + } + const drivers = [...new Set(names.flatMap((name) => { + const match = name.match(/^filter\.(.+)\.(?:clean|smudge|process|required)$/i); + return match?.[1] ? [match[1]] : []; + }))].sort((left, right) => left.localeCompare(right)); + const options = [ + "-c", "core.fsmonitor=false", + "-c", "core.symlinks=false" + ]; + for (const driver of drivers) { + options.push( + "-c", `filter.${driver}.smudge=`, + "-c", `filter.${driver}.process=`, + "-c", `filter.${driver}.required=false` + ); + } + return options; +} + +async function assertGitRepository(repoRoot: string): Promise { + try { + if ((await runGit(repoRoot, ["rev-parse", "--is-inside-work-tree"])).trim() !== "true") throw new Error("not a worktree"); + await runGit(repoRoot, ["rev-parse", "--verify", "HEAD"]); + } catch { + throw new Error(`Benchmark needs a local Git repository with at least one commit: ${repoRoot}`); + } +} + +async function repositoryLabel(repoRoot: string): Promise { + try { + const remote = (await runGit(repoRoot, ["remote", "get-url", "origin"])).trim(); + const match = remote.match(/github\.com[/:]([^/]+\/[^/]+?)(?:\.git)?$/i); + if (match?.[1]) return match[1]; + } catch { /* Local-only repositories have no origin; basename is non-sensitive and stable. */ } + return basename(repoRoot); +} + +async function runGit(cwd: string, args: string[]): Promise { + const { stdout } = await exec("git", args, { + cwd, + maxBuffer: GIT_BUFFER, + env: { ...process.env, GIT_TERMINAL_PROMPT: "0", GIT_CONFIG_NOSYSTEM: "1" } + }); + return stdout; +} + +function scoreCase(paths: string[], expected: string[]): BenchmarkCaseResult["arms"][BenchmarkArm] { + return { + top5Paths: paths, + top1Hit: expected.includes(paths[0] ?? ""), + top3Hit: expected.some((path) => paths.slice(0, 3).includes(path)), + top5Hit: expected.some((path) => paths.slice(0, 5).includes(path)) + }; +} + +function scoreArms(rows: BenchmarkCaseResult[]): Record { + return { + bm25: scoreRows(rows, "bm25"), + fixmap: scoreRows(rows, "fixmap"), + impact: scoreRows(rows, "impact") + }; +} + +function scoreRows(rows: BenchmarkCaseResult[], arm: BenchmarkArm): BenchmarkScore { + return { + cases: rows.length, + top1: rate(rows, arm, "top1Hit"), + top3: rate(rows, arm, "top3Hit"), + top5: rate(rows, arm, "top5Hit") + }; +} + +function rate(rows: BenchmarkCaseResult[], arm: BenchmarkArm, key: "top1Hit" | "top3Hit" | "top5Hit"): Rate { + const hits = rows.filter((row) => row.arms[arm][key]).length; + return { + hits, + of: rows.length, + rate: rows.length === 0 ? null : round(hits / rows.length), + interval95: wilsonInterval(hits, rows.length) + }; +} + +function wilsonInterval(hits: number, total: number): [number, number] | null { + if (total === 0) return null; + const z = 1.959963984540054; + const proportion = hits / total; + const denominator = 1 + (z * z) / total; + const center = (proportion + (z * z) / (2 * total)) / denominator; + const margin = (z / denominator) * Math.sqrt((proportion * (1 - proportion)) / total + (z * z) / (4 * total * total)); + return [round(Math.max(0, center - margin)), round(Math.min(1, center + margin))]; +} + +function round(value: number): number { + return Number(value.toFixed(3)); +} + +function uniquePaths(paths: string[]): string[] { + return [...new Set(paths)]; +} + +function increment(counts: Record, key: string): void { + counts[key] = (counts[key] ?? 0) + 1; +} + +export function renderRepositoryBenchmark(result: RepositoryBenchmark): string { + const unmentioned = result.cohorts.unmentioned; + const percent = (rate: number | null) => rate === null ? "n/a" : `${Math.round(rate * 100)}%`; + const lines = [ + "# FixMap Repository Benchmark", + "", + `Evaluated ${result.eligibleCases} eligible historical changes from ${result.repository}.`, + "Every case used the target commit's parent snapshot and history ending at that parent.", + "", + "## Unmentioned tasks", + "", + "| Arm | Top 1 | Top 3 | Top 5 |", + "| --- | ---: | ---: | ---: |", + `| BM25 over code | ${percent(unmentioned.bm25.top1.rate)} | ${percent(unmentioned.bm25.top3.rate)} | ${percent(unmentioned.bm25.top5.rate)} |`, + `| FixMap context | ${percent(unmentioned.fixmap.top1.rate)} | ${percent(unmentioned.fixmap.top3.rate)} | ${percent(unmentioned.fixmap.top5.rate)} |`, + `| FixMap + Impact Graph | ${percent(unmentioned.impact.top1.rate)} | ${percent(unmentioned.impact.top3.rate)} | ${percent(unmentioned.impact.top5.rate)} |`, + "", + `Impact secondary-file recall: ${result.impactSecondary.of === 0 ? "n/a" : `${result.impactSecondary.hits}/${result.impactSecondary.of} (${percent(result.impactSecondary.recall)})`}.`, + "", + "## Safeguards", + "", + "- Source was scanned from each target's parent revision.", + "- Co-change history stopped at that parent revision.", + `- Merges and commits touching more than ${result.safeguards.maxChangedFiles} files were excluded.`, + "- BM25, FixMap, and Impact Graph saw the same scanned files.", + "- Primary hits were scored only against changed maintained non-test code; generated twins, backups, and tests were excluded. Secondary impact recall may include changed tests.", + "- Repository code, dependencies, scripts, and hooks were not executed.", + "", + "A historical commit message is not the same input as its original issue. Treat this as a repository-specific backtest, not proof of agent savings.", + "" + ]; + return lines.join("\n"); +} diff --git a/packages/cli/src/cli-runner.ts b/packages/cli/src/cli-runner.ts index 06bd102..650b45f 100644 --- a/packages/cli/src/cli-runner.ts +++ b/packages/cli/src/cli-runner.ts @@ -9,6 +9,7 @@ import { quoteCliValue as formatCliValue, renderComparisonMarkdown, renderExplanationMarkdown, + renderAgentReport, renderJsonReport, renderVerifyMarkdown, resolveExclusions, @@ -22,6 +23,8 @@ import { import { runDoctorChecks, renderDoctorReport, type DoctorReport } from "./doctor.js"; import { installAgentCommands, renderFeatureCatalog, type AgentTarget } from "./agent-setup.js"; import { clarifyMissingPath } from "./explain-path.js"; +import type { RepositoryBenchmark } from "./benchmark.js"; +import type { WatchRepositoryInput, WatchUpdate } from "./watch.js"; import { buildReportForRepository, isSafeGitRefName, @@ -40,7 +43,7 @@ export type CliOptions = { baseRef?: string | undefined; headRef?: string | undefined; checkoutRef?: string | undefined; - format: "markdown" | "json"; + format: "markdown" | "json" | "agent"; output?: string | undefined; explainPath?: string | undefined; reportPath?: string | undefined; @@ -64,6 +67,10 @@ export type CliDependencies = { stdout?: (text: string) => void; writeReport?: (path: string, contents: string) => Promise; readIssueFile?: (path: string | number) => string | Buffer; + benchmarkRepository?: (input: { repoRoot: string; last?: number; progress?: (message: string) => void }) => Promise; + renderBenchmark?: (result: RepositoryBenchmark) => string; + watchRepository?: (input: WatchRepositoryInput) => Promise; + renderWatchUpdate?: (update: WatchUpdate, format: "markdown" | "json") => string; }; export const USAGE = `FixMap maps an issue, prompt, or diff to context files, test routes, and review risks. @@ -82,6 +89,7 @@ Usage: fixmap plan --working-tree --include-untracked --limit 12 --exclude "docs/**" fixmap plan --no-cache --issue "Fix login" --repo . fixmap plan --issue "Fix login" --format json --output plan.json + fixmap plan --issue "Fix login" --format agent fixmap plan --issue "Fix login in auth middleware" --compare plan.json fixmap plan --base main --head HEAD --format json fixmap verify --report plan.json --diff main...HEAD @@ -89,6 +97,10 @@ Usage: fixmap verify --report plan.json --working-tree --fail-on warning fixmap doctor --format json fixmap validate plan.json + fixmap benchmark --repo . --last 50 + fixmap context --issue "Fix login" --budget 10000 + fixmap graph --issue "Fix login" --format mermaid + fixmap watch --report plan.json --repo . fixmap features fixmap setup [--agent claude|cursor|copilot|agents|all] [--repo ] fixmap mcp [--repo ] @@ -98,6 +110,10 @@ Commands: verify Compare a saved report against the diff that followed it doctor Check the FixMap install for stale global or npx shadows validate Validate a saved FixMap JSON report + benchmark Backtest BM25, FixMap, and Impact Graph on pre-change snapshots + context Package the highest-value source ranges within a token budget + graph Export the evidence-backed Impact Graph as Mermaid or JSON + watch Recheck working-tree drift and impact whenever edits change features List every FixMap capability and its command setup Install a discoverable /fixmap command for coding agents mcp Run FixMap as an MCP server over stdio for AI coding agents @@ -115,7 +131,7 @@ Options: --ref Branch or tag to scan when --repo is a remote GitHub URL --limit Maximum context files to report (default 8, max 20) --exclude Path pattern to leave out of ranking (repeatable) - --format Output format: markdown (default) or json + --format Output format: markdown (default), json, or compact agent --output Write the report or verification to a file instead of stdout --explain Explain why one file was ranked where it was, or left out --compare Compare this plan against an earlier JSON report @@ -136,6 +152,7 @@ const MCP_USAGE = `Usage: fixmap mcp [--repo ]\n\nRuns the FixMap MCP serv const FEATURES_USAGE = `Usage: fixmap features [--format markdown|json]\n\nLists every FixMap capability and the command that exposes it.\n`; const SETUP_USAGE = `Usage: fixmap setup [--agent claude|cursor|copilot|agents|all] [--repo ] [--force]\n\nInstalls a /fixmap command that lists and runs FixMap workflows.\n`; const VALIDATE_USAGE = `Usage: fixmap validate [--format markdown|json]\n\nChecks a saved report against FixMap's structural compatibility contract.\n`; +const BENCHMARK_USAGE = `Usage: fixmap benchmark [--repo ] [--last <1-100>] [--format markdown|json] [--output ]\n\nBacktests BM25, FixMap, and Impact Graph against historical parent snapshots without executing repository code.\n`; export async function runCli(args: string[], dependencies: CliDependencies = {}): Promise { const stdout = dependencies.stdout ?? ((text: string) => process.stdout.write(text)); @@ -320,6 +337,100 @@ export async function runCli(args: string[], dependencies: CliDependencies = {}) } } + if (args[0] === "benchmark") { + if (args[1] === "--help" || args[1] === "-h") { stdout(BENCHMARK_USAGE); return 0; } + let repoRoot = process.cwd(); + let last: number | undefined; + let format: "markdown" | "json" = "markdown"; + let output: string | undefined; + const seen = new Set(); + for (let index = 1; index < args.length; index += 1) { + const raw = args[index]!; + const separator = raw.indexOf("="); + const flag = separator === -1 ? raw : raw.slice(0, separator); + const inline = separator === -1 ? undefined : raw.slice(separator + 1); + if (!new Set(["--repo", "--last", "--format", "--output"]).has(flag) || seen.has(flag)) { + stderr(`${seen.has(flag) ? `Pass ${flag} only once.` : `Unknown benchmark option: ${raw}`}\n\n${BENCHMARK_USAGE}`); + return 1; + } + seen.add(flag); + const following = args[index + 1]; + const value = inline ?? (following && !following.startsWith("-") ? following : undefined); + if (inline === undefined && value !== undefined) index += 1; + if (!value?.trim()) { stderr(`${flag} requires a value.\n\n${BENCHMARK_USAGE}`); return 1; } + if (flag === "--repo") repoRoot = expandHomePath(value.trim()); + else if (flag === "--output") output = expandHomePath(value.trim()); + else if (flag === "--format") { + const normalized = value.trim().toLowerCase(); + if (normalized !== "markdown" && normalized !== "json") { + stderr(`--format must be markdown or json.\n\n${BENCHMARK_USAGE}`); + return 1; + } + format = normalized; + } else { + const parsed = Number(value); + if (!/^\d+$/.test(value) || !Number.isSafeInteger(parsed) || parsed < 1 || parsed > 100) { + stderr(`--last must be a whole number from 1 to 100.\n\n${BENCHMARK_USAGE}`); + return 1; + } + last = parsed; + } + } + if (/^https?:\/\//i.test(repoRoot)) { + stderr(`benchmark --repo needs a local Git checkout so history cutoffs can be enforced.\n\n${BENCHMARK_USAGE}`); + return 1; + } + try { + const benchmarkModule = dependencies.benchmarkRepository && dependencies.renderBenchmark + ? undefined + : await import("./benchmark.js"); + const result = await (dependencies.benchmarkRepository ?? benchmarkModule!.benchmarkRepository)({ + repoRoot, + ...(last === undefined ? {} : { last }), + progress: (message) => { + if (progressRequested(process.env.FIXMAP_PROGRESS) || process.stderr.isTTY) stderr(`${message}\n`); + } + }); + const rendered = format === "json" + ? `${JSON.stringify(result, null, 2)}\n` + : (dependencies.renderBenchmark ?? benchmarkModule!.renderRepositoryBenchmark)(result); + if (output) await (dependencies.writeReport ?? ((path, contents) => writeFile(path, contents, "utf8")))(output, rendered); + else stdout(rendered); + return 0; + } catch (error) { + stderr(`${error instanceof Error ? error.message : String(error)}\n`); + return 1; + } + } + + if (args[0] === "context") { + const { runContextCommand } = await import("./analysis-commands.js"); + return runContextCommand(args.slice(1), { + stdout, + stderr, + ...(dependencies.writeReport ? { writeOutput: dependencies.writeReport } : {}) + }); + } + + if (args[0] === "graph") { + const { runGraphCommand } = await import("./analysis-commands.js"); + return runGraphCommand(args.slice(1), { + stdout, + stderr, + ...(dependencies.writeReport ? { writeOutput: dependencies.writeReport } : {}) + }); + } + + if (args[0] === "watch") { + const { runWatchCommand } = await import("./watch-command.js"); + return runWatchCommand(args.slice(1), { + stdout, + stderr, + watchRepository: dependencies.watchRepository, + renderWatchUpdate: dependencies.renderWatchUpdate + }); + } + if (args[0] === "doctor") { if (args[1] === "--help" || args[1] === "-h") { stdout(DOCTOR_USAGE); return 0; } const doctorArgs = args.slice(1); @@ -625,7 +736,11 @@ export async function runCli(args: string[], dependencies: CliDependencies = {}) return unresolvedChangeRequest ? 1 : 0; } - const rendered = options.format === "json" ? renderJsonReport(report) : renderMarkdownReport(report); + const rendered = options.format === "json" + ? renderJsonReport(report) + : options.format === "agent" + ? renderAgentReport(report) + : renderMarkdownReport(report); if (options.output) { try { await (dependencies.writeReport ?? ((path, contents) => writeFile(path, contents, "utf8")))( @@ -794,7 +909,7 @@ export function parseArgs(args: string[]): CliOptions { let baseRef: string | undefined; let headRef: string | undefined; let checkoutRef: string | undefined; - let format: "markdown" | "json" = "markdown"; + let format: "markdown" | "json" | "agent" = "markdown"; let output: string | undefined; let explainPath: string | undefined; let reportPath: string | undefined; @@ -881,10 +996,10 @@ export function parseArgs(args: string[]): CliOptions { } else if (arg === "--format") { consumeValue(); const normalized = value?.trim().toLowerCase(); - if (normalized === "markdown" || normalized === "json") { + if (normalized === "markdown" || normalized === "json" || normalized === "agent") { format = normalized; } else { - invalidValues.push(`--format received ${JSON.stringify(value ?? "(missing)")}; expected "markdown" or "json"`); + invalidValues.push(`--format received ${JSON.stringify(value ?? "(missing)")}; expected "markdown", "json", or "agent"`); } } else if (arg === "--report") { consumeValue(); @@ -1244,6 +1359,7 @@ async function runVerify( workingTree: options.workingTree, includeUntracked: options.includeUntracked, useCache: !options.noCache, + includeHistory: true, internalExclude: localPlanArtifactExclusions(options) }); const unresolvedDiff = repo.diagnostics.find((diagnostic) => diagnostic.code === "diff-unavailable"); diff --git a/packages/cli/src/mcp.ts b/packages/cli/src/mcp.ts index ce2eac9..be2e56a 100644 --- a/packages/cli/src/mcp.ts +++ b/packages/cli/src/mcp.ts @@ -6,9 +6,13 @@ import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js" import { CallToolRequestSchema, ListToolsRequestSchema } from "@modelcontextprotocol/sdk/types.js"; import { compareReports, + buildFixMapGraph, explainFile, renderComparisonMarkdown, renderExplanationMarkdown, + renderAgentReport, + renderContextPackMarkdown, + renderFixMapGraphMermaid, renderJsonReport, renderMarkdownReport, renderVerifyMarkdown, @@ -21,6 +25,7 @@ import { } from "@aryam/fixmap-core"; import { renderDoctorReport, runDoctorChecks } from "./doctor.js"; import { clarifyMissingPath } from "./explain-path.js"; +import { analyzeRepository, contextFromAnalysis } from "./analysis-source.js"; import { buildReportForRepository, isSafeGitRefName, @@ -37,7 +42,7 @@ type PlanArguments = { head?: string; repo?: string; ref?: string; - format?: "markdown" | "json"; + format?: "markdown" | "json" | "agent"; limit?: number; exclude?: string[]; workingTree?: boolean; @@ -64,6 +69,11 @@ type PlanArgumentsValidation = | { success: true; value: PlanArguments } | { success: false; message: string }; +type AnalysisToolArguments = Omit & { + format?: "markdown" | "json" | "mermaid"; + budget?: number; +}; + type VerifyArguments = { report: FixMapReport; reportPath?: string; @@ -116,7 +126,7 @@ const PLAN_TOOL = { ref: { type: "string", description: "Branch or tag to scan when repo is a remote GitHub URL" }, format: { type: "string", - description: "Output format: markdown (default) or json, case-insensitive" + description: "Output format: markdown (default), json, or compact agent, case-insensitive" }, limit: { type: "number", @@ -182,6 +192,35 @@ const EXPLAIN_TOOL = { } }; +const CONTEXT_TOOL = { + name: "fixmap_context", + title: "FixMap context", + description: "Select task-aware source ranges from FixMap's primary and impact files within a deterministic source-token budget. Use this after Plan and before editing; it does not call a model or execute repository code.", + inputSchema: { + type: "object" as const, + properties: { + ...PLAN_TOOL.inputSchema.properties, + budget: { type: "number", description: "Estimated source-token budget, a whole number from 256 to 200000; default 10000" }, + format: { type: "string", description: "Output format: markdown (default) or json, case-insensitive" } + }, + additionalProperties: false + } +}; + +const GRAPH_TOOL = { + name: "fixmap_graph", + title: "FixMap graph", + description: "Export FixMap's evidence-backed Impact Graph, including import direction, routed tests, and repeated co-change relationships, as Mermaid or JSON.", + inputSchema: { + type: "object" as const, + properties: { + ...PLAN_TOOL.inputSchema.properties, + format: { type: "string", description: "Output format: mermaid (default) or json, case-insensitive" } + }, + additionalProperties: false + } +}; + const VERIFY_TOOL = { name: "fixmap_verify", title: "FixMap verify", @@ -251,10 +290,41 @@ export function createFixMapMcpServer( const server = new Server({ name: "fixmap", version: readVersion() }, { capabilities: { tools: {} } }); server.setRequestHandler(ListToolsRequestSchema, async () => ({ - tools: [PLAN_TOOL, VERIFY_TOOL, EXPLAIN_TOOL, COMPARE_TOOL, DOCTOR_TOOL] + tools: [PLAN_TOOL, CONTEXT_TOOL, GRAPH_TOOL, VERIFY_TOOL, EXPLAIN_TOOL, COMPARE_TOOL, DOCTOR_TOOL] })); server.setRequestHandler(CallToolRequestSchema, async (request) => { + if (request.params.name === CONTEXT_TOOL.name || request.params.name === GRAPH_TOOL.name) { + const kind = request.params.name === CONTEXT_TOOL.name ? "context" : "graph"; + const parsed = parseAnalysisToolArguments(request.params.arguments ?? {}, kind); + if (!parsed.success) return { isError: true, content: [{ type: "text", text: `Invalid arguments: ${parsed.message}` }] }; + const args = parsed.value; + try { + const analysis = await analyzeRepository({ + repo: args.repo ?? (tryParseGitHubIssueSource(args.issue ?? "") ? undefined : defaultRepo), + checkoutRef: args.ref, + issueText: args.issue, + diffSpec: args.diff, + baseRef: args.base, + headRef: args.head, + workingTree: args.workingTree, + includeUntracked: args.includeUntracked, + useCache: !args.noCache, + limit: args.limit, + exclude: args.exclude + }, repositorySourceDependencies); + if (kind === "context") { + const pack = contextFromAnalysis(analysis, args.budget ?? 10_000); + const text = args.format === "json" ? `${JSON.stringify(pack, null, 2)}\n` : renderContextPackMarkdown(pack); + return { ...(pack.snippets.length === 0 ? { isError: true } : {}), content: [{ type: "text", text }] }; + } + const graph = buildFixMapGraph(analysis.report); + const text = args.format === "json" ? `${JSON.stringify(graph, null, 2)}\n` : renderFixMapGraphMermaid(graph); + return { ...(graph.nodes.length === 0 ? { isError: true } : {}), content: [{ type: "text", text }] }; + } catch (error) { + return { isError: true, content: [{ type: "text", text: error instanceof Error ? error.message : String(error) }] }; + } + } if (request.params.name === COMPARE_TOOL.name) { const record = request.params.arguments as Record | undefined; const unknown = Object.keys(record ?? {}).filter((key) => !["previous", "current", "format"].includes(key)); @@ -313,6 +383,7 @@ export function createFixMapMcpServer( workingTree: args.workingTree, includeUntracked: args.includeUntracked, useCache: !args.noCache, + includeHistory: true, internalExclude: args.reportPath ? [resolve(args.reportPath)] : undefined }); const diffFailure = repo.diagnostics.find((diagnostic) => diagnostic.code === "diff-unavailable"); @@ -442,7 +513,11 @@ export function createFixMapMcpServer( } } - const text = args.format === "json" ? renderJsonReport(report) : renderMarkdownReport(report); + const text = args.format === "json" + ? renderJsonReport(report) + : args.format === "agent" + ? renderAgentReport(report) + : renderMarkdownReport(report); return { content: [{ type: "text", text }] }; }); @@ -478,7 +553,7 @@ export function parsePlanArguments(input: unknown): PlanArgumentsValidation { if (record.workingTree === true && (record.diff || record.base || record.head)) return { success: false, message: 'use either "workingTree" or diff/base/head, not both.' }; if (record.diff && (record.base || record.head)) return { success: false, message: 'use either "diff" or base/head, not both.' }; if (record.head && !record.base) return { success: false, message: '"head" requires "base".' }; - const format = normalizeFormat(record.format); if (!format.success) return format; + const format = normalizePlanFormat(record.format); if (!format.success) return format; const limit = validateLimit(record.limit); if (!limit.success) { return limit; @@ -515,6 +590,47 @@ export function parsePlanArguments(input: unknown): PlanArgumentsValidation { return { success: true, value }; } +function parseAnalysisToolArguments( + input: unknown, + kind: "context" | "graph" +): { success: true; value: AnalysisToolArguments } | { success: false; message: string } { + if (typeof input !== "object" || input === null || Array.isArray(input)) { + return { success: false, message: "tool arguments must be an object." }; + } + const record = input as Record; + const allowed = new Set(["issue", "diff", "base", "head", "repo", "ref", "format", "limit", "exclude", "workingTree", "includeUntracked", "noCache", ...(kind === "context" ? ["budget"] : [])]); + const unknown = Object.keys(record).filter((key) => !allowed.has(key)); + if (unknown.length > 0) return { success: false, message: `unknown argument${unknown.length === 1 ? "" : "s"}: ${unknown.join(", ")}.` }; + + const format = record.format === undefined ? undefined : typeof record.format === "string" ? record.format.trim().toLowerCase() : ""; + const formats = kind === "context" ? ["markdown", "json"] : ["mermaid", "json"]; + if (record.format !== undefined && (typeof format !== "string" || !formats.includes(format))) return { success: false, message: `"format" must be "${formats.join('" or "')}".` }; + const normalizedFormat: "markdown" | "json" | "mermaid" | undefined = + format === "markdown" || format === "json" || format === "mermaid" ? format : undefined; + if (record.budget !== undefined && ( + typeof record.budget !== "number" || !Number.isSafeInteger(record.budget) || record.budget < 256 || record.budget > 200_000 + )) return { success: false, message: '"budget" must be a whole number from 256 to 200000.' }; + + const baseRecord = { ...record }; + delete baseRecord.format; + delete baseRecord.budget; + const parsed = parsePlanArguments(baseRecord); + if (!parsed.success) return parsed; + if (!parsed.value.issue && !parsed.value.diff && !parsed.value.base && !parsed.value.workingTree) { + return { success: false, message: "provide issue, diff, base/head, or workingTree so FixMap has a task signal." }; + } + const { format: _planFormat, ...baseValue } = parsed.value; + void _planFormat; + return { + success: true, + value: { + ...baseValue, + ...(normalizedFormat !== undefined ? { format: normalizedFormat } : {}), + ...(typeof record.budget === "number" ? { budget: record.budget } : {}) + } + }; +} + function validateLimit( candidate: unknown ): { success: true; value: number | undefined } | { success: false; message: string } { @@ -657,11 +773,22 @@ export function parseVerifyArguments(input: unknown): VerifyArgumentsValidation }; } +function normalizePlanFormat(candidate: unknown): { success: true; value: "markdown" | "json" | "agent" | undefined } | { success: false; message: string } { + if (candidate === undefined) return { success: true, value: undefined }; + if (typeof candidate !== "string") return { success: false, message: '"format" must be "markdown", "json", or "agent".' }; + const value = candidate.trim().toLowerCase(); + return value === "markdown" || value === "json" || value === "agent" + ? { success: true, value } + : { success: false, message: '"format" must be "markdown", "json", or "agent".' }; +} + function normalizeFormat(candidate: unknown): { success: true; value: "markdown" | "json" | undefined } | { success: false; message: string } { if (candidate === undefined) return { success: true, value: undefined }; if (typeof candidate !== "string") return { success: false, message: '"format" must be either "markdown" or "json".' }; const value = candidate.trim().toLowerCase(); - return value === "markdown" || value === "json" ? { success: true, value } : { success: false, message: '"format" must be either "markdown" or "json".' }; + return value === "markdown" || value === "json" + ? { success: true, value } + : { success: false, message: '"format" must be either "markdown" or "json".' }; } type LoadedReport = diff --git a/packages/cli/src/repository-source.ts b/packages/cli/src/repository-source.ts index 5f9e740..4b8803d 100644 --- a/packages/cli/src/repository-source.ts +++ b/packages/cli/src/repository-source.ts @@ -575,7 +575,7 @@ export async function buildReportForRepository( ); } -async function findLocalGitHubRepositoryUrl(repoRoot: string): Promise { +export async function findLocalGitHubRepositoryUrl(repoRoot: string): Promise { try { const { stdout } = await exec( "git", diff --git a/packages/cli/src/watch-command.ts b/packages/cli/src/watch-command.ts new file mode 100644 index 0000000..7c5c06a --- /dev/null +++ b/packages/cli/src/watch-command.ts @@ -0,0 +1,116 @@ +import { readFileSync } from "node:fs"; +import { homedir } from "node:os"; +import { resolve } from "node:path"; +import { stripByteOrderMark, validateFixMapReport, type FixMapReport } from "@aryam/fixmap-core"; +import { renderWatchUpdate, watchRepository, type WatchRepositoryInput, type WatchUpdate } from "./watch.js"; + +const USAGE = `Usage: fixmap watch --report [--repo ] [--interval <250-60000>] [--include-untracked] [--format markdown|json] [--fail-on error|warning] [--once]\n\nWatches a local Git working tree, verifies each changed state against the saved plan, and recalculates impact without executing repository code. Press Ctrl+C to stop.\n`; + +export async function runWatchCommand(args: string[], dependencies: { + stdout: (text: string) => void; + stderr: (text: string) => void; + watchRepository?: ((input: WatchRepositoryInput) => Promise) | undefined; + renderWatchUpdate?: ((update: WatchUpdate, format: "markdown" | "json") => string) | undefined; +}): Promise { + if (args[0] === "--help" || args[0] === "-h") { dependencies.stdout(USAGE); return 0; } + let repoRoot = process.cwd(); + let reportPath: string | undefined; + let intervalMs = 1_500; + let includeUntracked = false; + let once = false; + let format: "markdown" | "json" = "markdown"; + let failOn: "error" | "warning" = "error"; + const seen = new Set(); + const valueFlags = new Set(["--repo", "--report", "--interval", "--format", "--fail-on"]); + const booleanFlags = new Set(["--include-untracked", "--once"]); + + for (let index = 0; index < args.length; index += 1) { + const raw = args[index]!; + const separator = raw.indexOf("="); + const flag = separator === -1 ? raw : raw.slice(0, separator); + const inline = separator === -1 ? undefined : raw.slice(separator + 1); + if ((!valueFlags.has(flag) && !booleanFlags.has(flag)) || seen.has(flag)) { + dependencies.stderr(`${seen.has(flag) ? `Pass ${flag} only once.` : `Unknown watch option: ${raw}`}\n\n${USAGE}`); + return 1; + } + seen.add(flag); + if (booleanFlags.has(flag)) { + if (inline !== undefined) { dependencies.stderr(`${flag} does not take a value.\n\n${USAGE}`); return 1; } + if (flag === "--once") once = true; + else includeUntracked = true; + continue; + } + const following = args[index + 1]; + const value = inline ?? (following && !following.startsWith("-") ? following : undefined); + if (inline === undefined && value !== undefined) index += 1; + if (!value?.trim()) { dependencies.stderr(`${flag} requires a value.\n\n${USAGE}`); return 1; } + if (flag === "--repo") repoRoot = expandHomePath(value.trim()); + else if (flag === "--report") reportPath = expandHomePath(value.trim()); + else if (flag === "--format") { + const normalized = value.trim().toLowerCase(); + if (normalized !== "markdown" && normalized !== "json") { dependencies.stderr(`--format must be markdown or json.\n\n${USAGE}`); return 1; } + format = normalized; + } else if (flag === "--fail-on") { + const normalized = value.trim().toLowerCase(); + if (normalized !== "error" && normalized !== "warning") { dependencies.stderr(`--fail-on must be error or warning.\n\n${USAGE}`); return 1; } + failOn = normalized; + } else { + const parsed = Number(value); + if (!/^\d+$/.test(value) || !Number.isSafeInteger(parsed) || parsed < 250 || parsed > 60_000) { + dependencies.stderr(`--interval must be a whole number from 250 to 60000 milliseconds.\n\n${USAGE}`); + return 1; + } + intervalMs = parsed; + } + } + + if (!reportPath) { dependencies.stderr(`watch requires --report with a saved JSON plan.\n\n${USAGE}`); return 1; } + if (/^https?:\/\//i.test(repoRoot)) { dependencies.stderr(`watch --repo needs a local Git checkout.\n\n${USAGE}`); return 1; } + const report = readReport(reportPath, dependencies.stderr); + if (!report) return 1; + + const controller = new AbortController(); + const stop = () => controller.abort(); + process.once("SIGINT", stop); + process.once("SIGTERM", stop); + try { + dependencies.stderr(once ? "Checking the current working tree once.\n" : `Watching ${resolve(repoRoot)} every ${intervalMs}ms. Press Ctrl+C to stop.\n`); + const last = await (dependencies.watchRepository ?? watchRepository)({ + repoRoot, + report, + intervalMs, + includeUntracked, + once, + signal: controller.signal, + onUpdate: (update) => dependencies.stdout((dependencies.renderWatchUpdate ?? renderWatchUpdate)(update, format)) + }); + if (!once || !last) return 0; + return last.verification.findings.some((finding) => + finding.severity === "error" || (failOn === "warning" && finding.severity === "warning") + ) ? 1 : 0; + } catch (error) { + dependencies.stderr(`${error instanceof Error ? error.message : String(error)}\n`); + return 1; + } finally { + process.off("SIGINT", stop); + process.off("SIGTERM", stop); + } +} + +function readReport(path: string, stderr: (text: string) => void): FixMapReport | undefined { + try { + const parsed = JSON.parse(stripByteOrderMark(readFileSync(path, "utf8"))) as unknown; + const loaded = validateFixMapReport(parsed, `"${path}"`); + if (!loaded.success) { stderr(`${loaded.message}\n`); return undefined; } + return loaded.report; + } catch (error) { + stderr(`Could not read watch report "${path}": ${error instanceof Error ? error.message : String(error)}\n`); + return undefined; + } +} + +function expandHomePath(path: string): string { + if (path === "~") return homedir(); + if (path.startsWith("~/") || path.startsWith("~\\")) return resolve(homedir(), path.slice(2)); + return path; +} diff --git a/packages/cli/src/watch.ts b/packages/cli/src/watch.ts new file mode 100644 index 0000000..d60e998 --- /dev/null +++ b/packages/cli/src/watch.ts @@ -0,0 +1,163 @@ +import { execFile } from "node:child_process"; +import { createHash } from "node:crypto"; +import { createReadStream } from "node:fs"; +import { lstat, readlink } from "node:fs/promises"; +import { isAbsolute, relative, resolve, sep } from "node:path"; +import { setTimeout as delay } from "node:timers/promises"; +import { promisify } from "node:util"; +import { renderVerifyMarkdown, scanRepo, verifyPlan, type FixMapReport, type VerifyResult } from "@aryam/fixmap-core"; + +const exec = promisify(execFile); +const GIT_MAX_BUFFER = 24 * 1024 * 1024; + +export type WatchUpdate = { + watchVersion: 1; + sequence: number; + observedAt: string; + verification: VerifyResult; +}; + +export type WatchRepositoryInput = { + repoRoot: string; + report: FixMapReport; + intervalMs?: number | undefined; + includeUntracked?: boolean | undefined; + once?: boolean | undefined; + signal?: AbortSignal | undefined; + onUpdate: (update: WatchUpdate) => void | Promise; + fingerprint?: ((repoRoot: string, includeUntracked: boolean) => Promise) | undefined; + scan?: typeof scanRepo | undefined; + wait?: ((milliseconds: number, signal?: AbortSignal) => Promise) | undefined; +}; + +/** + * Continuously verifies the working tree against a saved plan. The lightweight Git + * fingerprint prevents full repository scans while nothing has changed; every emitted + * update is backed by a fresh scan and a recalculated Impact Graph. + */ +export async function watchRepository(input: WatchRepositoryInput): Promise { + const intervalMs = input.intervalMs ?? 1_500; + const includeUntracked = input.includeUntracked === true; + const fingerprint = input.fingerprint ?? fingerprintWorkingTree; + const scan = input.scan ?? scanRepo; + const wait = input.wait ?? waitForInterval; + let previousFingerprint: string | undefined; + let history: Awaited>["history"]; + let sequence = 0; + let lastUpdate: WatchUpdate | undefined; + + while (!input.signal?.aborted) { + const currentFingerprint = await fingerprint(input.repoRoot, includeUntracked); + if (currentFingerprint !== previousFingerprint) { + const repo = await scan({ + repoRoot: input.repoRoot, + workingTree: true, + includeUntracked, + useCache: false, + includeHistory: history === undefined + }); + const unresolved = repo.diagnostics.find((entry) => entry.code === "diff-unavailable"); + if (unresolved) throw new Error(`${unresolved.message} Watch needs a local Git working tree.`); + history ??= repo.history; + if (!repo.history && history) repo.history = history; + + sequence += 1; + lastUpdate = { + watchVersion: 1, + sequence, + observedAt: new Date().toISOString(), + verification: verifyPlan(input.report, repo) + }; + await input.onUpdate(lastUpdate); + previousFingerprint = currentFingerprint; + } + + if (input.once) return lastUpdate; + try { + await wait(intervalMs, input.signal); + } catch (error) { + if (input.signal?.aborted || (error instanceof Error && error.name === "AbortError")) return lastUpdate; + throw error; + } + } + return lastUpdate; +} + +export function renderWatchUpdate(update: WatchUpdate, format: "markdown" | "json"): string { + if (format === "json") return `${JSON.stringify(update)}\n`; + return [ + `## Watch update ${update.sequence} — ${update.observedAt}`, + "", + renderVerifyMarkdown(update.verification).trimEnd(), + "" + ].join("\n"); +} + +export async function fingerprintWorkingTree(repoRoot: string, includeUntracked: boolean): Promise { + const git = async (args: string[]) => (await exec("git", args, { + cwd: repoRoot, + encoding: "buffer", + maxBuffer: GIT_MAX_BUFFER, + windowsHide: true, + env: { + ...process.env, + GIT_CONFIG_COUNT: "2", + GIT_CONFIG_KEY_0: "core.fsmonitor", + GIT_CONFIG_VALUE_0: "false", + GIT_CONFIG_KEY_1: "diff.external", + GIT_CONFIG_VALUE_1: "" + } + })).stdout; + + try { + const [status, untracked] = await Promise.all([ + git(["status", "--porcelain=v1", "-z", includeUntracked ? "--untracked-files=all" : "--untracked-files=no"]), + includeUntracked ? git(["ls-files", "--others", "--exclude-standard", "-z"]) : Promise.resolve(Buffer.alloc(0)) + ]); + let diff: Buffer; + let contentPaths = untracked; + try { + diff = await git(["diff", "--no-ext-diff", "--no-textconv", "--binary", "HEAD", "--"]); + } catch (error) { + if (!isMissingHead(error)) throw error; + diff = Buffer.alloc(0); + contentPaths = Buffer.concat([contentPaths, await git(["ls-files", "--cached", "-z"])]); + } + const hash = createHash("sha256").update(status).update(diff).update(contentPaths); + if (contentPaths.length > 0) await hashWorkingFiles(repoRoot, contentPaths, hash); + return hash.digest("hex"); + } catch (error) { + throw new Error(`Could not inspect the working tree: ${error instanceof Error ? error.message : String(error)}`); + } +} + +function isMissingHead(error: unknown): boolean { + if (!(error instanceof Error)) return false; + const stderr = "stderr" in error ? String((error as Error & { stderr?: unknown }).stderr ?? "") : ""; + return /bad revision ['"]?HEAD|unknown revision.*HEAD|ambiguous argument ['"]?HEAD/i.test(`${error.message}\n${stderr}`); +} + +async function hashWorkingFiles(repoRoot: string, paths: Buffer, hash: ReturnType): Promise { + for (const path of paths.toString("utf8").split("\0").filter(Boolean).slice(0, 25_000)) { + const absolute = resolve(repoRoot, path); + const distance = relative(repoRoot, absolute); + if (!distance || distance === ".." || distance.startsWith(`..${sep}`) || isAbsolute(distance)) continue; + hash.update(path).update("\0"); + try { + const info = await lstat(absolute); + if (info.isSymbolicLink()) { + hash.update("symlink\0").update(await readlink(absolute)); + } else if (info.isFile()) { + for await (const chunk of createReadStream(absolute)) hash.update(chunk as Buffer); + } + } catch { + // A file can disappear between `git ls-files` and the read. Its next fingerprint + // records the new state; this pass stays alive instead of crashing the monitor. + hash.update("unavailable\0"); + } + } +} + +async function waitForInterval(milliseconds: number, signal?: AbortSignal): Promise { + await delay(milliseconds, undefined, signal ? { signal } : undefined); +} diff --git a/packages/cli/test/analysis-commands.test.ts b/packages/cli/test/analysis-commands.test.ts new file mode 100644 index 0000000..66a15e4 --- /dev/null +++ b/packages/cli/test/analysis-commands.test.ts @@ -0,0 +1,67 @@ +import { describe, expect, it, vi } from "vitest"; +import { runContextCommand, runGraphCommand } from "../src/analysis-commands.js"; +import type { AnalyzedRepository } from "../src/analysis-source.js"; + +const analysis: AnalyzedRepository = { + task: "resetPassword emails fail", + report: { + reportVersion: 1, + summary: "one file", + contextFiles: [{ rank: 1, path: "src/reset.ts", score: 20, confidence: "high", reasons: ["defines resetPassword"] }], + impact: { + seeds: ["src/reset.ts"], + files: [{ path: "test/reset.test.ts", score: 8, confidence: "high", evidence: [{ kind: "test-route", seed: "src/reset.ts", reason: "routed test" }] }], + inspectionOrder: ["src/reset.ts", "test/reset.test.ts"], + history: { available: false, eligibleCommits: 0, shallow: false, truncated: false } + }, + testRoutes: [], risks: [], changedFiles: [], diagnostics: [] + }, + repo: { + root: "/repo", + files: [ + { path: "src/reset.ts", extension: ".ts", sizeBytes: 60, isTest: false, isSource: true, kind: "code", textSample: "export function resetPassword() { return true; }\n", textSampleComplete: true }, + { path: "test/reset.test.ts", extension: ".ts", sizeBytes: 40, isTest: true, isSource: true, kind: "code", textSample: "test('resetPassword', () => {});\n", textSampleComplete: true } + ], + packageScripts: [], changedFiles: [], diffText: "", packageManager: "npm", diagnostics: [] + } +}; + +function capture() { + const stdout: string[] = []; + const stderr: string[] = []; + return { stdout, stderr, io: { stdout: (text: string) => stdout.push(text), stderr: (text: string) => stderr.push(text) } }; +} + +describe("context and graph commands", () => { + it("builds a JSON context pack with the requested budget", async () => { + const output = capture(); + const analyze = vi.fn(async () => analysis); + expect(await runContextCommand(["--issue", "resetPassword emails fail", "--budget", "512", "--format", "json"], { ...output.io, analyze })).toBe(0); + expect(analyze).toHaveBeenCalledWith(expect.objectContaining({ issueText: "resetPassword emails fail" })); + expect(JSON.parse(output.stdout.join(""))).toMatchObject({ contextVersion: 1, budgetTokens: 512 }); + }); + + it("exports Mermaid and JSON graphs", async () => { + const mermaid = capture(); + const json = capture(); + expect(await runGraphCommand(["--issue", "resetPassword"], { ...mermaid.io, analyze: async () => analysis })).toBe(0); + expect(mermaid.stdout.join("")).toContain("flowchart TD"); + expect(await runGraphCommand(["--issue", "resetPassword", "--format=json"], { ...json.io, analyze: async () => analysis })).toBe(0); + expect(JSON.parse(json.stdout.join(""))).toMatchObject({ graphVersion: 1 }); + }); + + it("rejects missing task signals, command-specific options, local refs, and unsupported formats", async () => { + const missing = capture(); + const budget = capture(); + const graphBudget = capture(); + const localRef = capture(); + const format = capture(); + expect(await runContextCommand([], missing.io)).toBe(1); + expect(await runContextCommand(["--issue", "x", "--budget", "255"], budget.io)).toBe(1); + expect(await runGraphCommand(["--issue", "x", "--budget", "512"], graphBudget.io)).toBe(1); + expect(graphBudget.stderr.join("")).toContain("Unknown graph option"); + expect(await runContextCommand(["--issue", "x", "--repo", ".", "--ref", "main"], localRef.io)).toBe(1); + expect(localRef.stderr.join("")).toContain("--ref only applies"); + expect(await runGraphCommand(["--issue", "x", "--format", "dot"], format.io)).toBe(1); + }); +}); diff --git a/packages/cli/test/benchmark.test.ts b/packages/cli/test/benchmark.test.ts new file mode 100644 index 0000000..207b29f --- /dev/null +++ b/packages/cli/test/benchmark.test.ts @@ -0,0 +1,68 @@ +import { execFile } from "node:child_process"; +import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { promisify } from "node:util"; +import { describe, expect, it } from "vitest"; +import { benchmarkRepository, renderRepositoryBenchmark } from "../src/benchmark.js"; + +const exec = promisify(execFile); + +describe("repository benchmark", () => { + it("uses parent snapshots and history cut off before the target change", { timeout: 30_000 }, async () => { + const root = await mkdtemp(join(tmpdir(), "fixmap-repository-benchmark-")); + try { + await exec("git", ["init", "-b", "main"], { cwd: root }); + await exec("git", ["config", "user.name", "FixMap Test"], { cwd: root }); + await exec("git", ["config", "user.email", "fixmap@example.invalid"], { cwd: root }); + await mkdir(join(root, "src", "auth"), { recursive: true }); + await mkdir(join(root, "dist", "auth"), { recursive: true }); + await mkdir(join(root, "test", "auth"), { recursive: true }); + await writeFile(join(root, ".gitattributes"), "*.ts filter=fixmap-danger\n"); + await writeFile(join(root, "package.json"), JSON.stringify({ scripts: { test: "vitest run" } })); + await writeFile(join(root, "src", "auth", "reset.ts"), "export const resetPassword = () => 'first';\n"); + await writeFile(join(root, "dist", "auth", "reset.js"), "export const resetPassword = () => 'first';\n"); + await writeFile(join(root, "src", "session.ts"), "export const sessionExpiry = 10;\n"); + await writeFile(join(root, "test", "auth", "reset.test.ts"), "import '../../src/auth/reset';\n"); + await commit(root, "initial authentication implementation"); + + await writeFile(join(root, "src", "auth", "reset.ts"), "import { sessionExpiry } from '../session'; export const resetPassword = () => sessionExpiry;\n"); + await writeFile(join(root, "dist", "auth", "reset.js"), "export const resetPassword = () => 20;\n"); + await writeFile(join(root, "src", "session.ts"), "export const sessionExpiry = 20;\n"); + await writeFile(join(root, "test", "auth", "reset.test.ts"), "import '../../src/auth/reset'; test('expiry', () => true);\n"); + await commit(root, "fix password reset session expiration"); + + await writeFile(join(root, "src", "auth", "reset.ts"), "import { sessionExpiry } from '../session'; export const resetPassword = () => sessionExpiry + 1;\n"); + await writeFile(join(root, "dist", "auth", "reset.js"), "export const resetPassword = () => 31;\n"); + await writeFile(join(root, "src", "session.ts"), "export const sessionExpiry = 30;\n"); + await writeFile(join(root, "test", "auth", "reset.test.ts"), "import '../../src/auth/reset'; test('expiration', () => true);\n"); + await commit(root, "correct password reset expiration behavior"); + + // A local checkout would execute this configured filter unless the benchmark + // neutralizes repository-defined drivers before creating its parent worktree. + await exec("git", ["config", "filter.fixmap-danger.smudge", "this-command-must-not-exist-fixmap"], { cwd: root }); + await exec("git", ["config", "filter.fixmap-danger.required", "true"], { cwd: root }); + + const result = await benchmarkRepository({ repoRoot: root, last: 1 }); + + expect(result.eligibleCases).toBe(1); + expect(result.safeguards).toMatchObject({ parentSnapshots: true, historyCutoff: "target-parent", primaryTargets: "changed-maintained-non-test-code", checkoutFiltersDisabled: true, repositoryCodeExecuted: false }); + expect(result.cases[0]?.mentionsExpectedPath).toBe(false); + expect(result.cases[0]?.expected).toEqual(expect.arrayContaining(["src/auth/reset.ts", "src/session.ts"])); + expect(result.cases[0]?.expected).not.toContain("dist/auth/reset.js"); + expect(result.cases[0]?.arms.impact.top5Paths).toContain("src/session.ts"); + expect(result.cases[0]?.arms.impact.top5Paths).not.toContain("package-lock.json"); + expect(renderRepositoryBenchmark(result)).toContain("Every case used the target commit's parent snapshot"); + + const worktrees = (await exec("git", ["worktree", "list", "--porcelain"], { cwd: root })).stdout; + expect(worktrees.match(/^worktree /gm)).toHaveLength(1); + } finally { + await rm(root, { recursive: true, force: true }); + } + }); +}); + +async function commit(root: string, message: string): Promise { + await exec("git", ["add", "."], { cwd: root }); + await exec("git", ["commit", "-m", message], { cwd: root }); +} diff --git a/packages/cli/test/cli-runner.test.ts b/packages/cli/test/cli-runner.test.ts index 126abf2..8731bb4 100644 --- a/packages/cli/test/cli-runner.test.ts +++ b/packages/cli/test/cli-runner.test.ts @@ -7,6 +7,8 @@ import { describe, expect, it, vi } from "vitest"; import { parseArgs, runCli } from "../src/cli-runner.js"; import { installAgentCommands } from "../src/agent-setup.js"; import type { FixMapReport } from "@aryam/fixmap-core"; +import type { RepositoryBenchmark } from "../src/benchmark.js"; +import type { WatchUpdate } from "../src/watch.js"; const exec = promisify(execFile); @@ -95,6 +97,35 @@ describe("CLI argument handling", () => { expect(buildReport).toHaveBeenCalledWith(expect.objectContaining({ issueText: url })); }); + it("renders the compact agent format with stable workflow headings", async () => { + const io = capture(); + const impactReport: FixMapReport = { + ...report, + impact: { + seeds: ["README.md"], + files: [{ + path: "docs/guide.md", + score: 6, + confidence: "medium", + evidence: [{ kind: "imported-by", seed: "README.md", reason: "this file imports README.md" }] + }], + inspectionOrder: ["README.md", "docs/guide.md"], + history: { available: false, eligibleCommits: 0, shallow: false, truncated: false } + } + }; + + expect(await runCli(["plan", "--issue", "improve docs", "--format", "agent"], { + ...io.dependencies, + buildReport: vi.fn(async () => impactReport) + })).toBe(0); + + const output = io.stdout.join(""); + for (const heading of ["EDIT CANDIDATE:", "INSPECT:", "TEST:", "RISK:", "AVOID:", "UNCERTAINTY:"]) { + expect(output).toContain(heading); + } + expect(output).toContain("docs/guide.md"); + }); + it.each([ ["--version"], ["-v"], @@ -126,7 +157,7 @@ describe("CLI argument handling", () => { const io = capture(); expect(await runCli(["features"], io.dependencies)).toBe(0); - for (const feature of ["Plan", "Explain", "Compare", "Verify", "Validate", "Doctor", "MCP", "Focus", "Live changes", "Fresh scan"]) { + for (const feature of ["Plan", "Context Pack", "Graph export", "Explain", "Compare", "Verify", "Watch", "Validate", "Doctor", "MCP", "Focus", "Live changes", "Fresh scan"]) { expect(io.stdout.join("")).toContain(`**${feature}**`); } expect(io.stdout.join("")).toContain("fixmap setup"); @@ -138,15 +169,85 @@ describe("CLI argument handling", () => { expect(io.stdout.join("")).toContain("Usage: fixmap features [--format markdown|json]"); }); + it("runs the repository benchmark through its isolated command surface", async () => { + const io = capture(); + const result = { + benchmarkVersion: 1, + generatedAt: "2026-08-12T00:00:00.000Z", + repository: "C:/repo", + requestedCommits: 5, + eligibleCases: 1, + skipped: {}, + safeguards: { parentSnapshots: true, historyCutoff: "target-parent", maxChangedFiles: 30, sameScannedCorpus: true, repositoryCodeExecuted: false }, + cohorts: {}, + impactSecondary: { hits: 0, of: 0, recall: null }, + cases: [] + } as unknown as RepositoryBenchmark; + const benchmarkRepository = vi.fn(async () => result); + + expect(await runCli(["benchmark", "--repo", "C:/repo", "--last", "5", "--format", "json"], { + ...io.dependencies, + benchmarkRepository, + renderBenchmark: () => "unused" + })).toBe(0); + + expect(benchmarkRepository).toHaveBeenCalledWith(expect.objectContaining({ repoRoot: "C:/repo", last: 5 })); + expect(JSON.parse(io.stdout.join(""))).toMatchObject({ benchmarkVersion: 1, eligibleCases: 1 }); + }); + + it("rejects remote and out-of-range benchmark inputs before touching history", async () => { + const remote = capture(); + const invalid = capture(); + const benchmarkRepository = vi.fn(async () => { throw new Error("must not run"); }); + + expect(await runCli(["benchmark", "--repo", "https://github.com/o/r"], { ...remote.dependencies, benchmarkRepository })).toBe(1); + expect(await runCli(["benchmark", "--last", "101"], { ...invalid.dependencies, benchmarkRepository })).toBe(1); + expect(benchmarkRepository).not.toHaveBeenCalled(); + }); + + it("runs watch once through its bounded command surface", async () => { + const root = await mkdtemp(join(tmpdir(), "fixmap-watch-command-")); + const reportPath = join(root, "plan.json"); + await writeFile(reportPath, JSON.stringify(report)); + const io = capture(); + const update: WatchUpdate = { + watchVersion: 1, + sequence: 1, + observedAt: "2026-08-12T00:00:00.000Z", + verification: { summary: "No drift.", changedFiles: [], findings: [], diagnostics: [] } + }; + const watchRepository = vi.fn(async (input: Parameters[1]["watchRepository"]>>[0]) => { + await input.onUpdate(update); + return update; + }); + + expect(await runCli(["watch", "--report", reportPath, "--repo", root, "--once", "--format", "json"], { + ...io.dependencies, + watchRepository, + renderWatchUpdate: (value) => `${JSON.stringify(value)}\n` + })).toBe(0); + expect(watchRepository).toHaveBeenCalledWith(expect.objectContaining({ repoRoot: root, once: true })); + expect(JSON.parse(io.stdout.join(""))).toMatchObject({ watchVersion: 1, sequence: 1 }); + }); + + it("rejects unsafe watch inputs before starting a monitor", async () => { + const missing = capture(); + const remote = capture(); + const watchRepository = vi.fn(async () => undefined); + expect(await runCli(["watch", "--once"], { ...missing.dependencies, watchRepository })).toBe(1); + expect(await runCli(["watch", "--report", "plan.json", "--repo", "https://github.com/o/r"], { ...remote.dependencies, watchRepository })).toBe(1); + expect(watchRepository).not.toHaveBeenCalled(); + }); + it("keeps the npm package README aligned with the complete public feature catalog", async () => { const npmReadme = await readFile(new URL("../README.md", import.meta.url), "utf8"); for (const feature of [ - "Plan", "Explain", "Compare", "Verify", "Validate", "Doctor", "MCP", + "Plan", "Context Pack", "Graph export", "Explain", "Compare", "Verify", "Watch", "Validate", "Doctor", "MCP", "Focus controls", "Live changes", "Exact-state cache", "Slash-command discovery" ]) { expect(npmReadme).toContain(`**${feature}**`); } - for (const command of ["fixmap setup", "fixmap features", "fixmap validate", "--no-cache"]) { + for (const command of ["fixmap setup", "fixmap features", "fixmap validate", "fixmap context", "fixmap graph", "fixmap watch", "--no-cache"]) { expect(npmReadme).toContain(command); } }); @@ -386,7 +487,7 @@ describe("CLI argument handling", () => { const exitCode = await runCli(["plan", "--issue", "test", "--format", "yaml", "--mystery"], io.dependencies); expect(exitCode).toBe(1); - expect(io.stderr.join("")).toContain('--format received "yaml"; expected "markdown" or "json"'); + expect(io.stderr.join("")).toContain('--format received "yaml"; expected "markdown", "json", or "agent"'); expect(io.stderr.join("")).toContain("Unknown option(s): --mystery"); expect(io.stderr.join("")).not.toContain("Unknown option(s): yaml"); }); diff --git a/packages/cli/test/mcp.test.ts b/packages/cli/test/mcp.test.ts index 4c32404..3c777e6 100644 --- a/packages/cli/test/mcp.test.ts +++ b/packages/cli/test/mcp.test.ts @@ -40,7 +40,7 @@ describe("fixmap mcp server", () => { }); expect(parsePlanArguments({ issue: "task", format: "yaml" })).toEqual({ success: false, - message: '"format" must be either "markdown" or "json".' + message: '"format" must be "markdown", "json", or "agent".' }); expect(parsePlanArguments({ issue: "task", surprise: true })).toEqual({ success: false, @@ -149,13 +149,13 @@ describe("fixmap mcp server", () => { expect(report.contextFiles).toHaveLength(1); }); - it("advertises the complete plan, explain, compare, verify, and doctor workflow", async () => { + it("advertises the complete plan, context, graph, explain, compare, verify, and doctor workflow", async () => { const client = await connectClient(); const tools = await client.listTools(); expect(tools.tools.map((tool) => tool.name)).toEqual([ - "fixmap_plan", "fixmap_verify", "fixmap_explain", "fixmap_compare", "fixmap_doctor" + "fixmap_plan", "fixmap_context", "fixmap_graph", "fixmap_verify", "fixmap_explain", "fixmap_compare", "fixmap_doctor" ]); const plan = tools.tools.find((tool) => tool.name === "fixmap_plan"); const verify = tools.tools.find((tool) => tool.name === "fixmap_verify"); @@ -167,6 +167,12 @@ describe("fixmap mcp server", () => { expect(plan?.inputSchema.additionalProperties).toBe(false); expect(plan?.inputSchema.properties?.repo?.description).toContain("public GitHub HTTPS"); expect(plan?.inputSchema.properties?.issue?.description).toContain("GitHub issue URL"); + const context = tools.tools.find((tool) => tool.name === "fixmap_context"); + expect(context?.inputSchema.properties?.budget).toBeDefined(); + expect(context?.inputSchema.additionalProperties).toBe(false); + const graph = tools.tools.find((tool) => tool.name === "fixmap_graph"); + expect(graph?.inputSchema.properties?.format?.description).toContain("mermaid"); + expect(graph?.inputSchema.additionalProperties).toBe(false); expect(verify).toBeDefined(); expect(Object.keys(verify?.inputSchema.properties ?? {}).sort()).toEqual( ["report", "diff", "base", "head", "repo", "workingTree", "includeUntracked", "format", "noCache"].sort() @@ -189,6 +195,24 @@ describe("fixmap mcp server", () => { expect(doctor?.inputSchema.additionalProperties).toBe(false); }); + it("returns bounded Context Packs and Mermaid graphs through MCP", async () => { + const root = await createAuthFixture(); + const client = await connectClient(); + const context = await client.callTool({ + name: "fixmap_context", + arguments: { issue: "password reset emails fail", repo: root, budget: 512, format: "json" } + }); + expect(context.isError).toBeFalsy(); + expect(JSON.parse((context.content as Array<{ text: string }>)[0]!.text)).toMatchObject({ contextVersion: 1, budgetTokens: 512 }); + + const graph = await client.callTool({ + name: "fixmap_graph", + arguments: { issue: "password reset emails fail", repo: root, format: "mermaid" } + }); + expect(graph.isError).toBeFalsy(); + expect((graph.content as Array<{ text: string }>)[0]!.text).toContain("flowchart TD"); + }); + it("compares two reports through MCP", async () => { const client = await connectClient(); const base = { summary: "", testRoutes: [], risks: [], changedFiles: [], diagnostics: [] }; @@ -435,6 +459,22 @@ describe("fixmap mcp server", () => { expect(report.contextFiles[0]?.path).toBe("src/auth/reset-password.ts"); }); + it("returns compact agent output when asked", async () => { + const root = await createAuthFixture(); + const client = await connectClient(); + + const result = await client.callTool({ + name: "fixmap_plan", + arguments: { issue: "password reset emails fail", repo: root, format: "agent" } + }); + + const text = (result.content as Array<{ type: string; text: string }>)[0]?.text ?? ""; + expect(result.isError).toBeFalsy(); + expect(text).toContain("EDIT CANDIDATE:"); + expect(text).toContain("INSPECT:"); + expect(text).toContain("src/auth/reset-password.ts"); + }); + it("analyzes a public GitHub URL through an isolated temporary checkout", async () => { const client = await connectClient({ clonePublicRepository: async (_url, destination, _hooks, ref) => { diff --git a/packages/cli/test/watch.test.ts b/packages/cli/test/watch.test.ts new file mode 100644 index 0000000..dc5eda4 --- /dev/null +++ b/packages/cli/test/watch.test.ts @@ -0,0 +1,78 @@ +import { execFile } from "node:child_process"; +import { mkdtemp, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { promisify } from "node:util"; +import { describe, expect, it, vi } from "vitest"; +import type { FixMapReport, RepoMap } from "@aryam/fixmap-core"; +import { fingerprintWorkingTree, renderWatchUpdate, watchRepository } from "../src/watch.js"; + +const exec = promisify(execFile); + +const report: FixMapReport = { + summary: "One candidate.", + contextFiles: [{ rank: 1, path: "src/a.ts", score: 5, confidence: "medium", reasons: ["match"] }], + testRoutes: [], + risks: [], + changedFiles: [], + diagnostics: [] +}; + +const repo = (changedFiles: string[], history = false): RepoMap => ({ + root: "/repo", + files: [{ path: "src/a.ts", extension: ".ts", sizeBytes: 1, isTest: false, isSource: true, kind: "code", textSample: "a" }], + trackedFiles: ["src/a.ts"], + packageScripts: [], + changedFiles, + diffText: changedFiles.length > 0 ? "diff" : "", + packageManager: "npm", + diagnostics: [], + ...(history ? { history: { commits: [], eligibleCommits: 0, shallow: false, truncated: false } } : {}) +}); + +describe("working-tree watch", () => { + it("emits only changed states and reuses the initial history snapshot", async () => { + const fingerprints = ["a", "a", "b"]; + const scans = [repo([], true), repo(["src/a.ts"])]; + const updates: unknown[] = []; + const controller = new AbortController(); + const scan = vi.fn(async () => scans.shift()!); + + await watchRepository({ + repoRoot: "/repo", + report, + signal: controller.signal, + fingerprint: async () => fingerprints.shift()!, + scan, + wait: async () => { if (fingerprints.length === 0) controller.abort(); }, + onUpdate: (update) => { updates.push(update); } + }); + + expect(updates).toHaveLength(2); + expect(scan).toHaveBeenCalledTimes(2); + expect(scan.mock.calls[0]?.[0]).toMatchObject({ includeHistory: true, workingTree: true, useCache: false }); + expect(scan.mock.calls[1]?.[0]).toMatchObject({ includeHistory: false, workingTree: true, useCache: false }); + expect((updates[1] as { verification: { changedFiles: string[] } }).verification.changedFiles).toEqual(["src/a.ts"]); + }); + + it("renders JSON Lines and readable markdown updates", () => { + const update: Parameters[0] = { + watchVersion: 1, + sequence: 2, + observedAt: "2026-08-12T00:00:00.000Z", + verification: { summary: "No changes to verify.", changedFiles: [], findings: [], diagnostics: [] } + }; + expect(JSON.parse(renderWatchUpdate(update, "json"))).toMatchObject({ sequence: 2 }); + expect(renderWatchUpdate(update, "markdown")).toContain("Watch update 2"); + }); + + it("notices content changes to an existing untracked path", async () => { + const root = await mkdtemp(join(tmpdir(), "fixmap-watch-untracked-")); + await exec("git", ["init"], { cwd: root }); + await writeFile(join(root, "draft.ts"), "export const state = 'one';\n"); + const first = await fingerprintWorkingTree(root, true); + await writeFile(join(root, "draft.ts"), "export const state = 'two';\n"); + const second = await fingerprintWorkingTree(root, true); + expect(second).not.toBe(first); + }); +}); diff --git a/packages/core/README.md b/packages/core/README.md index 5dc67d3..c6085de 100644 --- a/packages/core/README.md +++ b/packages/core/README.md @@ -7,7 +7,16 @@ Most users want the CLI and MCP server instead: [`@aryam/fixmap`](https://www.np ## Usage ```ts -import { buildFixMapReport, renderMarkdownReport } from "@aryam/fixmap-core"; +import { + buildContextPack, + buildFixMapGraph, + buildFixMapReport, + renderAgentReport, + renderContextPackMarkdown, + renderFixMapGraphMermaid, + renderMarkdownReport, + scanRepo +} from "@aryam/fixmap-core"; const report = await buildFixMapReport({ repoRoot: "/path/to/repo", @@ -15,11 +24,16 @@ const report = await buildFixMapReport({ }); console.log(renderMarkdownReport(report)); +console.log(renderAgentReport(report)); + +const repo = await scanRepo({ repoRoot: "/path/to/repo", includeHistory: true }); +console.log(renderContextPackMarkdown(buildContextPack({ report, repo, task: "password reset emails fail", budgetTokens: 10_000 }))); +console.log(renderFixMapGraphMermaid(buildFixMapGraph(report))); ``` -`buildFixMapReport` runs the full pipeline: scan the repository, resolve exclusions, rank context files against the task, route to the most relevant test commands, and collect risk notes and diagnostics. Exact git states can reuse a seven-day scan cache, while `useCache: false` forces a fresh scan and reports the bypass. +`buildFixMapReport` runs the full pipeline: scan the repository, resolve exclusions, rank primary context against the task, build a separate likely-impact view, route to relevant test commands, and collect risk notes and diagnostics. Impact evidence includes imports, reverse dependents, routed tests, and repeated bounded Git co-change relationships; it is explicitly an inspection aid rather than a claim that every related file must change. Exact git states can reuse a seven-day scan cache, while `useCache: false` forces a fresh scan and reports the bypass. -The package also exports the lower-level scanner, excluder, ranker, grounding, import-graph, test-routing, risk, comparison, explanation, verification, structural validation, and Markdown/JSON rendering APIs. `@aryam/fixmap-core/browser` exposes the filesystem-free report, Compare, Explain, Verify, validation, and rendering logic for browser applications. +The package also exports the lower-level scanner, excluder, ranker, BM25 retriever, grounding, import graph, Context Pack and Impact Graph builders, test routing, risk, comparison, explanation, verification, structural validation, and Markdown/JSON/agent/Mermaid rendering APIs. Context budgets use the deterministic estimate `ceil(UTF-8 bytes / 4)` for source and report sample truncation explicitly. `@aryam/fixmap-core/browser` exposes the filesystem-free report, Context Pack, Impact Graph, Compare, Explain, Verify, validation, and rendering logic for browser applications. JSON reports use `reportVersion: 1`. Additive fields and diagnostic codes may appear within that version; consumers should ignore unknown fields and use diagnostic severity as the stable fallback. diff --git a/packages/core/package.json b/packages/core/package.json index ff9e435..88360c9 100644 --- a/packages/core/package.json +++ b/packages/core/package.json @@ -1,7 +1,7 @@ { "name": "@aryam/fixmap-core", - "version": "0.8.9", - "description": "Deterministic local-first repository scanner, context ranker, and report renderer for coding agents.", + "version": "0.9.0", + "description": "Deterministic local-first repository scanner, context ranker, Impact Graph, and report renderer for coding agents.", "license": "MIT", "repository": { "type": "git", diff --git a/packages/core/src/browser.ts b/packages/core/src/browser.ts index 0519e1e..1142197 100644 --- a/packages/core/src/browser.ts +++ b/packages/core/src/browser.ts @@ -10,7 +10,11 @@ export { compareReports, renderComparisonMarkdown } from "./compare.js"; export { quoteCliValue } from "./cli-quote.js"; export type { CliShell } from "./cli-quote.js"; export { rankContextFiles } from "./rank.js"; -export { buildReportFromRepo, buildRiskNotes, buildTestRoutes, renderJsonReport, renderMarkdownReport } from "./report.js"; +export { rankByBm25, retrievalQueryTerms, retrievalTokens, taskMentionsExpectedPath } from "./retrieval.js"; +export { buildReportFromRepo, buildRiskNotes, buildTestRoutes, renderAgentReport, renderJsonReport, renderMarkdownReport } from "./report.js"; +export { buildContextPack, estimateContextTokens, renderContextPackMarkdown, type ContextPack, type ContextSnippet } from "./context.js"; +export { buildFixMapGraph, renderFixMapGraphMermaid, type FixMapGraph } from "./graph.js"; +export { buildImpactMap } from "./impact.js"; export { tokenizePath, tokenizeText } from "./signals.js"; export { renderVerifyMarkdown, verifyPlan } from "./verify.js"; export { validateFixMapReport } from "./validate.js"; @@ -20,6 +24,9 @@ export type { PathExcluder } from "./exclude.js"; export type { ReportComparison } from "./compare.js"; export type { FixMapReport, + ImpactEvidence, + ImpactFile, + ImpactMap, RankedFile, RepoFile, RepoMap, diff --git a/packages/core/src/context.ts b/packages/core/src/context.ts new file mode 100644 index 0000000..3dd67cf --- /dev/null +++ b/packages/core/src/context.ts @@ -0,0 +1,270 @@ +import { extractTaskSignals } from "./signals.js"; +import { markdownCode } from "./markdown.js"; +import type { FixMapReport, RepoFile, RepoMap } from "./types.js"; + +export type ContextSnippet = { + path: string; + role: "primary" | "supporting"; + startLine: number; + endLine: number; + language: string; + reason: string; + confidence: "high" | "medium" | "low"; + estimatedTokens: number; + sourceTruncated: boolean; + content: string; +}; + +export type ContextPack = { + contextVersion: 1; + task: string; + budgetTokens: number; + estimatedSourceTokens: number; + tokenEstimate: "utf8-bytes-divided-by-4"; + snippets: ContextSnippet[]; + omitted: Array<{ path: string; reason: "budget" | "unavailable" | "empty" }>; +}; + +type Candidate = { + path: string; + role: ContextSnippet["role"]; + confidence: ContextSnippet["confidence"]; + reason: string; +}; + +const MAX_CANDIDATES = 15; +const MIN_SNIPPET_TOKENS = 48; + +/** + * Turns an already-grounded plan into bounded source ranges. This remains deliberately + * lexical: no model chooses a symbol, and the estimate is stable across tokenizers. + */ +export function buildContextPack(input: { + report: FixMapReport; + repo: RepoMap; + task: string; + budgetTokens: number; +}): ContextPack { + const candidates = contextCandidates(input.report); + const fileByPath = new Map(input.repo.files.map((file) => [file.path, file])); + const signals = extractTaskSignals({ + issueText: input.task, + diffText: input.repo.diffText, + changedFiles: input.repo.changedFiles + }); + const terms = [...signals.tokens]; + const identifiers = [...signals.identifiers, ...signals.memberMentions]; + const snippets: ContextSnippet[] = []; + const omitted: ContextPack["omitted"] = []; + let remaining = input.budgetTokens; + + for (const candidate of candidates) { + const file = fileByPath.get(candidate.path); + if (!file) { + omitted.push({ path: candidate.path, reason: "unavailable" }); + continue; + } + if (!file.textSample.trim()) { + omitted.push({ path: candidate.path, reason: "empty" }); + continue; + } + if (remaining < MIN_SNIPPET_TOKENS) { + omitted.push({ path: candidate.path, reason: "budget" }); + continue; + } + + const share = Math.min( + remaining, + Math.max(MIN_SNIPPET_TOKENS, Math.floor(input.budgetTokens * (candidate.role === "primary" ? 0.4 : 0.25))) + ); + const range = selectRange(file, terms, identifiers, share); + if (!range) { + omitted.push({ path: candidate.path, reason: "budget" }); + continue; + } + snippets.push({ + path: candidate.path, + role: candidate.role, + startLine: range.startLine, + endLine: range.endLine, + language: languageForPath(candidate.path), + reason: candidate.reason, + confidence: candidate.confidence, + estimatedTokens: range.estimatedTokens, + sourceTruncated: file.textSampleComplete === false, + content: range.content + }); + remaining -= range.estimatedTokens; + } + + return { + contextVersion: 1, + task: input.task, + budgetTokens: input.budgetTokens, + estimatedSourceTokens: snippets.reduce((total, snippet) => total + snippet.estimatedTokens, 0), + tokenEstimate: "utf8-bytes-divided-by-4", + snippets, + omitted + }; +} + +export function estimateContextTokens(text: string): number { + return Math.ceil(new TextEncoder().encode(text).byteLength / 4); +} + +export function renderContextPackMarkdown(pack: ContextPack): string { + const lines = [ + "# FixMap Context", + "", + "## Task", + "", + pack.task || "No task text was supplied; ranges were selected from diff evidence.", + "", + `Source budget: ${pack.budgetTokens.toLocaleString("en-US")} estimated tokens. Included: ${pack.estimatedSourceTokens.toLocaleString("en-US")}.`, + "Estimate: UTF-8 bytes divided by four; headings and metadata are outside the source budget.", + "" + ]; + + for (const role of ["primary", "supporting"] as const) { + const selected = pack.snippets.filter((snippet) => snippet.role === role); + if (selected.length === 0) continue; + lines.push(`## ${role === "primary" ? "Primary" : "Supporting"} Context`, ""); + for (const snippet of selected) { + lines.push( + `### ${markdownCode(snippet.path)}:${snippet.startLine}-${snippet.endLine}`, + "", + `${snippet.confidence} confidence · ~${snippet.estimatedTokens.toLocaleString("en-US")} tokens · ${snippet.reason}` + + (snippet.sourceTruncated ? " · selected from the scanner's bounded text sample" : ""), + "", + `${safeFence(snippet.content)}${snippet.language}`, + snippet.content, + safeFence(snippet.content), + "" + ); + } + } + + if (pack.omitted.length > 0) { + lines.push("## Omitted", ""); + for (const entry of pack.omitted) lines.push(`- ${markdownCode(entry.path)}: ${entry.reason}`); + lines.push(""); + } + return `${lines.join("\n").trimEnd()}\n`; +} + +function contextCandidates(report: FixMapReport): Candidate[] { + const candidates: Candidate[] = report.contextFiles.map((file) => ({ + path: file.path, + role: "primary", + confidence: file.confidence, + reason: file.reasons.slice(0, 2).join("; ") || "ranked primary context" + })); + const seen = new Set(candidates.map((candidate) => candidate.path)); + for (const file of report.impact?.files ?? []) { + if (seen.has(file.path)) continue; + seen.add(file.path); + candidates.push({ + path: file.path, + role: "supporting", + confidence: file.confidence, + reason: file.evidence.slice(0, 2).map((evidence) => evidence.reason).join("; ") || "impact evidence" + }); + } + return candidates.slice(0, MAX_CANDIDATES); +} + +function selectRange( + file: RepoFile, + terms: string[], + identifiers: string[], + allowance: number +): { startLine: number; endLine: number; content: string; estimatedTokens: number } | undefined { + const lines = file.textSample.replace(/\r\n?/g, "\n").split("\n"); + const whole = lines.join("\n"); + const wholeTokens = estimateContextTokens(whole); + if (wholeTokens <= allowance) { + return { startLine: 1, endLine: lines.length, content: whole, estimatedTokens: wholeTokens }; + } + + let anchor = 0; + let bestScore = Number.NEGATIVE_INFINITY; + for (let index = 0; index < lines.length; index += 1) { + const score = scoreLine(lines[index]!, terms, identifiers, index); + if (score > bestScore) { + bestScore = score; + anchor = index; + } + } + + let start = anchor; + let end = anchor; + let content = lines[anchor] ?? ""; + let tokens = estimateContextTokens(content); + if (tokens > allowance) { + const maxBytes = allowance * 4; + content = truncateUtf8(content, maxBytes); + tokens = estimateContextTokens(content); + } + let preferBefore = true; + while (tokens < allowance && (start > 0 || end < lines.length - 1)) { + const sides = preferBefore ? ["before", "after"] as const : ["after", "before"] as const; + let expanded = false; + for (const side of sides) { + const nextStart = side === "before" && start > 0 ? start - 1 : start; + const nextEnd = side === "after" && end < lines.length - 1 ? end + 1 : end; + if (nextStart === start && nextEnd === end) continue; + const proposed = lines.slice(nextStart, nextEnd + 1).join("\n"); + const proposedTokens = estimateContextTokens(proposed); + if (proposedTokens > allowance) continue; + start = nextStart; + end = nextEnd; + content = proposed; + tokens = proposedTokens; + expanded = true; + break; + } + if (!expanded) break; + preferBefore = !preferBefore; + } + if (tokens < MIN_SNIPPET_TOKENS && allowance >= MIN_SNIPPET_TOKENS && content.trim().length === 0) return undefined; + return { startLine: start + 1, endLine: end + 1, content, estimatedTokens: tokens }; +} + +function scoreLine(line: string, terms: string[], identifiers: string[], index: number): number { + const lower = line.toLowerCase(); + let score = -index / 100_000; + for (const identifier of identifiers) { + if (line.includes(identifier)) score += 12; + else if (lower.includes(identifier.toLowerCase())) score += 7; + } + for (const term of terms) if (lower.includes(term.toLowerCase())) score += 2; + if (/\b(?:class|function|interface|type|enum|def|fn|func|const|let|var)\b/.test(line)) score += 1; + return score; +} + +function truncateUtf8(text: string, maxBytes: number): string { + let output = ""; + let bytes = 0; + for (const character of text) { + const next = new TextEncoder().encode(character).byteLength; + if (bytes + next > maxBytes) break; + output += character; + bytes += next; + } + return output; +} + +function safeFence(content: string): string { + const longest = Math.max(0, ...[...content.matchAll(/`+/g)].map((match) => match[0].length)); + return "`".repeat(Math.max(3, longest + 1)); +} + +function languageForPath(path: string): string { + const basename = path.replaceAll("\\", "/").split("/").at(-1) ?? path; + const extension = basename.includes(".") ? basename.split(".").at(-1)!.toLowerCase() : ""; + return ({ + cjs: "javascript", js: "javascript", jsx: "jsx", mjs: "javascript", + ts: "typescript", tsx: "tsx", py: "python", rb: "ruby", rs: "rust", + yml: "yaml", md: "markdown", mdx: "mdx", sh: "bash", ps1: "powershell" + } as Record)[extension] ?? extension; +} diff --git a/packages/core/src/graph.ts b/packages/core/src/graph.ts new file mode 100644 index 0000000..3e90b00 --- /dev/null +++ b/packages/core/src/graph.ts @@ -0,0 +1,74 @@ +import type { FixMapReport, ImpactEvidence } from "./types.js"; + +export type FixMapGraph = { + graphVersion: 1; + nodes: Array<{ id: string; path: string; role: "primary" | "impact"; confidence: "high" | "medium" | "low" }>; + edges: Array<{ from: string; to: string; kind: ImpactEvidence["kind"]; label: string }>; +}; + +export function buildFixMapGraph(report: FixMapReport): FixMapGraph { + const primary = new Map(report.contextFiles.map((file) => [file.path, file.confidence])); + const paths = [...new Set([ + ...report.contextFiles.map((file) => file.path), + ...(report.impact?.files.map((file) => file.path) ?? []) + ])]; + const idByPath = new Map(paths.map((path, index) => [path, `n${index + 1}`])); + const nodes = paths.map((path) => { + const impact = report.impact?.files.find((file) => file.path === path); + return { + id: idByPath.get(path)!, + path, + role: primary.has(path) ? "primary" as const : "impact" as const, + confidence: primary.get(path) ?? impact?.confidence ?? "low" + }; + }); + const edges: FixMapGraph["edges"] = []; + for (const file of report.impact?.files ?? []) { + for (const evidence of file.evidence) { + const seedId = idByPath.get(evidence.seed); + const fileId = idByPath.get(file.path); + if (!seedId || !fileId) continue; + const reversed = evidence.kind === "imported-by"; + edges.push({ + from: reversed ? fileId : seedId, + to: reversed ? seedId : fileId, + kind: evidence.kind, + label: graphEdgeLabel(evidence) + }); + } + } + return { graphVersion: 1, nodes, edges }; +} + +export function renderFixMapGraphMermaid(graph: FixMapGraph): string { + const lines = ["flowchart TD"]; + for (const node of graph.nodes) { + lines.push(` ${node.id}["${escapeMermaid(node.path)}"]:::${node.role}`); + } + for (const edge of graph.edges) { + const connector = edge.kind === "co-change" ? "-.-" : "-->"; + lines.push(` ${edge.from} ${connector}|"${escapeMermaid(edge.label)}"| ${edge.to}`); + } + lines.push( + " classDef primary fill:#163d2d,stroke:#74f0ba,color:#ffffff,stroke-width:2px", + " classDef impact fill:#17233a,stroke:#7aa2f7,color:#ffffff" + ); + return `${lines.join("\n")}\n`; +} + +function graphEdgeLabel(evidence: ImpactEvidence): string { + if (evidence.kind === "imports") return "imports"; + if (evidence.kind === "imported-by") return "imports"; + if (evidence.kind === "test-route") return "routed test"; + return evidence.occurrences ? `co-change ×${evidence.occurrences}` : "co-change"; +} + +function escapeMermaid(value: string): string { + return value + .replaceAll("&", "&") + .replaceAll('"', """) + .replaceAll("<", "<") + .replaceAll(">", ">") + .replaceAll("\r", " ") + .replaceAll("\n", " "); +} diff --git a/packages/core/src/impact.ts b/packages/core/src/impact.ts new file mode 100644 index 0000000..85990e5 --- /dev/null +++ b/packages/core/src/impact.ts @@ -0,0 +1,156 @@ +import { buildImportGraph } from "./import-graph.js"; +import { isBackupPath, isGeneratedPath } from "./paths.js"; +import type { + ImpactEvidence, + ImpactFile, + ImpactMap, + RepoMap, + TestRoute +} from "./types.js"; + +const DEFAULT_IMPACT_LIMIT = 12; +const MAX_IMPACT_SEEDS = 3; +const MIN_CO_CHANGE_OCCURRENCES = 2; + +type Candidate = { + path: string; + score: number; + evidence: ImpactEvidence[]; +}; + +/** + * Builds a separate impact view instead of smuggling relationship evidence into the task + * ranking. A task match answers "where should I start?"; this answers "what should I inspect + * around that start?". Keeping both lists explicit prevents a historical companion from + * masquerading as a file named by the task. + */ +export function buildImpactMap( + repo: RepoMap, + requestedSeeds: string[], + testRoutes: TestRoute[] = [], + limit = DEFAULT_IMPACT_LIMIT +): ImpactMap { + const repositoryPaths = new Set(repo.files.map((file) => file.path)); + const seeds = [...new Set(requestedSeeds)] + .filter((path) => repositoryPaths.has(path)) + .slice(0, MAX_IMPACT_SEEDS); + const seedSet = new Set(seeds); + const candidates = new Map(); + + const addEvidence = (path: string, score: number, evidence: ImpactEvidence): void => { + if (seedSet.has(path) || !repositoryPaths.has(path) || isGeneratedPath(path) || isBackupPath(path)) return; + const current = candidates.get(path) ?? { path, score: 0, evidence: [] }; + if (!current.evidence.some((entry) => entry.kind === evidence.kind && entry.seed === evidence.seed)) { + current.evidence.push(evidence); + current.score += score; + } + candidates.set(path, current); + }; + + const graph = buildImportGraph(repo.files); + for (const seed of seeds) { + for (const imported of [...(graph.imports.get(seed) ?? [])].sort((a, b) => a.localeCompare(b))) { + addEvidence(imported, 4, { + kind: "imports", + seed, + reason: `${seed} imports this file` + }); + } + for (const importer of [...(graph.importedBy.get(seed) ?? [])].sort((a, b) => a.localeCompare(b))) { + addEvidence(importer, 6, { + kind: "imported-by", + seed, + reason: `this file imports ${seed}` + }); + } + } + + for (const route of testRoutes.filter((entry) => entry.kind === "test")) { + for (const path of route.relatedFiles) { + const seed = nearestSeed(path, seeds) ?? seeds[0]; + if (!seed) continue; + addEvidence(path, 7, { + kind: "test-route", + seed, + reason: `routed test for ${seed} via ${route.command}` + }); + } + } + + const history = repo.history; + if (history) { + for (const seed of seeds) { + const seedCommits = history.commits.filter((commit) => commit.files.includes(seed)); + const coOccurrences = new Map(); + for (const commit of seedCommits) { + for (const path of commit.files) { + if (path !== seed && repositoryPaths.has(path)) { + coOccurrences.set(path, (coOccurrences.get(path) ?? 0) + 1); + } + } + } + for (const [path, occurrences] of coOccurrences) { + if (occurrences < MIN_CO_CHANGE_OCCURRENCES) continue; + const strength = occurrences / Math.max(seedCommits.length, 1); + const score = Math.min(8, 2 + Math.round(strength * 6)); + addEvidence(path, score, { + kind: "co-change", + seed, + reason: + `changed alongside ${seed} in ${occurrences} of its ${seedCommits.length} eligible ` + + `${seedCommits.length === 1 ? "change" : "changes"}`, + occurrences, + seedChanges: seedCommits.length + }); + } + } + } + + const files = [...candidates.values()] + .map(toImpactFile) + .sort((left, right) => right.score - left.score || left.path.localeCompare(right.path)) + .slice(0, Math.max(0, limit)); + + return { + seeds, + files, + inspectionOrder: [...seeds, ...files.map((file) => file.path)], + history: { + available: Boolean(history), + eligibleCommits: history?.commits.length ?? 0, + shallow: history?.shallow ?? false, + truncated: history?.truncated ?? false + } + }; +} + +function toImpactFile(candidate: Candidate): ImpactFile { + const kinds = new Set(candidate.evidence.map((entry) => entry.kind)); + const strongestCoChange = candidate.evidence + .filter((entry) => entry.kind === "co-change") + .reduce((best, entry) => Math.max(best, (entry.occurrences ?? 0) / Math.max(entry.seedChanges ?? 1, 1)), 0); + const confidence: ImpactFile["confidence"] = + kinds.has("test-route") || kinds.size >= 2 || strongestCoChange >= 0.6 + ? "high" + : kinds.has("imported-by") || kinds.has("imports") || strongestCoChange >= 0.3 + ? "medium" + : "low"; + return { + path: candidate.path, + score: candidate.score, + confidence, + evidence: candidate.evidence.sort((left, right) => left.kind.localeCompare(right.kind) || left.seed.localeCompare(right.seed)) + }; +} + +function nearestSeed(path: string, seeds: string[]): string | undefined { + const pathParts = path.split("/"); + return [...seeds] + .map((seed) => { + const seedParts = seed.split("/"); + let common = 0; + while (common < pathParts.length && common < seedParts.length && pathParts[common] === seedParts[common]) common += 1; + return { seed, common }; + }) + .sort((left, right) => right.common - left.common || left.seed.localeCompare(right.seed))[0]?.seed; +} diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts index b1f6559..a727906 100644 --- a/packages/core/src/index.ts +++ b/packages/core/src/index.ts @@ -1,4 +1,4 @@ -export { buildFixMapReport, resolveExclusions } from "./plan.js"; +export { buildFixMapAnalysis, buildFixMapReport, resolveExclusions } from "./plan.js"; export { buildPathExcluder, NO_EXCLUSIONS } from "./exclude.js"; export type { PathExcluder } from "./exclude.js"; export { compareReports, renderComparisonMarkdown } from "./compare.js"; @@ -15,10 +15,15 @@ export { buildRankingShape } from "./grounding.js"; export { buildImportGraph, findImportProximity } from "./import-graph.js"; +export { buildImpactMap } from "./impact.js"; export { detectPrimaryLanguage } from "./languages.js"; export type { LanguageDetection, PrimaryLanguage } from "./languages.js"; +export { isBackupPath, isGeneratedPath, moduleStem } from "./paths.js"; export { rankContextFiles } from "./rank.js"; -export { buildReportFromRepo, buildRiskNotes, buildSummary, buildTestRoutes, pathsForRiskArea, renderJsonReport, renderMarkdownReport } from "./report.js"; +export { rankByBm25, retrievalQueryTerms, retrievalTokens, taskMentionsExpectedPath } from "./retrieval.js"; +export { buildReportFromRepo, buildRiskNotes, buildSummary, buildTestRoutes, pathsForRiskArea, renderAgentReport, renderJsonReport, renderMarkdownReport } from "./report.js"; +export { buildContextPack, estimateContextTokens, renderContextPackMarkdown, type ContextPack, type ContextSnippet } from "./context.js"; +export { buildFixMapGraph, renderFixMapGraphMermaid, type FixMapGraph } from "./graph.js"; export { scanRepo } from "./repo-scan.js"; export { validateFixMapReport } from "./validate.js"; export type { ValidatedFixMapReport } from "./validate.js"; @@ -27,11 +32,16 @@ export { stripByteOrderMark } from "./text.js"; export type { FixMapInput, FixMapReport, + HistoryCommit, + ImpactEvidence, + ImpactFile, + ImpactMap, IdentifierGrounding, PackageScript, RankedFile, RepoFile, RepoMap, + RepositoryHistory, RiskNote, ScanDiagnostic, TaskAnalysis, diff --git a/packages/core/src/plan.ts b/packages/core/src/plan.ts index 61527e7..9e33a0b 100644 --- a/packages/core/src/plan.ts +++ b/packages/core/src/plan.ts @@ -10,7 +10,7 @@ import type { FixMapInput, FixMapReport } from "./types.js"; export async function buildFixMapReport( input: Pick< FixMapInput, - "repoRoot" | "issueText" | "diffSpec" | "baseRef" | "headRef" | "workingTree" | "includeUntracked" | "useCache" + "repoRoot" | "issueText" | "diffSpec" | "baseRef" | "headRef" | "workingTree" | "includeUntracked" | "useCache" | "includeHistory" > & { limit?: number | undefined; exclude?: string[] | undefined; @@ -18,7 +18,25 @@ export async function buildFixMapReport( internalExclude?: string[] | undefined; } ): Promise { - const repo = await scanRepo(input); + return (await buildFixMapAnalysis(input)).report; +} + +/** + * Builds a report and returns the exact repository snapshot that produced it. Consumers such as + * Context Packs must not rescan between ranking paths and selecting their source ranges: an active + * working tree could change between those reads and produce a mixed-state result. + */ +export async function buildFixMapAnalysis( + input: Pick< + FixMapInput, + "repoRoot" | "issueText" | "diffSpec" | "baseRef" | "headRef" | "workingTree" | "includeUntracked" | "useCache" | "includeHistory" + > & { + limit?: number | undefined; + exclude?: string[] | undefined; + internalExclude?: string[] | undefined; + } +): Promise<{ report: FixMapReport; repo: Awaited> }> { + const repo = await scanRepo({ ...input, includeHistory: input.includeHistory !== false }); const requestedExclude = await resolveExclusions(input.repoRoot, input.exclude ?? []); const internalExclude = buildPathExcluder( (input.internalExclude ?? []).map((pattern) => normalizeAbsolutePattern(input.repoRoot, pattern)) @@ -59,7 +77,7 @@ export async function buildFixMapReport( } } - return report; + return { report, repo }; } function combineExclusions(primary: PathExcluder, internal: PathExcluder): PathExcluder { diff --git a/packages/core/src/repo-scan.ts b/packages/core/src/repo-scan.ts index 248641c..d8a63a8 100644 --- a/packages/core/src/repo-scan.ts +++ b/packages/core/src/repo-scan.ts @@ -6,7 +6,7 @@ import { dirname, extname, isAbsolute, join, relative, resolve, sep } from "node import { promisify } from "node:util"; import { ALWAYS_IGNORED_DIRS, GENERATED_DIRS, SOURCE_FILE_EXTENSIONS, isGeneratedPath } from "./paths.js"; import { DIAGNOSTIC_SPEC_LIMIT, truncateForDiagnostic } from "./text.js"; -import type { FixMapInput, PackageScript, RepoFile, RepoMap } from "./types.js"; +import type { FixMapInput, HistoryCommit, PackageScript, RepoFile, RepoMap, RepositoryHistory } from "./types.js"; // Vendored source is intentionally scannable and receives a ranking penalty. Git mode has // always kept tracked vendor/ files; walk mode must not silently use a smaller corpus. @@ -46,9 +46,12 @@ const MAX_TEXT_SAMPLE_BYTES = 64_000; const MAX_DIFF_TEXT_CHARS = 200_000; const MAX_SCANNED_FILES = 25_000; const GIT_MAX_BUFFER = 10 * 1024 * 1024; +const GIT_HISTORY_MAX_BUFFER = 24 * 1024 * 1024; +const MAX_HISTORY_COMMITS = 1_000; +const MAX_HISTORY_FILES_PER_COMMIT = 30; const exec = promisify(execFile); type ScanState = { count: number; limitReported: boolean }; -const SCAN_CACHE_VERSION = 3; +const SCAN_CACHE_VERSION = 4; const SCAN_CACHE_MAX_AGE_MS = 7 * 24 * 60 * 60 * 1000; const SCAN_CACHE_MAX_FUTURE_SKEW_MS = 5 * 60 * 1000; const SCAN_CACHE_FILE = /^[a-f0-9]{24}-[a-f0-9]{24}\.json$/; @@ -63,12 +66,13 @@ type CachedScan = { packageScripts: PackageScript[]; packageManager: RepoMap["packageManager"]; diagnostics: RepoMap["diagnostics"]; + history: RepositoryHistory | null; }; export async function scanRepo( input: Pick< FixMapInput, - "repoRoot" | "baseRef" | "headRef" | "diffSpec" | "workingTree" | "includeUntracked" | "useCache" + "repoRoot" | "baseRef" | "headRef" | "diffSpec" | "workingTree" | "includeUntracked" | "useCache" | "includeHistory" > & { internalExclude?: string[] | undefined } ): Promise { const repoRoot = resolve(input.repoRoot); @@ -95,7 +99,7 @@ export async function scanRepo( ? cacheRoot : undefined; const cacheDecision = input.useCache === true - ? await buildScanCacheLocation(repoRoot, cacheRoot, internalPaths) + ? await buildScanCacheLocation(repoRoot, cacheRoot, internalPaths, input.includeHistory === true) : undefined; const cacheLocation = cacheDecision?.location; if (input.useCache === false) { @@ -116,11 +120,13 @@ export async function scanRepo( let trackedFiles: string[]; let packageScripts: PackageScript[]; let packageManager: RepoMap["packageManager"]; + let history: RepositoryHistory | undefined; if (cached) { files = cached.files; trackedFiles = cached.trackedFiles; packageScripts = cached.packageScripts; packageManager = cached.packageManager; + history = cached.history ?? undefined; diagnostics.push(...cached.diagnostics, { code: "cache-hit", severity: "info", @@ -131,6 +137,9 @@ export async function scanRepo( trackedFiles = await listTrackedPaths(repoRoot, internalPaths); packageScripts = await readPackageScripts(repoRoot, files, diagnostics); packageManager = detectPackageManager(files, diagnostics); + history = input.includeHistory === true + ? await readRepositoryHistory(repoRoot, new Set(files.map((file) => file.path)), diagnostics) + : undefined; if (cacheLocation) { await writeScanCache(cacheLocation, { version: SCAN_CACHE_VERSION, @@ -140,7 +149,8 @@ export async function scanRepo( trackedFiles, packageScripts, packageManager, - diagnostics: [...diagnostics] + diagnostics: [...diagnostics], + history: history ?? null }); } } @@ -148,6 +158,10 @@ export async function scanRepo( const diff = input.workingTree ? await readWorkingTree(repoRoot, input.includeUntracked === true, diagnostics, internalPaths) : await readDiff(repoRoot, diffSpec, diagnostics, internalPaths); + const orderedDiagnostics = [ + ...diagnostics.filter((entry) => !entry.code.startsWith("impact-history-")), + ...diagnostics.filter((entry) => entry.code.startsWith("impact-history-")) + ]; return { root: repoRoot, @@ -157,7 +171,8 @@ export async function scanRepo( changedFiles: diff.changedFiles, diffText: diff.diffText, packageManager, - diagnostics + diagnostics: orderedDiagnostics, + ...(history ? { history } : {}) }; } @@ -217,7 +232,8 @@ function gitPathspec(internalPaths: ReadonlySet): string[] { async function buildScanCacheLocation( root: string, cacheRoot: string, - internalPaths: ReadonlySet + internalPaths: ReadonlySet, + includeHistory: boolean ): Promise { if (sameFilesystemPath(cacheRoot, root) || containedPath(root, cacheRoot) !== undefined) { return { @@ -253,6 +269,7 @@ async function buildScanCacheLocation( head.trim(), status, dirtyDiff, + includeHistory ? "history" : "no-history", ...[...internalPaths].sort((a, b) => a.localeCompare(b)) ].join("\0")); return { location: { @@ -286,6 +303,7 @@ async function readScanCache(location: ScanCacheLocation): Promise { + if (!isRecord(commit) || typeof commit.hash !== "string" || !/^[a-f0-9]{40}$/i.test(commit.hash) || + typeof commit.committedAt !== "number" || !Number.isSafeInteger(commit.committedAt) || commit.committedAt < 0 || + !Array.isArray(commit.files)) return false; + return commit.files.every(isCachedRelativePath); + }); +} + function isCachedRepoFile(candidate: unknown): candidate is RepoFile { if (!isRecord(candidate)) return false; const validSkipReason = candidate.textSampleSkipReason === "too-large" || @@ -1102,6 +1135,108 @@ const NO_GIT_HISTORY = "this repository has no commits yet, so there is nothing to diff against. " + "Commit the initial work first, or run with --issue alone to rank from the task text."; +async function readRepositoryHistory( + root: string, + repositoryPaths: ReadonlySet, + diagnostics: RepoMap["diagnostics"] +): Promise { + try { + const [{ stdout: shallowText }, { stdout: countText }, { stdout: logText }] = await Promise.all([ + exec("git", ["rev-parse", "--is-shallow-repository"], { cwd: root, maxBuffer: GIT_MAX_BUFFER }), + exec("git", ["rev-list", "--count", "--no-merges", "HEAD"], { cwd: root, maxBuffer: GIT_MAX_BUFFER }), + exec("git", [ + "-c", "core.quotepath=false", + "log", + "--no-merges", + "-n", String(MAX_HISTORY_COMMITS), + "--format=%x1e%H%x1f%ct", + "--name-only", + "-z", + "HEAD" + ], { cwd: root, maxBuffer: GIT_HISTORY_MAX_BUFFER }) + ]); + + const parsed = parseHistoryLog(logText, repositoryPaths); + const totalCommits = Number.parseInt(countText.trim(), 10); + const shallow = shallowText.trim() === "true"; + const truncated = Number.isFinite(totalCommits) && totalCommits > parsed.inspectedCommits; + const history: RepositoryHistory = { + commits: parsed.commits, + inspectedCommits: parsed.inspectedCommits, + skippedLargeCommits: parsed.skippedLargeCommits, + shallow, + truncated + }; + + if (shallow) { + diagnostics.push({ + code: "impact-history-shallow", + severity: "info", + message: + `Impact history is shallow (${parsed.inspectedCommits.toLocaleString()} visible non-merge ` + + `${parsed.inspectedCommits === 1 ? "commit" : "commits"}). Import and test relationships remain available, ` + + "but co-change evidence may be incomplete." + }); + } + if (truncated) { + diagnostics.push({ + code: "impact-history-truncated", + severity: "info", + message: + `Impact history inspected the newest ${parsed.inspectedCommits.toLocaleString()} of ${totalCommits.toLocaleString()} ` + + `non-merge commits. Commits touching more than ${MAX_HISTORY_FILES_PER_COMMIT} files were excluded from co-change evidence.` + }); + } + return history; + } catch (error) { + const checkoutState = isMissingGit(error) ? undefined : await describeGitCheckout(root); + diagnostics.push({ + code: "impact-history-unavailable", + severity: "info", + message: checkoutState === "not-repository" + ? "Impact history is unavailable because this directory is not a Git checkout; import and test relationships are still reported." + : checkoutState === "no-history" + ? "Impact history is unavailable because this repository has no commits; import and test relationships are still reported." + : `Impact history could not be read (${truncateForDiagnostic(gitErrorDetail(error), DIAGNOSTIC_SPEC_LIMIT * 2)}); import and test relationships are still reported.` + }); + return undefined; + } +} + +function parseHistoryLog( + logText: string, + repositoryPaths: ReadonlySet +): { commits: HistoryCommit[]; inspectedCommits: number; skippedLargeCommits: number } { + const commits: HistoryCommit[] = []; + let inspectedCommits = 0; + let skippedLargeCommits = 0; + + for (const record of logText.split("\x1e")) { + if (!record) continue; + const fields = record.split("\0"); + const header = fields.shift()?.replace(/^\r?\n/, "") ?? ""; + const separator = header.indexOf("\x1f"); + if (separator === -1) continue; + const hash = header.slice(0, separator).trim(); + const committedAt = Number.parseInt(header.slice(separator + 1).trim(), 10); + if (!/^[a-f0-9]{40}$/i.test(hash) || !Number.isSafeInteger(committedAt) || committedAt < 0) continue; + + inspectedCommits += 1; + const allFiles = [...new Set(fields + .map((path) => path.replace(/^\r?\n/, "")) + .filter(Boolean) + .map(normalizePath))]; + if (allFiles.length > MAX_HISTORY_FILES_PER_COMMIT) { + skippedLargeCommits += 1; + continue; + } + const currentFiles = allFiles.filter((path) => repositoryPaths.has(path)); + if (currentFiles.length === 0) continue; + commits.push({ hash, committedAt, files: currentFiles }); + } + return { commits, inspectedCommits, skippedLargeCommits }; +} + /** * `execFile` puts "Command failed: git ..." in `message` and git's own explanation in * `stderr`, so matching on the message alone never saw the reason. Both are checked. diff --git a/packages/core/src/report.ts b/packages/core/src/report.ts index 3c4fd8a..5f2a620 100644 --- a/packages/core/src/report.ts +++ b/packages/core/src/report.ts @@ -5,6 +5,7 @@ import { import type { RankingShape, TaskGrounding } from "./grounding.js"; import type { PathExcluder } from "./exclude.js"; import { detectPrimaryLanguage, manifestTestCommand, suggestedRunner } from "./languages.js"; +import { buildImpactMap } from "./impact.js"; import { DEFAULT_CONTEXT_FILE_LIMIT, rankContextFiles, rankContextFilesDetailed } from "./rank.js"; import { extractTaskSignals, tokenizePath } from "./signals.js"; import { findGatedTestDiagnostics } from "./test-gates.js"; @@ -44,13 +45,15 @@ export function buildReportFromRepo( const contextPaths = contextFiles.map((file) => file.path); const testRoutes = buildTestRoutes(repo, contextPaths); const routedTestPaths = [...new Set(testRoutes.flatMap((route) => route.relatedFiles))]; + const impact = buildImpactMap(repo, contextPaths, testRoutes); return { reportVersion: 1, - summary: buildSummary(contextFiles.length, testRoutes.length), + summary: buildSummary(contextFiles.length, testRoutes.length, impact.files.length), contextFiles, testRoutes, risks: buildRiskNotes(contextPaths, repo.changedFiles), + impact, changedFiles: repo.changedFiles, diagnostics: [ ...repo.diagnostics, @@ -540,10 +543,11 @@ function findRelatedTests(repo: RepoMap, contextPaths: string[]): string[] { return [...changedTests, ...overlapping].slice(0, 8); } -export function buildSummary(contextFileCount: number, testRouteCount: number): string { +export function buildSummary(contextFileCount: number, testRouteCount: number, impactFileCount = 0): string { const files = contextFileCount === 1 ? "context file" : "context files"; const routes = testRouteCount === 1 ? "test route" : "test routes"; - return `FixMap found ${contextFileCount} ${files} and generated ${testRouteCount} ${routes}.`; + const impact = impactFileCount === 1 ? "impact file" : "impact files"; + return `FixMap found ${contextFileCount} ${files}, ${impactFileCount} ${impact}, and generated ${testRouteCount} ${routes}.`; } export function renderMarkdownReport(report: FixMapReport): string { @@ -556,6 +560,19 @@ export function renderMarkdownReport(report: FixMapReport): string { "", ...listOrEmpty(report.contextFiles.map((file) => `- ${markdownCode(file.path)} (${file.confidence} confidence, score ${file.score}): ${file.reasons.join("; ")}`)), "", + "## Impact Graph", + "", + ...listOrEmpty((report.impact?.files ?? []).map((file) => + `- ${markdownCode(file.path)} (${file.confidence} confidence, impact ${file.score}): ${file.evidence.map((entry) => entry.reason).join("; ")}` + )), + ...(report.impact ? [ + "", + `Inspection order: ${report.impact.inspectionOrder.map(markdownCode).join(" → ") || "None"}.`, + `History evidence: ${report.impact.history.available + ? `${report.impact.history.eligibleCommits.toLocaleString()} eligible commits${report.impact.history.shallow ? " (shallow)" : ""}${report.impact.history.truncated ? " (bounded)" : ""}` + : "not available; import and test evidence only"}.` + ] : []), + "", "## Test Routes", "", ...listOrEmpty(report.testRoutes.map((route) => { @@ -595,6 +612,47 @@ export function renderJsonReport(report: FixMapReport): string { return `${JSON.stringify(report, null, 2)}\n`; } +/** Compact, stable headings for an agent context window. Evidence stays attached to each path. */ +export function renderAgentReport(report: FixMapReport): string { + const editCandidate = report.contextFiles[0]; + const inspectByPath = new Map(); + for (const file of report.contextFiles.slice(1, 4)) { + inspectByPath.set(file.path, `context: ${file.reasons[0] ?? "ranked evidence"}`); + } + for (const file of (report.impact?.files ?? []).slice(0, 5)) { + if (!inspectByPath.has(file.path)) { + inspectByPath.set(file.path, `impact: ${file.evidence[0]?.reason ?? "related repository evidence"}`); + } + } + const avoided = [...new Set(report.diagnostics + .filter((entry) => entry.code === "generated-paths-dominant" || entry.code === "paths-excluded") + .flatMap((entry) => entry.paths ?? []))]; + const uncertainty = report.diagnostics + .filter((entry) => entry.severity !== "info" || entry.code.startsWith("impact-history-")) + .slice(0, 3) + .map((entry) => entry.message); + const lines = [ + "EDIT CANDIDATE:", + editCandidate ? `${editCandidate.path} # ${editCandidate.confidence}; ${editCandidate.reasons[0] ?? "ranked evidence"}` : "none", + "", + "INSPECT:", + ...listOrEmpty([...inspectByPath].map(([path, reason]) => `${path} # ${reason}`)), + "", + "TEST:", + ...listOrEmpty(report.testRoutes.map((route) => `${route.command}${route.relatedFiles[0] ? ` # ${route.relatedFiles[0]}` : ""}`)), + "", + "RISK:", + ...listOrEmpty(report.risks.map((risk) => `${risk.severity} ${risk.area} # ${risk.reason}`)), + "", + "AVOID:", + ...listOrEmpty(avoided), + "", + "UNCERTAINTY:", + ...listOrEmpty(uncertainty) + ]; + return `${lines.join("\n")}\n`; +} + function listOrEmpty(lines: string[]): string[] { return lines.length > 0 ? lines : ["- None found"]; } diff --git a/packages/core/src/retrieval.ts b/packages/core/src/retrieval.ts new file mode 100644 index 0000000..01728aa --- /dev/null +++ b/packages/core/src/retrieval.ts @@ -0,0 +1,92 @@ +import type { RepoFile } from "./types.js"; + +const STOPWORDS = new Set(`a about above after again against all am an and any are as at be because been before being +below between both but by can cannot could did do does doing down during each few for from further had has have having +he her here hers him his how i if in into is it its itself just me more most my no nor not of off on once only or other +ought our out over own same she should so some such than that the their them then there these they this those through +to too under until up very was we were what when where which while who whom why with would you your +bug issue issues error errors expected actual behavior behaviour reproduce reproduction steps version versions node npm +report repo repository description example code please thanks title type severity confidence location line lines +following above below see also would should could may might must will can also using used use uses`.split(/\s+/)); +const PATH_BOUNDARY = /[A-Za-z0-9_/-]/; +const REPO_ROOT_ANCHOR = /(?:\/(?:blob|tree|blame|raw)\/[^\s/]+\/|raw\.githubusercontent\.com\/[^\s/]+\/[^\s/]+\/[^\s/]+\/)$/i; + +export function retrievalTokens(text: string): string[] { + const tokens: string[] = []; + for (const raw of text.match(/[A-Za-z0-9_$]+/g) ?? []) { + const lower = raw.toLowerCase(); + if (lower.length >= 3) tokens.push(lower); + const parts = raw.split(/(?<=[a-z0-9])(?=[A-Z])|_/).filter((part) => part.length >= 3); + if (parts.length > 1) tokens.push(...parts.map((part) => part.toLowerCase())); + } + return tokens; +} + +export function retrievalQueryTerms(task: string): string[] { + return [...new Set(retrievalTokens(task))].filter((term) => !STOPWORDS.has(term)); +} + +/** Standard BM25 (k1=1.2, b=0.75) over the scanner's code-only candidate corpus. */ +export function rankByBm25(files: RepoFile[], task: string, limit = 5): string[] { + const candidates = files.filter((file) => file.isSource && !file.isTest && file.kind === "code"); + const terms = retrievalQueryTerms(task); + const documents = candidates.map((file) => { + const counts = new Map(); + for (const token of retrievalTokens(`${file.path}\n${file.textSample}`)) { + counts.set(token, (counts.get(token) ?? 0) + 1); + } + return { path: file.path, counts, length: [...counts.values()].reduce((sum, count) => sum + count, 0) }; + }); + if (documents.length === 0 || terms.length === 0) return []; + const averageLength = documents.reduce((sum, document) => sum + document.length, 0) / documents.length || 1; + const documentFrequency = new Map(terms.map((term) => [ + term, + documents.reduce((count, document) => count + (document.counts.has(term) ? 1 : 0), 0) + ])); + + return documents + .map((document) => { + let score = 0; + for (const term of terms) { + const frequency = document.counts.get(term) ?? 0; + if (frequency === 0) continue; + const df = documentFrequency.get(term) ?? 0; + const idf = Math.log(1 + (documents.length - df + 0.5) / (df + 0.5)); + score += idf * ((frequency * 2.2) / (frequency + 1.2 * (0.25 + (0.75 * document.length) / averageLength))); + } + return { path: document.path, score }; + }) + .filter((entry) => entry.score > 0) + .sort((left, right) => right.score - left.score || left.path.localeCompare(right.path)) + .slice(0, Math.max(0, limit)) + .map((entry) => entry.path); +} + +export function taskMentionsExpectedPath(task: string, expectedPaths: string[]): boolean { + const normalizedTask = task.replace(/\\/g, "/").toLowerCase(); + return expectedPaths.some((expectedPath) => { + const normalizedPath = expectedPath.replace(/\\/g, "/").toLowerCase(); + const segments = normalizedPath.split("/"); + if (findAnchoredPath(normalizedTask, normalizedPath)) return true; + // A multi-segment suffix can identify the file while a bare basename cannot reliably do so. + for (let start = 1; start < segments.length - 1; start += 1) { + if (findAnchoredPath(normalizedTask, segments.slice(start).join("/"))) return true; + } + return false; + }); +} + +function findAnchoredPath(haystack: string, needle: string): boolean { + if (!needle) return false; + let from = 0; + for (;;) { + const index = haystack.indexOf(needle, from); + if (index === -1) return false; + const before = index === 0 ? "" : haystack[index - 1] ?? ""; + const after = haystack[index + needle.length] ?? ""; + const preceding = haystack.slice(Math.max(0, index - 200), index); + const leftOk = !PATH_BOUNDARY.test(before) || REPO_ROOT_ANCHOR.test(preceding); + if (leftOk && !PATH_BOUNDARY.test(after)) return true; + from = index + 1; + } +} diff --git a/packages/core/src/types.ts b/packages/core/src/types.ts index 2a51c5d..ebb60de 100644 --- a/packages/core/src/types.ts +++ b/packages/core/src/types.ts @@ -11,6 +11,8 @@ export type FixMapInput = { includeUntracked?: boolean | undefined; /** Reuse an exact git-state scan from the OS cache. Non-git directories never cache. */ useCache?: boolean | undefined; + /** Read bounded Git history for impact evidence. Disable only when latency matters more than historical coverage. */ + includeHistory?: boolean | undefined; }; export type TextSampleSkipReason = "too-large" | "not-text" | "unreadable"; @@ -72,7 +74,10 @@ export type ScanDiagnostic = { | "cache-bypass" | "cache-skip" | "task-checklist-filtered" - | "package-manager-conflict"; + | "package-manager-conflict" + | "impact-history-unavailable" + | "impact-history-shallow" + | "impact-history-truncated"; message: string; severity: "info" | "warning" | "error"; /** @@ -93,6 +98,22 @@ export type RepoMap = { diffText: string; packageManager: "npm" | "pnpm" | "yarn" | "bun"; diagnostics: ScanDiagnostic[]; + /** Bounded, pre-HEAD history used only as evidence. It never contains file contents. */ + history?: RepositoryHistory; +}; + +export type HistoryCommit = { + hash: string; + committedAt: number; + files: string[]; +}; + +export type RepositoryHistory = { + commits: HistoryCommit[]; + inspectedCommits: number; + skippedLargeCommits: number; + shallow: boolean; + truncated: boolean; }; export type RankedFile = { @@ -123,6 +144,33 @@ export type RiskNote = { severity: "low" | "medium" | "high"; }; +export type ImpactEvidence = { + kind: "imports" | "imported-by" | "co-change" | "test-route"; + seed: string; + reason: string; + occurrences?: number; + seedChanges?: number; +}; + +export type ImpactFile = { + path: string; + score: number; + confidence: "high" | "medium" | "low"; + evidence: ImpactEvidence[]; +}; + +export type ImpactMap = { + seeds: string[]; + files: ImpactFile[]; + inspectionOrder: string[]; + history: { + available: boolean; + eligibleCommits: number; + shallow: boolean; + truncated: boolean; + }; +}; + export type IdentifierGrounding = { identifier: string; status: @@ -159,6 +207,8 @@ export type FixMapReport = { contextFiles: RankedFile[]; testRoutes: TestRoute[]; risks: RiskNote[]; + /** Additive v1 field: evidence-backed files likely worth inspecting after the primary context. */ + impact?: ImpactMap; changedFiles: string[]; diagnostics: ScanDiagnostic[]; analysis?: TaskAnalysis; @@ -172,6 +222,7 @@ export type VerifyFinding = { | "leading-file-untouched" | "no-test-changed" | "new-risk-area" + | "impact-file-unreviewed" | "plan-partially-stale" | "planned-file-deleted" | "plan-repository-mismatch"; @@ -191,4 +242,5 @@ export type VerifyResult = { * comparison; `diagnostics` are what FixMap noticed while looking. */ diagnostics: ScanDiagnostic[]; + impact?: ImpactMap; }; diff --git a/packages/core/src/validate.ts b/packages/core/src/validate.ts index 8c23ecc..b125ba2 100644 --- a/packages/core/src/validate.ts +++ b/packages/core/src/validate.ts @@ -115,6 +115,33 @@ export function validateFixMapReport(candidate: unknown, label: string): Validat }; } + if (record.impact !== undefined) { + const impact = record.impact; + const history = isRecord(impact) ? impact.history : undefined; + if (!isRecord(impact) || !isRepositoryRelativePathArray(impact.seeds) || + !Array.isArray(impact.files) || !isRepositoryRelativePathArray(impact.inspectionOrder) || + !isRecord(history) || typeof history.available !== "boolean" || + typeof history.eligibleCommits !== "number" || !Number.isSafeInteger(history.eligibleCommits) || history.eligibleCommits < 0 || + typeof history.shallow !== "boolean" || typeof history.truncated !== "boolean") { + return { success: false, message: `${label} has an invalid impact graph envelope.` }; + } + const invalidImpact = impact.files.findIndex((file) => { + if (!isRecord(file) || !isRepositoryRelativePath(file.path) || + typeof file.score !== "number" || !Number.isFinite(file.score) || file.score < 0 || + (file.confidence !== "high" && file.confidence !== "medium" && file.confidence !== "low") || + !Array.isArray(file.evidence)) return true; + return file.evidence.some((evidence) => !isRecord(evidence) || + !["imports", "imported-by", "co-change", "test-route"].includes(String(evidence.kind)) || + !isRepositoryRelativePath(evidence.seed) || typeof evidence.reason !== "string" || !evidence.reason.trim() || + (evidence.occurrences !== undefined && (!Number.isSafeInteger(evidence.occurrences) || (evidence.occurrences as number) < 0)) || + (evidence.seedChanges !== undefined && (!Number.isSafeInteger(evidence.seedChanges) || (evidence.seedChanges as number) < 0)) + ); + }); + if (invalidImpact !== -1) { + return { success: false, message: `${label} has an invalid impact.files entry at index ${invalidImpact}.` }; + } + } + if (!isRepositoryRelativePathArray(record.changedFiles)) { return { success: false, message: `${label} has invalid changedFiles; every entry must be a safe repository-relative path.` }; } diff --git a/packages/core/src/verify.ts b/packages/core/src/verify.ts index acfa057..6896c16 100644 --- a/packages/core/src/verify.ts +++ b/packages/core/src/verify.ts @@ -8,6 +8,7 @@ // say which. The output says what differs and leaves that judgement alone. import { isBackupPath, isGeneratedPath, moduleStem } from "./paths.js"; +import { buildImpactMap } from "./impact.js"; import { buildRiskNotes, pathsForRiskArea } from "./report.js"; import type { FixMapReport, RepoMap, VerifyFinding, VerifyResult } from "./types.js"; import { markdownCode } from "./markdown.js"; @@ -170,7 +171,24 @@ export function verifyPlan(report: FixMapReport, repo: RepoMap): VerifyResult { }); } - // 5. Risk the plan never mentioned, because the change reached further than the map did. + // 5. Recompute relationships from what actually changed. This is deliberately an + // informational inspection prompt, not a claim that every dependent must be edited. + const impact = buildImpactMap(repo, changed, report.testRoutes); + const highImpactOutsidePlan = impact.files.filter((entry) => + entry.confidence === "high" && !planned.has(entry.path) && !changed.includes(entry.path) && !isTest(entry.path) + ); + if (highImpactOutsidePlan.length > 0) { + findings.push({ + code: "impact-file-unreviewed", + severity: "info", + paths: highImpactOutsidePlan.slice(0, 8).map((entry) => entry.path), + message: + `${highImpactOutsidePlan.length === 1 ? "One high-evidence impact file is" : `${highImpactOutsidePlan.length} high-evidence impact files are`} ` + + "outside both the original plan and this diff. They are not required edits, but inspect the recorded import/history evidence before finishing." + }); + } + + // 6. Risk the plan never mentioned, because the change reached further than the map did. const plannedAreas = new Set(report.risks.map((risk) => risk.area)); const newRisks = buildRiskNotes(changed, changed).filter((risk) => !plannedAreas.has(risk.area)); for (const risk of newRisks) { @@ -186,7 +204,8 @@ export function verifyPlan(report: FixMapReport, repo: RepoMap): VerifyResult { summary: buildVerifySummary(changed.length, findings), changedFiles: changed, findings, - diagnostics: repo.diagnostics + diagnostics: repo.diagnostics, + impact }; } @@ -234,5 +253,13 @@ export function renderVerifyMarkdown(result: VerifyResult): string { } lines.push("", "## Changed Files", ""); lines.push(...(result.changedFiles.length > 0 ? result.changedFiles.map((path) => `- ${markdownCode(path)}`) : ["- None found"])); + if (result.impact) { + lines.push("", "## Recalculated Impact", ""); + lines.push(...(result.impact.files.length > 0 + ? result.impact.files.map((file) => + `- ${markdownCode(file.path)} (${file.confidence} confidence): ${file.evidence.map((entry) => entry.reason).join("; ")}` + ) + : ["- None found"])); + } return `${lines.join("\n")}\n`; } diff --git a/packages/core/test/context.test.ts b/packages/core/test/context.test.ts new file mode 100644 index 0000000..4c621e1 --- /dev/null +++ b/packages/core/test/context.test.ts @@ -0,0 +1,66 @@ +import { describe, expect, it } from "vitest"; +import { buildContextPack, estimateContextTokens, renderContextPackMarkdown } from "../src/context.js"; +import type { FixMapReport, RepoMap } from "../src/types.js"; + +const report: FixMapReport = { + reportVersion: 1, + summary: "Reset context.", + contextFiles: [{ rank: 1, path: "src/reset.ts", score: 20, confidence: "high", reasons: ["defines resetPassword"] }], + impact: { + seeds: ["src/reset.ts"], + files: [{ path: "test/reset.test.ts", score: 10, confidence: "high", evidence: [{ kind: "test-route", seed: "src/reset.ts", reason: "routed test" }] }], + inspectionOrder: ["src/reset.ts", "test/reset.test.ts"], + history: { available: false, eligibleCommits: 0, shallow: false, truncated: false } + }, + testRoutes: [], risks: [], changedFiles: [], diagnostics: [] +}; + +const repo: RepoMap = { + root: "/repo", + files: [ + { path: "src/reset.ts", extension: ".ts", sizeBytes: 100, isTest: false, isSource: true, kind: "code", textSample: "const unrelated = 1;\n\nexport function resetPassword(email: string) {\n return sendResetEmail(email);\n}\n", textSampleComplete: true }, + { path: "test/reset.test.ts", extension: ".ts", sizeBytes: 100, isTest: true, isSource: true, kind: "code", textSample: "test('reset password', () => {\n expect(resetPassword('a')).toBeTruthy();\n});\n", textSampleComplete: true } + ], + packageScripts: [], changedFiles: [], diffText: "", packageManager: "npm", diagnostics: [] +}; + +describe("context packs", () => { + it("selects primary and supporting source under a stable budget", () => { + const pack = buildContextPack({ report, repo, task: "resetPassword emails fail", budgetTokens: 256 }); + expect(pack.snippets.map((snippet) => snippet.role)).toEqual(["primary", "supporting"]); + expect(pack.snippets[0]).toMatchObject({ path: "src/reset.ts", startLine: 1, language: "typescript" }); + expect(pack.estimatedSourceTokens).toBeLessThanOrEqual(256); + expect(pack.snippets[0]?.content).toContain("resetPassword"); + }); + + it("reports budget omissions without exceeding the source budget", () => { + const pack = buildContextPack({ report, repo, task: "resetPassword", budgetTokens: 48 }); + expect(pack.estimatedSourceTokens).toBeLessThanOrEqual(48); + expect(pack.omitted).toContainEqual({ path: "test/reset.test.ts", reason: "budget" }); + }); + + it("renders fenced ranges and uses the documented byte estimate", () => { + expect(estimateContextTokens("12345678")).toBe(2); + const markdown = renderContextPackMarkdown(buildContextPack({ report, repo, task: "resetPassword", budgetTokens: 256 })); + expect(markdown).toContain("`src/reset.ts`:1-6"); + expect(markdown).toContain("```typescript"); + expect(markdown).toContain("UTF-8 bytes divided by four"); + }); + + it("keeps expanding on the other side when one neighboring line exceeds the allowance", () => { + const asymmetricRepo: RepoMap = { + ...repo, + files: [{ + ...repo.files[0]!, + textSample: `${"x".repeat(800)}\nexport function resetPassword() {}\nreturn sendResetEmail();\n`, + textSampleComplete: true + }] + }; + const primaryOnly: FixMapReport = { ...report, impact: undefined }; + + const pack = buildContextPack({ report: primaryOnly, repo: asymmetricRepo, task: "resetPassword", budgetTokens: 64 }); + + expect(pack.snippets[0]?.content).toContain("sendResetEmail"); + expect(pack.estimatedSourceTokens).toBeLessThanOrEqual(64); + }); +}); diff --git a/packages/core/test/entrypoints.test.ts b/packages/core/test/entrypoints.test.ts index 7ba1f83..cf8e5cc 100644 --- a/packages/core/test/entrypoints.test.ts +++ b/packages/core/test/entrypoints.test.ts @@ -7,6 +7,8 @@ describe("public entrypoint parity", () => { "buildReportFromRepo", "renderJsonReport", "renderMarkdownReport", + "renderAgentReport", + "buildImpactMap", "validateFixMapReport", "quoteCliValue" ])("exports deterministic API %s from both node and browser entries", (name) => { @@ -15,7 +17,9 @@ describe("public entrypoint parity", () => { }); it("keeps filesystem scanning node-only", () => { + expect(nodeEntry).toHaveProperty("buildFixMapAnalysis"); expect(nodeEntry).toHaveProperty("scanRepo"); expect(browserEntry).not.toHaveProperty("scanRepo"); + expect(browserEntry).not.toHaveProperty("buildFixMapAnalysis"); }); }); diff --git a/packages/core/test/graph.test.ts b/packages/core/test/graph.test.ts new file mode 100644 index 0000000..08cfeb3 --- /dev/null +++ b/packages/core/test/graph.test.ts @@ -0,0 +1,40 @@ +import { describe, expect, it } from "vitest"; +import { buildFixMapGraph, renderFixMapGraphMermaid } from "../src/graph.js"; +import type { FixMapReport } from "../src/types.js"; + +describe("impact graph export", () => { + it("preserves relationship direction and renders deterministic Mermaid", () => { + const report: FixMapReport = { + summary: "graph", + contextFiles: [{ rank: 1, path: "src/a.ts", score: 10, confidence: "high", reasons: ["match"] }], + impact: { + seeds: ["src/a.ts"], + files: [ + { path: "src/b.ts", score: 8, confidence: "medium", evidence: [{ kind: "imports", seed: "src/a.ts", reason: "a imports b" }] }, + { path: "src/c.ts", score: 7, confidence: "medium", evidence: [{ kind: "imported-by", seed: "src/a.ts", reason: "c imports a" }] } + ], + inspectionOrder: ["src/a.ts", "src/b.ts", "src/c.ts"], + history: { available: false, eligibleCommits: 0, shallow: false, truncated: false } + }, + testRoutes: [], risks: [], changedFiles: [], diagnostics: [] + }; + const graph = buildFixMapGraph(report); + expect(graph.edges).toEqual([ + { from: "n1", to: "n2", kind: "imports", label: "imports" }, + { from: "n3", to: "n1", kind: "imported-by", label: "imports" } + ]); + const mermaid = renderFixMapGraphMermaid(graph); + expect(mermaid).toContain('n1["src/a.ts"]:::primary'); + expect(mermaid).toContain('n3 -->|"imports"| n1'); + }); + + it("escapes line breaks in Mermaid labels", () => { + const mermaid = renderFixMapGraphMermaid({ + graphVersion: 1, + nodes: [{ id: "n1", path: "src/a\nfile.ts", role: "primary", confidence: "high" }], + edges: [] + }); + expect(mermaid).toContain("src/a file.ts"); + expect(mermaid).not.toContain("src/a\nfile.ts"); + }); +}); diff --git a/packages/core/test/impact.test.ts b/packages/core/test/impact.test.ts new file mode 100644 index 0000000..1a90e6a --- /dev/null +++ b/packages/core/test/impact.test.ts @@ -0,0 +1,108 @@ +import { describe, expect, it } from "vitest"; +import { buildImpactMap } from "../src/impact.js"; +import { taskMentionsExpectedPath } from "../src/retrieval.js"; +import type { RepoFile, RepoMap } from "../src/types.js"; + +function file(path: string, textSample = ""): RepoFile { + return { + path, + extension: path.slice(path.lastIndexOf(".")), + sizeBytes: textSample.length, + isTest: /(?:^|\/)tests?\//.test(path) || path.includes(".test."), + isSource: true, + kind: "code", + textSample + }; +} + +function repository(withHistory = true): RepoMap { + return { + root: "/repo", + files: [ + file("src/auth/reset.ts", "import { token } from './token'; export const reset = token;"), + file("src/auth/token.ts", "export const token = true;"), + file("src/api/auth.ts", "import { reset } from '../auth/reset'; export { reset };"), + file("src/session.ts", "export const session = true;"), + file("test/auth/reset.test.ts", "import { reset } from '../../src/auth/reset'; test('reset', () => reset);"), + file("docs/auth.md", "reset documentation") + ], + packageScripts: [{ name: "test", command: "vitest run", packageDir: "" }], + changedFiles: [], + diffText: "", + packageManager: "npm", + diagnostics: [], + ...(withHistory ? { + history: { + inspectedCommits: 4, + skippedLargeCommits: 0, + shallow: false, + truncated: false, + commits: [ + { hash: "a".repeat(40), committedAt: 4, files: ["src/auth/reset.ts", "src/session.ts"] }, + { hash: "b".repeat(40), committedAt: 3, files: ["src/auth/reset.ts", "src/session.ts", "docs/auth.md"] }, + { hash: "c".repeat(40), committedAt: 2, files: ["src/auth/reset.ts", "src/session.ts", "docs/auth.md"] }, + { hash: "d".repeat(40), committedAt: 1, files: ["src/api/auth.ts"] } + ] + } + } : {}) + }; +} + +describe("buildImpactMap", () => { + it("separates dependents, dependencies, routed tests, and historical companions from seeds", () => { + const impact = buildImpactMap(repository(), ["src/auth/reset.ts"], [{ + kind: "test", + command: "npm test", + reason: "root test script", + relatedFiles: ["test/auth/reset.test.ts"] + }]); + + expect(impact.seeds).toEqual(["src/auth/reset.ts"]); + expect(impact.files.map((entry) => entry.path)).not.toContain("src/auth/reset.ts"); + expect(impact.files.find((entry) => entry.path === "src/auth/token.ts")?.evidence[0]?.kind).toBe("imports"); + expect(impact.files.find((entry) => entry.path === "src/api/auth.ts")?.evidence[0]?.kind).toBe("imported-by"); + expect(impact.files.find((entry) => entry.path === "src/session.ts")?.evidence).toContainEqual(expect.objectContaining({ + kind: "co-change", + occurrences: 3, + seedChanges: 3 + })); + expect(impact.files.find((entry) => entry.path === "test/auth/reset.test.ts")?.evidence.map((entry) => entry.kind)) + .toEqual(expect.arrayContaining(["imported-by", "test-route"])); + expect(impact.history).toEqual({ available: true, eligibleCommits: 4, shallow: false, truncated: false }); + }); + + it("degrades to import and test evidence when history is unavailable", () => { + const impact = buildImpactMap(repository(false), ["src/auth/reset.ts"]); + + expect(impact.history.available).toBe(false); + expect(impact.files.some((entry) => entry.evidence.some((evidence) => evidence.kind === "co-change"))).toBe(false); + expect(impact.files.map((entry) => entry.path)).toEqual(expect.arrayContaining(["src/api/auth.ts", "src/auth/token.ts"])); + }); + + it("requires repeated co-change before reporting a historical relationship", () => { + const repo = repository(); + repo.history!.commits = [ + { hash: "e".repeat(40), committedAt: 1, files: ["src/auth/reset.ts", "src/session.ts"] } + ]; + const impact = buildImpactMap(repo, ["src/auth/reset.ts"]); + + expect(impact.files.find((entry) => entry.path === "src/session.ts")).toBeUndefined(); + }); +}); + +describe("repository benchmark path cohorts", () => { + it("recognizes anchored full paths and multi-segment suffixes but not bare basenames", () => { + const expected = ["packages/toolkit/src/query/react/buildHooks.ts"]; + expect(taskMentionsExpectedPath("Failure in packages/toolkit/src/query/react/buildHooks.ts:20", expected)).toBe(true); + expect(taskMentionsExpectedPath("tsc points at src/query/react/buildHooks.ts(20,4)", expected)).toBe(true); + expect(taskMentionsExpectedPath("buildHooks.ts returns the wrong type", expected)).toBe(false); + expect(taskMentionsExpectedPath("look at test/packages/toolkit/src/query/react/buildHooks.ts", expected)).toBe(false); + }); + + it("recognizes a repository-root path inside a GitHub permalink", () => { + expect(taskMentionsExpectedPath( + "https://github.com/o/r/blob/main/src/auth/reset.ts#L20", + ["src/auth/reset.ts"] + )).toBe(true); + }); +}); diff --git a/packages/core/test/plan.test.ts b/packages/core/test/plan.test.ts index 6721b7a..f1378ac 100644 --- a/packages/core/test/plan.test.ts +++ b/packages/core/test/plan.test.ts @@ -2,7 +2,7 @@ import { mkdtemp, mkdir, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { describe, expect, it } from "vitest"; -import { buildFixMapReport } from "../src/plan.js"; +import { buildFixMapAnalysis, buildFixMapReport } from "../src/plan.js"; import { renderMarkdownReport } from "../src/report.js"; async function createAuthFixture(): Promise { @@ -20,6 +20,16 @@ async function createAuthFixture(): Promise { } describe("buildFixMapReport", () => { + it("returns the exact scanned repository snapshot with an analysis", async () => { + const root = await createAuthFixture(); + + const analysis = await buildFixMapAnalysis({ repoRoot: root, issueText: "password reset emails fail" }); + + expect(analysis.report.contextFiles[0]?.path).toBe("src/auth/reset-password.ts"); + expect(analysis.repo.files.find((file) => file.path === "src/auth/reset-password.ts")?.textSample) + .toContain("sendResetEmail"); + }); + it("produces a full report from a task description", async () => { const root = await createAuthFixture(); diff --git a/packages/core/test/repo-scan.test.ts b/packages/core/test/repo-scan.test.ts index 203e9a9..18c9407 100644 --- a/packages/core/test/repo-scan.test.ts +++ b/packages/core/test/repo-scan.test.ts @@ -940,3 +940,47 @@ describe("scanRepo", () => { expect(diagnostic?.paths).toEqual(["packages/shared"]); }); }); + +describe("repository impact history", () => { + it("reads bounded pre-HEAD co-change evidence and excludes oversized commits", async () => { + const root = await mkdtemp(join(tmpdir(), "fixmap-history-")); + try { + await exec("git", ["init"], { cwd: root }); + await exec("git", ["config", "user.name", "FixMap Test"], { cwd: root }); + await exec("git", ["config", "user.email", "fixmap@example.invalid"], { cwd: root }); + await mkdir(join(root, "src"), { recursive: true }); + await writeFile(join(root, "src", "seed.ts"), "export const seed = 1;\n"); + await writeFile(join(root, "src", "peer.ts"), "export const peer = 1;\n"); + await exec("git", ["add", "."], { cwd: root }); + await exec("git", ["commit", "-m", "add seed and peer"], { cwd: root }); + + await writeFile(join(root, "src", "seed.ts"), "export const seed = 2;\n"); + await writeFile(join(root, "src", "peer.ts"), "export const peer = 2;\n"); + await exec("git", ["add", "."], { cwd: root }); + await exec("git", ["commit", "-m", "update seed and peer"], { cwd: root }); + + await mkdir(join(root, "bulk"), { recursive: true }); + await Promise.all(Array.from({ length: 31 }, (_, index) => + writeFile(join(root, "bulk", `file-${index}.ts`), `export const value${index} = ${index};\n`) + )); + await writeFile(join(root, "src", "seed.ts"), "export const seed = 3;\n"); + await exec("git", ["add", "."], { cwd: root }); + await exec("git", ["commit", "-m", "large generated sweep"], { cwd: root }); + + const repo = await scanRepo({ repoRoot: root, includeHistory: true }); + + expect(repo.history).toMatchObject({ + inspectedCommits: 3, + skippedLargeCommits: 1, + shallow: false, + truncated: false + }); + expect(repo.history?.commits).toHaveLength(2); + expect(repo.history?.commits.every((commit) => + commit.files.includes("src/seed.ts") && commit.files.includes("src/peer.ts") + )).toBe(true); + } finally { + await rm(root, { recursive: true, force: true }); + } + }); +}); diff --git a/packages/core/test/validate.test.ts b/packages/core/test/validate.test.ts index 48bfaa8..1f0db18 100644 --- a/packages/core/test/validate.test.ts +++ b/packages/core/test/validate.test.ts @@ -148,6 +148,34 @@ describe("validateFixMapReport", () => { expect(result.success).toBe(true); }); + it("accepts a valid additive impact graph and rejects unsafe impact paths", () => { + const impact = { + seeds: ["src/reset.ts"], + files: [{ + path: "src/session.ts", + score: 8, + confidence: "high", + evidence: [{ + kind: "co-change", + seed: "src/reset.ts", + reason: "changed alongside src/reset.ts in 3 of its 4 eligible changes", + occurrences: 3, + seedChanges: 4 + }] + }], + inspectionOrder: ["src/reset.ts", "src/session.ts"], + history: { available: true, eligibleCommits: 20, shallow: false, truncated: false } + }; + + expect(validateFixMapReport({ ...envelope, impact }, "report").success).toBe(true); + const invalid = validateFixMapReport({ + ...envelope, + impact: { ...impact, files: [{ ...impact.files[0], path: "../outside.ts" }] } + }, "report"); + expect(invalid.success).toBe(false); + if (!invalid.success) expect(invalid.message).toContain("impact.files entry"); + }); + it("rejects an unknown identifier grounding status", () => { const result = validateFixMapReport({ ...envelope, diff --git a/scripts/evaluate-agent-study.mjs b/scripts/evaluate-agent-study.mjs new file mode 100644 index 0000000..60cd3cc --- /dev/null +++ b/scripts/evaluate-agent-study.mjs @@ -0,0 +1,77 @@ +import { readFile } from "node:fs/promises"; +import { fileURLToPath } from "node:url"; +import { dirname, join, resolve } from "node:path"; + +const root = resolve(dirname(fileURLToPath(import.meta.url)), ".."); +const protocolPath = join(root, "benchmarks", "agent-study", "protocol.json"); +const protocol = JSON.parse(await readFile(protocolPath, "utf8")); +const requiredArms = ["baseline", "fixmap-available", "fixmap-instructed", "fixmap-impact"]; +const requiredMetrics = [ + "taskResolved", "correctFileInFirstThreeOpened", "toolCallsToFirstRelevantFile", + "filesOpenedBeforeFirstEdit", "incorrectFilesEdited", "totalToolCalls", "inputTokens", + "outputTokens", "testsSelectedCorrectly", "finalPatchAccepted", "fixmapPlanUsed", "verifyUsefulWarnings" +]; + +if (protocol.protocolVersion !== 1 || protocol.status !== "protocol-only" || + JSON.stringify(protocol.arms) !== JSON.stringify(requiredArms) || + requiredMetrics.some((metric) => !protocol.metrics.includes(metric)) || + Object.values(protocol.requirements).some((value) => value !== true)) { + throw new Error("Agent-study protocol is incomplete or has drifted from its frozen four-arm contract."); +} + +const inputIndex = process.argv.indexOf("--input"); +if (inputIndex === -1) { + process.stdout.write("Agent-study protocol valid. No run data supplied; no effectiveness result is claimed.\n"); + process.exit(0); +} +const inputPath = process.argv[inputIndex + 1]; +if (!inputPath) throw new Error("--input requires a JSONL run file."); +const rows = (await readFile(resolve(inputPath), "utf8")) + .split(/\r?\n/).filter(Boolean).map((line, index) => { + try { return JSON.parse(line); } catch (error) { throw new Error(`Invalid JSONL at line ${index + 1}: ${error.message}`); } + }); +if (rows.length === 0) throw new Error("Agent-study input contains no runs."); + +const keys = new Set(); +for (const [index, row] of rows.entries()) { + for (const field of ["taskId", "arm", "model", "modelVersion", "repository", "revision", "transcript", ...requiredMetrics]) { + if (!(field in row)) throw new Error(`Run ${index + 1} is missing ${field}.`); + } + if (!requiredArms.includes(row.arm)) throw new Error(`Run ${index + 1} has unknown arm ${JSON.stringify(row.arm)}.`); + if (typeof row.transcript !== "string" || !row.transcript.trim()) throw new Error(`Run ${index + 1} has no transcript reference.`); + const key = `${row.taskId}\0${row.arm}`; + if (keys.has(key)) throw new Error(`Duplicate task/arm run: ${row.taskId} / ${row.arm}.`); + keys.add(key); +} + +const tasks = [...new Set(rows.map((row) => row.taskId))]; +for (const task of tasks) { + const taskRows = rows.filter((row) => row.taskId === task); + const arms = taskRows.map((row) => row.arm).sort(); + if (JSON.stringify(arms) !== JSON.stringify([...requiredArms].sort())) { + throw new Error(`Task ${task} does not contain exactly one run from every arm.`); + } + for (const field of ["model", "modelVersion", "repository", "revision"]) { + if (new Set(taskRows.map((row) => row[field])).size !== 1) throw new Error(`Task ${task} does not hold ${field} constant.`); + } +} + +const aggregate = Object.fromEntries(requiredArms.map((arm) => { + const armRows = rows.filter((row) => row.arm === arm); + const rate = (field) => armRows.filter((row) => row[field] === true).length / armRows.length; + const median = (field) => { + const values = armRows.map((row) => row[field]).filter(Number.isFinite).sort((a, b) => a - b); + if (values.length === 0) return null; + const middle = Math.floor(values.length / 2); + return values.length % 2 ? values[middle] : (values[middle - 1] + values[middle]) / 2; + }; + return [arm, { + runs: armRows.length, + taskResolutionRate: rate("taskResolved"), + acceptedPatchRate: rate("finalPatchAccepted"), + firstThreeFileRate: rate("correctFileInFirstThreeOpened"), + medianToolCallsToRelevantFile: median("toolCallsToFirstRelevantFile"), + medianTotalToolCalls: median("totalToolCalls") + }]; +})); +process.stdout.write(`${JSON.stringify({ protocolVersion: 1, tasks: tasks.length, aggregate }, null, 2)}\n`); diff --git a/server.json b/server.json index 4e07472..7143ba8 100644 --- a/server.json +++ b/server.json @@ -1,17 +1,17 @@ { "$schema": "https://static.modelcontextprotocol.io/schemas/2025-12-11/server.schema.json", "name": "io.github.aryamthecodebreaker/fixmap", - "description": "Deterministic local-first repo context for coding agents with one-command GitHub issue analysis.", + "description": "Deterministic local-first context and impact maps for coding agents from tasks, issues, and diffs.", "repository": { "url": "https://github.com/aryamthecodebreaker/FixMap", "source": "github" }, - "version": "0.8.9", + "version": "0.9.0", "packages": [ { "registryType": "npm", "identifier": "@aryam/fixmap", - "version": "0.8.9", + "version": "0.9.0", "transport": { "type": "stdio" },