feat(cli): selftest command and a compatibility report form (#179)

.github/ISSUE_TEMPLATE/compatibility.yml

@@ -0,0 +1,49 @@
+name: Compatibility report
+description: Send the result of `solidworks-mcp-python selftest` on your SolidWorks
+labels: ["compatibility"]
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Thanks for checking your setup. With SolidWorks running and idle, run:
+
+        ```
+        uvx solidworks-mcp-python selftest
+        ```
+
+        It takes about three minutes, opens and closes a few documents of its own, and leaves yours alone. It prints a link that opens this form with the summary filled in; otherwise paste the summary it printed below. Failing results are just as useful as passing ones.
+
+  - type: textarea
+    id: summary
+    attributes:
+      label: Selftest summary
+      description: The Markdown summary the selftest printed, from "### Selftest" to the last area.
+      placeholder: "### Selftest: 4 of 4 areas passed"
+    validations:
+      required: true
+
+  - type: input
+    id: solidworks
+    attributes:
+      label: SolidWorks version and service pack
+      placeholder: e.g. 2024 SP5.0
+    validations:
+      required: true
+
+  - type: input
+    id: windows
+    attributes:
+      label: Windows version
+      placeholder: e.g. Windows 11 24H2
+
+  - type: input
+    id: client
+    attributes:
+      label: MCP client and model, if you use one
+      placeholder: e.g. Claude Desktop + Sonnet, VS Code + Copilot
+
+  - type: textarea
+    id: notes
+    attributes:
+      label: Anything else?
+      description: Add-ins, a non-English install, PDM, anything that might matter.

.github/ISSUE_TEMPLATE/feedback.yml

@@ -7,6 +7,8 @@ body:
       value: |
         Thanks for trying the server. Short answers are fine; skip anything you don't know.
         For a specific crash or wrong result, a Bug Report gets you a faster fix.
+        To report how the server runs on your SolidWorks version, run
+        `uvx solidworks-mcp-python selftest` and use the Compatibility report form.
 
   - type: textarea
     id: tried

CHANGELOG.md

@@ -14,6 +14,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- **`solidworks-mcp-python selftest`** (#179; `--selftest` works too).
+  With SolidWorks running, it builds a part, an assembly, a drawing view
+  and a STEP export through the MCP tools, checks each against a hand
+  calculation, and prints a report with a link that opens the new
+  Compatibility report form already filled in. It never touches a
+  document you had open. The docs list the releases checked so far.
 - **Tools the U-joint needed** (#209), each checked on SolidWorks 2026:
   - `add_mate` takes a face as a point on it (`face_a`, `face_b`) in that
     part's own coordinates, so a pin mates concentric in its hole;

README.md

@@ -196,7 +196,7 @@ Each GIF is a live test driving SolidWorks 2026 through the MCP tools, with the
 ## Supported Today
 
 - Windows + SolidWorks COM automation for the main CAD lifecycle.
-- Tested on SolidWorks 2026. Earlier versions may work but are untested; if you try one, please [open an issue](https://github.com/andrewbartels1/SolidworksMCP-python/issues/new/choose) with what worked.
+- Tested on SolidWorks 2026. Earlier versions may work but are untested. To check yours, run `uvx solidworks-mcp-python selftest` with SolidWorks open, then send the result with the [compatibility report form](https://github.com/andrewbartels1/SolidworksMCP-python/issues/new?template=compatibility.yml). The [compatibility page](https://andrewbartels1.github.io/SolidworksMCP-python/compatibility/) lists the releases checked so far.
 - Modeling, sketching, drawing, analysis, export, automation, templates, and macro tools.
 - Every tool's support level, and the reason a tool isn't stable yet, is on the [progress page](https://andrewbartels1.github.io/SolidworksMCP-python/progress/).
 

docs/compatibility.md

@@ -0,0 +1,40 @@
+# Compatibility
+
+The SolidWorks releases the selftest has been run on, and how each area
+did. The server is developed against SolidWorks 2026; every other row comes
+from a user's report.
+
+## Check your release
+
+With SolidWorks open and idle, run:
+
+```powershell
+uvx solidworks-mcp-python selftest
+```
+
+In about three minutes it checks four areas through the same MCP tools a
+client calls, each against a hand calculation:
+
+| Area | What it does | What it checks |
+| --- | --- | --- |
+| Part | A 40 × 20 × 10 mm block, then a Ø10 hole through it, then a 2 mm fillet on one edge; a Ø9.5 × 20 mm pin | Volume after each feature (8,000, then 7,214.60, then 7,180.27 mm³; the pin 1,417.64 mm³) and each bounding box |
+| Assembly | The block and the pin, mated by two faces: concentric in the hole, bottoms flush | 2 components, 2 mates, no interference |
+| Drawing | A new drawing with one isometric view of the block | Exactly one view |
+| Export | The block as STEP | The file exists and isn't empty |
+
+It saves only into a temporary folder it deletes afterwards. It closes only
+the documents it opened, without saving them, and never touches one that
+was open before it started.
+
+It prints PASS, FAIL or SKIP per area, then a summary and a link that opens
+the [compatibility report form](https://github.com/andrewbartels1/SolidworksMCP-python/issues/new?template=compatibility.yml)
+with the summary filled in. A failing report is as useful as a passing one:
+it shows what to fix. `--report selftest.md` also writes the summary to a
+file. The exit code is 0 when every area passes, 1 when any fails or is
+skipped, and 2 when it can't run (SolidWorks not installed or not running).
+
+## Releases checked
+
+| SolidWorks | Date | Part | Assembly | Drawing | Export | Report |
+| --- | --- | --- | --- | --- | --- | --- |
+| 2026 SP4.0 (34.4.0), English | 2026-11-08 | PASS 8/8 | PASS 3/3 | PASS 1/1 | PASS 1/1 | the maintainer's run, and the live test `tests/live/test_live_sw_selftest.py` |

docs/getting-started/installation.md

@@ -63,6 +63,14 @@ Open SolidWorks, restart your client as setup told you, and ask *"What model is
 
 VS Code labels every line the server writes to stderr as `[warning]`. That is where its normal log goes, since stdout carries the MCP messages, so those labels aren't problems.
 
+To check the whole setup without a client, run the selftest with SolidWorks open and idle:
+
+```powershell
+uvx solidworks-mcp-python selftest
+```
+
+It builds a small part, an assembly, a drawing view and a STEP export through the same tools a client calls, checks each against a hand calculation, and prints PASS or FAIL per area. It opens and closes a few documents of its own and leaves yours alone. On a release other than 2026, please send the result with the [compatibility report form](https://github.com/andrewbartels1/SolidworksMCP-python/issues/new?template=compatibility.yml); see [Compatibility](../compatibility.md).
+
 ## Install from source
 
 To work on the server itself, install it from a clone. You need [Python 3.13+](https://www.python.org/downloads/windows/), installed with **Add python.exe to PATH** ticked, and Git.

mkdocs.yml

@@ -20,6 +20,7 @@ nav:
   - Home: index.md
   - Demos: demos.md
   - Progress: progress.md
+  - Compatibility: compatibility.md
   - Getting Started:
       - Installation: getting-started/installation.md
       - Quick Start: getting-started/quickstart.md

openspec/changes/selftest-and-compatibility-report/.openspec.yaml

@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-11-07

openspec/changes/selftest-and-compatibility-report/design.md

@@ -0,0 +1,191 @@
+## Context
+
+- The console script is `server.cli_main`. It already routes the `setup`
+  and `skill` subcommands to their own Typer apps, and runs the server for
+  anything else.
+- `PyWin32Adapter.connect()` binds to a running SolidWorks with
+  `GetActiveObject`, and **launches one** with `Dispatch` when none is
+  running. The selftest must not start SolidWorks (spec: exit 2), so it
+  checks first.
+- The checks it needs already exist as adapter reads and tools:
+  `get_model_geometry_summary` (volume, bounding box, component and mate
+  counts), `verify_model`, `check_interference`, `list_drawing_views`,
+  `list_open_documents`, `activate_document`, `close_model(save=False)`,
+  and `health_check` (`sw_version`). The language and the default
+  templates have no adapter read yet.
+- The live tests' `_ScratchSession` (`tests/live/conftest.py`) already
+  holds the rule this command needs: close only the documents that are new
+  since a baseline, and only if they were never saved or were saved to a
+  file the session wrote. It lives in the tests, so the selftest needs its
+  own copy in `src/`.
+- Face mates by a point on each part (`add_mate` with `face_a`/`face_b`)
+  arrive with #209 (PR #216). This work branches from main after it merges.
+
+## Goals / Non-Goals
+
+**Goals:**
+
+- One run takes under three minutes on SolidWorks 2026 with the default
+  settle pauses.
+- Every number in a check is worked out in `expected.py` with its formula
+  written beside it, so a reviewer can redo it by hand.
+- A failure says which tool call failed and what it returned.
+
+**Non-Goals:**
+
+- No `--mock` mode for users: the mock has no geometry (its volumes and
+  bounding boxes are `None`), so a mock run would prove nothing. The mock
+  is used only by the unit tests.
+- No automatic upload: the user reads the report and chooses to send it.
+- No change to the per-tool support levels (#214). A report is evidence a
+  maintainer reads before moving a tool up a level.
+
+## Decisions
+
+### 1. A subcommand, with `--selftest` as an alias
+
+`selftest` matches `setup` and `skill`. `cli_main` also maps a leading
+`--selftest` to it, because #179 and any early docs use that spelling. The
+alternative, a `--selftest` option on the server's Typer command, would
+mix the selftest's own options (`--report`) into the server's `--help`.
+
+### 2. Through the MCP tools, in process
+
+The selftest builds the same FastMCP server the CLI runs
+(`SolidWorksMCPServer` with the loaded config) and calls tools through an
+in-memory `fastmcp.Client`, as `tests/live/test_live_sw_ujoint_demo.py`
+does. That covers the tool boundary, input validation, the circuit
+breaker and the adapter: everything a client's call goes through. Calling
+the adapter directly would be quicker to write but would miss the
+problems users actually hit at the tool layer.
+
+### 3. Areas behind one small interface
+
+```
+selftest/
+  __init__.py
+  cli.py        # Typer app: options, precheck, exit codes
+  runner.py     # runs areas in order, skips dependents, always cleans up
+  areas.py      # PartArea, AssemblyArea, DrawingArea, ExportArea
+  expected.py   # hand calculations, one named constant per check
+  documents.py  # DocumentGuard: baseline, temp folder, close-only-ours
+  report.py     # area lines, Markdown summary, prefilled issue link
+  results.py    # Check, AreaResult, ToolCaller (raises ToolFailed)
+  environment.py# environment section from get_application_info etc.
+```
+
+Each area has a `name`, the names of the areas it `needs`, and
+`async run(context) -> AreaResult`. An `AreaResult` holds `Check`s
+(`name`, `expected`, `measured`, `tolerance`, `passed`) and the tool calls
+made. The runner does not know what an area does, and the report does not
+know how a check was measured, so adding an area touches only `areas.py`
+and `expected.py`.
+
+An area is skipped when a file it needs (the saved block or pin) is
+missing, not when an earlier area failed a check, so one wrong value
+doesn't hide the other results.
+
+### 4. The part, the assembly and the hand calculations
+
+- **Block:** a 40 × 20 mm rectangle on the Top Plane, centred on the
+  origin, extruded 10 mm. Volume 8,000 mm³; bounding box 40 × 10 × 20 mm.
+- **Hole:** a Ø10 circle at the origin, cut through all. Volume
+  8,000 − π·5²·10 = 7,214.60 mm³; the bounding box is unchanged.
+- **Fillet:** r = 2 mm on one 40 mm top edge, picked by its midpoint
+  (0, 10, 10) mm. It removes (1 − π/4)·2²·40 = 34.34 mm³, leaving
+  7,180.27 mm³.
+- **Pin:** a Ø9.5 circle extruded 20 mm. Volume π·4.75²·20 = 1,417.64 mm³.
+- **Assembly:** the block and the pin, saved in the temporary folder. A
+  concentric face mate (the hole's wall at (5, 5, 0), the pin's side at
+  (4.75, 10, 0)), then a coincident aligned mate of the two bottom faces
+  ((−15, 0, 0) and (0, 0, 0)). Expected: 2 components, 2 mates, 0
+  interferences (0.25 mm radial clearance).
+- **Drawing:** a new drawing from the default template, one isometric
+  view of the block. Expected: 1 view.
+- **Export:** the block as STEP into the temporary folder. Expected: the
+  file exists and its size is above zero.
+
+Tolerances: volume within 0.01 % (relative), lengths within 0.01 mm.
+
+### 5. Leaving the user's documents alone
+
+`DocumentGuard` is an async context manager:
+
+1. On entry it reads `list_open_documents` as the baseline and makes a
+   folder with `tempfile.mkdtemp(prefix="solidworks-mcp-selftest-")`.
+2. Every path the areas save to is inside that folder.
+3. On exit, also after an exception or Ctrl+C, it lists the open documents
+   again. It closes each one that is not in the baseline and either has no
+   path or a path inside its folder, using `activate_document` and then
+   `close_model(save=False)`. Then it deletes the folder.
+
+A document in the baseline is never activated, so the user's active
+document only changes while the selftest's own documents are open in
+front of it.
+
+### 6. Not starting SolidWorks
+
+Before connecting, `cli.py` checks that `SLDWORKS.exe` is running. It uses
+`tasklist /FI "IMAGENAME eq SLDWORKS.exe" /NH`, so it needs no new
+dependency and makes no COM call. No process means exit 2 with the
+message from the spec. A non-Windows platform, or no SolidWorks install
+(`setup.solidworks.installed_versions()` returns empty), is also exit 2.
+
+### 7. `get_application_info`, the one new adapter read
+
+`get_application_info() -> AdapterResult[dict]` returns `revision`,
+`language`, and `templates` (`part`, `assembly` and `drawing`, each with
+`file_name` and `exists`). In the pywin32 adapter it is three
+`ISldWorks` reads (`RevisionNumber`, `GetCurrentLanguage`, and
+`GetUserPreferenceStringValue` with swDefaultTemplatePart = 8,
+swDefaultTemplateAssembly = 9 and swDefaultTemplateDrawing = 10, read from
+`swconst.tlb`), run inside `_handle_com_operation`. Each read is wrapped
+in `_attempt` so one unreadable value is `None`, not an error. The mock
+returns fixed values. Only file names leave the adapter, never full paths
+(the spec's privacy rule).
+
+Python, pywin32 and package versions come from `platform`,
+`importlib.metadata` and `solidworks_mcp.version`, not the adapter.
+
+### 8. The report and the issue link
+
+- Area lines: `PASS  part       6/6 checks`, `SKIP  drawing    needs part`.
+- The Markdown summary is a fenced environment table, then one table per
+  area (check, expected, measured, tolerance, result), then the tools
+  called.
+- GitHub issue forms take field values from the query string. The link is
+  `…/issues/new?template=compatibility.yml&labels=compatibility&summary=<url-encoded summary>`.
+  It is printed only while the whole URL is under 8,000 characters. Past
+  that, the plain form link is printed with "paste the summary above". A
+  full report is about 2,500 characters, so the fallback is for failures
+  with long error messages.
+
+### 9. The compatibility page
+
+`docs/compatibility.md` is a table maintained by hand: release and
+service pack, date, the four area results, and a report link. A maintainer
+adds the row when a report arrives. The README's "Tested on" line links
+to it. A generated page was considered, but reports arrive as issues, so
+there's no structured source to generate it from yet.
+
+## Risks / Trade-offs
+
+- [The user's active document changes while the selftest runs] → The
+  guard never activates or closes a baseline document. The README and the
+  command's opening line say the selftest opens and closes a few documents
+  of its own.
+- [Default templates missing or in another language] → `create_part`
+  already falls back to the installed templates. The environment section
+  shows what was found, which is often the answer to a failing report.
+- [Face-mate behaviour differs on older releases] → That is the point of
+  the selftest. The assembly area fails with the tool's message, and the
+  other areas still run.
+- [A crash in SolidWorks during the run] → The guard's cleanup is
+  best-effort. Its failures are printed, never raised over the area
+  results, and the temporary folder is still removed.
+- [`tasklist` output is localised] → It is matched only for the image
+  name, which is not translated.
+
+## Migration Plan
+
+Additive. Nothing to migrate or roll back beyond reverting the PR.