From c9defa01d812a613c16b0dc591353eaf7e29fc35 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Fri, 25 Sep 2026 06:03:38 +0800 Subject: [PATCH 01/26] docs(design): typing `import a.` freezes clangd 23.1; the plan for 0.0.4 --- ...2026-09-25-import-hang-status-highlight.md | 596 ++++++++++++++++++ 1 file changed, 596 insertions(+) create mode 100644 .agents/docs/2026-09-25-import-hang-status-highlight.md diff --git a/.agents/docs/2026-09-25-import-hang-status-highlight.md b/.agents/docs/2026-09-25-import-hang-status-highlight.md new file mode 100644 index 0000000..17f6b93 --- /dev/null +++ b/.agents/docs/2026-09-25-import-hang-status-highlight.md @@ -0,0 +1,596 @@ +# Typing `import a.` freezes the editor; what the status means; highlighting `import` + +Status: proposal for review · measured 2026-09-25 on mcppls 0.0.3 (linux-x64 payload, bundled clangd +23.1.0 `ea7d852a`), `main` at c508e62, project `~/test/mcpp/hello` + +| # | Item | Proposal | Section | +|---|---|---|---| +| H1 | clangd 23.1 spins forever on a module name that ends in `.` at end of line | mcppls never sends that text to clangd: a same-line `;` placeholder that keeps positions | §1, §3 | +| H2 | The guards added in 0.0.2/0.0.3 did not recover | a busy clangd is not always a compiling clangd: bound it, and stop treating an edit to the file itself as a rebuild | §2, §4 | +| H3 | Half-typed imports churn the plan: a stand-in for `hello.`, clangd restarts | validate module names; don't create a stand-in for an import that is still being typed | §5 | +| S | "Some features are limited" for a missing `;` | separate *your code has an error* from *the server is working* and from *the server lost a feature* | §6 | +| K | `import` is not highlighted | mcppls fixes it in two layers: an injected grammar in the extension, and module semantic tokens from the server (a custom `module` type with fallbacks); colors stay with themes | §7 | +| W | Temporary fixes are scattered and unlabeled | one workaround module; each entry names its upstream bug, how to remove it, and a canary test | §9 | +| C | Other C++ extensions on the same files | keep asking before any change; add commands, re-checks and a notice; semantic tokens follow the spec | §7, §10 | + +## 1. H1: root cause + +**What was seen (live, the user's session).** clangd pid 1661254 at 87% CPU for minutes. Its thread +`Worker:main.cpp` was in state R with about 200 s of CPU; every other thread was asleep. ptrace is +restricted (`yama/ptrace_scope=1`), so no live stack. The server log +(`~/.cache/mcppls/logs/server-20260924-211510.237-be62.log`) shows the edit, then nothing answered: + +``` +21:15:23 restarting clangd: units are compiled with other arguments +21:15:26 clangd could not find module hello <- while typing `import hello` +21:15:27 module hello has a stand-in ... trying module hello again +21:15:56 clangd: main.cpp:3:14: error: expected identifier after '.' in module name +21:15:58 module hello. has a stand-in <- a stand-in for an invalid name +21:16:18 ... did not answer textDocument/documentHighlight in time +21:16:55 .. 21:20:08 did not answer inlayHint / codeAction / semanticTokens / documentSymbol ... (every request) +``` + +**Reproduced without mcppls.** The bundled clangd, the same compile database, and a small LSP driver +that erases `greet;` from `import hello.greet;` and types it back one key at a time. Every request +is answered until the buffer holds `import hello.`. From then on nothing is answered, even after the +text is fixed, and `Worker:main.cpp` stays at 100% of a core. The verbose log shows +`Reusing preamble version 1 for version 7`, then "building AST" never finishes. + +**Minimal case.** `clangd --check=FILE` with `-std=c++23`, no project and no modules support, gives +the same result: + +Each case was given 8–20 s. A blank cell was not run. + +| File contents | clangd 23.1.0 (bundled) | 22.1.8 | 24.0.0git `510126255` | +|---|---|---|---| +| `import hello.⏎`, `import hello.⏎int x;⏎`, `import std;⏎import hello.⏎…` | **hangs** | error, 0.1 s | error, 0.1 s | +| `import a.b.⏎`, `export import hello.⏎` | **hangs** | | | +| `export module a.⏎`, `module a.⏎`, `module;⏎export module m.⏎` | **hangs** | | | +| `import hello. ⏎`, `import hello.\t⏎`, `import hello.// c⏎` | **hangs** | | | +| `import hello.⏎;⏎` (the `;` is on the next line) | **hangs** | | | +| `import hello.;`, `import hello. ;`, `import hello.;// c` | ok, error | | | +| `import hello`, `import hello.greet` (no `;`), `import hello:`, `import hello:p.` | ok, error | | | + +**Cause.** In clang 23.1, a module name in an `import` or `module` directive that ends in `.`, +with nothing after the dot on that line except whitespace or a comment, sends the preprocessor into a +loop. This is the new P1857 code, which lexes these lines as directives. It is not a modules-support +problem in clangd: the same file hangs without `--experimental-modules-support`. + +- **Why it happens every time.** Typing any dotted name at the end of a line goes through + `import a.`, and so does `export module a.b;` in an interface unit. +- **Why nothing gets it back.** `$/cancelRequest` cannot interrupt a parse. clangd builds a file's + versions one after another on one worker (ASTWorker), so the fixed text queues behind the spin + forever. Only killing clangd frees it. +- **Upstream.** + - LLVM 23.1.1 and 23.1.2: none of their 74 commits has a title about the lexer or `import`. They + were not run. + - clangd/clangd has published no release after 23.1.0, only snapshots from main + (`snapshot_20260913` is the latest). + - Main is fixed: the 24.0.0git build above passes. + - A likely fix, not yet confirmed by bisect: `6dcfc17b1b` "[clang] Move the diagnostic of + unexpected token after module name from phase 4 to phase 7 (#187846)", 2026-08-21. It is on main, + in the 24git build, and not in 23.1.2. + - I found no upstream issue for this hang. + +## 2. Why the stability work did not help + +Each guard did what it was designed to do. The design assumed that a busy clangd is a clangd making +progress. + +1. **StuckWatch detects only an idle clangd** (`clangd.cpp:1108-1180`, `guard.cpp` `StuckWatch::check`, + `IDLE_SHARE = 0.05`). "A long compile keeps a core busy, and is left alone" + (`docs/50-troubleshooting.md`). A spin keeps a core busy too, so the verdict is always "busy, not + stuck". There is no upper bound on how long busy may last. +2. **The set-aside path is switched off by the edit that causes the hang.** + - `Quarantine::timed_out` (`clangd.cpp:839-848`) needs unanswered requests from at least 2 files + to call clangd *stalled*. With only `main.cpp` open, every unanswered request is for that one + file. + - The per-file path is gated by `changed_recently_(path)`, which is true for + `GENERAL_PATIENCE` = **120 s after any edit to the file itself**. + - The hang is caused by an edit, and the user keeps editing to fix it, so the 120 s restarts on + every key. The verdict stays `wait`. +3. **Recovery is slow and can bring the hang back.** This is read from the code, not measured. + - Once typing stops, recovery takes 120 s, then 2 more timeouts, then set-aside, then + `Reclaim::if_busy` restarts clangd (`clangd.cpp:1820-1835`, the case the code itself calls + "the spin in experiment S17"). That is about 2.5–3 minutes with every feature blank. + - When the set-aside term ends (`Quarantine::due`, 2 minutes the first time), the file goes back + to clangd with whatever text it has. If that text still has `import hello.`, clangd spins + again. The earlier hand-back on an edit (`clangd.cpp:582`) compares import *structure*, and + mcppls's scanner does not count `hello.` as an import, so it cannot tell that text is the + problem either. + - Three restarts in ten minutes reach the cap, and clangd is then left spinning. +4. **Timed-out requests get empty answers** (`request.reply(Answer {})`). Semantic tokens, hover, + symbols and code actions all go blank, which is the "plugin unusable" the user saw. + +## 3. H1 fix: do not send clangd text that hangs it (first, small) + +In the clangd engine, before `didOpen` and each `didChange` goes to clangd: + +- **Detect.** Use `project::scan`'s lexer on lines that start with `[export] import` or + `[export] module`. A module name (or partition) whose last token is `.` and is followed only by + whitespace or a comment up to the end of the line is a hazard. +- **Rewrite what clangd sees.** Insert `;` immediately after that dot, on the same line. Measured on + 23.1.0: every hazard in §1 then finishes at once with the proper `pp_module_expected_ident` error. + - **It never changes valid code.** Under P1857 an `import` or `module` directive ends at the end + of its line, so a dot there is always an error. The rewrite only turns a hang into that error. + - **Positions.** Every other line is unchanged, and so is the edited line up to and including the + dot. Only the trailing whitespace or comment after it moves right by one column. Nothing is + there to request, and a result that clangd places there is moved back by one. + - **Do not delete the dot.** `import hello` makes clangd look for module `hello`, which starts the + stand-in churn of §5. +- **Sync.** While a file's clangd text differs from the editor's text, or did at the last change, + send clangd full-text `didChange` built from `document.text`. Today the engine forwards the + client's message as is (`clangd.cpp:589`). +- **Diagnostics.** Keep clangd's `pp_module_expected_ident` ("expected identifier after '.' in module + name"). Its wording is already right; only its range moves to the dot. +- **A workaround, registered as `WA-CLANGD-001` (§9).** The code lives in the workaround module. + - Its registry entry turns it on for 23.1.x, through a trait `hangsOnTrailingDotModuleName`. + - A clangd given with `--clangd PATH` is matched by its version. + - A canary test tells us when it can go. +- **Upstream.** + 1. Bisect and confirm the fix commit. + 2. File the LLVM issue and ask for a backport to 23.1.x. + 3. Update `packaging/payload.lock.json` once clangd/clangd publishes a fixed release. + - Not recommended as the default: bundling a clangd/clangd snapshot from main. + `snapshot_20260913` postdates the candidate fix, so it should contain it, but that was not run. + It would swap one known bug for an unreleased build; at most, offer it as an opt-in. + +## 4. H2 fix: a clangd that stays busy without progress is also stuck + +A defence against the next spin nobody has found yet. Target: **features back within about 15 s, +even while the user keeps typing.** + +- **Budget the work that should be short.** clangd reports what each file is doing (`fileStatus_`). + - Building a preamble or modules can take minutes. It keeps today's patience and progress checks. + - A main-file AST build on a reused preamble is short: milliseconds for `hello`. clangd is spinning + on the file when all of these hold: + - the file's state has not changed for its budget; + - the CPU is busy; + - a newer version of the same file is waiting. + - **Budget.** `max(20 s, 5 × the file's last successful AST build)`. A heavy template file that + really takes 15 s is not called stuck. The numbers are a starting point, to tune on the + real-project stress runs (0.0.2 plan). + - **Cost of a false positive.** A restart, plus rebuilding module BMIs. That is why the budget + follows each file's own history instead of a fixed number. +- **An edit to the file itself is not a rebuild.** Split `changed_recently_`: + - Edits to module sources in the file's import closure keep `GENERAL_PATIENCE`. + - An edit to the file itself gets a short grace, about 10 s. +- **One file is enough.** If every unanswered request is for one file, clangd answers nothing else + for `STALL_WINDOW`, and the CPU is busy, that file is stuck. Today this needs 2 files. +- **Recovery.** + - Set the file aside and restart at once (no deferral). + - Remember a hash of the text clangd hung on. That exact text never goes back to clangd; any other + text goes back immediately, without waiting out a term. + - **These restarts still count toward the cap**; a new text can hang again. What changes is what + happens at the cap: + - the file that caused the restarts stays with mcppls's engine until its text changes; + - clangd is restarted once more, without that file; + - clangd is never left spinning. +- **Meanwhile, native answers.** While the file is aside, the native engine answers (module + navigation, `import` completion, module diagnostics). Timed-out requests fall back to it instead + of an empty answer where it can answer. + +## 5. H3: plan hygiene for half-typed imports + +- **Validate module names** from the model's `requires` and from scanning: `ident(.ident)*` with an + optional `:ident(.ident)*` partition. An invalid name gets no stand-in, only a diagnostic. + - The `hello.` stand-in came from mcpp's rescan after autosave (`files.autoSave: afterDelay`). + mcppls's own scanner rejects a trailing dot (`scan.cpp:190-205`). + - Report this to mcpp. +- **No stand-in for an import that is still being typed.** A name that does not resolve in a file + the user is editing is usually a typo in progress. + - Report it as a diagnostic on the import, e.g. "module `hello` not found; did you mean + `hello.greet`?". + - Create a stand-in only after the text has been stable for a while, or for imports in files that + are not open. + - **Exception: a unit that provides a module** (`.cppm`, partitions). Building such a unit with an + import that does not resolve is what deadlocks clangd 23.1 (`plan.cpp`, "building it is what + deadlocks"). It keeps today's rule, a stand-in or leaving it out (`WA-CLANGD-002`, §9). A plain + source file such as `main.cpp` does not need one: without a stand-in, clangd just reports the + module as not found. +- **To investigate: the 21:15:23 restart during an import edit** ("units are compiled with other + arguments"). Changing a file's imports should not restart clangd. Add a test once the cause is + found; this analysis did not confirm it. + +## 6. S: the status must say *whose* problem it is + +**Today.** +- `compute_state()` (`workspace.cpp:1107-1122`) returns `degraded` for any engine issue, model issue + or plan issue. +- `plan.issues` includes `unresolved-module` and `module-build-failed`, which a typo or an import + still being typed produces. +- The extension shows every `degraded` state as "Some features are limited" on a warning background + (`editors/vscode/src/status.ts:117`). + +So a missing `;`, a slow rebuild, and a clangd that has lost the file all look the same. With +autosave, each half-typed state reaches the plan, so the warning flickers while typing. + +**Measured.** +- On a copy of the project with `import hello.greet` (no `;`) on disk, `mcppls report` gives state + `ready` and no issues. The final text alone is fine; clangd reports + `expected_semi_after_module_or_import`, as it does `expected_semi_after_expr` for a statement. +- In the live session, the intermediate texts `import hello` and `import hello.` each put a plan + issue in place (log at 05:15:27, 05:15:58, 05:20:02). Each one turned the status to "limited". +- These are code problems reported as a lost feature. + +**Rule: attribute every problem to where it comes from.** +- Each plan or engine issue carries the file and range that caused it. +- If the cause is text in a document the user has open (a syntax error, an unterminated `import`, an + import name that resolves to nothing, a module that fails because of its own code), it is a + **code** problem. + - It becomes a diagnostic on that range, with a clear source (clang, or mcppls for module names). + - It never changes the status. +- Only causes outside the user's text are **limited**: toolchain, environment, a dependency's + missing generated module, clangd failing. +- The model layer has to follow the same rule. mcpp's scan result and the plan's `unresolved-module` + or `module-build-failed` issues have to say which import of which file caused them, so they can + be shown as code problems. + +**Proposal: three categories, each shown in its own place.** + +| Category | Examples | Where | Status bar | +|---|---|---|---| +| **Code**: the user's source is wrong | `import hello.`, `import helo;`, a module that does not compile because of its code | a diagnostic on the line (Problems panel) | stays ✓ ready | +| **Working**: temporary | rebuilding after an edit, preparing modules, restarting clangd | spinner, "Analyzing main.cpp", shown only after about 1 s | spinner | +| **Limited**: a feature is actually lost | clangd hung and restarted, file set aside, restart cap reached, clangd cannot run, kit or SDK missing, stale model | warning with the scope and the loss: "main.cpp: basic features only; clangd stopped responding" | ⚠ / ✖ | + +- **A code problem with a consequence.** A module that does not compile because of its own code + sends its importers to mcppls's engine (existing containment, RP1.1). + - It stays a **code** problem: the fix is in the user's code, not in the setup. + - The `module-failed` diagnostic on each importer's `import` says what it costs: "module-level + features only until `hello.greet` compiles". + - The status bar stays ✓. It does not also warn. +- **Restarting clangd because it hung** is **working** while the restart runs. The cause stays + afterwards as a *limited* notice naming the file, until that file answers again. +- **Protocol (S3 extension, backward compatible).** + - `ModuleIssue` gets an optional `category` (`code` | `engine` | `environment` | `project`) and + `files`. + - `compute_state` ignores `code` issues, which become diagnostics instead. + - The status text comes from the top issue, not from a generic phrase. + - Document this in `docs/specs/s3-lsp-extensions.md`. +- **Hysteresis.** Show `degraded` only after it has lasted about 3 s. Hard failures such as + `engine-incompatible` or `payload-corrupt` show at once. +- **Wording.** + - "Some features are limited" goes away. + - `error` ("Only module-level features are available") stays, but names why. + +## 7. K: highlighting `import` — mcppls fixes it, in two layers + +**Measured.** +- **Grammar.** VS Code's built-in C++ grammar (better-cpp-syntax `071dd6e`) defines a + `module_import` rule, but nothing includes it (0 references), so `import std;` gets no keyword + scope. `export` and `module` are colored, as storage modifiers. +- **Semantic tokens.** clangd 23.1 on `main.cpp`, with modules built: tokens start at line 4 + (`main`). The `import` lines get **no tokens at all**, neither the keyword nor `std` or + `hello.greet`. clangd's legend has `namespace` but no `keyword` type. +- **This machine.** `import` is colored here only because the separate mcpp-vscode extension + injects `source.cpp.mcpp-modules`. Without it, a user of mcppls sees `import` uncolored. +- **Earlier claim.** The 0.0.3 plan (§7) said highlighting "needs nothing from us". That is wrong for + module syntax. + +**Who owns what.** + +| Layer | What it knows | Owner | In mcppls | +|---|---|---|---| +| Lexical/syntax (keywords, strings, comments) | the text only; must work on every key and with no server | the editor: TextMate (VS Code), tree-sitter (Neovim, Zed, Helix) | layer 1, and only for the module syntax the editor's grammar misses | +| Semantic (what a name *is*: class, function, module; declaration; imported) | the whole project | the language server, through `semanticTokens` | layer 2: mcppls's own module feature | +| Colors | nothing about code | the theme and the user's settings; LSP carries no colors, by design | defaults and documentation only | + +Both layers are mcppls's own code: the VS Code extension and the server. + +**Layer 1: an injection grammar in the VS Code extension.** It colors at once, while typing, and +even when the server is down. +- `source.cpp.mcppls-modules`, `injectTo: source.cpp`, selector `L:source.cpp`. +- It covers: + - `module;` + - `[export] module name[:part];` + - `module :private;` + - `[export] import name | :part |
| "header";` +- Anchored at line start, and it accepts incomplete names, so `import hello.` is colored while it is + typed. +- Scopes are the same as mcpp-vscode (`keyword.control.import.cpp`, `keyword.control.module.cpp`, + `keyword.control.export.cpp`, `entity.name.namespace.module.cpp`), so installing both is harmless. +- This layer makes up for VS Code's grammar, so it is registered in §9 as `WA-VSCODE-001`. Remove it + when the built-in grammar colors `import std;`; a canary grammar test checks that. + +**Layer 2: semantic tokens from the server.** This covers every editor (Neovim included), module +names too, and still answers when clangd is out. It stays within LSP 3.17: +- **Legend owned by mcppls.** + - mcppls declares its own fixed legend in `initialize`: clangd's types, plus `keyword`, a + predefined `SemanticTokenTypes` value. + - It maps clangd's token-type indices into that legend on every response. It does not rely on + appending to clangd's legend, so a clangd restart, a clangd given with `--clangd`, or no clangd + at all cannot shift the indices. +- **What the native engine tokenizes.** From its own scan of the document (`project::scan_source`): + - `export`, `module` and `import` as `keyword`; + - module and partition names as a custom type **`module`**, with the `declaration` modifier in + `export module` and a custom `partition` modifier on partitions. +- **Custom types need a fallback in every client.** LSP allows custom types, but a theme only colors + what it knows. + - The VS Code extension declares `contributes.semanticTokenTypes`: `module` with + `superType: namespace`. Any theme then colors it as a namespace until the user picks a color. + - The Neovim plugin sets `@lsp.type.module` with `default = true`, linked to `@module`. + - A client that does not say it knows `module` (in `initializationOptions`, sent by mcppls's own + plugins) gets `namespace` instead. Zed, Helix and the rest then still get a sensible color. +- **Later, and separately designed: `imported` and `exported` modifiers** on the names clangd + colors (for example, "this function comes from module `hello.greet`"). This is what only a + modules-aware server can do. It needs mapping each name to its module through the native index + (`engine/native/exports.cpp`), and is not part of this change. +- **clangd wins every position it covers.** mcppls adds tokens only where clangd has none. Tokens + stay sorted and never overlap (overlap is only allowed with the client's + `overlappingTokenSupport`). Measured: clangd has nothing on the `import` lines, so today nothing is + ever dropped. +- **Requests.** + - Advertise `full` without `delta`. mcppls re-encodes every answer, so it cannot pass on clangd's + `resultId`s. + - A `full/delta` request from an older client gets a full result, which LSP allows + (`SemanticTokens | SemanticTokensDelta`). + - `range`: filter to the range. +- **Refresh.** Send `workspace/semanticTokens/refresh`, if the client has `refreshSupport`, when + clangd restarts, when a file is set aside or handed back, and when modules finish preparing. +- **The same color in both layers (VS Code).** + - As far as I recall Dark+, the grammar's `keyword.control.*` is purple and semantic `keyword` + maps to scope `keyword`, which is blue. If so, `import` would change color once the tokens + arrive. To confirm in the test below. + - The extension contributes `semanticTokenScopes` for `cpp`: `keyword` → + `keyword.control.cpp`. clangd emits no `keyword` tokens (its legend has none), so nothing + else changes. + - Themes with their own `semanticTokenColors.keyword` still decide. Check Dark+, Light+ and one + third-party theme, looking for no color change between the grammar and the tokens. +- **Switch and customization.** + - `mcppls.semanticTokens.modules` (default on), also an `initializationOptions` field and a + Neovim option. It is for people who prefer their own grammar or tree-sitter colors. + - Colors are customized the standard way, and the docs show how: + - VS Code: `editor.semanticTokenColorCustomizations.rules` (`"module": …`, + `"*.partition": …`); + - Neovim: `vim.api.nvim_set_hl(0, '@lsp.type.module', …)`. +- **No bundled color themes.** + - A theme replaces the colors of the whole editor, and people choose theirs on purpose. + - Themes would need dark, light and high-contrast variants to maintain. + - They cannot be shared across editors (Neovim colorschemes are another system). + - If ever wanted, a theme is a separate optional extension, not part of the language server. +- **Neovim.** Tokens show through the `@lsp.type.keyword` and `@lsp.type.namespace` groups. The + smoke test on Neovim 0.10 to 0.12 checks they are linked by default. +- **clangd missing or hung, or the file set aside.** Answer the native tokens alone, instead of + today's `null`. +- **This is not a workaround.** It is a feature, permanent: clangd does not tokenize module syntax at + all. + +**Tests.** +- A `vscode-tmgrammar-test` fixture for layer 1. +- A conformance case for layer 2, with clangd both present and absent: + - `import`/`module`/`export` come back as `keyword`; + - module names come back as `module`, or as `namespace` for a client that did not ask for + `module`; + - clangd's token types come back mapped to the right legend entries; + - no two tokens overlap. +- A VS Code check that `import` keeps one color from the grammar through the semantic tokens, in + Dark+ and Light+. +- This also fills the gap the 0.0.3 plan recorded: no test covered semantic tokens. + +## 8. Acceptance + +- **Conformance `typing-import`, with the real bundled clangd.** + - Type `import hello.greet;` in `main.cpp` and `export module a.b;` in a `.cppm`, one key at a + time. + - Every request is answered within `INTERACTIVE_LIMIT`. + - clangd is idle within 5 s after typing stops. + - No restart and no stand-in. +- **Sanitizer unit tests.** Every hazard in §1 is rewritten, and nothing else is. Positions are + preserved. +- **Guard test.** A fault-injected clangd that spins on a marker text. + - Features are back within about 15 s while edits continue. + - The text it hung on is not resent. + - At the restart cap, the file stays with mcppls's engine and clangd is not left spinning. + - A file that really takes 15 s to build, while it is being edited, is not called stuck. +- **Status tests.** + - `import helo;` gives a diagnostic and the state stays `ready`. + - Typing `import hello.greet;` one key at a time, with autosave on, keeps the state `ready` + through every intermediate text. A missing `;` is only a syntax diagnostic. + - A spinning clangd gives `degraded` with text that names the file. + - Changes shorter than the hysteresis window do not flicker. +- **Highlighting**: the grammar fixture and the semantic-token case (§7). +- **Canary tests** for every workaround in §9. + +## 9. W: workarounds in one module, each one labeled + +**Why.** Some of these fixes exist only because of a bug in someone else's code (clangd 23.1, VS +Code's grammar, mcpp's scanner). They should go away when that bug does. Today such code is spread +across `clangd.cpp` and `plan.cpp`, gated by `EngineTraits` bools (`clangd.cpp:28-45`). Nothing +records why each one exists or when it can go. + +**Design.** +- **Where.** + - `src/engine/clangd/workarounds.cppm` / `workarounds.cpp`, module + `mcppls.engine.clangd.workarounds`, for engine workarounds. + - `editors/vscode/src/workarounds.ts`, plus the grammar file, for the extension. +- **A registry, one entry per workaround.** + + ```cpp + struct Workaround { + std::string_view id; // "WA-CLANGD-001": grep-able, used in comments, logs, report + std::string_view title; // what it works around, one line + std::string_view engine; // "clangd" + VersionRange affects; // data, e.g. [23.1.0, 24.0.0); compared with the running engine's version + std::string_view upstream; // issue / fix commit URL, or "unfiled" (then filing is a to-do) + std::string_view evidence; // doc section and fixture, e.g. ".agents/docs/2026-09-25-...#1" + std::string_view added; // mcppls version and date + std::string_view removeWhen; // the condition, e.g. "bundled and minimum clangd contain 6dcfc17b1b" + std::string_view canary; // the test that fails once the upstream bug is gone + }; + ``` + +- **Gating comes from the registry.** `traits_for_version` sets its bools from the entries that apply + to the engine's version, so the registry is the only list. +- **One code path per workaround.** The code lives in the module, for example + `sanitize_module_names(text) -> {text, insertions}`. The call site carries one comment: + `// WA-CLANGD-001, see workarounds.cppm`. +- **Visible.** + - At startup, one log line lists the active workarounds for the engine version. + - `mcppls report` lists them under `engines[].details.workarounds`. + - Bug reports then show which ones were on. +- **Canary tests drive removal.** Each entry has a test that runs the upstream bug with the + workaround off, and expects the bug. + - Example for `WA-CLANGD-001`: `clangd --check` on `import hello.` must time out. + - When a clangd update in `payload.lock.json` fixes the bug, the canary fails with "WA-CLANGD-001 + is no longer needed: remove it". + - Removing a workaround is then a failing test, not someone's memory. + +**Initial entries.** + +| ID | What | Affects | Remove when | Canary | +|---|---|---|---|---| +| WA-CLANGD-001 | Same-line `;` after a trailing-dot module name (§3) | clangd [23.1.0, 24): 23.1.0 measured, .1/.2 assumed from commit titles | the bundled and minimum supported clangd contain the fix (bisect; candidate `6dcfc17b1b`) | `--check` on `import hello.` times out | +| WA-VSCODE-001 | Module-syntax injection grammar (§7 layer 1) | VS Code built-in cpp grammar `071dd6e` | the built-in grammar colors `import std;` | tmgrammar test without the injection | +| WA-CLANGD-002..005 | Existing: stand-ins for unresolved imports (`hangsOnUnresolvedImports`), module preparation, module hints, MSVC STL aligned allocation | clangd 23.1.x | to be written when moved: each needs its upstream reference and a canary | to be written | + +**Not workarounds, kept out of the registry.** These stay whatever clangd does: +- H2, the busy-without-progress guard (§4). It is the defense for the next unknown bug; its numbers + are tuning, not a workaround. +- H3, module-name validation and no stand-in for an import being typed (§5). This is input + checking; reporting the mcpp scanner bug is a separate to-do. +- S, status categories (§6). +- K layer 2, semantic tokens (§7). + +## 10. Coexistence with other C++ language extensions + +**What exists today.** +- **VS Code** (`editors/vscode/src/conflicts.ts`). Once per workspace, it asks to turn off the + language features of cpptools (`C_Cpp.intelliSenseEngine: "disabled"`) and vscode-clangd + (`clangd.enable: false`). The change goes in the *workspace* settings, and an e2e "conflicts" + scenario tests it. +- **Neovim** (`editors/nvim/lua/mcppls/init.lua:242-280`). It says once when `clangd` or `ccls` also + attaches to a buffer. +- **Other editors.** The server itself sends one `showMessage` explaining how to disable the + editor's own clangd (`workspace.cpp:1138-1170`). Zed gets the exact settings. A client that + detects conflicts itself turns this off with `conflictArbitration: "client"`. + +**How semantic tokens behave next to another server.** VS Code does not merge semantic tokens +from several extensions: it uses one provider's result for a document. With vscode-clangd also +active, either its tokens or mcppls's are shown, never both. Layer 1 (the grammar) does not depend +on this, which is one more reason to keep it. Colors do not get mixed up, but mcppls's module tokens +may not show while another C++ server is active. + +**Gaps.** +1. **The question is asked once.** After "Keep" or closing the message, duplicates remain and nothing + says so again. There is also no way to do it later. +2. **Checked only at activation.** Installing or enabling vscode-clangd, or turning IntelliSense back + on, is not noticed. +3. **Only settings can be changed.** The VS Code API cannot disable another extension; changing its + settings is the only lever. An extension with no "enable" setting (ccls, for example) can only be + pointed out. +4. **Observed on this machine.** vscode-clangd is installed and enabled, and `clangd.enable` is not + set. No second clangd runs only because there is no `clangd` on `PATH`. Once there is one, two + engines serve the same files, and the status says nothing. + +**Proposal.** +- **Keep the rule: never change another extension without asking.** + - Only settings, in the workspace by default. + - Reversible, and written to the log. + - cpptools: turn off IntelliSense only, not the extension. Its debugger keeps working. +- **Commands.** + - `mcppls: Turn Off Other C++ Language Features`, with a choice of this workspace or everywhere + (user settings). + - `mcppls: Restore Other C++ Language Features`, which puts back the previous values. + - The first-run question stays as a shortcut to the first command. +- **Check again when things change.** Listen to `vscode.extensions.onDidChange` and + `workspace.onDidChangeConfiguration`. A new conflict gives a status notice (category + `environment`, §6): "clangd extension also active: results may appear twice". It has the "Turn + off" action and no warning background; this is a notice, not a lost feature. +- **Candidates.** + - cpptools: `C_Cpp.intelliSenseEngine`. + - vscode-clangd: `clangd.enable`. + - Extensions with no enable setting (ccls): a notice whose action opens the extension in the + Extensions view (`@id:`), so the user can pick "Disable (Workspace)". + - mcpp-vscode is not a conflict: it has no language server, and its grammar uses the same scopes. +- **Neovim.** Keep the notice, and add an option `disable_conflicting = true`, off by default. It + stops `clangd`/`ccls` clients on buffers mcppls serves, with a message saying which one it + stopped. +- **Tests.** + - Extend the e2e conflicts scenario: installing or enabling the clangd stub later gives the notice. + - Restore puts back the previous values. + - The command writes to the scope the user chose. + +## 11. Decisions for review + +| # | Question | Recommendation | Alternative | +|---|---|---|---| +| D1 | How to keep clangd 23.1 away from `import a.` | rewrite what clangd sees (§3) | hold the version back from clangd: simpler, but diagnostics and tokens go stale while typing | +| D2 | Bundle a clangd snapshot from main? | no; keep 23.1.0 plus WA-CLANGD-001 | opt-in snapshot for people willing to test | +| D3 | Budget for a main-file AST build | `max(20 s, 5 × last build)`, then tune on stress runs | a fixed number: simpler, and it misjudges heavy files | +| D4 | Restarts caused by a hung text | count toward the cap; at the cap, keep the file out and restart without it | exempt them from the cap: faster, with a risk of restart storms | +| D5 | Status categories | `code` / `working` / `limited`, status from the top non-code issue | keep `degraded` and only reword it | +| D6 | Module names as a custom `module` type | yes, with `superType: namespace` and a `namespace` fallback | `namespace` only: no customization hook | +| D7 | Bundled color themes | no | a separate optional theme extension, later | +| D8 | "Turn off other C++ features" scope | this workspace by default, "everywhere" on request | global by default | +| D9 | Neovim `disable_conflicting` | available, off by default | on by default | + +Order: H1 (inside the workaround module, the module created with it) and K layer 1, which are small +and fix what the user sees; then H2, then S, then K layer 2 and H3, and C. Moving the existing +workarounds into the registry comes after. Upstream work (bisect, LLVM issue, backport, clangd +update, the mcpp scanner report) runs in parallel. + +## 12. Implementation plan (0.0.4) + +One pull request, one version: **0.0.4**. Every item in §3–§10 is in it. + +**Measured while planning** (so the design follows the facts): +- **The spin's signature.** A normal rebuild of `main.cpp` shows `parsing includes, parsing main + file` for about 10 ms, then `idle`. During the spin that state never changes again, while newer + versions keep arriving. +- **No stand-in needed for a plain `.cpp`.** With an import nothing provides and no stand-in, + clangd answers normally: one `Module 'nosuch' not found` diagnostic, hover and tokens work. A + stand-in is therefore needed only when an importer itself provides a module. +- **Unresolved imports already have a diagnostic.** The native index publishes `unresolved-module` + ("module 'x' not found") on the import. A code-category issue needs no new diagnostic. +- **Timed-out requests already fall back.** `Answer {}` is "unavailable", which hands the request + to the next engine. The native engine answers what it can (module navigation, hover, completion, + outline). + +**Shared contracts** (fixed before the parallel work starts): + +| Contract | Shape | +|---|---| +| `initializationOptions.semanticTokens` | `{ "modules": bool = true, "moduleType": bool = false }`. `modules`: the server adds module-syntax tokens. `moduleType`: the client knows the custom type `module` and modifier `partition`; otherwise module names are sent as `namespace`. | +| Semantic token legend | Owned by the server: a fixed base (LSP standard types and modifiers, clangd's own extras, `module`, `partition`), with any other name clangd declares appended at `initialize`. clangd's indices are mapped by name on every response. `full` and `range`, no `delta`. | +| Status issue `category` | `"code"`, `"engine"`, `"environment"` or `"project"`, optional. Only non-`code` issues can make the state `degraded`. A client with no category treats the issue as non-code, as before. | +| Status hysteresis | The server holds a `ready` → `degraded` change for 3 s, so a passing condition never reaches any editor. `error` is sent at once. | +| VS Code contributions | Grammar `source.cpp.mcppls-modules`; `semanticTokenTypes` `module` (superType `namespace`); `semanticTokenModifiers` `partition`; `semanticTokenScopes` for `cpp`: `keyword` → `keyword.control.cpp`, `module` → `entity.name.namespace.module.cpp`. Setting `mcppls.semanticTokens.modules`. | +| Neovim | `semantic_tokens_modules` (default true) and `moduleType = true` sent. `@lsp.type.module` linked to `@module` by default. Option `disable_conflicting` (default false). | + +**Tasks and dependencies.** + +``` +T0 contracts (above) + ├─ A server core (main branch of work) + │ A1 workaround module: registry, WA-CLANGD-001 sanitizer, 002–005 registered, report/log ─┐ + │ A2 H2: self-edit grace, spin detection, poisoned text, restart at the cap (after A1) ────┤ + │ A3 H3: stand-ins only for module-providing importers, module-name validation ─────────────┤ + │ A4 S: issue categories, state from non-code issues, hysteresis ───────────────────────────┤ + │ A5 conformance: `type-text` and `clangd-check` kinds; `typing-import`, `workaround-canaries` fixtures ─┤ + ├─ B semantic tokens in the server (parallel, own worktree) ───────────────────────────────────┤ + ├─ C VS Code: grammar, token contributions, setting, conflicts, status rendering (parallel) ───┤ + └─ D Neovim: options, highlight link, disable_conflicting (parallel) ──────────────────────────┤ + ▼ + E specs (S3), user docs (en, zh-CN), CHANGELOG, version 0.0.4, design record ── F integrate, test all, PR, CI, review, merge, release, verify +``` + +**How each goal dimension is met.** +- **Architecture.** Workarounds live in one module, driven by a registry. Semantic tokens use the + existing `merge` role. Categories are decided where issues are made, not in the client. +- **Stability.** The spin can no longer be triggered (H1). Any future spin is found in about 20 s + and recovered even while the user types (H2). Churn from half-typed imports is gone (H3). +- **Simplicity.** H3 needs no timers (the provider rule). H2 needs no CPU reading (history and a + waiting newer version decide). +- **User experience.** Nothing freezes. The status says whose problem it is. `import` is colored. + Conflicts can be turned off or restored at any time. +- **Compatibility and seamless upgrade.** + - Every new protocol field is optional. + - A 0.0.3 extension with a 0.0.4 server sees fewer false `degraded` states; a 0.0.4 extension with + an older server behaves as before. + - No cache or setting has to change. + - The sanitizer turns off by clangd version. +- **Cross-platform.** Nothing platform-specific: text rewriting, timers, JSON. Conformance runs on + Linux x64 and arm64, macOS and Windows as before. +- **Consistency.** State, hysteresis and token types are decided by the server, so every editor + sees the same thing. From c5b1f75bbee2bba096b93dfa3c6883864b2f217f Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Fri, 25 Sep 2026 06:12:42 +0800 Subject: [PATCH 02/26] fix(engine): clangd is never given a module name that ends in a dot at the end of its line clangd 23.1 spins forever on `import hello.` (or `export module a.`) when nothing but white space or a comment follows the dot on that line, and every later version of the file queues behind the spin: typing any dotted import went through that text, so the editor froze each time. Such a directive is always an error, and clangd reports the same line with `;` right after the dot at once, so that is the text it is given; its diagnostics are mapped back to the editor's text, and a unit opened from disk is rewritten the same way. The rewrite is WA-CLANGD-001 in a new registry of what the server does only because of a defect in clangd (mcppls.engine.clangd.workarounds): each entry names the versions that need it, the upstream defect, the evidence, when it can go and the canary that will say so. The engine's traits are now read from it, the four existing compensations are registered as WA-CLANGD-002 to 005, and the log and the report list the ones a clangd version needs. --- src/engine/clangd.cpp | 104 ++++++++++++--- src/engine/clangd/workarounds.cpp | 207 +++++++++++++++++++++++++++++ src/engine/clangd/workarounds.cppm | 68 ++++++++++ src/engine/engine.cppm | 1 + tests/test_workarounds.cpp | 99 ++++++++++++++ 5 files changed, 464 insertions(+), 15 deletions(-) create mode 100644 src/engine/clangd/workarounds.cpp create mode 100644 src/engine/clangd/workarounds.cppm create mode 100644 tests/test_workarounds.cpp diff --git a/src/engine/clangd.cpp b/src/engine/clangd.cpp index 70859b5..0e71406 100644 --- a/src/engine/clangd.cpp +++ b/src/engine/clangd.cpp @@ -19,6 +19,7 @@ import mcppls.engine.clangd.definition; import mcppls.engine.clangd.guard; import mcppls.engine.clangd.primer; import mcppls.engine.clangd.process; +import mcppls.engine.clangd.workarounds; import mcppls.engine.native.index; namespace mcppls::engine::clangd { @@ -27,22 +28,18 @@ namespace log = base::log; namespace midx = mcppls::index; EngineTraits traits_for_version(std::string_view version) { - EngineTraits traits { + // Every compensation for clangd's own defects is a registered workaround (import-hang plan §9). + return EngineTraits { .importNavigation = false, .pushesDiagnostics = true, - .hangsOnUnresolvedImports = true, - .needsModulePreparation = true, - .needsModuleHints = true, - .msvcStlNeedsNoAlignedAllocation = true, + .hangsOnUnresolvedImports = needs(UNRESOLVED_IMPORT_STAND_INS, version), + .needsModulePreparation = needs(MODULE_PREPARATION, version), + .needsModuleHints = needs(MODULE_HINTS, version), + .msvcStlNeedsNoAlignedAllocation = needs(MSVC_STL_ALIGNED_ALLOCATION, version), + .hangsOnTrailingDotModuleName = needs(TRAILING_DOT_MODULE_NAME, version), .kitStdlibVersion = std::string { version }, - .tested = false, + .tested = version == "23.1.0", }; - if (version == "23.1.0") { - traits.tested = true; - } else if (version.starts_with("23.1.")) { - traits.msvcStlNeedsNoAlignedAllocation = false; - } - return traits; } bool is_interactive(std::string_view method) { @@ -180,6 +177,9 @@ class ClangdEngine final : public Engine { Quarantine quarantine_; // path keys std::optional lastAnswerAt_; // clangd's last answer to any client request StuckWatch stuck_; + // WA-CLANGD-001: the `;` insertions in the text clangd has of each open document (client URI), for + // the documents whose text it was given rewritten; mapped back out of what it reports. + std::map, std::less<>> rewritten_; // The request a watch was last tried for (when it was sent): one try each, so a platform that // cannot read clangd's CPU does not wake the loop again and again for the same request. std::optional stuckTriedFor_; @@ -347,9 +347,21 @@ class ClangdEngine final : public Engine { { "filesAwaitingDiagnostics", awaitingDiagnostics_.size() }, { "logLinesLeftOut", linesLeftOut_ }, { "databaseDirectory", databaseDirectory_ }, + { "workarounds", workarounds_json_() }, }; } + // import-hang plan §9: the registered workarounds this clangd needs, as the report shows them. + Json workarounds_json_() const { + Json list = Json::array(); + for (const auto& workaround : workarounds()) { + if (!needs(workaround, options_.version)) continue; + list.push_back(Json { { "id", workaround.id }, { "title", workaround.title }, { "upstream", workaround.upstream }, + { "removeWhen", workaround.removeWhen } }); + } + return list; + } + void start(Host& host) override { host_ = &host; sink_ = host.event_sink(ENGINE_ID); @@ -362,6 +374,13 @@ class ClangdEngine final : public Engine { // clangd starts without a database; the first plan is written before any document reaches it. platform::fs::remove_all(base::join_path(databaseDirectory_, "compile_commands.json")); log::info("clangd {} at {}", options_.version.empty() ? "?" : options_.version, options_.executable.empty() ? "(none)" : options_.executable); + // import-hang plan §9: which of clangd's known defects this server works around for this version. + if (!options_.executable.empty()) { + const auto active = active_workarounds(options_.version); + std::string ids; + for (const auto id : active) ids += std::format("{}{}", ids.empty() ? "" : ", ", id); + log::info("workarounds for clangd {}: {}", options_.version.empty() ? "?" : options_.version, ids.empty() ? "none" : ids); + } start_process_(); } @@ -586,7 +605,7 @@ class ClangdEngine final : public Engine { if (accepting_ && !excluded_path_(document.path)) open_or_hold_(document, true); break; } - if (accepting_ && !excluded_path_(document.path) && event.message != nullptr) (void)send_(*event.message); + if (accepting_ && !excluded_path_(document.path) && event.message != nullptr) send_change_(document, *event.message); break; case DocumentChange::closed: { const bool wasExcluded { excluded_path_(document.path) || quarantined_(document.path) || held_path_(document.path) @@ -600,6 +619,7 @@ class ClangdEngine final : public Engine { awaitingSince_.erase(document.uri); diagnosed_.erase(document.uri); fileStatus_.erase(document.uri); + rewritten_.erase(document.uri); if (accepting_ && !wasExcluded && event.message != nullptr) (void)send_(*event.message); release_prime_units_if_idle_(); break; @@ -1219,9 +1239,56 @@ class ClangdEngine final : public Engine { } } + // The text clangd is given for an open document: the document's own, or, where clangd 23.1 would spin on it + // (WA-CLANGD-001), the same text with `;` after each trailing dot, remembered so what clangd says maps back. + std::string engine_text_(const std::string& uri, std::string_view text) { + if (traits_.hangsOnTrailingDotModuleName) { + if (auto sanitized = sanitize_module_names(text); sanitized.changed()) { + if (!rewritten_.contains(uri)) { + log::debug("{} is given to clangd with ';' after a module name that ends in '.' ({}, {})", host_->path_of_uri(uri), + TRAILING_DOT_MODULE_NAME, host_->root_directory()); + } + rewritten_[uri] = std::move(sanitized.insertions); + return std::move(sanitized.text); + } + } + rewritten_.erase(uri); + return std::string { text }; + } + + // A change goes to clangd as the client sent it, unless clangd's text is, or was until now, a rewrite of + // the document's: then clangd gets the whole text as engine_text_ makes it. + void send_change_(const DocumentView& document, const Json& message) { + const bool wasRewritten { rewritten_.contains(document.uri) }; + std::string text { engine_text_(document.uri, document.text) }; + if (!wasRewritten && !rewritten_.contains(document.uri)) { + (void)send_(message); + return; + } + (void)send_(lsp::make_notification("textDocument/didChange", + Json { { "textDocument", Json { { "uri", document.uri }, { "version", document.version } } }, + { "contentChanges", Json::array({ Json { { "text", std::move(text) } } }) } })); + } + + static void map_out_of_rewrite_(std::span insertions, Json& diagnostics) { + const auto map_position = [&](Json& position) { + if (!position.is_object()) return; + const auto line = lsp::int_at(position, "line"); + const auto character = lsp::int_at(position, "character"); + if (!line || !character) return; + const auto original = to_original(insertions, TextPosition { static_cast(*line), static_cast(*character) }); + position["character"] = original.character; + }; + for (auto& diagnostic : diagnostics) { + if (!diagnostic.is_object() || !diagnostic.contains("range")) continue; + map_position(diagnostic["range"]["start"]); + map_position(diagnostic["range"]["end"]); + } + } + void open_in_engine_(const DocumentView& document) { Json params { { "textDocument", Json { { "uri", document.uri }, { "languageId", document.languageId }, - { "version", document.version }, { "text", std::string { document.text } } } } }; + { "version", document.version }, { "text", engine_text_(document.uri, document.text) } } } }; if (send_(lsp::make_notification("textDocument/didOpen", std::move(params)))) { databaseRead_ = true; if (!diagnosed_.contains(document.uri)) { @@ -1399,7 +1466,9 @@ class ClangdEngine final : public Engine { } else { diagnosed_.insert(uri); const auto version = lsp::int_at(params, "version"); - host_->publish_engine_diagnostics(ENGINE_ID, uri, params.value("diagnostics", Json::array()), version); + Json diagnostics = params.value("diagnostics", Json::array()); + if (const auto rewritten = rewritten_.find(uri); rewritten != rewritten_.end()) map_out_of_rewrite_(rewritten->second, diagnostics); + host_->publish_engine_diagnostics(ENGINE_ID, uri, std::move(diagnostics), version); } host_->status_changed(); return; @@ -2131,6 +2200,11 @@ class ClangdEngine final : public Engine { auto text = platform::fs::read_file(path); if (!text) return false; const std::string uri { base::path_to_uri(path) }; + // WA-CLANGD-001: a file saved mid-edit can hold the text clangd spins on. Nobody reads positions in a + // unit opened without the editor, so the rewrite needs no mapping back. + if (traits_.hangsOnTrailingDotModuleName) { + if (auto sanitized = sanitize_module_names(*text); sanitized.changed()) *text = std::move(sanitized.text); + } Json params { { "textDocument", Json { { "uri", uri }, { "languageId", "cpp" }, { "version", 1 }, { "text", std::move(*text) } } } }; if (!send_(lsp::make_notification("textDocument/didOpen", std::move(params)))) return false; databaseRead_ = true; diff --git a/src/engine/clangd/workarounds.cpp b/src/engine/clangd/workarounds.cpp new file mode 100644 index 0000000..053b89a --- /dev/null +++ b/src/engine/clangd/workarounds.cpp @@ -0,0 +1,207 @@ +module mcppls.engine.clangd.workarounds; + +import std; +import mcppls.base.text; + +namespace mcppls::engine::clangd { + +namespace { + +constexpr std::array REGISTRY { { + { + .id = TRAILING_DOT_MODULE_NAME, + .title = "a module name ending in '.' at the end of its line spins clangd forever; clangd is given the line with ';' after the dot", + .fixedIn = "", + .upstream = "unfiled; fixed on llvm-project main, likely by 6dcfc17b1b (#187846), not in 23.1.2", + .evidence = ".agents/docs/2026-09-25-import-hang-status-highlight.md §1; conformance fixtures typing-import, workaround-canaries", + .added = "0.0.4", + .removeWhen = "the bundled clangd and the oldest clangd the server supports finish `import a.` at once", + .canary = "conformance/fixtures/workaround-canaries: clangd --check on `import hello.` does not finish", + }, + { + .id = UNRESOLVED_IMPORT_STAND_INS, + .title = "building a module unit whose import resolves to nothing can deadlock clangd; such imports get an empty stand-in unit", + .fixedIn = "", + .upstream = "unfiled", + .evidence = "robustness design C2, experiments S12 and S17; conformance fixture module-faults", + .added = "0.0.1", + .removeWhen = "clangd builds a unit with an unresolved import to a diagnostic instead of stalling", + .canary = "", + }, + { + .id = MODULE_PREPARATION, + .title = "clangd builds the modules a file needs one file at a time; the server prepares them in parallel first", + .fixedIn = "", + .upstream = "unfiled", + .evidence = "cold-start plan 4.4; conformance fixture timing", + .added = "0.0.1", + .removeWhen = "clangd builds independent modules concurrently on its own", + .canary = "", + }, + { + .id = MODULE_HINTS, + .title = "clangd scans every file of the database to find a module's unit; the database names each unit instead", + .fixedIn = "", + .upstream = "unfiled (ProjectModules.cpp, CompileCommandsProjectModules)", + .evidence = "usable plan W7; conformance fixture timing", + .added = "0.0.1", + .removeWhen = "clangd looks a module's unit up without scanning the whole database", + .canary = "", + }, + { + .id = MSVC_STL_ALIGNED_ALLOCATION, + .title = "clangd rejects the MSVC STL's aligned allocation; units using the MSVC STL turn it off", + .fixedIn = "23.1.1", + .upstream = "llvm-project#218152, fixed in 23.1.1", + .evidence = ".agents/docs/design.md §7; conformance fixtures cmake-msvc-std, mcpp-msvc", + .added = "0.0.1", + .removeWhen = "the bundled clangd is 23.1.1 or later", + .canary = "", + }, +} }; + +// "23.1.0" -> {23, 1, 0}; anything else -> nullopt. +std::optional> parse_version(std::string_view version) { + std::array parts {}; + for (std::size_t i { 0 }; i < parts.size(); ++i) { + const auto end { version.find('.') }; + const std::string_view part { version.substr(0, end) }; + const auto [ptr, error] { std::from_chars(part.data(), part.data() + part.size(), parts[i]) }; + if (error != std::errc {} || ptr != part.data() + part.size() || part.empty()) return std::nullopt; + if (i + 1 < parts.size()) { + if (end == std::string_view::npos) return std::nullopt; + version.remove_prefix(end + 1); + } else if (end != std::string_view::npos) { + return std::nullopt; + } + } + return parts; +} + +bool in_known_line(std::string_view version) { + return version.starts_with(KNOWN_LINE) && version.size() > KNOWN_LINE.size() && version[KNOWN_LINE.size()] == '.'; +} + +// A byte of a UTF-8 sequence is part of a name: C++ identifiers may be written in any script. +bool name_char(char c) { return base::is_identifier_char(c) || c == '.' || c == ':' || static_cast(c) >= 0x80; } + +bool keyword_at(std::string_view line, std::size_t at, std::string_view keyword) { + if (line.substr(at, keyword.size()) != keyword) return false; + const std::size_t after { at + keyword.size() }; + return after == line.size() || !base::is_identifier_char(line[after]); +} + +std::size_t skip_blanks(std::string_view line, std::size_t at) { + while (at < line.size() && (line[at] == ' ' || line[at] == '\t')) ++at; + return at; +} + +// Where `;` goes on `line` (after its dot), or npos. `line` has no line terminator. +std::size_t insertion_point(std::string_view line) { + std::size_t at { skip_blanks(line, 0) }; + if (keyword_at(line, at, "export")) at = skip_blanks(line, at + 6); + if (keyword_at(line, at, "import")) { + at += 6; + } else if (keyword_at(line, at, "module")) { + at += 6; + } else { + return std::string_view::npos; + } + const std::size_t nameStart { skip_blanks(line, at) }; + if (nameStart == at || nameStart >= line.size()) return std::string_view::npos; + std::size_t end { nameStart }; + while (end < line.size() && name_char(line[end])) ++end; + if (end == nameStart || line[end - 1] != '.') return std::string_view::npos; + // Nothing but blanks or a comment may follow the dot on this line. + const std::size_t rest { skip_blanks(line, end) }; + if (rest < line.size() && !line.substr(rest).starts_with("//") && !line.substr(rest).starts_with("/*")) return std::string_view::npos; + return end; +} + +} // namespace + +std::span workarounds() { return REGISTRY; } + +const Workaround* find_workaround(std::string_view id) { + const auto found { std::ranges::find(REGISTRY, id, &Workaround::id) }; + return found == REGISTRY.end() ? nullptr : &*found; +} + +bool needs(const Workaround& workaround, std::string_view version) { + if (!in_known_line(version)) return true; + if (workaround.fixedIn.empty()) return true; + const auto have { parse_version(version) }; + const auto fixed { parse_version(workaround.fixedIn) }; + if (!have || !fixed) return true; + return *have < *fixed; +} + +bool needs(std::string_view id, std::string_view version) { + const Workaround* workaround { find_workaround(id) }; + return workaround != nullptr && needs(*workaround, version); +} + +std::vector active_workarounds(std::string_view version) { + std::vector ids; + for (const auto& workaround : REGISTRY) { + if (needs(workaround, version)) ids.push_back(workaround.id); + } + return ids; +} + +Sanitized sanitize_module_names(std::string_view text) { + Sanitized result; + bool inBlockComment { false }; + int lineNumber { 0 }; + std::size_t lineStart { 0 }; + std::size_t copiedUpTo { 0 }; + while (lineStart <= text.size()) { + std::size_t lineEnd { text.find('\n', lineStart) }; + const bool last { lineEnd == std::string_view::npos }; + if (last) lineEnd = text.size(); + std::string_view line { text.substr(lineStart, lineEnd - lineStart) }; + if (line.ends_with('\r')) line.remove_suffix(1); + // A directive inside a block comment is no directive; one that opens a comment afterwards still is. + const bool startsInComment { inBlockComment }; + for (std::size_t at { 0 }; at + 1 < line.size(); ++at) { + if (!inBlockComment && line[at] == '/' && line[at + 1] == '/') break; + if (!inBlockComment && line[at] == '/' && line[at + 1] == '*') { + inBlockComment = true; + ++at; + } else if (inBlockComment && line[at] == '*' && line[at + 1] == '/') { + inBlockComment = false; + ++at; + } + } + if (!startsInComment) { + if (const std::size_t point { insertion_point(line) }; point != std::string_view::npos) { + const std::size_t offset { lineStart + point }; + result.text.append(text.substr(copiedUpTo, offset - copiedUpTo)); + result.text.push_back(';'); + copiedUpTo = offset; + result.insertions.push_back(Insertion { lineNumber, static_cast(base::utf16_length(line.substr(0, point))) }); + } + } + if (last) break; + lineStart = lineEnd + 1; + ++lineNumber; + } + if (result.insertions.empty()) { + result.text.clear(); + return result; + } + result.text.append(text.substr(copiedUpTo)); + return result; +} + +TextPosition to_original(std::span insertions, TextPosition position) { + // An insertion moves what follows it on its line one code unit to the right; the `;` itself + // maps to where it went in, the end of the name. + int shift { 0 }; + for (const auto& insertion : insertions) { + if (insertion.line == position.line && position.character > insertion.character) ++shift; + } + return TextPosition { position.line, position.character - shift }; +} + +} // namespace mcppls::engine::clangd diff --git a/src/engine/clangd/workarounds.cppm b/src/engine/clangd/workarounds.cppm new file mode 100644 index 0000000..a4b5995 --- /dev/null +++ b/src/engine/clangd/workarounds.cppm @@ -0,0 +1,68 @@ +// What the server does only because of a defect in a clangd it drives (import-hang plan §9): one +// registry, one entry per workaround, each saying which versions need it, the upstream defect, +// the evidence, when it can go and the canary that says so. The engine's traits are read from +// here, so a workaround is on exactly where its entry says, and nowhere else. Code that carries +// one out names its id (`WA-CLANGD-001`) at the call site. +export module mcppls.engine.clangd.workarounds; + +import std; + +export namespace mcppls::engine::clangd { + +// The clangd release line the payload pins and the conformance suite runs. A version of this line +// needs a workaround until the entry's `fixedIn`; a version of any other line has not been seen +// by the suite, so it gets every workaround. +inline constexpr std::string_view KNOWN_LINE { "23.1" }; + +struct Workaround { + std::string_view id; // WA-CLANGD-: grep-able, in logs and the report + std::string_view title; // what it works around, in one line + std::string_view fixedIn; // the first release of KNOWN_LINE that no longer needs it; empty: none yet + std::string_view upstream; // the upstream issue or fix, or "unfiled" + std::string_view evidence; // where it was found and measured + std::string_view added; // the mcppls version that added it + std::string_view removeWhen; // when the workaround can go + std::string_view canary; // the check that fails once the defect is gone; empty: none yet +}; + +inline constexpr std::string_view TRAILING_DOT_MODULE_NAME { "WA-CLANGD-001" }; +inline constexpr std::string_view UNRESOLVED_IMPORT_STAND_INS { "WA-CLANGD-002" }; +inline constexpr std::string_view MODULE_PREPARATION { "WA-CLANGD-003" }; +inline constexpr std::string_view MODULE_HINTS { "WA-CLANGD-004" }; +inline constexpr std::string_view MSVC_STL_ALIGNED_ALLOCATION { "WA-CLANGD-005" }; + +std::span workarounds(); +const Workaround* find_workaround(std::string_view id); +// Whether clangd `version` needs `workaround`. +bool needs(const Workaround& workaround, std::string_view version); +bool needs(std::string_view id, std::string_view version); +// The ids of every workaround clangd `version` needs, in registry order. +std::vector active_workarounds(std::string_view version); + +// WA-CLANGD-001. clangd 23.1 never finishes a file in which a module name of an `import` or +// `module` directive ends in `.` with nothing after the dot on its line but white space or a +// comment: the build spins at a full core, and every later version of the file queues behind it. +// Such a directive is always an error (P1857: the directive ends with its line), and so is the +// same line with `;` right after the dot, which clangd reports at once. `text` is what clangd is +// given instead; each insertion is where a `;` went in, in the rewritten text's coordinates. +struct Insertion { + int line { 0 }; + int character { 0 }; // UTF-16, like every LSP position +}; + +struct Sanitized { + std::string text; + std::vector insertions; + bool changed() const { return !insertions.empty(); } +}; + +Sanitized sanitize_module_names(std::string_view text); + +// A position clangd reported in the rewritten text, in the text the editor has. +struct TextPosition { + int line { 0 }; + int character { 0 }; +}; +TextPosition to_original(std::span insertions, TextPosition position); + +} // namespace mcppls::engine::clangd diff --git a/src/engine/engine.cppm b/src/engine/engine.cppm index b86aa6c..d598620 100644 --- a/src/engine/engine.cppm +++ b/src/engine/engine.cppm @@ -38,6 +38,7 @@ struct EngineTraits { bool needsModulePreparation { false }; // the server prepares modules in parallel for it bool needsModuleHints { false }; // its database names the unit of each module bool msvcStlNeedsNoAlignedAllocation { false }; // MSVC STL contexts turn aligned allocation off + bool hangsOnTrailingDotModuleName { false }; // `import a.` at the end of a line spins it; it is given `import a.;` std::string kitStdlibVersion; // the libc++ version a semantic kit must have for it (S4-4-5); empty: any bool tested { false }; // a version this server's conformance suite runs against }; diff --git a/tests/test_workarounds.cpp b/tests/test_workarounds.cpp new file mode 100644 index 0000000..2b64f40 --- /dev/null +++ b/tests/test_workarounds.cpp @@ -0,0 +1,99 @@ +import std; +import mcppls.testing; +import mcppls.engine; +import mcppls.engine.clangd; +import mcppls.engine.clangd.workarounds; + +namespace cld = mcppls::engine::clangd; + +int main() { + using namespace mcppls::testing; + + "every workaround is registered once, says why, and says when it can go"_test = [] { + std::set ids; + for (const auto& workaround : cld::workarounds()) { + expect(ids.insert(workaround.id).second) << workaround.id; + expect(workaround.id.starts_with("WA-CLANGD-")) << workaround.id; + expect(!workaround.title.empty() && !workaround.upstream.empty() && !workaround.evidence.empty()) << workaround.id; + expect(!workaround.added.empty() && !workaround.removeWhen.empty()) << workaround.id; + } + // Workarounds added from 0.0.4 on carry a canary: the check that fails once the defect is gone. + const auto* trailingDot = cld::find_workaround(cld::TRAILING_DOT_MODULE_NAME); + expect(fatal(trailingDot != nullptr)); + expect(!trailingDot->canary.empty()); + expect(cld::find_workaround("WA-CLANGD-999") == nullptr); + }; + + "a workaround applies to its line until the fix, and to every version the suite has not seen"_test = [] { + expect(cld::needs(cld::MSVC_STL_ALIGNED_ALLOCATION, "23.1.0")); + expect(!cld::needs(cld::MSVC_STL_ALIGNED_ALLOCATION, "23.1.1")); + expect(!cld::needs(cld::MSVC_STL_ALIGNED_ALLOCATION, "23.1.12")); + expect(cld::needs(cld::MSVC_STL_ALIGNED_ALLOCATION, "22.1.8")) << "another line gets every workaround"; + expect(cld::needs(cld::MSVC_STL_ALIGNED_ALLOCATION, "23.10.0")) << "23.10 is not the 23.1 line"; + expect(cld::needs(cld::MSVC_STL_ALIGNED_ALLOCATION, "")) << "an unknown version gets every workaround"; + for (const auto version : { "23.1.0", "23.1.2", "22.1.8", "24.0.0" }) expect(cld::needs(cld::TRAILING_DOT_MODULE_NAME, version)) << version; + expect(!cld::needs("WA-CLANGD-999", "23.1.0")); + const auto active = cld::active_workarounds("23.1.1"); + expect(std::ranges::find(active, cld::TRAILING_DOT_MODULE_NAME) != active.end()); + expect(std::ranges::find(active, cld::MSVC_STL_ALIGNED_ALLOCATION) == active.end()); + }; + + "the traits are the registry's"_test = [] { + const auto pinned = cld::traits_for_version("23.1.0"); + expect(pinned.tested && pinned.hangsOnTrailingDotModuleName && pinned.hangsOnUnresolvedImports); + expect(pinned.needsModulePreparation && pinned.needsModuleHints && pinned.msvcStlNeedsNoAlignedAllocation); + expect(!cld::traits_for_version("23.1.1").msvcStlNeedsNoAlignedAllocation); + expect(cld::traits_for_version("23.1.1").hangsOnTrailingDotModuleName); + }; + + "every line clangd 23.1 spins on gets a ';' right after its dot"_test = [] { + // The table of .agents/docs/2026-09-25-import-hang-status-highlight.md §1: each hangs clangd 23.1.0. + const std::vector> hazards { + { "import hello.\n", "import hello.;\n" }, + { "import hello.", "import hello.;" }, + { "import hello.\nint x;\n", "import hello.;\nint x;\n" }, + { "import a.b.\n", "import a.b.;\n" }, + { "export import hello.\n", "export import hello.;\n" }, + { "export module a.\n", "export module a.;\n" }, + { "module a.\n", "module a.;\n" }, + { "module;\nexport module m.\n", "module;\nexport module m.;\n" }, + { "import hello. \n", "import hello.; \n" }, + { "import hello.\t\n", "import hello.;\t\n" }, + { "import hello.// c\n", "import hello.;// c\n" }, + { "import hello.\n;\n", "import hello.;\n;\n" }, + { " import hello.\r\n", " import hello.;\r\n" }, + { "import hello./* c */\n", "import hello.;/* c */\n" }, + }; + for (const auto& [text, expected] : hazards) { + const auto sanitized = cld::sanitize_module_names(text); + expect(sanitized.changed()) << text; + expect(sanitized.text == expected) << text << " became " << sanitized.text; + } + }; + + "everything else is left as it is"_test = [] { + for (const std::string_view text : { + "import hello.greet;\n", "import hello.greet\n", "import hello;\n", "import hello\n", "import hello.;\n", "import hello. ;\n", + "import hello.;// c\n", "import hello:\n", "import :part;\n", "module;\n", "module :private;\n", "import ;\n", + "auto x = a.\nb;\n", "// import hello.\n", "/*\nimport hello.\n*/\n", "int import_ = 1; // import x.\n", "important.\n", + "std::string s = \"import a.\";\n", "exported module a.\n", "", "\n" }) { + const auto sanitized = cld::sanitize_module_names(text); + expect(!sanitized.changed()) << text; + expect(sanitized.text.empty()) << "nothing is copied when nothing changes: " << text; + } + }; + + "insertions are recorded in UTF-16 and map positions back"_test = [] { + const auto sanitized = cld::sanitize_module_names("import std;\nimport hello.\n\nexport module mé.\n"); + expect(fatal(sanitized.insertions.size() == 2u)); + expect(sanitized.insertions[0].line == 1 && sanitized.insertions[0].character == 13); + expect(sanitized.insertions[1].line == 3 && sanitized.insertions[1].character == 17) << "é is one UTF-16 unit"; + const std::span insertions { sanitized.insertions }; + // Before and at the insertion nothing moves; after it, one unit back; other lines never move. + expect(cld::to_original(insertions, { 1, 7 }).character == 7); + expect(cld::to_original(insertions, { 1, 13 }).character == 13); + expect(cld::to_original(insertions, { 1, 14 }).character == 13); + expect(cld::to_original(insertions, { 0, 20 }).character == 20); + expect(cld::to_original(insertions, { 3, 18 }).character == 17); + }; +} From d802a97ae8d4f6c3786f72d2dd2f6dc091e7b6d2 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Fri, 25 Sep 2026 06:21:57 +0800 Subject: [PATCH 03/26] fix(engine): a file clangd will not finish is found in seconds and clangd restarted, even while the user keeps typing The guards of 0.0.2 and 0.0.3 took a busy clangd for a compiling one: the stuck watch only acts on a clangd that uses no CPU, and a file was not set aside for 120 s after any edit to it, so a spin started by an edit, with the user still typing to fix it, was never acted on, and even once typing stopped recovery took minutes and could hand the same text back. Now a file's main-file build is given a budget from its own history: five times its last build, never under 20 s; past it, with a version or a request waiting on it, clangd will not finish it, busy or not. The file goes to mcppls's engine with the text clangd spun on remembered, and clangd is restarted at once, past the restart cap if need be, since a clangd left spinning answers nothing and holds a core; the file goes back the moment its text is any other, and the same text never does. An edit to the file itself now counts as a rebuild for 10 s, not 120 s; a change to a module it imports keeps the long patience. The file-quarantined issue names the files. `--disable-workaround` turns a registered workaround off, which is how the spin is reproduced on purpose. --- src/cli/commands.cpp | 2 + src/cli/options.cpp | 2 + src/engine/clangd.cpp | 114 ++++++++++++++++++++++++++------ src/engine/clangd.cppm | 4 +- src/engine/clangd/guard.cpp | 75 +++++++++++++++++++++ src/engine/clangd/guard.cppm | 49 ++++++++++++++ src/orchestrator/workspace.cppm | 2 + tests/test_server.cpp | 56 ++++++++++++++++ 8 files changed, 283 insertions(+), 21 deletions(-) diff --git a/src/cli/commands.cpp b/src/cli/commands.cpp index 5229bed..c7d9c92 100644 --- a/src/cli/commands.cpp +++ b/src/cli/commands.cpp @@ -184,6 +184,7 @@ std::vector daemon_arguments(const cmdline::ParsedArgs& args) { if (auto value = args.value(name)) forwarded.insert(forwarded.end(), { std::format("--{}", name), *value }); } for (const auto& pattern : args.option_or_empty("model-exclude").values) forwarded.insert(forwarded.end(), { "--model-exclude", pattern }); + for (const auto& id : args.option_or_empty("disable-workaround").values) forwarded.insert(forwarded.end(), { "--disable-workaround", id }); for (const std::string_view flag : { "untrusted", "no-discover" }) { if (args.is_flag_set(flag)) forwarded.push_back(std::format("--{}", flag)); } @@ -221,6 +222,7 @@ int run(int argc, char* argv[]) { (void)app.option("tool-environment").takes_value().global(true).help("Which environment build tools run in: auto (the login shell on POSIX) or editor"); (void)app.option("producer-timeout").takes_value().global(true).help("Seconds a build tool may take to describe the project (default 60, or 600 when online)"); (void)app.option("engine").takes_value().global(true).help("The core semantic engine: clangd (default) or none, mcppls's own module features only"); + (void)app.option("disable-workaround").takes_value().multiple().global(true).help("Turn off a registered clangd workaround (WA-CLANGD-, see mcppls report); repeatable"); // Language clients pass these by convention; this server always speaks over its standard streams. (void)app.option("stdio").global(true).help("Accepted for language clients; standard input and output are always used"); (void)app.option("clientProcessId").takes_value().global(true).help("Accepted for language clients; not used"); diff --git a/src/cli/options.cpp b/src/cli/options.cpp index e7b9206..234485f 100644 --- a/src/cli/options.cpp +++ b/src/cli/options.cpp @@ -30,6 +30,7 @@ orchestrator::EngineFactories engine_factories(const orchestrator::SessionOption clangd.payloadCorrupt = payloadCorrupt; clangd.verboseLog = options.verboseEngineLog; clangd.requestTimeout = options.requestTimeout; + clangd.disabledWorkarounds = options.disabledWorkarounds; return engine::clangd::make_engine(std::move(clangd)); }; return factories; @@ -50,6 +51,7 @@ orchestrator::SessionOptions session_options(const cmdline::ParsedArgs& args) { options.engineFromCommandLine = true; } options.engineFactories = engine_factories; + options.disabledWorkarounds = args.option_or_empty("disable-workaround").values; // This very program, for the reviews an editor asks for: named as the process started it, else found on PATH. if (const auto arguments = platform::env::arguments(); !arguments.empty()) { const std::string started { arguments.front() }; diff --git a/src/engine/clangd.cpp b/src/engine/clangd.cpp index 0e71406..74ce391 100644 --- a/src/engine/clangd.cpp +++ b/src/engine/clangd.cpp @@ -27,16 +27,17 @@ namespace mcppls::engine::clangd { namespace log = base::log; namespace midx = mcppls::index; -EngineTraits traits_for_version(std::string_view version) { +EngineTraits traits_for_version(std::string_view version, std::span disabled) { // Every compensation for clangd's own defects is a registered workaround (import-hang plan §9). + const auto on = [&](std::string_view id) { return needs(id, version) && std::ranges::find(disabled, id) == disabled.end(); }; return EngineTraits { .importNavigation = false, .pushesDiagnostics = true, - .hangsOnUnresolvedImports = needs(UNRESOLVED_IMPORT_STAND_INS, version), - .needsModulePreparation = needs(MODULE_PREPARATION, version), - .needsModuleHints = needs(MODULE_HINTS, version), - .msvcStlNeedsNoAlignedAllocation = needs(MSVC_STL_ALIGNED_ALLOCATION, version), - .hangsOnTrailingDotModuleName = needs(TRAILING_DOT_MODULE_NAME, version), + .hangsOnUnresolvedImports = on(UNRESOLVED_IMPORT_STAND_INS), + .needsModulePreparation = on(MODULE_PREPARATION), + .needsModuleHints = on(MODULE_HINTS), + .msvcStlNeedsNoAlignedAllocation = on(MSVC_STL_ALIGNED_ALLOCATION), + .hangsOnTrailingDotModuleName = on(TRAILING_DOT_MODULE_NAME), .kitStdlibVersion = std::string { version }, .tested = version == "23.1.0", }; @@ -177,6 +178,7 @@ class ClangdEngine final : public Engine { Quarantine quarantine_; // path keys std::optional lastAnswerAt_; // clangd's last answer to any client request StuckWatch stuck_; + SpinWatch spin_; // import-hang plan §4: a file clangd will not finish, busy or not // WA-CLANGD-001: the `;` insertions in the text clangd has of each open document (client URI), for // the documents whose text it was given rewritten; mapped back out of what it reports. std::map, std::less<>> rewritten_; @@ -198,6 +200,7 @@ class ClangdEngine final : public Engine { // GENERAL_PATIENCE while module preparation makes no progress. static constexpr std::chrono::seconds FAILED_MODULE_PATIENCE { 5 }; static constexpr std::chrono::seconds GENERAL_PATIENCE { 120 }; + static constexpr std::chrono::seconds SELF_EDIT_GRACE { 10 }; std::map> awaitingSince_; // client URI -> when it was handed to clangd std::map> modulesFailedAt_; // module -> when clangd said it did not compile std::optional stuckCheckAt_; @@ -220,6 +223,8 @@ class ClangdEngine final : public Engine { struct Aside { std::string structure; // what it provided and imported then bool moduleFailed { false }; + // The text clangd spun on (SpinWatch): that exact text never goes back to clangd, and any other does at once. + std::optional spunOn; }; std::map> aside_; // path key std::map> fileStatus_; // client URI -> clangd's last textDocument/clangd.fileStatus state @@ -255,7 +260,7 @@ class ClangdEngine final : public Engine { public: explicit ClangdEngine(Options options) - : options_ { std::move(options) }, traits_ { traits_for_version(options_.version) }, stuck_ { options_.stuckWatch } {} + : options_ { std::move(options) }, traits_ { traits_for_version(options_.version, options_.disabledWorkarounds) }, stuck_ { options_.stuckWatch } {} std::string_view id() const override { return ENGINE_ID; } std::span methods() const override { return methods_; } @@ -356,8 +361,9 @@ class ClangdEngine final : public Engine { Json list = Json::array(); for (const auto& workaround : workarounds()) { if (!needs(workaround, options_.version)) continue; + const bool off { std::ranges::find(options_.disabledWorkarounds, workaround.id) != options_.disabledWorkarounds.end() }; list.push_back(Json { { "id", workaround.id }, { "title", workaround.title }, { "upstream", workaround.upstream }, - { "removeWhen", workaround.removeWhen } }); + { "removeWhen", workaround.removeWhen }, { "turnedOff", off } }); } return list; } @@ -378,7 +384,13 @@ class ClangdEngine final : public Engine { if (!options_.executable.empty()) { const auto active = active_workarounds(options_.version); std::string ids; - for (const auto id : active) ids += std::format("{}{}", ids.empty() ? "" : ", ", id); + for (const auto id : active) { + const bool off { std::ranges::find(options_.disabledWorkarounds, id) != options_.disabledWorkarounds.end() }; + ids += std::format("{}{}{}", ids.empty() ? "" : ", ", id, off ? " (turned off)" : ""); + } + for (const auto& id : options_.disabledWorkarounds) { + if (find_workaround(id) == nullptr) log::warning("--disable-workaround {}: no such workaround", id); + } log::info("workarounds for clangd {}: {}", options_.version.empty() ? "?" : options_.version, ids.empty() ? "none" : ids); } start_process_(); @@ -598,7 +610,11 @@ class ClangdEngine final : public Engine { if (quarantined_(document.path)) { const std::string key { base::path_key(document.path) }; // What clangd stopped on is still there: the file stays aside until its time is up or what it imports changes. - if (const auto aside = aside_.find(key); aside != aside_.end() && aside->second.structure == structure_of_text_(document.text)) break; + // A file clangd spun on goes back as soon as its text is any other than the one it spun on. + if (const auto aside = aside_.find(key); aside != aside_.end()) { + if (aside->second.spunOn ? *aside->second.spunOn == text_hash_(document.text) + : aside->second.structure == structure_of_text_(document.text)) break; + } quarantine_.release(key); aside_.erase(key); update_quarantine_issue_(); @@ -620,6 +636,7 @@ class ClangdEngine final : public Engine { diagnosed_.erase(document.uri); fileStatus_.erase(document.uri); rewritten_.erase(document.uri); + spin_.forget(document.uri); if (accepting_ && !wasExcluded && event.message != nullptr) (void)send_(*event.message); release_prime_units_if_idle_(); break; @@ -799,6 +816,7 @@ class ClangdEngine final : public Engine { for (const auto& [module, at] : primeDeadlines_) consider(at); consider(restartAt_); consider(stuckCheckAt_); + if (accepting_) consider(spin_.next_due()); // only acted on while accepting (handle_spins_) // While a reading of clangd's CPU is on its way, its event is what wakes the loop. if (!cpuReadInFlight_) { consider(stuck_.due()); @@ -827,6 +845,7 @@ class ClangdEngine final : public Engine { const auto now = Clock::now(); // Before the requests that expire now are answered: they are part of what clangd left unanswered. watch_for_stuck_(now); + if (accepting_) handle_spins_(now); std::vector expired; for (const auto& [id, request] : pending_) { if (request.deadline <= now) expired.push_back(id); @@ -890,6 +909,17 @@ class ClangdEngine final : public Engine { request_restart_("clangd did not answer initialize"); } for (const auto& key : quarantine_.due(now)) { + // The text clangd spun on is never given back to it, however long it has been aside. + if (const auto aside = aside_.find(key); aside != aside_.end() && aside->second.spunOn) { + const auto documents = host_->documents(); + const bool unchanged { std::ranges::any_of(documents, [&](const DocumentView& document) { + return !document.path.empty() && base::path_key(document.path) == key && text_hash_(document.text) == *aside->second.spunOn; + }) }; + if (unchanged) { + quarantine_.put(key, now); + continue; + } + } aside_.erase(key); for (const auto& document : host_->documents()) { if (document.path.empty() || base::path_key(document.path) != key) continue; @@ -1100,6 +1130,7 @@ class ClangdEngine final : public Engine { if (restartHistory_.size() > 20) restartHistory_.pop_front(); host_->record_event("engine-restart", Json { { "reason", std::string { reason } } }); restartAt_.reset(); + spin_.restarted(); // Requests to the old process are answered by the other engines. auto old = std::move(pending_); pending_.clear(); @@ -1229,6 +1260,7 @@ class ClangdEngine final : public Engine { const auto now = Clock::now(); pending_[engineId] = PendingRequest { Purpose::client, id, method, uri != nullptr && uri->is_string() ? uri->get() : std::string {}, std::min(now + own_timeout_(method), limit), generation_, limit, std::move(reply), now }; + if (uri != nullptr && uri->is_string()) spin_.asked(uri->get(), now); Json forwarded = message; forwarded["id"] = engineId; if (!send_(forwarded)) { @@ -1259,6 +1291,7 @@ class ClangdEngine final : public Engine { // A change goes to clangd as the client sent it, unless clangd's text is, or was until now, a rewrite of // the document's: then clangd gets the whole text as engine_text_ makes it. void send_change_(const DocumentView& document, const Json& message) { + spin_.sent(document.uri, text_hash_(document.text), Clock::now()); const bool wasRewritten { rewritten_.contains(document.uri) }; std::string text { engine_text_(document.uri, document.text) }; if (!wasRewritten && !rewritten_.contains(document.uri)) { @@ -1286,7 +1319,10 @@ class ClangdEngine final : public Engine { } } + static std::size_t text_hash_(std::string_view text) { return std::hash {}(text); } + void open_in_engine_(const DocumentView& document) { + spin_.sent(document.uri, text_hash_(document.text), Clock::now()); Json params { { "textDocument", Json { { "uri", document.uri }, { "languageId", document.languageId }, { "version", document.version }, { "text", engine_text_(document.uri, document.text) } } } }; if (send_(lsp::make_notification("textDocument/didOpen", std::move(params)))) { @@ -1423,6 +1459,7 @@ class ClangdEngine final : public Engine { const std::string uri { host_->client_uri(params->value("uri", std::string {})) }; if (host_->has_document(uri)) { fileStatus_[uri] = params->value("state", std::string {}); + spin_.state(uri, fileStatus_[uri], Clock::now()); } else if (const std::string path { host_->path_of_uri(uri) }; !path.empty()) { if (const auto unit = background_.find(base::path_key(path)); unit != background_.end()) unit->second.state = params->value("state", std::string {}); } @@ -1652,14 +1689,16 @@ class ClangdEngine final : public Engine { // Whether the file itself, or a source of any module it imports (transitively), changed within // GENERAL_PATIENCE: what clangd is busy with is then this change, not this file being stuck. bool changed_recently_(std::string_view path, Clock::time_point now) const { - const auto touched = [&](std::string_view file) { + const auto touched = [&](std::string_view file, Clock::duration within) { const auto at = touchedAt_.find(base::path_key(file)); - return at != touchedAt_.end() && now - at->second < GENERAL_PATIENCE; + return at != touchedAt_.end() && now - at->second < within; }; - if (touched(path)) return true; + // import-hang plan §4: an edit to the file itself rebuilds only the file, on a preamble it already has, which + // takes milliseconds to seconds. Only a change to a module it imports makes clangd rebuild modules first. + if (touched(path, SELF_EDIT_GRACE)) return true; for (const auto& module : host_->imports_of(path)) { for (const auto& source : closure_sources_(module)) { - if (touched(source)) return true; + if (touched(source, GENERAL_PATIENCE)) return true; } } return false; @@ -1866,14 +1905,34 @@ class ClangdEngine final : public Engine { bool quarantined_(std::string_view path) const { return !path.empty() && quarantine_.contains(base::path_key(path)); } + // import-hang plan §4: clangd has been building a file far longer than it ever took while the editor has moved + // on. Whatever the cause (WA-CLANGD-001 was one, found in the field), the build will not end, so the file goes to + // mcppls's engine with the text clangd spun on remembered, and clangd is restarted without it. + void handle_spins_(Clock::time_point now) { + for (const auto& spin : spin_.check(now)) { + std::string path; + for (const auto& document : host_->documents()) { + if (document.uri == spin.uri) path = document.path; + } + if (path.empty() || quarantined_(path)) continue; + const auto seconds = [](std::chrono::milliseconds duration) { return std::chrono::duration(duration).count(); }; + log::warning("clangd ({}) has built {} for {:.0f} s, past its {:.0f} s budget, while newer versions waited: it will not finish it", + host_->root_directory(), base::file_name(path), seconds(spin.building), seconds(spin.budget)); + host_->record_event("engine-spin", Json { { "file", path }, { "buildingSeconds", seconds(spin.building) }, { "budgetSeconds", seconds(spin.budget) } }); + set_aside_(path, "clangd would not finish building it", Reclaim::now, false, spin.textHash); + } + } + // Whether a restart gets back what clangd spends on a file it is no longer given. - enum class Reclaim { no, if_busy }; + // `now`: what clangd spends on it is never coming back (SpinWatch): restart at once, past the gate and the cap. + enum class Reclaim { no, if_busy, now }; - void set_aside_(const std::string& path, std::string_view why, Reclaim reclaim, bool moduleFailed = false) { + void set_aside_(const std::string& path, std::string_view why, Reclaim reclaim, bool moduleFailed = false, + std::optional spunOn = std::nullopt) { const std::string key { base::path_key(path) }; if (aside_.contains(key) && quarantine_.contains(key)) return; // set aside already if (!quarantine_.contains(key)) quarantine_.put(key, Clock::now()); - Aside aside { {}, moduleFailed }; + Aside aside { {}, moduleFailed, spunOn }; std::optional state; for (const auto& document : host_->documents()) { if (document.path.empty() || base::path_key(document.path) != key) continue; @@ -1890,6 +1949,17 @@ class ClangdEngine final : public Engine { update_quarantine_issue_(); // clangd does not stop building a file it is no longer given: a build that never ends (the spin in experiment S17) keeps a // core and one of clangd's workers for as long as clangd runs. A fresh clangd, without the file, gets both back. + if (reclaim == Reclaim::now && accepting_) { + // Not deferred, not spaced out, not capped: a clangd left spinning answers nothing for this file and holds a core, + // and the file it spun on stays with mcppls's engine, so the new clangd cannot be sent the same way. + if (restartGate_.at_cap(Clock::now())) { + log::warning("restarting clangd ({}) past the restart cap: it cannot be left spinning on {}, which stays with mcppls's engine", + host_->root_directory(), base::file_name(path)); + host_->record_event("engine-restart-past-cap", Json { { "file", path } }); + } + restart_(std::format("clangd would not finish {}", base::file_name(path))); + return; + } if (reclaim == Reclaim::if_busy && accepting_ && (!state || engine_working(*state))) { // Right after a source changed, clangd being busy is clangd rebuilding what the change // touched -- a module this file imports, one that may be about to fail and be contained @@ -2032,10 +2102,14 @@ class ClangdEngine final : public Engine { void update_quarantine_issue_() { std::erase_if(issues_, [](const Issue& issue) { return issue.code == "file-quarantined"; }); - if (const std::size_t count { quarantine_.size() }; count > 0) { + if (const auto members = quarantine_.members(); !members.empty()) { + // import-hang plan §6: the files by name, and what they still get. + std::string names; + for (std::size_t i { 0 }; i < members.size() && i < 3; ++i) names += std::format("{}{}", i == 0 ? "" : ", ", base::file_name(members[i])); + if (members.size() > 3) names += std::format(" and {} more", members.size() - 3); issues_.push_back(Issue { "file-quarantined", - std::format("clangd stopped answering for {} file{}; mcppls's engine answers for {} until {} changes", count, count == 1 ? "" : "s", - count == 1 ? "it" : "them", count == 1 ? "it" : "they"), "mcppls.restartServer" }); + std::format("clangd stopped responding on {}; module-level features only for {} until {} changes", names, + members.size() == 1 ? "it" : "them", members.size() == 1 ? "it" : "they"), "mcppls.restartServer" }); } host_->status_changed(); } diff --git a/src/engine/clangd.cppm b/src/engine/clangd.cppm index 35fe321..9b4c22e 100644 --- a/src/engine/clangd.cppm +++ b/src/engine/clangd.cppm @@ -16,7 +16,8 @@ inline constexpr std::string_view ENGINE_ID { "clangd" }; // The traits table (design 5.4), keyed by clangd version. 23.1.0 is the pinned payload; later 23.1 // releases no longer need aligned allocation turned off (llvm-project#218152, fixed in 23.1.1) but // have not run through the conformance suite; any other version gets every compensation. -EngineTraits traits_for_version(std::string_view version); +// `disabled`: registered workarounds (WA-CLANGD-) turned off whatever the version. +EngineTraits traits_for_version(std::string_view version, std::span disabled = {}); struct Options { std::string executable; // empty, or a file that does not exist: the engine is unavailable @@ -30,6 +31,7 @@ struct Options { std::chrono::milliseconds stuckAfter { std::chrono::seconds { 3 } }; std::chrono::milliseconds stuckWatch { std::chrono::seconds { 5 } }; std::vector extraArguments; + std::vector disabledWorkarounds; // registered workarounds turned off (import-hang plan §9) std::function()> processFactory; // empty: a real clangd process }; diff --git a/src/engine/clangd/guard.cpp b/src/engine/clangd/guard.cpp index e991184..71fe364 100644 --- a/src/engine/clangd/guard.cpp +++ b/src/engine/clangd/guard.cpp @@ -128,6 +128,81 @@ std::vector Quarantine::members() const { return files; } +void SpinWatch::state(std::string_view uri, std::string_view state, GuardClock::time_point now) { + auto& file = files_[std::string { uri }]; + const bool building { state.find("parsing main file") != std::string_view::npos }; + if (building && !file.buildingSince) { + file.buildingSince = now; + file.buildingHash = file.lastHash; + file.buildingSentAt = file.lastSentAt; + file.reported = false; + } else if (!building && file.buildingSince) { + file.lastBuild = now - *file.buildingSince; + file.buildingSince.reset(); + } +} + +void SpinWatch::sent(std::string_view uri, std::size_t textHash, GuardClock::time_point now) { + auto& file = files_[std::string { uri }]; + file.lastDemand = now; + file.lastSentAt = now; + file.lastHash = textHash; +} + +void SpinWatch::asked(std::string_view uri, GuardClock::time_point now) { + const auto it = files_.find(uri); + if (it != files_.end()) it->second.lastDemand = now; +} + +void SpinWatch::forget(std::string_view uri) { + if (const auto it = files_.find(uri); it != files_.end()) files_.erase(it); +} + +void SpinWatch::restarted() { + for (auto& [uri, file] : files_) { + file.buildingSince.reset(); + file.reported = false; + } +} + +bool SpinWatch::waited_on_(const File& file) { + // A request asked, or a version sent, after the version being built was sent waits on this build (a + // request is often sent before clangd says it started building). + if (!file.buildingSince || !file.lastDemand) return false; + return file.buildingSentAt ? *file.lastDemand > *file.buildingSentAt : *file.lastDemand > *file.buildingSince; +} + +std::optional SpinWatch::due_(const File& file) { + if (!file.buildingSince || !file.lastBuild || file.reported) return std::nullopt; + const GuardClock::duration budget { std::max(MIN_BUDGET, *file.lastBuild * HISTORY_FACTOR) }; + return *file.buildingSince + budget; +} + +std::vector SpinWatch::check(GuardClock::time_point now) { + std::vector spins; + for (auto& [uri, file] : files_) { + const auto due = due_(file); + // Past its budget, and the editor has asked for something since the version being built: that waits behind it. + if (!due || now < *due || !waited_on_(file)) continue; + file.reported = true; + spins.push_back(Spin { uri, std::chrono::duration_cast(now - *file.buildingSince), + std::chrono::duration_cast(*due - *file.buildingSince), file.buildingHash }); + } + return spins; +} + +std::optional SpinWatch::next_due() const { + // Only builds something waits behind: a deadline check() would not act on would come back at + // once, forever. A version sent later is an event, and the timers run after every event. + std::optional earliest; + for (const auto& [uri, file] : files_) { + if (!waited_on_(file)) continue; + const auto due = due_(file); + if (due && (!earliest || *due < *earliest)) earliest = due; + } + return earliest; +} + void StuckWatch::suspect(GuardClock::time_point now, std::optional cpuSeconds) { if (since_ || !cpuSeconds) return; since_.emplace(now, *cpuSeconds); diff --git a/src/engine/clangd/guard.cppm b/src/engine/clangd/guard.cppm index 9138fd3..7a81973 100644 --- a/src/engine/clangd/guard.cppm +++ b/src/engine/clangd/guard.cppm @@ -77,6 +77,55 @@ private: std::deque> unanswered_; // (timed out, sent, uri) }; +// A file clangd will not finish (import-hang plan §4): its main-file build ("parsing main file" in +// clangd's file status) has lasted past its budget while something waits on it: a newer version of +// the file, or a request about it. A +// main-file build on a built preamble takes milliseconds to seconds, so the budget is the file's own +// history, HISTORY_FACTOR times its last build, and never less than MIN_BUDGET: a file that really +// takes long takes long every time, and is not called stuck. A file with no finished build yet has +// no history and is left to the other guards (its first build may be preparing modules). Busy or +// idle does not matter: either way the build is not going to end. +class SpinWatch { +public: + static constexpr std::chrono::seconds MIN_BUDGET { 20 }; + static constexpr int HISTORY_FACTOR { 5 }; + + struct Spin { + std::string uri; + std::chrono::milliseconds building; // how long the build has run + std::chrono::milliseconds budget; + std::size_t textHash { 0 }; // the text being built when it started + }; + + // clangd's file status for `uri`. + void state(std::string_view uri, std::string_view state, GuardClock::time_point now); + // A version of `uri` whose text hashes to `textHash` was given to clangd. + void sent(std::string_view uri, std::size_t textHash, GuardClock::time_point now); + // A request about `uri` was sent to clangd: it waits on the file's build too. + void asked(std::string_view uri, GuardClock::time_point now); + void forget(std::string_view uri); + // A new clangd builds nothing yet; what earlier builds took stays. + void restarted(); + // The files found spinning by `now`, each reported once per build. + std::vector check(GuardClock::time_point now); + std::optional next_due() const; + +private: + struct File { + std::optional buildingSince; + std::size_t buildingHash { 0 }; + std::optional buildingSentAt; // when the version being built was sent + std::optional lastSentAt; + std::optional lastBuild; + std::optional lastDemand; // the latest version sent, or request asked + std::size_t lastHash { 0 }; + bool reported { false }; + }; + static std::optional due_(const File& file); + static bool waited_on_(const File& file); + std::map> files_; +}; + // clangd answering nothing while it uses next to no CPU is stuck, not slow: whatever it waits for // is not coming. Seen on slow CI runners after a module's source changed twice within a second: its // build never finished, clangd answered no request for minutes, and used 2 s of CPU in 105 s. A long diff --git a/src/orchestrator/workspace.cppm b/src/orchestrator/workspace.cppm index b20538d..6983481 100644 --- a/src/orchestrator/workspace.cppm +++ b/src/orchestrator/workspace.cppm @@ -48,6 +48,8 @@ struct SessionOptions { std::chrono::seconds producerTimeout { 0 }; bool verboseEngineLog { false }; std::chrono::milliseconds requestTimeout { std::chrono::seconds { 60 } }; + // Registered workarounds turned off (import-hang plan §9): to see whether one is still needed. + std::vector disabledWorkarounds; // This program, to run `mcppls review` for an editor's review command (overall design 7.7). std::string serverExecutable; // Given by the composition root; a test can substitute engines that start no process. diff --git a/tests/test_server.cpp b/tests/test_server.cpp index 934beff..8291dee 100644 --- a/tests/test_server.cpp +++ b/tests/test_server.cpp @@ -697,6 +697,62 @@ int main() { fs::remove_all(root); }; + "a file whose build runs far past its own history while the editor waits is spinning"_test = [] { + // import-hang plan §4: the budget is five times the file's last build, never under 20 s. + using namespace std::chrono_literals; + const auto t0 = cld::GuardClock::now(); + const std::string uri { "file:///p/src/main.cpp" }; + cld::SpinWatch watch; + // A first build with no history is never judged: it may be preparing modules. + watch.sent(uri, 1, t0); + watch.state(uri, "parsing includes, parsing main file", t0); + watch.asked(uri, t0 + 1s); + expect(watch.check(t0 + 10min).empty()) << "no history, no verdict"; + expect(!watch.next_due().has_value()); + watch.state(uri, "idle", t0 + 10min + 50ms); // it did finish; history is 10 min + 50 ms + + cld::SpinWatch quick; + quick.sent(uri, 1, t0); + quick.state(uri, "parsing includes, parsing main file", t0); + quick.state(uri, "idle", t0 + 10ms); // history: 10 ms, so the budget is the 20 s floor + quick.sent(uri, 2, t0 + 1s); // `import hello.` + quick.asked(uri, t0 + 1s + 1ms); // a request sent before clangd says it started + quick.state(uri, "parsing includes, parsing main file", t0 + 1s + 5ms); + expect(quick.next_due() == t0 + 1s + 5ms + 20s); + expect(quick.check(t0 + 20s).empty()) << "within its budget"; + const auto spins = quick.check(t0 + 1s + 5ms + 20s); + expect(fatal(spins.size() == 1u)); + expect(spins[0].uri == uri && spins[0].textHash == 2u) << "the text being built is the one remembered"; + expect(spins[0].budget == 20s); + expect(quick.check(t0 + 1h).empty()) << "reported once per build"; + expect(!quick.next_due().has_value()); + + // Nothing waiting on the build: no verdict and no deadline, so the event loop is not woken for it. + cld::SpinWatch idle; + idle.sent(uri, 1, t0); + idle.state(uri, "parsing main file", t0); + idle.state(uri, "idle", t0 + 10ms); + idle.state(uri, "parsing main file", t0 + 1s); + expect(idle.check(t0 + 1h).empty() && !idle.next_due().has_value()); + + // A file that always takes long keeps a budget of five times what it takes. + cld::SpinWatch heavy; + heavy.sent(uri, 1, t0); + heavy.state(uri, "parsing main file", t0); + heavy.state(uri, "idle", t0 + 15s); + heavy.sent(uri, 2, t0 + 20s); + heavy.state(uri, "parsing main file", t0 + 20s); + heavy.sent(uri, 3, t0 + 21s); + expect(heavy.check(t0 + 20s + 74s).empty()) << "75 s is its budget"; + expect(heavy.check(t0 + 20s + 75s).size() == 1u); + + // A restart builds nothing yet; forgetting a file drops it. + quick.restarted(); + expect(!quick.next_due().has_value()); + heavy.forget(uri); + expect(heavy.check(t0 + 10h).empty()); + }; + "a clangd that answers nothing and uses no CPU is stuck, and a busy one is not"_test = [] { using namespace std::chrono_literals; const auto t0 = cld::GuardClock::now(); From a6b46cba55f30c8d58f5cd77bf02989dd4ffbfa7 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Fri, 25 Sep 2026 06:28:13 +0800 Subject: [PATCH 04/26] fix(plan): an import still being typed gets no stand-in, and a name no module can have is never planned With autosave, each half-typed import reached the plan: `import hello` got an empty stand-in module and a rewritten engine database, and a build tool's scan of `import hello.` even got one for the module `hello.`. A name nothing provides, imported by a file the editor changed within the last five seconds, now waits for its stand-in until the file is quiet, and the workspace plans again then; a unit that provides a module still gets its stand-in at once, since building it with an import it cannot resolve is what stalls clangd. Names that are not dotted identifiers with at most one partition are dropped from what a unit imports. Plan issues now say whose problem they are: an unresolved import and a module that does not build are the code's, a missing SDK or toolchain the environment's. --- src/normalize/plan.cpp | 66 +++++++++++++++++++++++++--------- src/normalize/plan.cppm | 9 +++++ src/orchestrator/workspace.cpp | 20 +++++++++++ src/project/scan.cpp | 22 ++++++++++++ src/project/scan.cppm | 3 ++ tests/test_normalize.cpp | 59 ++++++++++++++++++++++++++++++ tests/test_scan.cpp | 5 +++ 7 files changed, 167 insertions(+), 17 deletions(-) diff --git a/src/normalize/plan.cpp b/src/normalize/plan.cpp index 92972a1..634e9fc 100644 --- a/src/normalize/plan.cpp +++ b/src/normalize/plan.cpp @@ -4,6 +4,7 @@ import std; import nlohmann.json; import mcppls.os; import mcppls.base.error; +import mcppls.base.log; import mcppls.base.path; import mcppls.base.text; import mcppls.base.version; @@ -23,6 +24,16 @@ namespace mcppls::normalize { namespace { +// A build tool's scan of a file saved mid-edit can name `hello.` as an import (import-hang plan §5): no module +// has such a name, and planning a stand-in for it only churns the engine database. +void drop_invalid_module_names(std::vector& names, std::string_view source) { + std::erase_if(names, [&](const std::string& name) { + if (project::is_module_name(name)) return false; + base::log::debug("ignoring import '{}' of {}: not a module name", name, source); + return true; + }); +} + struct Candidate { const spec::Set* set { nullptr }; const spec::TranslationUnit* unit { nullptr }; @@ -188,6 +199,7 @@ EnginePlan plan_engine(const PlanInput& input) { } candidate.required = unit.requiredModules; if (candidate.required.empty()) candidate.required = project::required_names(scan()); + drop_invalid_module_names(candidate.required, candidate.source); if (!candidate.provided.empty()) { candidate.module = candidate.provided.substr(0, candidate.provided.find(':')); } else if (candidate.role == spec::Role::module_implementation && scan().declaration) { @@ -230,8 +242,10 @@ EnginePlan plan_engine(const PlanInput& input) { } candidate.driver = candidate.c ? clangCDriver : clangDriver; } else { - plan.issues.push_back(PlanIssue { "toolchain-not-found", - std::format("no usable compiler or semantic kit for {}", base::file_name(candidate.source)), candidate.source, {} }); + PlanIssue issue { "toolchain-not-found", std::format("no usable compiler or semantic kit for {}", base::file_name(candidate.source)), + candidate.source, {} }; + issue.category = "environment"; + plan.issues.push_back(std::move(issue)); continue; } candidates.push_back(std::move(candidate)); @@ -272,6 +286,7 @@ EnginePlan plan_engine(const PlanInput& input) { candidate.role = project::role_of(scanned); candidate.provided = project::provided_name(scanned); candidate.required = project::required_names(scanned); + drop_invalid_module_names(candidate.required, candidate.source); candidate.module = scanned.declaration ? scanned.declaration->module : std::string {}; candidate.arguments = without_module_mode(std::move(candidate.arguments)); if (spec::is_importable(candidate.role)) { @@ -321,6 +336,9 @@ EnginePlan plan_engine(const PlanInput& input) { // nothing usable provides gets an empty unit (step 5b), every import resolves and no unit leaves; // without one, the providers that cannot be built leave and every other unit stays. const bool standIns { !input.stubDirectory.empty() }; + const auto editing = [&](std::string_view source) { + return std::ranges::any_of(input.editingSources, [&](const std::string& path) { return base::same_path(path, source); }); + }; std::set> stubbed; // modules that get a stand-in std::set> unusableProviders; // modules whose planned providers clangd cannot find std::vector excluded(candidates.size(), false); @@ -343,22 +361,30 @@ EnginePlan plan_engine(const PlanInput& input) { if (sdkBlocksStd && candidates[i].usesKit && is_std_module(name)) { excluded[i] = true; if (reported.insert("sdk-missing\n" + name).second) { - plan.issues.push_back(PlanIssue { "sdk-missing", - "the macOS SDK was not found; files that import the standard library cannot be built", candidates[i].source, name }); + PlanIssue issue { "sdk-missing", "the macOS SDK was not found; files that import the standard library cannot be built", candidates[i].source, name }; + issue.category = "environment"; + plan.issues.push_back(std::move(issue)); } continue; } const bool provider { spec::is_importable(candidates[i].role) }; if (const auto failed = input.unresolvedModules.find(name); failed != input.unresolvedModules.end()) { - if (standIns) { + // import-hang plan §5: a name nothing provides, imported by a file being edited, is most likely still being + // typed: no stand-in until the file is quiet. A unit that provides a module gets one at once, since building + // it with an import it cannot resolve is what stalls clangd. + const bool deferred { standIns && !provider && !providers.contains(name) && editing(candidates[i].source) }; + if (deferred) { + plan.standInsDeferred = true; + } else if (standIns) { stubbed.insert(name); unusableProviders.insert(name); } else if (provider) { excluded[i] = true; } if (reported.insert(candidates[i].source + "\n" + name).second) { - plan.issues.push_back(PlanIssue { "module-build-failed", std::format("module {} could not be built: {}", name, failed->second), - candidates[i].source, name }); + PlanIssue issue { "module-build-failed", std::format("module {} could not be built: {}", name, failed->second), candidates[i].source, name }; + issue.category = "code"; + plan.issues.push_back(std::move(issue)); } continue; } @@ -370,21 +396,27 @@ EnginePlan plan_engine(const PlanInput& input) { resolved = resolution.from == spec::ResolvedFrom::module_metadata; } if (resolved) continue; - // A provider with an import that cannot resolve cannot be built, and building it is what deadlocks. - if (standIns) stubbed.insert(name); - else if (input.excludeUnresolvedImports && provider) excluded[i] = true; + // A provider with an import that cannot resolve cannot be built, and building it is what deadlocks. A file + // being edited waits for its stand-in until it is quiet (import-hang plan §5). + const bool deferred { standIns && !provider && editing(candidates[i].source) }; + if (deferred) plan.standInsDeferred = true; + const bool standIn { standIns && !deferred }; + if (standIn) stubbed.insert(name); + else if (!standIns && input.excludeUnresolvedImports && provider) excluded[i] = true; if (reported.insert(candidates[i].source + "\n" + name).second) { // real-project plan RP2.3: a stand-in is the last resort, tried only after the project layer // already looked for the module's real generated source (mcppls.project.generated) // and did not find it; the issue says so, since "cannot be resolved" alone reads like // a typo in the import rather than a dependency that was never built. - plan.issues.push_back(PlanIssue { "unresolved-module", - standIns ? std::format("module {} cannot be resolved; an empty stand-in is used so its importers still build -- " - "if it is generated by a dependency's build, upgrade mcpp to {} or newer, or build the " - "project once so the real file exists", - name, base::MINIMUM_MCPP_VERSION) - : std::format("module {} cannot be resolved", name), - candidates[i].source, name }); + PlanIssue issue { "unresolved-module", + standIn ? std::format("module {} cannot be resolved; an empty stand-in is used so its importers still build -- " + "if it is generated by a dependency's build, upgrade mcpp to {} or newer, or build the " + "project once so the real file exists", + name, base::MINIMUM_MCPP_VERSION) + : std::format("module {} cannot be resolved: nothing the project builds provides it", name), + candidates[i].source, name }; + issue.category = "code"; + plan.issues.push_back(std::move(issue)); } } } diff --git a/src/normalize/plan.cppm b/src/normalize/plan.cppm index d2382c8..ab8f5e5 100644 --- a/src/normalize/plan.cppm +++ b/src/normalize/plan.cppm @@ -33,6 +33,10 @@ struct PlanIssue { std::string message; std::string file; std::string module; + // Whose problem it is (S3 status issue category, import-hang plan §6): "code" when the fix is in the + // file's own source (an import of a module nothing provides, a module that does not compile), else + // "project" or "environment". + std::string category { "project" }; }; // A module the engine database provides, for scheduling its build (usable plan W7). @@ -50,6 +54,7 @@ struct EnginePlan { std::vector> stubSources; // stand-in file -> its content std::vector stubModules; std::vector openSources; // open files no set describes, planned with the nearest unit's arguments + bool standInsDeferred { false }; // a stand-in was held back for a file being edited: plan again once it is not std::vector issues; std::vector excludedFiles; // providers left out because they cannot be built std::string contextSet; // empty: every set @@ -79,6 +84,10 @@ struct PlanInput { // Files the editor has open that no set describes: they join the database with the arguments of the nearest C++ // unit, so their imports resolve or get stand-ins instead of clangd guessing (robustness design C2). std::vector openSources; + // Open files the editor is changing right now (import-hang plan §5): an import of theirs that nothing provides is + // most likely still being typed, so it gets no stand-in yet, unless the file provides a module itself (building + // such a unit with an unresolved import is what stalls clangd). + std::vector editingSources; // Engine decisions (overall design 5.4), set by the core engine's configure_plan. Providers whose // imports cannot resolve, and providers importing them, stay out of the database: clangd 23.1 // deadlocks building them (robustness design, experiments S2, S6). Other units always stay. diff --git a/src/orchestrator/workspace.cpp b/src/orchestrator/workspace.cpp index 86097e4..0a8f44b 100644 --- a/src/orchestrator/workspace.cpp +++ b/src/orchestrator/workspace.cpp @@ -231,6 +231,10 @@ struct Workspace::Impl final : engine::Host { // Timers. std::optional reloadAt; std::optional replanAt; + // import-hang plan §5: when each open file (path key) was last changed in the editor. An import that nothing + // provides in a file changed within EDITING_WINDOW is most likely still being typed. + std::map> editedAt; + static constexpr std::chrono::seconds EDITING_WINDOW { 5 }; std::optional loadGiveUpAt; // the producer has not answered: take what there is (design 4.1) std::optional lastResortAt; // nothing at all came: serve without a database rather than nothing std::optional sdkCheckAt; @@ -1033,8 +1037,22 @@ struct Workspace::Impl final : engine::Host { if (!document->path.empty() && project::is_cxx_source_name(document->path) && base::is_within(document->path, root)) openSources.insert(document->path); } input.openSources.assign(openSources.begin(), openSources.end()); + const auto now = Clock::now(); + std::optional lastEdit; + for (const Document* document : documents_.all()) { + if (document->path.empty()) continue; + const auto edited = editedAt.find(base::path_key(document->path)); + if (edited == editedAt.end() || now - edited->second >= EDITING_WINDOW) continue; + input.editingSources.push_back(document->path); + if (!lastEdit || edited->second > *lastEdit) lastEdit = edited->second; + } if (coreEngine != nullptr) coreEngine->configure_plan(input); normalize::EnginePlan newPlan { normalize::plan_engine(input) }; + // A stand-in held back for a file being edited is planned once the file has been quiet for EDITING_WINDOW. + if (newPlan.standInsDeferred && lastEdit) { + const auto quiet = *lastEdit + EDITING_WINDOW + std::chrono::milliseconds { 100 }; + if (!replanAt || quiet < *replanAt) replanAt = quiet; + } plan = std::move(newPlan); openedOutsideModel = { plan.openSources.begin(), plan.openSources.end() }; plannedFiles.clear(); @@ -1474,6 +1492,7 @@ void Workspace::did_change(const Json& message, const Json& params) { ++impl_->snapshotGeneration; const Document* document { impl_->documents_.find(uri) }; if (!document->path.empty()) { + impl_->editedAt[base::path_key(document->path)] = Clock::now(); impl_->index.update(document->path, document->text); impl_->note_structure_change(document->path); } @@ -1487,6 +1506,7 @@ void Workspace::did_close(const Json& message, const Json& params) { if (found == nullptr) return; const Document document { *found }; impl_->documents_.close(uri); + if (!document.path.empty()) impl_->editedAt.erase(base::path_key(document.path)); ++impl_->snapshotGeneration; if (!document.path.empty()) { if (auto text = platform::fs::read_file(document.path)) impl_->index.update(document.path, *text); diff --git a/src/project/scan.cpp b/src/project/scan.cpp index 409bd99..c2c43c0 100644 --- a/src/project/scan.cpp +++ b/src/project/scan.cpp @@ -328,6 +328,28 @@ std::string imported_name(const ScanResult& result, const ImportDeclaration& imp return result.declaration->module + ":" + import.partition; } +bool is_module_name(std::string_view name) { + const auto dotted = [](std::string_view part) { + if (part.empty()) return false; + bool atStart { true }; + for (const char c : part) { + if (c == '.') { + if (atStart) return false; + atStart = true; + continue; + } + const bool utf8 { static_cast(c) >= 0x80 }; + if (!utf8 && !base::is_identifier_char(c)) return false; + if (atStart && c >= '0' && c <= '9') return false; + atStart = false; + } + return !atStart; + }; + const std::size_t colon { name.find(':') }; + if (colon == std::string_view::npos) return dotted(name); + return dotted(name.substr(0, colon)) && dotted(name.substr(colon + 1)); +} + std::vector required_names(const ScanResult& result) { std::vector names; auto add = [&](std::string name) { diff --git a/src/project/scan.cppm b/src/project/scan.cppm index 09f1e97..753d1e4 100644 --- a/src/project/scan.cppm +++ b/src/project/scan.cppm @@ -41,6 +41,9 @@ std::string provided_name(const ScanResult& result); // Imported module names with partitions qualified ("m:p"); an implementation unit // `module m;` implicitly requires "m". Header units are not included. std::vector required_names(const ScanResult& result); +// "a.b" or "a.b:c.d": dotted identifiers, with at most one partition. A build tool's scan of a file +// saved mid-edit can report `hello.` (import-hang plan §5); such a name is no module. +bool is_module_name(std::string_view name); // The full name an import refers to, given the importing unit's declaration. std::string imported_name(const ScanResult& result, const ImportDeclaration& import); diff --git a/tests/test_normalize.cpp b/tests/test_normalize.cpp index dd2fce1..0f22469 100644 --- a/tests/test_normalize.cpp +++ b/tests/test_normalize.cpp @@ -379,6 +379,65 @@ int main() { expect(plan.excludedFiles == std::vector { "/p/src/lost.cppm" }) << std::format("{}", plan.excludedFiles); }; + "an import still being typed gets no stand-in, and a name no module can have is never planned"_test = [] { + // import-hang plan §5: typing `import hello.greet;` goes through `import hello` and `import hello.`; with autosave + // each reached the plan, got a stand-in and rewrote the engine database. A unit that provides a module still gets + // its stand-in at once: building it with an import it cannot resolve is what stalls clangd. + const std::map sources { + { "/p/src/main.cpp", "import hello;\nint main() {}\n" }, + { "/p/src/greet.cppm", "export module hello.greet;\nimport half;\n" }, + { "/p/src/other.cpp", "import gone;\nint f() { return 0; }\n" }, + }; + s::Database database; + database.hasIde = true; + s::Set set; + set.name = "hello"; + set.hasIde = true; + set.toolchain = "gcc-16.1.0-x86_64-linux-gnu"; + for (const auto& [path, text] : sources) { + s::TranslationUnit unit; + unit.source = path; + unit.workDirectory = "/p"; + unit.arguments = { "/opt/gcc/bin/g++", "-std=c++23", "-fmodules", "-c", path }; + // What a build tool's scan of a file saved mid-edit reported: `hello.` is no module name. + if (path == "/p/src/other.cpp") unit.requiredModules = { "gone", "hello.", ".x", "a..b", "a:b:c" }; + set.units.push_back(std::move(unit)); + } + database.sets.push_back(set); + std::map> facts { { set.toolchain, gcc_facts() } }; + n::PlanInput input; + input.database = &database; + input.facts = &facts; + input.engineDriverDirectory = "/payload/clangd/bin"; + input.stubDirectory = "/cache/stubs"; + input.scanner = [&](std::string_view path) { + const auto it = sources.find(std::string { path }); + return it == sources.end() ? p::ScanResult {} : p::scan_source(it->second); + }; + // clangd said it cannot find `hello`, as it does for an import being typed. + input.unresolvedModules = { { "hello", "Don't get the module unit for module hello" } }; + input.editingSources = { "/p/src/main.cpp", "/p/src/greet.cppm" }; + const auto editing = n::plan_engine(input); + auto stubs = editing.stubModules; + std::ranges::sort(stubs); + expect(stubs == std::vector { "gone", "half" }) << "only the provider's and the quiet file's: " << std::format("{}", stubs); + expect(editing.standInsDeferred); + expect(editing.excludedFiles.empty()) << std::format("{}", editing.excludedFiles); + for (const auto& issue : editing.issues) { + expect(issue.module != "hello." && issue.module != ".x" && issue.module != "a..b" && issue.module != "a:b:c") << issue.module; + if (issue.code == "unresolved-module" || issue.code == "module-build-failed") expect(issue.category == "code") << issue.code; + } + expect(std::ranges::any_of(editing.issues, [](const n::PlanIssue& issue) { return issue.code == "module-build-failed" && issue.module == "hello"; })) + << "the import is still reported"; + + input.editingSources.clear(); + const auto quiet = n::plan_engine(input); + stubs = quiet.stubModules; + std::ranges::sort(stubs); + expect(stubs == std::vector { "gone", "half", "hello" }) << "once quiet, every stand-in: " << std::format("{}", stubs); + expect(!quiet.standInsDeferred); + }; + "a file the editor opened that no set describes joins with the nearest unit's arguments"_test = [] { // robustness design C2: clangd guessed the command of xlings' apps/gui/main.cpp, a target of a feature the build did not // enable, and without a stand-in for the module it imports that nothing provides, kept a core busy for good. diff --git a/tests/test_scan.cpp b/tests/test_scan.cpp index e1c7dd6..0a025be 100644 --- a/tests/test_scan.cpp +++ b/tests/test_scan.cpp @@ -28,6 +28,11 @@ int main() { expect(!result.uncertain); }; + "a module name is dotted identifiers with at most one partition"_test = [] { + for (const std::string_view name : { "std", "hello.greet", "a.b:c", "a:b.c", "_x.y2", "m\u00e9.a" }) expect(is_module_name(name)) << name; + for (const std::string_view name : { "", "hello.", ".x", "a..b", "a:b:c", ":p", "a:", "1a", "a.1b", "a-b", "a b" }) expect(!is_module_name(name)) << name; + }; + "partitions and implementation units"_test = [] { expect(role_of(scan_source("export module a.b:c;")) == Role::module_partition_interface); expect(provided_name(scan_source("export module a.b:c;")) == "a.b:c"); From 44f74c05d5ae81ebaf416d52185cb6ace2337c95 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Fri, 25 Sep 2026 06:35:24 +0800 Subject: [PATCH 05/26] feat(status): a problem in the user's code is a diagnostic, not a server that lost a feature A missing `;` or an import still being typed turned the status to "Some features are limited": any plan issue made the state degraded, so a typo, a slow rebuild and a clangd that lost the file all looked alike, and with autosave the warning flickered as the user typed. Every status issue now carries a category -- code, engine, environment or project -- and only the categories other than code make the state degraded: an import nothing provides and a module that does not compile are the code's and are told where they are, as diagnostics; a missing SDK, an untrusted workspace or a clangd that cannot run are the environment's. A change from ready to degraded goes out only once it has lasted three seconds, so a file set aside and handed back as the user types never reaches the editor; error goes out at once. Conformance gets the checks that prove the import-hang plan on every platform: `type-text` types a line one key at a time and requires every step to be answered in time, `clangd-check` is a workaround's canary, and the fixtures typing-import, typing-import-spin (the real spin, with WA-CLANGD-001 turned off) and workaround-canaries run in CI; module-faults and failure-at-base now expect ready, naming their code issues. --- .github/workflows/ci.yml | 8 +- conformance/README.md | 7 +- .../fixtures/failure-at-base/scenario.json | 4 +- .../fixtures/module-faults/scenario.json | 2 +- .../fixtures/typing-import-spin/scenario.json | 98 ++++++++++++ .../typing-import-spin/src/greet/detail.cppm | 5 + .../typing-import-spin/src/greet/greet.cppm | 7 + .../fixtures/typing-import-spin/src/main.cpp | 7 + .../fixtures/typing-import/scenario.json | 151 ++++++++++++++++++ .../typing-import/src/greet/detail.cppm | 5 + .../typing-import/src/greet/greet.cppm | 7 + .../fixtures/typing-import/src/main.cpp | 7 + conformance/fixtures/untrusted/scenario.json | 4 +- .../workaround-canaries/scenario.json | 17 ++ .../workaround-canaries/src/compile_flags.txt | 2 + .../workaround-canaries/src/greet/detail.cppm | 5 + .../workaround-canaries/src/greet/greet.cppm | 7 + .../fixtures/workaround-canaries/src/hang.cpp | 1 + .../fixtures/workaround-canaries/src/main.cpp | 7 + src/bin/conformance.cpp | 78 +++++++++ src/engine/clangd.cpp | 10 +- src/engine/engine.cppm | 4 + src/orchestrator/workspace.cpp | 40 +++-- 23 files changed, 460 insertions(+), 23 deletions(-) create mode 100644 conformance/fixtures/typing-import-spin/scenario.json create mode 100644 conformance/fixtures/typing-import-spin/src/greet/detail.cppm create mode 100644 conformance/fixtures/typing-import-spin/src/greet/greet.cppm create mode 100644 conformance/fixtures/typing-import-spin/src/main.cpp create mode 100644 conformance/fixtures/typing-import/scenario.json create mode 100644 conformance/fixtures/typing-import/src/greet/detail.cppm create mode 100644 conformance/fixtures/typing-import/src/greet/greet.cppm create mode 100644 conformance/fixtures/typing-import/src/main.cpp create mode 100644 conformance/fixtures/workaround-canaries/scenario.json create mode 100644 conformance/fixtures/workaround-canaries/src/compile_flags.txt create mode 100644 conformance/fixtures/workaround-canaries/src/greet/detail.cppm create mode 100644 conformance/fixtures/workaround-canaries/src/greet/greet.cppm create mode 100644 conformance/fixtures/workaround-canaries/src/hang.cpp create mode 100644 conformance/fixtures/workaround-canaries/src/main.cpp diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 70b6547..7c47220 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -277,7 +277,7 @@ jobs: - platform: linux-x64 part: 1 of 2 os: ubuntu-24.04 - fixtures: mcpp-split mcpp-all-cppm verify-changes mingw mcpp-split-gcc mcpp-watch multi-root mcpp-llvm mcpp-watch@polling mcpp-emit s1-two-sets failure-at-base@vscode generated-module-negotiated + fixtures: mcpp-split mcpp-all-cppm verify-changes mingw mcpp-split-gcc mcpp-watch multi-root mcpp-llvm mcpp-watch@polling mcpp-emit s1-two-sets failure-at-base@vscode generated-module-negotiated typing-import typing-import-spin workaround-canaries - platform: linux-x64 part: 2 of 2 os: ubuntu-24.04 @@ -289,18 +289,18 @@ jobs: - platform: darwin-arm64 os: macos-14 extras: true - fixtures: inferred engine-none module-faults untrusted mcpp-llvm mcpp-split mcpp-all-cppm verify-changes mcpp-watch multi-root failure-at-base clangd-cannot-load failure-at-base@zed generated-module generated-module-old-mcpp generated-module-negotiated + fixtures: inferred engine-none module-faults untrusted mcpp-llvm mcpp-split mcpp-all-cppm verify-changes mcpp-watch multi-root failure-at-base clangd-cannot-load failure-at-base@zed generated-module generated-module-old-mcpp generated-module-negotiated typing-import typing-import-spin workaround-canaries # No mcpp on the arm64 runner (its tools are the cross-built ones), so the fixtures that # need no build tool and no compiler of their own: the semantic kit, clangd and the server # on aarch64, with module faults, a corrupt payload and polling included. - platform: linux-arm64 os: ubuntu-24.04-arm cross-tools: true - fixtures: inferred engine-none module-faults untrusted payload-corrupt clangd-cannot-load failure-at-base watch-polling + fixtures: inferred engine-none module-faults untrusted payload-corrupt clangd-cannot-load failure-at-base watch-polling typing-import typing-import-spin workaround-canaries - platform: win32-x64 part: 1 of 2 os: windows-2022 - fixtures: mcpp-split mcpp-all-cppm verify-changes mcpp-split-msvc cmake-msvc-std compdb-clangxx-msvc-std multi-root compdb-clang-cl-std mcpp-llvm-msvc failure-at-base@vscode + fixtures: mcpp-split mcpp-all-cppm verify-changes mcpp-split-msvc cmake-msvc-std compdb-clangxx-msvc-std multi-root compdb-clang-cl-std mcpp-llvm-msvc failure-at-base@vscode typing-import typing-import-spin workaround-canaries - platform: win32-x64 part: 2 of 2 os: windows-2022 diff --git a/conformance/README.md b/conformance/README.md index 1e6cc49..9c48bfb 100644 --- a/conformance/README.md +++ b/conformance/README.md @@ -64,6 +64,9 @@ checks fail at once with that reason instead of each waiting out its timeout. | `s1-two-sets` | A workspace carrying its own S1 build database (`--database`, usable plan W9.2): two sets compile the same file under `-DVARIANT=1` and `-DVARIANT=2`; `cxxModules/setContext` switches which one answers | | `watch-polling` | Run with `--no-dynamic-watch` (usable plan W9.3): a new module interface written straight into the workspace must still reach the module graph within seconds, through the polling fallback rather than a client-driven `workspace/didChangeWatchedFiles` | | `clangd-cannot-load` | 0.0.3 plan B1: its `prepare` step puts a stand-in clangd in the workspace (mcppls-mock-mcpp with an `unavailable` config) that writes a loader's message, a `GLIBCXX` version not found, to standard error and exits 1; `--clangd` points the server at it. `initialize` must be answered within 20 s (it used to wait for good), the status must reach `error` with issue `engine-incompatible`, and mcppls's own module features must work | +| `typing-import` | Import-hang plan §8: `import hello.greet;` in `main.cpp` and `export module hello.greet;` in its interface typed one key at a time, through `import hello.` and `export module hello.`, which clangd 23.1 never finishes building (WA-CLANGD-001); again with every step saved, as autosave does. Every request is answered within 5 s, the status never turns `degraded`, hover works right after, and nothing restarts clangd or is set aside | +| `typing-import-spin` | The same typing with WA-CLANGD-001 turned off (`--disable-workaround`), so clangd really spins: the file is found spinning within its 20 s budget (event `engine-spin`), set aside with that text remembered, clangd is restarted, and features come back while typing goes on (import-hang plan §4). If a clangd update removes the defect, its `engine-spin` check fails as well | +| `workaround-canaries` | Import-hang plan §9: one `clangd-check` per registered workaround with a canary, run against the payload's clangd. A failure here means a clangd update fixed that defect and the workaround it names can be removed | | `payload-corrupt` | Its `prepare` step copies the payload the runner was given and truncates clangd in the copy (usable plan W9.4); `server-arguments` then points `--payload` at that broken copy, and status must reach `error` with issue `payload-corrupt` | | `multi-root` | Two workspace folders (usable plan W9.1): an `inferred` root and an mcpp-built `mcpp-llvm` root (level 3, from mcpp's own build database), each getting its own project model and clangd, each `cxxModules/status` telling them apart by `project.root` | @@ -144,7 +147,7 @@ always has been. | Kind | Passes when | |---|---| -| `status` | `cxxModules/status` reaches `ready`, `degraded` or `error` and matches `source`, `profile-kind`, `state`, `level`, `tier` (`project.tier`, the README's L1..L4, real-project plan RP3.2), `issue-code` (with `issue-command`, that issue's command; with `issue-message`, a part of its message), `notice-code` and `engine-name`/`engines-include` when given, and a `profile-compiler` prefix (a settled status that does not match yet is looked at again for up to three seconds, since a server coalesces changes that keep its state); `"folder"` picks one root's own status in a multi-root fixture (usable plan W9.1), absent picks whichever root's arrived most recently | +| `status` | `cxxModules/status` reaches `ready`, `degraded` or `error` and matches `source`, `profile-kind`, `state`, `level`, `tier` (`project.tier`, the README's L1..L4, real-project plan RP3.2), `issue-code` (with `issue-command`, that issue's command; with `issue-message`, a part of its message; with `issue-category`, its S3 category: `code`, `engine`, `environment` or `project`), `notice-code` and `engine-name`/`engines-include` when given, and a `profile-compiler` prefix (a settled status that does not match yet is looked at again for up to three seconds, since a server coalesces changes that keep its state); `"folder"` picks one root's own status in a multi-root fixture (usable plan W9.1), absent picks whichever root's arrived most recently | | `workspace-unchanged` | no file under the workspace was added, changed or removed after the prepare steps | | `responds` | a request (`method`, default `textDocument/definition`) at `at` is answered, empty answers included, within the check's time | | `module-cache-reused` | every file clangd published for `module` (default `std`) before the server started is still there unchanged, and none was added (SC4); passes on a cold start unless `--expect-warm` | @@ -164,6 +167,8 @@ always has been. | `execute-command` | `workspace/executeCommand` with `"command"` and `"arguments"` is answered without an error (the editor's review commands, design 7.7) | | `cli` | S5 section 7: `mcppls ` with the runner's payload and the fixture's server arguments, run to completion in the workspace, exits with `"exit"` (default 0) and prints one JSON document meeting `"expect"` | | `stress` | real-project stress testing (real-project plan RP0): seeded random use — see below — meets every key present in `"budget"` | +| `type-text` | line `line` of `file` takes each of `steps` in turn, `interval-ms` apart (default 120), the whole buffer sent each time; after each, `request` (default `textDocument/documentSymbol`) is answered within `answer-within` seconds (default 5); with `save`, each step is also written to disk and reported as saved and changed, as autosave does; fails when the status turned to a state listed in `states-never` meanwhile (import-hang plan §8) | +| `clangd-check` | the runner's own clangd (`--clangd`, else the payload's) run with `--check` on `file` hangs (`expect: "hangs"`, not finished after `seconds`, default 10) or finishes (`"finishes"`); a workaround's canary expects its defect, and fails with `says` once a clangd update fixed it (import-hang plan §9) | | `report` | robustness design O3: `cxxModules/report` meets `"expect"`, retried within the check's time like an `mcp`/`cli` result (a plan or an engine may still be on its way) | An expectation of `mcp`, `cli` and `report` names a JSON pointer in `"path"`, where a `*` segment stands for every diff --git a/conformance/fixtures/failure-at-base/scenario.json b/conformance/fixtures/failure-at-base/scenario.json index 99ab640..5662e65 100644 --- a/conformance/fixtures/failure-at-base/scenario.json +++ b/conformance/fixtures/failure-at-base/scenario.json @@ -1,6 +1,6 @@ { "name": "failure-at-base", - "description": "Real-project plan RP1.1/RP1.3, at scale: a straight import chain of 100 generated modules whose base (gen.chain0) does not compile, the shape of the xlings incident this plan is named for (a dependency's failure closure reaching every transitive importer). Files outside the chain keep answering normally; a plain importer of the chain's last module is answered by mcppls's own engine within a second and carries a module-failed diagnostic; the status settles within 60s naming modules-doomed; nothing restarts clangd or is set aside.", + "description": "Real-project plan RP1.1/RP1.3, at scale: a straight import chain of 100 generated modules whose base (gen.chain0) does not compile, the shape of the xlings incident this plan is named for (a dependency's failure closure reaching every transitive importer). Files outside the chain keep answering normally; a plain importer of the chain's last module is answered by mcppls's own engine within a second and carries a module-failed diagnostic; the status settles within 60s, ready, naming modules-doomed as a problem of the code (import-hang plan §6); nothing restarts clangd or is set aside.", "server-arguments": ["--no-discover"], "prepare": [ ["{conformance}", "prepare", "failure-at-base", "100"] @@ -10,7 +10,7 @@ "method": "textDocument/hover", "timeout": 1 }, { "id": "H1-outside-the-closure-module", "kind": "hover-contains", "file": "src/healthy/a.cppm", "at": [3, 15], "expect": "healthyValue" }, { "id": "H2-outside-the-closure-definition", "kind": "definition", "file": "src/healthy/b.cpp", "at": [2, 31], "expect": "src/healthy/a.cppm" }, - { "id": "S1-settles-degraded", "kind": "status", "state": "degraded", "issue-code": "modules-doomed", "timeout": 60 }, + { "id": "S1-settles-ready-naming-the-failure", "kind": "status", "state": "ready", "issue-code": "modules-doomed", "issue-category": "code", "timeout": 60 }, { "id": "F2-deep-importer-names-the-failed-module", "kind": "diagnostic-code", "file": "src/gen-importer.cpp", "expect": "module-failed", "timeout": 30 }, { "id": "F3-no-restart-nothing-set-aside", "kind": "report", "timeout": 10, "expect": [{ "path": "/roots/0/engines/*/details/restarts", "max-items": 0 }, { "path": "/roots/0/engines/*/details/filesSetAside", "max-items": 0 }] }, diff --git a/conformance/fixtures/module-faults/scenario.json b/conformance/fixtures/module-faults/scenario.json index c170505..f5f9a68 100644 --- a/conformance/fixtures/module-faults/scenario.json +++ b/conformance/fixtures/module-faults/scenario.json @@ -3,7 +3,7 @@ "description": "Faults stay where they are (robustness design C2-C6): a module chain with an import nothing provides, a module that does not compile, and a file importing both a broken chain and a working module. Files unrelated to a fault keep every feature, a file importing a broken module keeps what does resolve, nothing waits, and a module that breaks and heals while the server runs neither stalls clangd nor leaves the project without it. The only restart allowed is of a clangd that stopped answering (answering nobody, or StuckWatch: answering nothing while using no CPU): clangd 23.1's own build of a module whose source changed twice within a second can hang, on slow machines, and a restart is how that ends. A restart for any other reason, the fault handling's own included, fails F8.", "server-arguments": ["--no-discover"], "checks": [ - { "id": "S1", "kind": "status", "source": "inferred", "state": "degraded", "issue-code": "unresolved-module", "issue-message": "nowhere", "engine-name": "clangd" }, + { "id": "S1", "kind": "status", "source": "inferred", "state": "ready", "issue-code": "unresolved-module", "issue-message": "nowhere", "issue-category": "code", "engine-name": "clangd" }, { "id": "F0-stand-in", "kind": "report", "expect": [{ "path": "/roots/0/plan/standIns", "contains": "nowhere" }, { "path": "/roots/0/plan/leftOutCount", "equals": 0 }] }, { "id": "F1-working-module", "kind": "hover-contains", "file": "src/good/user.cppm", "at": [3, 28], "expect": "answer", "timeout": 90 }, { "id": "F1-definition", "kind": "definition", "file": "src/good/user.cppm", "at": [3, 28], "expect": "src/good/a.cppm", "timeout": 60 }, diff --git a/conformance/fixtures/typing-import-spin/scenario.json b/conformance/fixtures/typing-import-spin/scenario.json new file mode 100644 index 0000000..c7bb229 --- /dev/null +++ b/conformance/fixtures/typing-import-spin/scenario.json @@ -0,0 +1,98 @@ +{ + "name": "typing-import-spin", + "description": "Import-hang plan \u00a74 against the real defect: WA-CLANGD-001 turned off, so clangd 23.1 spins on `import hello.`. The file is found spinning within its 20 s budget, set aside with that text remembered, clangd is restarted, and features come back while typing goes on. If a clangd update removes the defect, T2 fails too: WA-CLANGD-001 can then go.", + "server-arguments": [ + "--no-discover", + "--disable-workaround", + "WA-CLANGD-001" + ], + "checks": [ + { + "id": "S1", + "kind": "status", + "source": "inferred", + "state": "ready", + "engine-name": "clangd" + }, + { + "id": "T0-before", + "kind": "hover-contains", + "file": "src/main.cpp", + "at": [ + 4, + 31 + ], + "expect": "greet", + "timeout": 90 + }, + { + "id": "T1-type-into-the-spin", + "kind": "type-text", + "file": "src/main.cpp", + "line": 1, + "steps": [ + "import hello.greet", + "import hello.gree", + "import hello.gre", + "import hello.gr", + "import hello.g", + "import hello." + ], + "interval-ms": 120, + "answer-within": 30 + }, + { + "id": "T1-keep-typing", + "kind": "type-text", + "file": "src/main.cpp", + "line": 1, + "steps": [ + "import hello.g", + "import hello.gr", + "import hello.gre", + "import hello.gree", + "import hello.greet", + "import hello.greet;" + ], + "interval-ms": 500, + "answer-within": 30 + }, + { + "id": "T2-spin-found", + "kind": "report", + "timeout": 60, + "expect": [ + { + "path": "/roots/0/events/*/kind", + "equals": "engine-spin" + }, + { + "path": "/roots/0/engines/*/details/workarounds/*/turnedOff", + "equals": true + } + ] + }, + { + "id": "T3-features-back", + "kind": "hover-contains", + "file": "src/main.cpp", + "at": [ + 4, + 31 + ], + "expect": "greet", + "timeout": 60 + }, + { + "id": "T4-handed-back", + "kind": "report", + "timeout": 30, + "expect": [ + { + "path": "/roots/0/engines/*/details/filesSetAside", + "max-items": 0 + } + ] + } + ] +} diff --git a/conformance/fixtures/typing-import-spin/src/greet/detail.cppm b/conformance/fixtures/typing-import-spin/src/greet/detail.cppm new file mode 100644 index 0000000..56cb780 --- /dev/null +++ b/conformance/fixtures/typing-import-spin/src/greet/detail.cppm @@ -0,0 +1,5 @@ +export module hello.greet:detail; +import std; +export namespace hello::detail { + std::string prefix() { return "Hello, "; } +} diff --git a/conformance/fixtures/typing-import-spin/src/greet/greet.cppm b/conformance/fixtures/typing-import-spin/src/greet/greet.cppm new file mode 100644 index 0000000..21a6536 --- /dev/null +++ b/conformance/fixtures/typing-import-spin/src/greet/greet.cppm @@ -0,0 +1,7 @@ +export module hello.greet; +export import :detail; +import std; + +export namespace hello { + std::string greet(std::string_view who) { return detail::prefix() + std::string(who); } +} diff --git a/conformance/fixtures/typing-import-spin/src/main.cpp b/conformance/fixtures/typing-import-spin/src/main.cpp new file mode 100644 index 0000000..71169db --- /dev/null +++ b/conformance/fixtures/typing-import-spin/src/main.cpp @@ -0,0 +1,7 @@ +import std; +import hello.greet; + +int main(int argc, char* argv[]) { + std::println("{}", hello::greet("mcpp")); + return 0; +} diff --git a/conformance/fixtures/typing-import/scenario.json b/conformance/fixtures/typing-import/scenario.json new file mode 100644 index 0000000..d2133da --- /dev/null +++ b/conformance/fixtures/typing-import/scenario.json @@ -0,0 +1,151 @@ +{ + "name": "typing-import", + "description": "Import-hang plan \u00a78: typing `import hello.greet;` and `export module hello.greet;` one key at a time, through `import hello.` and `export module hello.`, which clangd 23.1 never finishes (WA-CLANGD-001). Every request is answered within 5 s, the status never turns degraded, and features work right after.", + "server-arguments": [ + "--no-discover" + ], + "checks": [ + { + "id": "S1", + "kind": "status", + "source": "inferred", + "state": "ready", + "engine-name": "clangd" + }, + { + "id": "T0-before", + "kind": "hover-contains", + "file": "src/main.cpp", + "at": [ + 4, + 31 + ], + "expect": "greet", + "timeout": 90 + }, + { + "id": "T1-type-import", + "kind": "type-text", + "file": "src/main.cpp", + "line": 1, + "steps": [ + "import hello.greet", + "import hello.gree", + "import hello.gre", + "import hello.gr", + "import hello.g", + "import hello.", + "import hello.g", + "import hello.gr", + "import hello.gre", + "import hello.gree", + "import hello.greet", + "import hello.greet;" + ], + "interval-ms": 120, + "answer-within": 5, + "states-never": [ + "degraded", + "error" + ] + }, + { + "id": "T2-type-import-saved", + "kind": "type-text", + "file": "src/main.cpp", + "line": 1, + "steps": [ + "import hello.greet", + "import hello.gree", + "import hello.gre", + "import hello.gr", + "import hello.g", + "import hello.", + "import hello.g", + "import hello.gr", + "import hello.gre", + "import hello.gree", + "import hello.greet", + "import hello.greet;" + ], + "interval-ms": 120, + "answer-within": 5, + "save": true, + "states-never": [ + "degraded", + "error" + ] + }, + { + "id": "T3-features-after", + "kind": "hover-contains", + "file": "src/main.cpp", + "at": [ + 4, + 31 + ], + "expect": "greet", + "timeout": 20 + }, + { + "id": "T4-type-module-declaration", + "kind": "type-text", + "file": "src/greet/greet.cppm", + "line": 0, + "steps": [ + "export module hello.greet", + "export module hello.gree", + "export module hello.gre", + "export module hello.gr", + "export module hello.g", + "export module hello.", + "export module hello.g", + "export module hello.gr", + "export module hello.gre", + "export module hello.gree", + "export module hello.greet", + "export module hello.greet;" + ], + "interval-ms": 120, + "answer-within": 5, + "states-never": [ + "error" + ] + }, + { + "id": "T5-features-after", + "kind": "hover-contains", + "file": "src/main.cpp", + "at": [ + 4, + 31 + ], + "expect": "greet", + "timeout": 30 + }, + { + "id": "T6-workaround-on", + "kind": "report", + "expect": [ + { + "path": "/roots/0/engines/*/details/workarounds/*/id", + "equals": "WA-CLANGD-001" + } + ] + }, + { + "id": "T7-no-spin-no-restart", + "kind": "report", + "expect": [ + { + "path": "/roots/0/engines/*/details/restarts", + "max-items": 0 + }, + { + "path": "/roots/0/engines/*/details/filesSetAside", + "max-items": 0 + } + ] + } + ] +} diff --git a/conformance/fixtures/typing-import/src/greet/detail.cppm b/conformance/fixtures/typing-import/src/greet/detail.cppm new file mode 100644 index 0000000..56cb780 --- /dev/null +++ b/conformance/fixtures/typing-import/src/greet/detail.cppm @@ -0,0 +1,5 @@ +export module hello.greet:detail; +import std; +export namespace hello::detail { + std::string prefix() { return "Hello, "; } +} diff --git a/conformance/fixtures/typing-import/src/greet/greet.cppm b/conformance/fixtures/typing-import/src/greet/greet.cppm new file mode 100644 index 0000000..21a6536 --- /dev/null +++ b/conformance/fixtures/typing-import/src/greet/greet.cppm @@ -0,0 +1,7 @@ +export module hello.greet; +export import :detail; +import std; + +export namespace hello { + std::string greet(std::string_view who) { return detail::prefix() + std::string(who); } +} diff --git a/conformance/fixtures/typing-import/src/main.cpp b/conformance/fixtures/typing-import/src/main.cpp new file mode 100644 index 0000000..71169db --- /dev/null +++ b/conformance/fixtures/typing-import/src/main.cpp @@ -0,0 +1,7 @@ +import std; +import hello.greet; + +int main(int argc, char* argv[]) { + std::println("{}", hello::greet("mcpp")); + return 0; +} diff --git a/conformance/fixtures/untrusted/scenario.json b/conformance/fixtures/untrusted/scenario.json index e89b324..1f4b6d3 100644 --- a/conformance/fixtures/untrusted/scenario.json +++ b/conformance/fixtures/untrusted/scenario.json @@ -10,7 +10,9 @@ "kind": "status", "source": "inferred", "profile-kind": "semantic-kit", - "state": "degraded" + "state": "degraded", + "issue-code": "untrusted-workspace", + "issue-category": "environment" }, { "id": "C2", diff --git a/conformance/fixtures/workaround-canaries/scenario.json b/conformance/fixtures/workaround-canaries/scenario.json new file mode 100644 index 0000000..b9b59c8 --- /dev/null +++ b/conformance/fixtures/workaround-canaries/scenario.json @@ -0,0 +1,17 @@ +{ + "name": "workaround-canaries", + "description": "Import-hang plan \u00a79: each registered clangd workaround's defect, run against the payload's clangd. A check here failing means an update of clangd fixed the defect, and the workaround it names can be removed (src/engine/clangd/workarounds.cpp).", + "server-arguments": [ + "--no-discover" + ], + "checks": [ + { + "id": "WA-CLANGD-001", + "kind": "clangd-check", + "file": "src/hang.cpp", + "expect": "hangs", + "seconds": 10, + "says": "WA-CLANGD-001 is no longer needed: this clangd finishes `import hello.` at once; remove it from src/engine/clangd/workarounds.cpp and the typing-import-spin fixture" + } + ] +} diff --git a/conformance/fixtures/workaround-canaries/src/compile_flags.txt b/conformance/fixtures/workaround-canaries/src/compile_flags.txt new file mode 100644 index 0000000..dfad364 --- /dev/null +++ b/conformance/fixtures/workaround-canaries/src/compile_flags.txt @@ -0,0 +1,2 @@ +-std=c++23 +-xc++ diff --git a/conformance/fixtures/workaround-canaries/src/greet/detail.cppm b/conformance/fixtures/workaround-canaries/src/greet/detail.cppm new file mode 100644 index 0000000..56cb780 --- /dev/null +++ b/conformance/fixtures/workaround-canaries/src/greet/detail.cppm @@ -0,0 +1,5 @@ +export module hello.greet:detail; +import std; +export namespace hello::detail { + std::string prefix() { return "Hello, "; } +} diff --git a/conformance/fixtures/workaround-canaries/src/greet/greet.cppm b/conformance/fixtures/workaround-canaries/src/greet/greet.cppm new file mode 100644 index 0000000..21a6536 --- /dev/null +++ b/conformance/fixtures/workaround-canaries/src/greet/greet.cppm @@ -0,0 +1,7 @@ +export module hello.greet; +export import :detail; +import std; + +export namespace hello { + std::string greet(std::string_view who) { return detail::prefix() + std::string(who); } +} diff --git a/conformance/fixtures/workaround-canaries/src/hang.cpp b/conformance/fixtures/workaround-canaries/src/hang.cpp new file mode 100644 index 0000000..ceda8a5 --- /dev/null +++ b/conformance/fixtures/workaround-canaries/src/hang.cpp @@ -0,0 +1 @@ +import hello. diff --git a/conformance/fixtures/workaround-canaries/src/main.cpp b/conformance/fixtures/workaround-canaries/src/main.cpp new file mode 100644 index 0000000..71169db --- /dev/null +++ b/conformance/fixtures/workaround-canaries/src/main.cpp @@ -0,0 +1,7 @@ +import std; +import hello.greet; + +int main(int argc, char* argv[]) { + std::println("{}", hello::greet("mcpp")); + return 0; +} diff --git a/src/bin/conformance.cpp b/src/bin/conformance.cpp index f2ddab2..f9125b4 100644 --- a/src/bin/conformance.cpp +++ b/src/bin/conformance.cpp @@ -1051,9 +1051,11 @@ class Scenario { if (auto issueCode = check.find("issue-code"); issueCode != check.end()) { const std::string wantedCommand { check.value("issue-command", std::string {}) }; const std::string wantedMessage { check.value("issue-message", std::string {}) }; // a part of the message + const std::string wantedCategory { check.value("issue-category", std::string {}) }; // S3: code | engine | environment | project matched = matched && std::ranges::any_of(snapshot.value("issues", Json::array()), [&](const Json& issue) { if (issue.value("code", std::string {}) != issueCode->get()) return false; if (!wantedMessage.empty() && !issue.value("message", std::string {}).contains(wantedMessage)) return false; + if (!wantedCategory.empty() && issue.value("category", std::string {}) != wantedCategory) return false; return wantedCommand.empty() || issue.value("command", Json::object()).value("command", std::string {}) == wantedCommand; }); } @@ -1169,6 +1171,82 @@ class Scenario { auto [held, detail] = expectations_hold(value, check.value("expect", Json::array())); return { held, held ? lsp::dump(value).substr(0, 160) : detail }; } + if (kind == "type-text") { + // import-hang plan §8: a person typing a line one key at a time. Line `line` of `file` takes each of + // `steps` in turn, `interval-ms` apart (the whole buffer is sent, as editors with full sync do); after each, + // `request` (default documentSymbol) must be answered within `answer-within` seconds. With `save`, each step + // is also written to disk and reported as saved and changed, as autosave does. The check fails when the + // status turned to any state of `states-never` meanwhile. + open(file); + const int line { check.value("line", 0) }; + const std::chrono::milliseconds interval { check.value("interval-ms", 150) }; + const std::string method { check.value("request", std::string { "textDocument/documentSymbol" }) }; + const std::chrono::seconds answerWithin { check.value("answer-within", 5) }; + const bool save { check.value("save", false) }; + const auto startedAt = Clock::now(); + const std::size_t timelineBefore { client_.statusTimeline.size() }; + double worst { 0 }; + for (const auto& step : check.value("steps", Json::array())) { + const std::string current { text_of(file) }; + std::vector lines; + for (const auto each : base::split_lines(current)) lines.emplace_back(each); + if (line < 0 || static_cast(line) >= lines.size()) return { false, std::format("line {} is not in {}", line, file) }; + lines[static_cast(line)] = step.get(); + std::string text; + for (const auto& each : lines) text += each + "\n"; + change(file, text); + if (save) { + (void)fs::write_file_atomic(base::join_path(workspace_, file), text); + client_.notify("textDocument/didSave", Json { { "textDocument", Json { { "uri", uri(file) } } } }); + client_.notify("workspace/didChangeWatchedFiles", Json { { "changes", Json::array({ Json { { "uri", uri(file) }, { "type", 2 } } }) } }); + } + const auto asked = Clock::now(); + Json params { { "textDocument", Json { { "uri", uri(file) } } } }; + if (method != "textDocument/documentSymbol" && method != "textDocument/semanticTokens/full") { + params["position"] = Json { { "line", line }, { "character", 0 } }; + } + if (!client_.request(method, params, answerWithin)) { + return { false, std::format("{} was not answered within {} s after the line became '{}'", method, answerWithin.count(), step.get()) }; + } + worst = std::max(worst, std::chrono::duration(Clock::now() - asked).count()); + client_.drain(interval); + } + for (std::size_t i { timelineBefore }; i < client_.statusTimeline.size(); ++i) { + const auto& [at, state] = client_.statusTimeline[i]; + for (const auto& never : check.value("states-never", Json::array())) { + if (state == never.get()) { + return { false, std::format("the status turned {} {:.1f} s into the typing", state, std::chrono::duration(at - startedAt).count()) }; + } + } + } + return { true, std::format("{} steps, slowest answer {:.2f} s", check.value("steps", Json::array()).size(), worst) }; + } + if (kind == "clangd-check") { + // import-hang plan §9, a workaround's canary: the runner's own clangd (--clangd, else the payload's) is run with + // --check on `file`; `expect` is "hangs" (it has not finished after `seconds`, default 10) or "finishes". A canary + // expects the defect its workaround exists for; once an update of clangd fixes it, the check fails with `says`. + const std::string clangd { !options_.clangd.empty() ? options_.clangd + : base::join_path(options_.payload, "clangd/bin/clangd") + std::string { mcppls::os::EXECUTABLE_SUFFIX } }; + if (!fs::is_regular_file(clangd)) return { false, "no clangd: pass --clangd or --payload" }; + mcppls::platform::SpawnOptions spawn; + spawn.program = clangd; + spawn.arguments = { std::format("--check={}", base::join_path(workspace_, file)), "--check-tidy-time=0" }; + spawn.workDirectory = workspace_; + const std::chrono::seconds limit { check.value("seconds", 10) }; + auto running = std::async(std::launch::async, [spawn, limit]() mutable { return mcppls::platform::run(std::move(spawn), limit); }); + while (running.wait_for(std::chrono::milliseconds { 200 }) != std::future_status::ready) client_.drain(std::chrono::milliseconds { 0 }); + auto result = running.get(); + if (!result) return { false, result.error().message }; + const bool hung { result->timedOut }; + const std::string expected { check.value("expect", std::string { "hangs" }) }; + const std::string says { check.value("says", std::string {}) }; + if (expected == "hangs" && !hung) { + return { false, std::format("{} (clangd exited {} in under {} s: {})", says.empty() ? std::string { "clangd finished; the defect is gone" } : says, + result->exitCode, limit.count(), (result->output + result->error).substr(0, 200)) }; + } + if (expected == "finishes" && hung) return { false, std::format("clangd did not finish {} in {} s", file, limit.count()) }; + return { true, hung ? std::format("clangd has not finished {} after {} s, as expected", file, limit.count()) : "clangd finished" }; + } if (kind == "responds") { // An answer of any kind, an empty one included, within the check's time: a file the engine // cannot serve must be answered at once rather than left waiting (usable plan W1.7, W5.4). diff --git a/src/engine/clangd.cpp b/src/engine/clangd.cpp index 74ce391..d3e3071 100644 --- a/src/engine/clangd.cpp +++ b/src/engine/clangd.cpp @@ -1023,7 +1023,7 @@ class ClangdEngine final : public Engine { closedBackground_.clear(); if (options_.payloadCorrupt) { unavailable_ = true; - add_issue_(Issue { "payload-corrupt", "the extension's payload is corrupt or was modified; reinstall the extension", "mcppls.showLogs" }); + add_issue_(Issue { "payload-corrupt", "the extension's payload is corrupt or was modified; reinstall the extension", "mcppls.showLogs", "environment" }); flush_deferred_without_engine_(); host_->engine_settled(ENGINE_ID, Json::object()); host_->status_changed(); @@ -1031,7 +1031,7 @@ class ClangdEngine final : public Engine { } if (options_.executable.empty() || !platform::fs::is_regular_file(options_.executable)) { unavailable_ = true; - add_issue_(Issue { "engine-missing", "clangd was not found; only module-level features are available", "mcppls.showLogs" }); + add_issue_(Issue { "engine-missing", "clangd was not found; only module-level features are available", "mcppls.showLogs", "environment" }); flush_deferred_without_engine_(); host_->engine_settled(ENGINE_ID, Json::object()); host_->status_changed(); @@ -1590,7 +1590,7 @@ class ClangdEngine final : public Engine { add_issue_(Issue { "engine-incompatible", std::format("the bundled clangd cannot run on this system ({}); only module-level features are available. " "Supported systems are listed in the install guide", line), - "mcppls.showLogs" }); + "mcppls.showLogs", "environment" }); host_->record_event("engine-incompatible", Json { { "line", line } }); flush_deferred_without_engine_(); host_->engine_settled(ENGINE_ID, Json::object()); @@ -1614,7 +1614,7 @@ class ClangdEngine final : public Engine { host_->record_event("std-fallback-kit", Json { { "module", parsed.module }, { "reason", parsed.reason } }); add_issue_(Issue { "std-fallback-kit", std::format("clangd could not build the toolchain's standard library module ({}); files are read with the semantic kit", parsed.reason), - "mcppls.showLogs" }); + "mcppls.showLogs", "environment" }); host_->request_replan(); host_->status_changed(); } @@ -1815,7 +1815,7 @@ class ClangdEngine final : public Engine { void update_doom_issue_() { std::erase_if(issues_, [](const Issue& issue) { return issue.code == "modules-doomed"; }); if (doomedModules_.empty()) return; - issues_.push_back(Issue { "modules-doomed", doom_issue_message_(), "mcppls.showLogs" }); + issues_.push_back(Issue { "modules-doomed", doom_issue_message_(), "mcppls.showLogs", "code" }); } // One diagnostic, on the import (or the module declaration, for a unit of a doomed module diff --git a/src/engine/engine.cppm b/src/engine/engine.cppm index d598620..97b859b 100644 --- a/src/engine/engine.cppm +++ b/src/engine/engine.cppm @@ -47,6 +47,10 @@ struct Issue { std::string code; std::string message; std::string command; // a client command id, optional + // Whose problem it is (S3 status issue category, import-hang plan §6): "engine" (the engine lost something), + // "environment" (the machine or the payload), or "code" (the user's source; told as diagnostics, never a + // degraded state). + std::string category { "engine" }; }; struct EngineStatus { diff --git a/src/orchestrator/workspace.cpp b/src/orchestrator/workspace.cpp index 0a8f44b..aa9ee33 100644 --- a/src/orchestrator/workspace.cpp +++ b/src/orchestrator/workspace.cpp @@ -63,6 +63,10 @@ constexpr std::array WATCH_POLL_SKIP_DIRECTORIES { "target" // Changes to cxxModules/status that keep its state are sent at most this often (S3 4). constexpr std::chrono::milliseconds STATUS_COALESCE { 250 }; +// import-hang plan §6: a change from a working state to degraded goes out only once it has lasted this +// long, so a condition that passes by itself (a file set aside and handed back as the user types) never +// reaches the editor. error goes out at once. +constexpr std::chrono::milliseconds DEGRADED_HOLD { 3000 }; // The module structure of a scan, for deciding whether an edit changes the engine database. std::string structure_of(const project::ScanResult& scan) { @@ -262,6 +266,8 @@ struct Workspace::Impl final : engine::Host { State lastSentState { State::starting }; std::optional lastStatusSentAt; std::optional statusFlushAt; // a coalesced change goes out then + std::optional degradedSince; // when compute_state() turned degraded, while that is held back + State lastReportedState { State::starting }; // the state last let through DEGRADED_HOLD Impl(std::string root_, std::string key_, SessionOptions options_, engine::PayloadPaths payload_, bool payloadCorrupt_, bool kitEnabled_, std::string compilerOverride_, std::shared_ptr events_, ClientSink& client_) @@ -1132,10 +1138,14 @@ struct Workspace::Impl final : engine::Host { if (core && core->preparing) return State::preparing; // usable plan W5.4: degraded regardless of whether any open file happens to need std yet. if (kit && spec::requires_macos_sdk(*kit) && macosSdk.empty()) return State::degraded; + // import-hang plan §6: a problem in the user's own code is told as a diagnostic where it is, never as a + // server that lost a feature; only the other categories make the state degraded. + const auto notCode = [](const auto& issue) { return issue.category != "code"; }; bool engineIssues { false }; - for (const auto& engine : engines) engineIssues = engineIssues || !engine->status().issues.empty(); + for (const auto& engine : engines) engineIssues = engineIssues || std::ranges::any_of(engine->status().issues, notCode); + const bool planIssues { std::ranges::any_of(plan.issues, notCode) }; // S2 5: a kept model the producer could not confirm may be stale: said, not hidden in a ready state. - if (engineIssues || !model->issues.empty() || !plan.issues.empty() || !staleModelReason.empty()) return State::degraded; + if (engineIssues || !model->issues.empty() || planIssues || !staleModelReason.empty()) return State::degraded; return State::ready; } @@ -1241,6 +1251,16 @@ struct Workspace::Impl final : engine::Host { void update_status() { if (!initializeAnswered) return; // see the field's own comment const State state { compute_state() }; + if (state == State::degraded && lastReportedState != State::degraded) { + const auto now = Clock::now(); + if (!degradedSince) degradedSince = now; + if (now < *degradedSince + DEGRADED_HOLD) { + if (!statusFlushAt || *degradedSince + DEGRADED_HOLD < *statusFlushAt) statusFlushAt = *degradedSince + DEGRADED_HOLD; + return; + } + } + if (state != State::degraded) degradedSince.reset(); + lastReportedState = state; // Before the gate below, not after it. `clientSupportsStatus` means the client understands // this repository's own `cxxModules/status` — which is its VS Code extension and nothing // else. Every client this progress exists for (Zed, nvim, Helix, …) fails that test, so @@ -1249,14 +1269,15 @@ struct Workspace::Impl final : engine::Host { if (state == State::ready || state == State::degraded) say_one_engine_per_file_once(); if (!clientSupportsStatus) return; Json issues = Json::array(); - auto add = [&](std::string_view code, std::string_view message, std::string_view command, std::string_view title = "Fix") { + auto add = [&](std::string_view code, std::string_view message, std::string_view command, std::string_view title = "Fix", + std::string_view category = "project") { if (issues.size() >= 20) return; - Json issue { { "code", std::string { code } }, { "message", std::string { message } } }; + Json issue { { "code", std::string { code } }, { "message", std::string { message } }, { "category", std::string { category } } }; if (!command.empty()) issue["command"] = Json { { "title", std::string { title } }, { "command", std::string { command } } }; issues.push_back(std::move(issue)); }; for (const auto& engine : engines) { - for (const auto& issue : engine->status().issues) add(issue.code, issue.message, issue.command); + for (const auto& issue : engine->status().issues) add(issue.code, issue.message, issue.command, "Fix", issue.category); } if (!staleModelReason.empty()) { // Decision 8: no version table, no comparison --- one sentence that points at the one @@ -1272,7 +1293,7 @@ struct Workspace::Impl final : engine::Host { std::format("the build description needs a download: {}. Run the build tool in your terminal, " "or turn on mcppls.buildTool = online. It may also be an older build tool: updating it is worth trying", needsDownload), - "mcppls.runBuildToolInTerminal", "Run in Terminal"); + "mcppls.runBuildToolInTerminal", "Run in Terminal", "environment"); } if (producerElapsed) { add("producer-slow", std::format("reading the build description ({}, {} s)", @@ -1290,11 +1311,12 @@ struct Workspace::Impl final : engine::Host { for (const auto& issue : plan.issues) { // sdk-missing is reported once below, workspace-wide, with the fix command (W5.4). if (issue.code == "sdk-missing") continue; - add(issue.code, std::format("{} ({})", issue.message, base::file_name(issue.file)), ""); + add(issue.code, std::format("{} ({})", issue.message, base::file_name(issue.file)), "", "Fix", issue.category); } - if (!options.trusted) add("untrusted-workspace", "the workspace is not trusted: build tools and compilers are not run", ""); + if (!options.trusted) add("untrusted-workspace", "the workspace is not trusted: build tools and compilers are not run", "", "Fix", "environment"); if (kit && spec::requires_macos_sdk(*kit) && macosSdk.empty()) { - add("sdk-missing", "the macOS SDK was not found; install the Command Line Tools", "mcppls.installCommandLineTools", "Install Command Line Tools"); + add("sdk-missing", "the macOS SDK was not found; install the Command Line Tools", "mcppls.installCommandLineTools", "Install Command Line Tools", + "environment"); } Json notices = Json::array(); if (model) { From 69ff9b6a25aa76150e6066049dccedff0c5c2016 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Fri, 25 Sep 2026 06:27:16 +0800 Subject: [PATCH 06/26] feat(nvim): import/module get colored, and a conflicting clangd can be stopped for you The Neovim plugin now asks the server for module-syntax semantic tokens (initializationOptions.semanticTokens = { modules, moduleType = true }), where the new semantic_tokens_modules setup() option (default true) feeds `modules`; a user's own init_options.semanticTokens, if set, replaces the plugin's default outright, and conflictArbitration stays forced afterwards as before. @lsp.type.module is linked to @module, and @lsp.type.keyword to @keyword, both as Neovim `default` links so a colorscheme or the user's own nvim_set_hl wins; they are set at setup and put back on every ColorScheme, since colorschemes clear existing links when they load. @lsp.type.keyword already links to @keyword on Neovim 0.10 through 0.12 on its own (checked on 0.10.4 and 0.12.5 locally), so this is mostly a safety net for @lsp.type.module, the custom type this plugin's moduleType = true asks the server to send. A new disable_conflicting option (default false, unchanged behavior) has the plugin stop a clangd or ccls client itself when it attaches to a buffer mcppls also serves, instead of only saying so once: a client serving only such buffers is stopped outright, one also serving another buffer is only detached from this one, and either way the plugin says, once per client, which one it stopped. The default notice now mentions the option. The stop/detach is deferred one tick (vim.schedule): Neovim's own Client:on_attach() re-sets attached_buffers[bufnr] right after firing the same LspAttach event, so detaching synchronously from inside that handler would be undone as soon as it returns. editors/nvim/tests/smoke.lua covers all of this: the init options plugin sends and how a user's own init_options and semantic_tokens_modules interact; the highlight defaults and that they don't override a user's own; and both disable_conflicting branches (stop and detach-only), against the existing in-process fake clangd/ccls pattern. README documents the new options and the highlight groups, with how to recolor them. Ran editors/nvim/tests/smoke.lua (setup and enable modes) locally on Neovim 0.12.5 and 0.10.4; all checks pass on both. CI also runs 0.11.5, which was not available to test here, but nothing in this change uses APIs newer than 0.10. --- editors/nvim/README.md | 45 ++++++++++++++- editors/nvim/lua/mcppls/init.lua | 83 ++++++++++++++++++++++++---- editors/nvim/tests/smoke.lua | 94 ++++++++++++++++++++++++++++++-- 3 files changed, 202 insertions(+), 20 deletions(-) diff --git a/editors/nvim/README.md b/editors/nvim/README.md index 31cb171..27368f0 100644 --- a/editors/nvim/README.md +++ b/editors/nvim/README.md @@ -53,8 +53,17 @@ the same configuration. mcppls runs clangd itself, with the module database it built. Do not also start clangd (or ccls) for C and C++: two servers over one file means two engines answering. If one is attached to a -buffer mcppls serves, the plugin says so once, naming it. With nvim-lspconfig that means not -calling `lspconfig.clangd.setup()`; with 0.11's mechanism, `vim.lsp.enable('clangd', false)`. +buffer mcppls serves, the plugin says so once, naming it, and mentions `disable_conflicting`. With +nvim-lspconfig that means not calling `lspconfig.clangd.setup()`; with 0.11's mechanism, +`vim.lsp.enable('clangd', false)`. + +Set `disable_conflicting = true` to have the plugin do this itself instead: when `clangd` or `ccls` +attaches to a buffer mcppls also serves, the plugin stops it there and says, once per client, which +one it stopped. If that client serves only buffers mcppls also serves, it is stopped outright +(`client:stop()`); otherwise only this buffer is detached from it +(`vim.lsp.buf_detach_client()`), so it keeps running for whatever else it serves. This is a +per-buffer decision, not a global one: `vim.lsp.enable('clangd', false)`, above, is the way to stop +it from starting at all. ## Options @@ -67,13 +76,43 @@ require('mcppls').setup({ -- compiler = 'clang++', -- semanticKit = 'auto', -- or 'off' }, - detect_conflicts = true, -- say so when clangd or ccls is attached beside mcppls + detect_conflicts = true, -- say so when clangd or ccls is attached beside mcppls + disable_conflicting = false, -- stop clangd/ccls on buffers mcppls also serves, instead of only saying so + semantic_tokens_modules = true, -- ask the server for module-syntax semantic tokens (see "Highlighting `import`" below) }) ``` The root is the nearest directory with a root marker, else the current directory. `.cppm`, `.ixx`, `.mpp`, `.ccm` and `.cxxm` files are C++. +## Highlighting `import` + +Neither Neovim's own C++ syntax nor clangd's semantic tokens color `import`, `module`, `export` or +a module name: clangd's legend has no `keyword` type, and it emits nothing for module syntax. mcppls +fixes this from its own scan of the document, sent as extra semantic tokens on top of clangd's +(design `.agents/docs/2026-09-25-import-hang-status-highlight.md` §7): + +- `import`, `module` and `export` come back as the standard semantic type `keyword`, shown through + the `@lsp.type.keyword` highlight group. +- A module or partition name comes back as a custom type `module` (with a custom modifier + `partition` on the part after `:`), shown through `@lsp.type.module` and `@lsp.mod.partition`. + +`semantic_tokens_modules` (default `true`) turns this off if you would rather rely on your own +tree-sitter query or grammar for module syntax; it is sent to the server as +`initializationOptions.semanticTokens.modules`, and your own `init_options.semanticTokens`, if you +set one, is used as given instead. + +The plugin links `@lsp.type.module` to `@module` and makes sure `@lsp.type.keyword` is linked to +`@keyword`, both as Neovim's own `default` links: your colorscheme's own choice, or anything you set +yourself, is used instead, and these defaults are restored on `ColorScheme` (colorschemes clear +their own links when they load). To pick your own colors: + +```lua +vim.api.nvim_set_hl(0, '@lsp.type.module', { fg = '#...' }) -- module and partition names +vim.api.nvim_set_hl(0, '@lsp.mod.partition', { italic = true }) -- only the part after `:` +vim.api.nvim_set_hl(0, '@lsp.type.keyword', { link = 'Keyword' }) -- import / module / export +``` + ## Commands and statusline | Command | | diff --git a/editors/nvim/lua/mcppls/init.lua b/editors/nvim/lua/mcppls/init.lua index c5b5511..cf417c8 100644 --- a/editors/nvim/lua/mcppls/init.lua +++ b/editors/nvim/lua/mcppls/init.lua @@ -93,6 +93,15 @@ function M.config() -- The server sends cxxModules/status only to a client that says it reads it (S3). capabilities.experimental = vim.tbl_extend('force', capabilities.experimental or {}, { cxxModules = { version = 1, status = true } }) + -- `modules`: ask the server for module-syntax semantic tokens (design + -- .agents/docs/2026-09-25-import-hang-status-highlight.md §7, §12). `moduleType = true`: this + -- plugin knows the custom `module` type and `partition` modifier (set_highlight_defaults below), + -- so the server need not fall back to `namespace`. + local modules = options.semantic_tokens_modules + if modules == nil then + modules = true + end + local defaults = { semanticTokens = { modules = modules, moduleType = true } } return { name = 'mcppls', cmd = { M.server_path() or 'mcppls', 'serve' }, @@ -103,9 +112,10 @@ function M.config() on_dir(M.root(bufnr)) end, capabilities = capabilities, - -- This plugin tells the user about a second C/C++ server itself (watch_conflicts below), so the - -- server need not say it on every start. - init_options = vim.tbl_extend('force', options.init_options or {}, { conflictArbitration = 'client' }), + -- `defaults` first, so a user's own init_options.semanticTokens replaces it outright; then + -- conflictArbitration is forced last, as before: this plugin tells the user about a second + -- C/C++ server itself (watch_conflicts below), so the server need not say it on every start. + init_options = vim.tbl_extend('force', defaults, options.init_options or {}, { conflictArbitration = 'client' }), handlers = { ['cxxModules/status'] = on_status }, } end @@ -244,33 +254,74 @@ end M.conflicting = { clangd = true, ccls = true } local conflict_told = false +local conflict_stopped = {} -- client id -> true, once disable_conflicting has dealt with it + +--- Detach `client` from `bufnr`, or stop it outright when that is its only buffer, so no idle +--- server is left with nothing attached. The least surprising choice: a client also serving other +--- buffers keeps running for them; one serving only buffers mcppls also serves is not worth +--- leaving alive (README "Options"). +local function stop_conflicting(client, bufnr) + if vim.tbl_count(client.attached_buffers or {}) <= 1 then + client_stop(client) + else + vim.lsp.buf_detach_client(bufnr, client.id) + end +end + local function watch_conflicts(group) vim.api.nvim_create_autocmd('LspAttach', { group = group, callback = function(args) - if conflict_told or options.detect_conflicts == false then + if options.detect_conflicts == false then return end local get = vim.lsp.get_clients or vim.lsp.get_active_clients - local names = {} + local mcppls_here = false + local conflicting = {} for _, c in ipairs(get({ bufnr = args.buf })) do - names[c.name] = true + if c.name == 'mcppls' then + mcppls_here = true + elseif M.conflicting[c.name] then + conflicting[#conflicting + 1] = c + end end - if not names.mcppls then + if not mcppls_here then return end - for name in pairs(M.conflicting) do - if names[name] then + for _, c in ipairs(conflicting) do + if options.disable_conflicting then + if not conflict_stopped[c.id] then + conflict_stopped[c.id] = true + local bufnr, name = args.buf, c.name + -- Deferred to the next tick: Neovim's own Client:on_attach() sets + -- attached_buffers[bufnr] again right after firing this same LspAttach, so detaching + -- synchronously here would be undone as soon as this callback returns. + vim.schedule(function() + stop_conflicting(c, bufnr) + vim.notify(string.format('mcppls already serves this buffer; stopped %s for it.', name), vim.log.levels.WARN) + end) + end + elseif not conflict_told then conflict_told = true vim.notify(string.format('mcppls runs its own clangd; %s is also attached to this buffer, so two engines answer. ' - .. 'Stop starting %s for C and C++ (e.g. vim.lsp.enable(%q, false)).', name, name, name), vim.log.levels.WARN) - return + .. 'Stop starting %s for C and C++ (e.g. vim.lsp.enable(%q, false)), or set disable_conflicting = true ' + .. 'to have mcppls stop it itself.', c.name, c.name, c.name), vim.log.levels.WARN) end end end, }) end +-- `@lsp.type.module` (module and partition names) and `@lsp.type.keyword` (`import`, `module`, +-- `export`) as `default` links, so a colorscheme or the user's own nvim_set_hl still wins +-- (design .agents/docs/2026-09-25-import-hang-status-highlight.md §7, §12). Neovim already links +-- `@lsp.type.keyword` to `@keyword` on 0.10 through 0.12 (checked on 0.10.4 and 0.12.5); this is a +-- safety net in case some version, or a colorscheme's `hi clear`, ever leaves it unset. +local function set_highlight_defaults() + vim.api.nvim_set_hl(0, '@lsp.type.module', { link = '@module', default = true }) + vim.api.nvim_set_hl(0, '@lsp.type.keyword', { link = '@keyword', default = true }) +end + local commands_defined = false local function define_commands() if commands_defined then @@ -278,18 +329,26 @@ local function define_commands() end commands_defined = true watch_conflicts(vim.api.nvim_create_augroup('mcppls-conflicts', { clear = true })) + set_highlight_defaults() + -- Colorschemes clear existing links when they load, so put the defaults back each time. + vim.api.nvim_create_autocmd('ColorScheme', { + group = vim.api.nvim_create_augroup('mcppls-highlights', { clear = true }), + callback = set_highlight_defaults, + }) vim.api.nvim_create_user_command('McpplsStatus', show_status, { desc = 'mcppls: the build description, engine and issues' }) vim.api.nvim_create_user_command('McpplsRestart', restart, { desc = 'mcppls: restart the server' }) vim.api.nvim_create_user_command('McpplsReload', reload, { desc = 'mcppls: read the build description again' }) end --- Start mcppls on C and C++ buffers. ---- @param opts? { server?: string, init_options?: table, filetypes?: string[], root_markers?: string[], detect_conflicts?: boolean } +--- @param opts? { server?: string, init_options?: table, filetypes?: string[], root_markers?: string[], detect_conflicts?: boolean, semantic_tokens_modules?: boolean, disable_conflicting?: boolean } function M.setup(opts) opts = opts or {} options.server = opts.server options.init_options = opts.init_options options.detect_conflicts = opts.detect_conflicts + options.semantic_tokens_modules = opts.semantic_tokens_modules + options.disable_conflicting = opts.disable_conflicting if opts.filetypes then M.filetypes = opts.filetypes end diff --git a/editors/nvim/tests/smoke.lua b/editors/nvim/tests/smoke.lua index 1a117fd..3bf8afb 100644 --- a/editors/nvim/tests/smoke.lua +++ b/editors/nvim/tests/smoke.lua @@ -177,9 +177,55 @@ check('completion after hello:: offers greet', ok) check('nothing was shown to the user', #shown == 0, vim.inspect(shown)) --- A second C++ server on the same buffer: an in-process stand-in named clangd. The plugin says so, --- once, and names it. -local function fake_server() +-- initializationOptions.semanticTokens: `modules` follows the new `semantic_tokens_modules` +-- setup() option (default true); `moduleType` is always true, since this plugin knows the custom +-- `module` type (design .agents/docs/2026-09-25-import-hang-status-highlight.md §12, "Neovim"). +-- config() is pure (it starts nothing), so these are cheap to check without touching the live +-- client; the reuse below only reattaches main.cpp to the same already-running server +-- (reuse_client_default in Neovim's own vim.lsp.start matches by name and root_dir, not +-- init_options). +local cfg = mcppls.config() +check('semanticTokens.modules defaults to true', cfg.init_options.semanticTokens.modules == true, vim.inspect(cfg.init_options)) +check('moduleType is always sent as true', cfg.init_options.semanticTokens.moduleType == true, vim.inspect(cfg.init_options)) + +mcppls.setup({ server = server, semantic_tokens_modules = false }) +cfg = mcppls.config() +check('semantic_tokens_modules = false is respected', cfg.init_options.semanticTokens.modules == false, vim.inspect(cfg.init_options.semanticTokens)) + +mcppls.setup({ server = server, semantic_tokens_modules = false, init_options = { semanticTokens = { modules = true, moduleType = false } } }) +cfg = mcppls.config() +check("a user's own init_options.semanticTokens wins outright", cfg.init_options.semanticTokens.modules == true and cfg.init_options.semanticTokens.moduleType == false, + vim.inspect(cfg.init_options.semanticTokens)) +check("conflictArbitration stays forced even with a user's own init_options", cfg.init_options.conflictArbitration == 'client', cfg.init_options.conflictArbitration) + +-- Back to the plugin's defaults for the checks below. +mcppls.setup({ server = server }) +check('nothing was shown while checking init_options', #shown == 0, vim.inspect(shown)) + +-- `@lsp.type.module` and `@lsp.type.keyword` are `default` links, set at setup and put back on +-- every ColorScheme (colorschemes clear existing links when they load), so a colorscheme or the +-- user's own nvim_set_hl wins over them. +local function hl(name) + return vim.api.nvim_get_hl(0, { name = name }) +end +check('@lsp.type.module links to @module by default', hl('@lsp.type.module').link == '@module', vim.inspect(hl('@lsp.type.module'))) +check('@lsp.type.keyword links to @keyword', hl('@lsp.type.keyword').link == '@keyword', vim.inspect(hl('@lsp.type.keyword'))) + +local user_color = tonumber('0x123456') +vim.api.nvim_set_hl(0, '@lsp.type.module', { fg = user_color }) +vim.api.nvim_exec_autocmds('ColorScheme', { modeline = false }) +check("a user's own @lsp.type.module survives the ColorScheme default being reapplied", + hl('@lsp.type.module').fg == user_color, vim.inspect(hl('@lsp.type.module'))) + +-- A second C++ server on the same buffer: an in-process stand-in named clangd/ccls. `notify('exit')` +-- drives the dispatcher's on_exit the way a real server's exit would, so a full `client:stop()` +-- is observable the same way a real one is. +local function fake_server(dispatchers) + local function exit() + if dispatchers and dispatchers.on_exit then + vim.schedule(function() dispatchers.on_exit(0, 0) end) + end + end return { request = function(method, _, callback) if method == 'initialize' then @@ -189,18 +235,56 @@ local function fake_server() end return true, 1 end, - notify = function() return true end, + notify = function(method) + if method == 'exit' then exit() end + return true + end, is_closing = function() return false end, - terminate = function() end, + terminate = exit, } end +local function stop_fake(c) + if vim.fn.has('nvim-0.11') == 1 then c:stop(true) else c.stop(true) end +end + +-- The plugin says so, once, and names it (disable_conflicting stays the default, false). vim.lsp.start({ name = 'clangd', cmd = fake_server, root_dir = workspace }, { bufnr = 0 }) vim.wait(10000, function() return #shown > 0 end, 100) check('a second C++ server is named, once', #shown == 1 and shown[1]:find('clangd', 1, true) ~= nil, vim.inspect(shown)) +check('and it mentions disable_conflicting', shown[1] and shown[1]:find('disable_conflicting', 1, true) ~= nil, vim.inspect(shown)) vim.lsp.start({ name = 'clangd', cmd = fake_server, root_dir = workspace .. sep .. 'other' }, { bufnr = 0 }) vim.wait(2000) check('and only once', #shown == 1, #shown) +-- Clear both stand-in `clangd` clients before the disable_conflicting checks below, so they do not +-- also get caught by the next LspAttach on this buffer. +local get = vim.lsp.get_clients or vim.lsp.get_active_clients +for _, c in ipairs(get({ name = 'clangd' })) do + stop_fake(c) +end +vim.wait(5000, function() return #get({ name = 'clangd' }) == 0 end, 100) + +-- disable_conflicting = true: the conflicting client is stopped for the buffer instead, and the +-- plugin still says so, once, naming it. A client that serves only buffers mcppls also serves is +-- stopped outright; one that serves another buffer too is only detached from this one (README). +local shown_before = #shown +mcppls.setup({ server = server, disable_conflicting = true }) + +vim.lsp.start({ name = 'ccls', cmd = fake_server, root_dir = workspace .. sep .. 'ccls-solo' }, { bufnr = 0 }) +check('disable_conflicting stops a single-buffer conflicting client', + vim.wait(10000, function() return #get({ name = 'ccls' }) == 0 end, 100), + vim.inspect(get({ name = 'ccls' }))) +check('and says so once, naming it', #shown == shown_before + 1 and shown[#shown]:find('ccls', 1, true) ~= nil, vim.inspect(shown)) + +local scratch = vim.api.nvim_create_buf(false, true) +vim.lsp.start({ name = 'ccls', cmd = fake_server, root_dir = workspace .. sep .. 'ccls-multi' }, { bufnr = scratch }) +vim.lsp.start({ name = 'ccls', cmd = fake_server, root_dir = workspace .. sep .. 'ccls-multi' }, { bufnr = 0 }) +vim.wait(5000, function() return #shown > shown_before + 1 end, 100) +check('disable_conflicting only detaches a client that also serves another buffer', + vim.wait(5000, function() return #get({ bufnr = 0, name = 'ccls' }) == 0 end, 100) and #get({ bufnr = scratch, name = 'ccls' }) == 1, + string.format('buf0 count=%d scratch count=%d', #get({ bufnr = 0, name = 'ccls' }), #get({ bufnr = scratch, name = 'ccls' }))) +check('exactly one more notice, naming it', #shown == shown_before + 2 and shown[#shown]:find('ccls', 1, true) ~= nil, vim.inspect(shown)) + local c = client() if c then if vim.fn.has('nvim-0.11') == 1 then c:stop() else c.stop() end From d12a1e54e2e387f543725d2e99277cc4d0e1eb17 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Fri, 25 Sep 2026 06:37:06 +0800 Subject: [PATCH 07/26] docs: what the status means now, the typing freeze, spin recovery and --disable-workaround --- docs/30-settings.md | 1 + docs/50-troubleshooting.md | 34 ++++++++++++++++++++++++++++---- docs/zh-CN/30-settings.md | 1 + docs/zh-CN/50-troubleshooting.md | 12 +++++++++-- 4 files changed, 42 insertions(+), 6 deletions(-) diff --git a/docs/30-settings.md b/docs/30-settings.md index 93b4e02..904d275 100644 --- a/docs/30-settings.md +++ b/docs/30-settings.md @@ -50,6 +50,7 @@ Options that apply to every subcommand: | `--untrusted` | — | Run no build tool and no compiler | | `--no-discover` | — | Do not look for compilers; loose sources use the semantic kit | | `--log-level debug\|info\|warning\|error` | `info` | | +| `--disable-workaround WA-CLANGD-` | — | Turn off one of the registered workarounds for clangd's defects (repeatable), to see whether it is still needed; `mcppls report` lists them under `engines[].details.workarounds` | `print-environment` exists for the server itself: it is what the login shell is asked to run when `mcppls.toolEnvironment` is `auto`. diff --git a/docs/50-troubleshooting.md b/docs/50-troubleshooting.md index 7e85c18..e24067b 100644 --- a/docs/50-troubleshooting.md +++ b/docs/50-troubleshooting.md @@ -14,7 +14,7 @@ carrying what almost every question turns out to need: | `toolEnvironment` | Which environment build tools were started in, and the **names** of the variables that differ from the editor's (never the values) | | `toolRuns` | The last twenty external runs, each with its command, duration and outcome | | `plan` | What the engine was given: entries, stand-ins, what was left out and why | -| `engines` | clangd's state, restarts, files set aside | +| `engines` | clangd's state, restarts, files set aside, and `workarounds`: the clangd defects this server works around for this version | | `events` | A journal of the session | | `logTail` | The end of the log | @@ -40,9 +40,35 @@ your build. **A module does not compile.** Only what imports it, directly or not, is affected: those files are answered at once by mcppls's own engine (module navigation, symbols, `import` completion), carry one `module-failed` diagnostic on the import that leads to the failure, and are not sent to clangd until -the failed module's own source or command changes; everything else keeps clangd. The status says -*degraded* with "N modules cannot be prepared because M failed", never *preparing* for good. The -engine's `doomedModules` and `filesRoutedToOwnEngine` in the report list them. +the failed module's own source or command changes; everything else keeps clangd. This is a problem +in the code, so it is told where it is, as diagnostics: the status stays *ready* (listing +`modules-doomed`, category `code`) and is never *preparing* for good. The engine's `doomedModules` +and `filesRoutedToOwnEngine` in the report list them. + +**What the status says, and what it does not.** A mistake in your code (a missing `;`, an import +of a module nothing provides, a module that does not compile) is a diagnostic where it is, in the +Problems list; the status stays *ready*. *degraded* means the server lost something it would +otherwise give you, and names what and where: "clangd stopped responding on main.cpp", +"the workspace is not trusted", "the macOS SDK was not found". Each status issue carries a +`category` (`code`, `engine`, `environment`, `project`) saying whose problem it is; only the ones +other than `code` make the state *degraded*, and only once that has lasted three seconds, so a +condition that passes by itself never reaches the status bar. + +**The editor froze while typing an `import` (0.0.3 and earlier).** clangd 23.1 never finishes a +file in which a module name ends in `.` at the end of its line (`import hello.`, `export module a.`), +and every later version of the file waits behind it: typing any dotted import went through that +text. mcppls 0.0.4 gives clangd the line with `;` right after the dot instead, which clangd reports +at once (workaround `WA-CLANGD-001`); `engines[].details.workarounds` in the report lists it. + +**"clangd would not finish main.cpp".** A file's build ran past its budget — five times its own +last build, never under 20 s — while the editor waited on it: clangd will not finish it, busy or +not. The file is answered by mcppls's own engine, with module-level features, until its text +changes (the exact text clangd stopped on is never given back to it), and clangd is restarted at +once without it. The `events` journal has an `engine-spin` entry with the numbers. + +**Is a workaround still needed?** `--disable-workaround WA-CLANGD-` (repeatable) turns one off; +the log's first lines name the ones in use. Each has a canary in the conformance suite +(`workaround-canaries`) that fails once a clangd update fixes its defect. **The editor found a different compiler than my terminal.** `toolEnvironment.source` should say `login-shell`. If it says `editor`, the reason is in `toolEnvironment.reason` — an editor started diff --git a/docs/zh-CN/30-settings.md b/docs/zh-CN/30-settings.md index ee73d56..5c13f52 100644 --- a/docs/zh-CN/30-settings.md +++ b/docs/zh-CN/30-settings.md @@ -52,5 +52,6 @@ mcppls version | `--untrusted` | — | 不运行任何构建工具，也不运行编译器 | | `--no-discover` | — | 不查找编译器；零散源码使用语义工具包 | | `--log-level debug\|info\|warning\|error` | `info` | | +| `--disable-workaround WA-CLANGD-` | — | 关掉一个针对 clangd 缺陷登记的规避措施（可重复），用来确认它是否还有必要；`mcppls report` 在 `engines[].details.workarounds` 下列出它们 | `print-environment` 是给服务端自己用的：`mcppls.toolEnvironment` 为 `auto` 时，服务端让登录 shell 运行的就是这个命令。 diff --git a/docs/zh-CN/50-troubleshooting.md b/docs/zh-CN/50-troubleshooting.md index 80f3744..3f09262 100644 --- a/docs/zh-CN/50-troubleshooting.md +++ b/docs/zh-CN/50-troubleshooting.md @@ -15,7 +15,7 @@ | `toolEnvironment` | 构建工具是在哪个环境中启动的，以及与编辑器环境不同的那些变量的**名称**（不含值） | | `toolRuns` | 最近二十次外部运行，每条都带命令、耗时和结果 | | `plan` | 交给引擎的内容：条目、占位单元、被省略了什么以及原因 | -| `engines` | clangd 的状态、重启次数、被搁置的文件 | +| `engines` | clangd 的状态、重启次数、被搁置的文件，以及 `workarounds`：本服务端针对这个 clangd 版本规避的 clangd 缺陷 | | `events` | 本次会话的事件日志 | | `logTail` | 日志的末尾部分 | @@ -29,7 +29,15 @@ **原本正常，后来某个模块突然解析不了了。** `plan.standIns` 列出了没有任何单元提供的模块——mcppls 给它们分配占位单元，这样一个坏掉的模块不会拖垮项目的其余部分，日志里也会逐个写明模块名和原因。真正的问题在 `plan.issues` 里，或者在你的构建本身。 -**某个模块编译不过。** 受影响的只有直接或间接导入它的文件：这些文件由 mcppls 自己的引擎立即应答（模块跳转、符号、`import` 补全），在引向失败的那条 import 上带一条 `module-failed` 诊断，并且在失败模块自己的源码或编译命令变化之前不会再交给 clangd；其余文件照常由 clangd 应答。状态会显示 *degraded* 和"N modules cannot be prepared because M failed"，不会一直停在 *preparing*。报告里引擎的 `doomedModules` 和 `filesRoutedToOwnEngine` 会列出它们。 +**某个模块编译不过。** 受影响的只有直接或间接导入它的文件：这些文件由 mcppls 自己的引擎立即应答（模块跳转、符号、`import` 补全），在引向失败的那条 import 上带一条 `module-failed` 诊断，并且在失败模块自己的源码或编译命令变化之前不会再交给 clangd；其余文件照常由 clangd 应答。这是代码本身的问题，所以在出问题的地方以诊断的形式告诉你：状态保持 *ready*（列出类别为 `code` 的 `modules-doomed`），也不会一直停在 *preparing*。报告里引擎的 `doomedModules` 和 `filesRoutedToOwnEngine` 会列出它们。 + +**状态说明什么，不说明什么。** 代码里的错误（少了 `;`、import 了没有任何单元提供的模块、某个模块编译不过）以诊断的形式出现在出错的位置，也就是 Problems 列表里；状态保持 *ready*。*degraded* 表示服务端丢了本来能给你的功能，并说明丢了什么、在哪里：“clangd stopped responding on main.cpp”、“the workspace is not trusted”、“the macOS SDK was not found”。每个状态 issue 都带有 `category`（`code`、`engine`、`environment`、`project`），说明这是谁的问题；只有 `code` 以外的类别会让状态变成 *degraded*，而且要持续三秒才会显示，所以自己很快就会消失的情况不会出现在状态栏上。 + +**输入 `import` 时编辑器卡死（0.0.3 及更早版本）。** clangd 23.1 遇到模块名以 `.` 结尾、而且 `.` 就在行尾的文件（`import hello.`、`export module a.`）时永远处理不完，这个文件之后的所有版本都排在它后面等待；而输入任何带点的模块名都会经过这个状态。mcppls 0.0.4 改为把这一行在点后补上 `;` 再交给 clangd，clangd 会立即报告这个错误（规避措施 `WA-CLANGD-001`）；报告里的 `engines[].details.workarounds` 会列出它。 + +**“clangd would not finish main.cpp”。** 某个文件的构建超出了预算——该文件上次构建耗时的五倍，最少 20 秒——而编辑器还在等它：不管 clangd 忙不忙，它都不会完成这个文件了。这个文件改由 mcppls 自己的引擎应答（提供模块层面的功能），直到它的文本发生变化（让 clangd 卡住的那份文本永远不会再交给它），同时立即重启一个不带这个文件的 clangd。`events` 日志里有一条带具体数字的 `engine-spin`。 + +**某个规避措施还需要吗？** `--disable-workaround WA-CLANGD-`（可重复）可以关掉一个；日志开头几行会列出正在使用的规避措施。每个规避措施在一致性测试里都有一个对应的检测项（`workaround-canaries`），clangd 更新修好了对应缺陷后，这个检测项就会失败。 **编辑器找到的编译器和我终端里的不一样。** `toolEnvironment.source` 应该是 `login-shell`。如果是 `editor`，原因在 `toolEnvironment.reason` 里——从桌面项启动的编辑器不带任何 shell 配置。`mcppls.toolEnvironment` 控制这一行为。 From 943ec7f94c48e9a624431ff592d370043a4d5846 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Fri, 25 Sep 2026 06:37:29 +0800 Subject: [PATCH 08/26] chore(version): 0.0.4 --- editors/claude-code/.claude-plugin/marketplace.json | 2 +- editors/claude-code/mcppls-lsp/.claude-plugin/plugin.json | 2 +- editors/clion/gradle.properties | 2 +- editors/vscode/package-lock.json | 4 ++-- editors/vscode/package.json | 2 +- editors/zed/extension.toml | 2 +- mcpp.toml | 2 +- modules/base/src/version.cppm | 2 +- 8 files changed, 9 insertions(+), 9 deletions(-) diff --git a/editors/claude-code/.claude-plugin/marketplace.json b/editors/claude-code/.claude-plugin/marketplace.json index e6029a0..bbe46bf 100644 --- a/editors/claude-code/.claude-plugin/marketplace.json +++ b/editors/claude-code/.claude-plugin/marketplace.json @@ -11,7 +11,7 @@ "displayName": "C++ Modules Language Server", "source": "./mcppls-lsp", "description": "Registers mcppls as the language server for C and C++ sources, including C++20/23 named modules.", - "version": "0.0.3", + "version": "0.0.4", "author": { "name": "Sunrisepeak", "url": "https://github.com/Sunrisepeak/mcpp-language-server" diff --git a/editors/claude-code/mcppls-lsp/.claude-plugin/plugin.json b/editors/claude-code/mcppls-lsp/.claude-plugin/plugin.json index 435462e..172091e 100644 --- a/editors/claude-code/mcppls-lsp/.claude-plugin/plugin.json +++ b/editors/claude-code/mcppls-lsp/.claude-plugin/plugin.json @@ -1,7 +1,7 @@ { "name": "mcppls-lsp", "displayName": "C++ Modules Language Server", - "version": "0.0.3", + "version": "0.0.4", "description": "Registers mcppls as the language server for C and C++ sources, including C++20/23 named modules, and its MCP tools (symbols, references, modules, verification, review). Replaces clangd-lsp for a project; do not enable both at once.", "author": { "name": "Sunrisepeak", diff --git a/editors/clion/gradle.properties b/editors/clion/gradle.properties index 22f476a..351542a 100644 --- a/editors/clion/gradle.properties +++ b/editors/clion/gradle.properties @@ -2,7 +2,7 @@ # ones within the same major line; sinceBuild/untilBuild in plugin.xml is what actually gates it. platformType = CL platformVersion = 2025.2 -pluginVersion = 0.0.3 +pluginVersion = 0.0.4 org.gradle.jvmargs = -Xmx2g # The IDE ships the Kotlin standard library; bundling a second copy in the plugin is what JetBrains # asks plugins not to do. diff --git a/editors/vscode/package-lock.json b/editors/vscode/package-lock.json index e818c38..e714e74 100644 --- a/editors/vscode/package-lock.json +++ b/editors/vscode/package-lock.json @@ -1,12 +1,12 @@ { "name": "mcpp-language-server", - "version": "0.0.3", + "version": "0.0.4", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "mcpp-language-server", - "version": "0.0.3", + "version": "0.0.4", "license": "Apache-2.0", "dependencies": { "vscode-languageclient": "^10.1.1" diff --git a/editors/vscode/package.json b/editors/vscode/package.json index 65cd3ff..be77357 100644 --- a/editors/vscode/package.json +++ b/editors/vscode/package.json @@ -2,7 +2,7 @@ "name": "mcpp-language-server", "displayName": "C++ Modules Language Server", "description": "C++20/23 named modules that just work: go to definition, completion, hover and references across modules for any compiler, with clangd and a standard library kit built in. (mcppls)", - "version": "0.0.3", + "version": "0.0.4", "publisher": "sunrisepeak", "license": "Apache-2.0", "icon": "icon.png", diff --git a/editors/zed/extension.toml b/editors/zed/extension.toml index 44f3344..72fe2ec 100644 --- a/editors/zed/extension.toml +++ b/editors/zed/extension.toml @@ -1,6 +1,6 @@ id = "mcppls" name = "C++ Modules Language Server" -version = "0.0.3" +version = "0.0.4" schema_version = 1 description = "C++20 named modules that work on any compiler: navigation, completion and diagnostics from mcpp-language-server" repository = "https://github.com/Sunrisepeak/mcpp-language-server" diff --git a/mcpp.toml b/mcpp.toml index f9b9662..2dd7198 100644 --- a/mcpp.toml +++ b/mcpp.toml @@ -34,7 +34,7 @@ libarchive = "3.8.7" [package] name = "mcpp-language-server" -version = "0.0.3" +version = "0.0.4" description = "Compiler-agnostic C++ modules language server" license = "Apache-2.0" authors = ["Sunrisepeak"] diff --git a/modules/base/src/version.cppm b/modules/base/src/version.cppm index 81c03d8..fa142c5 100644 --- a/modules/base/src/version.cppm +++ b/modules/base/src/version.cppm @@ -9,7 +9,7 @@ export namespace mcppls::base { // checked against mcpp.toml (the one source) by `mcppls-devtools version --check`, not kept in step by // hand. Three constants that lived here and nothing read were removed rather than left to drift: // the S1 profile version is spec::PROFILE_VERSION, the kit manifest version is spec::KIT_VERSION. -inline constexpr std::string_view VERSION { "0.0.3" }; +inline constexpr std::string_view VERSION { "0.0.4" }; // The clangd the payload ships. Checked against packaging/payload.lock.json by the same command. inline constexpr std::string_view CLANGD_VERSION { "23.1.0" }; // The oldest mcpp that answers `mcpp emit build-database` — the `mcpp.build-database` kind, which From 2217284b2f9e71d52671a02ccb064e2834201fd3 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Fri, 25 Sep 2026 06:38:10 +0800 Subject: [PATCH 09/26] docs(design): the import-hang plan as it was built --- ...2026-09-25-import-hang-status-highlight.md | 78 ++++++++++++------- 1 file changed, 48 insertions(+), 30 deletions(-) diff --git a/.agents/docs/2026-09-25-import-hang-status-highlight.md b/.agents/docs/2026-09-25-import-hang-status-highlight.md index 17f6b93..c5202d8 100644 --- a/.agents/docs/2026-09-25-import-hang-status-highlight.md +++ b/.agents/docs/2026-09-25-import-hang-status-highlight.md @@ -1,6 +1,6 @@ # Typing `import a.` freezes the editor; what the status means; highlighting `import` -Status: proposal for review · measured 2026-09-25 on mcppls 0.0.3 (linux-x64 payload, bundled clangd +Status: implemented in 0.0.4 (§13 records what was built and measured) · measured 2026-09-25 on mcppls 0.0.3 (linux-x64 payload, bundled clangd 23.1.0 `ea7d852a`), `main` at c508e62, project `~/test/mcpp/hello` | # | Item | Proposal | Section | @@ -143,10 +143,13 @@ even while the user keeps typing.** - **Budget the work that should be short.** clangd reports what each file is doing (`fileStatus_`). - Building a preamble or modules can take minutes. It keeps today's patience and progress checks. - A main-file AST build on a reused preamble is short: milliseconds for `hello`. clangd is spinning - on the file when all of these hold: - - the file's state has not changed for its budget; - - the CPU is busy; - - a newer version of the same file is waiting. + on the file when both of these hold: + - the build ("parsing main file") has lasted past its budget; + - something waits on it: a newer version of the file, or a request about it, sent after the + version being built. + - **No CPU condition** (changed while building it). Busy or idle, a build five times longer than + the file's own history, with the editor waiting on it, is not going to end. Idle stalls are + also still caught by StuckWatch. - **Budget.** `max(20 s, 5 × the file's last successful AST build)`. A heavy template file that really takes 15 s is not called stuck. The numbers are a starting point, to tune on the real-project stress runs (0.0.2 plan). @@ -155,8 +158,7 @@ even while the user keeps typing.** - **An edit to the file itself is not a rebuild.** Split `changed_recently_`: - Edits to module sources in the file's import closure keep `GENERAL_PATIENCE`. - An edit to the file itself gets a short grace, about 10 s. -- **One file is enough.** If every unanswered request is for one file, clangd answers nothing else - for `STALL_WINDOW`, and the CPU is busy, that file is stuck. Today this needs 2 files. +- **One file is enough.** The spin check is per file: no second file has to go unanswered. - **Recovery.** - Set the file aside and restart at once (no deferral). - Remember a hash of the text clangd hung on. That exact text never goes back to clangd; any other @@ -166,9 +168,11 @@ even while the user keeps typing.** - the file that caused the restarts stays with mcppls's engine until its text changes; - clangd is restarted once more, without that file; - clangd is never left spinning. + - A spin needs a new build to reach its budget (at least 20 s), so restarts cannot come faster + than one per 20 s. - **Meanwhile, native answers.** While the file is aside, the native engine answers (module - navigation, `import` completion, module diagnostics). Timed-out requests fall back to it instead - of an empty answer where it can answer. + navigation, `import` completion, module diagnostics). Timed-out requests already fell back to it + (`Answer {}` means "unavailable", and the next engine answers), so nothing had to change there. ## 5. H3: plan hygiene for half-typed imports @@ -179,18 +183,29 @@ even while the user keeps typing.** - Report this to mcpp. - **No stand-in for an import that is still being typed.** A name that does not resolve in a file the user is editing is usually a typo in progress. - - Report it as a diagnostic on the import, e.g. "module `hello` not found; did you mean - `hello.greet`?". - - Create a stand-in only after the text has been stable for a while, or for imports in files that - are not open. - - **Exception: a unit that provides a module** (`.cppm`, partitions). Building such a unit with an - import that does not resolve is what deadlocks clangd 23.1 (`plan.cpp`, "building it is what - deadlocks"). It keeps today's rule, a stand-in or leaving it out (`WA-CLANGD-002`, §9). A plain - source file such as `main.cpp` does not need one: without a stand-in, clangd just reports the - module as not found. -- **To investigate: the 21:15:23 restart during an import edit** ("units are compiled with other - arguments"). Changing a file's imports should not restart clangd. Add a test once the cause is - found; this analysis did not confirm it. + - The diagnostic already exists: the native index publishes `unresolved-module` ("module 'x' not + found") on the import. + - A file changed in the editor within the last **5 s** gets no stand-in for such an import. The + plan reports that it held one back, and the workspace plans again once the file is quiet. + - **Exception: a unit that provides a module** (`.cppm`, partitions) gets its stand-in at once. + Building such a unit with an import that does not resolve is what deadlocks clangd 23.1 + (`plan.cpp`, "building it is what deadlocks"; `WA-CLANGD-002`, §9). + - **Why not "never for a plain `.cpp`".** That was the first version, and it was dropped: + - Measured on `hello` (8 database entries), clangd answers a plain file with an unresolved + import normally. + - But the plan's own tests record xlings' `apps/gui/main.cpp` keeping a core busy without a + stand-in. + - Without a name to look up, clangd 23.1 scans the whole database for the module's unit, which + on a real project is seconds per lookup. + - So only the editing window is exempt. +- **The 21:15:23 restart ("units are compiled with other arguments") was not reproduced.** Typing + and saving `import hello.greet;` key by key restarts nothing: + - 0.0.3 restarted only to reclaim the spin; + - 0.0.4 restarts nothing (`typing-import` asserts it). + + It came 13 s after the session started, when the plan made from the cached model was followed by + the producer's. It belongs to that startup, not to import edits. It stays open, with no claim + about its cause. ## 6. S: the status must say *whose* problem it is @@ -410,16 +425,19 @@ records why each one exists or when it can go. struct Workaround { std::string_view id; // "WA-CLANGD-001": grep-able, used in comments, logs, report std::string_view title; // what it works around, one line - std::string_view engine; // "clangd" - VersionRange affects; // data, e.g. [23.1.0, 24.0.0); compared with the running engine's version - std::string_view upstream; // issue / fix commit URL, or "unfiled" (then filing is a to-do) - std::string_view evidence; // doc section and fixture, e.g. ".agents/docs/2026-09-25-...#1" - std::string_view added; // mcppls version and date - std::string_view removeWhen; // the condition, e.g. "bundled and minimum clangd contain 6dcfc17b1b" - std::string_view canary; // the test that fails once the upstream bug is gone + std::string_view fixedIn; // first release of KNOWN_LINE ("23.1") that no longer needs it; empty: none yet + std::string_view upstream; // issue / fix commit, or "unfiled" + std::string_view evidence; // doc section and fixtures + std::string_view added; // mcppls version + std::string_view removeWhen; // the condition + std::string_view canary; // the check that fails once the upstream bug is gone; empty: none yet }; ``` + A version of the pinned line (23.1) needs an entry until its `fixedIn`. A version of any other line + has not been through the conformance suite, so it gets every workaround, as the traits table did + before. `--disable-workaround` turns one off. + - **Gating comes from the registry.** `traits_for_version` sets its bools from the entries that apply to the engine's version, so the registry is the only list. - **One code path per workaround.** The code lives in the module, for example @@ -440,9 +458,9 @@ records why each one exists or when it can go. | ID | What | Affects | Remove when | Canary | |---|---|---|---|---| -| WA-CLANGD-001 | Same-line `;` after a trailing-dot module name (§3) | clangd [23.1.0, 24): 23.1.0 measured, .1/.2 assumed from commit titles | the bundled and minimum supported clangd contain the fix (bisect; candidate `6dcfc17b1b`) | `--check` on `import hello.` times out | +| WA-CLANGD-001 | Same-line `;` after a trailing-dot module name (§3) | every 23.1.x (none fixed; 23.1.0 measured, .1/.2 by commit titles), and every untested line | the bundled and minimum supported clangd contain the fix (bisect; candidate `6dcfc17b1b`) | `--check` on `import hello.` times out | | WA-VSCODE-001 | Module-syntax injection grammar (§7 layer 1) | VS Code built-in cpp grammar `071dd6e` | the built-in grammar colors `import std;` | tmgrammar test without the injection | -| WA-CLANGD-002..005 | Existing: stand-ins for unresolved imports (`hangsOnUnresolvedImports`), module preparation, module hints, MSVC STL aligned allocation | clangd 23.1.x | to be written when moved: each needs its upstream reference and a canary | to be written | +| WA-CLANGD-002..005 | Existing, now registered: stand-ins for unresolved imports, module preparation, module hints, MSVC STL aligned allocation (005 `fixedIn` 23.1.1, llvm-project#218152) | 23.1.x (005: 23.1.0 only), and every untested line | written in each entry | none yet (an empty `canary`; a test requires one for entries added from 0.0.4 on) | **Not workarounds, kept out of the registry.** These stay whatever clangd does: - H2, the busy-without-progress guard (§4). It is the defense for the next unknown bug; its numbers From 5ca3d0f2aa7bcadc585e1178bf82af33a557abcc Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Fri, 25 Sep 2026 06:39:09 +0800 Subject: [PATCH 10/26] fix(engine): the spin log and the rebuild patience say what they now mean --- src/engine/clangd.cpp | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/engine/clangd.cpp b/src/engine/clangd.cpp index d3e3071..547ef47 100644 --- a/src/engine/clangd.cpp +++ b/src/engine/clangd.cpp @@ -1686,8 +1686,8 @@ class ClangdEngine final : public Engine { bool doomed_path_(std::string_view path) const { return !path.empty() && doomedFiles_.contains(base::path_key(path)); } - // Whether the file itself, or a source of any module it imports (transitively), changed within - // GENERAL_PATIENCE: what clangd is busy with is then this change, not this file being stuck. + // Whether the file itself changed within SELF_EDIT_GRACE, or a source of any module it imports (transitively) + // within GENERAL_PATIENCE: what clangd is busy with is then this change, not this file being stuck. bool changed_recently_(std::string_view path, Clock::time_point now) const { const auto touched = [&](std::string_view file, Clock::duration within) { const auto at = touchedAt_.find(base::path_key(file)); @@ -1906,7 +1906,7 @@ class ClangdEngine final : public Engine { bool quarantined_(std::string_view path) const { return !path.empty() && quarantine_.contains(base::path_key(path)); } // import-hang plan §4: clangd has been building a file far longer than it ever took while the editor has moved - // on. Whatever the cause (WA-CLANGD-001 was one, found in the field), the build will not end, so the file goes to + // on. Whatever the cause (the defect behind WA-CLANGD-001 was one, found in the field), the build will not end, so the file goes to // mcppls's engine with the text clangd spun on remembered, and clangd is restarted without it. void handle_spins_(Clock::time_point now) { for (const auto& spin : spin_.check(now)) { @@ -1916,7 +1916,7 @@ class ClangdEngine final : public Engine { } if (path.empty() || quarantined_(path)) continue; const auto seconds = [](std::chrono::milliseconds duration) { return std::chrono::duration(duration).count(); }; - log::warning("clangd ({}) has built {} for {:.0f} s, past its {:.0f} s budget, while newer versions waited: it will not finish it", + log::warning("clangd ({}) has built {} for {:.0f} s, past its {:.0f} s budget, while the editor waited on it: it will not finish it", host_->root_directory(), base::file_name(path), seconds(spin.building), seconds(spin.budget)); host_->record_event("engine-spin", Json { { "file", path }, { "buildingSeconds", seconds(spin.building) }, { "budgetSeconds", seconds(spin.budget) } }); set_aside_(path, "clangd would not finish building it", Reclaim::now, false, spin.textHash); From b50fe76d53a6df1412016012811be7bff97d80c9 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Fri, 25 Sep 2026 06:35:44 +0800 Subject: [PATCH 11/26] feat(vscode): highlight module syntax at once, name why the status is limited, and let people manage other C++ extensions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three pieces of the 0.0.4 import-hang-status-highlight design (§6, §7, §9, §10), scoped to editors/vscode: the server-side halves (issue categories, module semantic tokens) are separate, parallel tracks. - An injected grammar (syntaxes/mcppls-modules.tmLanguage.json, WA-VSCODE-001 in the new src/workarounds.ts registry) colors module, import, export and module names on every keystroke, with no server involved, because VS Code's own cpp grammar defines the rule for this but never uses it. It accepts a name that is still being typed (import hello. colors import and hello) and never matches import or module used as an ordinary identifier. - package.json declares the semanticTokenTypes/Modifiers/Scopes contributions and the mcppls.semanticTokens.modules setting the S3 extension's semantic-token contract needs, and the client now sends initializationOptions.semanticTokens accordingly, so the server's own module semantic tokens (a parallel track) have somewhere to land. - The status bar no longer says the generic "Some features are limited": src/statusText.ts (kept free of vscode, like serverLog.ts, for unit testing) picks the first non-code issue's message instead, shortened for the bar and in full in the tooltip/detail, and skips any issue categorized "code" (the user's own text being wrong, which becomes a diagnostic instead). An issue with no category -- an older server -- is treated as non-code, as every issue was before this field existed. The error state keeps its sentence and appends why. - mcppls.turnOffOtherCppFeatures and mcppls.restoreOtherCppFeatures (src/conflicts.ts, src/conflictCandidates.ts) let a person turn other C++ extensions' language features off, or put them back, in this workspace or everywhere, at any time -- not only through the one-time question. A conflict becoming active again after activation (reinstalled, re-enabled, or its setting turned back on) gets a non-modal notice, once per conflict per session. ccls and any other extension with no enable setting can only be pointed at, in the Extensions view. Nothing here changes another extension's settings without the user choosing it, and every path is reversible. --- editors/vscode/package.json | 50 ++++ editors/vscode/src/commands.ts | 3 + editors/vscode/src/conflictCandidates.ts | 87 +++++++ editors/vscode/src/conflicts.ts | 241 +++++++++++++++--- editors/vscode/src/extension.ts | 20 +- editors/vscode/src/prompt.ts | 19 +- editors/vscode/src/status.ts | 50 ++-- editors/vscode/src/statusText.ts | 90 +++++++ editors/vscode/src/workarounds.ts | 48 ++++ .../syntaxes/mcppls-modules.tmLanguage.json | 39 +++ 10 files changed, 584 insertions(+), 63 deletions(-) create mode 100644 editors/vscode/src/conflictCandidates.ts create mode 100644 editors/vscode/src/statusText.ts create mode 100644 editors/vscode/src/workarounds.ts create mode 100644 editors/vscode/syntaxes/mcppls-modules.tmLanguage.json diff --git a/editors/vscode/package.json b/editors/vscode/package.json index be77357..3dfda9c 100644 --- a/editors/vscode/package.json +++ b/editors/vscode/package.json @@ -77,6 +77,41 @@ ] } ], + "grammars": [ + { + "scopeName": "source.cpp.mcppls-modules", + "injectTo": [ + "source.cpp" + ], + "path": "./syntaxes/mcppls-modules.tmLanguage.json" + } + ], + "semanticTokenTypes": [ + { + "id": "module", + "superType": "namespace", + "description": "A C++ module or module partition name" + } + ], + "semanticTokenModifiers": [ + { + "id": "partition", + "description": "A module partition name, rather than a whole module" + } + ], + "semanticTokenScopes": [ + { + "language": "cpp", + "scopes": { + "keyword": [ + "keyword.control.cpp" + ], + "module": [ + "entity.name.namespace.module.cpp" + ] + } + } + ], "commands": [ { "command": "mcppls.selectContext", @@ -124,6 +159,16 @@ "command": "mcppls.runBuildToolInTerminal", "title": "Run the Build Tool in a Terminal", "category": "C++ Modules" + }, + { + "command": "mcppls.turnOffOtherCppFeatures", + "title": "Turn Off Other C++ Language Features", + "category": "C++ Modules" + }, + { + "command": "mcppls.restoreOtherCppFeatures", + "title": "Restore Other C++ Language Features", + "category": "C++ Modules" } ], "configuration": { @@ -171,6 +216,11 @@ "default": true, "description": "Offer once to turn off the language features of other C++ extensions in this workspace, so results are not shown twice." }, + "mcppls.semanticTokens.modules": { + "type": "boolean", + "default": true, + "description": "Module keywords and names from the server's semantic tokens. Turn this off to use only your own grammar or tree-sitter colors for module syntax." + }, "mcppls.trace.server": { "type": "string", "enum": [ diff --git a/editors/vscode/src/commands.ts b/editors/vscode/src/commands.ts index b149b79..0f74a60 100644 --- a/editors/vscode/src/commands.ts +++ b/editors/vscode/src/commands.ts @@ -2,6 +2,7 @@ import * as vscode from 'vscode'; import type { LanguageClient } from 'vscode-languageclient/node'; +import { restoreOtherCppFeatures, turnOffOtherCppFeatures } from './conflicts'; import { describeProfile, SemanticProfile } from './status'; export interface ServerAccess { @@ -276,5 +277,7 @@ export function registerCommands(context: vscode.ExtensionContext, access: Serve vscode.commands.registerCommand('mcppls.showLogs', () => access.showLogs()), vscode.commands.registerCommand('mcppls.collectReport', () => collectReport(access)), vscode.commands.registerCommand('mcppls.runBuildToolInTerminal', () => runBuildToolInTerminal(access)), + vscode.commands.registerCommand('mcppls.turnOffOtherCppFeatures', () => turnOffOtherCppFeatures(context, access.log)), + vscode.commands.registerCommand('mcppls.restoreOtherCppFeatures', () => restoreOtherCppFeatures(context, access.log)), ); } diff --git a/editors/vscode/src/conflictCandidates.ts b/editors/vscode/src/conflictCandidates.ts new file mode 100644 index 0000000..6d100f3 --- /dev/null +++ b/editors/vscode/src/conflictCandidates.ts @@ -0,0 +1,87 @@ +// Which other C++ extensions conflict with this one, and the pure part of deciding whether one is +// currently active. Kept free of `vscode` so the decision logic (design 2026-09-25 §10) is testable +// in plain Node, the same reason src/serverLog.ts is. +// +// Two shapes, because VS Code gives no way to disable a language feature that has no setting for +// it: a `SettableConflict` is turned off by writing its setting; an `UnsettableConflict` (for +// example ccls, which has no "enable" toggle) can only be pointed out, so the user disables the +// extension itself from the Extensions view. + +export interface SettableConflict { + readonly extensionId: string; + readonly displayName: string; + readonly section: string; + readonly key: string; + readonly disabledValue: unknown; +} + +export interface UnsettableConflict { + readonly extensionId: string; + readonly displayName: string; +} + +// mcpp-vscode (mcpp-community.mcpp-vscode) is deliberately absent: it has no language server, and +// its module grammar uses the same scopes as this extension's own, so nothing is shown twice. +export const SETTABLE_CANDIDATES: readonly SettableConflict[] = [ + { extensionId: 'ms-vscode.cpptools', displayName: 'C/C++', section: 'C_Cpp', key: 'intelliSenseEngine', disabledValue: 'disabled' }, + { extensionId: 'llvm-vs-code-extensions.vscode-clangd', displayName: 'clangd', section: 'clangd', key: 'enable', disabledValue: false }, +]; + +export const UNSETTABLE_CANDIDATES: readonly UnsettableConflict[] = [ + { extensionId: 'ccls-project.ccls', displayName: 'ccls' }, +]; + +// `isInstalled` and `currentValue` are the two things only VS Code can answer (installed *and* +// enabled extensions; the effective setting value); everything else here is a pure decision. +export function activeSettableConflicts( + isInstalled: (extensionId: string) => boolean, + currentValue: (section: string, key: string) => unknown, +): SettableConflict[] { + return SETTABLE_CANDIDATES.filter( + (candidate) => isInstalled(candidate.extensionId) && currentValue(candidate.section, candidate.key) !== candidate.disabledValue, + ); +} + +export function activeUnsettableConflicts(isInstalled: (extensionId: string) => boolean): UnsettableConflict[] { + return UNSETTABLE_CANDIDATES.filter((candidate) => isInstalled(candidate.extensionId)); +} + +// The ids of every conflict active right now, settable or not -- what "a newly active conflict" +// (design §10) is compared against from one recheck to the next. +export function activeConflictIds( + isInstalled: (extensionId: string) => boolean, + currentValue: (section: string, key: string) => unknown, +): Set { + const ids = new Set(); + for (const conflict of activeSettableConflicts(isInstalled, currentValue)) { + ids.add(conflict.extensionId); + } + for (const conflict of activeUnsettableConflicts(isInstalled)) { + ids.add(conflict.extensionId); + } + return ids; +} + +// --- Restore bookkeeping ---------------------------------------------------- +// +// What "turn off" remembers before it changes a setting, so "restore" can put exactly that back -- +// including "there was no override at this scope", which is `value: undefined` and means the +// restore removes the override rather than writing a value that merely matches today's default. + +export type ConflictScope = 'workspace' | 'global'; + +export interface StoredConflictValue { + readonly section: string; + readonly key: string; + readonly value: unknown; +} + +// `valueAtScope` reads the setting's own override at exactly the scope being turned off (VS Code's +// `inspect(key).workspaceValue` / `.globalValue`), not the effective value `.get()` would return, +// so a setting with no override here is remembered as `undefined` rather than as its default. +export function snapshotForDisable( + conflicts: readonly SettableConflict[], + valueAtScope: (section: string, key: string) => unknown, +): StoredConflictValue[] { + return conflicts.map((conflict) => ({ section: conflict.section, key: conflict.key, value: valueAtScope(conflict.section, conflict.key) })); +} diff --git a/editors/vscode/src/conflicts.ts b/editors/vscode/src/conflicts.ts index f67edf7..527e924 100644 --- a/editors/vscode/src/conflicts.ts +++ b/editors/vscode/src/conflicts.ts @@ -1,9 +1,26 @@ -// Another C++ extension serving the same files shows every result twice. -// Ask once per workspace whether to turn their language features off here. +// Another C++ extension serving the same files shows every result twice (design 2026-09-25 §10). +// Once per workspace, this asks whether to turn their language features off here; two commands let +// the user do that (or undo it) at any time, and a listener notices when a conflict becomes active +// after activation and says so once, without asking anything or changing a setting on its own. +// +// Never change another extension's settings without the user choosing it: every path below that +// writes a setting is reached only through an explicit answer to a question or a command the user ran. import * as vscode from 'vscode'; +import { + activeConflictIds, + activeSettableConflicts, + activeUnsettableConflicts, + ConflictScope, + SETTABLE_CANDIDATES, + SettableConflict, + snapshotForDisable, + StoredConflictValue, + UNSETTABLE_CANDIDATES, + UnsettableConflict, +} from './conflictCandidates'; import { DISABLE, KEEP } from './conflictAnswers'; -import { askOnce } from './prompt'; +import { askOnce, pickOnce } from './prompt'; // Re-exported so the rest of this file, and existing importers of the // answers from here (the conflicts scenario's test suite), have one place @@ -11,31 +28,66 @@ import { askOnce } from './prompt'; export { DISABLE, KEEP }; export const CONFLICT_ANSWER_KEY = 'mcppls.conflictAnswer'; +const STORAGE_KEY = 'mcppls.conflictPreviousValues'; export type ConflictCheck = 'skipped-setting' | 'already-answered' | 'none-found' | 'asked'; -interface Conflict { - extensionId: string; - displayName: string; - section: string; - key: string; - disabledValue: unknown; +function isInstalled(extensionId: string): boolean { + // getExtension answers only for installed extensions that are enabled. + return vscode.extensions.getExtension(extensionId) !== undefined; } -const CANDIDATES: readonly Conflict[] = [ - { extensionId: 'ms-vscode.cpptools', displayName: 'C/C++', section: 'C_Cpp', key: 'intelliSenseEngine', disabledValue: 'disabled' }, - { extensionId: 'llvm-vs-code-extensions.vscode-clangd', displayName: 'clangd', section: 'clangd', key: 'enable', disabledValue: false }, -]; +function currentValue(section: string, key: string): unknown { + return vscode.workspace.getConfiguration(section).get(key); +} + +// The override a setting has *at one specific scope*, as opposed to `currentValue`'s effective +// value -- so "there was no override here" is remembered as `undefined`, not as whatever the +// default happens to be, and restoring it means removing the override rather than reapplying a +// value that merely matched today's default. +function valueAtScope(section: string, key: string, scope: ConflictScope): unknown { + const inspected = vscode.workspace.getConfiguration(section).inspect(key); + return scope === 'workspace' ? inspected?.workspaceValue : inspected?.globalValue; +} -function activeConflicts(): Conflict[] { - return CANDIDATES.filter((candidate) => { - // getExtension answers only for installed extensions that are enabled. - if (!vscode.extensions.getExtension(candidate.extensionId)) { - return false; +function storeFor(context: vscode.ExtensionContext, scope: ConflictScope): vscode.Memento { + return scope === 'workspace' ? context.workspaceState : context.globalState; +} + +function targetFor(scope: ConflictScope): vscode.ConfigurationTarget { + return scope === 'workspace' ? vscode.ConfigurationTarget.Workspace : vscode.ConfigurationTarget.Global; +} + +function scopeLabel(scope: ConflictScope): string { + return scope === 'workspace' ? 'this workspace' : 'user settings'; +} + +// Writes each conflict's disabled value at `scope`, remembering what was there before so +// `restoreOtherCppFeatures` (or the command's own undo, for restore) can put it back exactly. +async function disableConflicts( + context: vscode.ExtensionContext, + conflicts: readonly SettableConflict[], + scope: ConflictScope, + log: (line: string) => void, +): Promise { + if (conflicts.length === 0) { + return; + } + const store = storeFor(context, scope); + const snapshot = snapshotForDisable(conflicts, (section, key) => valueAtScope(section, key, scope)); + const existing = store.get(STORAGE_KEY) ?? []; + // A conflict disabled again before being restored keeps its ORIGINAL previous value, not the + // disabled value it is about to be overwritten with. + const kept = existing.filter((entry) => !snapshot.some((next) => next.section === entry.section && next.key === entry.key)); + await store.update(STORAGE_KEY, [...kept, ...snapshot]); + for (const conflict of conflicts) { + try { + await vscode.workspace.getConfiguration(conflict.section).update(conflict.key, conflict.disabledValue, targetFor(scope)); + log(`Set ${conflict.section}.${conflict.key} to ${JSON.stringify(conflict.disabledValue)} (${scopeLabel(scope)}).`); + } catch (error) { + log(`Could not change ${conflict.section}.${conflict.key}: ${error instanceof Error ? error.message : String(error)}`); } - const value = vscode.workspace.getConfiguration(candidate.section).get(candidate.key); - return value !== candidate.disabledValue; - }); + } } // In test mode this still runs every real check (installed extensions, @@ -48,7 +100,7 @@ export async function checkConflicts(context: vscode.ExtensionContext, log: (lin if (context.workspaceState.get(CONFLICT_ANSWER_KEY) !== undefined) { return 'already-answered'; } - const conflicts = activeConflicts(); + const conflicts = activeSettableConflicts(isInstalled, currentValue); if (conflicts.length === 0) { return 'none-found'; } @@ -63,17 +115,146 @@ export async function checkConflicts(context: vscode.ExtensionContext, log: (lin ); // Closing the message counts as an answer too: the question is asked once. await context.workspaceState.update(CONFLICT_ANSWER_KEY, answer === DISABLE ? 'disabled' : answer === KEEP ? 'kept' : 'dismissed'); - if (answer !== DISABLE) { - return 'asked'; + if (answer === DISABLE) { + await disableConflicts(context, conflicts, 'workspace', log); } - for (const conflict of conflicts) { + return 'asked'; +} + +const SCOPE_ITEMS: readonly { label: string; value: ConflictScope }[] = [ + { label: 'This Workspace', value: 'workspace' }, + { label: 'Everywhere (User Settings)', value: 'global' }, +]; + +// Command: `mcppls: Turn Off Other C++ Language Features`. The first-run question (checkConflicts, +// above) is a shortcut to this, always at workspace scope; this command is the same action, chosen +// deliberately, with the scope the user picks. +export async function turnOffOtherCppFeatures(context: vscode.ExtensionContext, log: (line: string) => void): Promise { + const scope = await pickOnce('turnOffScope', SCOPE_ITEMS, 'Where to turn off other C++ extensions’ language features'); + if (!scope) { + return; + } + const settable = activeSettableConflicts(isInstalled, currentValue); + const unsettable = activeUnsettableConflicts(isInstalled); + if (settable.length === 0 && unsettable.length === 0) { + void vscode.window.showInformationMessage('mcppls: no other C++ language features are currently active.'); + return; + } + await disableConflicts(context, settable, scope, log); + if (settable.length > 0) { + void vscode.window.showInformationMessage( + `mcppls: turned off ${settable.map((conflict) => conflict.displayName).join(' and ')} (${scopeLabel(scope)}).`, + ); + } + for (const conflict of unsettable) { + offerToOpenExtension(conflict, log); + } +} + +// Command: `mcppls: Restore Other C++ Language Features`. Puts back exactly what +// turnOffOtherCppFeatures (or the first-run question) remembered, per scope. +export async function restoreOtherCppFeatures(context: vscode.ExtensionContext, log: (line: string) => void): Promise { + const scopesWithData = (['workspace', 'global'] as const) + .filter((scope) => (storeFor(context, scope).get(STORAGE_KEY) ?? []).length > 0); + if (scopesWithData.length === 0) { + void vscode.window.showInformationMessage('mcppls: nothing to restore.'); + return; + } + let scope: ConflictScope; + if (scopesWithData.length === 1) { + scope = scopesWithData[0]; + } else { + const picked = await pickOnce( + 'restoreScope', + SCOPE_ITEMS.filter((item) => scopesWithData.includes(item.value)), + 'Which scope to restore', + ); + if (!picked) { + return; + } + scope = picked; + } + const store = storeFor(context, scope); + const saved = store.get(STORAGE_KEY) ?? []; + const target = targetFor(scope); + for (const item of saved) { try { - await vscode.workspace.getConfiguration(conflict.section) - .update(conflict.key, conflict.disabledValue, vscode.ConfigurationTarget.Workspace); - log(`Set ${conflict.section}.${conflict.key} to ${JSON.stringify(conflict.disabledValue)} in the workspace settings.`); + await vscode.workspace.getConfiguration(item.section).update(item.key, item.value, target); + log(`Restored ${item.section}.${item.key} to ${JSON.stringify(item.value)} (${scopeLabel(scope)}).`); } catch (error) { - log(`Could not change ${conflict.section}.${conflict.key}: ${error instanceof Error ? error.message : String(error)}`); + log(`Could not restore ${item.section}.${item.key}: ${error instanceof Error ? error.message : String(error)}`); } } - return 'asked'; + await store.update(STORAGE_KEY, undefined); + void vscode.window.showInformationMessage(`mcppls: restored other C++ extensions’ language features (${scopeLabel(scope)}).`); +} + +function offerToOpenExtension(conflict: UnsettableConflict, log: (line: string) => void): void { + void vscode.window.showInformationMessage( + `mcppls: ${conflict.displayName} has no setting to turn off; disable the extension itself from the Extensions view.`, + 'Open Extension', + ).then((choice) => { + if (choice !== 'Open Extension') { + return; + } + log(`Opening the Extensions view for ${conflict.extensionId}.`); + void vscode.commands.executeCommand('workbench.extensions.search', `@id:${conflict.extensionId}`); + }); +} + +// A non-modal notice -- an information message, never a warning or error modal -- for a conflict +// that was not active when this session last looked. Shown at most once per conflict per session: +// see watchForNewConflicts below for how "last looked" is tracked. +function notifyNewConflict(context: vscode.ExtensionContext, extensionId: string, log: (line: string) => void): void { + const settable = SETTABLE_CANDIDATES.find((candidate) => candidate.extensionId === extensionId); + if (settable) { + void vscode.window.showInformationMessage( + `mcppls: ${settable.displayName} is also active: results may appear twice.`, + 'Turn Off', + ).then((choice) => { + if (choice === 'Turn Off') { + void disableConflicts(context, [settable], 'workspace', log); + } + }); + return; + } + const unsettable = UNSETTABLE_CANDIDATES.find((candidate) => candidate.extensionId === extensionId); + if (unsettable) { + offerToOpenExtension(unsettable, log); + } +} + +// Listens for a conflict becoming active after activation -- another C++ extension installed or +// enabled, or its setting turned back on -- and notices once per conflict per session. "Once" is +// tracked as a transition, not a permanent flag: a conflict that goes away and comes back is +// treated as newly active again, while repeated events while nothing has changed (many unrelated +// settings fire onDidChangeConfiguration too) notice nothing. +// +// The very first call only records what is active; it works whichever runs first, this call or an +// event firing on its own, because a transition needs a previous snapshot to compare against, and +// there is deliberately no notice for whatever was already active when this session started +// looking (that was already surfaced, or silently skipped, by checkConflicts above). +export function watchForNewConflicts(context: vscode.ExtensionContext, log: (line: string) => void): vscode.Disposable { + let known: Set | undefined; + + const recheck = (): void => { + if (!vscode.workspace.getConfiguration('mcppls').get('detectConflicts', true)) { + return; + } + const active = activeConflictIds(isInstalled, currentValue); + if (known !== undefined) { + for (const id of active) { + if (!known.has(id)) { + notifyNewConflict(context, id, log); + } + } + } + known = active; + }; + + recheck(); + return vscode.Disposable.from( + vscode.extensions.onDidChange(recheck), + vscode.workspace.onDidChangeConfiguration(recheck), + ); } diff --git a/editors/vscode/src/extension.ts b/editors/vscode/src/extension.ts index a6d85a5..f2b52f3 100644 --- a/editors/vscode/src/extension.ts +++ b/editors/vscode/src/extension.ts @@ -26,11 +26,12 @@ import { } from 'vscode-languageclient/node'; import { CommandLineToolsController, withInstallCommandFallback } from './commandLineTools'; import { registerCommands, reloadBuildDescription } from './commands'; -import { checkConflicts, ConflictCheck } from './conflicts'; +import { checkConflicts, ConflictCheck, watchForNewConflicts } from './conflicts'; import { resolveLaunch } from './payload'; import { ServerLogLevel, ServerLogRouter } from './serverLog'; import { promptTestHarness, PromptKind } from './prompt'; import { CxxModulesStatus, ModuleState, StatusController } from './status'; +import { describeActiveWorkarounds } from './workarounds'; // build description design 4.4: a value this extension does not know must not turn the network on. function buildToolSetting(value: string | undefined): string { @@ -250,6 +251,14 @@ class ServerHost implements vscode.Disposable { // from also explaining it: a server cannot see its siblings through LSP, so it tells // clients that arbitrate nothing — which is every editor but this one. conflictArbitration: 'client', + // Design 2026-09-25 §7/§12: `modules` is this setting; `moduleType` is fixed true + // because this extension always declares the custom `module` semantic token type + // (package.json contributes.semanticTokenTypes) with a `namespace` fallback for + // themes that do not colour it. + semanticTokens: { + modules: configuration.get('semanticTokens.modules', true), + moduleType: true, + }, }, errorHandler: { error: () => ({ action: ErrorAction.Continue, handled: true }), @@ -467,6 +476,13 @@ export function activate(context: vscode.ExtensionContext): TestApi { host = new ServerHost(context, status, commandLineTools, runConflictCheck); activeHost = host; context.subscriptions.push(status, host); + // Workaround registry design (§9): visible once per activation, so a bug report shows which of + // this extension's own workarounds (as opposed to the server's) were on. + host.log(describeActiveWorkarounds()); + // Coexistence design (§10): a conflict that becomes active after activation -- another C++ + // extension installed, enabled, or its setting turned back on -- gets a notice, once per + // conflict per session, distinct from the one-time question above. + context.subscriptions.push(watchForNewConflicts(context, (line) => host.log(line))); const serverAccess = { runningClient: () => host.runningClient(), @@ -480,7 +496,7 @@ export function activate(context: vscode.ExtensionContext): TestApi { vscode.workspace.onDidChangeConfiguration((event) => { if (event.affectsConfiguration('mcppls.compiler') || event.affectsConfiguration('mcppls.semanticKit') || event.affectsConfiguration('mcppls.engine') || event.affectsConfiguration('mcppls.buildTool') - || event.affectsConfiguration('mcppls.toolEnvironment')) { + || event.affectsConfiguration('mcppls.toolEnvironment') || event.affectsConfiguration('mcppls.semanticTokens.modules')) { void host.restart(); } }), diff --git a/editors/vscode/src/prompt.ts b/editors/vscode/src/prompt.ts index f0196f6..4ef09df 100644 --- a/editors/vscode/src/prompt.ts +++ b/editors/vscode/src/prompt.ts @@ -30,7 +30,9 @@ import * as vscode from 'vscode'; -export type PromptKind = 'conflict' | 'commandLineTools'; +// 'turnOffScope' and 'restoreScope' are the quick picks `mcppls.turnOffOtherCppFeatures` and +// `mcppls.restoreOtherCppFeatures` (src/conflicts.ts) show for which settings scope to act on. +export type PromptKind = 'conflict' | 'commandLineTools' | 'turnOffScope' | 'restoreScope'; const TEST_MODE = process.env.MCPPLS_TEST === '1'; const SUBSTITUTION_GRACE_MS = 5000; @@ -93,3 +95,18 @@ export function askOnce(kind: PromptKind, message: string, ...items: string[]): } return vscode.window.showInformationMessage(message, ...items); } + +// A quick pick among a small set of named values, with the same test-mode substitution as askOnce +// above (and for the same reason: nothing in the automated test host ever picks an item). `items` +// are shown in order; the resolved value is whichever item's label was chosen, or undefined for +// Escape or the grace-period default. +export function pickOnce( + kind: PromptKind, + items: readonly { label: string; value: T }[], + placeHolder: string, +): Thenable { + if (promptTestHarness) { + return promptTestHarness.ask(kind).then((answer) => items.find((item) => item.value === answer)?.value); + } + return vscode.window.showQuickPick(items, { placeHolder }).then((choice) => choice?.value); +} diff --git a/editors/vscode/src/status.ts b/editors/vscode/src/status.ts index 95a71f4..a471cf7 100644 --- a/editors/vscode/src/status.ts +++ b/editors/vscode/src/status.ts @@ -2,6 +2,7 @@ // status item for C++ files, driven by the server's cxxModules/status notification. import * as vscode from 'vscode'; +import { stateTexts } from './statusText'; export type ModuleState = 'starting' | 'loading' | 'preparing' | 'ready' | 'degraded' | 'error'; @@ -22,6 +23,10 @@ export interface ModuleIssue { code: string; message: string; command?: IssueCommand; + // S3 extension (design 2026-09-25 §6): "code" is the user's own text being wrong, shown as a + // diagnostic instead, and never colours the status. Absent means non-code, for a server from + // before this field existed; see statusText.ts. + category?: 'code' | 'engine' | 'environment' | 'project'; } export interface CxxModulesStatus { @@ -101,25 +106,6 @@ export function describeProfile(profile: SemanticProfile | undefined): string { return profile.compiler && profile.compiler.length > 0 ? profile.compiler : profile.stdlib ?? ''; } -function stateText(status: CxxModulesStatus): string | undefined { - switch (status.state) { - case 'starting': - return 'Starting'; - case 'loading': - return 'Loading the project'; - case 'preparing': - return status.progress && status.progress.total > 0 - ? `Preparing modules ${status.progress.done}/${status.progress.total}` - : 'Preparing modules'; - case 'ready': - return undefined; - case 'degraded': - return 'Some features are limited'; - case 'error': - return 'Only module-level features are available'; - } -} - export class StatusController implements vscode.Disposable { private readonly item: vscode.LanguageStatusItem; // A LanguageStatusItem lives behind the `{}` icon: a person has to go looking for it, and @@ -154,8 +140,10 @@ export class StatusController implements vscode.Disposable { this.paint('starting', detail); } - // The status bar half of the same state. - private paint(state: ModuleState | 'starting', detail: string | undefined): void { + // The status bar half of the same state. `detail` is the short label shown next to the icon; + // `tooltipDetail` (the full text, when it differs -- only `degraded` shortens anything, see + // statusText.ts) is what the tooltip shows, defaulting to `detail` when there is nothing fuller. + private paint(state: ModuleState | 'starting', detail: string | undefined, tooltipDetail: string | undefined = detail): void { const { text, background, foreground } = barFor(state, detail); const busy = BUSY_STATES.includes(state as ModuleState); this.bar.text = text; @@ -164,7 +152,7 @@ export class StatusController implements vscode.Disposable { // here, rather than resetting the colour, keeps one steady rhythm across those repaints // instead of restarting the cycle a few times a second. this.bar.color = busy ? this.pulseColor() : foreground; - this.bar.tooltip = detail ? `mcppls — ${detail}` : 'mcppls'; + this.bar.tooltip = tooltipDetail ? `mcppls — ${tooltipDetail}` : 'mcppls'; this.setPulsing(busy); } @@ -224,9 +212,10 @@ export class StatusController implements vscode.Disposable { this.item.text = label.length > 0 ? `C++ Modules · ${label}` : 'C++ Modules'; const details: string[] = []; - const state = stateText(status); - if (state) { - details.push(state); + const texts = stateTexts(status); + if (texts.full) { + // The full, unshortened text; see statusText.ts for why only `degraded` differs from `short`. + details.push(texts.full); } if (status.project) { const tier = status.project.tier; @@ -238,10 +227,9 @@ export class StatusController implements vscode.Disposable { details.push(`${status.engine.name} ${status.engine.version}`.trim()); } const issues = status.issues ?? []; - if ((status.state === 'degraded' || status.state === 'error') && issues.length > 0) { - details.push(issues[0].message); - } else if (status.notices && status.notices.length > 0) { + if (!texts.full && status.notices && status.notices.length > 0) { // Shown in the item's hover only: a notice changes neither the state nor the severity. + // Only when there is no issue text already in `details` above -- same precedence as before. details.push(status.notices[0].message); } this.item.detail = details.join(' · '); @@ -257,8 +245,10 @@ export class StatusController implements vscode.Disposable { ? { title: withCommand.command.title, command: withCommand.command.command, arguments: withCommand.command.arguments } : status.state === 'degraded' || status.state === 'error' ? COLLECT_REPORT : SHOW_LOGS; - // The status bar says the one thing that matters now; the item behind `{}` keeps the rest. - this.paint(status.state, stateText(status) ?? (describeProfile(status.profile) || undefined)); + // The status bar says the one thing that matters now, shortened when there is a fuller + // version in the tooltip; the item behind `{}` keeps the rest. + const shortDetail = texts.short ?? (describeProfile(status.profile) || undefined); + this.paint(status.state, shortDetail, texts.full ?? shortDetail); for (const waiter of [...this.waiters]) { if (waiter.states.includes(status.state)) { diff --git a/editors/vscode/src/statusText.ts b/editors/vscode/src/statusText.ts new file mode 100644 index 0000000..fa379d7 --- /dev/null +++ b/editors/vscode/src/statusText.ts @@ -0,0 +1,90 @@ +// How a `cxxModules/status` notification's issues become the words the status bar and the language +// status item show (design 2026-09-25 §6). Kept free of `vscode`, the same reason +// src/serverLog.ts is, so the categorisation and wording rules are testable in plain Node. + +export type IssueCategory = 'code' | 'engine' | 'environment' | 'project'; + +export interface StatusIssue { + code: string; + message: string; + category?: IssueCategory; +} + +// A `code` issue is the user's own text being wrong (an unterminated `import`, a name that does not +// resolve): it becomes a diagnostic on that range and never colours the status bar. Everything else, +// including an issue with no `category` at all (an older server, from before this field existed), +// is treated as non-code, exactly as every issue was before this field existed. +export function isCodeIssue(issue: StatusIssue): boolean { + return issue.category === 'code'; +} + +export function firstNonCodeIssue(issues: readonly StatusIssue[]): StatusIssue | undefined { + return issues.find((issue) => !isCodeIssue(issue)); +} + +// A sensible short form for the status bar's one line: the first sentence if that already fits, else +// a word-boundary truncation. The full message always stays available in the tooltip and the +// language status item's detail (`full` below), so nothing here is actually lost, only not shown twice. +export function shorten(message: string, maxLength = 72): string { + const trimmed = message.trim(); + if (trimmed.length <= maxLength) { + return trimmed; + } + // A sentence-ending punctuation mark counts only when followed by whitespace or the end of the + // string, so a dot inside a filename ("main.cpp") is not mistaken for the end of a sentence. + const ending = /[.!?;](?=\s|$)/.exec(trimmed); + const candidate = ending ? trimmed.slice(0, ending.index + 1) : trimmed; + if (candidate.length <= maxLength) { + return candidate; + } + const cut = candidate.slice(0, maxLength - 1); + const lastSpace = cut.lastIndexOf(' '); + const kept = lastSpace > maxLength / 3 ? cut.slice(0, lastSpace) : cut; + return `${kept.trimEnd()}…`; +} + +export interface StateTexts { + // The status bar's compact suffix; undefined when a state has none (ready, with no profile to show). + short: string | undefined; + // The tooltip and the language status item's detail; the full text behind `short`. Equal to + // `short` for every state except `degraded`, which is the only one that shortens anything. + full: string | undefined; +} + +function same(text: string | undefined): StateTexts { + return { short: text, full: text }; +} + +export interface StatusForText { + state: 'starting' | 'loading' | 'preparing' | 'ready' | 'degraded' | 'error'; + progress?: { done: number; total: number }; + issues?: readonly StatusIssue[]; +} + +// The generic "Some features are limited" wording is gone: `degraded` now names the first non-code +// issue instead. `error` keeps its fixed sentence -- module-level features really are all that is +// left, whatever the cause -- and appends why. +export function stateTexts(status: StatusForText): StateTexts { + switch (status.state) { + case 'starting': + return same('Starting'); + case 'loading': + return same('Loading the project'); + case 'preparing': + return same(status.progress && status.progress.total > 0 + ? `Preparing modules ${status.progress.done}/${status.progress.total}` + : 'Preparing modules'); + case 'ready': + return same(undefined); + case 'degraded': { + const issue = firstNonCodeIssue(status.issues ?? []); + const full = issue ? issue.message : 'Limited'; + return { short: issue ? shorten(issue.message) : full, full }; + } + case 'error': { + const base = 'Only module-level features are available'; + const issue = firstNonCodeIssue(status.issues ?? []); + return same(issue ? `${base}: ${issue.message}` : base); + } + } +} diff --git a/editors/vscode/src/workarounds.ts b/editors/vscode/src/workarounds.ts new file mode 100644 index 0000000..33196ba --- /dev/null +++ b/editors/vscode/src/workarounds.ts @@ -0,0 +1,48 @@ +// The extension's own workaround registry (design 2026-09-25 §9), mirroring the server's +// (src/engine/clangd/workarounds.cppm, a parallel track): one entry per fix that exists only +// because of a bug in someone else's code, so it is grep-able, named at startup, and has a canary +// test that fails once it can be removed. +// +// The one entry here today is WA-VSCODE-001: VS Code's own C++ grammar defines a `module_import` +// rule but never includes it from anywhere, so `import std;` gets no keyword scope at all (measured +// on better-cpp-syntax 071dd6e -- see the design doc, §7). syntaxes/mcppls-modules.tmLanguage.json +// is the fix: an injection grammar that colors the module syntax VS Code's grammar misses. Remove +// both this entry and that grammar file once the built-in grammar scopes `import std;` itself; the +// canary in test/suite/workaroundCanary.test.ts fails first, naming this id, when that happens. +// +// (JSON grammars cannot carry comments, which is why this note lives here rather than beside the +// `include` it is about.) + +export interface Workaround { + readonly id: string; + readonly title: string; + readonly affects: string; + readonly upstream: string; + readonly evidence: string; + readonly added: string; + readonly removeWhen: string; + readonly canary: string; +} + +export const WORKAROUNDS: readonly Workaround[] = [ + { + id: 'WA-VSCODE-001', + title: "an injected grammar colors `import`/`module`/`export` and module names, which VS Code's own cpp grammar does not", + affects: "VS Code's built-in cpp grammar (better-cpp-syntax 071dd6e): it defines module_import but no rule includes it", + upstream: 'unfiled', + evidence: '.agents/docs/2026-09-25-import-hang-status-highlight.md#7', + added: '0.0.4, 2026-09-25', + removeWhen: "the built-in cpp grammar scopes `import std;` itself", + canary: 'test/suite/workaroundCanary.test.ts: fails once module_import is referenced by an include', + }, +]; + +// One line at activation naming the active workarounds, so a bug report shows which ones were on +// (design §9: "Visible"). Every entry here is unconditional today (there is exactly one, and it is +// not gated by any version), unlike the server's, which gates by the running engine's version. +export function describeActiveWorkarounds(): string { + if (WORKAROUNDS.length === 0) { + return 'No workarounds active.'; + } + return `Active workarounds: ${WORKAROUNDS.map((workaround) => workaround.id).join(', ')}.`; +} diff --git a/editors/vscode/syntaxes/mcppls-modules.tmLanguage.json b/editors/vscode/syntaxes/mcppls-modules.tmLanguage.json new file mode 100644 index 0000000..c520df1 --- /dev/null +++ b/editors/vscode/syntaxes/mcppls-modules.tmLanguage.json @@ -0,0 +1,39 @@ +{ + "$schema": "https://raw.githubusercontent.com/martinring/tmlanguage/master/tmlanguage.json", + "name": "mcppls C++ Modules", + "comment": "WA-VSCODE-001 (see src/workarounds.ts): VS Code's built-in cpp grammar defines a module_import rule but nothing includes it, so import/module/export and module names get no scope on their own. This injection grammar covers only the module syntax the built-in grammar misses; its capture-group scopes match mcpp-community.mcpp-vscode's own module grammar exactly, so installing both extensions is harmless. Anchored at the start of the line (leading whitespace allowed) so 'x = import;' and 'obj.import(1)' are never matched -- only a line that itself begins with (an optional 'export' then) 'module' or 'import' is. Each pattern accepts an incomplete, still-being-typed name (for example 'import hello.' colors 'import' and 'hello', leaving the trailing dot unscoped) because nothing here requires a trailing ';' or an end of line.", + "scopeName": "source.cpp.mcppls-modules", + "injectionSelector": "L:source.cpp", + "patterns": [ + { "include": "#module-declaration" }, + { "include": "#import-declaration" } + ], + "repository": { + "module-declaration": { + "comment": "module; | [export] module name[:partition][;] | module :partition[;]", + "match": "^\\s*(?:(export)\\s+)?(module)\\b(?=[\\s;:/]|$)\\s*(?:([A-Za-z_][A-Za-z0-9_]*(?:\\.[A-Za-z_][A-Za-z0-9_]*)*)(?:\\s*(:)\\s*([A-Za-z_][A-Za-z0-9_]*))?|(:)\\s*([A-Za-z_][A-Za-z0-9_]*))?(?:\\s*;)?", + "captures": { + "1": { "name": "keyword.control.export.cpp" }, + "2": { "name": "keyword.control.module.cpp" }, + "3": { "name": "entity.name.namespace.module.cpp" }, + "4": { "name": "punctuation.separator.module-partition.cpp" }, + "5": { "name": "entity.name.namespace.module.partition.cpp" }, + "6": { "name": "punctuation.separator.module-partition.cpp" }, + "7": { "name": "entity.name.namespace.module.partition.cpp" } + } + }, + "import-declaration": { + "comment": "[export] import name | :partition |
| \"header\" [;]", + "match": "^\\s*(?:(export)\\s+)?(import)\\b(?=[\\s;:<\"/]|$)\\s*(?:(:)([A-Za-z_][A-Za-z0-9_]*)|([A-Za-z_][A-Za-z0-9_]*(?:\\.[A-Za-z_][A-Za-z0-9_]*)*(?::[A-Za-z_][A-Za-z0-9_]*)?)|(<[^>]*>)|(\"[^\"]*\"))?(?:\\s*;)?", + "captures": { + "1": { "name": "keyword.control.export.cpp" }, + "2": { "name": "keyword.control.import.cpp" }, + "3": { "name": "punctuation.separator.module-partition.cpp" }, + "4": { "name": "entity.name.namespace.module.partition.cpp" }, + "5": { "name": "entity.name.namespace.module.cpp" }, + "6": { "name": "string.quoted.other.header.cpp" }, + "7": { "name": "string.quoted.double.header.cpp" } + } + } + } +} From 67091b9526ca4f710bcf6df80dd5d4d719a9dbc9 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Fri, 25 Sep 2026 06:36:00 +0800 Subject: [PATCH 12/26] test(vscode): cover the injected grammar, the workaround canary, status wording and the conflict commands Unit (mocha, plain Node, no VS Code): - test/unit/statusText.test.ts: category filtering, the shortened-vs-full wording rules, and the removal of the generic "Some features are limited" phrase. - test/unit/conflictCandidates.test.ts: which extensions count as active conflicts, and the restore snapshot (an unset override is remembered as undefined, not as a default value). End to end (main suite): - test/suite/grammar.test.ts: opens a scratch .cpp file and reads back _workbench.captureSyntaxTokens for every form the grammar covers, including import hello. still being typed, and the two required negatives (x = import;, obj.import(1);, int module = 5;). - test/suite/workaroundCanary.test.ts: WA-VSCODE-001's canary -- fails, naming the workaround, the day VS Code's own cpp grammar includes its module_import rule from anywhere. (That rule ships under a hash-prefixed repository key, e.g. d9bc4796b0b_module_import, not the bare name the design doc found in the unmangled upstream source, so the check matches by suffix.) - test/suite/semanticTokens.test.ts: decodes the server's semantic tokens and checks the keyword and module types on the fixture's import lines. mcppls 0.0.3's legend has neither yet (confirmed live), so every assertion here skips, not fails, until the parallel server track lands them. Conflicts scenario (test/suite-conflicts): re-enabling clangd.enable after the first-run answer disabled it gets a non-modal notice; mcppls.turnOffOtherCppFeatures (workspace scope, substituted the same way as the conflictAnswer prompt) turns it back off; mcppls.restoreOtherCppFeatures puts back exactly the value from right before that turn-off (true, what the test set it to) rather than the very first answer's value, and removes cpptools's override entirely since it never had one. --- .../test/suite-conflicts/conflicts.test.ts | 38 +++++ editors/vscode/test/suite/grammar.test.ts | 149 ++++++++++++++++++ .../vscode/test/suite/semanticTokens.test.ts | 137 ++++++++++++++++ .../test/suite/workaroundCanary.test.ts | 63 ++++++++ .../test/unit/conflictCandidates.test.ts | 82 ++++++++++ editors/vscode/test/unit/statusText.test.ts | 113 +++++++++++++ 6 files changed, 582 insertions(+) create mode 100644 editors/vscode/test/suite/grammar.test.ts create mode 100644 editors/vscode/test/suite/semanticTokens.test.ts create mode 100644 editors/vscode/test/suite/workaroundCanary.test.ts create mode 100644 editors/vscode/test/unit/conflictCandidates.test.ts create mode 100644 editors/vscode/test/unit/statusText.test.ts diff --git a/editors/vscode/test/suite-conflicts/conflicts.test.ts b/editors/vscode/test/suite-conflicts/conflicts.test.ts index 3ef0c8c..4e2808d 100644 --- a/editors/vscode/test/suite-conflicts/conflicts.test.ts +++ b/editors/vscode/test/suite-conflicts/conflicts.test.ts @@ -77,6 +77,44 @@ suite('conflicting C++ extensions', function () { 'clangd.enable': false, }); }); + + // design 2026-09-25 §10 "Check again when things change": re-enabling a setting this + // extension turned off is a conflict becoming active again, and gets a notice -- distinct + // from the one-time question above, and never a warning or error modal. + test('re-enabling clangd.enable shows a non-modal notice, once', async () => { + const before = api.notificationCount(); + await vscode.workspace.getConfiguration('clangd').update('enable', true, vscode.ConfigurationTarget.Workspace); + await new Promise((resolve) => setTimeout(resolve, 3000)); + assert.ok( + api.notificationCount() > before, + `expected a new notice once clangd.enable was re-enabled; count stayed at ${api.notificationCount()}`, + ); + }); + + // design §10 "Commands": mcppls.turnOffOtherCppFeatures acts on whatever is active right + // now, at the scope chosen by the quick pick (substituted here the same way as the + // conflictAnswer prompt; see src/prompt.ts's pickOnce). cpptools is still off from the + // first-run answer above, so only clangd (re-enabled by the previous test) is active. + test('mcppls.turnOffOtherCppFeatures turns the newly re-enabled conflict back off, at workspace scope', async () => { + api.setPromptAnswer('turnOffScope', 'workspace'); + await vscode.commands.executeCommand('mcppls.turnOffOtherCppFeatures'); + assert.deepStrictEqual(readWorkspaceSettings(), { + 'C_Cpp.intelliSenseEngine': 'disabled', + 'clangd.enable': false, + }); + }); + + // design §10 "Restore Other C++ Language Features": puts back exactly what the most + // recent turn-off remembered -- here, `true` (what the test itself set clangd.enable to, + // right before the command above disabled it again), not `false` (the very first + // ask-once answer's value) and not cpptools's setting at all, which that command left + // alone because cpptools was not active when it ran. cpptools never had an override + // before the very first disable either, so restoring it removes the key entirely rather + // than writing a value that happens to match today's default. + test('mcppls.restoreOtherCppFeatures puts back the value from right before the last turn-off', async () => { + await vscode.commands.executeCommand('mcppls.restoreOtherCppFeatures'); + assert.deepStrictEqual(readWorkspaceSettings(), { 'clangd.enable': true }); + }); } else { test('keeping both changes no settings', () => { assert.deepStrictEqual(readWorkspaceSettings(), {}); diff --git a/editors/vscode/test/suite/grammar.test.ts b/editors/vscode/test/suite/grammar.test.ts new file mode 100644 index 0000000..8362be7 --- /dev/null +++ b/editors/vscode/test/suite/grammar.test.ts @@ -0,0 +1,149 @@ +// Layer 1 of highlighting `import` (design 2026-09-25 §7, WA-VSCODE-001): the injection grammar +// (syntaxes/mcppls-modules.tmLanguage.json) colors the module syntax VS Code's own cpp grammar +// misses, on every keystroke and with no server involved. This opens a scratch .cpp file (a real +// file on disk, outside the fixture workspace, because `_workbench.captureSyntaxTokens` needs a +// filesystem provider and cannot tokenize an untitled document) and reads back each line's scopes. +// +// See test/suite/workaroundCanary.test.ts for the other half: the check that VS Code's own grammar +// still needs this layer at all. + +import * as assert from 'assert'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import * as vscode from 'vscode'; + +interface CapturedToken { + c: string; // the token's text + t: string; // its full scope stack, space-separated, outermost first +} + +// captureSyntaxTokens returns one flat list for the whole file, with no token (not even an empty +// one) marking where a line ends -- token text simply runs from one line straight into the next. +// So a token is assigned to a line by walking the *known* line lengths and consuming that many +// characters of token text per line; nothing here ever crosses a line, since each pattern in the +// grammar only ever matches within one line. +function tokensByLine(sourceLines: readonly string[], tokens: readonly CapturedToken[]): { text: string; scopes: string[] }[][] { + const lines: { text: string; scopes: string[] }[][] = sourceLines.map(() => []); + let lineIndex = 0; + let consumed = 0; + for (const token of tokens) { + while (lineIndex < sourceLines.length && consumed >= sourceLines[lineIndex].length) { + lineIndex += 1; + consumed = 0; + } + if (lineIndex >= sourceLines.length) { + break; + } + lines[lineIndex].push({ text: token.c, scopes: token.t.split(' ') }); + consumed += token.c.length; + } + return lines; +} + +function findToken(line: { text: string; scopes: string[] }[], text: string): { text: string; scopes: string[] } | undefined { + return line.find((token) => token.text === text); +} + +async function captureLines(lines: readonly string[]): Promise<{ text: string; scopes: string[] }[][]> { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'mcppls-grammar-')); + const file = path.join(dir, 'probe.cpp'); + fs.writeFileSync(file, lines.join('\n')); + const uri = vscode.Uri.file(file); + const document = await vscode.workspace.openTextDocument(uri); + await vscode.window.showTextDocument(document); + const captured = await vscode.commands.executeCommand('_workbench.captureSyntaxTokens', uri); + return tokensByLine(lines, captured); +} + +suite('module-syntax highlighting: the injected grammar (WA-VSCODE-001)', function () { + this.timeout(60_000); + + let lines: { text: string; scopes: string[] }[][]; + + suiteSetup(async () => { + lines = await captureLines([ + 'module;', + 'export module a.b;', + 'module :private;', + 'import std;', + 'import hello.greet;', + 'import hello.', + 'export import hello:part;', + 'import ;', + 'import "foo.h";', + 'x = import;', + 'obj.import(1);', + 'int module = 5;', + ]); + }); + + test('bare "module;" colors the keyword', () => { + const token = findToken(lines[0], 'module'); + assert.ok(token?.scopes.includes('keyword.control.module.cpp'), JSON.stringify(lines[0])); + }); + + test('"export module a.b;" colors export, module, and the dotted name', () => { + const line = lines[1]; + assert.ok(findToken(line, 'export')?.scopes.includes('keyword.control.export.cpp')); + assert.ok(findToken(line, 'module')?.scopes.includes('keyword.control.module.cpp')); + assert.ok(findToken(line, 'a.b')?.scopes.includes('entity.name.namespace.module.cpp')); + }); + + test('"module :private;" colors the bare partition', () => { + const line = lines[2]; + assert.ok(findToken(line, 'module')?.scopes.includes('keyword.control.module.cpp')); + assert.ok(findToken(line, ':')?.scopes.includes('punctuation.separator.module-partition.cpp')); + assert.ok(findToken(line, 'private')?.scopes.includes('entity.name.namespace.module.partition.cpp')); + }); + + test('"import std;" colors the keyword and the module name', () => { + const line = lines[3]; + assert.ok(findToken(line, 'import')?.scopes.includes('keyword.control.import.cpp'), JSON.stringify(line)); + assert.ok(findToken(line, 'std')?.scopes.includes('entity.name.namespace.module.cpp')); + }); + + test('"import hello.greet;" colors the whole dotted name', () => { + const line = lines[4]; + assert.ok(findToken(line, 'hello.greet')?.scopes.includes('entity.name.namespace.module.cpp')); + }); + + test('typing "import hello." colors import and hello before the ";" exists', () => { + const line = lines[5]; + assert.ok(findToken(line, 'import')?.scopes.includes('keyword.control.import.cpp'), JSON.stringify(line)); + assert.ok(findToken(line, 'hello')?.scopes.includes('entity.name.namespace.module.cpp'), JSON.stringify(line)); + }); + + test('"export import hello:part;" colors export and import', () => { + const line = lines[6]; + assert.ok(findToken(line, 'export')?.scopes.includes('keyword.control.export.cpp')); + assert.ok(findToken(line, 'import')?.scopes.includes('keyword.control.import.cpp')); + assert.ok(findToken(line, 'hello:part')?.scopes.includes('entity.name.namespace.module.cpp')); + }); + + test('"import ;" colors the angle-bracket header', () => { + const line = lines[7]; + assert.ok(findToken(line, '')?.scopes.includes('string.quoted.other.header.cpp')); + }); + + test('\'import "foo.h";\' colors the quoted header', () => { + const line = lines[8]; + assert.ok(findToken(line, '"foo.h"')?.scopes.includes('string.quoted.double.header.cpp')); + }); + + test('"import" used as an ordinary identifier mid-line is never colored as the keyword', () => { + const assignment = findToken(lines[9], 'import'); + assert.ok(assignment, JSON.stringify(lines[9])); + assert.ok(!assignment.scopes.includes('keyword.control.import.cpp'), JSON.stringify(lines[9])); + + const memberCall = findToken(lines[10], 'import'); + assert.ok(memberCall, JSON.stringify(lines[10])); + assert.ok(!memberCall.scopes.includes('keyword.control.import.cpp'), JSON.stringify(lines[10])); + }); + + test('"module" used as an ordinary identifier is never colored as the module keyword', () => { + const token = findToken(lines[11], 'module'); + assert.ok(token, JSON.stringify(lines[11])); + assert.ok(!token.scopes.includes('keyword.control.module.cpp'), JSON.stringify(lines[11])); + }); +}); diff --git a/editors/vscode/test/suite/semanticTokens.test.ts b/editors/vscode/test/suite/semanticTokens.test.ts new file mode 100644 index 0000000..ff2916c --- /dev/null +++ b/editors/vscode/test/suite/semanticTokens.test.ts @@ -0,0 +1,137 @@ +// Layer 2 of highlighting `import` (design 2026-09-25 §7, §12 "Shared contracts"): the server's own +// semantic tokens for module syntax -- a `keyword` type for `import`/`module`/`export` and a custom +// `module` type (declared by this extension in package.json's `semanticTokenTypes`, with a +// `namespace` fallback) for module and partition names. +// +// NEEDS THE NEW SERVER: mcppls 0.0.3 (the payload this checkout builds against today) has no +// `keyword` or `module` entry in its semantic token legend at all -- confirmed live before writing +// this file: clangd's legend carries only its own C++ symbol kinds. Every assertion below is +// skipped, not failed, until a 0.0.4 server adds them; see MCPPLS_SERVER / MCPPLS_PAYLOAD in +// test/runTest.ts's header to point this suite at one once it exists. + +import * as assert from 'assert'; +import * as vscode from 'vscode'; +import type { TestApi } from '../../src/extension'; + +const EXTENSION_ID = 'sunrisepeak.mcpp-language-server'; +const READY_TIMEOUT_MS = 120_000; + +interface Legend { + tokenTypes: string[]; + tokenModifiers: string[]; +} + +interface DecodedToken { + line: number; + startChar: number; + length: number; + type: string; + modifiers: string[]; +} + +// The LSP `SemanticTokens.data` delta encoding: repeating groups of +// [deltaLine, deltaStartChar, length, tokenTypeIndex, tokenModifiersBitset]. +function decode(data: ArrayLike, legend: Legend): DecodedToken[] { + const tokens: DecodedToken[] = []; + let line = 0; + let char = 0; + for (let i = 0; i + 4 < data.length + 1; i += 5) { + const deltaLine = data[i]; + const deltaStart = data[i + 1]; + const length = data[i + 2]; + const typeIndex = data[i + 3]; + const modifierBits = data[i + 4]; + line += deltaLine; + char = deltaLine === 0 ? char + deltaStart : deltaStart; + const modifiers = legend.tokenModifiers.filter((_, bit) => (modifierBits & (1 << bit)) !== 0); + tokens.push({ line, startChar: char, length, type: legend.tokenTypes[typeIndex] ?? `#${typeIndex}`, modifiers }); + } + return tokens; +} + +function tokensOnLine(tokens: readonly DecodedToken[], line: number): DecodedToken[] { + return tokens.filter((token) => token.line === line); +} + +suite('module-syntax highlighting: server semantic tokens (design §7 layer 2)', function () { + this.timeout(READY_TIMEOUT_MS + 30_000); + + let legend: Legend | undefined; + let tokens: DecodedToken[] = []; + + suiteSetup(async function () { + const extension = vscode.extensions.getExtension(EXTENSION_ID); + assert.ok(extension, `${EXTENSION_ID} is not installed in the test instance`); + const api = await extension.activate(); + await api.waitForState(['ready', 'degraded'], READY_TIMEOUT_MS); + + const folder = vscode.workspace.workspaceFolders?.[0]; + assert.ok(folder, 'the fixture workspace is not open'); + const uri = vscode.Uri.joinPath(folder.uri, 'src', 'main.cpp'); + const document = await vscode.workspace.openTextDocument(uri); + await vscode.window.showTextDocument(document); + + legend = await vscode.commands.executeCommand('vscode.provideDocumentSemanticTokensLegend', uri); + if (!legend || !legend.tokenTypes.includes('keyword') || !legend.tokenTypes.includes('module')) { + // The precondition this whole suite needs; every test below skips instead of failing. + legend = undefined; + return; + } + const raw = await vscode.commands.executeCommand('vscode.provideDocumentSemanticTokens', uri); + tokens = raw ? decode(raw.data, legend) : []; + }); + + // Fixture (test/runTest.ts / conformance/fixtures/inferred/src/main.cpp): + // line 0: import std; + // line 1: import hello.greet; + + test('"import" on line 0 is the "keyword" type', function () { + if (!legend) { + this.skip(); + return; + } + const line = tokensOnLine(tokens, 0); + const keyword = line.find((token) => token.type === 'keyword'); + assert.ok(keyword, `no "keyword" token on line 0: ${JSON.stringify(line)}`); + }); + + test('"std" on line 0 is the "module" type', function () { + if (!legend) { + this.skip(); + return; + } + const line = tokensOnLine(tokens, 0); + const module = line.find((token) => token.type === 'module'); + assert.ok(module, `no "module" token on line 0: ${JSON.stringify(line)}`); + }); + + test('"import" on line 1 is the "keyword" type, and "hello.greet" is "module"', function () { + if (!legend) { + this.skip(); + return; + } + const line = tokensOnLine(tokens, 1); + assert.ok(line.some((token) => token.type === 'keyword'), `no "keyword" token on line 1: ${JSON.stringify(line)}`); + assert.ok(line.some((token) => token.type === 'module'), `no "module" token on line 1: ${JSON.stringify(line)}`); + }); + + test('no two tokens overlap', function () { + if (!legend) { + this.skip(); + return; + } + const byLine = new Map(); + for (const token of tokens) { + byLine.set(token.line, [...(byLine.get(token.line) ?? []), token]); + } + for (const [line, lineTokens] of byLine) { + const sorted = [...lineTokens].sort((a, b) => a.startChar - b.startChar); + for (let i = 1; i < sorted.length; i += 1) { + assert.ok( + sorted[i].startChar >= sorted[i - 1].startChar + sorted[i - 1].length, + `overlapping tokens on line ${line}: ${JSON.stringify(sorted[i - 1])} and ${JSON.stringify(sorted[i])}`, + ); + } + } + }); +}); diff --git a/editors/vscode/test/suite/workaroundCanary.test.ts b/editors/vscode/test/suite/workaroundCanary.test.ts new file mode 100644 index 0000000..eed1175 --- /dev/null +++ b/editors/vscode/test/suite/workaroundCanary.test.ts @@ -0,0 +1,63 @@ +// Canary for WA-VSCODE-001 (design 2026-09-25 §9, src/workarounds.ts): the injection grammar in +// syntaxes/mcppls-modules.tmLanguage.json exists only because VS Code's own built-in cpp grammar +// defines a `module_import` repository rule but no `include` anywhere in the grammar actually uses +// it, so `import std;` gets no keyword scope from VS Code alone. This fails on purpose, naming the +// workaround, the moment that stops being true -- which is the signal to remove both the workaround +// entry and the grammar file (or at least the parts of it VS Code has taken over). + +import * as assert from 'assert'; +import * as fs from 'fs'; +import * as path from 'path'; +import * as vscode from 'vscode'; + +interface TmGrammar { + repository?: Record; +} + +// The shipped grammar's own build step prefixes repository keys with a hash to avoid collisions +// when grammars are merged (measured: `module_import` ships as something like +// `d9bc4796b0b_module_import`), so both the key's existence and its use are matched by suffix +// rather than by the exact name the design doc found in the (unmangled) upstream source. +function isModuleImportKey(key: string): boolean { + return key === 'module_import' || key.endsWith('_module_import'); +} + +// Every `"include": "..."` string anywhere in the grammar, found structurally rather than by +// scanning the raw text, since `#module_import` could otherwise appear inside a comment, another +// rule's `match`, or a `name` and be mistaken for a real reference. +function includedRepositoryKeys(grammar: unknown, into: Set = new Set()): Set { + if (Array.isArray(grammar)) { + for (const item of grammar) { + includedRepositoryKeys(item, into); + } + } else if (grammar && typeof grammar === 'object') { + for (const [key, value] of Object.entries(grammar as Record)) { + if (key === 'include' && typeof value === 'string' && value.startsWith('#')) { + into.add(value.slice(1)); + } else { + includedRepositoryKeys(value, into); + } + } + } + return into; +} + +test('WA-VSCODE-001 canary: the built-in cpp grammar still does not include module_import', async function () { + this.timeout(30_000); + const grammarPath = path.join(vscode.env.appRoot, 'extensions', 'cpp', 'syntaxes', 'cpp.tmLanguage.json'); + const raw = fs.readFileSync(grammarPath, 'utf8'); + const grammar = JSON.parse(raw) as TmGrammar; + const repositoryKeys = Object.keys(grammar.repository ?? {}); + assert.ok( + repositoryKeys.some(isModuleImportKey), + 'the built-in grammar no longer defines a module_import rule at all; re-check this canary by hand', + ); + + const included = [...includedRepositoryKeys(grammar)]; + assert.ok( + !included.some(isModuleImportKey), + 'WA-VSCODE-001 is no longer needed: remove the injection grammar ' + + '(syntaxes/mcppls-modules.tmLanguage.json) and its entry in src/workarounds.ts -- ' + + "VS Code's own cpp grammar now includes module_import.", + ); +}); diff --git a/editors/vscode/test/unit/conflictCandidates.test.ts b/editors/vscode/test/unit/conflictCandidates.test.ts new file mode 100644 index 0000000..9794830 --- /dev/null +++ b/editors/vscode/test/unit/conflictCandidates.test.ts @@ -0,0 +1,82 @@ +// Conflict-candidate detection and restore bookkeeping (design 2026-09-25 §10), in plain Node: no +// VS Code, same reason test/unit/serverLog.test.ts is. The `vscode`-dependent glue (conflicts.ts) +// exercises the same logic through the real APIs in the e2e conflicts scenario. +import * as assert from 'assert'; +import { + activeConflictIds, + activeSettableConflicts, + activeUnsettableConflicts, + snapshotForDisable, + UNSETTABLE_CANDIDATES, +} from '../../src/conflictCandidates'; + +suite('settable conflict candidates', () => { + test('not installed is never a conflict', () => { + assert.deepStrictEqual(activeSettableConflicts(() => false, () => 'default'), []); + }); + + test('installed and at its enabled value is a conflict', () => { + const found = activeSettableConflicts(() => true, () => 'default'); + assert.strictEqual(found.length, 2); + assert.deepStrictEqual(found.map((c) => c.extensionId), ['ms-vscode.cpptools', 'llvm-vs-code-extensions.vscode-clangd']); + }); + + test('installed but already at its disabled value is not a conflict', () => { + const currentValue = (section: string, key: string): unknown => { + if (section === 'C_Cpp' && key === 'intelliSenseEngine') return 'disabled'; + if (section === 'clangd' && key === 'enable') return false; + return undefined; + }; + assert.deepStrictEqual(activeSettableConflicts(() => true, currentValue), []); + }); + + test('one installed and off, the other installed and on: only the active one is a conflict', () => { + const currentValue = (section: string): unknown => (section === 'clangd' ? false : 'default'); + const found = activeSettableConflicts(() => true, currentValue); + assert.strictEqual(found.length, 1); + assert.strictEqual(found[0].extensionId, 'ms-vscode.cpptools'); + }); +}); + +suite('unsettable conflict candidates', () => { + test('ccls, with no enable setting, is a candidate', () => { + assert.deepStrictEqual(UNSETTABLE_CANDIDATES.map((c) => c.extensionId), ['ccls-project.ccls']); + }); + + test('active only when installed', () => { + assert.deepStrictEqual(activeUnsettableConflicts(() => false), []); + assert.strictEqual(activeUnsettableConflicts(() => true).length, 1); + }); +}); + +suite('activeConflictIds', () => { + test('combines settable and unsettable ids', () => { + const ids = activeConflictIds(() => true, () => 'default'); + assert.deepStrictEqual( + [...ids].sort(), + ['ccls-project.ccls', 'llvm-vs-code-extensions.vscode-clangd', 'ms-vscode.cpptools'].sort(), + ); + }); + + test('empty when nothing is installed', () => { + assert.deepStrictEqual(activeConflictIds(() => false, () => 'default'), new Set()); + }); +}); + +suite('snapshotForDisable (restore bookkeeping)', () => { + test('remembers each conflict\'s current override at the scope being turned off', () => { + const conflicts = activeSettableConflicts(() => true, () => 'default'); + const valueAtScope = (section: string, key: string): unknown => (section === 'clangd' && key === 'enable' ? true : undefined); + const snapshot = snapshotForDisable(conflicts, valueAtScope); + assert.deepStrictEqual(snapshot, [ + { section: 'C_Cpp', key: 'intelliSenseEngine', value: undefined }, + { section: 'clangd', key: 'enable', value: true }, + ]); + }); + + test('no override at that scope is remembered as undefined, not as a default value', () => { + const conflicts = activeSettableConflicts(() => true, () => 'default'); + const snapshot = snapshotForDisable(conflicts, () => undefined); + assert.ok(snapshot.every((entry) => entry.value === undefined)); + }); +}); diff --git a/editors/vscode/test/unit/statusText.test.ts b/editors/vscode/test/unit/statusText.test.ts new file mode 100644 index 0000000..bb779fb --- /dev/null +++ b/editors/vscode/test/unit/statusText.test.ts @@ -0,0 +1,113 @@ +// The status wording rules (design 2026-09-25 §6, S3 extension `category`), in plain Node: no VS +// Code, same reason test/unit/serverLog.test.ts is. +import * as assert from 'assert'; +import { firstNonCodeIssue, isCodeIssue, shorten, stateTexts, StatusIssue } from '../../src/statusText'; + +suite('status issue categories', () => { + test('an issue with no category is treated as non-code (older servers)', () => { + const issue: StatusIssue = { code: 'engine-timeout', message: 'clangd stopped responding' }; + assert.strictEqual(isCodeIssue(issue), false); + assert.strictEqual(firstNonCodeIssue([issue]), issue); + }); + + test('category "code" is excluded from the non-code search', () => { + const codeIssue: StatusIssue = { code: 'unresolved-module', message: 'import helo; not found', category: 'code' }; + const engineIssue: StatusIssue = { code: 'engine-timeout', message: 'clangd stopped responding', category: 'engine' }; + assert.strictEqual(firstNonCodeIssue([codeIssue, engineIssue]), engineIssue); + assert.strictEqual(firstNonCodeIssue([codeIssue]), undefined); + }); + + test('every other category counts as non-code', () => { + for (const category of ['engine', 'environment', 'project'] as const) { + const issue: StatusIssue = { code: 'x', message: 'm', category }; + assert.strictEqual(isCodeIssue(issue), false); + } + }); +}); + +suite('shorten', () => { + test('leaves a short message alone', () => { + assert.strictEqual(shorten('clangd stopped responding'), 'clangd stopped responding'); + }); + + test('cuts at the first sentence when that already fits', () => { + assert.strictEqual( + shorten('main.cpp: basic features only. clangd stopped responding and was restarted.'), + 'main.cpp: basic features only.', + ); + }); + + test('falls back to a word-boundary truncation for one long sentence', () => { + const long = 'a'.repeat(30) + ' ' + 'b'.repeat(60); + const result = shorten(long, 40); + assert.ok(result.length <= 40, `expected <= 40 chars, got ${result.length}: ${result}`); + assert.ok(result.endsWith('…'), `expected an ellipsis, got: ${result}`); + assert.ok(!result.includes('bbbbb'), 'must not cut mid-word into the long run of b'); + }); +}); + +suite('status text: state, category and wording rules', () => { + test('the generic "Some features are limited" phrase is gone', () => { + const texts = stateTexts({ state: 'degraded', issues: [{ code: 'engine-timeout', message: 'clangd stopped responding' }] }); + assert.notStrictEqual(texts.short, 'Some features are limited'); + assert.notStrictEqual(texts.full, 'Some features are limited'); + }); + + test('degraded shows the first non-code issue, shortened for the bar, in full for the tooltip', () => { + const long = 'main.cpp: basic features only. clangd stopped responding and was restarted after a 20 second stall.'; + const texts = stateTexts({ state: 'degraded', issues: [{ code: 'engine-timeout', message: long }] }); + assert.strictEqual(texts.short, shorten(long)); + assert.strictEqual(texts.full, long); + }); + + test('degraded skips a code issue and uses the first non-code one instead', () => { + const texts = stateTexts({ + state: 'degraded', + issues: [ + { code: 'unresolved-module', message: 'import helo; not found', category: 'code' }, + { code: 'engine-timeout', message: 'clangd stopped responding', category: 'engine' }, + ], + }); + assert.strictEqual(texts.full, 'clangd stopped responding'); + }); + + test('degraded with only a code issue (a defensive fallback, not expected in practice) does not crash', () => { + const texts = stateTexts({ + state: 'degraded', + issues: [{ code: 'unresolved-module', message: 'import helo; not found', category: 'code' }], + }); + assert.strictEqual(texts.full, 'Limited'); + assert.notStrictEqual(texts.full, 'Some features are limited'); + }); + + test('error keeps its sentence and appends why', () => { + const texts = stateTexts({ state: 'error', issues: [{ code: 'engine-crashed', message: 'clangd exited repeatedly' }] }); + assert.strictEqual(texts.full, 'Only module-level features are available: clangd exited repeatedly'); + assert.strictEqual(texts.short, texts.full); + }); + + test('error with no issues keeps the sentence alone', () => { + const texts = stateTexts({ state: 'error', issues: [] }); + assert.strictEqual(texts.full, 'Only module-level features are available'); + }); + + test('error names why from the first non-code issue, skipping a code one', () => { + const texts = stateTexts({ + state: 'error', + issues: [ + { code: 'unresolved-module', message: 'import helo; not found', category: 'code' }, + { code: 'engine-crashed', message: 'clangd exited repeatedly', category: 'engine' }, + ], + }); + assert.strictEqual(texts.full, 'Only module-level features are available: clangd exited repeatedly'); + }); + + test('ready has no text', () => { + assert.deepStrictEqual(stateTexts({ state: 'ready' }), { short: undefined, full: undefined }); + }); + + test('preparing shows progress, and it is the same in short and full form', () => { + const texts = stateTexts({ state: 'preparing', progress: { done: 9, total: 646 } }); + assert.deepStrictEqual(texts, { short: 'Preparing modules 9/646', full: 'Preparing modules 9/646' }); + }); +}); From ca92e9c12b3dcada43c314bc4c96faf05a2b97be Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Fri, 25 Sep 2026 06:36:06 +0800 Subject: [PATCH 13/26] docs(vscode): document highlighting, the semantic tokens setting and the new conflict commands README: a Highlighting section explaining the two layers and how to customize the color, the new mcppls.semanticTokens.modules and detectConflicts wording in the settings table, the two new commands, and the conflicts section covering turn off/restore at any time, the re-check notice, and that an extension with no enable setting can only be pointed at. --- editors/vscode/README.md | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/editors/vscode/README.md b/editors/vscode/README.md index fc0ec13..1d6e182 100644 --- a/editors/vscode/README.md +++ b/editors/vscode/README.md @@ -15,7 +15,16 @@ Everything ships inside the extension: the language server, clangd 23.1 and the ## Status -When a C++ file is open, the language status area shows one item, for example `C++ Modules · gcc 16.1.0`. It is busy while modules are prepared, and turns into a warning with a suggested fix when something limits the results. +When a C++ file is open, the language status area shows one item, for example `C++ Modules · gcc 16.1.0`. It is busy while modules are prepared. A warning or error names the file and the actual reason (for example "clangd stopped responding"), never a generic "some features are limited"; an error not in your own code that mcppls's diagnostics do not already cover. + +## Highlighting `import`, `module` and `export` + +Module syntax is colored in two layers, both mcppls's own: + +- Immediately, and even with the server not running: an injected grammar colors `module`, `import`, `export` and module names as you type, including a name you are still typing (`import hello.` colors `import` and `hello`). This exists because VS Code's own C++ grammar defines the rule for it but never uses it — see **C++ Modules: Show Logs** at startup for the active workaround, `WA-VSCODE-001`. +- From the server, once it is running: semantic tokens add module and partition names as a custom `module` type (colored as a namespace until you customize it) and module keywords as the standard `keyword` type, for every editor the server supports, not only this one. Turn this off with `mcppls.semanticTokens.modules` to use only your own grammar or tree-sitter colors. + +Customize the color the standard way: `editor.semanticTokenColorCustomizations.rules` (for example `"module": { "foreground": "#..." }`, or `"*.partition"` for partitions). ## Commands @@ -28,6 +37,8 @@ When a C++ file is open, the language status area shows one item, for example `C | C++ Modules: Install Command Line Tools | Run `xcode-select --install` (macOS only) | | C++ Modules: Collect Diagnostic Report | Open the server's status and this extension's version, settings and other installed C++ extensions as JSON, ready to copy or attach to an issue | | C++ Modules: Run the Build Tool in a Terminal | Run the project's build command (`mcpp build` or the CMake configure step) in your own terminal, where a proxy or credentials you set by hand actually are | +| C++ Modules: Turn Off Other C++ Language Features | Turn off the language features of other active C++ extensions, in this workspace or everywhere (user settings) | +| C++ Modules: Restore Other C++ Language Features | Put back whatever the command above (or the one-time question) last changed, in the same scope | With `mcppls.ai.enabled`: @@ -46,16 +57,21 @@ All settings are optional. | `mcppls.semanticKit` | `auto` | `off` never uses the built-in standard library kit | | `mcppls.engine` | `clangd` | `none` runs without clangd: module-level features only | | `mcppls.ai.enabled` | `false` | Show the review commands | -| `mcppls.detectConflicts` | `true` | Offer once to turn off other C++ extensions' language features in the workspace | +| `mcppls.detectConflicts` | `true` | Offer once to turn off other C++ extensions' language features in the workspace, and notice again if one becomes active later | +| `mcppls.semanticTokens.modules` | `true` | Module keywords and names from the server's semantic tokens; turn off to use only your own grammar or tree-sitter colors for module syntax | | `mcppls.trace.server` | `off` | Trace the language server protocol in the log | | `mcppls.buildTool` | `offline` | How mcppls may run the project's build tool (mcpp, CMake) to learn how it is built: `offline` runs it without the network, offering to run it in a terminal when it needs a download; `online` lets it reach the network, with up to ten minutes; `off` never runs it, using only the cache or scanned sources | | `mcppls.toolEnvironment` | `auto` | Which environment build tools are started in: `auto` reads the login shell's environment once in the background on Linux and macOS (Windows always matches the editor); `editor` always uses the editor process's own environment | ## Other C++ extensions -If the Microsoft C/C++ extension or the clangd extension also serves C++ files, results appear twice. The extension asks once whether to turn their language features off for the workspace; nothing changes without your answer. +If the Microsoft C/C++ extension or the clangd extension also serves C++ files, results appear twice. The extension asks once whether to turn their language features off for the workspace; nothing changes without your answer. cpptools keeps its debugger either way — only its IntelliSense engine is turned off. + +Turn them off or put them back at any time with **C++ Modules: Turn Off Other C++ Language Features** and **C++ Modules: Restore Other C++ Language Features**. If one of them becomes active again later — reinstalled, re-enabled, or its setting turned back on — a notice says so once, with a "Turn Off" action; it is never a warning or error, and nothing changes without you choosing it. + +An extension with no setting to turn off (for example ccls) can only be pointed out: the notice's action opens it in the Extensions view, where you can disable it yourself. -This extension and `mcpp-community.mcpp-vscode` ("mcpp") are separate, and both are worth having installed together: `mcpp` handles building, toolchains and project operations, while this extension handles C++ module semantics and drives its own pinned clangd. +This extension and `mcpp-community.mcpp-vscode` ("mcpp") are separate, and both are worth having installed together: `mcpp` handles building, toolchains and project operations, while this extension handles C++ module semantics and drives its own pinned clangd. mcpp is not a conflict: it has no language server, and its module grammar uses the same scopes as this extension's own. ## macOS Command Line Tools From 6b77d0601b130ef64ce4161d3455b62f92ec7a28 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Fri, 25 Sep 2026 06:39:44 +0800 Subject: [PATCH 14/26] docs(editors): highlighting, the new VS Code setting and commands, and Neovim's disable_conflicting --- docs/10-editors.md | 17 +++++++++++++++-- docs/30-settings.md | 4 +++- docs/zh-CN/10-editors.md | 6 ++++-- docs/zh-CN/30-settings.md | 4 +++- 4 files changed, 25 insertions(+), 6 deletions(-) diff --git a/docs/10-editors.md b/docs/10-editors.md index 12a9ad4..c39f7f4 100644 --- a/docs/10-editors.md +++ b/docs/10-editors.md @@ -34,7 +34,17 @@ and both are worth having: **Alongside other C++ extensions.** Microsoft's C/C++ extension and the official clangd extension both want to be the language server for the same files. On first run mcppls offers, once, to turn -their language features off for this workspace; `mcppls.detectConflicts` controls that offer. +their language features off for this workspace; `mcppls.detectConflicts` controls that offer. Any +time later, *Turn Off Other C++ Language Features* does it for this workspace or everywhere and +*Restore Other C++ Language Features* undoes it; the C/C++ extension's debugger keeps working. A +conflicting extension that becomes active later is named in a notice. An extension has no way to +disable another one: only their own settings are changed, and only when you choose to. + +**Highlighting.** `import`, `module`, `export` and module names are colored twice over: by a +grammar the extension adds (at once, as you type), and by the server's semantic tokens (module +names as the token type `module`, which themes color as a namespace unless you give it a color of +its own in `editor.semanticTokenColorCustomizations`). VS Code's own C++ grammar leaves `import` +uncolored. ## Claude Code @@ -65,7 +75,10 @@ on PATH, or the payload `--install` puts in the user data directory — and star buffers through Neovim's own LSP client. Put `editors/nvim` on the runtimepath and call `require('mcppls').setup()`; on 0.11 and later `vim.lsp.enable('mcppls')` works too. It adds `:McpplsStatus`, `:McpplsRestart`, `:McpplsReload` and a statusline component. Do not also start -clangd for C and C++: the plugin names a second C++ server once if one attaches. +clangd for C and C++: the plugin names a second C++ server once if one attaches, and with +`disable_conflicting = true` stops it for you. Module keywords and names come from the server's +semantic tokens (`@lsp.type.keyword`, `@lsp.type.module`; `semantic_tokens_modules = false` turns +them off). ## CLion diff --git a/docs/30-settings.md b/docs/30-settings.md index 904d275..8a470c9 100644 --- a/docs/30-settings.md +++ b/docs/30-settings.md @@ -10,7 +10,8 @@ | `mcppls.semanticKit` | `auto` (default), `off` | Whether the bundled kit may be used at all | | `mcppls.engine` | `clangd` (default), `none` | The core engine. mcppls's own module engine runs either way; `none` means module features only | | `mcppls.ai.enabled` | `false` (default) | Whether the model-backed half of change review may be used. Off means the server makes no model calls | -| `mcppls.detectConflicts` | `true` (default) | Offer once to turn off another C++ extension's language features in this workspace | +| `mcppls.detectConflicts` | `true` (default) | Offer once to turn off another C++ extension's language features in this workspace, and say so when one becomes active later | +| `mcppls.semanticTokens.modules` | `true` (default) | Color `import`, `module`, `export` and module names from the server's semantic tokens. Off: only the grammar's colors | | `mcppls.trace.server` | `off` (default), `messages`, `verbose` | Log the LSP traffic to the C++ Modules output channel (at Trace level); `verbose` adds the server's debug log (at Debug level). Set the channel's log level to see them | ## Commands @@ -22,6 +23,7 @@ | C++ Modules: Show Module Graph | The project's modules and what imports what | | C++ Modules: Select Context | Switch which set of the build database the file is seen through | | C++ Modules: Restart Language Server / Show Logs | The usual two | +| C++ Modules: Turn Off Other C++ Language Features / Restore Other C++ Language Features | Turn off the C/C++ extension's IntelliSense (its debugger keeps working) and the clangd extension, in this workspace or everywhere; restore puts back exactly what was there | | C++ Modules: Review Changes / Clear Review | Change review over the working tree, published as diagnostics (needs `mcppls.ai.enabled` for the model-backed rules) | ## Command line diff --git a/docs/zh-CN/10-editors.md b/docs/zh-CN/10-editors.md index 76f864a..afe106c 100644 --- a/docs/zh-CN/10-editors.md +++ b/docs/zh-CN/10-editors.md @@ -23,7 +23,9 @@ mcpp run -p devtools -- uninstall --editor vscode|zed|clion|all # 卸载 | **mcpp** | 构建、工具链、项目操作 | | **C++ Modules**（本项目） | C++ 模块语义，驱动自己锁定版本的 clangd | -**和其他 C++ 扩展一起用。** Microsoft 的 C/C++ 扩展和官方 clangd 扩展都想当同一批文件的语言服务端。mcppls 第一次运行时会提示一次，问要不要把它们在这个工作区里的语言功能关掉；这个提示由 `mcppls.detectConflicts` 控制。 +**和其他 C++ 扩展一起用。** Microsoft 的 C/C++ 扩展和官方 clangd 扩展都想当同一批文件的语言服务端。mcppls 第一次运行时会提示一次，问要不要把它们在这个工作区里的语言功能关掉；这个提示由 `mcppls.detectConflicts` 控制。之后任何时候都可以用 *Turn Off Other C++ Language Features* 在当前工作区或全局关掉它们，用 *Restore Other C++ Language Features* 恢复；C/C++ 扩展的调试器照常可用。之后又有冲突扩展启用时，会有一条提示说明。扩展没有办法禁用别的扩展：mcppls 只改它们自己的设置，而且只在你选择之后才改。 + +**语法高亮。** `import`、`module`、`export` 和模块名有两层上色：扩展自带的语法文件（打开即生效，边输入边上色），以及服务端的语义 token（模块名的 token 类型是 `module`，主题默认按命名空间上色，也可以在 `editor.semanticTokenColorCustomizations` 里单独指定颜色）。VS Code 自带的 C++ 语法不给 `import` 上色。 ## Claude Code @@ -41,7 +43,7 @@ Zed 自带 C/C++ 的 clangd，两个都跑在同一个文件上，就成了两 ## Neovim -[`editors/nvim/`](../../editors/nvim/README.md) 里的插件（Neovim 0.10 及以上）会找到 `mcppls`——PATH 上的，或者 `--install` 放在用户数据目录下的 payload——然后通过 Neovim 自带的 LSP 客户端，为 C 和 C++ buffer 启动它。把 `editors/nvim` 加进 runtimepath，调用 `require('mcppls').setup()` 即可；0.11 及以上也可以用 `vim.lsp.enable('mcppls')`。插件提供 `:McpplsStatus`、`:McpplsRestart`、`:McpplsReload` 三个命令和一个状态栏组件。不要再为 C/C++ 另外启动 clangd：如果有第二个 C++ 服务器挂到同一个 buffer 上，插件会提示一次。 +[`editors/nvim/`](../../editors/nvim/README.md) 里的插件（Neovim 0.10 及以上）会找到 `mcppls`——PATH 上的，或者 `--install` 放在用户数据目录下的 payload——然后通过 Neovim 自带的 LSP 客户端，为 C 和 C++ buffer 启动它。把 `editors/nvim` 加进 runtimepath，调用 `require('mcppls').setup()` 即可；0.11 及以上也可以用 `vim.lsp.enable('mcppls')`。插件提供 `:McpplsStatus`、`:McpplsRestart`、`:McpplsReload` 三个命令和一个状态栏组件。不要再为 C/C++ 另外启动 clangd：如果有第二个 C++ 服务器挂到同一个 buffer 上，插件会提示一次；设置 `disable_conflicting = true` 后插件会替你停掉它。模块关键字和模块名来自服务端的语义 token（`@lsp.type.keyword`、`@lsp.type.module`；`semantic_tokens_modules = false` 可关闭）。 ## CLion diff --git a/docs/zh-CN/30-settings.md b/docs/zh-CN/30-settings.md index 5c13f52..6474691 100644 --- a/docs/zh-CN/30-settings.md +++ b/docs/zh-CN/30-settings.md @@ -12,7 +12,8 @@ | `mcppls.semanticKit` | `auto`（默认）, `off` | 内置工具包是否可以被使用 | | `mcppls.engine` | `clangd`（默认）, `none` | 核心引擎。无论如何，mcppls 自己的模块引擎都会运行；`none` 表示只提供模块相关功能 | | `mcppls.ai.enabled` | `false`（默认） | 是否启用变更审查里依赖模型的那部分。关闭时服务端不发起任何模型调用 | -| `mcppls.detectConflicts` | `true`（默认） | 在此工作区中提议关闭另一个 C++ 扩展的语言功能（只提议一次） | +| `mcppls.detectConflicts` | `true`（默认） | 在此工作区中提议关闭另一个 C++ 扩展的语言功能（只提议一次），之后又有冲突扩展启用时会提示 | +| `mcppls.semanticTokens.modules` | `true`（默认） | 用服务端的语义 token 给 `import`、`module`、`export` 和模块名上色。关闭后只用语法文件的颜色 | | `mcppls.trace.server` | `off`（默认）, `messages`, `verbose` | 把 LSP 通信记录到 C++ Modules 输出通道（Trace 级别）；`verbose` 还会打开服务端的 debug 日志（Debug 级别）。要看到它们，需把该输出通道的日志级别调到对应级别 | ## 命令 @@ -24,6 +25,7 @@ | C++ Modules: Show Module Graph | 项目的模块及其相互导入关系 | | C++ Modules: Select Context | 切换该文件所使用的那一套构建数据库 | | C++ Modules: Restart Language Server / Show Logs | 重启语言服务端、查看日志 | +| C++ Modules: Turn Off Other C++ Language Features / Restore Other C++ Language Features | 关闭 C/C++ 扩展的 IntelliSense（调试器照常可用）和 clangd 扩展，可选当前工作区或全局；恢复时原样放回之前的设置 | | C++ Modules: Review Changes / Clear Review | 对工作区变更做审查，以诊断形式发布（依赖模型的规则需要开启 `mcppls.ai.enabled`） | ## 命令行 From bee9f6b429ce59d9a0df58eeb74d4820d24bce5d Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Fri, 25 Sep 2026 06:41:01 +0800 Subject: [PATCH 15/26] spec(s3): issue categories, the degraded hold, and module syntax in semantic tokens --- conformance/traceability.json | 57 ++++++++++++++++++++++++++++++++- docs/specs/CHANGELOG.md | 14 ++++++++ docs/specs/s3-lsp-extensions.md | 28 +++++++++++++++- 3 files changed, 97 insertions(+), 2 deletions(-) diff --git a/conformance/traceability.json b/conformance/traceability.json index 860a60e..a3c4fb4 100644 --- a/conformance/traceability.json +++ b/conformance/traceability.json @@ -926,6 +926,61 @@ "contains": "s.project.tier and (' L' .. s.project.tier)" } ], + "S3-4-10": [ + { + "script": "src/engine/engine.cppm", + "contains": "std::string category { \"engine\" };" + }, + { + "script": "src/normalize/plan.cppm", + "contains": "std::string category { \"project\" };" + }, + { + "check": "untrusted/S1" + } + ], + "S3-4-11": [ + { + "check": "module-faults/S1" + }, + { + "check": "failure-at-base/S1-settles-ready-naming-the-failure" + }, + { + "script": "src/orchestrator/workspace.cpp", + "contains": "const auto notCode = [](const auto& issue) { return issue.category != \"code\"; };" + } + ], + "S3-4-12": [ + { + "script": "src/engine/native/index.cpp", + "contains": "\"unresolved-module\", std::format(\"module '{}' not found\", name)" + }, + { + "check": "typing-import/T1-type-import" + } + ], + "S3-4-13": [ + { + "script": "editors/vscode/src/statusText.ts", + "contains": "return issue.category === 'code';" + } + ], + "S3-4-14": [ + { + "script": "editors/vscode/src/statusText.ts", + "contains": "const issue = firstNonCodeIssue(status.issues ?? []);" + } + ], + "S3-4-15": [ + { + "check": "typing-import/T1-type-import" + }, + { + "script": "src/orchestrator/workspace.cpp", + "contains": "constexpr std::chrono::milliseconds DEGRADED_HOLD { 3000 };" + } + ], "S3-5.5-1": [ { "check": "module-faults/F0-stand-in" @@ -1634,4 +1689,4 @@ "contains": "ignore it, so it is a courtesy, not the mechanism" } ] -} \ No newline at end of file +} diff --git a/docs/specs/CHANGELOG.md b/docs/specs/CHANGELOG.md index b53da21..d7741f2 100644 --- a/docs/specs/CHANGELOG.md +++ b/docs/specs/CHANGELOG.md @@ -2,6 +2,20 @@ Changes to the specifications in this directory. Each specification is versioned independently. +## 2026-09-25 — S3: issue categories, the degraded hold, module syntax in semantic tokens + +Added `category` (optional) to `CxxModulesIssue`: `code`, `engine`, `environment` or `project`, +whose problem an issue is. Issues of category `code` — the user's own source being wrong — never +make a root `degraded` or `error`; they are reported as diagnostics where they are, and a client +treats an issue without a category as before (S3-4-10 to S3-4-14). A change to `degraded` is held +back until it has lasted a short interval, so a condition that passes by itself never reaches a +client (S3-4-15). Listed `modules-doomed` among the codes. + +Added section 6.1: servers add semantic tokens for module syntax (keywords as `keyword`, module +names as `module` for clients that declare `initializationOptions.semanticTokens.moduleType`, else +`namespace`), and `initializationOptions.semanticTokens.modules: false` turns them off (S3-6.1-1 +to S3-6.1-3). All additive: protocol version stays 1. + ## 2026-09-24 — S3: issue `engine-incompatible` Listed `engine-incompatible` among `CxxModulesIssue` codes: the core engine cannot run on this diff --git a/docs/specs/s3-lsp-extensions.md b/docs/specs/s3-lsp-extensions.md index e75baa4..7f9bd0d 100644 --- a/docs/specs/s3-lsp-extensions.md +++ b/docs/specs/s3-lsp-extensions.md @@ -97,9 +97,15 @@ interface CxxModulesIssue { | "std-fallback-kit" // the engine could not build the toolchain's standard library module; a semantic kit reads the files | "file-quarantined" // the engine stopped answering for some files; they are answered from the module index until they change | "engine-incompatible" // the engine cannot run on this machine at all (its program loader refused it); module-level features remain + | "modules-doomed" // modules that cannot be prepared because a module they import does not compile | string; message: string; command?: Command; // an optional action that fixes the issue + category?: "code" // the user's own source is wrong: told as diagnostics where it is + | "engine" // a semantic engine lost something (stopped responding, restarted too often) + | "environment" // the machine, the payload or the workspace's trust + | "project" // the build description (stale, ambiguous, slow) + | string; } ``` @@ -111,7 +117,7 @@ States: | `loading` | Detecting the project and loading or inferring its model. | | `preparing` | The semantic engine is building the modules a file imports. | | `ready` | All features are available. | -| `degraded` | Some features are reduced, for example an inferred model or an engine timeout. `issues` says why. | +| `degraded` | Some features are reduced, for example an inferred model or an engine timeout. `issues` says why. Issues of category `code` alone never make a root `degraded`. | | `error` | Only syntactic features remain. `issues` says why. | A server **MUST** send the notification whenever any field changes, **SHOULD** coalesce changes that occur within a short interval, and **MUST** send at least one notification after `initialized`. `project.source` names where the model came from: an mcpp project, a CMake project, an S1 database, a `compile_commands.json`, or inference from sources alone. `profile.kind` is `semantic-kit` when the server analyzes the project with an [S4](s4-semantic-kit.md) semantic kit because no suitable compiler was found. S3-4-1, S3-4-2, S3-4-3 @@ -120,6 +126,10 @@ A server **MUST** send the notification whenever any field changes, **SHOULD** c A server that manages more than one workspace root (multiple `workspaceFolders`, or folders added or removed later through `workspace/didChangeWorkspaceFolders`) **MUST** send one notification per root, each with that root's own `project.root`, rather than one notification describing all of them; a client that presents status per folder tells them apart by it. This is a backward-compatible addition: `project.root` already existed in protocol version 1, and a single-root server's one notification already satisfied "at least one notification" above. A request that names a document (for example `cxxModules/setContext`) is answered by the root that owns it; `cxxModules/graph` and a bare-name `cxxModules/moduleInfo` name no document and so, until a later protocol version adds a way to select one, are answered by the first root. S3-4-4 +Each issue **SHOULD** carry a `category` saying whose problem it is. S3-4-10 A problem of the user's own source — a syntax error, an import of a module nothing provides, a module that does not compile — is category `code`: a server **MUST NOT** report `degraded` or `error` because of `code` issues alone S3-4-11, and **SHOULD** report such a problem as a diagnostic at its location instead, though it may still list the issue. S3-4-12 A client **MUST** treat an issue without `category` as not `code` (servers from before the field existed) S3-4-13, and **SHOULD NOT** present a `code` issue as a loss of features. S3-4-14 + +A server **SHOULD NOT** send `degraded` for a condition that ends by itself within a few seconds (a file set aside and handed back while the user types): it holds a change to `degraded` until it has lasted a short interval, and sends `error` at once. S3-4-15 + A server whose semantic capabilities come from more than one engine **SHOULD** list each in `engines` with its role and state, and **MUST** name the engine that provides the core C++ semantics in `engine`, or `"none"` when the root has none. A client **MUST** accept engine names other than `"clangd"`. S3-4-5, S3-4-6, S3-4-7 ## 5. Requests @@ -227,6 +237,22 @@ The content of each `roots` entry is the server's own and may change between ser Diagnostics produced from the module index use these `code` values: `unresolved-module`, `ambiguous-module` and `partition-outside-module`. A server **SHOULD** name the semantic profile in the `source` of diagnostics it forwards from the semantic engine, for example `"mcppls · gcc 16"`, so that a user can tell which compiler's semantics a diagnostic reflects. S3-6-1 +### 6.1 Module syntax in semantic tokens + +A C++ semantic engine may send no tokens for module syntax at all (clangd 23.1 does not), so a server **SHOULD** add them to `textDocument/semanticTokens` results: the `export`, `module` and `import` keywords of module declarations and import declarations as the standard type `keyword`, and module and partition names. Where the core engine's tokens cover a position, the core engine's win; the result stays sorted and non-overlapping. When the core engine gives no result, the module-syntax tokens are the result. S3-6.1-1 + +```ts +// Client → server: InitializeParams.initializationOptions +interface CxxModulesInitializationOptions { + semanticTokens?: { + modules?: boolean; // default true: the server adds module-syntax tokens + moduleType?: boolean; // default false: the client knows the token type "module" and the modifier "partition" + }; +} +``` + +A module name is sent with the token type `module`, and a partition name also with the modifier `partition`, only to a client that declared `moduleType: true`; to any other client a server **MUST** send module and partition names as `namespace`, so that a theme that knows only the standard types still colors them. A server **MUST NOT** add module-syntax tokens for a client that declared `modules: false`. The legend is the server's: it maps the core engine's token types and modifiers into it by name. S3-6.1-2, S3-6.1-3 + ## 7. Versioning The protocol version is an integer. Version 1 is defined by this document. A later version adds optional fields and new messages only; a change that is not backward compatible requires a new method prefix. From 1cd7ad4e3606c4afd2d9d46f16a7c7412aefd1a4 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Fri, 25 Sep 2026 06:48:11 +0800 Subject: [PATCH 16/26] feat(orchestrator): semantic tokens get their own server-owned legend clangd 23.1 has no keyword type and tokenizes no import/module/export line at all (measured 2026-09-25); its own legend also has duplicate names (variable appears twice). mcppls.orchestrator.tokens is a new, pure module: a fixed base legend (the LSP standard types and modifiers, clangd's own extras, and mcppls's own module/partition), a mapping from a core engine's own token-type indices and modifier bits into it by name (any other name the core engine declares is appended, and duplicates collapse to the first entry with that name), the LSP relative encode/decode, and a merge where the core engine's tokens win every position they cover and native tokens only fill the gaps. Covered by tests/test_tokens.cpp: legend construction and its indices, mapping with duplicates and an appended name, the encode/decode round trip (sorted, relative), and the merge (overlap dropped, disjoint gap kept, null only when neither engine answered). --- src/orchestrator/tokens.cpp | 160 +++++++++++++++++++++++++++++++++++ src/orchestrator/tokens.cppm | 76 +++++++++++++++++ tests/test_tokens.cpp | 124 +++++++++++++++++++++++++++ 3 files changed, 360 insertions(+) create mode 100644 src/orchestrator/tokens.cpp create mode 100644 src/orchestrator/tokens.cppm create mode 100644 tests/test_tokens.cpp diff --git a/src/orchestrator/tokens.cpp b/src/orchestrator/tokens.cpp new file mode 100644 index 0000000..dc6b55c --- /dev/null +++ b/src/orchestrator/tokens.cpp @@ -0,0 +1,160 @@ +module mcppls.orchestrator.tokens; + +import std; +import nlohmann.json; + +namespace mcppls::orchestrator::tokens { + +namespace { + +// The LSP standard token types (metaModel 3.17's SemanticTokenTypes), clangd 23.1's own extras +// (measured 2026-09-25: `unknown`, `concept`, `bracket`, `label` -- everything else it declares, +// including its duplicates, is already an LSP standard name), and `module`, mcppls's own. +constexpr std::array BASE_TYPES { + "namespace", "type", "class", "enum", "interface", "struct", "typeParameter", "parameter", "variable", "property", + "enumMember", "event", "function", "method", "macro", "keyword", "modifier", "comment", "string", "number", + "regexp", "operator", "decorator", + "unknown", "concept", "bracket", "label", + "module", +}; + +// The LSP standard modifiers, clangd's own extras, and `partition`, mcppls's own. +constexpr std::array BASE_MODIFIERS { + "declaration", "definition", "readonly", "static", "deprecated", "abstract", "async", "modification", "documentation", "defaultLibrary", + "deduced", "virtual", "dependentName", "usedAsMutableReference", "usedAsMutablePointer", "constructorOrDestructor", "userDefined", + "functionScope", "classScope", "fileScope", "globalScope", + "partition", +}; + +std::size_t index_of_or_append(std::vector& list, const std::string& name) { + if (const auto found = std::ranges::find(list, name); found != list.end()) return static_cast(std::distance(list.begin(), found)); + list.push_back(name); + return list.size() - 1; +} + +} // namespace + +std::span base_types() { return BASE_TYPES; } +std::span base_modifiers() { return BASE_MODIFIERS; } + +std::size_t type_index(std::string_view name) { + for (std::size_t i = 0; i < BASE_TYPES.size(); ++i) { + if (BASE_TYPES[i] == name) return i; + } + return 0; +} + +std::uint32_t modifier_bit(std::string_view name) { + for (std::size_t i = 0; i < BASE_MODIFIERS.size(); ++i) { + if (BASE_MODIFIERS[i] == name) return 1u << i; + } + return 0; +} + +Legend build_legend(const Json& coreCapabilities) { + Legend legend; + legend.types.assign(BASE_TYPES.begin(), BASE_TYPES.end()); + legend.modifiers.assign(BASE_MODIFIERS.begin(), BASE_MODIFIERS.end()); + + Json coreTypes = Json::array(); + Json coreModifiers = Json::array(); + if (coreCapabilities.is_object()) { + if (const auto provider = coreCapabilities.find("semanticTokensProvider"); provider != coreCapabilities.end() && provider->is_object()) { + if (const auto declared = provider->find("legend"); declared != provider->end() && declared->is_object()) { + coreTypes = declared->value("tokenTypes", Json::array()); + coreModifiers = declared->value("tokenModifiers", Json::array()); + } + } + } + if (!coreTypes.is_array()) coreTypes = Json::array(); + if (!coreModifiers.is_array()) coreModifiers = Json::array(); + + legend.coreTypeToServer.reserve(coreTypes.size()); + for (const auto& entry : coreTypes) { + legend.coreTypeToServer.push_back(entry.is_string() ? index_of_or_append(legend.types, entry.get()) : 0); + } + legend.coreModifierBitToServer.reserve(coreModifiers.size()); + for (const auto& entry : coreModifiers) { + legend.coreModifierBitToServer.push_back(entry.is_string() ? index_of_or_append(legend.modifiers, entry.get()) : 0); + } + return legend; +} + +Json provider_capability(const Legend& legend) { + return Json { { "legend", Json { { "tokenTypes", legend.types }, { "tokenModifiers", legend.modifiers } } }, + { "full", true }, { "range", true } }; +} + +std::vector decode(const Json& semanticTokensResult) { + std::vector tokens; + if (!semanticTokensResult.is_object()) return tokens; + const auto found = semanticTokensResult.find("data"); + if (found == semanticTokensResult.end() || !found->is_array()) return tokens; + const Json& data = *found; + int line { 0 }; + int column { 0 }; + for (std::size_t i = 0; i + 5 <= data.size(); i += 5) { + if (!data[i].is_number_integer() || !data[i + 1].is_number_integer() || !data[i + 2].is_number_integer() + || !data[i + 3].is_number_integer() || !data[i + 4].is_number_integer()) { + break; // malformed: stop rather than guess at the rest. + } + const int deltaLine { data[i].get() }; + const int deltaStart { data[i + 1].get() }; + const int length { data[i + 2].get() }; + const auto type { static_cast(std::max(0, data[i + 3].get())) }; + const auto modifiers { static_cast(std::max(0, data[i + 4].get())) }; + if (deltaLine == 0) column += deltaStart; else { line += deltaLine; column = deltaStart; } + tokens.push_back(Token { line, column, length, type, modifiers }); + } + return tokens; +} + +Json encode(std::vector tokens) { + std::ranges::stable_sort(tokens, [](const Token& a, const Token& b) { + return a.line != b.line ? a.line < b.line : a.startChar < b.startChar; + }); + Json data = Json::array(); + int line { 0 }; + int column { 0 }; + for (const auto& token : tokens) { + const int deltaLine { token.line - line }; + const int deltaStart { deltaLine == 0 ? token.startChar - column : token.startChar }; + data.push_back(deltaLine); + data.push_back(deltaStart); + data.push_back(token.length); + data.push_back(static_cast(token.type)); + data.push_back(static_cast(token.modifiers)); + line = token.line; + column = token.startChar; + } + return Json { { "data", std::move(data) } }; +} + +Json remap_core_tokens(const Json& coreResult, const Legend& legend) { + std::vector tokens { decode(coreResult) }; + for (auto& token : tokens) { + token.type = token.type < legend.coreTypeToServer.size() ? legend.coreTypeToServer[token.type] : token.type; + std::uint32_t remapped { 0 }; + for (std::size_t bit = 0; bit < legend.coreModifierBitToServer.size(); ++bit) { + if ((token.modifiers & (1u << bit)) != 0) remapped |= (1u << legend.coreModifierBitToServer[bit]); + } + token.modifiers = remapped; + } + return encode(std::move(tokens)); +} + +Json merge(const Json& core, const Json& native) { + if (core.is_null() && native.is_null()) return Json(nullptr); + std::vector coreTokens { decode(core) }; + std::vector merged { coreTokens }; + for (const auto& candidate : decode(native)) { + const bool covered { std::ranges::any_of(coreTokens, [&](const Token& existing) { + if (existing.line != candidate.line) return false; + return candidate.startChar < existing.startChar + existing.length && existing.startChar < candidate.startChar + candidate.length; + }) }; + if (!covered) merged.push_back(candidate); + } + return encode(std::move(merged)); +} + +} // namespace mcppls::orchestrator::tokens diff --git a/src/orchestrator/tokens.cppm b/src/orchestrator/tokens.cppm new file mode 100644 index 0000000..b33c464 --- /dev/null +++ b/src/orchestrator/tokens.cppm @@ -0,0 +1,76 @@ +// Semantic tokens (design doc 2026-09-25 K/§7, contract T0): the legend mcppls owns, mapping the +// core engine's own legend into it by name, the LSP relative encoding, and the merge that lets the +// core engine's tokens win every position they cover while mcppls's own module-syntax tokens fill +// the gaps. Pure functions only: the workspace (which knows the core engine's capabilities and the +// document text) is what has state, this module only transforms what it is given. +export module mcppls.orchestrator.tokens; + +import std; +import nlohmann.json; + +export namespace mcppls::orchestrator::tokens { + +using Json = nlohmann::json; + +// The server's fixed base legend, before any name the core engine declares that is not already in +// it gets appended (build_legend). Positions never change, so the native tokenizer can use +// type_index/modifier_bit as compile-time-stable facts rather than looking anything up at runtime. +std::span base_types(); +std::span base_modifiers(); + +// The index (types) or bit (modifiers) of a fixed-base name; 0 for a name not in the base (a +// programming error, since every name the native tokenizer asks for is one of the base's own). +std::size_t type_index(std::string_view name); +std::uint32_t modifier_bit(std::string_view name); + +// The server's legend, and how to map a core engine's own token-type indices and modifier bits +// into it, by name (duplicates in the core engine's own list collapse to the first entry with that +// name, since they all map into the same, deduplicated, target). A name the core engine declares +// that is not already in the base is appended, so its position is stable across a restart of the +// core engine, or none at all, only for as long as this Legend itself is kept. +struct Legend { + std::vector types; + std::vector modifiers; + std::vector coreTypeToServer; // core engine's token-type index -> this Legend's + std::vector coreModifierBitToServer; // core engine's modifier bit position -> this Legend's +}; + +// `coreCapabilities` is the core engine's own `initialize` capabilities (or an empty object: no +// core engine, or it declared no semanticTokensProvider.legend): the base alone, then. +Legend build_legend(const Json& coreCapabilities); + +// The `semanticTokensProvider` entry this server advertises: `legend`'s types and modifiers, +// `full: true` (no delta -- this server never hands out a resultId a delta could build on) and +// `range: true`. +Json provider_capability(const Legend& legend); + +// One token, decoded to absolute position: `length` and `startChar` are UTF-16 code units, as LSP +// requires; a token never spans two lines. +struct Token { + int line { 0 }; + int startChar { 0 }; + int length { 0 }; + std::size_t type { 0 }; + std::uint32_t modifiers { 0 }; +}; + +// Decodes a `SemanticTokens` result's `data` (LSP's relative encoding) into absolute tokens, in +// the order given. The type/modifier numbers are given back exactly as they came: legend-agnostic, +// so it decodes either a core engine's own result or one already in this server's legend. +// Tolerates a missing or malformed `data`: never throws, gives back what it could decode. +std::vector decode(const Json& semanticTokensResult); + +// Encodes tokens (any order) into a `SemanticTokens` result: sorted ascending by line then column, +// relative-encoded, no `resultId`. +Json encode(std::vector tokens); + +// Decodes a core engine's own `SemanticTokens` result and re-encodes it with every token's +// type/modifiers mapped into `legend`'s indices (build_legend's mapping), dropping its `resultId`. +Json remap_core_tokens(const Json& coreResult, const Legend& legend); + +// The merged answer: `core` (already mapped into the server's legend, e.g. by remap_core_tokens) +// wins every position it covers; `native`'s tokens fill in only where `core` has none. Sorted, +// disjoint, relative-encoded. Null only when both inputs are null (neither engine answered). +Json merge(const Json& core, const Json& native); + +} // namespace mcppls::orchestrator::tokens diff --git a/tests/test_tokens.cpp b/tests/test_tokens.cpp new file mode 100644 index 0000000..9519b82 --- /dev/null +++ b/tests/test_tokens.cpp @@ -0,0 +1,124 @@ +// Semantic tokens (design doc 2026-09-25 K/§7): the server's legend, its mapping of a core +// engine's own token-type indices and modifier bits (including duplicates), the relative +// encode/decode round trip, and the merge that lets the core engine win every position it covers. +import std; +import nlohmann.json; +import mcppls.testing; +import mcppls.orchestrator.tokens; + +using Json = nlohmann::json; +using mcppls::orchestrator::tokens::Legend; +using mcppls::orchestrator::tokens::Token; +namespace tokens = mcppls::orchestrator::tokens; + +int main() { + using namespace mcppls::testing; + + "the base legend has the fixed types and modifiers, in order"_test = [] { + const Legend legend { tokens::build_legend(Json::object()) }; + expect(legend.types.front() == "namespace"); + expect(legend.types.back() == "module"); + expect(std::ranges::find(legend.types, "keyword") != legend.types.end()); + expect(std::ranges::find(legend.types, "unknown") != legend.types.end()); // clangd's own extra + expect(legend.modifiers.front() == "declaration"); + expect(legend.modifiers.back() == "partition"); + expect(legend.coreTypeToServer.empty() && legend.coreModifierBitToServer.empty()); + expect(tokens::type_index("keyword") == static_cast(std::ranges::find(legend.types, "keyword") - legend.types.begin())); + expect(tokens::modifier_bit("partition") == (1u << (legend.modifiers.size() - 1))); + }; + + "clangd's legend, including its own duplicates, maps by name"_test = [] { + // Measured 2026-09-25 on the bundled clangd 23.1: 'variable' appears twice. + const Json clangdCapabilities { { "semanticTokensProvider", + Json { { "legend", Json { { "tokenTypes", Json::array({ "variable", "variable", "namespace" }) }, + { "tokenModifiers", Json::array({ "declaration", "declaration", "static" }) } } } } } }; + const Legend legend { tokens::build_legend(clangdCapabilities) }; + // No name clangd declares here is missing from the fixed base, so the legend is unchanged. + expect(legend.types.size() == tokens::build_legend(Json::object()).types.size()); + expect(fatal(legend.coreTypeToServer.size() == 3u)); + expect(legend.coreTypeToServer[0] == legend.coreTypeToServer[1]); // both "variable" collapse to the same index + expect(legend.types[legend.coreTypeToServer[0]] == "variable"); + expect(legend.types[legend.coreTypeToServer[2]] == "namespace"); + expect(fatal(legend.coreModifierBitToServer.size() == 3u)); + expect(legend.coreModifierBitToServer[0] == legend.coreModifierBitToServer[1]); + expect(legend.modifiers[legend.coreModifierBitToServer[0]] == "declaration"); + }; + + "a name the core engine declares that mcppls does not already have is appended"_test = [] { + const Json capabilities { { "semanticTokensProvider", Json { { "legend", Json { { "tokenTypes", Json::array({ "namespace", "somethingNew" }) }, + { "tokenModifiers", Json::array({ "somethingElse" }) } } } } } }; + const Legend legend { tokens::build_legend(capabilities) }; + const auto base = tokens::build_legend(Json::object()); + expect(legend.types.size() == base.types.size() + 1); + expect(legend.types.back() == "somethingNew"); + expect(legend.coreTypeToServer[1] == legend.types.size() - 1); + expect(legend.modifiers.size() == base.modifiers.size() + 1); + expect(legend.modifiers.back() == "somethingElse"); + }; + + "encode and decode round-trip, sorted and relative"_test = [] { + std::vector input { { 3, 5, 4, 2, 0 }, { 1, 0, 6, 1, 0 }, { 1, 7, 3, 0, 5 } }; + const Json encoded = tokens::encode(input); + expect(fatal(encoded.contains("data") && encoded["data"].is_array())); + const auto decoded = tokens::decode(encoded); + expect(fatal(decoded.size() == 3u)); + // Sorted ascending by line then column. + expect(decoded[0].line == 1 && decoded[0].startChar == 0 && decoded[0].type == 1u); + expect(decoded[1].line == 1 && decoded[1].startChar == 7 && decoded[1].modifiers == 5u); + expect(decoded[2].line == 3 && decoded[2].startChar == 5 && decoded[2].length == 4); + }; + + "decode tolerates a missing or malformed data array"_test = [] { + expect(tokens::decode(Json(nullptr)).empty()); + expect(tokens::decode(Json::object()).empty()); + expect(tokens::decode(Json { { "data", Json::array({ 0, 0, 1 }) } }).empty()); // short entry + expect(tokens::decode(Json { { "data", Json::array({ 0, 0, 1, "x", 0 }) } }).empty()); // wrong type + }; + + "remap_core_tokens drops resultId and maps into the server's indices"_test = [] { + const Json clangdCapabilities { { "semanticTokensProvider", Json { { "legend", + Json { { "tokenTypes", Json::array({ "variable", "namespace" }) }, { "tokenModifiers", Json::array({ "declaration" }) } } } } } }; + const Legend legend { tokens::build_legend(clangdCapabilities) }; + const Json clangdResult { { "resultId", "1" }, { "data", Json::array({ 0, 0, 5, 1, 1 }) } }; // namespace, declaration + const Json remapped = tokens::remap_core_tokens(clangdResult, legend); + expect(!remapped.contains("resultId")); + const auto decoded = tokens::decode(remapped); + expect(fatal(decoded.size() == 1u)); + expect(legend.types[decoded[0].type] == "namespace"); + expect((decoded[0].modifiers & tokens::modifier_bit("declaration")) != 0); + }; + + "merge: the core engine wins every position it covers, native fills the gaps"_test = [] { + // core: one token at [0,0)-[0,6) ("import" as keyword, say); native additionally offers a + // token that overlaps it (dropped) and one that does not (kept). + const Json core = tokens::encode(std::vector { { 0, 0, 6, tokens::type_index("keyword"), 0 } }); + const Json native = tokens::encode(std::vector { { 0, 0, 6, tokens::type_index("module"), 0 }, // overlaps core: dropped + { 0, 7, 3, tokens::type_index("module"), 0 } }); // does not: kept + const Json merged = tokens::merge(core, native); + const auto decoded = tokens::decode(merged); + expect(fatal(decoded.size() == 2u)); + expect(decoded[0].startChar == 0 && decoded[0].type == tokens::type_index("keyword")); + expect(decoded[1].startChar == 7 && decoded[1].type == tokens::type_index("module")); + // Sorted, and adjacent tokens (touching but not overlapping) are not treated as covered. + expect(decoded[0].line == decoded[1].line); + }; + + "merge: with no core engine, native's tokens answer alone (not null)"_test = [] { + const Json native = tokens::encode(std::vector { { 2, 0, 6, tokens::type_index("keyword"), 0 } }); + const Json merged = tokens::merge(Json(nullptr), native); + expect(!merged.is_null()); + expect(tokens::decode(merged).size() == 1u); + }; + + "merge: with neither engine, the answer is null"_test = [] { expect(tokens::merge(Json(nullptr), Json(nullptr)).is_null()); }; + + "provider_capability advertises full and range, no delta"_test = [] { + const Json capability = tokens::provider_capability(tokens::build_legend(Json::object())); + expect(capability.value("full", false) == true); + expect(capability.value("range", false) == true); + expect(!capability.contains("delta")); + expect(capability["legend"]["tokenTypes"].is_array() && !capability["legend"]["tokenTypes"].empty()); + }; + + return report(); +} From aebbc9568115d64646f355fe0cfe325a0c4ce67e Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Fri, 25 Sep 2026 06:48:24 +0800 Subject: [PATCH 17/26] feat(engine): the native engine answers semantic tokens for module syntax, complete or not project::scan_syntax_tokens is a new tokenizer, next to scan_source's lexer: export, module and import keywords, and the module or partition name that follows each, from the text alone -- even a name still being typed (import hello. keeps its keyword and moduleName tokens; module :private and a plain module; produce only the keyword). Unlike scan_source it needs no terminating ';', and a dotted name never crosses a line, so a name cut off by a trailing dot or the end of a line still gives a valid token. The native engine now declares textDocument/semanticTokens/full and /range (merge role, gated by a new TokenOptions.modules) and answers with those tokens: module names as the custom `module` type with a `declaration` modifier for the unit's own declaration and a `partition` modifier on partitions, or as `namespace` with no partition modifier when TokenOptions.moduleType is off -- the fallback for a client that never asked for the custom type. engine::Host gets a new semantic_tokens_changed() hook (default no-op, so nothing else implementing Host needs to change) for the workspace's coalesced workspace/semanticTokens/refresh, wired up in the next commit. Covered by tests/test_scan.cpp (every form in the design doc, including the incomplete and pathological ones, and that no token spans two lines) and a new case in tests/test_server.cpp (moduleType on/off, modules=false, range filtering, and a merge through routing::merge_results). --- src/engine/engine.cppm | 6 +++ src/engine/native.cpp | 69 +++++++++++++++++++++++++++++++- src/engine/native.cppm | 8 +++- src/project/scan.cpp | 91 ++++++++++++++++++++++++++++++++++++++++++ src/project/scan.cppm | 15 +++++++ tests/test_scan.cpp | 78 ++++++++++++++++++++++++++++++++++++ tests/test_server.cpp | 65 ++++++++++++++++++++++++++++++ 7 files changed, 329 insertions(+), 3 deletions(-) diff --git a/src/engine/engine.cppm b/src/engine/engine.cppm index 97b859b..8f10fde 100644 --- a/src/engine/engine.cppm +++ b/src/engine/engine.cppm @@ -121,6 +121,12 @@ public: virtual void engine_settled(std::string_view engineId, const Json& serverCapabilities) = 0; virtual void status_changed() = 0; virtual void request_replan() = 0; + // Semantic tokens (design doc 2026-09-25 K/§7): an engine's next answer for a file may differ + // from what it last gave (clangd (re)started and finished its handshake, a file was set aside + // or handed back, module preparation finished). The workspace sends + // `workspace/semanticTokens/refresh`, coalesced, when the client declared + // `workspace.semanticTokens.refreshSupport`; otherwise this is a no-op. + virtual void semantic_tokens_changed() {} virtual std::vector documents() const = 0; virtual bool has_document(std::string_view clientUri) const = 0; // One file, one name (v1 design 14.3): the URI engines are given, and back to the client's. diff --git a/src/engine/native.cpp b/src/engine/native.cpp index bf68543..b5539e6 100644 --- a/src/engine/native.cpp +++ b/src/engine/native.cpp @@ -7,6 +7,8 @@ import mcppls.base.version; import mcppls.lsp.jsonrpc; import mcppls.lsp.protocol; import mcppls.normalize.plan; +import mcppls.project.scan; +import mcppls.orchestrator.tokens; import mcppls.engine; import mcppls.engine.native.index; @@ -14,6 +16,8 @@ namespace mcppls::engine::native { namespace { +namespace tokens = mcppls::orchestrator::tokens; + std::optional position_of(const Json* params) { if (params == nullptr) return std::nullopt; const Json* position { lsp::find(*params, "position") }; @@ -24,9 +28,61 @@ std::optional position_of(const Json* params) { return base::Position { static_cast(*line), static_cast(*character) }; } +std::optional range_of(const Json* params) { + if (params == nullptr) return std::nullopt; + const Json* range { lsp::find(*params, "range") }; + if (range == nullptr) return std::nullopt; + const Json* start { lsp::find(*range, "start") }; + const Json* end { lsp::find(*range, "end") }; + if (start == nullptr || end == nullptr) return std::nullopt; + const auto startLine = lsp::int_at(*start, "line"); + const auto startCharacter = lsp::int_at(*start, "character"); + const auto endLine = lsp::int_at(*end, "line"); + const auto endCharacter = lsp::int_at(*end, "character"); + if (!startLine || !startCharacter || !endLine || !endCharacter) return std::nullopt; + return base::Range { base::Position { static_cast(*startLine), static_cast(*startCharacter) }, + base::Position { static_cast(*endLine), static_cast(*endCharacter) } }; +} + +// Whether a (single-line, per scan_syntax_tokens) token falls inside a requested range. +bool overlaps_range(const base::Range& requested, const base::Range& token) { + if (token.start.line < requested.start.line || token.start.line > requested.end.line) return false; + if (token.start.line == requested.start.line && token.end.character <= requested.start.character) return false; + if (token.start.line == requested.end.line && token.start.character >= requested.end.character) return false; + return true; +} + +// Native module-syntax tokens (design doc 2026-09-25 K/§7): export/module/import keywords and the +// module or partition names that follow -- from a scan of the document text alone, complete or +// not. `range`, given, limits the answer the way `textDocument/semanticTokens/range` requires. +Json semantic_tokens_of(std::string_view text, bool moduleType, const std::optional& range) { + std::vector out; + for (const auto& syntax : project::scan_syntax_tokens(text)) { + if (range && !overlaps_range(*range, syntax.range)) continue; + tokens::Token token { syntax.range.start.line, syntax.range.start.character, syntax.range.end.character - syntax.range.start.character, 0, 0 }; + switch (syntax.kind) { + case project::SyntaxTokenKind::keyword: + token.type = tokens::type_index("keyword"); + break; + case project::SyntaxTokenKind::moduleName: + token.type = tokens::type_index(moduleType ? "module" : "namespace"); + if (syntax.isDeclaration) token.modifiers |= tokens::modifier_bit("declaration"); + break; + case project::SyntaxTokenKind::partitionName: + token.type = tokens::type_index(moduleType ? "module" : "namespace"); + if (syntax.isDeclaration) token.modifiers |= tokens::modifier_bit("declaration"); + if (moduleType) token.modifiers |= tokens::modifier_bit("partition"); + break; + } + out.push_back(token); + } + return tokens::encode(std::move(out)); +} + class NativeEngine final : public Engine { private: const index::ModuleIndex& index_; + TokenOptions tokenOptions_; std::vector methods_ { { std::string { lsp::method::TEXT_DOCUMENT_DEFINITION }, Role::answer, 100 }, { std::string { lsp::method::TEXT_DOCUMENT_DECLARATION }, Role::answer, 100 }, @@ -34,6 +90,8 @@ class NativeEngine final : public Engine { { std::string { lsp::method::TEXT_DOCUMENT_COMPLETION }, Role::answer, 100 }, { std::string { lsp::method::TEXT_DOCUMENT_DOCUMENT_SYMBOL }, Role::merge, 100 }, { std::string { lsp::method::WORKSPACE_SYMBOL }, Role::merge, 100 }, + { std::string { lsp::method::TEXT_DOCUMENT_SEMANTIC_TOKENS_FULL }, Role::merge, 100 }, + { std::string { lsp::method::TEXT_DOCUMENT_SEMANTIC_TOKENS_RANGE }, Role::merge, 100 }, }; // The module index's answer, or null when the position is not one it answers for. @@ -43,6 +101,13 @@ class NativeEngine final : public Engine { if (method == lsp::method::WORKSPACE_SYMBOL) { return index_.workspace_symbols(request.params != nullptr ? request.params->value("query", std::string {}) : std::string {}); } + if (method == lsp::method::TEXT_DOCUMENT_SEMANTIC_TOKENS_FULL || method == lsp::method::TEXT_DOCUMENT_SEMANTIC_TOKENS_RANGE) { + // initializationOptions.semanticTokens.modules == false: mcppls adds no module-syntax + // tokens of its own (contract T0); the core engine's own tokens, if any, still show. + if (!tokenOptions_.modules || request.path.empty()) return nullptr; + const auto range = method == lsp::method::TEXT_DOCUMENT_SEMANTIC_TOKENS_RANGE ? range_of(request.params) : std::nullopt; + return semantic_tokens_of(request.text, tokenOptions_.moduleType, range); + } if (request.path.empty()) return nullptr; const auto position = position_of(request.params); if (!position) return nullptr; @@ -55,7 +120,7 @@ class NativeEngine final : public Engine { } public: - explicit NativeEngine(const index::ModuleIndex& index) : index_ { index } {} + explicit NativeEngine(const index::ModuleIndex& index, TokenOptions tokenOptions) : index_ { index }, tokenOptions_ { tokenOptions } {} std::string_view id() const override { return ENGINE_ID; } std::span methods() const override { return methods_; } @@ -91,6 +156,6 @@ class NativeEngine final : public Engine { } // namespace -std::unique_ptr make_engine(const index::ModuleIndex& index) { return std::make_unique(index); } +std::unique_ptr make_engine(const index::ModuleIndex& index, TokenOptions tokenOptions) { return std::make_unique(index, tokenOptions); } } // namespace mcppls::engine::native diff --git a/src/engine/native.cppm b/src/engine/native.cppm index b705e3c..6db2346 100644 --- a/src/engine/native.cppm +++ b/src/engine/native.cppm @@ -11,7 +11,13 @@ export namespace mcppls::engine::native { inline constexpr std::string_view ENGINE_ID { "mcppls" }; +// initializationOptions.semanticTokens (design doc 2026-09-25 K/§7, contract T0). +struct TokenOptions { + bool modules { true }; // native module-syntax tokens (export/module/import, module names) at all + bool moduleType { false }; // the client knows the custom `module` type and `partition` modifier +}; + // `index` is the workspace's, and outlives the engine. -std::unique_ptr make_engine(const index::ModuleIndex& index); +std::unique_ptr make_engine(const index::ModuleIndex& index, TokenOptions tokenOptions = {}); } // namespace mcppls::engine::native diff --git a/src/project/scan.cpp b/src/project/scan.cpp index c2c43c0..ffa469d 100644 --- a/src/project/scan.cpp +++ b/src/project/scan.cpp @@ -233,8 +233,99 @@ void skip_attributes(Lexer& lexer, std::optional& token) { } } +// module-name, read leniently: whatever complete identifiers were read before the name broke off +// (a trailing dot, a token that is not an identifier, the end of the file) -- and never across a +// physical line, even if the raw token stream would otherwise happily continue past a newline (as +// scan_source's own parse_name does; that is fine there, since an incomplete name there is simply +// not recorded, but here it would make a token that spans two lines, which LSP does not allow). +// Advances `token` to wherever reading stopped, so the caller's own scan can go on from there. +std::optional> read_dotted_lenient(std::string_view text, Lexer& lexer, std::optional& token) { + if (!token || token->kind != TokenKind::identifier) return std::nullopt; + const std::size_t begin { token->offset }; + const std::size_t lineEnd { [&] { + const auto newline = text.find('\n', begin); + return newline == std::string_view::npos ? text.size() : newline; + }() }; + std::size_t end { token->offset + token->text.size() }; + token = lexer.next(false); + while (token && token->kind == TokenKind::punctuation && token->text == "." && token->offset <= lineEnd) { + std::optional afterDot { lexer.next(false) }; + if (!afterDot || afterDot->kind != TokenKind::identifier || afterDot->offset > lineEnd) { + token = afterDot; + break; + } + end = afterDot->offset + afterDot->text.size(); + token = lexer.next(false); + } + return std::make_pair(begin, end); +} + } // namespace +std::vector scan_syntax_tokens(std::string_view text) { + std::vector tokens; + Lexer lexer { text }; + int braceDepth { 0 }; + std::optional token { lexer.next(false) }; + const auto push = [&](SyntaxTokenKind kind, std::size_t begin, std::size_t end, bool isDeclaration = false) { + if (end <= begin) return; + tokens.push_back(SyntaxToken { kind, base::Range { base::position_at(text, begin), base::position_at(text, end) }, isDeclaration }); + }; + while (token) { + if (token->kind == TokenKind::punctuation) { + if (token->text == "{") ++braceDepth; + if (token->text == "}" && braceDepth > 0) --braceDepth; + token = lexer.next(false); + continue; + } + if (token->kind != TokenKind::identifier || braceDepth != 0) { + token = lexer.next(false); + continue; + } + bool exported { false }; + std::size_t exportBegin { 0 }; + std::size_t exportEnd { 0 }; + if (token->text == "export" && token->startsLine) { + exportBegin = token->offset; + exportEnd = token->offset + token->text.size(); + token = lexer.next(false); + if (!token || token->kind != TokenKind::identifier || (token->text != "module" && token->text != "import")) continue; + exported = true; + } else if (!token->startsLine || (token->text != "module" && token->text != "import")) { + token = lexer.next(false); + continue; + } + const bool isImport { token->text == "import" }; + if (exported) push(SyntaxTokenKind::keyword, exportBegin, exportEnd); + push(SyntaxTokenKind::keyword, token->offset, token->offset + token->text.size()); + token = lexer.next(isImport); + if (!token) break; + + if (isImport && (token->kind == TokenKind::header_name || token->kind == TokenKind::string)) { + // `import
;` / `import "header";`: the header text is a string/header-name + // literal, not a module-type token; other layers already color it. + token = lexer.next(false); + continue; + } + if (isImport && token->kind == TokenKind::punctuation && token->text == ":") { + // `import :partition;`: a partition of the current translation unit's own module, no + // module name of its own. + token = lexer.next(false); + if (const auto partition = read_dotted_lenient(text, lexer, token)) push(SyntaxTokenKind::partitionName, partition->first, partition->second); + continue; + } + const auto name = read_dotted_lenient(text, lexer, token); + if (name) push(SyntaxTokenKind::moduleName, name->first, name->second, !isImport); + // A colon only introduces a partition once a module name was actually read: `module + // :private;`'s colon is the private-module-fragment syntax, not `module`'s own partition. + if (name && token && token->kind == TokenKind::punctuation && token->text == ":") { + token = lexer.next(false); + if (const auto partition = read_dotted_lenient(text, lexer, token)) push(SyntaxTokenKind::partitionName, partition->first, partition->second, !isImport); + } + } + return tokens; +} + ScanResult scan_source(std::string_view text) { ScanResult result; Lexer lexer { text }; diff --git a/src/project/scan.cppm b/src/project/scan.cppm index 753d1e4..9bb7533 100644 --- a/src/project/scan.cppm +++ b/src/project/scan.cppm @@ -35,6 +35,21 @@ struct ScanResult { ScanResult scan_source(std::string_view text); +// A minimal token for syntax highlighting a module declaration or import (design doc 2026-09-25 +// K/§7): produced from the text alone, complete or not, so a person still typing `import hello.` +// sees `import` and `hello` colored while they type. Unlike scan_source, this never requires a +// terminating `;`, and a name cut short (a trailing dot, the end of a line, end of file) is given +// the span of whatever was actually read -- never a token that spans two lines. +enum class SyntaxTokenKind { keyword, moduleName, partitionName }; + +struct SyntaxToken { + SyntaxTokenKind kind { SyntaxTokenKind::keyword }; + base::Range range; + bool isDeclaration { false }; // the name of a `module` or `export module` declaration itself, not an import +}; + +std::vector scan_syntax_tokens(std::string_view text); + spec::Role role_of(const ScanResult& result); // "m" or "m:p" for units that can be imported; empty otherwise. std::string provided_name(const ScanResult& result); diff --git a/tests/test_scan.cpp b/tests/test_scan.cpp index 0a025be..982e551 100644 --- a/tests/test_scan.cpp +++ b/tests/test_scan.cpp @@ -126,5 +126,83 @@ import real; expect(!is_cxx_source_name("a.h") && !is_cxx_source_name("CMakeLists.txt")); }; + // design doc 2026-09-25 K/§7: syntax tokens for the server's native semantic tokens, produced + // from the text alone -- complete or not -- and used by the native engine's tokenizer. + const auto text_at = [](std::string_view text, Range range) -> std::string { + if (range.start.line != range.end.line) return ""; + const auto lines = mcppls::base::split_lines(text); + if (static_cast(range.start.line) >= lines.size()) return ""; + const auto line = lines[static_cast(range.start.line)]; + return std::string { line.substr(static_cast(range.start.character), + static_cast(range.end.character - range.start.character)) }; + }; + + "syntax tokens: a plain import"_test = [text_at] { + const std::string text { "import std;\n" }; + const auto tokens = scan_syntax_tokens(text); + expect(fatal(tokens.size() == 2u)); + expect(tokens[0].kind == SyntaxTokenKind::keyword && text_at(text, tokens[0].range) == "import"); + expect(tokens[1].kind == SyntaxTokenKind::moduleName && text_at(text, tokens[1].range) == "std" && !tokens[1].isDeclaration); + }; + + "syntax tokens: export module with a partition"_test = [text_at] { + const std::string text { "export module a.b:part;\n" }; + const auto tokens = scan_syntax_tokens(text); + expect(fatal(tokens.size() == 4u)); + expect(tokens[0].kind == SyntaxTokenKind::keyword && text_at(text, tokens[0].range) == "export"); + expect(tokens[1].kind == SyntaxTokenKind::keyword && text_at(text, tokens[1].range) == "module"); + expect(tokens[2].kind == SyntaxTokenKind::moduleName && text_at(text, tokens[2].range) == "a.b" && tokens[2].isDeclaration); + expect(tokens[3].kind == SyntaxTokenKind::partitionName && text_at(text, tokens[3].range) == "part" && tokens[3].isDeclaration); + }; + + "syntax tokens: module fragments with no name"_test = [] { + expect(scan_syntax_tokens("module;\n").size() == 1u); // just the keyword + const auto privateFragment = scan_syntax_tokens("module :private;\n"); + expect(fatal(privateFragment.size() == 1u)); + expect(privateFragment[0].kind == SyntaxTokenKind::keyword); + }; + + "syntax tokens: a partition-only import"_test = [text_at] { + const std::string text { "import :part;\n" }; + const auto tokens = scan_syntax_tokens(text); + expect(fatal(tokens.size() == 2u)); + expect(tokens[0].kind == SyntaxTokenKind::keyword && text_at(text, tokens[0].range) == "import"); + expect(tokens[1].kind == SyntaxTokenKind::partitionName && text_at(text, tokens[1].range) == "part" && !tokens[1].isDeclaration); + }; + + "syntax tokens: export import (a re-export) and header imports"_test = [text_at] { + const std::string text { "export import x.y;\nimport ;\nimport \"config.h\";\n" }; + const auto tokens = scan_syntax_tokens(text); + expect(fatal(tokens.size() == 5u)); + expect(tokens[0].kind == SyntaxTokenKind::keyword && text_at(text, tokens[0].range) == "export"); + expect(tokens[1].kind == SyntaxTokenKind::keyword && text_at(text, tokens[1].range) == "import"); + expect(tokens[2].kind == SyntaxTokenKind::moduleName && text_at(text, tokens[2].range) == "x.y"); + expect(tokens[3].kind == SyntaxTokenKind::keyword && text_at(text, tokens[3].range) == "import"); // + expect(tokens[4].kind == SyntaxTokenKind::keyword && text_at(text, tokens[4].range) == "import"); // "config.h" + }; + + "syntax tokens: an incomplete import while typing never crashes and never spans lines"_test = [text_at] { + const std::string trailingDot { "import hello.\n" }; + const auto tokens = scan_syntax_tokens(trailingDot); + expect(fatal(tokens.size() == 2u)); + expect(tokens[0].kind == SyntaxTokenKind::keyword && text_at(trailingDot, tokens[0].range) == "import"); + expect(tokens[1].kind == SyntaxTokenKind::moduleName && text_at(trailingDot, tokens[1].range) == "hello"); + + // The dot's continuation must not reach across the newline into the next statement. + const std::string nextLine { "import hello.\nint x;\n" }; + const auto acrossLines = scan_syntax_tokens(nextLine); + expect(fatal(acrossLines.size() == 2u)); + expect(text_at(nextLine, acrossLines[1].range) == "hello"); + for (const auto& token : acrossLines) expect(token.range.start.line == token.range.end.line); + + expect(scan_syntax_tokens("export module a.\n").size() == 3u); // export, module, "a" + expect(scan_syntax_tokens("import hello. ;\n").size() == 2u); // import, "hello" + expect(scan_syntax_tokens("import\n = 2;\n").size() == 1u); // just "import"; "=" is not a name + }; + + "syntax tokens: inside braces are not module syntax"_test = [] { + expect(scan_syntax_tokens("void f() {\n import x;\n}\n").empty()); + }; + return report(); } diff --git a/tests/test_server.cpp b/tests/test_server.cpp index 8291dee..d2a0f0b 100644 --- a/tests/test_server.cpp +++ b/tests/test_server.cpp @@ -25,6 +25,7 @@ import mcppls.orchestrator.client; import mcppls.orchestrator.documents; import mcppls.orchestrator.journal; import mcppls.orchestrator.routing; +import mcppls.orchestrator.tokens; import mcppls.orchestrator.workspace; using Json = nlohmann::json; @@ -33,6 +34,7 @@ namespace idx = mcppls::index; namespace orch = mcppls::orchestrator; namespace eng = mcppls::engine; namespace cld = mcppls::engine::clangd; +namespace tok = mcppls::orchestrator::tokens; namespace { @@ -354,6 +356,69 @@ int main() { expect(outline.size() == 2u && outline[0]["name"] == "hello.greet:detail") << outline.dump(); }; + // design doc 2026-09-25 K/§7: the native engine's own semantic tokens, and the merge with a + // (simulated) core engine's already-remapped answer. + "native engine: semantic tokens, moduleType, modules=false, and the merge"_test = [] { + const auto index = fixture_index(); + const std::string text { "export module hello.greet:detail;\nimport std;\n" }; + auto view = [&](std::string_view method, const Json& params) { return eng::RequestView { method, ¶ms, "/p/src/greet/detail.cppm", text }; }; + const Json empty = Json::object(); + + // moduleType == false (default): module names are `namespace`, no `partition` modifier. + // Six tokens: export, module, "hello.greet" (declaration), "detail" (declaration + // partition) on line 0; import, "std" on line 1. + const auto plain = eng::native::make_engine(index, eng::native::TokenOptions { .modules = true, .moduleType = false }); + std::optional answer; + plain->request(view("textDocument/semanticTokens/full", empty), Json::object(), [&](eng::Answer a) { answer = std::move(a); }); + expect(fatal(answer.has_value() && answer->kind == eng::Answer::Kind::result)); + auto tokens = tok::decode(answer->value); + expect(fatal(tokens.size() == 6u)) << tokens.size(); + expect(tokens[0].type == tok::type_index("keyword")); // export + expect(tokens[1].type == tok::type_index("keyword")); // module + expect(tokens[2].type == tok::type_index("namespace") && tokens[2].modifiers == tok::modifier_bit("declaration")); // hello.greet + expect(tokens[3].type == tok::type_index("namespace") && tokens[3].modifiers == tok::modifier_bit("declaration")); // detail (no partition bit: moduleType is off) + expect(tokens[4].type == tok::type_index("keyword")); // import + expect(tokens[5].type == tok::type_index("namespace") && tokens[5].modifiers == 0u); // std (not a declaration) + + // moduleType == true: the custom `module` type, and the `partition` modifier. + const auto withModuleType = eng::native::make_engine(index, eng::native::TokenOptions { .modules = true, .moduleType = true }); + answer.reset(); + withModuleType->request(view("textDocument/semanticTokens/full", empty), Json::object(), [&](eng::Answer a) { answer = std::move(a); }); + tokens = tok::decode(answer->value); + expect(fatal(tokens.size() == 6u)); + expect(tokens[2].type == tok::type_index("module")); + expect(tokens[3].type == tok::type_index("module") && (tokens[3].modifiers & tok::modifier_bit("partition")) != 0); + + // modules == false: mcppls adds no module-syntax tokens at all, and does not claim the method. + const auto disabled = eng::native::make_engine(index, eng::native::TokenOptions { .modules = false, .moduleType = false }); + expect(!disabled->claims(view("textDocument/semanticTokens/full", empty))); + + // range: only the second line's tokens (both "import" and "std" are on it). + const Json rangeParams { { "range", Json { { "start", Json { { "line", 1 }, { "character", 0 } } }, { "end", Json { { "line", 2 }, { "character", 0 } } } } } }; + answer.reset(); + plain->request(view("textDocument/semanticTokens/range", rangeParams), Json::object(), [&](eng::Answer a) { answer = std::move(a); }); + tokens = tok::decode(answer->value); + expect(fatal(tokens.size() == 2u)); + expect(tokens[0].line == 1 && tokens[1].line == 1); + + // Merge, through routing::merge_results itself: a (simulated) core engine's own answer, + // already remapped into the server's legend by the workspace (test_tokens.cpp covers that + // remapping on its own), wins the position it covers; native's answer fills the rest. + answer.reset(); + plain->request(view("textDocument/semanticTokens/full", empty), Json::object(), [&](eng::Answer a) { answer = std::move(a); }); + expect(fatal(answer.has_value())); + const Json coreResult = tok::encode(std::vector { { 0, 0, 6, tok::type_index("keyword"), 0 } }); // covers "export" on line 0 + const std::vector> mergedInputs { { "fake-core", coreResult }, { "mcppls", answer->value } }; + const Json merged = orch::merge_results("textDocument/semanticTokens/full", mergedInputs); + const auto mergedTokens = tok::decode(merged); + expect(std::ranges::any_of(mergedTokens, [](const tok::Token& t) { return t.line == 0 && t.startChar == 0 && t.type == tok::type_index("keyword"); })); + // Sorted and non-overlapping: no two tokens on the same line share any column. + for (std::size_t i = 1; i < mergedTokens.size(); ++i) { + if (mergedTokens[i].line != mergedTokens[i - 1].line) continue; + expect(mergedTokens[i].startChar >= mergedTokens[i - 1].startChar + mergedTokens[i - 1].length); + } + }; + "clangd's module build failures are recognized"_test = [] { const auto failure = cld::parse_module_failure( R"(E[03:15:19.435] Failed to build module greet; due to Failed to compile C:\Program Files\VS\modules\std.ixx. Use '--log=verbose' to view detailed failure reasons.)"); From ad7415c2bbfb374a2eb00e57861279bf5df10df4 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Fri, 25 Sep 2026 06:48:36 +0800 Subject: [PATCH 18/26] feat(orchestrator): the workspace remaps and merges semantic tokens, and refreshes them on change merge_capabilities now advertises semanticTokensProvider from the server's own legend (full: true, range: true, no delta), built from whatever the core engine declared -- never appended to the core engine's own legend, so a restart, a --clangd override or no core engine at all cannot shift an index a client has already seen. routing::merge_results merges textDocument/semanticTokens/full and /range through tokens::merge, kept pure: the workspace remaps the core engine's raw answer into the server's legend right where it arrives (route_client_request), before it ever reaches merge_results, so routing itself only ever combines two answers already in the same index space. A full/delta request is rewritten to full before routing sees it, since this server hands out no resultId a delta could build on. initializationOptions.semanticTokens ({modules, moduleType}) is read in session.cpp and carried on SessionOptions to the native engine's factory in cli/options.cpp, the same path compiler/kit/engine already take. Workspace gains semantic_tokens_changed(): coalesced to at most one workspace/semanticTokens/refresh every ~500ms, sent only once initialize is answered and only when the client declared workspace.semanticTokens.refreshSupport. clangd calls it at the one place that already tracks every set-aside and hand-back (update_quarantine_issue_), at its own handshake, and when module preparation goes idle -- three one-line calls, no other change to clangd.cpp, since another track is changing that file in parallel. --- src/cli/options.cpp | 4 ++- src/engine/clangd.cpp | 9 ++++++ src/orchestrator/routing.cpp | 11 +++++++ src/orchestrator/workspace.cpp | 55 +++++++++++++++++++++++++++++++-- src/orchestrator/workspace.cppm | 5 +++ src/server/session.cpp | 9 ++++++ 6 files changed, 90 insertions(+), 3 deletions(-) diff --git a/src/cli/options.cpp b/src/cli/options.cpp index 234485f..3f61f71 100644 --- a/src/cli/options.cpp +++ b/src/cli/options.cpp @@ -20,7 +20,9 @@ using namespace mcpplibs; orchestrator::EngineFactories engine_factories(const orchestrator::SessionOptions& options, const engine::PayloadPaths& payload, bool payloadCorrupt) { orchestrator::EngineFactories factories; - factories.modules = [](const index::ModuleIndex& index) { return engine::native::make_engine(index); }; + // initializationOptions.semanticTokens (design doc 2026-09-25 K/§7, contract T0). + const engine::native::TokenOptions tokenOptions { options.semanticTokensModules, options.semanticTokensModuleType }; + factories.modules = [tokenOptions](const index::ModuleIndex& index) { return engine::native::make_engine(index, tokenOptions); }; if (options.engine == "none") return factories; if (options.engine != "clangd") base::log::warning("unknown engine {}; using clangd", options.engine); factories.core = [options, payload, payloadCorrupt]() -> std::unique_ptr { diff --git a/src/engine/clangd.cpp b/src/engine/clangd.cpp index 547ef47..c5ba469 100644 --- a/src/engine/clangd.cpp +++ b/src/engine/clangd.cpp @@ -72,6 +72,10 @@ class ClangdEngine final : public Engine { { std::string { EVERY_METHOD }, Role::answer, 0 }, { std::string { lsp::method::TEXT_DOCUMENT_DOCUMENT_SYMBOL }, Role::merge, 0 }, { std::string { lsp::method::WORKSPACE_SYMBOL }, Role::merge, 0 }, + // design doc 2026-09-25 K/§7, contract T0: mcppls's own module-syntax tokens fill the gaps + // clangd leaves (it tokenizes no import/module/export line at all, measured on 23.1.0). + { std::string { lsp::method::TEXT_DOCUMENT_SEMANTIC_TOKENS_FULL }, Role::merge, 0 }, + { std::string { lsp::method::TEXT_DOCUMENT_SEMANTIC_TOKENS_RANGE }, Role::merge, 0 }, }; std::string databaseDirectory_; @@ -1431,6 +1435,7 @@ class ClangdEngine final : public Engine { earlyExits_ = 0; accept_traffic_if_ready_(); host_->status_changed(); + host_->semantic_tokens_changed(); // design doc 2026-09-25 K/§7: clangd (re)started break; } case Purpose::client: { @@ -2112,6 +2117,9 @@ class ClangdEngine final : public Engine { members.size() == 1 ? "it" : "them", members.size() == 1 ? "it" : "they"), "mcppls.restartServer" }); } host_->status_changed(); + // design doc 2026-09-25 K/§7: called at every point a file is set aside or handed back, so + // whichever engine now answers semantic tokens for it, the client asks again. + host_->semantic_tokens_changed(); } // A restart now, or as soon as the gate allows (robustness design C4). @@ -2518,6 +2526,7 @@ class ClangdEngine final : public Engine { if (heldPrimeUnits_.empty() || primer_.busy() || !awaitingDiagnostics_.empty() || !held_.empty()) return; log::info("module preparation idle ({}): closing {} prime units", host_->root_directory(), heldPrimeUnits_.size()); close_prime_units_(); + host_->semantic_tokens_changed(); // design doc 2026-09-25 K/§7: module preparation finished } void close_prime_units_() { diff --git a/src/orchestrator/routing.cpp b/src/orchestrator/routing.cpp index 0e0336c..7e33be3 100644 --- a/src/orchestrator/routing.cpp +++ b/src/orchestrator/routing.cpp @@ -4,6 +4,7 @@ import std; import nlohmann.json; import mcppls.engine; import mcppls.lsp.jsonrpc; +import mcppls.orchestrator.tokens; namespace mcppls::orchestrator { @@ -57,6 +58,12 @@ Json merge_results(std::string_view method, std::span=0.2 <1" } }; // usable plan W9.1: without this, a client has no reason to ever send diff --git a/src/orchestrator/workspace.cpp b/src/orchestrator/workspace.cpp index aa9ee33..fdbea2d 100644 --- a/src/orchestrator/workspace.cpp +++ b/src/orchestrator/workspace.cpp @@ -36,6 +36,7 @@ import mcppls.orchestrator.client; import mcppls.orchestrator.documents; import mcppls.orchestrator.instance; import mcppls.orchestrator.routing; +import mcppls.orchestrator.tokens; namespace mcppls::orchestrator { @@ -171,6 +172,12 @@ struct Workspace::Impl final : engine::Host { engine::Engine* moduleEngine { nullptr }; Json coreCapabilities = Json::object(); bool settledReported { false }; + // Semantic tokens (design doc 2026-09-25 K/§7): rebuilt whenever the core engine settles (even + // with none), so a request's core-engine tokens are always remapped by the same legend + // `merge_capabilities` advertised for this same `coreCapabilities`. + tokens::Legend tokensLegend { tokens::build_legend(Json::object()) }; + bool clientSupportsTokensRefresh { false }; + std::optional tokensRefreshAt; // coalesced: at most one refresh per ~500ms // Diagnostics published by engines other than mcppls's own, per engine and client URI. std::map>, std::less<>> engineDiagnostics; std::map> publishedDiagnostics; @@ -333,6 +340,10 @@ struct Workspace::Impl final : engine::Host { void engine_settled(std::string_view engineId, const Json& serverCapabilities) override { if (coreEngine != nullptr && engineId != coreEngine->id()) return; if (coreEngine != nullptr) coreCapabilities = serverCapabilities; + // Rebuilt from the very capabilities merge_capabilities is about to see (or already saw, + // for the first root), so a request's core-engine tokens are always remapped by the same + // legend the client was told about. + tokensLegend = tokens::build_legend(coreCapabilities); if (settledReported || !onEngineSettled) return; settledReported = true; auto callback = std::move(onEngineSettled); @@ -344,6 +355,15 @@ struct Workspace::Impl final : engine::Host { void request_replan() override { schedule_replan(); } void record_event(std::string_view kind, Json detail) override { journal.add(kind, std::move(detail)); } + // Semantic tokens (design doc 2026-09-25 K/§7): coalesced to at most one + // workspace/semanticTokens/refresh every ~500ms, and never before this root's own initialize + // was answered (the same gate update_status uses). + void semantic_tokens_changed() override { + if (!clientSupportsTokensRefresh || !initializeAnswered) return; + if (tokensRefreshAt) return; + tokensRefreshAt = Clock::now() + std::chrono::milliseconds { 500 }; + } + std::vector documents() const override { std::vector views; for (const Document* document : documents_.all()) views.push_back(view_of(*document)); @@ -429,6 +449,7 @@ struct Workspace::Impl final : engine::Host { consider(sdkCheckAt); consider(statusFlushAt); consider(leaseRenewAt); + consider(tokensRefreshAt); for (const auto& engine : engines) consider(engine->next_deadline()); return deadline; } @@ -492,7 +513,15 @@ struct Workspace::Impl final : engine::Host { job.clientId = message["id"]; job.method = message.value("method", std::string {}); job.message = message; - job.params = message.contains("params") ? message["params"] : Json::object(); + // Semantic tokens (design doc 2026-09-25 K/§7): this server advertises `full` with no + // `delta` (contract T0), so it never hands out a resultId a delta request could build on. + // A client that sends one anyway is answered like `full`, which LSP allows. + if (job.method == "textDocument/semanticTokens/full/delta") { + job.method = "textDocument/semanticTokens/full"; + job.message["method"] = job.method; + if (job.message.contains("params") && job.message["params"].is_object()) job.message["params"].erase("previousResultId"); + } + job.params = job.message.contains("params") ? job.message["params"] : Json::object(); const std::string uri { uri_of_params(job.params) }; job.path = uri.empty() ? std::string {} : path_of_uri(uri); const Document* document { uri.empty() ? nullptr : documents_.find(uri) }; @@ -505,9 +534,22 @@ struct Workspace::Impl final : engine::Host { if (!selection.mergers.empty()) { job.merging = true; job.awaiting = selection.mergers.size(); + // Semantic tokens: the core engine's own answer arrives in its own legend's indices; + // remapped into this server's legend right here, once, so routing::merge_results (and + // everything downstream) only ever sees the server's own index space (routing itself + // stays a pure function of what several engines already answered). + const bool remapCoreTokens { coreEngine != nullptr + && (job.method == "textDocument/semanticTokens/full" || job.method == "textDocument/semanticTokens/range") }; for (engine::Engine* merger : selection.mergers) { const std::string engineId { merger->id() }; - merger->request(job.view, job.message, [this, jobId, engineId](engine::Answer answer) { merge_answer(jobId, engineId, std::move(answer)); }); + engine::Reply reply { [this, jobId, engineId](engine::Answer answer) { merge_answer(jobId, engineId, std::move(answer)); } }; + if (remapCoreTokens && engineId == coreEngine->id()) { + reply = [this, jobId, engineId](engine::Answer answer) { + if (answer.kind == engine::Answer::Kind::result) answer.value = tokens::remap_core_tokens(answer.value, tokensLegend); + merge_answer(jobId, engineId, std::move(answer)); + }; + } + merger->request(job.view, job.message, std::move(reply)); if (!jobs.contains(jobId)) return; } return; @@ -1389,6 +1431,10 @@ struct Workspace::Impl final : engine::Host { statusFlushAt.reset(); update_status(); } + if (tokensRefreshAt && *tokensRefreshAt <= now) { + tokensRefreshAt.reset(); + client.notify(lsp::method::WORKSPACE_SEMANTIC_TOKENS_REFRESH, Json::object()); + } if (sdkCheckAt && *sdkCheckAt <= now) { sdkCheckAt.reset(); if (kit && spec::requires_macos_sdk(*kit) && macosSdk.empty()) { @@ -1444,6 +1490,11 @@ bool Workspace::owns_path(std::string_view path) const { return !path.empty() && void Workspace::start(Json clientParams, bool clientSupportsStatus, bool usePolling, std::function onEngineSettled) { impl_->clientParams = std::move(clientParams); impl_->clientSupportsStatus = clientSupportsStatus; + // Semantic tokens (design doc 2026-09-25 K/§7): workspace.semanticTokens.refreshSupport. + if (const Json* supported = lsp::find_path(impl_->clientParams, { "capabilities", "workspace", "semanticTokens", "refreshSupport" }); + supported != nullptr && supported->is_boolean()) { + impl_->clientSupportsTokensRefresh = supported->get(); + } impl_->onEngineSettled = std::move(onEngineSettled); impl_->dynamicWatch = !usePolling; if (usePolling) impl_->start_watch_polling(); diff --git a/src/orchestrator/workspace.cppm b/src/orchestrator/workspace.cppm index 6983481..5bc0799 100644 --- a/src/orchestrator/workspace.cppm +++ b/src/orchestrator/workspace.cppm @@ -50,6 +50,11 @@ struct SessionOptions { std::chrono::milliseconds requestTimeout { std::chrono::seconds { 60 } }; // Registered workarounds turned off (import-hang plan §9): to see whether one is still needed. std::vector disabledWorkarounds; + // initializationOptions.semanticTokens (design doc 2026-09-25 K/§7, contract T0): native + // module-syntax tokens are on by default; the custom `module` type and `partition` modifier + // only for a client that says it knows them. + bool semanticTokensModules { true }; + bool semanticTokensModuleType { false }; // This program, to run `mcppls review` for an editor's review command (overall design 7.7). std::string serverExecutable; // Given by the composition root; a test can substitute engines that start no process. diff --git a/src/server/session.cpp b/src/server/session.cpp index 0a9e21c..32ab3e7 100644 --- a/src/server/session.cpp +++ b/src/server/session.cpp @@ -263,6 +263,15 @@ class Session { environment && (*environment == "auto" || *environment == "editor")) { options_.toolEnvironment = *environment; } + // design doc 2026-09-25 K/§7, contract T0: initializationOptions.semanticTokens. + if (const Json* semanticTokens = lsp::find(*init, "semanticTokens"); semanticTokens != nullptr && semanticTokens->is_object()) { + if (const auto modules = semanticTokens->find("modules"); modules != semanticTokens->end() && modules->is_boolean()) { + options_.semanticTokensModules = modules->get(); + } + if (const auto moduleType = semanticTokens->find("moduleType"); moduleType != semanticTokens->end() && moduleType->is_boolean()) { + options_.semanticTokensModuleType = moduleType->get(); + } + } } // The environment the user's build tools run in is resolved once, in the background, before // anything needs it (design 4.3): an editor started from a desktop entry has none of the From b6ff4255aa15d550066604b0c0fd5e683653f8c9 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Fri, 25 Sep 2026 06:48:45 +0800 Subject: [PATCH 19/26] feat(conformance): a semantic-tokens check kind, for both module-name types The runner decodes textDocument/semanticTokens/full (or /range, with "range") with the legend initialize gave, and passes when every entry of "expect" ({"line", "text", "type", "modifiers"?}) is among the decoded tokens. A scenario's own "initialization-options" object is now merged into the runner's initializationOptions, so a fixture can ask for something no --client profile sends. inferred (with the real bundled clangd) asks for {"semanticTokens": {"moduleType": true}} and checks a declaration's module and partition names come back as the custom `module` type with `declaration`/`partition` modifiers, merged with clangd, which contributes nothing on those lines. engine-none (no core engine) keeps the default and checks the `namespace` fallback instead, so both forms in the design doc are exercised. conformance/README.md's kind table documents the new kind and the new scenario field. --- conformance/README.md | 5 + .../fixtures/engine-none/scenario.json | 7 +- conformance/fixtures/inferred/scenario.json | 13 ++- src/bin/conformance.cpp | 94 ++++++++++++++++++- 4 files changed, 114 insertions(+), 5 deletions(-) diff --git a/conformance/README.md b/conformance/README.md index 9c48bfb..a262ffd 100644 --- a/conformance/README.md +++ b/conformance/README.md @@ -121,6 +121,10 @@ are `[line, character]`, zero-based, UTF-16. A check with `"text"` opens its fil content; a check with `"optional": true` reports `SKIP` instead of failing, and `"timeout": SECONDS` waits less than the run's `--timeout`. `"file"` and `"folder"` on a check, like every other path a scenario names, are relative to the fixture's own root, never to a specific workspace folder. +`"initialization-options"` on the scenario is an object merged into the runner's own +`initializationOptions` (over whatever `--client` profile set), so a fixture can ask for something +`--client` does not, such as `{"semanticTokens": {"moduleType": true}}` (design doc 2026-09-25 +K/§7). `"initialize-within": SECONDS` on the scenario fails the run when `initialize` is answered later than that (the runner itself waits up to 120 s): a server that answers eventually is not enough where the point is that it answers at once (`clangd-cannot-load`). @@ -159,6 +163,7 @@ always has been. | `completion-contains` | a completion label starts with `expect`; `insert: [line, text]` adds a line first, `edit` changes another open buffer without saving it | | `references-span` | the references include every path in `expect` | | `document-symbol-contains` | the outline has a top-level symbol named `expect` | +| `semantic-tokens` | `textDocument/semanticTokens/full` (or `/range`, with `"range"`) for `"file"` (optionally with an unsaved `"text"`), decoded with the legend `initialize` gave, has every entry of `"expect"` (`{"line", "text", "type", "modifiers"?}`; `"modifiers"` is a list, and optional) among its tokens (design doc 2026-09-25 K/§7) | | `module-graph-contains` | `cxxModules/graph` lists module `expect`; retries within the check's own timeout, so it doubles as "a change reaches the graph within N seconds" (usable plan W9.3's `watch-polling`) | | `set-context` | sends `cxxModules/setContext` with `"context"` (usable plan W9.2), then a hover at `"at"` contains `expect`, retried the same way as `hover-contains` | | `write-file` | writes `"content"` (default: a fresh `export module ;`; `"content-from"` copies another workspace file) to `"file"` directly, the way a file system watcher — or, without one, the server's own polling fallback — would notice it, without the runner opening it as a document (usable plan W9.3); with `"expect-reload": true`, also waits for the status to pass through `loading` again (S2-5-1) | diff --git a/conformance/fixtures/engine-none/scenario.json b/conformance/fixtures/engine-none/scenario.json index 8993050..bdccff0 100644 --- a/conformance/fixtures/engine-none/scenario.json +++ b/conformance/fixtures/engine-none/scenario.json @@ -28,6 +28,11 @@ "expect": [{ "path": "/partitions", "contains": "hello.greet:detail" }] }, { "id": "D1-daemon-status", "kind": "cli", "args": ["daemon", "status"], "expect": [{ "path": "/sessions", "equals": 1 }, { "path": "/status/engine/name", "equals": "none" }] }, { "id": "D1-daemon-stop", "kind": "cli", "args": ["daemon", "stop"], "expect": [{ "path": "/ok", "equals": true }] }, - { "id": "R1-elsewhere", "kind": "responds", "file": "src/main.cpp", "at": [4, 10], "timeout": 5 } + { "id": "R1-elsewhere", "kind": "responds", "file": "src/main.cpp", "at": [4, 10], "timeout": 5 }, + { "id": "K1-semantic-tokens", "kind": "semantic-tokens", "file": "src/main.cpp", "expect": [ + { "line": 0, "text": "import", "type": "keyword" }, + { "line": 1, "text": "import", "type": "keyword" }, + { "line": 1, "text": "hello.greet", "type": "namespace" } + ] } ] } diff --git a/conformance/fixtures/inferred/scenario.json b/conformance/fixtures/inferred/scenario.json index 1d7ed1e..b12a6f0 100644 --- a/conformance/fixtures/inferred/scenario.json +++ b/conformance/fixtures/inferred/scenario.json @@ -2,6 +2,7 @@ "name": "inferred", "description": "Loose module sources, no build system, no compiler: the semantic kit provides libc++ semantics.", "server-arguments": ["--no-discover"], + "initialization-options": { "semanticTokens": { "moduleType": true } }, "checks": [ { "id": "S1", "kind": "status", "source": "inferred", "profile-kind": "semantic-kit", "state": "ready", "engine-name": "clangd", "engines-include": ["mcppls", "clangd"] }, { "id": "I1-second-instance", "kind": "second-instance", "notice-code": "shared-workspace" }, @@ -19,6 +20,16 @@ { "id": "C6", "kind": "completion-contains", "file": "src/main.cpp", "insert": [5, " hello::"], "at": [5, 11], "expect": "greet" }, { "id": "C7", "kind": "completion-contains", "file": "src/main.cpp", "at": [5, 11], "expect": "greet2", "edit": { "file": "src/greet/greet.cppm", "replace": "export namespace hello {", "with": "export namespace hello {\n int greet2() { return 2; }" } }, - { "id": "C9", "kind": "references-span", "file": "src/main.cpp", "at": [4, 31], "expect": ["src/main.cpp", "src/greet/greet.cppm"] } + { "id": "C9", "kind": "references-span", "file": "src/main.cpp", "at": [4, 31], "expect": ["src/main.cpp", "src/greet/greet.cppm"] }, + { "id": "K1-semantic-tokens", "kind": "semantic-tokens", "file": "src/greet/greet.cppm", "expect": [ + { "line": 0, "text": "export", "type": "keyword" }, + { "line": 0, "text": "module", "type": "keyword" }, + { "line": 0, "text": "hello.greet", "type": "module", "modifiers": ["declaration"] }, + { "line": 1, "text": "export", "type": "keyword" }, + { "line": 1, "text": "import", "type": "keyword" }, + { "line": 1, "text": "detail", "type": "module", "modifiers": ["partition"] }, + { "line": 2, "text": "import", "type": "keyword" }, + { "line": 2, "text": "std", "type": "module" } + ] } ] } diff --git a/src/bin/conformance.cpp b/src/bin/conformance.cpp index f9125b4..1a43883 100644 --- a/src/bin/conformance.cpp +++ b/src/bin/conformance.cpp @@ -25,10 +25,12 @@ import mcppls.platform.process; import mcppls.platform.task; import mcppls.lsp.jsonrpc; import mcppls.lsp.connection; +import mcppls.orchestrator.tokens; namespace base = mcppls::base; namespace fs = mcppls::platform::fs; namespace lsp = mcppls::lsp; +namespace tokens = mcppls::orchestrator::tokens; using Json = nlohmann::json; using Clock = std::chrono::steady_clock; @@ -930,6 +932,7 @@ class Scenario { std::unique_ptr mcp_; // started by the first mcp check std::unique_ptr mcpDaemon_; // the first mcp check "via": "daemon" std::string mcpFailure_; + Json semanticTokensLegend_ = Json::object(); // initialize's capabilities.semanticTokensProvider.legend McpClient* mcp_client(bool daemon) { auto& kept = daemon ? mcpDaemon_ : mcp_; @@ -953,9 +956,10 @@ class Scenario { Scenario(Client& client, const Options& options, std::vector serverArguments, std::string workspace, std::chrono::seconds timeout, std::map prepared, std::string cacheDirectory, bool expectWarm, - std::map> moduleFilesBefore) + std::map> moduleFilesBefore, Json semanticTokensLegend = Json::object()) : client_ { client }, options_ { options }, serverArguments_ { std::move(serverArguments) }, workspace_ { std::move(workspace) }, timeout_ { timeout }, prepared_ { std::move(prepared) }, - cacheDirectory_ { std::move(cacheDirectory) }, expectWarm_ { expectWarm }, moduleFilesBefore_ { std::move(moduleFilesBefore) } {} + cacheDirectory_ { std::move(cacheDirectory) }, expectWarm_ { expectWarm }, moduleFilesBefore_ { std::move(moduleFilesBefore) }, + semanticTokensLegend_ ( std::move(semanticTokensLegend) ) {} std::string uri(std::string_view relative) const { return base::path_to_uri(base::join_path(workspace_, relative)); } @@ -982,6 +986,44 @@ class Scenario { { "contentChanges", Json::array({ Json { { "text", text } } }) } }); } + // A `textDocument/semanticTokens/full` (or `/range`) result, decoded with the legend + // `initialize` gave, and the text of the token it names -- from `content`, which must be the + // buffer the request was answered against. + struct DecodedToken { + int line { 0 }; + int startChar { 0 }; + int length { 0 }; + std::string type; + std::vector modifiers; + std::string text; + }; + + std::vector decode_semantic_tokens(const Json& result, const std::string& content) const { + const Json types = semanticTokensLegend_.value("tokenTypes", Json::array()); + const Json modifiers = semanticTokensLegend_.value("tokenModifiers", Json::array()); + const auto lines = base::split_lines(content); + std::vector decoded; + for (const auto& token : tokens::decode(result)) { + DecodedToken entry; + entry.line = token.line; + entry.startChar = token.startChar; + entry.length = token.length; + entry.type = token.type < types.size() && types[token.type].is_string() ? types[token.type].get() : std::string {}; + for (std::size_t bit = 0; bit < modifiers.size(); ++bit) { + if ((token.modifiers & (1u << bit)) != 0 && modifiers[bit].is_string()) entry.modifiers.push_back(modifiers[bit].get()); + } + if (token.line >= 0 && static_cast(token.line) < lines.size()) { + const std::string_view lineText { lines[static_cast(token.line)] }; + if (token.startChar >= 0 && static_cast(token.startChar) <= lineText.size()) { + const std::size_t available { lineText.size() - static_cast(token.startChar) }; + entry.text = std::string { lineText.substr(static_cast(token.startChar), std::min(available, static_cast(std::max(0, token.length)))) }; + } + } + decoded.push_back(std::move(entry)); + } + return decoded; + } + // Repeats a request until `accept` holds, because the engine may still be preparing modules. std::pair retry(std::string_view method, const std::function& params, const std::function& accept) { const auto deadline = Clock::now() + timeout_; @@ -1482,6 +1524,40 @@ class Scenario { }); return { ok, lsp::dump(result).substr(0, 160) }; } + if (kind == "semantic-tokens") { + // design doc 2026-09-25 K/§7: every entry of "expect" ({"line", "text", "type", + // "modifiers"?}) must be one of the decoded tokens; "modifiers" (a list) is optional. + open(file); + const std::string content { text_of(file) }; + const Json expected = check.value("expect", Json::array()); + const auto range = check.find("range"); + const std::string method { range != check.end() ? std::string { "textDocument/semanticTokens/range" } : std::string { "textDocument/semanticTokens/full" } }; + auto [ok, result] = retry(method, + [&] { + Json params { { "textDocument", Json { { "uri", uri(file) } } } }; + if (range != check.end()) params["range"] = *range; + return params; + }, + [&](const Json& value) { + const auto decoded = decode_semantic_tokens(value, content); + return std::ranges::all_of(expected, [&](const Json& want) { + const int line { want.value("line", -1) }; + const std::string text { want.value("text", std::string {}) }; + const std::string type { want.value("type", std::string {}) }; + std::vector modifiers; + for (const auto& modifier : want.value("modifiers", Json::array())) modifiers.push_back(modifier.get()); + return std::ranges::any_of(decoded, [&](const DecodedToken& token) { + return token.line == line && token.text == text && token.type == type + && std::ranges::all_of(modifiers, [&](const std::string& modifier) { return std::ranges::find(token.modifiers, modifier) != token.modifiers.end(); }); + }); + }); + }); + std::string detail; + for (const auto& token : decode_semantic_tokens(result, content)) { + detail += std::format("[{}:{} '{}' {} {}] ", token.line, token.startChar, token.text, token.type, lsp::dump(token.modifiers)); + } + return { ok, detail.substr(0, std::min(detail.size(), 200)) }; + } if (kind == "report") { // robustness design O3: cxxModules/report, held to "expect" like a tool's result, retried within the check's time // (a plan or an engine may still be on its way). @@ -1779,6 +1855,12 @@ int run(Options options) { } break; } + // A scenario's own "initialization-options" object, merged on top of whatever the client + // profile above set -- design doc 2026-09-25 K/§7's conformance cases ask for + // {"semanticTokens": {"moduleType": true}} this way, without a client profile of their own. + if (const auto extra = scenario.find("initialization-options"); extra != scenario.end() && extra->is_object()) { + for (auto entry = extra->begin(); entry != extra->end(); ++entry) initializationOptions[entry.key()] = entry.value(); + } // usable plan W9.1: a fixture with several roots names them, relative to the fixture's own // root, in "folders"; a check names a file or a folder the same way, relative to that root, // regardless of how many workspace folders the fixture actually declares. @@ -1810,7 +1892,13 @@ int run(Options options) { plainLike ? " (plain client)" : ""); client.notify("initialized", Json::object()); - Scenario runner { client, options, serverArguments, workspace, options.timeout, std::move(prepared), cacheDirectory, options.expectWarm, std::move(moduleFilesBefore) }; + // design doc 2026-09-25 K/§7: the legend this server just advertised, so a "semantic-tokens" + // check can decode a result's type/modifier indices back into names. + const Json semanticTokensLegend = lsp::find_path(*initialized, { "capabilities", "semanticTokensProvider", "legend" }) != nullptr + ? (*initialized)["capabilities"]["semanticTokensProvider"]["legend"] + : Json::object(); + Scenario runner { client, options, serverArguments, workspace, options.timeout, std::move(prepared), cacheDirectory, options.expectWarm, + std::move(moduleFilesBefore), semanticTokensLegend }; int failures { advertised ? 0 : 1 }; // "initialize-within": seconds. The handshake is answered at all, and in time (0.0.3 plan B1). if (const auto within = scenario.find("initialize-within"); within != scenario.end() && within->is_number()) { From c6ceb5e86ad1fae30c7320e97f586075714cf557 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Fri, 25 Sep 2026 06:58:02 +0800 Subject: [PATCH 20/26] spec(s3): evidence for module syntax in semantic tokens --- conformance/traceability.json | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/conformance/traceability.json b/conformance/traceability.json index a3c4fb4..255f80b 100644 --- a/conformance/traceability.json +++ b/conformance/traceability.json @@ -999,6 +999,39 @@ "test": "tests/test_server.cpp: merging" } ], + "S3-6.1-1": [ + { + "check": "inferred/K1-semantic-tokens" + }, + { + "check": "engine-none/K1-semantic-tokens" + }, + { + "test": "tests/test_tokens.cpp: merge: the core engine wins every position it covers, native fills the gaps" + }, + { + "test": "tests/test_tokens.cpp: merge: with no core engine, native's tokens answer alone (not null)" + } + ], + "S3-6.1-2": [ + { + "check": "inferred/K1-semantic-tokens" + }, + { + "check": "engine-none/K1-semantic-tokens" + }, + { + "test": "tests/test_server.cpp: native engine: semantic tokens, moduleType, modules=false, and the merge" + } + ], + "S3-6.1-3": [ + { + "test": "tests/test_server.cpp: native engine: semantic tokens, moduleType, modules=false, and the merge" + }, + { + "test": "tests/test_tokens.cpp: clangd's legend, including its own duplicates, maps by name" + } + ], "S4-3-1": [ { "validate": "S4 schema rejects unknown kit-version" From cca319660aafd1aa2cc2c720d608160071c8e3e3 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Fri, 25 Sep 2026 07:07:24 +0800 Subject: [PATCH 21/26] fix(orchestrator): semantic tokens advertise range only where clangd answers it, and refresh is a request clangd 23.1 declares no semantic-tokens range request, so advertising `range` had VS Code send ones clangd rejected with "method not found"; `range` is now advertised only when the core engine declares it too (mcppls's own engine alone always answers it). `workspace/semanticTokens/refresh` is a request in LSP, not a notification: it goes out with an id, and its empty answer is dropped like the watcher registrations'. The grammar test's probe imported a partition by its qualified name after `import std;`, which is ill-formed and which clangd 23.1 (and main at 510126255) never finishes under a project's module command: the probe, opened in a real editor, held the session in `preparing` for two minutes. It now imports `:part`, the form partitions are imported by, and is closed once its tokens are read. --- editors/vscode/test/suite/grammar.test.ts | 16 +++++++++++----- src/orchestrator/routing.cpp | 10 +++++++++- src/orchestrator/tokens.cpp | 4 ++-- src/orchestrator/tokens.cppm | 7 ++++--- src/orchestrator/workspace.cpp | 5 ++++- tests/test_tokens.cpp | 13 +++++++++++-- 6 files changed, 41 insertions(+), 14 deletions(-) diff --git a/editors/vscode/test/suite/grammar.test.ts b/editors/vscode/test/suite/grammar.test.ts index 8362be7..56ab728 100644 --- a/editors/vscode/test/suite/grammar.test.ts +++ b/editors/vscode/test/suite/grammar.test.ts @@ -53,6 +53,8 @@ async function captureLines(lines: readonly string[]): Promise<{ text: string; s const document = await vscode.workspace.openTextDocument(uri); await vscode.window.showTextDocument(document); const captured = await vscode.commands.executeCommand('_workbench.captureSyntaxTokens', uri); + // The probe is no part of the fixture's project: close it, so the suites after this one see only the project. + await vscode.commands.executeCommand('workbench.action.closeActiveEditor'); return tokensByLine(lines, captured); } @@ -69,7 +71,10 @@ suite('module-syntax highlighting: the injected grammar (WA-VSCODE-001)', functi 'import std;', 'import hello.greet;', 'import hello.', - 'export import hello:part;', + // `:part`, the form a partition is imported by. The probe is opened in a real editor, so clangd sees it too: + // `import std;` followed by the ill-formed `export import hello:part;` never finishes in clangd 23.1 (nor on + // main at 510126255) under a project's module command, and would hold this session in `preparing` for two minutes. + 'export import :part;', 'import ;', 'import "foo.h";', 'x = import;', @@ -114,11 +119,12 @@ suite('module-syntax highlighting: the injected grammar (WA-VSCODE-001)', functi assert.ok(findToken(line, 'hello')?.scopes.includes('entity.name.namespace.module.cpp'), JSON.stringify(line)); }); - test('"export import hello:part;" colors export and import', () => { + test('"export import :part;" colors export, import and the partition', () => { const line = lines[6]; - assert.ok(findToken(line, 'export')?.scopes.includes('keyword.control.export.cpp')); - assert.ok(findToken(line, 'import')?.scopes.includes('keyword.control.import.cpp')); - assert.ok(findToken(line, 'hello:part')?.scopes.includes('entity.name.namespace.module.cpp')); + assert.ok(findToken(line, 'export')?.scopes.includes('keyword.control.export.cpp'), JSON.stringify(line)); + assert.ok(findToken(line, 'import')?.scopes.includes('keyword.control.import.cpp'), JSON.stringify(line)); + assert.ok(findToken(line, ':')?.scopes.includes('punctuation.separator.module-partition.cpp'), JSON.stringify(line)); + assert.ok(findToken(line, 'part')?.scopes.includes('entity.name.namespace.module.partition.cpp'), JSON.stringify(line)); }); test('"import ;" colors the angle-bracket header', () => { diff --git a/src/orchestrator/routing.cpp b/src/orchestrator/routing.cpp index 7e33be3..d309394 100644 --- a/src/orchestrator/routing.cpp +++ b/src/orchestrator/routing.cpp @@ -120,7 +120,15 @@ Json merge_capabilities(const Json& engineCapabilities) { // Semantic tokens (design doc 2026-09-25 K/§7, contract T0): the legend is this server's own, // built from whatever the core engine (if any) declared, so a restart or a missing core engine // cannot shift an index a client has already seen. - capabilities["semanticTokensProvider"] = tokens::provider_capability(tokens::build_legend(engineCapabilities)); + // Range only when the core engine answers range requests too (mcppls's own engine always does). + bool range { true }; + if (engineCapabilities.is_object()) { + if (const auto provider = engineCapabilities.find("semanticTokensProvider"); provider != engineCapabilities.end() && provider->is_object()) { + const auto declared = provider->find("range"); + range = declared != provider->end() && (declared->is_object() || (declared->is_boolean() && declared->get())); + } + } + capabilities["semanticTokensProvider"] = tokens::provider_capability(tokens::build_legend(engineCapabilities), range); if (!capabilities.contains("experimental") || !capabilities["experimental"].is_object()) capabilities["experimental"] = Json::object(); capabilities["experimental"]["cxxModules"] = Json { { "version", 1 }, { "databaseSpec", ">=0.2 <1" } }; // usable plan W9.1: without this, a client has no reason to ever send diff --git a/src/orchestrator/tokens.cpp b/src/orchestrator/tokens.cpp index dc6b55c..9f45509 100644 --- a/src/orchestrator/tokens.cpp +++ b/src/orchestrator/tokens.cpp @@ -80,9 +80,9 @@ Legend build_legend(const Json& coreCapabilities) { return legend; } -Json provider_capability(const Legend& legend) { +Json provider_capability(const Legend& legend, bool range) { return Json { { "legend", Json { { "tokenTypes", legend.types }, { "tokenModifiers", legend.modifiers } } }, - { "full", true }, { "range", true } }; + { "full", true }, { "range", range } }; } std::vector decode(const Json& semanticTokensResult) { diff --git a/src/orchestrator/tokens.cppm b/src/orchestrator/tokens.cppm index b33c464..d938f57 100644 --- a/src/orchestrator/tokens.cppm +++ b/src/orchestrator/tokens.cppm @@ -40,9 +40,10 @@ struct Legend { Legend build_legend(const Json& coreCapabilities); // The `semanticTokensProvider` entry this server advertises: `legend`'s types and modifiers, -// `full: true` (no delta -- this server never hands out a resultId a delta could build on) and -// `range: true`. -Json provider_capability(const Legend& legend); +// `full: true` (no delta -- this server never hands out a resultId a delta could build on), and +// `range` only when `range` says every engine can answer one: clangd 23.1 has no range request, +// and advertising it had clients send one clangd rejects with "method not found". +Json provider_capability(const Legend& legend, bool range); // One token, decoded to absolute position: `length` and `startChar` are UTF-16 code units, as LSP // requires; a token never spans two lines. diff --git a/src/orchestrator/workspace.cpp b/src/orchestrator/workspace.cpp index fdbea2d..258bace 100644 --- a/src/orchestrator/workspace.cpp +++ b/src/orchestrator/workspace.cpp @@ -178,6 +178,7 @@ struct Workspace::Impl final : engine::Host { tokens::Legend tokensLegend { tokens::build_legend(Json::object()) }; bool clientSupportsTokensRefresh { false }; std::optional tokensRefreshAt; // coalesced: at most one refresh per ~500ms + std::uint64_t tokensRefreshes { 0 }; // for the refresh requests' ids // Diagnostics published by engines other than mcppls's own, per engine and client URI. std::map>, std::less<>> engineDiagnostics; std::map> publishedDiagnostics; @@ -1433,7 +1434,9 @@ struct Workspace::Impl final : engine::Host { } if (tokensRefreshAt && *tokensRefreshAt <= now) { tokensRefreshAt.reset(); - client.notify(lsp::method::WORKSPACE_SEMANTIC_TOKENS_REFRESH, Json::object()); + // A request, not a notification (LSP 3.16): its answer carries nothing, and an id the session cannot + // parse as an engine's is dropped, like the watcher registrations' answers. + client.send(lsp::make_request(std::format("w:{}:t{}", key, ++tokensRefreshes), lsp::method::WORKSPACE_SEMANTIC_TOKENS_REFRESH, nullptr)); } if (sdkCheckAt && *sdkCheckAt <= now) { sdkCheckAt.reset(); diff --git a/tests/test_tokens.cpp b/tests/test_tokens.cpp index 9519b82..ef82c06 100644 --- a/tests/test_tokens.cpp +++ b/tests/test_tokens.cpp @@ -5,6 +5,7 @@ import std; import nlohmann.json; import mcppls.testing; import mcppls.orchestrator.tokens; +import mcppls.orchestrator.routing; using Json = nlohmann::json; using mcppls::orchestrator::tokens::Legend; @@ -112,12 +113,20 @@ int main() { "merge: with neither engine, the answer is null"_test = [] { expect(tokens::merge(Json(nullptr), Json(nullptr)).is_null()); }; - "provider_capability advertises full and range, no delta"_test = [] { - const Json capability = tokens::provider_capability(tokens::build_legend(Json::object())); + "provider_capability advertises full, no delta, and range only when every engine answers it"_test = [] { + const Json capability = tokens::provider_capability(tokens::build_legend(Json::object()), true); expect(capability.value("full", false) == true); expect(capability.value("range", false) == true); expect(!capability.contains("delta")); expect(capability["legend"]["tokenTypes"].is_array() && !capability["legend"]["tokenTypes"].empty()); + expect(tokens::provider_capability(tokens::build_legend(Json::object()), false).value("range", true) == false); + // clangd 23.1 declares full (with delta) and no range: a range request would be rejected by it. + const Json clangd = Json::parse(R"({"semanticTokensProvider":{"full":{"delta":true},"legend":{"tokenTypes":["variable"],"tokenModifiers":[]}}})"); + expect(mcppls::orchestrator::merge_capabilities(clangd)["semanticTokensProvider"].value("range", true) == false); + const Json ranged = Json::parse(R"({"semanticTokensProvider":{"full":true,"range":true,"legend":{"tokenTypes":[],"tokenModifiers":[]}}})"); + expect(mcppls::orchestrator::merge_capabilities(ranged)["semanticTokensProvider"].value("range", false) == true); + expect(mcppls::orchestrator::merge_capabilities(Json::object())["semanticTokensProvider"].value("range", false) == true) + << "mcppls's own engine alone answers range requests"; }; return report(); From 0f21c148c4539e5abf5d349d11ce835a55c58844 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Fri, 25 Sep 2026 07:19:49 +0800 Subject: [PATCH 22/26] test(vscode): the grammar probe's export import comes before import std, which clangd 23.1 never finishes otherwise --- editors/vscode/test/suite/grammar.test.ts | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/editors/vscode/test/suite/grammar.test.ts b/editors/vscode/test/suite/grammar.test.ts index 56ab728..bca6407 100644 --- a/editors/vscode/test/suite/grammar.test.ts +++ b/editors/vscode/test/suite/grammar.test.ts @@ -68,13 +68,14 @@ suite('module-syntax highlighting: the injected grammar (WA-VSCODE-001)', functi 'module;', 'export module a.b;', 'module :private;', - 'import std;', + // The probe is opened in a real editor, so clangd sees it too, with the command it makes up for a file outside + // the project from the nearest unit's. Under that command clangd 23.1 (and main at 510126255) never finishes a + // file in which `import std;` comes before an `export import` of something nothing provides, and the session + // would sit in `preparing` for two minutes: this `export import` comes first. + 'export import :part;', 'import hello.greet;', 'import hello.', - // `:part`, the form a partition is imported by. The probe is opened in a real editor, so clangd sees it too: - // `import std;` followed by the ill-formed `export import hello:part;` never finishes in clangd 23.1 (nor on - // main at 510126255) under a project's module command, and would hold this session in `preparing` for two minutes. - 'export import :part;', + 'import std;', 'import ;', 'import "foo.h";', 'x = import;', @@ -103,7 +104,7 @@ suite('module-syntax highlighting: the injected grammar (WA-VSCODE-001)', functi }); test('"import std;" colors the keyword and the module name', () => { - const line = lines[3]; + const line = lines[6]; assert.ok(findToken(line, 'import')?.scopes.includes('keyword.control.import.cpp'), JSON.stringify(line)); assert.ok(findToken(line, 'std')?.scopes.includes('entity.name.namespace.module.cpp')); }); @@ -120,7 +121,7 @@ suite('module-syntax highlighting: the injected grammar (WA-VSCODE-001)', functi }); test('"export import :part;" colors export, import and the partition', () => { - const line = lines[6]; + const line = lines[3]; assert.ok(findToken(line, 'export')?.scopes.includes('keyword.control.export.cpp'), JSON.stringify(line)); assert.ok(findToken(line, 'import')?.scopes.includes('keyword.control.import.cpp'), JSON.stringify(line)); assert.ok(findToken(line, ':')?.scopes.includes('punctuation.separator.module-partition.cpp'), JSON.stringify(line)); From a9dc28f36bca7967a8ffee36f445b1194b095f0c Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Fri, 25 Sep 2026 07:20:30 +0800 Subject: [PATCH 23/26] docs(design): the import-hang plan's implementation record, and a second clangd 23.1 defect --- ...2026-09-25-import-hang-status-highlight.md | 49 +++++++++++++++++++ .agents/docs/design.md | 14 ++++++ 2 files changed, 63 insertions(+) diff --git a/.agents/docs/2026-09-25-import-hang-status-highlight.md b/.agents/docs/2026-09-25-import-hang-status-highlight.md index c5202d8..b017214 100644 --- a/.agents/docs/2026-09-25-import-hang-status-highlight.md +++ b/.agents/docs/2026-09-25-import-hang-status-highlight.md @@ -612,3 +612,52 @@ T0 contracts (above) Linux x64 and arm64, macOS and Windows as before. - **Consistency.** State, hysteresis and token types are decided by the server, so every editor sees the same thing. + +## 13. What was built, and what building it found (0.0.4) + +**Built.** Everything in §3–§10, in one pull request. Tracks B (semantic tokens in the server), +C (VS Code) and D (Neovim) ran in parallel against the contracts of §12, and were merged onto track A +(the server core, specs and docs). + +**Measured on this machine** (linux-x64, bundled clangd 23.1.0; the `hello` project and the +`inferred` fixture): + +| | 0.0.3 | 0.0.4 | +|---|---|---| +| Typing `import hello.greet;` key by key, with autosave | no answer for 30 s at `import hello.`; `degraded` with `module-build-failed`, `unresolved-module`, `file-quarantined`; stand-ins for `hello.`, `hello.g`, `hello.gr` | slowest answer 0.44 s; status stays `ready`; no stand-in, no restart | +| The same with WA-CLANGD-001 turned off (the real spin) | — | spin found at its 20 s budget; clangd restarted; features back, slowest answer while still typing 3.2 s | +| `typing-import` / `typing-import-spin` / `workaround-canaries` | — | pass; the canary fails, as it should, on clangd 24.0.0git | +| Unit tests (dev and release profiles) | — | 26/26 programs | +| VS Code e2e: main / conflicts / stress | — | 24 passing, 1 pending (macOS only) / 6 + 3 / 1 | +| Neovim smoke, setup and enable | — | 28/28 each | + +**Changed while building it:** +1. **Spin detection keys on demand, not on "a newer version waits".** + - A user who stops typing at `import hello.` sends no newer version, but the editor's requests for + the file do wait on the build. + - Those requests count, compared with when the version being built was sent: a request is often + sent before clangd reports that it started building. +2. **"No stand-in for a plain `.cpp`" became "none for a file being edited".** §5 has the xlings + evidence that made the narrower rule necessary. +3. **Semantic tokens advertise `range` only when the core engine has it.** clangd 23.1 declares no + range request, so VS Code's range requests came back "method not found". + `workspace/semanticTokens/refresh` is sent as the request LSP defines it to be. +4. **A second clangd 23.1 defect, found by the VS Code grammar test's probe file.** + - clangd never finishes a file in which `import std;` comes before an `export import` of something + nothing provides, when the command names std's unit (`-fmodule-file=std=…`, WA-CLANGD-004's + hints). + - Minimal case: `export module m;` `import std;` `export import :p;`. It also hangs on 24.0.0git. + - Inside a project it does not happen: typing `export import :detail;` after `import std;` in an + interface answered within 0.62 s, with no restart (a scratch fixture, measured). + - A file outside the database gets the command clangd interpolates from its nearest unit, hints + included, so a stray file with that ill-formed code reaches it. + - The first-diagnostics guard sets such a file aside after 120 s. It is recorded as a known limit + (design record §7), not worked around, and is to be filed upstream with the first one. +5. **VS Code's built-in grammar keeps the dead rule under a hash-prefixed key** + (`d9bc4796b0b_module_import`). The WA-VSCODE-001 canary matches it by suffix. + +**Still open:** +- Bisect the fix of the first defect (candidate `6dcfc17b1b`), file both defects upstream, and ask for + a 23.1.x backport. +- Update the bundled clangd once a fixed release exists; the canaries will say when. +- The 21:15:23 restart of the user's session (§5). diff --git a/.agents/docs/design.md b/.agents/docs/design.md index 35dab06..6ffc6c4 100644 --- a/.agents/docs/design.md +++ b/.agents/docs/design.md @@ -177,6 +177,13 @@ before anything is published (`docs/92-release.md`). turn out to matter (0.0.3 plan §5.2). - clangd 23.1 rejects MSVC STL's aligned allocation; the plan turns aligned allocation off for units using MSVC STL (`msvcStlNeedsNoAlignedAllocation`) until upstream fixes it. +- Every compensation for a clangd defect is a registered workaround (`WA-CLANGD-`, + `src/engine/clangd/workarounds.cpp`, import-hang plan §9); `mcppls report` lists the ones in use. +- clangd 23.1 (and main at 510126255) never finishes a file in which `import std;` comes before an + `export import` of something nothing provides, under a command that names `std`'s unit. Only a file + outside the database gets such a command, the one clangd interpolates from its nearest unit, so + only a stray file with ill-formed code reaches it. The first-diagnostics guard sets it aside after + two minutes (import-hang plan §13). - An mcpp project built for Windows through openkal needs `--target x86_64-windows-gnu`, which no editor setting passes to mcpp yet. - openkal cannot lower a child's scheduling priority, so clangd's cold-start module builds compete @@ -243,5 +250,12 @@ before its handshake still lets the server initialize; a loader's refusal is `en not a crash · L the icon · S being found as mcppls · P Open VSX from CI, after local verification · A Linux arm64 with the official LLVM clangd · X one platform table. +**"import-hang plan" — [2026-09-25-import-hang-status-highlight.md](2026-09-25-import-hang-status-highlight.md).** +§1 clangd 23.1 spins on a module name ending in `.` at the end of its line · §2 why the guards did +not see it · §3 WA-CLANGD-001, the same-line `;` · §4 the spin guard, a build's budget from its own +history · §5 no stand-in for an import still being typed · §6 status issue categories, the degraded +hold · §7 module syntax colored by an injected grammar and by the server's semantic tokens · §9 the +workaround registry and its canaries · §10 living with other C++ extensions. + **"tooling architecture".** 3.2 the workspace layout · 5.1 what mcpp, mcppls and devtools each do · 5.5 how devtools finds the server it just built · M0–M6 its migration steps. From d6c4bb2b59ea87b6513922ddb13c58f423291bf8 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Fri, 25 Sep 2026 07:20:38 +0800 Subject: [PATCH 24/26] docs(changelog): 0.0.4 --- CHANGELOG.md | 77 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 77 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 86f654f..a4dd719 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,83 @@ release's notes are that section. Versions are three-part semantic versions, `MAJOR.MINOR.PATCH`, and every editor plugin carries the product version unchanged. +## [0.0.4] — 2026-09-25 + +Typing an `import` no longer freezes the editor, the status bar says whose problem it is, and +`import` is colored. The investigation, the plan and its measurements are in +`.agents/docs/2026-09-25-import-hang-status-highlight.md`. + +### Engine + +- **Typing a dotted import froze everything.** clangd 23.1 never finishes a file in which a module + name ends in `.` at the end of its line (`import hello.`, `export module a.`): it spins at a full + core, and every later version of the file waits behind it. Typing `import hello.greet;` went + through that text every time. The feature requests for the file went unanswered for 30 s at a + time, and the status turned *degraded*. clangd is now given that line with `;` right after the + dot, which it reports at once as the error it is; nothing else about the text changes. + Measured: every keystroke answered within 0.4 s, where 0.0.3 answered nothing for 30 s. +- **A file clangd will not finish is found and recovered even while the user keeps typing.** Before, + the guards took a busy clangd for a compiling one, and an edit to the file postponed setting it + aside for two minutes; the edit that caused the hang, and the edits fixing it, kept postponing it. + - A file's build now has a budget: five times its own last build, never under 20 s. + - Past the budget, with the editor waiting on the file, clangd is restarted at once, past the + restart cap if need be. + - The file goes to mcppls's own engine, which gives module-level features, and returns to clangd + as soon as its text is anything other than the text clangd stopped on. + - Event `engine-spin`. +- **Workarounds for clangd's own defects are registered in one place** + (`src/engine/clangd/workarounds.cpp`). Each entry has the versions it applies to, the upstream + defect, and when it can go. The report lists the ones in use (`engines[].details.workarounds`), + and `--disable-workaround WA-CLANGD-` turns one off. +- **Half-typed imports no longer churn the engine database.** A module nothing provides, imported + by a file changed in the last five seconds, gets its stand-in only once the file is quiet; a + unit that provides a module still gets its stand-in at once. A name that is no module name + (`hello.`, as a build tool's scan of a file saved mid-edit can report) is never planned. + +### Status + +- **A problem in your code is a diagnostic, not a lost feature.** A missing `;`, an import of a + module nothing provides, or a module that does not compile is reported where it is, in the + Problems list, and the status stays *ready*. *degraded* now means the server lost something, and + it says what and where, for example "clangd stopped responding on main.cpp; module-level features + only for it until it changes". Status issues carry a `category` (`code`, `engine`, + `environment`, `project`; S3). +- A change to *degraded* is shown only once it has lasted three seconds, so a condition that passes + by itself never flickers in the status bar. + +### Editors + +- **`import`, `module` and `export` are colored**, and so are module names: + - The server sends semantic tokens for module syntax, which clangd sends none for (a custom type + `module`, with `namespace` for clients that do not ask for it), with and without clangd. + - The VS Code extension adds a grammar that colors them as you type, since VS Code's own C++ + grammar leaves `import` uncolored. + - Settings: `mcppls.semanticTokens.modules` in VS Code, `semantic_tokens_modules` in Neovim. +- **Other C++ extensions** can be turned off, or back on, at any time: + - Commands *Turn Off Other C++ Language Features* and *Restore Other C++ Language Features*, in + this workspace or everywhere. + - A notice when one becomes active later. + - The C/C++ extension's debugger keeps working. + - Neovim has `disable_conflicting`. + +### Known limits + +- clangd 23.1 also never finishes a file in which `import std;` comes before an `export import` of + something nothing provides. Only a file outside the project, holding such ill-formed code, gets the + command that exposes it. mcppls sets such a file aside after two minutes, and the defect is to be + reported upstream with the first. `.agents/docs/2026-09-25-import-hang-status-highlight.md` §13 has + the details. + +### Testing + +- Conformance kinds `type-text` (a line typed one key at a time, each step answered in time, the + status never turning *degraded*) and `clangd-check` (a workaround's canary). +- Fixtures on every platform: + - `typing-import`; + - `typing-import-spin`: the real spin, with the workaround off, recovered within its budget; + - `workaround-canaries`: it fails once a clangd update fixes the defect. +- `module-faults` and `failure-at-base` now expect *ready*, naming their code issues. + ## [0.0.3] — 2026-09-24 Linux arm64 is a platform, the extension is on Open VSX and is found by searching *mcppls*, and a From b9fdc0a5ed7b7d167d118749919906fbf1ceb3f4 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Fri, 25 Sep 2026 07:40:49 +0800 Subject: [PATCH 25/26] fix(engine): a spin found on a file already set aside still restarts clangd at once and remembers the text Review found that when the timeouts of a file's requests set it aside before its spin was found, the spin was ignored: the restart that setting-aside had put off stayed put off, and without the text clangd spun on remembered, the end of the file's term could hand that same text back. A spin found on a file set aside for anything else now records the text and restarts clangd at once, past the cap if need be, the same as one found first. A build tool's scan naming `export module hello.` no longer plans a provider of `hello.` either. --- src/engine/clangd.cpp | 34 ++++++++++++++++++++++++---------- src/normalize/plan.cpp | 5 +++++ tests/test_normalize.cpp | 4 ++++ 3 files changed, 33 insertions(+), 10 deletions(-) diff --git a/src/engine/clangd.cpp b/src/engine/clangd.cpp index c5ba469..1ca24bb 100644 --- a/src/engine/clangd.cpp +++ b/src/engine/clangd.cpp @@ -1919,15 +1919,36 @@ class ClangdEngine final : public Engine { for (const auto& document : host_->documents()) { if (document.uri == spin.uri) path = document.path; } - if (path.empty() || quarantined_(path)) continue; + if (path.empty()) continue; const auto seconds = [](std::chrono::milliseconds duration) { return std::chrono::duration(duration).count(); }; log::warning("clangd ({}) has built {} for {:.0f} s, past its {:.0f} s budget, while the editor waited on it: it will not finish it", host_->root_directory(), base::file_name(path), seconds(spin.building), seconds(spin.budget)); host_->record_event("engine-spin", Json { { "file", path }, { "buildingSeconds", seconds(spin.building) }, { "budgetSeconds", seconds(spin.budget) } }); + if (quarantined_(path)) { + // Set aside already, by the timeouts of its requests, which may have put the restart off: the same build is + // now known to be a spin, so its text is remembered and clangd is restarted at once all the same. + const std::string key { base::path_key(path) }; + aside_[key].spunOn = spin.textHash; + deferredReclaims_.erase(key); + reclaim_spin_(path); + continue; + } set_aside_(path, "clangd would not finish building it", Reclaim::now, false, spin.textHash); } } + // Not deferred, not spaced out, not capped: a clangd left spinning answers nothing for the file and holds a core, and + // the file it spun on stays with mcppls's engine, so the new clangd cannot be sent the same way. + void reclaim_spin_(const std::string& path) { + if (!accepting_) return; + if (restartGate_.at_cap(Clock::now())) { + log::warning("restarting clangd ({}) past the restart cap: it cannot be left spinning on {}, which stays with mcppls's engine", + host_->root_directory(), base::file_name(path)); + host_->record_event("engine-restart-past-cap", Json { { "file", path } }); + } + restart_(std::format("clangd would not finish {}", base::file_name(path))); + } + // Whether a restart gets back what clangd spends on a file it is no longer given. // `now`: what clangd spends on it is never coming back (SpinWatch): restart at once, past the gate and the cap. enum class Reclaim { no, if_busy, now }; @@ -1954,15 +1975,8 @@ class ClangdEngine final : public Engine { update_quarantine_issue_(); // clangd does not stop building a file it is no longer given: a build that never ends (the spin in experiment S17) keeps a // core and one of clangd's workers for as long as clangd runs. A fresh clangd, without the file, gets both back. - if (reclaim == Reclaim::now && accepting_) { - // Not deferred, not spaced out, not capped: a clangd left spinning answers nothing for this file and holds a core, - // and the file it spun on stays with mcppls's engine, so the new clangd cannot be sent the same way. - if (restartGate_.at_cap(Clock::now())) { - log::warning("restarting clangd ({}) past the restart cap: it cannot be left spinning on {}, which stays with mcppls's engine", - host_->root_directory(), base::file_name(path)); - host_->record_event("engine-restart-past-cap", Json { { "file", path } }); - } - restart_(std::format("clangd would not finish {}", base::file_name(path))); + if (reclaim == Reclaim::now) { + reclaim_spin_(path); return; } if (reclaim == Reclaim::if_busy && accepting_ && (!state || engine_working(*state))) { diff --git a/src/normalize/plan.cpp b/src/normalize/plan.cpp index 634e9fc..567e620 100644 --- a/src/normalize/plan.cpp +++ b/src/normalize/plan.cpp @@ -197,6 +197,11 @@ EnginePlan plan_engine(const PlanInput& input) { } else { candidate.provided = project::provided_name(scan()); } + // The same mid-edit scan can name `export module hello.`: no module has that name either. + if (!candidate.provided.empty() && !project::is_module_name(candidate.provided)) { + base::log::debug("ignoring module '{}' provided by {}: not a module name", candidate.provided, candidate.source); + candidate.provided.clear(); + } candidate.required = unit.requiredModules; if (candidate.required.empty()) candidate.required = project::required_names(scan()); drop_invalid_module_names(candidate.required, candidate.source); diff --git a/tests/test_normalize.cpp b/tests/test_normalize.cpp index 0f22469..b3ecb34 100644 --- a/tests/test_normalize.cpp +++ b/tests/test_normalize.cpp @@ -387,6 +387,7 @@ int main() { { "/p/src/main.cpp", "import hello;\nint main() {}\n" }, { "/p/src/greet.cppm", "export module hello.greet;\nimport half;\n" }, { "/p/src/other.cpp", "import gone;\nint f() { return 0; }\n" }, + { "/p/src/mid.cppm", "export module mid;\n" }, }; s::Database database; database.hasIde = true; @@ -401,6 +402,7 @@ int main() { unit.arguments = { "/opt/gcc/bin/g++", "-std=c++23", "-fmodules", "-c", path }; // What a build tool's scan of a file saved mid-edit reported: `hello.` is no module name. if (path == "/p/src/other.cpp") unit.requiredModules = { "gone", "hello.", ".x", "a..b", "a:b:c" }; + if (path == "/p/src/mid.cppm") unit.providedModules = { { "mid.", "" } }; set.units.push_back(std::move(unit)); } database.sets.push_back(set); @@ -429,6 +431,8 @@ int main() { } expect(std::ranges::any_of(editing.issues, [](const n::PlanIssue& issue) { return issue.code == "module-build-failed" && issue.module == "hello"; })) << "the import is still reported"; + expect(std::ranges::none_of(editing.entries, [](const n::EngineEntry& entry) { return entry.provides == "mid."; })) + << "a provided name no module can have is never planned either"; input.editingSources.clear(); const auto quiet = n::plan_engine(input); From 658e878a20fcb62177748abb03ee8c94d6577c76 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Fri, 25 Sep 2026 07:47:49 +0800 Subject: [PATCH 26/26] fix(conformance): on Windows clangd 23.1 crashes on `import hello.` instead of spinning, and the checks say so CI found the first defect is a crash on Windows (exit 0x80000003) rather than a spin: the canary took the quick exit for a fixed clangd, and typing-import-spin waited for an engine-spin event that cannot come there. The canary now counts a crash as the defect still being there, a check can name the operating systems it holds on (`only-on`), and typing-import-spin checks the crash restart on Windows and the spin elsewhere. WA-CLANGD-001 keeps clangd from crashing there too. --- ...2026-09-25-import-hang-status-highlight.md | 8 ++++++- CHANGELOG.md | 2 +- conformance/README.md | 6 ++--- .../fixtures/typing-import-spin/scenario.json | 24 ++++++++++++++++++- src/bin/conformance.cpp | 16 ++++++++++++- 5 files changed, 49 insertions(+), 7 deletions(-) diff --git a/.agents/docs/2026-09-25-import-hang-status-highlight.md b/.agents/docs/2026-09-25-import-hang-status-highlight.md index b017214..5646661 100644 --- a/.agents/docs/2026-09-25-import-hang-status-highlight.md +++ b/.agents/docs/2026-09-25-import-hang-status-highlight.md @@ -653,7 +653,13 @@ C (VS Code) and D (Neovim) ran in parallel against the contracts of §12, and we included, so a stray file with that ill-formed code reaches it. - The first-diagnostics guard sets such a file aside after 120 s. It is recorded as a known limit (design record §7), not worked around, and is to be filed upstream with the first one. -5. **VS Code's built-in grammar keeps the dead rule under a hash-prefixed key** +5. **On Windows the first defect is a crash, not a spin.** clangd 23.1 exits with 0x80000003 + (`STATUS_BREAKPOINT`) on `import hello.`, found by CI. + - WA-CLANGD-001 prevents that crash too. + - The canary counts a crash as the defect. + - `typing-import-spin` checks the crash restart there (`engine-exit`) instead of `engine-spin`, + through a new per-check `only-on`. +6. **VS Code's built-in grammar keeps the dead rule under a hash-prefixed key** (`d9bc4796b0b_module_import`). The WA-VSCODE-001 canary matches it by suffix. **Still open:** diff --git a/CHANGELOG.md b/CHANGELOG.md index a4dd719..dd2e5e9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,7 +19,7 @@ Typing an `import` no longer freezes the editor, the status bar says whose probl name ends in `.` at the end of its line (`import hello.`, `export module a.`): it spins at a full core, and every later version of the file waits behind it. Typing `import hello.greet;` went through that text every time. The feature requests for the file went unanswered for 30 s at a - time, and the status turned *degraded*. clangd is now given that line with `;` right after the + time, and the status turned *degraded*. On Windows, the same text crashed clangd instead. clangd is now given that line with `;` right after the dot, which it reports at once as the error it is; nothing else about the text changes. Measured: every keystroke answered within 0.4 s, where 0.0.3 answered nothing for 30 s. - **A file clangd will not finish is found and recovered even while the user keeps typing.** Before, diff --git a/conformance/README.md b/conformance/README.md index a262ffd..5651073 100644 --- a/conformance/README.md +++ b/conformance/README.md @@ -65,7 +65,7 @@ checks fail at once with that reason instead of each waiting out its timeout. | `watch-polling` | Run with `--no-dynamic-watch` (usable plan W9.3): a new module interface written straight into the workspace must still reach the module graph within seconds, through the polling fallback rather than a client-driven `workspace/didChangeWatchedFiles` | | `clangd-cannot-load` | 0.0.3 plan B1: its `prepare` step puts a stand-in clangd in the workspace (mcppls-mock-mcpp with an `unavailable` config) that writes a loader's message, a `GLIBCXX` version not found, to standard error and exits 1; `--clangd` points the server at it. `initialize` must be answered within 20 s (it used to wait for good), the status must reach `error` with issue `engine-incompatible`, and mcppls's own module features must work | | `typing-import` | Import-hang plan §8: `import hello.greet;` in `main.cpp` and `export module hello.greet;` in its interface typed one key at a time, through `import hello.` and `export module hello.`, which clangd 23.1 never finishes building (WA-CLANGD-001); again with every step saved, as autosave does. Every request is answered within 5 s, the status never turns `degraded`, hover works right after, and nothing restarts clangd or is set aside | -| `typing-import-spin` | The same typing with WA-CLANGD-001 turned off (`--disable-workaround`), so clangd really spins: the file is found spinning within its 20 s budget (event `engine-spin`), set aside with that text remembered, clangd is restarted, and features come back while typing goes on (import-hang plan §4). If a clangd update removes the defect, its `engine-spin` check fails as well | +| `typing-import-spin` | The same typing with WA-CLANGD-001 turned off (`--disable-workaround`), so clangd really spins (Linux, macOS; on Windows the same text crashes it instead): a spin is found within its 20 s budget (event `engine-spin`), the file set aside with that text remembered and clangd restarted, a crash is restarted as before (`engine-exit`), and either way features come back while typing goes on (import-hang plan §4). If a clangd update removes the defect, its T2 check fails as well | | `workaround-canaries` | Import-hang plan §9: one `clangd-check` per registered workaround with a canary, run against the payload's clangd. A failure here means a clangd update fixed that defect and the workaround it names can be removed | | `payload-corrupt` | Its `prepare` step copies the payload the runner was given and truncates clangd in the copy (usable plan W9.4); `server-arguments` then points `--payload` at that broken copy, and status must reach `error` with issue `payload-corrupt` | | `multi-root` | Two workspace folders (usable plan W9.1): an `inferred` root and an mcpp-built `mcpp-llvm` root (level 3, from mcpp's own build database), each getting its own project model and clangd, each `cxxModules/status` telling them apart by `project.root` | @@ -118,7 +118,7 @@ sees it names `["{conformance}", "prepare", "", ...]`: the generators live (`mcppls-conformance prepare --help`), so a conformance host needs nothing the runner does not bring — no interpreter. Positions are `[line, character]`, zero-based, UTF-16. A check with `"text"` opens its file with that unsaved -content; a check with `"optional": true` reports `SKIP` instead of failing, and `"timeout": SECONDS` +content; a check with `"only-on": ["linux", "macos", "windows"]` runs only on those operating systems and reports `SKIP` elsewhere; a check with `"optional": true` reports `SKIP` instead of failing, and `"timeout": SECONDS` waits less than the run's `--timeout`. `"file"` and `"folder"` on a check, like every other path a scenario names, are relative to the fixture's own root, never to a specific workspace folder. `"initialization-options"` on the scenario is an object merged into the runner's own @@ -173,7 +173,7 @@ always has been. | `cli` | S5 section 7: `mcppls ` with the runner's payload and the fixture's server arguments, run to completion in the workspace, exits with `"exit"` (default 0) and prints one JSON document meeting `"expect"` | | `stress` | real-project stress testing (real-project plan RP0): seeded random use — see below — meets every key present in `"budget"` | | `type-text` | line `line` of `file` takes each of `steps` in turn, `interval-ms` apart (default 120), the whole buffer sent each time; after each, `request` (default `textDocument/documentSymbol`) is answered within `answer-within` seconds (default 5); with `save`, each step is also written to disk and reported as saved and changed, as autosave does; fails when the status turned to a state listed in `states-never` meanwhile (import-hang plan §8) | -| `clangd-check` | the runner's own clangd (`--clangd`, else the payload's) run with `--check` on `file` hangs (`expect: "hangs"`, not finished after `seconds`, default 10) or finishes (`"finishes"`); a workaround's canary expects its defect, and fails with `says` once a clangd update fixed it (import-hang plan §9) | +| `clangd-check` | the runner's own clangd (`--clangd`, else the payload's) run with `--check` on `file` does not finish (`expect: "hangs"`: not finished after `seconds`, default 10, or crashed) or finishes normally (`"finishes"`); a workaround's canary expects its defect, and fails with `says` once a clangd update fixed it (import-hang plan §9) | | `report` | robustness design O3: `cxxModules/report` meets `"expect"`, retried within the check's time like an `mcp`/`cli` result (a plan or an engine may still be on its way) | An expectation of `mcp`, `cli` and `report` names a JSON pointer in `"path"`, where a `*` segment stands for every diff --git a/conformance/fixtures/typing-import-spin/scenario.json b/conformance/fixtures/typing-import-spin/scenario.json index c7bb229..991dcbb 100644 --- a/conformance/fixtures/typing-import-spin/scenario.json +++ b/conformance/fixtures/typing-import-spin/scenario.json @@ -1,6 +1,6 @@ { "name": "typing-import-spin", - "description": "Import-hang plan \u00a74 against the real defect: WA-CLANGD-001 turned off, so clangd 23.1 spins on `import hello.`. The file is found spinning within its 20 s budget, set aside with that text remembered, clangd is restarted, and features come back while typing goes on. If a clangd update removes the defect, T2 fails too: WA-CLANGD-001 can then go.", + "description": "Import-hang plan \u00a74 against the real defect: WA-CLANGD-001 turned off, so clangd 23.1 spins on `import hello.` (Linux, macOS) or crashes on it (Windows). A spin is found within its 20 s budget, the file set aside with that text remembered and clangd restarted; a crash is restarted as before; either way features come back while typing goes on. If a clangd update removes the defect, T2 fails too: WA-CLANGD-001 can then go.", "server-arguments": [ "--no-discover", "--disable-workaround", @@ -70,6 +70,28 @@ "path": "/roots/0/engines/*/details/workarounds/*/turnedOff", "equals": true } + ], + "only-on": [ + "linux", + "macos" + ] + }, + { + "id": "T2-crash-restarted", + "kind": "report", + "timeout": 60, + "only-on": [ + "windows" + ], + "expect": [ + { + "path": "/roots/0/events/*/kind", + "equals": "engine-exit" + }, + { + "path": "/roots/0/engines/*/details/workarounds/*/turnedOff", + "equals": true + } ] }, { diff --git a/src/bin/conformance.cpp b/src/bin/conformance.cpp index 1a43883..f628e1b 100644 --- a/src/bin/conformance.cpp +++ b/src/bin/conformance.cpp @@ -1279,7 +1279,10 @@ class Scenario { while (running.wait_for(std::chrono::milliseconds { 200 }) != std::future_status::ready) client_.drain(std::chrono::milliseconds { 0 }); auto result = running.get(); if (!result) return { false, result.error().message }; - const bool hung { result->timedOut }; + // The defect shows as a hang on Linux and macOS, and as a crash on Windows (0x80000003): either is clangd not + // finishing. A normal exit, with or without errors, is --check's 0 to 3. + const bool crashed { !result->timedOut && (result->exitCode < 0 || result->exitCode > 125) }; + const bool hung { result->timedOut || crashed }; const std::string expected { check.value("expect", std::string { "hangs" }) }; const std::string says { check.value("says", std::string {}) }; if (expected == "hangs" && !hung) { @@ -1287,6 +1290,7 @@ class Scenario { result->exitCode, limit.count(), (result->output + result->error).substr(0, 200)) }; } if (expected == "finishes" && hung) return { false, std::format("clangd did not finish {} in {} s", file, limit.count()) }; + if (crashed) return { true, std::format("clangd crashed on {} (exit {}), as expected", file, result->exitCode) }; return { true, hung ? std::format("clangd has not finished {} after {} s, as expected", file, limit.count()) : "clangd finished" }; } if (kind == "responds") { @@ -1919,6 +1923,16 @@ int run(Options options) { say("SKIP {} status (a plain client asks for no cxxModules/status)", id); continue; } + // `only-on`: the operating systems a check holds on ("linux", "macos", "windows"); a defect that shows + // differently elsewhere (a crash instead of a spin) is checked by its own entry there. + if (const auto onlyOn = check.find("only-on"); onlyOn != check.end() && onlyOn->is_array()) { + const std::string_view here { mcppls::os::FAMILY == mcppls::os::Family::windows ? "windows" + : mcppls::os::FAMILY == mcppls::os::Family::macos ? "macos" : "linux" }; + if (std::ranges::none_of(*onlyOn, [&](const Json& os) { return os.is_string() && os.get() == here; })) { + say("SKIP {} {} (only on {})", id, check.value("kind", std::string {}), lsp::dump(*onlyOn)); + continue; + } + } if (const auto reason = client.unusable(); !reason.empty()) { if (!optional) ++failures; say("{} {} {} (not run) {}", optional ? "SKIP" : "FAIL", id, check.value("kind", std::string {}), reason);