Repository navigation
fix(accessibility): scan fixes from the 2 Nov stability report - #541
Conversation
…gh the user's own Local tunnel, rename on a taken name, say why plan rules are off - startAccessibilityScan returns a scan that outlasts the 5-minute wait as still running, with its scan id, run id and dashboard link, instead of "Failed to start accessibility scan" with no ids. A cancelled run ends the wait, and the wait's timer is cleared. - startAccessibilityScan takes an optional localIdentifier: the scan goes through a BrowserStack Local tunnel the user started, and no tunnel is started or killed. The hosted refusal of local URLs says how to start one. - A scan name already in use is retried once with a UTC timestamp appended, and the result opens with the name used. - When advanced or experimental rules were not asked off but the run reports them off, the settings line says the plan most likely doesn't include them. - apiClient gives each of a host's addresses 2 s to connect instead of Node's 250 ms, and tries a request once more when its connection never opened. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y a scan failed
- startAccessibilityScan replaces characters central-scanner refuses in a scan name ("/",
"|", "&", ",", ")" and more, and "--") instead of failing the scan. Accents are dropped,
parentheses become square brackets, and a name with nothing usable left falls back to
the page's host. The result says the name was changed and what it was started as.
- A failed scan now says why: the run's own reason from the scans list, and each page that
did not scan with its reason from the run's scan logs, up to 10 pages. Without either, it
still points to the dashboard.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the
|
…ability No textual conflicts. Brings in main, with GRR regional routing (#489), so the scan failure details this branch reads (the scans list and a run's scan logs) now go to getRegionalHost("accessibilityApi") instead of the US host. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ack hint as they are on #541 Restores src/lib/local.ts, src/lib/feedback-hint.ts and src/tools/live-utils/start-session.ts to their #541 versions, which puts back killExistingBrowserStackLocalProcesses and drops ensureOwnTunnel, stopOwnTunnels, killOwnTunnelsNow and NO_FEEDBACK_HINT. startScan again starts a tunnel with ensureLocalBinarySetup for a local URL and calls killExistingBrowserStackLocalProcesses for a public one. The refusals for the hosted MCP and for USE_OWN_LOCAL_BINARY_PROCESS with a local URL stay ScanRefusedError plain answers, and account refusals keep their wording without the feedback-hint opt-out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarve-shreyas
left a comment
There was a problem hiding this comment.
Reviewed 6abd4a4..cbe1c17 (this PR on top of #527). npm run build passes locally on this head: 142 files, 2,104 tests.
No blocking findings from me. Checked and fine:
ScanTimeoutErrorhands the ids back, the timer is cleared on finish/error, andcancelledends the wait.- An own
localIdentifierskips both start and kill, and sendslocalTestingInfofor public URLs too. - Taken-name retry only fires on a 422 that says "already exists" or gives no reason; the timestamp stays inside the allowed character set and 255 chars.
cleanScanNamecollapses runs so--can't appear, and falls back to the host.failureDetailsis best-effort (errors swallowed) and only takeslastScanDetails.errorwhen it is this run.planDowngradeNoteonly fires when the rules were asked for or left at their default.
One non-blocking note inline on apiClient.
| // A connection that never opened sent nothing, so one more try is safe even for a | ||
| // POST; a second failure is reported as before. | ||
| if (!isConnectFailure(error)) throw error; | ||
| res = await fn(this.axiosAgent); |
There was a problem hiding this comment.
Non-blocking: this retry and the new directAgent apply to every tool that uses apiClient, not just accessibility. I'm fine with that, since a connect failure sent nothing. But it's worth a line in the PR description so reviewers of other tools know about it.
Also, a host that really is down now takes up to (2 s × addresses) × 2 before failing, where it used to fail in under a second. The default case is fine, but tools that loop over many calls (scan-list walks, polling) could stall much longer when an upstream is down. Maybe skip the retry when the first attempt already spent its full per-address budget, or cap the total.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A failure after every address has timed out means the host is down. Retrying it doubled the wait, which loops over many calls (scan-list walks, polling) would pay on every call. Only a failure within 5 s is tried again now. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sarve-shreyas
left a comment
There was a problem hiding this comment.
Re-reviewed cbe1c17..dd1edcd (dd1edcd plus the merge of #527). npm run build passes locally: 142 files, 2,104 tests.
Resolved:
- The connect retry now only fires when the failure came within 5 s (
CONNECT_RETRY_WINDOW_MS). A host that's really down waits out the OS connect timeout on its last address, goes past the window, and isn't retried. The fastETIMEDOUT/ENETUNREACHcase this PR targets still is.
No open findings from me on this PR. Small ask: a line in the description noting that the retry and directAgent apply to every tool, not just accessibility.
sarve-shreyas
left a comment
There was a problem hiding this comment.
Approved at dd1edcd. The connect retry now only fires on a fast failure. Build passes locally, 2,104 tests. Small ask, not blocking: note in the description that the apiClient retry and agent apply to every tool.
…compare, re-scan, Local tunnels, report links (#553) * fix(accessibility): say when a run did not check its pages, instead of reading it as clean fetchAccessibilityIssues read runs that checked nothing as clean or complete reports: - A failed Website Scanner run exported an empty report and read as "0 issues ✅ All issues retrieved"; a failed Automated Tests build (the API's status for one that completed with no test cases) threw "Failed to fetch report: null". A failed scan run, build or Workflow Analyzer report is now an error that says so, with the scan's reasons from its logs, and no export is started. An app build's "failed" is its tests' result, so it is read as before. Comparing scan runs refuses a failed run, whose empty report read every issue of the other run as resolved or new. - A page that did not load had an error page (Chrome's, or BrowserStack Local's privoxy page) scanned in its place, and its issues came back as the page's. The report parser now sets such pages apart, as the comparison already did, keeping their rows out of the issues, totals, summary and duplicates; the read (and startAccessibilityScan) leads with a note. - A completed scan's pages that failed to scan were simply absent. Its scan logs are read with the export and the first page lists them (redirects aside, as those pages were scanned where they redirected to). With pages skipped, the last page no longer claims every issue. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accessibility): quote a scan's failure reasons and page lines as untrusted data The reasons a run or a page failed, and the pages a completed run could not scan, come from the scan logs and the site: the page's address, its HTTP status, where it redirected. They are now quoted in an untrusted-data block, as the report's issues and page addresses are, with the sentences around them ("N pages were not scanned …:", "Fix the cause and scan again.") left outside it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accessibility): steer compareWith on reports with no compare to reading both compareWith on a Workflow Analyzer report, an Automated Tests build or a manual app report answered with an error ("... is neither"), and agents spent a call working out that they should read both reports and compare them by hand. The answer now says so directly, as guidance rather than an error: read both reports with fetchAccessibilityIssues and match issues by rule and CSS selector on each page (by rule and component for app reports); an issue in the later report only is new, in the earlier only is fixed, in both is still open. compareWith naming a report that is not an app build, for an app build, gets the same guidance. The compareWith description now says which reports it compares (Scanner runs of one scan, app builds) and to read both for any other. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(accessibility): compare Automated Tests builds with compareWith compareWith on an Automated Tests (web) build answered "can't be compared here yet" and then, since 6a777ac, told the agent to read both builds and compare them by hand. In MCP dogfooding (3 Nov) 3 of 3 agents passed correct testhub ids and got no compare. compareWith on such a build (buildId, or its report link as reportUrl) now takes another build's id, its report link, or "previous", and compares them: - The older build, by start time, is "before" and the newer "after", whichever order they were passed in. When compareWith is the newer one the answer says so in one line. Both builds are named (name, project, #number, id, start), the names quoted as data. - Two builds of one build name and project are compared by the Automated Tests compare API (GET /api/automated-tests/v1/builds/compare/issues, newReport = after, baseReport = before). It takes the builds in the order given and does not check it. "previous" calls it without baseReport, and the answer names the build the API picked. Its issues are keyed by the API's own signatureKey, and the answer says it matches by that, not by rule, element and page alone. The API told apart a css-orientation-lock issue on <html> before and after a fix added lang to it, and listed it as both fixed and new. With a rule|element|page key on both, agents recounted the dashboard's 6/13/1 to 4/11/3. - Any other pair, or a pair the API refuses (issue-count threshold, "TestRun Not Found") or does not answer (504 after its 30 s, or no answer within 40 s), is compared by matching the two builds' issue exports by rule, element and page, as Scanner runs are. The answer says which method was used, in one line. - Either way the answer gives three lists with counts: new (only in after), fixed (only in before), still open (in both). Each list shows at most 50 issues and says how many more. For the API, a list cut short on its first page (100 rows) is read on its own. - Issues found on a BrowserStack Local or Chrome network error page are set apart as "page did not load". The diff also catches a page whose error-page issues are all duplicates of another page's, as a grouped build's are; before, that page's earlier issues read as fixed. - A build the build list calls "failed" (completed with no test cases) has no accessibility results: the answer says so as guidance and does not compare. "previous" with no earlier completed run of the build's name is said plainly, with no request to report it. The compareWith description now names web builds (21,495 of the 21,500 budget). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(accessibility): re-run an earlier scan of the same page instead of starting a new one startAccessibilityScan always created a new scan, so re-scanning a page after a fix made a second scan whose runs could not be compared with the first (fetchAccessibilityIssues compareWith failed with "Scan logs not found"). - New optional scanId. When that scan is for this same page (compared as startScan saves URLs: local hosts as bs-local.com, host case and a trailing slash ignored), start a new run of it (POST /scans/:id/new_scan) with its saved settings, wait for it like a new scan, and end with how to compare with compareWith "previous". Without scanId a new scan is created as before. - scanId for another page (or several pages): a new scan, saying why. - Settings passed that differ from the scan's saved ones: no run; each difference and the two ways forward are listed. Only settings the call passed are compared; a login or tunnel can only be checked for being saved at all, which is said. - A re-run keeps the scan's name (a different name passed is mentioned), and a Local scan's re-run says the tunnel must run with the saved identifier. - Re-run refusals as plain results: a run still going (with its run id), a deleted login config, an unknown scan, the plan's message. - Every new scan says how to scan the same page again with its scanId. - The 50-scan plan limit on create is said plainly instead of "open an issue on GitHub". - Shorter wording for a few accessibility argument descriptions, to stay within the description budget with the new argument. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(local): never kill BrowserStack Local tunnels this MCP did not start; local-URL limits as plain answers Every public-URL accessibility scan and Live session ran `pkill -f BrowserStackLocal` (`taskkill /IM BrowserStackLocal.exe` on Windows), as did starting a tunnel for a local URL, so a tunnel the user had up (their own, or their test suite's) was killed by an unrelated call. - lib/local.ts no longer kills any process by name. A public URL takes no tunnel action at all. - A local-URL scan without localIdentifier runs through this MCP's own tunnel, started through browserstack-local under a unique mcp-<uuid> identifier, kept in module state and reused while it runs. It is replaced (the old one stopped through its own handle) only once it has died or the credentials change, and stopped by its own pid when the process exits or is signalled. The binary runs tunnels with different --local-identifier values side by side, so the user's keeps running; `force` is dropped from the configured options. Should ours fail to start while another Local is running, the error says to pass that tunnel's name as localIdentifier. - Live sessions (the other caller) start their unidentified tunnel only when no other BrowserStack Local is running, and otherwise use the one that is, leaving it alone. - A local URL the MCP cannot tunnel to (the hosted MCP, or USE_OWN_LOCAL_BINARY_PROCESS) is answered as a plain result with #541's guidance (start BrowserStack Local and pass its name as localIdentifier, or scan a public URL), with no GitHub or submitFeedback line. - The accessibility tools' error wrapper adds a full stop only when the message lacks one, so it no longer writes "..". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accessibility): search 25 pages of scans for a report link, not 15 A Website Scanner dashboard link is resolved by walking the scan list newest first. At the list's default 20 per page, 15 pages reached ~300 scans, so a link to an older scan came back "not found". 25 pages reach ~500, at about 1 s a page (~26 s worst case). Stopgap until the issues API reads by thBuildId (accessibility #5689). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accessibility): no 5-minute wait on a finalizing report; account refusals as plain answers; expert questions within 500 chars fetchAccessibilityIssues waited up to five minutes inside the call for a report still finalizing (or an app build still running its rules). Clients cut off a call that long: the claude.ai connector aborts a tool call silent for 305 s, progress notifications or not, so the agent got nothing, and one reported a fix as verified that had added 26 issues. - A first read of a report that is "finalizing" or "rule_processing" now answers at once. It is not an error: it says no issues were read yet, so the page is not to be called clean nor a fix verified, and gives the exact call to make again in a minute or two (pageUrl and compareWith kept), or to call the report stuck after about 10 minutes. The wait loop and REPORT_STATUS_POLL_MS / REPORT_STATUS_WAIT_MS are gone. A later page, or a report whose tests are still running, is read as before with its "still processing" note. - A cancelled Website Scanner run is answered like a failed one (failedRunResponse): an error saying it has no results and that 0 issues does not mean the page is clean, instead of reading its empty export as "0 issues, all retrieved". - Refusals the user resolves on their own account are explained instead of reported as faults: the plan's saved-scan limit ("make over N scans"), a plan without the External APIs add-on, and Website Scanner's "user info missing" (open Website Scanner once in the dashboard first). Each says what to tell the user and quotes the API's words, with no "open an issue on GitHub" line. They carry NO_FEEDBACK_HINT, a new symbol-keyed flag in lib/feedback-hint.ts that is never serialised, so withFeedbackHint does not send the agent to submitFeedback. (startAccessibilityScan's 402 scan-limit refusal is still answered first by the scanner's ScanRefusedError; this covers the message wherever else it reaches the error handler.) - accessibilityExpert keeps a question within the knowledge base's 500-character limit, which refused longer ones ("Query is too long"). A longer question keeps its last sentence, where agents put the question itself, and as much of its opening as fits, cut at a word break; the answer starts with a note that it was shortened. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(accessibility): drop the new test files from this PR Removes the six new test files and the test cases this PR added to existing files. Keeps only the edits that existing tests need because behaviour changed here: no 5-minute wait on a finalizing report, compareWith guidance on reports with no compare, failed-page wording, and no tunnel killing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(accessibility): fetchAccessibilityIssues no longer waits on a processing report Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accessibility): resolve a Scanner link or build id passed as compareWith, and compare runs oldest first compareScanRuns took compareWith verbatim as a run id, so a Website Scanner report link or a run's thBuildId failed with "Scan run id must be an Integer", a run id of another scan with "Scan logs not found", and a newer run id swapped new and resolved without a word. compareWith is now resolved as a source is: a link or 40-character id through sourceFromReportUrl / resolveBuild and findScanRun. A run of the same scan is compared with the older run as "before", saying so when the order was swapped. Another scan's run, a build or a report, and a run id the scan's full run list does not hold, get a plain answer naming both reads' arguments. compareWith's description drops "older": the runs are ordered here now. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accessibility): do not compare a run or build that is still processing Comparing against a run still processing gave a clean-looking diff ("0 new, 0 resolved, 2 still open") while its issues were still being added, which could read a fix as verified. The Website Scanner, Automated Tests and app build compares now check both sides' statuses first: while either is processing or finalizing, the answer is a plain "not finished yet" naming the run or build and its status, telling the agent to have the user ask again in a few minutes and not to call again in this turn. The Automated Tests compare's "only the issues found so far are compared" warning goes, as such a build is no longer compared. An app build compare is dispatched before the finalizing read, so a build still running its rules gets this answer rather than "call again in a minute or two". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accessibility): find a Website Scanner run given only its build id in the scan list A Website Scanner run's thBuildId passed as buildId, with no link, was refused whenever the API could not yet read a run by its build id: "pass the scan's report link as reportUrl instead", plus the GitHub and submitFeedback lines. An agent then searched for the run itself, and on an old scan gave up. Such an id now walks the scan list with findScanRun, as a link does, under the same 25-page cap and with the same "found in the scan list instead" note, naming the scan and run it found. Only when the walk finds nothing (or the API that reads runs by build id has none) is the answer a plain not-found, saying what to pass instead, without the GitHub or submitFeedback lines. findScanRun's not-found is now a ScanNotInListError, with the same message. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accessibility): read an Automate build's hashed_id passed as buildId as its accessibility build An SDK run prints the Automate build's hashed_id ("stopBinSession hashed_id=…"), and an agent passing it as buildId was refused, with only a Website Scanner build id offered as an alternative. It then spent recovery calls on fetchBuildInsights and listSessions before finding a session link. A 40-hex id that is neither an Automated Tests nor an App Accessibility build is now tried as an Automate build, then an App Automate build, through the same session mapping their dashboard links use, before the scan-list walk. The answer says which build the id was read as. An id none of them knows says so in the plain not-found answer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accessibility): a report still running with no export yet is "no report yet", not "Failed to fetch report: null" A build stuck in "running" since 2025-09-16 was read as processing, so its export ran anyway and failed with error null: "Failed to fetch report: null. Please open an issue on GitHub", and the agent went searching for another product before handing the question back to the user. When a report is still processing and its export fails, the answer is now a plain "still running (status, started <date>), no report yet". If it started more than 24 hours ago it is called probably stuck, to be run again, rather than something to call again for. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accessibility): name the Workflow Analyzer report before this one for compareWith "previous" compareWith "previous" on a Workflow Analyzer report only said there is no built-in compare and to read both, without saying which report came before. Agents went looking for a way to list reports, hit routing refusals, and the hosted run gave no answer. The earlier report is now found here: the account's report list (newest first) locates this report, and the summaries of the completed reports after it give their start and pages. The newest one before it that scanned one of its pages is named, or else the newest on its site, with its id, its dashboard link and the two reads to make. When none is among the newest reports, the answer says so and to ask the user which report to compare with. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accessibility): keep a manual app report's page under 40k characters, describe cursor, and find its previous report A 70-issue manual report came back as one page of 51,132 characters, over Claude Code's 50,000-character limit, so the client saved it to a file. The page budget counted only the bare issue rows, while the answer prints them as indented JSON with the rules and screens they name. An app page (build or manual report) now counts each issue as printed, and each rule's text and screen's screenshot the first time the page names it, against a 35,000 character budget: with the summary, notes and wrappers the whole answer stays under about 40,000. The cursor still counts the bare issues. Agents then passed cursor as an issue index (40, 35) and got the same page again. cursor is now described as the next_page value from the previous answer, not an issue index. compareWith "previous" on a manual report now names the newest finished manual report of the same app recorded before it (found through the report list, by the report's name and then its app's package), with its id, its dashboard link and the two reads to make, as for a Workflow Analyzer report. Descriptions (total unchanged at 21,500): - cursor: "Pagination offset (default 0); with compareWith, issues already listed." (71) -> "The next_page value from the previous answer (default 0), not an issue index." (77) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accessibility): match a Workflow Analyzer report's site only on a page address that parses Follow-up to the "previous" Workflow Analyzer lookup: a page address that does not parse has no site, and two such addresses no longer count as one site. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accessibility): say compareWith "previous" works for every report compareWith's description ended "Others: read both", so for a Workflow Analyzer or manual app report agents never passed "previous" and went searching for the earlier report themselves: in re-runs, 4 of 5 sessions skipped it, and one hosted run again gave no answer. "previous" now finds the earlier report for those too, and the tool itself answers any pair it has no compare for with how to read both, so the description says "previous" works for any report. compareWith: 'Runs of one scan, web/app builds: "previous", scanRunId, build id/link. Others: read both.' (90) -> '"previous" (any report), or a run of the same scan or a web/app build: its id or link.' (86). Total 21,500 -> 21,496. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * revert(local): leave BrowserStack Local tunnel handling and the feedback hint as they are on #541 Restores src/lib/local.ts, src/lib/feedback-hint.ts and src/tools/live-utils/start-session.ts to their #541 versions, which puts back killExistingBrowserStackLocalProcesses and drops ensureOwnTunnel, stopOwnTunnels, killOwnTunnelsNow and NO_FEEDBACK_HINT. startScan again starts a tunnel with ensureLocalBinarySetup for a local URL and calls killExistingBrowserStackLocalProcesses for a public one. The refusals for the hosted MCP and for USE_OWN_LOCAL_BINARY_PROCESS with a local URL stay ScanRefusedError plain answers, and account refusals keep their wording without the feedback-hint opt-out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(accessibility): move this PR's report, compare and re-scan helpers into accessiblity-utils accessibility.ts had grown to 3,337 lines. The helpers this PR added move, unchanged, into accessiblity-utils: - error-messages.ts: asSentence and the account-refusal wording - expert-query.ts: fitting an accessibilityExpert question to 500 characters - report-answers.ts: failed, cancelled, finalizing and no-report-yet answers, unknown build ids, and pages not scanned or that showed an error page - compare-with.ts: what compareWith names (a scan run, the "previous" report), the read-both guidance and the not-finished-yet answer - compare-lists.ts: the export diff, compared lists and not-loaded note that scan-run and web-build comparisons share - web-build-compare.ts: comparing Automated Tests (web) builds - scan-rerun.ts: the saved-settings checks, notes and hints of a re-run shared.ts takes the small helpers of accessibility.ts these need too (the response builders, source names, progress, the scanner client, WCAG names), so neither side imports the other. accessibility.ts re-exports EXPERT_QUERY_MAX_CHARS. No behaviour or tool description changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accessibility): no stale exports, faster link lookups, Automate probe falls through - An export started while a report is still processing no longer has its task remembered: continued after the report finished, it handed back the mid-processing snapshot as final. - A run found by its build id is cached (per account, bounded), since a build id names one fixed run: a link read again, or both sides of a compare, skip the scan-list walk. The EXPORT_WAIT_MS comment no longer claims a whole call stays under 60 s. - A 40-hex id that Automate or App Automate can't answer about (an HTTP failure other than 404, or an Automate build with no sessions) moves on to the next API instead of ending the call, and the not-found answer says why. Once a session is found, what it says is still the answer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accessibility): don't offer re-runs that can't reach a local page; say what a re-run ignores - A local page this MCP tunnelled itself is saved with an identifier nobody was shown, and the tunnel is gone after the next public scan or a restart. Its scan no longer gets the "re-run with scanId" hint, and a re-run asked for a scan saved with Local but no localIdentifier starts a new scan instead, saying why and how to re-run it with its own tunnel. - A re-run can't take a login or a Local identifier: each passed one is now named as not used, with how to scan with it (a new scan), instead of only "could not be checked". - The saved scan's pages and name come from the account, which another user may have set: they are now quoted as untrusted data in re-run answers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accessibility): refuse to compare with a cancelled scan run, as with a failed one A cancelled run has no results and exports an empty report, so compared with it every issue of the other run read as resolved or new. Reads already treat a cancelled run as having no results; compares now do too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accessibility): don't silently take compareWith as the older build when start times are missing A build with no start time made the order check always false, so a newer build passed as compareWith was used as "before" and new and fixed issues swapped without a word. The order now falls back to the build number for runs of one build name and project; with neither, the passed order is kept and the answer says so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accessibility): shorten CJK expert questions by sentence, and say what was kept The sentence split knew only . ? !, so a Chinese or Japanese question was one "sentence" cut at 500, losing the question at its end. 。?! now end sentences too. A cut no longer leaves half a surrogate pair, and the note says whether the opening alone or the opening and last sentence were sent, as it happened. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accessibility): name the plan's scan limit the API gave, not a hard-coded 50 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * style: prettier Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accessibility): put the saved scan's name after the settings list in a re-run refusal The quoted name sat between "these differ:" and the list, so it read as one of the differences. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(accessibility): cover web-build compare, re-runs, failed runs and the Automate id probe Brings back the web-build compare, re-run and failed-run tests dropped in 7ee4c3d, updated to this PR's current behaviour, and adds cases for the review fixes: - web-build compare: refusing a build still processing, ordering by build number when a start time is missing, and keeping the passed order with a note otherwise; - re-runs: a Local-saved scan without localIdentifier starts a new scan, a local page the MCP tunnelled gets no re-run hint, a login or tunnel passed is named as not used, the saved pages and name are quoted as untrusted, and the scan limit is the API's number; - a 40-hex Automate or App Automate build id: found, falling through on 403/5xx or an empty build, and naming why Automate could not be asked; - compare refused for a cancelled run; - exports: no task kept while a report is processing, a run read by build id reused for 5 minutes at most, and a link's run found once not looked up again; - expert questions: the note matches what was cut, CJK sentence ends, no split character. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Stack: #527 → #541 (this, based on #527) → #553. GitHub moves this PR to
mainonce #527 merges. The scan-logs call added here now uses the account's region (#489).Why
The MCP daily report for 2 Nov put accessibility stability at 55.4%. Most of that was one account probing the scanner, but even without it the accessibility tools sat at 67–80% all week. This PR fixes the failures that need only an MCP change. The others (central-scanner's "user info missing", the External API add-on, report links that need accessibility changes) are tracked separately.
What changed
A scan that outlasts the 5-minute wait is handed back, not reported as failed
Failed to start accessibility scan: …as an error, with no scan id or run id. The scan kept running and finished. In an isolated test the agent then told the user no scan existed.ScanTimeoutErrorand the tool returns a normal result:fetchAccessibilityIssueswith those ids once it has finished, and don't start the scan again.cancelledrun now ends the wait instead of waiting out the 5 minutes, and the 5-minute timer is cleared once a scan finishes.Local URLs through a BrowserStack Local tunnel the user started (
localIdentifier)localTestingInfowith any tunnel id.startAccessibilityScannow takes an optionallocalIdentifier. With it, MCP sends the scan through that tunnel and does not start or kill any tunnel of its own.USE_OWN_LOCAL_BINARY_PROCESS, which refused local URLs, and internal hosts that are notlocalhost.BrowserStackLocal --key <your access key> --local-identifier <a name>, then pass that name aslocalIdentifier.A taken scan name no longer fails the scan
<name> YYYY-MM-DD HH:MM:SS UTC. Only a 422 that says "already exists", or gives no reason, counts as a taken name; a 422 about a setting is passed on unchanged.Scan names the API would refuse are cleaned instead of failing the scan
_ : * . @ ! $ ' { } [ ] ( -in a scan name, and never--. Agents name scans after page titles and paths, so 21 scans in a week failed on "Scan name is invalid".Café→Cafe), parentheses become square brackets (the API takes(but not)), and a run of replaced characters becomes-, or-between words. A name with nothing usable left, such as one in another script, falls back to the page's host.The name "…" has characters scan names can't contain, so this scan was started as "…".A cleaned name that is also taken gets the timestamp below, and the note says both.A failed scan says why
lastScanDetails.error, only when that is this run). It also lists each page that did not scan, with its reason from the run's scan logs (e.g.[HTTP 403] Forbidden,Failed to start the test), up to 10 pages and then a count.Saying why advanced or experimental rules came back off
submitFeedback.Connections that never open (
apiClient)connect ETIMEDOUT <IPv4>; connect ENETUNREACH <IPv6>: 17 calls across 7 tools in a week.autoSelectFamilyAttemptTimeout). I checked that the agent option reaches the socket on Node 22.apiClient, not only accessibility. A host that is really down now fails after roughly 2–5 s instead of under 1 s.Verified
npm run buildsteps: lint, prettier,tscand vitest all pass (130 files, 1876 tests). New tests cover each item above.claude -psessions with a strict MCP config (only this server, no built-in tools), against a real account:zz-5m-timeout-test 2026-11-03 09:51:04 UTC(scan 4300475) and the result opened with the rename note. The agent told the user.zz-mcp-9a-testand a test page onlocalhost:8765, the agent passedlocalIdentifieritself. The scan reached the page through the tunnel and found the 8 planted issues (scan 4300483). The tunnel was still running afterwards.Pricing / Plans | Q4 (draft), v2, the scan started asPricing - Plans - Q4 [draft] - v2, and the result opened with the rename note.Why: The requested page is not reachable. Please ensure the domain is accessible.and the page withFailed to start the test.Failed to start. The agent then calledfetchAccessibilityIssueswith those ids itself and told the user the scan was still finishing. On 2.4.1 the same test ended with the agent saying no scan existed.🤖 Generated with Claude Code