Skip to content

fix(accessibility): scan fixes from the 2 Nov stability report - #541

Merged
Dave3130 merged 6 commits into
fix/a11y-report-linksfrom
fix/a11y-scan-stability
Nov 6, 2026
Merged

Dave3130 merged 6 commits into
fix/a11y-report-linksfrom
fix/a11y-scan-stability

Conversation

@Aaryan430

@Aaryan430 Aaryan430 commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Stack: #527 → #541 (this, based on #527) → #553. GitHub moves this PR to main once #527 merges. The scan-logs call added here now uses the account's region (#489).

Why

The MCP daily report for 2 Nov put accessibility stability at 55.4%. Most of that was one account probing the scanner, but even without it the accessibility tools sat at 67–80% all week. This PR fixes the failures that need only an MCP change. The others (central-scanner's "user info missing", the External API add-on, report links that need accessibility changes) are tracked separately.

What changed

A scan that outlasts the 5-minute wait is handed back, not reported as failed

  • Before, the wait rejected with "Scan timed out after 5 minutes" and the tool answered Failed to start accessibility scan: … as an error, with no scan id or run id. The scan kept running and finished. In an isolated test the agent then told the user no scan existed.
  • Now the wait ends with a ScanTimeoutError and the tool returns a normal result:
    • the scan is still running and was not cancelled;
    • the scan id, run id and dashboard link;
    • call fetchAccessibilityIssues with those ids once it has finished, and don't start the scan again.
  • A cancelled run now ends the wait instead of waiting out the 5 minutes, and the 5-minute timer is cleared once a scan finishes.

Local URLs through a BrowserStack Local tunnel the user started (localIdentifier)

  • The hosted MCP runs on BrowserStack's servers, so it cannot start a tunnel on the user's machine, and it refused every local URL.
  • The scan API already takes localTestingInfo with any tunnel id. startAccessibilityScan now takes an optional localIdentifier. With it, MCP sends the scan through that tunnel and does not start or kill any tunnel of its own.
  • This also covers USE_OWN_LOCAL_BINARY_PROCESS, which refused local URLs, and internal hosts that are not localhost.
  • Without it, the hosted refusal now says how to start a tunnel: the Local app, or BrowserStackLocal --key <your access key> --local-identifier <a name>, then pass that name as localIdentifier.

A taken scan name no longer fails the scan

  • Scan names are unique per account, so an agent re-scanning a page under the same name was refused.
  • MCP now retries once as <name> YYYY-MM-DD HH:MM:SS UTC. Only a 422 that says "already exists", or gives no reason, counts as a taken name; a 422 about a setting is passed on unchanged.
  • The timestamp uses only characters central-scanner allows in a name, and the name is shortened to stay within its 255 characters. The result opens with the name the scan was started under.

Scan names the API would refuse are cleaned instead of failing the scan

  • central-scanner allows only letters, digits, spaces and _ : * . @ ! $ ' { } [ ] ( - in a scan name, and never --. Agents name scans after page titles and paths, so 21 scans in a week failed on "Scan name is invalid".
  • MCP now replaces any other character before creating the scan. Accents are dropped (Café → Cafe), parentheses become square brackets (the API takes ( but not )), and a run of replaced characters becomes -, or - between words. A name with nothing usable left, such as one in another script, falls back to the page's host.
  • The result opens with The name "…" has characters scan names can't contain, so this scan was started as "…". A cleaned name that is also taken gets the timestamp below, and the note says both.

A failed scan says why

  • It used to say only "The dashboard says why", which left an agent nothing to tell the user (19 failed scans in a week).
  • The result now gives the run's own reason, taken from the scans list (lastScanDetails.error, only when that is this run). It also lists each page that did not scan, with its reason from the run's scan logs (e.g. [HTTP 403] Forbidden, Failed to start the test), up to 10 pages and then a count.
  • If neither read gives anything, it still points to the dashboard. Both reads are best-effort, so a failed read doesn't add an error on top of the failed scan.

Saying why advanced or experimental rules came back off

  • On a public-API scan, a plan without advanced or experimental rules turns them off and runs the scan anyway, without saying so. A customer reported this through submitFeedback.
  • When those rules were asked for, or left at their default (on), and the run reports them off, the settings line now adds that the account's plan most likely doesn't include them, and that the issues don't cover them.
  • The two rule parameters say "off if the plan lacks them".

Connections that never open (apiClient)

  • On the hosted server, calls failed within 0.5–1.5 s with connect ETIMEDOUT <IPv4>; connect ENETUNREACH <IPv6>: 17 calls across 7 tools in a week.
  • Node gives each of a host's addresses 250 ms by default. On a slow route the IPv4 attempt is dropped that early, and IPv6 fails where there is no route. Likely cause; not confirmed on the hosted pod.
  • The default HTTPS agent now gives each address 2 s (autoSelectFamilyAttemptTimeout). I checked that the agent option reaches the socket on Node 22.
  • A request whose connection never opened within 5 s is tried once more. Nothing was sent, so this is safe for POSTs too. A request that got a response, failed after connecting, or failed only after every address timed out (the host is down) is not repeated.
  • The retry and the 2 s per-address connect timeout apply to every tool's calls through apiClient, not only accessibility. A host that is really down now fails after roughly 2–5 s instead of under 1 s.

Verified

  • npm run build steps: lint, prettier, tsc and vitest all pass (130 files, 1876 tests). New tests cover each item above.
  • Live, in isolated claude -p sessions with a strict MCP config (only this server, no built-in tools), against a real account:
    • Taken name: asked to scan under an existing name, the scan started as zz-5m-timeout-test 2026-11-03 09:51:04 UTC (scan 4300475) and the result opened with the rename note. The agent told the user.
    • Own tunnel: with BrowserStack Local started separately under zz-mcp-9a-test and a test page on localhost:8765, the agent passed localIdentifier itself. The scan reached the page through the tunnel and found the 8 planted issues (scan 4300483). The tunnel was still running afterwards.
    • Name cleaned: asked to name a scan Pricing / Plans | Q4 (draft), v2, the scan started as Pricing - Plans - Q4 [draft] - v2, and the result opened with the rename note.
    • Failed scan: a scan of a domain that does not resolve (scan 4300543) returned Why: The requested page is not reachable. Please ensure the domain is accessible. and the page with Failed to start the test.
    • 5-minute wait: with a test-only copy of this build that sends the page plus 10 more URLs (so the scan takes over 5 minutes), the tool returned the still-running result with scan 4300473 / run 2420007 instead of Failed to start. The agent then called fetchAccessibilityIssues with those ids itself and told the user the scan was still finishing. On 2.4.1 the same test ended with the agent saying no scan existed.
  • Not tested live: the plan note (the test account's plan includes advanced rules) and the connect retry (the failure can't be reproduced locally). Both are covered by unit tests.

🤖 Generated with Claude Code

Aaryan430 and others added 2 commits November 3, 2026 15:26
…gh the user's own Local tunnel, rename on a taken name, say why plan rules are off

- startAccessibilityScan returns a scan that outlasts the 5-minute wait as still running,
  with its scan id, run id and dashboard link, instead of "Failed to start accessibility
  scan" with no ids. A cancelled run ends the wait, and the wait's timer is cleared.
- startAccessibilityScan takes an optional localIdentifier: the scan goes through a
  BrowserStack Local tunnel the user started, and no tunnel is started or killed. The
  hosted refusal of local URLs says how to start one.
- A scan name already in use is retried once with a UTC timestamp appended, and the
  result opens with the name used.
- When advanced or experimental rules were not asked off but the run reports them off,
  the settings line says the plan most likely doesn't include them.
- apiClient gives each of a host's addresses 2 s to connect instead of Node's 250 ms, and
  tries a request once more when its connection never opened.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y a scan failed

- startAccessibilityScan replaces characters central-scanner refuses in a scan name ("/",
  "|", "&", ",", ")" and more, and "--") instead of failing the scan. Accents are dropped,
  parentheses become square brackets, and a name with nothing usable left falls back to
  the page's host. The result says the name was changed and what it was started as.
- A failed scan now says why: the run's own reason from the scans list, and each page that
  did not scan with its reason from the run's scan logs, up to 10 pages. Without either, it
  still points to the dashboard.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Aaryan430 Aaryan430 changed the title fix(accessibility): hand back scans that outlast the wait, scan through the user's own Local tunnel, rename on a taken name, say why plan rules are off fix(accessibility): scan fixes from the 2 Nov stability report Nov 3, 2026
@coderabbitai

coderabbitai Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Central YAML (base), Organization UI (inherited), Workspace UI (inherited); Enforced overrides: Organization
  • Review profile: ASSERTIVE
  • Plan: Enterprise
  • Run ID: 497d66f1-4e3e-4574-98f7-d219b2bd3e61

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

…ability

No textual conflicts. Brings in main, with GRR regional routing (#489), so
the scan failure details this branch reads (the scans list and a run's scan
logs) now go to getRegionalHost("accessibilityApi") instead of the US host.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Aaryan430
Aaryan430 changed the base branch from main to fix/a11y-report-links November 5, 2026 09:07
Aaryan430 added a commit that referenced this pull request Nov 5, 2026
…ack hint as they are on #541

Restores src/lib/local.ts, src/lib/feedback-hint.ts and
src/tools/live-utils/start-session.ts to their #541 versions, which puts
back killExistingBrowserStackLocalProcesses and drops ensureOwnTunnel,
stopOwnTunnels, killOwnTunnelsNow and NO_FEEDBACK_HINT.

startScan again starts a tunnel with ensureLocalBinarySetup for a local URL
and calls killExistingBrowserStackLocalProcesses for a public one. The
refusals for the hosted MCP and for USE_OWN_LOCAL_BINARY_PROCESS with a
local URL stay ScanRefusedError plain answers, and account refusals keep
their wording without the feedback-hint opt-out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@sarve-shreyas sarve-shreyas left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 6abd4a4..cbe1c17 (this PR on top of #527). npm run build passes locally on this head: 142 files, 2,104 tests.

No blocking findings from me. Checked and fine:

  • ScanTimeoutError hands the ids back, the timer is cleared on finish/error, and cancelled ends the wait.
  • An own localIdentifier skips both start and kill, and sends localTestingInfo for public URLs too.
  • Taken-name retry only fires on a 422 that says "already exists" or gives no reason; the timestamp stays inside the allowed character set and 255 chars.
  • cleanScanName collapses runs so -- can't appear, and falls back to the host.
  • failureDetails is best-effort (errors swallowed) and only takes lastScanDetails.error when it is this run.
  • planDowngradeNote only fires when the rules were asked for or left at their default.

One non-blocking note inline on apiClient.

Comment thread src/lib/apiClient.ts
// A connection that never opened sent nothing, so one more try is safe even for a
// POST; a second failure is reported as before.
if (!isConnectFailure(error)) throw error;
res = await fn(this.axiosAgent);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: this retry and the new directAgent apply to every tool that uses apiClient, not just accessibility. I'm fine with that, since a connect failure sent nothing. But it's worth a line in the PR description so reviewers of other tools know about it.

Also, a host that really is down now takes up to (2 s × addresses) × 2 before failing, where it used to fail in under a second. The default case is fine, but tools that loop over many calls (scan-list walks, polling) could stall much longer when an upstream is down. Maybe skip the retry when the first attempt already spent its full per-address budget, or cap the total.

Aaryan430 and others added 2 commits November 6, 2026 12:20
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A failure after every address has timed out means the host is down. Retrying
it doubled the wait, which loops over many calls (scan-list walks, polling)
would pay on every call. Only a failure within 5 s is tried again now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@sarve-shreyas sarve-shreyas left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed cbe1c17..dd1edcd (dd1edcd plus the merge of #527). npm run build passes locally: 142 files, 2,104 tests.

Resolved:

  • The connect retry now only fires when the failure came within 5 s (CONNECT_RETRY_WINDOW_MS). A host that's really down waits out the OS connect timeout on its last address, goes past the window, and isn't retried. The fast ETIMEDOUT/ENETUNREACH case this PR targets still is.

No open findings from me on this PR. Small ask: a line in the description noting that the retry and directAgent apply to every tool, not just accessibility.

@sarve-shreyas sarve-shreyas left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved at dd1edcd. The connect retry now only fires on a fast failure. Build passes locally, 2,104 tests. Small ask, not blocking: note in the description that the apiClient retry and agent apply to every tool.

…compare, re-scan, Local tunnels, report links (#553)

* fix(accessibility): say when a run did not check its pages, instead of reading it as clean

fetchAccessibilityIssues read runs that checked nothing as clean or complete reports:

- A failed Website Scanner run exported an empty report and read as "0 issues ✅ All issues
  retrieved"; a failed Automated Tests build (the API's status for one that completed with
  no test cases) threw "Failed to fetch report: null". A failed scan run, build or Workflow
  Analyzer report is now an error that says so, with the scan's reasons from its logs, and
  no export is started. An app build's "failed" is its tests' result, so it is read as before.
  Comparing scan runs refuses a failed run, whose empty report read every issue of the other
  run as resolved or new.
- A page that did not load had an error page (Chrome's, or BrowserStack Local's privoxy page)
  scanned in its place, and its issues came back as the page's. The report parser now sets
  such pages apart, as the comparison already did, keeping their rows out of the issues,
  totals, summary and duplicates; the read (and startAccessibilityScan) leads with a note.
- A completed scan's pages that failed to scan were simply absent. Its scan logs are read with
  the export and the first page lists them (redirects aside, as those pages were scanned where
  they redirected to). With pages skipped, the last page no longer claims every issue.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(accessibility): quote a scan's failure reasons and page lines as untrusted data

The reasons a run or a page failed, and the pages a completed run could not scan, come from
the scan logs and the site: the page's address, its HTTP status, where it redirected. They
are now quoted in an untrusted-data block, as the report's issues and page addresses are,
with the sentences around them ("N pages were not scanned …:", "Fix the cause and scan
again.") left outside it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(accessibility): steer compareWith on reports with no compare to reading both

compareWith on a Workflow Analyzer report, an Automated Tests build or a manual app
report answered with an error ("... is neither"), and agents spent a call working
out that they should read both reports and compare them by hand. The answer now says
so directly, as guidance rather than an error: read both reports with
fetchAccessibilityIssues and match issues by rule and CSS selector on each page (by
rule and component for app reports); an issue in the later report only is new, in
the earlier only is fixed, in both is still open. compareWith naming a report that
is not an app build, for an app build, gets the same guidance.

The compareWith description now says which reports it compares (Scanner runs of one
scan, app builds) and to read both for any other.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(accessibility): compare Automated Tests builds with compareWith

compareWith on an Automated Tests (web) build answered "can't be compared here yet" and
then, since 6a777ac, told the agent to read both builds and compare them by hand. In MCP
dogfooding (3 Nov) 3 of 3 agents passed correct testhub ids and got no compare.
compareWith on such a build (buildId, or its report link as reportUrl) now takes another
build's id, its report link, or "previous", and compares them:

- The older build, by start time, is "before" and the newer "after", whichever order they
  were passed in. When compareWith is the newer one the answer says so in one line. Both
  builds are named (name, project, #number, id, start), the names quoted as data.
- Two builds of one build name and project are compared by the Automated Tests compare API
  (GET /api/automated-tests/v1/builds/compare/issues, newReport = after, baseReport =
  before). It takes the builds in the order given and does not check it. "previous" calls
  it without baseReport, and the answer names the build the API picked.
  Its issues are keyed by the API's own signatureKey, and the answer says it matches by
  that, not by rule, element and page alone. The API told apart a css-orientation-lock issue
  on <html> before and after a fix added lang to it, and listed it as both fixed and new.
  With a rule|element|page key on both, agents recounted the dashboard's 6/13/1 to 4/11/3.
- Any other pair, or a pair the API refuses (issue-count threshold, "TestRun Not Found") or
  does not answer (504 after its 30 s, or no answer within 40 s), is compared by matching the
  two builds' issue exports by rule, element and page, as Scanner runs are. The answer says
  which method was used, in one line.
- Either way the answer gives three lists with counts: new (only in after), fixed (only in
  before), still open (in both). Each list shows at most 50 issues and says how many more.
  For the API, a list cut short on its first page (100 rows) is read on its own.
- Issues found on a BrowserStack Local or Chrome network error page are set apart as "page
  did not load". The diff also catches a page whose error-page issues are all duplicates of
  another page's, as a grouped build's are; before, that page's earlier issues read as fixed.
- A build the build list calls "failed" (completed with no test cases) has no accessibility
  results: the answer says so as guidance and does not compare. "previous" with no earlier
  completed run of the build's name is said plainly, with no request to report it.

The compareWith description now names web builds (21,495 of the 21,500 budget).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(accessibility): re-run an earlier scan of the same page instead of starting a new one

startAccessibilityScan always created a new scan, so re-scanning a page after a fix
made a second scan whose runs could not be compared with the first
(fetchAccessibilityIssues compareWith failed with "Scan logs not found").

- New optional scanId. When that scan is for this same page (compared as startScan
  saves URLs: local hosts as bs-local.com, host case and a trailing slash ignored),
  start a new run of it (POST /scans/:id/new_scan) with its saved settings, wait for
  it like a new scan, and end with how to compare with compareWith "previous".
  Without scanId a new scan is created as before.
- scanId for another page (or several pages): a new scan, saying why.
- Settings passed that differ from the scan's saved ones: no run; each difference and
  the two ways forward are listed. Only settings the call passed are compared; a login
  or tunnel can only be checked for being saved at all, which is said.
- A re-run keeps the scan's name (a different name passed is mentioned), and a Local
  scan's re-run says the tunnel must run with the saved identifier.
- Re-run refusals as plain results: a run still going (with its run id), a deleted
  login config, an unknown scan, the plan's message.
- Every new scan says how to scan the same page again with its scanId.
- The 50-scan plan limit on create is said plainly instead of "open an issue on GitHub".
- Shorter wording for a few accessibility argument descriptions, to stay within the
  description budget with the new argument.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(local): never kill BrowserStack Local tunnels this MCP did not start; local-URL limits as plain answers

Every public-URL accessibility scan and Live session ran `pkill -f BrowserStackLocal`
(`taskkill /IM BrowserStackLocal.exe` on Windows), as did starting a tunnel for a local
URL, so a tunnel the user had up (their own, or their test suite's) was killed by an
unrelated call.

- lib/local.ts no longer kills any process by name. A public URL takes no tunnel
  action at all.
- A local-URL scan without localIdentifier runs through this MCP's own tunnel, started
  through browserstack-local under a unique mcp-<uuid> identifier, kept in module state
  and reused while it runs. It is replaced (the old one stopped through its own handle)
  only once it has died or the credentials change, and stopped by its own pid when the
  process exits or is signalled. The binary runs tunnels with different
  --local-identifier values side by side, so the user's keeps running; `force` is
  dropped from the configured options. Should ours fail to start while another Local is
  running, the error says to pass that tunnel's name as localIdentifier.
- Live sessions (the other caller) start their unidentified tunnel only when no other
  BrowserStack Local is running, and otherwise use the one that is, leaving it alone.
- A local URL the MCP cannot tunnel to (the hosted MCP, or USE_OWN_LOCAL_BINARY_PROCESS)
  is answered as a plain result with #541's guidance (start BrowserStack Local and pass
  its name as localIdentifier, or scan a public URL), with no GitHub or submitFeedback
  line.
- The accessibility tools' error wrapper adds a full stop only when the message lacks
  one, so it no longer writes "..".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(accessibility): search 25 pages of scans for a report link, not 15

A Website Scanner dashboard link is resolved by walking the scan list newest
first. At the list's default 20 per page, 15 pages reached ~300 scans, so a
link to an older scan came back "not found". 25 pages reach ~500, at about
1 s a page (~26 s worst case). Stopgap until the issues API reads by
thBuildId (accessibility #5689).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(accessibility): no 5-minute wait on a finalizing report; account refusals as plain answers; expert questions within 500 chars

fetchAccessibilityIssues waited up to five minutes inside the call for a report still
finalizing (or an app build still running its rules). Clients cut off a call that long:
the claude.ai connector aborts a tool call silent for 305 s, progress notifications or
not, so the agent got nothing, and one reported a fix as verified that had added 26 issues.

- A first read of a report that is "finalizing" or "rule_processing" now answers at once.
  It is not an error: it says no issues were read yet, so the page is not to be called
  clean nor a fix verified, and gives the exact call to make again in a minute or two
  (pageUrl and compareWith kept), or to call the report stuck after about 10 minutes.
  The wait loop and REPORT_STATUS_POLL_MS / REPORT_STATUS_WAIT_MS are gone. A later page,
  or a report whose tests are still running, is read as before with its "still
  processing" note.
- A cancelled Website Scanner run is answered like a failed one (failedRunResponse): an
  error saying it has no results and that 0 issues does not mean the page is clean,
  instead of reading its empty export as "0 issues, all retrieved".
- Refusals the user resolves on their own account are explained instead of reported as
  faults: the plan's saved-scan limit ("make over N scans"), a plan without the External
  APIs add-on, and Website Scanner's "user info missing" (open Website Scanner once in the
  dashboard first). Each says what to tell the user and quotes the API's words, with no
  "open an issue on GitHub" line. They carry NO_FEEDBACK_HINT, a new symbol-keyed flag in
  lib/feedback-hint.ts that is never serialised, so withFeedbackHint does not send the
  agent to submitFeedback. (startAccessibilityScan's 402 scan-limit refusal is still
  answered first by the scanner's ScanRefusedError; this covers the message wherever else
  it reaches the error handler.)
- accessibilityExpert keeps a question within the knowledge base's 500-character limit,
  which refused longer ones ("Query is too long"). A longer question keeps its last
  sentence, where agents put the question itself, and as much of its opening as fits,
  cut at a word break; the answer starts with a note that it was shortened.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(accessibility): drop the new test files from this PR

Removes the six new test files and the test cases this PR added to existing
files. Keeps only the edits that existing tests need because behaviour changed
here: no 5-minute wait on a finalizing report, compareWith guidance on reports
with no compare, failed-page wording, and no tunnel killing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(accessibility): fetchAccessibilityIssues no longer waits on a processing report

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(accessibility): resolve a Scanner link or build id passed as compareWith, and compare runs oldest first

compareScanRuns took compareWith verbatim as a run id, so a Website Scanner
report link or a run's thBuildId failed with "Scan run id must be an Integer",
a run id of another scan with "Scan logs not found", and a newer run id swapped
new and resolved without a word.

compareWith is now resolved as a source is: a link or 40-character id through
sourceFromReportUrl / resolveBuild and findScanRun. A run of the same scan is
compared with the older run as "before", saying so when the order was swapped.
Another scan's run, a build or a report, and a run id the scan's full run list
does not hold, get a plain answer naming both reads' arguments.

compareWith's description drops "older": the runs are ordered here now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(accessibility): do not compare a run or build that is still processing

Comparing against a run still processing gave a clean-looking diff ("0 new, 0
resolved, 2 still open") while its issues were still being added, which could
read a fix as verified. The Website Scanner, Automated Tests and app build
compares now check both sides' statuses first: while either is processing or
finalizing, the answer is a plain "not finished yet" naming the run or build
and its status, telling the agent to have the user ask again in a few minutes
and not to call again in this turn.

The Automated Tests compare's "only the issues found so far are compared"
warning goes, as such a build is no longer compared. An app build compare is
dispatched before the finalizing read, so a build still running its rules
gets this answer rather than "call again in a minute or two".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(accessibility): find a Website Scanner run given only its build id in the scan list

A Website Scanner run's thBuildId passed as buildId, with no link, was refused
whenever the API could not yet read a run by its build id: "pass the scan's
report link as reportUrl instead", plus the GitHub and submitFeedback lines.
An agent then searched for the run itself, and on an old scan gave up.

Such an id now walks the scan list with findScanRun, as a link does, under the
same 25-page cap and with the same "found in the scan list instead" note,
naming the scan and run it found. Only when the walk finds nothing (or the API
that reads runs by build id has none) is the answer a plain not-found, saying
what to pass instead, without the GitHub or submitFeedback lines.

findScanRun's not-found is now a ScanNotInListError, with the same message.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(accessibility): read an Automate build's hashed_id passed as buildId as its accessibility build

An SDK run prints the Automate build's hashed_id ("stopBinSession
hashed_id=…"), and an agent passing it as buildId was refused, with only a
Website Scanner build id offered as an alternative. It then spent recovery
calls on fetchBuildInsights and listSessions before finding a session link.

A 40-hex id that is neither an Automated Tests nor an App Accessibility build
is now tried as an Automate build, then an App Automate build, through the same
session mapping their dashboard links use, before the scan-list walk. The
answer says which build the id was read as. An id none of them knows says so
in the plain not-found answer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(accessibility): a report still running with no export yet is "no report yet", not "Failed to fetch report: null"

A build stuck in "running" since 2025-09-16 was read as processing, so its
export ran anyway and failed with error null: "Failed to fetch report: null.
Please open an issue on GitHub", and the agent went searching for another
product before handing the question back to the user.

When a report is still processing and its export fails, the answer is now a
plain "still running (status, started <date>), no report yet". If it started
more than 24 hours ago it is called probably stuck, to be run again, rather
than something to call again for.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(accessibility): name the Workflow Analyzer report before this one for compareWith "previous"

compareWith "previous" on a Workflow Analyzer report only said there is no
built-in compare and to read both, without saying which report came before.
Agents went looking for a way to list reports, hit routing refusals, and the
hosted run gave no answer.

The earlier report is now found here: the account's report list (newest first)
locates this report, and the summaries of the completed reports after it give
their start and pages. The newest one before it that scanned one of its pages
is named, or else the newest on its site, with its id, its dashboard link and
the two reads to make. When none is among the newest reports, the answer says
so and to ask the user which report to compare with.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(accessibility): keep a manual app report's page under 40k characters, describe cursor, and find its previous report

A 70-issue manual report came back as one page of 51,132 characters, over
Claude Code's 50,000-character limit, so the client saved it to a file. The
page budget counted only the bare issue rows, while the answer prints them as
indented JSON with the rules and screens they name. An app page (build or
manual report) now counts each issue as printed, and each rule's text and
screen's screenshot the first time the page names it, against a 35,000
character budget: with the summary, notes and wrappers the whole answer stays
under about 40,000. The cursor still counts the bare issues.

Agents then passed cursor as an issue index (40, 35) and got the same page
again. cursor is now described as the next_page value from the previous
answer, not an issue index.

compareWith "previous" on a manual report now names the newest finished
manual report of the same app recorded before it (found through the report
list, by the report's name and then its app's package), with its id, its
dashboard link and the two reads to make, as for a Workflow Analyzer report.

Descriptions (total unchanged at 21,500):
- cursor: "Pagination offset (default 0); with compareWith, issues already
  listed." (71) -> "The next_page value from the previous answer (default 0),
  not an issue index." (77)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(accessibility): match a Workflow Analyzer report's site only on a page address that parses

Follow-up to the "previous" Workflow Analyzer lookup: a page address that does
not parse has no site, and two such addresses no longer count as one site.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(accessibility): say compareWith "previous" works for every report

compareWith's description ended "Others: read both", so for a Workflow
Analyzer or manual app report agents never passed "previous" and went
searching for the earlier report themselves: in re-runs, 4 of 5 sessions
skipped it, and one hosted run again gave no answer. "previous" now finds the
earlier report for those too, and the tool itself answers any pair it has no
compare for with how to read both, so the description says "previous" works
for any report.

compareWith: 'Runs of one scan, web/app builds: "previous", scanRunId, build
id/link. Others: read both.' (90) -> '"previous" (any report), or a run of the
same scan or a web/app build: its id or link.' (86). Total 21,500 -> 21,496.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* revert(local): leave BrowserStack Local tunnel handling and the feedback hint as they are on #541

Restores src/lib/local.ts, src/lib/feedback-hint.ts and
src/tools/live-utils/start-session.ts to their #541 versions, which puts
back killExistingBrowserStackLocalProcesses and drops ensureOwnTunnel,
stopOwnTunnels, killOwnTunnelsNow and NO_FEEDBACK_HINT.

startScan again starts a tunnel with ensureLocalBinarySetup for a local URL
and calls killExistingBrowserStackLocalProcesses for a public one. The
refusals for the hosted MCP and for USE_OWN_LOCAL_BINARY_PROCESS with a
local URL stay ScanRefusedError plain answers, and account refusals keep
their wording without the feedback-hint opt-out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(accessibility): move this PR's report, compare and re-scan helpers into accessiblity-utils

accessibility.ts had grown to 3,337 lines. The helpers this PR added move,
unchanged, into accessiblity-utils:

- error-messages.ts: asSentence and the account-refusal wording
- expert-query.ts: fitting an accessibilityExpert question to 500 characters
- report-answers.ts: failed, cancelled, finalizing and no-report-yet answers,
  unknown build ids, and pages not scanned or that showed an error page
- compare-with.ts: what compareWith names (a scan run, the "previous" report),
  the read-both guidance and the not-finished-yet answer
- compare-lists.ts: the export diff, compared lists and not-loaded note that
  scan-run and web-build comparisons share
- web-build-compare.ts: comparing Automated Tests (web) builds
- scan-rerun.ts: the saved-settings checks, notes and hints of a re-run

shared.ts takes the small helpers of accessibility.ts these need too (the
response builders, source names, progress, the scanner client, WCAG names),
so neither side imports the other. accessibility.ts re-exports
EXPERT_QUERY_MAX_CHARS. No behaviour or tool description changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(accessibility): no stale exports, faster link lookups, Automate probe falls through

- An export started while a report is still processing no longer has its task
  remembered: continued after the report finished, it handed back the
  mid-processing snapshot as final.
- A run found by its build id is cached (per account, bounded), since a build
  id names one fixed run: a link read again, or both sides of a compare, skip
  the scan-list walk. The EXPORT_WAIT_MS comment no longer claims a whole call
  stays under 60 s.
- A 40-hex id that Automate or App Automate can't answer about (an HTTP failure
  other than 404, or an Automate build with no sessions) moves on to the next
  API instead of ending the call, and the not-found answer says why. Once a
  session is found, what it says is still the answer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(accessibility): don't offer re-runs that can't reach a local page; say what a re-run ignores

- A local page this MCP tunnelled itself is saved with an identifier nobody
  was shown, and the tunnel is gone after the next public scan or a restart.
  Its scan no longer gets the "re-run with scanId" hint, and a re-run asked
  for a scan saved with Local but no localIdentifier starts a new scan
  instead, saying why and how to re-run it with its own tunnel.
- A re-run can't take a login or a Local identifier: each passed one is now
  named as not used, with how to scan with it (a new scan), instead of only
  "could not be checked".
- The saved scan's pages and name come from the account, which another user
  may have set: they are now quoted as untrusted data in re-run answers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(accessibility): refuse to compare with a cancelled scan run, as with a failed one

A cancelled run has no results and exports an empty report, so compared with
it every issue of the other run read as resolved or new. Reads already treat a
cancelled run as having no results; compares now do too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(accessibility): don't silently take compareWith as the older build when start times are missing

A build with no start time made the order check always false, so a newer build
passed as compareWith was used as "before" and new and fixed issues swapped
without a word. The order now falls back to the build number for runs of one
build name and project; with neither, the passed order is kept and the answer
says so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(accessibility): shorten CJK expert questions by sentence, and say what was kept

The sentence split knew only . ? !, so a Chinese or Japanese question was one
"sentence" cut at 500, losing the question at its end. 。?! now end sentences
too. A cut no longer leaves half a surrogate pair, and the note says whether
the opening alone or the opening and last sentence were sent, as it happened.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(accessibility): name the plan's scan limit the API gave, not a hard-coded 50

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* style: prettier

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(accessibility): put the saved scan's name after the settings list in a re-run refusal

The quoted name sat between "these differ:" and the list, so it read as one of the
differences.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(accessibility): cover web-build compare, re-runs, failed runs and the Automate id probe

Brings back the web-build compare, re-run and failed-run tests dropped in 7ee4c3d,
updated to this PR's current behaviour, and adds cases for the review fixes:
- web-build compare: refusing a build still processing, ordering by build number when a
  start time is missing, and keeping the passed order with a note otherwise;
- re-runs: a Local-saved scan without localIdentifier starts a new scan, a local page the
  MCP tunnelled gets no re-run hint, a login or tunnel passed is named as not used, the
  saved pages and name are quoted as untrusted, and the scan limit is the API's number;
- a 40-hex Automate or App Automate build id: found, falling through on 403/5xx or an
  empty build, and naming why Automate could not be asked;
- compare refused for a cancelled run;
- exports: no task kept while a report is processing, a run read by build id reused for
  5 minutes at most, and a link's run found once not looked up again;
- expert questions: the note matches what was cut, CJK sentence ends, no split character.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@Dave3130
Dave3130 merged commit cb12556 into fix/a11y-report-links Nov 6, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants