fix(service-automation): a connection marked reconnect required says the cause that actually happened (#28863)

.changeset/28863-reconnect-required-cause.md

@@ -0,0 +1,14 @@
+---
+'@objectstack/service-automation': patch
+---
+
+fix(service-automation): a connection marked "reconnect required" says the cause that actually happened (#28863)
+
+Clause-②: no — prose, and the stored `status_reason` text of two refresh paths. No accept set, status value, error code or export changes meaning.
+Metadata-protocol: none
+
+A connector connection becomes `reconnect_required` four ways: the authorization server refuses the grant on refresh (`invalid_grant`); it issued no refresh token; the connection keeps no client to refresh with; or the upstream refuses the connection's token while an `mcp` instance discovers its tools. The connection's stored reason, the `sys_connector_connection.status` field help and the `ConnectorConnectionStore#markReconnectRequired` docblock all named only the first.
+
+- **The stored reason, per path.** `reconnectRequiredReason` takes an optional cause (`'refused'`, `'no_refresh_token'`, `'no_client'`; `'refused'` when none is given, the old sentence unchanged). The no-refresh-token path now stores "…its authorization server issued no refresh token, so an access token the connection has to renew cannot be renewed…", and the no-client path "…its connection keeps no client to refresh with, so an access token it has to renew cannot be renewed…". Neither says the server refused anything, and neither says the access token expired: a refresh also runs on a cache miss and on a dispatch's 401, while the token may still be valid. The dispatch's 503 refusal carries the same sentence.
+- **The field help and the docblock** list all four causes and point to `status_reason` for which one happened.
+- Unchanged: the refused path's sentence, the MCP discovery path's own reason, and every status, code and write.

content/docs/automation/connectors.mdx

@@ -686,9 +686,13 @@ a rotating refresh token is never spent twice.
   not a sign-in account, and signing in never touches it. A run dispatches
   with its own organization's connection; a run with no organization uses the
   instance's only connection, and is refused when there are several. When the
-  vendor refuses the grant (`invalid_grant`), the connection is marked
-  **reconnect required** and the instance degraded until an administrator
-  connects it again. A refresh, and the mark "reconnect required", are each
+  grant stops working, the connection is marked **reconnect required** and the
+  instance degraded until an administrator connects it again. That happens when
+  the vendor refuses the grant on refresh (`invalid_grant`), when the
+  connection keeps nothing to renew an access token it has to renew with (the
+  vendor issued no refresh token, or no client is kept to refresh with), or when
+  the upstream refuses the connection's token while an `mcp` instance discovers
+  its tools. The connection's `statusReason` says which. A refresh, and the mark "reconnect required", are each
   written only over the revision they read, in one conditional write, so a
   stale one never overwrites a newer connect on any replica. A revoke deletes
   the stored connection only: the server does not revoke the refresh token at
@@ -865,7 +869,7 @@ retry):
 | `provider` names no installed factory | **Hard boot error** naming the entry, the provider, and the plugin that supplies it — add the executor to `plugins:` |
 | Invalid `providerConfig` (missing `baseUrl` / `spec` / `transport`, wrong shape) | **Hard boot error** from the provider factory |
 | `auth` is a grant-bearing kind (`oauth2`, `app-credential`) on a provider that does not ask the credential handle (a plugin's provider registered without `{ credentialHandle: true }`) | **Not fatal**: the instance registers **degraded**, naming the kind, and the upstream is never called without its credential; a datasource borrowing it is refused with `422` |
-| An `authorization_code` instance nobody connected, or whose grant the vendor refused | **Not fatal**: the instance registers **degraded**, "not connected" or "reconnect required" — connect it (`POST …/connectors/:name/connection`), and it materializes without a restart |
+| An `authorization_code` instance nobody connected, or whose grant stopped working (the vendor refused it on refresh, no refresh token or client to renew an access token with, or the upstream refused its token at MCP tool discovery) | **Not fatal**: the instance registers **degraded**, "not connected" or "reconnect required" — connect it (`POST …/connectors/:name/connection`), and it materializes without a restart |
 | The token endpoint is unreachable or answers `5xx` / `429` | **Not fatal**: the dispatch fails `CONNECTOR_UPSTREAM_UNAVAILABLE` and is retried on the existing backoff; the connection stays connected |
 | `auth` is `signed` on a provider that does not say it applies the request signer | **Hard boot error** naming the instance, the scheme and the provider: its calls would go out unsigned — declare it through `rest` or `openapi` |
 | `credentialRef` (or a `signed` instance's `vendorPublicKeyRef`) resolves to nothing | **Not fatal**: the instance registers **degraded**, naming the reference — set it (`PUT …/credential-refs/:ref`), and it materializes without a restart |

packages/services/service-automation/src/connector-connections.ts

@@ -322,11 +322,15 @@ export class ConnectorConnectionStore {
     }
 
     /**
-     * Record that the authorization server refused the grant `record` held
-     * (revoked, or expired with no usable refresh). `reason` is the platform's
-     * sentence, never the server's text. Left alone when the row moved since
-     * — checked by the write itself ({@link updateAtRevision}): a reconnect
-     * that landed between the refusal and this write keeps its fresh grant.
+     * Record that the grant `record` held stopped working: the authorization
+     * server refused it on refresh, the connection keeps nothing to renew an
+     * access token it has to renew with (no refresh token was issued, or no
+     * client to refresh with), or the upstream refused the connection's token
+     * while an `mcp` instance discovered its tools. `reason` is the platform's
+     * sentence naming which, never the server's text. Left alone when the row
+     * moved since — checked by the write itself ({@link updateAtRevision}): a
+     * reconnect that landed between the mark and this write keeps its fresh
+     * grant.
      */
     async markReconnectRequired(record: ConnectorConnectionRecord, reason: string): Promise<void> {
         const engine = this.resolveEngine();

packages/services/service-automation/src/oauth2-grant/grant-handle.test.ts

@@ -413,3 +413,48 @@ describe('ADR-0143 D2.1 — an MCP instance whose connection keeps the client it
         expect(onGrantRefused).toHaveBeenCalledWith('org_a');
     });
 });
+
+describe('[#28863] the stored reason of a connection marked reconnect required names the cause that happened', () => {
+    const storedReason = (held: Map<string, ConnectorConnectionRecord>) => String(held.get('org_a')?.statusReason);
+
+    it('no refresh token: the reason names the missing refresh token, claims no refusal and no expiry', async () => {
+        const idp = tokenEndpoint();
+        const { store, held } = fakeConnections([{ org: 'org_a' }]);
+        const handle = createOAuth2GrantHandle({ connector: 'crm', config: AUTH_CODE, connections: store, fetch: idp.fetch });
+        const refused = await handle.get({ organizationId: 'org_a' }).catch((e) => e);
+        const reason = storedReason(held);
+        expect(reason).toContain("connector instance 'crm' needs to be reconnected for organization 'org_a'");
+        expect(reason).toContain('issued no refresh token, so an access token the connection has to renew cannot be renewed');
+        expect(reason).not.toMatch(/refused/);
+        expect(reason).not.toMatch(/expired/);
+        expect(refused.message).toBe(reason);
+        expect(idp.state.requests).toHaveLength(0);
+    });
+
+    it('no client: the reason names the missing client, claims no refusal and no expiry', async () => {
+        const idp = tokenEndpoint();
+        const { store, held } = fakeConnections([{ org: 'org_a', refreshToken: 'dummy-rt-a' }]);
+        const handle = createOAuth2GrantHandle({ connector: 'notion', config: { grant: 'authorization_code' }, connections: store, fetch: idp.fetch });
+        const refused = await handle.get({ organizationId: 'org_a' }).catch((e) => e);
+        const reason = storedReason(held);
+        expect(reason).toContain("connector instance 'notion' needs to be reconnected for organization 'org_a'");
+        expect(reason).toContain('keeps no client to refresh with, so an access token it has to renew cannot be renewed');
+        expect(reason).not.toMatch(/refused/);
+        expect(reason).not.toMatch(/expired/);
+        expect(refused.message).toBe(reason);
+        expect(idp.state.requests).toHaveLength(0);
+    });
+
+    it('control: a refresh the authorization server refuses (`invalid_grant`) stores a reason naming the refusal', async () => {
+        const idp = tokenEndpoint();
+        idp.state.answer = () => ({ status: 400, body: { error: 'invalid_grant' } });
+        const { store, held } = fakeConnections([{ org: 'org_a', refreshToken: 'dummy-rt-a' }]);
+        const handle = createOAuth2GrantHandle({ connector: 'crm', config: AUTH_CODE, connections: store, fetch: idp.fetch });
+        const refused = await handle.get({ organizationId: 'org_a' }).catch((e) => e);
+        const reason = storedReason(held);
+        expect(reason).toContain('its authorization server refused the grant (revoked, or expired with no usable refresh)');
+        expect(reason).not.toContain('dummy-rt-a');
+        expect(refused.message).toBe(reason);
+        expect(idp.state.requests).toHaveLength(1);
+    });
+});

packages/services/service-automation/src/oauth2-grant/grant-handle.ts

@@ -119,11 +119,40 @@ export function notConnectedReason(connector: string, organizationId?: string):
         + 'dispatch uses that connection (ADR-0143 D4)';
 }
 
-/** The dispatch refusal of an organization whose grant the authorization server refused. */
-export function reconnectRequiredReason(connector: string, organizationId: string): string {
-    return `connector instance '${connector}' needs to be reconnected for organization '${organizationId}': its authorization `
-        + 'server refused the grant (revoked, or expired with no usable refresh). Reconnect required: an administrator connects '
-        + 'it again, and the next dispatch uses the new connection (ADR-0143 D4)';
+/**
+ * [#28863] Why a refresh left a connection `reconnect_required`, as
+ * {@link reconnectRequiredReason} words it: the authorization server refused
+ * the grant (`invalid_grant`); it issued no refresh token; or the connection
+ * keeps no client to refresh with. The last two never ask the server. (The
+ * fourth way a connection is marked, the upstream refusing its token at MCP
+ * tool discovery, words its own reason in `plugin.ts`.)
+ */
+export type ReconnectRequiredCause = 'refused' | 'no_refresh_token' | 'no_client';
+
+const RECONNECT_REQUIRED_CAUSES: Readonly<Record<ReconnectRequiredCause, string>> = {
+    refused: 'its authorization server refused the grant (revoked, or expired with no usable refresh)',
+    // Not "expired": a refresh also runs on a cache miss (a fresh replica, a
+    // revision another replica moved) and on a dispatch's 401, while the
+    // access token may still be valid.
+    no_refresh_token: 'its authorization server issued no refresh token, so an access token the connection has to renew '
+        + 'cannot be renewed',
+    no_client: 'its connection keeps no client to refresh with, so an access token it has to renew cannot be renewed',
+};
+
+/**
+ * The dispatch refusal, and the stored `status_reason`, of an organization
+ * whose connection needs to be reconnected. `cause` names what happened on
+ * the path that marked it; with none, the authorization server's refusal, as
+ * before.
+ */
+export function reconnectRequiredReason(
+    connector: string,
+    organizationId: string,
+    cause: ReconnectRequiredCause = 'refused',
+): string {
+    return `connector instance '${connector}' needs to be reconnected for organization '${organizationId}': `
+        + `${RECONNECT_REQUIRED_CAUSES[cause]}. Reconnect required: an administrator connects it again, and the next dispatch `
+        + 'uses the new connection (ADR-0143 D4)';
 }
 
 /** Create the handle of one `oauth2` instance. */
@@ -266,21 +295,21 @@ export function createOAuth2GrantHandle(deps: OAuth2GrantHandleDeps): OAuth2Gran
                 + 'read back now; the next dispatch retries');
         }
         if (refreshToken === undefined) {
-            throw await grantRefused(current, reconnectRequiredReason(connector, organizationId)
-                + '. Its authorization server issued no refresh token, so an expired access token cannot be renewed');
+            throw await grantRefused(current, reconnectRequiredReason(connector, organizationId, 'no_refresh_token'));
         }
         const endpoint = await endpointOf(current);
         if (endpoint === undefined) {
-            throw await grantRefused(current, reconnectRequiredReason(connector, organizationId)
-                + '. Its connection keeps no client to refresh it with');
+            throw await grantRefused(current, reconnectRequiredReason(connector, organizationId, 'no_client'));
         }
         const params: Record<string, string> = { grant_type: 'refresh_token', refresh_token: refreshToken };
         if (endpoint.resource) params.resource = endpoint.resource;
         let token: OAuth2TokenResponse;
         try {
             token = await tokenRequest(params, endpoint);
         } catch (err) {
-            if (err instanceof OAuth2GrantRefusedError) throw await grantRefused(current, reconnectRequiredReason(connector, organizationId));
+            if (err instanceof OAuth2GrantRefusedError) {
+                throw await grantRefused(current, reconnectRequiredReason(connector, organizationId, 'refused'));
+            }
             throw err;
         }
         const written = await deps.connections.recordRefresh(current, {

packages/services/service-automation/src/sys-connector-connection.object.ts

@@ -95,7 +95,7 @@ export const SysConnectorConnection = ObjectSchema.create({
       required: true,
       defaultValue: 'connected',
       description:
-        "`connected` while the grant works; `reconnect_required` once the authorization server refused it (revoked, or expired with no usable refresh). Only a new connect returns it to `connected`.",
+        "`connected` while the grant works; `reconnect_required` once the grant stopped working: the authorization server refused it on refresh, the connection keeps nothing to renew an access token it has to renew with (no refresh token was issued, or no client to refresh with), or the upstream refused the connection's token while an `mcp` instance discovered its tools. `status_reason` says which. Only a new connect returns it to `connected`.",
     }),
 
     status_reason: Field.textarea({