feat(spec): a save of a connector's served form drops a leftover undeclared auth key and names it

.changeset/28640-withheld-not-carried-forward.md

@@ -0,0 +1,17 @@
+---
+'@objectstack/spec': patch
+'@objectstack/metadata-protocol': patch
+---
+
+feat(spec): a save of a connector's served form drops the leftover inline value under an undeclared `auth` key, and the save answer names each key it dropped
+
+Clause-②: yes (widening)
+Metadata-protocol: none
+
+A connector row stored before the instance-auth shapes went strict can still hold a value under an `auth` key its kind does not declare, typically an inline secret such as `auth.token`. The read already withheld it. A save of the served form, however, re-grafted it from the stored row, so the cleartext stayed at rest through every edit, although no door accepts the key and nothing at runtime reads it. Now:
+
+- **The redactor can mark a withheld position as not carried forward.** `MetadataRedactionResult` (`@objectstack/spec/kernel`) gains an optional `notCarriedForwardKeys`: the positions among `redactedKeys` that a save drops instead of re-grafting. It is a subset, so `redactedKeys` still names every withheld position and a reader that knows only `redactedKeys` withholds exactly what it did before. Absent means none: the `datasource` built-in, the schema-derived redactors and every registered redactor keep carrying every withheld position forward, with nothing to change.
+- **The connector redactor marks its undeclared `auth` keys.** Each one is named in both lists. A cleared value there (`null`, `''`, an empty map) is now withheld too, because the strict shape refuses the key whatever it holds, so serving it made a plain read then save fail with `unrecognized_keys`. Every other withheld connector position (the four header and environment maps, `authentication`, a credential URL, an undeclared top-level, `retryConfig` or `actions` key, a non-object `auth`) is carried forward exactly as before.
+- **The save drops it and says so.** The `/meta` save (`saveMetaItem`, draft saves included) never re-grafts a not-carried-forward position, so the stored row loses it, and the answer's `advisories` carries one `warning` per dropped position: rule `withheld-key-dropped`, `where` naming the item, and `path` where the value sat at rest (`connectors[0].auth.token`). The notice never carries the value. The key is still omitted when there is nothing to report, so a save that drops nothing answers the same bytes as before. A value the incoming body itself holds at that position is the author's word and is not a drop.
+
+If a connector still needs a credential that sat under such a key, store the secret outside the metadata and name it with `auth.credentialRef`.

content/docs/references/api/protocol.mdx

@@ -2874,7 +2874,7 @@ The authoring surface of a metadata write. Closed: `copilot` is an AI assistant'
 | **seq** | `integer` | ✅ | Monotonic sequence number of the metadata event this write appended to the item history (sys_metadata_history.event_seq). Orders writes; unlike `version` it is not an OCC token. |
 | **state** | `Enum<'draft' \| 'active'>` | ✅ | Lifecycle the body was written into: "draft" when the request asked for draft mode (`?mode=draft`), otherwise "active" (published and live). A draft is staged only — it is not served to the runtime until published. |
 | **projectionApplied** | `{ success: boolean; error?: string }` | optional | Outcome of the awaited ADR-0094 mutation projector — the post-persist step that materializes this metadata into its derived data-plane read model (e.g. `datasource` → its live connection pool). Present ONLY when a projector is registered for this metadata type, which is why it is optional: its absence means "no projector ran", never "the projection failed". Best-effort by design — a projector failure is reported here and logged, never thrown, so a caller that needs the read model to be live must check `projectionApplied.success` rather than rely on the 200. |
-| **advisories** | `{ rule: string; path: string; where: string; message: string; … }[]` | optional | Non-gating findings from the runtime authoring gate — the same shared author-time rules `os validate` / `os build` / `os lint` run, applied to this body on its way to `active`. The write SUCCEEDED; these are what the gate has to say about it anyway (closing D3). Present ONLY when at least one advisory was raised — an empty array is never emitted, so a clean save's response bytes are unchanged and absence means "nothing to report", never "the gate did not run". Advisory by construction: every entry has `severity` `warning` or `info`, because an `error` finding refuses the write and arrives as the 422 `invalid_metadata` envelope instead of here. A caller that ignores this key behaves exactly as before. Runtime-only: the CLI surfaces the same findings on its own stdout, and a Studio / MCP / AI author has no CLI at all, which is the gap this key exists to close. The gate runs on both write doors (D1), and both report: `POST /meta/:type/:name/publish` carries the same key on `PublishMetaItemResponseSchema`. |
+| **advisories** | `{ rule: string; path: string; where: string; message: string; … }[]` | optional | Non-gating findings from the runtime authoring gate — the same shared author-time rules `os validate` / `os build` / `os lint` run, applied to this body on its way to `active`. The write SUCCEEDED; these are what the gate has to say about it anyway (closing D3). Present ONLY when at least one advisory was raised — an empty array is never emitted, so a clean save's response bytes are unchanged and absence means "nothing to report", never "the gate did not run". Advisory by construction: every entry has `severity` `warning` or `info`, because an `error` finding refuses the write and arrives as the 422 `invalid_metadata` envelope instead of here. A caller that ignores this key behaves exactly as before. Runtime-only: the CLI surfaces the same findings on its own stdout, and a Studio / MCP / AI author has no CLI at all, which is the gap this key exists to close. The gate runs on both write doors (D1), and both report: `POST /meta/:type/:name/publish` carries the same key on `PublishMetaItemResponseSchema`. The save also reports here what it did to the row at rest that the author did not write, on a draft save too: one `warning` with rule `withheld-key-dropped` for each stored position it dropped because the type's read redactor withholds it and marks it not carried forward (a connector's undeclared `auth` keys, which no door accepts and nothing reads). Its `path` names where the value sat at rest, in the same spelling (`connectors[0].auth.token`). |
 | **message** | `string` | optional |  |
 
 ### Nested Shape: `SaveMetaItemResponse.projectionApplied`

packages/metadata-protocol/src/metadata-redaction.connector-undeclared-keys.test.ts

@@ -8,7 +8,12 @@
 // Every value below is an obviously-fake placeholder.
 
 import { describe, it, expect } from 'vitest';
-import { carryForwardRedactedValues, redactMetadataItem } from './metadata-redaction.js';
+import {
+  carryForwardRedactedValues,
+  carryForwardRedactedValuesReportingDrops,
+  redactMetadataItem,
+  redactedPathsCarriedForward,
+} from './metadata-redaction.js';
 
 const P = (n: number) => `TEST_PLACEHOLDER_UNDECLARED_${n}`;
 
@@ -84,3 +89,55 @@ describe('carry-forward of the keys a connector does not declare', () => {
     expect(out).toEqual({ ...malformed, label: 'Vendor catalog (edited)' });
   });
 });
+
+// [#28640 ②B] An undeclared `auth` key is withheld AND not carried forward: a
+// save of the served form drops it, and the drop is reported.
+describe('an undeclared `auth` key is not carried forward', () => {
+  const legacy = {
+    name: 'vendor_catalog',
+    label: 'Vendor catalog',
+    type: 'api',
+    provider: 'rest',
+    apiKey: P(1),
+    providerConfig: { baseUrl: 'https://api.vendor.example/v1', headers: { Authorization: P(6) } },
+    auth: { type: 'bearer', credentialRef: 'VENDOR_TOKEN', token: P(7) },
+  };
+  const servedLegacy = () => JSON.parse(JSON.stringify(redactMetadataItem('connector', legacy))) as Record<string, any>;
+
+  it('the save drops it, names it, and still carries every other withheld position (undeclared root key, header map)', () => {
+    const { item, dropped } = carryForwardRedactedValuesReportingDrops('connector', servedLegacy(), legacy);
+    expect(item).toEqual({ ...legacy, auth: { type: 'bearer', credentialRef: 'VENDOR_TOKEN' } });
+    expect(dropped).toEqual(['auth.token']);
+    expect(carryForwardRedactedValues('connector', servedLegacy(), legacy)).toEqual(item);
+  });
+
+  it('the runtime gate is never told it will be restored', () => {
+    expect(redactedPathsCarriedForward('connector', servedLegacy(), legacy).sort())
+      .toEqual(['apiKey', 'providerConfig.headers']);
+  });
+
+  it('a value the incoming body holds at that position is the author\'s word, not a drop', () => {
+    // `headerName` is undeclared on `bearer` and declared on `api-key`: the
+    // author switched kinds and wrote it.
+    const atRest = { ...legacy, auth: { type: 'bearer', credentialRef: 'VENDOR_TOKEN', headerName: P(8) } };
+    const body = JSON.parse(JSON.stringify(redactMetadataItem('connector', atRest))) as Record<string, any>;
+    expect(body.auth).toEqual({ type: 'bearer', credentialRef: 'VENDOR_TOKEN' });
+    body.auth = { type: 'api-key', credentialRef: 'VENDOR_KEY', headerName: 'X-Key' };
+    const { item, dropped } = carryForwardRedactedValuesReportingDrops('connector', body, atRest);
+    expect(dropped).toEqual([]);
+    expect((item as Record<string, any>).auth).toEqual({ type: 'api-key', credentialRef: 'VENDOR_KEY', headerName: 'X-Key' });
+  });
+
+  it('a body that removes `auth` drops it too, and says so', () => {
+    const body = servedLegacy();
+    delete body.auth;
+    const { item, dropped } = carryForwardRedactedValuesReportingDrops('connector', body, legacy);
+    expect(item).not.toHaveProperty('auth');
+    expect(dropped).toEqual(['auth.token']);
+  });
+
+  it('CONTROL: a row with no undeclared `auth` key reports no drop', () => {
+    const { dropped } = carryForwardRedactedValuesReportingDrops('connector', served(), stored);
+    expect(dropped).toEqual([]);
+  });
+});

packages/metadata-protocol/src/metadata-redaction.ts

@@ -48,7 +48,13 @@
  *
  * ⛔ It does NOT create cleartext, and does not migrate it out either: it
  * preserves what is already at rest. Getting stored cleartext OUT of the store
- * is #8081 item 3's migration and is deliberately not attempted here.
+ * is #8081 item 3's migration and is deliberately not attempted here — with
+ * ONE exception the redactor itself declares. A position it names in
+ * `notCarriedForwardKeys` (#28640 ②B, ruling 6566912252: a connector's
+ * undeclared `auth` keys, which no door accepts and nothing reads) is never
+ * re-grafted, so a save of the served form drops it from the row, and the
+ * save names each dropped position on its answer
+ * ({@link carryForwardRedactedValuesReportingDrops}).
  *
  * ## Why nothing is stamped on the wire
  *
@@ -448,6 +454,11 @@ function sameValue(a: unknown, b: unknown): boolean {
  *  - the incoming body has no container for that path at all ⇒ nothing is
  *    grafted, because a removed container is also the author's word.
  *
+ * Those three are for a path in `redactedKeys` only. A path the redactor also
+ * names in `notCarriedForwardKeys` (#28640 ②B) is never grafted, whatever the
+ * incoming body holds there: a save of the served form drops it, and
+ * {@link carryForwardRedactedValuesReportingDrops} reports the drop.
+ *
  * A path through an ARRAY (a flow's start node, `nodes.<i>.config.secret`,
  * #20552) is walked by the stored element's `id`, not by its index, so a body
  * that reorders the array still carries the value onto the element it came
@@ -520,9 +531,39 @@ export function redactedPathsCarriedForward(type: string, incoming: unknown, sto
 }
 
 /**
- * The one decision behind {@link carryForwardRedactedValues} and
- * {@link redactedPathsCarriedForward}: the grafted body, and where each graft
- * landed. `landed` is empty exactly when `out` is `incoming` by reference.
+ * [#28640 ②B] {@link carryForwardRedactedValues}, and what the same save DROPS
+ * from the row at rest: every position the type's redactor names in
+ * `notCarriedForwardKeys` for the stored body (withheld AND not carried
+ * forward — a connector's undeclared `auth` keys) where the stored body holds
+ * a value and the incoming body holds none. Such a position is never
+ * re-grafted, so a save of the served form persists the row without it; the
+ * dotted paths (the redactor's spelling, sorted) are what the save names on
+ * its answer, so the drop is loud rather than silent.
+ *
+ * A position the incoming body DOES hold a value at is not dropped: that
+ * value is the author's word and is judged at the door like any other (an
+ * `auth` kind the author switched to may declare the key). Not on the
+ * package entry: the save door's half, consumed inside this package only.
+ *
+ * @param type     request-shaped metadata type (plural or singular).
+ * @param incoming the body about to be persisted.
+ * @param stored   the body currently at rest, RAW (never a served copy).
+ */
+export function carryForwardRedactedValuesReportingDrops<T>(
+    type: string,
+    incoming: T,
+    stored: unknown,
+): { item: T; dropped: string[] } {
+    const plan = planCarryForward(type, incoming, stored);
+    return { item: plan.out, dropped: plan.dropped };
+}
+
+/**
+ * The one decision behind {@link carryForwardRedactedValues},
+ * {@link redactedPathsCarriedForward} and
+ * {@link carryForwardRedactedValuesReportingDrops}: the grafted body, where
+ * each graft landed, and which not-carried-forward stored positions the body
+ * drops. `landed` is empty exactly when `out` is `incoming` by reference.
  */
 /**
  * A `redactedKeys` entry resolved against the STORED body into the container
@@ -559,22 +600,39 @@ function resolveRedactedPath(
     return undefined;
 }
 
-function planCarryForward<T>(type: string, incoming: T, stored: unknown): { out: T; landed: string[] } {
-    const unchanged = { out: incoming, landed: [] as string[] };
-    if (!incoming || typeof incoming !== 'object' || Array.isArray(incoming)) return unchanged;
-    if (!stored || typeof stored !== 'object' || Array.isArray(stored)) return unchanged;
+function planCarryForward<T>(type: string, incoming: T, stored: unknown): { out: T; landed: string[]; dropped: string[] } {
+    const none = { out: incoming, landed: [] as string[], dropped: [] as string[] };
+    if (!incoming || typeof incoming !== 'object' || Array.isArray(incoming)) return none;
+    if (!stored || typeof stored !== 'object' || Array.isArray(stored)) return none;
     const redactor = redactorFor(type);
-    if (!redactor) return unchanged;
+    if (!redactor) return none;
 
     // What a read exit WOULD have served for the row at rest. Computed from the
     // stored body rather than remembered from a response, so the comparison
     // holds for any caller — Studio, the CLI, a raw `curl` — and needs no
     // session state.
     const served = redactor(stored as Record<string, unknown>);
-    if (served.redactedKeys.length === 0) return unchanged;
+    if (served.redactedKeys.length === 0) return none;
+
+    // [#28640 ②B] The withheld positions the redactor marks NOT carried
+    // forward: never re-grafted, so the save drops each one the stored row
+    // holds and the incoming body does not — and reports it (`dropped`).
+    const notCarriedForward = new Set(served.notCarriedForwardKeys ?? []);
+    const dropped: string[] = [];
+    const unchanged = { out: incoming, landed: [] as string[], dropped };
 
     const grafts: Array<{ at: string[]; key: string; value: unknown }> = [];
     for (const path of served.redactedKeys) {
+        if (notCarriedForward.has(path)) {
+            const resolved = resolveRedactedPath(stored, path);
+            if (!resolved) continue;
+            const target = locate(incoming, resolved.hops);
+            const authored = target !== undefined
+                && Object.prototype.hasOwnProperty.call(target.container, resolved.key)
+                && target.container[resolved.key] !== undefined;
+            if (!authored) dropped.push(path);
+            continue;
+        }
         // Dotted, item-relative — the registry's documented contract for
         // `redactedKeys` (`config.password`; an array hop is an index into the
         // stored body, `nodes.0.config.secret`, resolved to an identity by
@@ -621,7 +679,7 @@ function planCarryForward<T>(type: string, incoming: T, stored: unknown): { out:
         if (kept.length === 0) return unchanged;
         const withheld = new Set(redactor(out as Record<string, unknown>).redactedKeys);
         const next = kept.filter((graft) => withheld.has(landing(graft)));
-        if (next.length === kept.length) return { out: out as T, landed: kept.map(landing) };
+        if (next.length === kept.length) return { out: out as T, landed: kept.map(landing), dropped };
         kept = next;
     }
 }