p2p: a publisher can push its body, so a node nobody can reach can still sell

README.md

@@ -28,6 +28,9 @@ node's identity.
 The explorer UI is a separate build ([ainize-web](https://github.com/ainblockchain/ainize-web)); point
 `webDist` at it to have this process serve it too.
 
+Publishers behind NAT can offer bodies to opt-in peers using the
+[outbound P2P relay](docs/blob-relay.md), including retrying already-announced knowledge.
+
 ## The rules this node will not bend
 
 - **Publishing is not selling.** Two independent nodes must load the patch into the real model and

deploy/build-source-refresh.sh

@@ -0,0 +1,28 @@
+#!/usr/bin/env bash
+set -euo pipefail
+NODE=$(cd "$(dirname "$0")/.." && pwd)
+CORE=$(realpath "${1:?pass the compatible ainize-core worktree}")
+CLI=$(realpath "${2:?pass the compatible ainize-cli worktree}")
+OUT=${3:?pass a new build evidence directory}
+IMAGE=${4:?pass a new local image tag}
+BASE=${AINIZE_BASE_IMAGE:?pass the existing dependency/runtime image ID or tag}
+mkdir "$OUT"
+OUT=$(realpath "$OUT")
+mkdir "$OUT/context"
+BASE_ID=$(docker image inspect "$BASE" --format '{{.Id}}')
+printf '%s\n' "$BASE_ID" > "$OUT/base-image.txt"
+for name in core node cli; do
+  source=$NODE
+  if [ "$name" = core ]; then source=$CORE; fi
+  if [ "$name" = cli ]; then source=$CLI; fi
+  mkdir "$OUT/context/$name"
+  cp -a "$source/src" "$source/package.json" "$source"/tsconfig*.json "$OUT/context/$name/"
+  git -C "$source" rev-parse HEAD > "$OUT/$name-commit.txt"
+done
+cp -a "$NODE/trainer" "$OUT/context/node/"
+cp "$NODE/deploy/source-refresh.Dockerfile" "$OUT/context/Dockerfile"
+(cd "$OUT/context" && find . -type f -print0 | sort -z | xargs -0 sha256sum) > "$OUT/source-sha256.txt"
+DOCKER_BUILDKIT=0 docker build --cpu-period 100000 --cpu-quota 200000 --cpuset-cpus "${AINIZE_CPUSET:-0-7}" \
+  --memory 4g --memory-swap 4g --build-arg "BASE_IMAGE=$BASE_ID" -t "$IMAGE" "$OUT/context" > "$OUT/build.log" 2>&1
+docker image inspect "$IMAGE" > "$OUT/image.json"
+echo "Built $IMAGE. This command does not replace a running node or publish an image."

deploy/cert-docker/compose.ainize.json

@@ -0,0 +1,35 @@
+{
+  "name": "ain-cert-ainize",
+  "services": {
+    "node": {
+      "image": "${AINIZE_IMAGE:-ain-cert-ainize:relay-runtime-20260911}",
+      "network_mode": "host",
+      "cpus": 2,
+      "cpuset": "0-7",
+      "mem_limit": "8g",
+      "memswap_limit": "8g",
+      "user": "${AINIZE_UID:?set AINIZE_UID}:${AINIZE_GID:?set AINIZE_GID}",
+      "group_add": ["${DOCKER_GID:?set DOCKER_GID}"],
+      "environment": {
+        "AINIZE_HOME": "/mnt/newdata/gov/kpi/ainize/home-docker"
+      },
+      "volumes": [
+        "/mnt/newdata/gov/kpi/ainize/home-docker:/mnt/newdata/gov/kpi/ainize/home-docker",
+        "/mnt/newdata/qwen3.8:/mnt/newdata/qwen3.8",
+        "/mnt/newdata/gov/kpi/harness/dart-datasets:/datasets:ro",
+        "/mnt/newdata/gov/kpi/evidence:/mnt/newdata/gov/kpi/evidence",
+        "/var/run/docker.sock:/var/run/docker.sock:ro"
+      ],
+      "healthcheck": {
+        "test": ["CMD", "node", "-e", "fetch('http://localhost:3410/healthz').then(r=>process.exit(r.ok?0:1)).catch(()=>process.exit(1))"],
+        "interval": "15s",
+        "timeout": "5s",
+        "retries": 4,
+        "start_period": "30s"
+      },
+      "restart": "no",
+      "labels": { "org.ain.cert.role": "ainize-node" },
+      "logging": { "driver": "json-file", "options": { "max-size": "20m", "max-file": "4" } }
+    }
+  }
+}

deploy/cert-docker/upgrade-ainize.sh

@@ -0,0 +1,58 @@
+#!/usr/bin/env bash
+set -euo pipefail
+KPI=$(cd "$(dirname "$0")/.." && pwd)
+RUN_ID=${RUN_ID:-ainize_upgrade_$(date -u +%Y%m%dT%H%M%SZ)}
+[[ "$RUN_ID" =~ ^[A-Za-z0-9_-]+$ ]] || exit 1
+OUT="$KPI/evidence/$RUN_ID"
+BACKUP="$KPI/secrets/$RUN_ID"
+SNAPSHOT="$KPI/pr/an-relay/deploy/runtime-snapshot.mjs"
+TRAINER_ROOT=${AINIZE_TRAINER_ROOT:-/mnt/newdata/qwen3.8/.teach}
+mkdir "$OUT"
+mkdir -m 700 "$BACKUP"
+docker top ain-cert-ainize-node-1 -eo pid,stat,args > "$OUT/processes-before.txt"
+if awk '$0 ~ /node .*ainize-lifecycle[.]js/ && $2 !~ /T/ {found=1} END {exit !found}' "$OUT/processes-before.txt"; then
+  echo 'lifecycle observer is active; arrange a terminal/idle maintenance window first' >&2
+  exit 1
+fi
+for container in flashnext flashtrain; do
+  docker inspect "$container" --format '{{.Id}} {{.State.StartedAt}} {{.State.Pid}}' > "$OUT/$container-before.txt"
+done
+node "$SNAPSHOT" jobs "$KPI/ainize/home-docker" "$OUT/jobs-before.json"
+curl --fail --silent --show-error --max-time 30 http://localhost:3410/api/info > "$OUT/info-before.json"
+cp "$SNAPSHOT" "$OUT/runtime-snapshot.mjs"
+sha256sum "$OUT/runtime-snapshot.mjs" > "$OUT/source.sha256"
+node "$OUT/runtime-snapshot.mjs" capture "$KPI/ainize/home-docker" "$OUT/jobs-before.json" "$OUT/info-before.json" "$OUT/inventory-before.json" "$TRAINER_ROOT"
+docker inspect ain-cert-ainize-node-1 --format '{{json .State}}' > "$OUT/state-before.json"
+docker inspect ain-cert-ainize-node-1 --format '{{json .Image}}' > "$OUT/image-before.json"
+export AINIZE_UID="$(id -u)" AINIZE_GID="$(id -g)" DOCKER_GID="$(stat -c %g /var/run/docker.sock)"
+docker compose -f "$KPI/docker/compose.ainize.json" config --format json > "$OUT/compose.json"
+TARGET_IMAGE=$(node -e 'console.log(JSON.parse(require("fs").readFileSync(process.argv[1])).services.node.image)' "$OUT/compose.json")
+docker image inspect "$TARGET_IMAGE" --format '{{json .Id}}' > "$OUT/target-image.json"
+docker compose -f "$KPI/docker/compose.ainize.json" stop -t 60 node > "$OUT/stop.log" 2>&1
+umask 077
+tar -C "$KPI/ainize" -cf "$BACKUP/home-docker.tar" home-docker
+tar -C "$TRAINER_ROOT" -cf "$BACKUP/trained-bodies.tar" .
+umask 022
+sha256sum "$BACKUP/home-docker.tar" "$BACKUP/trained-bodies.tar" > "$OUT/backup.sha256"
+docker compose -f "$KPI/docker/compose.ainize.json" up -d --no-deps node > "$OUT/start.log" 2>&1
+docker inspect ain-cert-ainize-node-1 --format '{{json .Image}}' > "$OUT/image-after.json"
+cmp "$OUT/target-image.json" "$OUT/image-after.json"
+docker inspect ain-cert-ainize-node-1 --format '{{json .HostConfig}}' > "$OUT/limits-after.json"
+for attempt in {1..60}; do
+  if curl --fail --silent --max-time 5 http://localhost:3410/readyz > "$OUT/ready.json"; then
+    node -e 'const value=JSON.parse(require("fs").readFileSync(process.argv[1])); if (value.ready !== true && value.ok !== true) throw Error("not backend readiness JSON")' "$OUT/ready.json"
+    node "$OUT/runtime-snapshot.mjs" jobs "$KPI/ainize/home-docker" "$OUT/jobs-after.json"
+    curl --fail --silent --show-error --max-time 30 http://localhost:3410/api/info > "$OUT/info-after.json"
+    node "$OUT/runtime-snapshot.mjs" capture "$KPI/ainize/home-docker" "$OUT/jobs-after.json" "$OUT/info-after.json" "$OUT/inventory-after.json" "$TRAINER_ROOT"
+    node "$OUT/runtime-snapshot.mjs" verify "$OUT/inventory-before.json" "$OUT/inventory-after.json" > "$OUT/preservation.json"
+    for container in flashnext flashtrain; do
+      docker inspect "$container" --format '{{.Id}} {{.State.StartedAt}} {{.State.Pid}}' > "$OUT/$container-after.txt"
+      cmp "$OUT/$container-before.txt" "$OUT/$container-after.txt"
+    done
+    echo "Ainize ready; private backup preserved at $BACKUP"
+    exit 0
+  fi
+  sleep 2
+done
+echo 'readiness observation expired; inspect the same container, do not restart blindly'
+exit 1

deploy/runtime-snapshot.mjs

@@ -0,0 +1,122 @@
+import assert from 'node:assert/strict';
+import { createHash } from 'node:crypto';
+import { createReadStream } from 'node:fs';
+import { lstat, readdir, readFile, realpath, stat as fileStat, writeFile } from 'node:fs/promises';
+import path from 'node:path';
+import { pathToFileURL } from 'node:url';
+
+export function requireIdle(info, jobs) {
+  const terminal = new Set(['READY', 'FAILED', 'CANCELLED', 'NEEDS_MORE', 'REJECTED', 'ANNOUNCED', 'PENDING_REVIEW']);
+  assert.ok(Array.isArray(jobs.items) && jobs.items.length < 500, 'job list is missing or may be truncated');
+  assert.ok(jobs.items.every(job => terminal.has(job.status)), 'unfinished teach job; do not restart');
+  const runtime = info.runtime;
+  assert.equal(runtime?.available, true, 'runtime is unavailable');
+  assert.deepEqual(runtime.applied, [], 'runtime has an applied patch');
+  assert.equal(runtime.queue?.running, null);
+  assert.equal(runtime.queue.waiting, 0);
+  assert.equal(runtime.queue.lock, null);
+  assert.deepEqual(runtime.queue.queued, []);
+}
+
+export function jobBindings(jobs) {
+  return jobs.items.map(job => ({ id: job.id, status: job.status, dataset: job.dataset, mode: job.mode,
+    context_patch_ids: job.context_patch_ids, draft_id: job.draft_id, result: job.result, checks: job.checks,
+  })).sort((left, right) => left.id.localeCompare(right.id));
+}
+
+export async function operatorJobs(home, request = fetch) {
+  const credentials = path.join(home, 'cli.json');
+  assert.equal((await lstat(credentials)).mode & 0o077, 0, 'operator credential file must be private');
+  const state = JSON.parse(await readFile(credentials, 'utf8'));
+  const origin = new URL(state.nodeUrl);
+  assert.ok(origin.protocol === 'https:' || origin.protocol === 'http:' && ['localhost', '127.0.0.1', '[::1]'].includes(origin.hostname));
+  assert.ok(!origin.username && !origin.password && origin.pathname === '/' && !origin.search && !origin.hash, 'origin-only operator URL required');
+  assert.ok(typeof state.token === 'string' && state.token, 'operator login required');
+  const response = await request(new URL('/api/me/teach/jobs', origin), { headers: { authorization: `Bearer ${state.token}` }, redirect: 'error', signal: AbortSignal.timeout(30_000) });
+  assert.ok(response.ok && response.headers.get('content-type')?.includes('application/json'), `operator jobs returned HTTP ${response.status}`);
+  const chunks = [];
+  let size = 0;
+  for await (const chunk of response.body) {
+    size += chunk.length;
+    assert.ok(size <= 8 * 1024 ** 2, 'operator job list is too large');
+    chunks.push(chunk);
+  }
+  const jobs = JSON.parse(Buffer.concat(chunks).toString('utf8'));
+  assert.ok(Array.isArray(jobs.items) && jobs.items.length < 500, 'operator job list may be truncated');
+  return jobs;
+}
+
+export async function inventory(home, trainerRoot) {
+  const files = {};
+  const allowed = trainerRoot ? await realpath(trainerRoot) : null;
+  async function visit(relative, base = home, prefix = '') {
+    let filename = path.join(base, relative);
+    let stat = await lstat(filename);
+    let target;
+    if (stat.isSymbolicLink()) {
+      assert.ok(allowed && base === home && /^[a-f0-9]{64}\.npz$/.test(path.basename(relative)), `refusing symlink in evidence inventory: ${relative}`);
+      filename = await realpath(filename);
+      assert.ok(filename.startsWith(allowed + path.sep), `symlink escapes the explicit trainer root: ${relative}`);
+      target = path.relative(allowed, filename);
+      stat = await fileStat(filename);
+      assert.ok(stat.isFile(), 'linked body must be a regular file');
+    }
+    if (stat.isDirectory()) {
+      for (const name of (await readdir(filename)).sort()) await visit(path.join(relative, name), base, prefix);
+    } else {
+      assert.ok(stat.isFile(), `unexpected file type: ${relative}`);
+      const hash = createHash('sha256');
+      for await (const chunk of createReadStream(filename)) hash.update(chunk);
+      const sha256 = hash.digest('hex');
+      if (target) assert.equal(path.basename(relative, '.npz'), sha256, 'linked body does not match its content-addressed name');
+      files[prefix + relative] = { bytes: stat.size, sha256, ...(target ? { trainerTarget: target } : {}) };
+    }
+  }
+  for (const directory of ['data/teach/datasets', 'data/drive/patches']) await visit(directory);
+  if (allowed) await visit('', allowed, 'trainer/');
+  assert.ok(Object.keys(files).length > 0, 'empty data inventory');
+  return files;
+}
+
+export function compareSnapshots(before, after) {
+  assert.deepEqual(after.jobs, before.jobs, 'job IDs, datasets, trained bodies or checks changed');
+  for (const [filename, metadata] of Object.entries(before.files)) {
+    assert.deepEqual(after.files[filename], metadata, `existing dataset or trained body changed: ${filename}`);
+  }
+  assert.deepEqual(after.runtime, before.runtime, 'model runtime identity changed');
+  assert.equal(after.nodeAddress, before.nodeAddress, 'publisher identity changed');
+}
+
+async function main() {
+  const [command, ...args] = process.argv.slice(2);
+  if (command === 'jobs') {
+    const [home, destination] = args;
+    assert.ok(destination, 'jobs <home> <destination.json>');
+    await writeFile(destination, JSON.stringify(await operatorJobs(home), null, 2) + '\n', { flag: 'wx', mode: 0o600 });
+  } else if (command === 'capture') {
+    const [home, jobsPath, infoPath, destination, trainerRoot] = args;
+    assert.ok(destination, 'capture <home> <jobs.json> <info.json> <destination.json> [trainer-root]');
+    const jobs = JSON.parse(await readFile(jobsPath, 'utf8'));
+    const info = JSON.parse(await readFile(infoPath, 'utf8'));
+    requireIdle(info, jobs);
+    const runtime = Object.fromEntries(['api', 'model', 'hook', 'repo', 'patch_dir'].map(key => [key, info.runtime[key]]));
+    const files = await inventory(home, trainerRoot);
+    if (trainerRoot) {
+      for (const job of jobs.items.filter(item => item.result?.sha256 && item.checks?.executed)) {
+        const body = files[`trainer/${job.id}/lesson.npz`];
+        assert.equal(body?.sha256, job.result.sha256, `checked job body changed: ${job.id}`);
+        assert.equal(body.bytes, job.result.size_bytes, `checked job size changed: ${job.id}`);
+      }
+    }
+    const snapshot = { at: new Date().toISOString(), nodeAddress: info.node.address, runtime, jobs: jobBindings(jobs), files };
+    await writeFile(destination, JSON.stringify(snapshot, null, 2) + '\n', { flag: 'wx' });
+  } else if (command === 'verify') {
+    const [beforePath, afterPath] = args;
+    const before = JSON.parse(await readFile(beforePath, 'utf8'));
+    const after = JSON.parse(await readFile(afterPath, 'utf8'));
+    compareSnapshots(before, after);
+    console.log(JSON.stringify({ verified: true, jobs: before.jobs.length, preservedFiles: Object.keys(before.files).length }));
+  } else throw new Error('expected capture or verify');
+}
+
+if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) main().catch(error => { console.error(error.message); process.exitCode = 1; });

deploy/source-refresh.Dockerfile

@@ -0,0 +1,15 @@
+ARG BASE_IMAGE
+FROM ${BASE_IMAGE}
+USER root
+RUN rm -rf /opt/ainize/ainize-core/src /opt/ainize/ainize-core/dist /opt/ainize/ainize-node/src /opt/ainize/ainize-node/dist /opt/ainize/ainize-cli/src /opt/ainize/ainize-cli/dist
+COPY core/src /opt/ainize/ainize-core/src
+COPY core/package.json core/tsconfig*.json /opt/ainize/ainize-core/
+COPY node/src /opt/ainize/ainize-node/src
+COPY node/package.json node/tsconfig*.json /opt/ainize/ainize-node/
+COPY node/trainer /opt/ainize/ainize-node/trainer
+COPY cli/src /opt/ainize/ainize-cli/src
+COPY cli/package.json cli/tsconfig*.json /opt/ainize/ainize-cli/
+RUN cd /opt/ainize/ainize-core && npm run build && cd /opt/ainize/ainize-node && npm run build && cd /opt/ainize/ainize-cli && npm run build
+WORKDIR /opt/ainize/ainize-node
+ENTRYPOINT ["node"]
+CMD ["/opt/ainize/ainize-node/dist/bin.js"]

deploy/source-refresh.md

@@ -0,0 +1,30 @@
+# Certification runtime refresh
+
+These helpers are for the existing `/mnt/newdata/gov/kpi` deployment, not a clean-machine installer or a public-server deployment. They do not restart model containers or publish container images.
+
+`build-source-refresh.sh` freezes compatible core/node/CLI source and rebuilds all three against the immutable ID of an existing dependency/Python runtime image. It saves source hashes, Git provenance, build output and image ID. CPU quota2, cpuset0–7 and RAM/swap4GiB bound the build. Changing dependencies requires rebuilding the dependency image first; this helper does not run an unlocked `npm install`.
+
+```bash
+AINIZE_BASE_IMAGE=sha256:563e96f6725136939bd6bac5cf6a6480fcc535f71fb51c0ab02ab19d7d8891d7 \
+  bash deploy/build-source-refresh.sh /path/to/ainize-core /path/to/ainize-cli \
+  /path/to/new-evidence ain-cert-ainize:relay-runtime-20260911
+```
+
+The verified build combines core `695a8ad6` (0.1.3), node runtime `257bfe85` (0.1.2 plus hardening/watchdog fix), and CLI `9acc9de` (0.1.1 including HF URL import and VERIFIED compatibility). Keeping the older CLI source caused a real `LISTED`/`VERIFIED` TypeScript error; that failed build is preserved. Rebuilding with this helper returned the exact same final image ID `sha256:da23af1a5c75cfee61bdad8403f39cad31b6bd2860e7af590a6ffbf03f31c4b0`. Dependency checks and all three package builds pass.
+
+## Safe API-only replacement
+
+Install `cert-docker/upgrade-ainize.sh` and `cert-docker/compose.ainize.json` verbatim in the deployment's `kpi/docker/`. The existing clone at `kpi/pr/an-relay` supplies `deploy/runtime-snapshot.mjs`. The compose default is the verified local image; `AINIZE_IMAGE` can select an explicit available image. This is not permission to replace another deployment's configuration.
+
+Coordinate the lifecycle observer first. Pause it intentionally while it is only observing a known TRAINING job; do not interrupt an inference audit or cancel the actual job. Wait for the server's real terminal job state and an idle, empty model stack. The upgrade script refuses active observers and nonterminal jobs. A preflight check is not a distributed lock against another operator, so the runtime must be reserved for this maintenance.
+
+```bash
+cd /mnt/newdata/gov
+RUN_ID=ainize_runtime_upgrade_20260911 bash kpi/docker/upgrade-ainize.sh
+```
+
+The script snapshots the job IDs, dataset bindings, patch hashes/checks, publisher identity, data files and trainer files. Published drive entries can be symlinks into `.teach/<job>/lesson.npz`: only SHA-named links under the explicit trainer root are followed and their bytes must match the name. Other links are refused. All checked job bodies must match their recorded SHA and size. Neither a symlink nor a public node's empty blob list proves the original body is gone.
+
+Only the Ainize API container stops. Its private home and the linked trainer tree are backed up separately under `kpi/secrets/<RUN_ID>` (0700 directory/0600 archives). Backups are never release assets. After startup, actual JSON readiness, exact target image, every existing data/body hash, unchanged job binding and publisher/model identity are checked. `flashnext` and `flashtrain` container IDs/start times/PIDs must remain identical. An expired readiness observation is a failure to inspect, not a reason for blind restart. Resume the original lifecycle RUN_ID only after preservation checks pass; do not replace its frozen source or submit duplicate jobs.
+
+Offline validation: `ainize_runtime_maintenance_final_tests_20260911` passed55 tests (30 relay/client/watchdog/snapshot +25 cluster/chat guards). Additional operator enumeration tests are recorded separately. The new snapshot tests include linked-body handling and refusal of foreign/mutated state. Enumeration uses the operator-only `/api/me/teach/jobs`, not the signature-only visitor endpoint or a teaching key's subset. An actual CHECKING job caused the guard to refuse the restart, leaving the same API instance running (`ainize_runtime_preflight_operator_reject_20260911`). An earlier `docker top` call missing its mandatory PID field is retained as a wrapper failure, not a successful guard test. Test fixtures and a successful local rebuild do not establish public-server deployment, P2P delivery, Live answer quality or all100 dataset evaluations.