Skip to content

p2p: a publisher can push its body, so a node nobody can reach can still sell - #5

Open
kimminhyun-ai wants to merge 8 commits into
mainfrom
feat/blob-relay
Open

p2p: a publisher can push its body, so a node nobody can reach can still sell#5
kimminhyun-ai wants to merge 8 commits into
mainfrom
feat/blob-relay

Conversation

@kimminhyun-ai

Copy link
Copy Markdown
Contributor

Blob transfer is pull-only, so a publisher behind NAT or a firewall announces successfully and nobody can ever fetch what it announced. No error is raised: the anchor gossips, the catalogue lists it, and the status never leaves ANNOUNCED. This is how ainize.ai came to show a knowledge whose live test cannot run.

  • POST /p2p/blob/:sha — accepts a body on behalf of its author. Off unless p2p.relayBlobs, bounded by p2p.maxRelayBytes.
  • P2P.offerBlob() — best-effort push to peers, never throws.
  • Market.offerBody() — called after announce; warns plainly when no peer took it.

Accepting is safe without trusting the caller: the sha must name an anchor already known from the gossiped ledger (not open storage), the uploader must sign as that anchor's author, and importFile rehashes and refuses a mismatch.

Discovery and fetch are unchanged — holders() already works off advertised PeerInfo.blobs.

Depends on ainize-core#3 (@ainize/core 0.1.2). Build clean, 223 tests pass.

nvidia and others added 2 commits September 11, 2026 08:32
…ill sell

Every blob transfer here is a pull — the fetcher goes to the holder. That is
right for a CONSUMER behind a firewall and wrong for a PUBLISHER behind one: the
verifier has to reach in, cannot, and the anchor sits at ANNOUNCED for ever.
Nothing errors anywhere. The anchor gossips, the catalogue lists it, and the
body is simply unobtainable — which is how ainize.ai came to show a knowledge
whose live test cannot run, because the seller node that held it no longer runs.

Three parts:

  POST /p2p/blob/:sha   accepts a body on behalf of its author. Off unless
                        `p2p.relayBlobs`, bounded by `p2p.maxRelayBytes`.
  P2P.offerBlob()       pushes to peers, best-effort, never throws.
  Market.offerBody()    called after announce; logs who took it, and warns
                        plainly when nobody did.

Accepting is not a matter of trusting the caller. The sha must name an anchor
this node already knows from the gossiped ledger, so it is not open storage; the
uploader must sign as that anchor's author, so only the publisher can place its
own bytes; and importFile rehashes and refuses a mismatch — the signed anchor
already fixes the hash, this checks the bytes against it.

Discovery and fetch needed no change: holders() already works off the blobs a
peer advertises in PeerInfo, so a relay shows up as a holder and fetchBlob finds
it unmodified.

Needs @ainize/core 0.1.2 for p2p.relayBlobs / p2p.maxRelayBytes.
@kimminhyun-ai

Copy link
Copy Markdown
Contributor Author

Native outbound P2P relay hardening

Source: d156d36253e7ec877ff21680ba0f44e31fde3771, existing node PR #5 / feat/blob-relay.
Companion core source used for validation: 85cbef76da935b6ad8b9b1daf3fe6bc14701ea5a (core PR #3).

This is a source prerelease and validation evidence, not an npm publication,
public-server deployment, verified marketplace status, or a successful public Live test.

  • Admission/authentication before file parsing; positive cumulative storage budget and concurrent reservations.
  • Exact signed size/hash/dimensions, bounded NPZ inflation, corruption checks, cleanup on failure/disconnection.
  • Streamed outbound offers, checked JSON receipts, bounded responses, no redirect forwarding, configured peers only.
  • Operator-only POST /api/patches/:id/relay retries already-announced bodies without retraining or another anchor.
  • Wire-compatible with the original POST /p2p/blob/:sha feature; no Funnel or public seller URL prerequisite.

Validation: 39/39 Docker tests pass (14 relay tests + 25 existing cluster/chat-guard regressions).
Both source trees compile. Docker CPU quota 2, CPU set 0–7, RAM/swap ceiling 4 GiB,
network none, read-only root/source, executable 2 GiB tmpfs, no GPU.
Test finish: 2026-09-11T09:01:04.148513822Z. Image digest:
sha256:563e96f6725136939bd6bac5cf6a6480fcc535f71fb51c0ab02ab19d7d8891d7.
Attached source checksums identify the immutable test inputs; the commit contains those same source files.

Failures retained locally: r1 executable-tmpfs setup, r2 incomplete test verifier config,
r3 multipart part-limit regression (7 pass / 5 fail), then r4 37/37 and final r5 39/39.
Synthetic test fixtures are not DART100 performance evidence.

Deployment caveat: at 08:58:53 UTC both https://ainize.ai and https://www.ainize.ai
still answered 404 Cannot POST for the new route. The actual public endpoint must
deploy/forward this route and enable p2p.relayBlobs with a positive p2p.maxRelayBytes.
Only then can the existing two DART bodies be pushed and the real public Live test repeated.
Core npm 0.1.2 alone is insufficient: current main's 0.1.2 and the feature branch use
the same version string but different config fields. Merge/build core PR #3 as well;
do not silently roll back the main branch's VERIFIED status rename.

No verification quorum, payment entitlement, dataset visibility or PII checks are waived.

@kimminhyun-ai

Copy link
Copy Markdown
Contributor Author

Preserve and recover existing P2P knowledge bodies

Source commit: a4753e0142f27d5c2ff0412d6a3d81ea3ef3dccd, pushed to the existing feat/blob-relay / node PR #5.
Core test source: 85cbef76da935b6ad8b9b1daf3fe6bc14701ea5a (core PR #3).

This is a source prerelease with diagnostic evidence, not npm publication,
public-server deployment, or a successful public Live test.

Direct evidence, not a missing-file assumption

The original publisher ain-cert-ainize-node-1 is running and healthy on the
experiment machine (PID 3616107, started 2026-09-11T07:39:04.930747533Z).
Its /p2p/blobs had six bodies at 09:10 UTC. Both already-published DART bodies
exist there and match the public author-signed anchors exactly:

  • f9f665f6fa1a6b37963a4845107c0c0a5d3b970bcd2af6e8f40938a0fbdf7acc: 3,679,278 bytes, 2,856 rows × 160.
  • fb1cd41e2f6a26f785d72460a2eac4a62688ee4c70e5bee43187d734eeca2e64: 3,719,206 bytes, 2,887 rows × 160.

At 09:11:52 and 09:12:53 UTC, actual author-signed multipart body offers to www
and apex respectively both returned HTML 404 Cannot POST, not JSON success.
The attached signed-offer-status.json includes exact endpoints and responses.
An empty remote inventory does not mean the original files were deleted. Remote
localhost:3410 also cannot identify a process on another machine. No retraining
or additional anchor/dataset publication was performed.

Changes

  • Standalone scripts/retry-public-blob.mjs offers an existing free public body
    while the old author node/trainer keeps running. It verifies the anchor signature,
    author/hash/size and bounded receipts, then checks a full authenticated read-back.
    It never logs identity secrets or auth headers, or reports HTML 200/404 as success.
  • A persistent blob retention flag prevents verifier cleanup and GC from deleting
    an accepted relay body immediately after attestation. Retention survives reopening
    the database, is rechecked after asynchronous catalog access, and is cleared by
    explicit removal. Ordinary verifier copies still follow their original policy.
  • Retention is not a paid license, an attestation, or permission to apply knowledge.
  • Korean evidence/runbook: docs/blob-relay-evidence-ko.md.

Validation

49/49 tests pass: 18 relay/retention regressions, six recovery-client transport
fixtures, and 25 existing cluster/chat-guard regressions. Both source trees compile.
Docker CPU quota 2, CPU set 0–7, memory and memory+swap ceiling 4 GiB, network none,
read-only source/root, executable 2 GiB tmpfs, no GPU. Test finish:
2026-09-11T09:19:59.05798069Z. Immutable test source hashes are attached and match
the released source/tests/recovery client. The earlier 45/48-pass intermediate
runs and the initial recovery client's old-core import failure remain local evidence.

The actual signed offers used separate Docker containers with CPU quota 1,
CPU set 0–7, memory and memory+swap ceiling 1 GiB, selected config/body read-only,
and no GPU. They failed and are not counted as successful transfers or inference.

The deployment still needs node PR #5 plus the core relay configuration fields.
Core's version string 0.1.2 alone does not identify those fields. Diagnose the
internal POST http://127.0.0.1:3400/p2p/blob/<sha> response and actual deployed
commit before attributing the public 404 to a particular backend/proxy layer.

@kimminhyun-ai

Copy link
Copy Markdown
Contributor Author

Updated to257bfe853b9bbc1f08ded002a3646fd01891a3a1: incorporated main20e599a6 and core0.1.3/VERIFIED, resolved relay merge conflicts, preserved bounded authenticated imports and retention, and fixed a watchdog stack-snapshot race (3 red→green regressions;52 selected Docker tests pass). Full source and raw evidence: https://github.com/ainblockchain/ainize-node/releases/tag/p2p-relay-runtime-compat-20260911 . At10:12:59UTC both public domains still return HTML404 Cannot POST for POST /p2p/blob/:sha; /readyz is frontend HTML, not backend readiness JSON. Two original publisher bodies remain available locally and previous signed full-body POST failures are retained. Please distinguish source availability from actual backend deployment/route matching; public replication/Live is not yet claimed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants