Rework shielded transactions

[l0r1s]

Summary

Companion PR to opentensor/polkadot-sdk#6

Reworks the MEV Shield from a store-then-decrypt model to a decrypt-at-proposal model.

Previously, encrypted transactions were stored on-chain (Submissions) and decrypted by the block author in a separate step, leaving a window where decrypted transactions sat in the pool before inclusion. Now, encrypted transactions travel through the pool as opaque ciphertext and the block proposer decrypts them inline during block building, including the inner extrinsic in the same block. There is no point where a decrypted transaction is visible before it lands in a finalized block.

Encryption uses ML-KEM-768 + XChaCha20-Poly1305 with per-block ephemeral keys rotated via an inherent.

Note: E2E tests have been moved to the setup-e2e branch for ease of review.

Breaking changes for clients

This PR introduces breaking changes that require client-side updates.

Encrypted message format

The ciphertext payload now includes the next public key hash as a prefix. The new wire format is:

key_hash (16 bytes) || kem_len (2 bytes LE) || kem_ct (variable) || nonce (24 bytes) || aead_ct (variable)

• key_hash: xxhash128(NextKey) — the 16-byte hash of the ML-KEM public key used for encryption. Clients must compute this from the NextKey they encrypt against.
• kem_len: 2-byte little-endian length of the KEM ciphertext that follows.
• kem_ct: ML-KEM-768 encapsulation ciphertext (typically 1088 bytes).
• nonce: 24-byte random nonce for XChaCha20-Poly1305.
• aead_ct: The authenticated ciphertext containing the encoded inner extrinsic.

submit_encrypted API change

The old unused commitment parameter has been removed. The new signature is:

submit_encrypted(origin, ciphertext)

New transaction extension: CheckShieldedTxValidity

A new transaction extension validates encrypted messages at two levels:

• Pool validation — rejects ciphertext that cannot be parsed (malformed structure). Returns FailedShieldedTxParsing error.
• In-block validation — the block proposer verifies that key_hash matches either CurrentKey or NextKey. If it doesn't match, the transaction is dropped silently — the block proposer returns an Invalid transaction error without a custom error code because the block proposer does not propagate the inner error code of the extension.

This means clients should expect:

• A clear FailedShieldedTxParsing error if the ciphertext format is wrong.
• A generic Invalid error (no specific code) if the key_hash doesn't match any active key — e.g. when using a stale or wrong key.

Rollout plan

The upgrade will be deployed in 3 parts:

1. Hotfix — disable the current shield implementation.
2. Node upgrade — deploy the new node changes including host functions for ML-KEM-768 and XChaCha20-Poly1305 crypto.
3. Runtime upgrade — activate the new runtime with the ShieldApi runtime API used by the block proposer.

Changes

pallet-shield (reworked)

• announce_next_key inherent: rotates CurrentKey <- NextKey each block and publishes the next block author's ML-KEM public key
• submit_encrypted extrinsic: accepts an encrypted ciphertext wrapper — the block proposer decrypts and includes the inner extrinsic in the same block
• try_decode_shielded_tx / try_unshield_tx: runtime helpers called by the block proposer via the ShieldApi runtime API
• CheckShieldedTxValidity transaction extension: pool validation checks ciphertext structure; block import additionally validates key_hash against CurrentKey/NextKey
• FindAuthors trait for resolving current and next block author
• Migration to clear removed v1 storage (Submissions, KeyHashByBlock)

stp-io (new crate — host functions)

• mlkem768_decapsulate: ML-KEM-768 decapsulation via the ShieldKeystore extension
• aead_decrypt: XChaCha20-Poly1305 decryption via the ShieldKeystore extension
• Exported as SubtensorHostFunctions for registration in the node

Node integration

• ShieldKeystore created in service.rs and threaded through to the proposer and inherent providers
• ShieldInherentDataProvider added to Aura consensus configuration
• KeyRotationService generates a new ML-KEM keypair on each own-block import
• SubtensorHostFunctions registered in the client executor

Runtime integration

• ShieldApi runtime API implemented
• CheckShieldedTxValidity added to the transaction extension pipeline

Tests

• Unit: key rotation, try_decode_shielded_tx, try_unshield_tx, depth-limit protection, inherent creation, CheckShieldedTxValidity extension (key_hash matching, malformed rejection, pool vs in-block source)

E2E (branch: setup-e2e)

14 tests across 3 files covering 3-node and 6-node topologies:

Key rotation (3-node)

• NextKey and CurrentKey are populated and rotate across blocks
• AuthorKeys stores per-author keys

Encrypted transactions (3-node)

• Happy path: wrapper and inner tx included in the same block
• Failed inner tx: wrapper succeeds but inner transfer has no effect
• Malformed ciphertext rejected at pool level
• Wrong key hash not included by block proposer
• Stale key not included after rotation
• Multiple encrypted txs in the same block

Scaling (6-node)

• Network scales to 6 nodes with full peering
• Key rotation continues with more peers
• Encrypted tx works with 6 nodes
• Multiple encrypted txs in the same block with 6 nodes

Edge cases

• CurrentKey fallback: encrypted tx persists across blocks
• Valid ciphertext with invalid inner call

[open-junius]

Seems it introduces the problem that the Ink contract doesn't work. I am not sure what is issue yet. Everything is fine in my local. #2450 created to debug it.