BIP Draft: Segregated Data - #2222
Conversation
There was a problem hiding this comment.
Hi @MrHash, thank you for your submission.
-
Looking through the discussion at https://delvingbitcoin.org/t/bip-draft-segregated-data-a-prunable-script-isolated-block-region-for-data-carriage/2641/40, the feedback was extensive and entirely negative.
-
The first line of the README in this repository states: "People wishing to submit a BIP should first describe their idea to the bitcoindev@googlegroups.com mailing list to gather feedback on viability and community interest before working on a formal description." BIP3 also describes this in more detail. Please see the README and BIP 3 for the process.
-
In your PR description, you write, "Attempted to post for discussion on mailing list without success." When did you send your post? Sometimes it can take a fortnight or more for the mods to get through the queue and patience may be needed.
In summary, with negative community feedback in Delving, and with no discussion on the mailing list, this PR was opened prematurely, and it ought to see positive conceptual feedback from the community before opening a PR here. Let's close this for now.
|
@jonatack I attempted to start a discussion at the mailing list over 1 month ago but my submission was denied and i don't know why. I reached out to @RubenSomsen directly on socials to find out why but unfortunately didn't get a response. Instead I created a conversation on Delving which seemed to be the only option at that point. There were several misunderstandings which were not followed on after I clarified. In the end, the useful technical feedback was fully taken on board and the BIPs have been updated in the last weeks into their current form which resolve the technical issues that were raised. Finally @murchandamus suggested I make the PR which I am now doing. Negative feedback seems to have been based on either opinion on economic principle or just plain misunderstanding, partly due to imprecision in explaination by me. I also had some brief conversations with @adam3us on X where he gave some technical suggestions and feedback which I have incoporated. I've tried to follow the procedure but i've faced obstacles, and again here. I'd appreciate if you would re-open the PR for consideration properly, otherwise clear the barriers for me to post to the mailing list. In any case, negative opinion is not grounds alone for closing a technical proposal like this. thanks |
|
@MrHash the feedback on Delving was universally negative, and your reviewers there don't have unlimited time to iterate on each of your responses. I'll ask another mod about the mailing list submission. |
|
Murch suggested that I make the PR so that's what I did, because I had no idea if trying to send to the mailing list would be blocked again. I'm not trying to bother devs at all. The negativity on Delving revolved around a strange misconception that this allowed data carriage for free, which is 100% wrong. As I said, technical feedback was appreciated and the proposal fixed. Meanwhile i've been having entirely positive feedback from Adam Back on X. |
|
I took the time just now to reach out to the ML mods. One confirmed that your submission was not approved, more or less for not making sense and wasting the community's time. Another mod replied that the discussion is already ongoing on delving, so it's better to continue the conversation there.
I would suggest continuing the existing discussion on Delving, with a link to the updated/fixed version on your GitHub. That said, none of the feedback I am seeing from any of the people involved is encouraging. Good luck. |
|
From what it sounds like you are saying that if i try to submit a second time to the mailing list with the update proposal in order to reach a broader audience that it will be censored again? |
|
@MrHash The mailing list tries for a balance in order to keep it useful and readable for experienced reviewers who have limited time. That review is the bottleneck, and you are asking for it repeatedly despite negative technical feedback from all of them so far. This has been a recurring problematic pattern in submission to the list and here. I've already suggested a way forward in my previous post. |
|
@jonatack To be clear, i am not trying to waste the community's time. I'm trying to present a valid technical proposal which has been misunderstood at its surface. I have been instructed to make a PR and I attach the proof of proposed next steps. 
I also attach a screenshot of Dr Back promoting the proposal after review: 
There is more detailed proposal discussion here: https://x.com/adam3us/status/2075619857714856354 and https://x.com/adam3us/status/2071164137569145207 Given that clearly you can see there is significant and useful positive feedback, I would kindly ask that you stop treating this or my following of the procedure as a waste of community time, but as a serious albeit unusual and unfamiliar proposal from outside the existing Core team. |
|
@MrHash please read my previous comments, as I do not intend to repeat them, and kindly stop wasting our time. Posts on X have no bearing here. If Murch wishes to unlock and re-open this and review it again here, he is free to do so. |
|
FWIW, this updated proposal does not appear to make any more sense than the previous one that saw only negative technical feedback on Delving, and I see why the mail list mods did not accept it. The newer proposal would make data optional at every depth and keep consensus validation identical, fixing the reorg risk AJ Towns described. Nodes would only check a commitment plus the declared size against the weight limit; everything else would be policy. It would be a soft fork for something that doesn’t need consensus rules. One can already commit to arbitrary data hashes today. The data still consumes block weight even though no one is required to keep or serve it. Most nodes would ignore it, making availability unreliable. Use cases that care about permanence would stick to normal witness or OP_RETURN data. It would be adding protocol complexity for a feature most nodes will turn off and that has weak incentives to be used at all. Some of the best reviewers in the space were in that Delving discussion and I believe you received generous feedback there that is as expert as you'll see and need. |
|
Thanks @jonatack for your feedback here so far. I did indeed promise that I would take another look at the submission here, so I’m reopening this. |
|
@murchandamus Sure. My take so far is that @MrHash has been sea-lioning and handwaving away the consistently (polite but) negative feedback on Delving Bitcoin. (I made a poll here out of interest, and there are some comments below it, though it's of course not of any more value than the X screenshot above.) |
murchandamus
left a comment
murchandamus
left a comment
There was a problem hiding this comment.
This proposal is overly verbose. It could be improved by focusing on describing the proposed feature and foregoing speculation about adoption and implementation details.
The document is missing a Rationale section, and in several instances content that should appear in the Specification appears in the Glossary, Motivation, and Key Design Decisions, whereas the Specification contains unnecessary speculative content.
This proposal essentially describes an optional network service that enables participating nodes to retain and relay optional data payloads associated with transactions. Participating nodes only accept data payloads for which the sender has paid a fee corresponding to the data payload’s length. I surmise that this proposal may garner more support if it endorsed such a framing. The seeming expectation that nodes would broadly participate in this scheme seems naive and undermines the prospect of this proposal. I remain unconvinced that users participating in data embedding schemes would be convinced to adopt this scheme to get a small fee reduction in exchange for a large data availability hit compared to simply continuing use of the already established schemes.
|
|
||
| ## Abstract | ||
|
|
||
| This BIP defines how **Segregated Data (SegData)** *entries* propagate between *peers*. The consensus BIP places entries in a block-level `segdata` *region* committed via the coinbase and validates every block from its base serialisation alone, so no node needs the region to validate, at the tip or at any depth. Entry presence, integrity, and retention are therefore relay policy. This BIP specifies how entries travel bundled with their referencing transactions, how a block's region is reconstructed from held entries, how nodes serve recent entries and advertise deeper retention, and how a node opts out in part or in full. |
There was a problem hiding this comment.
Relay policy strikes me as the wrong term here. I would perhaps describe it as an optional node service.
|
|
||
| **Entry-bearing relay**: Relay that carries SegData entries, bundled with their transactions and regions, as opposed to entry-stripped relay, which carries only the standard base serialisation. | ||
|
|
||
| **Recent-block floor**: The most recent 288 blocks, within which a SegData peer MUST retain and serve every **available** entry. |
There was a problem hiding this comment.
This Glossary section seems to contain parts of the Specification.
|
|
||
| **Recent-block floor**: The most recent 288 blocks, within which a SegData peer MUST retain and serve every **available** entry. | ||
|
|
||
| **Retention policy**: A node's local choice of how much history to keep and which entries to exclude, above the mandatory recent-block floor. It determines what the node can serve, and its depth is what the coverage tier advertises. The amount and exclusion controls are defined in the consensus BIP (§Prunability, §Reference Implementation), and this BIP standardises only the coverage signal, not their format. |
There was a problem hiding this comment.
(§Prunability, §Reference Implementation)
I don’t see a section titled “Prunability”. Please use actual links to link to referenced sections. If you wish leave yourself reminders for outstanding todos, you could e.g., leave a “(TK)” in the corresponding spots.
|
|
||
| To enable prunability of SegData entries, a SegData *reference* output carries only a 4-byte marker and 32-byte hash of the entry. The transaction alone does not carry the bytes a block builder needs to populate `segdata`. Consensus never requires the region, but the network must move entries from originator to miners, and on to any node that opts to receive them. | ||
|
|
||
| SegData entries are deliberately outside transaction identity (see D2 below), which enables deduplication and per-entry retention but means the bytes do not travel with the transaction by construction. |
There was a problem hiding this comment.
Please link to referenced sections.
| To enable prunability of SegData entries, a SegData *reference* output carries only a 4-byte marker and 32-byte hash of the entry. The transaction alone does not carry the bytes a block builder needs to populate `segdata`. Consensus never requires the region, but the network must move entries from originator to miners, and on to any node that opts to receive them. | ||
|
|
||
| SegData entries are deliberately outside transaction identity (see D2 below), which enables deduplication and per-entry retention but means the bytes do not travel with the transaction by construction. | ||
|
|
There was a problem hiding this comment.
This section is describing the approach, however the motivation section should explain why a BIP is being written, how the existing situation presents a problem, and why the proposed idea resolves the issue or improves upon the current situation.
|
|
||
| SegData entries are deliberately outside transaction identity (see D2 below), which enables deduplication and per-entry retention but means the bytes do not travel with the transaction by construction. | ||
|
|
||
| ## Key design decisions |
There was a problem hiding this comment.
At first glance this section seems to contain a mix of content that should either appear in the Specification or in the Rationale sections.
| The default policy is to receive and relay the region, so that data stays available and the node can serve and retain it. A node MAY decline mempool entry relay while still relaying ordinary transactions, MAY take new blocks base-only rather than extended, and MAY skip or prune historical entries per its retention policy, whose depth its coverage tier then advertises (D8). A node that receives no entries anywhere still tracks consensus exactly, differing only in what it can serve. Entry-stripped relay is the rule for pre-activation and non-negotiated peers, and the explicit choice for any peer. | ||
|
|
||
| The full opt-out is a single participation switch (for concreteness, a Bitcoin Core `permitsegdata=0`). In this case the node advertises neither service bit, negotiates no entry relay, takes base-only blocks, and retains nothing, so it holds and serves no entries while validating fully. This is independent of `blocksonly`, which governs mempool transaction relay in general. A node running `blocksonly` with SegData still permitted continues to take and serve block regions. |
There was a problem hiding this comment.
Here and generally, given that this BIP does not describe a consensus change, this proposal would overall be more approachable, if it took care to frame prescribed behavior as applying to software projects implementing the proposal rather than sounding like it were making claims about node behavior in general.
| - **`NODE_SEGDATA`** = `(1 << 12)`: the peer relays SegData entries and serves the full `segdata` archive since activation. The guarantee is one of depth, the full range since activation, not of every individual entry. Per-entry exclusions are permitted and answered with `notfound`, and only systematic failure to serve within range forfeits the claim (§Retention-coverage advertising). A node not covering the full range MUST NOT set the bit, the BIP-159 rule for `NODE_NETWORK`. Service bits travel in `addr` relay, so syncing nodes discover archival coverage before connecting. | ||
| - **`NODE_SEGDATA_LIMITED`** = `(1 << 13)`: the peer relays SegData entries and guarantees serving `segdata` for the most recent 288 blocks (D7). The direct analogue of `NODE_NETWORK_LIMITED`, with the same 288-block window and the same reorg-serving rationale. Nodes setting `NODE_SEGDATA` SHOULD also set `NODE_SEGDATA_LIMITED`, as archival nodes signal both BIP-159 bits today. | ||
|
|
||
| Both bits imply entry-relay capability. By default a node retains SegData for the blocks it stores, so a default-configured unpruned node is archival and sets both bits. A node retaining less sets only `NODE_SEGDATA_LIMITED` and advertises any intermediate coverage (§Retention-coverage advertising). If a connection's advertised coverage states a tier lower than the bits imply, that coverage governs for the connection. Bits are discovery hints in `addr` gossip and may be stale, and the advertised coverage is the per-connection commitment. |
There was a problem hiding this comment.
Default behavior of nodes regarding optional features is an implementation decision. I would suggest describing the services without making claims about implementation details.
|
|
||
| Both bits imply entry-relay capability. By default a node retains SegData for the blocks it stores, so a default-configured unpruned node is archival and sets both bits. A node retaining less sets only `NODE_SEGDATA_LIMITED` and advertises any intermediate coverage (§Retention-coverage advertising). If a connection's advertised coverage states a tier lower than the bits imply, that coverage governs for the connection. Bits are discovery hints in `addr` gossip and may be stale, and the advertised coverage is the per-connection commitment. | ||
|
|
||
| - **`sendsegdata`**: an empty negotiation message sent between `version` and `verack`, following the [BIP-339](bip-0339.mediawiki) (`wtxidrelay`) pattern. A node sends it only to SegData peers (either bit) at a protocol version of 70017 or higher (proposed, pending assignment). Entry-bearing relay is active on a connection only when both sides have sent it. Otherwise the connection carries entry-stripped relay (D3). Sending `sendsegdata` after `verack` is a protocol violation. |
There was a problem hiding this comment.
This approach has meanwhile been specified as BIP434: Peer Feature Negotiation.
|
|
||
| In the mempool a node MAY hold a shared entry once or once per referencing transaction, the latter matching the full-attribution feerate below. Maximum entry size is relay policy. A node rejects a zero-length entry as nonstandard, since an empty entry carries no data yet still costs a reference output. This is relay policy rather than consensus because per-entry length lives in the region, not the base serialisation, so an empty entry a miner includes anyway is harmless and simply not relayed onward in the mempool. | ||
|
|
||
| For feerate, a transaction's weight includes the full `segdata` weight of every entry it references, even one shared with another mempool transaction. This over-estimates a shared entry (consensus counts it once per block) but never under-estimates, so feerate sorting, eviction, and RBF stay well-defined and conservative. `minrelaytxfee` applies to that full weight including the entry. Standardness checks the same attributed weight, so a reference-bearing transaction and the entries it pulls in are bounded by `MAX_STANDARD_TX_WEIGHT` like any other standard transaction. Block builders MAY refine with package-style accounting at template construction. |
There was a problem hiding this comment.
It would be preferable to skip over prescribing all these implementation details by just saying that the segdata counts toward the transaction’s weight as one weight per byte which would imply the described behavior in this section.
Thanks Jon, I wasn't aware that defending my proposal by actually listing the incentives was sea-lioning, i thought i was just following the process as a first time contributor, but i apologize if i'm making mistakes. The main criticism seems to be regarding adoption "nobody would store data" which they already do, given a choice, which is not really based on any evidence i have seen. The central point about contributing to the decentralization of the network has been overlooked without discussion. The results of your poll are interesting even in the plainest framing, which suggest this may not be a waste of time after all. 
|
|
@murchandamus i will review all your comments and try to adapt the proposal as requested. thank you for your time as always. |
murchandamus
left a comment
murchandamus
left a comment
There was a problem hiding this comment.
I read the second document. Please see comments in-line.
|
|
||
| ## Abstract | ||
|
|
||
| This BIP defines a new soft-fork-activated block *region*, **Segregated Data (SegData)**, dedicated to carrying script-isolated data *entries* outside the transaction and witness structures. A coinbase output commits to a Merkle root of the entry hashes and the region's total byte length. Consensus checks only that the *commitment* matches the *references* the transactions declare, and that the block's weight, including the SegData region, is within the limit. No node needs the region itself to validate, so its presence, integrity, and availability are relay policy, and an operator may decline or discard it in part or in full at any depth while the commitment stays permanent, placing the data burden on those who choose to retain it rather than on all. |
There was a problem hiding this comment.
Still find the characterization as “relay policy” a misnomer. How about:
| This BIP defines a new soft-fork-activated block *region*, **Segregated Data (SegData)**, dedicated to carrying script-isolated data *entries* outside the transaction and witness structures. A coinbase output commits to a Merkle root of the entry hashes and the region's total byte length. Consensus checks only that the *commitment* matches the *references* the transactions declare, and that the block's weight, including the SegData region, is within the limit. No node needs the region itself to validate, so its presence, integrity, and availability are relay policy, and an operator may decline or discard it in part or in full at any depth while the commitment stays permanent, placing the data burden on those who choose to retain it rather than on all. | |
| This BIP defines a new soft-fork-activated block *region*, **Segregated Data (SegData)**, dedicated to carrying script-isolated data *entries* outside the transaction and witness structures. A coinbase output commits to a Merkle root of the entry hashes and the region's total byte length. Consensus checks only that the *commitment* matches the *references* the transactions declare, and that the block's weight, including the SegData region, is within the limit. No node needs the region itself to validate, so its presence, integrity, and availability are optional, and an operator may decline or discard it in part or in full at any depth while the commitment stays permanent, placing the data burden on those who choose to retain it rather than on all. |
|
|
||
| ## Glossary | ||
|
|
||
| **Arbitrary data**: Bytes placed on-chain for their content rather than to authorise or define a transfer of value. |
There was a problem hiding this comment.
Terminology nit: Is it “on-chain” when it is optional?
|
|
||
| **Arbitrary data**: Bytes placed on-chain for their content rather than to authorise or define a transfer of value. | ||
|
|
||
| **Commitment**: A coinbase output which commits to the Merkle root of all entry hashes and the total length of the SegData region. |
There was a problem hiding this comment.
“Commitment” is underspecified, please use a more descriptive term. E.g., Data Region Commitment or similar.
| 1. **OP_RETURN tail bytes.** Retained by every non-pruned node at the full transaction weight. | ||
| 2. **Witness fields.** Retained by every non-pruned node at the witness discount. | ||
| 3. **Vanity P2PKH / P2SH addresses.** Bytes encoded in the address hash, occupying a UTXO-set entry on every node indefinitely. | ||
| 4. **Bare multisig outputs.** Bytes substituted for the public keys of non-P2SH multisig, occupying a UTXO-set entry on every node indefinitely, at more bytes per output than an address hash. |
There was a problem hiding this comment.
Nit: A common construction uses 1-of-3 multisig with one real public key and keys used as data carrier. Such outputs are spendable.
| 4. **Bare multisig outputs.** Bytes substituted for the public keys of non-P2SH multisig, occupying a UTXO-set entry on every node indefinitely, at more bytes per output than an address hash. | ||
| 5. **Script opcodes.** Bytes encoded as the opcode sequence of an executing script rather than as pushed data, retained by every non-pruned node and executed on validation. | ||
|
|
||
| Because it shares the same structures as monetary data, every node receives, validates, and retains arbitrary data on the same terms, so storage and IBD bandwidth scale with total data-carriage demand for all alike. |
There was a problem hiding this comment.
Avoid falling back to pronouns such as “it”, rather repeat the concrete term. (We are writing technical documentation, not prose.)
| Because it shares the same structures as monetary data, every node receives, validates, and retains arbitrary data on the same terms, so storage and IBD bandwidth scale with total data-carriage demand for all alike. | |
| Because arbitrary data shares the same structures as monetary data, every node receives, validates, and retains arbitrary data on the same terms, so storage and IBD bandwidth scale with total data-carriage demand for all alike. |
Also
so storage and IBD bandwidth scale with total data-carriage demand for all alike.
doesn’t seem right. Arbitrary data has to compete for limited blockspace, so clearly storage and IBD bandwidth do not scale with demand.
|
|
||
| Censorship would mean preventing publication, or erasing what was published. SegData does neither. What goes into a block is still chosen by miners and priced by fees, exactly as today. No consensus rule reads an entry's contents, and once a block is mined its commitment is permanent for every node. Choosing not to retain an entry does not remove it from the chain or stop it being included. | ||
|
|
||
| What selective retention affects is availability, not the record. A node keeps SegData by default, as it does witness today, and its operator may now decline to host a copy, which the protocol never required anyway, since pruned nodes already discard block data. An entry may then end up with fewer copies than witness produces, but the availability floor is unchanged: one surviving copy, held by anyone anywhere, remains distributable and verifiable against the permanent commitment. |
There was a problem hiding this comment.
A node keeps SegData by default, as it does witness today
You spend a lot of time establishing above that SegData is not consensus relevant. Default behavior of nodes regarding optional features is an implementation detail. Making claims about default behavior detracts from this document.
|
|
||
| ### SegData reference in transactions | ||
|
|
||
| A transaction MAY include one or more SegData references. A reference is encoded as a 36-byte witness-v2 program, the same 4-byte marker + 32-byte hash as encoded in the commitment: |
There was a problem hiding this comment.
Why witness-v2 instead of v1?
|
|
||
| Block weight remains derivable for software outside the node. Although the `segdata` region is absent from the base serialisation, the SegData commitment carries the region byte length, so any holder of the base bytes computes a block's full weight from a proof-of-work-committed field without fetching the region. | ||
|
|
||
| Per-transaction weight is a new break. BIP-141 kept the witness inside the transaction, so a transaction stayed computable from its own bytes. SegData puts the entry bytes outside the transaction, so a SegData transaction's weight, and therefore its feerate, needs the entry sizes and their attribution from a node (§Reference Implementation item 4). Committing the length in each reference (§Open Questions) would remove this break for SegData-aware software. |
There was a problem hiding this comment.
I see, that answers my question from above. This of course means that anyone wanting to mine SegData transactions would need to participate in relay of SegData.
I still think that it would be more reliable if the references committed to the weight, as then even upgraded nodes that opt out of retention of SegData would be able to enforce the consensus rules. Otherwise, a miner lying about the size of the segdata could split the opt-out nodes from the opt-in nodes.
| A reference implementation of the consensus-layer rules in this BIP requires: | ||
|
|
||
| 1. Block validation changes: commitment-output detection, canonical-commitment selection, and reference-output validation (value, unspendability, and Merkle-root match). The `segdata` region itself is not parsed by consensus (§Uniform validation); region receipt and matching is a peer-services concern. | ||
| 2. Weight-accounting changes: per the extended formula in §Weight accounting. | ||
| 3. Storage layer changes: per-entry addressable `segdata` storage with a single allocation parameter, following the `-prune` idiom: the default retains everything, as unpruned block storage does, and the parameter opts into less. Storage MAY deduplicate identical entries across blocks, since each block's canonical entry list is derivable from its reference outputs and reconstructs the region from a content-addressed store exactly. | ||
| 4. RPC interface: inspection of retained entries, plus a manual `prunesegdata` call discarding a block's `segdata` or a single entry within it, refusing targets the node is committed to serve under its retention policy, as `pruneblockchain` refuses heights inside the pruned-node minimum. Selective retention policy thereby lives in external tooling driving this RPC, not in the node. Block and mempool reporting exposes weight with and without the SegData term: the full consensus weight, the three-term BIP-141 weight byte-holding tooling can still derive and cross-check, and the SegData component per block and per transaction (attributed entry weight). This keeps weight statistics comparable across activation and makes each block's data-carriage share directly observable. | ||
|
|
||
| P2P propagation, service-bit advertising, request/response messages for SegData entries, and operator-policy retention mechanics belong to the companion peer-services BIP. |
There was a problem hiding this comment.
Seems out of place for a consensus BIP.
| - *Risk*: the explicit `dat0` marker makes SegData references trivially identifiable, so a miner, a relay policy, or a regulator mandating one can filter the entire class by pattern match, with no content inspection. This is cheaper than identifying data hidden in witness envelopes or key fields, so SegData marginally lowers the cost of class-level filtering. | ||
| - *Mitigation*: inclusion is the ordinary censorship-resistance problem. A fee-paying SegData transaction is profit for whoever mines it, so durable exclusion requires majority hashrate and otherwise only delays confirmation, exactly as for any transaction. Relay-layer filtering impedes propagation but is trivially routed around. The same transparency that eases filtering is what makes SegData sheddable, and declining to *retain* an entry, unlike declining to mine it, is not censorship at all. | ||
|
|
||
| The above are non-exhaustive. Community review is expected to surface additional considerations. |
There was a problem hiding this comment.
As mentioned above, lying about the size of the SegData in the commitment would be a vector to split the network which should be addressed
3 participants
Consensus and Peer Services BIPs for the Segregated Data proposal.
Following on from discussion on DelvingBitcoin. Attempted to post for discussion on mailing list without success.
I've spent plenty of time trying to get it precise, hopefully there are no glaring errors. Happy to answer any further technical questions here or on the mailing list if directed.