[do not merge] V in Data

[zliu41]

This PR adds the V constructor to Data. It hasn't been decided whether or not to pursue this, but if yes, then this is how it could be done.

Backwards compatibility is not trivial and not obvious. We need to think through it and review the implementation carefully, to make sure it won't cause any split between old and new node versions.

The idea is:

• The CBOR Serialise instance for Data NO LONGER ROUNDTRIPS: encode accepts V, but decode rejects it. There's a separate decodeDataAcceptingValues function that accepts it.
  • Why not do the reverse - make decode accept V, and add a separate decodeDataLegacy that rejects V? Because it complicates ledger integration - the ledger must replace all existing decode calls with decodeDataLegacy.
  • Why not make encode reject V, and add a separate encodeDataAcceptingValues? We can do that for the sake of satisfying the roundtrip property, but it isn't otherwise needed.
• The Flat instance for Data STILL ROUNDTRIPS, and both encoding and decoding accept V. To prevent V in Plutus V1-V3, a separate check is added in PlutusLedgerApi.Common.SerialisedScript.

@lehins The ledger integration should be fairly straightforward (though I'm not certain). It basically amounts to: before Dijkstra, use decode or decodeData. After Dijkstra, use decodeDataAcceptingValues.

cc @colll78 @kwxm @SeungheonOh

Also see this comment: #7910 (comment)

[zliu41]

Most of the Value builtins are moved here, to avoid cyclic dependency between Value and Data.

[SeungheonOh]

Implementation looks reasonable, I'll give a more thorough review tomorrow.

I'm not entirely sure what the purpose of this addition is. The only benefit of having V on Data is that the fact that ledger can inject Data directly and we won't have to run valueData : Data.Map -> Value(or rather, this unValueData just becomes very cheap) but most of the time this cost doesn't seem to be that high to begin with so there's not much to gain.

Also, it would be nice if you can write out the reason why Serialise instance must be changed to no round trip. I assume it's because for decoding it must run the normalisation but I'm not entirely sure and clarification would be nice.

[Unisay]

checkConstant only guards bare con data, so a V inside a compound constant like list data or pair data data gets through: those hit the _ -> Nothing branch, their inner Data is never inspected, and the Flat decoder accepts V at any depth. So a pre-Dijkstra script with such a constant is accepted here but rejected by an older node that doesn't know tag 1401. That's an old/new split.

Confirmed at (PlutusV3, newestPV):

1. (con data (I 1))                   -> ACCEPTED   [sanity]
2. (con data (V empty))               -> REJECTED   ("The V constructor of data is not available in language PlutusV3 ...")
3. (con (list data) [I 1])            -> ACCEPTED   [control: list data is allowed here]
4. (con (list data) [V empty])        -> ACCEPTED   [the same V that (2) rejects slips through]
5. (con (pair data data) (I 1, I 2))  -> ACCEPTED   [control: pair data data is allowed here]
6. (con (pair data data) (I 1, V ..)) -> ACCEPTED   [same slip-through via pair]

The gate needs to reject V in a constant of any type containing Data, not only in con data.

[Unisay]

Suggested change

	| 1280 <= t && t < 1401 ->
	| 1280 <= t && t < valueTag ->

[zliu41]

I'm not entirely sure what the purpose of this addition is. The only benefit of having V on Data is that the fact that ledger can inject Data directly and we won't have to run valueData : Data.Map -> Value(or rather, this unValueData just becomes very cheap)

The more you do with the Value, the more the cost of unValueData is amortized. If you only perform one or two operations on the Value, then unValueData cost could very well be dominant.

[colll78]

This looks good to me. I strongly encourage adding V to Data.

[lehins]

I have some questions about this proposed change:

• are there any numbers to back up the performance gains / reduction in cost that this change would bring for DApp developers?

• Do those numbers justify a change to representation of such a critical part as Data

• The ledger integration should be fairly straightforward (though I'm not certain). It basically amounts to: before Dijkstra, use decode or decodeData. After Dijkstra, use decodeDataAcceptingValues.

  This approach would still allow older PlutuV1-V3 to use this new Data representation with V in it starting with Dijkstra era. Is this acceptable? Looking at your other comment about Flat decoding for scripts it does seem like it is.

• What about invariant on the content of Value that are enforced by Ledger. Would these decoders also provide such guarantees?

  • no zero values
  • no empty sub-Maps for a PolicyId
  • no asset names over 32 bytes

[zliu41]

are there any numbers to back up the performance gains

@Unisay You did some evaluation on the overhead of unValueData before. Can you point to the data?

This approach would still allow older PlutuV1-V3 to use this new Data representation with V in it starting with Dijkstra era.

Right, as currently implemented, V1-V3 won't be able to contain a Data constant that uses V, but it can be given a datum or redeemer that uses V.

I considered simply making V available to V1-V3 with no restriction. This would be more consistent, however, I think that would require creating 3 new semantic variants (one for V4, one for V1/2, one for V3), which is a LOT of code to maintain.

And this is just one of the reasons why this is a complex change, and must be thought through very carefully to ensure there's no unintended consequences. Hence why I said it would take time to merge and I'm leaning against doing this.

Would these decoders also provide such guarantees?

Yes

[Unisay]

Data is in #7738: standalone unValueData per shape, full matrix in the top comment. Goldens on branch yura/issue-2177-value-builtin-budget-spec; the unValueData cost model hasn't changed since, so the numbers still apply.

Summary: the conversion is 64% of the builtin lookup cost at 1 token and 98% at 101 tokens (21.7M of 22.1M CPU); lookupCoin itself is a near-constant 320-375K. Conversion costs ~192K CPU per data node, while a hand-rolled walk over the raw Data costs ~380K per visited node. So the builtin is cheaper per node but pays for all of them upfront, which is why lookups on large values can beat it by short-circuiting. With V in Data that upfront cost disappears and the builtin wins at every size and position. For unionValue there's no crossover even today: 43x CPU over unionWith at 101 tokens.