cbor: Scapy-native fields, codec hardening, and cbor2 CI - #5125
Open
polybassa wants to merge 64 commits into
Open
cbor: Scapy-native fields, codec hardening, and cbor2 CI#5125polybassa wants to merge 64 commits into
polybassa wants to merge 64 commits into
Conversation
This was referenced Sep 1, 2026
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #5125 +/- ##
==========================================
- Coverage 80.85% 80.66% -0.20%
==========================================
Files 390 390
Lines 97734 99117 +1383
==========================================
+ Hits 79021 79950 +929
- Misses 18713 19167 +454
🚀 New features to boost your workflow:
|
polybassa
force-pushed
the
split/cbor-scapy-native
branch
from
September 1, 2026 11:57
5f9b34d to
414ac55
Compare
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
It makes wide-reaching changes across CBOR core encoding/decoding and packet cache semantics, which warrants careful human validation beyond static review.
Pull request overview
This PR refactors Scapy’s CBOR support toward Scapy-native field conventions (Python-native values, ASN.1-like hooks), strengthens codec correctness (maps, floats, optionals, nesting/indefinite handling, deterministic encoding), and adds an isolated cbor2-based interoperability/differential test lane in tox and CI.
Changes:
- Rework CBOR field/container APIs to use Scapy-style
any2i/i2m/m2iplusbuild_result/dissect_result, with improved raw-cache fidelity and parent/child ownership handling. - Harden CBOR codec behavior (indefinite containers, duplicate keys, preferred float widths, deterministic map ordering, max nesting) and introduce CBOR map pair preservation (
CBORMapData). - Add pinned
cbor2==6.1.4differential UTScapy campaign plus a dedicated tox env and GitHub Actions job.
File summaries
| File | Description |
|---|---|
tox.ini |
Adds a dedicated cbor2 tox env for pinned differential tests; removes cbor2 from main deps. |
test/scapy/layers/requirements-cbor2.txt |
Documents/pins optional cbor2 dependency for interop testing. |
test/scapy/layers/generate_cbor2_corpus.py |
Adds helper script to generate reproducible CBOR corpora using cbor2. |
test/scapy/layers/cbor_cbor2_interop.uts |
Adds extensive cbor2-vs-Scapy differential/interoperability coverage. |
test/fields.uts |
Adds a regression test around ConditionalField.__getattr__. |
test/configs/windows2.utsc |
Excludes the cbor2 interop campaign from default Windows2 runs and marks keyword. |
test/configs/windows.utsc |
Excludes the cbor2 interop campaign from default Windows runs and marks keyword. |
test/configs/solaris.utsc |
Excludes the cbor2 interop campaign from default Solaris runs and marks keyword. |
test/configs/linux.utsc |
Excludes the cbor2 interop campaign from default Linux runs and marks keyword. |
test/configs/bsd.utsc |
Excludes the cbor2 interop campaign from default BSD runs and marks keyword. |
scapy/cborpacket.py |
Adds CBOR-specific packet cache integration, default-copy deepening, and parent reattachment on copy. |
scapy/cbor/cborfields.py |
Major CBOR field API rewrite (native values, optional/conditional semantics, sequences, deterministic map schema behavior). |
scapy/cbor/cborcodec.py |
Hardens low-level codec: nesting limit, indefinite support, strictness checks, float width preference, deterministic encoding helpers. |
scapy/cbor/cbor.py |
Introduces CBORMapData, richer equality/hash semantics, float encoding retention, and native wrapper classes. |
scapy/cbor/__init__.py |
Exposes new CBOR types/fields/results in the public API surface. |
README.md |
Fixes packaging status badge markdown formatting. |
.github/workflows/unittests.yml |
Adds a CI job running tox -e cbor2 on Python 3.12. |
Review details
- Files reviewed: 17/18 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
polybassa
pushed a commit
to polybassa/scapy-1
that referenced
this pull request
Sep 9, 2026
Capture map-key equivalence, deterministic floats, hash/tag field semantics, and multi-map unknown ownership before changing the implementation. Co-authored-by: Cursor <cursoragent@cursor.com>
polybassa
force-pushed
the
split/cbor-scapy-native
branch
from
September 9, 2026 05:01
414ac55 to
4a552ff
Compare
polybassa
pushed a commit
to polybassa/scapy-1
that referenced
this pull request
Sep 9, 2026
Capture map-key equivalence, deterministic floats, hash/tag field semantics, and multi-map unknown ownership before changing the implementation. Co-authored-by: Cursor <cursoragent@cursor.com> AI-Assisted: yes (Composer)
polybassa
force-pushed
the
split/cbor-scapy-native
branch
from
September 9, 2026 06:32
4a552ff to
f9e6104
Compare
Rewrite the CBOR packet/field layer toward ASN.1-style Scapy APIs, harden maps/floats/optionals, and add pinned cbor2 differential coverage behind an isolated tox/CI job. AI-Assisted: yes (Cursor) Co-authored-by: Cursor <cursoragent@cursor.com>
Rename CBORF_PACKET/SEQUENCE_OF/ARRAY_OF constructor kwargs to pkt_cls so typing.Generic.__new__ does not collide on 3.7, and drop the BPv7-only BundleEidField unit from the CBOR campaign. AI-Assisted: yes (Cursor) Co-authored-by: Cursor <cursoragent@cursor.com>
Capture map-key equivalence, deterministic floats, hash/tag field semantics, and multi-map unknown ownership before changing the implementation. Co-authored-by: Cursor <cursoragent@cursor.com> AI-Assisted: yes (Composer)
Use semantic CBOR key equivalence for duplicate detection and lookup, rebuild deterministic floats from values, drop identity hashing, and treat schema tag numbers as metadata rather than packet fields. Co-authored-by: Cursor <cursoragent@cursor.com> AI-Assisted: yes (Composer)
Arbitrary CBOR values use the lossless object tree; typed floats stay plain Python floats with wire fidelity via the packet raw cache. Co-authored-by: Cursor <cursoragent@cursor.com> AI-Assisted: yes (Composer)
Replace packet-global _cbor_unknown_map_pairs with a dedicated mutable field on each CBORF_MAP so nested and sibling maps keep independent extensions across copy and rebuild. Co-authored-by: Cursor <cursoragent@cursor.com> AI-Assisted: yes (Composer)
Unknown map state now lives in normal fields, so CBOR packets can use Scapy's default build path. Keep only parent reattachment in copy(). Co-authored-by: Cursor <cursoragent@cursor.com> AI-Assisted: yes (Composer)
Reject nonterminal unbounded SEQUENCE_OF, support count_from/max_count, and replace the 1<<30 sentinel with conf.max_list_count. Co-authored-by: Cursor <cursoragent@cursor.com> AI-Assisted: yes (Composer)
…ing packets Default deterministic scanning rejects indefinite containers, drop internal result types from the public surface, and simplify BYTE_STRING_PACKET decode. Co-authored-by: Cursor <cursoragent@cursor.com> AI-Assisted: yes (Composer)
Use CBOR_FLOAT, CBOR_SEMANTIC_TAG, CBOR_SIMPLE_VALUE, and a CBOR_UNDEFINED singleton directly in the object model and update tests accordingly. Co-authored-by: Cursor <cursoragent@cursor.com> AI-Assisted: yes (Composer)
RFC 8949 map-key equivalence for floats must not collapse every NaN. Identity now uses sign plus a width-normalized significand, preferring CBOR_FLOAT wire bytes when available, while +0.0 and -0.0 still match. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>
Share NaN component and preferred-width helpers between the deterministic validator and encoder so CBOR_FLOAT NaNs shorten without losing payload identity, including unknown-map rebuilds. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>
Remove unused CBORMapData identity helpers, document equivalence-based lookup, delete the ConditionalField getsource test, and restore the unrelated README packaging badge. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>
Decoded floats keep _encoded until mutated. Clearing it on val assignment prevents rebuilds from emitting stale float bytes. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>
Unwrap transparent wrappers before the terminal-sequence schema check so optional/conditional nesting cannot hide an ambiguous unbounded SEQUENCE_OF. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>
Use set membership on normalized map keys for duplicate detection and as_dict collision tracking, and require packet classes (not instances) for SEQUENCE_OF / ARRAY_OF pkt_cls. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>
Fingerprint CBORF_ANY values including CBOR_FLOAT._encoded so same-value .val assignment invalidates the packet raw cache without changing semantic CBOR_Object equality. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>
Reject non-Packet classes that only define CBOR_root and plain Packet types without CBOR_root in SEQUENCE_OF / ARRAY_OF pkt_cls validation. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>
Centralize initial-byte encoding so major/AI EnumElements need no call-site int casts, and use that API directly for indefinite arrays and break. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>
Enforce MAX_CBOR_NESTING in the lightweight pre-scan (nested under cbor_count_items, via memoryview), cap indefinite ARRAY counts by schema budget, and move class-owned helpers off the module surface. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>
Extract _cbor_skip_item for count and item-span paths with codec-matching chunk rules, apply the same depth/chunk checks to the non-det walker, and fold remaining single-use helpers plus map-equality duplication. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>
Keep structural cbor_item_span for framing, but validate RawVal and raw packet bytes with a full decode; nest shortest-argument checks and cover the non-det walker depth/chunk paths. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>
Make build_result/dissect_result canonical, drop dead badsequence and unused enum/flags exports, simplify map pair collection, and keep the deterministic scanner out of production. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>
Treat all fixed-map member names as reserved on encode, even when optional/conditional fields omit them, and tidy copy/optional helpers. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>
Delegate CBORF_SEMANTIC_TAG.is_empty to the inner field so optional tagged scalars with default=None omit the tag on fresh build. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>
Teach Packet.prepare_cached_fields/do_init_cached_fields to honor ismutable via Field.do_copy, and stop mutating CBOR packets from getters. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>
Extract _raw_packet_cache_is_valid from Packet.self_build so CBOR can reuse the same fingerprinting and only clear its item-count sidecar. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>
Rename result types and counted hooks to underscore names, keep public build/dissect ASN.1-like on leaves, and reserve item accounting for compounds and CBOR_Packet raw-cache fidelity. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>
Remove allows_none and redundant min/max_items, give NULL/UNDEFINED real leaf builds, skip re-decode for trusted CBOR_Packet encodes, and honor conf.debug_dissector for nested packet construction errors. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>
AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>
Align BYTE_STRING_PACKET with other nested-packet exception handling, reduce do_init_cached_fields to packet re-parenting, cover Packet ismutable defaults via FlagsField, and drop leftover is_absent / redundant ANY.build helpers. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>
polybassa
force-pushed
the
split/cbor-scapy-native
branch
from
September 10, 2026 06:30
b433d7e to
06a065b
Compare
Snapshot nested CBOR packet fields by recursive fingerprints instead of shallow dict copies, and fingerprint ARRAY_OF/ANY and unknown-map pairs so wire-sensitive float mutations invalidate the parent cache. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>
Pre-count sequences and indefinite arrays with structural bounds so count_from and conditionals are not evaluated against defaults, and recompute live suffix reservations as discriminators are dissected. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>
Promoting ismutable FlagValue defaults into fields shadowed overloaded_fields, so NBNSHeader built NM_FLAGS=0 and NBNS_am failed. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>
Drop the CBOR_Packet counted-build fast path so nested PACKET encode includes post_build/payload, and route ANY through public build() like NULL/UNDEFINED. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>
Reuse BYTE_STRING decode for packet wrappers, mark only optionals absent, drop an unreachable budget raise, override _raw_packet_cache_is_valid, and dedupe half-float / float-key paths. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>
* cbor: reserve structural mins until discriminators are ready Suffix budgeting uses structural_min_items so default count_from and conditionals cannot starve leading fields; live mins apply once string count_from sources are in pkt.fields. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com> * cbor: stream unframed SEQUENCE without trailing pre-count Stop after schema fields so counted SEQUENCE_OF leaves valid, bare break, and truncated suffixes unchanged; framed ARRAY budgets stay strict. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com> * cbor: propagate holds_packets from packet-valued item fields ARRAY_OF/SEQUENCE_OF with CBORF_PACKET elements register as packet storage and reuse nested fingerprints while encode/decode still follow item_field versus packet-class selection. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com> * cbor: drop unused packet-level cardinality cache Remove _cbor_build_counted and _cbor_raw_cache_items; nested encode keeps exact wire via Packet raw cache and bytes(child). AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com> * cbor: share byte-string wire helpers; restore Packet field typing Extract decode/encode helpers so CBORF_BYTE_STRING_PACKET stays CBORF_field[Packet] while reusing definite-length byte-string handling. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com> * cbor: simplify compound parse budgeting Move suffix reservation onto reserve_min_items, share field dissect helpers, and drop the compound-side ready/budget branching. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com> * cbor: fix flake8 blank lines after budget simplify AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Nils Weiss <nils.weiss@dissecto.com> Co-authored-by: Cursor <cursoragent@cursor.com>
regression.uts leaves conf.debug_dissector True, so the nested constructor test must pin False/True explicitly like BYTE_STRING_PACKET. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>
Drop count_from, shrink cardinality to structural_min, isolate ismutable defaults on CBOR_Packet, and fold encode_value into i2m. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>
Drop structural_min_items and skipped-optional validation so active conditionals and required tails keep ownership of budgeted items. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>
Skip top-level CBOR_ABSENT fingerprints, require direct-terminal SEQUENCE_OF, and drop SEQUENCE suffix lookahead. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>
Fail SEMANTIC_TAG(SEQUENCE_OF) at construction, let optional own presence matching, and drop the single-use _mark_absent helper. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>
Use CBOR_KeyKind norms and CBOR_FingerprintKind cache tags, drop redundant int() enum casts, half floats via struct >e, and make EnumElement deepcopy a singleton. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
Author
|
@gpotter2 I've done with a lot of refactorings and a manual review; Could you please have a look? |
Reject nested REMAINDER_OF, require usable CBOR_root for packet classes, snapshot generic CBOR_Object fingerprints, rename SEQUENCE/SEQUENCE_OF to ITEMS/REMAINDER_OF, and slim max_items. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>
Recurse remainder rejection only through unframed ITEMS, apply conf.max_list_count to MAP, and tighten Packet default-copy coverage. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>
Align remainder validation and MAP limit tests with the re-review's preferred form, including ITEMS(ARRAY(...), tail) coverage. AI-Assisted: yes (Composer) Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stack
PR 2 of 3. Depends on #5124 (CRC presets). Reviewable CBOR-only commits sit on top of that foundation; next is BPv7 contrib.
Test plan
./test/run_tests -t test/scapy/layers/cbor.uts -Ntox -e cbor2(Python 3.12)