Skip to content

needs_separator_when_before omits CDC, so serialized tokens re-parse wrong #434

Description

@hey-jj

TokenSerializationType::needs_separator_when_before returns false when a Number, DelimHash, DelimAt, or DelimMinus is followed by CDC. Concatenating the two serializations produces text that tokenizes into different tokens than the input.

Reproducer

cssparser 0.37.0, default features.

use cssparser::{Parser, ParserInput, ToCss};

fn main() {
    let mut pi = ParserInput::new("5 -->");
    let mut p = Parser::new(&mut pi);
    let a = p.next().unwrap().clone(); // Number { value: 5.0, int_value: Some(5) }
    let b = p.next().unwrap().clone(); // CDC

    // no separator requested
    assert!(!a
        .serialization_type()
        .needs_separator_when_before(b.serialization_type()));

    let mut s = String::new();
    a.to_css(&mut s).unwrap();
    b.to_css(&mut s).unwrap();
    assert_eq!(s, "5-->");
}

Observed vs expected

Observed: the call returns false and serialization yields 5-->. Re-parsing 5--> gives Dimension { value: 5.0, unit: "--" } followed by Delim('>').

Expected: true, which signals that an empty comment is needed so the output round-trips. The doc comment on needs_separator_when_before says it returns true if "an empty comment /**/ needs to be inserted between them so that they are not re-parsed as a single token". CSS Syntax Level 3 §Serialization requires the serialized form to round-trip.

Root cause

src/serializer.rs:519. The Ident and AtKeywordOrHash | Dimension rows list CDC. The rows covering Number, DelimHash, DelimAt, and DelimMinus do not.

Scope

Four pairs return false: Number then CDC gives 5-->, re-parsing as Dimension{5,"--"} + Delim('>'). DelimHash then CDC gives #-->, re-parsing as IDHash("--") + Delim('>'). DelimAt then CDC gives @-->, re-parsing as AtKeyword("--") + Delim('>'). DelimMinus then CDC gives --->, re-parsing as Ident("---") + Delim('>').

Any consumer using this API to re-emit CSS is affected. Stylo calls it for custom-property and var() substitution serialization. Present on main as of the 2026-07-21 commit.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions