Skip to content

Commit f3a0463

Browse files
dmitriplotnikovcopybara-github
authored andcommitted
[Pratt Parser] Add a fuzzer comparing ANTLR and Pratt parser outputs and fix uncovered discrepancies
Added `CelPrattParserFuzzer` to fuzz CEL inputs against both `AntlrParser` and `PrattParser`, asserting error parity and AST equality. Parser differences uncovered and fixed in `PrattParser` / `Lexer`: 1. **Mixed unary operator chains (e.g., `-!ll`, `!-x`)**: ANTLR requires consecutive unary operators to be homogeneous (`!`/`-`) and only allows `-` after `!` when immediately followed by an integer or floating-point literal (e.g., `!-42`), whereas Pratt previously allowed arbitrary mixtures of `!` and `-` without parentheses. 2. **Vertical tab (`\v`, ASCII 11)**: ANTLR does not treat `\v` as whitespace, whereas `Lexer` and `PrattParser` previously skipped it. 3. **Unquoted `.in` field selector**: ANTLR treats `in` as a keyword token and rejects unquoted `x.in` (requiring backtick-quoted `` x.`in` ``), whereas Pratt previously accepted unquoted `.in` after `.`. 4. **Chained optional select (`T.?a.?a`) AST positions**: ANTLR records the position of the field constant in `_?._` at the start of the `member` expression (`T`), whereas Pratt stopped at intermediate `.?`/`[]`/`()` nodes. 5. **Numeric literals immediately followed by identifier characters (e.g., `9in-x`)**: ANTLR tokenizes numeric literals (`NUM_INT`, `NUM_UINT`, `NUM_FLOAT`) without rejecting trailing identifier characters so `9in-x` parses as `9 in -x`, whereas Pratt's `Lexer` previously rejected trailing identifier characters at lexing time. 6. **Invalid quoted field selectors inside `has(...)` (e.g., `` has(a.`$b`) ``)**: When `normalizeIdent()` rejects an invalid backtick-quoted field name, `PrattParser` previously still constructed a `CelSelect` with an empty field string, causing `CelExprFactory.newSelect()` to throw `IllegalArgumentException` during `has()` macro expansion instead of returning an unset error expression like `AntlrParser`. Intentional parser differences ignored by `CelPrattParserFuzzer` (where `PrattParser` behavior is preferred): 1. **Raw byte string literal prefixes (`rb'...'`, `rB'...'`, `Rb'...'`, `RB'...'`)**: ANTLR only accepts `br`/`bR`/`Br`/`BR` prefix order, whereas Pratt accepts both `br` and `rb`. 2. **Standalone commas in empty collection literals (`[,]`, `{,}`, `Msg{,}`)**: ANTLR accepts empty collections containing only a comma, whereas Pratt requires at least one element/entry before a trailing comma. 3. **Leading-dot identifier positions (`.R`)**: Pratt records the position of leading-dot identifiers at the `.` token, whereas ANTLR records it at the identifier token after `.`. PiperOrigin-RevId: 987007823
1 parent 5d344bc commit f3a0463

10 files changed

Lines changed: 718 additions & 129 deletions

parser/src/main/java/dev/cel/parser/Lexer.java

Lines changed: 1 addition & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -428,7 +428,6 @@ private void consumeWhitespaceAndComments() {
428428
case '\n':
429429
case ' ':
430430
case '\r':
431-
case 11: // \v
432431
case '\t':
433432
position++;
434433
break;
@@ -589,18 +588,12 @@ private Token consumeNumericLiteral() {
589588
}
590589
} else {
591590
advance(1);
592-
if (c == '0' && consume('x')) {
591+
if (c == '0' && (consume('x') || consume('X'))) {
593592
if (!consumeHexDigits()) {
594593
return setError(
595594
start, position, "integral literal missing digits after hexadecimal separator");
596595
}
597596
TokenType tokenType = consumeIntegralSuffix();
598-
if (consumeIf(Lexer::isIdentTrailing)) {
599-
return setError(
600-
start,
601-
position,
602-
tokenType.getSymbol() + " literal has unexpected trailing characters");
603-
}
604597
return makeToken(tokenType, start, position);
605598
}
606599
consumeDigits();
@@ -622,10 +615,6 @@ && isDigit(content.get(position + 1))) {
622615
}
623616
}
624617
TokenType tokenType = floatingPoint ? TokenType.FLOAT : consumeIntegralSuffix();
625-
if (consumeIf(Lexer::isIdentTrailing)) {
626-
return setError(
627-
start, position, tokenType.getSymbol() + " literal has unexpected trailing characters");
628-
}
629618
return makeToken(tokenType, start, position);
630619
}
631620

0 commit comments

Comments
 (0)