Fix: digit-leading generalized-identifier segment mislexed as numeric literal

[acinep]

Summary

A run of digits inside an unquoted [fieldName] selector, immediately preceded by whitespace and immediately followed by . then a non-digit identifier character (e.g. [Maintenance vendors 20000101.Department]), was mislexed. readNumericLiteral greedily consumed the digit run as a NumericLiteral token, leaving a bare . that tokenizeDefault has no valid interpretation for and throws on.

TaskUtils.tryLexParse - the entry point most consumers actually use - surfaces this as a Lex-stage error via errorLineMap, even though calling Lexer.tryLex + Lexer.trySnapshot directly on the identical text happens to still succeed (that inconsistency is what made this tricky to pin down).

This is a real-world pattern, not a contrived edge case: it's exactly the naming convention Table.ExpandTableColumn itself generates for an expanded/merged column ("<other query's display name>.<column name>"), so any M document with a merge/expand step where the other query's display name ends in digits immediately before the generated . hits this.

Repro

import * as PQP from "@microsoft/powerquery-parser";

const result = await PQP.TaskUtils.tryLexParse(PQP.DefaultSettings, `let x = [Maintenance vendors 20000101.Department] in x`);
// Before this fix: stage=Lex, resultKind=Error, error="Error on line(s): 0"
// After this fix:  stage=Parse, resultKind=Ok

Fix

When a digit-first token's matched numeric span (Pattern.Numeric) is immediately followed by . then a non-digit identifier-part character - the one sequence a genuine numeric literal can never produce, since a decimal point is always followed by more digits - re-read the whole span as an identifier instead. This mirrors the digit-tolerant continuation that identifier tokens starting with a letter already get via IdentifierUtils.getIdentifierLength (e.g. [Foo Bar.Baz] already lexed correctly before this change; only the digit-leading case was missing the equivalent handling).

readNumericLiteral is left untouched and still used for the one call site where a leading . is unambiguous (.5-style literals - M identifiers never begin with a literal dot, so there's no ambiguity to resolve there).

Testing

• Added 3 new tests to lexSimple.test.ts covering: a bare digit-leading identifier segment, one inside a full field selector alongside an ordinary Identifier/Equal/etc. token stream, and confirmation that ordinary numeric literals (including ones immediately followed by more digits) are unaffected.
• Full existing suite: npm test (399 → 402 passing, 1 pending, unchanged) and npm run test:resources (102 passing) both green with zero regressions.
• Verified against the two real-world M queries that surfaced this (via a downstream consumer, not included here) - both now lex and parse successfully end-to-end.

[acinep]

@microsoft-github-policy-service agree