refactor: restructure token handling with NonReservedWord() sentinels

Replace the manual updateKeywords task and hardcoded keyword whitelist
with a NonReservedWord() BNF production bracketed by MIN/MAX sentinel
tokens. Non-reserved keywords are now determined by O(1) range check
in isIdentifierAhead() instead of enumerating them in
RelObjectNameWithoutValue().

- Refactor ParserKeywordsUtils: derive keywords dynamically from
  CCJSqlParserConstants + grammar file, remove hardcoded
  ALL_RESERVED_KEYWORDS array and RESTRICTED_* flags
- Consolidate ConditionalKeywordsTest into KeywordsTest
- Simplify updateKeywords Gradle task (generates RST doc only)
- Update contribution.rst and usage.rst documentation
- allow nested comments

Fixes #1175

Signed-off-by: Andreas Reichel <andreas@manticore-projects.com>
Signed-off-by: manticore-projects <andreas@manticore-projects.com>

Author: manticore-projects

.github/workflows/ci.yml

@@ -87,7 +87,7 @@ jobs:
         run: sudo apt-get install -y xsltproc sphinx-common
 
       - name: Install Python dependencies
-        run: pip install furo myst_parser sphinx_substitution_extensions sphinx_issues sphinx_inline_tabs pygments
+        run: pip install manticore_sphinx_theme myst_parser sphinx_substitution_extensions sphinx_issues sphinx_inline_tabs pygments
 
       - name: Build Sphinx documentation with Gradle
         run: FLOATING_TOC=false ./gradlew -DFLOATING_TOC=false gitChangelogTask renderRR xslt xmldoc sphinx

build.gradle

@@ -91,7 +91,7 @@ tasks.register('generateBuildInfo') {
         def buildTime = Instant.now().toString()
 
         def content = """\
-            |package ai.starlake.jsqltranspiler;
+            |package net.sf.jsqlparser;
             |
             |public final class BuildInfo {
             |  public static final String NAME = "${project.name}";
@@ -539,7 +539,7 @@ Version {{name}}
 
 tasks.register('updateKeywords', JavaExec) {
     group = "Execution"
-    description = "Run the main class with JavaExecTask"
+    description = "Generate the Reserved Keywords documentation"
     classpath = sourceSets.main.runtimeClasspath
     args = [
             file('src/main/jjtree/net/sf/jsqlparser/parser/JSqlParserCC.jjt').absolutePath

src/main/java/net/sf/jsqlparser/expression/OracleHint.java

@@ -24,7 +24,7 @@ public class OracleHint extends ASTNodeAccessImpl implements Expression {
 
     private static final Pattern SINGLE_LINE = Pattern.compile("--\\+ *([^ ].*[^ ])");
     private static final Pattern MULTI_LINE =
-            Pattern.compile("\\/\\*\\+ *([^ ].*[^ ]) *\\*+\\/", Pattern.MULTILINE | Pattern.DOTALL);
+            Pattern.compile("/\\*\\+ *([^ ].*[^ ]) *\\*+/", Pattern.MULTILINE | Pattern.DOTALL);
 
     private String value;
     private boolean singleLine = false;

src/main/java/net/sf/jsqlparser/parser/ParserKeywordsUtils.java

@@ -97,49 +97,27 @@ The JSQLParser is generated by ``JavaCC`` based on the provided Grammar. The Gra
 Manage Reserved Keywords
 ------------------------------
 
-Since JSQLParser is built by JavaCC from a Token based Grammar, ``Reserved Keywords`` need a special treatment. All Tokens of the Grammar would become ``Reserved Keywords`` -- unless explicitly allowed and white-listened.
+Since JSQLParser is built by JavaCC from a Token based Grammar, ``Reserved Keywords`` need a special treatment. All Tokens of the Grammar would become ``Reserved Keywords`` -- unless explicitly allowed as identifiers.
+
+The Grammar uses a ``NonReservedWord()`` BNF production with inline token declarations, bracketed by ``MIN_NON_RESERVED_WORD`` and ``MAX_NON_RESERVED_WORD`` sentinel tokens. JavaCC assigns consecutive token kind values to the inline declarations, which enables an efficient O(1) range check in ``isIdentifierAhead()`` to determine whether a token can be used as an unquoted identifier.
 
 .. code-block:: sql
-    :caption: White-list Keyword example
+    :caption: Non-reserved keyword example
 
-    -- <K_OVERLAPS:"OVERLAPS"> is a Token, recently defined in the Grammar
-    -- Although it is not restricted by the SQL Standard and could be used for Column, Table and Alias names
-    -- Explicitly white-listing OVERLAPS by adding it to the  RelObjectNameWithoutValue() Production will allow for parsing the following statement
+    -- <K_OVERLAPS:"OVERLAPS"> is defined as a non-reserved keyword inside the NonReservedWord() production
+    -- It can be used for Column, Table and Alias names without quoting
 
     SELECT Overlaps( overlaps ) AS overlaps
     FROM overlaps.overlaps overlaps
     WHERE overlaps = 'overlaps'
         AND (CURRENT_TIME, INTERVAL '1' HOUR) OVERLAPS (CURRENT_TIME, INTERVAL -'1' HOUR)
     ;
 
-So we will need to define and white-list any Keywords which may be allowed for Object Names (such as `Schema`, `Table`, `Column`, `Function`, `Alias`). This White-List must be updated whenever the Tokens of the Grammar change (e. |_| g. when adding a new Token or Production).
-
-There is a task ``updateKeywords`` for Gradle and Maven, which will:
-
-    1) Parse the Grammar in order to find all Token definitions
-    2) Read the list of explicitly ``Reserved Keywords`` from ``net/sf/jsqlparser/parser/ParserKeywordsUtils.java``
-    3) Derive the list of ``White-Listed Keywords`` as difference between ``All Tokens`` and ``Reserved Keywords``
-    4) Modifies the Grammar Productions ``RelObjectNameWithoutValue...`` adding all Tokens according to ``White-Listed Keywords``
-    5) Run two special Unit Tests to verify parsing of all ``White-Listed Keywords`` (as `Schema`, `Table`, `Column`, `Function` or `Alias`)
-    6) Update the web page about the Reserved Keywords
-
-
-.. tab:: Gradle
-
-    .. code-block:: shell
-        :caption: Gradle `updateKeywords` Task
-
-            gradle updateKeywords
-
-.. tab:: Maven
-
-    .. code-block:: shell
-        :caption: Maven `updateKeywords` Task
-
-            mvn exec:java
-
+When adding a new keyword token to the Grammar:
 
-Without this Gradle Task, any new Token or Production will become a ``Reserved Keyword`` automatically and can't be used for Object Names without quoting.
+    1) If the keyword should be usable as an unquoted identifier (the common case), add its inline token declaration to the ``NonReservedWord()`` production. It will automatically be placed between the sentinel tokens and recognised by the range check.
+    2) If the keyword must be reserved (e. |_| g. core SQL syntax like ``SELECT``, ``FROM``, ``WHERE``), add it to the ``Reserved SQL Keywords`` TOKEN block **after** the ``MAX_NON_RESERVED_WORD`` sentinel.
+    3) Verify that existing tests pass and that the keyword can be used as a ``Schema``, ``Table``, ``Column``, ``Function`` or ``Alias`` name where expected.
 
 
 Commit a Pull Request
@@ -196,4 +174,4 @@ Please consider using `Conventional Commits` and structure your commit message a
    * - **revert**
      - reverts one or many previous commits
 
-Please visit `Better Programming <https://betterprogramming.pub/write-better-git-commit-messages-to-increase-your-productivity-89fa773e8375>`_ for more information and guidance.
+Please visit `Better Programming <https://betterprogramming.pub/write-better-git-commit-messages-to-increase-your-productivity-89fa773e8375>`_ for more information and guidance.