From c8565774aa5b2794b6c11590c6073ee639f77b93 Mon Sep 17 00:00:00 2001 From: Jake Barnby Date: Thu, 13 Aug 2026 22:33:01 +1200 Subject: [PATCH] docs(readme): sync with code, document AST layer and correct dialect claims The README had drifted from the code in ways that would mislead a reader rather than merely omit detail. Broken or wrong: - CI badges pointed at tests.yml/integration.yml, neither of which exists; the repo has ci.yml, baseline.yml, linter.yml, static-analysis.yml. - The aggregations example claimed `HAVING \`total\` > ?`. The builder expands an aggregate alias back to its expression, so it emits `HAVING COUNT(*) > ?` -- standard SQL forbids aliases in HAVING. - The wire-protocol section listed seven write keywords that actually classify as Unknown (RENAME, REPLACE, LOAD, MERGE, EXECUTE, HANDLER, IMPORT) while omitting the real one (LOCK); put SAVEPOINT/RELEASE/SET under TransactionEnd when they return Type::Transaction; omitted the Transaction case entirely; named COM_INIT_DB/COM_QUIT, which do not exist; and claimed the PostgreSQL parser handles X and startup messages, which fall through to Unknown, while omitting B and E. Overstated UnsupportedException claims -- these are worse than omissions because the documented failure mode was loud and the real one is silent: - Column-level ttl() is ignored outside ClickHouse, not rejected. - MongoDB drops CHECK constraints and generated columns silently, and maps SERIAL to int rather than throwing. - SQLite supports foreign keys inline via foreignKey($column); only the ALTER-based form is absent. The inline form was undocumented. - ClickHouse procedures/triggers are absent from the class, so calling them is a fatal error, not a catchable exception. Dialect drift: MariaDB was missing from five capability lists it inherits from MySQL; Upsert was attributed to Builder\SQL, which does not implement it; SQLite was missing from String Aggregates; vector was "PostgreSQL only" but also exists on ClickHouse and MongoDB; partitions were claimed on ClickHouse, which has only its own partitionBy(). The Feature Matrix is rebuilt from the actual implements clauses. The AST and tokenizer subsystems had no coverage at all, so this adds a section for them, plus the executor pattern, materialized views, renameIndex, database and analyze DDL, and the Query factories and modifiers that were missing. Query::elemMatch() is documented as compilable only by a custom Compiler, since every bundled builder rejects it, and groupByTimeBucket is scoped to ClickHouse. Every documented output in the new and corrected content was produced by executing the code. Co-Authored-By: Claude Opus 5 --- README.md | 302 ++++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 260 insertions(+), 42 deletions(-) diff --git a/README.md b/README.md index cc1525a..fbb288e 100644 --- a/README.md +++ b/README.md @@ -1,11 +1,10 @@ # Utopia Query -[![Tests](https://github.com/utopia-php/query/actions/workflows/tests.yml/badge.svg)](https://github.com/utopia-php/query/actions/workflows/tests.yml) -[![Integration Tests](https://github.com/utopia-php/query/actions/workflows/integration.yml/badge.svg)](https://github.com/utopia-php/query/actions/workflows/integration.yml) +[![CI](https://github.com/utopia-php/query/actions/workflows/ci.yml/badge.svg)](https://github.com/utopia-php/query/actions/workflows/ci.yml) [![Linter](https://github.com/utopia-php/query/actions/workflows/linter.yml/badge.svg)](https://github.com/utopia-php/query/actions/workflows/linter.yml) [![Static Analysis](https://github.com/utopia-php/query/actions/workflows/static-analysis.yml/badge.svg)](https://github.com/utopia-php/query/actions/workflows/static-analysis.yml) -A PHP library for building type-safe, dialect-aware queries and DDL statements. Provides a fluent builder API with parameterized output for MySQL, MariaDB, PostgreSQL, SQLite, ClickHouse, and MongoDB, plus wire protocol parsers and a serializable `Query` value object for passing query definitions between services. +A PHP library for building type-safe, dialect-aware queries and DDL statements. Provides a fluent builder API with parameterized output for MySQL, MariaDB, PostgreSQL, SQLite, ClickHouse, and MongoDB, plus a SQL tokenizer and AST for inspecting and rewriting existing SQL, wire protocol parsers, and a serializable `Query` value object for passing query definitions between services. ## Installation @@ -53,6 +52,7 @@ composer require utopia-php/query - [Conditional Building](#conditional-building) - [Builder Cloning and Callbacks](#builder-cloning-and-callbacks) - [Debugging](#debugging) + - [Executors](#executors) - [Hooks](#hooks) - [Dialect-Specific Features](#dialect-specific-features) - [MySQL](#mysql) @@ -73,11 +73,18 @@ composer require utopia-php/query - [Partitions](#partitions) - [Comments](#comments) - [Views](#views) + - [Databases and Maintenance](#databases-and-maintenance) - [Procedures and Triggers](#procedures-and-triggers) - [PostgreSQL Schema Extensions](#postgresql-schema-extensions) - [ClickHouse Schema](#clickhouse-schema) - [SQLite Schema](#sqlite-schema) - [MongoDB Schema](#mongodb-schema) +- [SQL Tokenizer and AST](#sql-tokenizer-and-ast) + - [Tokenizer](#tokenizer) + - [Parsing to an AST](#parsing-to-an-ast) + - [Serializing an AST](#serializing-an-ast) + - [Walking and Rewriting](#walking-and-rewriting) + - [Builder Round-Trip](#builder-round-trip) - [Wire Protocol Parsers](#wire-protocol-parsers) - [SQL Parser](#sql-parser) - [MySQL Parser](#mysql-parser) @@ -135,19 +142,31 @@ Query::exists(['name', 'email']); Query::notExists('legacyField'); // Date helpers +Query::createdBefore('2024-01-01'); Query::createdAfter('2024-01-01'); +Query::createdBetween('2024-01-01', '2024-06-30'); +Query::updatedBefore('2024-01-01'); +Query::updatedAfter('2024-01-01'); Query::updatedBetween('2024-01-01', '2024-06-30'); ``` +Most text and containment filters have a negated counterpart: `notSearch()`, `notStartsWith()`, `notEndsWith()`, `notContains()`, `notBetween()`. + > **Note:** `Query::contains()` is deprecated — use `Query::containsString()` for string substring matching or `Query::containsAny()` for array/relationship attributes. ### Ordering and Pagination ```php +use Utopia\Query\NullsPosition; + Query::orderAsc('createdAt'); Query::orderDesc('score'); Query::orderRandom(); +// Control where NULLs sort +Query::orderAsc('deletedAt', NullsPosition::Last); +Query::orderDesc('score', NullsPosition::First); + Query::limit(25); Query::offset(50); @@ -169,6 +188,8 @@ Query::or([ ]); ``` +A `Query::elemMatch($attribute, $queries)` factory also exists, for matching a single array element against all of the given predicates. It serializes and parses like any other query, but **no bundled builder compiles it** — every dialect throws `UnsupportedException`. It is only useful to consumers that implement their own [`Compiler`](#compiler-interface). + ### Spatial Queries ```php @@ -204,8 +225,11 @@ Query::jsonPath('metadata', 'address.city', '=', 'London'); ```php Query::select(['name', 'email', 'createdAt']); +Query::distinct(); ``` +Most builder clauses have a `Query` factory equivalent, so a whole statement can be expressed as a serializable array and passed to `queries()` — aggregations (`count`, `countDistinct`, `sum`, `avg`, `min`, `max`, `stddev`, `variance`, `bitAnd`, …), `groupBy`, `having`, joins (`join`, `leftJoin`, `rightJoin`, `crossJoin`, `fullOuterJoin`, `naturalJoin`), set operations (`union`, `unionAll`), and `groupByTimeBucket` (ClickHouse only). Builder-only clauses with no `Query` counterpart include `whereRaw()`, `whereColumn()`, `window()`, and the locking methods. + ### Raw Expressions ```php @@ -237,11 +261,17 @@ $queries = Query::parseQueries([$json1, $json2]); $parsed = Query::groupByType($queries); // $parsed->filters, $parsed->selections, $parsed->aggregations, $parsed->groupBy, // $parsed->having, $parsed->joins, $parsed->unions, $parsed->limit, $parsed->offset, -// $parsed->cursor, $parsed->cursorDirection, $parsed->distinct +// $parsed->cursor, $parsed->cursorDirection, $parsed->distinct, $parsed->timeBuckets // Filter by method type $cursors = Query::getByType($queries, [Method::CursorAfter, Method::CursorBefore]); +// Shorthand for the cursor queries specifically +$cursors = Query::getCursorQueries($queries); + +// Stable hash of query shape, ignoring bound values — for cache keys and plan logging +$key = Query::fingerprint($queries); + // Merge (later limit/offset/cursor overrides earlier) $merged = Query::merge($defaultQueries, $userQueries); @@ -368,9 +398,11 @@ $result = (new Builder()) ->build(); // SELECT COUNT(*) AS `total`, SUM(`price`) AS `total_price`, `status` -// FROM `orders` GROUP BY `status` HAVING `total` > ? +// FROM `orders` GROUP BY `status` HAVING COUNT(*) > ? ``` +A `having()` predicate may reference an aggregate by the alias given to the aggregate call itself (`total` above); the builder expands that alias back to the underlying expression, since standard SQL does not allow `HAVING` to reference a `SELECT` alias. + **Distinct:** ```php @@ -385,7 +417,7 @@ $result = (new Builder()) ### Statistical Aggregates -Available on MySQL, PostgreSQL, SQLite, and ClickHouse via the `StatisticalAggregates` interface: +Available on MySQL, MariaDB, PostgreSQL, SQLite, and ClickHouse via the `StatisticalAggregates` interface: ```php use Utopia\Query\Builder\PostgreSQL as Builder; @@ -403,7 +435,7 @@ $result = (new Builder()) ### Bitwise Aggregates -Available on MySQL, PostgreSQL, SQLite, and ClickHouse via the `BitwiseAggregates` interface: +Available on MySQL, MariaDB, PostgreSQL, SQLite, and ClickHouse via the `BitwiseAggregates` interface: ```php $result = (new Builder()) @@ -416,7 +448,7 @@ $result = (new Builder()) ### Conditional Aggregates -Available on MySQL, PostgreSQL, SQLite, and ClickHouse via the `ConditionalAggregates` interface: +Available on MySQL, MariaDB, PostgreSQL, SQLite, and ClickHouse via the `ConditionalAggregates` interface: ```php use Utopia\Query\Builder\PostgreSQL as Builder; @@ -436,7 +468,7 @@ Also available: `avgWhen()`, `minWhen()`, `maxWhen()`. ### String Aggregates -Available on MySQL, PostgreSQL, and ClickHouse via the `StringAggregates` interface: +Available on MySQL, MariaDB, PostgreSQL, SQLite, and ClickHouse via the `StringAggregates` interface: ```php use Utopia\Query\Builder\MySQL as Builder; @@ -468,7 +500,7 @@ $result = (new Builder()) ### Group By Modifiers -Available on MySQL, PostgreSQL, and ClickHouse via the `GroupByModifiers` interface: +Available on MySQL, MariaDB, PostgreSQL, and ClickHouse via the `GroupByModifiers` interface: ```php use Utopia\Query\Builder\MySQL as Builder; @@ -749,7 +781,7 @@ $result = (new Builder()) ### Upsert -Available on MySQL, PostgreSQL, and SQLite builders (`Builder\SQL` subclasses): +Available on MySQL, MariaDB, PostgreSQL, and SQLite via the `Upsert` and `UpsertSelect` interfaces. MongoDB implements `Upsert` with its own document semantics (see [MongoDB](#mongodb)): `onConflict()` takes the conflict key columns and the columns to update on conflict: @@ -799,7 +831,7 @@ $result = (new Builder()) ### Locking -Available on MySQL, PostgreSQL, and SQLite builders: +Available on MySQL, MariaDB, PostgreSQL, and SQLite via the `Locking` interface: ```php $result = (new Builder()) @@ -817,7 +849,7 @@ PostgreSQL also supports table-specific locking: `forUpdateOf('accounts')`, `for ### Transactions -Available on MySQL, PostgreSQL, and SQLite builders: +Available on MySQL, MariaDB, PostgreSQL, and SQLite via the `Transactions` interface: ```php $builder = new Builder(); @@ -831,7 +863,7 @@ $builder->rollback(); // ROLLBACK ### EXPLAIN -Available on all builders. MySQL and PostgreSQL provide extended options: +Available on every SQL builder and ClickHouse. MySQL and PostgreSQL provide extended options: ```php use Utopia\Query\Builder\MySQL as Builder; @@ -855,6 +887,8 @@ $result = (new PgBuilder()) ->explain(analyze: true, verbose: true, buffers: true, format: 'JSON'); ``` +> **Note:** the MongoDB builder inherits `explain()` but merely prefixes `EXPLAIN ` to the JSON operation document, which is not a valid MongoDB command. Use the driver's own explain support instead. + ### Conditional Building `when()` applies a callback only when the condition is true: @@ -902,6 +936,34 @@ $sql = (new Builder()) // SELECT * FROM `users` WHERE `status` IN ('active') LIMIT 10 ``` +### Executors + +The library never talks to a database itself — it emits `Statement` objects. Attaching an executor closure is optional and lets you collapse build-and-run into one call. The closure receives the `Statement` and returns rows or an affected-row count: + +```php +$executor = function (Statement $statement) use ($pdo): array|int { + $prepared = $pdo->prepare($statement->query); + $prepared->execute($statement->bindings); + + return $statement->readOnly ? $prepared->fetchAll() : $prepared->rowCount(); +}; + +$rows = (new Builder()) + ->setExecutor($executor) + ->from('users') + ->filter([Query::equal('status', ['active'])]) + ->execute(); // array|int +``` + +An executor can also be attached to a single `Statement` after the fact. `withExecutor()` returns a new `Statement` — `Statement` is readonly: + +```php +$statement = (new Builder())->from('users')->build(); +$rows = $statement->withExecutor($executor)->execute(); +``` + +Calling `execute()` on a `Statement` with no executor throws `BadMethodCallException`. `Schema::setExecutor()` accepts the same closure shape for DDL statements. + ### Hooks Hooks extend the builder with reusable, testable classes for attribute resolution and condition injection. @@ -1816,15 +1878,20 @@ Unsupported features are not on the class — consumers type-hint the interface | Selects, Filters, Aggregates, Joins, Unions, CTEs, Inserts, Updates, Deletes, Hooks | x | | | | | | | | | Windows | x | | | | | | | | | `whereRaw` / `whereColumn` | | x | | | | | x | | -| Locking, Transactions, Upsert | | x | | | | | | | -| Spatial, Full-Text Search | | x | | | | | | | +| Locking, Transactions | | x | | | | | | | +| Locking `OF` (`forUpdateOf`/`forShareOf`) | | | | | x | | | | +| Upsert | | | x | x | x | x | | x | +| Upsert Select | | | x | x | x | x | | | +| Insert or Ignore | | | x | x | x | x | | x | +| Spatial | | | x | x | x | | | | +| Full-Text Search | | | x | x | x | | | x | | Statistical Aggregates | | | x | x | x | x | x | | | Bitwise Aggregates | | | x | x | x | x | x | | | Conditional Aggregates | | | x | x | x | x | x | | | JSON (incl. `setJsonPath`) | | | x | x | x | x | | | | Hints | | | x | x | | | x | | | Lateral Joins | | | x | x | x | | | | -| String Aggregates | | | x | x | x | | x | | +| String Aggregates | | | x | x | x | x | x | | | Group By Modifiers | | | x | x | x | | x | | | Sequences (`nextVal`/`currVal`) | | | | x | x | | | | | `RETURNING` | | | | x | x | | | | @@ -1843,14 +1910,14 @@ Unsupported features are not on the class — consumers type-hint the interface | `groupByTimeBucket` | | | | | | | x | | | Named-typed `{name:Type}` bindings | | | | | | | x | | | Approximate Aggregates (incl. `quantiles`) | | | | | | | x | | -| Upsert (Mongo-style) | | | | | | | | x | -| Full-Text Search (Mongo) | | | | | | | | x | | Field Updates | | | | | | | | x | | Array Push Modifiers | | | | | | | | x | | Conditional Array Updates | | | | | | | | x | | Pipeline Stages | | | | | | | | x | | Atlas Search | | | | | | | | x | +MongoDB implements the same `Upsert` and `FullTextSearch` interfaces as the SQL dialects, so `instanceof` checks pass, but both emit MongoDB operation documents with document semantics rather than SQL — see [MongoDB](#mongodb). + ## Schema Builder The schema builder generates DDL statements for table creation, alteration, indexes, views, and more. @@ -1890,13 +1957,13 @@ $result = $schema->table('users') ->createIfNotExists(); ``` -Available column types: `id`, `uuid`, `string`, `text`, `mediumText`, `longText`, `tinyInteger`, `smallInteger`, `integer`, `bigInteger`, `serial`, `bigSerial`, `smallSerial`, `float`, `decimal`, `boolean`, `datetime`, `timestamp`, `json`, `binary`, `enum`, `point`, `linestring`, `polygon`, `vector` (PostgreSQL only), `timestamps`. +Available column types: `id`, `uuid`, `string`, `text`, `mediumText`, `longText`, `tinyInteger`, `smallInteger`, `integer`, `bigInteger`, `serial`, `bigSerial`, `smallSerial`, `float`, `decimal`, `boolean`, `datetime`, `timestamp`, `json`, `binary`, `enum`, `point`, `linestring`, `polygon`, `vector` (PostgreSQL, ClickHouse, MongoDB), `timestamps`. -Column modifiers: `nullable()`, `default($value)`, `defaultRaw($expression)`, `unsigned()`, `unique()`, `primary()`, `autoIncrement()`, `after($column)`, `comment($text)`, `collation($collation)`, `check($expression)`, `generatedAs($expression)` + `stored()` / `virtual()`, `ttl($expression)` (ClickHouse), `userType($name)` (PostgreSQL). +Column modifiers: `nullable()`, `default($value)`, `defaultRaw($expression)`, `unsigned()`, `unique()`, `primary()`, `autoIncrement()`, `after($column)`, `comment($text)`, `collation($collation)`, `check($expression)`, `generatedAs($expression)` + `stored()` / `virtual()`, `srid($srid)` (spatial columns), `dimensions($dimensions)` (vector columns), `ttl($expression)` (ClickHouse), `userType($name)` (PostgreSQL). **Raw default expressions** — use `defaultRaw($expression)` for dialect-specific server-generated defaults that `default()` would otherwise quote as a string literal (`now()`, `CURRENT_TIMESTAMP`, `gen_random_uuid()`, `generateUUIDv4()`, `UUID()`, …). The expression is emitted verbatim and must come from a trusted source; it must not be empty or contain a semicolon. Takes precedence over `default()` when both are set. -**SERIAL types** — auto-incrementing integers. PostgreSQL emits native `SERIAL` / `BIGSERIAL` / `SMALLSERIAL`; MySQL/MariaDB compile to `INT AUTO_INCREMENT` / `BIGINT AUTO_INCREMENT` / `SMALLINT AUTO_INCREMENT`; SQLite maps to `INTEGER`. ClickHouse and MongoDB throw `UnsupportedException`: +**SERIAL types** — auto-incrementing integers. PostgreSQL emits native `SERIAL` / `BIGSERIAL` / `SMALLSERIAL`; MySQL/MariaDB compile to `INT AUTO_INCREMENT` / `BIGINT AUTO_INCREMENT` / `SMALLINT AUTO_INCREMENT`; SQLite maps to `INTEGER`; MongoDB maps them to the BSON `int` type. ClickHouse throws `UnsupportedException`: ```php $result = $schema->table('orders') @@ -1971,6 +2038,9 @@ $result = $schema->table('order_items') ```php $result = $schema->createIndex('users', 'idx_email', ['email'], unique: true); $result = $schema->dropIndex('users', 'idx_email'); + +// Rename an existing index (MySQL, PostgreSQL) — RenameIndex interface +$result = $schema->renameIndex('users', 'idx_email', 'idx_users_email'); ``` PostgreSQL supports index methods, operator classes, and concurrent creation: @@ -2008,9 +2078,22 @@ $result = $schema->dropForeignKey('orders', 'fk_user'); Available actions: `ForeignKeyAction::Cascade`, `SetNull`, `SetDefault`, `Restrict`, `NoAction`. +`addForeignKey()`/`dropForeignKey()` come from the `ForeignKeys` interface (MySQL, PostgreSQL). Foreign keys can also be declared inline at table-creation time with `foreignKey($column)`, which is additionally available on SQLite: + +```php +$result = $schema->table('posts') + ->integer('user_id') + ->foreignKey('user_id')->references('id')->on('users') + ->onDelete(ForeignKeyAction::Cascade) + ->create(); + +// CREATE TABLE `posts` (`user_id` INTEGER NOT NULL, +// FOREIGN KEY (`user_id`) REFERENCES `users` (`id`) ON DELETE CASCADE) +``` + ### Partitions -Available on MySQL, PostgreSQL, and ClickHouse: +The `partitionBy*` strategies below and `createPartition()` are available on MySQL and PostgreSQL only. ClickHouse partitions through its own `Table\ClickHouse::partitionBy(string $expression)` (see [ClickHouse Schema](#clickhouse-schema)) and implements `DropPartition` but neither `Partitioning` nor `CreatePartition`. ```php // Define partition strategy in table creation @@ -2023,7 +2106,7 @@ $result = $schema->table('events') // Create a child partition (MySQL, PostgreSQL) $result = $schema->createPartition('events', 'events_2024', "VALUES LESS THAN ('2025-01-01')"); -// Drop a partition +// Drop a partition (MySQL, PostgreSQL, ClickHouse) $result = $schema->dropPartition('events', 'events_2024'); ``` @@ -2053,6 +2136,8 @@ $result = $schema->commentOnColumn('users', 'email', 'Primary contact email'); ### Views +`createView()` and `dropView()` come from the `Views` interface, implemented on every dialect. `createOrReplaceView()` comes from `ReplaceView` (MySQL and PostgreSQL only). + ```php $query = (new Builder())->from('users')->filter([Query::equal('active', [true])]); @@ -2061,6 +2146,26 @@ $result = $schema->createOrReplaceView('active_users', $query); $result = $schema->dropView('active_users'); ``` +**Materialized views** — via the `MaterializedViews` interface, currently ClickHouse only. The body accepts a `Builder` (whose bindings ride along on the returned `Statement`) or a raw SQL string. `$targetTable` is the ClickHouse destination table for the materialized aggregate: + +```php +$result = $schema->createMaterializedView('daily_totals', $query, targetTable: 'daily_totals_store'); +$result = $schema->dropMaterializedView('daily_totals'); +``` + +> **Security:** when `$body` is a string it is inlined verbatim with no escaping or validation. Pass only SQL you fully control, and prefer the `Builder` overload whenever any part of the body is parameterized. + +### Databases and Maintenance + +```php +// Databases interface (MySQL, PostgreSQL, ClickHouse, MongoDB) +$result = $schema->createDatabase('analytics'); +$result = $schema->dropDatabase('analytics'); + +// AnalyzeTable interface (MySQL, PostgreSQL, MongoDB) +$result = $schema->analyzeTable('users'); +``` + ### Procedures and Triggers ```php @@ -2149,9 +2254,9 @@ $result = $schema->table('events') // CREATE TABLE `events` (...) ENGINE = MergeTree() ORDER BY (...) ``` -ClickHouse uses `Nullable(type)` wrapping for nullable columns, `Enum8(...)` for enums, `Tuple(Float64, Float64)` for points, and `TYPE minmax GRANULARITY 3` for indexes. Foreign keys, stored procedures, triggers, generated columns, and CHECK constraints throw `UnsupportedException`. +ClickHouse uses `Nullable(type)` wrapping for nullable columns, `Enum8(...)` for enums, `Tuple(Float64, Float64)` for points, and `TYPE minmax GRANULARITY 3` for indexes. Foreign keys, generated columns, and CHECK constraints throw `UnsupportedException`. Stored procedures and triggers are absent from the class entirely — check with `instanceof` rather than catching. -Supports the `TableComments`, `ColumnComments`, `DropPartition`, `Views`, and `Databases` interfaces. +Supports the `TableComments`, `ColumnComments`, `DropPartition`, `Views`, `MaterializedViews`, and `Databases` interfaces. **Engine selection** — choose from 10 variants of the `Engine` enum: @@ -2202,7 +2307,7 @@ $schema->table('events') // differ from the primary key. ``` -TTL expressions are emitted verbatim; they must not be empty or contain semicolons. Dialects other than ClickHouse throw `UnsupportedException`. +TTL expressions are emitted verbatim; they must not be empty or contain semicolons. Table-level `ttl()` exists only on `Table\ClickHouse`. Column-level `ttl()` is on the shared `Column` class but only ClickHouse reads it — on other dialects it is silently ignored rather than raising. **Skip-index algorithms** — every ClickHouse index is a data-skipping index that accelerates WHERE pruning by letting the engine skip whole granules. Pick the algorithm that matches the column shape via the `algorithm` argument on `Table::index()`: @@ -2414,7 +2519,7 @@ These OLAP-shaped modifiers live on the ClickHouse-specific `Column\ClickHouse` use Utopia\Query\Schema\SQLite as Schema; ``` -SQLite uses simplified type mappings: `INTEGER` for booleans, `TEXT` for datetimes/JSON, `REAL` for floats, `BLOB` for binary. Auto-increment uses `AUTOINCREMENT`. Vector and spatial types are not supported. Foreign keys, stored procedures, and triggers throw `UnsupportedException`. SERIAL types map to `INTEGER`. Both `STORED` and `VIRTUAL` generated columns are supported. +SQLite uses simplified type mappings: `INTEGER` for booleans, `TEXT` for datetimes/JSON, `REAL` for floats, `BLOB` for binary. Auto-increment uses `AUTOINCREMENT`. Vector and `Array`/`Tuple` types throw `UnsupportedException`; spatial column types compile to `TEXT` with no geometry semantics. Foreign keys are supported inline at table creation via `foreignKey($column)`; the ALTER-based `addForeignKey()`/`dropForeignKey()`, stored procedures, and triggers are absent from the class rather than throwing. SERIAL types map to `INTEGER`. Both `STORED` and `VIRTUAL` generated columns are supported. ### MongoDB Schema @@ -2482,11 +2587,121 @@ $result = $schema->createDatabase('analytics'); $result = $schema->dropDatabase('analytics'); ``` -Column types map to BSON types: `string` → `string`, `integer`/`bigInteger` → `int`, `float`/`double` → `double`, `boolean` → `bool`, `datetime`/`timestamp` → `date`, `json` → `object`, `binary` → `binData`. Composite primary keys, CHECK constraints, generated columns, SERIAL types, and user-defined types all throw `UnsupportedException`. +Column types map to BSON types: `string` → `string`, `integer`/`bigInteger` → `int`, `float`/`double` → `double`, `boolean` → `bool`, `datetime`/`timestamp` → `date`, `json` → `object`, `binary` → `binData`. Composite primary keys and user-defined types throw `UnsupportedException`, as does dropping or renaming a column. SERIAL types map to `int`. CHECK constraints and generated columns are silently dropped — the JSON Schema validator has no equivalent, so enforce them in application code. + +## SQL Tokenizer and AST + +Everything above generates SQL. This layer goes the other way: it takes existing SQL text and turns it into an inspectable, rewritable tree. Use it to validate columns against an allow-list, inject tenant predicates into queries you did not author, rename tables, or translate a statement from one dialect's quoting to another's. + +```php +use Utopia\Query\Tokenizer\Tokenizer; +use Utopia\Query\AST\Parser; +use Utopia\Query\AST\Serializer; +``` + +### Tokenizer + +`Tokenizer::tokenize()` returns a flat list of `Token` objects, each with a `type` (`TokenType`), a `value`, and a `position`. `Tokenizer::filter()` strips whitespace and comment tokens, leaving only meaningful ones: + +```php +$tokenizer = new Tokenizer(); +$tokens = Tokenizer::filter($tokenizer->tokenize( + "SELECT name, email FROM users WHERE status = 'active' ORDER BY name ASC LIMIT 10" +)); + +$tokens[0]->type; // TokenType::Keyword +$tokens[0]->value; // 'SELECT' +``` + +`TokenType` covers `Keyword`, `Identifier`, `QuotedIdentifier`, `Integer`, `Float`, `String`, `Boolean`, `Null`, `Operator`, `LeftParen`, `RightParen`, `Comma`, `Semicolon`, `Dot`, `Star`, `Placeholder`, `NamedPlaceholder`, `NumberedPlaceholder`, `LineComment`, `BlockComment`, `Whitespace`, and `Eof`. + +Dialect subclasses handle dialect-specific quoting and operators — `Tokenizer\MySQL`, `Tokenizer\MariaDB`, `Tokenizer\PostgreSQL`, `Tokenizer\SQLite`, and `Tokenizer\ClickHouse`. + +### Parsing to an AST + +`AST\Parser` is a recursive-descent parser over the token list. It produces an `AST\Statement\Select`: + +```php +$ast = (new Parser())->parse($tokens); + +$ast->from->name; // 'users' +count($ast->columns); // 2 +$ast->where; // AST\Expression\Binary +``` + +The node hierarchy is made of readonly value objects. Expressions implement the `AST\Expression` marker interface: `Literal`, `Raw`, `Star`, `Placeholder`, `Reference\Column`, `Call\Func`, and `Expression\{Aliased, Between, Binary, CaseWhen, Cast, Conditional, Exists, In, Subquery, Unary, Window}`. Structural nodes include `Reference\Table`, `JoinClause`, `OrderByItem`, `SubquerySource`, `Definition\Cte`, `Definition\Window`, and `Specification\Window`. + +`Statement\Select` is immutable; use its `with()` helper to derive a modified copy. + +### Serializing an AST + +`AST\Serializer` renders a `Select` back to SQL. Dialect subclasses override identifier quoting: + +```php +use Utopia\Query\AST\Serializer\MySQL; +use Utopia\Query\AST\Serializer\PostgreSQL; + +(new MySQL())->serialize($ast); +// SELECT `name`, `email` FROM `users` WHERE `status` = 'active' ORDER BY `name` ASC LIMIT 10 + +(new PostgreSQL())->serialize($ast); +// SELECT "name", "email" FROM "users" WHERE "status" = 'active' ORDER BY "name" ASC LIMIT 10 +``` + +Subclasses are provided for `MySQL`, `MariaDB`, `PostgreSQL`, `SQLite`, and `ClickHouse`. `serializeExpression()` renders a single expression node in isolation. + +### Walking and Rewriting + +`AST\Walker` applies a `Visitor` to every node and returns a new, possibly transformed `Select`. The `Visitor` interface has three methods — `visitExpression()`, `visitTableReference()`, and `visitSelect()` — each returning the original node to keep it or a replacement to swap it. + +Three visitors ship with the library: + +```php +use Utopia\Query\AST\Walker; +use Utopia\Query\AST\Visitor\ColumnValidator; +use Utopia\Query\AST\Visitor\FilterInjector; +use Utopia\Query\AST\Visitor\TableRenamer; +use Utopia\Query\AST\Expression\Binary; +use Utopia\Query\AST\Reference\Column; +use Utopia\Query\AST\Literal; + +$walker = new Walker(); + +// Rename tables — everywhere they appear, including joins +$result = $walker->walk($ast, new TableRenamer(['users' => 'accounts'])); +// SELECT `name`, `email` FROM `accounts` WHERE `status` = 'active' ... + +// AND an extra predicate onto the WHERE clause +$result = $walker->walk($ast, new FilterInjector( + new Binary(new Column('tenant_id'), '=', new Literal(42)) +)); +// SELECT `name`, `email` FROM `users` WHERE `status` = 'active' AND `tenant_id` = 42 ... + +// Reject any column outside the allow-list +$result = $walker->walk($ast, new ColumnValidator(['name'], allowStar: false)); +// throws Utopia\Query\Exception: Column 'email' is not in the allowed list +``` + +`ColumnValidator` also rejects `SELECT *` unless constructed with `allowStar: true`. + +### Builder Round-Trip + +Any builder can hand out its AST and be reconstructed from one, so a fluent query can be rewritten through the visitor pipeline and then rebuilt: + +```php +$builder = (new Builder()) + ->from('users') + ->select(['id', 'name']) + ->filter([Query::equal('status', ['active'])]); + +$ast = $builder->toAst(); // AST\Statement\Select +$rebuilt = Builder::fromAst($ast); // static +$rebuilt->build()->query; // SELECT `id`, `name` FROM `users` WHERE `status` IN (?) +``` ## Wire Protocol Parsers -The `Parser` interface classifies raw database traffic into query types (`Read`, `Write`, `TransactionBegin`, `TransactionEnd`, `Unknown`). This is useful for connection proxies, audit logging, and read/write splitting. +The `Parser` interface classifies raw database traffic into query types (`Read`, `Write`, `TransactionBegin`, `TransactionEnd`, `Transaction`, `Unknown`). This is useful for connection proxies, audit logging, and read/write splitting. ```php use Utopia\Query\Parser; @@ -2509,11 +2724,13 @@ $type = $parser->classifySQL('COMMIT'); // Type::TransactionEnd Read keywords: `SELECT`, `SHOW`, `DESCRIBE`, `DESC`, `EXPLAIN`, `WITH` (when followed by a read), `TABLE`, `VALUES`. -Write keywords: `INSERT`, `UPDATE`, `DELETE`, `ALTER`, `DROP`, `CREATE`, `TRUNCATE`, `RENAME`, `REPLACE`, `LOAD`, `GRANT`, `REVOKE`, `MERGE`, `CALL`, `EXECUTE`, `DO`, `HANDLER`, `IMPORT`. +Write keywords: `INSERT`, `UPDATE`, `DELETE`, `CREATE`, `DROP`, `ALTER`, `TRUNCATE`, `GRANT`, `REVOKE`, `LOCK`, `CALL`, `DO`. + +Transaction keywords: `BEGIN`, `START` → `TransactionBegin`; `COMMIT`, `ROLLBACK` → `TransactionEnd`; `SAVEPOINT`, `RELEASE`, `SET` → `Transaction`. -Transaction keywords: `BEGIN`, `START` → `TransactionBegin`; `COMMIT`, `ROLLBACK`, `SAVEPOINT`, `RELEASE` → `TransactionEnd`. +Anything else — including `RENAME`, `REPLACE`, `LOAD`, `MERGE`, and `EXECUTE` — falls through to `Type::Unknown`. -Special handling: `COPY` is classified based on direction (`FROM STDIN` = Write, `TO STDOUT` = Read). `SET` is classified as `TransactionEnd` (session configuration). +Special handling: `COPY` is classified based on direction (`FROM STDIN` = Write, `TO STDOUT` = Read). ### MySQL Parser @@ -2526,7 +2743,7 @@ $parser = new MySQL(); $type = $parser->parse($rawPacketData); // Type::Read, Write, TransactionBegin, etc. ``` -Recognizes MySQL command bytes including `COM_QUERY` (classifies via SQL text), `COM_STMT_PREPARE`, `COM_STMT_EXECUTE`, `COM_INIT_DB`, `COM_QUIT`, and others. +Recognizes `COM_QUERY` (`0x03`, classified via its SQL text), `COM_STMT_PREPARE` (`0x16`), `COM_STMT_EXECUTE` (`0x17`), `COM_STMT_SEND_LONG_DATA` (`0x18`), `COM_STMT_CLOSE` (`0x19`), and `COM_STMT_RESET` (`0x1A`). The prepared-statement commands are routed to the primary. ### PostgreSQL Parser @@ -2539,7 +2756,7 @@ $parser = new PostgreSQL(); $type = $parser->parse($rawMessageData); // Type::Read, Write, TransactionBegin, etc. ``` -Handles message types including `Q` (simple query), `P` (parse/prepared statement), `X` (terminate), and startup messages. +Handles message types `Q` (simple query, classified via its SQL text), `P` (parse), `B` (bind), and `E` (execute). Other message types, including terminate and startup messages, return `Type::Unknown`. ### MongoDB Parser @@ -2590,17 +2807,18 @@ This is the pattern used by [utopia-php/database](https://github.com/utopia-php/ All code contributions should go through a pull request and be approved by a core developer before being merged. ```bash -composer install # Install dependencies -composer test # Run tests -composer lint # Check formatting -composer format # Auto-format code -composer check # Run static analysis (PHPStan level max) +composer install # Install dependencies +composer test # Run unit tests in parallel (excludes the performance group) +composer test:performance # Run the performance group +composer lint # Check formatting +composer format # Auto-format code +composer check # Run static analysis (PHPStan level max) ``` -**Integration tests** require Docker: +**Integration tests** require Docker. The compose file brings up MySQL, MariaDB, PostgreSQL (pgvector), ClickHouse, and MongoDB; SQLite runs in-memory with no container. Connection details are fixed in `tests/Integration/IntegrationTestCase.php`, so the containers must be up or the suite fails rather than skips: ```bash -docker compose -f docker-compose.test.yml up -d # Start MySQL, PostgreSQL, ClickHouse +docker compose -f docker-compose.test.yml up -d # Start the database containers composer test:integration # Run integration tests docker compose -f docker-compose.test.yml down # Stop containers ```