Skip to content

testing.md

Latest commit

 

History

History
146 lines (115 loc) · 7.94 KB

testing.md

File metadata and controls

146 lines (115 loc) · 7.94 KB
Raw

Testing and release checks

The release target is the builder's supported API, not the entire ClickHouse SQL language. The standalone suite uses public package exports from a fresh build and executes deterministic SELECT/CTE fixtures. It does not create tables or write data.

Commands

Run these from the repository root:

bun run build
bun run typecheck
bun run test

test includes SQL generation, type/codec regressions, the dialect coverage manifest, and documentation checks. Live tests skip when no endpoint is configured. typecheck also checks the compile-time assertion files under src/, live fixtures under tests/, and TypeScript release scripts under scripts/; build first because the latter import public exports.

For live tests against an existing local server:

EFFECT_ORM_CLICKHOUSE_URL=http://127.0.0.1:8123 \
EFFECT_ORM_CLICKHOUSE_USER=maple \
EFFECT_ORM_CLICKHOUSE_PASSWORD=maple \
bun run test:clickhouse

test:clickhouse builds the package and requires an endpoint. Missing configuration, an unreachable server, a failed assertion, or a decoding error fails the command. The user defaults to default and the password to an empty string.

bun run test:package builds and packs the package, installs the tarball outside the workspace with its Effect peer, typechecks a consumer with strict declarations, and executes imports from the public entry points under Node. This needs npm registry access. It uses the installed Effect and TypeScript versions, with explicit Node, DOM, and disposable type libraries required by the Effect declarations.

bun run test:release requires the same ClickHouse environment and runs build, typecheck, all tests (including live tests), documentation checks, and the isolated tarball check. prepublishOnly runs it too, so publishing cannot silently skip live checks. Packing for inspection uses npm pack --ignore-scripts and never publishes.

What the manifest guarantees

tests/dialect-cases.ts is the fixture manifest. Each case names the features it covers and asserts complete decoded result rows. tests/dialect-coverage.test.ts discovers the function barrel, built query/union methods, and public type descriptors, and rejects missing coverage, stale entries, duplicate case IDs, and redundant exemptions. Add a case or a specific exemption when extending those APIs.

The inventory covers the wrapped ClickHouse functions (including /expr names), query and union methods, and built-in type descriptors. It is not a percentage of ClickHouse's grammar, every overload, every input combination, or every low-level expression/SQL factory. Those also have unit and consumer checks. Custom and untyped descriptors are explicit extension points: their factories are smoke-tested, but the caller's SQL type and schema remain the caller's responsibility.

Cases cover normal results plus selected empty inputs, nullable/nested values, 64-bit identity preservation through toString, parameter escaping, grouped windows, and joins against nullable unions. Each case runs under both quoted/unquoted 64-bit JSON output and default/nullable outer joins. The original publishing regression suite additionally checks unmatched joins under both settings and DateTime64 bounds. Fixtures pin the session timezone to UTC, matching the timestamp codecs' wire contract. Both JSON and JSONEachRow response formats are exercised.

One builder suite, every dialect

tests/core-cases.ts holds the builder cases every dialect runs: selection, the shared operators, params, grouping, joins on tables and subqueries, CTEs, unions, routing, and format. A case builds its query from a CoreContext, which supplies the dialect's column types, aggregate catalog and compile, and reads the same fixture rows on every database (a WITH over values(...) on ClickHouse and VALUES on Postgres, so nothing is written).

  • tests/core.clickhouse.test.ts runs them live, under both join_use_nulls settings.
  • tests/core.postgres.test.ts runs them on PGlite (Postgres 17) on every vitest run.
  • tests/core-sql.test.ts snapshots the exact SQL and parameters per dialect.

Where the databases genuinely disagree, the case says so: expectedBy gives a target its own rows (/ is integer division on Postgres; ClickHouse fills a missing join row with defaults unless join_use_nulls=1), and rejects names a dialect that must refuse to compile it (format on Postgres). A case a dialect cannot run yet goes in coreSkips with a reason. The core manifest in tests/dialect-coverage.test.ts requires every query and union method, expression and condition operator, and param kind to have a core case.

Postgres functions and types have their own manifest: every export of the ./postgres entry that is its own (not the shared query builder, which the ClickHouse manifest covers) is run by a case in tests/dialect-cases.postgres.ts (tests/dialect.postgres.test.ts) or exempted with a reason.

Tests preserve documented behavior: arithmetic chains follow SQL precedence, not call order, and windowFunnel with strict_order rejects intervening events.

Fuzzing

tests/fuzz/ generates random inputs with fast-check and checks them against oracles that do not trust the builder:

  • literals.fuzz.test.ts: a random value of each column type reaches the database twice. One copy goes through the database's own JSON parser as the reference, the other through the builder, both as a comparison literal and as a param in the SELECT list. The two must agree.
  • queries.fuzz.test.ts: random query plans (joins, nested conditions, grouping, having, ordering, paging, subqueries) run on the core fixture and must match model.ts, a JS evaluator with SQL three-valued logic. A plan that reports single-tenant must have read one org's rows only.
  • compile.fuzz.test.ts: needs no database. Valid plans always compile, compile deterministically, leak no undefined, NaN or __PARAM_, and number Postgres placeholders $1..$n. Any JS value placed where a value goes is compiled or refused with a QueryBuilderError, never thrown and never a defect.

bun run test runs a small budget (a few seconds). Use bun run fuzz for longer runs:

FUZZ_SCALE=50 bun run fuzz          # 50× every property's default run count
FUZZ_RUNS=10000 bun run fuzz -t "plans match the model"
FUZZ_VERBOSE=1 bun run fuzz         # print checked / skipped counts per property

A failure prints the shrunk input, the SQL it compiled to, the database's reply, and a FUZZ_SEED=… FUZZ_PATH=… command that replays it. Pin the input in tests/fuzz/regressions.ts, which is replayed before any random input.

Bugs found but not yet fixed live in tests/fuzz/known-issues.ts. Each entry recognises the inputs that reach its bug and gives a reason, so the suite stays green while the rest of the input space keeps being fuzzed. FUZZ_STRICT=<id> re-enables one entry's inputs, to check a fix before deleting the entry; FUZZ_STRICT=1 re-enables them all. Differences between the databases that are not bugs are encoded in the model instead, each with the difference it represents.

Version and CI policy

Every push to main and every pull request runs the complete release check against pinned 26.2.19.43 and 26.8.2.7 servers in .github/workflows/ci.yml. These are tested compatibility points; older versions are not established by this suite. The workflow is also manually dispatchable and reusable by a publishing workflow. It validates but does not publish. Its results apply only to the commit tested.

Upstream references

ClickHouse's stateless functional tests and testing guide are useful sources of additional edge cases. Port relevant operations into the DSL and retain provenance when copying fixtures. Executing upstream SQL unchanged would not validate the builder. The current small fixtures were written for this package.