Skip to content

perf: harden runtime guards, expand mypyc surface, and optimize PGO builds - #921

Merged
cofin merged 28 commits into
mainfrom
feat/closeout-for-release
Nov 17, 2026
Merged

cofin merged 28 commits into
mainfrom
feat/closeout-for-release

Conversation

@cofin

@cofin cofin commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Compiles 39 more modules with mypyc (234 to 273, including every adapter driver), speeds up the statement and driver hot paths, and tightens migration, config-discovery and storage handling. It also fixes the behaviour differences that compiling the drivers exposed, so a compiled wheel behaves the same as a source install.

What changed

  • More of the library is compiled. All 24 adapter drivers, the session contexts, routing helpers, framework extension helpers, and the Snowflake and Teradata migration runners join the mypyc build.
  • Compiled wheels match source installs. Fixed cases that only went wrong when compiled:
    • migrations ran without a lock, and a deferred session could not report its dialect;
    • async drivers could hang on a failed or cancelled statement;
    • asyncpg, DuckDB and Spanner rejected list or tuple parameter sequences;
    • raise ... from ... lost the original error. Raise sites in compiled modules now attach the cause themselves, through a small with_cause() helper in sqlspec.exceptions.
  • Faster hot paths. Statement caching, a fast path for positional parameters, cheaper result materialization, and a leaner session lifecycle across adapters. In a compiled wheel, a 10,000-row single-statement SQLite insert loop takes about 21 ms through SQLSpec, down from about 60 ms.
  • Safer defaults.
    • Migration tracking tables, versions and loaders are validated more strictly. On PostgreSQL the tracking table is refused when it has rewrite rules or row-level security that applies to the migrating role.
    • Config and project-root discovery check who owns the files and directories they read.
    • FSSpecBackend treats *, ? and [ in keys literally on copy, move and delete.
    • aiomysql and asyncmy turn off multi-statement execution on each connection.
    • Lifecycle hooks registered after startup now fire.
  • Linux wheel builds. Profile-guided builds keep the hardening compiler flags, add link-time optimization, and train on 13 workloads instead of a SQLite-only run.
  • Compiled wheels are checked more thoroughly in CI. The wheel test installs the driver SDKs, imports and constructs every compiled driver, and checks compiled-only behaviour such as exception causes. Python 3.12 is added to the release wheel test matrix.
  • Benchmarks. The SQLite and subsystem performance baselines are refreshed and now include the declared-parameter scenario.

How you use it

Most of this needs no action. A few behaviours change:

  • MySQL (aiomysql, asyncmy): execute() now rejects stacked statements. Use execute_script() for multi-statement SQL, or opt back in with connection_config={"client_flag": CLIENT.MULTI_STATEMENTS}.
  • Litestar: get_annotation, provide_db_session and provide_db_connection raise KeyError when a config class is shared by several registered configurations. Pass the configuration instance or its session, connection or pool key.
  • BigQuery in framework plugins: configurations use commit_mode="manual", and an explicit autocommit mode raises ImproperConfigurationError at plugin creation. Call commit() in the handler to keep work done inside an explicit transaction.
  • Object storage keys: code that relied on FSSpecBackend expanding glob characters in copy, move or delete must list the matching keys first.

Docs

  • docs/changelog.md describes the behaviour changes above.
  • docs/databases/mysql.md documents the multi-statement default for aiomysql and asyncmy.
  • docs/learn/migrations.md updates the tracking-table and schema-extraction rules.

@codecov-commenter

codecov-commenter commented Nov 17, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 83.96816% with 564 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.63%. Comparing base (c2886c3) to head (7927ee5).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
sqlspec/adapters/oracledb/driver.py 77.98% 55 Missing and 15 partials ⚠️
sqlspec/driver/_routing.py 51.16% 51 Missing and 12 partials ⚠️
sqlspec/core/result/_base.py 75.52% 35 Missing and 12 partials ⚠️
sqlspec/utils/config_tools.py 78.68% 24 Missing and 2 partials ⚠️
sqlspec/migrations/loaders.py 76.04% 22 Missing and 1 partial ⚠️
sqlspec/adapters/asyncpg/driver.py 76.34% 17 Missing and 5 partials ⚠️
sqlspec/adapters/psycopg/driver.py 87.26% 18 Missing and 2 partials ⚠️
sqlspec/core/cache.py 81.30% 16 Missing and 4 partials ⚠️
sqlspec/observability/_common.py 29.16% 14 Missing and 3 partials ⚠️
sqlspec/adapters/cockroach_psycopg/driver.py 78.68% 9 Missing and 4 partials ⚠️
... and 66 more
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #921      +/-   ##
==========================================
- Coverage   84.66%   84.63%   -0.04%     
==========================================
  Files         628      629       +1     
  Lines      140808   142408    +1600     
  Branches    30685    30807     +122     
==========================================
+ Hits       119219   120529    +1310     
- Misses      14999    15244     +245     
- Partials     6590     6635      +45     
Flag Coverage Δ
cloud-adbc-storage 29.80% <35.73%> (+0.06%) ⬆️
integration 56.64% <61.17%> (+0.03%) ⬆️
mssql-db2 30.72% <35.73%> (+0.05%) ⬆️
mysql-mariadb-embedded 38.06% <46.93%> (+0.10%) ⬆️
oracle 30.11% <38.06%> (+0.11%) ⬆️
postgres-cockroach 39.78% <49.03%> (+0.05%) ⬆️
py3.11 84.47% <83.93%> (-0.04%) ⬇️
py3.12 84.48% <83.93%> (-0.04%) ⬇️
py3.13 84.31% <83.93%> (-0.20%) ⬇️
py3.14 84.22% <83.85%> (-0.03%) ⬇️
unit 81.91% <81.80%> (+0.05%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
sqlspec/adapters/adbc/_typing.py 90.08% <100.00%> (+0.33%) ⬆️
sqlspec/adapters/adbc/adk/store.py 77.54% <100.00%> (ø)
sqlspec/adapters/adbc/pool.py 97.84% <100.00%> (ø)
sqlspec/adapters/adbc/type_converter.py 100.00% <100.00%> (ø)
sqlspec/adapters/aiomysql/_typing.py 97.67% <100.00%> (+2.80%) ⬆️
sqlspec/adapters/aiomysql/adk/store.py 87.19% <100.00%> (+0.03%) ⬆️
sqlspec/adapters/aiomysql/core.py 80.73% <100.00%> (+0.69%) ⬆️
sqlspec/adapters/aiosqlite/_typing.py 96.29% <100.00%> (+3.56%) ⬆️
sqlspec/adapters/arrow_odbc/core.py 89.36% <100.00%> (+0.02%) ⬆️
sqlspec/adapters/arrow_odbc/litestar/store.py 86.32% <100.00%> (ø)
... and 207 more

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

cofin added 22 commits November 17, 2026 04:50
…t in compiled builds

Compiled driver classes store dialect on the instance, so reading it from the class returned a descriptor: migrations ran without a lock and a deferred request session raised on dialect before its connection opened. The lock backend now reads the instance value for driver adapters, and the deferred session falls back to the configuration's statement config.

The PostgreSQL tracking-table guard now rejects row-level security only when it applies to the migrating role.
…cancelled statements

In compiled builds, an async with block whose exit suspends left the task pending forever when the body failed through a future or was cancelled. The affected sites in the asyncpg, psycopg, Db2, Oracle, Spanner, mysql-connector, aiomysql, and asyncmy drivers now enter the context directly and exit it outside any exception handler.

Also:
- psycopg execute_stack accepts list parameters in compiled builds
- Cockroach async retries keep the driver error as the cause when the rollback also fails
- asyncpg, Cockroach asyncpg, and Db2 mapped exceptions set their cause explicitly
- asyncpg releases a failed transaction claim on pooled connections
- MySQL drivers accept tuple fetch results, and aiomysql maps only client-raised TypeErrors
- psycopg skips percent escaping work for SQL without pyformat placeholders and reads the cursor description once
…and DuckDB overwrite loads

- aiosqlite load_from_arrow and load_from_storage raise the mapped error instead of hanging in compiled builds
- the SQLite and aiosqlite rowid cache is keyed by table and bounded, so sqlcommenter comments no longer cause a probe per insert
- DuckDB load_from_arrow(overwrite=True) falls back to a positional insert when column names differ
- aiosqlite runs statements and begin() with one worker round trip instead of three
- Teradata rejects a boolean batch size in compiled builds
…estore the validator cache

- the cached cursor path converts tuple fetchall() results before row collection
- base driver errors set their cause explicitly, which compiled builds otherwise drop
- cached statements check observability live, so hooks registered on an open session fire
- building a statement cache holder no longer disables the shared parameter validator cache
…through world-writable ancestors

- pyproject.toml and project-root discovery no longer stop at a world-writable ancestor; trust is decided by marker ownership, and permission checks are skipped on platforms without POSIX ownership
- discovery logs why it stopped or refused a marker
- s3a, abfs, and abfss URIs resolve object keys without the bucket prefix
- fsspec copy, move, and delete treat glob characters in keys literally and fall back to rm() when rm_file() is not implemented
…time and name invalid settings

- session and connection lifecycle hooks registered after the runtime is built now fire
- Starlette, FastAPI, Flask, and Sanic validate routing settings and normalize status sets and flags before building compiled state, so invalid values raise the named configuration error
- verify_primary_pin returns a failed verification for a missing or non-string token in compiled builds
- the Prometheus observer caches label children
- the Litestar ambiguous config type error points at the configuration instance
…he wheel smoke test

- the smoke script checks the migration lock backend, deferred session dialect, failing and cancelled loads and stacks, psycopg stack parameters, and tuple fetch results, each bounded by a timeout
- Linux wheel tests install a hashed driver SDK profile and fail when a guaranteed driver module is skipped
- profile training fails on a skipped workload in strict mode and always fails on import errors raised by sqlspec itself
…t path and handle list-valued settings

- a sticky world-writable directory such as /tmp is never used as the migration project root, even when it holds a trusted marker
- recursive fsspec deletes of a literal key work on filesystems whose find() returns a list for a single file
- psycopg detects upper-case %S placeholders before skipping percent escaping
- framework status overrides given as a list or tuple are stored as a set
…nts without comments

Statements whose text ends with a comment are looked up by table only, so per-request sqlcommenter attributes neither probe per insert nor add cache entries, and every other insert keeps the single dictionary lookup.
… when hooks are registered

The cached-statement path reads a per-driver flag again instead of evaluating the runtime's idle state on every statement. The runtime counts hook registrations, and a driver recomputes its flag when the count changes, so hooks registered on an open session still fire.
…fore reading it

Discovery opens the file once, reads ownership and permissions from the open descriptor, and parses from the same handle, so a file swapped in after discovery is refused instead of read.
…t debug level

Running migrations as a different user from the one that owns the source
tree, as a container does with a mounted checkout, printed a warning for
every migration directory. The search already ended there silently before
refusals were logged; a refused marker still warns.
A compiled module does not apply the from clause of a raise statement, so
the error raised from a compiled wheel had no __cause__ and showed the
handled error as unrelated context. The sites that build the error inline
now attach the cause with with_cause(), and the sites that raise a local
error set __cause__ before raising, as the database error mappers already
do.
…in, commit, and rollback

sqlite3 tracks the flag itself and rejects the assignment, so every
transaction boundary raised and discarded an AttributeError.
The create-only checks wrote fixed keys, so they failed whenever the
bucket outlived the session that first wrote them.
…anager

Leaving the cursor context built and entered a contextlib.suppress object
for every statement. A plain try/except does the same cleanup for about
280 ns less per statement.
…of five runs

A single timing of each size failed on a loaded runner.
…heel smoke run

The wheel-test jobs now install the BigQuery, Spanner, mssql-python,
SQL Anywhere and Db2 SDKs so their driver modules are imported from the
built wheel instead of skipped. Db2 is installed on x86_64 only, where
ibm_db publishes a wheel, and sqlanydb is built from its hash-pinned
source distribution with a hash-pinned setuptools.

The smoke script constructs every adapter driver class around a
placeholder connection, accepts --require-sdk for SDKs required on a
single job, and checks that an error raised while handling another keeps
its direct cause or its suppression.
Recaptured interpreted at 1000 rows, pool size 1, seven iterations and
three warmups on Python 3.11. The SQLite baseline gains the raw and
SQLSpec declared_parameters rows and the subsystem baseline gains the
two declared-parameter check probes.
@cofin
cofin merged commit a4d10f0 into main Nov 17, 2026
46 of 47 checks passed
@cofin
cofin deleted the feat/closeout-for-release branch November 17, 2026 17:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants