You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 4ca3a67
Browse filesBrowse the repository at this point in the historyBrowse files
fix(postgres): harden COPY staging cleanup, JSON fallback, and CR quoting
- Wrap DataFrame replace_query, ctas, merge, and _scd_type_2 calls in a single transaction so COPY staging DDL, target writes, and cleanup commit or roll back atomically
- Protect COMMENT ON statements inside active transactions with a SAVEPOINT so swallowed comment permission/policy failures do not leave psycopg2 in PQTRANS_INERROR and silently roll back DataFrame loads on commit
- Track standalone staging table commits in _df_to_source_queries, roll back aborted 25P02 transactions before retrying DROP TABLE, and never mask propagating caller errors with secondary drop failures
- Fall back to VALUES when non-TEXT staging columns target JSON or JSONB while preserving COPY for TEXT JSON strings
- Use CRLF CSV row terminators in _copy_df so Python 3.9 csv.QUOTE_MINIMAL quotes bare carriage returns, and clarify staging table behavior in Postgres docs
Signed-off-by: Vamsi-klu <nrvamsi13@gmail.com>
When using the `psycopg2` driver, SQLMesh loads pandas DataFrames (such as those returned by Python models and used for seeds) with `COPY ... FROM STDIN (FORMAT csv)` instead of batched `INSERT ... VALUES` statements, which is much faster for large DataFrames. The data is copied into a temporary staging table, which is created in the same schema as the target table and then cast into the target table's column types. The staging table is dropped afterwards, including when the load fails. Because it is created in the target's schema, the connecting user needs the `CREATE` privilege on that schema, as for other SQLMesh DDL.
29
+
When using the `psycopg2` driver, SQLMesh loads pandas DataFrames (such as those returned by Python models and used for seeds) with `COPY ... FROM STDIN (FORMAT csv)` instead of batched `INSERT ... VALUES` statements, which is much faster for large DataFrames. The data is copied into a regular schema-qualified staging table (`__temp_...`, rather than a session-local PostgreSQL `TEMP` table) created in the same schema as the target table and then cast into the target table's column types. SQLMesh DataFrame loads run inside a transaction and drop the staging table at the end of the load (or roll back its creation if the load fails inside the transaction); if a staging query is run outside an enclosing transaction, default schema privileges apply while the table exists and an abrupt connection drop or process crash may leave a `__temp_*` table behind that operators can remove. Because the staging table is created in the target's schema, the connecting user needs the `CREATE` privilege on that schema, as for other SQLMesh DDL.
30
30
31
31
No configuration is required and there is no option to disable this behavior. SQLMesh automatically falls back to `INSERT ... VALUES` when:
32
32
33
33
- A column has a type or values that cannot be loaded with `COPY` identically. Supported columns are integers (up to 64 bits signed, up to 32 bits unsigned), `float64`, booleans, timestamps (with or without time zone), and strings. Other types, such as `float32`, 64-bit unsigned integers, `Decimal`, dates, bytes, dicts, lists, and categoricals, trigger the fallback.
34
-
- A column's staging type is incompatible with the target column's cast (for example, an integer or numeric DataFrame column targeting a `BOOLEAN`, `BIT`, or `OID`/`REG*` type).
34
+
- A column's staging type is incompatible with the target column's cast (for example, a non-text DataFrame column targeting `JSON` or `JSONB`, or an integer or numeric column targeting `BOOLEAN`, `BIT`, or `OID`/`REG*`).
35
35
- The DataFrame has a single column and one of its values is `\.`.
36
36
- The driver does not support `copy_expert` (for example, a driver other than `psycopg2`).
f"Identifier name '{name}' (length {name_bytes} bytes) exceeds {self.dialect.capitalize()}'s max identifier limit of {self.MAX_IDENTIFIER_LENGTH} bytes"
0 commit comments