Repository navigation
FIX: bind executemany money-range Decimals as SQL_NUMERIC - #752
VyrnSynx (vyrnsynx) wants to merge 18 commits into
Conversation
executemany auto-detect skipped the money-range VARCHAR shortcut by deriving a batch-wide NUMERIC precision/scale, matching execute() so comparisons against smaller numeric columns no longer overflow. setinputsizes DECIMAL/NUMERIC string binding is unchanged. Fixes microsoft#745
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
There was a problem hiding this comment.
🟡 Changes recommended
The new batch precision/scale path needs stricter validation and the current executemany numeric override still conflates numeric precision with string buffer sizing in a way that can produce invalid SQL Server precision (>38) for some Decimal shapes.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR fixes the remaining executemany() auto-detection path where MONEY/SMALLMONEY-range Decimal values could still be bound as SQL_VARCHAR, causing SQL Server to overflow when comparing against smaller numeric/decimal columns. The change aligns executemany() with the already-fixed execute() behavior by binding auto-detected Decimal columns as SQL_NUMERIC using a batch-wide precision/scale, while still sending values as SQL_C_CHAR strings.
Changes:
- Update
executemany()auto-detect to bindDecimalcolumns asSQL_NUMERIC(skipping the MONEY-range VARCHAR shortcut) and compute a batch-wide(precision, scale). - Add unit/integration tests covering the GH-745 overflow regression and mixed-sign batch behavior.
- Document the behavior change in the changelog and update explanatory test/module docs.
File summaries
| File | Description |
|---|---|
mssql_python/cursor.py |
Adds batch-wide Decimal precision/scale derivation and applies it to executemany() auto-detect numeric binding. |
tests/test_020_money_smallmoney.py |
Adds DB integration coverage for GH-745 (no overflow on executemany comparisons; mixed-sign batch still works). |
tests/test_004_cursor.py |
Adds unit tests for batch precision/scale derivation and verifies executemany binds money-range Decimals as SQL_NUMERIC. |
CHANGELOG.md |
Records GH-745 fix and clarifies setinputsizes behavior remains unchanged. |
Review details
Suppressed comments (1)
mssql_python/cursor.py:2729
- executemany’s SQL_NUMERIC/SQL_DECIMAL override reuses ParamInfo.columnSize both as the numeric precision (passed as cbColDef to SQLBindParameter) and as the max SQL_C_CHAR buffer length. Setting columnSize to max_decimal_len (which includes sign and decimal point) can push the declared numeric precision above SQL Server’s max 38 (e.g., scale-38 values format to 40 chars), causing bind failures unrelated to the actual numeric precision/scale. Consider decoupling numeric precision from string buffer sizing (e.g., compute buffer sizes from actual encoded string lengths in BindParameterArray when SQLType is NUMERIC/DECIMAL but keep cbColDef=precision<=38).
# Ensure columnSize also accommodates the longest string form
# (mixed-sign batches, GH-557).
if max_decimal_len > paraminfo.columnSize:
paraminfo.columnSize = max_decimal_len
- Files reviewed: 4/4 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Sumit Sarabhai (sumitmsft)
left a comment
There was a problem hiding this comment.
Reviewed the change that routes executemany auto-detected Decimal columns to SQL_NUMERIC (GH-745). The core approach is sound and the new _decimal_sql_precision_scale / _batch_decimal_precision_scale helpers correctly derive a batch-wide precision/scale (and fix a latent sample-only sizing gap). Verified the precision/scale math matches _get_numeric_data, and black --check passes.
One blocking issue: the retained max_decimal_len override inside the NUMERIC block sets the NUMERIC precision from a formatted string length, which can push precision past 38 and break near-max-precision money-range Decimals that previously worked. Two smaller reliability points below.
Recommendation: Request changes.
| if batch_scale > paraminfo.decimalDigits: | ||
| paraminfo.decimalDigits = batch_scale | ||
| # Ensure columnSize also accommodates the longest string form | ||
| # (mixed-sign batches, GH-557). |
There was a problem hiding this comment.
For a SQL_NUMERIC param, columnSize is the numeric precision passed to SQLBindParameter (ddbc_bindings.cpp, 6th arg), not a string length. The max_decimal_len > columnSize override just below counts the decimal point and sign, so it sets precision to batch_precision + 1/2. Now that money-range Decimals flow into this NUMERIC block, a full-scale value like Decimal("1E-38") or Decimal("0."+"1"*38) becomes NUMERIC(40,38) and the driver rejects it (max precision 38), a regression from the prior VARCHAR path and inconsistent with the execute() fix (GH-740). batch_precision already sizes precision correctly; the string-length need is handled separately in the SQL_VARCHAR block below. Please drop this override in the NUMERIC branch and add an executemany test for Decimal("1E-38") into NUMERIC(38,38).
| # Ensure columnSize accommodates the longest string representation | ||
| # One NUMERIC(precision, scale) must fit every Decimal in the | ||
| # batch (GH-745). Sample-only precision/scale is not enough. | ||
| batch_precision, batch_scale = self._batch_decimal_precision_scale(column) |
There was a problem hiding this comment.
batch_precision is applied to columnSize without a <=38 check. The sample-level guard in _map_sql_type doesn't cover the whole batch, so a mixed batch (many integer digits in one row, many fractional in another) can produce batch_precision > 38 even when each value is individually storable, and the driver then fails with an opaque message. Consider raising the existing "precision too high" ValueError when batch_precision > 38. Don't clamp, since that would silently truncate.
| # against a smaller numeric column does not overflow. executemany still | ||
| # string-binds via SQL_C_CHAR below; setinputsizes DECIMAL stays on the | ||
| # GH-503 string path above. | ||
| decimal_as_numeric = isinstance(sample_value, decimal.Decimal) |
There was a problem hiding this comment.
Deriving decimal_as_numeric from just the sample type forces the whole column onto the NUMERIC/string-decimal path. A heterogeneous column (Decimal sample plus a stray non-numeric value) that previously bound as VARCHAR will now raise in the Decimal conversion loop below. Consider gating this on "all non-NULL values are Decimal," or documenting heterogeneous columns as unsupported on this path.
…inding Address review feedback on the executemany SQL_NUMERIC path: keep columnSize as numeric precision (not string length), raise when batch precision exceeds 38, force NUMERIC only when every non-NULL value is Decimal, and cover the cases with unit tests.
|
Sumit Sarabhai (@sumitmsft) Thanks for the review — addressed in the latest commit:
Also added unit coverage for near-max precision ( |
|
@microsoft-github-policy-service agree |
Resolve conflict in execute(): keep upstream native DDBCSQLExecute / DDBCSQLExecDirect path from microsoft#736/microsoft#758; retain microsoftGH-745 executemany SQL_NUMERIC batch precision/scale changes.
There was a problem hiding this comment.
🔵 Needs a closer look
It changes core executemany() parameter binding semantics in a DB driver, so a final human review is needed to confirm no subtle regressions across ODBC/SQL Server edge cases.
Review details
- Files reviewed: 4/4 changed files
- Comments generated: 2
- Review effort level: Lite
| non_null_values = [v for v in column if v is not None] | ||
| decimal_as_numeric = bool(non_null_values) and all( | ||
| isinstance(v, decimal.Decimal) for v in non_null_values | ||
| ) |
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
|
VyrnSynx (@vyrnsynx) - requesting you to please look into the failing tests, let us know if you need any help |
Gaurav Sharma (bewithgaurav)
left a comment
There was a problem hiding this comment.
requesting changes. the issue comparison still errors, while normal decimal batches that work on the merge base now fail before reaching sql server. I reproduced this locally, and the same regression appears in the linux and windows CI jobs.
| # One NUMERIC(precision, scale) must fit every Decimal in the | ||
| # batch (GH-745). Sample-only precision/scale is not enough. | ||
| # columnSize is NUMERIC precision for SQLBindParameter, not a | ||
| # string buffer length — do not widen it with max_decimal_len. |
There was a problem hiding this comment.
this uses sql numeric precision as the array's character-buffer width too. precision only counts digits, but the fixed-point string also needs space for the sign, decimal point, leading zero, and terminator. Decimal("1E-38") gets (38, 38) here but formats to 40 characters, so PyStringArrayDataBuffer rejects it before sql server sees it.
I ran the same cases on this head and the exact merge base. Decimal("1E-38") and [-12.34, 56.78] both complete on the base; this head raises RuntimeError: Input string exceeds allowed column size at parameter index 0. the linux and windows jobs show the same 13 decimal failures too.
can we keep sql precision and scale separate from the SQL_C_CHAR buffer stride, sizing the buffer from the longest encoded fixed-point value plus its terminator?
Keep SQLBindParameter columnSize as digit precision/scale, and size the
executemany CHAR array stride from the longest fixed-point encoding
(bufferSize) so values like Decimal("1E-38") and mixed-sign short
batches are not rejected by the array buffer.
|
Gaurav Sharma (@bewithgaurav) thanks for catching this — you were right that we were treating SQL numeric precision as the CHAR array width. Pushed a follow-up that keeps Added/updated unit coverage for those, plus an executemany |
There was a problem hiding this comment.
🔵 Needs a closer look
It changes core executemany type inference and native array-binding buffer sizing in C++, which is regression-prone and warrants final human review despite strong new test coverage.
Review details
Suppressed comments (2)
mssql_python/cursor.py:2632
executemany()builds a fullnon_null_valueslist just to decide whether all non-NULL values areDecimal. For large batches this doubles per-column memory and does extra work; a single-pass check avoids allocating another list.
non_null_values = [v for v in column if v is not None]
decimal_as_numeric = bool(non_null_values) and all(
isinstance(v, decimal.Decimal) for v in non_null_values
)
mssql_python/pybind/ddbc_bindings.cpp:2418
- This bounds check is against
dataWidth(array buffer stride), but the thrown message still refers to "column size". SincecolumnSizecan represent NUMERIC precision rather than string length, the message should reference buffer width to match the check.
if (encodedStr.size() > dataWidth) {
LOG("BindParameterArray: String/binary too "
"long - param_index=%d, row=%zu, size=%zu, "
"max=%zu",
paramIndex, i, encodedStr.size(), dataWidth);
ThrowStdException("Input exceeds column size at index " +
std::to_string(i));
- Files reviewed: 6/6 changed files
- Comments generated: 2
- Review effort level: Lite
| for row in seq_of_parameters: | ||
| value = row[col_index] | ||
| if value is None: | ||
| continue |
| if (wstr.length() > dataWidth) { | ||
| ThrowStdException("Input string exceeds allowed column size " | ||
| "at parameter index " + | ||
| std::to_string(paramIndex)); | ||
| } |
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
Gaurav Sharma (bewithgaurav)
left a comment
There was a problem hiding this comment.
the original decimal buffer issue is fixed. requesting changes for a new regression in how conversion errors are handled, detailed inline.
Provisional CHAR buffer sizing for setinputsizes DECIMAL/NUMERIC now only measures already-Decimal values. Non-Decimal conversion stays in the protected loop so MemoryError/RuntimeError become sanitized ValueError with row/column details; bufferSize is then widened from that converted fixed-point text.
There was a problem hiding this comment.
🟡 Changes recommended
Batch precision/scale derivation currently ignores non-Decimal values even though executemany later converts non-Decimal inputs for DECIMAL/NUMERIC columns, which can yield under-sized NUMERIC metadata and runtime errors for mixed-type batches.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (1)
mssql_python/cursor.py:2627
- In executemany() auto-detect,
non_null_values = [v for v in column if v is not None]allocates an extra list the size of the batch just to decide whether all non-NULLs are Decimals. For large batches this adds avoidable memory/time overhead; a single-pass scan can compute the same result without an intermediate list.
non_null_values = [v for v in column if v is not None]
decimal_as_numeric = bool(non_null_values) and all(
isinstance(v, decimal.Decimal) for v in non_null_values
)
- Files reviewed: 6/6 changed files
- Comments generated: 1
- Review effort level: Lite
There was a problem hiding this comment.
🟡 Changes recommended
A critical buffer-sizing issue and additional correctness and traceback issues remain unresolved.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (2)
mssql_python/cursor.py:2965
- The fetch methods previously used a bare
raisehere, butraise eadds this handler as a new traceback frame and changes the reported failure location. This unrelated change makes fetch/conversion failures harder to diagnose; preserve the original traceback with a bareraise(and apply the same correction infetchmany()andfetchall()).
except Exception as e:
# On error, don't increment rownumber - rethrow the error
raise e
mssql_python/cursor.py:2372
_compute_column_type()returns early when anyintis present (lines 2358-2362), so this new guard is never reached for a mixed column such as[1, Decimal("NaN")]. That column is then typed as an integer and the non-finite Decimal is passed through the integer binding instead of raising the promisedValueError, unlike the all-Decimal cases covered by the tests. Validate Decimal finiteness before the integer fast path (or in the conversion/type-selection path).
if not v.is_finite():
raise ValueError("Cannot bind non-finite Decimal (NaN/Infinity) as SQL NUMERIC")
- Files reviewed: 6/6 changed files
- Comments generated: 2
- Review effort level: Lite
| if max_encoded: | ||
| prior = getattr(ptype, "bufferSize", 0) or 0 | ||
| ptype.bufferSize = max(prior, max_encoded, 1) |
PR Performance ReportPerformance could not be assessed. No matching performance run appeared within the 220-minute wait for |
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
An unresolved moderate edge case can leak MemoryError during Decimal sizing.
Review effort: Lite
Findings: 1
Open (7)
bufferSizeis documented/consumed as a byte stride:BindParameterArraycompares it with… Usingraise ehere resets the traceback/context and can make debugging fetch failures harder. Use… In the post-conversion widening pass,decimal.Decimal(val)is only caught forDecimalException.… This check now enforces the per-row buffer width (dataWidth), but the thrown message still says… This overflow check now compares againstdataWidth(buffer width), but the exception still says… In thesetinputsizesDECIMAL/NUMERIC path,numeric_buffer_sizescansseq_of_parametersand… In executemany() auto-detect, building a fullnon_null_valueslist forces an extra allocation and…
Bring PR microsoft#752 up to date with microsoft/mssql-python main (through RELEASE: 1.16.0 / microsoft#840) so the branch is no longer behind.
There was a problem hiding this comment.
🟡 Changes recommended
Two moderate issues remain in cursor.py and should be addressed before approval.
8 open issues
bufferSizeis documented/consumed as a byte stride:BindParameterArraycompares it with… Avoid redundant Decimal parsing for explicit numeric input sizes · New Usingraise ehere resets the traceback/context and can make debugging fetch failures harder. Use… In the post-conversion widening pass,decimal.Decimal(val)is only caught forDecimalException.… This check now enforces the per-row buffer width (dataWidth), but the thrown message still says… This overflow check now compares againstdataWidth(buffer width), but the exception still says… In thesetinputsizesDECIMAL/NUMERIC path,numeric_buffer_sizescansseq_of_parametersand… In executemany() auto-detect, building a fullnon_null_valueslist forces an extra allocation and…
Review effort: Lite
| as_decimal = decimal.Decimal(val) | ||
| except decimal.DecimalException: | ||
| continue | ||
| if not as_decimal.is_finite(): | ||
| continue | ||
| precision, scale = self._decimal_sql_precision_scale(as_decimal) | ||
| found_numeric_text = True | ||
| max_scale = max(max_scale, scale) | ||
| max_int_digits = max(max_int_digits, precision - scale) |
…-decimal-sql-numeric # Conflicts: # mssql_python/cursor.py
There was a problem hiding this comment.
🟡 Changes recommended
A critical active issue remains because the new ParamInfo.bufferSize attribute is missing from the type stub, causing the strict typing check to fail.
9 open issues
Declare ParamInfo.bufferSize in the type stub · NewbufferSizeis documented/consumed as a byte stride:BindParameterArraycompares it with… Avoid redundant Decimal parsing for explicit numeric input sizes Usingraise ehere resets the traceback/context and can make debugging fetch failures harder. Use… In the post-conversion widening pass,decimal.Decimal(val)is only caught forDecimalException.… This check now enforces the per-row buffer width (dataWidth), but the thrown message still says… This overflow check now compares againstdataWidth(buffer width), but the exception still says… In thesetinputsizesDECIMAL/NUMERIC path,numeric_buffer_sizescansseq_of_parametersand… In executemany() auto-detect, building a fullnon_null_valueslist forces an extra allocation and…
Review effort: Lite
| .def_readwrite("paramCType", &ParamInfo::paramCType) | ||
| .def_readwrite("paramSQLType", &ParamInfo::paramSQLType) | ||
| .def_readwrite("columnSize", &ParamInfo::columnSize) | ||
| .def_readwrite("bufferSize", &ParamInfo::bufferSize) |
…-decimal-sql-numeric
There was a problem hiding this comment.
🟡 Changes recommended
The new ParamInfo.bufferSize assignments are missing from the native type stub, so the repository’s strict typing tests will fail until bufferSize is declared there.
10 open issues
Add bufferSize to the ParamInfo type stub · New Declare ParamInfo.bufferSize in the type stubbufferSizeis documented/consumed as a byte stride:BindParameterArraycompares it with… Avoid redundant Decimal parsing for explicit numeric input sizes Usingraise ehere resets the traceback/context and can make debugging fetch failures harder. Use… In the post-conversion widening pass,decimal.Decimal(val)is only caught forDecimalException.… This check now enforces the per-row buffer width (dataWidth), but the thrown message still says… This overflow check now compares againstdataWidth(buffer width), but the exception still says… In thesetinputsizesDECIMAL/NUMERIC path,numeric_buffer_sizescansseq_of_parametersand… In executemany() auto-detect, building a fullnon_null_valueslist forces an extra allocation and…
Review effort: Lite
| if numeric_buffer_size: | ||
| paraminfo.bufferSize = numeric_buffer_size |
| # Propagate non-finite errors; do not silently skip them. | ||
| precision, scale = self._decimal_sql_precision_scale(value) | ||
| found = True | ||
| max_scale = max(max_scale, scale) |
There was a problem hiding this comment.
Zero should contribute zero integer digits to the batch precision. Decimal("0") currently contributes one, so this valid batch is inferred as precision 39 and rejected:
cursor.execute("CREATE TABLE #decimal_boundary (v NUMERIC(38,38))")
cursor.executemany(
"INSERT INTO #decimal_boundary VALUES (?)",
[(Decimal("0"),), (Decimal("1E-38"),)],
)I reproduced this against separate Release builds of the exact merge base (52f30011) and HEAD (daf06929): base inserts both values losslessly; HEAD raises ValueError: ... maximum precision ... 38, but got 39. Reversing the rows or including NULL reproduces it too.
Please exclude zero's integer digits here and in the post-conversion aggregation around line 3018, while retaining the final minimum precision of 1. Both passes need the correction; otherwise the later pass reintroduces the rejection. Please add an end-to-end regression for this boundary.
|
VyrnSynx (@vyrnsynx) Thank you for raising this PR.. can you please resolve the comments that you have received |
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
Decimal("0") reported precision 1, so a batch with Decimal("1E-38")
was inferred as NUMERIC(39,38) and rejected. Zero has no integer
digits; keep the batch at NUMERIC(38,38). Apply the same rule in
the post-conversion widen pass.
There was a problem hiding this comment.
🔵 Needs a closer look
A critical active issue remains because the exposed ParamInfo.bufferSize property is missing from the native typing stub, causing strict typing failures and leaving typed consumers unable to use it.
10 open issues
Add bufferSize to the ParamInfo type stub Declare ParamInfo.bufferSize in the type stubbufferSizeis documented/consumed as a byte stride:BindParameterArraycompares it with… Avoid redundant Decimal parsing for explicit numeric input sizes Usingraise ehere resets the traceback/context and can make debugging fetch failures harder. Use… In the post-conversion widening pass,decimal.Decimal(val)is only caught forDecimalException.… This check now enforces the per-row buffer width (dataWidth), but the thrown message still says… This overflow check now compares againstdataWidth(buffer width), but the exception still says… In thesetinputsizesDECIMAL/NUMERIC path,numeric_buffer_sizescansseq_of_parametersand… In executemany() auto-detect, building a fullnon_null_valueslist forces an extra allocation and…
Review effort: Lite



Work Item / Issue Reference
Summary
executemanyauto-detect still used the MONEY/SMALLMONEY-range VARCHAR shortcut after #742 fixedexecute(). A money-rangeDecimalcompared against a smaller numeric column could still overflow on that path.This change binds auto-detected Decimal columns as
SQL_NUMERICwith a batch-wide precision/scale (values still go through the existingSQL_C_CHARstring conversion). ThesetinputsizesDECIMAL/NUMERIC string path (GH-503) and mixed-sign VARCHAR sizing (GH-557) are left alone.Fixes #745