Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
13147b2
FIX: bind executemany money-range Decimals as SQL_NUMERIC
vyrnsynx Sep 4, 2026
04a6ab7
FIX: drop NUMERIC max_decimal_len override and harden batch Decimal b…
vyrnsynx Sep 4, 2026
b196076
Merge branch 'main' into fix-745-executemany-decimal-sql-numeric
vyrnsynx Sep 5, 2026
09b2414
Merge branch 'main' into fix-745-executemany-decimal-sql-numeric
vyrnsynx Sep 8, 2026
977654f
FIX: size SQL_C_CHAR buffer separately from NUMERIC precision
vyrnsynx Sep 8, 2026
ef13ce4
FIX: size setinputsizes DECIMAL buffer via protected conversion
vyrnsynx Sep 8, 2026
763a8dd
FIX: widen executemany NUMERIC precision from converted text
vyrnsynx Sep 9, 2026
806541f
Merge branch 'main' into fix-745-executemany-decimal-sql-numeric
sumitmsft Sep 9, 2026
ac69e51
FIX: black-format non-finite Decimal ValueError raise
vyrnsynx Sep 9, 2026
b5f8241
Merge branch 'main' into fix-745-executemany-decimal-sql-numeric
bewithgaurav Sep 10, 2026
107a78c
Merge branch 'main' into fix-745-executemany-decimal-sql-numeric
vyrnsynx Sep 10, 2026
b921205
Merge branch 'main' into fix-745-executemany-decimal-sql-numeric
vyrnsynx Sep 13, 2026
b6c383c
Merge branch 'main' into fix-745-executemany-decimal-sql-numeric
vyrnsynx Oct 6, 2026
e52d849
DOC: drop stale #745 note from GH-740 changelog entry
vyrnsynx Oct 6, 2026
4235f4f
Merge branch 'main' into fix-745-executemany-decimal-sql-numeric
vyrnsynx Oct 17, 2026
11b2159
Merge remote-tracking branch 'upstream/main' into fix-745-executemany…
vyrnsynx Nov 19, 2026
daf0692
Merge remote-tracking branch 'upstream/main' into fix-745-executemany…
vyrnsynx Nov 20, 2026
5baee3c
Count Decimal 0 as zero integer digits in executemany batches
vyrnsynx Nov 21, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 14 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,19 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
decoding on Linux/macOS. Existing platform-specific decode-error fallbacks
remain unchanged. Actual MAX/LOB text decoding
is unchanged; its existing BOM and trailing-NUL loss is not fixed here.
- **GH-745:** `executemany` auto-detect binds money-range `Decimal`
values as `SQL_NUMERIC` with a batch-wide precision/scale (still via
`SQL_C_CHAR` string values), so a comparison against a smaller numeric
column no longer overflows. SQL precision/scale stay on `columnSize` /
`decimalDigits`; the CHAR array stride uses a separate `bufferSize`
sized from the longest fixed-point encoding (e.g. `Decimal("1E-38")`),
so near-max precision values are not rejected by the array buffer.
Zero contributes no integer digits to the batch precision, so
`Decimal("0")` plus `Decimal("1E-38")` stays `NUMERIC(38,38)`.
The `setinputsizes` DECIMAL/NUMERIC string path uses the same buffer-width
split; buffer width is derived from text produced by the protected
conversion path so failed conversions still raise a sanitized `ValueError`
with row/column details (no raw MemoryError/RuntimeError leakage).
Comment thread
Copilot marked this conversation as resolved.
Resolved
- **GH-769:** Corrected 11 `GetInfoConstants` IDs for scalar functions, outer
joins, driver handles, cursor attributes, catalog support, and parameter
descriptions. Added the ODBC name `SQL_TIMEDATE_FUNCTIONS` as an alias of
Expand Down Expand Up @@ -179,8 +192,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
`sql_variant` stores them as `numeric`, and because `numeric` outranks
`money`/`varchar` in data-type precedence, `WHERE money_or_varchar_col = ?` can
add a `CONVERT_IMPLICIT` on the column side that turns an index seek into a scan.
`executemany` intentionally keeps its batch `VARCHAR` string binding (GH-503);
the remaining money-range case there is tracked in #745.
The `executemany` auto-detect path is covered separately by GH-745.
- **GH-725:** The `timeout` parameter of `connect()` / `Connection(...)` now
correctly sets the **login (connection-attempt) timeout**
(`SQL_ATTR_LOGIN_TIMEOUT`), matching pyodbc and its own docstring. Previously
Expand Down
196 changes: 183 additions & 13 deletions mssql_python/cursor.py
Original file line number Diff line number Diff line change
Expand Up @@ -769,9 +769,9 @@ def _map_sql_type( # pylint: disable=too-many-arguments,too-many-positional-arg
- i: The index of the parameter in the list.
- decimal_as_numeric: When True, bind a Decimal as SQL_NUMERIC regardless of
value, skipping the MONEY/SMALLMONEY-range VARCHAR shortcut. The execute()
path sets this so a money-range Decimal compared against a numeric column
does not overflow (GH-740). executemany() leaves it False because it
string-binds Decimals for the whole batch (GH-503).
path and executemany() auto-detect path set this so a money-range Decimal
compared against a numeric column does not overflow (GH-740, GH-745).
setinputsizes DECIMAL/NUMERIC still string-binds (GH-503).
Returns:
- A tuple containing the SQL type, C type, column size, and decimal digits.
"""
Expand Down Expand Up @@ -907,11 +907,11 @@ def _map_sql_type( # pylint: disable=too-many-arguments,too-many-positional-arg
f"The maximum precision supported by SQL Server is 38, but got {precision}."
)

# Detect MONEY / SMALLMONEY range. Skipped on the execute() path
# (decimal_as_numeric=True), where a money-range Decimal must bind as
# SQL_NUMERIC so a comparison against a smaller numeric column returns no
# match instead of overflowing (GH-740). executemany keeps the VARCHAR
# shortcut because it string-binds Decimals for the batch (GH-503).
# Detect MONEY / SMALLMONEY range. Skipped when decimal_as_numeric=True
# (execute() and executemany auto-detect), where a money-range Decimal must
# bind as SQL_NUMERIC so a comparison against a smaller numeric column
# returns no match instead of overflowing (GH-740, GH-745). The
# setinputsizes DECIMAL path still string-binds (GH-503).
if not decimal_as_numeric and SMALLMONEY_MIN <= param <= SMALLMONEY_MAX:
logger.debug("_map_sql_type: DECIMAL -> SMALLMONEY - index=%d", i)
# smallmoney
Expand Down Expand Up @@ -2469,9 +2469,72 @@ def _transpose_rowwise_to_columnwise(

return columnwise, row_count

@staticmethod
def _decimal_sql_precision_scale(value: decimal.Decimal) -> Tuple[int, int]:
"""Return SQL NUMERIC (precision, scale) for a finite Decimal.

Matches the precision/scale rules used by _map_sql_type / _get_numeric_data.
"""
decimal_as_tuple = value.as_tuple()
digits_tuple = decimal_as_tuple.digits
num_digits = len(digits_tuple)
exponent = decimal_as_tuple.exponent
if isinstance(exponent, str):
raise ValueError("Cannot bind non-finite Decimal (NaN/Infinity) as SQL NUMERIC")
Comment thread
jahnvi480 marked this conversation as resolved.
Resolved
if exponent >= 0:
precision = num_digits + exponent
scale = 0
elif (-1 * exponent) <= num_digits:
precision = num_digits
scale = exponent * -1
else:
precision = exponent * -1
scale = exponent * -1
return precision, scale

@staticmethod
def _decimal_batch_int_digits(
value: decimal.Decimal, precision: int, scale: int
) -> int:
"""Integer digits this Decimal contributes to a batch NUMERIC type.

``Decimal("0")`` reports precision 1, but zero has no integer digits and
must not push a high-scale sibling (``Decimal("1E-38")``) over precision 38.
"""
if value.is_zero():
return 0
return precision - scale

def _batch_decimal_precision_scale(self, column: Sequence[Any]) -> Tuple[int, int]:
"""Derive one NUMERIC(precision, scale) that fits every Decimal in a column.

Used by executemany so money-range Decimals can bind as SQL_NUMERIC with a
single batch-wide type (GH-745) without shrinking any row's digits.

Non-finite Decimals (NaN/Infinity) raise ValueError rather than being
skipped. Callers must enforce SQL Server's precision limit (<= 38).
"""
max_scale = 0
max_int_digits = 0
found = False
for value in column:
if not isinstance(value, decimal.Decimal):
continue
# Propagate non-finite errors; do not silently skip them.
precision, scale = self._decimal_sql_precision_scale(value)
found = True
max_scale = max(max_scale, scale)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Zero should contribute zero integer digits to the batch precision. Decimal("0") currently contributes one, so this valid batch is inferred as precision 39 and rejected:

cursor.execute("CREATE TABLE #decimal_boundary (v NUMERIC(38,38))")
cursor.executemany(
    "INSERT INTO #decimal_boundary VALUES (?)",
    [(Decimal("0"),), (Decimal("1E-38"),)],
)

I reproduced this against separate Release builds of the exact merge base (52f30011) and HEAD (daf06929): base inserts both values losslessly; HEAD raises ValueError: ... maximum precision ... 38, but got 39. Reversing the rows or including NULL reproduces it too.

Please exclude zero's integer digits here and in the post-conversion aggregation around line 3018, while retaining the final minimum precision of 1. Both passes need the correction; otherwise the later pass reintroduces the rejection. Please add an end-to-end regression for this boundary.

max_int_digits = max(
max_int_digits, self._decimal_batch_int_digits(value, precision, scale)
)
if not found:
return 0, 0
return max(max_int_digits + max_scale, 1), max_scale

def _compute_column_type(
self, column: Sequence[Any]
) -> tuple[Any, int | None, int | None, int]:

"""
Determine representative value and integer min/max for a column.

Expand Down Expand Up @@ -2509,6 +2572,11 @@ def _compute_column_type(
max_decimal_formatted_len = 0
for v in non_nulls:
if isinstance(v, decimal.Decimal):
# Non-finite Decimals have a string exponent ('n'/'N'/'F'); comparing
# that to int raises TypeError before the NUMERIC ValueError path.
# Reject early with the same message used by _decimal_sql_precision_scale.
if not v.is_finite():
raise ValueError("Cannot bind non-finite Decimal (NaN/Infinity) as SQL NUMERIC")
max_decimal_formatted_len = max(max_decimal_formatted_len, len(format(v, "f")))
if not sample_value:
sample_value = v
Expand All @@ -2524,7 +2592,7 @@ def _compute_column_type(
# If length comparison fails, keep the current sample_value
pass
elif isinstance(v, decimal.Decimal) and isinstance(sample_value, decimal.Decimal):
# For Decimal objects, prefer the one that requires higher precision or scale
# Both values are finite (checked above). Prefer higher precision/scale.
v_tuple = v.as_tuple()
sample_tuple = sample_value.as_tuple()
v_exponent = cast(int, v_tuple.exponent)
Expand Down Expand Up @@ -2669,6 +2737,9 @@ def executemany( # pylint: disable=too-many-locals,too-many-branches,too-many-s
)

# Prepare parameter type information
# Columns configured via setinputsizes keep declared columnSize/decimalDigits
# through the post-conversion widen pass (bufferSize may still grow).
explicit_inputsize_cols = set()
with perf_phase("py::executemany::param_type_detection"):
for col_index in range(param_count):
column = (
Expand All @@ -2680,6 +2751,7 @@ def executemany( # pylint: disable=too-many-locals,too-many-branches,too-many-s

if self._inputsizes and col_index < len(self._inputsizes):
# Use explicitly set input sizes
explicit_inputsize_cols.add(col_index)
sql_type, c_type, column_size, decimal_digits = self._inputsizes[col_index]

# Default is_dae to False
Expand All @@ -2700,12 +2772,25 @@ def executemany( # pylint: disable=too-many-locals,too-many-branches,too-many-s
is_dae = True

# Sanitize precision/scale for numeric types
numeric_buffer_size = 0
if sql_type in (
ddbc_sql_const.SQL_DECIMAL.value,
ddbc_sql_const.SQL_NUMERIC.value,
):
column_size = max(1, min(int(column_size) if column_size > 0 else 18, 38))
decimal_digits = min(max(0, decimal_digits), column_size)
# Provisional SQL_C_CHAR stride: size only from values that
# are already Decimal. Do NOT convert non-Decimals here —
# that bypasses the protected conversion loop below and can
# leak MemoryError/RuntimeError (and value-bearing messages).
# After conversion, bufferSize is widened from the produced
# fixed-point text (same path that sanitizes failures).
max_encoded = 0
for row in seq_of_parameters:
value = row[col_index]
if isinstance(value, decimal.Decimal):
max_encoded = max(max_encoded, len(format(value, "f")))
numeric_buffer_size = max(max_encoded, column_size + 3, 1)

# For binary data columns with mixed content, we need to find max size
if sql_type in (
Expand Down Expand Up @@ -2736,6 +2821,8 @@ def executemany( # pylint: disable=too-many-locals,too-many-branches,too-many-s
paraminfo.columnSize = column_size
paraminfo.decimalDigits = decimal_digits
paraminfo.isDAE = is_dae
if numeric_buffer_size:
paraminfo.bufferSize = numeric_buffer_size
Comment on lines +2824 to +2825

# Ensure we never have SQL_C_DEFAULT (0) for C-type
if paraminfo.paramCType == 0:
Expand All @@ -2753,6 +2840,18 @@ def executemany( # pylint: disable=too-many-locals,too-many-branches,too-many-s
column
)

# GH-745: auto-detected Decimal columns bind as SQL_NUMERIC (skipping
# the money-range VARCHAR shortcut) so a money-range value compared
# against a smaller numeric column does not overflow. executemany still
# string-binds via SQL_C_CHAR below; setinputsizes DECIMAL stays on the
# GH-503 string path above.
# Only force NUMERIC when every non-NULL value in the column is Decimal;
# a heterogeneous column keeps the prior sample-driven path.
non_null_values = [v for v in column if v is not None]
decimal_as_numeric = bool(non_null_values) and all(
isinstance(v, decimal.Decimal) for v in non_null_values
)

dummy_row = list(sample_row)
paraminfo = self._create_parameter_types_list(
sample_value,
Expand All @@ -2761,6 +2860,7 @@ def executemany( # pylint: disable=too-many-locals,too-many-branches,too-many-s
col_index,
min_val=min_val,
max_val=max_val,
decimal_as_numeric=decimal_as_numeric,
)

# GH-610: all-NULL columns now pass SQL_UNKNOWN_TYPE to C++,
Expand All @@ -2778,9 +2878,26 @@ def executemany( # pylint: disable=too-many-locals,too-many-branches,too-many-s
ddbc_sql_const.SQL_NUMERIC.value,
):
paraminfo.paramCType = ddbc_sql_const.SQL_C_CHAR.value
# Ensure columnSize accommodates the longest string representation
if max_decimal_len > paraminfo.columnSize:
paraminfo.columnSize = max_decimal_len
# One NUMERIC(precision, scale) must fit every Decimal in the
# batch (GH-745). Sample-only precision/scale is not enough.
# columnSize is NUMERIC precision for SQLBindParameter, not a
# string buffer length — do not widen it with max_decimal_len.
batch_precision, batch_scale = self._batch_decimal_precision_scale(column)
if batch_precision > 38:
raise ValueError(
"Precision of the numeric value is too high. "
"The maximum precision supported by SQL Server is 38, "
f"but got {batch_precision}."
)
if batch_precision > paraminfo.columnSize:
paraminfo.columnSize = batch_precision
if batch_scale > paraminfo.decimalDigits:
paraminfo.decimalDigits = batch_scale
# SQL_C_CHAR array stride is separate from SQL precision.
# Fixed-point strings need room for sign, '.', and a leading
# zero (e.g. Decimal("1E-38") -> 40 chars with precision 38).
# Size from the longest encoded value in the batch.
paraminfo.bufferSize = max(max_decimal_len, 1)

# Correct column size for Decimal columns sent as SQL_VARCHAR (GH-557).
# The sample value's formatted string may be shorter than another
Expand Down Expand Up @@ -2880,8 +2997,61 @@ def executemany( # pylint: disable=too-many-locals,too-many-branches,too-many-s
raise ValueError(err_msg) from None
processed_parameters.append(processed_row)

# Now transpose the processed parameters
with perf_phase("py::executemany::param_processing"):
# Derive/widen SQL_C_CHAR bufferSize and (for auto-detect only) SQL
# NUMERIC precision/scale from text produced by the protected conversion.
# setinputsizes previously sized by converting independently (leaking raw
# MemoryError/RuntimeError); the auto-detect path already had a
# Decimal-only provisional size. Post-conversion strings are authoritative
# for precision on auto-detect (e.g. Decimal("1e15.00") + "2e16"). Explicit
# setinputsizes columnSize/decimalDigits stay as declared; bufferSize still
# grows so the CHAR array fits.
for col_index, ptype in enumerate(parameters_type):
if ptype.paramSQLType not in (
ddbc_sql_const.SQL_DECIMAL.value,
ddbc_sql_const.SQL_NUMERIC.value,
):
continue
max_encoded = 0
max_scale = 0
max_int_digits = 0
found_numeric_text = False
for row in processed_parameters:
val = row[col_index]
if not isinstance(val, str):
continue
max_encoded = max(max_encoded, len(val))
try:
as_decimal = decimal.Decimal(val)
except decimal.DecimalException:
continue
if not as_decimal.is_finite():
continue
precision, scale = self._decimal_sql_precision_scale(as_decimal)
found_numeric_text = True
max_scale = max(max_scale, scale)
max_int_digits = max(
max_int_digits,
self._decimal_batch_int_digits(as_decimal, precision, scale),
)
if max_encoded:
prior = getattr(ptype, "bufferSize", 0) or 0
ptype.bufferSize = max(prior, max_encoded, 1)
Comment on lines +3037 to +3039
# Honor explicit setinputsizes precision/scale; only auto-detect widens.
if found_numeric_text and col_index not in explicit_inputsize_cols:
batch_precision = max(max_int_digits + max_scale, 1)
if batch_precision > 38:
raise ValueError(
"Precision of the numeric value is too high. "
"The maximum precision supported by SQL Server is 38, "
f"but got {batch_precision}."
)
if batch_precision > ptype.columnSize:
ptype.columnSize = batch_precision
if max_scale > ptype.decimalDigits:
ptype.decimalDigits = max_scale

# Now transpose the processed parameters
columnwise_params, row_count = self._transpose_rowwise_to_columnwise(
processed_parameters
)
Expand Down
Loading