Skip to content

[DO NOT MERGE] Enable ConnectionPoolV2 by default - #4537

Open
mdaigle wants to merge 4 commits into
dotnet:dev/automation/channel-pool-reclaim-timerfrom
mdaigle:mdaigle-jubilant-robot
Open

[DO NOT MERGE] Enable ConnectionPoolV2 by default#4537
mdaigle wants to merge 4 commits into
dotnet:dev/automation/channel-pool-reclaim-timerfrom
mdaigle:mdaigle-jubilant-robot

Conversation

@mdaigle

@mdaigle mdaigle commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

CI results summary

This PR flips UseConnectionPoolV2 to true so the full CI matrix runs against ChannelDbConnectionPool by default. That surfaced three differences:

  1. Emancipated connection reclamation: ReclaimEmancipatedOnOpenTest exposed missing reclaim-on-open behavior in V2. The fix is now supplied by this PR's base, Channel Pool: Reclaim leaked connections #4529.
  2. TVP packet-wraparound test: TvpTest.TestPacketNumberWraparound exposed a latent Task<Task> test bug. Commit 443b4f00e unwraps the task and propagates early failures correctly.
  3. Physical-open timeout errors: AmbientTransactionFailureTest exposed a V2 timeout race on Windows. Commit d80ca6845 preserves the physical connection error instead of replacing it with a pool-exhaustion timeout.

CI is rerunning with all three findings addressed. The title remains [DO NOT MERGE] until the matrix confirms the default flip is clean.

Summary

Changes the default connection pool from WaitHandleDbConnectionPool to ChannelDbConnectionPool. Applications can restore the legacy pool explicitly:

AppContext.SetSwitch("Switch.Microsoft.Data.SqlClient.UseConnectionPoolV2", false);

This PR is stacked on #4529 (dev/automation/channel-pool-reclaim-timer), which contains the V2 emancipated-connection reclamation fix discovered by the first CI run.

Changes

  • Changes UseConnectionPoolV2's default to true.
  • Updates the switch documentation and default-value unit test.
  • Fixes TvpTest.TestPacketNumberWraparound so Task.Factory.StartNew's nested task is unwrapped and observed.
  • Preserves physical connection failures when the caller's timeout expires immediately before V2 starts physical connection creation.
  • Adds a deterministic unit regression for the expired-timeout physical-open path.

Ambient transaction failure root cause

The failing Windows legs expected the invalid server open to throw SqlException, but V2 returned:

InvalidOperationException: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool.

ChannelDbConnectionPool.GetInternalConnection creates a cancellation token from the caller's TimeoutTimer. It previously passed that token into OpenNewInternalConnection, which checks cancellation before calling CreatePooledConnection. With the test's one-second timeout, Windows could consume the remaining budget before that check. V2 then converted the resulting OperationCanceledException into ADP.PooledOpenTimeout(), even though the pool was empty and no pool contention occurred.

V1 does not preempt physical creation this way. It passes the remaining or expired TimeoutTimer into CreatePooledConnection, allowing the physical-open failure to remain a SqlException.

The fix keeps the caller's timeout-derived cancellation token on actual pool waits, but does not use it to preempt an empty-pool physical connection attempt. Background warmup still passes its own cancellation token. The physical factory receives the original timeout budget, preserving V1 behavior without extending the timeout.

TVP test root cause

The test used Task.Factory.StartNew(async () => ...), which returns Task<Task>. Task.WhenAny observed only the outer task and could continue after the async delegate's first suspension, while the TVP operation was still running. V2's async timing made that latent test bug reproducible.

The test now uses a non-async delegate plus .Unwrap() and awaits the action task when it wins, so early OpenAsync failures propagate directly and the enumerator assertion runs only after the operation completes.

Performance comparison

An interleaved best-of-three comparison covered 162 benchmarks with a 10% threshold:

  • 41 improvements, primarily under higher-concurrency pool stress.
  • 9 confirmed regressions.
  • 3 unconfirmed regressions.

Confirmed regressions

Benchmark Method Parameters Baseline V2 default Delta
SqlConnectionRunner OpenAsyncConnection MARS=True, Pooling=True 0.0035 ms 0.0087 ms +145.23%
ConnectionPoolContentionRunner SteadyStateOpenQueryClose Parallelism=50, MaxPoolSize=10 40.2628 ms 98.2759 ms +144.09%
SqlConnectionRunner OpenAsyncConnection MARS=False, Pooling=True 0.0036 ms 0.0076 ms +108.42%
ConnectionPoolStressRunner RapidFireOpenClose Parallelism=20, MaxPoolSize=50 13.5875 ms 21.5105 ms +58.31%
ConnectionPoolStressRunner RapidFireOpenClose Parallelism=25, MaxPoolSize=100 15.4399 ms 23.5592 ms +52.59%
ConnectionPoolStressRunner RapidFireOpenClose Parallelism=25, MaxPoolSize=50 14.9726 ms 22.5136 ms +50.37%
ConnectionPoolStressRunner RapidFireOpenClose Parallelism=10, MaxPoolSize=50 11.4258 ms 16.3225 ms +42.86%
ConnectionPoolStressRunner RapidFireOpenClose Parallelism=20, MaxPoolSize=100 13.4755 ms 18.1328 ms +34.56%
ConnectionPoolStressRunner RapidFireOpenClose Parallelism=10, MaxPoolSize=100 13.1124 ms 16.1011 ms +22.79%

The strongest remaining performance signals are fixed per-call overhead in pooled OpenAsync, allocation and synchronization cost in rapid open/close loops, and contention when MaxPoolSize is small relative to concurrency. Higher-concurrency workloads with adequately sized pools generally improved.

Validation

  • ChannelDbConnectionPoolTest: 64/64 passed on net8.0, net9.0, and net10.0.
  • AmbientTransactionFailureTest: 2/2 passed locally on net8.0 and net9.0.
  • Earlier default-switch and connection-pool unit selection: 332/332 passed on net8.0, net9.0, and net10.0.
  • The seven CI legs affected by TvpTest.TestPacketNumberWraparound passed after commit 443b4f00e.

Suggested release note

Changed the default connection pool implementation to the Channel-based pool. Set Switch.Microsoft.Data.SqlClient.UseConnectionPoolV2 to false to restore the legacy pool.

Checklist

  • Tests added or updated
  • Internal switch documentation updated
  • No public API changes
  • Full CI matrix passes with V2 enabled by default

@mdaigle
mdaigle requested a review from a team as a code owner August 13, 2026 19:20
Copilot AI lite review requested due to automatic review settings August 13, 2026 19:20
@github-project-automation github-project-automation Bot moved this to To triage in SqlClient Board Aug 13, 2026
@mdaigle mdaigle added the DO NOT MERGE PRs that are created for test reasons, should not be merged. label Aug 13, 2026
@mdaigle mdaigle added this to the 8.0.0 milestone Aug 13, 2026
@mdaigle mdaigle changed the title Enable ConnectionPoolV2 by default [DO NOT MERGE] Enable ConnectionPoolV2 by default Aug 13, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR flips the default of the internal Switch.Microsoft.Data.SqlClient.UseConnectionPoolV2 AppContext switch to true, making the Channel-based pool (ChannelDbConnectionPool) the default connection pooling implementation while keeping the legacy pool (WaitHandleDbConnectionPool) available via explicit opt-out.

Changes:

  • Change LocalAppContextSwitches.UseConnectionPoolV2 default from falsetrue, and update its XML doc accordingly.
  • Update the unit test that asserts the default switch values.
  • Update the internal feature documentation table for the switch default.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/LocalAppContextSwitches.cs Flips UseConnectionPoolV2 default to true and updates XML documentation.
src/Microsoft.Data.SqlClient/tests/UnitTests/Microsoft/Data/SqlClient/LocalAppContextSwitchesTest.cs Updates default-value assertion to expect UseConnectionPoolV2 == true.
.github/instructions/features.instructions.md Updates the documented default for UseConnectionPoolV2 in the AppContext switches table.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines 255 to 256
| `Switch.Microsoft.Data.SqlClient.UseCompatibilityAsyncBehaviour` | `false` | Uses legacy async behavior for compatibility |
| `Switch.Microsoft.Data.SqlClient.UseCompatibilityProcessSni` | `false` | Uses legacy SNI processing path |
@mdaigle

mdaigle commented Aug 13, 2026

Copy link
Copy Markdown
Contributor Author

CI investigation: the two failing legs (sqlclient_manual_azure_123_linux_net10, sqlclient_manual_azure_123_windows_net10) plus a third failing leg (sqlclient_manual_azure_123_linux_net9, currently showing as part of the check group) all fail on the same test: ConnectionPoolTest.ReclaimEmancipatedOnOpenTest, with InvalidOperationException: Timeout expired ... obtaining a connection from the pool.

This is not a pre-existing flake: the same test passes on #4504's CI (same base branch, UseConnectionPoolV2 still defaulting to false there). It only fails here because this PR flips the default to true, routing the test through ChannelDbConnectionPool.

Root cause: WaitHandleDbConnectionPool (v1) has an explicit ReclaimEmancipatedObjects() path invoked when the pool is exhausted, letting Open() reclaim a GC'd-but-undisposed connection instead of timing out. ChannelDbConnectionPool (v2) has no equivalent active reclamation path (only a comment referencing IsEmancipated), so it can't reclaim the emancipated connection and Open() times out waiting for a pool slot.

Per this PR's stated purpose, I'm not fixing pool internals here — documented the gap in the PR body so the pool-v2 workstream can add reclaim-on-open support to ChannelDbConnectionPool before this default flip ships.

@mdaigle

mdaigle commented Aug 13, 2026

Copy link
Copy Markdown
Contributor Author

Broader CI update: failures across nearly the full manual-test matrix

As more legs of sqlclient-pr completed, the failure count grew beyond the initial 2-3 legs. Pulled the full test-execution logs for every failing sqlclient_manual_*_123_* leg (azure and localhost variants, net8/net9/net10/net462, Linux and Windows) from build 166919. Two distinct test failures account for all of them — no other tests are failing:

1. ConnectionPoolTest.ReclaimEmancipatedOnOpenTest — fails on every leg (azure + localhost, all TFMs/OSes)

Same root cause already documented in the PR description: ChannelDbConnectionPool (V2) has no active reclaim-on-open path for "emancipated" (GC'd-without-Dispose) connections, unlike WaitHandleDbConnectionPool (V1)'s ReclaimEmancipatedObjects(). Open() times out instead of reclaiming the slot. Confirmed via #4504's own CI (same base branch, switch still false) that this test passes there — this is caused by the default flip, not flakiness.

2. TvpTest.TestPacketNumberWraparound — fails on every localhost leg only (not azure legs, since it's IsNotAzureServer-gated)

This is a distinct, pre-existing regression test for a specific TdsParserStateObject.WritePacket byte-counter-wraparound bug. It opens its own SqlConnection against the default (no MaxPoolSize override) TCP connection string via OpenAsync, then drives a custom 1,000,000-row TVP enumerator through ExecuteNonQueryAsync (swallowing errors from the sproc/table-type not existing, by design — the test only cares whether the full row-set gets enumerated before that error hits).

Observed failure signature across all 5 localhost logs is consistent and non-random: the enumerator only advances 1-18 elements out of 1,000,000 in 1.5-6.2 milliseconds (not the 60s timeout), e.g.: enumerator.Count=3, enumerator.MaxCount=1000000, elapsed=00:00:00.0062206

That signature (near-zero elapsed time, tiny count) points to the task faulting essentially immediately, most likely during connection.OpenAsync(cancellationToken), which is not wrapped in the test's try/catch (only ExecuteNonQueryAsync is), so any exception there (e.g. a pool-related timeout/error) would abort the task before any rows are read, matching what's observed.

I have not confirmed the exact exception (stdout only surfaces the assertion failure, not the swallowed/faulted exception detail), and I'm not certain this is the same root cause as #1. It may be a related knock-on effect (e.g. state left behind by the ReclaimEmancipatedOnOpenTest failure destabilizing the default connection pool for subsequent tests in the same run) or a separate, independent V2 pool gap. Flagging this clearly as a second, distinct finding rather than assuming it's explained by #1.

No other manual test failures were found in any of the ~15 failing legs beyond these two. Per this PR's scope, not attempting to fix pool internals — surfacing both findings here for the pool-v2 workstream to investigate before this default flip ships.

Copilot AI review requested due to automatic review settings August 13, 2026 21:58
@mdaigle
mdaigle force-pushed the mdaigle-jubilant-robot branch from 4e920b5 to 6c1ea47 Compare August 13, 2026 21:58
@mdaigle
mdaigle changed the base branch from dev/automation/channel-pool-v2-followups to dev/automation/channel-pool-reclaim-timer August 13, 2026 21:58

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (1)

.github/instructions/features.instructions.md:257

  • The AppContext switch default-value table is now inconsistent with the actual defaults in LocalAppContextSwitches.cs: UseCompatibilityAsyncBehaviour and UseCompatibilityProcessSni both default to true (compatibility mode), but the table still lists false. Since this PR already edits this section, please update these rows so the table reflects real defaults and explains that setting them to false enables the newer behaviors.
| `Switch.Microsoft.Data.SqlClient.UseCompatibilityAsyncBehaviour` | `false` | Uses legacy async behavior for compatibility |
| `Switch.Microsoft.Data.SqlClient.UseCompatibilityProcessSni` | `false` | Uses legacy SNI processing path |
| `Switch.Microsoft.Data.SqlClient.UseConnectionPoolV2` | `true` | Enables the new `ChannelDbConnectionPool` implementation; set to `false` to restore the legacy `WaitHandleDbConnectionPool` |

@mdaigle
mdaigle force-pushed the dev/automation/channel-pool-reclaim-timer branch from 513c4d3 to 6cf4e79 Compare August 19, 2026 17:34
Flip the default value of the UseConnectionPoolV2 AppContext switch
from false to true, making the new Channel-based connection pool
(ChannelDbConnectionPool) the default implementation. The legacy V1
pool (WaitHandleDbConnectionPool) remains available by explicitly
setting the switch to false.

- Update XML doc comment on the switch to reflect the new default
- Update features.instructions.md default value table
- Update LocalAppContextSwitchesTest default-value assertion

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@mdaigle
mdaigle force-pushed the mdaigle-jubilant-robot branch from 6c1ea47 to ee208e9 Compare August 19, 2026 17:35
Copilot AI review requested due to automatic review settings August 19, 2026 17:35

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Suppressed comments (1)

.github/instructions/features.instructions.md:257

  • The switch-default table still lists UseCompatibilityAsyncBehaviour and UseCompatibilityProcessSni as defaulting to false, but both are asserted as true defaults in LocalAppContextSwitchesTest and documented/implemented as defaultValue: true in LocalAppContextSwitches.cs. This table should be updated to avoid misleading contributors about the actual defaults.
| `Switch.Microsoft.Data.SqlClient.UseCompatibilityAsyncBehaviour` | `false` | Uses legacy async behavior for compatibility |
| `Switch.Microsoft.Data.SqlClient.UseCompatibilityProcessSni` | `false` | Uses legacy SNI processing path |
| `Switch.Microsoft.Data.SqlClient.UseConnectionPoolV2` | `true` | Enables the new `ChannelDbConnectionPool` implementation; set to `false` to restore the legacy `WaitHandleDbConnectionPool` |

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings August 19, 2026 23:40

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (2)

.github/instructions/features.instructions.md:257

  • The switch default-value table is inconsistent with the actual defaults in LocalAppContextSwitches.cs: both UseCompatibilityAsyncBehaviour and UseCompatibilityProcessSni default to true (see LocalAppContextSwitches.cs:539-575), but this table still lists them as false. This makes the switch reference misleading, especially now that UseConnectionPoolV2 is being updated here as well.
| `Switch.Microsoft.Data.SqlClient.UseCompatibilityAsyncBehaviour` | `false` | Uses legacy async behavior for compatibility |
| `Switch.Microsoft.Data.SqlClient.UseCompatibilityProcessSni` | `false` | Uses legacy SNI processing path |
| `Switch.Microsoft.Data.SqlClient.UseConnectionPoolV2` | `true` | Enables the new `ChannelDbConnectionPool` implementation; set to `false` to restore the legacy `WaitHandleDbConnectionPool` |

src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/LocalAppContextSwitches.cs:588

  • The PR description calls out a missing “reclaim emancipated connections on open” path in PoolV2 as the primary root cause of CI failures, but ChannelDbConnectionPool.GetInternalConnection already performs a reclaim sweep on the slow path before it parks on the idle channel (see ChannelDbConnectionPool.cs:1516-1526, ReclaimEmancipatedConnections()). Either the investigation summary is out of date, or the problem is that reclamation is not triggering / not freeing a usable connection; please update the PR description (or add a note) so the documented root cause matches the current code.
    /// </summary>
    public static bool UseConnectionPoolV2 =>
        AcquireAndReturn(
            UseConnectionPoolV2String,
            defaultValue: true,

Task.Factory.StartNew with an async lambda returns Task<Task>. Without
Unwrap(), Task.WhenAny observed only the outer task, which completed as
soon as the async lambda hit its first await, rather than waiting for
RunPacketNumberWraparound to actually finish. This masked itself under
the legacy WaitHandleDbConnectionPool's synchronous-leaning timing, but
was exposed by ChannelDbConnectionPool's genuinely asynchronous pooled
open path, producing spurious low-enumerator-count failures.

Also capture and await the winning task when it is actionTask so any
unexpected failure (e.g. a connection open failure) propagates as a
real exception instead of surfacing only as a generic count mismatch.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings August 20, 2026 17:02

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

Suppressed comments (3)

.github/instructions/features.instructions.md:257

  • The switch-default table now claims UseCompatibilityAsyncBehaviour and UseCompatibilityProcessSni default to false, but LocalAppContextSwitches defaults both to true (see LocalAppContextSwitches.cs:557 and :575). Since this PR updates this table, it should keep these defaults accurate to avoid misleading guidance.
| `Switch.Microsoft.Data.SqlClient.UseCompatibilityAsyncBehaviour` | `false` | Uses legacy async behavior for compatibility |
| `Switch.Microsoft.Data.SqlClient.UseCompatibilityProcessSni` | `false` | Uses legacy SNI processing path |
| `Switch.Microsoft.Data.SqlClient.UseConnectionPoolV2` | `true` | Enables the new `ChannelDbConnectionPool` implementation; set to `false` to restore the legacy `WaitHandleDbConnectionPool` |

src/Microsoft.Data.SqlClient/tests/ManualTests/SQL/ParameterTest/TvpTest.cs:72

  • Task.Factory.StartNew(..., LongRunning).Unwrap() is unnecessary here and adds thread-creation overhead; RunPacketNumberWraparound is already async and can be started directly and raced against the timeout task.
            // Task.Factory.StartNew with an async delegate returns a Task<Task>, so it must be
            // unwrapped before use in Task.WhenAny below. Without Unwrap(), WhenAny would observe
            // only the outer task (which completes as soon as the async lambda hits its first
            // await) instead of the actual completion of RunPacketNumberWraparound.
            Task actionTask = Task.Factory.StartNew(

src/Microsoft.Data.SqlClient/tests/ManualTests/SQL/ParameterTest/TvpTest.cs:86

  • If the timeout wins, any fault from actionTask is never observed (and the task may keep running briefly after test failure), which can hide the real failure cause and leak background work into later tests. Consider best-effort observing faults after cancellation on the timeout path.
            // Propagate any unexpected failure from the action task (e.g. a connection open
            // failure) instead of letting it surface only as a low enumerator count below.
            if (completedTask == actionTask)
            {
                await actionTask;

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings August 21, 2026 15:40

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

src/Microsoft.Data.SqlClient/tests/ManualTests/SQL/ParameterTest/TvpTest.cs:86

  • If the timeout task wins, the test cancels the CTS and continues without ever observing the eventual completion/fault of actionTask. If OpenAsync faults after cancellation (e.g., OperationCanceledException), that exception can become unobserved and show up later as UnobservedTaskException noise. Consider attaching a fault-only continuation in the timeout case to observe exceptions without blocking the test.
            if (completedTask == actionTask)
            {
                await actionTask;
            }

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

DO NOT MERGE PRs that are created for test reasons, should not be merged.

Projects

Status: To triage

Development

Successfully merging this pull request may close these issues.

2 participants