fix(e2e): reconcile IP connect on the existing shared bastion - #9165
Open
Ganeshkumar Ashokavardhanan (ganeshkumarashok) wants to merge 1 commit into
Open
fix(e2e): reconcile IP connect on the existing shared bastion#9165Ganeshkumar Ashokavardhanan (ganeshkumarashok) wants to merge 1 commit into
Ganeshkumar Ashokavardhanan (ganeshkumarashok) wants to merge 1 commit into
Conversation
#9149 enabled IP connect on the shared bastion, but only in the creation path. ensureSharedBastion returns early when the bastion already exists, so the long-lived shared bastion in the e2e subscription kept enableIpConnect unset and every SSH attempt kept failing with: failed to start bastion tunnel: error creating tunnel: 403 bastionssh.go connects through .../bh-hostConnect/<ip>, Bastion's IP-based connect flow, which is rejected unless IP connect is on. Reconcile the flag on the existing bastion so the fix actually reaches the deployed resource. The update is a no-op once the flag is set. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1ab1da0c-5f90-4ace-a154-0f9e7dbbbad8
Ganeshkumar Ashokavardhanan (ganeshkumarashok)
requested review from
Chou Hu (AbelHu),
Devinwong,
Sri Harsha (SriHarsha001),
Nishchay (awesomenix),
Calvin S. (calvin197),
Cameron Meissner (cameronmeissner),
Sylvain Boily (djsly),
Karen Chen (karenychen),
lilypan26,
Mark Ibrahim (mxj220),
Peter Damianov (pdamianov-dev),
Patrick W. Healy (phealy),
r2k1,
Runzhen (runzhen),
sulixu,
Tim Wright (timmy-wright),
Thibault Cohen (titilambert),
Xu Xue (xuexu6666) and
Zachary (zachary-bailey)
as code owners
August 7, 2026 20:26
Copilot started reviewing on behalf of
Ganeshkumar Ashokavardhanan (ganeshkumarashok)
August 7, 2026 20:27
View session
Contributor
Windows Unit Test Results 3 files 12 suites 49s ⏱️ Results for commit 5d3098d. |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes a reliability issue in the e2e shared infrastructure reconciliation: the shared Azure Bastion can predate the introduction of EnableIPConnect, and the existing “bastion already exists” fast-path previously returned without ever applying the newer required property—leading to persistent 403 failures when establishing SSH tunnels via Bastion’s IP-based connect flow.
Changes:
- Add a reconciliation step to enable
EnableIPConnecton an existing shared bastion before returning its DNS name. - Keep
EnableIPConnect: truein the bastion creation properties, with clarifying comments tying it to thebh-hostConnect/<ip>flow used bybastionssh.go.
Comment on lines
+290
to
+312
| func ensureBastionIPConnect(ctx context.Context, rg string, bastion armnetwork.BastionHost) error { | ||
| if bastion.Properties.EnableIPConnect != nil && *bastion.Properties.EnableIPConnect { | ||
| return nil | ||
| } | ||
| toolkit.Logf(ctx, "enabling IP connect on existing shared bastion %s", SharedBastionName) | ||
| bastion.Properties.EnableIPConnect = to.Ptr(true) | ||
| poller, err := config.Azure.BastionHosts.BeginCreateOrUpdate(ctx, rg, SharedBastionName, bastion, nil) | ||
| if err != nil { | ||
| return fmt.Errorf("enabling bastion IP connect: %w", err) | ||
| } | ||
| if _, err := poller.PollUntilDone(ctx, config.DefaultPollUntilDoneOptions); err != nil { | ||
| return fmt.Errorf("waiting for bastion IP connect: %w", err) | ||
| } | ||
| return nil | ||
| } | ||
|
|
||
| func ensureSharedBastion(ctx context.Context, rg, location string) (string, error) { | ||
| existing, err := config.Azure.BastionHosts.Get(ctx, rg, SharedBastionName, nil) | ||
| if err == nil { | ||
| if existing.Properties == nil || existing.Properties.DNSName == nil { | ||
| return "", fmt.Errorf("shared bastion %s exists but has no DNS name", SharedBastionName) | ||
| } | ||
| if err := ensureBastionIPConnect(ctx, rg, existing.BastionHost); err != nil { |
Comment on lines
+287
to
+289
| // ensureBastionIPConnect turns on IP connect for a bastion that predates it being set at | ||
| // creation time. Without it the /api/tokens call in bastionssh.go returns 403 forever, since | ||
| // the bastion is only created once and is otherwise never updated. |
Contributor
Author
|
Confirmed working: GPU E2E build 175689787 on this branch shows 0 bastion 403s (baseline 175585765 on Note this unmasks a pre-existing failure — all 12 GPU scenario attempts now fail on |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
ensureSharedBastionreconcilesenableIpConnecton the existing shared bastion, not just on the one it creates.Why
#9149 enabled IP connect, but only in the creation path.
ensureSharedBastionreturns early when the bastion already exists:The shared bastion in the e2e subscription was created before #9149, so it kept
enableIpConnectunset and #9149 never reached it. Confirmed on the live resource:Every SSH attempt therefore failed, 5/5 retries, on every scenario:
bastionssh.gobuilds.../providers/Microsoft.Network/bh-hostConnect/<ip>, which is Bastion's IP-based connect flow. Bastion rejects it with 403 unless IP connect is on. Bastion itself was healthy the whole time — the resource-based flow (az network bastion tunnel --target-resource-id ...) connected fine.Verification
I set the flag out of band on the shared bastion and re-ran
Test_Ubuntu2204_GPUA100locally. SSH succeeded on attempt 1/5 instead of failing 5/5, and validation proceeded through the kubelet, NIC, kernel and waagent validators:This change makes that self-healing so the fix survives without manual intervention. The update is a no-op once the flag is set, so it costs nothing on subsequent runs.