Skip to content

Fix lstk start waits forever when the container fails during startup#390

Merged
tiurin merged 5 commits into
mainfrom
devx-758-cli-waits-when-container-fails-during-startup
Jul 18, 2026
Merged

Fix lstk start waits forever when the container fails during startup#390
tiurin merged 5 commits into
mainfrom
devx-758-cli-waits-when-container-fails-during-startup

Conversation

@tiurin

@tiurin tiurin commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Motivation

lstk start kept waiting at "Starting LocalStack..." indefinitely when the container failed during startup (e.g. in offline/DNS-broken environments), and never surfaced the container's exit code or logs.

Changes

Before: ∞ "Starting LocalStack..." without any actionable information.

After: If after 20 seconds LocalStack hasn't started lstk prompts to keep waiting or stop, suggesting to look at lstk logs as well. In non-interactive mode the timeout is 60 seconds upon which lstk reports a failure and leaves the container running for inspection.

Note: 3 second timeout is used for the video for the sake of brevity. 20 seconds felt like a good default for me, LocalStack should generally start faster as a local tool. Happy to adjust if there is any better idea about the default.

devx-758.mov

Changes details

  • Register a next-exit container wait between create and start, so the exit code is captured even for instantly-exiting containers before --rm removes them; Runtime.Start now returns the exit channel and awaitStartup consumes it alongside the existing health/liveness poll
  • Bound the readiness wait with a startup deadline: 20s interactive, 60s non-interactive (more relaxed for CI use case to allow for slower machines and also CI is a more patient user), overridable via LSTK_STARTUP_TIMEOUT
  • On the interactive deadline, prompt Keep waiting [W] (re-arms the deadline) / Stop LocalStack and exit [S] (stops with progress feedback); non-interactive fails and leaves the container running for inspection
  • Surface both failure modes with the exit code and last container log lines (read after the log-tail drain so the tail is complete), with messaging that reflects whether the container was stopped; add a start_timeout telemetry error code

Tests

  • New integration tests: a container that exits during startup fails with the reported exit code; a running-but-never-healthy container fails after the configured timeout (previously hung forever)
  • New unit tests: exit-code delivery, poll fallback when the wait fails, timeout, the interactive keep-waiting/stop prompt flow, and per-mode default resolution

Closes DEVX-758

@tiurin tiurin added semver: patch docs: needed Pull request requires documentation updates docs: skip Pull request does not require documentation changes and removed docs: needed Pull request requires documentation updates labels Jul 16, 2026
@tiurin
tiurin force-pushed the devx-758-cli-waits-when-container-fails-during-startup branch from af37498 to dd5f249 Compare July 16, 2026 14:01
@tiurin tiurin changed the title Fix start waiting forever when the container fails during startup Fix lstk start waits forever when the container fails during startup Jul 16, 2026
@tiurin
tiurin marked this pull request as ready for review July 16, 2026 15:11
@tiurin
tiurin requested a review from a team as a code owner July 16, 2026 15:11

@tiurin tiurin left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will push a bit of a refactoring shortly.

Comment thread internal/container/start.go Outdated

healthURL := fmt.Sprintf("http://localhost:%s%s", c.Port, c.HealthPath)
err = awaitStartup(ctx, rt, sink, containerID, "LocalStack", healthURL, startupLogs)
err = awaitStartup(ctx, rt, sink, containerID, "LocalStack", healthURL, exitCh, startupTimeout, interactive)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not too happy with the function parameters count growing so much in awaitStatus and handleStartupTimeout/Faillure. Will refactor this to put dependencies that are constant for the whole Start invocation to a struct: rt, sink, tel, timeout, interactive. And then keep only per-call data as function agruments: containerID, healthURL, exitCh, c, err, logs

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Refactored in 6ffb089

@tiurin
tiurin force-pushed the devx-758-cli-waits-when-container-fails-during-startup branch 3 times, most recently from 6ffb089 to 5092d07 Compare July 17, 2026 15:44

@anisaoshafi anisaoshafi left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your contribution, Misha 👏🏼

small issue: before if CTRL+C called while awaiting it still did emit and send a telemetry event which is a sort of regression in this PR. Not a big deal, but if there's a reason for it, would be nice to assert it in some test so we recall the decision, or otherwise just try to track that event.

tiurin added a commit that referenced this pull request Jul 17, 2026
An interrupted start (Ctrl+C while awaiting readiness) deliberately renders
no styled error, but the early return also dropped the start_error lifecycle
telemetry event that was emitted before startup monitoring was introduced.
Restore the emit (error code start_failed, matching the previous behavior)
and add an integration test that interrupts a start mid-wait and asserts the
event is delivered, guarding the decision from the PR #390 review.

Co-Authored-By: Claude <noreply@anthropic.com>
tiurin and others added 5 commits July 17, 2026 19:35
lstk start could wait indefinitely at "Starting LocalStack..." when the
container failed during startup (DEVX-758), and never reported the
container's exit code.

- Register a next-exit ContainerWait between create and start, so even an
  instantly-exiting container's exit code is captured before AutoRemove
  removes it; Runtime.Start now returns the exit channel.
- Bound the readiness wait with a startup deadline: 20s interactive (shows
  a recoverable keep-waiting/stop prompt that re-arms on "keep waiting"),
  60s non-interactive (fails, leaving the container running for
  inspection). LSTK_STARTUP_TIMEOUT overrides both.
- Surface both failure modes as styled ErrorEvents with the exit code and
  the last container log lines, read after the log-tail drain so the tail
  is complete; add a start_timeout telemetry error code.

Co-Authored-By: Claude <noreply@anthropic.com>
…ults

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
awaitStartup, handleStartupTimeout, and handleStartupFailure had grown to
6-9 parameters. Move the collaborators and mode that are constant across a
Start invocation (rt, sink, tel, timeout, interactive) onto a startupMonitor
receiver, keeping only per-call data as method arguments. Also drop the
unused name parameter of awaitStartup.

Co-Authored-By: Claude <noreply@anthropic.com>
An interrupted start (Ctrl+C while awaiting readiness) deliberately renders
no styled error, but the early return also dropped the start_error lifecycle
telemetry event that was emitted before startup monitoring was introduced.
Restore the emit (error code start_failed, matching the previous behavior)
and add an integration test that interrupts a start mid-wait and asserts the
event is delivered, guarding the decision from the PR #390 review.

Co-Authored-By: Claude <noreply@anthropic.com>
@tiurin
tiurin force-pushed the devx-758-cli-waits-when-container-fails-during-startup branch from 4178173 to 79db60e Compare July 17, 2026 17:35
@tiurin

tiurin commented Jul 17, 2026

Copy link
Copy Markdown
Contributor Author

Thank you for your contribution, Misha 👏🏼

small issue: before if CTRL+C called while awaiting it still did emit and send a telemetry event which is a sort of regression in this PR. Not a big deal, but if there's a reason for it, would be nice to assert it in some test so we recall the decision, or otherwise just try to track that event.

Thanks for spotting this @anisaoshafi! Restored the telemetry event in 79db60e

@tiurin
tiurin merged commit 2d5a15b into main Jul 18, 2026
17 checks passed
@tiurin
tiurin deleted the devx-758-cli-waits-when-container-fails-during-startup branch July 18, 2026 10:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs: skip Pull request does not require documentation changes semver: patch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants