Ship LocalStack's bundled extensions in every release package

.github/workflows/ci.yml

@@ -122,6 +122,15 @@ jobs:
           version: "~> v2"
           args: check
 
+      # `goreleaser check` only validates config syntax, so it cannot see that
+      # packaging bundled extensions and downloading them have to land
+      # together. This does, and fails the PR instead of the release.
+      - name: Check bundled packaging is in step with the download
+        run: scripts/bundled-extensions/check-bundled-packaging-sync.sh
+
+      - name: Test release scripts
+        run: make test-scripts
+
   test-unit:
     name: Unit Tests
     runs-on: ubuntu-latest
@@ -319,6 +328,43 @@ jobs:
             exit 1
           fi
 
+      # Bundled extensions. Downloads the multi-call binary for every platform
+      # and the descriptions file from the private extensions repo, verifies
+      # them against its checksum manifest, and stages them under bundled/ for
+      # GoReleaser and the npm step below. The version file says `latest`, so
+      # the script resolves that to one concrete tag here, and every later step
+      # uses that tag. On a re-run of an already-published lstk release the tag
+      # is read back from that release's notes instead of re-resolved, so the
+      # re-run ships the same extension binaries the original did.
+      - name: Fetch bundled extensions
+        id: bundle
+        run: |
+          recorded="$(gh release view "${GITHUB_REF_NAME}" --json body --jq .body 2>/dev/null \
+            | sed -n 's/^Bundled extensions: \([^ ]*\).*/\1/p' | head -n1 || true)"
+          if [ -n "${recorded}" ]; then
+            echo "Release ${GITHUB_REF_NAME} already records bundle ${recorded}; pinning to it."
+            export LSTK_EXTENSIONS_TAG="${recorded}"
+          fi
+          scripts/bundled-extensions/fetch-bundled-extensions.sh | tee fetch.log
+          tag="$(sed -n 's/^Resolved extensions bundle: \([^ ]*\) .*/\1/p' fetch.log)"
+          [ -n "${tag}" ] || { echo "could not determine the resolved bundle tag"; exit 1; }
+          commit="$(GH_TOKEN="${LSTK_EXTENSIONS_READ_TOKEN}" gh api "repos/${LSTK_EXTENSIONS_REPO}/commits/${tag}" --jq .sha)"
+          echo "tag=${tag}" >> "${GITHUB_OUTPUT}"
+          echo "commit=${commit}" >> "${GITHUB_OUTPUT}"
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          LSTK_EXTENSIONS_READ_TOKEN: ${{ secrets.LSTK_EXTENSIONS_READ_TOKEN }}
+          LSTK_EXTENSIONS_REPO: localstack/lstk-bundled-extensions
+
+      # A command described in lstk-extensions.toml that the bundle does not
+      # provide would show in help and fail when run; a binary whose commands
+      # are all undescribed would be unreachable. Either fails the release. The
+      # bundle's side of the comparison comes from asking it — the gate runs
+      # `bundled-extensions list` — so it has to be pointed at the runner's own
+      # platform directory.
+      - name: Check descriptions match the bundled binary
+        run: scripts/bundled-extensions/check-descriptions.sh bundled/linux_amd64
+
       - name: Run GoReleaser
         uses: goreleaser/goreleaser-action@v7
         with:
@@ -356,10 +402,36 @@ jobs:
       - name: Install signal-forwarding launcher
         run: cp npm/launcher.js dist/npm/lstk/index.js
 
+      # The launcher execs the Go binary from the platform package, so that is
+      # where lstk's bundled dir resolves to and where the bundled extensions
+      # must live — not the wrapper. The script also registers them in each
+      # package's `files` allowlist: the publisher emits "files": [], which npm
+      # packs as package.json + bin only, so a bare copy would be dropped.
+      - name: Add bundled extensions to the npm platform packages
+        run: scripts/bundled-extensions/add-bundled-to-npm.sh dist/npm bundled
+
       - name: Publish to NPM
         run: |
           for dir in dist/npm/lstk-*/ dist/npm/lstk/; do
             npm publish "$dir" --access public
           done
         env:
           NODE_AUTH_TOKEN: ${{ secrets.NPM_AUTH_TOKEN }}
+
+      # Permanent record of which bundle this release shipped. Job logs expire;
+      # the release notes do not, and they are what someone investigating a
+      # bundled-extension bug months later will read. Idempotent so a re-run
+      # does not append a second line.
+      - name: Record the bundle in the release notes
+        run: |
+          gh release view "${GITHUB_REF_NAME}" --json body --jq .body > notes.md
+          if grep -q '^Bundled extensions: ' notes.md; then
+            echo "Release notes already record the bundle."
+            exit 0
+          fi
+          printf '\n\n---\n\nBundled extensions: %s (commit %s)\n' "${TAG}" "${COMMIT}" >> notes.md
+          gh release edit "${GITHUB_REF_NAME}" --notes-file notes.md
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          TAG: ${{ steps.bundle.outputs.tag }}
+          COMMIT: ${{ steps.bundle.outputs.commit }}

.gitignore

@@ -12,3 +12,12 @@ test/integration/test-samples/**/.terraform/
 test/integration/test-samples/**/.terraform.lock.hcl
 test/integration/test-samples/**/*.tfstate
 test/integration/test-samples/**/*.tfstate.*
+
+# Bundled-extension staging tree, populated by
+# scripts/bundled-extensions/fetch-bundled-extensions.sh at release-build
+# time: per-platform binaries in bundled/<os>_<arch>/ and
+# bundled/lstk-extensions.toml. Only the version file is tracked; downloaded
+# artifacts must never be committed.
+# It deliberately lives outside dist/, which `goreleaser --clean` wipes.
+/bundled/*
+!/bundled/extensions.version

.goreleaser.yaml

@@ -44,6 +44,23 @@ archives:
     files:
       - completions/*
       - manpages/*
+      # LocalStack's bundled extensions: one multi-call binary per platform plus
+      # the descriptions file, at the archive root next to lstk. Staged under
+      # bundled/ by scripts/bundled-extensions/fetch-bundled-extensions.sh,
+      # which the release job runs first; a local snapshot build needs it too
+      # (see docs/extensions-bundling.md). A glob matching nothing fails the
+      # build.
+      #
+      # The glob is the binary alone (bare on Unix, .exe on Windows). Nothing
+      # else in the staged platform directory is shipped: lstk finds bundled
+      # commands through lstk-extensions.toml and dispatches by argv[0], so a
+      # per-command file at the archive root would serve no purpose.
+      - src: "bundled/{{ .Os }}_{{ .Arch }}/bundled-extensions*"
+        strip_parent: true
+        info:
+          mode: 0o755
+      - src: bundled/lstk-extensions.toml
+        strip_parent: true
 
 checksum:
   name_template: checksums.txt
@@ -71,5 +88,8 @@ homebrew_casks:
       post:
         install: |
           if OS.mac?
-            system_command "/usr/bin/xattr", args: ["-dr", "com.apple.quarantine", "#{staged_path}/lstk"]
+            # The whole staged dir, not only lstk: the bundled extensions binary
+            # sits next to it and would otherwise be blocked by Gatekeeper on its
+            # first run.
+            system_command "/usr/bin/xattr", args: ["-dr", "com.apple.quarantine", "#{staged_path}"]
           end

CLAUDE.md

@@ -170,7 +170,7 @@ Shared plumbing lives in `cmd/proxy.go` (`leadingFlags`, `stripLeadingProxyFlags
 
 # Extensions
 
-lstk supports Git-style extensions: when `lstk <name>` is not a built-in command or alias, lstk resolves and execs an external `lstk-<name>` executable, forwarding arguments verbatim and propagating the exit code. Built-ins always win. Resolution order is built-ins → bundled dir (the directory of the symlink-resolved lstk executable) → `PATH`; there is no manifest. Runtime context is conveyed via `LSTK_EXT_API_VERSION` and `LSTK_EXT_CONTEXT` (JSON: `configDir`, optional `authToken`, `nonInteractive`, `json`, optional `sessionId` — lstk's telemetry session id, omitted when telemetry is disabled, so an extension's own telemetry can join lstk's `ext:<name>` event — optional `machineId` — lstk's anonymized machine id (the prepared hash), omitted alongside `sessionId` when telemetry is disabled, so an extension reports the same machine without re-deriving it — optional `endpointUrl` — the resolved `--endpoint-url`/`LSTK_ENDPOINT_URL`/`AWS_ENDPOINT_URL` value, conveyed verbatim and unvalidated (dispatch never rejects or probes it, unlike the built-ins' `rejectEndpointURL`), omitted when no source is set — and an `emulators` array, which stays local-Docker discovery and is independent of `endpointUrl`) — see `extension.Context`/`Environ` in `internal/extension/context.go`; dispatch and help listing are in `cmd/extension.go`. Automated distribution/co-update of bundled extensions is deferred to the `add-bundled-extension-distribution` change. See [extensions-authoring.md](docs/extensions-authoring.md) for the author-facing contract.
+lstk supports Git-style extensions: when `lstk <name>` is not a built-in command or alias, lstk resolves and execs an external `lstk-<name>` executable, forwarding arguments verbatim and propagating the exit code. Built-ins always win. Resolution order is built-ins → bundled dir (the directory of the symlink-resolved lstk executable) → `PATH`; there is no manifest. Runtime context is conveyed via `LSTK_EXT_API_VERSION` and `LSTK_EXT_CONTEXT` (JSON: `configDir`, optional `authToken`, `nonInteractive`, `json`, optional `sessionId` — lstk's telemetry session id, omitted when telemetry is disabled, so an extension's own telemetry can join lstk's `ext:<name>` event — optional `machineId` — lstk's anonymized machine id (the prepared hash), omitted alongside `sessionId` when telemetry is disabled, so an extension reports the same machine without re-deriving it — optional `endpointUrl` — the resolved `--endpoint-url`/`LSTK_ENDPOINT_URL`/`AWS_ENDPOINT_URL` value, conveyed verbatim and unvalidated (dispatch never rejects or probes it, unlike the built-ins' `rejectEndpointURL`), omitted when no source is set — and an `emulators` array, which stays local-Docker discovery and is independent of `endpointUrl`) — see `extension.Context`/`Environ` in `internal/extension/context.go`; dispatch and help listing are in `cmd/extension.go`. LocalStack's own bundled extensions ship as **one multi-call binary**, `bundled-extensions`, next to `lstk`, plus an `lstk-extensions.toml` listing the commands it provides; lstk execs it with `argv[0]` = `lstk-<name>`. The invariants are on `extension.BundledBinaryName`, `LoadBundle` and `Extension.Argv0`; the release pipeline (helper scripts in `scripts/bundled-extensions/`, `make test-scripts`) is in [extensions-bundling.md](docs/extensions-bundling.md) and the author-facing contract in [extensions-authoring.md](docs/extensions-authoring.md). Set-wise co-update (`internal/update`) is still pending in the `add-bundled-extension-distribution` change.
 
 # Signal Forwarding to Wrapped Tools
 

Makefile

@@ -5,7 +5,7 @@ endif
 BUILD_DIR=bin
 export CGO_ENABLED=0
 
-.PHONY: build clean test test-integration lint govulncheck mock-generate otel
+.PHONY: build clean test test-integration test-scripts lint govulncheck mock-generate otel
 
 # Always invoke `go build` and let Go's build cache handle incrementality; a
 # file target on bin/lstk would be skipped when the binary exists, even with
@@ -23,6 +23,11 @@ test:
 test-integration: build
 	@RUN="$(RUN)" ./scripts/test-integration.sh
 
+# Bash suites for the bundled-extension release helper scripts. They only ever run
+# on the Linux release runner, so a bash suite is the faithful test here.
+test-scripts:
+	@./scripts/bundled-extensions/test-scripts.sh
+
 otel:
 	docker compose -f docker-compose.tracing.yaml up -d
 

bundled/extensions.version

@@ -0,0 +1,23 @@
+# Which extensions bundle this lstk release ships.
+#
+# One value line (blank lines and #-comments are ignored), in one of two forms:
+#
+#   latest       Take the newest published release of the private extensions
+#                repository. This is the default: there is no routine bump to
+#                remember, and a release can never go silently stale. The
+#                release job resolves it to a concrete tag ONCE and records
+#                that tag in the published release notes, so an lstk version
+#                still maps to exactly one bundle.
+#
+#   v2026.08.19  Lock this build to that exact release tag of the private
+#                extensions repository (that repository tags by date, not
+#                semver). Use it to hold a bad bundle back. To re-run an
+#                already-published lstk release against the bundle it
+#                originally shipped, do not edit this file — pass that
+#                release's recorded tag to
+#                scripts/bundled-extensions/fetch-bundled-extensions.sh instead
+#                (--tag / LSTK_EXTENSIONS_TAG).
+#
+# This is the only tracked file under bundled/. The downloaded binaries and
+# descriptions file are staged alongside it and are gitignored.
+latest

cmd/extension.go

@@ -54,7 +54,15 @@ func dispatchExtension(ctx context.Context, cfg *env.Env, tel *telemetry.Client,
 			})
 			return output.NewSilentError(fmt.Errorf("unknown command %q for lstk", name))
 		}
-		return err
+		// Anything else is a broken bundled install (the multi-call binary is
+		// present but its command list is not loadable) — an lstk problem, not
+		// an unknown command, so it gets the styled error and a way out.
+		output.NewPlainSink(os.Stderr).Emit(output.ErrorEvent{
+			Title:   "bundled extensions are not usable",
+			Summary: err.Error(),
+			Actions: []output.ErrorAction{{Label: "Reinstall lstk to restore them:", Value: "lstk update"}},
+		})
+		return output.NewSilentError(err)
 	}
 
 	emulators := resolveEmulators(ctx, cfg, logger)
@@ -153,33 +161,32 @@ func emulatorCandidates() []config.ContainerConfig {
 }
 
 // registerExtensionHelp wires an "extensions" template function that renders the
-// Extensions section of `lstk --help`. It scans the bundled dir + PATH for
-// `lstk-*` executables (de-duplicated, bundled wins) and attaches descriptions
-// for bundled extensions from the hand-authored descriptions file; PATH and
-// custom extensions, and bundled names missing from the file, are name-only.
-// Rendering never executes an extension. A scan happens on each help render so
-// freshly installed extensions appear without restarting.
+// Extensions section of `lstk --help`. It lists the bundle's commands, then
+// `lstk-*` executables in the bundled dir and on PATH (de-duplicated, bundled
+// wins); bundled entries come with the description Resolver.List attached from
+// the hand-authored descriptions file, while PATH and custom extensions, and
+// bundled names missing from the file, are name-only. Rendering never executes
+// an extension. A scan happens on each help render so freshly installed
+// extensions appear without restarting.
 func registerExtensionHelp(logger log.Logger) {
 	cobra.AddTemplateFunc("extensions", func(namePadding int) string {
-		resolver := extension.NewResolver(logger)
-		list := resolver.List()
+		list := extension.NewResolver(logger).List()
 		if len(list) == 0 {
 			return ""
 		}
-		descriptions := extension.LoadDescriptions(resolver.BundledDir, logger)
-		return formatExtensionList(list, descriptions, namePadding)
+		return formatExtensionList(list, namePadding)
 	})
 }
 
 // formatExtensionList renders the extension help lines so they align with the
 // command sections above them. It mirrors Cobra's own scheme (see the usage
 // template's "{{rpad .Name .NamePadding}} {{.Short}}"): each name is right-padded
-// to namePadding, then a single space, then its description (bundled extensions
-// only, from the descriptions file). namePadding is the root command's
+// to namePadding, then a single space, then its Description (set for bundled
+// extensions only). namePadding is the root command's
 // .NamePadding, so the description column matches the Commands/Tools sections; a
 // name longer than namePadding widens its own row exactly as Cobra's per-row
 // rpad does. Lines are sorted by name (List already sorts).
-func formatExtensionList(list []extension.Extension, descriptions map[string]string, namePadding int) string {
+func formatExtensionList(list []extension.Extension, namePadding int) string {
 	width := namePadding
 	for _, ext := range list {
 		if len(ext.Name) > width {
@@ -189,11 +196,7 @@ func formatExtensionList(list []extension.Extension, descriptions map[string]str
 
 	var b strings.Builder
 	for _, ext := range list {
-		desc := ""
-		if ext.Bundled {
-			desc = descriptions[ext.Name]
-		}
-		if desc != "" {
+		if desc := ext.Description; desc != "" {
 			fmt.Fprintf(&b, "  %-*s %s\n", width, ext.Name, desc)
 		} else {
 			fmt.Fprintf(&b, "  %s\n", ext.Name)

docs/extensions-authoring.md

@@ -5,6 +5,7 @@ lstk supports Git-style extensions. When you run `lstk <name>` and `<name>` is n
 ## The contract at a glance
 
 - **Name it `lstk-<name>`** and put it on `PATH`. `lstk <name> ...` will run it; `lstk help` will list it.
+- **You are invoked as `lstk-<name>`.** Your `argv[0]` is the base name `lstk-<name>` (plus `.exe` when your file carries that suffix on Windows), never a full path, and it is the same whether lstk found you on `PATH`, next to its own binary, or inside LocalStack's bundle. Strip a trailing `.exe` before comparing. A `<name>` starts with a letter or digit and uses only letters, digits, hyphens and underscores.
 - **Your arguments are forwarded verbatim.** Everything after `<name>` is yours — lstk does not parse it. Define and parse your own flags however you like, including flags that happen to share a name with an lstk global flag.
 - **lstk's global flags are consumed before the name.** `lstk --non-interactive <name> --foo` runs your extension with just `--foo`; the resolved global state reaches you via environment variables (below), not on your command line.
 - **Exit code and streams pass through.** Your exit status becomes lstk's exit status, and your stdin/stdout/stderr are wired straight to the terminal.
@@ -111,6 +112,29 @@ Reuse `machineId` rather than deriving your own: it is already the final hashed
 
 **Absence is ambiguous, by design.** Both fields are omitted when lstk's telemetry is disabled — a disabled lstk computes neither, so they always appear and disappear together — and both are also absent on an lstk released before they existed. You cannot tell those two cases apart, so don't try. Treat absence as "no correlation available" and carry on: generate or derive your own ids if you need them, and never make either field a hard requirement.
 
+## How LocalStack's bundled extensions differ
+
+Everything above applies to bundled extensions too, with one mechanical
+difference: they are not separate `lstk-<name>` files. LocalStack ships one
+multi-call binary, `bundled-extensions`, next to `lstk`, and lstk executes it
+with `argv[0]` set to `lstk-<name>` for whichever command was requested. The
+list of commands it provides is the descriptions file `lstk-extensions.toml`
+beside it; a name that file does not list is never handed to the bundle. Read
+`os.Args[0]` (or your language's equivalent) to find out which extension you are
+being asked to be. The value is exactly `lstk-<name>`, with no path and no
+`.exe`, and lstk only ever hands the bundle a name the toml lists, so a lookup
+miss inside the binary means the toml and the binary disagree: report it
+loudly rather than guessing.
+
+The binary is the only copy on disk — there is no `lstk-<name>` file per
+command — so it is reachable only as `lstk <name>`. It nonetheless sits in the
+install directory next to `lstk`, where somebody will eventually find it and
+run it directly. Handle that: when `LSTK_EXT_API_VERSION` is unset there is no
+runtime context to work from, so print a short message saying the binary is
+part of lstk and naming the command to use instead, then exit non-zero. Do not
+fall back to defaults. See [extensions-bundling.md](extensions-bundling.md)
+for how the bundle is built and shipped.
+
 ## Help descriptions
 
 `lstk --help` lists installed extensions by command name. One-line descriptions are shown **only for extensions LocalStack bundles with lstk**, from a static descriptions file LocalStack ships with them. Third-party and `PATH`-installed extensions are listed by name only (the same as Git's `git help -a`). lstk never executes an extension to render help, so listing is always side-effect-free.