Ship LocalStack's bundled extensions in every release package

[carillan81]

NOTE:
This PR needs to be merged when the extensions are released and never before. The moment this PR is merged, the package will include the extensions in the next release.

Motivation

LocalStack's own extensions (lstk doctor, lstk deploy) exist but do not ship with lstk. Today someone has to place the files next to the binary by hand. This makes every release carry them, on all three install channels.

Solution

The extensions ship as one binary, bundled-extensions, next to lstk. lstk tells it which extension to be by launching it under the name lstk-, the same trick busybox and git use. Which commands exist comes from lstk-extensions.toml shipped beside it, so that file is required whenever the binary is present. If it is missing or broken, lstk says so and points at lstk update rather than reporting an unknown command.

The release pipeline downloads the bundle from the private extensions repo, verifies every file against its checksums, and refuses to release if the descriptions file and the binary disagree about which commands exist. The binary is asked directly for its command list (bundled-extensions list), so the toml can never promise a command the binary does not answer to. Goreleaser puts both files in the archives, the Homebrew cask inherits them, and a script copies them into each npm platform package.

A release ships exactly two files per platform: the binary and the descriptions file. There are no lstk- alias links. lstk never used them to resolve anything, and they behaved differently on every channel (npm drops symlinks when packaging, Windows extractors turn them into text files), so dropping them makes all three channels identical. A leftover link from an older install is inert: lstk still takes its command list from the toml and dispatches by argv[0].

Testing

New bash suites for the release scripts (make test-scripts), unit and integration tests for the dispatch, and a PR check that stops the two halves of the packaging config being merged separately. Verified against a real goreleaser build and a real brew and npm install.

Docs

New docs/extensions-bundling.md: what ships, where the files live per install method, the release pipeline, what the private repo must publish, and how to diagnose a broken install.

docs/extensions-authoring.md gains a section on how bundled extensions differ for authors.

CLAUDE.md and the openspec change are updated.

No existing documented command changes, so nothing else needs updating.

[peter-smith-phd]

Thanks @carillan81, I've read through the relevant .md files, and have made a few comments. Most notably, I see that we're still talking about the lstk-<name> symlink in many places, but I thought we had decided against that idea?

Once we got that sorted out, I'd like to do a lot of manual testing myself (using the code in this PR). That'll convince me whether we're ready to launch this or not. I think we're heading in the right direction with this PR, but I'd just like to double-check everything.

[carillan81]

Thanks @carillan81, I've read through the relevant .md files, and have made a few comments. Most notably, I see that we're still talking about the lstk-<name> symlink in many places, but I thought we had decided against that idea?

My understanding was that we don't use symlinks to run extensions from lstk but we keep them as a cosmetic and auxilliary tool (run extensions without lstk providing context manually). Revisiting this, it's really not needed to bundle the symlinks and introduces some drift between installation methods (symlinks work on some, not others). I will then remove everything related to symlinks from the bundling mechanism.
I'll add the changes when they are ready (some hours I hope)

[carillan81]

Hey @peter-smith-phd, I pushed some more changes. Those changes are aligned with this other PR in the lstk-bundled-extensions repo.
In order to be able to use this complete the other PR needs to be merged and a release generated.
I think most of the questions are addressed.

[peter-smith-phd]

Thanks @carillan81, it's looking really good. I've provided more feedback, although it's pretty minor this time.

The only thing I want to do is manually test this on MacOS, Windows, and Linux, for the three different distribution methods. I got part way through doing this, but ran out of time today. I should be able to get it tested tomorrow.

One more thing - given that we're publicly launching lstk on Tuesday, there'll be a lot of people downloading lstk next week. I suggest we hold off merging this until at least the following week.

[peter-smith-phd]

Thanks, I'll definitely be testing this manually - if not today, by the end of this week 😃

[peter-smith-phd]

Thanks, I've managed to successfully test on MacOS, Linux, and Windows, using binaries, npm and brew. Everything worked as expected for a fresh install.

I'm now confident this will work, although let's still delay the merge until the end of next week, given that we have the lstk marketing launch coming up on Tuesday. We don't want to accidentally break things shortly after the launch.

Thanks for doing this!