Skip to content

feat(storage): sign Azure layer-file uploads with a SAS - #3634

Closed
michalsuba-e2b wants to merge 3 commits into
feat/layer-upload-cache-hit-tolerancefrom
feat/azure-signed-upload-headers
Closed

michalsuba-e2b wants to merge 3 commits into
feat/layer-upload-cache-hit-tolerancefrom
feat/azure-signed-upload-headers

Conversation

@michalsuba-e2b

@michalsuba-e2b michalsuba-e2b commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Azure template builds fail on every COPY: Put Blob requires the x-ms-blob-type request header, which a SAS cannot carry and the upload client never sent. The Azure provider now signs a real upload SAS, and the response carries the request headers the client must send.

  • UploadSignedURL returns UploadURL{URL, Headers}; GCS/S3/fs return no headers — responses byte-identical, guard-tested (MinIO live for S3).
  • Azure signs a user-delegation SAS (token credential) or a service SAS (shared key), create+write, start backdated for clock skew; a SAS-only connection string keeps failing with a typed error naming the config needed.
  • Contract: proto InitLayerFileUploadResponse.uploadHeaders = 3 + OpenAPI headers, both additive; the api handler passes them through only when non-empty.
  • Tests: Azurite end-to-end forcing test (PUT without the header fails, with it succeeds), fake-transport user-delegation signing assertions, additive-response handler test.
  • Verified live on an Azure BYOC env (miso9) with a dev build: header-less PUT → 400 MissingRequiredHeader, with header → 201; SDK counterpart fix(sdk): apply the upload headers the API returns with a file upload link E2B#1870.

Stacked on #3633. CI note: validate-openapi is red on main for all three specs; this branch's +5 spec lines change nothing in its output.

Sponsor: @michalsuba-e2b

🤖 Generated with Claude Code

…load headers

Azure's Put Blob requires the request header "x-ms-blob-type: BlockBlob". A SAS can
only pin response headers, so no signed URL alone can satisfy it and the provider
refused to issue one at all — every template build with a COPY instruction failed on
Azure at the get-signed-URL step.

UploadSignedURL now returns {URL, Headers} instead of a bare URL, and the header
travels with the URL through the gRPC and public API responses for the upload client
to apply. Azure mints a create+write blob SAS with whichever credential the provider
already authenticated with: a user delegation key for a token credential (the managed
identity path), or the shared key when one is configured. A SAS-only connection string
can sign neither and now says so at construction and at the call. GCS, S3 and the
filesystem provider return nil headers and byte-identical URLs.

The proto and OpenAPI additions are additive — url stays optional, and a client that
ignores the new field parses the response unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@michalsuba-e2b

michalsuba-e2b commented Sep 11, 2026

Copy link
Copy Markdown
Contributor Author

CI attribution, for the reviewer (current run, after the main merge — 38 pass / 4 fail):

The only failures are validate-openapi, and they are red on main, not here. spec/openapi.yml lints to the identical 1 error, 81 warnings at origin/main and on this branch (+5 lines); the error is e2b-consistency/operation-summary-no-get-prefix on /clusters/{clusterID}/rigs/{rigID}/errors. openapi-dashboard.yml and openapi-edge.yml are untouched here. The job is pull_request-gated on spec/**, so it only surfaces on PRs that edit a spec.

Everything else is green, including the two flakes from the previous run: arm64-tests packages/orchestrator (the 10s-deadline TestCacheExportToDiffWithMetadata_ProceedsUnderReadLock timing test) and the integration-tests shards — TestTemplateBuildCOPY, the only integration test that reaches InitLayerFileUpload, passes.

@michalsuba-e2b

Copy link
Copy Markdown
Contributor Author

Live-environment validation on an Azure BYOC env (miso9), dev orchestrator-ee built from this branch stack: InitLayerFileUpload through edge returns the UD SAS + uploadHeaders={x-ms-blob-type: BlockBlob}; PUT without the header fails 400 MissingRequiredHeader, with it 201; repeat call returns present=true (#3633's tolerance). SDK-side (E2B#1870 uploadFile) exercised live against the minted URL: headers applied → success, headers stripped → the exact mandatory-header failure. Full evidence: https://linear.app/e2b/issue/BYOC-228#comment-259c9df2

michalsuba-e2b and others added 2 commits September 14, 2026 09:40
…rance' into feat/azure-signed-upload-headers

# Conflicts:
#	packages/api/internal/api/api.gen.go
Standing review rule: a comment states one constraint on one line; reasoning
lives here, not at the fix site. No behavior change; tests unchanged and green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@michalsuba-e2b
michalsuba-e2b marked this pull request as ready for review September 14, 2026 08:01
@michalsuba-e2b

Copy link
Copy Markdown
Contributor Author

Moved to belt (source of truth): e2b-dev/belt#3308 (stacked on e2b-dev/belt#3307). Same diff, paths remapped; live miso9 evidence and review threads referenced from there. Closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant