blob/gcsblob: support the gRPC API and Rapid Storage (zonal) buckets

[stanhu]

This pull request adds support for the Cloud Storage gRPC API to gcsblob, and on top of it, support for Rapid Storage (zonal) buckets.

Updated after review. Options.UseGRPC and Options.UseZonalAPIs are gone, replaced by a Dial plus a constructor following secrets/gcpkms. TestConformanceGRPC is now a record/replay test with golden files, and passes all 88 checks. Details in "What changed since the review" at the bottom.

API

func DialGRPC(ctx context.Context, ts gcp.TokenSource, opts ...option.ClientOption) (*storage.Client, func(), error)
func OpenBucketGRPC(client *storage.Client, bucketName string, opts *Options) (*blob.Bucket, error)

c, cleanup, err := gcsblob.DialGRPC(ctx, ts,
	storage.WithAppendableUploads(), storage.WithGRPCBidiReads())
if err != nil { ... }
defer cleanup()
b, err := gcsblob.OpenBucketGRPC(c, "my-rapid-bucket", nil)

OpenBucketGRPC takes no context, matching gcpkms.OpenKeeper: once a client exists there is nothing left to cancel. Options.Client, which has accepted a *storage.Client since v0.44.0, keeps working unchanged.

For callers whose entire configuration is a URL string, and who therefore have nowhere to put a client, there are two query parameters:

Parameter	Effect
grpc=true	Uses the gRPC transport.
zonal=true	Additionally enables the zonal bucket APIs. Implies grpc=true.

b, err := blob.OpenBucket(ctx, "gs://my-rapid-bucket?zonal=true")

Combining zonal=true with an explicit grpc=false is rejected rather than silently overridden. URLOpener gains a TokenSource, populated by lazyCredsOpener from the credentials it already resolves, because a gRPC client cannot reuse an HTTP one.

Why Rapid Storage needs more than a transport switch

Zonal buckets accept only appendable object uploads. Every ordinary write is rejected, on both transports:

gs://bucket?grpc=true   InvalidArgument: This bucket type only supports appendable objects
gs://bucket             googleapi: Error 400: This bucket requires appendable objects

An appendable object is uploaded over a bidirectional stream. It becomes visible as soon as the first bytes are flushed and stays open for further writes until something finalizes it. Ordinary uploads, one-shot or resumable, produce an object only once the whole payload has been sent. Zonal buckets support the appendable form and nothing else.

So two things are needed:

1. storage.WithAppendableUploads() and storage.WithGRPCBidiReads() on the client. The first makes ObjectHandle.NewWriter default Writer.Append to true, which selects the appendable upload path; the second switches reads to the bidirectional API. These were a single experimental.WithZonalBucketAPIs() until storage v1.67.0 split them and graduated both out of experimental.
2. Writer.FinalizeOnClose on the writer. An appendable object stays open by default, so Close leaves an unfinalized object exposing only the bytes that happened to be flushed. For a payload small enough to fit in one buffer that is a zero-length object, even though Close returned no error.

Reads already worked on both transports without any change.

Notes for review

FinalizeOnClose is set unconditionally in NewTypedWriter. This looked risky to me too, so I traced it. The field never appears in http_client.go, so the JSON/HTTP writer cannot see it. In grpc_writer.go the only read is gRPCAppendBidiWriteBufferSender.send, and pickBufferSender returns that sender only when Writer.Append is set. It is dead code on every path gcsblob uses today.

Buckets now close the client they created. Close was a no-op. A gRPC client owns a connection pool, so every OpenBucketURL with grpc/zonal leaked one for the process lifetime. Clients supplied by the caller, through OpenBucketGRPC or Options.Client, are left alone.

Emulator handling on the gRPC path is left to the storage library. Reusing STORAGE_EMULATOR_HOST would not work: it is the HTTP endpoint, and a local emulator needs a separate port for gRPC. Passing it to option.WithEndpoint alongside option.WithoutAuthentication would also still dial over TLS, because skipping credentials does not make the transport plaintext. defaultGRPCOptions already reads STORAGE_EMULATOR_HOST_GRPC, strips the scheme, dials insecurely and disables client metrics, and those defaults are merged ahead of caller-supplied options. lazyCredsOpener checks that variable too.

gRPC on its own is not a speedup. From an n2-standard-4 in us-central1-c, 1 KiB objects each read exactly once, n=1000:

Bucket	JSON/HTTP p50	JSON/HTTP p99	gRPC p50	gRPC p99
Rapid Storage (zonal), zonal=true	*	*	12.0 ms	34.5 ms
NAM4 dual-region	35.4 ms	69.2 ms	35.5 ms	73.2 ms
US multi-region	61.2 ms	150.9 ms	59.2 ms	124.6 ms

* Not comparable: a zonal bucket rejects every write over JSON/HTTP.

On the standard buckets plain gRPC is a wash, and on a smaller e2-standard-4 it was consistently slower, so the docs say to measure before enabling it. The win is the zonal bucket: 2.9x faster than NAM4 and 5.1x faster than multi-region.

Testing

go test ./blob/gcsblob/ passes and the existing replay tests are unaffected. New unit tests cover the grpc and zonal parameters, their invalid values, the zonal=true plus grpc=false conflict, and OpenBucketGRPC including that the caller's client survives bucket.Close.

TestConformanceGRPC runs the full drivertest suite over gRPC. Against a US multi-region bucket all 88 checks pass, repeatedly.

It records to and replays from testdata/TestConformanceGRPC, the same way TestConformance does. 79 golden files, 748 KB. Four consecutive offline replays with credentials pointed at nonexistent paths give identical results.

Getting there needed three fixes to grpcreplay, all now merged and released in v1.5.0, which master already depends on:

go-replayers#70	storage reads objects with a zero-copy codec, so RecvMsg is handed a *mem.BufferSlice rather than a proto.Message and grpcreplay panicked on the first read.
go-replayers#71	A subtest that makes no RPCs produced a 9-byte golden file the replayer rejected with "missing initial state". Seven subtests here do that.
go-replayers#72	Concurrent writes over bidirectional streams were matched to recorded streams by method alone, non-deterministically. Replaying one recording three times gave three different answers.

The golden files need regenerating. grpcBucketName is currently a personal bucket, because the bucket name is part of every recorded gRPC request and a replay only matches against the bucket that was recorded. It is deliberately a separate constant from bucketName so that regenerating the gRPC files cannot invalidate the JSON/HTTP ones. The constant and testdata/TestConformanceGRPC have to be regenerated together, with --record.

I also verified against a real Rapid Storage bucket that writes, reads, attributes, range reads and 1 MiB writes all succeed where they fail outright on master, and that objects written through zonal=true come back finalized, size=23 with a non-zero Finalized, against size=0 without the FinalizeOnClose line.

TestConformanceGRPCZonal is skipped

It is present but skipped unconditionally, because it cannot pass. Deleting one t.Skip enables it once specific conformance tests can be disabled. Against a Rapid Storage bucket, 55 checks pass and 33 fail, none of them driver bugs:

• Rapid storage class objects do not support rewrite accounts for three of the five failing groups. Only TestCopy is about copying; TestKeys and TestAs copy incidentally, and TestKeys alone contributes 19 failures because it copies once per weird key.
• Listing with a delimiter other than / fails with Invalid argument. That is a hierarchical namespace restriction, which Rapid Storage inherits by requiring HNS, rather than anything to do with zonal buckets or gRPC.
• TestWrite hits the per-object mutation rate limit. Not a Rapid Storage limit at all, just the general GCS cap, and it only appeared when running from a VM in the bucket's zone, fast enough to trip it. The failure count varied with distance: 27 from a laptop, 33 from in-zone.

drivertest cannot express any of this. Its only opt-out is the Unimplemented error code, and all six places that honor it guard SignedURL; testCopy treats any error from Copy as a failure. The test's doc comment names the exact subtests to disable and why, so it can be switched on by deleting one t.Skip once selective disabling exists.

What this PR does not do

It does not deliver the sub-millisecond reads Rapid Storage is advertised for.

Sub-millisecond is possible, but under certain conditions. You only get it on later reads of one particular object, from a process that has kept a bidirectional read stream open to that object. The first read of any given object costs 11 to 12 ms no matter which API you use.

Same 1 KiB object, same client, n=1000:

Read	p50	min
Never read before, via NewReader. This is what the driver does.	12.0 ms	6.7 ms
Never read before, by opening a MultiRangeDownloader for it	11.1 ms	6.3 ms
Same object again, on the stream already open	0.95 ms	0.59 ms
Same object again, with a ReadHandle cached from the earlier read	4.3 ms	2.6 ms

Getting there means keeping state alive between reads. Two ways:

• Cache open MultiRangeDownloaders per object on the driver's bucket. Reaches the third row. Needs idle expiry, cleanup from bucket.Close, and an adapter from Add's callback to io.Reader.
• Cache ReadHandles per object. Much smaller, but only reaches the fourth row.

Neither has anywhere to live today. driver.Bucket.NewRangeReader hands back a fresh driver.Reader per call, and the portable layer makes a new one for every read and discards it on a Seek.

What changed since the review

• Options.UseGRPC and Options.UseZonalAPIs removed. Replaced by DialGRPC and OpenBucketGRPC, per your gcpkms suggestion.
• The half-used gcp.HTTPClient is gone from the gRPC path. You were right that it was odd. It was not entirely ignored, its OAuth2 token source was reused, but everything else about it was dropped, and when the transport was not exactly an *oauth2.Transport the code silently fell back to option.WithoutAuthentication. A caller with a wrapped transport got an anonymous client and 403s at request time. Now a nil token source means unauthenticated explicitly, and a gRPC URL with neither credentials nor anonymous=true is an error.
• Client ownership fixed. See the note above; Close used to leak.
• Rebased onto master, which picks up 12ac9a59 and fixes the golangci-lint failure. That failure was never related to this change; it was Go 1.27 against golangci-lint v2.12.
• TestConformanceGRPC and TestConformanceGRPCZonal added, with the caveats above. The zonal one is skipped unconditionally with a comment naming the subtests that need disabling and why, per your suggestion.
• TestConformanceGRPC is now a record/replay test with committed golden files, passing 88 of 88. This needed three grpcreplay fixes, all merged and released in v1.5.0.
• Migrated off experimental.WithZonalBucketAPIs, which storage v1.67.0 removed. It is now storage.WithAppendableUploads() plus storage.WithGRPCBidiReads(), both of which have graduated out of experimental, so this PR no longer depends on an experimental package.

Two things I would rather you decide:

1. Should Options.Client be marked deprecated now that OpenBucketGRPC exists? It is released API, added in ab74300b, so it cannot be withdrawn, but it is now a second way to do the same thing.
2. Should the grpc and zonal URL parameters stay? You said the URL option is fine, and I have kept them, but they are the part of this PR that adds surface area. The argument for them is that a caller configured purely by a connection string has no other way to reach a Rapid Storage bucket; anyone writing Go already has OpenBucketGRPC.

[vangent]

Can you merge with HEAD? I think that might fix the golangci-lint problem.

[vangent]

So, the "client *gcp.HTTPClient" passed in to the constructor here is getting ignored? That seems odd.

Maybe enforce that client is nil to make that more clear?

[vangent]

OpenKeeper for KMS has a constructor that takes a client (so that the caller can do whatever with it); maybe that's a better pattern here?

I.e., the grpc=true URL option is fine, and controls what the URL opener does, but there's no "UseGRPC" Option; instead, there are two separate OpenBucket constructors, one for HTTP and one for gRPC, where the latter takes a storage.Client, and we provide a Dial to create it pre-wrapped similar to KMS ("cloudkms.NewKeyManagementClient(ctx, option.WithTokenSource(ts), useragent.ClientOption("secrets"))").

[stanhu]

@vangent I've updated this pull request to have:

func OpenBucket(ctx context.Context, client *gcp.HTTPClient, bucketName string, opts *Options) (*blob.Bucket, error)
func OpenBucketGRPC(client *storage.Client, bucketName string, opts *Options) (*blob.Bucket, error) {

I've kept grpc=true and zonal=true for query parameter support because this is the standard way to access a bucket with OpenBucketURL.

[vangent]

I don't want to merge this without running it through the conformance test.

You should be able to make a new function here, TestConformanceGRPC (and maybe another one, TestConformanceGRPCZonal), that uses a different newHarness-equivalent function (or refactor newHarness) that creates a gRPC client etc.

To generate the golden files locally you'll need to update the constants at the top of the file and run with --record. I'll ask you to revert the constant changes before merging, and I'll re-generate the golden files with our bucket after that.

[stanhu]

TestConformanceGRPCZonal currently fails for a number of reasons:

TestCopy/Works
  got unexpected error copying blob: (code=InvalidArgument):
  Rapid storage class objects do not support rewrite.

TestListDelimiters/backslash
  (code=InvalidArgument): Invalid argument.        # non-"/" delimiter on an HNS bucket

TestWrite/write_with_explicit_ContentType_overrides_discovery
  NewWriter or Close got err (code=ResourceExhausted):
  The object <rapid bucket>/blob-for-reading exceeded the rate limit for object
  mutation operations (create, update, and delete).

https://docs.cloud.google.com/storage/docs/rapid/rapid-bucket mentions that object rewrites are not supported (https://docs.cloud.google.com/storage/docs/json_api/v1/objects/rewrite).

Rapid Storage also requires / as the delimeter (https://cloud.google.com/blog/products/storage-data-transfer/understanding-new-cloud-storage-hierarchical-namespace), so those tests fail as well.

For now I'll omit TestConformanceGRPCZonal until there's a better way to selectively disable conformance tests.

[vangent]

Can you go ahead and add the test, but comment it out with some comments explaining the above? When I have time I'll try to make it easier to disable specific conformance tests (with explanation) so that they can be enabled.

[vangent]

Or "skip" rather than comment out.

[stanhu]

@vangent Done!

To generate the golden files locally you'll need to update the constants at the top of the file and run with --record. I'll ask you to revert the constant changes before merging, and I'll re-generate the golden files with our bucket after that.

In order to generate the golden files for gRPC, we need this fix for go-replayers: google/go-replayers#70

[vangent]

Done.

[vangent]

Let me know when you're ready for another round of review, you'll need to upload the golden files as part of the PR for it to pass I think.

[codecov]

Codecov Report

❌ Patch coverage is 89.02439% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.04%. Comparing base (5dbc7eb) to head (959234e).

Files with missing lines	Patch %	Lines
blob/gcsblob/gcsblob.go	89.02%	9 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3772      +/-   ##
==========================================
+ Coverage   79.97%   80.04%   +0.07%     
==========================================
  Files         104      104              
  Lines       12219    12284      +65     
==========================================
+ Hits         9772     9833      +61     
- Misses       2446     2450       +4     
  Partials        1        1              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[vangent]

I updated deps so that we've got the latest go-replayers including your contribution, so you should be able to add the gRPC replay golden files now.

[stanhu]

I updated deps so that we've got the latest go-replayers including your contribution, so you should be able to add the gRPC replay golden files now.

Thanks! It looks like go-replayers still needs a few other fixes:

• grpcreplay: write the header when a recording captures no RPCs go-replayers#71
• grpcreplay: match bidi streams on their first send, not just the method go-replayers#72

[vangent]

I updated deps so that we've got the latest go-replayers including your contribution, so you should be able to add the gRPC replay golden files now.

Thanks! It looks like go-replayers still needs a few other fixes:

• grpcreplay: write the header when a recording captures no RPCs go-replayers#71
• grpcreplay: match bidi streams on their first send, not just the method go-replayers#72

Done

[stanhu]

Thanks! I've updated the conformance tests. To re-record the golden files:

1. Point grpcBucketName at a bucket to which you can write
2. rm -rf blob/gcsblob/testdata/TestConformanceGRPC
3. go test ./blob/gcsblob/ -run 'TestConformanceGRPC$' -record

Recording needs Application Default Credentials with write access to that bucket. Restricting -run keeps it from also re-recording TestConformance, which needs a private key for SignedURL. The bucket name is part of every recorded request, so grpcBucketName and the golden files always have to be regenerated together.