CloudFetch and M2M token exchange are unbounded and uninterruptible (v1.13.0)

[hannahapuan]

Version

v1.13.0

Summary

CloudFetch result downloads are unbounded and uninterruptible, and an expired
presigned link ends the read with no recovery path. Four issues, the first three
in the download path and the fourth on the auth path.

1. The caller's context is discarded

internal/rows/rows.go:622 and :624 pass r.ctx into NewArrowRowScanner,
but internal/rows/arrowbased/arrowRows.go:124 substitutes context.Background()
when constructing the iterator:

bi, err2 = NewCloudBatchIterator(context.Background(), rowSet.ResultLinks, ...)

So the context given to QueryContext never reaches the download. Cancelling the
query, or its deadline expiring, cannot stop an in-progress file fetch.

2. The download client has no timeout and no public setter

internal/rows/arrowbased/batchloader.go:62 falls back to http.DefaultClient,
which has no Timeout:

httpClient := http.DefaultClient
if cfg.HTTPClient != nil {
    httpClient = cfg.HTTPClient
}

CloudFetchConfig.HTTPClient has no ConnOption. The only way to populate it is
WithTransport, which also replaces the Thrift transport as a side effect, and
the http.Client it builds (connector.go:421) sets no Timeout either.

Related: #307 asked for a configurable CloudFetch transport and was closed as
completed. WithTransport addresses the transport case, but a RoundTripper
cannot set http.Client.Timeout, so the timeout case above is still open.

Together with (1), a stalled download has neither a timeout nor working
cancellation. Because downloads are lazy (cloudIPCStreamIterator.Next(),
batchloader.go:200), the caller is blocked inside row iteration, so a deferred
rows.Close() cannot fire to break the stall.

3. Expired links are terminal

batchloader.go:453 returns errors.New(dbsqlerr.ErrLinkExpired). That constant
appears nowhere else in the module apart from its own definition: nothing handles
it and nothing re-issues links. A result set large enough that its presigned URLs
expire part-way through reading cannot be completed, only restarted.

CloudFetchConfig.MinTimeToExpiry defaults to 0 (internal/config/config.go:557
clamps only negative values) and likewise has no ConnOption, so a caller cannot
widen the safety margin.

4. The token exchange is unbounded and serialized

auth/oauth/m2m/m2m.go:41 holds authClient.mx across the token fetch, and the
token source is built with context.Background() (m2m.go:73), so
clientcredentials falls back to http.DefaultClient with no timeout. A hung
token endpoint therefore stalls every in-flight request on that connector
indefinitely, with no way for the caller to bound it. Endpoint discovery is
capped at 10s (auth/oauth/oauth.go:21), but the exchange itself is not.

Also, a doc mismatch

WithCloudFetch's comment (connector.go:429) says "Default is false", but
CloudFetchConfig.WithDefaults() sets UseCloudFetch = true unconditionally
(internal/config/config.go:546). CloudFetch is on by default.

Suggested fixes

• Pass the scanner's context through to NewCloudBatchIterator instead of
  context.Background().
• Expose HTTPClient and MinTimeToExpiry as ConnOptions, so a caller can bound
  downloads without displacing the Thrift transport. Note that WithTransport is
  not a substitute: a RoundTripper cannot set http.Client.Timeout, so it can
  only bound each hop of a redirect chain rather than the request as a whole.
• Bound the token exchange, and consider not holding the mutex across it.
• Re-fetch result links on expiry, or if that is out of scope, document the
  condition as terminal so callers can plan for it.

[hannahapuan]

Confirmed this still reproduces on v1.15.1, not just v1.13.0. Checked all four call sites, code is unchanged, only line numbers moved:

• arrowRows.go:125 still passes context.Background() to NewCloudBatchIterator
• batchloader.go:62 still falls back to http.DefaultClient, and there's still no ConnOption for HTTPClient or MinTimeToExpiry
• batchloader.go:453 still returns ErrLinkExpired with nothing handling it; config.go:752 still clamps only negative values
• m2m.go still holds the mutex across a context.Background() token exchange

One retraction: the WithCloudFetch doc mismatch I mentioned is fixed in v1.15.1, it now correctly says "Default is true." Ignore that part.