cubestore: web identity S3 credentials are only refreshed when the token file changes, so they expire after ~1h

[karthikb-appcard]

Summary

spawn_creds_refresh_loop in remotefs/s3.rs does refresh S3 credentials, but in web identity mode it re-derives them only when the token file's mtime changes:

// In web identity mode, only refresh when the token file changed.
if let (Some(ref file), Some(_)) = (&token_file, &role_arn) {
    let current_modified = std::fs::metadata(file).ok().and_then(|m| m.modified().ok());
    if current_modified == last_modified {
        continue;
    }
    ...
}

Credentials::from_sts returns AssumeRoleWithWebIdentity credentials, which expire in about an hour. But the projected service account token is only rotated by the kubelet near its own expiry, and on EKS the default (eks.amazonaws.com/token-expiration) is 86400s, rotated at roughly 80% of that.

So for ~19 hours the token file is unchanged, the loop continues past every opportunity to refresh, and the credentials it derived at startup are already dead. The effect is exactly one working hour of uploads per pod lifetime, then every upload fails until the pod restarts.

Observed

On a cubestore router using IRSA:

• uploads succeed for ~1 hour after each pod start, then fail continuously with ExpiredToken, at the cachestore upload cadence:

ERROR [cubestore::util] Error during Cachestore upload: CubeError { message: "AWS S3 error:
Got HTTP 400 with content '<Error><Code>ExpiredToken</Code>
<Message>The provided token has expired.</Message>..."

• four separate restarts over a month each produced one working hour and nothing after it
• the projected volume showed expirationSeconds: 86400, and the loop was running (the 30s web-identity poll interval is applied) — it just never passed the mtime check

Consequences beyond failed uploads: remote snapshot pruning cannot advance, because delete_old_snapshots() only deletes what was successfully uploaded, and the remote metastore copy goes stale indefinitely.

Workaround

Shorten the projected token's lifetime so the kubelet rotates it often enough to drive the refresh:

# ServiceAccount annotation; 600 is the EKS minimum
eks.amazonaws.com/token-expiration: "600"

The kubelet then rewrites the file every ~480s, the mtime changes, and the loop re-derives. This fully resolves it for us (zero ExpiredToken since), at the cost of ~180 extra STS calls per day. It requires recreating pods, since the expiry is baked into the projected volume at admission.

That works, but it means correct behaviour depends on a cluster-side annotation that has no obvious connection to cubestore, and the default configuration is broken in a way that looks like partial support: it works just long enough to pass a smoke test.

Suggested direction

The mtime check is a reasonable fast path, but it should not be the only trigger. Options, roughly in order of preference:

1. Track the credentials' own expiry and refresh when they are close to it, independent of the token file.
2. Refresh on a timer in web identity mode as well, with an interval well under the STS session length (the token file remains valid, so re-calling from_sts with the same JWT is enough).
3. At minimum, document that web identity mode requires a short token-expiration, since nothing in the current behaviour hints at it.

Happy to send a PR for whichever shape you prefer — I did not want to pick the refresh policy on your behalf.

Not the same as #6795

#6795 states that assuming an IAM role is not implemented. That reads as stale: the CUBESTORE_AWS_WEB_IDENTITY_TOKEN_FILE / CUBESTORE_AWS_ROLE_ARN path is honoured and does work, for one hour. Support is present but the refresh is gated, which is a narrower and more confusing failure than no support at all. I could not find an existing issue covering the refresh gap specifically.

Version: cubejs/cubestore:v1.6.57, and the code above is unchanged on master.