Azure: indexer fails with storage error(kind=Unauthorized) exactly 24h after startup when using Workload Identity

[cds-nitaliya]

Describe the bug

When Quickwit authenticates to Azure Blob Storage via Azure Workload Identity (no access_key configured), the indexer loses access to object storage exactly 24 hours after process start, and never recovers on its own.

WARN uploader: quickwit_indexing::actors::uploader: Failed to upload split. Killing!
     cause=failed uploading key 01K….split in bucket azure://<container>/quickwit-indexes/<index>
    0: storage error(kind=Unauthorized, source=Azure error wrapper(inner=non-io error occurred which will not be retried))

Every split upload fails from that moment on. Because the error is classified non-retryable, the indexing pipeline is killed and never recovers — the only remedy is restarting the pod, after which it works for exactly another 24 hours.

The root cause is that azure_identity 0.21 reads AZURE_FEDERATED_TOKEN_FILE once, at credential construction, and reuses that assertion for the lifetime of the credential.

Steps to reproduce

1. Run Quickwit on AKS with Azure Workload Identity enabled (azure.workload.identity/use: "true", service account annotated with a client ID, webhook injecting AZURE_CLIENT_ID / AZURE_TENANT_ID / AZURE_FEDERATED_TOKEN_FILE).
2. Configure Azure storage without an access key, so the token-credential path is taken:

   default_index_root_uri: azure://<container>/quickwit-indexes
   storage:
     azure:
       account: <account>

3. Run an indexer with a continuously ingesting source and wait 24 hours.

Expected behavior

The credential is refreshed transparently and indexing continues indefinitely, as it does with S3 / IRSA.

Actual behavior

At T+24h every object storage write fails with kind=Unauthorized until the process is restarted.

Configuration

1. quickwit --version:

   quickwit version: 0.9.0 (x86_64-unknown-linux-gnu 2026-07-28T09:58:34Z 4b2775e)
   

   Image: quickwit/quickwit:edge@sha256:687cea395588fb328a413b3ef5ef58889597a164eae43f4006f69e89d505ab6a

2. Relevant node.yaml:

   default_index_root_uri: azure://<container>/quickwit-indexes
   storage:
     azure:
       account: <account>   # no access_key -> token credential path

   Loaded config confirms access_key: None:

   storage_configs: StorageConfigs([Azure(AzureStorageConfig { account_name: Some("<account>"), access_key: None })])
   

---

Root cause analysis

AzureBlobStorage delegates credential selection to azure_identity when no access key is set:

https://github.com/quickwit-oss/quickwit/blob/main/quickwit/quickwit-storage/src/object_storage/azure_blob_storage.rs#L178-L184

With the three workload-identity env vars present, the resolution chain is
create_credential() → DefaultAzureCredential → EnvironmentCredential → WorkloadIdentityCredential.

In azure_identity 0.21.0, that credential reads the token file exactly once, inside create(), and stores it:

// azure_identity-0.21.0/src/token_credentials/workload_identity_credentials.rs
let token = std::fs::read_to_string(token_file.clone())?;   // read ONCE
return Ok(WorkloadIdentityCredential::new(..., token));
...
async fn get_token(&self, scopes: &[&str]) -> Result<AccessToken> {
    federated_credentials_flow::perform(..., self.token.secret(), ...)  // frozen copy, forever
}

Two lifetimes then collide. Both measured on a live cluster:

Token	Lifetime	Measurement
Projected service-account token (the client assertion)	1 hour	kubectl create token --audience api://AzureADTokenExchange → exp - iat = 3600; pod spec shows expirationSeconds: 3600
Entra ID access token for https://storage.azure.com/.default	24 hours	live token exchange → expires_in: 86399

Sequence:

1. T+0 — the token file is read once, exchanged, and a 24h access token is cached in TokenCache.
2. T+0 → T+24h — kubelet rotates the token file hourly, but Quickwit never re-reads it. No refresh is attempted, so the growing staleness stays invisible and everything works normally.
3. T+24h — TokenCache::is_expired() fires and the exchange is retried with an assertion that expired 23 hours earlier. Entra rejects it (AADSTS700024: Client assertion is not within its valid time range).
4. DefaultAzureCredential falls through to App Service / IMDS / Azure CLI credentials, none of which are available in the pod, so the whole chain fails and surfaces as kind=Unauthorized.
5. No recovery, because the frozen assertion lives in a credential owned by a long-lived pipeline. Only a process restart re-reads the file.

Observed timing matches to the second — pod started 2026-08-08T09:12:10Z, first Unauthorized at 2026-08-09T09:12:19Z: 24h 00m 09s. Across a week of restarts the interval was consistently 24.2–25.4h (the drift is human reaction time).

Only the indexer is affected. StorageResolver::resolve() constructs a fresh Storage — and therefore a fresh credential and a fresh file read — on every call. The janitor and searcher re-resolve per operation and stayed healthy for 67h+ uptime on the same service account; the indexer resolves once when the indexing pipeline spawns and holds that credential for the process lifetime.

Upstream status

This is a known, fixed bug in the Azure SDK:

• Issue: Azure Workload Identity - Expired token Azure/azure-sdk-for-rust#1739 — "Azure Workload Identity - Expired token", same AADSTS700024 signature
• Fix: Periodically read workload identity token from file Azure/azure-sdk-for-rust#1997 — "Periodically read workload identity token from file" (re-reads every 10 minutes)

The fix shipped in azure_identity 0.22.0.

Why simply bumping the dependency does not work

azure_storage and azure_storage_blobs were never published beyond 0.21.0 — the legacy Rust SDK storage crates are end-of-life. azure_storage 0.21 requires azure_core ^0.21, and StorageCredentials::token_credential() takes an Arc<dyn TokenCredential> from azure_core 0.21, whereas azure_identity ≥ 0.22 implements azure_core ≥ 0.22's trait — a different, incompatible trait. The new-SDK replacement azure_storage_blob is currently 1.1.0-beta.1.

So the fix cannot arrive via https://github.com/quickwit-oss/quickwit/blob/main/quickwit/Cargo.toml#L378-L385 without migrating off the storage crates.

Proposed fix

A small, self-contained wrapper inside quickwit-storage that implements azure_core 0.21's TokenCredential and re-reads AZURE_FEDERATED_TOKEN_FILE (mirroring what upstream #1997 does), used in place of azure_identity::create_credential() at azure_blob_storage.rs:182. No dependency changes, no API break, and it can be scoped to the workload-identity case with a fallback to the existing behaviour otherwise.