auth: import FileTokenCache into CLI and wire DualWrite

[simonfaltum]

Stack

Entry point for the opt-in secure token storage work. Review and merge top-to-bottom:

1. auth: import FileTokenCache into CLI and wire DualWrite #5056 auth: import FileTokenCache into CLI and wire DualWrite (this PR)
2. libs/auth/storage: add dormant secure-storage foundation #5008 libs/auth/storage: add dormant secure-storage foundation
3. auth: wire secure-storage cache into CLI #5013 auth: wire secure-storage cache into login/token/logout

This PR is also the first of a separate 3-PR sequence that moves file token cache ownership from the SDK to the CLI. That sequence (SDK PR removing the internal dual-write, then SDK bump in the CLI) can proceed in parallel with #5008 and #5013.

Why

First of 3 PRs moving file-based OAuth token cache ownership from the Go SDK to the CLI. Today the SDK owns both the cache interface and the file-backed implementation, including the dual-write-under-host-key convention. Long-term we want the SDK to stop owning persistence: the OAuth flow and cache interface stay, but file format and host-key conventions move to the CLI. This unblocks secure storage backends and Renaud's Session model on a cleaner foundation.

This PR imports the cache into the CLI and wires it everywhere. Nothing is deleted from the SDK yet. PRs 2 and 3 (SDK PR, then SDK bump) finish the move.

Changes

Before: CLI relied on the SDK's default FileTokenCache. Dual-write to the legacy host-based key happened inside PersistentAuth.Challenge() and refresh() via the SDK's internal dualWrite.

Now: CLI owns its own FileTokenCache at libs/auth/storage/file_cache.go, a near-verbatim copy from the SDK (same JSON schema, same path ~/.databricks/token-cache.json, same permissions). A new storage.DualWrite(cache, arg, token) helper centralises the dual-write convention. Every u2m.NewPersistentAuth call site in the CLI now passes u2m.WithTokenCache(...).

The SDK is unchanged. It still dual-writes internally, so the two writes hit the same file with the same keys and bytes, i.e. idempotent. Zero user-visible behavior change.

Files touched:

• libs/auth/storage/file_cache.go + test (new)
• libs/auth/storage/dualwrite.go + test (new)
• cmd/auth/login.go, cmd/auth/token.go, cmd/auth/logout.go
• libs/auth/credentials.go
• NEXT_CHANGELOG.md

Lint-driven deltas from SDK:

• os.UserHomeDir() is forbidden in the CLI, uses env.UserHomeDir(ctx) instead. Required threading ctx through NewFileTokenCache.
• os.IsNotExist(err) is forbidden, uses errors.Is(err, fs.ErrNotExist).

Known edge case: Tokens that exist only under the legacy host key (users who logged in before profile-keyed writes existed and never re-ran auth login --profile) keep working for now because the SDK's internal dualWrite still runs. After PR 2 (SDK stops dual-writing), re-login will be required for those users. Minimal impact.

Test plan

• Unit tests for file_cache_test.go (port of SDK tests)
• Unit tests for dualwrite_test.go covering no host-key provider, distinct host key, host-key-equals-primary, discovery with populated/empty GetDiscoveredHost
• make checks and make test pass
• Manual smoke test of databricks auth login, auth token, auth logout on a live profile before merging

[github-actions]

Approval status: pending

/cmd/auth/ - needs approval

Files: cmd/auth/login.go, cmd/auth/logout.go, cmd/auth/token.go
Suggested: @mihaimitrea-db
Also eligible: @tanmay-db, @tejaskochar-db, @renaudhartert-db, @hectorcast-db, @parthban-db, @Divyansh-db, @chrisst, @rauchy

/libs/auth/ - needs approval

5 files changed
Suggested: @mihaimitrea-db
Also eligible: @tanmay-db, @tejaskochar-db, @renaudhartert-db, @hectorcast-db, @parthban-db, @Divyansh-db, @chrisst, @rauchy

General files (require maintainer)

Files: NEXT_CHANGELOG.md
Based on git history:

• @denik -- recent work in ./, cmd/auth/

_{Any maintainer (@andrewnester, @anton-107, @denik, @pietern, @shreyas-goenka, @renaudhartert-db) can approve all areas.
See OWNERS for ownership rules.}