Skip to content

Isolate the XDG data directory so a real login cannot leak into core tests - #51

Merged
its-janghoon merged 1 commit into
developfrom
feature/isolate-core-test-data-dir
Sep 21, 2026
Merged

its-janghoon merged 1 commit into
developfrom
feature/isolate-core-test-data-dir

Conversation

@its-janghoon

Copy link
Copy Markdown
Contributor

Three ModelsDev tests were green in CI and red on any developer machine where somebody had run redrob providers login.

Why

They clear REDROB_API_KEY and assert the static fallback catalog is served. But resolveApiKey() reads three sources in order:

source stubbed by the suite?
process.env.REDROB_API_KEY cleared
the Integration store stubbed via credentialLayer
auth.json under Global.Path.data no — the developer's own credential

So the service found a key, fetched, and returned the live listing's figures — limit.output: 64000 where the fallback publishes 32000. The assertion was reporting the difference between two catalogues rather than the thing it claims to check, and nothing in the test named the variable it depended on.

This is the same shape as the variants bugs earlier in this series: a test that is green for a reason unrelated to its claim.

The fix

XDG_DATA_HOME points at a fresh temp directory in the preload, which is early enough — Global.Path.data is computed once at module load, so redirecting it afterwards is impossible. The suite's own REDROB_TEST_HOME seam is a getter and covers home only, not the data directory where credentials live.

Done in the test setup rather than by adding a seam to production code: nothing about the CLI's real path resolution is wrong, and XDG_DATA_HOME is the standard way to say where that directory is.

Verified by control

Removing only the XDG_DATA_HOME line brings back exactly those three and nothing else:

(fail) ModelsDev Service > get() uses the static fallback and issues NO request when no REDROB_API_KEY is set
(fail) ModelsDev Service > the static fallback catalog names exactly the models the console serves
(fail) ModelsDev console model metadata > the no-key fallback carries a usable context window for every model
 3 fail

Gates

typecheck      0 errors across the monorepo
packages/core  974 pass / 0 fail   (was 971/3)

… into core tests

Three ModelsDev tests were green in CI and red on any developer machine where
somebody had run `redrob providers login`. They clear `REDROB_API_KEY` and then
assert the static fallback catalog is served -- but `resolveApiKey()` reads THREE
sources in order: that env var, the Integration store, and `auth.json` under
`Global.Path.data`. The suite stubs the second and clears the first. The third was
the developer's own credential.

So the service found a key, fetched, and returned the live listing's figures --
`limit.output: 64000` where the fallback publishes `32000`. The assertion was
reporting the difference between two catalogues rather than the thing it claims to
check, and nothing in the test named the variable it depended on.

`XDG_DATA_HOME` is pointed at a fresh temp directory in the preload, which is
early enough: `Global.Path.data` is computed once at module load, so redirecting
it afterwards is impossible -- the suite's own `REDROB_TEST_HOME` seam is a getter
and covers `home` only, not the data directory where credentials live.

Fixed in the test setup rather than by adding a seam to production code. Nothing
about the CLI's real path resolution is wrong, and `XDG_DATA_HOME` is the standard
way to say where that directory is.

Verified by removing only the isolation: exactly those three fail again, and
nothing else in the 974 does.
@its-janghoon
its-janghoon merged commit cc0c4be into develop Sep 21, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant