Skip to content

Add offline end-to-end generation test for the generate helper - #13

Open
tylrcc wants to merge 1 commit into
DaBestCode:mainfrom
tylrcc:fix/issue-2
Open

tylrcc wants to merge 1 commit into
DaBestCode:mainfrom
tylrcc:fix/issue-2

Conversation

@tylrcc

@tylrcc tylrcc commented Nov 11, 2026

Copy link
Copy Markdown

Closes #2

What this adds

GenerateHelperEndToEndTests in tests/test_transformers.py: an offline, CPU-only end-to-end test that goes through the public randkv.transformers.generate helper (the existing generation tests construct RandKVCache directly instead).

  • Tiny local model: a randomly initialized Qwen3ForCausalLM (2 layers, hidden 32, vocab 32, fixed seed) built from a config in the test — no checkpoint download.
  • Local tokenizer inputs: a fixed in-test character vocabulary maps the prompt to token ids — no downloaded tokenizer, no network at all.
  • Enough tokens for multiple eviction rounds: greedy generation of 24 tokens with RandKVConfig(budget=24, buffer_size=4).

Assertions

The helper creates its cache internally, so the test model records the past_key_values it is handed and the test inspects that cache:

  • output is prompt + 24 tokens, prompt preserved;
  • at least two eviction rounds per layer (4+ observed), positions actually evicted;
  • cache bound: physical tokens per layer never exceed budget + buffer (28), while absolute tokens seen reach 41;
  • monotonic absolute position tracking: retained positions per head are strictly increasing, the prompt positions are all protected, and the newest token is always retained;
  • determinism: two runs from the same seed produce identical output.

Verification

  • pytest tests/ → 45 passed (43 existing + 2 new).
  • Each new test's call takes ~0.5s locally, well under the 10-second bar; Python 3.10/3.13-compatible (no new syntax or dependencies).

GenerateHelperEndToEndTests exercises the public
randkv.transformers.generate path end to end on CPU:

- a tiny randomly initialized Qwen3 model (2 layers, vocab 32),
  built locally - no checkpoint download;
- tokenizer inputs from a fixed in-test character vocabulary - no
  downloaded tokenizer;
- greedy generation of 24 tokens against a budget of 24 + buffer 4,
  forcing multiple eviction rounds;
- assertions on the cache the helper creates (captured via the
  past_key_values handed to the model): physical size never exceeds
  budget + buffer, at least two eviction rounds per layer, absolute
  position tracking advances past the physical bound, retained
  positions per head are strictly increasing with the prompt fully
  protected and the newest token retained;
- a determinism check: same seed, same output.

Each test runs in about half a second.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add an offline tiny-model Transformers integration test

1 participant