Skip to content

Fix sampler accepts and the Mirostat vocabulary size in LlamaLanguageModel - #429

Merged
mattt merged 3 commits into
mainfrom
mattt/fix-llama-samplers
Nov 18, 2026
Merged

mattt merged 3 commits into
mainfrom
mattt/fix-llama-samplers

Conversation

@mattt

@mattt mattt commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator

This PR fixes two defects in how LlamaLanguageModel samples tokens. The text and image generation loops called llama_sampler_accept after llama_sampler_sample, which already accepts the token that it returns. Of the samplers in the package's chain, only the penalties sampler has an accept that affects output here, and it recorded each generated token twice. Its window of repeatLastN entries then held half as many generated tokens (32 instead of the default 64), and each occurrence in that shorter window counted twice for the frequency penalty. The repeat and presence penalties check only whether a token is in the window, so only the shorter window changed them. Separately, llama_sampler_init_mirostat got the context size as its n_vocab argument instead of the vocabulary size.

This PR removes the extra accept calls, passes the vocabulary size to Mirostat v1, and builds the sampler chain for all three generation paths with one function, makeSamplerChain(vocabularySize:options:). Because the package does not accept prompt tokens into the sampler, generated text can change from the 2nd generated token for callers that set a frequency penalty. For callers that use only repeat or presence penalties, including the default repeat penalty of 1.1, it can change from the 34th generated token with the default repeatLastN. Generated text can also change for callers that use Mirostat v1.

@mattt
mattt merged commit 0e547a5 into main Nov 18, 2026
14 checks passed
@mattt
mattt deleted the mattt/fix-llama-samplers branch November 18, 2026 14:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant