Add knap, a byte level BPE tokenizer in pure Mojo

[olaflaitinen]

Adds recipes/knap/, a recipe and its test file for Knap, a byte level BPE tokenizer written in pure Mojo.

• Repository: https://github.com/olaflaitinen/knap
• Release: https://github.com/olaflaitinen/knap/releases/tag/v1.0.0
• Licence: EUPL-1.2
• Maintainer: @olaflaitinen

What the package contains

Path	What it is
$PREFIX/lib/mojo/knap.mojoc	The precompiled library, importable with no include flag
$PREFIX/bin/knap	A command line tool, so installing gives a working knap count rather than a library you have to write a program against
$PREFIX/share/{bash-completion,zsh/site-functions,fish/vendor_completions.d}/	Completions for the three shells

No vocabulary is bundled. Vocabulary files are third party data under their own terms, and bundling them would attach a licence question to this package that fetching them does not.

What it does

All seven tiktoken encodings: cl100k_base, o200k_base, o200k_harmony, p50k_base, p50k_edit, r50k_base and gpt2. Byte identical with tiktoken over a 110 MB corpus, 191762320 tokens, plus 20 million differentially fuzzed inputs with zero divergences. No Python interpreter at run time.

Verification

Built locally with rattler-build from this recipe at this path, and the test section run against the installed artefact: knap-1.0.0-hb0f4dca_0.conda, 341 KiB, completions present, knap version printing 1.0.0, and mojo run -I $PREFIX/lib/mojo smoke.mojo importing the package and decoding through it.

One deliberate deviation, and why

The run requirement is mojo-compiler ==1.0.0 rather than pin_compatible('mojo-compiler').

pin_compatible derives a range from whatever resolved at build time, and a range is the one thing a precompiled Mojo artefact cannot honour: the compiler's documentation says a .mojoc file may not be compatible with another compiler version, and the ABI is explicitly not stable. An exact pin turns that into a solver error at install time instead of a link error deep in a consumer's build. The reasoning is written into the recipe rather than left in this thread, so it survives.

Happy to change it if the channel prefers consistency here.

Maintenance

The recipe is maintained in the Knap repository at conda.recipe/recipe.yaml and copied here. A gate there checks on every push that the declared version agrees with CITATION.cff, that the compiler pin agrees with pixi.toml and pyproject.toml, that source.rev is a commit in that repository, and that every path the build script reads exists at that commit. That last check was added after the recipe pinned a commit older than its own build script and the package build died on a file that had been added months later.

[Copilot]

🟡 Changes recommended

Two moderate recipe validation issues remain unresolved.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds Knap, a pure Mojo byte-level BPE tokenizer, as a reproducible conda package with its library, CLI, completions, and smoke tests.

Changes:

• Pins Knap source and Mojo compiler compatibility.
• Builds and installs the library, CLI, and shell completions.
• Adds installation and decoding validation.

File summaries

File	Summary	Findings
recipes/knap/smoke.mojo	Verifies installed-library import and decoding.	None.
recipes/knap/recipe.yaml	Defines the build, dependencies, metadata, and package tests.	Two moderate findings: validate knap version against ${{ version }} (2 votes); remove -I so default library discovery is tested (1 vote).

Review details

Suppressed comments (1)

recipes/knap/recipe.yaml:115

• Because this invocation supplies -I, the smoke test can pass even if the installed .mojoc is not discoverable through the package's promised default ${PREFIX}/lib/mojo path. The build comment immediately above says that path needs no include flag, and other precompiled-package smoke tests in this repository compile without one; remove the override so this acceptance test covers the advertised install behavior.

        mojo run -I "${PREFIX}/lib/mojo" smoke.mojo

• Files reviewed: 2/2 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[olaflaitinen]

Both findings were right, and both are now fixed in b4df4a8.

knap version was not validated. The test ran it and discarded the output, which proves the binary starts and nothing else. A package built from the wrong revision would have passed. It now compares that output against the version the recipe declares, which are two independent sources of the same fact:

test "$("${PREFIX}/bin/knap" version)" = "${{ version }}"

The -I override made the smoke test vacuous. This is the sharper of the two. The build script installs into ${PREFIX}/lib/mojo specifically so the compiler finds the package with no include flag, and the comment above it says so. Passing -I meant the test would have passed even if that promise were broken, so the one thing it existed to cover was the one thing it could not fail on. Removed.

I checked rather than assumed that default discovery actually works, since if it did not, the right fix would have been to correct the claim instead. Building the recipe from recipes/knap/recipe.yaml:

+ test 1.0.0 = 1.0.0
+ mojo run smoke.mojo
knap package smoke test: import and decode both work
all tests passed!
Artifact: output/linux-64/knap-1.0.0-hb0f4dca_0.conda (341.80 KiB)

So the promise held and is now proven rather than stated.

The remaining deviation from the house style is the exact compiler pin in run, which I described in the pull request body. Happy to switch to pin_compatible if the channel prefers consistency; the reasoning for the exact pin is that a precompiled .mojoc is tied to the compiler that produced it, so a range would turn a solver error at install time into a link error inside a consumer's build.