fix: decode tokens to avoid re-encoding them

[stephantul]

Fixes #402

This PR decodes encoded tokens again so their surface-level forms can be safely ingested by a tokenizer. This fixes the leading space issue in #402.

[greptile-apps]

Confidence Score: 4/5

[Medium impact] Changes token encoding logic in the tokenizer.

The PR appears safe to merge, with a non-blocking request to test encoded leading spaces.

_{Reviews (1) · Last reviewed commit: "fix: decode tokens to avoid re-encoding ..." · Reviewed by Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

Files with missing lines	Coverage Δ
model2vec/tokenizer/tokenizer.py	100.00% <100.00%> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.