bpe-openai: use Regex::find instead of captures for pretokenization#121

aneubeck merged 3 commits into

github:maingithub/rust-gems:mainfrom

dmatth1:pretok-find-fast-splitdmatth1/rust-gems:pretok-find-fast-splitCopy head branch name to clipboard