Skip to content

Honor unk_token in byte-level BPE - #2736

Open
galleonli wants to merge 1 commit into
huggingface:mainfrom
galleonli:fix/byte-level-bpe-unknowns
Open

galleonli wants to merge 1 commit into
huggingface:mainfrom
galleonli:fix/byte-level-bpe-unknowns

Conversation

@galleonli

Copy link
Copy Markdown

Fixes #2732.

BamiBERT currently encodes a\0b as ab because byte-level BPE drops the NUL before it applies merges. This keeps the configured unknown token in the symbol stream and honors fuse_unk for consecutive missing bytes. The whole-word lookup also checks that a multibyte character can actually be built from its byte tokens.

With BamiBERT, the IDs for that input are now [68, 3, 69], matching 0.23.3. The patched encoder also matched 0.23.3 on all 795 test strings with the BamiBERT, Profluent-E1-300M and Bingsu CLIP tokenizer configurations.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[1.0.0-rc.4] A byte-level BPE with an unk_token drops unknown bytes, where 0.23 emits the unk token

1 participant