Tokenizer Extension for Amharic Whisper

This is a research note on a small change with a large effect: extending Whisper's tokenizer to recognize Ge'ez characters. We ran a controlled four-condition study on Amharic and found that the standard fine-tuning recipe leaves a portion of the model's output capacity unused. Tokenizer extension recovers it without changing the training data, the model architecture, or the compute budget; only the tokenizer and its embedding rows differ.

The widget below lets you watch a sentence run into the wall.

The cause turns on two architectural facts. Whisper's decoder has a hard 448-token output ceiling, set by the dimension of its learned positional embeddings. And Whisper's tokenizer, trained on a corpus dominated by English and other high-resource European scripts, has no Ge'ez tokens of its own. Each Ge'ez character decomposes into its three UTF-8 bytes, and each byte costs a token. A long Amharic sentence accumulates tokens roughly six times faster than its Latin-script equivalent, and the decoder runs out of room while the encoder is still listening.

There is a second story tucked inside the first. When you fine-tune a Whisper-style model on a Hugging Face pipeline, the reference transcript is tokenized with the same tokenizer for loss computation. References that overflow 443 tokens are truncated at the token level, mid-character. Word error rate is then computed against the truncated reference. Missing trailing content cannot produce deletion errors against the model's hypothesis, so reported WER is artificially low. The pipeline systematically flatters byte-fallback configurations. Tokenizer-modification comparisons reported under the standard pipeline understate the benefit by a margin that turns out to be substantial.

How substantial depends on which utterance you look at.

The story of the WER gap is mostly contained in the 7.76% of utterances where the default tokenizer's reference exceeds the budget. On those, a script-aware tokenizer cuts the error rate by 7.44 points, from 17.51% to 10.07%. Most of the gap is rescue from truncation, not better modeling. On the other 92.24% of utterances where neither tokenizer overflows, there is still a smaller advantage of 1.38 points. The default tokenizer's six-times-more autoregressive steps per character mean six times more chances for the decoder to compound an error.

What we did

The result above came from a four-condition design. Same training data¹, same architecture, same compute, same hyperparameters. Only the tokenizer and its embedding rows changed. The four conditions: Whisper's unmodified default, an extended tokenizer with 7,432 Ge'ez-aware tokens added to Whisper's vocabulary, a replaced tokenizer trained from scratch on Amharic only, and a control that adds 7,432 tokens drawn from Hangul, a script that never appears in our training or evaluation transcript.

+The control is what isolates tokenization quality from parameter count. It adds the same parameter count as the extended tokenizer but the added rows never receive gradient signal during training, since Hangul codepoints never appear in our Amharic data. If the control tracks the default and the extension outperforms both, the gain is from tokenization rather than from added embedding capacity. The control did track the default within noise. The extension outperformed both. Replacement, despite having the most efficient tokenizer of all four at roughly one token per two characters, lost anyway because reinitializing the decoder discards too much pretrained knowledge. Extension preserves the pretrained signal and captures most of the tokenization gain at the same time, and is the better engineering choice.

Decode latency drops by approximately threefold. Same model, same audio, fewer autoregressive steps per character.

Who else this affects

The languages on the right inherit the same architectural ceiling as Amharic, with strictly worse pretraining priors and no language tag at inference time. The tokenizer extension demonstrated here is script-level rather than language-specific. It applies wherever the script encodes. Tigrinya, Tigre, Bilen, the Gurage cluster, Ge'ez itself: dropping the same extended tokenizer onto a fine-tune for any of them recovers the same capacity, with the truncation arithmetic identical and the per-step gradient signal denser than under byte-fallback².

The 448-token cap is a design constant; it penalizes scripts in proportion to their distance from Whisper's tokenizer training distribution. The languages that lose the most output capacity are the ones least likely to have the engineering resources to diagnose the loss. Whisper does not have an Amharic problem; it has a tokenizer-architecture interaction that affects every byte-fallback language it encounters.

The full paper is available on ResearchGate.