From 10.5 to 27 BLEU on Twi English: Notes on Fine Tuning NLLB for a Ghanaian Language Pair

A small, honest write up of two Twi English translation models I published to Hugging Face, what worked, what did not, and what the numbers actually mean.

Context

Machine translation for the languages I grew up around has a specific shape of failure. Google Translate covers Twi. It also covers it badly. The Bible has been translated. Wikipedia has some articles. Beyond that, the parallel data available for training is scattered, small, and often noisy. The gap between "supported" and "usable" is the interesting one, and it is where most of the work happens.

I trained and published two models on Hugging Face for the Twi to English direction. This is the write up.

Models:

ninte/twi-en-marianmt: a MarianMT baseline
ninte/twi-en-nllb-v2: a fine tuned NLLB 200 distilled 600M

The dataset

Roughly 10,000 parallel Twi to English sentences, aggregated from several open corpora and light cleaning. The exact split for the two models was 10,370 for the MarianMT baseline and 9,600 for the NLLB fine tuning, after another pass of filtering.

The number to keep in mind is the size. Ten thousand pairs is not a lot. Modern MT training runs happen on billions of pairs. Every design choice below is about squeezing signal out of a corpus that a bigger project would consider too small to work with.

Baseline: MarianMT

I started with Helsinki NLP's MarianMT because it is small, fast to fine tune on consumer GPUs, and its Twi English variant already existed as a starting point. Fine tuning for a few epochs on the cleaned pairs gave a functional model that scored 10.5 BLEU on the held out test set.

10.5 is a low number in absolute terms. The way to read it is not "the model is 10 percent good." It is: the model produces syntactically plausible English for most inputs, gets the topic right most of the time, and is wrong on specific lexical choices roughly as often as it is right. For a first pass with a small corpus, it is the reasonable baseline to beat.

The move to NLLB

NLLB 200 was trained on 200 languages, including many low resource ones. Fine tuning it for a specific low resource direction takes advantage of two things at once: the model already understands the shape of low resource translation, and its multilingual encoder brings enough cross lingual signal that a small target corpus has an outsized effect.

I fine tuned NLLB 200 distilled 600M on 9,600 Twi English pairs. The final BLEU on the same held out test set was 27.

The delta from 10.5 to 27 is not a marketing number. It is what happens when you swap a specialised small model for a large multilingual one that already carries prior information about the target task. This is close to what I would predict from the literature, and the point of running the experiment was less to be surprised and more to have the model in hand.

What 27 BLEU actually means

BLEU is a proxy. Its correlation with human judgment on low resource languages is worse than on high resource ones, because the reference translations themselves are small and the test set does not span the linguistic diversity of the language. On sentences that resemble the training distribution the model is usable. On dialect variation, code switching with English, or topical domains that were not in the training corpus, it fails in predictable ways.

The failure modes I saw most often:

Named entities that appear zero times in training get mapped to phonetically similar English words.
Idiomatic Twi phrases that do not decompose word by word are translated literally.
Tone information, which Twi carries but which the ASCII orthography does not always encode, is lost when the ambiguity could have been resolved from context.

None of these are surprising. They are the standard failure modes for a small corpus, and they are what a future project would target.

What I would change

More parallel data, but not naively. The obvious next step is more pairs. The less obvious step is that scraped pairs from parallel corpora contain a lot of near duplicates and English source translations that are themselves suspect. Growth in effective training size will come from careful curation, not raw scraping.

Include Asante Twi and Akuapem Twi as separate labels. The current models treat "Twi" as a single variety, which is a simplification the data does not actually support. Adding a language tag conditional on dialect would let the model handle each variety more honestly.

Back translation. Standard trick for low resource MT. Generate synthetic Twi from a monolingual English corpus using the current model, filter aggressively for quality, add to training. Nine thousand real pairs plus fifty thousand carefully filtered synthetic pairs would very likely push BLEU further.

A cleaner evaluation suite. The BLEU number I report was on the split I had available. A held out test set that spans religious, news, conversational, and code switched inputs, released alongside the model, would make the numbers easier to interpret. This is a small amount of work with a large amount of value for the field.

Why publish

The strongest reason to put these models on Hugging Face is that Twi speakers who need a starting point for their own work should not have to redo the boring parts. The corpus, the fine tuning code, and the checkpoints are the reusable pieces. The BLEU number is the least interesting artifact.

The second reason is that machine translation research for underrepresented languages has a long history of being announced and never released. Publishing something imperfect that other people can build on is closer to the norm this space needs than perfect work that stays private.

What comes next

Adelani group work on Ghanaian and West African languages under the MasakhaNER and AfriBERTa line is the standard I want to hold future iterations of this to. That includes rigorous evaluation, honest reporting, and resources that are useful outside the specific paper they were built for. This piece of work is a starting point rather than a finished thing.

Links

Models: ninte/twi-en-marianmt, ninte/twi-en-nllb-v2
Portfolio: i-ninte.github.io/portfolio

Kwabena Obeng

Comments (0)

Leave a comment