Engineering LLMs for Low-Resource Languages: How a Sinhala MMLU entry reached 64.25% on the progression leaderboard and 61.87% in the official final evaluation

How structured inference-time methods improved performance, why Sinhala tokenisation presents challenges, and what Sri Lankan students and engineers can learn from AI shared tasks

By Mohamed Saheed Mohamed Riswan

When people test a large language model (LLM) such as Gemini, Gemma or Qwen in English, the interaction can feel effortless. Ask it a university-level question in physics, history or computer science, and it may respond with considerable fluency.

Switch the same question into Sinhala (or Tamil, or another underrepresented language), and performance can change. Models that score well on English benchmarks may struggle with subject-specific questions, misinterpret nuanced phrasing or use more tokens to process the same content.

Why does this happen? And how can engineers and students improve LLM performance in low-resource languages without access to a multi-million-dollar GPU cluster?

At the ICTer 2026 Sinhala MMLU Shared Task, hosted by the University of Colombo School of Computing and the Informatics Institute of Technology, participants were challenged to improve multi-task language understanding in Sinhala. This article examines the benchmark, the engineering approach that moved one entry from a 50.93% baseline to 64.25% on the progression leaderboard, and its 61.87% result in the official final evaluation. It also considers possible tokenisation and model-adaptation challenges, before outlining practical lessons for undergraduates, graduates and developers in Sri Lanka interested in AI shared tasks.

What Is the Sinhala MMLU Shared Task (and why does it matter)?

MMLU (Massive Multitask Language Understanding) is a widely used benchmark for evaluating how well LLMs handle knowledge and reasoning across academic and professional subjects, including STEM, the humanities and social sciences.

The Sinhala MMLU task evaluates a model’s ability to understand questions and select answers in Sinhala across three difficulty bands: Easy, Medium and Hard.

For Sri Lanka’s digital ecosystem, including education, healthcare, legal technology and public services, language capability is an important part of making AI more accessible. Shared tasks such as Sinhala MMLU provide a common benchmark through which researchers, students and software engineers can test approaches against a standardised evaluation dataset.

From a 50.93% baseline to 64.25% on the Progression Leaderboard: What worked (and what did not)

In an LLM competition, it is tempting to treat progress as a prompt-guessing exercise. A more systematic approach is to treat the benchmark as a software engineering pipeline, measuring each change against a consistent baseline.

A Two-Stage approach to Low-Resource LLM Engineering: Inference-Time Reasoning and Tokeniser and Parameter Adaptation.

Lesson 1: Cross-family ensembling can reduce accuracy

A common assumption in machine learning competitions is: “If one model is good, combining two different model families (like Qwen + Gemma) in a majority vote will be even better.”

In this set of experiments, the results did not support the assumption that combining two model families would necessarily improve performance:

• Gemma Direct Baseline: 93% macro-average accuracy
• Cross-family voting (Qwen + Gemma): 46.30% macro-average accuracy (4.63 percentage points below the baseline)

One possible explanation is that model families differ in their exposure to Sinhala text and their handling of domain terminology. If one model is less reliable or differently calibrated on Sinhala questions, combining its predictions with those of a stronger model may introduce errors rather than improve the result. The experiment shows that this particular voting approach performed worse; it does not, on its own, establish why.

Lesson 2: Standard Chain-of-Thought (CoT) vs. Pass-1 Structured Thinking

Next, we tested standard Chain-of-Thought (Gemma CoT), asking the model to “think step by step” before answering:

• Gemma Direct: 93%
• Gemma standard CoT: 51.92% (a 0.99 percentage-point gain)

One possible limitation of open-ended reasoning is that the model may produce inconsistent explanations or shift terminology as it generates its response. The results here show only a modest improvement from the standard CoT prompt; further testing would be needed to identify the reasons.

Observed Progression Leaderboard Results — Pass-1 structured thinking was associated with a 10.00-point increase, while later consensus-voting configurations raised the leaderboard score further.

The largest increase in this sequence—a 10.00 percentage-point rise from 51.92% to 61.92%—followed the introduction of a Pass-1 thinking configuration:

1. Explicit problem decomposition: The prompt asks the model to identify the core subject, assess the factual basis of each multiple-choice option (A, B, C and D), eliminate distractors and then provide a final answer.
2. Structured output extraction: In multiple-choice benchmarks, formatting differences can cause answer-extraction errors. The team used deterministic post-processing to make answer extraction more consistent and reduce the risk of lost predictions.

Lesson 3: Calibrated Consensus Voting (v5 to v9: a further 2.33-point leaderboard gain)

After the single-pass configuration reached 61.92% on the progression leaderboard, the team tested same-family, multi-pass consensus voting. The reported progression was v5: 62.68%; v6: 63.17%; v8: 63.89%; and v9: 64.25%. The 64.25% figure was the progression leaderboard result; the entry’s macro-average accuracy in the official final evaluation was 61.87%.

Across these configurations, the progression leaderboard score increased by a further 2.33 percentage points. However, because the changes were not tested in isolation, the results do not establish that consensus voting alone accounted for the full increase.

Progression Leaderboard Accuracy by Difficulty Band — The highest score was recorded on Hard questions (71.13%).

The reported progression leaderboard results by difficulty band were as follows:

The Hard band recorded the highest score (71.13%), while Medium was the weakest (56.35%). One possible explanation is that the structured approach was better suited to questions requiring formal or multi-step reasoning. Medium questions may have placed greater emphasis on nuanced wording or contextual knowledge. These are interpretations of the results rather than conclusions established by the scores alone.

What could improve next: Tokenisation and Model Adaptation

As the progression leaderboard result rose from a 50.93% baseline to 64.25%, while the official final evaluation score was 61.87% – how could performance improve further? One area worth investigating is how the model’s tokeniser handles Sinhala text.

During the ICTer 2026 workshop, Juyeong Ji of Google Japan delivered a technical session on Gemma’s SentencePiece tokeniser, which has a vocabulary of 262,144 tokens. The session highlighted tokenisation features that may be relevant when working with languages that have less representation in training data.

ICTer 2026 workshop session by Juyeong Ji (Google Japan) on the Gemma 4 SentencePiece tokeniser, its 262,144-token vocabulary, byte fallback and reserved tokens.

The challenge: Subword fragmentation and byte fallback

An LLM processes text as numerical token IDs rather than as words in the way people read them. In English, some common words, such as “phones” or “engineering”, may be represented by a single token, although tokenisation varies by model and context.

Tokenisers are shaped by the data on which they are trained. If a language or script is less represented in that data, some words may be split into smaller subword units rather than represented as single tokens. Depending on the tokeniser, byte fallback may also be used for characters or sequences that are not otherwise represented.

1. Subword fragmentation: A Sinhala word may be split into several smaller units. The number of tokens depends on the word and the tokeniser; the extent of fragmentation should be measured rather than assumed.
2. Byte fallback: Some tokenisers can represent otherwise unsupported text using byte-level tokens. Whether and how often this occurs should be checked for the specific model and input.

Why might this matter?

• Token and context overhead: If Sinhala text is split into more tokens than an equivalent English passage, it may use more of the model’s context window and increase processing costs. The size of this difference needs to be measured for each model and dataset.
• Potential effect on processing: Fragmented tokenisation can make text representation less efficient. However, the claim that this directly reduces a model’s capacity for semantic reasoning would require further evidence and should not be inferred from token counts alone.

A possible research direction: Reserved token slots

One idea raised in discussions of Gemma’s vocabulary is whether reserved token IDs could be used when adapting a tokeniser for Sinhala. The existence and practical availability of reserved slots, and the work required to use them, should be verified against the specific model version and its documentation. Reserved IDs should not be assumed to be ready-made embedding capacity for new words.

If researchers establish that a model can be adapted in this way, a possible workflow would include:

1. Vocabulary curation: Analyse Sinhala corpora, such as textbooks, Wikipedia, news and domain-specific datasets, to identify frequent words, stems and compound forms that are heavily fragmented by the existing tokeniser.
2. Tokeniser and embedding adaptation: If the model architecture and licence permit it, investigate assigning selected Sinhala units to new token IDs and updating the associated tokeniser and model embeddings. This is a model-adaptation task, not simply a matter of filling unused slots.
3. Continued pre-training and post-training: Newly introduced tokens need suitable learned representations. Depending on the adaptation method, this may involve continued pre-training on Sinhala text, followed by instruction fine-tuning or other post-training methods.

Such an approach could reduce token overhead in some cases, but its effect on language understanding and reasoning would need to be tested. It should be treated as a research direction, not a guaranteed route to higher accuracy.

A practical playbook: How to approach AI shared tasks

You do not need a PhD or a large corporate research lab to contribute to AI shared tasks such as ICTer, Kaggle competitions, SemEval or Hugging Face benchmarks. For computer science undergraduates, recent graduates and software engineers moving into AI, the following steps offer a practical starting point:

Step 1: Build a reproducible evaluation harness on day one

Before trying fancy prompts or complex ensembles, write a clean, reproducible script that:

• Runs the unmodified baseline model (v0).
• Logs exact accuracy across every category and difficulty band.
• Saves every prompt, raw model response, parsed answer, and error case to a structured JSON/CSV file.

Rule of thumb: If you cannot measure whether a change improves accuracy by 0.5 percentage points, it is difficult to know whether the change has helped.

Step 2: Change one variable at a time

The progression (50.93%, 51.92%, 61.92%, 62.68%, 63.17%, 63.89% and 64.25%) records the leaderboard results across successive configurations. When cross-family Qwen + Gemma voting returned 46.30%, the result indicated that this approach was not beneficial in the tested setup. Keeping detailed logs made it possible to compare configurations and focus subsequent experiments on the structured-thinking approach.

Step 3: Audit parser and formatting failures

In multiple-choice and structured NLP benchmarks, some incorrect results may stem from answer-extraction errors—for example, when a model gives an answer in a format the parser does not recognise. Reviewing a sample of raw outputs can help identify these problems and show whether they are affecting the score.

Step 4: Test structured thinking before larger ensembles

Rather than starting with an expensive five-model ensemble, first test what can be achieved with a single model and a clearly structured prompt. Then evaluate whether consensus voting adds a measurable improvement. The best approach will depend on the task and the results of controlled experiments.

Step 5: Look beyond prompting to tokenisation and data quality

Prompt design is one part of LLM engineering. Understanding tokenisation, data quality, embedding adaptation and fine-tuning can help developers investigate why a model performs differently across languages. Tools such as interactive SentencePiece tokenisers can be used to inspect how a model processes input text.

Why Sri Lankan students and engineers should take part

Large technology companies are investing in foundation models, but locally grounded research remains important for improving Sinhala and Tamil language support, educational benchmarks and regional domain coverage.

The Sinhala MMLU Shared Task at ICTer illustrates how systematic experimentation and careful evaluation can help individual contributors and student teams investigate the capabilities and limitations of open-weight models such as Gemma.

When the next shared task, hackathon or open benchmark is announced, consider getting involved: explore the dataset, establish a baseline and begin testing ideas.

Featured image courtesy: Pahasara (UCSC Media) — ICTer 2026