Teaching a 144M model to decide like a big one

Most questions to an assistant start with a routing decision: which tool answers this, which agent should take it, is it about someone else, does it ask to change something, and which language is it in. CHEF makes those calls well, but each call costs a network round trip and around 400 ms. We wanted the same decisions made locally, in the browser or on a small CPU server, fast enough that the user never notices.

So we built a custom model with 144 million parameters. It trains in about 25 minutes on one GPU and ships as a 57 MB file.

• 144MparametersModernBERT encoder plus a two-layer decision head
• ~25 minper training runone RTX 5090, four epochs
• 57 MBin the browser4-bit ONNX, 54 MB gzipped
• 127 msmedian decisionon a CPU, against 392 ms for CHEF

A model that picks from a list#

A decision is a choice between options, so the model never has to write text. In a single forward pass it reads the state, the question and one marker per option. It gives each marker a score, and a softmax over the scores gives the answer and its probability.

Because there is no decoding loop, the latency is just the cost of encoding the input. After training we fit a temperature on held-out rows, so "0.9 sure" means right about nine times in ten, and you can act on confident answers and pass the rest to a bigger model. The same model also handles any number of options, from two to thirteen.

The body is an open multilingual encoder, Julia-1, which builds on mmBERT-small. It is a ModernBERT with 22 layers, a hidden size of 384 and 6 attention heads. Every third layer uses full attention and the rest use a 128-token sliding window. A small two-layer head on top turns each option marker into a score. That comes to 144.3M parameters, and we cap the input at 1,024 tokens.

CHEF as the teacher#

Everything here starts from CHEF, which did two jobs.

First, it labelled the data. We generated questions with a cheap LLM and had CHEF answer each one exactly as the production system would ask it, with the same option groups in the same order and the same wording. For every row we kept CHEF's full probability distribution as well as its top pick.

Second, it refereed. The generator also wrote down which answer it meant each question to have, and we kept a row only when CHEF agreed. When they disagreed, a second cheap LLM judged the pair, and anything still unsettled went to a review file instead of into training. About 3,200 rows ended up there.

Going straight from CHEF to 144M parameters was a big jump, so we added a step in between. We fully fine-tuned a 0.8B model (Qwen3.5-0.8B-Base) on the CHEF-labelled data. On this narrow task it beat CHEF on our replay bench (96.5% against 92.9%), which made it a strong teacher. That 0.8B model then labelled every row with its full distribution, and the 144M student learned from those soft labels.

The data#

All generation ran on deepseek-v4.1-flash through OpenRouter, which is cheap. The prompts asked for:

• questions for each option, in Norwegian, Swedish, Danish and English, in batches of eight options so the generator could see the neighbours each option has to be told apart from;
• near misses, meaning questions that sound like an option but belong to none of them, so the model learns to answer "other";
• questions typed the way people type: short and long, with typos and missing diacritics.

We added production questions to the generated rows, split by session so one conversation can't end up on both sides of the split, along with hand-checked real questions.

Two augmentations made a clear difference. In the first, each question got four extra rows with the options reordered, and the right option kept in only half of them. The model can't learn that the answer is usually first, and it learns to say "none of these". In the second, half the rows got a "sloppy twin": a lowercased copy with the closing punctuation dropped, and sometimes the æøå (å→a, ø→o) and the commas too. A hash of the text picks which rows get a twin, so the choice is the same on every run. The version that added sloppy twins moved live answers from 187 right and 21 strictly wrong to 197 and 13.

We also trimmed the vocabulary. The encoder knows 256,000 tokens, and most of them never appear in Nordic or English text. We kept every token that appears at least three times in a Wikipedia sample of the four languages, plus any token in the training data. That left 118,937 tokens and shrank the tokenizer from 32.8 to 6.8 MiB. All 4,077 sample rows still tokenized to the same ids as before.

By the last version the training mix held about 330,000 rows.

Training rows per version

Training#

Training uses plain PyTorch and Hugging Face transformers on a single RTX 5090 under WSL.

Most of the loss comes from the teacher's distribution because it carries more than the hard label does. When the teacher splits 60/40 between two options, the student learns that those options are close. A hard label throws that away.

A run took 9 minutes at the start and 27 minutes by the end, because the data grew with every version. Runs that short let us try an idea, measure it and drop it in the same afternoon.

After training, a grid search between 0.4 and 4.0 finds the temperature that best calibrates the validation rows. The last model settled on 0.8.

Two seeds per candidate#

Two runs of the same recipe with different seeds differed by about nine right answers on the live eval. That was more than many of our ideas gained. From then on we trained every candidate on two seeds and judged it on the mean. For the last versions we averaged the two seeds' weights into one model, and those averaged versions have the best scores in the series.

Shipping it#

We export to ONNX (opset 18) and quantize the encoder. The decision head stays in full precision.

*4 threads, batch of 16.

For the browser we pack the 4-bit embedding so it is safe for WebGPU and store the head in fp16. That gives a 57 MB file, or 54 MB gzipped. The server uses int8. On 4 threads it took 349 ms per decision against 620 ms, and it chose differently from full precision on only 1 decision in 70.

How well it works#

We compared it with CHEF on a replay bench of 142 real calls:

Intent, top-1 accuracy

1. CHEF92.9%
2. 0.8B teacher96.5%
3. Custom 144M, 4-bit91.5%

The small model lands about a point and a half below CHEF on intent and gets every language call right. It runs on a CPU with no network, in a third of CHEF's time. Its calibration error (ECE 0.033) is close to CHEF's (0.029), so you can rely on its confidence the same way.

Median latency per decision

1. CHEF (API)392 ms
2. 0.8B teacher93 ms
3. Custom 144M, 4-bit, CPU127 ms

On held-out data the final model picked the right tool 1,875 times out of 1,893 (99.0%). Only 4 of those decisions were both confident and wrong. It identified the language 627 times out of 634.

In the live eval, which runs 383 questions end to end through the real system, the number answered correctly and instantly went from 159 at version 7 to 220 at version 22. Strictly wrong answers stayed between 12 and 16. The other questions fall back to a bigger model, and calibrated confidence is what makes that handoff safe.

Live eval, 383 questions

• Answered right, instantly
• Strictly wrong

The loop#

The model went through 22 versions in about two weeks, and almost every version followed the same loop:

1. Run the live eval and read the wrong answers as well as the score.
2. Group the misses. Is it one kind of question the model has never seen, two options it confuses, or a style of typing?
3. Generate rows aimed at that group, including contrast triples: three questions that look alike but have different answers.
4. Have CHEF label them, keep only the rows it agrees with, and retrain on two seeds.
5. Keep the version if the mean improves and strictly wrong answers don't rise.

Step 5 caught real regressions. One version added contrasts between similar policies, and its strictly wrong answers jumped to 50. We dropped it the same day.

Train your own#

The recipe doesn't depend on our task. If you have a large model that makes good typed decisions, whether that's CHEF or your own, these are the steps:

1. Write down the decisions. List every option with a one-line description, and fix how the question and options are shown to the model. Training and runtime must match exactly.
2. Generate questions with a cheap LLM for each option and each language, including near misses and real-world typing. Record the answer each question was meant to have.
3. Label the questions with the big model and keep its full probability distribution. Keep a row only when the big model agrees with the generator, and send disagreements to review or to a judge model.
4. Hold out a test set before anything else. Split by a hash of the text, by session or by whole eval set, so translations and rewrites can't leak into training.
5. Augment the rows with reshuffled option lists, and with sloppy twins chosen by a hash.
6. If you like, fine-tune a teacher of about 1B parameters on the labels, then use its distributions to label every row, including the augmented ones. Drop augmented rows where the teacher disagrees with the expected answer.
7. Trim the vocabulary to the languages you need, and check that the token ids don't change.
8. Train the small encoder on two seeds with a loss that is mostly soft. Average the seeds, then fit a temperature.
9. Export to ONNX and quantize: int8 for CPU servers, 4-bit for the browser. Measure accuracy after quantizing.
10. Run the live eval, read the misses and go back to step 2.

In production, set a confidence threshold. Answer instantly above it, and hand everything below it to the big model. In our case the small model came within about a point and a half of CHEF's intent accuracy at a third of the latency.