Train a calibrated decision model with kev

Many production decisions are not open questions. Should this agent keep running, pause for a person, or stop? Is this ticket billing, shipping or technical? How risky was this trace, on a four-level rubric? The options are known, and your application needs more than an answer: it needs to know how sure the answer is, so that it can act alone when the model is sure and ask a person when it is not.

A chat model can be prompted or fine-tuned to answer such questions, but it writes text: it can answer outside your options, it takes seconds per answer, and the probability behind its words is not something you can read or trust. A decision model works differently. It reads each question with each of its candidate answers, gives every candidate a score, and turns the scores into probabilities that a fitted temperature calibrates on your own data. It never answers outside the candidates, it scores a decision in milliseconds on a small GPU, and its probabilities mean what they say. In the System 1 and System 2 picture of thinking, it is the fast, intuitive System 1: the quick, cheap call you make thousands of times a day, with a chat model kept for the slow reasoning that needs it.

This guide trains one with Tensorant, end to end, on a public dataset. Every number below comes from a real run in production on 23 and 24 October 2026, on GPUs in our own RunPod account.

What you will build

You will fine-tune skundu42/kev, a ModernBERT scorer of about 400 million parameters, on LocalLLaMA/typed-decisions. Its cases are agent traces: a state (the agent, its task, a summary of what it did), and five questions about each trace:

The gold answers are probabilities, not single labels: several annotators answered each question and disagreed, so action might be 58% continue, 29% human_review, 9% observe and 4% stop. A model trained on that learns where people disagree, and says so.

What the run cost and what it reached:

You need the Pro plan for decision models, a RunPod account and a RunPod network volume connected under Connections, and the member or owner role. See Decision models for the details.

1. Choose a decision model, or a chat model

Use a decision model when the answer is one of a few options you can name, and the probability matters:

• Routing and triage: which team, which queue, which tool.
• Gating: let an agent continue, flag it, or stop it; approve or hold.
• Rubric scoring: risk, urgency or quality on a fixed scale.
• Classification with a confidence threshold: act alone above 90%, ask a person below.

Use a chat model, trained with SFT, DPO or RL, when the answer is text you need written: a reply, a summary, an explanation, a JSON object. The two work well together: the decision model takes the frequent, cheap calls and sends the hard cases on.

The candidates can change from one question to the next. The model reads them with the question, so one model can route tickets in one product and score risk in another, as long as your training data covers both.

2. Create the decision project

1. Open the project switcher and choose New project.
2. Under Hugging Face base model, enter skundu42/kev.
3. Wait for the model check. It reads "Decision model · ModernBERT · head kept · ≈396M parameters". Head kept means training continues from kev's own scoring head, and kev's own temperature scores it before training.
4. Name the project, for example "Agent trace triage", and choose Create project.

A project's kind comes from its base model and never changes. A decision project's sidebar has Data, Tune and Use, like a chat project, without recipes or document generation.

kev starts from ModernBERT. You can also start from jhu-clsp/ettin-encoder-400m or answerdotai/ModernBERT-base, which get a new scoring head. Starting from kev, training refines a model that already scores candidates, rather than teaching a new head from nothing.

3. Import the cases

1. In Sources, choose Add sources and open the Hugging Face tab.
2. Enter LocalLLaMA/typed-decisions, choose the subset all and the split train.
3. Import as shows Typed decisions · state, questions, gold: the import found the state, questions and gold columns by name, and uses workflow as each case's category.
4. Choose Training and validation, keep Approve the cases as they arrive, and import.
5. Import the test split the same way. A test split always stays held out.

The import checks every row before it stores it. A row whose gold doesn't sum to 1, or names a candidate the question doesn't have, is skipped and listed in the import report, with the question it concerns and never your text.

For your own data, each row is a case with three fields:

{
  "state": {"ticket": 1042, "message": "My parcel arrived damaged and I want my money back."},
  "questions": {
    "route": {"type": "choice", "instructions": "Which team should handle this ticket?",
              "criteria": {"billing": "Payments and refunds", "shipping": "Delivery problems", "technical": null}},
    "refund": {"type": "noul", "instructions": "The customer is owed a refund.",
               "criteria": {"false": "No refund is owed.", "true": "A refund is owed."}}
  },
  "gold": {
    "route": {"probabilities": {"billing": 0.7, "shipping": 0.3, "technical": 0.0}},
    "refund": {"label": "true"}
  }
}

A gold can be probabilities, when several people answered, or one label. If you have a CSV of texts and labels instead, import it as Labelled texts: each text becomes a case with one choice question over your labels. See Prepare your data.

Two habits pay off later:

• Keep disagreement in the gold. If two of three reviewers chose human_review, write 0.67 and 0.33, not a single label. The model learns to be unsure exactly where people are, which is what makes its probabilities useful.
• Write candidate descriptions. "human_review: Queue this trace for a human to review" teaches the model more than "human_review" alone, and the model reads the description every time it scores the candidate.

4. Review and create a version

Open Review to read the cases: each shows its state, then each question with bars for the gold probabilities of its candidates. Fix a wrong case by choosing another Gold label, or reject it.

Then create a version under Versions. A version keeps each case whole, with all its questions in one split, and keeps cases that share a state together, so no test case leaks into training through a near copy. From typed-decisions' 1,200 training cases, the version holds 1,080 to train and 120 to validate, and the 400 imported test cases stay as the test split.

Readiness for training counts the tokens of every question and candidate pair with kev's own tokenizer. For this dataset the longest pair is 642 tokens, well under the default token limit of 1,024. A decision is never cut short: if a pair is longer than the limit, the run stops before training and says how many are over.

5. Run a trial first

A trial costs a few cents and tells you whether the full run fits the GPU, how long it will take and what it will cost.

1. In Runs, choose Configure a run, and under Run choose Training trial · a small sample.
2. Choose the version, the GPU you plan to use and the full run's settings. Keep the trial's defaults: 128 training and 16 validation cases, 20 steps, 20 minutes.
3. Save the trial draft and launch it.

Our trial on the RTX PRO 4000:

The trial's own GPU ran 10 minutes for 3 minutes of training: starting a GPU, pulling the training image and loading the model take several minutes, and a full run spends them again, plus saving the model. The real run trained for 25.5 minutes, close to the estimate, and billed 41 minutes in all. Read a trial's estimate as the time the GPU works, and add the start-up the trial itself took.

The peak memory is the number to look at before you raise the token limit or the candidate limit. At 2.5 GiB, kev with LoRA fits the cheapest 16 GB GPUs with room to spare.

6. Train

Choose Configure a run again and keep Train model. The settings that matter most:

Save the draft, look at Readiness for training, and launch it. The run then works through its steps on one GPU, and the page follows them:

1. Checking decisions: every pair against the token limit, before the model downloads.
2. Scoring base model: kev as it is, on the 2,000 test decisions, with its own temperature (1.373).
3. Training: one decision a step with all its candidates, 32 decisions an update. It checks the validation loss every tenth of the steps and keeps the best model.
4. Calibrating: it fits the temperature that gives the lowest loss on the validation split. Ours came out at 1.016.
5. Scoring tuned model: the trained model on the same 2,000 test decisions, at that temperature.
6. Exporting: the model kev loads, with its temperature.

The validation split picks the kept model and the temperature; the test split only scores. Nothing the run reports was tuned on the cases it was scored on.

7. Read the scores

When the run shows Model ready, its page shows five scores of the tuned model on the test split, each against the base model:

Accuracy barely moved, and that is expected: with split gold, almost every top answer has some gold probability, so accuracy saturates near 100% and says little. The real change is in the other four. Log loss fell by 30%, and the calibration error fell from 13.8% to 2.6%: base kev was overconfident on this data, and the tuned model's confidence now matches how often people agree with its top answer. That is what lets an application set a threshold, such as "act alone above 0.8", and get roughly the error rate it expects.

Compare evaluations opens both evaluations side by side. Break the scores down by Question or Question type to see where the model gained and where it still struggles. Open a decision to see each candidate's probability beside its gold.

Probabilities switches between Calibrated and Uncalibrated. The uncalibrated log loss of the tuned model was 0.9231 against 0.9231 calibrated: training alone had already brought the model close to calibrated, and the temperature of 1.016 only nudged it. For base kev, calibration mattered more: its own temperature of 1.373 took its log loss from 1.427 to 1.313.

8. Gate it with an experiment

An experiment turns "is this model better?" into a rule you set once and apply to every run. A decision experiment can train a new model, or gate a finished run without training anything.

1. In Experiments, choose New experiment, and the version.
2. Under Model, choose Reuse Model 1 on v1 to gate the run you just finished. It trains nothing and costs nothing.
3. Under Acceptance, choose Log loss as the main metric. Set a target of at most 1.0, a minimum improvement of 0.2, 10 minimum test cases, and tick Accept only a clear improvement.
4. Optionally tick Keep an accepted model serving, with a GPU and limits, to start a test session of the model when it passes.
5. Check readiness and the cost, then approve.

Ours finished in a minute: "Accepted: meets your thresholds. On log loss the tuned model scores 0.9231 vs 1.3135, an improvement of +0.3904 (likely +0.3672 to +0.4125), lower being better. 1,709 decisions improved, 291 got worse." The likely range comes from drawing the 400 test cases again 2,000 times, each case with all its decisions, so a gain that could be chance is not accepted. With a target of at most 0.5, the same model was rejected, with the reason in the same words.

Use the same experiment for every new version of your data. Accuracy is a poor gate for split gold, as step 7 showed; gate on log loss, or on calibration error when what you need most is trustworthy confidence.

9. Use the model

Use this model on the run offers:

• Download model: a ZIP with the weights in full precision (1.6 GB for kev), its tokenizer and kev_config.json, which holds the token limit and the fitted temperature. kev's DecisionModel loads it on your own GPU.
• Publish to Hugging Face, to your organization's account.
• Serve it with Tensorant: a test session for the Playground, and an API model your applications call on POST /v1/decisions. The next guide, Serve a decision model and call it from your app, does that, and checks that the served probabilities match the run's.

What to watch for

• GPU availability changes by the minute. A cheap GPU listed as free can be gone when the run starts; the run then fails at once with "RunPod has no GPU of this type free". Launch again on another GPU: kev with LoRA fits any 16 GB card.
• The token limit is a memory decision. Raise it only after a trial on the GPU you will use.
• Validation needs at least 50 decisions to fit a temperature. With fewer, the run keeps the base model's.
• Rerun the experiment, not your judgment. When the data changes, a new version and the same experiment tell you whether the new model is better, by the rule you already agreed on.