Decision models

A decision model answers typed questions about a text: which team should handle a request, whether it needs a refund, how urgent it is on a scale. Each question comes with its candidate answers. The model scores every candidate and returns a probability for each, calibrated with a temperature fitted on your own data. kev is one: a ModernBERT encoder of about 400 million parameters.

A decision project trains such a model on your own decisions, with LoRA or full training on one GPU. Each run scores the base model and the tuned model on your test cases, fits the temperature that calibrates the probabilities, and keeps a model that you download, publish to your Hugging Face account and load with kev. Decision models need the Pro plan: see Plan. Tensorant doesn't serve them yet: see What decision projects don't have yet.

When to use a decision model

Use a decision model for routing, triage, moderation and scoring, where you know the options and need probabilities you can act on, such as sending a case to a person below 70%. It never answers outside its candidates, and the candidates can differ from one question to the next: the model reads them with the question.

Before you start

• A decision project. See Create a decision project.
• The Pro plan, to create the project and launch its runs. See Plan.
• A version with training, validation and test cases. See Prepare your data.
• A GPU account and storage, connected by an owner. See Training runs.
• A training run left in your plan this month. A decision run counts as any training run does, and so does an evaluate run.
• The member or owner role.

Create a decision project

A project's kind comes from its base model when you create it, and never changes.

1. Open the project switcher and choose New project.
2. Under Hugging Face base model, enter a decision model, or pick one under Decision models (on the Pro plan) in the suggestions: skundu42/kev, jhu-clsp/ettin-encoder-400m or answerdotai/ModernBERT-base. See Supported models.
3. Wait for the model check. For kev it says "Decision model · ModernBERT · head kept · ≈396M parameters".
4. Enter a Project name and choose Create project.

What the check says about the model's head:

Encoders of other families, BERT, RoBERTa, DeBERTa v2 and v3, XLM-RoBERTa, ELECTRA and DistilBERT, are refused: "Tensorant trains decision models on ModernBERT and Ettin encoders." On a plan without decision models, the dialog shows the plan message with Upgrade, and Create project is unavailable.

A decision project's sidebar has Data (Sources, Review, Versions) and Tune (Runs, Evaluations). Home shows its next step: add sources, review the cases, create a version, then train or evaluate.

What decision projects don't have yet

A decision project has no Playground, Models, Experiments or Recipes: Tensorant doesn't serve decision models yet, and doesn't generate decisions from documents. It trains with the decision objective alone, so no SFT, DPO or RL, and has no training trials. Agents and the CLI can't work in decision projects yet. The API refuses the work of those pages, DPO, RL and every request of an agent with "Decision projects can't use this yet."

To use a trained model, download it or publish it to Hugging Face, and load it with kev.

How a run works

A case is one state, such as a support ticket or an agent's trace, with one or more typed questions about it. Each question is one decision: its candidates, and the gold probability of each.

A decision is scored the way kev scores it. Its question becomes one prompt, such as this one for the first question of the typed-decisions row below:

Decision type: choice
Instructions: What should the observability system do with this trace?
State:
{"agent": {"autonomy": "checkpointed", "model": "internal-agent-v1"}, "task": "Rotate the expired TLS certificate on the staging load balancer.", "trace_summary": {"constraint_violations": 0, "irreversible_actions": 0, "steps": 11, "tool_errors": 0}}
Criteria:
0: continue: Let the agent proceed without interruption.
1: human_review: Queue this trace for a human to review.
2: observe: Keep running, but flag the trace for later sampling.
3: stop: Halt the agent now.

Each candidate is read as the pair of that prompt and the candidate, and gets one score. The probabilities are the softmax of the scores divided by the temperature. A temperature above 1 makes the probabilities less sure, below 1 more sure.

A run, on one GPU:

1. Checks every decision, before the model's weights download: each prompt and candidate pair must fit in the Token limit, and each decision may have at most Most candidates. A decision is never cut short: the run stops instead, naming how many decisions are over. See When something goes wrong.
2. Scores the base model on the version's test split, with the model's own temperature (1 for a model without one). This is its first evaluation.
3. Trains with LoRA or full training, one decision a step with all its candidates. The loss compares each decision's probabilities with its gold probabilities, so gold that splits between candidates trains the model to be unsure between them. Every tenth of the steps it measures the loss on the validation split and saves the model, and it keeps the one with the lowest validation loss. With LoRA, the added weights are then merged into the model.
4. Calibrates: it fits the temperature that gives the lowest loss on the validation split, between 0.05 and 20. With fewer than 50 validation decisions it keeps the base model's temperature, and the log says so.
5. Scores the tuned model on the test split, with that temperature. This is its second evaluation.
6. Exports the model kev loads.

The validation split picks the kept model and fits the temperature, and the test split is used only to score. Unlike a chat run, a decision run reads the test cases on the GPU machine, to score them there.

Prepare your data

A decision project imports cases in one of two shapes. It imports no documents: its import takes JSON Lines, JSON, CSV, TSV, YAML or Excel rows.

Typed decisions

Each row is a case: a state, its questions by identifier, and the gold answer of each question. This is a row of LocalLLaMA/typed-decisions, shortened to three of its five questions and a shorter state, and written as objects for reading:

{
  "workflow": "agent_trace_observability",
  "state": {"agent": {"autonomy": "checkpointed", "model": "internal-agent-v1"}, "task": "Rotate the expired TLS certificate on the staging load balancer.", "trace_summary": {"constraint_violations": 0, "irreversible_actions": 0, "steps": 11, "tool_errors": 0}},
  "questions": {
    "action": {"type": "choice", "instructions": "What should the observability system do with this trace?", "criteria": {"continue": "Let the agent proceed without interruption.", "human_review": "Queue this trace for a human to review.", "observe": "Keep running, but flag the trace for later sampling.", "stop": "Halt the agent now."}},
    "needs_review": {"type": "noul", "instructions": "This trace requires human review.", "criteria": {"false": "No human attention is warranted.", "true": "A human should inspect this run."}},
    "risk": {"type": "score", "instructions": "How risky was the agent's behaviour in this trace?", "criteria": ["Benign: read-only or clearly safe actions.", "Low: routine writes within scope.", "Moderate: irreversible or out-of-scope actions.", "High: destructive, security-relevant, or policy-violating actions."]}
  },
  "gold": {
    "action": {"label": "continue", "probabilities": {"continue": 0.583333, "human_review": 0.29, "observe": 0.09, "stop": 0.036667}},
    "needs_review": {"label": "false", "probabilities": {"false": 0.643333, "true": 0.356667}},
    "risk": {"label": "1", "probabilities": {"0": 0.3, "1": 0.616667, "2": 0.076667, "3": 0.006667}}
  }
}

The dataset keeps state, questions and gold as JSON text in its columns, as a CSV cell can. The import reads questions and gold from JSON text or objects alike. A state that is text is kept as it is; any other JSON value becomes JSON text with sorted keys, as kev writes it.

A question has a type, its instructions and its criteria, which give its candidates:

The gold of a question is the first of these it has:

• probabilities: a probability for every candidate key, summing to 1. Several annotators who disagree, as in typed-decisions, give a split gold.
• label: one candidate key, which gets probability 1.
• noul, for a noul question: the probability of true.
• score, for a score question: the level that gets probability 1, counted from 0.

Other fields of a gold entry, such as typed-decisions' confidence, are left out. Probabilities are stored with 4 decimals: 0.583333 becomes 0.5833.

Map the columns under Column mapping, with Import as set to Typed decisions · state, questions, gold: State column, Questions column, Gold column and Category column (optional). A column left empty is found by its name: state, questions and gold. Rows with these three columns are proposed as typed decisions, and a workflow column, as typed-decisions has, becomes the category.

Labelled texts

Each row is a text and its label, such as a CSV of support messages:

text,label
I was charged twice this month.,billing
The app crashes when I open settings.,technical
How do I change the email on my account?,account

With Import as set to Labelled texts · a text and its label, each row becomes a case: its text is the state, and one choice question asks for its label among the import's labels.

• Labels lists the candidates: 2 to 16 distinct labels. It is filled in from the label column of the first rows: remove a label with its ×, add one with Add a label, or choose Use the labels of the rows. A row whose label is not listed is skipped.
• Question identifier (optional) names the question in results: label unless you change it.
• Instructions (optional) say what the model is asked about each text: "Choose the label that fits the text." unless you write your own.
• A number or true or false label is matched as written, such as 1 or true.
• A Hugging Face class column holds class numbers, so its labels are the numbers (0, 1...), not the class names.

Map Text column, Label column and Category column (optional). Labels have no descriptions: to describe the candidates, import typed decisions instead.

Import the cases

1. In Sources, choose Add sources, and upload files, paste rows or load a Hugging Face dataset, as for any import. See Sources.
2. Check the mapping. Preview of the first N rows. The import checks every row. shows the cases as the mapping reads them. For files and pasted rows, it also marks the rows that can't be used, with the reason.
3. Choose what to use the rows for: training and validation, or held out for test. See Training or evaluation.
4. Keep Approve the cases as they arrive ticked, or untick it to read each case in Review first.
5. Import.

A row that can't be used is skipped and listed in the import report with its reason, which names question identifiers and never your text, such as "Question route: gold probabilities must sum to 1". See When you import.

To import typed-decisions itself, load LocalLLaMA/typed-decisions on the Hugging Face tab and choose the subset all, or one workflow's. Import the train split for Training and validation, then the test split, which always stays held out. all has 1,200 training and 400 test cases of 5 questions each.

Review and versions

In Review, a case shows its State (JSON indented), then each question with its type, its instructions, and its candidates with bars for their gold probabilities. Gold label names the candidate with the highest. Choosing another label sets its probability to 1 and the others to 0, and Save returns the case to pending. Probabilities themselves can't be edited. Approve and Reject work as for examples, and Category is under More.

A version of a decision project holds approved cases. It keeps each case whole, all its questions in one split, and keeps cases that share a state, or the same decision, together in one split. Versions shows each split's cases and their decisions.

Readiness for training counts the tokens of every prompt and candidate pair of the cases it checks, with the model's own tokenizer: Longest pair (tokens), Decisions over the limit, Token limit and Decisions checked. It checks a sample of the cases, as for examples, and a run checks every decision before it trains.

Configure a decision run

Saving a run never starts a GPU.

1. Open Runs under Tune and choose Configure a run.
2. Under Run, keep Train model. To score a model without training, see Evaluate a model.
3. Training objective is Decision · candidate scoring. Choose the Training method: LoRA, or Full training · every weight, more GPU memory. Choosing one sets Learning rate to its own default.
4. Choose the Dataset version.
5. Choose the Training provider, Training GPU, Time limit (hours) and Price limit ($/hour), as for any training run. A decision run has no baseline to choose: it scores the base model itself.
6. Under Decision settings, check Token limit, Most candidates, Learning rate and Epochs. See Settings.
7. Look at Readiness for training below the form.
8. Choose Save training draft, then launch it as you would any run. See Launch a run.

Number of GPUs shows 2, 4 and 8 with "(decision models train on one GPU)", and they can't be chosen. On a plan without decision models, the form shows the plan message with Upgrade, or See plans if you are not an owner. A draft still saves.

Settings

Under Decision settings:

Under Advanced settings:

Every decision run also uses a weight decay of 0.01, a warmup over the first 3% of the steps, AdamW, bfloat16 arithmetic over full-precision weights, gradient checkpointing and seed 42.

GPU memory

The model check estimates a decision model's GPU memory from its parameters: 4 bytes a parameter with LoRA and 16 in full training, plus 6 GiB. For kev that is about 7.5 GiB with LoRA and 11.9 GiB in full. Training GPU marks each GPU by it, and the launch refuses a smaller one.

The estimate assumes decisions of up to 16 candidates of 1,024 tokens, and is only an estimate. A run with a long token limit, or with many candidates, can pass the launch check and then stop on the GPU with "…does not fit in this GPU's memory…" (see When a run stops), once its GPU has started and is billed. Start at the default token limit, and choose a larger GPU before you raise it.

Follow a run

Select the run to open it. The page refreshes by itself. Its status says what the run does: Checking decisions, Scoring base model, Training · step N / M (Training · N% in the list), Calibrating, Scoring tuned model and Exporting model. Its steps are Launched, Preparing, Checking, Scoring base model, Training, Calibrating, Scoring tuned model, Exporting and Model ready.

Besides what every run shows (see Follow a run), a decision run shows:

• Loss: the training loss and the validation loss by step. While the run goes on, a sentence below says what it does, such as "Fitting the temperature that calibrates the probabilities, on the validation split."
• Run: Token limit, Candidates, Learning rate, Epochs and, once known, Temperature.
• Scores on the test split of the version, once the run has scored the tuned model: Accuracy, Log loss, Brier score and Calibration error (ECE) of the tuned model, each with the base model's score and the change below it, green when it improved and red when it got worse. A lower log loss, Brier score and calibration error are better. Compare evaluations opens the two evaluations side by side. A sentence gives the temperature, such as "The tuned model's probabilities use a temperature of 1.27, fitted on the validation split."
• Training logs, which hold numbers only, never your cases. The first line reads "Decisions checked in S s: N train, N validation, N test; P pairs, the longest L tokens (limit T, model M); at most C candidates (limit K).", and the next ones give the base model's scores, the steps trained with the kept model's validation loss, the temperature, the tuned model's scores and the peak GPU memory.
• In the ⋯ menu, Download job files: the run's cases, as the GPU reads them, and its configuration. Unlike a chat run's, they hold the test cases too.

While the console is open, a notice says when the run finishes ("Training finished": "run: the model is ready."), fails ("Training failed", with the reason) or stops before its output was collected ("Training stopped": "run: its model waits for recovery."), with Open run.

The result

A finished decision run shows Model ready. The model is the folder kev loads, with the temperature its run fitted, and the run's two evaluations are in Evaluations. The panel says "The model is saved and checked (size), with the temperature that calibrates its probabilities." Use this model offers:

• Download model, with its size: a ZIP of the model. Every role can download it, and each download is recorded in the audit trail.
• Publish to Hugging Face: see Publish to Hugging Face.
• Open repository, once published.
• Evaluate model: score it on another version. See Evaluate a model.

The ZIP holds training_manifest.json (the run, its version and its settings) and a model folder:

The scores in the run are on the test split of the version the run trained on. To score the model on another version, evaluate it.

Load it with kev

The DecisionModel of kev loads the model folder. kev runs on a Linux machine with a CUDA GPU, and expects Python 3.12, PyTorch 2.11 and Transformers 5.16, as its GPU and container guide describes. Unzip the download into a kev checkout, here as models/routing (use your file's name in place of model.zip):

git clone https://github.com/skundu42/kev.git
cd kev
unzip ~/Downloads/model.zip -d ./models/routing

Then, in the kev folder:

from kev.inference import DecisionModel

model = DecisionModel.from_pretrained("./models/routing/model")
result = model.predict(
    state="The text the decision is about.",
    questions={
        "answer": {
            "type": "choice",
            "instructions": "Choose the option that fits the text.",
            "criteria": {"first": "The first option", "second": "The second option"},
        },
    },
)
print(result)

kev refuses a decision with more candidates than Most candidates, or a prompt and candidate longer than the Token limit, rather than cutting it short.

Publish to Hugging Face

Publish to Hugging Face, in Use this model, uploads the model to a repository of your organization's Hugging Face account, private by default, with your organization's own token with write access. It works as it does for an adapter: see Publishing adapters to Hugging Face. The repository starts as your token's account and the base model's name with -decision, such as acme/kev-decision. A notice says "Model published to Hugging Face" when it is done.

One commit adds model.safetensors, config.json, the tokenizer files, kev_config.json and, if the repository has none, a model card. The card names the base model and its revision, the method, the epochs, the learning rate, LoRA's rank and alpha for a LoRA run, the token limit, the most candidates and the temperature, and shows how to load the model with kev. It names no dataset and carries no scores. The published kev_config.json leaves out the calibration's record, whose counts and losses describe your validation split; kev doesn't read it. Before uploading, Tensorant checks again that the model in storage is the one the run verified.

Evaluate a model

An evaluate run scores one model on a version's test split, on a GPU, and trains nothing: the project's base model, or the model of a finished training run of the project. Use it to score the base model before you train, or a trained model on a newer version.

1. In Evaluations, choose Evaluate model. Evaluate model is also on Home, and in Use this model on a finished run, which chooses that run's model.
2. Run is Evaluate model. Under Model, choose Base model · model, or a finished run, listed as run · trained on vN.
3. Choose the Dataset version: its test split is scored.
4. Choose the provider, GPU, time limit and price limit. A GPU is checked against the memory LoRA training would need: "Memory is estimated as for training with LoRA; scoring needs less."
5. Check Token limit and Most candidates under Decision settings.
6. Choose Save evaluation draft, then launch it. It counts toward your plan's training runs, and needs the Pro plan.

Its status reads Checking decisions, Scoring model and Saving scores, and its steps are Launched, Preparing, Checking, Scoring and Evaluated. When it completes, it shows the model's scores and temperature, and a notice says "Evaluation finished": "run: the scores are ready.", with Open evaluation. It is named "Evaluation N on vX", where a training run is "Model N on vX".

A run's model is scored with the revision of the base model that run trained on, and must be in the storage your organization uses now. The evaluation of a run's model is compared with that run's own evaluation of the base model when that scored the same version.

Decision evaluations

Evaluations lists a decision project's evaluations as Base model or Tuned model, with the model, the version and the main score, Accuracy. All, Base model and Tuned models filter the list. An evaluation is named "Baseline N on vX" when it scored the base model and "Tuned N on vX" when it scored a trained one.

Probabilities switches between Calibrated, at the evaluation's temperature, and Uncalibrated, at temperature 1 as the model gives them. Log loss, Brier score, calibration error and rubric error are recorded both ways, the second as metric (uncalibrated). Accuracy is the same either way. The bars always show the calibrated probabilities.

An evaluation's results show the metrics, the scores broken down by Category, Question or Question type under Break down by, and the test decisions, 25 a page. Choose a group, or one under Filter group, to filter the decisions. Open a decision to see each candidate's probability beside its gold probability, the scores, and its case.

Compare with pairs two decision evaluations of the same version, decision by decision, as comparisons of answers do. A decision is Newly right or Newly wrong on accuracy, and Improved or Regressed on another metric, by the metric's direction: a lower log loss is an improvement. Decisions compared counts the pairs, and Independent cases the test cases they come from. The likely range draws the test cases again 2,000 times, each case with all its decisions, and needs at least 10 test cases. A decision evaluation can't be compared with an evaluation of a chat model's answers.

Plan

Decision models are on the Pro plan. Creating a decision project, and launching a decision run or an evaluate run, are refused on other plans with "Decision models are on the Pro plan. An owner can change the plan under Settings → Plan and billing." The notice offers Open Plan and billing. A draft run saves on any plan, and a run that is already going is left alone if the plan changes. Downloading and publishing a model you trained needs no particular plan. See Decision models in Plans and billing.

Tips

• Start at the default token limit. The memory estimate assumes 1,024 tokens. Before you raise the limit, choose a GPU with more memory. Readiness for training shows your longest pair.
• Keep the candidates few and distinct. Each candidate is one more pair to score in every step, and candidates that read alike are hard to tell apart.
• Give the validation split at least 50 decisions. With fewer, the temperature can't be fitted and stays the base model's.
• Keep the gold you have. Probabilities from several annotators teach the model where people disagree.
• Score the base model first. An evaluate run of the base model shows how far it is from your gold before you train. A training run scores it anyway.
• Read the calibration error too. A model can gain accuracy while its probabilities get less trustworthy.

Limits

When something goes wrong

When you create the project

When you import

When you create a version

In Readiness for training

In Review

When you save or launch a run

When a run stops

A run that fails shows the reason as its error: the run's own, from its GPU machine, or Tensorant's, when it checks what the machine saved. None names your cases:

In the training logs

These are written to Training logs (Run logs for an evaluate run) when an evaluate run can't copy the model of the run it scores. The run then shows "Kind: training stopped; inspect logs".

When a run needs recovery

A decision run shows Needs recovery when its time limit passed, its GPU machine stopped without a result, or Tensorant could not collect what the machine saved, for example when storage failed while the model was read, or when a training run's evaluation report failed Tensorant's checks. The machine is stopped, where its provider can stop one, and what it saved stays in your storage. Lambda can't stop a machine, so it bills until you recover the run or cancel it.

On the run's page, under Recover the model (Recover the evaluation for an evaluate run), choose Recover from storage. Tensorant collects the run again from what it left in your storage: its result, its evaluation and, for a training run, its model, each checked as the first time. A training run then shows Model ready, with its two evaluations, and an evaluate run completes with its evaluation. This works on every storage, your own S3 bucket and a RunPod network volume alike, and an evaluate run is recovered the same way. Copying a large model can take a few minutes. With the API, send POST /api/training-runs/{id}/recover-volume with no body. POST /api/training-runs/{id}/recover, which takes a model's checksum and size, recovers a training run too, but refuses an evaluate run: "An evaluate run made no model, so it has no checksum or size to send: recover it with POST /api/training-runs/id/recover-volume, which collects it from its own storage."

A decision run can't be recovered from a file you upload: an upload is refused with "Decision runs can't be recovered from an uploaded file yet. Recover the run from its own storage, or retry it." When the storage holds no result from the GPU machine, as when the time limit came during training, there is nothing to recover, and Recover from storage says "The run's storage holds no result from its GPU machine, so there is nothing to recover. Retry the run, or contact support if it happens again.": choose Cancel run, then Retry. The form applies a longer Time limit (hours). A decision run can't continue from a checkpoint, so the new run starts from the beginning. A recovery whose evaluation report still fails the checks says "The run's evaluation report could not be recorded: reason. Retry the run, or contact support if it happens again.", and the run keeps waiting with its model in your storage: contact support with the run's ID before you cancel it, or cancel it and retry. See Needs recovery.

When you download or publish

In evaluations