Skip to content

Scoring and ranking

Score and rank candidates with a calibrated decision model

Fine-tune a ModernBERT or Ettin encoder such as kev to score each candidate answer with a calibrated probability. For routing, triage and moderation.

The problem

Why not let a chat model pick?

A chat model answers with text it writes. Asked to pick a route, it can name one that is not on your list, and it gives your options no probabilities that your code can compare with a threshold.

For routing, triage, moderation and scoring, you already know the options. You need a probability you can act on, such as sending a case to a person when the top answer is below 70%. See When to use a decision model.

How it works

What a decision model does

  • A decision model reads a text, such as a support ticket, with a typed question and its candidate answers. It scores every candidate and returns a probability for each, calibrated with a temperature fitted on your data.
  • It never answers outside its candidates. The candidates can differ from one question to the next, because the model reads them with the question. See When to use a decision model.
  • A run trains on one GPU, with LoRA or full training. It scores the base model and the tuned model on your test cases, and fits the temperature on the validation split. See How a run works.

Models

Models to start with

Tensorant trains decision models on ModernBERT and Ettin encoders of up to 1.2 billion parameters. See Supported models.

  • skundu42/kev

    A ModernBERT scorer of about 400 million parameters, fine-tuned from the Ettin encoder below. Training continues from its one-score head and its own temperature.

  • jhu-clsp/ettin-encoder-400m

    An Ettin encoder. Training adds a new head that gives each candidate one score.

  • answerdotai/ModernBERT-base

    ModernBERT's base model. Training adds a new one-score head, as it does for the Ettin encoder.

See every supported model

Steps

How to do it in Tensorant

  1. Create a decision project

    Enter a decision model as the Hugging Face base model when you create the project. A project's kind comes from its base model and never changes.

  2. Import decisions or labelled texts

    Import typed decisions, each a state with its questions and the gold answer to each, or labelled texts, such as a CSV with text and label columns. See Decision projects.

  3. Freeze a version

    Approve the cases and create a version. It keeps each case whole, and cases that share a state in one split, so no test case shares a state with a training case.

  4. Train and evaluate

    Configure a decision run. It scores the base model, trains, fits the temperature and scores the tuned model, with accuracy, log loss, Brier score and calibration error for each.

  5. Serve it or download it

    Publish the model under a name and call it on POST /v1/decisions. See Serve decision models. Or download it as a ZIP that kev loads, or publish it to your Hugging Face account. See The result.

Plans

What you need

  • Decision models are on every organization: creating the project, launching its runs and serving its models. A model you trained downloads, or publishes to Hugging Face. See Images, RL and decision models in Pricing and billing.
  • Each training run, and each run that only scores a model, uses one GPU. A training run pays the training-token fee for the candidate pairs it trains on; a run that only scores a model pays none. See Before you start.

Guides

Related guides

Browse all guides

Use cases

Other use cases

See every use case

Start with one dataset.

Create an account and import your data, or book a demo and we will show you the product.