Serve a decision model and call it from your app

A decision model is only useful once your application can ask it. This guide takes the model trained in Train a calibrated decision model with kev and serves it from GPUs in your own RunPod account. You will try it in the Playground, publish it under a name, call POST /v1/decisions from Python with batches and retries, and turn its probabilities into actions: proceed, ask a person, or stop.

Every number here was measured in production on 24 October 2026, serving that model ("Model 1 on v1", kev tuned on LocalLLaMA/typed-decisions) from RunPod's EU-RO-1 datacenter.

How serving works

Tensorant serves decision models with a small decision server on the same image that trained them. The server encodes each question and candidate exactly as training did, scores the pairs in bfloat16 over full-precision weights, and applies the temperature the run fitted. Several decision models of your organization can share one GPU.

There are two ways to run a model:

You can create an API model without a test session first. A test session is the quick way to look at the model before anyone depends on it.

What you need: the Pro plan, RunPod compute and a network volume under Connections, a finished decision run showing Model ready, and the owner role to publish. Members can start test sessions and use the Playground.

1. Try it in a test session

1. In the decision project, open Models under Use and choose Test a model. In a decision project it opens Test a decision model.
2. Under Model to serve, choose your run, here "Model 1 on v1". You can also serve the project's base model, or another Hugging Face scorer such as kev.
3. Choose a GPU. The form says the model needs about 4 GB, so any 16 GB card will do: we used an RTX A4500 at $0.25/h. Keep 1 hour and 15 idle minutes.
4. Review the cost and start it.

The session took 8.4 minutes to reach Ready. Most of that is RunPod pulling the training image onto the machine and loading the 1.6 GB model; a machine that has run the image before starts faster. Details shows what the server uses: "Token limit 1,024 tokens" and "Temperature 1.016, fitted by its run".

If the start fails at once with "RunPod has no GPU of this type free in your volume's datacenter right now", RunPod's list was ahead of its stock. Choose another GPU and start again: this happened to us twice in one morning with the cheapest cards.

2. Ask it in the Playground

Choose Open Playground on the session. A decision project's Playground has two forms:

• State and questions: the state, then each question's identifier, type, instructions and candidates. Load a case fills the form from one of the project's cases, without its gold.
• Texts and labels: one text a line, and the labels, for a quick classification.

Load a case and choose Ask. Our first case answered in 1.5 s: "5 decisions · 18 pairs · 3,374 tokens". Each question shows its candidates as bars of their probabilities, the top pick and the confidence:

action        Top pick human_review   15.7% confidence
  continue       19.9%
  human_review   36.7%
  observe        36.7%
  stop            6.7%
needs_review  Top pick No             37.2% yes
outcome       Top pick partial        22.0% confidence
risk          Top pick Level 0.84 expected

This is the point of a decision model: on this trace it is torn between human_review and observe, and says so. An application can see that and decide not to act alone.

3. Publish it

An owner publishes from the session's ⋯ menu (Create API model from this) or from Create API model on Models. Give it a name, which is what your application sends as model, and choose its availability:

We published the same model twice, once on demand and once always on (an RTX PRO 4000 at $0.57/h, for 1 day). This guide calls them triage-wake and triage. The always-on model was ready 9 minutes after it was created. Both appear under API models with a Decision tag, and GET /v1/models lists them with "type": "decision".

The cold start of an on-demand decision model is longer than you might expect, because each new worker pulls a large GPU image. If anything waits on the answer, use always on, or warm the model before the traffic comes. See Serverless vs dedicated inference for the tradeoff in general.

4. Create an API key

On API keys, choose Create key. Name it after the application, limit it to the models it calls, and set its requests a minute and at once. The key is shown once: store it as a secret, here TENSORANT_API_KEY.

A decision request counts one request for each input or text it holds, where an input is one state with all its questions: a batch of 32 inputs is 32 requests toward the plan's requests a month, and the plan's and the key's requests a minute. Our check below sent the 400 test cases twice, in batches, and the plan's counter went from 8 to 818.

5. Ask from your application

One decision is a state and its questions, in the same shape as the training data, without the gold:

curl https://api.tensorant.io/v1/decisions \
  -H "Authorization: Bearer $TENSORANT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "triage",
    "state": {"agent": {"autonomy": "checkpointed"}, "task": "Rotate the expired TLS certificate on staging.",
              "trace_summary": {"steps": 11, "tool_errors": 0, "irreversible_actions": 0}},
    "questions": {
      "action": {"type": "choice", "instructions": "What should the observability system do with this trace?",
                 "criteria": {"continue": "Let the agent proceed without interruption.",
                              "human_review": "Queue this trace for a human to review.",
                              "observe": "Keep running, but flag the trace for later sampling.",
                              "stop": "Halt the agent now."}}
    }
  }'

The answer has one entry for each question, under answers, with the probability of every candidate, and usage. For example:

{
  "answers": {
    "action": {"type": "choice", "choice": "continue",
               "probabilities": {"continue": 0.81, "human_review": 0.11, "observe": 0.06, "stop": 0.02},
               "confidence": 0.75}
  },
  "usage": {"decisions": 1, "pairs": 4, "tokens": 412}
}

A noul (yes or no) question answers with noul, the probability of yes, and a score question with score, the expected level. Write the questions exactly as you wrote them in training: the same instructions and the same candidate descriptions. The model was trained on that wording, and the state is read as JSON text with sorted keys, as it was in training.

6. Send batches, and retry the right errors

For volume, send up to 32 decisions in inputs. The answer's results keep your order, and a decision that can't be scored comes back in its place as an error while the others are answered:

import os
import time
import httpx

API = "https://api.tensorant.io/v1/decisions"
KEY = os.environ["TENSORANT_API_KEY"]
RETRY = {502, 503}  # the model did not answer, is starting, or is busy


def decide_batch(model, cases, attempts=8):
    """Up to 32 cases ({"state", "questions"}), answered in order. An item that can't be scored is
    {"error": {"code", "message"}}, and the rest are answered."""
    body = {"model": model, "inputs": cases}
    with httpx.Client(timeout=httpx.Timeout(10, read=180)) as client:
        for attempt in range(attempts):
            response = client.post(API, json=body, headers={"Authorization": f"Bearer {KEY}"})
            if response.status_code == 200:
                return response.json()["results"]
            if response.status_code not in RETRY and response.status_code != 429:
                response.raise_for_status()  # 400, 401, 404, 409, 413: fix the request or the key
            # model_starting says how long to wait; otherwise back off
            wait = float(response.headers.get("retry-after", min(2 ** attempt, 30)))
            time.sleep(wait)
        raise TimeoutError("The model did not answer after retries")

What to retry, and what not to:

How fast batches are: on one RTX PRO 4000, a batch of 32 of these cases (160 questions, 568 question and candidate pairs, about 174,000 tokens) took 5.1 to 5.3 seconds, about 30 questions answered a second. Shorter states and fewer candidates go faster: the time follows the number of question and candidate pairs, and their length.

For plain classification, skip the questions and send texts with labels:

{"model": "ticket-router", "texts": ["Where is my parcel?", "Charge me again?"], "labels": ["billing", "shipping"]}

A model trained on labelled texts knows its own labels, and you can leave labels out.

7. Turn probabilities into actions

The reason to use a decision model is the probabilities. Decide what each range means for your product, and write it down as code:

def route(answer, act_above=0.80, stop_above=0.50):
    """What to do with one trace, from its "action" question's probabilities."""
    p = answer["answers"]["action"]["probabilities"]
    if p["stop"] >= stop_above:
        return "stop"                  # a likely stop is acted on early
    top, value = max(p.items(), key=lambda item: item[1])
    if value >= act_above:
        return top                     # sure enough to act alone
    return "human_review"              # unsure: a person decides

Calibration is what makes these thresholds honest. The model in this guide had a calibration error of 2.6% on its test split: when it puts 80% on a candidate, people's gold puts close to that much on it too. Set the thresholds on your own test results, not on a guess: open the tuned evaluation in Evaluations, break it down by question, and look at how often decisions above each threshold were right. Read the top probability, not confidence: confidence measures how far the top candidate is above an even split, which suits a quick display but not a threshold.

8. Check that serving matches training

A model that scores differently in production than in its evaluation would make every threshold above wrong. We checked: we sent all 400 test cases (2,000 decisions) of the run's version to triage in batches of 32, and compared each probability with what the run recorded for the same decision.

The small differences come from bfloat16 arithmetic: the server groups pairs into batches differently from the training run, and kev's own benchmarks show such regrouping moves probabilities by up to 0.03 to 0.06. They change the top answer only where two candidates were already within 0.05 of each other, which your thresholds should treat as unsure anyway.

You can run the same check on your own model: export the tuned evaluation's results, send the test cases' states and questions through your API model, and compare.

9. Run it day to day

• Watch usage and spend. API keys shows each key's requests and tokens a month; Models shows each model's GPU time and estimated spend, settled with RunPod's billing later.
• Stop what you don't use. Stop or delete an API model from its ⋯ menu; a test session stops by itself when idle. Always-on models end on their date unless you extend them.
• Replace the model in place. Edit settings on an API model changes what it serves or its GPU while its name stays the same. An always-on model keeps answering until its replacement is ready.
• A model that can't load says why. Models shows Cannot start with the cause, such as a missing model file or a GPU too small, and the API answers 503 model_unavailable until it is fixed.
• When you train a better model, gate it with the same experiment as in the training guide, then point the API model at the new run. Your application's code and thresholds don't change.

See Serve decision models for every field, limit and error.