Serve decision models

A trained decision model answers typed questions with a probability for each candidate. You serve one the way you serve a chat model, with the same pages: a test session on one GPU while you try it, and an API model that your applications call by name. The difference is the route. A decision model answers POST /v1/decisions, and a chat model answers POST /v1/chat/completions.

Serving decision models needs the Pro plan. The server scores with the model's own temperature and the training program's own encoding, holding the weights in full precision and scoring in bfloat16. Its probabilities agree closely with the ones your run measured, but not exactly: scoring in bfloat16 moves them slightly with how pairs are batched.

Before you start

• A decision model to serve. A finished training run of a decision project, with Model ready and GPU released, or another decision model from Hugging Face (a ModernBERT or Ettin scorer, such as skundu42/kev). An evaluate run scores a model and makes none: serve the run it scored.
• RunPod compute and a network volume, connected in Connections. The model server's code and the model are delivered from your volume.
• The Pro plan. See Plan.
• The owner role to publish. Members can start test sessions and use the Playground. See roles.
• Room in your plan: a free deployment slot for a test session or an always-on model, and room for the API model under API models. See What a plan limits.

Test a decision model

A test session is a decision model on one GPU of your RunPod account, for you and your team to try in the Playground.

1. Open Models under Use in the sidebar and choose Test a model; in a decision project it opens Test a decision model.
2. Choose what to serve: a model trained in this project (a run whose GPU is still being released is listed and can't be chosen yet), the project's base model, or another Hugging Face decision model with its revision.
3. Choose the GPU, the Maximum hours, the Idle minutes and the price limit. A decision model runs on one GPU, and the list marks the GPUs without enough memory to serve it.
4. Check the cost and start it. A paid start asks you to confirm, as every deployment does.

A test session has no context length and none of the advanced serving settings of a chat deployment. The model's own token limit is the longest pair it reads: the Token limit its run trained with, up to 7,999 tokens. Details shows it, with the temperature the server uses.

A model from Hugging Face must be a scorer with one output, such as kev. A model whose head is another shape is refused before a GPU starts. The same stops, slots and limits as for deployments apply, and the session stops when it is idle or reaches its maximum hours.

Publish a decision model

Choose Create API model on Models, or Publish in a test session's menu, and fill in the same form as for a chat model: the source, a name, and On demand or Always on. The name is what applications send as model. See API models for names, availability, ending and deleting.

• On demand serves the model from a Serverless endpoint that wakes when a request arrives. The API holds the first request after a rest for up to a minute while a worker starts. If the model is still starting, it answers 503 model_starting with Retry-After: 15, and you send the request again. How long a cold start takes has not been measured yet.
• Always on keeps the model on a GPU in a deployment slot.

A published decision model shares a GPU with your other decision models of the same kind, GPU and availability, and loads each model as it is asked for. It has no context length, advanced serving settings or GPU count. Published decision models carry a Decision tag, and count towards the plan's limit of API models.

Try it in the Playground

A decision project's Playground asks a running test session or a published decision model. Choose it, then fill in one of two forms:

• State and questions. Enter the state, then each question's identifier, type, instructions and candidates: a line each as key: description for a choice, the two texts for a yes or no, or the levels of a score. Load a case fills the form from one of the project's cases. It never loads the gold.
• Texts and labels. Enter one text a line, and the labels separated by commas, or none to use the model's own labels. Optionally add instructions.

Each question's answer shows its candidates as bars of their probabilities, the top pick, and the confidence. The panel header shows the time taken and the usage, such as "5 decisions · 13 pairs · 156 tokens". Get code shows the request for a published model.

A question to a published model counts against the plan's API limits, as a question from your application does. A question to a test session costs only the session's GPU time. Nothing you enter is saved.

The API

The route answers decision models of your organization, by the name of the published model. For keys, limits per key and the plan's limits, see Keys and usage. The base URL, the Authorization header and the error format are those of chat completions.

Send a decision model's request to /v1/decisions and a chat model's to /v1/chat/completions. The wrong one gets 400 wrong_route, naming the right one.

One decision

curl https://api.tensorant.io/v1/decisions \
  -H "Authorization: Bearer $TENSORANT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ticket-router",
    "state": "My parcel arrived damaged and I want my money back.",
    "questions": {
      "route": {
        "type": "choice",
        "instructions": "Which team should handle this ticket?",
        "criteria": {"billing": "Payments and refunds", "shipping": "Delivery problems", "technical": null}
      },
      "refund": {
        "type": "noul",
        "instructions": "The customer is owed a refund.",
        "criteria": {"false": "No refund is owed.", "true": "A refund is owed."}
      },
      "urgency": {
        "type": "score",
        "instructions": "How urgent is this ticket?",
        "criteria": ["Can wait", "Normal", "Soon", "Now"]
      }
    }
  }'

The state is text, or any JSON value, which the model reads as JSON text with sorted keys. Each question has a type, its instructions and its criteria, as in typed decisions without the gold:

The answer has one entry for each question, under answers:

{
  "answers": {
    "route": {
      "type": "choice",
      "choice": "billing",
      "probabilities": {"billing": 0.71, "shipping": 0.27, "technical": 0.02},
      "confidence": 0.565
    },
    "refund": {"type": "noul", "noul": 0.88, "probabilities": {"false": 0.12, "true": 0.88}},
    "urgency": {
      "type": "score",
      "score": 1.75,
      "probabilities": {"0": 0.05, "1": 0.3, "2": 0.5, "3": 0.15},
      "confidence": 0.33
    }
  },
  "usage": {"decisions": 3, "pairs": 9, "tokens": 412}
}

• probabilities are keyed by candidate name, calibrated with the model's temperature, and sum to 1. The names of a choice are the keys of criteria; the keys of a score are the levels' positions from 0.
• confidence is how far the top probability is above an even split, from 0 to 1. A noul question has none: its noul is the probability of yes.
• usage counts the questions scored (decisions), the question and candidate pairs the model read (pairs), and their tokens.

The answers don't depend on the order of the candidates in your request: the model scores them in its own order, and the answer is keyed by name.

A batch

Send up to 32 decisions in inputs, each with its own state and questions:

{"model": "ticket-router", "inputs": [{"state": "...", "questions": {}}, {"state": "...", "questions": {}}]}

The answer is {"results": [...]}, one entry for each input in the order you sent, each as the answer of one decision above. An input that can't be scored comes back in its place as {"error": {"code": "too_long", "message": "..."}} while the others are answered. The codes are too_long (a pair longer than the model's token limit, never cut short), too_many_candidates, invalid and not_finite. The messages name positions and question identifiers, never your text.

Texts and labels

For classifying texts, send text, or texts (up to 32), with labels:

{"model": "ticket-router", "texts": ["Where is my parcel?", "Charge me again?"], "labels": ["billing", "shipping"]}

Each text becomes one choice question over the labels, 2 to 16 of them, each up to 200 characters. Optional instructions (default "Choose the label that fits the text.") replace the question. Without labels, the model's own labels are used, when its training data had them: a version whose every case is one choice question over the same candidates, as a text-and-label import makes. A model without a label set of its own, and a Hugging Face model, get 400 labels_required: "This model has no label set of its own. Send labels with the texts."

The answer for text is one item; for texts it is {"results": [...]}. Each item is:

{"label": "shipping", "probabilities": {"billing": 0.04, "shipping": 0.96}, "confidence": 0.92, "usage": {"decisions": 1, "pairs": 2, "tokens": 31}}

Fields

Send one shape. Any other field gets 400 unsupported_parameter, and a missing or malformed field gets 400 invalid_request with a message that names the input by position and the question by identifier. A bad input refuses the whole request, before it counts toward your monthly requests.

What a request counts

Each decision in inputs, and each text, is one request against your plan's requests a minute and requests a month, and against the key's own Requests a minute. A batch of 32 is 32 requests. A batch with more inputs than the key's Requests a minute, or than your plan's requests a minute, is refused with 429 rate_limit_exceeded and "Send fewer at once", without a Retry-After, because waiting would not help. A request in flight is one request at once, however many inputs it has.

Tokens a month are reserved before the model answers, at each pair's token limit, and settled with the tokens the model read. A decision that can't be scored counts as a request and an error. The plan's usage and your keys' usage show these as for chat.

If the plan has fewer requests left than the request has inputs, the whole request is refused with 429 quota_exceeded: "This request's inputs do not fit the API requests a month its plan (Plan) allows."

Limits

Errors

The errors have the format and the common codes of chat completions: 401 invalid_api_key, 404 model_not_found, 408 request_timeout, 429 rate_limit_exceeded and quota_exceeded, with a Retry-After where one helps. These are particular to decision models.

Plan

Test sessions, API models, the Playground and the API route of decision models are on the Pro plan, with the training they follow. They use no new limit: decisions count against API requests a month, Requests a minute, Requests at once and API tokens a month, and published decision models against the API models limit. See Plan limits.

An organization that leaves the plan keeps its models, but a request to a decision model is refused with 409 plan_upgrade_required and the Playground and Test a model are refused until an owner upgrades.

When something goes wrong