Reinforcement learning
Fine-tune LLMs with reinforcement learning on tasks you can check
Train an LLM with GRPO against a reward that checks its answers: a label, a number, JSON that fits a schema or a Python function you write.
The problem
What if you can check an answer but not write the ideal one?
SFT learns from answers you wrote or approved, so each example needs the whole answer you want. For many tasks you can check an answer more easily than you can write the ideal one: a final number, a label from a list, JSON that fits a schema.
RL learns from a score for each answer the model writes itself. The reference can be just the part the reward checks, such as a final number, and not a full written solution. See When to use RL.
How it works
How an RL run learns
- In each round, the model writes several answers to each training prompt, a reward scores every answer from 0 to 1, and the adapter moves toward the answers that scored higher than the others to the same prompt. See RL training. The reinforcement learning page follows one prompt through a round.
- RL works best when the model already gets some answers right and some wrong. If it almost never gets one right, train with SFT first, then start RL from that adapter.
- The reward is a check that runs on the GPU machine, so the LLM judge can't be a reward. RL is not for answers that only a person or another model can judge. See When to use RL.
In practice
Rewards you can use
- For Q&A answers, the reward can be Token F1, normalized exact match or JSON validity. Token F1 gives a partly right answer partial credit, so the model can tell a better answer from a worse one.
- For classification, it can be classification accuracy against your allowed labels, normalized exact match or JSON validity.
- For structured JSON, it can be JSON-schema correctness, JSON field accuracy on the fields you list, normalized exact match or JSON validity.
- For numeric answers, it can be numeric accuracy within the tolerances you set, normalized exact match or JSON validity.
- You can also write the reward in Python, as a
score(answer, reference)function that returns a number from 0 to 1. It runs in a sandbox on the GPU machine. See Custom metrics. - A full run starts with its baseline's scoring and main metric as the reward, so it trains toward what you measure. Change any field to train toward something else. See Choose the reward.
Models
Models to start with
RL trains text models with LoRA on one GPU. RL on mixture-of-experts models is not yet proven. See Supported models.
Qwen/Qwen3-4B-Instruct-2507
A 4B instruct model that answers directly, without a reasoning block, so the reward scores the answer alone.
unsloth/Llama-3.2-1B-Instruct
An ungated copy of Llama 3.2 1B Instruct, the smaller of these two.
Steps
How to do it in Tensorant
Freeze a version
Create a version of text examples. RL reads it as it is: each example's last message is the reference the reward checks, and the conversation before it is the prompt. See Prepare your data.
Score a baseline
Run a baseline evaluation of the version. A full RL run needs one, and its scoring and main metric become the reward unless you choose another.
Run an RL trial
Choose RL as the training objective and run a trial first. It runs one round on up to 8 prompts, and reports the peak GPU memory, the seconds each part took and an estimate of the full run's time and GPU cost.
Run it and compare
Launch the full run. It keeps the adapter of the round with the best validation score. Then choose Score against baseline to compare the adapter with the base model on the test examples. See The result.
Publish it
The result is a regular LoRA adapter. Give it a name in API models and call it as you would any other adapter.
Plans
What you need
- RL training is on every organization: launching an RL run or trial, and creating or starting an experiment that trains with RL. Each run pays the training-token fee for the prompts and answers it trains on. See Plan and fee in RL training.
- GPUs come from Tensorant compute, charged per second from prepaid credits, which an owner adds under Settings, Plan and billing (the first payment turns Tensorant compute on), or from a GPU account you connect with the Own compute subscription. Each round samples several answers to every prompt, so RL uses more GPU time per prompt than SFT. See Before you start.
Private and sensitive data
Medical, legal and financial records: what Tensorant stores, where it runs, and what can stay in your cloud.
Classification and routing
Answer with one label from your list: intents, ticket routes, what needs a person.
JSON extraction
Pull the fields your code needs out of invoices, forms and emails, as JSON that fits your schema.
Replace a costly LLM API
Train an open model on the answers your API model gives, and switch when it scores as well.
Small language models
Fine-tune a 1B to 4B model for one narrow job, and download the adapter.
Scoring and ranking
Score each candidate answer with a calibrated probability, with a decision model such as kev.
Migrate from OpenAI fine-tuning
OpenAI ends fine-tuning in January 2027. Move your data, train an open model and compare the two.
Start with one dataset.
Create an account and import your data, or book a demo and we will show you the product.