RL training (GRPO)

RL (reinforcement learning) trains an adapter on the model's own answers. In each round the model writes several answers to a set of training prompts, a reward scores every answer from 0 to 1, and the adapter is updated toward the answers that scored higher than the other answers to the same prompt.

Choose RL · reinforcement learning (GRPO) under Training objective, beside SFT and DPO. RL trains a LoRA adapter on one GPU, for text models. You can save an RL draft on any plan, but launching one needs the Pro plan: see Plan.

When to use RL

Use RL when an answer can be checked: a number, a label from a list, JSON that follows a schema, or anything a short Python function can score. The reference can be just the part the reward checks, such as a final number, and not a full written solution.

RL learns from the differences between answers, so it works best when the model already gets some answers right and some wrong. If it almost never gets one right, train with SFT first, then start RL from that adapter. RL is not for answers that only a person or another model can judge: the LLM judge and RadEval can't be a reward.

Before you start

• A version with training and validation examples, in text. See Prepare your data.
• For a full run, a completed baseline evaluation of that version. The reward comes from it unless you choose another. A trial needs none, but then you choose the reward yourself.
• The Pro plan, to launch an RL run or trial. See Plan.
• A text model. A vision model, or a version with images, is refused.
• A GPU account and storage, connected by an owner. See Training runs.
• A training run left in your plan this month. An RL run counts like any other full run. A trial counts only on connections Tensorant provides.
• The member or owner role.

How a run works

A run is a number of rounds. Before the first round, it scores the adapter it starts from on a fixed sample of up to 128 of your validation prompts. That is round 0. Then, in each round, it:

1. Takes the next Prompts per round training prompts, none twice in a round. When the prompts run out, it goes through them again in a new order.
2. Has the model write Answers per prompt answers to each prompt. The answers are sampled with randomness, so they differ from each other.
3. Scores every answer with the reward.
4. Trains the adapter in one pass over the answers to the prompts whose answers did not all score the same. It moves toward the answers that scored higher than the other answers to the same prompt. A prompt whose answers all scored alike has nothing to teach, so it is skipped.
5. Scores the same validation sample again, with the model's most likely answer to each prompt.

The run keeps the adapter of the round with the best validation score. See How a run ends.

For example, take Numeric accuracy as the reward and 80 as the reference answer to a question about a train's speed. Four of the model's eight answers might score like this:

The update moves the model toward the bare number. If all eight answers had scored 0, or all 1, there would be nothing to compare and the prompt would be skipped.

Prepare your data

RL reads the versions you already have. There is no new format.

• Prompt. An example's conversation without its last message: an optional system message, then user and assistant turns, ending with a user turn.
• Reference. The last message, an assistant reply. The reward checks every sampled answer against it. For a DPO preference pair, the prompt is the shared prompt and the reference is the chosen answer, as in evaluations. The rejected answer is not used.
• Which examples. Training examples supply the prompts of the rounds. A fixed sample of up to 128 validation examples measures progress. Test examples are never used: they stay held out for evaluations.
• Text only. RL trains on text. A vision model is refused, and so is a version with images on a text model.

This training example has the question as its prompt and 80 as its reference:

{"messages":[{"role":"user","content":"A train travels 120 km in 1.5 hours. What is its speed in km/h?"},{"role":"assistant","content":"80"}]}

Saving a draft checks every training and validation reference against the reward, as an evaluation checks its test references: a classification reference must be one of the labels, a numeric reference a plain number, and so on. See Reference answers are checked first. A reference that doesn't suit the reward is refused with "Incompatible reference answers:" and up to 20 examples.

Each prompt and the longest sampled answer must fit in the Training window (tokens). Launch readiness checks a sample of your examples for free, and before it loads the model the run checks every row, naming the first that doesn't fit. See When something goes wrong.

Choose the reward

The reward is one metric that scores each sampled answer from 0 to 1. You choose it with the same fields as an evaluation: what Answers are, and a Reward metric of that kind of answer.

Your own custom metrics are listed after these in Reward metric. Allowed labels, JSON schema, Fields to compare (optional), Absolute tolerance and Relative tolerance appear for the kinds of answer that need them, and work as in an evaluation. The LLM judge and RadEval are never offered.

The default is the baseline's. For a run with a baseline, the reward starts as that baseline's scoring and main metric, and Reward says "Taken from the baseline's scoring and main metric. Change a field to train toward another reward." Change any field to train toward something else. If the baseline's main metric is the LLM judge or an additional evaluator such as RadEval, there is no default: "The baseline's main metric, metric, can't be an RL reward. Choose a reward metric." A trial that has no baseline has none either, and the form asks for one: "Choose a reward for RL: the metric that scores each sampled answer."

A built-in reward gives an answer the score an evaluation would. Two things about how it scores matter:

• Partial credit helps. Token F1 scores a partly right answer between 0 and 1, so the model can tell a better answer from a worse one. Normalized exact match scores only 0 or 1: a model that is nearly always wrong gets nothing to learn from.
• An answer cut off scores 0. An answer that reaches Longest sampled answer (tokens) before it finishes scores 0, so the model learns to finish within the limit.

Custom metrics

Your organization's registered Python metrics are listed in Reward metric, after the built-in ones. Under Custom metrics, choose Manage metrics to add one; a metric you register there becomes the chosen reward metric. A metric defines score(answer, reference) and returns a number from 0 to 1, as for evaluations: see Upload your own metric. With none yet, Custom metrics says "No custom metrics yet. Upload a Python metric to use it as the reward."

A custom reward differs from a custom evaluation metric in these ways:

• It runs on your GPU machine. A sandbox there runs the metric in separate processes that hold none of your credentials, not your storage keys or your Hugging Face token. The machine downloads the sandbox's runtime when training starts, so it needs internet access for that: a network that only reaches Hugging Face is not enough.
• The run keeps the version you saved. Editing a metric makes a new version and leaves a saved run alone. A baseline's custom main metric keeps the version the baseline used. Choose an active version: an archived one is refused for a new run.
• A failing answer scores 0. If the metric raises an error on an answer, or returns something that is not a number from 0 to 1, that answer scores 0 and counts as a failure, where an evaluation fails the whole batch. A metric that runs out of time or computation, or prints too much, fails all the answers of its batch of up to 8. If more than a tenth of a round's graded answers fail, the run stops with the metric's error. Once the starting adapter is scored, that stop keeps the best adapter so far: see When a round fails. Make the metric handle any text, including an empty answer.
• It is tried before the model loads. The run scores one reference against itself first, so a metric that doesn't compile, or doesn't define score(answer, reference), stops the run before the model loads.
• Each reference can have at most 32,000 characters.

Configure an RL run

Saving a run never starts a GPU.

1. Open Runs under Tune and choose Configure a run.
2. Under Training objective, choose RL · reinforcement learning (GRPO). Training method becomes LoRA and Number of GPUs becomes 1. Both stay locked, and Learning rate changes to 0.000005.
3. Under Run, choose Full training or Training trial · a small sample. Choose the Dataset version, and for a full run a Baseline of that version.
4. Choose the Training provider, Training GPU, Time limit (hours) and Price limit ($/hour), as for any training run.
5. Under Reinforcement learning, choose where to Start from and the numbers of a round. See Settings.
6. Under Reward, check what scores each answer. See Choose the reward.
7. Under Advanced settings, check Learning rate and Training window (tokens).
8. Choose Save training draft, or Save trial draft. Launch it as you would any run: see Launch a run.

If the project's model is a vision model, the form says "RL trains text models. model is a vision model: choose SFT or DPO, or a text model." and the draft can't be saved. On a plan without RL, the option reads "RL · reinforcement learning (GRPO) (Pro)" and the form shows the plan message with Upgrade, or See plans if you are not an owner.

Settings

Under Reinforcement learning:

Under Advanced settings, these work differently from other objectives:

The time limit, price limit, disks and the model revision work as for other runs. A round samples several answers to every prompt before it updates, so give Time limit (hours) room: the default is 2 hours, and a trial's estimate helps you choose. The hint under it says that an RL run stops training about 3 minutes before the limit, to save the adapter: see How a run ends.

Every RL run also uses a constant learning rate and no LoRA dropout, and samples answers at temperature 1, whatever sampling defaults the model ships with. Nothing pulls the adapter back toward the model it started from, so raise the learning rate in small steps.

Start from an adapter

By default an RL run starts from the base model with a new adapter. Start from can instead continue the adapter of a finished run. A common recipe is SFT first, so the model answers in the right shape, then RL to improve how often the answers are right.

• The list holds the finished full runs of this project that have an adapter (Adapter ready or Completed) and trained the project's model. A trial is not listed: its adapter is only a check. If you typed a Model revision, the list holds only runs trained on it. With main, the run must have trained the same revision this run will pin, and saving refuses one that didn't.
• The new run continues that adapter, so its adapter size and scaling apply: Adapter size (LoRA rank) and Adapter scaling (LoRA alpha) take the source run's values and can't be changed.
• The run reads the adapter from your storage when it starts and checks it against the checksum and size saved with the source run. The adapter must be in the storage your organization uses now.
• The source run stays as it is. The new adapter is a new run, and the run shows Starts from the run you chose.
• The source can be an SFT, DPO or RL run.

Try an RL trial first

A trial runs one round on a few prompts, to check formatting, the reward, GPU memory and speed before you pay for a long run. It is not a quality evaluation. Set up the form for the full run you plan, set Run to Training trial · a small sample, and follow Try a small trial first. Use RL for the trial: an SFT trial does not predict RL memory or speed. A trial needs the Pro plan too.

An RL trial has these bounds, in place of Training rows and Optimizer steps:

The trial checks the format of every prompt it sampled, trains one round on at most 8 training prompts, and scores at most 8 validation prompts, never test examples. The launch confirmation says "A trial: one round on up to 8 prompts and N minutes, then the GPU is released.", where N is the trial's Minutes, or your Time limit (hours) in minutes when that is lower. Its Training trial report shows:

• Formatting, Seconds per round and Peak GPU memory.
• Sampling, Grading, Update and Validation: the seconds each part of the round took.
• After a successful trial, Full training time, noted in rounds, and Full GPU cost for the full run's settings, at the trial's GPU price. The estimate takes the seconds one round took, scales them from the trial's prompts to your Prompts per round, multiplies them by your Rounds, and adds the preparation time and the validation scoring, which happens before the first round and after each. Starting the GPU and saving the adapter take a little more.

Under the tiles it gives advice. It says "The estimate holds for the same GPU, model revision, answers per prompt, longest sampled answer, context length and adapter settings. A small sample can miss long rows that run out of memory." When memory runs short it says "The GPU ran out of memory. Choose a larger GPU, fewer answers per prompt or a shorter longest sampled answer." or "The trial used over 90% of this GPU's memory. Choose a larger GPU, fewer answers per prompt or a shorter longest sampled answer." When the full run would pass its limit, it says "Full training would take longer than its time limit. The full run's form raises the limit." A trial that did not finish, such as one that reached its Minutes, says "The trial did not finish. Fix its error and run it again before full training." A trial whose reward metric fails too often, or whose GPU runs out of memory, does not finish either: it fails, where a full run stops and keeps its adapter (see When a round fails), because its one round is what it measures.

Once the trial's GPU is released, Configure the full run opens the form with the trial's settings. The full run starts again from its own starting adapter or the base model, not from the trial's adapter.

Follow a run

Select the run to open it. The page refreshes by itself, and the status reads Training · round N / M while it trains. Besides what every run shows (see Follow a run), an RL run shows:

• Reward: a chart of two lines by round, from 0% to 100%. Reward is the average score of the answers sampled in that round. Validation is the average score of the validation sample's answers, with the adapter as the round left it; round 0 has this one only. Until the first round is scored, the chart says "Rewards appear here once the trainer scores its first round."
• How the run ended, under the chart, once it has, with the reason in a note when a round failed. See How a run ends.
• Run: Rounds, Prompts per round, Answers per prompt, Longest sampled answer, the Reward (the metric and the kind of answers, such as "Token F1 · Q&A") and Starts from.
• Time left (est.), with the share of rounds done.
• Rounds and Samples, described below.
• Training logs, with a line for each round, such as "Round 3 of 20: reward 0.4100, signal 62%, cut off 2%, 0 metric failures, updated, loss 0.0120, validation reward 0.4700 (best round 3); seconds: sampling 81.2, grading 2.1, update 64.0, validation 9.5." The line's signal is the table's Learned from. The line adds whether the adapter updated, and the loss, and leaves out the answer tokens. The logs hold numbers only, never your prompts, answers or references.

The Reward line moves a lot: each round samples other prompts, with randomness. Validation scores the same prompts each time with the model's most likely answers, so it is steadier. It is what the run keeps the best round by.

Rounds

Rounds is a table with the newest round first. Point at a heading to read what it means, and at Seconds in a row to see where the time went.

A Learned from that stays low means most prompts have answers that score alike: give the reward partial credit, or start from an SFT adapter. A high Cut off means the answers are longer than Longest sampled answer (tokens): raise it. Any Metric failures point at a custom metric that needs fixing.

Samples

Samples shows what the model answered. Choose a Round from the latest ten. Each round lists its first three prompts. Open one to read the Prompt, its Reference, and every Answer with its score. An answer is tagged Cut off when it reached the longest sampled answer, and Metric failed when the reward failed on it. A prompt shows its last user message. Text is shortened: prompts after 1,000 characters, references and answers after 600. If they can't be read, the page says "The samples could not be loaded:" and the reason, with Retry.

Read a few before you trust the numbers. A model can learn to score well in a way you did not mean. To read every answer of a round, see Samples in your storage.

How a run ends

A run ends when it has done all its rounds, or earlier. The line under the chart says which:

When the time runs out. Training stops about 3 minutes before the Time limit (hours), to leave time to save the adapter. Before each round, the run checks that the slowest round so far, plus a quarter, and two minutes to save the adapter, would still end by then, and it stops there when it wouldn't. A round that is still going when that time comes is stopped, and only that round's work is lost. The first round always starts, since nothing is known yet about how long a round takes, so the time can run out inside it.

If no round ever learned anything, the run fails instead of completing: "RL found no learning signal in rounds A-B: every answer to every prompt scored the same (value). Start from an SFT adapter, give partial credit in the reward, or check the reward."

When a round fails

Two failures end a full run the way the time limit does, once the run has scored its starting adapter and so has an adapter to keep: the reward metric failing on more than a tenth of a round's graded answers, and the GPU running out of memory while it samples a round's answers or trains on them. The round that failed is lost, as a round the time limit interrupts is. The run packages the adapter of its best round so far, which is the starting adapter when no round beat it, and shows Adapter ready. A failure in round 1 leaves only round 0 to keep, so the run completes with its starting adapter: the line reads "Stopped after round 0 of M: ... · best round 0", and the page adds "No round beat the starting point on validation, so the run kept its starting adapter."

The line under the chart says which failure it was, and a note under it gives the run's own words, which name no prompt, answer or reference. For a metric it reads, for example, "Custom metric name (vV) failed on N of M answers in round R: kinds. Make it handle any text." The round it names is the one that failed, the one after the last round in the line. For memory it reads "Sampling G answers of up to T tokens does not fit in this GPU's memory. Choose a GPU with more memory, fewer answers per prompt or a shorter longest sampled answer." when the answers to sample did not fit, or "Training on a micro batch of N answers of up to T tokens does not fit in this GPU's memory. Choose a GPU with more memory, a smaller micro batch size or a shorter longest sampled answer." when the update did (with a micro batch of 1 it leaves out the smaller micro batch size). The Training logs say "RL stopped (metric_failures) after round N of M; best round B." or "RL stopped (out_of_memory) after round N of M; best round B."

Everything else still fails the run: any other error, a failure while the starting adapter is scored, and any trial, whose one round is what it measures. A run that stopped this way is not failed, so it has no Retry and nothing starts it again by itself. To train on, fix the cause and configure a new run: Duplicate keeps the settings, and Start from the run continues its adapter.

• Too many metric failures. Make the metric handle any text, register the fix as a new version and choose it. See When a custom reward fails.
• Out of GPU memory. If sampling ran out, lower Answers per prompt or Longest sampled answer (tokens). If the update did, lower Micro batch size (under Advanced settings) or Longest sampled answer (tokens). Either way, a GPU with more memory helps too. A trial shows the memory a round needs before a full run does.

The result

The result is a regular LoRA adapter, from the round with the best validation score. The run shows Adapter ready, and Use this adapter tests, deploys, publishes or downloads it as any other adapter does. See When training finishes and The adapter.

The scores in the chart are scores on training prompts and on the validation sample that picked the round. They are not a held-out result. Choose Score against baseline to compare the adapter with the base model on the test examples: see Evaluate an adapter.

A failed or cancelled run can be retried. Retry starts a new run from round 1 with the same settings, the same starting adapter and the same version of a custom reward. See Retry a run. A run that stopped on its metric's failures or its GPU's memory is Adapter ready, not failed, so it has no Retry: see When a round fails.

Samples in your storage

The console shows a few samples. The run also writes every prompt, answer and reward of each round to a file, round-001.jsonl and so on. Where the files end up depends on your storage:

• A RunPod network volume, with RunPod training. They stay on the volume at runs/run ID/rollouts/, written as each round finishes.
• Your own S3 storage, or a RunPod volume with another training provider. They are uploaded to the same path in your storage after each round. When a run stops early (on its time limit, its metric's failures or its GPU's memory), its adapter is saved first and the files of its last rounds go up after it, so they can be missing if the GPU is released before they arrive.
• Connections Tensorant provides. They are not kept. The samples in the console are all there is.

Each line of a file holds a prompt, its reference, and every answer with its reward, whether it was cut off, the failure the metric gave if any, and its length in tokens. These files hold your prompts and answers: treat them as your data.

In an experiment

An experiment offers RL like DPO. Choose RL · reinforcement learning (GRPO) under Training objective in Setup. The experiment then measures the base model, trains with RL, and compares the two on the same test examples.

• The reward. The Compute step shows "Reward: the evaluation's main metric (metric)". Each sampled answer is scored by the metric the comparison uses, so the training pushes toward what you measure. Choose Choose another reward for a different one, and Use the evaluation's main metric to go back. If the main metric is the LLM judge or RadEval, the step asks you to choose a reward. A reused baseline fixes the evaluation settings, and so the default reward.
• The settings. Start from, Rounds, Prompts per round, Answers per prompt and Longest sampled answer (tokens) are the first fields under Advanced settings on the Compute step.
• The plan. Creating or starting an experiment that trains with RL needs the Pro plan. An experiment that reuses an adapter trains nothing, so it needs no RL plan, whatever objective trained that adapter.
• Following it. While training, the experiment shows the round, such as "Round 3 of 20", and once the loop ended, how it ended. An experiment goes on with the best adapter of a run that stopped on its time limit, its metric's failures or its GPU's memory, as it does with any adapter that is ready.

From the API or an agent

Requests use training.objective: "rl", training.method: "lora", training.gpu_count: 1 and a training.rl block. Leaving out training.rl uses the defaults below, and omitting the learning rate uses 5e-6.

{
  "rounds": 20,
  "prompts_per_round": 64,
  "answers_per_prompt": 8,
  "max_answer_tokens": 512,
  "start_run": null,
  "reward": null
}

A reward of null is the baseline's scoring and main metric. To name one, give an object with preset (qa, classification, structured_json or numeric), metric (a metric of that preset, such as token_f1, or custom. followed by a custom metric version's ID), the settings the preset needs (labels, response_schema, fields, absolute_tolerance, relative_tolerance), and custom_metric (that ID, or null). A saved run always holds a full reward.

An agent starts RL with start_training, which takes the settings as rounds, prompts_per_round, answers_per_prompt, max_answer_tokens and start_run, and always takes the baseline's main metric as the reward. See Training with RL.

Plan

RL training is on the Pro plan. A draft run saves on any plan. Launching an RL run or trial, and creating or starting an experiment that trains with RL, is refused on other plans with "RL training is on the Pro plan. An owner can change the plan under Settings → Plan and billing." The notice offers Open Plan and billing. An RL run counts toward your plan's training runs like any other, and a run that is already going is left alone if the plan changes. Using an adapter trained with RL, from testing it to publishing it, needs no particular plan. See RL training in Plans and billing.

Tips

• Run a trial first. A round samples several answers to every prompt and then trains on them, so RL uses more GPU time per prompt than SFT. A trial shows the memory, the seconds of each part and an estimate for the whole run. RL on mixture-of-experts models, gpt-oss included, is not yet proven: see Supported models.
• Start from SFT when the model rarely gets an answer right. RL can only reinforce what the model already sometimes does.
• Make the reward give partial credit when you can: Token F1 rather than exact match, or a custom metric that scores closeness.
• Keep the answers short. The shorter Longest sampled answer (tokens), the faster and lighter a round. Raise it only when answers are cut off.
• Thinking models. A model whose chat template reasons before it answers, such as Qwen3, writes its reasoning as part of the answer. The reasoning counts toward Longest sampled answer (tokens) and is scored as part of the answer, as evaluations score served answers. Raise Longest sampled answer (tokens) for such a model, or its answers are cut off and score 0. The built-in metrics score the whole text, so use a custom metric that reads the final answer out of it.
• Judge the adapter on the test examples. Use Score against baseline, not the reward the run reported.
• Read the samples. The model learns exactly what the metric scores, including its loopholes.
• Set the time limit with room. A run that reaches its time limit keeps its best adapter so far, but it has trained fewer rounds. The limit counts from launch, so GPU start-up and the model download use it, and training stops about 3 minutes before it. A run needs to recover only if the limit comes before round 0 (the starting adapter's score on the validation sample) is done, because nothing is kept yet; so does a trial that reaches its limit.

Limits

When something goes wrong

When you save or launch

In the training logs

A run that stops on its GPU shows "Kind: training stopped; inspect logs", such as "RuntimeError: training stopped; inspect logs". Read Training logs for the reason. Two of the reasons below, a metric that fails too often and a GPU that runs out of memory, do not fail a full run that has scored its starting adapter: it stops and shows Adapter ready. See When a round fails.

When a custom reward fails

When a custom reward fails, the log names the kind of failure. In a round it lists the kinds, such as execution, result. Before the first round the run tries the metric on one reference, and any failure then stops the run before the model loads, with the reason in words.

When the time limit passes

An RL run that is in a round when its time limit comes is not lost. It stops about 3 minutes early, keeps its best adapter so far and shows Adapter ready: see How a run ends. Recovery is needed only when no adapter was kept yet, and for a trial that reaches its limit before it finishes. For an RL run, no adapter yet means that its time limit came before round 0 was done: while the GPU started, the model and the data downloaded, a custom reward was set up, the adapter it starts from was copied or the starting adapter was scored. A trial's report then shows what it had measured, with no estimate. The messages below are the run's own error, not lines of Training logs.