Skip to content

Reinforcement learning

Fine-tune LLMs with reinforcement learning on tasks you can check

Train an LLM with GRPO against a reward that checks its answers: a label, a number, JSON that fits a schema or a Python function you write.

The problem

What if you can check an answer but not write the ideal one?

SFT learns from answers you wrote or approved, so each example needs the whole answer you want. For many tasks you can check an answer more easily than you can write the ideal one: a final number, a label from a list, JSON that fits a schema.

RL learns from a score for each answer the model writes itself. The reference can be just the part the reward checks, such as a final number, and not a full written solution. See When to use RL.

How it works

How an RL run learns

  • In each round, the model writes several answers to each training prompt, a reward scores every answer from 0 to 1, and the adapter moves toward the answers that scored higher than the others to the same prompt. See RL training. The reinforcement learning page follows one prompt through a round.
  • RL works best when the model already gets some answers right and some wrong. If it almost never gets one right, train with SFT first, then start RL from that adapter.
  • The reward is a check that runs on the GPU machine, so the LLM judge can't be a reward. RL is not for answers that only a person or another model can judge. See When to use RL.

In practice

Rewards you can use

  • For Q&A answers, the reward can be Token F1, normalized exact match or JSON validity. Token F1 gives a partly right answer partial credit, so the model can tell a better answer from a worse one.
  • For classification, it can be classification accuracy against your allowed labels, normalized exact match or JSON validity.
  • For structured JSON, it can be JSON-schema correctness, JSON field accuracy on the fields you list, normalized exact match or JSON validity.
  • For numeric answers, it can be numeric accuracy within the tolerances you set, normalized exact match or JSON validity.
  • You can also write the reward in Python, as a score(answer, reference) function that returns a number from 0 to 1. It runs in a sandbox on the GPU machine. See Custom metrics.
  • A full run starts with its baseline's scoring and main metric as the reward, so it trains toward what you measure. Change any field to train toward something else. See Choose the reward.

Models

Models to start with

RL trains text models with LoRA on one GPU. RL on mixture-of-experts models is not yet proven. See Supported models.

  • Qwen/Qwen3-4B-Instruct-2507

    A 4B instruct model that answers directly, without a reasoning block, so the reward scores the answer alone.

  • unsloth/Llama-3.2-1B-Instruct

    An ungated copy of Llama 3.2 1B Instruct, the smaller of these two.

See every supported model

Steps

How to do it in Tensorant

  1. Freeze a version

    Create a version of text examples. RL reads it as it is: each example's last message is the reference the reward checks, and the conversation before it is the prompt. See Prepare your data.

  2. Score a baseline

    Run a baseline evaluation of the version. A full RL run needs one, and its scoring and main metric become the reward unless you choose another.

  3. Run an RL trial

    Choose RL as the training objective and run a trial first. It runs one round on up to 8 prompts, and reports the peak GPU memory, the seconds each part took and an estimate of the full run's time and GPU cost.

  4. Run it and compare

    Launch the full run. It keeps the adapter of the round with the best validation score. Then choose Score against baseline to compare the adapter with the base model on the test examples. See The result.

  5. Publish it

    The result is a regular LoRA adapter. Give it a name in API models and call it as you would any other adapter.

Plans

What you need

  • RL training is on every organization: launching an RL run or trial, and creating or starting an experiment that trains with RL. Each run pays the training-token fee for the prompts and answers it trains on. See Plan and fee in RL training.
  • GPUs come from Tensorant compute, charged per second from prepaid credits, which an owner adds under Settings, Plan and billing (the first payment turns Tensorant compute on), or from a GPU account you connect with the Own compute subscription. Each round samples several answers to every prompt, so RL uses more GPU time per prompt than SFT. See Before you start.

Guides

Related guides

Browse all guides

Use cases

Other use cases

See every use case

Start with one dataset.

Create an account and import your data, or book a demo and we will show you the product.