Experiments

An experiment answers one question: does fine-tuning make this model better on your data? It measures the base model, trains an adapter, measures the adapter on the same test examples, tells you whether it meets your thresholds, and releases the GPUs. You set it up once and approve its cost, and it runs each step for you.

Use an experiment for your first adapter and whenever you want a fair, repeatable comparison. Everything it starts also appears where it would if you did it by hand: its deployments under Deployments, its run under Runs and its evaluations under Evaluations.

Before you start

• A frozen version with training, validation and test examples.
• RunPod compute and a RunPod network volume, connected by an owner in Connections.
• A free training slot and a free deployment slot. The Compute card in Connections shows your limits. Always-on published models share the deployment slots.
• The member or owner role. Viewers can follow experiments but not start them. See roles.

What an experiment does

1. Baseline. Deploys the base model, evaluates it on the version's test examples, then stops the deployment.
2. Train. Trains an adapter on the training examples, then releases the training GPU.
3. Deploy. Deploys the adapter with the same GPU, context length and serving settings as the base model.
4. Compare. Evaluates the adapter on the same test examples with the same settings.
5. Cleanup. Stops the tuned deployment (unless you keep it), confirms nothing is still running, and records whether the adapter meets your thresholds.

Before each launch, the experiment checks GPU capacity and prices again, because they change.

Create an experiment

1. Open Experiments under Tune and choose New experiment.
2. Fill in Setup, Compute limits, Evaluation and Acceptance thresholds. The sections are described below.
3. Choose Check readiness. Each check shows pass, warning or fail. See Readiness checks.
4. When every check passes, choose Save draft.

Saving does not start anything. It fixes the model revision to one commit, so later changes on Hugging Face do not affect the experiment.

Setup

Compute limits

These are hard limits for the training GPU and the two deployments.

When a GPU costs more than its limit, choose Raise the limit to $price/h to match it.

Advanced serving settings apply to both deployments. FP8 quantization is not available here, because it is not yet verified with adapters. See Serving settings.

Advanced training parameters holds the training settings as JSON (epochs, learning rate, rank, alpha, context length, batch size, gradient accumulation and disk sizes). The defaults are a good start. See Hyperparameters for what each one does.

Evaluation

Both evaluations use these settings, so the comparison is fair. See Presets and metrics.

An experiment's evaluations have no LLM judge. To score with a judge, run the evaluations by hand. See The LLM judge.

Acceptance thresholds

Set what "good enough" means for your use case.

Two more choices:

• Continue automatically through evaluation, training and cleanup after approval. See Automatic or stage by stage.
• Keep the tuned model running after the comparison. See Keep the tuned model running.

Approve and start

1. Select the draft and choose Review and launch.
2. Check the GPUs, their hours and price limits, and the Maximum GPU list-price estimate.
3. If warnings are listed, read them and tick I reviewed these launch and dataset warnings.
4. Keep Approved maximum compute estimate ($) at the estimate, or raise it.
5. Choose Approve and start.

The approved amount is recorded with the experiment. It does not stop spending: each stage's hour and price limits do.

Automatic or stage by stage

With Continue automatically… ticked, the experiment runs from start to finish without stopping.

Without it, the experiment pauses after each step and shows Paused. Review the step, then choose Review and continue and Approve and start to run the next one.

The experiment also pauses when a GPU has no capacity at launch time, or when another run or deployment is using the slot it needs. Wait, then choose Review and continue.

Follow an experiment

All experiments lists each experiment with its stage and status: Draft, Running, Paused, Cancelling, Cancelled, Failed or Completed. Select one to see:

• The stages, with the current one marked, and the Next action.
• This experiment needs attention, with the reason, when something stopped it.
• Training details, which opens the run with its loss chart and logs. See Follow a run.
• Compare answers, once it completed, which opens the comparison. See Compare two evaluations.

Read the result

When the experiment completes, it says Meets configured acceptance thresholds or Does not meet configured acceptance thresholds, with the reasons. It also shows:

• Tuned score: the adapter's average score on the primary metric.
• Change: the difference from the base model, in percentage points.
• Regressed answers: how many test examples scored lower with the adapter than with the base model.

An adapter misses the thresholds when its score is too low, its improvement is too small, the test set has too few independent groups, or (if you required it) the 95% range does not lie fully above zero.

Choose Compare answers to see which examples improved and which got worse. That tells you what data to add next. See Read a comparison.

Meeting the thresholds shows the adapter did better on this test set. It does not guarantee results in real use.

Keep the tuned model running

If you ticked Keep the tuned model running after the comparison, the tuned deployment stays up when the experiment completes. It becomes an ordinary deployment.

• Open playground lets you try it. See Playground.
• Stop it stops it now.

It stops by itself when idle or at its time limit. Until then it uses a deployment slot and RunPod bills it.

After an experiment

Follow-ups

A completed experiment shows Next dataset iteration, where you note what to fix in the next version.

1. Under New follow-up, describe the Observed failure pattern and the New data or review action.
2. Optionally set a category, priority and status.
3. Optionally link new examples you collected for it. Only training examples created in this project after the experiment can be linked.
4. Choose Record follow-up.

Collect new examples opens Sources, and Open Review opens Review.

Launch presets

Once an experiment completed and its GPUs are released, you can save its settings as a preset: enter a Launch preset name and choose Save successful settings. A preset keeps the model revision and the training and serving settings. It records settings that ran, not whether the adapter met your thresholds.

Presets are shared across your organization's projects. New experiments on the same base model offer them under Successful launch preset.

Cancel an experiment

1. Choose Cancel and clean up.
2. Choose Confirm cancellation.

The experiment stops its deployments and training run, then shows Cancelled. Your datasets and any finished adapter are kept. If its training run needs recovery, the experiment waits until you recover or cancel that run. See Needs recovery.

A failed experiment cleans up the same way. It cannot be resumed: fix the cause and create a new one. Its completed evaluations stay under Evaluations.

Costs

GPUs run in your RunPod account and RunPod bills them. The Maximum GPU list-price estimate assumes each of the three GPU stages runs for its full hours at its price limit:

training hours × training $/hour + 2 × hours per deployment × inference $/hour

With the defaults: 2 × $2 + 2 × 1 × $2 = $8. Most stages end sooner: each deployment stops when its evaluation is done, and training ends when the adapter is ready.

The estimate is not a billing cap. It leaves out the network volume, CPU and disk fees, and the time it takes to release a GPU. A paused deployment and a kept tuned model are billed while they run.

Tips

• Start with the default training settings. Change one thing at a time between experiments, so you know what made the difference.
• Use Continue automatically unless you want to inspect each stage. A paused deployment keeps costing money.
• Require the 95% interval only when your test set has at least 10 independent groups. With fewer, the adapter cannot meet the thresholds.
• After an experiment that ran well, save a launch preset so the next one starts from settings that worked.
• Use follow-ups to plan your next version from the answers that regressed.

Limits

When something goes wrong