Skip to content

Replace a costly LLM API

Replace a costly LLM API with a fine-tuned open model

Train an open model on answers your API model already gives, score both on the same test set, and serve yours behind an OpenAI-compatible API.

The problem

Why pay a frontier model for a narrow, repeated job?

A large general model is billed per token on every request. When your application sends it the same kind of request all day, such as tagging a ticket or pulling fields out of a form, you pay a large general model's price for a task that may need far less model.

A smaller open model can learn that one task from the answers your API model already gives. Tensorant scores both models on the same test examples, so you switch when the tuned model scores as well on your task.

If the model you call is an OpenAI fine-tune, Migrate from OpenAI fine-tuning covers moving its training files and comparing it with an open model.

How it works

How the swap is measured

  • In published studies, small open models fine-tuned for one narrow task outscored GPT-4 and GPT-4o on that task. Our case studies cover LoRA Land and Anyscale and OmniSQL's text-to-SQL models, including the tasks where the larger model stayed ahead.
  • You add your API model as a model endpoint and score it on your version's test examples. An experiment then scores your tuned model on the same examples.
  • You open the tuned model's evaluation and compare it with the API model's. The comparison gives the change in points with a likely range, and lists each answer that improved or got worse.

Models

Models to start with

Start with a small instruct model. Every family Tensorant trains is in Supported models.

  • Qwen/Qwen3-4B-Instruct-2507

    A 4B instruct model that answers directly, without a reasoning block. Try it first.

  • mistralai/Mistral-7B-Instruct-v0.3

    A 7B instruct model to try when a 4B model falls short on your test set.

  • unsloth/gpt-oss-20b-BF16

    A 20B mixture-of-experts model for harder tasks. This BF16 copy trains on an 80 GB GPU.

See every supported model

Steps

How to do it in Tensorant

  1. Collect examples

    Import the prompts and answers your application logged, as conversations or question and answer columns. See Sources. Or have a strong model write examples from your documents with a recipe. Either way, check your model provider's terms on training another model with its outputs first.

  2. Freeze a version

    Approve the examples in Review and create a version. It holds its test examples out of training, and both models are scored on them.

  3. Score the API model

    Add the API model as a model endpoint and run an evaluation of the version on it. Each test example it scores is one request, which the provider bills. See Your own endpoints.

  4. Run an experiment

    Run an experiment on an open model, with the scoring settings you gave the API model. It scores the base model, trains an adapter and scores the adapter on the same test examples.

  5. Compare the two

    Open the adapter's evaluation and choose the API model's evaluation under Compare with. See Compare any two evaluations.

  6. Publish it and switch

    Give the adapter a name in API models. In your application, change the OpenAI SDK's base_url to https://api.tensorant.io/v1, along with the key and the model name. See the API quickstart.

Plans

What you need

  • Text fine-tuning, evaluations and publishing are on every organization, with the same limits for all. Each training run pays a training-token fee from credits. See Limits on every organization.
  • GPUs come from Tensorant compute, charged per second from prepaid credits, which an owner adds under Settings, Plan and billing (the first payment turns Tensorant compute on), or from a GPU account you connect with the Own compute subscription. See connections.
  • A published model on Tensorant compute is charged per second while its machine runs. On demand, the machine starts when a request arrives and keeps running for a warm period after the last one, 60 seconds unless you change it, and that time is charged too. See On demand or always on.

Guides

Related guides

Browse all guides

Use cases

Other use cases

See every use case

Start with one dataset.

Create an account and import your data, or book a demo and we will show you the product.