Replace a costly LLM API
Replace a costly LLM API with a fine-tuned open model
Train an open model on answers your API model already gives, score both on the same test set, and serve yours behind an OpenAI-compatible API.
The problem
Why pay a frontier model for a narrow, repeated job?
A large general model is billed per token on every request. When your application sends it the same kind of request all day, such as tagging a ticket or pulling fields out of a form, you pay a large general model's price for a task that may need far less model.
A smaller open model can learn that one task from the answers your API model already gives. Tensorant scores both models on the same test examples, so you switch when the tuned model scores as well on your task.
If the model you call is an OpenAI fine-tune, Migrate from OpenAI fine-tuning covers moving its training files and comparing it with an open model.
How it works
How the swap is measured
- In published studies, small open models fine-tuned for one narrow task outscored GPT-4 and GPT-4o on that task. Our case studies cover LoRA Land and Anyscale and OmniSQL's text-to-SQL models, including the tasks where the larger model stayed ahead.
- You add your API model as a model endpoint and score it on your version's test examples. An experiment then scores your tuned model on the same examples.
- You open the tuned model's evaluation and compare it with the API model's. The comparison gives the change in points with a likely range, and lists each answer that improved or got worse.
Models
Models to start with
Start with a small instruct model. Every family Tensorant trains is in Supported models.
Qwen/Qwen3-4B-Instruct-2507
A 4B instruct model that answers directly, without a reasoning block. Try it first.
mistralai/Mistral-7B-Instruct-v0.3
A 7B instruct model to try when a 4B model falls short on your test set.
unsloth/gpt-oss-20b-BF16
A 20B mixture-of-experts model for harder tasks. This BF16 copy trains on an 80 GB GPU.
Steps
How to do it in Tensorant
Collect examples
Import the prompts and answers your application logged, as conversations or question and answer columns. See Sources. Or have a strong model write examples from your documents with a recipe. Either way, check your model provider's terms on training another model with its outputs first.
Freeze a version
Approve the examples in Review and create a version. It holds its test examples out of training, and both models are scored on them.
Score the API model
Add the API model as a model endpoint and run an evaluation of the version on it. Each test example it scores is one request, which the provider bills. See Your own endpoints.
Run an experiment
Run an experiment on an open model, with the scoring settings you gave the API model. It scores the base model, trains an adapter and scores the adapter on the same test examples.
Compare the two
Open the adapter's evaluation and choose the API model's evaluation under Compare with. See Compare any two evaluations.
Publish it and switch
Give the adapter a name in API models. In your application, change the OpenAI SDK's
base_urltohttps://api.tensorant.io/v1, along with the key and the model name. See the API quickstart.
Plans
What you need
- Text fine-tuning, evaluations and publishing are on every organization, with the same limits for all. Each training run pays a training-token fee from credits. See Limits on every organization.
- GPUs come from Tensorant compute, charged per second from prepaid credits, which an owner adds under Settings, Plan and billing (the first payment turns Tensorant compute on), or from a GPU account you connect with the Own compute subscription. See connections.
- A published model on Tensorant compute is charged per second while its machine runs. On demand, the machine starts when a request arrives and keeps running for a warm period after the last one, 60 seconds unless you change it, and that time is charged too. See On demand or always on.
Case studies
Case study: fine-tuned 7B models beat GPT-4 on narrow tasks
How small LoRA-tuned open models outscored GPT-4 in the LoRA Land and Anyscale studies, and how to run that same comparison in Tensorant.
Case studies
Case study: a fine-tuned 7B model beats GPT-4o at text-to-SQL
How fine-tuned Qwen2.5-Coder 7B models outperformed GPT-4o on Spider and BIRD benchmarks, and how to test your own SQL model in Tensorant.
Fine-tuning
Estimate LLM fine-tuning cost before launching
Budget training, base-versus-tuned evaluation, storage, iteration, and inference with Tensorant's provider controls and transparent cost formulas.
Private and sensitive data
Medical, legal and financial records: what Tensorant stores, where it runs, and what can stay in your cloud.
Classification and routing
Answer with one label from your list: intents, ticket routes, what needs a person.
JSON extraction
Pull the fields your code needs out of invoices, forms and emails, as JSON that fits your schema.
Small language models
Fine-tune a 1B to 4B model for one narrow job, and download the adapter.
Scoring and ranking
Score each candidate answer with a calibrated probability, with a decision model such as kev.
Reinforcement learning
Train with GRPO against a reward that checks each answer: a label, a number, JSON or your own code.
Migrate from OpenAI fine-tuning
OpenAI ends fine-tuning in January 2027. Move your data, train an open model and compare the two.
Start with one dataset.
Create an account and import your data, or book a demo and we will show you the product.