Small language models
Fine-tune small language models for edge and on-device use
Fine-tune Qwen3 4B, SmolLM2 or Llama 3.2 1B for one narrow job, compare the result with the base model, and download the LoRA adapter.
The problem
Why train a small model?
A model of 1 to 4 billion parameters needs less memory and compute for each answer than a large one. It costs less to run and fits on smaller hardware.
Out of the box, a small model handles a narrow job less reliably than a large one. Fine-tuning trains it on examples of that one job, and an experiment measures whether the tuned model does the job better than the base model did.
How it works
What fine-tuning adds
- The model trains on examples of the one job you need, each an input and the answer you expect. Import them as question and answer pairs or as conversations. See how rows are recognized.
- A trial trains for a few steps on a small sample first. It checks chat formatting, GPU memory and speed, and estimates the full run's time and GPU cost before you pay for full training.
- An experiment scores the base model, trains an adapter and scores the adapter on the same test examples. You see whether the small model got better at the job, answer by answer.
- Tensorant estimates that LoRA training needs about 7.6 GiB of GPU memory for a model of 1.5 billion parameters, and about 13.7 GiB for 4 billion. QLoRA needs less. See GPU memory.
In practice
What you download
- The adapter downloads as a ZIP with
adapter_config.json,adapter_model.safetensors, the tokenizer files,training_manifest.jsonand the training configuration. See The adapter. - It loads with any tool that loads PEFT LoRA adapters, on the same base model and revision it was trained on.
- A QLoRA run still exports an adapter, not a standalone 4-bit model.
- You can also publish the adapter to your own Hugging Face account, in a private repository by default. See Publishing adapters to Hugging Face.
Models
Models to start with
These models have 1 to 4 billion parameters. Every family Tensorant trains is in Supported models.
Qwen/Qwen3-4B-Instruct-2507
The largest of these four. It answers directly, without a reasoning block.
HuggingFaceTB/SmolLM2-1.7B-Instruct
A small instruct model built on Llama's architecture. It answers directly, without reasoning first.
unsloth/Llama-3.2-1B-Instruct
An ungated copy of Llama 3.2 1B Instruct, the smallest of these four.
Qwen/Qwen3-1.7B
It writes its reasoning before the answer, as part of its answer. For a job that needs only a label or JSON, choose one of the Instruct models.
Steps
How to do it in Tensorant
Create a project
Choose a small model as the base model when you create the project. The console checks it on Hugging Face and shows its size.
Import and review
Import examples of the job in Sources, then approve them in Review.
Freeze a version
Create a version. It splits the approved examples into training, validation and test examples, and holds the test examples out for scoring.
Run a trial, then an experiment
A trial checks formatting, memory and speed on a small sample. Then an experiment trains the adapter and scores it against the base model.
Download the adapter
Open the run or its experiment, choose Use this adapter, then Download adapter. See The adapter.
Plans
What you need
- Text fine-tuning, evaluations and adapter downloads are on every organization. Each training run pays a training-token fee from credits. See Training-token fee.
- GPUs come from Tensorant compute, charged per second from prepaid credits, which an owner adds under Settings, Plan and billing (the first payment turns Tensorant compute on), or from a GPU account you connect with the Own compute subscription. See connections.
Fine-tuning
LoRA vs QLoRA: choose a fine-tuning method
Understand LoRA and QLoRA memory tradeoffs, choose practical settings, and compare adapters fairly in Tensorant before deployment.
Fine-tuning
Estimate LLM fine-tuning cost before launching
Budget training, base-versus-tuned evaluation, storage, iteration, and inference with Tensorant's provider controls and transparent cost formulas.
Case studies
Case study: fine-tuned 7B models beat GPT-4 on narrow tasks
How small LoRA-tuned open models outscored GPT-4 in the LoRA Land and Anyscale studies, and how to run that same comparison in Tensorant.
Private and sensitive data
Medical, legal and financial records: what Tensorant stores, where it runs, and what can stay in your cloud.
Classification and routing
Answer with one label from your list: intents, ticket routes, what needs a person.
JSON extraction
Pull the fields your code needs out of invoices, forms and emails, as JSON that fits your schema.
Replace a costly LLM API
Train an open model on the answers your API model gives, and switch when it scores as well.
Scoring and ranking
Score each candidate answer with a calibrated probability, with a decision model such as kev.
Reinforcement learning
Train with GRPO against a reward that checks each answer: a label, a number, JSON or your own code.
Migrate from OpenAI fine-tuning
OpenAI ends fine-tuning in January 2027. Move your data, train an open model and compare the two.
Start with one dataset.
Create an account and import your data, or book a demo and we will show you the product.