Skip to content

Small language models

Fine-tune small language models for edge and on-device use

Fine-tune Qwen3 4B, SmolLM2 or Llama 3.2 1B for one narrow job, compare the result with the base model, and download the LoRA adapter.

The problem

Why train a small model?

A model of 1 to 4 billion parameters needs less memory and compute for each answer than a large one. It costs less to run and fits on smaller hardware.

Out of the box, a small model handles a narrow job less reliably than a large one. Fine-tuning trains it on examples of that one job, and an experiment measures whether the tuned model does the job better than the base model did.

How it works

What fine-tuning adds

  • The model trains on examples of the one job you need, each an input and the answer you expect. Import them as question and answer pairs or as conversations. See how rows are recognized.
  • A trial trains for a few steps on a small sample first. It checks chat formatting, GPU memory and speed, and estimates the full run's time and GPU cost before you pay for full training.
  • An experiment scores the base model, trains an adapter and scores the adapter on the same test examples. You see whether the small model got better at the job, answer by answer.
  • Tensorant estimates that LoRA training needs about 7.6 GiB of GPU memory for a model of 1.5 billion parameters, and about 13.7 GiB for 4 billion. QLoRA needs less. See GPU memory.

In practice

What you download

  • The adapter downloads as a ZIP with adapter_config.json, adapter_model.safetensors, the tokenizer files, training_manifest.json and the training configuration. See The adapter.
  • It loads with any tool that loads PEFT LoRA adapters, on the same base model and revision it was trained on.
  • A QLoRA run still exports an adapter, not a standalone 4-bit model.
  • You can also publish the adapter to your own Hugging Face account, in a private repository by default. See Publishing adapters to Hugging Face.

Models

Models to start with

These models have 1 to 4 billion parameters. Every family Tensorant trains is in Supported models.

  • Qwen/Qwen3-4B-Instruct-2507

    The largest of these four. It answers directly, without a reasoning block.

  • HuggingFaceTB/SmolLM2-1.7B-Instruct

    A small instruct model built on Llama's architecture. It answers directly, without reasoning first.

  • unsloth/Llama-3.2-1B-Instruct

    An ungated copy of Llama 3.2 1B Instruct, the smallest of these four.

  • Qwen/Qwen3-1.7B

    It writes its reasoning before the answer, as part of its answer. For a job that needs only a label or JSON, choose one of the Instruct models.

See every supported model

Steps

How to do it in Tensorant

  1. Create a project

    Choose a small model as the base model when you create the project. The console checks it on Hugging Face and shows its size.

  2. Import and review

    Import examples of the job in Sources, then approve them in Review.

  3. Freeze a version

    Create a version. It splits the approved examples into training, validation and test examples, and holds the test examples out for scoring.

  4. Run a trial, then an experiment

    A trial checks formatting, memory and speed on a small sample. Then an experiment trains the adapter and scores it against the base model.

  5. Download the adapter

    Open the run or its experiment, choose Use this adapter, then Download adapter. See The adapter.

Plans

What you need

  • Text fine-tuning, evaluations and adapter downloads are on every organization. Each training run pays a training-token fee from credits. See Training-token fee.
  • GPUs come from Tensorant compute, charged per second from prepaid credits, which an owner adds under Settings, Plan and billing (the first payment turns Tensorant compute on), or from a GPU account you connect with the Own compute subscription. See connections.

Guides

Related guides

Browse all guides

Use cases

Other use cases

See every use case

Start with one dataset.

Create an account and import your data, or book a demo and we will show you the product.