Case study: fine-tuned 7B models beat GPT-4 on narrow tasks

A general model has to work out your conventions from a prompt. A fine-tuned model learns them from examples. On a narrow task with a fixed output format, that difference can be large enough for a 7-billion-parameter model to beat a frontier model many times its size.

This case study looks at two published results: what was measured, why the small models won, and where they lost. Then it shows how to run the same kind of comparison with your own data and GPUs in Tensorant, against whichever frontier model you use today.

The results

LoRA Land (arXiv 2405.00732, Predibase, April 2024) fine-tuned 10 open base models of 7 billion parameters or fewer on 31 tasks: 310 adapters in total, all trained with 4-bit LoRA (QLoRA). The paper's summary: "4-bit LoRA fine-tuned models outperform base models by 34 points and GPT-4 by 10 points on average." Averaged over the 31 tasks, the best fine-tuned model per task scored 0.756 and GPT-4 (gpt-4-0613) scored 0.661. 224 of the 310 adapters beat GPT-4's score on their task.

Selected rows from the paper's Table 4:

Anyscale's Llama 2 study (blog post, August 2023) fully fine-tuned Llama 2 7B and 13B on three tasks. With the 13B model, accuracy rose "from, 58% to 98% on functional representations, 42% to 89% on SQL generation, and 28% to 47% on GSM." Those are before-and-after numbers for Llama 2 itself. Against GPT-4, the post reports that on SQL "both the Llama-7b and 13b fine-tuned models outperform the 70b-chat and GPT-4 models." On math word problems (GSM8k), the fine-tuned models improved but did not reach GPT-4.

The "functional representations" task is ViGGO run the other way from LoRA Land: read a sentence and write its meaning representation, such as give_opinion(name[Little Nightmares], rating[good], genres[adventure, platformer, puzzle], player_perspective[side view]). That direction has one correct answer per sentence, so it can be scored exactly. It is the task you will run below.

Why the small models won

The tasks were narrow and the output format was fixed. ViGGO's meaning representations use 9 functions and 14 attributes, and the attributes always appear in the same order. A prompt can state those rules. Several thousand examples also teach the cases a description leaves out: which attribute a phrase belongs to, how values are spelled, and when to leave an attribute out. Anyscale observed that "while GPT-4's accuracy significantly drops when attribute precedence is considered, the outputs of the fine-tuned models always follow precedence."

There was enough labeled data. ViGGO has 5,103 training examples. LoRA Land trained each adapter on the task's own training split. A frontier model given a short prompt has none of that.

The scoring rewarded the expected answer, not a plausible one. A frontier model that writes a reasonable answer in a different format loses points on ROUGE-L or accuracy. That is not unfair to it: an application parsing the output fails the same way.

The small models were cheap to train and serve. LoRA Land trained every adapter "on a single A10 GPU with 24 GB of memory", and served 25 fine-tuned Mistral 7B adapters from one A100 GPU, since adapters can share one copy of the base model.

Where GPT-4 stayed ahead

The paper is direct about it: "GPT-4 continues to outperform fine-tuned models in 6 out of 31 tasks, particularly in broader, more complex domains such as Python coding and MMLU." Coding, broad knowledge questions and multi-step math need more than a format a small model can learn from one task's examples.

Read the numbers with these limits in mind:

• Both studies ran their own evaluations. Predibase and Anyscale sell fine-tuning and serving.
• GPT-4 got simple prompts. LoRA Land used "simple zero or single-shot completion-style prompts for all tasks." A careful prompt with more examples would likely narrow some gaps, especially where GPT-4's low score suggests it did not learn the expected labels.
• The metrics are proxies. LoRA Land used ROUGE-L for generation tasks, even for SQL, and scored only the first 1,000 examples of large test sets.
• GPT-4 is no longer the frontier. The result says something about the shape of a task, not about a model generation. Measure it again on your task with the model you would actually use, which is what the rest of this guide does.

Step 1: Prepare ViGGO as conversations

ViGGO is public on Hugging Face under CC BY-SA 4.0. This Python script, which needs nothing beyond Python 3, downloads the training and test splits and writes them as chat conversations. The system message gives every model the full rules, including the attribute order, so the base model and the frontier model get a fair description of the task.

import csv
import io
import json
import urllib.request

SYSTEM = (
    "Convert the user's sentence about video games into a meaning representation. "
    "Reply with one line in the form function(attribute[value], attribute[value]) and no other text.\n"
    "Functions: inform, request, give_opinion, confirm, verify_attribute, suggest, "
    "request_explanation, recommend, request_attribute.\n"
    "Attributes, always in this order: name, exp_release_date, release_year, developer, esrb, "
    "rating, genres, player_perspective, has_multiplayer, platforms, available_on_steam, "
    "has_linux_release, has_mac_release, specifier.\n"
    "Include only the attributes the sentence mentions. Separate several values with commas."
)

def rows(split):
    url = f"https://huggingface.co/datasets/GEM/viggo/resolve/main/{split}.csv"
    with urllib.request.urlopen(url) as response:
        return list(csv.DictReader(io.StringIO(response.read().decode("utf-8-sig"))))

for split, name in [("train", "viggo-train.jsonl"), ("test", "viggo-test.jsonl")]:
    with open(name, "w", encoding="utf-8") as out:
        for row in rows(split):
            out.write(json.dumps({
                "messages": [
                    {"role": "system", "content": SYSTEM},
                    {"role": "user", "content": row["ref"]},
                    {"role": "assistant", "content": row["mr"]},
                ],
                "category": row["mr"].split("(")[0],
            }) + "\n")

It writes viggo-train.jsonl with 5,103 conversations and viggo-test.jsonl with 1,083. Each conversation's category is its function, such as inform or request, so results can be read per function.

Step 2: Create the project and import both files

1. Open the project switcher at the top of the sidebar and choose New project.
2. Choose Support assistant as the use case. It only sets starting defaults, and it starts scoring as Q&A, which this task needs.
3. Enter ViGGO as the Project name and Qwen/Qwen2.5-7B-Instruct as the Hugging Face base model, then choose Create project.
4. Open Sources, choose Add sources, and upload viggo-train.jsonl. Under Use these sources for, choose Training and validation, then Import files.
5. Choose Add sources again, upload viggo-test.jsonl, choose Test (held out), and import it.

Leave Approve examples that pass checks ticked. ViGGO repeats a few sentences word for word, so the import report may list some duplicates. See the import report.

Step 3: Create a version

Open Review and look through a sample of examples from both files. Then open Versions and choose Create version.

Rows from the test file always stay in test. Tensorant splits the training file itself: some of its rows go to validation, and about a tenth can also go to test. Your test score therefore covers ViGGO's test split plus a share of its training split, never the same sentence on both sides. See how groups are assigned.

Check Readiness for training with the default Training window (tokens) of 2,048. ViGGO conversations are short, so all of them fit.

Step 4: Train an adapter and score the base model

Open Experiments, choose New experiment, and select the version. An experiment evaluates the base model, trains the adapter and evaluates it on the same test examples with the same settings.

Normalized exact match counts an answer as correct only if it is the reference, ignoring case and spacing. It is stricter than LoRA Land's ROUGE-L, and closer to what a parser needs. Token F1 is recorded too and gives partial credit.

Choose a Training GPU marked Fits: for a 7B model with QLoRA, a 24 GB GPU is usually enough. Set your price and time limits, choose Check and review launch, read the cost estimate, and choose Approve and start. See Experiments.

Step 5: Score a frontier model on the same test

To bring a frontier model into the comparison, score it on the same version with the same settings.

1. An owner adds it in Connections under Model endpoints: a Name such as frontier, the provider's OpenAI-compatible Base address, the Model name and an API key. Choose Check connection. See Model endpoints.
2. Open Evaluations, choose Run evaluation, and keep Base model selected.
3. Choose the same Dataset version, and choose frontier under Model to score.
4. Set Answers are, Main metric and Longest answer (tokens) exactly as in the experiment. Leave System prompt (optional) empty so each test example keeps its own.
5. Choose Run baseline evaluation.

Tensorant can't tell which model an endpoint serves, so name it clearly. The provider receives every test prompt and bills you for each call. Evaluations send temperature: 0 and a max_tokens limit: choose a model whose API accepts both, which rules out some reasoning models. See Your own endpoints.

Step 6: Read the result

You now have three evaluations of the same test examples: the base model, the adapter and the frontier model.

1. In the experiment, choose Compare answers to see how the adapter changed the base model's answers.
2. Open the adapter's evaluation. Under Compare with, choose the frontier model's evaluation. The sentence at the top gives the change in points and a likely range. See Compare any two evaluations.
3. Read the per-category results. A strong average can hide a weak function, such as request_attribute, which has the fewest examples.
4. Read the answers the frontier model got wrong. Typical misses are attribute order, attribute names and value spelling: the conventions the adapter learned from examples.

Your numbers will not match the studies'. The base model, metric, prompt and test set all differ. What carries over is the method: the same test examples, the same scoring and the same settings for every model.

When to expect the same result on your task

Fine-tuning a small model tends to win when:

• The output has a fixed format or a fixed set of labels that your application checks.
• You have hundreds to thousands of reviewed examples that follow one policy.
• A wrong format costs as much as a wrong answer.
• The volume is high enough for per-request cost and response time to matter.

Prefer a frontier model, or try retrieval first, when the task needs broad knowledge, long reasoning or code, or when its rules change faster than you can collect examples. See Fine-tuning vs RAG.

Keep the test fair as you iterate. A test set you tune against stops measuring new cases: see data leakage in LLM evaluation. For a text-to-SQL version of this comparison, against GPT-4o, read the text-to-SQL case study.