Case study: a fine-tuned 7B model beats GPT-4o at text-to-SQL

Text-to-SQL is a good fit for fine-tuning. The model has to follow your schema exactly, the output is checkable, and the questions often contain company data that you would rather not send to a model provider. It is also a task where frontier models are strong, which makes it a demanding test.

This case study covers OmniSQL, a 2025 research result in which fine-tuned 7B, 14B and 32B open models matched or beat GPT-4o on standard text-to-SQL benchmarks. It explains what drove the result and where GPT-4o kept the lead, then shows how to fine-tune and test a SQL model of your own in Tensorant.

The result

OmniSQL (arXiv 2503.02240, Renmin University of China and ByteDance, March 2025, revised July 2025) fine-tuned Qwen2.5-Coder-Instruct models of 7B, 14B and 32B parameters. The authors' summary: "OmniSQL achieves state-of-the-art performance, matching or surpassing leading closed-source and open-source LLMs, including GPT-4o and DeepSeek-V3, despite its smaller size."

The paper measures execution accuracy: each generated query is run against the benchmark's database, and it counts as correct when it returns the same result as the reference query. A query can be written differently from the reference and still be correct.

Selected results with greedy decoding, from the paper's Tables 4 and 5:

The fully fine-tuned 7B model beat GPT-4o on all three main benchmarks and on the nine-dataset average. The LoRA version, the method Tensorant uses, beat GPT-4o on Spider and on the average, and came within 0.4 points on BIRD. With LoRA at 14B, OmniSQL-14B-LoRA scored 63.6 on BIRD dev, ahead of GPT-4o.

The authors' reason for the work will sound familiar: "Prompting-based methods often depend on closed-source LLMs, which are expensive, raise privacy concerns, and lack customization."

What drove the result

Training data that covers many databases. The models were trained on SynSQL-2.5M, a synthetic dataset of 2.5 million examples over more than 16,000 databases, together with the Spider and BIRD training sets. Each example pairs a question with its SQL and a step-by-step explanation. The breadth taught general habits, such as reading a schema carefully, rather than one database's answers.

A strong code model to start from. Untuned, Qwen2.5-Coder-7B-Instruct already scored 73.4 on Spider dev, above GPT-4o's 70.9. Spider is the easier benchmark, and part of the Spider win belongs to the base model.

The harder benchmark is where fine-tuning paid off. On BIRD, which has larger, messier databases, the 7B model rose from 50.9 to 63.9 and passed GPT-4o's 61.9. That gain came from training.

Less data still helps, up to a point. In an ablation, training the 7B model only on Spider and BIRD's own training sets, with explanations added, reached 86.6 on Spider test, ahead of GPT-4o's 83.2. On BIRD dev it reached 59.6, behind GPT-4o's 61.9. The large synthetic dataset closed that gap. Full training at that scale took "approximately 6, 12, and 20 days" for the three sizes on a machine with 8 A800 80 GB GPUs. A few thousand examples for your own schema cost far less.

Where GPT-4o stayed ahead

On three of the nine datasets, GPT-4o beat OmniSQL-7B:

All three use specialized or enterprise-scale databases. A fine-tuned model does best on the kind of questions and databases it was trained on, which is why training on your own schema matters.

Keep these limits in mind as well:

• The authors ran the evaluation, and list GPT-4o "as a reference" rather than as a tuned competitor.
• Reasoning models were left out "due to the prohibitively high inference costs."
• GPT-4o is no longer the newest frontier model. Measure again with the model you would actually use.

For an older, smaller example of the same pattern, Anyscale's 2023 Llama 2 study fine-tuned 7B and 13B models on the b-mc2/sql-create-context dataset and reported that "both the Llama-7b and 13b fine-tuned models outperform the 70b-chat and GPT-4 models." See the LoRA Land and Anyscale case study.

Step 1: Prepare question and SQL pairs

The best training data is your own: questions your users ask, the reviewed SQL that answers them, and the CREATE TABLE statements of the tables involved. Query logs from analysts are a good source. Keep each example's schema to the tables the question needs, so prompts stay short.

To try the workflow first, use b-mc2/sql-create-context (CC BY 4.0), the dataset from Anyscale's study. It has 78,577 questions built from WikiSQL and Spider, each with its tables' CREATE TABLE statements. This script samples 8,000 training and 1,000 test questions. It splits by table, so the test questions are about tables training never saw:

import hashlib
import json
import random
import urllib.request

SYSTEM = (
    "Write one SQLite query that answers the question, using only the tables given. "
    "Reply with the query on one line, with no code block, no explanation and no semicolon. "
    "Put text values in double quotes."
)
URL = "https://huggingface.co/datasets/b-mc2/sql-create-context/resolve/main/sql_create_context_v4.json"

with urllib.request.urlopen(URL) as response:
    rows = json.load(response)

def held_out(row):
    # Every question about the same tables lands on the same side of the split.
    return hashlib.sha256(row["context"].encode()).digest()[0] < 26

random.seed(7)
train = random.sample([row for row in rows if not held_out(row)], 8000)
test = random.sample([row for row in rows if held_out(row)], 1000)

for name, sample in [("sql-train.jsonl", train), ("sql-test.jsonl", test)]:
    with open(name, "w", encoding="utf-8") as out:
        for row in sample:
            out.write(json.dumps({"messages": [
                {"role": "system", "content": SYSTEM},
                {"role": "user", "content": f"Tables:\n{row['context']}\n\nQuestion: {row['question'].strip()}"},
                {"role": "assistant", "content": row["answer"]},
            ]}) + "\n")

The system message states the output rules, including the dataset's double-quote style, so every model is told the conventions it is scored on. The labels come from public datasets and have quirks, such as values written in lowercase. Use this dataset to learn the workflow, and make decisions on your own data.

Step 2: Create the project and import

1. Open the project switcher at the top of the sidebar and choose New project.
2. Choose Support assistant as the use case. It only sets starting defaults, and it starts scoring as Q&A, which this task needs.
3. Enter SQL writer as the Project name and Qwen/Qwen2.5-Coder-7B-Instruct, OmniSQL's starting model, as the Hugging Face base model. Choose Create project.
4. Open Sources, choose Add sources, upload sql-train.jsonl, choose Training and validation, and import it.
5. Upload sql-test.jsonl the same way, choosing Test (held out).

Open Versions and choose Create version. Check Readiness for training with the default Training window (tokens) of 2,048. If your own schemas are long, raise it until every example fits. See Versions.

Step 3: Train an adapter and score the base model

Open Experiments, choose New experiment, and select the version.

Choose a Training GPU marked Fits, set your price and time limits, choose Check and review launch, and approve the estimate. The experiment scores the base model, trains the adapter and scores it on the same test questions. See Experiments.

Normalized exact match counts a query as correct only when it is the reference query, ignoring case and spacing. That is stricter than execution accuracy: WHERE a = 1 AND b = 2 and WHERE b = 2 AND a = 1 return the same rows but don't match. Use it to compare models during development, and confirm the winner by running queries in Step 5.

Step 4: Score a frontier model on the same test

1. An owner adds the frontier model in Connections under Model endpoints: a Name such as frontier, the provider's OpenAI-compatible Base address, the Model name and an API key. See Model endpoints.
2. Open Evaluations, choose Run evaluation, keep Base model selected, choose the same Dataset version, and choose frontier under Model to score.
3. Set Answers are, Main metric and Longest answer (tokens) as in the experiment, leave System prompt (optional) empty, and choose Run baseline evaluation.
4. Open the adapter's evaluation and choose the frontier evaluation under Compare with.

The provider receives every test prompt, including your schemas, and bills you for each call. Evaluations send temperature: 0 and a max_tokens limit, which some reasoning models refuse. See Compare any two evaluations.

Read the frontier model's misses before drawing conclusions. If most are the same query written another way, exact match is understating it, and Step 5 gives the fairer number.

Step 5: Measure execution accuracy on your database

Execution accuracy needs a database to run queries against. With your own data, use a read-only copy of the database the questions are about. Publish the adapter first: an owner opens Models, chooses Publish a model, names it sql-writer and selects the adapter. See Publishing models and the adapter deployment guide.

This script sends each test question to both models, runs every query, and compares the rows each returns with the reference query's rows:

import json
import os
import sqlite3
from openai import OpenAI

tensorant = OpenAI(base_url="https://api.tensorant.io/v1", api_key=os.environ["TENSORANT_API_KEY"])
frontier = OpenAI(base_url=os.environ["FRONTIER_BASE_URL"], api_key=os.environ["FRONTIER_API_KEY"])
models = {"sql-writer": tensorant, os.environ["FRONTIER_MODEL"]: frontier}
database = sqlite3.connect("file:database-copy.db?mode=ro", uri=True)

def result(sql):
    try:
        # Sorted, so row order only matters to questions you check by hand.
        return sorted(map(repr, database.execute(sql).fetchall()))
    except sqlite3.Error:
        return None

correct, scored = dict.fromkeys(models, 0), 0
for line in open("sql-test.jsonl"):
    *prompt, reference = json.loads(line)["messages"]
    expected = result(reference["content"])
    if expected is None:
        continue
    scored += 1
    for name, client in models.items():
        reply = client.chat.completions.create(model=name, messages=prompt, temperature=0, max_tokens=256)
        correct[name] += result((reply.choices[0].message.content or "").strip()) == expected

for name, count in correct.items():
    print(f"{name}: {count / scored:.1%} execution accuracy on {scored} questions")

Install the client with pip install openai, and keep both keys in environment variables. A wake-on-request model can answer 503 model_starting while it starts: retry after the Retry-After delay. See cold starts and retries. Use a copy of the database, because generated queries can be slow, and a read-only connection, because they are not trusted.

When to expect the same result on your task

A fine-tuned SQL model tends to win when:

• Questions are about a known set of databases, and you can collect a few thousand reviewed question and SQL pairs for them.
• Your schema has conventions a prompt explains poorly: naming, which table holds the truth, which filters always apply.
• Schemas and questions contain data you would rather keep in your own account than send to a model provider.
• Query volume is high enough for per-request cost and response time to matter.

Prefer a frontier model when questions span many unfamiliar databases, or need long multi-step reasoning across large schemas: the cases where GPT-4o kept its lead in OmniSQL.

Before you trust any score, keep tables and paraphrased questions from the test set out of training. See data leakage in LLM evaluation.