Curated Behavior Cloning: Small LLMs Can Beat Large Ones at 5-30x Lower Cost

· Andrew Jesson, Gabriel Bianconi, Aaron Hill, Viraj Mehta

Summary

  • Fine-tuning small models on automatically curated high-quality outputs from a large model can beat large-model performance while reducing inference costs by up to 30x and inference time by up to 4x.

  • We benchmark closed-source (OpenAI, Google) and open-source (Qwen) models on multi-turn maze navigation (BabyAI), agentic RAG (Multi-Hop), and agentic tool use (τ-bench).

  • Learn how to reproduce this workflow for your own use case using open-source LLMOps tools, even without a GPU, along with best practices for production deployments.

The Performance-Cost Dilemma

Every LLM application designer faces a fundamental tradeoff: large models deliver superior performance at higher costs, while small models are affordable but often underperform. This choice becomes critical when scaling to production workloads where inference costs can quickly explode.

Consider a customer service agent handling thousands of conversations daily. Using GPT-4.1 or Claude Sonnet 4 might provide excellent responses, but at $2-$15 per 1M tokens, costs accumulate rapidly. Switch to a smaller model and costs plummet, but so might customer satisfaction.

What if you could have both?

Our research demonstrates a way to escape this tradeoff. As illustrated below, we find that fine-tuned small models can achieve 10-20x cost reductions while improving or maintaining competitive performance across diverse applications. Additionally, these models deliver substantially faster response times, improving user experience while reducing costs.

Cost per success comparison across tasks
Figure 1: Cost per success comparison across tasks. While GPT-4.1 (blue bars) delivers strong performance, it incurs substantially higher costs per successful task completion (cost per task ÷ success rate) compared to fine-tuned alternatives. Fine-tuned models — like Gemini 2.0 Flash or GPT-4.1 mini — achieve dramatic cost reductions while maintaining competitive task success rates.

We achieve these savings through a synthetic imitation learning approach that fine-tunes small models using high-quality responses from large models. Beyond cost savings, our experiments reveal that fine-tuned small models can:

Unlike inference-time optimization techniques that require costly inference-time compute, fine-tuning shifts the computational burden to a one-time training phase, making every subsequent inference dramatically cheaper.

The Approach

We systematically evaluate fine-tuning across three challenging task categories that represent real-world LLM applications:

  1. Multi-turn navigation (BabyAI GoTo)
  2. Retrieval Augmented Generation (RAG) (Multi-Hop)
  3. Agentic tool use (τ-bench retail and airline domains)

For each task type, we:

We use multiple fine-tuning providers (OpenAI, Google Vertex, Unsloth) and leverage TensorZero to seamlessly switch between providers and environments without infrastructure complexity.

Results

Our evaluation reveals significant cost efficiency improvements when using fine-tuned models compared to GPT-4.1. When measuring cost per success (cost per task ÷ success rate), all fine-tuned models demonstrate substantial reductions in the resources required to achieve successful task completion. The following table provides a detailed breakdown of cost-efficiency improvements across each individual task environment.

Table: Cost Savings Factor Relative to GPT-4.1

Fine-tuned Model BabyAI GoTo Multi-Hop τ-bench Airline τ-bench Retail
Gemini 2.0 Flash Lite 29.4× 23.1× 10.4× 15.2×
GPT-4.1 nano 21.2× 14.2× 7.3× 7.3×
Gemini 2.0 Flash 16.3× 13.6× 8.7× 9.4×
GPT-4o mini 14.4× 14.4× 5.2× 8.7×
Qwen3-8B 4.6× 9.1× 1.8× 4.0×
GPT-4.1 mini 6.0× 5.7× 3.4× 4.5×

Values show the reduction factor in cost per success (cost per task ÷ success rate) compared to GPT-4.1. Higher values indicate greater cost efficiency improvements.

Across all evaluation environments — treating τ-bench (airline and retail) as a single environment group — Gemini 2.0 Flash Lite (fine-tuned) achieves the highest cost efficiency with 21.8x lower cost per success than GPT-4.1, followed by GPT-4.1 nano (fine-tuned) at 14.2x and Gemini 2.0 Flash (fine-tuned) at 13.0x.

1. Multi-Turn Navigation (BabyAI GoTo)

Task: Navigate to target locations in a text-based game environment.

Success: Agent reaches the target within 32 steps.

Figure 2: Most fine-tuned small models (crosses) exceed GPT-4.1's performance while being 5-20x cheaper and 2-4x faster. Use the toggles to explore different metrics (pass^1 vs pass^5) and dimensions (cost vs response time). Fine-tuned models consistently outperform their zero-shot counterparts across both cost efficiency and response speed.

Key Finding: Fine-tuning eliminates the performance-cost tradeoff for this navigation task by consistently improving both the success rate (pass^1) and reliability (pass^5) across all model families except Qwen3-8B. All fine-tuned models outperform their zero-shot counterparts at significantly lower costs.

BabyAI Evaluation Details Fine-tuning data:

446 successful GPT-4.1 conversations as demonstrations (356 training, 90 validation).

A GPT-4.1 Demonstration (GCP Vertex Format)
{
  "systemInstruction": {
    "role": "system",
    "parts": [
      {
        "text": "You are an agent playing a simple navigation game.\nYour goal is to go to the green box.\nThe following are the possible actions you can take in the game, followed by a short description of each action:\n\nturn left: turn to the left\nturn right: turn to the right\ngo forward: take one step forward\npick up: pick up the object below you\ndrop: drop the object that you are holding\ntoggle: manipulate the object in front of you\n\nTips:\n- Once the desired object you want to interact or pickup in front of you, you can use the 'toggle' action to interact with it.\n- It doesn't make sense to repeat the same action over and over if the observation doesn't change.\n\nYou always have to output one of the above actions at a time and no other text. You always have to output an action until the episode terminates.\n\nPLAY!"
      }
    ]
  },
  "contents": [
    {
      "role": "user",
      "parts": [
        {
          "text": "Observation:\na wall 2 steps forward\na wall 2 steps right\na green box 3 steps left"
        }
      ]
    },
    {
      "role": "model",
      "parts": [
        {
          "text": "turn left"
        }
      ]
    },
    {
      "role": "user",
      "parts": [
        {
          "text": "Observation:\na wall 2 steps right\na green box 3 steps forward"
        }
      ]
    },
    {
      "role": "model",
      "parts": [
        {
          "text": "go forward"
        }
      ]
    },
    {
      "role": "user",
      "parts": [
        {
          "text": "Observation:\na wall 2 steps right\na green box 2 steps forward"
        }
      ]
    },
    {
      "role": "model",
      "parts": [
        {
          "text": "go forward"
        }
      ]
    }
  ]
}

Evaluation data:

100 test tasks × 5 repetitions.

Agentic RAG (Multi-Hop)

Task: Retrieve Wikipedia articles to verify a claim using a multi-policy search.

Success: Agent returns the top-5 ground-truth Wikipedia article titles.

Figure 3: Fine-tuned small models can outperform their large model teachers without policy specific feedback. Cross markers show fine-tuned variants achieving higher success rates than GPT-4.1 while operating at significantly lower costs and faster response times. Toggle between metrics and dimensions to explore the complete performance landscape.

Key Finding: Fine-tuned small models can surpass their large model teachers without policy-level feedback. Fine-tuned Gemini 2.0 Flash, GPT-4o mini, and GPT-4.1 mini exceed GPT-4.1’s success rate, with fine-tuned GPT-4.1 mini and GPT-4o mini also exceeding its reliability. Additionally, fine-tuned models deliver faster response times, with Qwen3-8B (fine-tuned) responding in ~513ms compared to GPT-4.1’s ~1305ms.

Multi-Hop Evaluation Details Architecture:

Two specialized policies working together:

  • Policy 1: Generates search queries given a claim and accumulated notes
  • Policy 2: Extracts relevant titles and creates notes for the next search

Training Approach:

We fine-tune two separate versions of each small model, one for each policy. Importantly, we only use successful conversation demonstrations from GPT-4.1, not policy-specific feedback.

Fine-tuning data:

671 successful GPT-4.1 conversations as demonstrations (536 training, 135 validation).

A GPT-4.1 Generate Query Demonstration (GCP Vertex Format)
{
  "systemInstruction": {
    "role": "system",
    "parts": [
      {
        "text": "You are a research assistant helping investigate the following claim:\n\n\"Gina Bramhill was born in a village. The 2011 population of the area that includes this village was 167,446.\"\n\nYour role is to generate a focused and effective search query that will retrieve new, relevant evidence about the claim.\n\nAt each step, you will receive:\n- A list of \"New Notes\" extracted from the latest round of retrieved documents.\n- Your own previously generated queries, if any.\n\nUse this information to guide what to search for next:\n- Avoid repeating queries that have already been used.\n- Consider what new facts or perspectives would help resolve the claim.\n- Build on prior notes to deepen or expand the investigation.\n\nPlease return your next search query in the following JSON format:\n\n{\n    \"query\": \"your next search query here\"\n}"
      }
    ]
  },
  "contents": [
    {
      "role": "user",
      "parts": [
        {
          "text": "New Notes:\n\n['']"
        }
      ]
    },
    {
      "role": "model",
      "parts": [
        {
          "text": "Query:\n\nGina Bramhill birthplace village population 2011"
        }
      ]
    },
    {
      "role": "user",
      "parts": [
        {
          "text": "New Notes:\n\n['Gina Bramhill was born in Eastoft, where she grew up on a farm. There is no information in these results about the population of the area that includes Eastoft, nor a reference to the figure 167,446.']"
        }
      ]
    },
    {
      "role": "model",
      "parts": [
        {
          "text": "Query:\n\n2011 population census Eastoft North Lincolnshire"
        }
      ]
    },
    {
      "role": "user",
      "parts": [
        {
          "text": "New Notes:\n\n['Eastoft is a village and civil parish in North Lincolnshire, England.', 'The population of North Lincolnshire unitary authority at the 2011 census was 167,446.', 'Eastoft is located within North Lincolnshire.']"
        }
      ]
    },
    {
      "role": "model",
      "parts": [
        {
          "text": "Query:\n\nNorth Lincolnshire civil parishes list 2011 census population Eastoft"
        }
      ]
    },
    {
      "role": "user",
      "parts": [
        {
          "text": "New Notes:\n\n['Eastoft is a village and civil parish in North Lincolnshire, England, confirming its classification as a village within that unitary authority.']"
        }
      ]
    },
    {
      "role": "model",
      "parts": [
        {
          "text": "{\n    \"query\": \"Gina Bramhill birthplace Eastoft relationship to North Lincolnshire population 2011\"\n}"
        }
      ]
    }
  ]
}
A GPT-4.1 Append Notes Demonstration (GCP Vertex Format)
{
  "systemInstruction": {
    "role": "system",
    "parts": [
      {
        "text": "You are an expert assistant helping to gather factual evidence for the following claim:\n\n\"Gina Bramhill was born in a village. The 2011 population of the area that includes this village was 167,446.\"\n\nYou will be shown search results from the latest query. These may contain relevant background, biographical details, event descriptions, or supporting facts.\n\nYour task is to:\n1. Extract **new, useful insights** that help evaluate the claim.\n2. Ensure that each new note is **concise**, **fact-based**, and **non-redundant** with earlier notes seen in the conversation.\n3. Identify and include the **titles** of the most relevant sources from which you derived the new notes.\n\nOnly include information that is directly relevant to the claim or helps resolve open gaps.\n\nPlease return your response in the following JSON format:\n\n{\n    \"new_notes\": [\"fact-based note 1\", \"fact-based note 2\", ...],\n    \"titles\": [\"title 1\", \"title 2\", ...]\n}"
      }
    ]
  },
  "contents": [
    {
      "role": "user",
      "parts": [
        {
          "text": "Search Results:\n\n{'Gina Bramhill | Gina Bramhill was born in Eastoft, where she grew up on a farm.  As a child, she appeared in several school plays.  She was trained at the Royal Academy of Dramatic Art.  Shortly after graduating she appeared as Bella in the movie Lotus Eaters.  2012 she got a role as the recurring character Eve Sands in the TV series Being Human.  In the same year Bramhill played one of the main roles in the drama pilot The Frontier.  In Coronation Street she portrayed the character Jodie Woodward.  She got a main role in the movie Pleasure Island, which was shown at the Cannes Film Festival in 2014.': 15.67646312713623, 'Užulėnis | Užulėnis is a village in Ukmergė District Municipality, Vilnius County, Lithuania.  According to the 2001 census, the village had a population of 90 people.  The population decreased to 68 at the time of the 2011 census.  The village is the birthplace of the first President of Lithuania Antanas Smetona.': 10.659812927246094, 'Vukmanić | Vukmanić is a village belonging to Karlovac in Croatia, population 207 (2011).  It was the birthplace of Ivan Ribar who preceded Josip Broz Tito as post-World War II leader of Yugoslavia.': 9.863264083862305, 'Azinhaga | Azinhaga (] ) is a village and a civil parish in the municipality of Golegã, located in Ribatejo, Portugal.  The population in 2011 was 1,620, in an area of 38.21\\xa0km².  It was the birthplace of the Nobel Prize-winning author José Saramago.': 9.802595138549805, \"Denby | Denby is a village in the English county of Derbyshire that is notable as the birthplace of John Flamsteed, England's first Astronomer Royal, and the location of the Denby Pottery Company.  The population at the 2001 Census was 1,827, increasing to 2,190 at the 2011 Census.\": 9.683469772338867, 'Smiljan | Smiljan (] ) is a village in the mountainous region of Western Lika in Croatia.  It is located 6 km northwest of Gospić, and fifteen kilometers from the Zagreb-Split highway; its population is 418 (2011).  Smiljan is famous as the birthplace of scientist Nikola Tesla.': 9.567204475402832, 'Varvara, Chalkidiki | Varvara is a village in Chalkidiki, in Greece.  It is located in the north of Chalkidiki, built at 550 m height, between the hills Derveniko and Sougkliani.  It is part of Aristotelis municipality.  Its population is 538 residents according to 2011 census.  It is 20 kilometers northwest of Arnaia.  In the east of the village there are the Aegean coasts and the ancient city Stagira, the birthplace of Aristotelis.': 9.453887939453123, 'Elaiochori, Achaea | Elaiochori (Greek: Ελαιοχώρι) is a village in the municipality of West Achaea, Greece.  It is located about 7\\xa0km south of Kato Achaia and 23\\xa0km southwest of Patras.  The nearest larger village is Petrochori, 3\\xa0km to the west.  The population of Elaiochori was 238 in 2011.  Elaiochori is the birthplace of the leader of the military government that ruled the country from 1967 to 1974, Georgios Papadopoulos.': 9.30756378173828, 'Balawa | Balawa is a village development committee in Mahottari District in the Janakpur Zone of south-eastern Nepal.  At the time of the 2011 Nepal census.  it had a population of 15332 people living in 4232 individual households.The birthplace of great Nepali scientist and writer Saqib Shamim is Balkawa.It the main palace for market and also it has a helth post and a ilaka police office for six village.': 9.21255588531494, 'Bogan Gate | Bogan Gate is a small village in Parkes Shire of the Central West of New South Wales, Australia.  At the 2011 census , Bogan Gate and the surrounding area had a population of 307.  Bogan Gate is derived from the local Aboriginal word meaning \"the birthplace of a notable headman of the local tribe\".': 9.128785133361816}"
        }
      ]
    },
    {
      "role": "model",
      "parts": [
        {
          "text": "New Notes:\n\n['Gina Bramhill was born in Eastoft, where she grew up on a farm. There is no information in these results about the population of the area that includes Eastoft, nor a reference to the figure 167,446.']\n\nTitles:\n\n['Gina Bramhill']"
        }
      ]
    },
    {
      "role": "user",
      "parts": [
        {
          "text": "Search Results:\n\n{'Eastoft | Eastoft is a village and civil parish in North Lincolnshire, England.  It is situated within the Isle of Axholme, 3 mi north-east from Crowle, and on the A161 road.': 16.75237274169922, 'North East Lincolnshire | North East Lincolnshire is a unitary authority area in the Yorkshire and the Humber region of England, bordering the unitary authority of North Lincolnshire and the non-metropolitan county of Lincolnshire.  The population of the Unitary Authority at the 2011 Census was 159,616.  These three administrative units make up the ceremonial county of Lincolnshire.': 13.91547966003418, 'Holme, North Lincolnshire | Holme is a linear settlement and civil parish in North Lincolnshire, England.  The population of the civil parish at the 2011 census was 113.': 13.73335075378418, 'Allington, Lincolnshire | Allington is a village and civil parish in the South Kesteven district of Lincolnshire, England.  It is situated 3 mi north-west of Grantham.  The 2001 Census gave a parish population of 728 in 329 households.  This increased to a population of 897 in 422 households in the 2011 census.': 13.411900520324709, 'North Lincolnshire | North Lincolnshire is a unitary authority area in the region of Yorkshire and the Humber in England.  The population of the Unitary Authority at the 2011 census was 167,446.  For ceremonial purposes it is part of Lincolnshire.  There are three significant towns: Scunthorpe (the administrative centre), Brigg and Barton-upon-Humber.': 13.348052978515623, 'Wilsford, Lincolnshire | Wilsford is a village and civil parish in the North Kesteven district of Lincolnshire, England.  The population of the civil parish was 400 at the 2011 census.': 13.272210121154783, 'Elsham, North Lincolnshire | Elsham is a village and civil parish in North Lincolnshire, England.  The population of the civil parish at the 2011 census was 400.  It is situated 4 mi north from Brigg, 1 mi north from the M180 and 1 mi west from the A15 road.': 13.251594543457031, 'North Carlton, Lincolnshire | North Carlton is a village and civil parish in the West Lindsey district of Lincolnshire, England.  The population of the civil parish at the 2011 census was 172.  It is situated about 5 mi north from the city and county town of Lincoln.': 13.234726905822754, 'Pinchbeck, Lincolnshire | Pinchbeck is a village and civil parish in the South Holland district of Lincolnshire, England.  The civil parish population was 5,153 at the 2001 census, increasing to 5,455 at the 2011 census.  It is situated 2 mi north from the centre of Spalding.': 13.147077560424805, 'Walcott, Lincolnshire | Walcott is a small village and civil parish in the North Kesteven district of Lincolnshire, England.  The population of the civil parish at the 2011 census was 566.  It is situated approximately 2 mi north from Billinghay and 7 mi north-east from the town of Sleaford.': 13.11349105834961}"
        }
      ]
    },
    {
      "role": "model",
      "parts": [
        {
          "text": "New Notes:\n\n['Eastoft is a village and civil parish in North Lincolnshire, England.', 'The population of North Lincolnshire unitary authority at the 2011 census was 167,446.', 'Eastoft is located within North Lincolnshire.']\n\nTitles:\n\n['Eastoft', 'North Lincolnshire']"
        }
      ]
    },
    {
      "role": "user",
      "parts": [
        {
          "text": "Search Results:\n\n{'Eastoft | Eastoft is a village and civil parish in North Lincolnshire, England.  It is situated within the Isle of Axholme, 3 mi north-east from Crowle, and on the A161 road.': 22.09337615966797, 'Holme, North Lincolnshire | Holme is a linear settlement and civil parish in North Lincolnshire, England.  The population of the civil parish at the 2011 census was 113.': 20.58799171447754, 'Wilsford, Lincolnshire | Wilsford is a village and civil parish in the North Kesteven district of Lincolnshire, England.  The population of the civil parish was 400 at the 2011 census.': 20.126848220825195, 'North Carlton, Lincolnshire | North Carlton is a village and civil parish in the West Lindsey district of Lincolnshire, England.  The population of the civil parish at the 2011 census was 172.  It is situated about 5 mi north from the city and county town of Lincoln.': 19.795473098754883, 'Elsham, North Lincolnshire | Elsham is a village and civil parish in North Lincolnshire, England.  The population of the civil parish at the 2011 census was 400.  It is situated 4 mi north from Brigg, 1 mi north from the M180 and 1 mi west from the A15 road.': 19.76159286499023, 'Pinchbeck, Lincolnshire | Pinchbeck is a village and civil parish in the South Holland district of Lincolnshire, England.  The civil parish population was 5,153 at the 2001 census, increasing to 5,455 at the 2011 census.  It is situated 2 mi north from the centre of Spalding.': 19.733495712280273, 'Walcott, Lincolnshire | Walcott is a small village and civil parish in the North Kesteven district of Lincolnshire, England.  The population of the civil parish at the 2011 census was 566.  It is situated approximately 2 mi north from Billinghay and 7 mi north-east from the town of Sleaford.': 19.623489379882812, 'Broughton, Lincolnshire | Broughton is a small town and civil parish situated on the Roman Ermine Street, in the North Lincolnshire district of Lincolnshire, England.  The population of the civil parish at the 2011 census was 5,726.  It is situated approximately 2 mi north-west from the town of Brigg.  The hamlets of Wressle, Castlethorpe, and part of Scawby Brook lie within the parish boundaries.': 19.578060150146484, 'Manton, North Lincolnshire | Manton is a village and civil parish in North Lincolnshire, England.  The population of the civil parish at the 2011 census was 123.  The village is situated just south from the town of Scunthorpe, and about 6 mi south-west from the town of Brigg.  The parish includes the hamlet of Cleatham.  Cleatham was a civil parish between 1866 and 1936.': 19.575958251953125, 'Haconby | Haconby is a village and civil parish in the South Kesteven district of Lincolnshire, England.  The population of the civil parish at the 2001 census was 448 increasing to 532 at the 2011 census.  It is situated on the western edge of the Lincolnshire Fens, 3 mi north from Bourne.': 19.557518005371097}"
        }
      ]
    },
    {
      "role": "model",
      "parts": [
        {
          "text": "New Notes:\n\n['Eastoft is a village and civil parish in North Lincolnshire, England, confirming its classification as a village within that unitary authority.']\n\nTitles:\n\n['Eastoft']"
        }
      ]
    },
    {
      "role": "user",
      "parts": [
        {
          "text": "Search Results:\n\n{'Gina Bramhill | Gina Bramhill was born in Eastoft, where she grew up on a farm.  As a child, she appeared in several school plays.  She was trained at the Royal Academy of Dramatic Art.  Shortly after graduating she appeared as Bella in the movie Lotus Eaters.  2012 she got a role as the recurring character Eve Sands in the TV series Being Human.  In the same year Bramhill played one of the main roles in the drama pilot The Frontier.  In Coronation Street she portrayed the character Jodie Woodward.  She got a main role in the movie Pleasure Island, which was shown at the Cannes Film Festival in 2014.': 21.69491958618164, 'Eastoft | Eastoft is a village and civil parish in North Lincolnshire, England.  It is situated within the Isle of Axholme, 3 mi north-east from Crowle, and on the A161 road.': 16.75237274169922, 'North East Lincolnshire | North East Lincolnshire is a unitary authority area in the Yorkshire and the Humber region of England, bordering the unitary authority of North Lincolnshire and the non-metropolitan county of Lincolnshire.  The population of the Unitary Authority at the 2011 Census was 159,616.  These three administrative units make up the ceremonial county of Lincolnshire.': 12.128313064575195, 'Holme, North Lincolnshire | Holme is a linear settlement and civil parish in North Lincolnshire, England.  The population of the civil parish at the 2011 census was 113.': 11.716436386108398, 'North Lincolnshire | North Lincolnshire is a unitary authority area in the region of Yorkshire and the Humber in England.  The population of the Unitary Authority at the 2011 census was 167,446.  For ceremonial purposes it is part of Lincolnshire.  There are three significant towns: Scunthorpe (the administrative centre), Brigg and Barton-upon-Humber.': 11.508501052856444, 'Elsham, North Lincolnshire | Elsham is a village and civil parish in North Lincolnshire, England.  The population of the civil parish at the 2011 census was 400.  It is situated 4 mi north from Brigg, 1 mi north from the M180 and 1 mi west from the A15 road.': 11.390216827392578, 'North Carlton, Lincolnshire | North Carlton is a village and civil parish in the West Lindsey district of Lincolnshire, England.  The population of the civil parish at the 2011 census was 172.  It is situated about 5 mi north from the city and county town of Lincoln.': 11.351000785827637, 'Wilsford, Lincolnshire | Wilsford is a village and civil parish in the North Kesteven district of Lincolnshire, England.  The population of the civil parish was 400 at the 2011 census.': 11.255294799804688, 'Walcott, Lincolnshire | Walcott is a small village and civil parish in the North Kesteven district of Lincolnshire, England.  The population of the civil parish at the 2011 census was 566.  It is situated approximately 2 mi north from Billinghay and 7 mi north-east from the town of Sleaford.': 11.252113342285156, 'Alford, Lincolnshire | Alford (pronounced \"Olford\") is a town in Lincolnshire, England, about 11 mi north-west of the coastal resort of Skegness, at the foot of the Lincolnshire Wolds, an Area of Outstanding Natural Beauty.  Its population was 3,459 in the 2011 Census.': 11.2147216796875}"
        }
      ]
    },
    {
      "role": "model",
      "parts": [
        {
          "text": "{\n    \"new_notes\": [\"Gina Bramhill was born in Eastoft, which is a village in North Lincolnshire, England.\", \"The population of North Lincolnshire unitary authority at the 2011 census was 167,446.\"],\n    \"titles\": [\"Gina Bramhill\", \"Eastoft\", \"North Lincolnshire\"]\n}"
        }
      ]
    }
  ]
}

Evaluation data:

115 test tasks × 5 repetitions.

3. Agentic Tool Use (τ-bench)

τ-bench benchmarks multi-turn, agentic tool-use agents in both retail and airline customer service environments. We augment the τ-bench system instruction with the Anthropic think-tool prompt in all evaluations.

Success: Agent completes the user’s request with correct database changes (measured by state comparison) and communicates all required information.

Key Finding: The τ-bench results reveal the practical limits and opportunities of fine-tuning. In the retail domain, fine-tuned models offer compelling cost-performance tradeoffs that could satisfy many production requirements. We only fine-tune using retail conversations because there is no training set for the airline domain. The airline results show that fine-tuned models generally maintain stable performance on related tasks, with some models showing modest improvements over their base counterparts.

Retail Domain (in-distribution)

Task: Handle multi-turn customer service conversations for order management including cancellations, modifications, returns, exchanges, and address updates while following domain-specific policies and authentication requirements.

Figure 4: While fine-tuned small models don't quite reach GPT-4.1's performance, they can recover 90% accuracy at 5x lower cost and significantly faster response times. All fine-tuned models improve substantially over their zero-shot baselines across both cost efficiency and response speed.
τ-bench Retail Evaluation Details

Fine-tuning data:

410 successful GPT-4.1 conversations as demonstrations (328 training, 82 validation).

A GPT-4.1 Retail Agent Demonstration (GCP Vertex Format)
{
  "systemInstruction": {
    "role": "system",
    "parts": [
      {
        "text": "# Retail agent policy\n\nAs a retail agent, you can help users cancel or modify pending orders, return or exchange delivered orders, modify their default user address, or provide information about their own profile, orders, and related products.\n\n- At the beginning of the conversation, you have to authenticate the user identity by locating their user id via email, or via name + zip code. This has to be done even when the user already provides the user id.\n\n- Once the user has been authenticated, you can provide the user with information about order, product, profile information, e.g. help the user look up order id.\n\n- You can only help one user per conversation (but you can handle multiple requests from the same user), and must deny any requests for tasks related to any other user.\n\n- Before taking consequential actions that update the database (cancel, modify, return, exchange), you have to list the action detail and obtain explicit user confirmation (yes) to proceed.\n\n- You should not make up any information or knowledge or procedures not provided from the user or the tools, or give subjective recommendations or comments.\n\n- You should transfer the user to a human agent if and only if the request cannot be handled within the scope of your actions.\n\n## Domain basic\n\n- All times in the database are EST and 24 hour based. For example \"02:30:00\" means 2:30 AM EST.\n\n- Each user has a profile of its email, default address, user id, and payment methods. Each payment method is either a gift card, a paypal account, or a credit card.\n\n- Our retail store has 50 types of products. For each type of product, there are variant items of different options. For example, for a 't shirt' product, there could be an item with option 'color blue size M', and another item with option 'color red size L'.\n\n- Each product has an unique product id, and each item has an unique item id. They have no relations and should not be confused.\n\n- Each order can be in status 'pending', 'processed', 'delivered', or 'cancelled'. Generally, you can only take action on pending or delivered orders.\n\n- Exchange or modify order tools can only be called once. Be sure that all items to be changed are collected into a list before making the tool call!!!\n\n## Cancel pending order\n\n- An order can only be cancelled if its status is 'pending', and you should check its status before taking the action.\n\n- The user needs to confirm the order id and the reason (either 'no longer needed' or 'ordered by mistake') for cancellation.\n\n- After user confirmation, the order status will be changed to 'cancelled', and the total will be refunded via the original payment method immediately if it is gift card, otherwise in 5 to 7 business days.\n\n## Modify pending order\n\n- An order can only be modified if its status is 'pending', and you should check its status before taking the action.\n\n- For a pending order, you can take actions to modify its shipping address, payment method, or product item options, but nothing else.\n\n### Modify payment\n\n- The user can only choose a single payment method different from the original payment method.\n\n- If the user wants the modify the payment method to gift card, it must have enough balance to cover the total amount.\n\n- After user confirmation, the order status will be kept 'pending'. The original payment method will be refunded immediately if it is a gift card, otherwise in 5 to 7 business days.\n\n### Modify items\n\n- This action can only be called once, and will change the order status to 'pending (items modifed)', and the agent will not be able to modify or cancel the order anymore. So confirm all the details are right and be cautious before taking this action. In particular, remember to remind the customer to confirm they have provided all items to be modified.\n\n- For a pending order, each item can be modified to an available new item of the same product but of different product option. There cannot be any change of product types, e.g. modify shirt to shoe.\n\n- The user must provide a payment method to pay or receive refund of the price difference. If the user provides a gift card, it must have enough balance to cover the price difference.\n\n## Return delivered order\n\n- An order can only be returned if its status is 'delivered', and you should check its status before taking the action.\n\n- The user needs to confirm the order id, the list of items to be returned, and a payment method to receive the refund.\n\n- The refund must either go to the original payment method, or an existing gift card.\n\n- After user confirmation, the order status will be changed to 'return requested', and the user will receive an email regarding how to return items.\n\n## Exchange delivered order\n\n- An order can only be exchanged if its status is 'delivered', and you should check its status before taking the action. In particular, remember to remind the customer to confirm they have provided all items to be exchanged.\n\n- For a delivered order, each item can be exchanged to an available new item of the same product but of different product option. There cannot be any change of product types, e.g. modify shirt to shoe.\n\n- The user must provide a payment method to pay or receive refund of the price difference. If the user provides a gift card, it must have enough balance to cover the price difference.\n\n- After user confirmation, the order status will be changed to 'exchange requested', and the user will receive an email regarding how to return items. There is no need to place a new order.\n\n## Using the think tool\n\nBefore taking any action or responding to the user after receiving tool results, use the think tool as a scratchpad to:\n\n- List the specific rules that apply to the current request\n\n- Check if all required information is collected\n\n- Verify that the planned action complies with all policies\n\n- Iterate over tool results for correctness \n\nHere are some examples of what to iterate over inside the think tool:\n\n<think_tool_example_1>\nUser wants to cancel flight ABC123\n- Need to verify: user ID, reservation ID, reason\n- Check cancellation rules:\n  * Is it within 24h of booking?\n  * If not, check ticket class and insurance\n- Verify no segments flown or are in the past\n- Plan: collect missing info, verify rules, get confirmation\n</think_tool_example_1>\n\n<think_tool_example_2>\nUser wants to book 3 tickets to NYC with 2 checked bags each\n- Need user ID to check:\n  * Membership tier for baggage allowance\n  * Which payments methods exist in profile\n- Baggage calculation:\n  * Economy class × 3 passengers\n  * If regular member: 1 free bag each → 3 extra bags = $150\n  * If silver member: 2 free bags each → 0 extra bags = $0\n  * If gold member: 3 free bags each → 0 extra bags = $0\n- Payment rules to verify:\n  * Max 1 travel certificate, 1 credit card, 3 gift cards\n  * All payment methods must be in profile\n  * Travel certificate remainder goes to waste\n- Plan:\n1. Get user ID\n2. Verify membership level for bag fees\n3. Check which payment methods in profile and if their combination is allowed\n4. Calculate total: ticket price + any bag fees\n5. Get explicit confirmation for booking\n</think_tool_example_2>"
      }
    ]
  },
  "contents": [
    {
      "role": "user",
      "parts": [
        {
          "text": "Hi! I need to cancel an order that I no longer need."
        }
      ]
    },
    {
      "role": "model",
      "parts": [
        {
          "text": "I’d be happy to help you cancel your order. To get started, I’ll need to verify your identity. Could you please provide the email address associated with your account?"
        }
      ]
    },
    {
      "role": "user",
      "parts": [
        {
          "text": "Sure! My email address is [email protected]."
        }
      ]
    },
    {
      "role": "model",
      "parts": [
        {
          "functionCall": {
            "name": "find_user_id_by_email",
            "args": {
              "email": "[email protected]"
            }
          }
        }
      ]
    },
    {
      "role": "user",
      "parts": [
        {
          "functionResponse": {
            "name": "find_user_id_by_email",
            "response": {
              "result": "daiki_johnson_9523"
            }
          }
        }
      ]
    },
    {
      "role": "model",
      "parts": [
        {
          "text": "Thank you for providing your email. To proceed with the cancellation, could you please provide the order ID of the order you wish to cancel? If you’re not sure about the order ID, let me know and I can help you locate it."
        }
      ]
    },
    {
      "role": "user",
      "parts": [
        {
          "text": "I want to cancel order #W1436802."
        }
      ]
    },
    {
      "role": "model",
      "parts": [
        {
          "functionCall": {
            "name": "get_order_details",
            "args": {
              "order_id": "#W1436802"
            }
          }
        }
      ]
    },
    {
      "role": "user",
      "parts": [
        {
          "functionResponse": {
            "name": "get_order_details",
            "response": {
              "result": "{\"order_id\": \"#W1436802\", \"user_id\": \"daiki_johnson_9523\", \"address\": {\"address1\": \"939 Elm Street\", \"address2\": \"Suite 261\", \"city\": \"Jacksonville\", \"country\": \"USA\", \"state\": \"FL\", \"zip\": \"32273\"}, \"items\": [{\"name\": \"Tea Kettle\", \"product_id\": \"9832717871\", \"item_id\": \"3909406921\", \"price\": 98.25, \"options\": {\"material\": \"glass\", \"capacity\": \"1 liter\", \"stovetop compatibility\": \"gas\"}}], \"fulfillments\": [], \"status\": \"pending\", \"payment_history\": [{\"transaction_type\": \"payment\", \"amount\": 98.25, \"payment_method_id\": \"paypal_2433177\"}]}"
            }
          }
        }
      ]
    },
    {
      "role": "model",
      "parts": [
        {
          "functionCall": {
            "name": "think",
            "args": {
              "thought": "User daiki_johnson_9523 requests to cancel order #W1436802. Rules: must check order status. Order #W1436802 is 'pending' and can be cancelled. Need to confirm the order ID, and the reason ('no longer needed' or 'ordered by mistake'). User already stated order is no longer needed. After confirmation, refund will be sent via original PayPal payment in 5-7 business days. Next: Confirm all details and get explicit user confirmation to proceed."
            }
          }
        }
      ]
    },
    {
      "role": "user",
      "parts": [
        {
          "functionResponse": {
            "name": "think",
            "response": {
              "result": ""
            }
          }
        }
      ]
    },
    {
      "role": "model",
      "parts": [
        {
          "text": "Order #W1436802 for a Tea Kettle is currently pending and eligible for cancellation. The reason you've provided is \"no longer needed.\" If you confirm, the order will be cancelled and your payment of $98.25 will be refunded to your original PayPal payment method within 5–7 business days.\n\nWould you like me to proceed with the cancellation? Please reply 'yes' to confirm or 'no' to cancel this request."
        }
      ]
    },
    {
      "role": "user",
      "parts": [
        {
          "text": "Yes, please go ahead and cancel the order."
        }
      ]
    },
    {
      "role": "model",
      "parts": [
        {
          "functionCall": {
            "name": "cancel_pending_order",
            "args": {
              "order_id": "#W1436802",
              "reason": "no longer needed"
            }
          }
        }
      ]
    },
    {
      "role": "user",
      "parts": [
        {
          "functionResponse": {
            "name": "cancel_pending_order",
            "response": {
              "result": "{\"order_id\": \"#W1436802\", \"user_id\": \"daiki_johnson_9523\", \"address\": {\"address1\": \"939 Elm Street\", \"address2\": \"Suite 261\", \"city\": \"Jacksonville\", \"country\": \"USA\", \"state\": \"FL\", \"zip\": \"32273\"}, \"items\": [{\"name\": \"Tea Kettle\", \"product_id\": \"9832717871\", \"item_id\": \"3909406921\", \"price\": 98.25, \"options\": {\"material\": \"glass\", \"capacity\": \"1 liter\", \"stovetop compatibility\": \"gas\"}}], \"fulfillments\": [], \"status\": \"cancelled\", \"payment_history\": [{\"transaction_type\": \"payment\", \"amount\": 98.25, \"payment_method_id\": \"paypal_2433177\"}, {\"transaction_type\": \"refund\", \"amount\": 98.25, \"payment_method_id\": \"paypal_2433177\"}], \"cancel_reason\": \"no longer needed\"}"
            }
          }
        }
      ]
    },
    {
      "role": "model",
      "parts": [
        {
          "text": "Your order #W1436802 for the Tea Kettle has been cancelled. You will receive a refund of $98.25 to your original PayPal payment method within 5–7 business days.\n\nIf you need any further assistance, feel free to ask!"
        }
      ]
    },
    {
      "role": "user",
      "parts": [
        {
          "text": "Thank you for your help! I also need to modify another order."
        }
      ]
    },
    {
      "role": "model",
      "parts": [
        {
          "text": "I’m glad I could help with your previous request! Please provide the order ID of the order you’d like to modify and let me know what changes you wish to make (such as updating the shipping address, changing the payment method, or modifying product options). I’ll check the status and guide you through the process."
        }
      ]
    },
    {
      "role": "user",
      "parts": [
        {
          "text": "The order I want to modify is #W5282037. I’d like to change the garden hose's material and color."
        }
      ]
    },
    {
      "role": "model",
      "parts": [
        {
          "functionCall": {
            "name": "get_order_details",
            "args": {
              "order_id": "#W5282037"
            }
          }
        }
      ]
    },
    {
      "role": "user",
      "parts": [
        {
          "functionResponse": {
            "name": "get_order_details",
            "response": {
              "result": "{\"order_id\": \"#W5282037\", \"user_id\": \"daiki_johnson_9523\", \"address\": {\"address1\": \"834 Park Avenue\", \"address2\": \"Suite 947\", \"city\": \"Denver\", \"country\": \"USA\", \"state\": \"CO\", \"zip\": \"80273\"}, \"items\": [{\"name\": \"Garden Hose\", \"product_id\": \"6679515468\", \"item_id\": \"3230708338\", \"price\": 99.51, \"options\": {\"length\": \"25ft\", \"material\": \"latex\", \"color\": \"green\"}}, {\"name\": \"Makeup Kit\", \"product_id\": \"5149340237\", \"item_id\": \"6254646215\", \"price\": 248.85, \"options\": {\"skin tone\": \"dark\", \"kit size\": \"basic\", \"brand\": \"Brand B\"}}], \"fulfillments\": [], \"status\": \"pending\", \"payment_history\": [{\"transaction_type\": \"payment\", \"amount\": 348.36, \"payment_method_id\": \"paypal_2433177\"}]}"
            }
          }
        }
      ]
    },
    {
      "role": "model",
      "parts": [
        {
          "functionCall": {
            "name": "get_product_details",
            "args": {
              "product_id": "6679515468"
            }
          }
        }
      ]
    },
    {
      "role": "user",
      "parts": [
        {
          "functionResponse": {
            "name": "get_product_details",
            "response": {
              "result": "{\"name\": \"Garden Hose\", \"product_id\": \"6679515468\", \"variants\": {\"5753502325\": {\"item_id\": \"5753502325\", \"options\": {\"length\": \"25ft\", \"material\": \"rubber\", \"color\": \"green\"}, \"available\": false, \"price\": 96.35}, \"8249784860\": {\"item_id\": \"8249784860\", \"options\": {\"length\": \"50ft\", \"material\": \"vinyl\", \"color\": \"green\"}, \"available\": false, \"price\": 96.42}, \"8481719475\": {\"item_id\": \"8481719475\", \"options\": {\"length\": \"100ft\", \"material\": \"latex\", \"color\": \"blue\"}, \"available\": true, \"price\": 98.61}, \"9829827210\": {\"item_id\": \"9829827210\", \"options\": {\"length\": \"25ft\", \"material\": \"vinyl\", \"color\": \"blue\"}, \"available\": true, \"price\": 90.43}, \"1518544029\": {\"item_id\": \"1518544029\", \"options\": {\"length\": \"100ft\", \"material\": \"rubber\", \"color\": \"black\"}, \"available\": false, \"price\": 95.39}, \"3369928769\": {\"item_id\": \"3369928769\", \"options\": {\"length\": \"25ft\", \"material\": \"vinyl\", \"color\": \"green\"}, \"available\": true, \"price\": 97.35}, \"4024196380\": {\"item_id\": \"4024196380\", \"options\": {\"length\": \"50ft\", \"material\": \"latex\", \"color\": \"black\"}, \"available\": true, \"price\": 102.9}, \"4764314102\": {\"item_id\": \"4764314102\", \"options\": {\"length\": \"50ft\", \"material\": \"rubber\", \"color\": \"green\"}, \"available\": false, \"price\": 96.51}, \"3230708338\": {\"item_id\": \"3230708338\", \"options\": {\"length\": \"25ft\", \"material\": \"latex\", \"color\": \"green\"}, \"available\": true, \"price\": 99.51}, \"5206946487\": {\"item_id\": \"5206946487\", \"options\": {\"length\": \"50ft\", \"material\": \"vinyl\", \"color\": \"black\"}, \"available\": true, \"price\": 95.08}}}"
            }
          }
        }
      ]
    },
    {
      "role": "model",
      "parts": [
        {
          "functionCall": {
            "name": "get_user_details",
            "args": {
              "user_id": "daiki_johnson_9523"
            }
          }
        }
      ]
    },
    {
      "role": "user",
      "parts": [
        {
          "functionResponse": {
            "name": "get_user_details",
            "response": {
              "result": "{\"name\": {\"first_name\": \"Daiki\", \"last_name\": \"Johnson\"}, \"address\": {\"address1\": \"834 Park Avenue\", \"address2\": \"Suite 947\", \"city\": \"Denver\", \"country\": \"USA\", \"state\": \"CO\", \"zip\": \"80273\"}, \"email\": \"[email protected]\", \"payment_methods\": {\"paypal_2433177\": {\"source\": \"paypal\", \"id\": \"paypal_2433177\"}}, \"orders\": [\"#W1436802\", \"#W5282037\", \"#W9502127\"]}"
            }
          }
        }
      ]
    },
    {
      "role": "model",
      "parts": [
        {
          "functionCall": {
            "name": "think",
            "args": {
              "thought": "User wants to modify Garden Hose item in pending order #W5282037. Rules: Only one modification permitted, must collect all items to be changed. Only variants of the same product are allowed. Available options for Garden Hose are:\n- 8481719475: 100ft, latex, blue (available)\n- 9829827210: 25ft, vinyl, blue (available)\n- 3369928769: 25ft, vinyl, green (available)\n- 4024196380: 50ft, latex, black (available)\n- 5206946487: 50ft, vinyl, black (available)\nUser only has PayPal as payment method; any price difference will use it. Need to confirm which material and color the user wants for the replacement. Plan: Ask user for specific material and color desired, then confirm the exact variant."
            }
          }
        }
      ]
    },
    {
      "role": "user",
      "parts": [
        {
          "functionResponse": {
            "name": "think",
            "response": {
              "result": ""
            }
          }
        }
      ]
    },
    {
      "role": "model",
      "parts": [
        {
          "text": "For your pending order #W5282037, you can modify the Garden Hose to a different material and color. Here are the available options for a 25ft or other sizes:\n\n1. 25ft, vinyl, blue\n2. 25ft, vinyl, green\n3. 100ft, latex, blue\n4. 50ft, latex, black\n5. 50ft, vinyl, black\n\nPlease let me know which material, color, and length you would like to change your Garden Hose to, so I can proceed to confirm the details with you. If you wish to keep the length as 25ft, you can choose option 1 or 2."
        }
      ]
    },
    {
      "role": "user",
      "parts": [
        {
          "text": "I’d like to change it to 25ft, vinyl, blue."
        }
      ]
    },
    {
      "role": "model",
      "parts": [
        {
          "functionCall": {
            "name": "think",
            "args": {
              "thought": "User has chosen to modify the Garden Hose in order #W5282037 from 25ft, latex, green to 25ft, vinyl, blue (item_id 9829827210). This is an available option. Only one modification is permitted, so confirm the user wants no other item changes in this order. Payment method will be PayPal; the price for the new item is $90.43, less than the original $99.51, so a refund of the difference will be issued to PayPal. Need to confirm with the user before proceeding."
            }
          }
        }
      ]
    },
    {
      "role": "user",
      "parts": [
        {
          "functionResponse": {
            "name": "think",
            "response": {
              "result": ""
            }
          }
        }
      ]
    },
    {
      "role": "model",
      "parts": [
        {
          "text": "You’ve chosen to change your Garden Hose in order #W5282037 from 25ft, latex, green to 25ft, vinyl, blue. The new hose costs $90.43, which is less than your original selection ($99.51). You will receive a refund of the price difference on your PayPal account.\n\nPlease confirm:\n- This is the only item you want to modify in this order (no other changes).\n- You’d like to proceed with this modification and refund.\n\nReply “yes” to confirm and continue, or let me know if you want to change anything else."
        }
      ]
    },
    {
      "role": "user",
      "parts": [
        {
          "text": "Yes, I confirm the modification and refund."
        }
      ]
    },
    {
      "role": "model",
      "parts": [
        {
          "functionCall": {
            "name": "modify_pending_order_items",
            "args": {
              "order_id": "#W5282037",
              "item_ids": [
                "3230708338"
              ],
              "new_item_ids": [
                "9829827210"
              ],
              "payment_method_id": "paypal_2433177"
            }
          }
        }
      ]
    },
    {
      "role": "user",
      "parts": [
        {
          "functionResponse": {
            "name": "modify_pending_order_items",
            "response": {
              "result": "{\"order_id\": \"#W5282037\", \"user_id\": \"daiki_johnson_9523\", \"address\": {\"address1\": \"834 Park Avenue\", \"address2\": \"Suite 947\", \"city\": \"Denver\", \"country\": \"USA\", \"state\": \"CO\", \"zip\": \"80273\"}, \"items\": [{\"name\": \"Garden Hose\", \"product_id\": \"6679515468\", \"item_id\": \"9829827210\", \"price\": 90.43, \"options\": {\"length\": \"25ft\", \"material\": \"vinyl\", \"color\": \"blue\"}}, {\"name\": \"Makeup Kit\", \"product_id\": \"5149340237\", \"item_id\": \"6254646215\", \"price\": 248.85, \"options\": {\"skin tone\": \"dark\", \"kit size\": \"basic\", \"brand\": \"Brand B\"}}], \"fulfillments\": [], \"status\": \"pending (item modified)\", \"payment_history\": [{\"transaction_type\": \"payment\", \"amount\": 348.36, \"payment_method_id\": \"paypal_2433177\"}, {\"transaction_type\": \"refund\", \"amount\": 9.079999999999998, \"payment_method_id\": \"paypal_2433177\"}]}"
            }
          }
        }
      ]
    },
    {
      "role": "model",
      "parts": [
        {
          "text": "Your order #W5282037 has been successfully modified. The Garden Hose will now be 25ft, vinyl, blue. You will receive a refund of $9.08 (the price difference) to your PayPal account.\n\nIf you need help with anything else or want to check on another order, let me know!"
        }
      ]
    },
    {
      "role": "user",
      "parts": [
        {
          "text": "Thank you! That’s all I needed for now. I appreciate your help!"
        }
      ]
    },
    {
      "role": "model",
      "parts": [
        {
          "text": "You’re very welcome! I’m glad I could assist you with your orders today. If you need any help in the future, feel free to reach out. Have a great day!"
        }
      ]
    }
  ]
}

Evaluation data: 115 test tasks × 5 repetitions.

The persistent performance gap between the small models and the large model likely reflects the task’s inherent complexity, requiring multi-step reasoning, business rule adherence, and precise database state management. Our relatively limited training dataset of 332 demonstrations may be too small for such complex agentic tasks. Additionally, we used default fine-tuning settings for each provider, so additional hyperparameter tuning could yield further gains.

Airline Domain (out-of-distribution)

Task: Handle multi-turn customer service conversations for flight reservations including bookings, modifications, cancellations, and customer inquiries while following complex airline policies regarding baggage allowances, cabin restrictions, and payment rules.

Figure 5: Fine-tuned models maintain generally stable performance on unseen airline tasks, with mixed results across models. Some show modest improvements over their zero-shot counterparts while others regress. However, fine-tuned models consistently deliver faster response times, demonstrating the potential for cross-domain transfer of efficiency improvements.
τ-bench Airline Evaluation Details

Fine-tuning data:

Zero airline-specific data (models fine-tuned only on retail tasks)

Evaluation data:

50 test tasks × 5 repetitions

The airline domain presents an interesting pattern where some fine-tuned models (like Gemini 2.0 Flash and GPT-4.1 nano) show modest improvements over their zero-shot counterparts, while others (GPT-4.1 mini, GPT-4o mini, and Gemini 2.0 Flash-Lite) experience performance regression. This mixed behavior on out-of-distribution tasks represents a compelling area for future research into transfer learning dynamics and domain adaptation strategies.

Why This Matters for Production Applications

Fine-tuning offers compelling cost savings and potential performance improvements, but requires empirical validation for each use case. The technique is accessible and cost-effective enough to justify experimentation, and tools like TensorZero make it straightforward to determine whether it is right for you. Here are the key benefits fine-tuning can deliver:

1. Dramatic Cost Reduction with Strategic Flexibility

A 10-20x cost reduction can fundamentally transform LLM application economics. With TensorZero, you can start with an expensive, high-performance model and seamlessly transition to a less expensive fine-tuned alternative without any infrastructure changes. You can serve more users, handle more queries, and experiment more freely. Our experiments demonstrate success across multiple model families (GPT-4o mini, Gemini, Qwen) and fine-tuning platforms, reducing vendor lock-in risks while providing negotiating leverage and ensuring continuity if any single provider changes pricing or availability.

2. Improved User Experience Through Faster Response Times

Beyond cost savings, fine-tuned small models consistently deliver faster response times, improving user experience and enabling real-time applications. Our measurements show 2-4x faster response times across all tasks, with some models like Qwen3-8B achieving sub-200ms responses on navigation tasks. Faster responses reduce user wait times, enable more interactive applications, and can improve conversion rates in customer-facing scenarios.

3. Cost Effective Alternative to Inference-Time Optimization

While techniques like best-of-n sampling, chain-of-thought prompting, or dynamic in-context learning can improve model performance, they multiply inference costs by requiring multiple model calls, longer contexts, or longer completions. Fine-tuning requires a one-time investment — using OpenAI GPT-4.1 mini, fine-tuning costs ranged from $2.30 (BabyAI) to $32 (τ-bench) — but then delivers performance improvements through single, fast inference calls. This upfront investment can pay for itself in hundreds of conversations or fewer.

4. Flexible Deployment Strategies

Fine-tuned models can enable sophisticated routing architectures:

Practical Implementation Guide

Ideal scenarios for fine-tuning:

Key considerations and limitations:

While fine-tuning offers compelling advantages, several factors warrant careful consideration. The approach requires initial data collection using expensive large models, but the upfront investment is recoverable through reduced inference expenses. Success depends on clearly defining task-specific success criteria, whether through automated metrics or human evaluation. Ongoing quality assurance becomes essential to detect performance drift over time, particularly as input distributions evolve. The benefits may not transfer perfectly to adjacent use cases, requiring robust evaluation and monitoring. Additionally, different model families exhibit varying fine-tuning characteristics, making provider selection an important strategic decision. While providers like OpenAI and Google Vertex automatically determine hyperparameters for most use cases, more niche or complex tasks may require manual tuning for optimal results.

Implementation roadmap with TensorZero:

TensorZero provides an MLOps platform for systematic fine-tuning optimization:

  1. Deploy observability: Install TensorZero Gateway as a drop-in replacement for your LLM API calls to identify high-cost, high-volume operations with <1ms latency overhead.

  2. Collect quality data: Configure feedback collection to capture 100-1000 successful conversations from expensive models, automatically stored in ClickHouse for easy fine-tuning export.

  3. Train and evaluate: Use multi-provider integrations to fine-tune multiple small models simultaneously, then run TensorZero Evaluations to identify the best performer.

    Example: Fine-tune Gemini 2.0 Flash-Lite on τ-bench Retail
    import os
    from time import sleep
    from tensorzero import (
      TensorZeroGateway,
      FloatMetricFilter,
      GCPVertexGeminiSFTConfig,
      OptimizationJobStatus,
    )
    

    CONFIG_PATH = "<your-config-path>/config/tensorzero.toml"
    # Function to optimize, e.g. "tau_bench_retail:act"
    FUNCTION_NAME = "tau_bench_retail:act"
    # Large model to get demonstrations from, e.g. "GPT-4.1-think-tool"
    VARIANT_NAME = "GPT-4.1-think-tool"
    # Model to fine-tune, e.g. "gemini-2.0-flash-lite-001"
    FT_MODEL = "gemini-2.0-flash-lite-001"
    # Bucket to store fine-tuned models, e.g. "gs://your-bucket-name"
    BUCKET_NAME = "gs://your-bucket-name"
    # Region to store fine-tuned models, e.g. "us-central1"
    REGION = "us-central1"
    # Project ID to store fine-tuned models, e.g. "your-project-id"
    PROJECT_ID = "your-project-id"
    # URL to ClickHouse database that stores your inferences
    CLICKHOUSE_URL = "<your-clickhouse-url>"
    

    # Build a TensorZero client
    client = TensorZeroGateway.build_embedded(
        config_file=CONFIG_PATH,
        clickhouse_url=CLICKHOUSE_URL,
    )
    

    # Filter for high-quality conversations
    filters = FloatMetricFilter(
        metric_name="reward",
        value=0.5,
        comparison_operator=">",
    )
    

    # List high-quality conversations
    inferences = client.experimental_list_inferences(
        function_name=FUNCTION_NAME,
        variant_name=VARIANT_NAME,
        filters=filters,
        output_source="inference",
    )
    

    # Render the conversations for fine-tuning
    rendered_samples = client.experimental_render_samples(
        stored_samples=inferences,
        variants={FUNCTION_NAME: VARIANT_NAME},
    )
    

    # Configure the fine-tuning job
    optimization_config = GCPVertexGeminiSFTConfig(
        model=FT_MODEL,
        bucket_name=BUCKET_NAME,
        region=REGION,
        project_id=PROJECT_ID,
    )
    

    # Launch the fine-tuning job
    optimization_job_handle = client.experimental_launch_optimization(
        train_samples=rendered_samples,
        val_samples=None,  # Optionally, create a validation split
        optimization_config=optimization_config,
    )
    

    # Poll the fine-tuning job until it completes
    while True:
        job_info = client.experimental_poll_optimization(
            job_handle=optimization_job_handle
        )
        if job_info.status == OptimizationJobStatus.Completed:
            break
        sleep(30)  # Poll every 30 seconds for production jobs

  4. Deploy gradually: Leverage the experimentation framework to route traffic from expensive to fine-tuned models, starting with 5% allocation and scaling based on performance metrics.

The Path Forward

Fine-tuning is reshaping the traditional performance-cost tradeoff in LLM applications. By leveraging the knowledge distilled in large models to train specialized small models, we can achieve the best of both worlds: enterprise-grade performance at startup-friendly costs.

As AI becomes the foundation of more software applications, the teams that master cost-effective deployment will build sustainable competitive advantages. The gap between prototype and production-ready AI applications is narrowing, but only for those who optimize intelligently.

Don’t just scale up. Scale smart.

Start building today. Check out our GitHub and Quick Start .