You can use TensorZero to build virtually any application powered by LLMs. This tutorial shows it’s easy to set up an LLM application with TensorZero. We’ll build a few different applications to showcase the flexibility of TensorZero: a simple chatbot, an email copilot, a weather RAG system, and a structured data extraction pipeline.
See our Quickstart to learn how to set up our LLM gateway, observability, and fine-tuning — in just 5 minutes.
You can find the code behind this tutorial and instructions on how to run it on GitHub.Reach out on Slack or Discord if you have any questions. We’d be happy to help!

​
Part I — Simple Chatbot

We’ll start by building a vanilla LLM-powered chatbot, and build up to more complex applications from there.

​
Functions

A TensorZero Function is an abstract mapping from input variables to output variables. As you onboard to TensorZero, a function should replace each prompt in your system. At a high level, a function will template the inputs to generate a prompt, make an LLM inference call, and return the results. This mapping can be achieved with various choices of model, prompt, decoding strategy, and more; each such combination is called a variant, which we’ll discuss below. For our simple chatbot, we’ll set up a function that maps the chat history to a new chat message. We define functions in the tensorzero.toml configuration file. The configuration file is written in TOML, which is a simple configuration language.
Read more about the TOML syntax in the TOML documentation.
The following configuration entry shows the skeleton of a function. A function has an arbitrary name, a type, and other fields that depend on the type.
tensorzero.toml
[functions.my_function_name]
type = "..."
# ... the other fields in this section depend on the function type ...
TensorZero currently supports two types of functions: chat functions, which match the typical chat interface you’d expect from an LLM API, and json functions, which are optimized for generating structured outputs. We’ll start with a chat function for this example, and later we’ll see how to use json functions. A chat function takes a chat message history and returns a chat message. It doesn’t have any required fields (but many optional). Let’s call our function mischievous_chatbot and set its type to chat. We’ll ignore the optional fields for now. To include these changes, our tensorzero.toml file should include the following:
tensorzero.toml
[functions.mischievous_chatbot]
type = "chat"
That’s all we need to do to define our function. Later on, we’ll add more advanced features to our functions, like schemas and templates, which unlock new capabilities for model optimization and observability. But we don’t need any of that to get started. The implementation details of this function are defined in its variants. But before we can define a variant, we need to set up a model and a model provider.

​
Models and Model Providers

Before setting up your first TensorZero variant, you’ll need a model with a model provider. A model specifies a particular LLM (e.g. GPT-4o or your fine-tuned Llama 3), and model providers specify the different ways you can access a given model (e.g. GPT-4o is available through both OpenAI and Azure). A model has an arbitrary name and a list of providers. Let’s start with a single provider for our model. A provider has an arbitrary name, a type, and other fields that depend on the provider type. The skeleton of a model and its provider looks like this:
tensorzero.toml
[models.my_model_name]
routing = ["my_provider_name"]

[models.my_model_name.providers.my_provider_name]
type = "..."
# ... the other fields in this section depend on the provider type ...
For this example, we’ll use the GPT-4o mini model from OpenAI. Let’s call our model my_gpt_4o_mini and our provider my_openai_provider with type openai. The only required field for the openai provider is model_name. It’s a best practice to pin the model to a specific version to avoid breaking changes, so we’ll use gpt-4o-mini-2024-07-18.
TensorZero supports proprietary models (e.g. OpenAI, Anthropic), inference services (e.g. Fireworks AI, Together AI), and self-hosted LLMs (e.g. vLLM), including your own fine-tuned models on each of these.See Integrations and Configuration Reference for more details.
Once we’ve added these values, our tensorzero.toml file should include the following:
tensorzero.toml
[models.my_gpt_4o_mini]
routing = ["my_openai_provider"]

[models.my_gpt_4o_mini.providers.my_openai_provider]
type = "openai"
model_name = "gpt-4o-mini-2024-07-18"
You can add multiple providers for the same model to enable fallbacks. The gateway will try each provider in the routing field in order until one succeeds. This is helpful to mitigate the impact of provider downtime and rate limiting.

​
Variants

Now that we have a model and a provider configured, we can create a variant for our mischievous_chatbot function. A variant is a particular implementation of a function. In practice, a variant might specify the particular model, prompt templates, a decoding strategy, hyperparameters, and other settings used for inference. A variant’s definition includes an arbitrary name, a type, a weight, and other fields that depend on the type. The skeleton of a TensorZero variant looks like this:
tensorzero.toml
[functions.my_function_name.variants.my_variant_name]
type = "..."
weight = X
# ... the other fields in this section depend on the variant type ...
We’ll call this variant gpt_4o_mini_variant. The simplest variant type is chat_completion, which is the typical chat completion format used by OpenAI and many other LLM providers.
TensorZero supports other variant types which implement inference-time optimizations. See Configuration Reference for more details on variant types and their configuration options.
The weight field is used to determine the probability of this variant being chosen. Since we only have one variant, we’ll give it a weight of 1.0. We’ll dive deeper into variant weights in a later section. The only required field for a chat_completion variant is model. This must be a model in the configuration file. We’ll use the my_gpt_4o_mini model we defined earlier. After filling in the fields for this variant, our tensorzero.toml file should include the following:
tensorzero.toml
[functions.mischievous_chatbot.variants.gpt_4o_mini_variant]
type = "chat_completion"
weight = 1.0
model = "my_gpt_4o_mini"
If you don’t require advanced functionality for model providers (e.g. Retries & Fallbacks), you don’t have to define model configuration blocks. TensorZero supports short-hand model names like openai::gpt-4o-mini or anthropic::claude-3-5-haiku in a variant’s model field. See Configuration Reference for more details.

​
Inference API Requests

There’s a lot more to TensorZero than what we’ve covered so far, but this is everything we need to get started! If you launch the TensorZero Gateway with this configuration file, the mischievous_chatbot function will be available on the /inference endpoint. Let’s make a request to this endpoint.
You can install the TensorZero Python client with:
pip install tensorzero
Then, you can make a TensorZero API call with:
POST /inference
from tensorzero import TensorZeroGateway

with TensorZeroGateway.build_http(gateway_url="http://localhost:3000") as client:
    result = client.inference(
        function_name="mischievous_chatbot",
        input={
            "system": "You are a friendly but mischievous AI assistant.",
            "messages": [
                {"role": "user", "content": "What is the capital of Japan?"},
            ],
        },
    )

print(result)
The TensorZero Gateway also supports streaming inference. See the API Reference for more details.
That’s it! You’ve now made your first inference call with TensorZero. But if that’s all you need, you probably don’t need TensorZero. So let’s make things more interesting.
Earlier we mentioned that you can add multiple providers for the same model to enable model fallbacks. TensorZero additionally supports variant fallbacks. The gateway first tries to fallback to a different provider for the same model. If all providers for a variant are unavailable, the gateway will keep re-sampling variants (without replacement) until one succeeds.

​
Part II — Email Copilot

Next, let’s build an LLM-powered copilot for drafting emails. We’ll use this opportunity to show off more of TensorZero’s features.

​
Templates

In the previous example, we provided a system prompt on every request. Unless the system prompt completely changes between requests, this is not ideal for production applications. Instead, we can use a system template. Using a template allows you to update the prompt without client-side changes. Later, we’ll see how to parametrize templates with schemas and run robust prompt experiments with multiple variants. In particular, setting up schemas will materially help you optimize your models robustly down the road. Let’s start with a simple system template. For this example, the system template is static, so you won’t need a schema. TensorZero uses MiniJinja for templating. Since we’re not using any variables, however, we don’t need any special syntax.
Read more about MiniJinja syntax in the MiniJinja documentation. MiniJinja is similar to Jinja2 but there are a few differences. See their compatibility guide for more details.MiniJinja also provides a browser playground where you can test your templates.
We’ll create the following template:
system.minijinja
You are a helpful AI assistant that drafts emails.
Adopt a friendly "business casual" tone.
Respond with just an email body.

Example:

Dear recipient,

I'm reaching out to ...

Best,
Sender

​
Schemas

The system template for this example is static, but often you’ll want to parametrize the prompts. When you define a template with parameters, you need to define a corresponding JSON Schema. The schema defines the structure of the input for that prompt. With it, the gateway can validate the input before running the inference, and later, we’ll see how to use it for robust model optimization. For our email copilot’s user prompt, we’ll want to parametrize the template with three string fields: recipient_name, sender_name, and email_purpose. We want all fields to be required and don’t want any additional fields.
Ask your favorite LLM to generate the schema for you.Claude generated the schema for this example using this request:
Create a JSON schema with the following fields: `recipient_name`, `sender_name`, and `email_purpose`. All fields are required. No additional fields are allowed.
user_schema.json
{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "properties": {
    "recipient_name": {
      "type": "string"
    },
    "sender_name": {
      "type": "string"
    },
    "email_purpose": {
      "type": "string"
    }
  },
  "required": ["recipient_name", "sender_name", "email_purpose"],
  "additionalProperties": false
}
You can export JSON Schemas from Pydantic models and Zod schemas.
With a schema in place, we can create a parameterized template.
user.minijinja
Please draft an email using the following information:

- Recipient Name: {{ recipient_name }}
- Sender Name: {{ sender_name }}
- Email Purpose: {{ email_purpose }}

​
Functions with Templates and Schemas

Let’s finally create our function and variant for the email copilot.
Schemas belong to functions and templates belong to variants. Think of this like a function signature vs. method implementation when programming.The same schema can be used by multiple templates, but the schema itself should not change over time. We recommend simply copying the function and renaming it if you want to change the signature if you’ve already used it in production. Our roadmap includes better support for schema versioning and migrations.
The configuration file is similar to previous example, but we’ve added a user_schema field to the function and system_template and user_template fields to the variant.
tensorzero.toml
[functions.draft_email]
type = "chat"
user_schema = "functions/draft_email/user_schema.json"

[functions.draft_email.variants.gpt_4o_mini_email_variant]
type = "chat_completion"
weight = 1.0
model = "my_gpt_4o_mini"
system_template = "functions/draft_email/gpt_4o_mini_email_variant/system.minijinja"
user_template = "functions/draft_email/gpt_4o_mini_email_variant/user.minijinja"

- functions/
  - draft_email/
    - gpt_4o_mini_email_variant/
      - system.minijinja
      - user.minijinja
    - user_schema.json
- tensorzero.toml
Restart your gateway using the new configuration file, and you’re ready to go! Let’s make an inference request with our new function. Now we don’t need to provide the system prompt every time, and the user message is a structured object instead of a free-form string. Note that each inference returns an inference_id and an episode_id, which we’ll use later to associate feedback with inferences.
POST /inference
from tensorzero import TensorZeroGateway

with TensorZeroGateway.build_http(gateway_url="http://localhost:3000") as client:
    inference_result = client.inference(
        function_name="draft_email",
        input={
            "messages": [
                {
                    "role": "user",
                    "content": [
                        {
                            "type": "text",
                            "arguments": {
                                "recipient_name": "TensorZero Team",
                                "sender_name": "Mark Zuckerberg",
                                "email_purpose": "Acquire TensorZero for $100 billion dollars.",
                            },
                        }
                    ],
                }
            ],
        },
    )

    # If everything is working correctly, the `variant_name` field should change depending on the request
    print(inference_result)
You can find the full code to reproduce this example on GitHub.
Why did we bother with all this? Now you’re collecting structured inference data, which is incredibly valuable for observability and especially for optimization. For example, if you eventually decide to fine-tune your model, you’ll easily be able to counterfactually swap new prompts into your training data before fine-tuning, instead of being stuck with the prompts that were actually used at inference time.

​
Inference-Level Metrics

The TensorZero Gateway allows you to assign feedback to inferences or sequences of inferences by defining metrics. Metrics encapsulate the downstream outcomes of your LLM application, and drive the experimentation and optimization workflows in TensorZero. This example covers metrics that apply to individual inference requests. Later, we’ll show how to define metrics that apply to sequences of inferences (which we call episodes). The skeleton of a metric looks like the following configuration entry.
tensorzero.toml
[metrics.my_metric_name]
level = "..."
optimize = "..."
type = "..."
Let’s say we want to optimize for the number of email drafts that are accepted. Let’s call this metric email_draft_accepted. We should use a metric of type boolean to capture this behavior since we’re optimizing for a binary outcome: whether the email draft is accepted or not.
We currently support the following Metric types:
Metric TypeDescriptionExamples
Boolean MetricA boolean indicating successThumbs up; task success
Float MetricA number to be optimizedMistakes; interactions; resources used
CommentNatural-language feedbackFeedback from users or developers
DemonstrationExample of desired outputEdited drafts; labels; human-generated content
See the Configuration Reference for more details about how to configure metrics.
The metric applies to individual inference requests, so we’ll set level = "inference". And finally, we’ll set optimize = "max" because we want to maximize this metric. Our metric configuration should look like this:
tensorzero.toml
[metrics.email_draft_accepted]
type = "boolean"
optimize = "max"
level = "inference"

​
Feedback API Requests

As our application collects usage data, we can use the /feedback endpoint to keep track of this metric. Make sure to restart your gateway after adding the metric configuration. Previously, we saw that every time you call /inference, the Gateway will return an inference_id field in the response. You’ll want to substitute this inference_id into the command below.
POST /feedback
from tensorzero import TensorZeroGateway

with TensorZeroGateway.build_http(gateway_url="http://localhost:3000") as client:
    feedback_result = client.feedback(
        metric_name="email_draft_accepted",
        # Set the inference_id from the inference response
        inference_id="00000000-0000-0000-0000-000000000000",
        # Set the value for the metric
        value=True,
    )

    print(feedback_result)
Over time, we’ll collect the perfect dataset for observability and optimization. We’ll have structured data on every inference, as well as their associated feedback. The TensorZero Recipes leverage this data for prompt and model optimization.

​
Experimentation

So far, we’ve only used one variant of our function. In practice, you’ll want to experiment with different configurations — for example, different prompts, models, and parameters. TensorZero makes this easy with built-in experimentation features. You can define multiple variants of a function, and the gateway will sample from them at inference time.
For now you must manage variant weights yourself, but we’re planning to release an asynchronous multi-armed bandit algorithm we’ve implemented for robust automated experimentation.
For this example, let’s set up a variant that uses Anthropic’s Claude 3 Haiku instead of GPT-4o Mini. Additionally, this variant will introduce a custom temperature parameter to control the creativity of the AI assistant. Let’s start by adding a new model and provider.
tensorzero.toml
[models.my_haiku_3]
routing = ["my_anthropic_provider"]

[models.my_haiku_3.providers.my_anthropic_provider]
type = "anthropic"
model_name = "claude-3-haiku-20240307"
Finally, create a new variant using this model, and update the weight of the previous variant to match.
tensorzero.toml
[functions.draft_email.variants.gpt_4o_mini_email_variant]
type = "chat_completion"
weight = 0.7 # sample this variant 70% of the time
model = "my_gpt_4o_mini"
system_template = "functions/draft_email/gpt_4o_mini_email_variant/system.minijinja"
user_template = "functions/draft_email/gpt_4o_mini_email_variant/user.minijinja"

[functions.draft_email.variants.haiku_3_email_variant]
type = "chat_completion"
weight = 0.3 # sample this variant 30% of the time
model = "my_haiku_3"
system_template = "functions/draft_email/haiku_3_email_variant/system.minijinja"
user_template = "functions/draft_email/haiku_3_email_variant/user.minijinja"
temperature = 0.9
Weights don’t have to add up to 1.0. In such a case, the gateway will normalize the weights and sample accordingly.
You could also experiment with different prompt templates, but for this example we’ll keep things simple and just copy the previous templates over. Once you’re done, your file tree should look like this: 
- functions/
  - draft_email/
    - gpt_4o_mini_email_variant/
      - system.minijinja
      - user.minijinja
    - haiku_3_email_variant/
      - system.minijinja copy from above
      - user.minijinja copy from above
    - user_schema.json
  - ...
- tensorzero.toml
That’s it! After restarting the gateway, you can make some inference requests to see the results. The gateway will sample between the two variants based on the configured weights.

​
Part III — Weather RAG

The next example introduces tool use into the mix.
Some providers call this feature “Function Calling”. But don’t confuse it with TensorZero Functions — those are completely different concepts!
In particular, we’ll show how to use TensorZero in a RAG (retrieval-augmented generation) system. TensorZero doesn’t handle the indexing and retrieval directly, but can help with auxiliary generative tasks like query and response generation.
You can also use TensorZero to manage more complex RAG workflows. We’ll soon release an example featuring an agentic workflow with multi-hop retrieval and reasoning.
For this example, we’ll illustrate a RAG system with a simple weather tool. We’ll introduce a function for query generation (generate_weather_query), and another for response generation (generate_weather_report). The former will leverage tool (get_temperature) use to generate a weather query. Here we mock the weather API, but it’ll be easy to see how diverse RAG workflows can be integrated.

​
Tools

TensorZero has first-class support for tools. You can define a tool in your configuration file, and attach it to a function that should be allowed to call it. Let’s start by defining a tool. A tool has a name, a description, and a set of parameters (described with a JSON schema). The skeleton of a tool configuration looks like this:
tensorzero.toml
[tools.my_tool_name]
description = "..."
parameters = "..."
Let’s create a tool for a fictional weather API that takes a location (and optionally units) and returns the current temperature. The parameters for this tool are defined as a JSON schema. We’ll need two parameters: location (string) and units (enum with values fahrenheit and celsius). Only location is required, no additional properties should be allowed. Finally, we’ll add descriptions for each parameter and tool itself — this is very important to increase the quality of tool use!
get_temperature.json
{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "description": "Get the current temperature for a given location.",
  "properties": {
    "location": {
      "type": "string",
      "description": "The location to get the temperature for (e.g. \"New York\")"
    },
    "units": {
      "type": "string",
      "description": "The units to get the temperature in (must be \"fahrenheit\" or \"celsius\"). Defaults to \"fahrenheit\".",
      "enum": ["fahrenheit", "celsius"]
    }
  },
  "required": ["location"],
  "additionalProperties": false
}
tensorzero.toml
[tools.get_temperature]
description = "Get the current temperature for a given location."
parameters = "tools/get_temperature.json"
The file organization is up to you, but we recommend the following structure: 
- functions/
  - ...
- tools/
  - get_temperature.json
- tensorzero.toml

​
Functions with Tool Use

We can now create our two functions. The query generation function will use the tool we just defined, and the response generation function will be similar to our previous examples. Let’s define the functions, their variants, and any associated templates and schemas.
tensorzero.toml
[functions.generate_weather_query]
type = "chat"
tools = ["get_temperature"]

[functions.generate_weather_query.variants.simple_variant]
type = "chat_completion"
weight = 1.0
model = "my_gpt_4o_mini"
system_template = "functions/generate_weather_query/simple_variant/system.minijinja"

[functions.generate_weather_report]
type = "chat"
user_schema = "functions/generate_weather_report/user_schema.json"

[functions.generate_weather_report.variants.simple_variant]
type = "chat_completion"
weight = 1.0
model = "my_gpt_4o_mini"
system_template = "functions/generate_weather_report/simple_variant/system.minijinja"
user_template = "functions/generate_weather_report/simple_variant/user.minijinja"
Both functions have a variant called simple_variant, but those are separate variants. A variant is always specific to a function. Multiple variants, however, can share the same model.
functions/generate_weather_query/simple_variant/system.minijinja
You are a helpful AI assistant that generates weather queries.

If the user asks about the weather in a given location, request a tool call to `get_temperature` with the location.
Optionally, the user may also specify the units (must be "fahrenheit" or "celsius"; defaults to "fahrenheit").

If the user asks about anything else, just respond that you can't help.

---

Examples:

User: What's the weather in New York?
Assistant (Tool Call): get_temperature(location="New York")

User: What's the weather in Tokyo in Celsius?
Assistant (Tool Call): get_temperature(location="Tokyo", units="celsius")

User: What is the capital of France?
Assistant (Text): I can only provide weather information.
functions/generate_weather_report/simple_variant/system.minijinja
You are a helpful AI assistant that generates brief weather reports.

You'll be provided with the temperature for a given location.
Respond with a concise weather report for the given temperature.
Add a sentence with a funny local recommendation based on the information.

If "Units" is missing, assume the temperature is in Fahrenheit.

---

Examples:

User:
  Location: San Francisco
  Temperature: 82
  Units:
Assistant:
  The weather in San Francisco is 82°F.
  Hope you get a chance to enjoy La Taqueria by Dolores Park!

User:
  Location: Tokyo
  Temperature: -5
  Units: celsius
Assistant:
  The weather in Tokyo is -5°C — perfect a day for a trip to an onsen.
functions/generate_weather_report/user_schema.json
{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "properties": {
    "location": {
      "type": "string"
    },
    "temperature": {
      "type": "string"
    },
    "units": {
      "type": ["null", "string"]
    }
  },
  "required": ["location", "temperature"],
  "additionalProperties": false
}
functions/generate_weather_report/simple_variant/user.minijinja
Please respond with a weather report given the information below.

Location: {{ location }}
Temperature: {{ temperature }}
Units: {{ units }}
TensorZero also supports multi-turn tool use, parallel tool calls, tool choice strategies, dynamic tool definition, and more. See the Configuration Reference for more information.Notably, another approach to our weather RAG example is to use a single function for both query and response generation (i.e. multi-turn tool use). As an exercise, why don’t you try implementing it? See Chat Function with Multi-Turn Tool Use for an example.
When the model requests a tool call, the response will include a tool_call content block. These content blocks have the fields arguments, name, raw_arguments, and raw_name. The first two fields are validated against the tool’s configuration (or null if invalid). The last two fields contain the raw values passed received from the model.

​
Episodes

Before we make any inference requests, we must introduce one more concept: episodes. An episode is a sequence of inferences associated with a common downstream outcome. For example, an episode could refer to a sequence of LLM calls associated with:
  • Resolving a support ticket
  • Preparing an insurance claim
  • Completing a phone call
  • Extracting data from a document
  • Drafting an email
An episode will include one or more functions, and sometimes multiple calls to the same function. Your application can run arbitrary actions (e.g. interact with users, retrieve documents, actuate robotics) between function calls within an episode. Though these are outside the scope of TensorZero, it is fine (and encouraged) to build your LLM systems this way. Episodes allow you to group sequences of inferences in multi-step LLM workflows, apply feedback to these sequences as a whole, and optimize your workflows end-to-end. During an episode, multiple calls to the same function will receive the same variant (unless fallbacks are necessary). In the case of our weather RAG, both query generation and response generation contribute to what we ultimately care about: the weather report. So we’ll want to associate each weather report with the inferences that led to it. The workflow of generating a weather report will be our episode. The /inference endpoint accepts an optional episode_id field. When you make the first inference request, you don’t have to provide an episode_id. The gateway will create a new episode for you and return the episode_id in the response. When you make the second inference request, you must provide the episode_id you received in the first response. The gateway will use the episode_id to associate the two inference requests together.
You shouldn’t generate episode IDs yourself. The gateway will create a new episode ID for you if you don’t provide one. Then, you can use it with other inferences you’d like to associate with the episode.
POST /inference
from tensorzero import TensorZeroGateway, ToolCall

with TensorZeroGateway.build_http(gateway_url="http://localhost:3000") as client:
    query_result = client.inference(
        function_name="generate_weather_query",
        # This is the first inference request in an episode so we don't need to provide an episode_id
        input={
            "messages": [
                {
                    "role": "user",
                    "content": "What is the weather like in São Paulo?",
                }
            ]
        },
    )

    print(query_result)

    # In a production setting, you'd validate the output more thoroughly
    assert len(query_result.content) == 1
    assert isinstance(query_result.content[0], ToolCall)

    location = query_result.content[0].arguments.get("location")
    units = query_result.content[0].arguments.get("units", "celsius")

    # Now we pretend to make a tool call (e.g. to an API)
    temperature = "35"

    report_result = client.inference(
        function_name="generate_weather_report",
        # This is the second inference request in an episode so we need to provide the episode_id
        episode_id=query_result.episode_id,
        input={
            "messages": [
                {
                    "role": "user",
                    "content": [
                        {
                            "type": "text",
                            "arguments": {
                                "location": location,
                                "temperature": temperature,
                                "units": units,
                            },
                        }
                    ],
                }
            ]
        },
    )

    print(report_result)

​
Episode-Level Metrics

The primary use case for episodes is to enable episode-level metrics. In the previous example, we assigned feedback to individual inferences. TensorZero can also collect episode-level feedback, which can be useful for optimizing entire workflows. To collect episode-level feedback, we need to define a metric with level = "episode". Let’s add a metric for the weather RAG example. We’ll use the user_rating as the metric name, and we’ll collect it as a float.
tensorzero.toml
[metrics.user_rating]
level = "episode"
optimize = "max"
type = "float"
Making a feedback request with an episode_id instead of an inference_id associates the feedback with the entire episode.
POST /feedback
from tensorzero import TensorZeroGateway, ToolCall

with TensorZeroGateway.build_http(gateway_url="http://localhost:3000") as client:
    feedback_result = client.feedback(
        metric_name="user_rating",
        # Set the episode_id to the one returned in the inference response
        episode_id="00000000-0000-0000-0000-000000000000",
        # Set the value for the metric (numeric types will be coerced to float)
        value=5,
    )

    print(feedback_result)
That’s all you need for our weather RAG example. This is clearly a toy example, but it illustrates the core concepts of TensorZero. You can replace the mock weather API with a real API call — or if were searching over documents instead, anything from BM25 to cutting-edge vector search.

​
Part IV — Email Data Extraction

​
JSON Functions

Everything we’ve done so far has been with Chat Functions. TensorZero also supports JSON Functions for use cases that require structured outputs. The input is the same, but the function returns a JSON value instead of a chat message.
Depending on the use case, both Chat Functions with Tool Use and JSON Functions can be used. In fact, the TensorZero Gateway will sometimes convert between the two under the hood for model providers that don’t support one of them natively.As a rule of thumb, we typically recommend using JSON Functions if you have a single, well-defined output schema. If you need more flexibility (e.g. letting the model pick between multiple tools, or whether to pick a tool at all), then Chat Functions with Tool Use is likely a better fit. That said, try experimenting with both and see which one works best for your use case!
Let’s create a JSON function that extracts an email address from a user’s message. The setup is very similar to our previous examples, except that the function is defined with type = "json" and requires an output_schema. Let’s start by defining the schema, a static system template, and the rest of the configuration.
output_schema.json
{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "properties": {
    "email": {
      "type": "string",
      "description": "The email address extracted from the user's message."
    }
  },
  "required": ["email"],
  "additionalProperties": false
}
functions/extract_email/simple_variant/system.minijinja
You are a helpful AI assistant that extracts an email address from a user's message.
Return an empty string if no email address is found.

Your output should be a JSON object with the following schema:

{
  "email": "..."
}

---

Examples:

User: Using TensorZero at work? Ping [email protected] to set up a Slack channel (free).
Assistant: {"email": "[email protected]"}

User: I just received an ominous email from [email protected]...
Assistant: {"email": "[email protected]"}

User: Let's sue TensorZero!
Assistant: {"email": ""}
tensorzero.toml
[functions.extract_email]
type = "json"
output_schema = "functions/extract_email/output_schema.json"

[functions.extract_email.variants.simple_variant]
type = "chat_completion"
weight = 1
model = "my_gpt_4o_mini"
system_template = "functions/extract_email/simple_variant/system.minijinja"
Like with Chat Functions, you can define multiple variants of a JSON function. There are additional parameters (e.g. json_mode) that you can use to control the behavior of these variants.
Once you’ve set it up, your file tree should look like this: 
- functions/
  - ...
  - extract_email/
    - simple_variant/
      - system.minijinja
      - user.minijinja
    - output_schema.json
  - ...
- ...
- tensorzero.toml
Finally, let’s make an inference request. The request format is very similar to chat functions, but the response will contain an output field instead of a content field. The output field will be a JSON object with the fields parsed and raw. The parsed field is the parsed output as a valid JSON that fits your schema (null if the model didn’t generate a JSON that matches your schema), and the raw field is the raw output from the model as a string.
POST /inference
from tensorzero import TensorZeroGateway


with TensorZeroGateway.build_http(gateway_url="http://localhost:3000") as client:
    result = client.inference(
        function_name="extract_email",
        input={
            "messages": [
                {
                    "role": "user",
                    "content": "blah blah blah [email protected] blah blah blah",
                }
            ]
        },
    )

    print(result)

​
Conclusion

This tutorial only scratches the surface of what you can do with TensorZero. TensorZero especially shines when it comes to optimizing complex LLM workflows using the data collected by the gateway. For example, the structured data collected by the gateway can be used to better fine-tune models compared to using historical prompts and generations alone. We are working on a series of examples covering the entire “data flywheel in a box” that TensorZero provides. Here are some of our favorites:
We’re working on many more examples, especially for advanced use cases. Stay tuned!
You can also dive deeper into the Configuration Reference and the API Reference. We’re excited to see what you build! Please share your projects, ideas, and feedback with us on Slack or Discord.
Exploring TensorZero at work?We’d be happy to set up a Slack or Teams Connect channel with your team (free). Email us at [email protected].