An episode is a sequence of inferences associated with a common downstream outcome. For example, an episode could refer to a sequence of LLM calls associated with:
  • Resolving a support ticket
  • Preparing an insurance claim
  • Completing a phone call
  • Extracting data from a document
  • Drafting an email
An episode will include one or more functions, and sometimes multiple calls to the same function. Your application can run arbitrary actions (e.g. interact with users, retrieve documents, actuate robotics) between function calls within an episode. Though these are outside the scope of TensorZero, it is fine (and encouraged) to build your LLM systems this way. The /inference endpoint accepts an optional episode_id field. When you make the first inference request, you don’t have to provide an episode_id. The gateway will create a new episode for you and return the episode_id in the response. When you make the second inference request, you must provide the episode_id you received in the first response. The gateway will use the episode_id to associate the two inference requests together.
You shouldn’t generate episode IDs yourself. The gateway will create a new episode ID for you if you don’t provide one. Then, you can use it with other inferences you’d like to associate with the episode.
You can also find the runnable code for this example on GitHub.

​
Scenario

In the Quickstart, we built a simple LLM application that writes haikus about artificial intelligence. Imagine we want to separately generate some commentary about the haiku, and present both pieces of content to users. We can associate both inferences with the same episode. Let’s define an additional function in our configuration file.
tensorzero.toml
[functions.analyze_haiku]
type = "chat"

[functions.analyze_haiku.variants.gpt_4o_mini]
type = "chat_completion"
model = "gpt_4o_mini"

​
Inferences & Episodes

This time, we’ll create a multi-step workflow that first generates a haiku and then analyzes it. We won’t provide an episode_id in the first inference request, so the gateway will generate a new one for us. We’ll then use that value in our second inference request.
run_with_tensorzero.py
from tensorzero import TensorZeroGateway

with TensorZeroGateway.build_http(gateway_url="http://localhost:3000") as client:
    haiku_response = client.inference(
        function_name="generate_haiku",
        # We don't provide an episode_id for the first inference in the episode
        input={
            "messages": [
                {
                    "role": "user",
                    "content": "Write a haiku about artificial intelligence.",
                }
            ]
        },
    )

    print(haiku_response)

    # When we don't provide an episode_id, the gateway will generate a new one for us
    episode_id = haiku_response.episode_id

    # In a production application, we'd first validate the response to ensure the model returned the correct fields
    haiku = haiku_response.content[0].text

    analysis_response = client.inference(
        function_name="analyze_haiku",
        # For future inferences in that episode, we provide the episode_id that we received
        episode_id=episode_id,
        input={
            "messages": [
                {
                    "role": "user",
                    "content": f"Write a one-paragraph analysis of the following haiku:\n\n{haiku}",
                }
            ]
        },
    )

    print(analysis_response)

​
Conclusion & Next Steps

Episodes are first-class citizens in TensorZero that enable powerful workflows for multi-step LLM systems. You can use them alongside other features like experimentation, metrics & feedback, and tool use (function calling). For example, you can track KPIs for entire episodes instead of individual inferences, and later jointly optimize your LLMs to maximize these metrics.