TensorZero

What is TensorZero?

TensorZero is an 9.6Kopen-source stack for industrial-grade LLM applications.

• Gateway: access every LLM provider through a unified API (<1ms p99 latency)
• Observability: monitor your LLM systems, programmatically or with a UI
• Optimization: optimize your prompts, models, and inference strategies
• Evaluations: benchmark individual inferences or end-to-end workflows
• Experimentation: deploy with built-in A/B testing, fallbacks, etc.

Take what you need, adopt incrementally, and complement with other tools.

How do I get started?

You can use TensorZero with its Python SDK, any OpenAI SDK (Python, Node, Go, etc.), or its HTTP API.

For example, with a single line of code, your OpenAI SDK starts supporting hundreds of LLMs across all major providers. From there, you can add observability, automatic fallbacks, A/B testing, and much more.

from openai import OpenAI
from tensorzero import patch_openai_client

client = OpenAI()

patch_openai_client(client, async_setup=False)

response = client.chat.completions.create(
    model="tensorzero::model_name::openai::gpt-5"
    # or: model="tensorzero::model_name::anthropic::claude-opus-4-1"
    # or Google, AWS, Azure, xAI, vLLM, Ollama, and many more...
    messages=[
        {
            "role": "user",
            "content": "Woah I can try hundreds of LLMs like this?",
        }
    ],
)

Our Quick Start shows how to set up a production-ready LLM application with observability and fine-tuning in just 5 minutes.

How much does TensorZero cost?

Nothing. TensorZero is 100% self-hosted and open-source. There are no paid features.

Is TensorZero production-ready?

Yes. Here’s a case study: Automating Code Changelogs at a Large Bank with LLMs

How can I ask questions or share feedback?

Reach out on Slack, Discord, or GitHub.

Do you offer enterprise support?

Yes, at no cost. Email us at hello@tensorzero.com to set up a dedicated support Slack channel for your team.

Why are you building TensorZero?

Our goal is to enable LLM applications to learn from real-world experience.

The current offering is the first step towards that vision: a data and learning flywheel that enables a feedback loop for optimizing LLM applications — turning production data into smarter, faster, and cheaper models.

Who is building TensorZero?

Our technical team includes a former Rust compiler maintainer, machine learning researchers (Stanford, CMU, Oxford, Columbia) with thousands of citations, and the chief product officer of a decacorn startup.

We’re backed by the same investors as leading open-source projects (e.g. ClickHouse, CockroachDB) and AI labs (e.g. OpenAI, Anthropic).

Can I contribute? Are you hiring?

We welcome open-source contributions. We’re also hiring in NYC.