It’s easy to get started with the TensorZero Gateway. To deploy the TensorZero Gateway, you need to:
See the TensorZero UI Deployment Guide for more details on how to deploy the TensorZero UI.

​
ClickHouse

The TensorZero Gateway stores inference and feedback data in a ClickHouse database. This data is later used for model observability, experimentation, and optimization.

​
Development

For development purposes, you can run a single-node ClickHouse instance locally (e.g. using Homebrew or Docker) or a cheap Development-tier cluster on ClickHouse Cloud. See the ClickHouse documentation for more details on configuring your ClickHouse deployment.

​
Production

​
Managed Services

For production deployments, the easiest setup is to use a managed service like ClickHouse Cloud. ClickHouse Cloud is also available through the AWS Marketplace, GCP Marketplace, and Azure Marketplace. Other options for managed ClickHouse deployments include Tinybird (serverless) and Altinity (hands-on support).

​
Self-Managed Deployment

You can alternatively run your own self-managed ClickHouse instance or cluster.
The TensorZero Gateway does not automatically enable data replication for ClickHouse tables.If you are using a self-managed distributed ClickHouse deployment, you must set up replication yourself. See the ClickHouse replication documentation for more details.ClickHouse Cloud automatically sets up data replication, so this step is not necessary if you’re using the managed service.

​
Configuration

After setting up your database, you need to configure the TENSORZERO_CLICKHOUSE_URL environment variable with the connection details. The variable takes a standard format.
.env
TENSORZERO_CLICKHOUSE_URL="http[s]://[username]:[password]@[hostname]:[port]/[database]"

# Example: ClickHouse running locally
TENSORZERO_CLICKHOUSE_URL="http://chuser:chpassword@localhost:8123/tensorzero"

# Example: ClickHouse Cloud
TENSORZERO_CLICKHOUSE_URL="https://USERNAME:[email protected]:8443/tensorzero"

# Example: TensorZero Gateway running in a container, ClickHouse running on host machine
TENSORZERO_CLICKHOUSE_URL="http://host.docker.internal:8123/tensorzero"
The TensorZero Gateway automatically applies database migrations on startup. See [gateway] in the Configuration Reference for the relevant configuration (e.g. customizing the port).

​
TensorZero Gateway

The TensorZero Python client includes a built-in embedded gateway, so you don’t need to run a separate service for it. The gateway is only needed if you want to use the OpenAI Python client or interact with TensorZero via its HTTP API (for other programming languages). The TensorZero UI also requires the gateway service.

​
Development

Running with Docker (Recommended)

​
Production

You can deploy the TensorZero Gateway alongside your application (e.g. as a sidecar container) or as a standalone service. A single gateway instance can handle over 1k QPS/core with sub-millisecond latency (see Benchmarks), so a simple deployment should suffice for the vast majority of applications. If you deploy it as an independent service, we recommend deploying at least two instances behind a load balancer for high availability. The gateway is stateless, so you can easily scale horizontally and don’t need to worry about persistence.

Running with Docker (Recommended)

​
Command Line Arguments

The TensorZero Gateway requires either --config-file to specify a custom configuration file (e.g. --config-file /path/to/tensorzero.toml) or --default-config to use default settings (i.e. no custom functions, metrics, etc.). You can also use --log-format to set the logging format to either pretty (default) or json.

​
Clients

See the Clients page for more details on how to interact with the TensorZero Gateway.

​
Configuration

To run the TensorZero Gateway, first you need to create a tensorzero.toml configuration file. Read more about the configuration file here.

​
Model Provider Credentials

In addition to the TENSORZERO_CLICKHOUSE_URL environment variable discussed above, the TensorZero Gateway accepts the following environment variables for provider credentials. Unless you specify an alternative credential location in your configuration file, these environment variables are required for the providers that are used in a variant with positive weight. If required credentials are missing, the gateway will fail on startup.
ProviderEnvironment Variable(s)
AnthropicANTHROPIC_API_KEY
AWS BedrockAWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION (optional)
AWS SageMakerAWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION (optional)
Azure OpenAIAZURE_OPENAI_API_KEY
FireworksFIREWORKS_API_KEY
GCP Vertex AI AnthropicGCP_VERTEX_CREDENTIALS_PATH (see below for details)
GCP Vertex AI GeminiGCP_VERTEX_CREDENTIALS_PATH (see below for details)
Google AI Studio GeminiGOOGLE_AI_STUDIO_GEMINI_API_KEY
GroqGROQ_API_KEY
HyperbolicHYPERBOLIC_API_KEY
MistralMISTRAL_API_KEY
OpenAIOPENAI_API_KEY
OpenRouterOPENROUTER_API_KEY
TogetherTOGETHER_API_KEY
xAIXAI_API_KEY
Notes:
  • AWS Bedrock supports many authentication methods, including environment variables, IAM roles, and more. See the AWS documentation for more details.
  • If you’re using the GCP Vertex provider, you also need to mount the credentials for a service account in JWT form (described here) to /app/gcp-credentials.json using an additional -v flag.
See .env.example for a complete example with every supported environment variable.