Skip to main content
This guide shows how to set up a minimal deployment to use the TensorZero Gateway with the Azure OpenAI Service.

​
Setup

For this minimal setup, you’ll need just two files in your project directory: 
- config/
  - tensorzero.toml
- docker-compose.yml
You can also find the complete code for this example on GitHub.
For production deployments, see our Deployment Guide.

​
Configuration

Create a minimal configuration file that defines a model and a simple chat function:
config/tensorzero.toml
[models.gpt_4o_mini_2024_07_18]
routing = ["azure"]

[models.gpt_4o_mini_2024_07_18.providers.azure]
type = "azure"
deployment_id = "gpt4o-mini-20240718"
endpoint = "https://your-azure-openai-endpoint.openai.azure.com"

[functions.my_function_name]
type = "chat"

[functions.my_function_name.variants.my_variant_name]
type = "chat_completion"
model = "gpt_4o_mini_2024_07_18"
See the list of models available on Azure OpenAI Service. If you need to configure the endpoint at runtime, you can set it to endpoint = "env::AZURE_OPENAI_ENDPOINT" to read from the environment variable AZURE_OPENAI_ENDPOINT on startup or endpoint = "dynamic::azure_openai_endpoint" to read from a dynamic credential azure_openai_endpoint on each inference.

​
Credentials

You must set the AZURE_OPENAI_API_KEY environment variable before running the gateway. You can customize the credential location by setting the api_key_location to env::YOUR_ENVIRONMENT_VARIABLE or dynamic::ARGUMENT_NAME. See the Credential Management guide and Configuration Reference for more information.

​
Deployment (Docker Compose)

Create a minimal Docker Compose configuration:
docker-compose.yml
# This is a simplified example for learning purposes. Do not use this in production.
# For production-ready deployments, see: https://www.tensorzero.com/docs/gateway/deployment

services:
  gateway:
    image: tensorzero/gateway
    volumes:
      - ./config:/app/config:ro
    command: --config-file /app/config/tensorzero.toml
    environment:
      - AZURE_OPENAI_API_KEY=${AZURE_OPENAI_API_KEY:?Environment variable AZURE_OPENAI_API_KEY must be set.}
    ports:
      - "3000:3000"
    extra_hosts:
      - "host.docker.internal:host-gateway"
You can start the gateway with docker compose up.

​
Inference

Make an inference request to the gateway: 
curl -X POST http://localhost:3000/inference \
  -H "Content-Type: application/json" \
  -d '{
    "function_name": "my_function_name",
    "input": {
      "messages": [
        {
          "role": "user",
          "content": "What is the capital of Japan?"
        }
      ]
    }
  }'

​
Other Features

​
Generate embeddings

The Azure OpenAI Service model provider supports generating embeddings. You can find a complete code example on GitHub.