Getting Started with vLLM

This guide shows how to set up a minimal deployment to use the TensorZero Gateway with self-hosted LLMs using vLLM.

We’re using Llama 3.1 in this example, but you can use virtually any model supported by vLLM.

Setup

This guide assumes that you are running vLLM locally with vllm serve meta-llama/Llama-3.1-8B-Instruct. Make sure to update the api_base and model_name in the configuration below to match your vLLM server and model.

For this minimal setup, you’ll need just two files in your project directory:

Directoryconfig/
- tensorzero.toml
docker-compose.yml

For production deployments, see our Deployment Guide.

Configuration

Create a minimal configuration file that defines a model and a simple chat function:

[models.llama3_1_8b_instruct]
routing = ["vllm"]

[models.llama3_1_8b_instruct.providers.vllm]
type = "vllm"
api_base = "http://host.docker.internal:8000/v1/"  # for vLLM running locally on the host
model_name = "meta-llama/Llama-3.1-8B-Instruct"
api_key_location = "none"  # by default, vLLM requires no API key

[functions.my_function_name]
type = "chat"

[functions.my_function_name.variants.my_variant_name]
type = "chat_completion"
model = "llama3_1_8b_instruct"

# Disable observability to keep this example minimal (not recommended in production)
[gateway]
disable_observability = true

Credentials

The api_key_location field in your model provider configuration specifies how to handle API key authentication:

If your endpoint does not require an API key (e.g. vLLM by default):
```
api_key_location = "none"
```
If your endpoint requires an API key, you have two options:
1. Configure it in advance through an environment variable:
```
api_key_location = "env::ENVIRONMENT_VARIABLE_NAME"
```
  You’ll need to set the environment variable before starting the gateway.
2. Provide it at inference time:
```
api_key_location = "dynamic::ARGUMENT_NAME"
```
  The API key can then be passed in the inference request.

See the Configuration Reference and the API reference for more details.

In this example, vLLM is running locally without authentication, so we use api_key_location = "none".

Deployment (Docker Compose)

Create a minimal Docker Compose configuration:

# This is a simplified example for learning purposes. Do not use this in production.
# For production-ready deployments, see: https://www.tensorzero.com/docs/gateway/deployment

services:
  gateway:
    image: tensorzero/gateway
    volumes:
      - ./config:/app/config:ro
    # environment:
    #   - VLLM_API_KEY=${VLLM_API_KEY:?Environment variable VLLM_API_KEY must be set.}
    ports:
      - "3000:3000"

You can start the gateway with docker compose up.

Inference

Make an inference request to the gateway:

curl -X POST http://localhost:3000/inference \
  -H "Content-Type: application/json" \
  -d '{
    "function_name": "my_function_name",
    "input": {
      "messages": [
        {
          "role": "user",
          "content": "What is the capital of Japan?"
        }
      ]
    }
  }'