Skip to main content
This page shows how to:
  • Generate embeddings with a unified API. TensorZero unifies many LLM APIs (e.g. OpenAI) and inference servers (e.g. Ollama).
  • Use any programming language. You can use any OpenAI SDK (Python, Node, Go, etc.) or the OpenAI-compatible HTTP API.
You can find a complete runnable example of this guide on GitHub.

​
Generate embeddings from OpenAI

Our example uses the OpenAI Python SDK, but you can use any OpenAI SDK or call the OpenAI-compatible HTTP API. See Call any LLM for an example using the OpenAI Node SDK.The TensorZero Python SDK doesn’t have an independent embedding endpoint at the moment.
You can point the OpenAI Python SDK to a TensorZero Gateway to generate embeddings with a unified API.
1

Set up the credentials for your LLM provider

For example, if you’re using OpenAI, you can set the OPENAI_API_KEY environment variable with your API key.
export OPENAI_API_KEY="sk-..."
See the Integrations page to learn how to set up credentials for other LLM providers.
2

Install the OpenAI Python SDK

You can install the OpenAI SDK with a Python package manager like pip.
pip install openai
3

Deploy the TensorZero Gateway

Let’s deploy the TensorZero Gateway using Docker. For simplicity, we’ll use the gateway without observability or custom configuration.
docker run \
  -e OPENAI_API_KEY \
  -p 3000:3000 \
  tensorzero/gateway \
  --default-config
See the Deploy the TensorZero Gateway page for more details.
4

Initialize the OpenAI client

Let’s initialize the OpenAI SDK and point it to the gateway we just launched.
from openai import OpenAI

client = OpenAI(base_url="http://localhost:3000/openai/v1", api_key="not-used")
5

Call the LLM

result = client.embeddings.create(
    input="Hello, world!",
    model="tensorzero::embedding_model_name::openai::text-embedding-3-small",
    # or: Azure, any OpenAI-compatible endpoint (e.g. Ollama, Voyager)
)

CreateEmbeddingResponse(
    data=[
        Embedding(
            embedding=[
                -0.019143931567668915,
                # ...
            ],
            index=0,
            object='embedding'
        )
    ],
    model='tensorzero::embedding_model_name::openai::text-embedding-3-small',
    object='list',
    usage=Usage(prompt_tokens=4, total_tokens=4)
)

​
Define a custom embedding model

You can define a custom embedding model in your TensorZero configuration file. For example, let’s define a custom embedding model for nomic-embed-text served locally by Ollama.
1

Deploy the Ollama embedding model

Download the embedding model and launch the Ollama server:
ollama pull nomic-embed-text
ollama serve
We assume that Ollama is running on your host machine at http://localhost:11434.
2

Define your custom embedding model

Add your custom model and model provider to your configuration file:
tensorzero.toml
[embedding_models.nomic-embed-text]
routing = ["ollama"]

[embedding_models.nomic-embed-text.providers.ollama]
type = "openai"
api_base = "http://host.docker.internal:11434/v1"
model_name = "nomic-embed-text"
api_key_location = "none"
See the Configuration Reference for details on configuring your embedding models.
3

Deploy the TensorZero Gateway with your configuration

Deploy the TensorZero Gateway with your configuration file. Make sure that the container has access to the Ollama server running on the host.
See the Deploy the TensorZero Gateway page for more details.
4

Call your custom embedding model

Use your custom model by referencing it with tensorzero::embedding_model_name::nomic-embed-text.For example, using the OpenAI Python SDK:
from openai import OpenAI

client = OpenAI(base_url="http://localhost:3000/openai/v1", api_key="not-used")

result = client.embeddings.create(
    input="Hello, world!",
    model="tensorzero::embedding_model_name::nomic-embed-text",
)

CreateEmbeddingResponse(
    data=[
        Embedding(
            embedding=[
                -0.019143931567668915,
                # ...
            ],
            index=0,
            object='embedding'
        )
    ],
    model='tensorzero::embedding_model_name::nomic-embed-text',
    object='list',
    usage=Usage(prompt_tokens=4, total_tokens=4)
)

​
Cache embeddings

The TensorZero Gateway supports caching embeddings to improve latency and reduce costs. When caching is enabled, identical embedding requests will be served from the cache instead of being sent to the model provider. 
result = client.embeddings.create(
    input="Hello, world!",
    model="tensorzero::embedding_model_name::openai::text-embedding-3-small",
    extra_body={
        "tensorzero::cache_options": {
            "enabled": "on",  # Enable reading from and writing to cache
            "max_age_s": 3600,  # Optional: cache entries older than 1 hour are ignored
        }
    }
)
Caching works for single embeddings. Batch embedding requests (multiple inputs) will write to the cache but won’t serve cached responses. See the Inference Caching guide for more details on cache modes and options.