Skip to main content

Documentation Index

Fetch the complete documentation index at: https://www.tensorzero.com/docs/llms.txt

Use this file to discover all available pages before exploring further.

TensorZero and Portkey offer diverse features to streamline LLM engineering, including an LLM gateway, observability tools, and more. TensorZero is fully open-source and self-hosted, while Portkey offers an open-source gateway but otherwise requires a paid commercial (hosted) service. Additionally, TensorZero has more features around LLM optimization (e.g. advanced fine-tuning workflows and inference-time optimizations), whereas Portkey has a broader set of features around the UI (e.g. prompt playground).

​
Similarities

  • Unified Inference API. Both TensorZero and Portkey offer a unified inference API that allows you to access LLMs from most major model providers with a single integration, with support for structured outputs, batch inference, tool use, streaming, and more.
    → TensorZero Gateway Quickstart
  • Automatic Fallbacks, Retries, & Load Balancing for Higher Reliability. Both TensorZero and Portkey offer automatic fallbacks, retries, and load balancing features to increase reliability.
    → Retries & Fallbacks with TensorZero
  • Schemas, Templates. Both TensorZero and Portkey offer schema and template features to help you manage your LLM applications.
    → Prompt Templates & Schemas with TensorZero
  • Multimodal Inference. Both TensorZero and Portkey support multimodal inference.
    → Multimodal Inference with TensorZero

​
Key Differences

​
TensorZero

  • Open-Source Observability. TensorZero offers built-in open-source observability features, collecting inference and feedback data in your own database. Portkey also offers observability features, but they are limited to their commercial (hosted) offering.
  • Built-in Evaluations. TensorZero offers built-in evaluation functionality, including heuristics and LLM judges. Portkey doesn’t offer any evaluation features.
    → TensorZero Evaluations Overview
  • Open-Source Inference Caching. TensorZero offers open-source inference caching features, allowing you to cache requests to improve latency and reduce costs. Portkey also offers inference caching features, but they are limited to their commercial (hosted) offering.
    → Inference Caching with TensorZero
  • Open-Source Fine-Tuning Workflows. TensorZero offers open-source built-in fine-tuning workflows, allowing you to create custom models using your own data. Portkey also offers fine-tuning features, but they are limited to their enterprise ($$$) offering.
    → LLM Optimization with TensorZero
  • Advanced Fine-Tuning Workflows. TensorZero offers advanced fine-tuning workflows, including the ability to curate datasets using feedback signals (e.g. production metrics) and the ability to use RLHF for reinforcement learning. Portkey doesn’t offer similar features.
    → LLM Optimization with TensorZero
  • Automated Experimentation (A/B Testing). TensorZero offers advanced A/B testing features, including automated experimentation, to help your identify the best models and prompts for your use cases. Portkey only offers simple canary and A/B testing features.
    → Run adaptive A/B tests with TensorZero
  • Inference-Time Optimizations. TensorZero offers built-in inference-time optimizations (e.g. dynamic in-context learning), allowing you to optimize your inference performance. Portkey doesn’t offer any inference-time optimizations.
    → Inference-Time Optimizations with TensorZero
  • Programmatic & GitOps-Friendly Orchestration. TensorZero can be fully orchestrated programmatically in a GitOps-friendly way. Portkey can manage some of its features programmatically, but certain features depend on its external commercial hosted service.
  • Open-Source Access Control. Both TensorZero and Portkey offer access control features like TensorZero API keys. Portkey only offers them in the commercial (hosted) offering, whereas TensorZero’s solution is fully open-source.
    → Set up auth for TensorZero

​
Portkey

  • Prompt Playground. Portkey offers a prompt playground in its commercial (hosted) offering, allowing you to test your prompts and models in a graphical interface. TensorZero doesn’t offer a prompt playground today (coming soon!).
  • Guardrails. Portkey offers guardrails features, including integrations with third-party guardrails providers and the ability to use custom guardrails using webhooks. For now, TensorZero doesn’t offer built-in guardrails, and instead requires you to manage integrations yourself.
  • Managed Service. Portkey offers a paid managed (hosted) service in addition to the open-source version. TensorZero is fully open-source and self-hosted.