TensorZero Raises $7.3M Seed Round to Build an Open-Source Stack for Industrial-Grade LLM Applications

“What will LLM engineering look like in a few years?”

We asked ourselves this question when we started TensorZero. Our answer is that LLMs will have to learn from real-world experience, just like humans do. The analogy we like here is, “If you take a really smart person and throw them at a completely new job, they won’t be great at it at first but will likely learn the ropes quickly from instruction or trial and error.”

This same process is very challenging for LLMs today. It will only get more complex as more models, APIs, tools, and techniques emerge, especially as teams tackle increasingly ambitious use cases. At some point, you won’t be able to judge business outcomes by staring at individual inferences, which is how most people approach LLM engineering today. You’ll have to reason about these end-to-end systems and their consequences as a whole. TensorZero is our answer to all this.

Our mission is to enable a data and learning flywheel for optimizing LLM applications: a feedback loop that turns production metrics and human feedback into smarter, faster, and cheaper models and agents.

Despite all the noise in the industry, companies building LLM applications still lack the right tools to meet complex cognitive and infrastructure needs and resort to stitching together whatever early solutions are available on the market. TensorZero provides production-grade, enterprise-ready components for building LLM applications that natively work together in a self-reinforcing loop out of the box.

Today, TensorZero’s 9.7Kopen-source stack unifies an LLM gateway, observability, optimization, evaluation, and experimentation. You can take what you need, adopt incrementally, and complement with other tools. Over time, these components enable you to set up a principled feedback loop for your LLM application. The data you collect is tied to your KPIs, ports across model providers, and compounds into a competitive advantage for your business.

Our vision is to automate much of LLM engineering. We’re laying the foundation for that with open-source TensorZero. For example, with our data model and end-to-end workflow, we will be able to proactively suggest new variants (e.g. a new fine-tuned model), backtest it on historical data (e.g. using diverse techniques from reinforcement learning), enable a gradual, live A/B test, and repeat the process.

With a tool like this, engineers can focus on higher-level workflows — deciding what data goes in and out of these models, how to measure success, which behaviors to incentivize and disincentivize, and so on — and leave the low-level implementation details to an automated system. This is the future we see for LLM engineering as a discipline.

For a more formal discussion on these topics, see our blog post Think of LLM Applications as POMDPs — Not Agents.

We started building TensorZero in January 2024 with that goal in mind. After a successful technical pilot with a healthcare voice agent, we decided to open-source the platform and published the first release in September 2024.

Recently, TensorZero reached #1 trending repository of the week globally on GitHub. We’re fortunate to have received contributions from dozens of developers worldwide, and it’s exciting to see TensorZero already powering cutting-edge LLM products at frontier AI startups and large organizations, including one of Europe’s largest banks (read the case study).

The team behind TensorZero brings deep expertise in machine learning and software infrastructure.

Viraj Mehta (CTO) received a PhD from Carnegie Mellon focused on reinforcement learning for nuclear fusion and LLMs; he holds a BS in math and an MS in computer science from Stanford.

Gabriel Bianconi (CEO) was the chief product officer at Ondo, one of the largest decentralized finance projects with over $1B in AUM; he holds BS and MS degrees in computer science from Stanford.

The team also includes former maintainers of major open-source projects such as the Rust compiler, machine learning researchers (Stanford, CMU, Oxford, Columbia) with thousands of citations, and other talented technical contributors.

Are you interested in building on open-source ML infrastructure? We’re looking for outstanding technical talent to join our team in NYC.

TensorZero’s $7.3M seed round will accelerate our efforts to build best-in-class open-source infrastructure for LLM engineers.