Even (very) noisy LLM evaluators are useful for improving AI agents

Rule-based and classical NLP metrics are often brittle and miss the semantic dimensions that matter.^{1, 2} Learned reward models are vulnerable to distribution shift³ and reward hacking.⁴ Studies of LLM-as-a-judge setups have repeatedly documented systematic biases and limitations: judges are heavily swayed by surface-level style,⁵ prefer longer responses to shorter ones of similar quality,⁶ are inconsistent across repeated evaluators and minor prompt variations,⁷ often align poorly with human judgments,⁸ and may correlate weakly with the downstream outcomes they’re meant to predict.⁹

For each agent, we model the distribution of true scores as a Beta distribution parameterized by its mean $\mu \in (0, 1)$ and a fixed concentration $\kappa > 0$:

The mean of the distribution is exactly $\mu$, and increasing $\kappa$ concentrates mass more tightly around $\mu$. We use $\kappa = 5$, $\mu_A = 0.6$, and $\mu_B = 0.3$, which gives each agent a unimodal distribution with moderate spread while keeping the two visually comparable.

For each evaluator with noise $\sigma$, the evaluator score for an output with true score $S$ is

with $\sigma = 0.03,\; 0.12,\; 0.25$ for the slightly, moderately, and very noisy evaluators, respectively. Each click of the “Draw new samples” button shows a fresh random realization with $N = 30$ trajectories per agent; the empirical means $\widehat{\mu}_A, \widehat{\mu}_B$ are the average evaluator scores within each agent’s sample.

The sample size required to reliably distinguish two agents scales inversely with the square of the performance gap between them — halving the gap roughly quadruples the number of samples you need. This squared scaling comes from how the sampling distribution of a mean tightens with $N$: the variance of a sample mean shrinks as $1/N$, so its standard error shrinks as $1/\sqrt{N}$, and reliably resolving a gap of size $\Delta$ requires the standard error to be small relative to $\Delta$ — i.e., $N$ to grow as $1/\Delta^2$. The interactive figure above is illustrative: with a 0.30 gap and only $N = 30$ samples per agent, even the noisiest of the three evaluators gets the ordering right essentially every draw. For agents that differ by 5 to 10 percentage points on an outcome of interest — a typical magnitude in practice — even a fairly noisy evaluator can give the correct ranking with high probability on a few hundred to a few thousand examples.

This section formalizes the claim that the empirical evaluator means can recover the true ordering of agents given enough samples, and discusses when this can fail.

Let $x$ be an agent output, and let $S(x)$ be the true score for $x$, meaning the thing we’d ideally like to measure. Each agent gives rise to a distribution over trajectories $x$, which can differ in arbitrary ways. Maybe $A$ tends to produce long, detailed responses, while $B$ tends to be short and crisp. Or maybe $A$ is more prone to hallucination than $B$. The average scores $\mu_A$ and $\mu_B$ are expected values over those distributions, which we’ll denote with $\mathbb{E}_A$ and $\mathbb{E}_B$, so that

Now let $V$ be an evaluator, so that $V(x)$ is the score the evaluator assigns to output $x$. Define a noise term $\varepsilon(x) := V(x) - S(x)$. Then we have

The empirical evaluator means converge in general to the true population evaluator means:

Therefore, in order for the empirical means to lead us to choose the better agent with high probability, we need the expected evaluator scores to mirror the true ordering of agents. That is, we desire the following property to hold:

for all pairs of candidate agents $A$ and $B$. From Eq. (1) and the linearity of expectation, we have

From this, we see that Eq. (2) can be satisfied in many different ways. For example, if the noise terms average out to 0 for both distributions (a common assumption), then we’d have

from which Eq. (2) would immediately follow. Even if the noise doesn’t average out to 0 — meaning the evaluator systematically under- or over-estimates the true scores — we’d still obtain the same result, as long as it averages out to the same value for both agents. More generally, Eq. (2) can hold as long as the per-agent noise gap $\mathbb{E}_A[\varepsilon(x)] - \mathbb{E}_B[\varepsilon(x)]$ doesn’t have the wrong sign and a large enough magnitude to reverse the true-score gap $\mathbb{E}_A[S(x)] - \mathbb{E}_B[S(x)]$. Equivalently, the evaluator can have arbitrary per-agent biases, as long as those biases don’t favor the worse agent strongly enough to overturn its true disadvantage.

In practice, of course, we never observe the expectations $\mathbb{E}_A[V(x)]$ and $\mathbb{E}_B[V(x)]$ directly; we only have the empirical means $\widehat{\mu}_A, \widehat{\mu}_B$ computed over a finite evaluation dataset, and on any given draw those empirical means might disagree with the true ordering. The claim is that, when Eq. (2) holds and trajectories are sampled appropriately, the probability of disagreement vanishes as the evaluation dataset grows:

where $N$ is the size of the evaluation dataset, and assuming without loss of generality that $\mu_A > \mu_B$. In plain terms: with a large enough evaluation dataset, we will correctly identify the better agent with arbitrarily high probability — even when the evaluator is noisy on individual trajectories. We omit a formal proof here, but (3) can be shown to hold under typical sampling regimes, such as iid sampling, stationary ergodic processes, or other forms of sufficiently weak dependence.

Where this breaks down. The conditions above are sufficient for the empirical mean evaluator scores to recover the true ordering with high probability, but they aren’t always satisfied in practice. A few common failure modes:

• Region-specific bias.¹⁰ If the evaluator’s bias varies across the score range — say, it gives polished-looking outputs extra credit regardless of correctness — and the agents under comparison concentrate their outputs in different regions, then $\mathbb{E}_A[\varepsilon(x)] \neq \mathbb{E}_B[\varepsilon(x)]$ and the per-agent noise gap can flip the sign of Eq. (2). More data causes the empirical means to converge to biased values rather than the truth.
• Distribution shift between offline and online. The offline test set may not match the distribution the deployed agent encounters in production. If the evaluator’s noise behaves differently on those two distributions, an offline ranking won’t necessarily predict online behavior — even when the offline argument goes through cleanly.
• Strong dependence or non-stationarity. The convergence claim (3) tolerates iid, ergodic, or weakly dependent sampling, but strongly correlated trajectories can prevent the empirical means from converging.

These failure modes aren’t specific to offline agent selection: they affect any use of an evaluator, online or offline.

Agent and evaluator generation. Each task’s 25 agent variants differ in their system prompt and the underlying model they use. Both the agents and evaluators were LLM-generated: the LLM was told that each evaluator should be as highly correlated as possible with the task’s ground-truth metric, and was given access to a training set when generating the evaluators. We used small models and didn’t aim to optimize either the agents or the evaluators — the point isn’t to show how well an LLM can perform at these tasks, but to illustrate how even noisy evaluators can be useful for agent selection.

Pearson vs Spearman. The numbers above are Pearson correlations, which capture linear association — a good fit for the output-level use case (e.g. guardrails), where the magnitudes of evaluator scores feed directly into downstream decisions. Spearman correlations capture rank association and are a more natural fit for the variant selection use case, where what matters is whether the evaluator’s ranking matches the ground-truth ranking. The two metrics agree qualitatively in every environment we measured (both increase from output-level to agent-level, both consistent with the central claim) and are mathematically equivalent in the special case that both variables are binary.

The largest divergence between the two metrics is on the Data Extraction (NER) task’s output-level cell, where ground truth is binary (exact match) and the Pearson value (0.08) is low because the evaluator’s discrete partial-credit scores compress more than they correlate linearly with the binary outcome.

Within-variant decomposition by de-meaning. The “output-level” bars in the chart above are not raw per-(variant, trace) correlations; those would mix two distinct effects: a between-variant effect (the evaluator agrees with ground truth on which variants are good on average) and a within-variant effect (the evaluator agrees with ground truth on which traces within a single variant are good). To isolate the within-variant effect, for each variant we subtract that variant’s mean from both the evaluator score and the ground-truth score, then compute the correlation across all (variant, trace) cells on the de-meaned values. What’s left is the part of the evaluator-vs-ground-truth signal that is not explained by knowing which variant produced the output — i.e., the signal an evaluator would need to reliably grade individual outputs in production.