Learn how to collect metrics and feedback about inferences or sequences of inferences.
Feedback Type | Examples |
---|---|
Boolean Metric | Thumbs up, task success |
Float Metric | Star rating, clicks, number of mistakes made |
Comment | Natural-language feedback from users or developers |
Demonstration | Edited drafts, labels, human-generated content |
/feedback
endpoint.
tensorzero.toml
configuration file.
The skeleton of a metric looks like the following configuration entry.
boolean
to capture this behavior since we’re optimizing for a binary outcome: whether we liked the haikus or not.
The metric applies to individual inference requests, so we’ll set level = "inference"
.
And finally, we’ll set optimize = "max"
because we want to maximize this metric.
Our metric configuration should look like this:
Full Configuration
inference_id
we receive from the first API call to link the two.
Sample Output
demonstration
.
You can’t assign demonstrations to an episode.
comment
.