Skip to main content
Not every quality signal needs an LLM judge. Manual Feedback lets a human rate a specific trace directly, alongside the automated evaluations already running on it.

​
How it works

  1. Open any LLM trace in Telemetry and click its Evaluation tab.
  2. In the Feedback panel, rate the response Good, Bad, or Neutral.
  3. Optionally add a comment explaining the rating.
  4. Click Submit Feedback.
Your feedback is stored as its own evaluation entry (manual_feedback) in the same evaluation pipeline as the automated evaluators - so it appears in the trace’s evaluation list right alongside Hallucination, Bias, and any other evaluator results, and rolls up into Evaluations → Analytics.
Score follows the same convention as the automated evaluators (low score = no issue): Good stores 0, Neutral stores 0.5, and Bad stores 1. Your comment is saved as the entry’s explanation.

​
When to use it

  • Spot-checking responses an automated evaluator didn’t flag, or disagreeing with one that did
  • Collecting reviewer or subject-matter-expert sign-off on specific traces
  • Capturing qualitative notes that don’t fit a built-in or custom evaluation type

Evaluators

Built-in and custom evaluation types

LLM-as-a-Judge

Automated scoring for hallucination, bias, toxicity, and more

Configuration

Auto Evaluation schedule and judge model