Evaluations

Automated AI evaluation to assess and monitor the quality, safety, and performance of your LLM outputs across development and production environments. Open Monitor → Evaluations for Analytics, Evaluators, and Configuration. For evaluation types and custom evaluators, see Evaluators.

Find the right feature

If you want to…	Use this OpenLIT feature
Automatically score every production trace	Configuration (Auto Evaluation)
Score one specific trace on demand	LLM-as-a-Judge from that trace’s Evaluation tab
Rate a trace yourself instead of an LLM judge	Manual Feedback - Good / Bad / Neutral plus a comment
Judge responses against your own ground truth, not just the model’s knowledge	Rule Engine + Context
Evaluate on criteria beyond the 11 built-in types	Custom evaluators
Only run certain evaluators for certain models, providers, or environments	Evaluators + Rule Engine
Test prompt or model changes before shipping	Programmatic evaluations via the SDK
Block a deploy on a quality regression	Programmatic evaluations in a CI/CD pipeline
See pass-rate trends across evaluators and time	Analytics

Setup

Online
Offline

Pick a judge model

Go to Evaluations → Settings, choose a provider and model to act as the judge (OpenAI, Anthropic, Google, Mistral, TypeSafe Jev, and others), and add its API key from Vault. TypeSafe Jev uses the System One API rather than chat completions. See TypeSafe Jev judge.

Enable evaluators

Switch to the Evaluation Types tab and turn on the evaluators you want. Hallucination, Bias, and Toxicity are enabled by default; Relevance, Coherence, Safety, and 5 others are opt-in.

Turn on Auto Evaluation

Back in Settings, enable Auto Evaluation with a cron schedule so every new trace gets scored automatically - or skip this and click Run Evaluation from any trace’s Evaluation tab to score it on demand.

Review results

Open any trace’s Evaluation tab for its score, classification, and reasoning, or check the Evaluations dashboard for aggregate trends across models and time.

Get an API key

In OpenLIT, go to Settings → API Keys and create a key. Offline evaluations call your OpenLIT server, so you’ll need this key plus your OpenLIT URL.

Run an evaluation

Python
Typescript

import openlit

openlit.init(
    openlit_url="http://localhost:3000",
    openlit_api_key="openlit-xxxxx",
)

result = openlit.eval(
    prompt="What is the capital of France?",
    response="The capital of France is Lyon.",
    contexts=["Paris is the capital and largest city of France."],
)

assert result.passed, f"Evaluation failed: {result.failed_evals}"

import openlit, { isPassed, getFailedEvals } from "openlit"

openlit.init({
  openlitUrl: "http://localhost:3000",
  openlitApiKey: "openlit-xxxxx",
})

const result = await openlit.eval({
  prompt: "What is the capital of France?",
  response: "The capital of France is Lyon.",
  contexts: ["Paris is the capital and largest city of France."],
})

console.log(isPassed(result), getFailedEvals(result))

This uses the exact same evaluators, custom types, and Rule Engine context matching configured in your dashboard - see the Programmatic evaluations quickstart for batch runs and CI/CD gating.

Evaluators

Built-in and custom evaluation types

Configuration

Auto Evaluation schedule and judge model

LLM-as-a-Judge

Use advanced LLMs to evaluate AI application quality with automated scoring

Programmatic evaluations

Quick start guide for implementing custom evaluations in your code

Getting Started

Concepts

Observability

Dashboards

Otter (AI Copilot)

Pricing & Models

Prompts and Experiments

Developer Resources

Evaluations

Find the right feature

Setup

Evaluators

Configuration

LLM-as-a-Judge

Programmatic evaluations

​Find the right feature

​Setup

Evaluators

Configuration

LLM-as-a-Judge

Programmatic evaluations

Find the right feature

Setup