/evaluations?tab=evaluators) lists evaluation types. Section title: Evaluation types. Description: Enable built-in evaluators or create custom ones for your use case.
Built-in types
OpenLIT ships 11 built-in types. Each can be enabled or disabled independently and linked to Rule Engine rules for conditional runs.
Badges you may see: Enabled / Disabled, Custom, Rule engine. Rows can show Linked from rules: when Rule Engine entities point at that type.
When context is provided via the Rule Engine, evaluations judge the LLM response against that context - not against real-world knowledge alone.
Custom evaluators
Click Create custom evaluator to open the create form (Type ID, Label, Description, Evaluation Prompt). Custom types appear under Custom evaluation types, run alongside built-in types in auto and manual evaluation, and can be deleted when you no longer need them. Click a built-in or custom card to open its detail page (/evaluations/types/[id]) - enable/disable, edit the prompt, and link Rule Engine rules. From Analytics, an evaluator row opens /evaluations/evaluators/[id] for pass-rate and recent results (Back to Analytics, Configure).
Legacy URLs /evaluations/types and /evaluations/settings redirect to the Evaluators and Configuration tabs.
Online vs offline
- Online - Auto Evaluation or Run Evaluation on a live trace (see Configuration and LLM-as-a-Judge).
- Offline - Programmatic evaluations via the SDK, typically in CI/CD.
- Human - Manual Feedback on a trace (Good / Bad / Neutral).
Frequently asked questions
What is LLM evaluation?
What is LLM evaluation?
Scoring LLM outputs for quality, safety, and correctness - automatically with an LLM-as-a-judge, with custom types, or with human feedback - so you can catch issues in production or before you ship.
Can I create custom evaluation types?
Can I create custom evaluation types?
Yes. Use Create custom evaluator on the Evaluators tab with your own prompt and criteria.