Get started with AI Observability

OpenLIT automatically instruments LLMs, VectorDBs, MCP, and frameworks by default.

This guide demonstrates how to implement real-time cost tracking, token usage monitoring, hallucination detection, and latency optimization for your AI applications with OpenTelemetry traces and metrics.

Deploy OpenLIT

Git clone OpenLIT repository

git clone git@github.com:openlit/openlit.git

Start Docker Compose

From the root directory of the OpenLIT Repo, Run the below command:

docker compose up -d

Install OpenLIT SDK

Want zero-code observability? Try the OpenLIT Controller

The Controller uses eBPF to automatically discover and instrument LLM traffic across Kubernetes, Docker, and Linux — no SDK required. Use SDKs for deeper application-level tracing.

Python
Typescript

pip install openlit

npm install openlit

Instrument your AI application

Python
Typescript

Not sure which method to choose? Check out Instrumentation Methods to understand the differences.

Manual instrumentation
Zero-code instrumentation

Via function parameters
Via environment variables

import openlit

openlit.init(otlp_endpoint="http://127.0.0.1:4318")

Examples:

from openai import OpenAI
import openlit

openlit.init(otlp_endpoint="http://127.0.0.1:4318")

client = OpenAI(
    api_key="YOUR_OPENAI_KEY"
)

chat_completion = client.chat.completions.create(
    messages=[
        {
            "role": "user",
            "content": "What is LLM Observability?",
        }
    ],
    model="gpt-3.5-turbo",
)

import os
from anthropic import Anthropic
import openlit

openlit.init(otlp_endpoint="http://127.0.0.1:4318")

client = Anthropic(
    # This is the default and can be omitted
    api_key=os.environ.get("ANTHROPIC_API_KEY"),
)

message = client.messages.create(
    max_tokens=1024,
    messages=[
        {
            "role": "user",
            "content": "Hello, What is LLM Observability?",
        }
    ],
    model="claude-3-opus-20240229",
)

import cohere
import openlit

openlit.init(otlp_endpoint="http://127.0.0.1:4318")

co = cohere.Client(
    api_key="YOUR_API_KEY",
)

chat = co.chat(
    message="hello world!",
    model="command"
)

from litellm import completion
import os
import openlit

openlit.init(otlp_endpoint="http://127.0.0.1:4318")

os.environ["HUGGINGFACE_API_KEY"] = "huggingface_api_key"

# e.g. Call 'WizardLM/WizardCoder-Python-34B-V1.0' hosted on HF Inference endpoints
response = completion(
  model="huggingface/WizardLM/WizardCoder-Python-34B-V1.0",
  messages=[{ "content": "Hello, how are you?","role": "user"}],
  api_base="https://my-endpoint.huggingface.cloud"
)

from langchain_core.messages import HumanMessage, SystemMessage
import openlit

openlit.init(otlp_endpoint="http://127.0.0.1:4318")

from langchain_openai import ChatOpenAI

model = ChatOpenAI(model="gpt-4o-mini")

messages = [
    SystemMessage(content="Translate the following from English into Italian"),
    HumanMessage(content="hi!"),
]

model.invoke(messages)

import ollama
import openlit

openlit.init(otlp_endpoint="http://127.0.0.1:4318")

response = ollama.chat(model='llama3.1', messages=[
  {
    'role': 'user',
    'content': 'Why is the sky blue?',
  },
])

Add the following two lines to your application code:

import openlit

openlit.init()

Run the following command to configure the OTEL export endpoint:

export OTEL_EXPORTER_OTLP_ENDPOINT = "http://127.0.0.1:4318"

Examples:

from openai import OpenAI
import openlit

openlit.init()

client = OpenAI(
    api_key="YOUR_OPENAI_KEY"
)

chat_completion = client.chat.completions.create(
    messages=[
        {
            "role": "user",
            "content": "What is LLM Observability?",
        }
    ],
    model="gpt-3.5-turbo",
)

import os
from anthropic import Anthropic
import openlit

openlit.init()

client = Anthropic(
    # This is the default and can be omitted
    api_key=os.environ.get("ANTHROPIC_API_KEY"),
)

message = client.messages.create(
    max_tokens=1024,
    messages=[
        {
            "role": "user",
            "content": "Hello, What is LLM Observability?",
        }
    ],
    model="claude-3-opus-20240229",
)

import cohere
import openlit

openlit.init()

co = cohere.Client(
    api_key="YOUR_API_KEY",
)

chat = co.chat(
    message="hello world!",
    model="command"
)

from litellm import completion
import os
import openlit

openlit.init()

os.environ["HUGGINGFACE_API_KEY"] = "huggingface_api_key"

# e.g. Call 'WizardLM/WizardCoder-Python-34B-V1.0' hosted on HF Inference endpoints
response = completion(
  model="huggingface/WizardLM/WizardCoder-Python-34B-V1.0",
  messages=[{ "content": "Hello, how are you?","role": "user"}],
  api_base="https://my-endpoint.huggingface.cloud"
)

from langchain_core.messages import HumanMessage, SystemMessage
import openlit

openlit.init()

from langchain_openai import ChatOpenAI

model = ChatOpenAI(model="gpt-4o-mini")

messages = [
    SystemMessage(content="Translate the following from English into Italian"),
    HumanMessage(content="hi!"),
]

model.invoke(messages)

import ollama
import openlit

openlit.init()

response = ollama.chat(model='llama3.1', messages=[
  {
    'role': 'user',
    'content': 'Why is the sky blue?',
  },
])

Via CLI arguments
Via environment variables

# Install OpenLIT
pip install openlit

# Configure via CLI arguments
openlit-instrument \
  --service-name my-ai-app \
  --environment production \
  --otlp-endpoint http://127.0.0.1:4318 \
  python your_app.py

# Configure via environment variables
export OTEL_SERVICE_NAME=my-ai-app
export OTEL_DEPLOYMENT_ENVIRONMENT=production
export OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318

# Run with zero code changes
openlit-instrument python your_app.py

SDK Integration

Via function parameters
Via environment variables

import Openlit from "openlit"

Openlit.init({ otlpEndpoint: "http://127.0.0.1:4318" })

Example Usage for monitoring OpenAI Usage:

import Openlit from "openlit"

Openlit.init({ otlpEndpoint: "http://127.0.0.1:4318" })

async function main() {
      const OpenAI = await import("openai").then((e) => e.default);
      const openai = new OpenAI({
          apiKey: YOUR_OPENAI_KEY,
      });
      const completion = await openai.chat.completions.create({
          model: "gpt-3.5-turbo",
          messages: [{ role: "system", content: "Who are you ?" }],
      });

      console.log(completion?.choices?.[0]);
  }

  main();

import openlit from "openlit"
import OpenAI from "openai"

openlit.init({ 
  otlpEndpoint: "http://127.0.0.1:4318", 
  instrumentations: {
    openai: OpenAI,
  } 
})

async function main() {
      const openai = new OpenAI({
          apiKey: YOUR_OPENAI_KEY,
      });
      const completion = await openai.chat.completions.create({
          model: "gpt-3.5-turbo",
          messages: [{ role: "system", content: "Who are you ?" }],
      });

      console.log(completion?.choices?.[0]);
  }

  main();

Add the following two lines to your application code:

import openlit from "openlit"

openlit.init()

Run the following command to configure the OTEL export endpoint:

export OTEL_EXPORTER_OTLP_ENDPOINT = "http://127.0.0.1:4318"

Example Usage for monitoring OpenAI Usage:

import openlit from "openlit"

openlit.init()

async function main() {
      const OpenAI = await import("openai").then((e) => e.default);
      const openai = new OpenAI({
          apiKey: YOUR_OPENAI_KEY,
      });
      const completion = await openai.chat.completions.create({
          model: "gpt-3.5-turbo",
          messages: [{ role: "system", content: "Who are you ?" }],
      });

      console.log(completion?.choices?.[0]);
  }

  main();

import openlit from "openlit"
import OpenAI from "openai"

openlit.init({
  instrumentations: {
    openai: OpenAI,
  } 
})

async function main() {
      const openai = new OpenAI({
          apiKey: YOUR_OPENAI_KEY,
      });
      const completion = await openai.chat.completions.create({
          model: "gpt-3.5-turbo",
          messages: [{ role: "system", content: "Who are you ?" }],
      });

      console.log(completion?.choices?.[0]);
  }

  main();

Refer to OpenLIT Python SDK repository or Typescript SDK repository for more advanced configurations and use cases.

Monitor, debug and test the quality of your AI applications

With real-time LLM observability data now flowing to OpenLIT, visualize comprehensive AI performance metrics including token costs, latency patterns, hallucination rates, and model accuracy to optimize your production AI applications.Just head over to OpenLIT at 127.0.0.1:3000 on your browser to start exploring. You can login using the default credentials

Email: user@openlit.io
Password: openlituser

You’re all set! Your AI applications now have observability with real-time performance monitoring, cost tracking, and AI safety evaluations. Send Observability telemetry to other OpenTelemetry backends If you wish to send telemetry directly from the SDK to another backend, you can stop the current Docker services by using the command below. For more details on sending the data to your existing OpenTelemetry backends, checkout our Supported Destinations guide.

docker compose down

If you have any questions or need support, reach out to our community.

Quickstart: LLM Evaluations

Get started with evaluating your LLM responses in 2 simple steps

Integrations

60+ AI integrations with automatic instrumentation and performance tracking

Create a dashboard

Create custom visualizations with flexible widgets, queries, and real-time AI monitoring

Getting Started

Observability

Evaluations

Dashboards

Otter (AI Copilot)

Pricing & Models

Prompts and Experiments

Developer Resources

Get started with AI Observability

Want zero-code observability? Try the OpenLIT Controller

Quickstart: LLM Evaluations

Integrations

Create a dashboard