To send OpenTelemetry metrics and traces generated by OpenLIT from your AI Application to Prometheus and Grafana Tempo, follow the below steps.

​
1. Configure OpenTelemetry Collector

Prometheus + Tempo requires an OpenTelemetry Collector to route metrics to Prometheus and traces to Tempo. Install OpenTelemetry Collector (if not already running) For detailed installation instructions, refer to the OpenTelemetry Collector Documentation. Configure the Collector
  1. Configure OTLP Receiver: Set up receiver on 0.0.0.0:4318 for HTTP and 0.0.0.0:4317 for gRPC.
  2. Define Exporters:
    • prometheusremotewrite for metrics → Prometheus
    • otlp for traces → Tempo
  3. Assign to Pipelines: Route metrics and traces to appropriate backends.
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
  memory_limiter:
    # 80% of maximum memory up to 2G
    limit_mib: 1500
    # 25% of limit up to 2G
    spike_limit_mib: 512
    check_interval: 5s

exporters:
  prometheusremotewrite:
    endpoint: "YOUR_PROMETHEUS_REMOTE_WRITE_URL"
    add_metric_suffixes: false
  otlp:
    endpoint: "YOUR_TEMPO_URL"

service:
  pipelines:
    traces:
      receivers: [ otlp ]
      processors: [ memory_limiter, batch ]
      exporters: [ otlp ]
    metrics:
      receivers: [ otlp ]
      processors: [ memory_limiter, batch ]
      exporters: [ prometheusremotewrite ]
Replace:
  1. YOUR_PROMETHEUS_REMOTE_WRITE_URL with your Prometheus remote write endpoint.
    • Example: https://prometheus.grafana.net/api/prom/push
  2. YOUR_TEMPO_URL with your Tempo endpoint.
    • Example: tempo.grafana.net:443

​
2. Instrument your application

For Kubernetes deployments with zero-code instrumentation: 
apiVersion: openlit.io/v1alpha1
kind: AutoInstrumentation
metadata:
  name: prometheus-tempo-instrumentation
  namespace: default
spec:
  selector:
    matchLabels:
      instrument: "true"
  python:
    instrumentation:
      provider: "openlit"
      version: "latest"
  otlp:
    endpoint: "YOUR_OTELCOL_URL:4318"
    timeout: 30
  resource:
    environment: "production"
    serviceName: "my-ai-service"
Replace:
  1. YOUR_OTELCOL_URL:4318 with the HTTP endpoint of your OpenTelemetry Collector.
    • Example: http://otel-collector.monitoring.svc.cluster.local:4318 (for in-cluster collector)
    • Example: http://127.0.0.1:4318 (for local development)
Ensure your OpenTelemetry Collector is properly deployed and configured to route metrics to Prometheus and traces to Tempo.
Refer to the OpenLIT Operator Documentation for more advanced configurations and use cases.

​
3. Import the pre-built Dashboard

  1. Log into your Grafana Instance. To install Grafana, refer to the Official documentation.
  2. Add Data Sources: Make sure Prometheus and Tempo are added as data sources in Grafana. To add a new data source, follow the steps in the Official documentation.
  3. Import Dashboard: Once Prometheus and Tempo are available as data sources in Grafana:
    • Click Dashboards in the primary menu
    • Click New and select Import in the drop-down menu
    • Copy the dashboard JSON provided in the accordion below
    • Paste the dashboard JSON text directly into the text area
    • Click Import
    • Save the dashboard
{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": {
          "type": "grafana",
          "uid": "-- Grafana --"
        },
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "type": "dashboard"
      }
    ]
  },
  "editable": true,
  "fiscalYearStartMonth": 0,
  "graphTooltip": 1,
  "id": 0,
  "links": [
    {
      "asDropdown": false,
      "icon": "bolt",
      "includeVars": false,
      "keepTime": false,
      "tags": [],
      "targetBlank": true,
      "title": "OpenLIT Github",
      "tooltip": "Github",
      "type": "link",
      "url": "https://github.com/openlit/openlit"
    }
  ],
  "panels": [
    {
      "fieldConfig": {
        "defaults": {},
        "overrides": []
      },
      "gridPos": {
        "h": 5,
        "w": 12,
        "x": 0,
        "y": 0
      },
      "id": 2,
      "options": {
        "colorMode": "value",
        "graphMode": "none",
        "justifyMode": "auto",
        "orientation": "auto",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "limit": 80,
          "noValue": "N/A",
          "values": false
        },
        "text": "Total LLM Requests",
        "textMode": "value"
      },
      "plugin": {
        "id": "stat",
        "name": "Stat"
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "prometheus"
          },
          "editorMode": "builder",
          "expr": "sum(gen_ai_total_requests)",
          "legendFormat": "__auto",
          "range": true,
          "refId": "A"
        }
      ],
      "type": "stat"
    },
    {
      "fieldConfig": {
        "defaults": {},
        "overrides": []
      },
      "gridPos": {
        "h": 5,
        "w": 12,
        "x": 12,
        "y": 0
      },
      "id": 3,
      "options": {
        "colorMode": "value",
        "graphMode": "none",
        "justifyMode": "auto",
        "orientation": "auto",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "limit": 80,
          "noValue": "N/A",
          "values": false
        },
        "text": "Total VectorDB Requests",
        "textMode": "value"
      },
      "plugin": {
        "id": "stat",
        "name": "Stat"
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "prometheus"
          },
          "editorMode": "builder",
          "expr": "sum(db_total_requests)",
          "legendFormat": "__auto",
          "range": true,
          "refId": "A"
        }
      ],
      "type": "stat"
    }
  ],
  "schemaVersion": 38,
  "tags": [
    "openlit",
    "genai",
    "observability"
  ],
  "templating": {
    "list": []
  },
  "time": {
    "from": "now-6h",
    "to": "now"
  },
  "timepicker": {},
  "timezone": "",
  "title": "GenAI Observability - Prometheus + Tempo",
  "uid": "openlit-prometheus-tempo",
  "version": 1
}