Installation

Docker (recommended)

The easiest way to run the collector. The image is published to GitHub Container Registry and supports linux/amd64 and linux/arm64.

docker pull ghcr.io/openlit/otel-gpu-collector:latest

For per-process GPU attribution (cmdline, PID, zombie/process.state, owner), run with host PID namespace access: Docker --pid=host, Compose pid: host, or Kubernetes hostPID: true. Device-level hw.gpu.* metrics work without it.

Tag	Description
`latest`	Most recent release
`1.2.3`	Specific version
`1.2`	Latest patch of a minor version

Pre-built binaries

Download a binary for your platform from the GitHub Releases page. Binaries are available for:

Platform	Architecture
Linux	amd64, arm64, armv7
macOS	amd64 (Intel), arm64 (Apple Silicon)
Windows	amd64, arm64

# Example: Linux amd64
curl -L https://github.com/openlit/openlit/releases/latest/download/opentelemetry-gpu-collector-<version>-linux-amd64 \
    -o opentelemetry-gpu-collector
chmod +x opentelemetry-gpu-collector

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \
./opentelemetry-gpu-collector

GPU device and process metrics work on Linux and Windows (NVIDIA via NVML; AMD/Intel via sysfs on Linux or DXGI+PDH on Windows). eBPF CUDA tracing and occupancy are Linux only. On macOS the binary runs with host and process metrics only.

Verify the SHA256 checksum from the SHA256SUMS.txt file in the release:

sha256sum -c SHA256SUMS.txt --ignore-missing

Build from source

Requirements: Go 1.21+, CGO enabled (required for NVML on Linux).

git clone https://github.com/openlit/openlit.git
cd openlit/opentelemetry-gpu-collector
make build
./opentelemetry-gpu-collector

For eBPF CUDA tracing support, also run:

make setup-bpf   # installs bpftool, generates vmlinux.h
make generate    # runs bpf2go code generation
make build

Kubernetes DaemonSet

Run one collector per GPU node. Use the OpenTelemetry-recommended pattern: downward API → K8S_NODE_NAME → OTEL_RESOURCE_ATTRIBUTES with host.name and k8s.node.name. On GKE, AKS, and EKS the collector also auto-detects k8s.cluster.name, cloud.provider, and host.type (instance type) from the Kubernetes Node object and/or cloud metadata.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: otel-gpu-collector
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: otel-gpu-collector-node-get
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: otel-gpu-collector-node-get
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: otel-gpu-collector-node-get
subjects:
  - kind: ServiceAccount
    name: otel-gpu-collector
    namespace: default # change to your namespace
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: otel-gpu-collector
spec:
  selector:
    matchLabels:
      app: otel-gpu-collector
  template:
    metadata:
      labels:
        app: otel-gpu-collector
    spec:
      serviceAccountName: otel-gpu-collector
      hostPID: true
      containers:
        - name: otel-gpu-collector
          image: ghcr.io/openlit/otel-gpu-collector:latest
          env:
            - name: K8S_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: OTEL_SERVICE_NAME
              value: otel-gpu-collector
            - name: OTEL_RESOURCE_ATTRIBUTES
              value: "host.name=$(K8S_NODE_NAME),k8s.node.name=$(K8S_NODE_NAME)"
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: "http://otel-collector:4317"
            # Optional for on-prem / self-managed clusters:
            # - name: K8S_CLUSTER_NAME
            #   value: my-cluster
            # eBPF CUDA tracing is on by default on Linux; set false to disable:
            # - name: OTEL_GPU_EBPF_ENABLED
            #   value: "false"
          volumeMounts:
            - name: pod-resources
              mountPath: /var/lib/kubelet/pod-resources
              readOnly: true
            # AMD/Intel DRM:
            # - name: dri
            #   mountPath: /dev/dri
          securityContext:
            capabilities:
              add: ["SYS_ADMIN"] # or privileged / CAP_BPF+CAP_PERFMON for eBPF
      volumes:
        - name: pod-resources
          hostPath:
            path: /var/lib/kubelet/pod-resources
        # - name: dri
        #   hostPath:
        #     path: /dev/dri

hostPID: true (same role as Docker --pid=host) is required for per-process and per-pod GPU attribution so the collector can see workload PIDs under /proc. Without it, device-level hw.gpu.* metrics still work. The nodes/get ClusterRole enables OpenCost-style host.type / cloud.provider discovery from node labels without relying on IMDS (helpful when EKS hop limit is 1).

See Configuration for identity env vars and detection order.

Upgrade

Docker

docker pull ghcr.io/openlit/otel-gpu-collector:latest
docker stop otel-gpu-collector
docker rm otel-gpu-collector
# re-run with same flags

Binary

Download the new binary from the Releases page, replace the existing file, and restart the process.

Uninstall

Docker

docker stop otel-gpu-collector
docker rm otel-gpu-collector
docker rmi ghcr.io/openlit/otel-gpu-collector:latest

Binary

rm /usr/local/bin/opentelemetry-gpu-collector

Troubleshooting

No GPU metrics - collector starts but reports no hw.gpu.* metrics

Confirm the host has a supported GPU: lspci | grep -E 'VGA|3D|Display'
For NVIDIA: verify libnvidia-ml.so is present: ldconfig -p | grep nvidia-ml
For Docker: ensure --gpus all (NVIDIA) or --device /dev/dri (AMD/Intel) is passed
Check logs: docker logs otel-gpu-collector for "discovered GPU" entries

eBPF tracing not working

On Linux it is enabled by default; confirm it is not disabled via OTEL_GPU_EBPF_ENABLED=false
Check kernel version: uname -r (requires 5.8+)
The process needs CAP_BPF and CAP_PERFMON, or run as root
For Docker: --cap-add CAP_BPF --cap-add CAP_PERFMON, --pid=host, and --ulimit memlock=-1:-1 (BPF maps need locked memory; no CUDA mount needed)
For Kubernetes: hostPID: true plus BPF capabilities (or SYS_ADMIN); raise memlock if map create returns EPERM
If no CUDA process is running yet, the collector rescans /proc every 30s
Verify a workload has loaded cudart: grep libcudart /proc/*/maps 2>/dev/null | head

Connection refused / no data reaching backend

Verify OTEL_EXPORTER_OTLP_ENDPOINT is reachable from the container: curl http://<endpoint>/health
For Docker networking: use the host IP or service name, not localhost
Check if gRPC vs HTTP/protobuf matches the backend: set OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf for HTTP backends (port 4318)

Intel GPU not detected

Verify the i915 or Xe driver is loaded: lsmod | grep -E 'i915|xe'
Check DRM entries exist: ls /sys/class/drm/
Requires Linux kernel 5.10+ for sysfs metric exposure
Fan speed requires kernel 6.16+

Getting Started

Reference

GPU Vendors

Docker (recommended)

Tags

Pre-built binaries

Build from source

Kubernetes DaemonSet

Upgrade

Docker

Binary

Uninstall

Docker

Binary

Troubleshooting

​Docker (recommended)

​Tags

​Pre-built binaries

​Build from source

​Kubernetes DaemonSet

​Upgrade

​Docker

​Binary

​Uninstall

​Docker

​Binary

​Troubleshooting

Docker (recommended)

Tags

Pre-built binaries

Build from source

Kubernetes DaemonSet

Upgrade

Docker

Binary

Uninstall

Docker

Binary

Troubleshooting