Skip to main content
The OpenTelemetry GPU Collector is a lightweight, single-binary metrics collector written in Go. It exports GPU hardware telemetry, host system metrics, and process metrics via OpenTelemetry (OTLP) - with no Python dependencies, no DCGM daemon, and no vendor-specific agents. It is fully configured via standard OpenTelemetry environment variables and follows the OTel semantic conventions for hardware metrics.

​
Goals

  • OpenTelemetry-native - uses standard OTEL_* env vars, exports via OTLP gRPC or HTTP to any OTel-compatible backend
  • Cross-vendor GPU support - NVIDIA (NVML), AMD, and Intel on Linux and Windows from a single binary
  • OTel semantic conventions - hw.gpu.* metric names, hw.id / hw.name / hw.vendor attributes per spec
  • Zero dependencies - no DCGM, no Python, no CUDA toolkit needed at runtime for hardware metrics
  • Resilient - continues exporting host metrics even when no GPUs are present; retries GPU discovery every 30s

​
What it collects

GPU Hardware Telemetry

Utilization, memory, temperature, power, energy, clocks, PCIe/interconnect throughput, throttle, ECC/XID/RAS — NVIDIA, AMD, and Intel on Linux and Windows

Process + workload attribution

Per-process VRAM/util plus cmdline, OS state (zombie), owner, uptime, and LLM framework labels (vllm, ollama, …). Kubernetes pod name/namespace via PodResources

eBPF CUDA Tracing

Kernel launch counts, grid/block sizes, memory allocations, and memory copies via uprobes on libcudart (on by default on Linux; discovers libs from /proc with host PID — no CUDA mount)

​
GPU vendor support

​
Platform support

​
How it works

1

Discovers GPUs

On Linux, the collector scans /sys/bus/pci/devices/ for GPU class codes. On Windows, it loads nvml.dll when present and enumerates adapters via DXGI.
2

Initialises vendor backend

Each detected GPU is handed to its vendor-specific backend. NVIDIA uses NVML. Linux AMD/Intel read sysfs/hwmon; Windows AMD/Intel use DXGI memory totals and PDH for utilization and per-process attribution.
3

Registers OTel instruments

Observable gauge and counter instruments are registered with the OTel SDK meter. On each collection tick, the SDK calls back into the collector to read fresh values from each GPU.
4

Exports via OTLP

Metrics are exported via OTLP to any compatible backend - OpenLIT, Grafana, Datadog, New Relic, or a standard OTel Collector.

Quickstart

Get the collector running in under 5 minutes with Docker

Configuration

Full reference for all environment variables