Metrics Reference

All metric names and attributes follow the OpenTelemetry semantic conventions for hardware and system metrics. See the semconv registry for the full instrument list.

GPU Hardware Telemetry

Collected for each detected GPU on Linux and Windows. Availability depends on vendor, OS, and GPU model.

Device metrics

Metric	Type	Unit	Description	NVIDIA	AMD	Intel
`hw.gpu.utilization`	Gauge	`1`	GPU compute/encoder/decoder utilization (0.0–1.0)	Yes	Yes	Yes*
`hw.gpu.memory.utilization`	Gauge	`1`	Fraction of GPU memory used (usage / limit)	Yes	Yes	Yes*
`hw.gpu.memory.controller.utilization`	Gauge	`1`	Memory controller busy fraction (extension; NVML/AMD)	Yes	Yes	-
`hw.gpu.memory.limit`	UpDownCounter	`By`	Total GPU memory	Yes	Yes	Yes*
`hw.gpu.memory.usage`	UpDownCounter	`By`	Used GPU memory	Yes	Yes	Yes*
`hw.gpu.memory.free`	UpDownCounter	`By`	Free GPU memory	Yes	Yes	Yes*
`hw.temperature`	Gauge	`Cel`	Die or memory temperature	Yes	Yes	Yes
`hw.fan.speed`	Gauge	`rpm`	Fan speed	-†	Yes	Yes*
`hw.fan.speed_ratio`	Gauge	`1`	Fan speed as fraction of max	Yes	-	-
`hw.power`	Gauge	`W`	Current power draw	Yes	Yes	Yes
`hw.power.limit`	Gauge	`W`	Power limit/cap	Yes	Yes	Yes
`hw.energy`	Counter	`J`	Cumulative energy consumed	Yes	Yes	Yes
`hw.gpu.speed`	Gauge	`Hz`	Clock frequency (`hw.gpu.clock_domain`)	Yes	Yes	Yes*
`hw.gpu.io`	Counter	`By`	Cumulative PCIe I/O bytes	Yes	Yes¶	Soft‡
`hw.gpu.interconnect.io`	Counter	`By`	Cumulative NVLink / XGMI bytes	Yes§	Yes¶	-
`hw.status`	UpDownCounter	`1`	Hardware status (up / throttled states)	Yes	Yes	Yes
`hw.gpu.allocated`	Gauge	`1`	`1` when process memory or util ≥ threshold	Yes	Yes	Yes
`hw.gpu.idle`	Gauge	`1`	Idle ratio (`1 - utilization`) when util known	Yes	Yes	Yes*
`hw.errors`	Counter	`{error}`	ECC, PCIe, XID, and RAS error counts	Yes	Yes (RAS + PCIe replay¶)	-

* Intel support depends on driver (i915/Xe) and kernel version. † NVIDIA NVML reports fan speed as a percentage — use hw.fan.speed_ratio; hw.fan.speed (rpm) is omitted for NVIDIA. § Requires OTEL_GPU_INTERCONNECT_ENABLED (default true). ¶ AMD XGMI / PCIe when the driver exposes counters. ‡ Soft / best-effort when the kernel exposes the counters. * Intel device util/VRAM/power/throttle/media use Level Zero Sysman when available (Linux/Windows); otherwise DRM/hwmon/PDH subset. Encoder/decoder util comes from media engine groups or Windows PDH VideoEncode/VideoDecode. † NVIDIA NVML exposes fan speed as a percentage, not RPM, so hw.fan.speed is not emitted for NVIDIA — use hw.fan.speed_ratio instead. ‡ Soft-omitted when the vendor library/driver path does not expose the counter. Core util/memory/power still collect without those libraries. Windows AMD has no PCIe/XGMI throughput API (ADL). § NVLink on Linux and Windows NVIDIA when the GPU exposes enabled NVLink links (rates after the second sample). ¶ AMD Linux via libamd_smi (amdsmi_get_gpu_pci_throughput, amdsmi_get_link_metrics, amdsmi_get_violation_status) when present; soft-omitted on VMs/consumer cards without those counters. AMD media util (MmActivity) is exported as hw.gpu.task=encoder when the driver only exposes a combined VCN counter (decoder stays unset).

Attributes

All GPU metrics carry these base attributes:

Attribute	Description	Example
`hw.id`	Unique device identifier (required by spec)	`GPU-a1b2c3d4-5678-...`
`hw.name`	Product name	`NVIDIA A100-SXM4-80GB`
`hw.vendor`	Vendor name	`nvidia`, `amd`, `intel`
`gpu.index`	Zero-based device index	`0`, `1`
`gpu.pci_address`	PCI bus address	`0000:01:00.0`

Additional per-metric attributes:

Metric	Attribute	Values
`hw.gpu.utilization`	`hw.gpu.task`	`general`, `encoder`, `decoder`
`hw.temperature`	`hw.sensor_location`	`die`, `memory`
`hw.gpu.speed`	`hw.gpu.clock_domain`	`graphics`, `memory`, `sm`
`hw.gpu.io`	`network.io.direction`	`receive`, `transmit`
`hw.gpu.interconnect.io`	`network.io.direction`	`receive`, `transmit`
`hw.gpu.interconnect.io`	`hw.gpu.interconnect.type`	`nvlink`, `xgmi`, `other`
`hw.status`	status / throttle attrs	up / throttled + reason labels
`hw.errors`	`error.type`	`corrected`, `uncorrected`, `pcie_replay`, `xid`, `ras_corrected`, `ras_uncorrected`
`hw.errors`	`hw.type`	`gpu`
MIG devices	`gpu.mig.enabled`, `gpu.mig.device_id`, `gpu.parent.uuid`, `gpu.mig.instance_id`, `gpu.mig.profile`	Linux NVIDIA only (MIG is not available on Windows)
Partitions (RDC)	`hw.parent`, `hw.gpu.partition`	AMD compute partitions when available

System Metrics

Collected on all platforms (Linux, macOS, Windows) via gopsutil. Follows the OTel semantic conventions for system metrics.

Metric	Type	Unit	Description	Attributes
`system.cpu.utilization`	Gauge	`1`	CPU utilization per logical core (0.0–1.0)	`cpu.logical_number`
`system.cpu.logical.count`	UpDownCounter	`{cpu}`	Number of logical CPU cores
`system.memory.usage`	UpDownCounter	`By`	Memory bytes by state	`system.memory.state`=
`system.memory.utilization`	Gauge	`1`	Memory utilization (0.0–1.0)
`system.disk.io`	Counter	`By`	Disk I/O bytes	`system.device`, `disk.io.direction`=
`system.disk.operations`	Counter	`{operation}`	Disk I/O operations	`system.device`, `disk.io.direction`=
`system.filesystem.usage`	UpDownCounter	`By`	Filesystem space by state	`system.device`, `system.filesystem.mountpoint`, `system.filesystem.type`, `system.filesystem.state`=
`system.filesystem.utilization`	Gauge	`1`	Filesystem utilization (0.0–1.0)	`system.device`, `system.filesystem.mountpoint`, `system.filesystem.type`
`system.network.io`	Counter	`By`	Network I/O bytes	`network.interface.name`, `network.io.direction`=
`system.network.errors`	Counter	`{error}`	Network errors	`network.interface.name`, `network.io.direction`=

system.memory.state values cached and buffers are only reported on Linux. Loopback interfaces (lo, lo0) are excluded from network metrics.

Process Metrics

Self-monitoring of the collector process. Follows the OTel semantic conventions for process metrics.

Metric	Type	Unit	Description	Attributes
`process.cpu.time`	Counter	`s`	Cumulative CPU time	`cpu.mode`=
`process.cpu.utilization`	Gauge	`1`	CPU utilization (0.0–1.0)
`process.memory.usage`	UpDownCounter	`By`	Resident memory (RSS)
`process.memory.virtual`	UpDownCounter	`By`	Virtual memory size
`process.thread.count`	UpDownCounter	`{thread}`	OS thread count
`process.unix.file_descriptor.count`	UpDownCounter	`{file_descriptor}`	Open file descriptors (Linux/macOS)
`process.runtime.go.goroutines`	Gauge	`{goroutine}`	Go goroutine count
`process.runtime.go.mem.heap_alloc`	Gauge	`By`	Go heap memory allocated

Per-process / per-pod GPU attribution

Collected on each scrape from NVML (NVIDIA) or DRM fdinfo (AMD/Intel). Requires host PID visibility — Docker --pid=host / Compose pid: host, or Kubernetes hostPID: true — so /proc shows workload PIDs. Without it, device-level hw.gpu.* metrics still work, but per-process cmdline, state (including zombies), and GPU process metrics will be missing or incomplete. These attributes support views like zombie-process triage: filter process.state == "zombie" and show process.command_line, process.pid, process.owner, and process.uptime (age).

Metric	Type	Unit	Description	NVIDIA	AMD	Intel
`process.gpu.memory.usage`	UpDownCounter	`By`	GPU memory used by a process on a device	Yes	Yes	Yes
`process.gpu.memory.utilization`	Gauge	`1`	Process memory / device memory limit (0–1); omitted when limit unknown	Yes	Yes	Yes
`process.gpu.utilization`	Gauge	`1`	Per-process GPU utilization (0.0–1.0)	Yes	Yes	Yes
`process.uptime`	Gauge	`s`	Process uptime for GPU-attributed PIDs	Yes	Yes	Yes

Attributes

Attribute	Description
`process.pid`	Host process ID
`process.executable.name`	Short name from `/proc/<pid>/comm` (or Windows image basename)
`process.command_line`	Truncated command line (see `OTEL_GPU_PROCESS_CMDLINE`)
`process.state`	`running`, `sleeping`, `zombie`, `stopped`, `dead`, `unknown`
`process.owner` / `process.owner.userid`	Username and UID/SID
`process.workload.kind`	`llm_inference`, `llm_training`, or `other`
`process.workload.framework`	`vllm`, `ollama`, `llama.cpp`, `sglang`, `tgi`, `triton`, `ray`, `pytorch`, `unknown`
`hw.id`, `hw.name`, `hw.vendor`, `gpu.index`, `gpu.pci_address`	Same device attrs as `hw.gpu.*`
`hw.gpu.task`	On utilization only: `general`, `encoder`, `decoder`
`k8s.pod.uid`	From cgroup path when present
`k8s.pod.name`, `k8s.namespace.name`, `k8s.container.name`	Via kubelet PodResources (GPU device join) and/or pod API lookup
`container.id`	Runtime container ID when resolvable

process.start_time is not exported as a metric attribute (cardinality). Use process.uptime for lifetime. On MIG devices, NVML often provides memory only (no per-process SM util). Process util may not sum to device util (time-slicing / sampling windows differ). Use process.state == zombie to find defunct processes still holding GPU contexts.

eBPF CUDA Metrics

Enable with OTEL_GPU_EBPF_ENABLED (on by default on Linux for serving+ / all modes; off in light). Soft-fails without caps. Discovers libcudart.so* (CUDA runtime API) and libcuda.so* (CUDA driver API) from common install paths and from /proc/*/maps (with Docker --pid=host / Kubernetes hostPID: true), so fleet DaemonSets do not need a CUDA toolkit mount. Driver-API probes extend coverage to frameworks that never load libcudart (llama.cpp, Ollama). Attaches uprobes/uretprobes for launches (cudaLaunchKernel, cudaLaunchKernelExC, cooperative, cuLaunchKernel, cuLaunchKernelEx), graph replays (cudaGraphLaunch, cuGraphLaunch), alloc/free, memcpy, stream/device/event sync, and cudaSetDevice. PIDs that map libcudart skip driver launch/graph events so runtime wrappers are not double-counted. AMD/Intel do not use this path — their per-process metrics come from DRM fdinfo.

Activity metrics

Metric	Type	Unit	Description	Attributes
`gpu.kernel.launch.calls`	Counter	`{call}`	CUDA kernel launch count (runtime and driver APIs)	`process.pid`, `process.executable.name`, `cuda.kernel.name`, `hw.id` / `gpu.index` when known
`gpu.kernel.grid.size`	Histogram	`{thread}`	Total threads in grid per launch	same
`gpu.kernel.block.size`	Histogram	`{thread}`	Threads per block per launch	same
`gpu.kernel.shared_memory`	Histogram	`By`	Dynamic shared memory per launch (`cudaLaunchKernel` sharedMem)	same
`gpu.kernel.duration`	Histogram	`s`	Model estimate of launch→sync duration (shared store with occupancy; cardinality-capped names)	same + optional `cuda.launch.kind=graph`
`gpu.graph.launch.calls`	Counter	`{call}`	CUDA graph replay invocations (`cudaGraphLaunch` / `cuGraphLaunch`). Counts replays, not the kernels executed inside each replay — that number is not observable at this API-tracing layer	`process.pid`, `process.executable.name`, `hw.id` / `gpu.index` when known
`gpu.memory.allocations`	Counter	`By`	Bytes allocated via cudaMalloc	`process.pid`, `process.executable.name`, `hw.id` / `gpu.index` when known
`gpu.memory.copies`	Histogram	`By`	Bytes per cudaMemcpyAsync	`process.pid`, `cuda.memcpy.kind`, `hw.id` / `gpu.index` when known

hw.id is set after cudaSetDevice for that thread, or immediately on single-GPU hosts. On multi-GPU hosts before cudaSetDevice, join via process.pid as before. Linux/NVIDIA (CUDA) only — same eBPF path for Kubernetes and non-Kubernetes when host PID is visible. AMD/Intel use device/process metrics via DRM fdinfo, not these kernel series.

Stream-sync occupancy model

These are CPU-side model estimates, not hardware SM occupancy (warps resident / max warps). Spans run from kernel launch to sync API return; thread-seconds are clamped to NVML CUDA core count and normalized so multi-process totals do not exceed device capacity.

Metric	Type	Unit	Description
`process.gpu.core.usage`	Gauge	`{cores}`	Normalized avg CUDA cores from launch→sync thread-seconds
`process.gpu.estimated.sm_active`	Gauge	`1`	Fraction of interval with any launch→sync span (`gpu.measurement.source=ebpf_model`)
`hw.gpu.core.limit`	Gauge	`{cores}`	NVML CUDA core count
`hw.gpu.estimated.sm_active`	Gauge	`1`	Device-wide union of launch→sync spans

Prefer NVML process.gpu.utilization for sampled SM%. These estimates are not DCGM hw.gpu.sm.utilization / hw.gpu.sm.occupancy. Use them when NVML process util is unavailable or when correlating launch→sync activity with device util. Known limits: no GPU completion signal (late sync inflates usage); cudaGraphLaunch / cuGraphLaunch is one composite span and gpu.graph.launch.calls counts replays, not kernels (no per-node kernels inside a graph); cudaStreamWaitEvent wait edges are not modeled; sync cudaMemcpy and default-stream sync are treated as device-wide (legacy default-stream semantics).

DCGM profiling (optional)

Enable with OTEL_GPU_DCGM_ENABLED=true. Soft-fails when libdcgm is missing. DCGM is optional — continuous clocks (graphics / sm / memory), util, and power work from NVML alone. Metrics carry gpu.measurement.source=dcgm. Set OTEL_GPU_DCGM_PREFER=true so DCGM owns overlapping IO / power / util / graphics clock / idle (NVML suppresses those series — single producer). Memory clock stays on NVML.

Metric	Type	Unit	Attributes
`hw.gpu.engine.utilization`	Gauge	`1`	`hw.gpu.engine`
`hw.gpu.sm.utilization`	Gauge	`1`	DCP SM active
`hw.gpu.sm.occupancy`	Gauge	`1`	DCP SM occupancy
`hw.gpu.pipe.utilization`	Gauge	`1`	`hw.gpu.pipe`=`tensor\|fp16\|fp32\|fp64`
`hw.gpu.memory.bandwidth.utilization`	Gauge	`1`
`hw.gpu.io` / `hw.gpu.interconnect.io`	Counter	`By`	Prefer only; rate-integrated PCIe / NVLink
`hw.gpu.speed`	Gauge	`Hz`	Prefer only; `hw.gpu.clock_domain=graphics` (field 100)
`hw.gpu.idle`	Gauge	`1`	Prefer only; derived from Prefer util
`hw.power` / `hw.gpu.utilization` / `hw.gpu.memory.controller.utilization`	Gauge	Prefer only	Fields 155 / 203 / 204 (204 → controller util)
`openlit.collector.gpu.dcgm.sample_valid`	UpDownCounter	`1`	0 on blank sample

Control plane: POST /v1/dcgm/pause and POST /v1/dcgm/resume (loopback OTEL_GPU_CONTROL_ADDR).

AMD RDC profiling (optional)

Enable with OTEL_GPU_RDC_ENABLED=true. Soft-fails when librdc is missing. Metrics carry gpu.measurement.source=rdc. Partition samples may set hw.parent.

Metric	Type	Unit	Notes
`hw.gpu.sm.utilization`	Gauge	`1`	SM/CU active
`hw.gpu.sm.occupancy`	Gauge	`1`	CU / SM occupancy
`hw.gpu.pipe.utilization`	Gauge	`1`	`fp16` / `fp32` / `fp64` from EVAL_FLOPS percent
`hw.gpu.simd.utilization`	Gauge	`1`	SIMD utilization

Kineto on-demand profiling (optional)

Enable with OTEL_GPU_KINETO_ENABLED=true. Does not emit continuous metrics. Use the control API:

POST /v1/profile/gpu — match GPU PIDs and write on-demand libkineto config
Requires OTEL_GPU_CONTROL_ADDR (loopback by default; set OTEL_GPU_CONTROL_ALLOW_REMOTE=true + token for multi-node)

Self-metric: openlit.collector.profile.requests. Multi-node fan-out (no always-on collector cost):

go run ./cmd/profile-fanout --hosts node1,node2 --port 1919 --token "$TOKEN" --duration-ms 500
# or SLURM:
go run ./cmd/profile-fanout --job-id 12345 --port 1919 --token "$TOKEN"

Intel PT on-demand (optional)

Enable with OTEL_HOST_INTEL_PT_ENABLED=true and a control address. Does not run continuously.

POST /v1/profile/cpu/pt with {"duration_ms":500}
Requires Linux intel_pt PMU and perf in PATH
Caps: duration, CPU count, AUX buffer pages (see configuration)

Fan-out: profile-fanout --profile cpu-pt ...

CPU PMU / uncore memory IO (optional)

Enable with OTEL_HOST_PMU_ENABLED=true.

Metric	Notes
`hw.cpu.instructions` / `hw.cpu.cycles` / cache / branch / TLB	Core PMU
`hw.cpu.memory.io`	When `memory_bandwidth` or `uncore` is in `OTEL_HOST_PMU_EVENTS`; IMC CAS×64 bytes

TPU scrape (optional)

Enable with OTEL_TPU_ENABLED=true. Scrapes a Prometheus-format TPU endpoint (OTEL_TPU_ENDPOINT, default http://127.0.0.1:2112/metrics) and re-exports allowlisted series. Soft-fails when the endpoint is unreachable.

NIC / RDMA (optional)

Flag	Default	Metrics surface
`OTEL_HOST_NIC_ENABLED`	`false`	Per-NIC hardware counters beyond basic `system.network.*`
`OTEL_HOST_RDMA_ENABLED`	`false`	RDMA device counters (`OTEL_HOST_RDMA_COUNTERS` allowlist)

Soft-omitted when sysfs/counters are unavailable.

Capability matrix

Feature	NVIDIA	AMD	Intel
Device `hw.gpu.` / spec `hw.`	Yes	Yes	Partial
`process.gpu.memory.usage` / `memory.utilization` / `utilization`	NVML	DRM fdinfo	DRM fdinfo
Pod UID on process metrics	cgroup	cgroup	cgroup
eBPF activity + occupancy	Yes (default on Linux)	—	—
DCGM profiling	Optional	—	—
RDC profiling	—	Optional (soft-skip without librdc)	—

Getting Started

Reference

GPU Vendors

Metrics Reference

GPU Hardware Telemetry

Device metrics

Attributes

System Metrics

Process Metrics

Per-process / per-pod GPU attribution

Attributes

eBPF CUDA Metrics

Activity metrics

Stream-sync occupancy model

DCGM profiling (optional)

AMD RDC profiling (optional)

Kineto on-demand profiling (optional)

Intel PT on-demand (optional)

CPU PMU / uncore memory IO (optional)

TPU scrape (optional)

NIC / RDMA (optional)

Capability matrix

​GPU Hardware Telemetry

​Device metrics

​Attributes

​System Metrics

​Process Metrics

​Per-process / per-pod GPU attribution

​Attributes

​eBPF CUDA Metrics

​Activity metrics

​Stream-sync occupancy model

​DCGM profiling (optional)

​AMD RDC profiling (optional)

​Kineto on-demand profiling (optional)

​Intel PT on-demand (optional)

​CPU PMU / uncore memory IO (optional)

​TPU scrape (optional)

​NIC / RDMA (optional)

​Capability matrix

GPU Hardware Telemetry

Device metrics

Attributes

System Metrics

Process Metrics

Per-process / per-pod GPU attribution

Attributes

eBPF CUDA Metrics

Activity metrics

Stream-sync occupancy model

DCGM profiling (optional)

AMD RDC profiling (optional)

Kineto on-demand profiling (optional)

Intel PT on-demand (optional)

CPU PMU / uncore memory IO (optional)

TPU scrape (optional)

NIC / RDMA (optional)

Capability matrix