Metrics Reference

All metric names and attributes follow the OpenTelemetry semantic conventions for hardware and system metrics.

GPU Hardware Telemetry

Collected for each detected GPU on Linux and Windows. Availability depends on vendor, OS, and GPU model.

Metrics

Metric	Type	Unit	Description	NVIDIA	AMD	Intel
`hw.gpu.utilization`	Gauge	`1`	GPU compute/encoder/decoder utilization (0.0–1.0)	Yes	Yes	Yes*
`hw.gpu.memory.utilization`	Gauge	`1`	Memory controller utilization (0.0–1.0)	Yes	Yes	-
`hw.gpu.memory.limit`	UpDownCounter	`By`	Total GPU memory	Yes	Yes	Yes*
`hw.gpu.memory.usage`	UpDownCounter	`By`	Used GPU memory	Yes	Yes	Yes*
`hw.gpu.memory.free`	UpDownCounter	`By`	Free GPU memory	Yes	Yes	Yes*
`hw.gpu.temperature`	Gauge	`Cel`	Die or memory temperature	Yes	Yes	Yes
`hw.gpu.fan_speed`	Gauge	`{rpm}`	Fan speed	-†	Yes	Yes*
`hw.gpu.power.draw`	Gauge	`W`	Current power draw	Yes	Yes	Yes
`hw.gpu.power.limit`	Gauge	`W`	Power limit/cap	Yes	Yes	Yes
`hw.gpu.energy.consumed`	Counter	`J`	Cumulative energy consumed	Yes	Yes	Yes
`hw.gpu.clock.graphics`	Gauge	`MHz`	Graphics/SM clock frequency	Yes	Yes	Yes*
`hw.gpu.clock.memory`	Gauge	`MHz`	Memory clock frequency	Yes	Yes	-
`hw.gpu.up`	Gauge	`1`	`1` when the device was successfully scraped	Yes	Yes	Yes
`hw.gpu.allocated`	Gauge	`1`	`1` when process memory or util ≥ threshold	Yes	Yes	Yes
`hw.gpu.idle`	Gauge	`1`	Idle ratio (`1 - utilization`) when util known	Yes	Yes	Yes*
`hw.gpu.pcie.throughput`	Gauge	`By/s`	PCIe RX/TX throughput	Yes	Yes¶	Soft‡
`hw.gpu.interconnect.throughput`	Gauge	`By/s`	NVLink / XGMI aggregate throughput	Yes§	Yes¶	-
`hw.gpu.throttled`	Gauge	`1`	`1` when thermal/power throttle is active	Yes	Yes¶	Yes*
`hw.errors`	Counter	`{error}`	ECC, PCIe, XID, and RAS error counts	Yes	Yes (RAS + PCIe replay¶)	-

* Intel device util/VRAM/power/throttle/media use Level Zero Sysman when available (Linux/Windows); otherwise DRM/hwmon/PDH subset. Encoder/decoder util comes from media engine groups or Windows PDH VideoEncode/VideoDecode. † NVIDIA NVML exposes fan speed as a percentage, not RPM, so hw.gpu.fan_speed is not emitted for NVIDIA. ‡ Soft-omitted when the vendor library/driver path does not expose the counter. Core util/memory/power still collect without those libraries. Windows AMD has no PCIe/XGMI throughput API (ADL). § NVLink on Linux and Windows NVIDIA when the GPU exposes enabled NVLink links (rates after the second sample). ¶ AMD Linux via libamd_smi (amdsmi_get_gpu_pci_throughput, amdsmi_get_link_metrics, amdsmi_get_violation_status) when present; soft-omitted on VMs/consumer cards without those counters. AMD media util (MmActivity) is exported as hw.gpu.task=encoder when the driver only exposes a combined VCN counter (decoder stays unset).

Attributes

All GPU metrics carry these base attributes:

Attribute	Description	Example
`hw.id`	Unique device identifier (required by spec)	`GPU-a1b2c3d4-5678-...`
`hw.name`	Product name	`NVIDIA A100-SXM4-80GB`
`hw.vendor`	Vendor name	`nvidia`, `amd`, `intel`
`gpu.index`	Zero-based device index	`0`, `1`
`gpu.pci_address`	PCI bus address	`0000:01:00.0`

Additional per-metric attributes:

Metric	Attribute	Values
`hw.gpu.utilization`	`hw.gpu.task`	`general`, `encoder`, `decoder`
`hw.gpu.temperature`	`sensor`	`die`, `memory`
`hw.gpu.pcie.throughput`	`network.io.direction`	`receive`, `transmit`
`hw.gpu.interconnect.throughput`	`network.io.direction`	`receive`, `transmit`
`hw.gpu.interconnect.throughput`	`hw.gpu.interconnect.type`	`nvlink`, `xgmi`, `other`
`hw.gpu.throttled`	`hw.gpu.throttle_reasons`	comma-separated reason labels
`hw.errors`	`error.type`	`corrected`, `uncorrected`, `pcie_replay`, `xid`, `ras_corrected`, `ras_uncorrected`
`hw.errors`	`hw.type`	`gpu`
MIG devices	`gpu.mig.enabled`, `gpu.mig.device_id`, `gpu.parent.uuid`, `gpu.mig.instance_id`, `gpu.mig.profile`	Linux NVIDIA only (MIG is not available on Windows)

System Metrics

Collected on all platforms (Linux, macOS, Windows) via gopsutil. Follows the OTel semantic conventions for system metrics.

Metric	Type	Unit	Description	Attributes
`system.cpu.utilization`	Gauge	`1`	CPU utilization per logical core (0.0–1.0)	`cpu.logical_number`
`system.cpu.logical.count`	UpDownCounter	`{cpu}`	Number of logical CPU cores
`system.memory.usage`	UpDownCounter	`By`	Memory bytes by state	`system.memory.state`=
`system.memory.utilization`	Gauge	`1`	Memory utilization (0.0–1.0)
`system.disk.io`	Counter	`By`	Disk I/O bytes	`system.device`, `disk.io.direction`=
`system.disk.operations`	Counter	`{operation}`	Disk I/O operations	`system.device`, `disk.io.direction`=
`system.filesystem.usage`	UpDownCounter	`By`	Filesystem space by state	`system.device`, `system.filesystem.mountpoint`, `system.filesystem.type`, `system.filesystem.state`=
`system.filesystem.utilization`	Gauge	`1`	Filesystem utilization (0.0–1.0)	`system.device`, `system.filesystem.mountpoint`, `system.filesystem.type`
`system.network.io`	Counter	`By`	Network I/O bytes	`network.interface.name`, `network.io.direction`=
`system.network.errors`	Counter	`{error}`	Network errors	`network.interface.name`, `network.io.direction`=

system.memory.state values cached and buffers are only reported on Linux. Loopback interfaces (lo, lo0) are excluded from network metrics.

Process Metrics

Self-monitoring of the collector process. Follows the OTel semantic conventions for process metrics.

Metric	Type	Unit	Description	Attributes
`process.cpu.time`	Counter	`s`	Cumulative CPU time	`cpu.mode`=
`process.cpu.utilization`	Gauge	`1`	CPU utilization (0.0–1.0)
`process.memory.usage`	UpDownCounter	`By`	Resident memory (RSS)
`process.memory.virtual`	UpDownCounter	`By`	Virtual memory size
`process.thread.count`	UpDownCounter	`{thread}`	OS thread count
`process.unix.file_descriptor.count`	UpDownCounter	`{file_descriptor}`	Open file descriptors (Linux/macOS)
`process.runtime.go.goroutines`	Gauge	`{goroutine}`	Go goroutine count
`process.runtime.go.mem.heap_alloc`	Gauge	`By`	Go heap memory allocated

Per-process / per-pod GPU attribution

Collected on each scrape from NVML (NVIDIA) or DRM fdinfo (AMD/Intel). Requires host PID visibility — Docker --pid=host / Compose pid: host, or Kubernetes hostPID: true — so /proc shows workload PIDs. Without it, device-level hw.gpu.* metrics still work, but per-process cmdline, state (including zombies), and GPU process metrics will be missing or incomplete. These attributes support views like zombie-process triage: filter process.state == "zombie" and show process.command_line, process.pid, process.owner, and process.uptime (age).

Metric	Type	Unit	Description	NVIDIA	AMD	Intel
`process.gpu.memory.usage`	UpDownCounter	`By`	GPU memory used by a process on a device	Yes	Yes	Yes
`process.gpu.utilization`	Gauge	`1`	Per-process GPU utilization (0.0–1.0)	Yes	Yes	Yes
`process.uptime`	Gauge	`s`	Process uptime for GPU-attributed PIDs	Yes	Yes	Yes

Attributes

Attribute	Description
`process.pid`	Host process ID
`process.executable.name`	Short name from `/proc/<pid>/comm` (or Windows image basename)
`process.command_line`	Truncated command line (see `OTEL_GPU_PROCESS_CMDLINE`)
`process.state`	`running`, `sleeping`, `zombie`, `stopped`, `dead`, `unknown`
`process.owner` / `process.owner.userid`	Username and UID/SID
`process.workload.kind`	`llm_inference`, `llm_training`, or `other`
`process.workload.framework`	`vllm`, `ollama`, `llama.cpp`, `sglang`, `tgi`, `triton`, `ray`, `pytorch`, `unknown`
`hw.id`, `hw.name`, `hw.vendor`, `gpu.index`, `gpu.pci_address`	Same device attrs as `hw.gpu.*`
`hw.gpu.task`	On utilization only: `general`, `encoder`, `decoder`
`k8s.pod.uid`	From cgroup path when present
`k8s.pod.name`, `k8s.namespace.name`, `k8s.container.name`	Via kubelet PodResources (GPU device join) and/or pod API lookup
`container.id`	Runtime container ID when resolvable

process.start_time is not exported as a metric attribute (cardinality). Use process.uptime for lifetime. On MIG devices, NVML often provides memory only (no per-process SM util). Process util may not sum to device util (time-slicing / sampling windows differ). Use process.state == zombie to find defunct processes still holding GPU contexts.

eBPF CUDA Metrics

Enable with OTEL_GPU_EBPF_ENABLED (on by default on Linux). Soft-fails without caps. Discovers libcudart.so* from common install paths and from /proc/*/maps (with Docker --pid=host / Kubernetes hostPID: true), so fleet DaemonSets do not need a CUDA toolkit mount. Attaches uprobes/uretprobes for launches, alloc/free, memcpy, stream/device sync, and cudaSetDevice. AMD/Intel do not use this path — their per-process metrics come from DRM fdinfo.

Activity metrics

Metric	Type	Unit	Description	Attributes
`gpu.kernel.launch.calls`	Counter	`{call}`	CUDA kernel launch count	`process.pid`, `process.executable.name`, `cuda.kernel.name`, `hw.id` / `gpu.index` when known
`gpu.kernel.grid.size`	Histogram	`{thread}`	Total threads in grid per launch	same
`gpu.kernel.block.size`	Histogram	`{thread}`	Threads per block per launch	same
`gpu.kernel.shared_memory`	Histogram	`By`	Dynamic shared memory per launch (`cudaLaunchKernel` sharedMem)	same
`gpu.memory.allocations`	Counter	`By`	Bytes allocated via cudaMalloc	`process.pid`, `process.executable.name`, `hw.id` / `gpu.index` when known
`gpu.memory.copies`	Histogram	`By`	Bytes per cudaMemcpyAsync	`process.pid`, `cuda.memcpy.kind`, `hw.id` / `gpu.index` when known

hw.id is set after cudaSetDevice for that thread, or immediately on single-GPU hosts. On multi-GPU hosts before cudaSetDevice, join via process.pid as before. Linux/NVIDIA (CUDA) only — same eBPF path for Kubernetes and non-Kubernetes when host PID is visible. AMD/Intel use device/process metrics via DRM fdinfo, not these kernel series.

Stream-sync occupancy (Datadog-parity model)

These are CPU-side model estimates, not hardware SM occupancy (warps resident / max warps). Spans run from kernel launch to sync API return; thread-seconds are clamped to NVML CUDA core count and normalized so multi-process totals do not exceed device capacity.

Metric	Type	Unit	Description
`process.gpu.core.usage`	Gauge	`{cores}`	Normalized avg CUDA cores from launch→sync thread-seconds
`process.gpu.sm_active`	Gauge	`1`	Fraction of interval with any launch→sync span for the process
`gpu.core.limit`	Gauge	`{cores}`	NVML CUDA core count
`gpu.sm_active`	Gauge	`1`	Device-wide union of launch→sync spans

Prefer NVML process.gpu.utilization for sampled SM%. Use occupancy metrics when correlating with Datadog Agent stream-sync behavior or when NVML process util is unavailable. Known limits: no GPU completion signal (late sync inflates usage); CUDA Graphs / unhooked launch paths under-report; sync cudaMemcpy and default-stream sync are treated as device-wide (legacy default-stream semantics).

Capability matrix

Feature	NVIDIA	AMD	Intel
Device `hw.gpu.*`	Yes	Yes	Partial
`process.gpu.memory.usage` / `utilization`	NVML	DRM fdinfo	DRM fdinfo
Pod UID on process metrics	cgroup	cgroup	cgroup
eBPF activity + occupancy	Yes (default on Linux)	—	—

Getting Started

Reference

GPU Vendors

Metrics Reference

GPU Hardware Telemetry

Metrics

Attributes

System Metrics

Process Metrics

Per-process / per-pod GPU attribution

Attributes

eBPF CUDA Metrics

Activity metrics

Stream-sync occupancy (Datadog-parity model)

Capability matrix

​GPU Hardware Telemetry

​Metrics

​Attributes

​System Metrics

​Process Metrics

​Per-process / per-pod GPU attribution

​Attributes

​eBPF CUDA Metrics

​Activity metrics

​Stream-sync occupancy (Datadog-parity model)

​Capability matrix

GPU Hardware Telemetry

Metrics

Attributes

System Metrics

Process Metrics

Per-process / per-pod GPU attribution

Attributes

eBPF CUDA Metrics

Activity metrics

Stream-sync occupancy (Datadog-parity model)

Capability matrix