> ## Documentation Index
> Fetch the complete documentation index at: https://docs.openlit.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Metrics Reference

> Complete list of all metrics exported by the OpenTelemetry GPU Collector

All metric names and attributes follow the [OpenTelemetry semantic conventions for hardware](https://opentelemetry.io/docs/specs/semconv/hardware/gpu/) and [system metrics](https://opentelemetry.io/docs/specs/semconv/system/).

## GPU Hardware Telemetry

Collected for each detected GPU on **Linux and Windows**. Availability depends on vendor, OS, and GPU model.

### Metrics

| Metric                           | Type          | Unit      | Description                                       | NVIDIA |            AMD           | Intel |
| -------------------------------- | ------------- | --------- | ------------------------------------------------- | :----: | :----------------------: | :---: |
| `hw.gpu.utilization`             | Gauge         | `1`       | GPU compute/encoder/decoder utilization (0.0–1.0) |   Yes  |            Yes           | Yes\* |
| `hw.gpu.memory.utilization`      | Gauge         | `1`       | Memory controller utilization (0.0–1.0)           |   Yes  |            Yes           |   -   |
| `hw.gpu.memory.limit`            | UpDownCounter | `By`      | Total GPU memory                                  |   Yes  |            Yes           | Yes\* |
| `hw.gpu.memory.usage`            | UpDownCounter | `By`      | Used GPU memory                                   |   Yes  |            Yes           | Yes\* |
| `hw.gpu.memory.free`             | UpDownCounter | `By`      | Free GPU memory                                   |   Yes  |            Yes           | Yes\* |
| `hw.gpu.temperature`             | Gauge         | `Cel`     | Die or memory temperature                         |   Yes  |            Yes           |  Yes  |
| `hw.gpu.fan_speed`               | Gauge         | `{rpm}`   | Fan speed                                         |   -†   |            Yes           | Yes\* |
| `hw.gpu.power.draw`              | Gauge         | `W`       | Current power draw                                |   Yes  |            Yes           |  Yes  |
| `hw.gpu.power.limit`             | Gauge         | `W`       | Power limit/cap                                   |   Yes  |            Yes           |  Yes  |
| `hw.gpu.energy.consumed`         | Counter       | `J`       | Cumulative energy consumed                        |   Yes  |            Yes           |  Yes  |
| `hw.gpu.clock.graphics`          | Gauge         | `MHz`     | Graphics/SM clock frequency                       |   Yes  |            Yes           | Yes\* |
| `hw.gpu.clock.memory`            | Gauge         | `MHz`     | Memory clock frequency                            |   Yes  |            Yes           |   -   |
| `hw.gpu.up`                      | Gauge         | `1`       | `1` when the device was successfully scraped      |   Yes  |            Yes           |  Yes  |
| `hw.gpu.allocated`               | Gauge         | `1`       | `1` when process memory or util ≥ threshold       |   Yes  |            Yes           |  Yes  |
| `hw.gpu.idle`                    | Gauge         | `1`       | Idle ratio (`1 - utilization`) when util known    |   Yes  |            Yes           | Yes\* |
| `hw.gpu.pcie.throughput`         | Gauge         | `By/s`    | PCIe RX/TX throughput                             |   Yes  |           Yes¶           | Soft‡ |
| `hw.gpu.interconnect.throughput` | Gauge         | `By/s`    | NVLink / XGMI aggregate throughput                |  Yes§  |           Yes¶           |   -   |
| `hw.gpu.throttled`               | Gauge         | `1`       | `1` when thermal/power throttle is active         |   Yes  |           Yes¶           | Yes\* |
| `hw.errors`                      | Counter       | `{error}` | ECC, PCIe, XID, and RAS error counts              |   Yes  | Yes (RAS + PCIe replay¶) |   -   |

\* Intel device util/VRAM/power/throttle/media use Level Zero Sysman when available (Linux/Windows); otherwise DRM/hwmon/PDH subset. Encoder/decoder util comes from media engine groups or Windows PDH VideoEncode/VideoDecode.

† NVIDIA NVML exposes fan speed as a percentage, not RPM, so `hw.gpu.fan_speed` is not emitted for NVIDIA.

‡ Soft-omitted when the vendor library/driver path does not expose the counter. Core util/memory/power still collect without those libraries. Windows AMD has no PCIe/XGMI throughput API (ADL).

§ NVLink on Linux and Windows NVIDIA when the GPU exposes enabled NVLink links (rates after the second sample).

¶ AMD Linux via `libamd_smi` (`amdsmi_get_gpu_pci_throughput`, `amdsmi_get_link_metrics`, `amdsmi_get_violation_status`) when present; soft-omitted on VMs/consumer cards without those counters. AMD media util (`MmActivity`) is exported as `hw.gpu.task=encoder` when the driver only exposes a combined VCN counter (decoder stays unset).

### Attributes

All GPU metrics carry these base attributes:

| Attribute         | Description                                 | Example                  |
| ----------------- | ------------------------------------------- | ------------------------ |
| `hw.id`           | Unique device identifier (required by spec) | `GPU-a1b2c3d4-5678-...`  |
| `hw.name`         | Product name                                | `NVIDIA A100-SXM4-80GB`  |
| `hw.vendor`       | Vendor name                                 | `nvidia`, `amd`, `intel` |
| `gpu.index`       | Zero-based device index                     | `0`, `1`                 |
| `gpu.pci_address` | PCI bus address                             | `0000:01:00.0`           |

Additional per-metric attributes:

| Metric                           | Attribute                                                                                           | Values                                                                               |
| -------------------------------- | --------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ |
| `hw.gpu.utilization`             | `hw.gpu.task`                                                                                       | `general`, `encoder`, `decoder`                                                      |
| `hw.gpu.temperature`             | `sensor`                                                                                            | `die`, `memory`                                                                      |
| `hw.gpu.pcie.throughput`         | `network.io.direction`                                                                              | `receive`, `transmit`                                                                |
| `hw.gpu.interconnect.throughput` | `network.io.direction`                                                                              | `receive`, `transmit`                                                                |
| `hw.gpu.interconnect.throughput` | `hw.gpu.interconnect.type`                                                                          | `nvlink`, `xgmi`, `other`                                                            |
| `hw.gpu.throttled`               | `hw.gpu.throttle_reasons`                                                                           | comma-separated reason labels                                                        |
| `hw.errors`                      | `error.type`                                                                                        | `corrected`, `uncorrected`, `pcie_replay`, `xid`, `ras_corrected`, `ras_uncorrected` |
| `hw.errors`                      | `hw.type`                                                                                           | `gpu`                                                                                |
| MIG devices                      | `gpu.mig.enabled`, `gpu.mig.device_id`, `gpu.parent.uuid`, `gpu.mig.instance_id`, `gpu.mig.profile` | Linux NVIDIA only (MIG is not available on Windows)                                  |

***

## System Metrics

Collected on all platforms (Linux, macOS, Windows) via [gopsutil](https://github.com/shirou/gopsutil). Follows the [OTel semantic conventions for system metrics](https://opentelemetry.io/docs/specs/semconv/system/system-metrics/).

| Metric                          | Type          | Unit          | Description                                | Attributes                                                                                                       |
| ------------------------------- | ------------- | ------------- | ------------------------------------------ | ---------------------------------------------------------------------------------------------------------------- |
| `system.cpu.utilization`        | Gauge         | `1`           | CPU utilization per logical core (0.0–1.0) | `cpu.logical_number`                                                                                             |
| `system.cpu.logical.count`      | UpDownCounter | `{cpu}`       | Number of logical CPU cores                |                                                                                                                  |
| `system.memory.usage`           | UpDownCounter | `By`          | Memory bytes by state                      | `system.memory.state`={used,free,cached,buffers}                                                                 |
| `system.memory.utilization`     | Gauge         | `1`           | Memory utilization (0.0–1.0)               |                                                                                                                  |
| `system.disk.io`                | Counter       | `By`          | Disk I/O bytes                             | `system.device`, `disk.io.direction`={read,write}                                                                |
| `system.disk.operations`        | Counter       | `{operation}` | Disk I/O operations                        | `system.device`, `disk.io.direction`={read,write}                                                                |
| `system.filesystem.usage`       | UpDownCounter | `By`          | Filesystem space by state                  | `system.device`, `system.filesystem.mountpoint`, `system.filesystem.type`, `system.filesystem.state`={used,free} |
| `system.filesystem.utilization` | Gauge         | `1`           | Filesystem utilization (0.0–1.0)           | `system.device`, `system.filesystem.mountpoint`, `system.filesystem.type`                                        |
| `system.network.io`             | Counter       | `By`          | Network I/O bytes                          | `network.interface.name`, `network.io.direction`={receive,transmit}                                              |
| `system.network.errors`         | Counter       | `{error}`     | Network errors                             | `network.interface.name`, `network.io.direction`={receive,transmit}                                              |

<Note>
  `system.memory.state` values `cached` and `buffers` are only reported on Linux. Loopback interfaces (`lo`, `lo0`) are excluded from network metrics.
</Note>

***

## Process Metrics

Self-monitoring of the collector process. Follows the [OTel semantic conventions for process metrics](https://opentelemetry.io/docs/specs/semconv/system/process-metrics/).

| Metric                               | Type          | Unit                | Description                         | Attributes               |
| ------------------------------------ | ------------- | ------------------- | ----------------------------------- | ------------------------ |
| `process.cpu.time`                   | Counter       | `s`                 | Cumulative CPU time                 | `cpu.mode`={user,system} |
| `process.cpu.utilization`            | Gauge         | `1`                 | CPU utilization (0.0–1.0)           |                          |
| `process.memory.usage`               | UpDownCounter | `By`                | Resident memory (RSS)               |                          |
| `process.memory.virtual`             | UpDownCounter | `By`                | Virtual memory size                 |                          |
| `process.thread.count`               | UpDownCounter | `{thread}`          | OS thread count                     |                          |
| `process.unix.file_descriptor.count` | UpDownCounter | `{file_descriptor}` | Open file descriptors (Linux/macOS) |                          |
| `process.runtime.go.goroutines`      | Gauge         | `{goroutine}`       | Go goroutine count                  |                          |
| `process.runtime.go.mem.heap_alloc`  | Gauge         | `By`                | Go heap memory allocated            |                          |

***

## Per-process / per-pod GPU attribution

Collected on each scrape from NVML (NVIDIA) or DRM fdinfo (AMD/Intel). Requires host PID visibility — Docker `--pid=host` / Compose `pid: host`, or Kubernetes `hostPID: true` — so `/proc` shows workload PIDs. Without it, device-level `hw.gpu.*` metrics still work, but per-process cmdline, state (including zombies), and GPU process metrics will be missing or incomplete.

These attributes support views like zombie-process triage: filter `process.state == "zombie"` and show `process.command_line`, `process.pid`, `process.owner`, and `process.uptime` (age).

| Metric                     | Type          | Unit | Description                              | NVIDIA | AMD | Intel |
| -------------------------- | ------------- | ---- | ---------------------------------------- | :----: | :-: | :---: |
| `process.gpu.memory.usage` | UpDownCounter | `By` | GPU memory used by a process on a device |   Yes  | Yes |  Yes  |
| `process.gpu.utilization`  | Gauge         | `1`  | Per-process GPU utilization (0.0–1.0)    |   Yes  | Yes |  Yes  |
| `process.uptime`           | Gauge         | `s`  | Process uptime for GPU-attributed PIDs   |   Yes  | Yes |  Yes  |

### Attributes

| Attribute                                                       | Description                                                                           |
| --------------------------------------------------------------- | ------------------------------------------------------------------------------------- |
| `process.pid`                                                   | Host process ID                                                                       |
| `process.executable.name`                                       | Short name from `/proc/<pid>/comm` (or Windows image basename)                        |
| `process.command_line`                                          | Truncated command line (see `OTEL_GPU_PROCESS_CMDLINE`)                               |
| `process.state`                                                 | `running`, `sleeping`, `zombie`, `stopped`, `dead`, `unknown`                         |
| `process.owner` / `process.owner.userid`                        | Username and UID/SID                                                                  |
| `process.workload.kind`                                         | `llm_inference`, `llm_training`, or `other`                                           |
| `process.workload.framework`                                    | `vllm`, `ollama`, `llama.cpp`, `sglang`, `tgi`, `triton`, `ray`, `pytorch`, `unknown` |
| `hw.id`, `hw.name`, `hw.vendor`, `gpu.index`, `gpu.pci_address` | Same device attrs as `hw.gpu.*`                                                       |
| `hw.gpu.task`                                                   | On utilization only: `general`, `encoder`, `decoder`                                  |
| `k8s.pod.uid`                                                   | From cgroup path when present                                                         |
| `k8s.pod.name`, `k8s.namespace.name`, `k8s.container.name`      | Via kubelet PodResources (GPU device join) and/or pod API lookup                      |
| `container.id`                                                  | Runtime container ID when resolvable                                                  |

<Note>
  `process.start_time` is not exported as a metric attribute (cardinality). Use `process.uptime` for lifetime. On MIG devices, NVML often provides **memory only** (no per-process SM util). Process util may not sum to device util (time-slicing / sampling windows differ). Use `process.state == zombie` to find defunct processes still holding GPU contexts.
</Note>

***

## eBPF CUDA Metrics

Enable with `OTEL_GPU_EBPF_ENABLED` (on by default on Linux). Soft-fails without caps. Discovers `libcudart.so*` from common install paths **and** from `/proc/*/maps` (with Docker `--pid=host` / Kubernetes `hostPID: true`), so fleet DaemonSets do not need a CUDA toolkit mount. Attaches uprobes/uretprobes for launches, alloc/free, memcpy, stream/device sync, and `cudaSetDevice`. AMD/Intel do not use this path — their per-process metrics come from DRM fdinfo.

### Activity metrics

| Metric                     | Type      | Unit       | Description                                                     | Attributes                                                                                     |
| -------------------------- | --------- | ---------- | --------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- |
| `gpu.kernel.launch.calls`  | Counter   | `{call}`   | CUDA kernel launch count                                        | `process.pid`, `process.executable.name`, `cuda.kernel.name`, `hw.id` / `gpu.index` when known |
| `gpu.kernel.grid.size`     | Histogram | `{thread}` | Total threads in grid per launch                                | same                                                                                           |
| `gpu.kernel.block.size`    | Histogram | `{thread}` | Threads per block per launch                                    | same                                                                                           |
| `gpu.kernel.shared_memory` | Histogram | `By`       | Dynamic shared memory per launch (`cudaLaunchKernel` sharedMem) | same                                                                                           |
| `gpu.memory.allocations`   | Counter   | `By`       | Bytes allocated via cudaMalloc                                  | `process.pid`, `process.executable.name`, `hw.id` / `gpu.index` when known                     |
| `gpu.memory.copies`        | Histogram | `By`       | Bytes per cudaMemcpyAsync                                       | `process.pid`, `cuda.memcpy.kind`, `hw.id` / `gpu.index` when known                            |

`hw.id` is set after `cudaSetDevice` for that thread, or immediately on single-GPU hosts. On multi-GPU hosts before `cudaSetDevice`, join via `process.pid` as before. **Linux/NVIDIA (CUDA) only** — same eBPF path for Kubernetes and non-Kubernetes when host PID is visible. AMD/Intel use device/process metrics via DRM fdinfo, not these kernel series.

### Stream-sync occupancy (Datadog-parity model)

These are **CPU-side model estimates**, not hardware SM occupancy (warps resident / max warps). Spans run from kernel launch to sync API return; thread-seconds are clamped to NVML CUDA core count and normalized so multi-process totals do not exceed device capacity.

| Metric                   | Type  | Unit      | Description                                                    |
| ------------------------ | ----- | --------- | -------------------------------------------------------------- |
| `process.gpu.core.usage` | Gauge | `{cores}` | Normalized avg CUDA cores from launch→sync thread-seconds      |
| `process.gpu.sm_active`  | Gauge | `1`       | Fraction of interval with any launch→sync span for the process |
| `gpu.core.limit`         | Gauge | `{cores}` | NVML CUDA core count                                           |
| `gpu.sm_active`          | Gauge | `1`       | Device-wide union of launch→sync spans                         |

**Prefer** NVML `process.gpu.utilization` for sampled SM%. Use occupancy metrics when correlating with Datadog Agent stream-sync behavior or when NVML process util is unavailable.

**Known limits:** no GPU completion signal (late sync inflates usage); CUDA Graphs / unhooked launch paths under-report; sync `cudaMemcpy` and default-stream sync are treated as device-wide (legacy default-stream semantics).

***

## Capability matrix

| Feature                                    |         NVIDIA         |     AMD    |    Intel   |
| ------------------------------------------ | :--------------------: | :--------: | :--------: |
| Device `hw.gpu.*`                          |           Yes          |     Yes    |   Partial  |
| `process.gpu.memory.usage` / `utilization` |          NVML          | DRM fdinfo | DRM fdinfo |
| Pod UID on process metrics                 |         cgroup         |   cgroup   |   cgroup   |
| eBPF activity + occupancy                  | Yes (default on Linux) |      —     |      —     |