Configuration

The collector is configured entirely via environment variables. All variables follow the standard OpenTelemetry SDK configuration spec where applicable.

Collector mode

OTEL_GPU_COLLECTOR_MODE selects a use-case preset (all | light | serving | training | deep). Default: all. Presets only fill unset feature envs; explicit flags always win. See Modes for the full matrix, cost notes, and hosting checklist.

Variable	Default	Description
`OTEL_GPU_COLLECTOR_MODE`	`all`	Feature preset. Invalid values warn and fall back to `all`

Environment Variables

Variable	Default	Description
`OTEL_EXPORTER_OTLP_ENDPOINT`	(required)	OTLP endpoint URL, e.g. `http://localhost:4318`
`OTEL_EXPORTER_OTLP_HEADERS`		Auth headers in `key=val,key2=val2` format
`OTEL_EXPORTER_OTLP_PROTOCOL`	`grpc`	`grpc` or `http/protobuf`
`OTEL_SERVICE_NAME`	(unset)	Optional. Only set if your backend requires `service.name`. This agent is not an application service — product identity uses `telemetry.distro.name=opentelemetry-gpu-collector` and `telemetry.distro.version`
`OTEL_RESOURCE_ATTRIBUTES`	`deployment.environment=default`	Resource attributes (`key=val,...`). Prefer setting `host.name`, `k8s.*`, `cloud.provider`, `host.type`, `cloud.region` here — overrides auto-detect. Org tags such as `team` or `datacenter` are not auto-detected; set them here.
`OTEL_METRIC_EXPORT_INTERVAL`	`60000`	Metric polling interval in milliseconds. For self-hosted LLM hosts, `15000` is recommended
`OTEL_GPU_EBPF_ENABLED`	mode-dependent (`true` on Linux for `serving`+)	eBPF CUDA activity tracing + stream-sync occupancy (Linux/NVIDIA only). Discovers `libcudart` and `libcuda` via FS + `/proc` maps (fleet-friendly with host PID). Soft-fails without caps; set `false` to disable
`OPENLIT_HOST_METRICS`	`true`	Collect system + collector-process host metrics. Set `false` for a GPU-only light footprint
`OTEL_GPU_FS_TYPES_EXCLUDE`	`squashfs,erofs,iso9660,cramfs,romfs,cd9660,CDFS,UDF`	Filesystem types excluded from `system.filesystem.*` metrics (case-sensitive). Default skips image-based and optical filesystems that are 100% full by construction (e.g. snap mounts). Set to an empty string to report all types
`OTEL_GPU_PROCESS_CMDLINE`	`true`	Export truncated `process.command_line` on GPU process metrics
`OTEL_GPU_PROCESS_CMDLINE_MAX_LEN`	`512`	Max characters for `process.command_line`
`OTEL_GPU_ALLOCATED_UTIL_THRESHOLD`	`0.05`	Util threshold (0–1) used with process memory for `hw.gpu.allocated`
`OTEL_GPU_INTERCONNECT_ENABLED`	`true`	Export NVLink/XGMI interconnect throughput when available
`K8S_NODE_NAME`		Kubernetes node name via downward API (`spec.nodeName`). Also accepts `OTEL_RESOURCE_ATTRIBUTES_NODE_NAME` (Operator) or legacy `NODE_NAME`
`K8S_CLUSTER_NAME`		Explicit cluster name when cloud auto-detect fails (on-prem). Alias: `OPENLIT_K8S_CLUSTER_NAME`. Only applied in Kubernetes
`OPENLIT_K8S_NODE_LOOKUP`	`true`	When `false`, skip `GET /api/v1/nodes/$K8S_NODE_NAME` for instance-type / provider discovery
`OPENLIT_K8S_POD_RESOURCES`	`true` in K8s	Use kubelet PodResources socket; joins GPU UUID → `k8s.pod.name` / namespace / container
`OPENLIT_K8S_POD_LOOKUP`	`true` in K8s when `K8S_NODE_NAME` is set	List pods on this node via the Kubernetes API (needs `list` on pods) for UID/container-id joins
`POD_RESOURCES_SOCKET`	OS default	Override kubelet PodResources socket / named pipe path
`OPENLIT_CLOUD_DETECT`	`true`	When `false`, skip AWS/GCP/Azure IMDS probes (recommended on bare metal to avoid link-local timeouts)
`OTEL_GPU_DCGM_ENABLED`	mode-dependent (`true` for `serving`+)	Optional NVIDIA DCGM profiling (soft-skipped without libdcgm / when DCGM cannot init)
`OTEL_GPU_DCGM_LIB_PATH`	`/lib64/libdcgm.so`	Path to libdcgm
`OTEL_GPU_DCGM_ADDRESS`		Empty = embedded; else hostengine address
`OTEL_GPU_DCGM_INTERVAL`	`10`	DCGM sample interval in seconds (warns if < 10s; DCP multiplexing)
`OTEL_GPU_DCGM_PREFER`	mode-dependent (`true` for `serving`+)	When true and DCGM is up: DCGM owns overlapping `hw.gpu.io` / interconnect / power / util / SM clock / `hw.gpu.memory.controller.utilization`; NVML suppresses those series. Spec `hw.gpu.memory.utilization` (usage/limit) stays on the vendor path
`OTEL_GPU_DCGM_FIELDS`	`50,100,155,203,204,1001–1012`	Comma-separated DCGM field IDs (`155`=power, `203`=GPU util, `204`=mem util, `1001–1012`=DCP)
`OTEL_GPU_RDC_ENABLED`	mode-dependent (`true` for `all`)	Optional AMD RDC profiling (soft-skipped without librdc)
`OTEL_GPU_RDC_LIB_PATH`	`librdc.so`	Path / soname for librdc
`OTEL_GPU_KINETO_ENABLED`	mode-dependent (`true` for `deep`/`all`)	On-demand Kineto IPC + control profiling
`OTEL_GPU_PROMETHEUS_ADDR`		When set (e.g. `:9464`), serve Prometheus `/metrics` alongside OTLP
`OTEL_GPU_CONTROL_ADDR`	mode-dependent (`127.0.0.1:1919` for `deep`/`all`)	Loopback control HTTP; empty disables
`OTEL_GPU_CONTROL_TOKEN`		Optional Bearer token for control API (required if allow-remote)
`OTEL_GPU_CONTROL_ALLOW_REMOTE`	`false`	Allow non-loopback bind for multi-node fan-out (requires token)
`OTEL_HOST_CPU_HIGHRES`	mode-dependent (`true` for `deep`/`all`)	100ms CPU ring buffer for `GET /v1/cpu/highres` (requires control addr; Linux)
`OTEL_HOST_PMU_ENABLED`	mode-dependent (`true` for `training`+)	Core CPU PMU counters (`hw.cpu.instructions`, …)
`OTEL_HOST_PMU_EVENTS`	`instructions,cycles` or +`memory_bandwidth` in training+	Include `memory_bandwidth` / `uncore` for IMC `hw.cpu.memory.io`
`OTEL_HOST_INTEL_PT_ENABLED`	mode-dependent (`true` for `deep`/`all`)	On-demand Intel PT via `POST /v1/profile/cpu/pt` (needs `perf` + intel_pt)
`OTEL_HOST_INTEL_PT_MAX_DURATION_MS`	`2000`	Hard cap for PT capture duration
`OTEL_HOST_INTEL_PT_MAX_CPUS`	`4`	Max CPUs traced per PT capture
`OTEL_HOST_INTEL_PT_MAX_BUFFER_PAGES`	`64`	AUX mmap pages passed to `perf -m`
`OTEL_TPU_ENABLED`	mode-dependent (`true` for `all`)	Scrape TPU Prometheus endpoint
`OTEL_HOST_KVM_ENABLED`	mode-dependent (`true` for `all`)	KVM exit / VM metrics
`OTEL_HOST_INTERRUPTS_ENABLED`	mode-dependent (`true` for `all`)	Host interrupt counters
`OTEL_HOST_NIC_ENABLED`	mode-dependent (`true` for `training`+)	Per-NIC `hw.network.` (ethtool/sysfs). When enabled, `system.network.` is skipped to avoid duplicates
`OTEL_HOST_RDMA_ENABLED`	mode-dependent (`true` for `training`+)	RDMA device counters

Host, Kubernetes, and cloud identity

Resource attributes follow OpenTelemetry semantic conventions for host, K8s, and cloud:

Attribute	When set
`host.name`	Always (from `OTEL_RESOURCE_ATTRIBUTES`, or `K8S_NODE_NAME` → GCE hostname in K8s → OS hostname)
`k8s.node.name`	When a node env is set (`K8S_NODE_NAME` / Operator / legacy `NODE_NAME`)
`k8s.cluster.name`	In Kubernetes only: `K8S_CLUSTER_NAME` / `OTEL_RESOURCE_ATTRIBUTES` → GKE / AKS / EKS metadata
`cloud.provider`	Auto: K8s `Node.spec.providerID` → AWS/GCP/Azure IMDS → DMI vendor hint
`cloud.platform`	e.g. `aws_eks`, `gcp_kubernetes_engine`, `aws_ec2`, `azure_aks`
`host.type`	Instance type (e.g. `g4dn.xlarge`, `a2-highgpu-1g`, `Standard_NC6s_v3`) from node labels or IMDS
`cloud.region` / `cloud.availability_zone`	Topology labels or IMDS
`cloud.account.id`	AWS account / GCP project / Azure subscription when available
`host.id`	Cloud instance ID from providerID or IMDS
`openlit.host.type.source`	Which tier filled `host.type`: `k8s_label`, `imds`, or `dmi`

Discovery order (later tiers only fill missing fields; never blocks startup; never emits "unknown"):

Explicit OTEL_RESOURCE_ATTRIBUTES (wins via SDK WithFromEnv)
Kubernetes Node GET (needs get on nodes + K8S_NODE_NAME) — OpenCost-style labels / providerID
Parallel AWS / GCP / Azure instance metadata (short timeout)
DMI sys_vendor hint for provider only

Use cloud.provider + host.type + cloud.region as join keys for future UI cost attribution. The collector does not compute prices. Kubernetes is detected via KUBERNETES_SERVICE_HOST. On GKE, AKS, and EKS the cluster name is read from the instance metadata service (short timeout; failures are ignored). Self-managed clusters should set k8s.cluster.name via OTEL_RESOURCE_ATTRIBUTES or K8S_CLUSTER_NAME.

Recommended Kubernetes DaemonSet (OTel-native)

env:
  - name: K8S_NODE_NAME
    valueFrom:
      fieldRef:
        fieldPath: spec.nodeName
  - name: OTEL_RESOURCE_ATTRIBUTES
    value: "host.name=$(K8S_NODE_NAME),k8s.node.name=$(K8S_NODE_NAME)"
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "http://otel-collector:4317"
  # Optional when cloud auto-detect is unavailable (on-prem / self-managed):
  # - name: K8S_CLUSTER_NAME
  #   value: my-cluster
  # Or append to OTEL_RESOURCE_ATTRIBUTES:
  #   ,k8s.cluster.name=my-cluster,cloud.provider=aws,host.type=g4dn.xlarge,cloud.region=us-east-1

If you only set K8S_NODE_NAME (without packing it into OTEL_RESOURCE_ATTRIBUTES), the collector still maps it to host.name and k8s.node.name automatically. For K8s node label / providerID discovery, grant the DaemonSet ServiceAccount:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: otel-gpu-collector-node-get
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: otel-gpu-collector-node-get
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: otel-gpu-collector-node-get
subjects:
  - kind: ServiceAccount
    name: otel-gpu-collector
    namespace: monitoring

On EKS, pods without hostNetwork may fail IMDSv2 when the node httpPutResponseHopLimit is 1. Prefer K8s node lookup (above), raise the hop limit to 2+, or run with hostNetwork: true. Timeouts are soft-fail and do not stop the collector.

Common configurations

Minimal - send to a local OTel Collector

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \
./opentelemetry-gpu-collector

Production - with service name, environment, and auth header

OTEL_SERVICE_NAME=gpu-worker \
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production,team=ml \
OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.example.com:4317 \
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer\ my-token \
OTEL_METRIC_EXPORT_INTERVAL=30000 \
./opentelemetry-gpu-collector

HTTP/protobuf instead of gRPC

OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf \
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 \
./opentelemetry-gpu-collector

Disable eBPF CUDA tracing

On Linux, eBPF CUDA tracing is on by default. It discovers libcudart and libcuda from the filesystem and from /proc/*/maps (no CUDA volume mount required when Docker --pid=host / Kubernetes hostPID: true is set). Soft-fails without CAP_BPF + CAP_PERFMON (or root). Containers typically also need --ulimit memlock=-1:-1 so BPF maps can be created. Set false to skip:

OTEL_GPU_EBPF_ENABLED=false \
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \
./opentelemetry-gpu-collector

eBPF activity/occupancy is NVIDIA/CUDA only. AMD and Intel use the same host-PID process attribution for DRM fdinfo metrics; they do not need libcudart.

Notes

OTEL_METRIC_EXPORT_INTERVAL is in milliseconds per the OTel spec. For a 30-second interval, set 30000.
deployment.environment is extracted from OTEL_RESOURCE_ATTRIBUTES and attached as a resource attribute. Any key-value pairs in OTEL_RESOURCE_ATTRIBUTES are also forwarded to the OTel SDK resource via resource.WithFromEnv().
If OTEL_EXPORTER_OTLP_ENDPOINT is not set, the collector starts but no metrics are exported. Check the logs for a warning.
Auto-detected host.name / k8s.* attributes are logged at startup as resolved resource identity.

Getting Started

Reference

GPU Vendors

Collector mode

Environment Variables

Host, Kubernetes, and cloud identity

Recommended Kubernetes DaemonSet (OTel-native)

Common configurations

Minimal - send to a local OTel Collector

Production - with service name, environment, and auth header

HTTP/protobuf instead of gRPC

Disable eBPF CUDA tracing

Notes

​Collector mode

​Environment Variables

​Host, Kubernetes, and cloud identity

​Recommended Kubernetes DaemonSet (OTel-native)

​Common configurations

​Minimal - send to a local OTel Collector

​Production - with service name, environment, and auth header

​HTTP/protobuf instead of gRPC

​Disable eBPF CUDA tracing

​Notes

Collector mode

Environment Variables

Host, Kubernetes, and cloud identity

Recommended Kubernetes DaemonSet (OTel-native)

Common configurations

Minimal - send to a local OTel Collector

Production - with service name, environment, and auth header

HTTP/protobuf instead of gRPC

Disable eBPF CUDA tracing

Notes