Skip to main content
Implemented optional features are documented under Metrics, Configuration, and Semantic conventions. Plans below are collector-side only — no application SDK, no framework instrumentation dependency. Correlation stays on hw.id, process.pid, time, cgroup/pod attrs, and control-plane artifacts.

​
Recently added

​
Deep plans (out of scope today)

​
1. HIP / ROCm eBPF activity (AMD)

Goal: Parity with CUDA eBPF for AMD — launch counts, grid/block, memcpy, stream sync occupancy estimates — without ROCm app SDK hooks. Approach
  1. Inventory stable userspace symbols in libamdhip64.so / libhsa-runtime64.so (launch, memcpy, stream sync, set-device).
  2. Reuse the existing uprobe tracer skeleton (internal/ebpf) with a HIP event ABI parallel to CUDA EventRecord.
  3. Device resolve via /dev/kfd + DRM minor → UUID/hw.id (same attr contract as NVML).
  4. Occupancy model: launch→sync thread-seconds clamped to device compute-unit count from sysfs/amdsmi (not DCGM SM fields).
  5. Soft attach policy: only when AMD GPUs present; missing HIP libs/caps soft-skip (instrument faults still fatal).
Non-goals: Per-wavefront hardware occupancy; HIP graph per-node expand (phase 2 with CUDA graphs). Milestones: symbol map + stub tracer → single-GPU occupancy → multi-GPU set-device → docs + DaemonSet caps.

​
2. True GPU completion (vs launch→sync estimates)

Goal: End spans on device completion, not CPU sync return — without CUPTI/app SDK. Options (pick one primary) Recommended path: A for CUDA when events exist; keep launch→sync as fallback labeled gpu.measurement.source=ebpf_model; never rename to imply hardware SM occupancy. Exit criteria: Histogram of gpu.kernel.duration with completion=event|sync_fallback attribute; documented error bars.

​
3. CUDA Graph per-node kernels

Goal: Expand cudaGraphLaunch composite span into per-node kernel spans. Approach
  1. Uprobe cudaGraphGetNodes / instantiate / launch paths; build node-id → kernel-name map at graph instantiate time.
  2. On launch, emit N child spans or N duration samples with cuda.graph.id + cuda.graph.node_id.
  3. Cardinality: hash node names; cap distinct nodes per process (same budget as kernel-name cap).
  4. Without stable instantiate hooks: keep composite span (current behavior).
Non-goals: Replay/capture API coverage in v1; CUPTI graph callbacks.

​
4. cudaStreamWaitEvent / cross-stream edges

Goal: Model wait edges so occupancy does not count blocked wait time as SM-busy. Approach
  1. Uprobe wait-event + record-event; store event→stream edges in BPF map.
  2. Occupancy engine subtracts wait intervals from active thread-seconds.
  3. Attribute cuda.wait.event on duration samples when edge known.
Risk: Map size under high event churn — bounded LRU + drop counter self-metric.

​
5. Intel GPU util / memory without XPU Manager

Goal: Close Intel util/memory gap using collector-only sources. Approach
  1. Probe sysfs/hwmon + DRM fdinfo (already used for process mem).
  2. Optional: dynamic load of Level Zero loader for engine utilization if present (dlopen; soft-skip when missing).
  3. Document engine util as extension hw.gpu.engine.utilization with gpu.measurement.source=level0.
Non-goals: Shipping XPU Manager as a dependency.

​
6. Broader AMD DF / UMC PMU encodings

Goal: Stable hw.cpu.memory.io across EPYC SKUs beyond current uncore_imc* / amd_umc* sysfs discovery. Approach: Per-family event tables keyed by CPUID; discovery still prefers sysfs; tables fill gaps; self-metric when fallback used.

​
7. Native Intel PT AUX (no perf binary)

Goal: Capture PT without shelling out to perf record. Approach: perf_event_open + AUX mmap in internal/intelpt; keep duration/CPU/buffer caps; decode optional/offline. Fail-closed when PT enabled and open fails.

​
8. Proprietary fleet warehouses

Goal: Not in scope. Export is OTLP (and optional Prometheus). Fleet warehouses / custom backends stay with the customer’s collector pipeline. Collector work: Keep resource identity + hw.id joins solid; no proprietary export protocol.

​
9. Process-side correlation without an SDK

Goal: Richer join keys without instrumenting apps. Approach (collector-only)
  1. Read /proc/<pid>/environ for TRACEPARENT / OTEL_RESOURCE_ATTRIBUTES when present (best-effort, capped).
  2. Join GPU series to cgroup → pod via existing PodResources path.
  3. Optional: annotate on-demand profile artifacts with pid cmdline + container id only.
Non-goals: Generating W3C trace IDs inside the collector; requiring OpenLIT SDK in training/serving jobs.

​
Tracked (smaller)

​
Feature failure policy

Two outcomes at init — never conflated: TPU scrape endpoint misses and DCGM/RDC blank samples are handled at scrape time (self-metrics), not as startup faults.