tool-rocprof

name: tool-rocprof description: ROCm profiling workflow for AMD GPU kernels using rocprofv3 and rocprof-compute. Use when profiling hot kernels, collecting counters, diagnosing memory-vs-compute-vs-stall bottlenecks, reading Perfetto traces, or validating low-precision AMD kernels.

ROCm Profiling Workflow

Use this skill when you need runtime evidence before changing an AMD kernel.

Pick the Right Tool

Rule of thumb:

• Start with rocprof-compute if you want a fast high-level bottleneck classification.
• Use rocprofv3 when you need exact trace timing, kernel overlap, or raw counters.
• Use both when the bottleneck is ambiguous.

Minimal Workflow

1. Identify the hot kernel or focused benchmark from the project skill.
2. Run rocprof-compute on that focused benchmark to get Speed-of-Light and occupancy context.
3. If the result is unclear, run rocprofv3 for timeline gaps, counter detail, or multi-kernel overlap.
4. Classify the bottleneck as memory-bound, compute-bound, or stall / concurrency-bound.
5. Make one change, then re-profile the same workload.

Useful Commands

# High-level metric collection
rocprof-compute profile --name workload_name -- python run_kernel.py
rocprof-compute analyze -p workload_name/<GPU_NAME> --cli   # <GPU_NAME> subdir is auto-named by the detected GPU, e.g. MI300X / MI350X

# Discover available metric blocks / help
rocprof-compute profile -h
rocprof-compute profile --list-metrics
rocprof-compute analyze -h

# Trace or raw-counter collection
rocprofv3 --help
rocprofv3 --list-counters
rocprofv3 <trace-or-counter-options> -- python run_kernel.py

# Supporting commands
rocm-smi
rocminfo
hipcc --resource-usage kernel.hip

The exact rocprofv3 flags vary by ROCm release, so verify them with local --help before scripting.

First Classification

Use Speed-of-Light or equivalent GPU activity to decide whether microarchitecture tuning is even the right problem:

Do not start with micro-tuning if the GPU is mostly idle.

Memory vs Compute vs Stall

Memory-bound signals

• HBM bandwidth utilization is high and near the device roofline.
• L2 hit rate is poor or coalescing looks worse than expected.
• LDS bank conflicts or VMEM latency counters are elevated.

Typical actions:

• Improve coalescing and vectorization.
• Add or improve LDS tiling.
• Reduce redundant global traffic.
• Revisit traversal order and cache reuse.

Compute-bound signals

• MFMA utilization is high or should be high, but achieved throughput is still below expectation.
• Instruction mix is dominated by MFMA or should be, yet issue efficiency is poor.
• VALU work appears inflated relative to the intended GEMM / tensor-core style path.

Typical actions:

• Make sure the kernel is actually using MFMA.
• Revisit tile sizes, K unrolling, or matrix-instruction selection.
• Reduce dependency chains and improve overlap between loads and MFMA.
• For Triton, inspect AMD-specific compiler behavior before blaming the algorithm.

Stall / concurrency-bound signals

• HBM and MFMA are both below roofline even though the GPU is busy.
• Occupancy is low because of VGPR, LDS, barrier, or wave-limit pressure.
• Scratch usage or spill indicators are non-zero.

Typical actions:

• Reduce register pressure or live ranges.
• Reduce LDS footprint or change block shape.
• Re-check launch bounds and occupancy.
• Inspect synchronization patterns and pipeline bubbles.

Practical Thresholds

These are starting heuristics, not laws:

Always compare against the target GPU's known good/bad ranges for gfx942 / gfx950: use the project knowledge base consulted during ANALYZE plus the agent's own hardware knowledge.

Counter Families Worth Watching

Counter names vary across ROCm releases. Use local tool output to confirm the exact spelling.

AMD Low-Precision Validation

Profiling is not enough for low-precision work. Every FP8 / FP6 / FP4 / MXFP round should explicitly record:

• target architecture
• exact input and output format
• accumulator format
• scale format and scale granularity
• tolerance used
• whether the failure is conversion-, scale-, or math-related

Critical rule:

• gfx942 FP8 is typically FNUZ
• gfx950 FP8 is OCP

Never label a result only as "fp8" when multiple encodings are possible.

Validation strategy:

1. Validate conversion and packing separately.
2. Validate scale handling separately for block-scaled paths.
3. Compare kernel math against a higher-precision or dequantized reference.
4. Only then trust performance numbers.

Recommended Decision Loop

1. Capture baseline profile on the focused benchmark.
2. Write down the bottleneck classification and one hypothesis.
3. Change exactly one thing in the kernel.
4. Re-run correctness first.
5. Re-run the same profile / benchmark path.
6. Accept only if the data supports the hypothesis.

Related Files

• ../../../../rules/iteration_rules.mdc
• ../../../kernel-optimize/workflow/optimize-loop.md
• ../../SKILL.md (Primus-Turbo development guide)