Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 51 additions & 27 deletions systems/frontier_user_guide.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3313,33 +3313,6 @@ Valgrind4hpc is available on Frontier under all compiler families:
Additional information about Valgrind4hpc usage can be found on the `HPE Cray Programming Environment User Guide Page <https://support.hpe.com/hpesc/public/docDisplay?docId=a00115110en_us&page=Debug_Applications_With_valgrind4hpc_To_Find_Common_Errors.html>`__.


Omnitrace/ROCm Systems Profiler
-------------------------------

OLCF provides installations of AMD's `Omnitrace <https://github.com/AMDResearch/omnitrace>`_ and the new re-branded `ROCm Systems Profiler <https://rocm.docs.amd.com/projects/rocprofiler-systems/en/latest/index.html>`_ profiling tools on Frontier.
AMD provides documentation on the usage of the ROCm Systems Profiler at `<https://rocm.docs.amd.com/projects/rocprofiler-systems/en/latest/index.html>`_.
This section details the installation and common pitfalls of the ``omnitrace`` and ``rocprofiler-systems`` modules on Frontier.

A ROCm module must be loaded before being able to view or load either the ``omnitrace`` or ``rocprofiler-systems`` modules.
As a rule of thumb, always load the profiler's module last (but specifically after you load a ROCm module).
If you load a new version of ROCm, you will need to re-load the profiler module.

To use ``omnitrace`` or ``rocprofiler-systems``, you may use the following commands

.. code::

module load rocm
module load omnitrace # or module load rocprofiler-systems

``rocprofiler-systems`` does not support all available ROCm modules. Please consult the table below for the compatibility matrix.

+---------------------+--------------------------+
| rocprofiler-compute | Supported ROCm Versions |
+=====================+==========================+
| 1.0.1 | 6.3.*, 6.4.* |
+---------------------+--------------------------+


Profiling Applications
======================

Expand Down Expand Up @@ -3461,6 +3434,57 @@ More detailed information on HPCToolkit can be found in the `HPCToolkit User's M
Getting Started with the ROCm Profiler
--------------------------------------

ROCm Systems Profiler
^^^^^^^^^^^^^^^^^^^^^

OLCF provides installations of AMD's `ROCm Systems Profiler <https://rocm.docs.amd.com/projects/rocprofiler-systems/en/latest/index.html>`_ profiling tool on Frontier.
AMD provides documentation on the usage of the ROCm Systems Profiler at `<https://rocm.docs.amd.com/projects/rocprofiler-systems/en/latest/index.html>`_.

We recommend using a newer ROCm and its accompanying ``rocprofiler-systems`` binaries, which ship alongside the ROCm installation under its ``bin`` folder, to benefit from the latest features and bug fixes.
If you would like to build ``rocprofiler-systems`` from source, follow the instructions at `<https://rocm.docs.amd.com/projects/rocprofiler-systems/en/latest/install/build.html>`_.

A ROCm module must be loaded before being able to use the ``rocprofiler-systems`` binaries.
Comment on lines +3443 to +3446

.. code::

module load rocm

**Looking up configuration options.**
Use ``rocprof-sys-avail`` to look up configuration settings, hardware counters, and data collection components. For example, ``rocprof-sys-avail --list-domains`` lists the API trace domains supported for profiling.

.. code:: bash

rocprof-sys-avail --list-domains

Read more about configuring runtime options at `<https://rocm.docs.amd.com/projects/rocprofiler-systems/en/latest/how-to/configuring-runtime-options.html>`_.

**Example usage.**

.. code:: bash

rocprof-sys-sample -- ./myapp
srun -n 2 rocprof-sys-run -- ./mpi-example
srun -N 2 -n 16 --ntasks-per-node=8 --gpus-per-node=8 --gpu-bind=closest rocprof-sys-run -- ./mpi-example
Comment on lines +3463 to +3467

**Using predefined presets.**
ROCm Systems Profiler provides predefined presets that simplify configuring the profiler for common workload scenarios. Read more at `<https://rocm.docs.amd.com/projects/rocprofiler-systems/en/latest/how-to/using-preset-profiles.html>`_.

Balanced profiling with moderate overhead and comprehensive data:

.. code:: bash

rocprof-sys-run --preset=balanced -- ./myapp

Optimized for MPI, OpenMP, and compute-intensive applications:

.. code:: bash

srun -n 4 rocprof-sys-sample --preset=trace-hpc -- ./mpi_app

**Understanding the output.**
Several output files can be generated, as detailed at `<https://rocm.docs.amd.com/projects/rocprofiler-systems/en/latest/how-to/understanding-rocprof-sys-output.html>`_.
The traces are generated as Perfetto traces and/or a rocPD database, which can be viewed in the `Perfetto UI <https://ui.perfetto.dev/>`_ and the `ROCm Optiq <https://github.com/ROCm/roc-optiq>`_ visualizer respectively.

Rocprof v1 (For ROCm<=6.5)
^^^^^^^^^^^^^^^^^^^^^^^^^^

Expand Down