diff --git a/README.md b/README.md index 0b522c339..4b939f84b 100644 --- a/README.md +++ b/README.md @@ -347,15 +347,21 @@ The `exec` command supports the following options: ```bash $ cvs --help -usage: cvs [-h] [--version] {run,list,generate,monitor,exec} ... +usage: cvs [-h] [--version] + {copy-config,generate,list,man,run,scp,monitor,exec} ... Cluster Validation Suite (CVS) positional arguments: - {run,list,generate,monitor,exec} - run Run a specific test (wrapper over pytest) - list List available tests + {copy-config,generate,list,man,run,scp,monitor,exec} + Available commands + copy-config List or copy config files from CVS package. Lists + configs if --output not specified. generate Generate configuration files or templates + list List available tests + man Explain the config parameters for a test + run Run a specific test (wrapper over pytest) + scp Copy files to all nodes in the cluster monitor Run cluster monitoring scripts exec Execute a command on all nodes in the cluster @@ -364,6 +370,28 @@ options: --version show program's version number and exit ``` +Config parameters are documented by `cvs man` rather than by comments inside +the sample config files: + +```bash +$ cvs man rccl_perf nic_model + +cvs man rccl_perf nic_model +================================================================================ + +Sweeps RCCL collectives across the cluster and checks bandwidth and latency. + + rccl.cvs_params + ------------------------------------------------------------------------------ + + • nic_model [string, default 'ainic'] + NIC family used to select model-specific validations. + example: 'thor', 'ainic', 'connectx' + +================================================================================ +Total: 1 parameter +``` + ```bash $ cvs run --help usage: cvs run [-h] --cluster_file CLUSTER_FILE --config_file diff --git a/cvs/lib/utils_lib.py b/cvs/lib/utils_lib.py index 55e2c811d..769e11724 100644 --- a/cvs/lib/utils_lib.py +++ b/cvs/lib/utils_lib.py @@ -311,7 +311,8 @@ def replace_in_string(value, path=""): error_msg += f"{'=' * 70}\n" error_msg += "ACTION REQUIRED:\n" error_msg += "Please edit your configuration file and replace all the '' placeholders\n" - error_msg += "with an appropriate value before running the tests.\n" + error_msg += "with an appropriate value before running the tests.\n\n" + error_msg += "Run 'cvs man ' to see what each config parameter expects.\n" error_msg += f"{'=' * 70}\n" log.error("%s", error_msg) diff --git a/docs/how-to/browse-config-parameters.rst b/docs/how-to/browse-config-parameters.rst new file mode 100644 index 000000000..9120c1538 --- /dev/null +++ b/docs/how-to/browse-config-parameters.rst @@ -0,0 +1,126 @@ +.. meta:: + :description: Look up what each CVS test config parameter means using cvs man + :keywords: CVS, man, config, configuration, parameters, defaults, documentation + +***************************************** +Look up test configuration parameters +***************************************** + +CVS provides a ``man`` command that explains every configuration parameter a +test accepts: its type, its real default, any constraints, and an example +value. Use it instead of reading the sample config files, which deliberately +carry no inline documentation. + +The reference is generated by introspecting the pydantic schemas in +``cvs/parsers/``, so it reflects what the code actually reads. In particular +the default shown is the value that takes effect when you omit the key, which +is not always the value that happens to appear in the sample config. + +List what is documented +======================= + +Run ``cvs man`` with no arguments to see which tests have a parameter +reference: + +.. code:: bash + + cvs man + +.. code:: text + + Config parameter references + ================================================================================ + + • rccl_perf + Sweeps RCCL collectives across the cluster and checks bandwidth and latency. + +Explain a test +============== + +Pass a test name to see every parameter it accepts, grouped by config section: + +.. code:: bash + + cvs man rccl_perf + +.. code:: text + + rccl.cvs_params + ------------------------------------------------------------------------------ + + • nic_model [string, default 'ainic'] + NIC family used to select model-specific validations. + example: 'thor', 'ainic', 'connectx' + +Explain a single parameter +========================== + +Add a parameter name to narrow the output. The name is matched exactly first, +then as a substring of the dotted path, so a partial name also works: + +.. code:: bash + + cvs man rccl_perf nic_model + cvs man preflight_checks rdma + +Machine-readable output +======================= + +``--json`` emits the same reference as JSON, for tooling that generates +configs or validates them: + +.. code:: bash + + cvs man rccl_perf --json + +.. code:: json + + { + "parameters": [ + { + "path": "rccl.cvs_params.nic_model", + "type": "string", + "required": false, + "description": "NIC family used to select model-specific validations.", + "default": "ainic", + "examples": ["thor", "ainic", "connectx"] + } + ], + "test": "rccl_perf", + "config_files": ["/path/to/cvs/input/config_file/rccl/rccl_config.json"] + } + +``config_files`` is the real filesystem path to the shipped sample, resolved +against wherever ``cvs`` is installed — not a path relative to your current +directory. + +Reading the output +================== + +.. list-table:: + :widths: 3 7 + :header-rows: 1 + + * - Notation + - Meaning + * - ``required`` + - The test fails if the key is missing; there is no usable default. + * - ``default `` + - The value used when the key is omitted, taken from the code. + * - ``constraints:`` + - Validation bounds, such as ``>= 2, <= 512``. + * - ```` in a path + - A mapping level whose keys you choose, as in + ``model_params.single_node...precision``. + * - ``[]`` in a path + - A list level, as in ``rvs.tests[].name``. + +.. note:: + + Not every suite has a reference yet. ``cvs man`` reports how many test + suites are still undocumented; those are being migrated one suite at a time. + +.. tip:: + + When a run aborts because a ```` placeholder was left in your + config, ``cvs man `` tells you what that parameter expects. diff --git a/docs/index.rst b/docs/index.rst index 18905a809..ef8062ba6 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -25,6 +25,7 @@ The component public repository is located at `https://github.com/ROCm/cvs ` + * :doc:`Look up test configuration parameters ` * :doc:`Run ad-hoc cluster-wide commands ` * :doc:`Copy files and directories to cluster nodes ` * :doc:`Monitor the health of GPU clusters ` diff --git a/docs/sphinx/_toc.yml.in b/docs/sphinx/_toc.yml.in index 10373bbcd..604e80114 100644 --- a/docs/sphinx/_toc.yml.in +++ b/docs/sphinx/_toc.yml.in @@ -17,6 +17,8 @@ subtrees: entries: - file: how-to/run-cvs-tests title: Run tests + - file: how-to/browse-config-parameters + title: Look up test configuration parameters - file: how-to/run-with-containers title: Run tests with the container backend - file: how-to/run-vllm-benchmarks diff --git a/test_cli.sh b/test_cli.sh index b56663a18..7dcaab3e4 100755 --- a/test_cli.sh +++ b/test_cli.sh @@ -1,6 +1,6 @@ #!/bin/bash # Test script for CVS package -# This script tests the cvs list, generate, copy-config, monitor, and exec commands +# This script tests the cvs list, generate, copy-config, man, monitor, and exec commands # Use the CVS environment variable if set, otherwise default to 'cvs' CVS="${CVS:-cvs}" @@ -85,6 +85,17 @@ done # Test: cvs copy-config --list run_test "cvs copy-config --list" "$CVS copy-config --list" +# Test: cvs man (list tests with a config parameter reference) +run_test "cvs man" "$CVS man" + +# Test: cvs man for each documented test, in both output formats +echo "Testing: cvs man for each documented test" +echo "====================" +for test in $($CVS man | grep "•" | awk '{print $2}'); do + run_test "cvs man $test" "$CVS man $test" + run_test "cvs man $test --json" "$CVS man $test --json" +done + # Test: cvs monitor (list all monitors) run_test "cvs monitor" "$CVS monitor"