Content Testing (contest)

This is a repository of publicly-available tests used for testing ComplianceAsCode/content on Red Hat Enterprise Linux.

Terminology

• FMF - Flexible Metadata Format, a test metadata format used by TMT

• TMT - Test Management Tool, a framework and a related CLI tool for running tests, see also user docs here or Under The Hood which explains the basic much better

• "test" is a FMF object with a test: in its YAML definiton, ie. /hardening/oscap/stig

  • (In this case, one directory /hardening/oscap defines multiple tests, all sharing the same source code, parametrized using environment variables in main.fmf.)

• "result" is a piece of data reported by a test, containing

  • name - either a test name, or a test name with something appended to it, ie. /hardening/oscap/stig or /hardening/oscap/stig/some_rule_name/etc
  • status - one of pass, fail, info, warn or error
  • note - additional freeform text details about the result
  • log - a list of logs associated with the result

Parameters

• CONTEST_VERBOSE

  • Set to an integer value to control the verbosity of reported results. This applies only to sub-results (/something after a test name), results for tests themselves (as seen by TMT) are always reported.
    • 0 outputs only fail and error
    • 1 (default) is fail, error and warn
    • 2 or greater to output everything

• CONTEST_WAIVER_DIR

  • Specify a relative path to a waiver directory containing waiver files.
    • The directory itself is traversed recursively (may contain further sub-directories).
  • All files and directories are read in a locale-specific sorted order, and their contents combined to a final list of waiver rules.
  • Files and directories starting with . are ignored.
  • Defaults to conf/waivers.

• CONTEST_LEAVE_GUEST_RUNNING

  • Set to 1 to break gurantees provided by class Guest(), that is make the context manager not honor __exit__ by leaving running guests (VMs) behind.
  • This is useful for debugging a failing OpenSCAP rule as you get the running virtual environment, as it was scanned, without an extra OS startup.
  • SSH instructions will be provided on stdout (python log output).
    • Alternatively, use ./contest-sshvm [vm-name], which discovers the per-guest QEMU user-mode networking port and connects through 127.0.0.1.
  • However any tests that use more than 1 VM and rely on a shut-down VM state between two context-managed blocks, will break.
    • Because the VM was left running after the first context manager block.
    • Fortunately, no such test currently exists (the use case is rare).

• CONTEST_VERBATIM_RESULTS

  • Set to 1 to avoid waiving known failures, leaving results exactly as tests reported them.
  • Useful when you want the actual result of ie. /per-rule/from-env, rather than the waived one.

• CONTEST_STRICT_WAIVERS

  • Set to 1 to force all waivers to be strict=True.
  • See WAIVERS.md for more.

• CONTEST_CONTENT

  • Specify a path to a content source directory (as cloned from CaC/content) to be used for testing.
  • The content should be already built (at least for the product under test). If it is not, an attempt will be made to build it in-place (rather than in a temporary directory) so that any future tests benefit from the built content.
    • Note that this may fail if the content is located on a read-only path.

• CONTEST_CONTENT_BRANCH

  • Specify a branch name of the CaC/content project.
  • This will download content from the specified branch and automatically pre-set CONTEST_CONTENT to point to it.
  • Essentially, this is like CONTEST_CONTENT but without you having to provide a cloned directory, Contest automatically clones it for you.
  • Do not specify CONTEST_CONTENT in combination with this option.

• CONTEST_CONTENT_PR

  • Specify a numerical Pull Request ID (no # or other letters) of the CaC/content project.
  • This is like CONTEST_CONTENT_BRANCH, but it uses content from the pull request instead of a branch.
  • Do not specify CONTEST_CONTENT in combination with this option.

• CONTEST_OSCAP_BRANCH

  • Specify a branch name of the OpenSCAP project.
  • This will add a Packit DNF repository (specific for the branch) to the target system, and upgrade openscap-scanner.
  • As such, openscap-scanner built by Packit has to have a newer NVR than the RPM provided by regular OS repositories.

• CONTEST_OSCAP_PR

  • Specify a numerical Pull Request ID (no # or other letters) of the OpenSCAP project.
  • This works like CONTEST_OSCAP_BRANCH, but it upgrades to a Packit-built version from the pull request, instead of a branch.
  • Wait for Packit to build the RPM before running tests with this variable, otherwise the test run will fail.

Included test categories

See TESTS.md.

Waiving failed results

In this context, "to waive" means to label a failing result as known-bad, something we have seen before and expect to fail.

Read WAIVERS.md to see where/how you can set up rules to automatically waive failures.

Triggering Packit testing on Github

• To run just host-os directly managed by Packit + tmt:

  /packit test --labels host-os
  

  This runs 3 jobs (oscap, ansible, other). To run (or rerun) only one of them, you can pass --identifier or -i with the specific name:

  /packit test -i host-os-oscap
  

• To run all possible tests using ATEX and containerized Contest:

  /packit test -i productization
  

  You can also pass some parameters:

  • PLAN to override the default /plans/daily
  • TESTS as comma-separated test name fmf-style expressions
  • RERUNS to override the default 1 automatic rerun of every failed test
  • CONTENT_PR to test a specific CaC/content PR instead of master branch
  • NO_EXCLUDES=1 to run even tests normally incompatible with containers or unsuitable for PR CI

  For example, to test all CIS profile variants (incl. non-daily):

  /packit test -i productization --env RERUNS=0 --env PLAN=/plans/weekly --env TESTS=/cis$,/cis_server,/cis_workstation
  

Workarounds

(TODO: Find a better place for this?)

Using upstream/shipped content kickstarts

These have some unfortunate metadata, such as

• hardcoded network interface names
• unnecessarily large /var/log/audit size
• oscap Anaconda addon configuration using scap-security-guide

which are removed by translate_ssg_kickstart() in virt.py.

Referencing library code

See https://rhsecuritycompliance.github.io/contest/ for online Sphinx version of the modules present in lib.

Debugging

(TODO: probably move to its own document?)

SSH into Anaconda

Anaconda-based remediation can be debugged on a virtual machine through QEMU user-mode networking. While the installer is running, create a localhost port for its SSH service with virsh qemu-monitor-command contest --hmp 'hostfwd_add tcp:127.0.0.1:0-:22'. Then run virsh qemu-monitor-command contest --hmp 'info usernet' and find the TCP[HOST_FORWARD] row with source 127.0.0.1 and destination port 22. Connect from the VM host with ssh -p <port> root@127.0.0.1. There is no password for the Anaconda environment, so this will just log you in.

SSH into installed VMs

You can use a handy script in the home directory of the VM host's user.
Simply run:

./contest-sshvm [vm-name]

The script will find the first contest-installed VM if vm-name is not given, it will check whether the VM is running (as a result of you starting it earlier or CONTEST_LEAVE_GUEST_RUNNING=1) and if not, it will start it and wait for sshd to start responding. It will then ssh you into the VM, using pre-generated SSH keys (no passwords needed).

License

Unless specified otherwise, any content within this repository is distributed under the GNU GPLv3 license, see the COPYING.txt file for more.