During an upgrade from OKD 4.19.0‑okd‑scos.19 to 4.20.0‑okd‑scos.17, two nodes (1 master, 1 worker) failed to rejoin the cluster.
Both machines rebooted automatically as part of the SCOS update, but never reached a Ready state afterward.
The nodes get stuck early in the boot process, repeatedly showing output from /tmp/configure-ovs.sh, and the system never proceeds to start kubelet or CRI-O properly.
The root cause appears to be missing OCI runtime (runc) in the upgraded SCOS rootfs, causing CRI‑O to crash immediately on startup.
This leaves the nodes permanently NotReady and unable to register back into the cluster.
I am attaching a screenshot taken directly from the physical node console, showing where the boot sequence stops.
And also the relevant journal logs (CRI‑O, kubelet, system boot logs) that demonstrate the failure.
[core@bsl-okd-test-worker01 ~]$ journalctl -u crio -b --no-pager | tail -n 50 Mar 05 05:16:00 bsl-okd-test-worker01 crio[61478]: time="2026-03-05T05:16:00.787094844Z" level=info msg="Updating config from drop-in file: /etc/crio/crio.conf.d/01-ctrcfg-defaultRuntime" Mar 05 05:16:00 bsl-okd-test-worker01 crio[61478]: time="2026-03-05T05:16:00.797222358Z" level=info msg="Starting CRI-O, version: 1.33.4, git: unknown(clean)" Mar 05 05:16:00 bsl-okd-test-worker01 crio[61478]: time="2026-03-05T05:16:00.797281317Z" level=info msg="Node configuration value for hugetlb cgroup is true" Mar 05 05:16:00 bsl-okd-test-worker01 crio[61478]: time="2026-03-05T05:16:00.797289708Z" level=info msg="Node configuration value for pid cgroup is true" Mar 05 05:16:00 bsl-okd-test-worker01 crio[61478]: time="2026-03-05T05:16:00.797338018Z" level=info msg="Node configuration value for memoryswap cgroup is true" Mar 05 05:16:00 bsl-okd-test-worker01 crio[61478]: time="2026-03-05T05:16:00.797347003Z" level=info msg="Node configuration value for cgroup v2 is true" Mar 05 05:16:00 bsl-okd-test-worker01 crio[61478]: time="2026-03-05T05:16:00.806057997Z" level=info msg="Node configuration value for systemd AllowedCPUs is true" Mar 05 05:16:00 bsl-okd-test-worker01 crio[61478]: time="2026-03-05T05:16:00.817355645Z" level=info msg="Using default capabilities: CAP_CHOWN, CAP_DAC_OVERRIDE, CAP_FSETID, CAP_FOWNER, CAP_SETGID, CAP_SETUID, CAP_SETPCAP, CAP_NET_BIND_SERVICE, CAP_KILL" Mar 05 05:16:00 bsl-okd-test-worker01 crio[61478]: time="2026-03-05T05:16:00.81744607Z" level=fatal msg="validating runtime config: runtime validation: \"runc\" not found in $PATH: exec: \"runc\": executable file not found in $PATH" Mar 05 05:16:00 bsl-okd-test-worker01 systemd[1]: crio.service: Main process exited, code=exited, status=1/FAILURE Mar 05 05:16:00 bsl-okd-test-worker01 systemd[1]: crio.service: Failed with result 'exit-code'. Mar 05 05:16:00 bsl-okd-test-worker01 systemd[1]: Failed to start crio.service - Container Runtime Interface for OCI (CRI-O). Mar 05 05:16:10 bsl-okd-test-worker01 systemd[1]: crio.service: Scheduled restart job, restart counter is at 2687. Mar 05 05:16:10 bsl-okd-test-worker01 systemd[1]: Starting crio.service - Container Runtime Interface for OCI (CRI-O)... Mar 05 05:16:11 bsl-okd-test-worker01 (crio)[61499]: crio.service: Referenced but unset environment variable evaluates to an empty string: CRIO_CONFIG_OPTIONS, CRIO_METRICS_OPTIONS, CRIO_NETWORK_OPTIONS, CRIO_RUNTIME_OPTIONS, CRIO_STORAGE_OPTIONS Mar 05 05:16:11 bsl-okd-test-worker01 crio[61499]: time="2026-03-05T05:16:11.036063601Z" level=info msg="Updating config from single file: /etc/crio/crio.conf" Mar 05 05:16:11 bsl-okd-test-worker01 crio[61499]: time="2026-03-05T05:16:11.036093188Z" level=info msg="Updating config from drop-in file: /etc/crio/crio.conf" Mar 05 05:16:11 bsl-okd-test-worker01 crio[61499]: time="2026-03-05T05:16:11.038457106Z" level=info msg="Updating config from path: /etc/crio/crio.conf.d" Mar 05 05:16:11 bsl-okd-test-worker01 crio[61499]: time="2026-03-05T05:16:11.038521978Z" level=info msg="Updating config from drop-in file: /etc/crio/crio.conf.d/00-default" Mar 05 05:16:11 bsl-okd-test-worker01 crio[61499]: time="2026-03-05T05:16:11.03887504Z" level=info msg="Updating config from drop-in file: /etc/crio/crio.conf.d/01-ctrcfg-defaultRuntime" Mar 05 05:16:11 bsl-okd-test-worker01 crio[61499]: time="2026-03-05T05:16:11.04894451Z" level=info msg="Starting CRI-O, version: 1.33.4, git: unknown(clean)" Mar 05 05:16:11 bsl-okd-test-worker01 crio[61499]: time="2026-03-05T05:16:11.049003315Z" level=info msg="Node configuration value for hugetlb cgroup is true" Mar 05 05:16:11 bsl-okd-test-worker01 crio[61499]: time="2026-03-05T05:16:11.049016159Z" level=info msg="Node configuration value for pid cgroup is true" Mar 05 05:16:11 bsl-okd-test-worker01 crio[61499]: time="2026-03-05T05:16:11.049063475Z" level=info msg="Node configuration value for memoryswap cgroup is true" Mar 05 05:16:11 bsl-okd-test-worker01 crio[61499]: time="2026-03-05T05:16:11.049073048Z" level=info msg="Node configuration value for cgroup v2 is true" Mar 05 05:16:11 bsl-okd-test-worker01 crio[61499]: time="2026-03-05T05:16:11.057477201Z" level=info msg="Node configuration value for systemd AllowedCPUs is true" Mar 05 05:16:11 bsl-okd-test-worker01 crio[61499]: time="2026-03-05T05:16:11.070147021Z" level=info msg="Using default capabilities: CAP_CHOWN, CAP_DAC_OVERRIDE, CAP_FSETID, CAP_FOWNER, CAP_SETGID, CAP_SETUID, CAP_SETPCAP, CAP_NET_BIND_SERVICE, CAP_KILL" Mar 05 05:16:11 bsl-okd-test-worker01 crio[61499]: time="2026-03-05T05:16:11.070279133Z" level=fatal msg="validating runtime config: runtime validation: \"runc\" not found in $PATH: exec: \"runc\": executable file not found in $PATH" Mar 05 05:16:11 bsl-okd-test-worker01 systemd[1]: crio.service: Main process exited, code=exited, status=1/FAILURE Mar 05 05:16:11 bsl-okd-test-worker01 systemd[1]: crio.service: Failed with result 'exit-code'. Mar 05 05:16:11 bsl-okd-test-worker01 systemd[1]: Failed to start crio.service - Container Runtime Interface for OCI (CRI-O). Mar 05 05:16:21 bsl-okd-test-worker01 systemd[1]: crio.service: Scheduled restart job, restart counter is at 2688. Mar 05 05:16:21 bsl-okd-test-worker01 systemd[1]: Starting crio.service - Container Runtime Interface for OCI (CRI-O)... Mar 05 05:16:21 bsl-okd-test-worker01 (crio)[61587]: crio.service: Referenced but unset environment variable evaluates to an empty string: CRIO_CONFIG_OPTIONS, CRIO_METRICS_OPTIONS, CRIO_NETWORK_OPTIONS, CRIO_RUNTIME_OPTIONS, CRIO_STORAGE_OPTIONS Mar 05 05:16:21 bsl-okd-test-worker01 crio[61587]: time="2026-03-05T05:16:21.28693912Z" level=info msg="Updating config from single file: /etc/crio/crio.conf" Mar 05 05:16:21 bsl-okd-test-worker01 crio[61587]: time="2026-03-05T05:16:21.286973028Z" level=info msg="Updating config from drop-in file: /etc/crio/crio.conf" Mar 05 05:16:21 bsl-okd-test-worker01 crio[61587]: time="2026-03-05T05:16:21.288606651Z" level=info msg="Updating config from path: /etc/crio/crio.conf.d" Mar 05 05:16:21 bsl-okd-test-worker01 crio[61587]: time="2026-03-05T05:16:21.288667715Z" level=info msg="Updating config from drop-in file: /etc/crio/crio.conf.d/00-default" Mar 05 05:16:21 bsl-okd-test-worker01 crio[61587]: time="2026-03-05T05:16:21.289002379Z" level=info msg="Updating config from drop-in file: /etc/crio/crio.conf.d/01-ctrcfg-defaultRuntime" Mar 05 05:16:21 bsl-okd-test-worker01 crio[61587]: time="2026-03-05T05:16:21.299052597Z" level=info msg="Starting CRI-O, version: 1.33.4, git: unknown(clean)" Mar 05 05:16:21 bsl-okd-test-worker01 crio[61587]: time="2026-03-05T05:16:21.299121985Z" level=info msg="Node configuration value for hugetlb cgroup is true" Mar 05 05:16:21 bsl-okd-test-worker01 crio[61587]: time="2026-03-05T05:16:21.299131283Z" level=info msg="Node configuration value for pid cgroup is true" Mar 05 05:16:21 bsl-okd-test-worker01 crio[61587]: time="2026-03-05T05:16:21.299187376Z" level=info msg="Node configuration value for memoryswap cgroup is true" Mar 05 05:16:21 bsl-okd-test-worker01 crio[61587]: time="2026-03-05T05:16:21.299197346Z" level=info msg="Node configuration value for cgroup v2 is true" Mar 05 05:16:21 bsl-okd-test-worker01 crio[61587]: time="2026-03-05T05:16:21.30868857Z" level=info msg="Node configuration value for systemd AllowedCPUs is true" Mar 05 05:16:21 bsl-okd-test-worker01 crio[61587]: time="2026-03-05T05:16:21.320678504Z" level=info msg="Using default capabilities: CAP_CHOWN, CAP_DAC_OVERRIDE, CAP_FSETID, CAP_FOWNER, CAP_SETGID, CAP_SETUID, CAP_SETPCAP, CAP_NET_BIND_SERVICE, CAP_KILL" Mar 05 05:16:21 bsl-okd-test-worker01 crio[61587]: time="2026-03-05T05:16:21.320801025Z" level=fatal msg="validating runtime config: runtime validation: \"runc\" not found in $PATH: exec: \"runc\": executable file not found in $PATH" Mar 05 05:16:21 bsl-okd-test-worker01 systemd[1]: crio.service: Main process exited, code=exited, status=1/FAILURE Mar 05 05:16:21 bsl-okd-test-worker01 systemd[1]: crio.service: Failed with result 'exit-code'. Mar 05 05:16:21 bsl-okd-test-worker01 systemd[1]: Failed to start crio.service - Container Runtime Interface for OCI (CRI-O). [core@bsl-okd-test-worker01 ~]$
Version
No response
Reproducibility
No response
Log Bundle
No response
During an upgrade from OKD 4.19.0‑okd‑scos.19 to 4.20.0‑okd‑scos.17, two nodes (1 master, 1 worker) failed to rejoin the cluster.
Both machines rebooted automatically as part of the SCOS update, but never reached a Ready state afterward.
The nodes get stuck early in the boot process, repeatedly showing output from /tmp/configure-ovs.sh, and the system never proceeds to start kubelet or CRI-O properly.
The root cause appears to be missing OCI runtime (runc) in the upgraded SCOS rootfs, causing CRI‑O to crash immediately on startup.
This leaves the nodes permanently NotReady and unable to register back into the cluster.
I am attaching a screenshot taken directly from the physical node console, showing where the boot sequence stops.
And also the relevant journal logs (CRI‑O, kubelet, system boot logs) that demonstrate the failure.
[core@bsl-okd-test-worker01 ~]$ journalctl -u crio -b --no-pager | tail -n 50 Mar 05 05:16:00 bsl-okd-test-worker01 crio[61478]: time="2026-03-05T05:16:00.787094844Z" level=info msg="Updating config from drop-in file: /etc/crio/crio.conf.d/01-ctrcfg-defaultRuntime" Mar 05 05:16:00 bsl-okd-test-worker01 crio[61478]: time="2026-03-05T05:16:00.797222358Z" level=info msg="Starting CRI-O, version: 1.33.4, git: unknown(clean)" Mar 05 05:16:00 bsl-okd-test-worker01 crio[61478]: time="2026-03-05T05:16:00.797281317Z" level=info msg="Node configuration value for hugetlb cgroup is true" Mar 05 05:16:00 bsl-okd-test-worker01 crio[61478]: time="2026-03-05T05:16:00.797289708Z" level=info msg="Node configuration value for pid cgroup is true" Mar 05 05:16:00 bsl-okd-test-worker01 crio[61478]: time="2026-03-05T05:16:00.797338018Z" level=info msg="Node configuration value for memoryswap cgroup is true" Mar 05 05:16:00 bsl-okd-test-worker01 crio[61478]: time="2026-03-05T05:16:00.797347003Z" level=info msg="Node configuration value for cgroup v2 is true" Mar 05 05:16:00 bsl-okd-test-worker01 crio[61478]: time="2026-03-05T05:16:00.806057997Z" level=info msg="Node configuration value for systemd AllowedCPUs is true" Mar 05 05:16:00 bsl-okd-test-worker01 crio[61478]: time="2026-03-05T05:16:00.817355645Z" level=info msg="Using default capabilities: CAP_CHOWN, CAP_DAC_OVERRIDE, CAP_FSETID, CAP_FOWNER, CAP_SETGID, CAP_SETUID, CAP_SETPCAP, CAP_NET_BIND_SERVICE, CAP_KILL" Mar 05 05:16:00 bsl-okd-test-worker01 crio[61478]: time="2026-03-05T05:16:00.81744607Z" level=fatal msg="validating runtime config: runtime validation: \"runc\" not found in $PATH: exec: \"runc\": executable file not found in $PATH" Mar 05 05:16:00 bsl-okd-test-worker01 systemd[1]: crio.service: Main process exited, code=exited, status=1/FAILURE Mar 05 05:16:00 bsl-okd-test-worker01 systemd[1]: crio.service: Failed with result 'exit-code'. Mar 05 05:16:00 bsl-okd-test-worker01 systemd[1]: Failed to start crio.service - Container Runtime Interface for OCI (CRI-O). Mar 05 05:16:10 bsl-okd-test-worker01 systemd[1]: crio.service: Scheduled restart job, restart counter is at 2687. Mar 05 05:16:10 bsl-okd-test-worker01 systemd[1]: Starting crio.service - Container Runtime Interface for OCI (CRI-O)... Mar 05 05:16:11 bsl-okd-test-worker01 (crio)[61499]: crio.service: Referenced but unset environment variable evaluates to an empty string: CRIO_CONFIG_OPTIONS, CRIO_METRICS_OPTIONS, CRIO_NETWORK_OPTIONS, CRIO_RUNTIME_OPTIONS, CRIO_STORAGE_OPTIONS Mar 05 05:16:11 bsl-okd-test-worker01 crio[61499]: time="2026-03-05T05:16:11.036063601Z" level=info msg="Updating config from single file: /etc/crio/crio.conf" Mar 05 05:16:11 bsl-okd-test-worker01 crio[61499]: time="2026-03-05T05:16:11.036093188Z" level=info msg="Updating config from drop-in file: /etc/crio/crio.conf" Mar 05 05:16:11 bsl-okd-test-worker01 crio[61499]: time="2026-03-05T05:16:11.038457106Z" level=info msg="Updating config from path: /etc/crio/crio.conf.d" Mar 05 05:16:11 bsl-okd-test-worker01 crio[61499]: time="2026-03-05T05:16:11.038521978Z" level=info msg="Updating config from drop-in file: /etc/crio/crio.conf.d/00-default" Mar 05 05:16:11 bsl-okd-test-worker01 crio[61499]: time="2026-03-05T05:16:11.03887504Z" level=info msg="Updating config from drop-in file: /etc/crio/crio.conf.d/01-ctrcfg-defaultRuntime" Mar 05 05:16:11 bsl-okd-test-worker01 crio[61499]: time="2026-03-05T05:16:11.04894451Z" level=info msg="Starting CRI-O, version: 1.33.4, git: unknown(clean)" Mar 05 05:16:11 bsl-okd-test-worker01 crio[61499]: time="2026-03-05T05:16:11.049003315Z" level=info msg="Node configuration value for hugetlb cgroup is true" Mar 05 05:16:11 bsl-okd-test-worker01 crio[61499]: time="2026-03-05T05:16:11.049016159Z" level=info msg="Node configuration value for pid cgroup is true" Mar 05 05:16:11 bsl-okd-test-worker01 crio[61499]: time="2026-03-05T05:16:11.049063475Z" level=info msg="Node configuration value for memoryswap cgroup is true" Mar 05 05:16:11 bsl-okd-test-worker01 crio[61499]: time="2026-03-05T05:16:11.049073048Z" level=info msg="Node configuration value for cgroup v2 is true" Mar 05 05:16:11 bsl-okd-test-worker01 crio[61499]: time="2026-03-05T05:16:11.057477201Z" level=info msg="Node configuration value for systemd AllowedCPUs is true" Mar 05 05:16:11 bsl-okd-test-worker01 crio[61499]: time="2026-03-05T05:16:11.070147021Z" level=info msg="Using default capabilities: CAP_CHOWN, CAP_DAC_OVERRIDE, CAP_FSETID, CAP_FOWNER, CAP_SETGID, CAP_SETUID, CAP_SETPCAP, CAP_NET_BIND_SERVICE, CAP_KILL" Mar 05 05:16:11 bsl-okd-test-worker01 crio[61499]: time="2026-03-05T05:16:11.070279133Z" level=fatal msg="validating runtime config: runtime validation: \"runc\" not found in $PATH: exec: \"runc\": executable file not found in $PATH" Mar 05 05:16:11 bsl-okd-test-worker01 systemd[1]: crio.service: Main process exited, code=exited, status=1/FAILURE Mar 05 05:16:11 bsl-okd-test-worker01 systemd[1]: crio.service: Failed with result 'exit-code'. Mar 05 05:16:11 bsl-okd-test-worker01 systemd[1]: Failed to start crio.service - Container Runtime Interface for OCI (CRI-O). Mar 05 05:16:21 bsl-okd-test-worker01 systemd[1]: crio.service: Scheduled restart job, restart counter is at 2688. Mar 05 05:16:21 bsl-okd-test-worker01 systemd[1]: Starting crio.service - Container Runtime Interface for OCI (CRI-O)... Mar 05 05:16:21 bsl-okd-test-worker01 (crio)[61587]: crio.service: Referenced but unset environment variable evaluates to an empty string: CRIO_CONFIG_OPTIONS, CRIO_METRICS_OPTIONS, CRIO_NETWORK_OPTIONS, CRIO_RUNTIME_OPTIONS, CRIO_STORAGE_OPTIONS Mar 05 05:16:21 bsl-okd-test-worker01 crio[61587]: time="2026-03-05T05:16:21.28693912Z" level=info msg="Updating config from single file: /etc/crio/crio.conf" Mar 05 05:16:21 bsl-okd-test-worker01 crio[61587]: time="2026-03-05T05:16:21.286973028Z" level=info msg="Updating config from drop-in file: /etc/crio/crio.conf" Mar 05 05:16:21 bsl-okd-test-worker01 crio[61587]: time="2026-03-05T05:16:21.288606651Z" level=info msg="Updating config from path: /etc/crio/crio.conf.d" Mar 05 05:16:21 bsl-okd-test-worker01 crio[61587]: time="2026-03-05T05:16:21.288667715Z" level=info msg="Updating config from drop-in file: /etc/crio/crio.conf.d/00-default" Mar 05 05:16:21 bsl-okd-test-worker01 crio[61587]: time="2026-03-05T05:16:21.289002379Z" level=info msg="Updating config from drop-in file: /etc/crio/crio.conf.d/01-ctrcfg-defaultRuntime" Mar 05 05:16:21 bsl-okd-test-worker01 crio[61587]: time="2026-03-05T05:16:21.299052597Z" level=info msg="Starting CRI-O, version: 1.33.4, git: unknown(clean)" Mar 05 05:16:21 bsl-okd-test-worker01 crio[61587]: time="2026-03-05T05:16:21.299121985Z" level=info msg="Node configuration value for hugetlb cgroup is true" Mar 05 05:16:21 bsl-okd-test-worker01 crio[61587]: time="2026-03-05T05:16:21.299131283Z" level=info msg="Node configuration value for pid cgroup is true" Mar 05 05:16:21 bsl-okd-test-worker01 crio[61587]: time="2026-03-05T05:16:21.299187376Z" level=info msg="Node configuration value for memoryswap cgroup is true" Mar 05 05:16:21 bsl-okd-test-worker01 crio[61587]: time="2026-03-05T05:16:21.299197346Z" level=info msg="Node configuration value for cgroup v2 is true" Mar 05 05:16:21 bsl-okd-test-worker01 crio[61587]: time="2026-03-05T05:16:21.30868857Z" level=info msg="Node configuration value for systemd AllowedCPUs is true" Mar 05 05:16:21 bsl-okd-test-worker01 crio[61587]: time="2026-03-05T05:16:21.320678504Z" level=info msg="Using default capabilities: CAP_CHOWN, CAP_DAC_OVERRIDE, CAP_FSETID, CAP_FOWNER, CAP_SETGID, CAP_SETUID, CAP_SETPCAP, CAP_NET_BIND_SERVICE, CAP_KILL" Mar 05 05:16:21 bsl-okd-test-worker01 crio[61587]: time="2026-03-05T05:16:21.320801025Z" level=fatal msg="validating runtime config: runtime validation: \"runc\" not found in $PATH: exec: \"runc\": executable file not found in $PATH" Mar 05 05:16:21 bsl-okd-test-worker01 systemd[1]: crio.service: Main process exited, code=exited, status=1/FAILURE Mar 05 05:16:21 bsl-okd-test-worker01 systemd[1]: crio.service: Failed with result 'exit-code'. Mar 05 05:16:21 bsl-okd-test-worker01 systemd[1]: Failed to start crio.service - Container Runtime Interface for OCI (CRI-O). [core@bsl-okd-test-worker01 ~]$Version
No response
Reproducibility
No response
Log Bundle
No response