ZimaOS version: 1.6.2 (kernel updated to 6.18.9 in this release, per changelog)
Hardware: Generic x86 NUC, SSD: TWSC TSC10N512-F7T10S (512GB SATA SSD, not present in smartctl device database), SMART overall health: PASSED, Lifetime writes: 18TB
Summary
The main data partition (/dev/sda8, ext4 — hosts /DATA, /var/lib/docker, /var/log, /opt, /var/lib/casaos_data, etc.) has dropped into emergency_ro (read-only) mode twice within 7 days on 1.6.2. Docker containers remain listed as “Up” in docker ps but silently fail on any write operation — no crash, no restart, just an inaccessible/broken state until reboot. The second incident was followed by a boot that took over 1 hour with no network response at all, requiring a hard power-cycle to recover.
Timeline
Incident 1 — 2026-07-22, 21:13:16
• dumpe2fs -h /dev/sda8 superblock recorded:
• First error function: __ext4_new_inode, line 1094, err: EROFS
• Last error function: ext4_journal_check_start, line 87, err: EIO (3 seconds later)
• FS Error count: 31
• Symptom chain observed: CrowdSec triggered a community blocklist update at 21:09:22 (3000 entries added/deleted). At 21:10:13–17, the NPMplus↔CrowdSec bouncer (crowdsec.lua:711 Allow()) started timing out querying the local LAPI (127.0.0.1:7422), with request times spiking to 6–7s and one context canceled: sql: transaction has already been committed or rolled back error in the CrowdSec log. The EROFS/EIO pair on /dev/sda8 occurred within this same window.
• Resolved by reboot. Post-reboot, filesystem showed clean, error counters reset by the kernel automatically on remount.
• Forced a fsck via tune2fs -c 1 /dev/sda8 as a precaution; came back clean on next boot.
Incident 2 — 2026-07-29, 23:47:00
• Same error signature:
• First error function: ext4_do_writepages, line 2944, err: EROFS
• Last error function: ext4_journal_check_start, line 87, err: EIO
• FS Error count: 14
• First visible symptom: docker restart <container> failed with chown ... read-only file system on an overlay2 work directory.
• dmesg -T | head -100 only showed cascading systemd-journald write failures starting ~21:57 (buffer had already rotated past the actual 23:47 trigger event in both incidents — never captured the root cause directly in dmesg).
• Rebooted via sudo reboot. This time the system took over 1 hour to become reachable again — no ping response, SSH refused/timed out (“Host is down” → “Operation timed out”), not visible as an active DHCP client. Recovered only after a hard power-cycle (physical button, no screen/KVM access available to observe the boot process directly).
What was ruled out
• Disk space: /DATA at 31% usage at time of incident 1, not a factor.
• Network/CrowdSec as root cause: confirmed via CrowdSec’s own LAPI logs that the service itself was responding normally (200 OK) throughout the period — the NPM bouncer timeouts and SQL contention were downstream symptoms of the same disk stall, not an independent CrowdSec bug.
• SMART: overall health “PASSED”, but this SSD model is not in smartctl’s database, so alert thresholds are uncalibrated for it — low confidence in this result.
• No Buffer I/O error, ata bus errors, or EXT4-fs error lines found via dmesg -T | grep in either incident — but in both cases the dmesg ring buffer had already rotated past the actual trigger timestamp before it could be inspected, so this is inconclusive rather than a clean bill of health.
Suspected cause
1.6.2 updated the Linux kernel to 6.18.9. Both incidents show an identical error signature (EROFS on an inode/writepage operation, immediately followed by EIO on ext4 journal transaction start) with no corroborating hardware-level error visible. The system runs sustained concurrent Docker I/O (30+ containers: full *arr stack, qBittorrent, Jellyfin, CrowdSec, Gluetun VPN, Postgres, etc.), and incident 1 coincided with a burst of write activity from a CrowdSec blocklist update. This points toward a possible kernel/ext4 regression under concurrent I/O load introduced with the 6.18.9 kernel bump, rather than a one-off hardware fault — though a marginal/degrading SSD contributing under stress cannot be ruled out either.
Separately, on the community forum, other users have reported storage read-only issues around the same period (e.g. “Zima os just set my storage to read-only”, May 2026; a v1.7.0 user reporting a media disk becoming “unsupported” post-upgrade, July 29 2026) — suggesting this may not be an isolated case, though none of the found reports match this exact EROFS/ext4_journal_check_start/EIO signature.
Steps to reproduce
Not reliably reproducible on demand. Both incidents occurred during normal 24/7 operation, roughly 7 days apart, under sustained concurrent container I/O — no specific user action triggered either one.
Requested
• Whether this is a known issue linked to the kernel 6.18.9 update in 1.6.2
• Whether a kernel rollback / downgrade to a pre-1.6.2 build is advisable while this is investigated, given the second incident’s abnormally long, unresponsive boot
• Any specific diagnostics IceWhale would like captured proactively before the next occurrence (dmesg ring buffer rotates too fast to catch the trigger event after the fact)
ZimaOS version: 1.6.2 (kernel updated to 6.18.9 in this release, per changelog)
Hardware: Generic x86 NUC, SSD: TWSC TSC10N512-F7T10S (512GB SATA SSD, not present in smartctl device database), SMART overall health: PASSED, Lifetime writes: 18TB
Summary
The main data partition (/dev/sda8, ext4 — hosts /DATA, /var/lib/docker, /var/log, /opt, /var/lib/casaos_data, etc.) has dropped into emergency_ro (read-only) mode twice within 7 days on 1.6.2. Docker containers remain listed as “Up” in docker ps but silently fail on any write operation — no crash, no restart, just an inaccessible/broken state until reboot. The second incident was followed by a boot that took over 1 hour with no network response at all, requiring a hard power-cycle to recover.
Timeline
Incident 1 — 2026-07-22, 21:13:16
Incident 2 — 2026-07-29, 23:47:00
What was ruled out
• Disk space: /DATA at 31% usage at time of incident 1, not a factor. • Network/CrowdSec as root cause: confirmed via CrowdSec’s own LAPI logs that the service itself was responding normally (200 OK) throughout the period — the NPM bouncer timeouts and SQL contention were downstream symptoms of the same disk stall, not an independent CrowdSec bug. • SMART: overall health “PASSED”, but this SSD model is not in smartctl’s database, so alert thresholds are uncalibrated for it — low confidence in this result. • No Buffer I/O error, ata bus errors, or EXT4-fs error lines found via dmesg -T | grep in either incident — but in both cases the dmesg ring buffer had already rotated past the actual trigger timestamp before it could be inspected, so this is inconclusive rather than a clean bill of health.Suspected cause
1.6.2 updated the Linux kernel to 6.18.9. Both incidents show an identical error signature (EROFS on an inode/writepage operation, immediately followed by EIO on ext4 journal transaction start) with no corroborating hardware-level error visible. The system runs sustained concurrent Docker I/O (30+ containers: full *arr stack, qBittorrent, Jellyfin, CrowdSec, Gluetun VPN, Postgres, etc.), and incident 1 coincided with a burst of write activity from a CrowdSec blocklist update. This points toward a possible kernel/ext4 regression under concurrent I/O load introduced with the 6.18.9 kernel bump, rather than a one-off hardware fault — though a marginal/degrading SSD contributing under stress cannot be ruled out either.
Separately, on the community forum, other users have reported storage read-only issues around the same period (e.g. “Zima os just set my storage to read-only”, May 2026; a v1.7.0 user reporting a media disk becoming “unsupported” post-upgrade, July 29 2026) — suggesting this may not be an isolated case, though none of the found reports match this exact EROFS/ext4_journal_check_start/EIO signature.
Steps to reproduce
Not reliably reproducible on demand. Both incidents occurred during normal 24/7 operation, roughly 7 days apart, under sustained concurrent container I/O — no specific user action triggered either one.
Requested