bootc based PXE image

[bcl]

I am exploring what needs to be done to be able to support making a PXE bootable bootc image type for https://issues.redhat.com/browse/HMS-8933

The basic steps that are needed:

• make a bootc container that includes dracut-live
• run bootc to install the selected container
• Add dracut configuration file to add dmsquash-live support
• rebuild the initrd to add dracut-live support so that it can mount a compressed filesystem
• Disable the use of containerfs in bootc which conflicts with mounting the filesystem from the squashfs rootfs
• copy the kernel and initrd from the bootc filesystem into the PXE tar
• copy the EFI/ directory from the bootc filesystem to assist users in setting up their PXE environment
• compress the bootc filesystem using squashfs/erofs
• Make an example grub.cfg to assist the user in setting up their PXE boot environment
• Add a README
• Create a compressed tar of these files

The package based version of this was fairly straightforward. Using a bootc container introduces some new problems:

• bootc really likes to install to devices, even with the use of to-filesystem, so a partitioned disk image needs to be used.
• squashfs expects an input from another pipeline as its source, not a mount. the squashfs/erofs commits in #2218 add support for a mount:// source.
• devices expect to be setup from a file in the current pipeline. #2218 adds support for using a file from a previous pipeline in read only mode. Or you can copy the file into the current pipeline, greatly increasing the amount of space used and slowing down the build.
• The user needs to add dracut-live to the container they specify so that dracut can include it in the new initrd. There is no way for osbuild to add new layers to a container (and if it did it would change the checksum, making it harder to track what container is in use).
• The kernel version used in the bootc container is not visible outside the container, this is needed in order to rebuild the initrd.
• The UUID used by ostree to store the initial commit is not visible outside the container, this is needed by the user to setup their grub.cfg boot arguments -- they can mount the rootfs.img themselves and look for /ostree/boot.1/default/UUID but it would be nicer to be able to inject that into the example grub.cfg
• dracut's dmsquash-live expected the root filesystem to have /usr in it as a flag that the filesystem mount worked. dracut needs a patch to also look for /ostree
• bootc defaults to using containerfs which causes a conflict when trying to setup the ostree commit from the squashfs filesystem. It is easily disabled by writing a file to the bootc filesystem

I have 2 example manifests that work, and are bootable. One uses the device input://tree/ change to mount the disk.raw from the previous pipeline. The other example copies the disk.raw into a new pipeline for mounting and compression, then copies files into another new pipeline to make the final directory tree for the tar.

The problems that remain unsolved are:

• kernel version number to use when running dracut on the bootc filesystem
• UUID of the ostree commit to use in the grub.cfg

[bcl]

Example manifest using device with input: from the previous pipeline.
bib-pxe.json

Example manifest using extra pipelines and copying disk.raw so that squashfs can mount and compress it.
bib-pxe-copy.json

[bcl]

@cgwalters @supakeen @achilleas-k may want to comment on this :)

[cgwalters]

This relates to bootc-dev/bootc#898 - if we did that, a whole lot of complexity drops out here

[achilleas-k]

Is this something we can expect to see soon? Should we try to reprioritise this to have a simpler solution in the end and avoid doing too much work now that wont be necessary in the long run?

https://issues.redhat.com/browse/RHELBU-3152 is already overdue (not sure when exactly we're planning to have this now), and I'm not sure how much we can shift things around, but it might be worth discussing.

[supakeen]

For what it's worth; even with support-for-bootc in dracut and thus initrd's we'd still likely need an image type ourselves, though it drops (part of) the requirement of us turning it into a rootfs ourselves.

[achilleas-k]

• devices expect to be setup from a file in the current pipeline. #2218 adds support for using a file from a previous pipeline in read only mode. Or you can copy the file into the current pipeline, greatly increasing the amount of space used and slowing down the build.

I'm willing to allow for some inefficiency in the initial implementation to avoid blocking on discussions about the best way to do the device-from-pipeline feature (we seem to have a lot of opinions there). Also, if we end up not needing it because of bootc-dev/bootc#898, I'd prefer to do less work now.

That said, I can imagine the feature itself (mounting a device from a previous pipeline to read from it) might have other use cases for us going forward, so it might still be worth discussing if we want it or not.