Bootc based anaconda installer containers

[mvo5]

One goal we have for bootc-image-buidler (and image-builder-cli) is to support building anaconda installer ISO that are build from bootc images (instead of our current RPM based approach).

In osbuild/bootc-image-builder#1059 we implemented a spike that allows to create a bootc container with anaconda that can be turned into an installer ISO by bootc-image-builder. The Containerfile looks something like this (check the linked PR for the full one):

FROM <original-container-e.g-bootc-centos9>
RUN dnf install -y \
     anaconda and a bunch more dependencies \
     && dnf clean all
# workaround(s)
RUN dnf reinstall -y shim-x64
RUN mkdir /var/mnt

The spike contains code to also specify an "--installer-payload" so that users can install a container without anaconda in it. So e.g. for centos-bootc-9 create a derived installer container from centos-9 with anaconda and then bootc-image-builder would put the specified installer payload (usually also either centos-bootc-9 directly or a derived container from it) on the ISO.

We would like to turn this from a spike into something real that is as friction-less to the user as possible and the main friction is currently to construct the correct containerfile so that anaconda has alll dependencies and the required workarounds are applied.

So to solve this I see the following options:

Dependencies rpm & fix workaround

Create a image-builder-anaconda-deps package and put into all distros we support that contains the dependencies required and also fix all cases that need workarounds

This is the most elegant solution IMHO, here the containerfile looks like

FROM <base-ref>
RUN dnf install  -y image-builder-anaconda-deps && dnf clean all

the downside of course is that it will take a bit of time to fix the workarounds and to get the new deps package into the right places.

bootc-base-imgctl equivalent for anaconda

Create a new bootc-anaconda-imgectl script similar to the existing bootc-base-imagectl (or create a new subcommand in that script) that encapsulates the install and workarounds.

This means the Containerfile would look something like this:

FROM <base-ref>
RUN bootc-anaconda-imagectl make-installer

The upside of tihs solution is that there is only a single place that we need to put the script (the base-images repo) and we could get it in quickly before the workarounds are fixed and we would be flexible when new workarounds are needed. The downside that the base-image repo is quite far from the "images/bootc-image-builder repo" where this is used (and the image-builder team has no write access there).

Bootc-image-builder automatically creates derrived container

Here the bootc anaconda container would be invsible to the user and it would be auto-created based on a template in images (or bootc-image-builder). It would simply assume that the containerfile works (and we would have tests to ensure this is the case for cs9,cs10,f42+,rhel-9,rhel-10). Least flexible approach and feels quite "magic".

Just Document what is needed

We could simply document what needs to be done and in parallel work on removing the required workarounds and simplifying the dependencies required in dnf. More friction though.

Something else ?

Happy to hear any ideas or suggestions!

[achilleas-k]

Sorry for the cop-out answer here but why not: All of the above?

Of course, not all at once, but I think there's a good plan here that involves all of the options. Details below.

Dependencies rpm & fix workaround

I think this is the ideal solution, but I agree that it would take longer to do than the others. So I would say let's keep it as the final long-term solution we would like to have.

bootc-base-imgctl equivalent for anaconda

Let's do this first as a quick solution. It will probably be simple enough to get out. Perhaps we could keep a development version of it under github.com/osbuild so we can point people to it when they have issues with the downstream version. But in the long run, we should plan to replace this with the first solution.

Bootc-image-builder automatically creates derived container

I think this can be done in parallel, or eventually as a helper of bootc-image-builder. I wouldn't do it initially, let's get the users used to building their own derived containers and then provide them with this helper. We could evaluate its usefulness based on what users are doing. Assuming we have a way of knowing, if it turns out that everyone is just following our instructions and not adding anything extra, we can roll it in as a standard, safe way of creating the ISO with a single command.

Just Document what is needed

Let's document what's needed anyway. Meaning, the docs shouldn't just include the containerfile that describes how to make the derived container for the ISO, but should also explain what happens when you dnf install image-builder-anaconda-deps or bootc-anaconda-imagectl make-installer, so it's easier for users to know what to expect and what might go wrong.

Something else ?

Can't think of anything right now.

[achilleas-k]

One clear issue with this is that eventually we'll end up having two ways of doing things, imgctl and the rpm. We could prepare for this by calling the first solution a development tool and deprecating it when we get the rpm. I realise that never works as expected and users tend to stick to what they learn first, but I think it's worth trying.

[thozza]

I'm sorry for the late response. I got to this discussion only now, after having a look at osbuild/images#1906

Since I'm missing the background on this spike, I'd like to ask about the actual need / reasons to base the installer environment on the bootc container. Why is this needed? I acknowledge that the installer environment should be based on the same distribution and version as the bootc container, but couldn't this be done by installing it traditionally as packages using the content from the repositories in the bootc container? In a sense, nobody will be running the installer environment for anything other than the installation, so it is a throwaway, one-time-use environment. The host system installed by the installer image is the part that must be entirely based on the bootc container. I'm definitely missing the motivation for making our lives more complicated than necessary to achieve this.

Dependencies rpm & fix workaround

I actually like this solution the least. I think this is the worst approach from the maintainability perspective.

My reasons:

• It would require keeping the virtual RPM maintained and in sync in all supported distributions and versions. This may mean potential RHEL ZStream updates. Fixing any issue would mean fixing it everywhere, potentially backporting (although an easy) patch. The complexity of the fixes isn't what I'm worried about; it is the potentially huge matrix of updates that need to happen and stay in sync.
• If the user needs to build the container image with Anaconda and dependencies, is it such a difference that they would have to specify multiple packages instead of a single one? If any tooling builds the container for the user, providing the virtual package makes little difference (if any).
• image-builder SRPM, in my opinion, is not the right place for this. If anything, anaconda could be the one defining a virtual package with dependencies that installs everything for a working installer environment. Alternatively, a COMPS group could also be a better solution.

bootc-base-imgctl equivalent for anaconda

I don't have any experience with this script, but from your description, it doesn't sound like an ideal solution. But it is also not the worst if contributions to the repo where it lives is relatively easy to contribute to.

Bootc-image-builder automatically creates derived container

I actually like this solution the most. From my point of view, we already do this for the package-based image-installer ISOs. We hide entirely from the user how we assemble the installer image; the user can only modify the payload via BP. In a sense, this is similar to the user providing the payload as a bootc container and we take care of the installer environment.

(This solution would enable the use of installer customizations in the BP even for bootc-based installer images; EDIT: I didn't realize that the way the fstree content is created does not affect our currently supported installer customizations, so this comment is not relevant)

Just Document what is needed

Documenting things never hurts, so I agree with documenting this in any case.

[achilleas-k]

We talked about this elsewhere but I'd like to summarise some of the discussion here for future reference (and anyone else that might be following).

couldn't this be done by installing it traditionally as packages using the content from the repositories in the bootc container? In a sense, nobody will be running the installer environment for anything other than the installation, so it is a throwaway, one-time-use environment. The host system installed by the installer image is the part that must be entirely based on the bootc container. I'm definitely missing the motivation for making our lives more complicated than necessary to achieve this.

What you describe is the current situation. When building an ISO for a bootc container, we mount the container to read its repository configurations during the depsolve so that we know we are using the same content sources that were used to create the container itself. There are a couple of issues with this:

1. RHEL containers don't have repository configurations. Repo configs are created automatically by subscription-manager using registration information from the host. To get this to work, we run the container, call a quick dnf <something> in the container to trigger subscription-manager to create the repo configs, and then use those to depsolve and download the RPMs for the ISO. This has been working OK so far, but it's a bit hacky and possibly brittle.
2. Users can make configuration changes to the repositories in the base container that can cause issues with the installation of the Anaconda packages. This new approach doesn't solve the issue directly, they'll have the same issues when they try to install the package for the derived container for Anaconda, but it puts the creation of the ISO container on the user in a way that lets them (or sorta forces them) to troubleshoot the situation with the tooling they're already working with (container builds).

There's also the matter of the future plans of supporting bootc-based image types in the Image Builder service. Pulling a user's Anaconda container and packaging it into an ISO for them with their base bootc container embedded, ready for installation, is probably going to be a lot less of an issue for us than reading the repositories from their base image and pulling RPMs from them, without being able to know or control how those repositories might be configured.

To be perfectly honest here, I'm personally more in favour of creating the ISO the way we know how and have been doing for ages. But a big part of the idea of building images in the bootc way is putting image building tools into the hands of users that feel familiar (Containerfile, podman build, or container build tools in general). And I think that goal generally makes our life harder, because building images is not trivial and enabling users to do it right with tools that weren't designed for it is a lot more work than providing them with a tool that's made for the task.