[rgarcia]

Problem

The builder agent unconditionally mounts a 3 GB tmpfs at /var/lib/buildkit (needed because the VM rootfs is overlayfs, and BuildKit's native overlayfs snapshotter can't create whiteout devices on a nested overlayfs). This hard-caps BuildKit's snapshot storage: any build whose layer data exceeds 3 GB fails, and because it's tmpfs, that storage also competes with the build's RAM budget.

Reproduced locally with a Dockerfile writing 3×1.2 GB layers:

Mounted tmpfs at /var/lib/buildkit for BuildKit snapshotter
#7 overlay  3.0G  2.4G  660M  79% /
#8 dd: error writing '/big3.bin': No space left on device
→ build failed

Fix

If /var/lib/buildkit is already a mountpoint (e.g. a dedicated volume attached to the builder VM), use it directly. Otherwise the tmpfs fallback is preserved exactly (same mount -t tmpfs -o size=3G command, same error messages) — the unmounted path is byte-for-byte the current behavior.

Changes

• Extract the mount setup in runBuild into ensureBuildkitRoot(root, isMounted, mountTmpfs) with the mount check and tmpfs command injected, so both paths are testable without privileged mounts.
• isMountPoint checks /proc/self/mounts; mountBuildkitTmpfs runs the unchanged tmpfs mount command.
• The nested-overlayfs rationale comment is preserved on ensureBuildkitRoot.

No changes to the host manager, API, volume attachment, registry behavior, or default runtime behavior.

Verification

Ran the same 3.6 GB build end-to-end with a 15 GB ext4 volume pre-attached at /var/lib/buildkit:

Using existing mount at /var/lib/buildkit for BuildKit
#7 overlay 14.7G  2.4G  11.5G  17% /
→ status: ready, image pushed

This also confirms the whiteout concern doesn't apply to the mounted path: BuildKit's native overlayfs snapshotter runs fine on an ext4 block volume.

Caveat: hypeman itself can't attach a volume there today — isSystemDirectory rejects /var/... mount paths, so the end-to-end test required temporarily attaching the volume from executeBuild and carving out /var/lib/buildkit from that guard. Wiring up the host side (volume attach + guard exception) is follow-up work; until then this path is only reachable via out-of-band orchestration.

Security considerations

The agent trusts whatever is mounted at /var/lib/buildkit, so the follow-up wiring MUST attach a fresh per-build volume, deleted with the VM. Sharing the snapshotter root across builds/tenants would enable cross-tenant cache poisoning (BuildKit caches RUN steps) and disclosure of prior builds' layers/source. Cross-tenant cache sharing already has a safe, scoped mechanism (registry-based cache_scope import/export). Documented on ensureBuildkitRoot.

Tests

• go test ./lib/builds/builder_agent/ — new tests: already-mounted root is accepted without attempting tmpfs; unmounted root mounts tmpfs; mount failure propagates.
• go test ./lib/builds/... and go build ./... pass.

---

Note

Medium Risk
Changes privileged mount behavior on builder VMs; wrong or shared mounts at /var/lib/buildkit could affect build isolation until host volume wiring is in place.

Overview
BuildKit snapshot storage no longer always gets a 3 GB tmpfs at /var/lib/buildkit. If that path is already a mountpoint (e.g. a dedicated block volume), the agent uses it as-is and skips tmpfs, avoiding the layer-size cap and RAM pressure from large builds.

Unmounted path is unchanged: ensureBuildkitRoot still creates the directory and runs the same mount -t tmpfs -o size=3G via mountBuildkitTmpfs, with the nested-overlayfs rationale kept on the helper. Mount detection uses isMountPoint reading /proc/self/mounts.

Tests inject isMounted and mountTmpfs to cover already-mounted (no tmpfs), unmounted (tmpfs + mkdir), and mount failure propagation.

^{Reviewed by Cursor Bugbot for commit b66a729. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Mount detection misses init volumes
  • Mount-point detection now matches both the guest path and its pre-chroot /overlay/newroot/... mount entry so attached BuildKit volumes are recognized and tmpfs is not mounted over them.

Or push these changes by commenting:

@cursor push b6c7734af7

Preview (b6c7734af7)

diff --git a/lib/builds/builder_agent/main.go b/lib/builds/builder_agent/main.go
--- a/lib/builds/builder_agent/main.go
+++ b/lib/builds/builder_agent/main.go
@@ -941,20 +941,38 @@
 }
 
 // isMountPoint reports whether path is a mountpoint according to /proc/self/mounts.
+// In exec mode, init mounts guest volumes before chroot at /overlay/newroot/<path>,
+// so check both the guest path and its pre-chroot namespace path.
 func isMountPoint(path string) (bool, error) {
 	data, err := os.ReadFile("/proc/self/mounts")
 	if err != nil {
 		return false, err
 	}
+	candidates := mountPathCandidates(path, "")
+	if processRoot, err := os.Readlink("/proc/self/root"); err == nil {
+		candidates = mountPathCandidates(path, processRoot)
+	}
 	for _, line := range strings.Split(string(data), "\n") {
 		fields := strings.Fields(line)
-		if len(fields) >= 2 && fields[1] == path {
-			return true, nil
+		if len(fields) >= 2 {
+			if _, ok := candidates[fields[1]]; ok {
+				return true, nil
+			}
 		}
 	}
 	return false, nil
 }
 
+func mountPathCandidates(path string, processRoot string) map[string]struct{} {
+	candidates := map[string]struct{}{
+		filepath.Clean(path): {},
+	}
+	if processRoot != "" {
+		candidates[filepath.Clean(filepath.Join(processRoot, path))] = struct{}{}
+	}
+	return candidates
+}
+
 func extractDigest(metadataPath string) (string, error) {
 	data, err := os.ReadFile(metadataPath)
 	if err != nil {

diff --git a/lib/builds/builder_agent/main_test.go b/lib/builds/builder_agent/main_test.go
--- a/lib/builds/builder_agent/main_test.go
+++ b/lib/builds/builder_agent/main_test.go
@@ -55,3 +55,10 @@
 
 	require.ErrorContains(t, err, "mount failed")
 }
+
+func TestMountPathCandidatesIncludesChrootMountPath(t *testing.T) {
+	candidates := mountPathCandidates("/var/lib/buildkit", "/overlay/newroot")
+
+	assert.Contains(t, candidates, "/var/lib/buildkit")
+	assert.Contains(t, candidates, "/overlay/newroot/var/lib/buildkit")
+}

_{You can send follow-ups to the cloud agent here.}

^{Reviewed by Cursor Bugbot for commit 7114b82. Configure here.}

[rgarcia]

Closing in favor of a fresh first-class Builder resource stack. Useful implementation and test work from this PR will be selectively reapplied in smaller reviewable layers.