[rgarcia]

Summary

Introduces a small local cache-volume manager for persistent per-scope BuildKit cache volumes, gated by build.cache.local.enabled (default false — the disabled runtime path is unchanged). This layer adds the lifecycle machinery only; attaching volumes to builds lands in the PR above.

Behavior:

• Volume IDs are build-cache-<sha256(effective-scope)>; each volume is created at a configurable fixed size (build.cache.local.size_gb, default 50 GB).
• Per-scope serialization via lockScope, so concurrent builds for one scope can be serialized by callers.
• Last-used metadata persisted to builds/cache-volumes.json (atomic write) and reloaded on restart.
• A reaper (started from Manager.Start only when enabled) deletes volumes idle past idle_ttl (default 24h) and evicts idle volumes LRU-first to satisfy host-wide max_bytes / max_volumes limits.
• Attached volumes and volumes guarded by an in-flight build (acquireVolume) are never evicted, including the window between creation and attachment.
• Enabling the local cache implies identity-derived tenant scopes (resolveEffectiveCacheScope treats cache.local.enabled like cache.derive_tenant_scope), since the per-scope volumes require a non-empty scope.

Changes

• lib/builds/cache_volume.go: CacheVolumeConfig, cacheVolumeManager (ID derivation, ensure/create, per-scope locks, in-use guards, last-used persistence, reaper with TTL + LRU byte/count eviction).
• lib/builds/manager.go: Config.Cache; manager constructed only when enabled; reaper launched in Start.
• cmd/api/api/builds.go: cache.local.enabled also enables tenant scope derivation.
• Config plumbing: build.cache.local.* in cmd/api/config (with validation), lib/providers, config.example.yaml.

Tests

lib/builds/cache_volume_test.go:

• ID derivation (deterministic, build-cache-<64 hex>, distinct per scope / tenant isolation)
• ensure: create at fixed/default size, reuse on second call, exactly one create under concurrent same-scope ensures
• locking: same scope serializes, different scopes independent
• eviction: idle TTL delete/keep, LRU under max_volumes and max_bytes, attached and in-use volumes never evicted (including limit pressure)
• last-used persistence across manager restarts; NewManager wiring (nil when disabled)

cmd/api/api/builds_cache_scope_test.go: cache.local.enabled implies the identity-derived scope for ordinary callers.

go test -race ./lib/builds/..., go test ./cmd/api/config/... ./lib/providers/..., and go build ./... pass.

---

Note

Medium Risk
Touches volume APIs, instance attachment rules, and build cache scope behavior when local cache is enabled; default-off path is unchanged but misconfiguration could derive tenant scopes or evict cache volumes unexpectedly.

Overview
Adds persistent per-scope BuildKit cache volume lifecycle behind build.cache.local.enabled (default off). When enabled, a cacheVolumeManager creates volumes as build-cache-<sha256(scope)>, tracks last use in cache-volumes.json, serializes work per scope, and runs a background reaper for idle TTL plus LRU eviction under max_bytes / max_volumes without touching attached or in-flight volumes.

Enabling local cache also turns on identity-derived tenant cache scopes (same as derive_tenant_scope), since each volume needs a non-empty scope. Config is wired through API config validation, providers, and config.example.yaml.

Hardens internal build volumes so callers cannot squat on or spoof system resources: reserved ID prefixes (build-cache-, build-disk-, etc.) and hypeman.system/ tags are rejected on volume create/archive and delete; non-internal instance attachments block those IDs unless allowSystemPaths is set.

^{Reviewed by Cursor Bugbot for commit 1e84163. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Failed deletes skew eviction accounting
  • Reaper accounting now only removes volumes from TTL/limit tracking after successful deletes so failed evictions remain counted for subsequent limit decisions.

Or push these changes by commenting:

@cursor push 84f9175271

Preview (84f9175271)

diff --git a/lib/builds/cache_volume.go b/lib/builds/cache_volume.go
--- a/lib/builds/cache_volume.go
+++ b/lib/builds/cache_volume.go
@@ -258,21 +258,23 @@
 		return v.CreatedAt
 	}
 
-	deleteLocked := func(v volumes.Volume) {
+	deleteLocked := func(v volumes.Volume) bool {
 		if err := c.volumeManager.DeleteVolume(ctx, v.Id); err != nil {
 			c.logger.Warn("failed to evict cache volume", "volume", v.Id, "error", err)
-			return
+			return false
 		}
 		c.logger.Info("evicted cache volume", "volume", v.Id)
 		delete(c.lastUsed, v.Id)
+		return true
 	}
 
 	// Idle TTL eviction
 	remaining := make([]volumes.Volume, 0, len(cacheVols))
 	for _, v := range cacheVols {
 		if now.Sub(lastUsedLocked(v)) > c.config.IdleTTL && deletableLocked(v) {
-			deleteLocked(v)
-			continue
+			if deleteLocked(v) {
+				continue
+			}
 		}
 		remaining = append(remaining, v)
 	}
@@ -292,9 +294,10 @@
 		overBytes := c.config.MaxBytes > 0 && totalBytes > c.config.MaxBytes
 		overCount := c.config.MaxVolumes > 0 && present > c.config.MaxVolumes
 		if (overBytes || overCount) && deletableLocked(v) {
-			deleteLocked(v)
-			totalBytes -= int64(v.SizeGb) * 1024 * 1024 * 1024
-			present--
+			if deleteLocked(v) {
+				totalBytes -= int64(v.SizeGb) * 1024 * 1024 * 1024
+				present--
+			}
 		}
 	}
 

diff --git a/lib/builds/cache_volume_test.go b/lib/builds/cache_volume_test.go
--- a/lib/builds/cache_volume_test.go
+++ b/lib/builds/cache_volume_test.go
@@ -329,6 +329,87 @@
 	assert.ErrorIs(t, err, volumes.ErrNotFound, "idle volume evicted instead")
 }
 
+func TestReap_TTLDeleteFailureStillCountsForLimits(t *testing.T) {
+	mgr, volumeMgr, now, tempDir := setupCacheVolumeManager(t, CacheVolumeConfig{
+		Enabled:    true,
+		SizeGB:     10,
+		IdleTTL:    time.Hour,
+		MaxVolumes: 1,
+	})
+	defer os.RemoveAll(tempDir)
+
+	ctx := context.Background()
+	base := *now
+
+	oldVol, err := mgr.ensureCacheVolume(ctx, "tenant-old")
+	require.NoError(t, err)
+	*now = base.Add(-3 * time.Hour)
+	mgr.touchLastUsed(oldVol)
+
+	freshVol, err := mgr.ensureCacheVolume(ctx, "tenant-fresh")
+	require.NoError(t, err)
+	*now = base.Add(-30 * time.Minute)
+	mgr.touchLastUsed(freshVol)
+	*now = base
+
+	volumeMgr.deleteFunc = func(ctx context.Context, id string) error {
+		if id == oldVol {
+			return volumes.ErrInUse
+		}
+		delete(volumeMgr.volumes, id)
+		return nil
+	}
+
+	mgr.reap(ctx)
+
+	_, err = volumeMgr.GetVolume(ctx, oldVol)
+	assert.NoError(t, err, "failed TTL delete keeps original volume")
+
+	_, err = volumeMgr.GetVolume(ctx, freshVol)
+	assert.ErrorIs(t, err, volumes.ErrNotFound, "limit eviction still runs with failed TTL delete")
+}
+
+func TestReap_LimitDeleteFailureStillEnforcesMaxVolumes(t *testing.T) {
+	mgr, volumeMgr, now, tempDir := setupCacheVolumeManager(t, CacheVolumeConfig{
+		Enabled:    true,
+		SizeGB:     10,
+		IdleTTL:    100 * time.Hour,
+		MaxVolumes: 1,
+	})
+	defer os.RemoveAll(tempDir)
+
+	ctx := context.Background()
+	base := *now
+	volIDs := make([]string, 3)
+	for i, scope := range []string{"tenant-1", "tenant-2", "tenant-3"} {
+		volID, err := mgr.ensureCacheVolume(ctx, scope)
+		require.NoError(t, err)
+		*now = base.Add(time.Duration(i-3) * time.Hour)
+		mgr.touchLastUsed(volID)
+		volIDs[i] = volID
+	}
+	*now = base
+
+	volumeMgr.deleteFunc = func(ctx context.Context, id string) error {
+		if id == volIDs[0] {
+			return volumes.ErrInUse
+		}
+		delete(volumeMgr.volumes, id)
+		return nil
+	}
+
+	mgr.reap(ctx)
+
+	_, err := volumeMgr.GetVolume(ctx, volIDs[0])
+	assert.NoError(t, err, "failed delete volume remains present")
+
+	_, err = volumeMgr.GetVolume(ctx, volIDs[1])
+	assert.ErrorIs(t, err, volumes.ErrNotFound, "next oldest volume evicted to satisfy limit")
+
+	_, err = volumeMgr.GetVolume(ctx, volIDs[2])
+	assert.ErrorIs(t, err, volumes.ErrNotFound, "eviction continues until count limit is met")
+}
+
 func TestCacheVolumeStatePersistence(t *testing.T) {
 	config := CacheVolumeConfig{Enabled: true, SizeGB: 10}
 	mgr, _, now, tempDir := setupCacheVolumeManager(t, config)

_{You can send follow-ups to the cloud agent here.}

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Reaper matches user volume IDs
  • The reaper now only considers volumes tracked by ensureCacheVolume via persisted last-used metadata, so user-created volumes that merely share the build-cache- prefix are no longer evicted.

Or push these changes by commenting:

@cursor push 692a791b81

Preview (692a791b81)

diff --git a/lib/builds/cache_volume.go b/lib/builds/cache_volume.go
--- a/lib/builds/cache_volume.go
+++ b/lib/builds/cache_volume.go
@@ -11,7 +11,6 @@ import (
 	"os"
 	"path/filepath"
 	"sort"
-	"strings"
 	"sync"
 	"time"
 

@@ -11,7 +11,6 @@ import (
 	"os"
 	"path/filepath"
 	"sort"
-	"strings"
 	"sync"
 	"time"
 
@@ -237,7 +236,10 @@ func (c *cacheVolumeManager) reap(ctx context.Context) {
 
 	cacheVols := make([]volumes.Volume, 0)
 	for _, v := range vols {
-		if strings.HasPrefix(v.Id, cacheVolumeIDPrefix) {
+		// Only volumes that passed through ensureCacheVolume are managed by
+		// this reaper, so user-created volumes that happen to share the prefix
+		// are never swept.
+		if _, tracked := c.lastUsed[v.Id]; tracked {
 			cacheVols = append(cacheVols, v)
 		}
 	}

@@ -237,7 +236,10 @@ func (c *cacheVolumeManager) reap(ctx context.Context) {
 
 	cacheVols := make([]volumes.Volume, 0)
 	for _, v := range vols {
-		if strings.HasPrefix(v.Id, cacheVolumeIDPrefix) {
+		// Only volumes that passed through ensureCacheVolume are managed by
+		// this reaper, so user-created volumes that happen to share the prefix
+		// are never swept.
+		if _, tracked := c.lastUsed[v.Id]; tracked {
 			cacheVols = append(cacheVols, v)
 		}
 	}

diff --git a/lib/builds/cache_volume_test.go b/lib/builds/cache_volume_test.go
--- a/lib/builds/cache_volume_test.go
+++ b/lib/builds/cache_volume_test.go
@@ -176,6 +176,41 @@ func TestReap_IdleTTLDeletesOldVolumes(t *testing.T) {
 	assert.NoError(t, err, "fresh volume kept")
 }
 
+func TestReap_DoesNotEvictUntrackedPrefixedVolumes(t *testing.T) {
+	mgr, volumeMgr, now, tempDir := setupCacheVolumeManager(t, CacheVolumeConfig{
+		Enabled: true,
+		SizeGB:  10,
+		IdleTTL: time.Hour,
+	})
+	defer os.RemoveAll(tempDir)
+
+	ctx := context.Background()
+	base := *now
+	oldCacheVol, err := mgr.ensureCacheVolume(ctx, "tenant-old")
+	require.NoError(t, err)
+
+	*now = base.Add(-2 * time.Hour)
+	mgr.touchLastUsed(oldCacheVol)
+	*now = base
+
+	userVolID := cacheVolumeIDPrefix + "user-volume"
+	_, err = volumeMgr.CreateVolume(ctx, volumes.CreateVolumeRequest{
+		Id:     &userVolID,
+		Name:   userVolID,
+		SizeGb: 10,
+	})
+	require.NoError(t, err)
+	volumeMgr.volumes[userVolID].CreatedAt = base.Add(-2 * time.Hour)
+
+	mgr.reap(ctx)
+
+	_, err = volumeMgr.GetVolume(ctx, oldCacheVol)
+	assert.ErrorIs(t, err, volumes.ErrNotFound, "tracked cache volume should still be evicted")
+
+	_, err = volumeMgr.GetVolume(ctx, userVolID)
+	assert.NoError(t, err, "untracked user volume with cache prefix must not be evicted")
+}
+
 func TestReap_NeverEvictsAttachedVolume(t *testing.T) {
 	mgr, volumeMgr, now, tempDir := setupCacheVolumeManager(t, CacheVolumeConfig{
 		Enabled: true,

@@ -176,6 +176,41 @@ func TestReap_IdleTTLDeletesOldVolumes(t *testing.T) {
 	assert.NoError(t, err, "fresh volume kept")
 }
 
+func TestReap_DoesNotEvictUntrackedPrefixedVolumes(t *testing.T) {
+	mgr, volumeMgr, now, tempDir := setupCacheVolumeManager(t, CacheVolumeConfig{
+		Enabled: true,
+		SizeGB:  10,
+		IdleTTL: time.Hour,
+	})
+	defer os.RemoveAll(tempDir)
+
+	ctx := context.Background()
+	base := *now
+	oldCacheVol, err := mgr.ensureCacheVolume(ctx, "tenant-old")
+	require.NoError(t, err)
+
+	*now = base.Add(-2 * time.Hour)
+	mgr.touchLastUsed(oldCacheVol)
+	*now = base
+
+	userVolID := cacheVolumeIDPrefix + "user-volume"
+	_, err = volumeMgr.CreateVolume(ctx, volumes.CreateVolumeRequest{
+		Id:     &userVolID,
+		Name:   userVolID,
+		SizeGb: 10,
+	})
+	require.NoError(t, err)
+	volumeMgr.volumes[userVolID].CreatedAt = base.Add(-2 * time.Hour)
+
+	mgr.reap(ctx)
+
+	_, err = volumeMgr.GetVolume(ctx, oldCacheVol)
+	assert.ErrorIs(t, err, volumes.ErrNotFound, "tracked cache volume should still be evicted")
+
+	_, err = volumeMgr.GetVolume(ctx, userVolID)
+	assert.NoError(t, err, "untracked user volume with cache prefix must not be evicted")
+}
+
 func TestReap_NeverEvictsAttachedVolume(t *testing.T) {
 	mgr, volumeMgr, now, tempDir := setupCacheVolumeManager(t, CacheVolumeConfig{
 		Enabled: true,

_{You can send follow-ups to the cloud agent here.}

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Reserved volumes still deletable
  • Added the reserved-prefix guard to DeleteVolume and a regression test so API callers now receive a conflict response instead of deleting internal build-managed volumes.

Or push these changes by commenting:

@cursor push 9b8f221c09

Preview (9b8f221c09)

diff --git a/cmd/api/api/volumes.go b/cmd/api/api/volumes.go
--- a/cmd/api/api/volumes.go
+++ b/cmd/api/api/volumes.go
@@ -181,6 +181,12 @@
 		}, nil
 	}
 	log := logger.FromContext(ctx)
+	if prefix := builds.ReservedVolumeIDPrefix(vol.Id); prefix != "" {
+		return oapi.DeleteVolume409JSONResponse{
+			Code:    "conflict",
+			Message: fmt.Sprintf("volume IDs with the prefix %q are reserved for internal use", prefix),
+		}, nil
+	}
 
 	err := s.VolumeManager.DeleteVolume(ctx, vol.Id)
 	if err != nil {

diff --git a/cmd/api/api/volumes_test.go b/cmd/api/api/volumes_test.go
--- a/cmd/api/api/volumes_test.go
+++ b/cmd/api/api/volumes_test.go
@@ -4,6 +4,7 @@
 	"testing"
 
 	"github.com/kernel/hypeman/lib/oapi"
+	"github.com/kernel/hypeman/lib/volumes"
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
 )
@@ -114,3 +115,26 @@
 	require.True(t, ok, "expected 400 for reserved ID")
 	assert.Contains(t, bad.Message, "reserved for internal use")
 }
+
+func TestDeleteVolume_ReservedIDPrefix(t *testing.T) {
+	t.Parallel()
+	svc := newTestService(t)
+
+	for _, id := range []string{"build-cache-abc", "build-disk-1", "build-source-1", "build-config-1"} {
+		_, err := svc.VolumeManager.CreateVolume(ctx(), volumes.CreateVolumeRequest{
+			Id:     &id,
+			Name:   id,
+			SizeGb: 1,
+		})
+		require.NoError(t, err)
+
+		resp, err := svc.DeleteVolume(ctxWithVolume(svc, id), oapi.DeleteVolumeRequestObject{Id: id})
+		require.NoError(t, err)
+		conflict, ok := resp.(oapi.DeleteVolume409JSONResponse)
+		require.True(t, ok, "expected 409 for reserved ID %q", id)
+		assert.Contains(t, conflict.Message, "reserved for internal use")
+
+		_, err = svc.VolumeManager.GetVolume(ctx(), id)
+		require.NoError(t, err, "reserved volume %q should not be deleted", id)
+	}
+}

_{You can send follow-ups to the cloud agent here.}

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Reaper deletes volumes by spoofable tag
  • The reaper now treats a volume as managed only when it has both the internal managed tag and the reserved build-cache- ID prefix, and a regression test confirms tagged non-cache volumes are never evicted.

Or push these changes by commenting:

@cursor push fc7e115734

Preview (fc7e115734)

diff --git a/lib/builds/cache_volume.go b/lib/builds/cache_volume.go
--- a/lib/builds/cache_volume.go
+++ b/lib/builds/cache_volume.go
@@ -11,6 +11,7 @@
 	"os"
 	"path/filepath"
 	"sort"
+	"strings"
 	"sync"
 	"time"
 
@@ -175,10 +176,11 @@
 	return volID, nil
 }
 
-// isManagedCacheVolume reports whether a volume carries the internal tag
-// marking it as created by the build cache manager.
+// isManagedCacheVolume reports whether a volume has both the reserved cache
+// ID prefix and the internal managed tag set by the build cache manager.
 func isManagedCacheVolume(vol *volumes.Volume) bool {
-	return vol.Tags[cacheVolumeManagedTagKey] == cacheVolumeManagedTagValue
+	return strings.HasPrefix(vol.Id, cacheVolumeIDPrefix) &&
+		vol.Tags[cacheVolumeManagedTagKey] == cacheVolumeManagedTagValue
 }
 
 // touchLastUsed records that a cache volume was used now.

diff --git a/lib/builds/cache_volume_test.go b/lib/builds/cache_volume_test.go
--- a/lib/builds/cache_volume_test.go
+++ b/lib/builds/cache_volume_test.go
@@ -150,6 +150,33 @@
 	assert.NoError(t, err, "unmanaged volume must never be evicted")
 }
 
+func TestReap_IgnoresTaggedNonCachePrefixVolume(t *testing.T) {
+	mgr, volumeMgr, now, tempDir := setupCacheVolumeManager(t, CacheVolumeConfig{
+		Enabled: true,
+		SizeGB:  10,
+		IdleTTL: time.Hour,
+	})
+	defer os.RemoveAll(tempDir)
+
+	// A caller-created volume spoofing the managed cache tag must not be
+	// treated as a cache volume unless it also has the reserved cache prefix.
+	spoofedVol := &volumes.Volume{
+		Id:     "user-data-1",
+		Name:   "user-data",
+		SizeGb: 10,
+		Tags: tags.Tags{
+			cacheVolumeManagedTagKey: cacheVolumeManagedTagValue,
+		},
+		CreatedAt: now.Add(-72 * time.Hour),
+	}
+	volumeMgr.volumes[spoofedVol.Id] = spoofedVol
+
+	mgr.reap(context.Background())
+
+	_, err := volumeMgr.GetVolume(context.Background(), spoofedVol.Id)
+	assert.NoError(t, err, "tag-spoofed non-cache volume must never be evicted")
+}
+
 func TestLockScope_SerializesSameScope(t *testing.T) {
 	mgr, _, _, tempDir := setupCacheVolumeManager(t, CacheVolumeConfig{Enabled: true})
 	defer os.RemoveAll(tempDir)

_{You can send follow-ups to the cloud agent here.}

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Ensure races with cache reaper
  • ensureCacheVolume now acquires the volume in-use guard for the full ensure path so the reaper cannot evict the volume between existence check and last-used touch.

Or push these changes by commenting:

@cursor push 1f421f1b64

Preview (1f421f1b64)

diff --git a/lib/builds/cache_volume.go b/lib/builds/cache_volume.go
--- a/lib/builds/cache_volume.go
+++ b/lib/builds/cache_volume.go
@@ -143,6 +143,8 @@
 // configured fixed size if it does not exist yet, and marks it used.
 func (c *cacheVolumeManager) ensureCacheVolume(ctx context.Context, scope string) (string, error) {
 	volID := cacheVolumeID(scope)
+	release := c.acquireVolume(volID)
+	defer release()
 
 	if vol, err := c.volumeManager.GetVolume(ctx, volID); err == nil {
 		if !isManagedCacheVolume(vol) {

_{You can send follow-ups to the cloud agent here.}

^{Reviewed by Cursor Bugbot for commit 1e84163. Configure here.}

[rgarcia]

Closing in favor of a fresh first-class Builder resource stack. Useful implementation and test work from this PR will be selectively reapplied in smaller reviewable layers.