Repository navigation
builds: run builds on Builders with persistent cache disks - #342
Conversation
✱ Stainless preview builds for hypemanThis PR will update the Edit this comment to update it. It will appear in the SDK's changelogs. ✅ hypeman-openapi studio · code · diff
|
8c5aaf3 to
295e1b2
Compare
295e1b2 to
4c5bce4
Compare
4c5bce4 to
b9e1a96
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
Autofix Details
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: Non-ready builders accepted at create
- CreateBuild now rejects existing builders whose status is not ready by returning builders.ErrInUse before queueing the build, with a regression test covering non-ready builder states.
Or push these changes by commenting:
@cursor push a3aa68f4d2
Preview (a3aa68f4d2)
diff --git a/lib/builds/builder_disk_test.go b/lib/builds/builder_disk_test.go
--- a/lib/builds/builder_disk_test.go
+++ b/lib/builds/builder_disk_test.go
@@ -149,6 +149,32 @@
require.ErrorIs(t, err, builders.ErrNotFound)
}
+// TestCreateBuild_BuilderNotReady verifies builds targeting an existing
+// builder that is not ready fail at create time.
+func TestCreateBuild_BuilderNotReady(t *testing.T) {
+ mgr, _, volumeMgr, tempDir := setupTestManager(t)
+ defer os.RemoveAll(tempDir)
+
+ ctx := context.Background()
+ builder, err := mgr.builderManager.CreateBuilder(ctx, builders.CreateBuilderRequest{DiskSizeGb: 20})
+ require.NoError(t, err)
+
+ volumeMgr.deleteFunc = func(context.Context, string) error {
+ return errors.New("delete failed")
+ }
+ require.NoError(t, mgr.builderManager.ResetDisk(ctx, builder.ID))
+ require.Eventually(t, func() bool {
+ got, err := mgr.builderManager.GetBuilder(ctx, builder.ID)
+ return err == nil && got.Status == builders.StatusError
+ }, 2*time.Second, 10*time.Millisecond)
+
+ _, err = mgr.CreateBuild(ctx, CreateBuildRequest{
+ Dockerfile: "FROM alpine",
+ BuilderID: builder.ID,
+ }, []byte("src"))
+ require.ErrorIs(t, err, builders.ErrInUse)
+}
+
// TestCreateBuild_GCBoundsWrittenForBuilder verifies the host computes
// BuildKit GC bounds from the builder disk size into the guest config, and
// leaves them zero for tmpfs-backed builds.
diff --git a/lib/builds/manager.go b/lib/builds/manager.go
--- a/lib/builds/manager.go
+++ b/lib/builds/manager.go
@@ -413,6 +413,9 @@
if err != nil {
return nil, err
}
+ if b.Status != builders.StatusReady {
+ return nil, builders.ErrInUse
+ }
builder = b
}You can send follow-ups to the cloud agent here.
b9e1a96 to
c9390a3
Compare
c9390a3 to
9108276
Compare
b61b3ef to
fcd962b
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.
Autofix Details
Bugbot Autofix prepared fixes for both issues found in the latest run.
- ✅ Fixed: Activity checker misses disk builds
- Builder activity checks now fall back to pending build metadata on disk so delete/prune/reaper treat unrecovered queued or building jobs as in-use.
- ✅ Fixed: Failed release bricks builder queue
- Builder release is now retried until success and the serial key is only released after that hold is actually cleared, preventing stale in-memory acquisitions from advancing same-builder builds.
Or push these changes by commenting:
@cursor push 79c87e01c2
Preview (79c87e01c2)
diff --git a/lib/builds/builder_disk_test.go b/lib/builds/builder_disk_test.go
--- a/lib/builds/builder_disk_test.go
+++ b/lib/builds/builder_disk_test.go
@@ -362,3 +362,79 @@
err = mgr.builderManager.DeleteBuilder(ctx, builder.ID)
assert.ErrorIs(t, err, builders.ErrInUse, "delete must be blocked by a queued-then-running build")
}
+
+// TestBuilderDeleteAndPruneBlockedByDiskPendingBuilds verifies that builders
+// remain protected across restarts while pending build metadata exists on disk
+// before recovery has re-enqueued the builds.
+func TestBuilderDeleteAndPruneBlockedByDiskPendingBuilds(t *testing.T) {
+ mgr, _, _, tempDir := setupTestManager(t)
+ defer os.RemoveAll(tempDir)
+
+ ctx := context.Background()
+ builder, err := mgr.builderManager.CreateBuilder(ctx, builders.CreateBuilderRequest{})
+ require.NoError(t, err)
+ mgr.builderManager.SetBuildActivityChecker(mgr.BuilderHasBuilds)
+
+ req := CreateBuildRequest{
+ Dockerfile: "FROM alpine",
+ BuilderID: builder.ID,
+ }
+ prepareBuildOnDisk(t, mgr, "build-on-disk", req)
+
+ require.True(t, mgr.BuilderHasBuilds(builder.ID), "disk-pending builds must be treated as in-flight")
+
+ err = mgr.builderManager.DeleteBuilder(ctx, builder.ID)
+ assert.ErrorIs(t, err, builders.ErrInUse, "delete must be blocked by disk-pending builds")
+
+ err = mgr.builderManager.ResetDisk(ctx, builder.ID)
+ assert.ErrorIs(t, err, builders.ErrInUse, "prune must be blocked by disk-pending builds")
+}
+
+type flakyReleaseBuilderManager struct {
+ builders.Manager
+ failReleases int
+ releaseCalls int
+}
+
+func (m *flakyReleaseBuilderManager) ReleaseBuild(ctx context.Context, id string, buildID string) error {
+ m.releaseCalls++
+ if m.failReleases > 0 {
+ m.failReleases--
+ return errors.New("transient release failure")
+ }
+ return m.Manager.ReleaseBuild(ctx, id, buildID)
+}
+
+// TestExecuteBuild_RetriesBuilderRelease verifies executeBuild retries
+// ReleaseBuild failures so transient persistence errors do not leave a stale
+// in-memory acquisition that bricks later same-builder builds.
+func TestExecuteBuild_RetriesBuilderRelease(t *testing.T) {
+ mgr, instanceMgr, _, tempDir := setupTestManager(t)
+ defer os.RemoveAll(tempDir)
+
+ ctx := context.Background()
+ builder, err := mgr.builderManager.CreateBuilder(ctx, builders.CreateBuilderRequest{DiskSizeGb: 20})
+ require.NoError(t, err)
+
+ flaky := &flakyReleaseBuilderManager{
+ Manager: mgr.builderManager,
+ failReleases: 1,
+ }
+ mgr.builderManager = flaky
+
+ req := CreateBuildRequest{
+ Dockerfile: "FROM alpine\nRUN echo hello",
+ BuilderID: builder.ID,
+ }
+ prepareBuildOnDisk(t, mgr, "build-1", req)
+ stoppedBuilderInstance(instanceMgr, nil)
+
+ policy := DefaultBuildPolicy()
+ _, err = mgr.executeBuild(ctx, "build-1", req, &policy)
+ require.NoError(t, err)
+ assert.Equal(t, 2, flaky.releaseCalls, "executeBuild must retry a failed builder release")
+
+ _, err = mgr.builderManager.AcquireForBuild(ctx, builder.ID, "build-2")
+ require.NoError(t, err, "builder hold should be cleared after retry succeeds")
+ require.NoError(t, mgr.builderManager.ReleaseBuild(ctx, builder.ID, "build-2"))
+}
diff --git a/lib/builds/manager.go b/lib/builds/manager.go
--- a/lib/builds/manager.go
+++ b/lib/builds/manager.go
@@ -593,9 +593,30 @@
// BuilderHasBuilds reports whether any build targeting the builder is
// queued or running.
func (m *manager) BuilderHasBuilds(builderID string) bool {
- return m.ActiveBuildForBuilder(builderID) != nil || len(m.QueuedBuildsForBuilder(builderID)) > 0
+ if m.ActiveBuildForBuilder(builderID) != nil || len(m.QueuedBuildsForBuilder(builderID)) > 0 {
+ return true
+ }
+ return m.builderHasPendingBuildOnDisk(builderID)
}
+// builderHasPendingBuildOnDisk reports whether restart-recoverable metadata on
+// disk still references the builder as queued or running.
+func (m *manager) builderHasPendingBuildOnDisk(builderID string) bool {
+ pending, err := listPendingBuilds(m.paths)
+ if err != nil {
+ // Be conservative: if we cannot inspect pending metadata, prevent
+ // delete/prune/reap from removing a potentially in-use builder.
+ m.logger.Warn("failed to inspect pending builds on disk", "builder", builderID, "error", err)
+ return true
+ }
+ for _, meta := range pending {
+ if meta.Request != nil && meta.Request.BuilderID == builderID {
+ return true
+ }
+ }
+ return false
+}
+
// storeSource stores the source tarball for a build
func (m *manager) storeSource(buildID string, data []byte) error {
sourceDir := m.paths.BuildSourceDir(buildID)
@@ -630,11 +651,6 @@
// Run the build in a builder VM
result, err := m.executeBuild(buildCtx, id, req, policy)
- // executeBuild's defers release the builder acquisition when it returns,
- // so same-builder builds may start while this build waits for image
- // conversion and re-tagging below.
- m.queue.ReleaseSerialKey(id)
-
duration := time.Since(start)
durationMS := duration.Milliseconds()
@@ -804,9 +820,10 @@
return nil, fmt.Errorf("acquire builder %s: %w", req.BuilderID, err)
}
defer func() {
- if err := m.builderManager.ReleaseBuild(context.Background(), req.BuilderID, id); err != nil {
- m.logger.Error("failed to release builder", "builder", req.BuilderID, "build", id, "error", err)
- }
+ m.releaseBuilderForBuild(req.BuilderID, id)
+ // Same-builder builds may start as soon as the builder hold is
+ // released, even while this build waits on post-build image work.
+ m.queue.ReleaseSerialKey(id)
}()
builderDiskVolID = builders.DiskVolumeID(req.BuilderID)
}
@@ -869,6 +886,33 @@
return result, nil
}
+// releaseBuilderForBuild retries ReleaseBuild until it succeeds. ReleaseBuild
+// intentionally keeps the builder acquired on persistence failure so the caller
+// can retry; without retries, the in-memory hold can outlive the build and
+// block future same-builder builds.
+func (m *manager) releaseBuilderForBuild(builderID, buildID string) {
+ backoff := 100 * time.Millisecond
+ for attempt := 1; ; attempt++ {
+ if err := m.builderManager.ReleaseBuild(context.Background(), builderID, buildID); err != nil {
+ m.logger.Error("failed to release builder; retrying",
+ "builder", builderID,
+ "build", buildID,
+ "attempt", attempt,
+ "retry_in", backoff,
+ "error", err)
+ time.Sleep(backoff)
+ if backoff < 5*time.Second {
+ backoff *= 2
+ if backoff > 5*time.Second {
+ backoff = 5 * time.Second
+ }
+ }
+ continue
+ }
+ return
+ }
+}
+
// waitForResult waits for the build result from the builder agent via vsock
func (m *manager) waitForResult(ctx context.Context, buildID string, inst *instances.Instance) (*BuildResult, error) {
// Wait a bit for the VM to start and the builder agent to listen on vsockYou can send follow-ups to the cloud agent here.
fcd962b to
ce9f7c9
Compare
399dd8d to
7875809
Compare
783be9b to
7075bd2
Compare
sjmiller609
left a comment
There was a problem hiding this comment.
bugs
- manager.go:597 — BuilderHasBuilds takes two separate queue locks; pending→active builds invisible mid-transition
- delete/prune guard relies on that racy predicate; builder deletable under a starting build
findings
- no e2e test: nothing boots a VM, runs buildkitd on ext4, verifies a cache hit. At least good e2e functional test should exist for each new major feature. Made some suggestions about this in one of the review comments.
- manager.go:407 — builder status unchecked at create; doomed builds 201 then fail async at acquire
- main.go:955 — stamp is full buildkitd --version output; sha-only rebuild wipes all caches (intended?)
- serialization invariant assembled from defer ordering across three functions; extract builderLease
- SetBuildActivityChecker setter-injection dodges dependency cycle; hoist check to handler, delete callback
abstract builder tooling - /var/lib/buildkit literal in three packages (agent, builds, instances); define once
nits
- buildSerialKey returns req.BuilderID verbatim; inline it
- activeSerialKeys keyed backwards; map[key]buildID gives O(1) lookups
- canStartLocked inlines serialKeyFreeLocked's scan; call the helper
- queue positions renumber for plain builds too; note in PR body
- no -race stress test on queue serialization invariant
- mock volume manager re-implements real semantics; builders tests use the real one
- resetRoot deletes lost+found; freshDisk preserves it — asymmetric
- 70%/90% GC split undocumented for users sizing disks
- builder_instance_id vs builder_id adjacent with different meanings; clarify descriptions
- active_build_id/queued_builds are point-in-time snapshots; say so in schema
- delete/prune-blocked test anchors current wiring; moves if check hoists to handler
Add optional builder_id to CreateBuild and the Build response; it is the only builder selector. A build with builder_id acquires the builder before attaching, attaches its disk read-write at the exact /var/lib/buildkit path through the internal AllowSystemVolumeMounts carveout, boots the disposable builder VM, and releases the builder after VM execution ends and before the image-conversion wait, stamping last_used_at even on failed builds. Builds for one builder are serialized by builder ID in the queue: a waiting same-builder build stays pending without consuming a global concurrency slot. Builds without builder_id keep the exact tmpfs and queue behavior. The builder API responses gain max_concurrency (fixed at 1), active_build_id, and queued_builds computed from the queue, and builder delete/prune now also reject while builds for the builder are queued or running, via an activity checker the build manager registers. The host computes BuildKit GC bounds from the disk size into the guest config (reservedSpace 70%, maxUsedSpace 90%). The builder agent stamps the persistent root with the buildkitd version and empties the root on incompatibility before buildkitd starts; a fresh disk (no stamp, nothing but lost+found) is stamped without a wipe. Cache contents are best-effort, and no secrets or source/config volumes live on the builder disk. Stale holder recovery happens at acquire: DeleteInstance's volume detach is warn-only, so a crashed build can leak an attachment record and leave its builder VM holding the disk. AcquireForBuild clears those stale attachments before marking the builder held (deleting the stale VM, tolerating its absence, and detaching surviving records), so the next build self-heals instead of failing until a restart runs reconciliation. The builds tests wire the builds and builders managers to one shared volume manager, matching production. Tests cover the disk attach lifecycle and cleanup, the unchanged no-builder path asserted exactly, create-time builder validation, GC bound computation, queue serialization and introspection, stale holder recovery success and loud failure through the real acquire path, delete/prune rejection while builds are queued or running, and the version stamp transitions.
ReleaseBuild keeps the builder acquired when stamping last_used_at fails so the caller can retry, but executeBuild's deferred release only logged the error: a transient metadata write failure leaked the hold and every later build for the builder failed AcquireForBuild with ErrInUse until restart. Retry the release a bounded number of times before giving up. BuilderHasBuilds only inspected the in-memory queue, while startup recovery re-enqueues persisted pending builds asynchronously after the builder image is ready. In that window delete, prune, and the idle reaper could remove a builder with queued or building jobs still on disk. Until recovery completes, BuilderHasBuilds also matches persisted pending builds, failing closed when the listing fails.
ReleaseSerialKey freed the builder's serial key but the pending successor still needed a free global slot, which the releasing build holds through image conversion. At MaxConcurrentBuilds=1 the successor could not start until the prior build fully finished, so the documented overlap (next build's VM starts during the prior build's image conversion) never happened. Release now starts the pending build holding the same serial key directly: the releasing build is past its exclusive phase, so VM concurrency is unchanged.
1ed417e to
156c878
Compare
|
@sjmiller609 addressed the new integration review in 156c878 (plus the lifecycle fix in #340 at 3ae855e):
I kept A real-VM BuildKit cache-hit/crash-recovery CI fixture is now called out explicitly in the PR follow-up. I did not add an arbitrary per-Builder queue cap in this stack because that needs a host-wide admission policy and a new create-build error contract. |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.
Bugbot Autofix prepared fixes for both issues found in the latest run.
- ✅ Fixed: Builder activity check misses post-VM
- Builder activity now always checks persisted queued/building metadata in addition to queue state, so post-VM conversion/re-tag phases still count as active.
- ✅ Fixed: Create build returns 500 when busy
- CreateBuild now maps builders.ErrInUse to a non-500 client error response instead of falling through to internal_error.
Or push these changes by commenting:
@cursor push 0782534869
Preview (0782534869)
diff --git a/cmd/api/api/builds.go b/cmd/api/api/builds.go
--- a/cmd/api/api/builds.go
+++ b/cmd/api/api/builds.go
@@ -276,6 +276,11 @@
Code: "not_found",
Message: "builder not found",
}, nil
+ case errors.Is(err, builders.ErrInUse):
+ return oapi.CreateBuild400JSONResponse{
+ Code: "conflict",
+ Message: "builder is not ready",
+ }, nil
default:
log.ErrorContext(ctx, "failed to create build", "error", err)
return oapi.CreateBuild500JSONResponse{
diff --git a/cmd/api/api/builds_test.go b/cmd/api/api/builds_test.go
new file mode 100644
--- /dev/null
+++ b/cmd/api/api/builds_test.go
@@ -1,0 +1,49 @@
+package api
+
+import (
+ "bytes"
+ "context"
+ "mime/multipart"
+ "testing"
+
+ "github.com/kernel/hypeman/lib/builders"
+ "github.com/kernel/hypeman/lib/builds"
+ "github.com/kernel/hypeman/lib/oapi"
+ "github.com/stretchr/testify/assert"
+ "github.com/stretchr/testify/require"
+)
+
+type createBuildErrManager struct {
+ builds.Manager
+ err error
+}
+
+func (m *createBuildErrManager) CreateBuild(context.Context, builds.CreateBuildRequest, []byte) (*builds.Build, error) {
+ return nil, m.err
+}
+
+func TestCreateBuild_ReturnsBadRequestWhenBuilderBusy(t *testing.T) {
+ svc := newTestService(t)
+ svc.BuildManager = &createBuildErrManager{
+ Manager: svc.BuildManager,
+ err: builders.ErrInUse,
+ }
+
+ var body bytes.Buffer
+ writer := multipart.NewWriter(&body)
+ sourcePart, err := writer.CreateFormFile("source", "source.tar.gz")
+ require.NoError(t, err)
+ _, err = sourcePart.Write([]byte("fake-source-data"))
+ require.NoError(t, err)
+ require.NoError(t, writer.Close())
+
+ resp, err := svc.CreateBuild(ctx(), oapi.CreateBuildRequestObject{
+ Body: multipart.NewReader(bytes.NewReader(body.Bytes()), writer.Boundary()),
+ })
+ require.NoError(t, err)
+
+ badReq, ok := resp.(oapi.CreateBuild400JSONResponse)
+ require.True(t, ok, "expected a 400 when builder is busy")
+ assert.Equal(t, "conflict", badReq.Code)
+ assert.Equal(t, "builder is not ready", badReq.Message)
+}
diff --git a/lib/builders/README.md b/lib/builders/README.md
--- a/lib/builders/README.md
+++ b/lib/builders/README.md
@@ -1,51 +1,50 @@
# Builders
-Builders give repeated Hypeman builds an explicit, persistent BuildKit cache.
-Without a Builder, every build uses a new 3 GB in-memory BuildKit root and loses
-its local cache when the build VM exits. With a Builder, builds reuse the same
-cache disk across otherwise disposable build VMs.
+Builders retain build cache across repeated Hypeman builds. Without a Builder,
+each build gets a temporary 3 GB cache that disappears with its build VM. With
+a Builder, disposable build VMs reuse one persistent cache disk.
-A Builder is useful when the same application, repository, or build pipeline is
-built repeatedly. Reusing one lets BuildKit retain downloaded base-image data,
-intermediate layers, and build metadata, reducing subsequent build time and
-registry traffic.
+A Builder is useful for an application, repository, or pipeline that is built
+repeatedly. Reusing it reduces repeated downloads and work in later builds.
## Identity and isolation
Each Builder has an opaque ID and owns one cache disk. Pass that ID as
-`builder_id` when creating a build. The ID—not the optional display name, API
-credential, or caller identity—is the cache boundary.
+`builder_id` when creating a build. The ID—not the optional name or the caller's
+credentials—is the cache boundary.
-Names and tags are organizational metadata. Names are optional and are not
-unique, so automation should always store and use the Builder ID.
+Names are optional and non-unique. Tags support filtering. Automation should
+store and use the Builder ID.
-Omitting `builder_id` preserves the original build behavior: the build runs with
-a temporary 3 GB BuildKit root and does not retain local BuildKit state.
+Omitting `builder_id` preserves the original build behavior: Hypeman uses a
+temporary 3 GB cache and does not retain its contents. There is no implicit or
+default Builder.
## Build behavior
A build that targets a Builder still runs in a new disposable VM. Hypeman
-attaches the Builder's disk at `/var/lib/buildkit` for the duration of the build,
-then detaches it when VM execution ends. Source archives, build secrets, and VM
-configuration are not stored on the Builder disk.
+attaches the Builder's cache disk for the build and detaches it when VM execution
+ends. Source archives, build secrets, and VM configuration are not stored on the
+Builder disk.
One build runs on a Builder at a time. Additional builds for the same Builder
-wait in submission order without consuming a global build-execution slot. Use
-separate Builders when independent build streams need to run concurrently or
-must not share cache state.
+wait in submission order. Use separate Builders when build streams need to run
+concurrently or must not share cache state.
Builder details report the active build, queued build IDs, and
-`max_concurrency`, which is fixed at `1`.
+`max_concurrency`. These are point-in-time values; `max_concurrency` is currently
+fixed at `1`.
## Cache lifecycle
-Creating a Builder eagerly creates its fixed-size ext4 cache disk. Disk size is
-immutable; create a new Builder to use a different size.
+Creating a Builder creates a fixed-size cache disk. Disk size cannot be changed;
+create a new Builder to use a different size.
-BuildKit garbage collection is derived from the disk size and reclaims cache
-before the disk fills. Cache contents are an optimization, not durable build
-output: Hypeman may recreate an empty disk after missing storage, interrupted
-cleanup, or an incompatible BuildKit version.
+The current build backend protects cache usage below 70% of disk capacity and
+allows it to grow to 90% before garbage collection reclaims entries. Cache
+contents are an optimization, not durable build output. Hypeman may recreate an
+empty disk after missing storage, interrupted cleanup, or an incompatible
+backend version.
`POST /builders/{id}/prune` clears cache contents while preserving the Builder's
ID, name, tags, and disk size. Pruning is asynchronous: the Builder reports
@@ -53,15 +52,15 @@
exclusive for a given Builder.
`DELETE /builders/{id}` permanently removes both the Builder and its cache disk.
-Deletion does not retain cache state and is rejected while the Builder has
-queued or active work.
+Deletion is rejected while the Builder has queued or active work.
-## Operational model
+## Hypeman restarts
-Builders are local to the Hypeman server that owns their disk. Their metadata
-survives process restarts, and startup recovery reconciles interrupted prune or
-delete operations and stale VM attachments.
+Builders' metadata survives Hypeman process restarts. Startup recovery
+reconciles interrupted prune or delete operations and stale VM attachments.
+Builds that were queued or running during the restart may be started again.
-The initial implementation deliberately keeps build VMs disposable. It does not
-provide warm Builder VMs, concurrent BuildKit solves on one Builder, cache
-migration between servers, or a durability guarantee for cached data.
+Builders are local to the Hypeman server that owns their disk. The initial
+implementation does not provide warm Builder VMs, concurrent builds on one
+Builder, cache migration between servers, or a durability guarantee for cached
+data.
diff --git a/lib/builders/manager.go b/lib/builders/manager.go
--- a/lib/builders/manager.go
+++ b/lib/builders/manager.go
@@ -671,7 +671,7 @@
if meta.Status != StatusReady {
return meta, false, nil
}
- if _, held := m.acquired[id]; held {
+ if _, held := m.acquired[id]; held || m.buildsInFlightLocked(id) {
return meta, false, nil
}
diff --git a/lib/builds/builder_agent/buildkitroot.go b/lib/builds/builder_agent/buildkitroot.go
--- a/lib/builds/builder_agent/buildkitroot.go
+++ b/lib/builds/builder_agent/buildkitroot.go
@@ -48,9 +48,18 @@
if err != nil {
return "", fmt.Errorf("buildkitd --version: %v: %s", err, output)
}
- return strings.TrimSpace(string(output)), nil
+ return buildkitCompatibilityVersion(string(output))
}
+func buildkitCompatibilityVersion(output string) (string, error) {
+ for _, field := range strings.Fields(output) {
+ if strings.HasPrefix(field, "v") && len(field) > 1 {
+ return field, nil
+ }
+ }
+ return "", fmt.Errorf("parse buildkitd version from %q", strings.TrimSpace(output))
+}
+
// reconcileBuildkitVersionStamp resets a persistent BuildKit root written by
// an incompatible BuildKit version: cache contents are best-effort, so on a
// version mismatch the root is emptied before buildkitd starts rather than
@@ -114,6 +123,9 @@
// match the current BuildKit version.
func resetRoot(root string, entries []os.DirEntry) error {
for _, entry := range entries {
+ if entry.Name() == "lost+found" {
+ continue
+ }
if err := os.RemoveAll(filepath.Join(root, entry.Name())); err != nil {
return fmt.Errorf("reset buildkit root: remove %s: %w", entry.Name(), err)
}
diff --git a/lib/builds/builder_agent/main_test.go b/lib/builds/builder_agent/main_test.go
--- a/lib/builds/builder_agent/main_test.go
+++ b/lib/builds/builder_agent/main_test.go
@@ -160,6 +160,19 @@
}
}
+func TestBuildkitCompatibilityVersion(t *testing.T) {
+ version, err := buildkitCompatibilityVersion("buildkitd github.com/moby/buildkit v0.23.2 abc123")
+ require.NoError(t, err)
+ assert.Equal(t, "v0.23.2", version)
+
+ version, err = buildkitCompatibilityVersion("buildkitd github.com/moby/buildkit v0.23.2 rebuilt-sha")
+ require.NoError(t, err)
+ assert.Equal(t, "v0.23.2", version, "same release with a different build SHA must keep the cache")
+
+ _, err = buildkitCompatibilityVersion("unexpected output")
+ require.Error(t, err)
+}
+
func TestReconcileBuildkitVersionStamp_MatchingVersion(t *testing.T) {
root := t.TempDir()
version := "buildkitd github.com/moby/buildkit v0.23.2 abc123"
@@ -190,6 +203,19 @@
assert.Equal(t, newVersion, strings.TrimSpace(string(stamped)))
}
+func TestReconcileBuildkitVersionStamp_ResetPreservesLostFound(t *testing.T) {
+ root := t.TempDir()
+ lostFound := filepath.Join(root, "lost+found")
+ require.NoError(t, os.MkdirAll(lostFound, 0755))
+ require.NoError(t, os.WriteFile(filepath.Join(root, "cache.db"), []byte("cache"), 0644))
+ require.NoError(t, os.WriteFile(filepath.Join(root, buildkitVersionStampFile), []byte("v0.22.0\n"), 0644))
+
+ err := reconcileBuildkitVersionStamp(root, func() (string, error) { return "v0.23.2", nil })
+ require.NoError(t, err)
+ assert.DirExists(t, lostFound)
+ assert.NoFileExists(t, filepath.Join(root, "cache.db"))
+}
+
func TestReconcileBuildkitVersionStamp_MissingStampResets(t *testing.T) {
root := t.TempDir()
require.NoError(t, os.WriteFile(filepath.Join(root, "leftover"), []byte("x"), 0644))
diff --git a/lib/builds/builder_disk_test.go b/lib/builds/builder_disk_test.go
--- a/lib/builds/builder_disk_test.go
+++ b/lib/builds/builder_disk_test.go
@@ -151,6 +151,33 @@
require.ErrorIs(t, err, builders.ErrNotFound)
}
+// TestCreateBuild_BuilderNotReady verifies a build is rejected before queueing
+// when its Builder cannot accept work.
+func TestCreateBuild_BuilderNotReady(t *testing.T) {
+ mgr, _, volumeMgr, tempDir := setupTestManager(t)
+ defer os.RemoveAll(tempDir)
+
+ ctx := context.Background()
+ builder, err := mgr.builderManager.CreateBuilder(ctx, builders.CreateBuilderRequest{})
+ require.NoError(t, err)
+
+ volumeMgr.deleteFunc = func(context.Context, string) error {
+ return errors.New("delete failed")
+ }
+ _, err = mgr.builderManager.ResetDisk(ctx, builder.ID)
+ require.NoError(t, err)
+ require.Eventually(t, func() bool {
+ got, err := mgr.builderManager.GetBuilder(ctx, builder.ID)
+ return err == nil && got.Status == builders.StatusError
+ }, 2*time.Second, 10*time.Millisecond)
+
+ _, err = mgr.CreateBuild(ctx, CreateBuildRequest{
+ Dockerfile: "FROM alpine",
+ BuilderID: builder.ID,
+ }, []byte("src"))
+ assert.ErrorIs(t, err, builders.ErrInUse)
+}
+
// TestCreateBuild_GCBoundsWrittenForBuilder verifies the host computes
// BuildKit GC bounds from the builder disk size into the guest config, and
// leaves them zero for tmpfs-backed builds.
@@ -295,11 +322,11 @@
started := make(chan string, 3)
req := CreateBuildRequest{BuilderID: "builder-a"}
- pos1 := mgr.queue.EnqueueSerial("build-1", req, mgr.buildSerialKey(req), func() {
+ pos1 := mgr.queue.EnqueueSerial("build-1", req, req.BuilderID, func() {
started <- "build-1"
<-release
})
- pos2 := mgr.queue.EnqueueSerial("build-2", req, mgr.buildSerialKey(req), func() {
+ pos2 := mgr.queue.EnqueueSerial("build-2", req, req.BuilderID, func() {
started <- "build-2"
<-release
})
@@ -432,3 +459,22 @@
mgr.RecoverPendingBuilds()
assert.False(t, mgr.BuilderHasBuilds("builder-a"))
}
+
+// TestBuilderHasBuilds_CountsDiskBuildingAfterRecovery verifies a build that
+// is still building on disk remains visible to builder activity checks even
+// after startup recovery has completed.
+func TestBuilderHasBuilds_CountsDiskBuildingAfterRecovery(t *testing.T) {
+ mgr, _, _, tempDir := setupTestManager(t)
+ defer os.RemoveAll(tempDir)
+
+ require.NoError(t, writeMetadata(mgr.paths, &buildMetadata{
+ ID: "build-post-vm",
+ Status: StatusBuilding,
+ Request: &CreateBuildRequest{BuilderID: "builder-a"},
+ CreatedAt: time.Now(),
+ }))
+ mgr.pendingRecovered.Store(true)
+
+ assert.True(t, mgr.BuilderHasBuilds("builder-a"), "building metadata must count after recovery")
+ assert.False(t, mgr.BuilderHasBuilds("builder-b"))
+}
diff --git a/lib/builds/manager.go b/lib/builds/manager.go
--- a/lib/builds/manager.go
+++ b/lib/builds/manager.go
@@ -148,8 +148,7 @@
createMu sync.Mutex
builderReady atomic.Bool
// pendingRecovered is set once RecoverPendingBuilds has re-enqueued
- // persisted pending builds; until then BuilderHasBuilds also scans
- // disk so delete, prune, and the idle reaper see them.
+ // persisted pending builds.
pendingRecovered atomic.Bool
// Status subscription system for SSE streaming
@@ -425,6 +424,9 @@
if err != nil {
return nil, err
}
+ if b.Status != builders.StatusReady {
+ return nil, builders.ErrInUse
+ }
builder = b
}
@@ -562,7 +564,7 @@
// Enqueue the build. Builds sharing a builder are serialized by builder
// ID so a waiting same-builder build stays pending instead of holding a
// global concurrency slot while blocked on the builder's disk.
- queuePos := m.queue.EnqueueSerial(id, req, m.buildSerialKey(req), func() {
+ queuePos := m.queue.EnqueueSerial(id, req, req.BuilderID, func() {
m.runBuild(context.Background(), id, req, policy)
})
@@ -575,13 +577,6 @@
return build, nil
}
-// buildSerialKey returns the queue serialization key for a build: builds
-// that attach the same builder disk share a key and never run concurrently.
-// Builds without a builder are unconstrained.
-func (m *manager) buildSerialKey(req CreateBuildRequest) string {
- return req.BuilderID
-}
-
// gcBoundsForDisk computes the BuildKit GC bounds for a builder disk:
// maxUsedSpace at 90% of the disk so GC reclaims before ENOSPC, and
// reservedSpace at 70% so reclaim keeps a useful cache floor.
@@ -603,15 +598,11 @@
}
// BuilderHasBuilds reports whether any build targeting the builder is
-// queued or running. Until startup recovery completes, persisted pending
-// builds are not in the queue yet, so they are matched on disk.
+// queued or running, including persisted queued/building metadata.
func (m *manager) BuilderHasBuilds(builderID string) bool {
- if m.ActiveBuildForBuilder(builderID) != nil || len(m.QueuedBuildsForBuilder(builderID)) > 0 {
+ if m.queue.HasSerialKey(builderID) {
return true
}
- if m.pendingRecovered.Load() {
- return false
- }
pending, err := listPendingBuilds(m.paths)
if err != nil {
// Fail closed: blocking a delete is recoverable, deleting a
diff --git a/lib/builds/queue.go b/lib/builds/queue.go
--- a/lib/builds/queue.go
+++ b/lib/builds/queue.go
@@ -102,15 +102,7 @@
if len(q.active) >= q.maxConcurrent {
return false
}
- if serialKey == "" {
- return true
- }
- for _, key := range q.activeSerialKeys {
- if key == serialKey {
- return false
- }
- }
- return true
+ return q.serialKeyFreeLocked(serialKey)
}
// startLocked marks a build active and launches it.
@@ -205,22 +197,15 @@
return q.pendingPositionLocked(buildID)
}
-// pendingPositionLocked returns the pending queue position for buildID.
-// Position counts only pending builds whose serial keys are currently free.
-// Builds blocked by active serial keys are skipped so later startable builds
-// are not reported behind work that cannot start yet.
+// pendingPositionLocked returns the physical pending queue position for
+// buildID. Scheduling may skip a blocked serial key, but positions retain the
+// existing submission-order semantics for every build.
func (q *BuildQueue) pendingPositionLocked(buildID string) *int {
- pos := 0
- for _, build := range q.pending {
- blocked := !q.serialKeyFreeLocked(build.SerialKey)
+ for i, build := range q.pending {
if build.BuildID == buildID {
- pos++
+ pos := i + 1
return &pos
}
- if blocked {
- continue
- }
- pos++
}
return nil
}
@@ -276,6 +261,29 @@
return len(q.active) + len(q.pending)
}
+// HasSerialKey reports whether an active or pending build uses serialKey.
+// Both states are inspected under one lock so pending-to-active transitions
+// cannot briefly appear idle to lifecycle guards.
+func (q *BuildQueue) HasSerialKey(serialKey string) bool {
+ if serialKey == "" {
+ return false
+ }
+ q.mu.Lock()
+ defer q.mu.Unlock()
+
+ for _, key := range q.activeSerialKeys {
+ if key == serialKey {
+ return true
+ }
+ }
+ for _, build := range q.pending {
+ if build.SerialKey == serialKey {
+ return true
+ }
+ }
+ return false
+}
+
// ActiveBuildForSerialKey returns the ID of the active build holding
// serialKey, or nil when no active build holds it.
func (q *BuildQueue) ActiveBuildForSerialKey(serialKey string) *string {
diff --git a/lib/builds/queue_test.go b/lib/builds/queue_test.go
--- a/lib/builds/queue_test.go
+++ b/lib/builds/queue_test.go
@@ -300,11 +300,11 @@
pos3 := queue.EnqueueSerial("build-3", CreateBuildRequest{}, "builder-a", startFn("build-3"))
pos4 := queue.EnqueueSerial("build-4", CreateBuildRequest{}, "builder-c", startFn("build-4"))
assert.Equal(t, 1, pos3)
- assert.Equal(t, 1, pos4, "queue position skips blocked pending builds")
+ assert.Equal(t, 2, pos4, "queue positions retain submission order")
pendingPos4 := queue.GetPosition("build-4")
require.NotNil(t, pendingPos4)
- assert.Equal(t, 1, *pendingPos4)
+ assert.Equal(t, 2, *pendingPos4)
<-started // build-1
<-started // build-2
@@ -395,10 +395,41 @@
assert.Equal(t, []string{"build-2"}, pending)
assert.NotNil(t, queue.PendingBuildsForSerialKey("builder-c"))
assert.Empty(t, queue.PendingBuildsForSerialKey("builder-c"))
+ assert.True(t, queue.HasSerialKey("builder-a"))
+ assert.True(t, queue.HasSerialKey("builder-b"))
+ assert.False(t, queue.HasSerialKey("builder-c"))
close(release)
}
+// TestBuildQueue_HasSerialKeyAcrossPendingToActiveTransition checks the
+// lifecycle guard never sees the serial key disappear as its next build starts.
+func TestBuildQueue_HasSerialKeyAcrossPendingToActiveTransition(t *testing.T) {
+ for i := 0; i < 100; i++ {
+ queue := NewBuildQueue(1)
+ release := make(chan struct{})
+ started := make(chan string, 2)
+
+ queue.EnqueueSerial("build-1", CreateBuildRequest{}, "builder-a", func() {
+ started <- "build-1"
+ <-release
+ })
+ require.Equal(t, "build-1", <-started)
+ queue.EnqueueSerial("build-2", CreateBuildRequest{}, "builder-a", func() {
+ started <- "build-2"
+ <-release
+ })
+ require.True(t, queue.HasSerialKey("builder-a"))
+
+ queue.ReleaseSerialKey("build-1")
+ require.Equal(t, "build-2", <-started)
+ assert.True(t, queue.HasSerialKey("builder-a"), "serial key must remain visible while the successor starts")
+
+ close(release)
+ require.Eventually(t, func() bool { return queue.ActiveCount() == 0 }, time.Second, time.Millisecond)
+ }
+}
+
// TestBuildQueue_ReleaseSerialKeyStartsSuccessorAtFullCapacity releases the
// serial key while the build keeps its global slot (post-build work) and
// verifies the serialized successor starts anyway: at MaxConcurrentBuilds=1
diff --git a/lib/oapi/oapi.go b/lib/oapi/oapi.go
--- a/lib/oapi/oapi.go
+++ b/lib/oapi/oapi.go
@@ -360,10 +360,10 @@
// Build defines model for Build.
type Build struct {
- // BuilderId Builder whose persistent cache disk backed this build
+ // BuilderId Persistent Builder resource whose cache backed this build
BuilderId *string `json:"builder_id"`
- // BuilderInstanceId Instance ID of the builder VM (for debugging)
+ // BuilderInstanceId Disposable VM instance that executed this build; distinct from builder_id
BuilderInstanceId *string `json:"builder_instance_id"`
// CompletedAt Build completion timestamp
@@ -442,7 +442,7 @@
// Builder defines model for Builder.
type Builder struct {
- // ActiveBuildId ID of the build currently running on this builder
+ // ActiveBuildId Point-in-time ID of the build currently running on this builder
ActiveBuildId *string `json:"active_build_id"`
// CreatedAt Creation timestamp (RFC3339)
@@ -457,13 +457,13 @@
// LastUsedAt When a build last ran on this builder
LastUsedAt *time.Time `json:"last_used_at"`
- // MaxConcurrency Maximum concurrent builds on this builder (fixed at 1)
+ // MaxConcurrency Maximum concurrent builds on this builder. Currently fixed at 1.
MaxConcurrency int `json:"max_concurrency"`
// Name Optional non-unique display name
Name *string `json:"name,omitempty"`
- // QueuedBuilds IDs of queued builds waiting for this builder, oldest first
+ // QueuedBuilds Point-in-time IDs of queued builds waiting for this builder, oldest first
QueuedBuilds []string `json:"queued_builds"`
// Status Builder lifecycle status
@@ -17586,7 +17586,7 @@
// Base64 encoded, gzipped, json marshaled Swagger object
var swaggerSpec = []string{
- "H4sIAAAAAAAC/+y9+3IbOZI3+ioIntloaYakqKttTXR8R5bcbm232lrL9pzdpg8FVoEkRlVANYCiRHf4",
+ "H4sIAAAAAAAC/+y9+3IbOZI3+ioIntkYaYakqKttdXR8R5bcbm232lrL9pzdpg8FVoEkRlVANYCiRHf4",
"332AfcR9ki+QAOqKIouyLtbYsRvTMqsKl0QikZnI/OWfnYDHCWeEKdk5/LMjgxmJMfx5pBQOZh94lMbk",
"LfkjJVLpnxPBEyIUJfBSzFOmRglWM/2vkMhA0ERRzjqHnXOsZuh6RgRBc2gFyRlPoxCNCYLvSNjpdsgN",
"jpOIdA47WzFTWyFWuNPtqEWif5JKUDbtfO52BMEhZ9HCdDPBaaQ6hxMcSdKtdHumm0ZYIv1JD77J2htz",
@@ -17610,302 +17610,302 @@
"azjNMYU9eULm1BwtZd3NLs0oFHROhEd8ZwqAEYXmPbSh97oWIYwzslmiFJvTkOI24iCEMY2oh3vOj0+R",
"eYxOT9DGjNyUO9l5Nn7eaW6S4djDCz+nMWY9vSH0sFz78G6x7V/3vCYKj+N0NBU8Teotn745O3uP4CFi",
"oOEWW3y+49NUk4COcBgKIqV//u5hcWyDwWBwiHcOB4P+wDfKOWEhF40kNY/9JN0ehGRJk61IatuvkfS3",
- "D6cnp0fomIuEC7DZVm6cInmK8yqyTXlVfPz/MqVRWOf6sf6Z+On00jxD1zMuCUqIkFQqLb0CHMwICqm8",
- "QmNjsIBpDm21YfesU3dy+Xo/dbrb6YlTTux36MMZ2tCCKyTjdDqlbLrZpteA6zXQ56tPc4DJIvuONqWV",
- "U41ufcgHguAV3ek3WnVW39+pYZ9RLJtad69oMR7TKKKSBJyFstgHZepgr3kyhV1qjsVaV6/0zygmUuIp",
- "QRvgdgJzwkhwrU1NMI1IuNlOg26azD/5uHBulfYUsEUPj4PtnV2vwIrxlIxCOrV+w+q5qH/XLKbbUQje",
- "9k8ENIh284AuBZnU+/sJzgvoRJAJEUTz+Bd2lwg+Jwxbk+kv0G/n/9nKHapb1pu6BcQ8z1//3O38kZKU",
- "jBIuqRlhTVzaJ5qNgNQIvvCPGR4tW+sCR0mFxfL9AW/cwU7MlcmVtLGuHa1P4enKT97pd6oCG+RxpsAU",
- "pECjXH6lNSmPSsKZsg8qLl4+RRFlxszREtmsBShzi4T8GHEQiXdEh4z89c2vx30L4WV+aGhNP+tmWn/E",
- "p0VqzggWakxKxGw4N21D+egayX9e2j6VAxJLMlouQc4pYyQEn7rd2OZNrTt7bRvYRVdUjeb6TPXtORjW",
- "L1Qh+0ZjUxEPriY0IqMZljPrhAxDahyq56WZeFTE0mUFBieAaxBUFzCaL34+2tk/QLYDDw2td1e/UJ9J",
- "4WvdvHkXKSzGOIq8vNHMbuuf0XUO8XNA7tBtOnsyDnSMaSRdx66mNc5TOTN/gezWo4KzT4sBzV6R/rvG",
- "uN3OTU+32ptjAVTXzReG9R+up8JvL/NOC7+eZ/0Xfnxrh1L46Sc3qsJvx/kAHVWMGeR1ccOk/XpbWV0r",
- "OE6tbwLphcv0RTjLv0iVOq4pUWjj7U/Hu7u7Lzbbq1NUXo0k/URG07Fnh+e6r9NBCzqw/kwfjlM6xeOF",
- "IrKPjjFjXKExQcEMsykJEZ4o/ZUda8mk3h+sulZZpp03KEXq0x5+8fzmBqsXB/RavvgUj8X0n7veG03w",
- "F2oJ0+xTw3Y19atIYOZZxNsdzTG+0ba84RLfdeaZuYBC2Ut2DWR1CGhjQm80pRXaLhl0XpL6rbjsKopx",
- "1ksZ/SOFNU4ivKjbcorguAd84COqkQ9mo0jfNpF6n5i33IyuMVV6h2S3nXZqXcSjUJ8oEyqk6rS/x1rr",
- "yCfirpSf0m7y60L1la+SrPG4zsbZuCciOiHBIohIXWw7wZyI1DpKQ6JPE/gzc/21FdBuKCUZ6348z7oo",
- "/XyS91f6/ZXp/HPXSDRiHzYGPCwXWcdL5dObmCp9tqfS3LgZPy6yF/V3IJ3yO10cRUT0ZJokEYWrDyeu",
- "usbdOSWMCM0VxommkDZmaFiJyMj3Wm/bt9vufTuvtSFqfGtW1LgeGxd0xRymEdc60wK5ieReuz46LdOu",
- "izA8gMuMMpWz66aCZw1tkP6030XDThLQ3mAwGPTwTm8w6A2GnbJvLNrrmTuDBCtFhB7g//877n066v3X",
- "oPfiY/7nqN/7+Le/+CjZ1t3n1Ag7zw0nZbrIDbboA6wOdJV/8NYCrjj8j41LfartgHVX+vi07hgwcw15",
- "cEVEn/KtiI4FFostNqXs5jDCikhVnvnyd70r4u6b6veDWExJ4aYCS8TlFhbB7PetORYUM/XRjBENOxFl",
- "6c0WjsODvWFns4tirAKtjqITGBHq9Vw7VgC5y4fSbUgf/cYVwsgc5lrsHsJL9X6nAscxFprH5ziiIbC3",
- "kWQ9SUOCMAsRZfBMv5ESiQRRqWBobzBwT0aFQWkjmks3lWEHXVM1Q7pbBNNCXCAs4oM9c0kYkER3qXiI",
- "Kze1BVLU94reKH/byv7acH/2P/5t8//85U6ZFRhtCZeyqebjNfm04uoGObMR8WsiAm0CR0TPVna1FUyV",
- "7MIyhGA9Is6ixd9RYLRj493hAhEWWkrDe2V2jhc9nNAeNUM1esOvhE3VrHN4sOsl7ob9o/fxr+6nBsKK",
- "NCIeKfSWp6CHweNi7JkbQ6Z/LVsRR900Aj9bTNmp+Wy7rqZ92Qq7iSxbaeNNb1xqfUpkF6MrBlIPQtQm",
- "WuxTA+ZECL0VjVw7PjtBGxG9IlagaZMQDdPBYDeAF+BPYn8JeBxjFprfNss6i3WzeNSV3zskmHHwFEUR",
- "XyfmC9RT0E5wtNSRsow0XmofZ+3W3S4/c6l6MWZYG4n5ANBY8CuiB2oiQSiR6IostKmwQFPdaG9OJYQh",
- "ETZHc2zumvpD9g5ua+AV90hCRAedExTz4MrEJ844aMpGLHbR9YxGRhUUBEe5tIwxZUOmBXRPBjzRWhqz",
- "r8HU0CVh80sU4wS2ORYE9riW/URQHNFPJs4UYktISLWMHjICGwMlWO/5IOAihDAwjggOZgUq/CDRpfEY",
- "XULzl5Rptr40G7MSUfln5837dy/fvP/tZPTm/NVvR6ejX179p/7ZfNQ5/P3PjoknzizVlwQLItBf/oT5",
- "fjb+RXB7dI5SNeOCfjJ3dBBFJRUo/jihfZ4Qhmk/4HGn2/lr8Z8fP390HjETvDDX28AzsM9eRdEoOx6R",
- "dOLugCWy94ouokWTTIuo1+fvt7T6lGAp1UzwdDorbwyru621JcDIoHw0Tnxj0tbF6dYbpDVLFFG9QTNN",
- "cnswOHu5JYcd/Y9994/NPjoxuxaGr2UQF1bBlTPNPllo8vH5e4SjiAf2EmvSFIXouvIJeMKUWCSc+rzo",
- "FeGUv1qXUb1e/nQNUbQ1pmxL6mXoBevRHfjm1r7cV2xOBWcxYQqBujSOiCzvld/enLwavfrtQ+dQHwRh",
- "Gti76PM3b991Dju7g8Gg42NQzUErZODr8/cm1g22DcGRmo2CGQmuVn34M7x7DK/CjlNJlE7Bxq2v3lFG",
- "GhSTmAtz/WG/QRuzspJitjyCdR12dl+/NHy5/RpY0q2nDSrKWjENV+LAXr/0MdpskRAxp9J3R/pz9swx",
- "TT2cvbQtrDnu+D13GVonRhDxNOwVuux2JlSQAMKH9b/+ILE20uafynFMnu/8V5etjJMVVgeOEsrIErPj",
- "K9Eur7m4ijgOwbFwl8qljcDzhH6bB+X1zawhxxK1jIoxZuE1DdVsFPJrpofsEcn2CcpezuTyjZ4Jjv73",
- "v//nw1luQ2+/HidWSG/v7H+hkK6IZd209/4rm0ia+KfxPvFP4sPZ//73/7iZPO4kjA5zK33Qrv8r00I1",
- "KNzmyhh/eUMoYXbwZxHOiltnCXyOHO+tjBv0yXg+JyLCi4LgtWPqbA9A+lVGJShkASH7XcHruFwM69ac",
- "fvC66sDZGfgF7XdnxVfurBAERO8oyfLTlm2Qt+bt3I70MJ2H517qs8Ce+204LWO07Z0z++dOneX8HCev",
- "aDICg2qEp1k8xrKUtosrmlgrDb4w2zSKjKAPU7Drxpyr/pCZkHW9N4FzyA0JgBekwgodnZ9KdE2jCLy7",
- "cGjUVQdt8xVyHeB1qfT/ipR10ThV2pDjiiBrUkMnKYwFXh4TlDLsAmQrZpWdYD3eGMhyRQQj0ciYTbIl",
- "ZcxHyH7USByY6gRLm2IjVJqU6XXyy9kF2jhZMBzTAP1iWj3jYRoRdGHCjTfL1OsOWSIgbll3otmR2n75",
- "BPFU9fikpwQhbogxNJb5x2305vz1+Xsb/ys3+0P2lmjCEhbaREWnUdgktpCzH7REJmG52WL/FaI3xXhL",
- "hhM5420314V9Pd9d7f1M3c48SNLyku50G5PY5lSoFEf6LC1ZGt6IX5OE67EoTY5v0bK151qe9KfKUZNt",
- "nXGmZcjI9ab7eXxqRhNu7VMreHlq3jXngviz3WBXtH/K3ECW+hRzL8QX9HVhGqlF85ufu25mt6DSaUaT",
- "iifybshzJAtem1bJs+ZkN8efRBuXOKF9y8f9gMeXXXT519IPeu8701Hrj9fIUAPkCdM/Fduv+qtWepLW",
- "SlctLg6Wt1+PI9mY+oDm20gJzKRJWpnhhPTRzyDEkSKxVk0gV0yiLNcDMX79d8SN0uo+HTI9NGliuC05",
- "Mn+ipFNG2XRTm3H6YMJhaJyOk1SlQr83pzKnZpl1nGOvphma0REjjyHDm7IgSkOCLp3z77KsFNVdg3WT",
- "3/oKaxasIQlYrmDMq604Vbp7PeFM0eSpMpkkdurlLJ+KA3JVsKMdSxaFc4v1v8jERTWRf+4xYfXk7AUt",
- "eIwLrusmD7FVVPze6yuygCV3nmpc81UXndR+V7IgkkdzYo/dopt7DFAF3ChOuYfb+Kqte1pv/2qSvs9x",
- "u2opNL1ak79sCnogCqTqucnmHGOtO5ck6qSQnpzpr6uVf0mA+GBaHiJQxy67xhYmYHkgppklQiEVJFC1",
- "5imbDhnEd1/aX/q2tUu9ybWOcifAD5BHDUp7cWlRYWWd2gfN6KnxmCpFwm5ZN7giJJGrJ6XVa3un4bl4",
- "EeRaUCfIXJZhS/WMsAkXAYmtkfBljoFXhcaWBJW0baIebm3oWxizyy/HNlYH1t6sB3jgS2nnVfSVsGK1",
- "mfDecpeXOIou0YZ9aRMJ8k9IzbVrxTjLmf3d8bljgSxk5cNZV3OklgKXM6WSkf4fOdK7+LLamP3W7fAc",
- "GeP5AOyrvb1du6rWqWoGXGm27D/1hiw3L41TvxsvXTVf6FHaGPA2qvxx/knuZL+iLGzbwC/63dURU87S",
- "uG8HbCJIL02mAkPO3V26X299pQ7UbJbgK3CIlkfDpVLxuJiAu7E8CK5MrDmPeiFWeJ0YODPcerhbvDBN",
- "GVvM6/doii28qAYUlqJtvWgkXxrf4MbiW5amjF5jQZJwpPjy9EI6Qe7dVumLkICs+Gg+oXx5vreNXSvh",
- "h5jjyNq1uoleElDrTgAdJ5iZ7C9DBFAaP5wVr3X7Q9aD4/cQnWQdZM1mTWLQLXFobsY2uCgMwmR2o/Fi",
- "E2H04ayP3mWj/UEibbDMiUtxnmGJxoQwlMLVApyGPXMWFweQSjg0VfVz6zsx2dCbcHvN7bN+hpkEXpoM",
- "AQrCHMe0Mh+DUgILZcMFMCt6wVp5rZZlErwlUyqVWJZNUMKD2O8Ntnvb+++2B4cD/f//1T7d4O4Tvn1t",
- "HZVliw0cLUqf4/enJzvWWbp56wSCO08J94uykzziFW2kkoieE5Pg6/bEuRbCSRviWG8dnnpP0aZ5ctyy",
- "dw0l3uk37yPX3RfTb9Pp1s9GrwrMlSmRhcnVLflFAnZnvksKGpyNTA6oJzmwC9EqLwXBV4DhUT+3Yzwl",
- "cmTOM3+oSypN/BW5sd4NwbmaSHMvXvZ6bu8923u+e7D3fDDwZFvXGZ4HdASx560G8Ob4FEV4kaUhbcCF",
- "ZojGER+XGX1/9+D5s8GL7Z224zBXeO3okBle7iu0YSnyN4ez6J6UBrWz8+xgd3d3cHCws9dqVNZf3GpQ",
- "zrdcUkme7T7b236+s9eKCj6F/pXLfq8q8D6olSODTqb/1ZMJCeiEBghyS5D+AG3EcISR7LaqvCfHOHSA",
- "NP6zQ2EayaURMaYz+6ZxtMVppGgSEfMMFqSVLxpmfgIteRH+GMsAgNZryWIGrIwAcXPJXkEl7IcS6c4M",
- "wlFBeaIkCg/NDl0p52A184F9bOIDO4eW3PCrNp16EZmTqMgE5ugyUDuCoIxPzKKVZuUufylLUi9LNJLy",
- "p1SALmoaRXjMU2WuGS1iU94JZCyA7THR4rqdnfsTF1crQ4v1SZwBU630Ch2BI31iXTVwiuMsddSmDxeU",
- "vuw60Fya2ucSvTVfGA9R/nOSllE5u9CT9SQxJIhUHCSpdRjaZtpql369BZylLn7A9JfLzgeKbepNTDjI",
- "3VrYYkoAkE2t1Fg0p5gAjQt4vXWmgv5wpSOlBd0ZuX4IokOuTU+zbU8ynNwPxZcFG2a+hvwlOIUFDUkf",
- "we6CqCeH3VHZaReKJwkJM/9Pf8hsqH/2kzQ3KPpDQwc1I1QgLuiUljsuO9juM2pxHVZ03HRrdix+2BSF",
- "JLMLH9+mN9ngExMEaGCJikmqdhE63c5FBlVnJVGZNG8zuL8aRfIo3NoQX5+/Xzf2MBF8Qn14qRALYZ9a",
- "y8xF5f26N7jobf+HibDV/AYqGmUmfiLmYQVZzr7f7uR5ff7+vGlMGdYbKo6uNqcs4mUZOK+jiL1UsreS",
- "1oJx7K8PlqyTXPd+4dNlJwLHZJxOJkSMYo9z7Sf9HJkXstTds5dlfVbrzW2t5vPS4oDZPMGBRc1qR32P",
- "Q64yjW6Bmh/9y/WWmGO4KXlbL5Ww79i87T76LUPXQ6/P30uURyl5PHWV/OimVIrz2ULSAEemRQOdQ1nR",
- "wQbM2VpDPs8/tK5Ij57sB2V0GwFtzKdJCtvw4m3v9M2HrTgk825pTBBZNOMR0ePeLEiLucvIzfM+SkJi",
- "3uTpMIwh226gAq2yHdyaSIX96qGO4gpHIxlxX7DGO/0QwUO08eEnk5CnR9BFSWkp9e8FKpT4+8C7YwD0",
- "oqHbC+iw6jItbXCv7VhG8zfulcL0Sp36torJgqjrOHVEVH5VXmh+tRqF0zTS3O+xS9SoOLWtIolMPgeC",
- "fA53/4zMp8ZrbV0jkiRYYEWihdEssqOvjM1A6leJ5IYEa2SKvNKvfzZIO6kgIzUTRM54VL6A3u3W0Zgl",
- "BEHOiQWoM3MqON4VRzEWV3AwOkUapcxQoBxDu7sKI2GmVLLGpH5+9+7cWNeKiDmOqlHYsna1ekIivEBj",
- "oq4JYW4qWCKMXvMMCbCaNiUbQMKEGiVEUF6mYWfX0++FCcxEU4EDAImkPHRY7nZJJJyabUlpe/Eg7AYB",
- "kbJhfbeXra/9dJJG7dbYN6ztlaUPgnUW+N3xuQO7ysDCHZl36lQ+J6JntpxDDV++tDtyOeya64pxRuqd",
- "CT4mgMNmQ9+LOVwusARw5vTnpbypgnCQxYQl2w/sAkOqrtnnH1spe9Xt7rtIjzELfaDrJuLZJKdO01gv",
- "iB6ySOHuiIYm6sTkyxq1vBi4LQgOKSNSVjL7glREnW6nN7GzOtzainiAoxmX6nBvd/v51vL4vaWBmzZO",
- "ZRTSZfadi2Yx8Q4uAc6An8OkyyyxhZOkhQfM0HHF+QDiqR4oBkDr+mwraHgucngwqCUp3uBAOehIcImV",
- "rjxxcdsmmiVL84EGs+Ik+y9eFPfnwHsHndcGcuy/VeN9PTMTTKZ5xLgbKnQ0TP7Jq1Fx4cMH5ELZFKQx",
- "cdFm2XnoYrnspUqps+eD58VZtirBAsKmss3tvvNM1bxdTkskBXLb/WsbgLiYss7htvRyCGNNlxU8pSVi",
- "jaWWE5QnhK1Fz/293Z316Nl2Iqcua7Iil3ygCMdnJ0YnCjhTmDIiUEwUtvWmCkIGfElaygBMFSYx5ClM",
- "/r5ctDTELxRRDm6LpXdXt98NQJpvTfxmiGLM6EQLZPtmsWc5wzv7B4cG7jckk739g36/v27u96s82bvV",
- "UmyZ/NZCGnhfzr5sHe4hxbvNXP7snB+9+1kLslQKc2htyTFlh4V/Z//MH8Af5p9jyryp4a0Qoumkhgxd",
- "jgfTBr/5/bBQ8MjpPa2qhPidwRAWCngUXqAlhadatzEcd3+IShmPmwHcJllxSfKeVlfeQG26BtrcGuw5",
- "L3OgCiDPxTSfFoDP9NPy+23n7oJ3bJ9QuCjHwq7fbN8KzVwuBXytoQYmhGUQr1Fk/go4m+vt6sN7LR2R",
- "7lkLUEE4Rix6YNZl8ces98KPx8WBFH53oISFnyzw68c1Q1KWKqT/qOuhq6WQU0dXbGa/7zE7FdqCbFsk",
- "Js/5/Ohn4W3Ctsq9v5n++x//nzx/9s/tP3798OE/56///eQ3+p8fovM3XwS6sByE61GRtO4MPAtilUoI",
- "Wm1Z6QyrwOOj0+ZfA4XtE+NxUMEMQILRmBwOWQ/9ShURODpEw04l62zYQRsELCX4Squ7uimbPLupPz43",
- "N4r64z+dGvy52kZos2SFXZAM/ECm45DHmLLNIRsy2xZyE5FgF+i/QhTgRIE7hzKk7d8FGguodGlvfPLO",
- "u+hPnCSfN4cMLk3IjRJ6BgkWKoN1dD0AU9hRmTBU+zrJ8uDNpcuQZad1hkJlrv36mfIP4R7VJB4/UZbb",
- "b9Zyej7w4XVBIoFeyAhAoVF2gUYlMHqW4YCeDzbr9twKGyPjoSXsBzuhXgLXMWWLvWQYGLo2gnvkXJQr",
- "wjO0bDJ7BIEFqTj89wK5hnJaZEts7g1MWok0184qkoWEks2Ot6IVrG7LCZl7V/gsapGI/spkHL379QIp",
- "ImKXA7oRaHJOaKDnBxGlVMpUsyLF6Oj47NVmv0UNX6BtNv4l6/gum2E1X9jewzZdL+fmLo5JF52eQMaX",
- "3aG5WguR2j9xgSIjYPJ9fYjeW+zfUlPIBIqalYwW+WWuOQGGnU3XYlKVFIfobaZN42wopbrB5fvhfF9C",
- "szaWx4SR11rv1ipyCqdJW9EGQeNYZXmH+sRtFgXt3TcOVkTv+YqzYe29Xbxfb3QlFNb+rjEa717d2V1P",
- "3XFV1pIZlj7unhUviOClJZVYacXrLZqjIUr9rqwumnUH9V1JVsTAfO6rC7nf295+t723vidkXXi9MsxJ",
- "AQIpQ9hrD413HxBzda/ADVWjxjhZpB/bqFhnYn44QzMs2Q8KHlYMze3dZ62qVele20aYFmNL+cQMKZNS",
- "DjMli4w06DFXNIpMwLGkU4Yj9AJtXJy+/uX01183UQ+9eXNWXYplX/jWpwXSnhMVr8/fwyUjliMXpNWc",
- "l4Tz3D5yQ6WSdUiZVrGOX4LsZz5tV5vATdK0kZcoWA4P+HMJws+LEbR5h7h+LkC1RsaHQOx7zMyfrw8t",
- "cCm+35eC9Fnj5Z4w+hqFuw/friznzc93i7Z3L8NZWQC9eNa7tMxbw9t1O9STknYktQgmITo9z0sg5B5O",
- "13xlTi92+tsHz/vbg0F/e9Cutk2wpO+zo+P2nQ92jGpxiMeHQXhIJl/gCLeMbZRxHF3jhURDZy4NO8Y+",
- "KxhmhW1rTapWl/Z1FMHbgQZWFZoGWEBQ7Fw0hLRlMZuUmxY5R1UInDiNTFprsZRmWVucYYlkYvDeDJR1",
- "pssOGQywa3Fn4KigMUE4CESa+zNchSOj+aaJ5fshE0QmnElTNb6PfiELiWIKNytZ9xBPJVEW2h4O2YZw",
- "aRBZvkOCU0lC/QPEGHddLKseGlUA2a0/GDI5S6Hc9WYfHXMm05gI6+pBYwpu6E0kU2PcwXiBGgstMCUN",
- "iRgy/ZoHUe7PTFE/PBgMBoOsDHfncFf/e+Djpnu9UelbxESTyAfYTcxiJwIolEgZSllIBHrLJVEKI2LY",
- "oRp3s+ZtzBdCJbrP2+lR9vNcgfJvzFXIiu0gE78Up25ZbdCLclXQ1mbqF9Trb5VX4BRVm1Fgvxqtcy1K",
- "UMDTKNS2z1ifdsY1RULrQZNE5QVX4YB8z66Y3qOlqdt4RsXRHykRC/Th7Kx0lyrIxNaTbDFxkBIN68CT",
- "tZZhZ4W3YOVobole+BCIhVVNpaAh3jk+YfHawmU2Gg5tcX2RW4zeaHTKzNJoPlkyp4rjOSTzUZr6DBH9",
- "yOEZvH9/elJiDowPtp8Pnr/oPR9vH/T2wsF2D2/vHvR29vFgshs8222o6Nw+G+X2CSZes9NXoteFmY5c",
- "uKsvCrEp2Lhy7tsAymvKQn5dOlq8EW3F3m203Kru67GwrYfgjaCH2pHQUoOUOIMDlARQOB4iSCsFXBvc",
- "aQfvBtsr3Gkr5QUMrkH+vhMpCwzQGEjizFEdFwZcXKzyOG8nTmFALlJ9FbWKnbcn2uBw/8Xh/pcSzUVb",
- "rxpjlZ0ecHGbQjgcWmUlnNulFBWcMg7drWP1DeM7ttHfnW4nC1CHv+GgrQQ/Zo9bZV00bdiuX4wsk98N",
- "yYenJVsArpYNaFV4qLWALG9tnCqU5bRq9eI44mmICg4tg+EDtz2nBbtANwOXL9bfZcDYTPS0th8A/ROw",
- "pynTghhuuXQjNlPxEL2Gd+ERjo3JZAdhENCLFzw4XJgLbr2/XNfGgFk+5Atru8A32pBB+l8wbU0G6/dc",
- "3oTRfA7Rbxy+ySwpxqsOVPM6mDD116vO1g2bQuhSzaEzq8Ydop8y1S1T/qyytyGJ/XNkBVaO8LBZyrO1",
- "K97R3JKvXCGFtNsxFO10O45QkGpaTzp9n3N9bf8VWdEX+UFwBHs5T+pLFY0soCnMhEpFA2mjvfXiNukX",
- "tjgDCUfGMGkKIjOZYtZ4yT5y6suHM7QB0GV/Q9Y7rP+1mQWclc66nRd7Lw6e7bw4aAVQkg9wtdp5DHmM",
- "9cGt1EGDJB1Zv0PT1I/P3xu/QmAsdrhusHMv5IMngmvRo2fuGix2/qL/oojLEvJ0HBWuryyIE6QnmQXz",
- "FwLPtrI/cOkPGs3pZML++BRc7fxT0Hj75kDujL1e2qwjv0vrtHiFXfP/knHPFFrwQ2cAQwnZiC7zlkiY",
- "AbogCgH/9BAOwHTI0g8tyzkMGktxL2Pt7e7uPn+2v9OKr+zoChtnBA4uz6FsR1DYYvAm2nh7cYG2Cgxn",
- "2nQ52YAFy6xZ6d9nyJblG5QV0v72YNfHJQ0Hd841tu153EjyD9Y0s5OyRIcsysxsq+1yL7V3dwfP9vaf",
- "77fbxtb1OhI3yyWMyzEw5LGQxcWV3wBt8t3ROYIMvgkOyn6T7Z3dvf2DZ8/XGpVaa1QAt21gctcY2PNn",
- "B/t7uzvb7WCSfKENFgCstGHLssuz6TxM4VkNDynqorfbdFr41CnDYG9JEGEaHwUuJrpy+hg43JEwr+WL",
- "0OZgsN7+2sHV4ttWjqPMHWQi6o1qwAVKWQbC3l99r3m7a8pmMW3Og9Vi3Bc4zzS5LJ6HqbpyC9olgswp",
- "T+UdNMSVyWqbRJyLtb5tslDeEplGytwlUok+nP0AQkQzF5KKJGWjybLfEtSTW05urQ1c4gk/VzcRq9Vq",
- "tFn6ZRPuNmzT7rKU99L2by60r0VVylbHFB7jKEihzgDO1lPPCmBCIGk3SaKFib6NIs4ZCmaYwY2DsFVN",
- "2BRhNONR2PdGROono4k3FoFfo4gbWNQrQhILwW8GoT/TOgudE7RRSBpGhpUqpdL2YyNVLMh6mRv3Y3/N",
- "Jyx96SRZsqqmJ1a8gNhpPin5GCM+lWAFKogr7leBohMsTLgwZqakxDw2xmMZZWlHn/aeIVakt+8INUcn",
- "n1iL1uoYkApqKIkDwaVEJKJTKF/w4aySYbgkKyXLM1wdJ1gebAvWNbeDnrMLzjTZuvKM70D0RNx/yZEI",
- "PAxZPUsi8Jw3MsYsBVD+AiOTm4QKwx7touxmXKpRhhyz5mClGgHgeipIDi+V5cVmDiD3jvdcdKLtNuSy",
- "4ay3+rrGVf6mmgbYLFO9FPVTq5vxoI+N69g5S+F6cvyfKtjLOuhOOUI3ldAqLQALoQ3GVUksFVCmN9tE",
- "nPhtVN1PzTy1BcV+3RtctAVeWo6zdI7V7JRNuCc7f41rSOd6tkGQCRExhZIDKCSMktAZj9l9pPVtQUpn",
- "JAkKU2IpZxRSgS3BsdnekGHPnFOMsmlF1lc7bOMPNmNYjscO/doX28QOSX/K2zuRAq1MtJ9EOE9+axU6",
- "SeXIf39Vb1iQaRphgargYkuGLBdxRNlVm9blIh7ziAZIf1C9ZJ7wKOLXI/1I/ghz2Ww1O/3BKM/dqFwa",
- "m8HZzB2zIJV+8yn8qGe5WckbBNfLlvl+CzAd2oRieQOQf6IRsfhb7xm9KTB6GbB4b2fQlM/a0Ggpk7WO",
- "3bau5LYs69vxDlbtKCtg67mlNKHElbvSsiNy5U0fxKovy96tu2LQhovucoDQZboWgJlbeULahctX4xjd",
- "aLYkCcq97z3ff3bQEhn7i3ydBq3grj2b83iJR7Nhpc7auM2e7z9/8WJ3b//FzloOKhfy2rA+TWGvxfWp",
- "1KmuOM32ITxssNagTNCrf0gNga/lAZVqTt96QJ+XbN2m4IJ8bzZdckbFlXT3LGUPaDsf4xJt6aikcuV1",
- "stEGmUwIGJUjQ7dePphK2mOrMQQ4wQFVC4/DBF+bQLzslQqyXxtvWnmwHpLati0KlJZcMh3nmSEbrnP0",
- "V+Nar/DC89YA+zIdN7nx31R7NU783AdUvCJqcUOT1wCtuwuy+VxjWYo1038HEBLqgt/rgcDmjeUoY9Uo",
- "TbgEtHUkCpEUPnTKyvlnPyouf2U5C27fkpJcpfiyI7R5C65lQ3tOZI8JHaxO76nIB3sA3u6r0bhY+mJp",
- "bZFSnYz81F2/3xYZTHVc2OwEW7+/Qi7HOh9WMdCAH+0YLMnztrsllmjgpkKYrscc4RHpZXEONoYXydT4",
- "V/Wet7CanuSS4IpPJmVsr/1mLEgAYYQ4dNcLVorEieoicgNmOglrQIKuZPq+HHYQF2jY2Y6HnYoT0JvZ",
- "EeObke2gnIA9WAbOmBV9qg5SuhmMIx5cmaoOSlAi+2iAYoKZRCmDzV/xUW4Plvvaup2ksDYZFCIxN8Q1",
- "sQVjGpMZnlNA4LUeqmkpjoXcUCUh3gbaOUQhB/u2XNLKzlC/ZvIuDvNJw6GD2cI2rBvU73HmAoLyd8Fc",
- "mkAhLfaJCN61GYVaYr95c9Y19z8QuWEGVgoPcRM1I9ACMuuiAieb/+4PvxpHZATjrsKTxnU6FvPjwE8t",
- "iCRKWrzCnB0qTIACnjJVxS2N24VwliPe60dSyiBWwt6eAQ6F7d0WwgxJYIsN191LJUa/BXNXwi4tpX1x",
- "l7s+FoZNAZ45v+f9rXWvVwcgERakWI3OtFMMizM+15FU3JYvyHb1iNwEhIRVgCP/K21DDe2X3lDDX7FN",
- "5M8Kxdm3IVysPrv+/cWew1ibqF0MiWSc9SCF2C2pTfc1UCg2obzMaCWoxUJa7cgHJ+V7oU0yGLlZTuvf",
- "yI0CPMgwjTR5m1jXiip7GK2i+K2TLpo2NBer66reQ5kME653q0IZNtLvQWpl2J/vpT5GbTkuiHLvXli+",
- "aS6LWkKyLnkEXYCke6V8RWl4p4vsiY62480Kz+3N/G4QizrWMl+E4ZiMEkEm9GYJt5gXjCVcTrHOd06p",
- "dK5EGzG+QXvPUDDDQlbGzuh0pqJF+f5yz4Nw8EVVYwRRhKk1igznq+k+rAcL2OUstu7Thi8KeAT+4sck",
- "HC0DBjzOXnPXsQlegNum0cf6bHdvMNjdGdwKGfCuajIX2mlKQSh8Z+9JSlE9xRayhK964a5rQSFvLCOT",
- "VILg+BAClRMcEBSRCUDXZAUTVx4Wta6XD95qUDY5PuN/t1B23dwVRhkTPOvKgiq6aXRcAFQZD6H4vD7s",
- "Jfg2mZgJakA3nhyF3d7g4N327uH+weH29n2g+WVEaoqOffZp+/pZtIMne9HzxbM/tmfPpjvxrtfwuofy",
- "3+Wcz2o1cDuHhIhqRbZqJUNJIspIT2YR5avTOpbIAhOksXL/r+fYNzNYqixclCdZ1BmwyolT4qwHAv+w",
- "o196O1Ed/unJ8mHfKkS7OhA/g1WHAvzUbjCAubv9pdV1U9by3HlfeLH1ybM0bWDV2ePL8oSt7V3lBor7",
- "+LkkGEs7bNmJXT/VPN7RKRdUzeLlx0P2WgaMCHFmn6QKy2ATfXQ6ZVB/sfhzFlZQNJP0x51uJ/q0V94z",
- "9vf2sCMWCDBjQLvURTWgxbU7lPdcTgV4JTcthIn809a4HvOP273tFxD8Fn3a+3HQe9FH/ygE4XUNtYrk",
- "23Zvl34dtKFhsaiJA8PffrFWhJqj5zIO+oX6SnLkB7GFCLQ8nhe/c2eFy0gqLXD+uLbGFRiBRo3zS1U7",
- "e5qNilpSSCK88JUsL7hiZcU+LDIZGpMpZbKNZ3Z3kLlm9+Nhp4+OLLAmWKt5bdNS81DVssAnNI5JSLVS",
- "aYz75ojPnZbetqrxsB7asvvKo571/frZi9U5pKsC1Fcdk/0vSFj6InO3nYm7LL0ZPGfOJgUAE3ixi+gE",
- "YVYpuWQLDNtMQ8gcAdicQ4cSk7OslQEyV/ycJ6SLplyhPMewpUctZc2ev2z85AY8qkuSig1D7NxJxniG",
- "XkKXia/TE5QIHqZBnmATwaDzlGiRVvBhlmj1q0OY7tOhAZlrEy7QaodGkwejnQeyab0r3kfNsM1LvT1Y",
- "vdT34gXpdtIkXC3DzEvtJNhaAKorUjY8Ppky2SuaYGEyH1tI9LdFCtaNXOMtDrRKlCbuCkXzVJ2TPBcq",
- "cIngwxI8IRHRx1S9EcSjMI8qpTKXoqtF6vbB81nTJSbcOdUH8gshibZVACAC+osxW3gHVq2nizYGrkia",
- "NFdaPQPAbqlVHtyzlZpY41K1L01c8WqbhPJiJegMSfRu6xLbL1dWjb8PP9xjKmlv7OVCBTPOIRpmGKyu",
- "f5MxA5iTLKqc13u+C3gfW7yzlnETDl41a6boZj7q/ZdxK6NR/3Drx7/9v72Pf/W6lyt2sySiF5IJhBJd",
- "kUUPQPyRttH7ZRQ4ACDWyvTUsgrBMTiNgitinFQxvimOd3+QCY3FbziuTQFisGLKsn+vnNDf/tIcwVQg",
- "43uQkytZ9ovxue+jJJTi7jjaiImYuqLVLvB+sz9kEMt/RRYSFSosWJXGMeoPMvtEq+jgtMQRujRqYJ+w",
- "+SUaUyhZI4dMW7U4CEiirQmLNE9N3UUO0kcQHBXbsZUeXKKcvXI0EQMEfTirQQi+ef/u5Zv3v52M3py/",
- "+u3odPTLq/+EII7rnukh7Gne29s/sNUWi5Tc9izxF6AZfxGKn4/dDBaYh78gKQVKWHoUZiohpdQFFBRe",
- "RhskTtTClW9yuS2b62GTHWUNesPZ7hhafvDiLirpvF9aOmfOo57WqBuQgb0OTEMLbzg2NGXC3DtNju2p",
- "p+L7hfUmTukUe3zZ3mKzd1Hxxg1oJWhcbf0b61X4g+NPqrDJRhoYUlVgfit2qVS95tj5WCtSo7ysZzki",
- "I2U2vYQWArbKuSQxU1u2MpUvpTXkAN65LKEo32UOsagHH63Ok1mqyhdmVhhJ89qcOY21olMvIdC5Js31",
- "jAhSWAj4IIejXZNkNtmjRaK0qR+TEJEHQrpMEShzLyhkj2TOBkeCLCGo7oFdDjd8hm+yHsB7j2Xtjgvm",
- "kQP/b79+CdCwb11VWjpxTcAwKvaEHwi1zEXLaOK4qr4YRa6qz9u87914VlYtkX5Ne6vCnHkfJdb08eM/",
- "MFU/cQEWSHNa8r3jqYJ1ExIBuCxVtNRWUKM0JuEoq83dtP9dOW6Tk5xVPs9rUzlrCwMTayG3un6QS5zN",
- "x1CntCYHCVJB1eICaviaCGGCBRFHqdnwrhSw/TnvGEo9ff4MfsqJJwvhNWFE0AAdnZ/CfowxAyUdfTgr",
- "lGcxlXpqGGqgXr45PrUWroPhA4uFKmA9F8x3dH7a6XbmRBgrrzPo7/QHsJkTwnBCO4ed3f52f9AxxZth",
- "iltQlpII+IdNMcxspdPQakIv3Uv6S4FjouCL3z3JehDMZl8HrRdPC3ZLgqmwhksSQRKhYRiqvwZ4XXeg",
- "HppT2VZgbu2mk2phUypI8sYu7kdQKmHvwDR3BgMLNqrs8QsJISYKfeufNiQx77eVVmdJ5EGbrVkWTrfM",
- "SP+529kbbK81pmVDgb3r6/g9w6macUE/ETAI99ckxK06PWUm1wsZ1DAbclPcccBIxb32+0e9ZjKNYywW",
- "jmBFaiVcNinGRCLs3jUYiEqiQIsKwMTvozeM2FqsWCFswmFFyiTEX9gPNYeWd4Fp2y1yhiPwkoeLOyNh",
- "qQ9nFn8uizO9XT7X+PnueCdj4/pC2kcO9dJw7QMw0Escuipkj7ZT9gYv7r/TY84mEQ0U6mUMbINcqYQo",
- "kwgwPB08ABfoj5QrjLIY8Se0pa3OOs7YrZsfRVt/0vCz2d4R8Xlez4mIMTMR9+adFZu+tp2NFzzfzktP",
- "Ncf4ALcNJ5XLkzcHFShy5S1aPLaqymD9ONprqJxMhJ1e+IiMv/cAO9xONisK9phbDipOoVSSp7Sd7K3O",
- "OFdCvLrca6K+Fp4fPOSRZYF9v8Fd9FQY+DXJNLx8tWqHwlYiUmYMYK8G+DbPgrPf/VBW/t7lTwqBGeBK",
... diff truncated: showing 800 of 1233 linesYou can send follow-ups to the cloud agent here.
Reviewed by Cursor Bugbot for commit 156c878. Configure here.
156c878 to
124184a
Compare
sjmiller609
left a comment
There was a problem hiding this comment.
A real-VM BuildKit cache-hit/crash-recovery CI fixture is now called out explicitly in the PR follow-up
Ok sounds good, and I do think the follow up is actually warranted


Add optional builder_id to CreateBuild and the Build response; it is the
only builder selector. A build with builder_id acquires the builder
before attaching, attaches its disk read-write at the exact
/var/lib/buildkit path through the internal AllowSystemVolumeMounts
carveout (now also covering the mount path), boots the disposable builder
VM, and releases the builder after VM execution ends and before the
image-conversion wait, stamping last_used_at even on failed builds.
Builds for one builder are serialized by builder ID in the queue: a
waiting same-builder build stays pending without consuming a global
concurrency slot. Scheduling may skip a serially blocked build, while reported
queue positions remain physical submission order. Builds without builder_id
keep the exact tmpfs and queue behavior.
The builder API responses gain max_concurrency (fixed at 1),
active_build_id, and queued_builds computed from the queue, and builder
delete/prune now also reject while builds for the builder are queued or
running, via an activity checker the build manager registers. The host
computes BuildKit GC bounds from the disk size into the guest config
(reservedSpace 70%, maxUsedSpace 90%). The builder agent stamps the
persistent root with the buildkitd version and empties the root on
incompatibility before buildkitd starts; a fresh disk (no stamp, nothing
but lost+found) is stamped without a wipe. Cache contents are
best-effort, and no secrets or source/config volumes live on the builder
disk.
Stale holder recovery happens at acquire: DeleteInstance's volume detach
is warn-only, so a crashed build can leak an attachment record and leave
its builder VM holding the disk. AcquireForBuild clears those stale
attachments before marking the builder held (deleting the stale VM,
tolerating its absence, and detaching surviving records), so the next
build self-heals instead of failing until a restart runs reconciliation.
The builds tests wire the builds and builders managers to one shared
volume manager, matching production.
Tests cover the disk attach lifecycle and cleanup, the unchanged
no-builder path asserted exactly, create-time builder validation, GC
bound computation, queue serialization and introspection, stale holder
recovery success and loud failure through the real acquire path,
delete/prune rejection while builds are queued or running, and the
version stamp transitions.
Add
lib/builders/README.mdto document the user-visible cache identity,build execution, concurrency, lifecycle, recovery, and durability model.
Follow-up (not blocking)
Add a CI fixture that boots the generic Builder image in a real VM and proves
cache reuse across two disposable VMs plus stale-holder recovery after a VM
crash. The current suite covers these host/agent lifecycle invariants under the
race detector but does not execute BuildKit end to end.
Stack created with GitHub Stacks CLI • Give Feedback 💬
Note
Medium Risk
Touches build scheduling, VM volume attachment, and builder lifecycle; mistakes could leak holds, corrupt cache, or allow unsafe delete/prune, but behavior is heavily tested and fails closed on activity checks.
Overview
Adds optional
builder_idon create-build andbuilder_idon build responses so jobs can reuse a Builder’s persistent cache disk instead of ephemeral tmpfs. Builds without it keep the previous behavior.Execution and queue: Same-builder jobs are serialized via queue serial keys (pending work does not consume a global slot while waiting). After VM work finishes,
ReleaseSerialKeylets the next job start while post-build steps run. The host acquires the builder, mounts its disk at/var/lib/buildkit(with system-mount carveout), computes BuildKit GC bounds from disk size, and releases with retries; stale VM/disk holders are cleared on acquire.API and lifecycle: Builder payloads expose
max_concurrency(1),active_build_id, andqueued_builds. Create-build returns 404/409 for missing or busy builders. Delete/prune/idle-reaper consult a build activity checker so queued or disk-pending work blocks teardown. The builder agent version-stamps persistent roots and wipes cache on BuildKit mismatch.Wiring passes
builders.Managerinto the build manager and registers the activity checker. Addslib/builders/README.mdand broad tests for queue, disk lifecycle, and recovery.Reviewed by Cursor Bugbot for commit 124184a. Bugbot is set up for automated code reviews on this repo. Configure here.