feat(blaze): add build-time data-plane providers - #3074
WeissonHan wants to merge 11 commits into
Conversation
Capability: select a source-level data-plane provider when assembling the daemon while preserving the file-backed provider in the standard binary. Add the provider contract, conformance helpers, daemon builder, strict lease identity, and typed opened resources. Route create and delete through the selected provider and fail startup instead of silently changing providers. This commit establishes direct lifecycle coordination. Recovery of ambiguous provider transitions across daemon crashes is completed by a later commit in this series. Assisted-by: Codex:GPT-5 Signed-off-by: Weisson <Weisson@linux.alibaba.com>
Capability: compare persisted sandbox records with provider inventory during startup and adopt a running sandbox only when all recorded identities agree. Persist lease and backend process identities, classify missing, duplicated, stale, or mismatched ownership as recovery-required, and retain unresolved resources instead of reporting successful cleanup. This commit establishes inventory and adoption. Durable write-ahead recovery for transitions that crash around provider calls is completed later in this series. Assisted-by: Codex:GPT-5 Signed-off-by: Weisson <Weisson@linux.alibaba.com>
Capability: coordinate provider-owned root and memory content with Blaze-owned backend artifacts under one management checkpoint identifier. Add capture, rollback, pruning, deletion, and retirement flows. Keep provider references outside management responses and require all-or-nothing root and memory ownership for provider checkpoints. This commit establishes the checkpoint workflow. Cross-operation write-ahead recovery and strict retirement identity are completed later in this series. Assisted-by: Codex:GPT-5 Signed-off-by: Weisson <Weisson@linux.alibaba.com>
Capability: preserve provider-owned data-plane content while hibernated and resume it through a fresh lease without changing the management sandbox ID. Coordinate guest capability checks, lifecycle hooks, backend stop and restore, compensation, and retirement. Keep provider references outside management responses and leave the standard file-backed hibernation path unchanged. This commit establishes suspend and resume coordination. Crash-safe transition recovery and strict retirement identity are completed later in this series. Assisted-by: Codex:GPT-5 Signed-off-by: Weisson <Weisson@linux.alibaba.com>
Capability: query provider-owned capacity and drain selected resource classes through optional management routes. Define stable resource-class identities, mutually exclusive accounting states, monotonic revisions, and idempotent drains. Validate results before publication and return not implemented when the selected provider lacks this extension. Capacity covers provider-owned data-plane resources only. It does not implement a reusable sandbox pool or runtime provider discovery. Assisted-by: Codex:GPT-5 Signed-off-by: Weisson <Weisson@linux.alibaba.com>
github-actions
Bot
added
component:blaze
|
@codex review This PR adds a build-time data-plane provider framework with lifecycle reconciliation, checkpoints, suspension, capacity controls, and crash-safe ownership transitions. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a9c1263aa4
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
@codex review This update preserves exact provider leases awaiting explicit cleanup, quarantines identity mismatches, and restores legacy file-backed cleanup without bypassing request-scoped ownership. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e9cf926f2b
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
@codex review This update preserves the existing sandbox management schema by separating durable recovery journals from API response DTOs and adds exact serialization coverage. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cb2a04db2f
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
@codex review This update documents every public inventory and reconciliation field and method and makes missing Rustdoc a compile-time error for the provider API. |
|
Codex Review: Didn't find any major issues. More of your lovely PRs please. Reviewed commit: ℹ️ About Codex in GitHubCodex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback". |
|
@codex review This PR adds build-time data-plane providers with lifecycle recovery, checkpoint and suspension integration, capacity controls, and contract validation tests. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f596463669
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
f596463 to
3bf37af
Compare
|
@codex review This PR adds build-time data-plane providers with durable lifecycle recovery, checkpoint and suspension integration, opened-resource handoff, and capability-gated capacity controls. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3bf37afe9c
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Capability: recover provider-backed lifecycle operations across daemon crashes and ambiguous external outcomes. Persist exact request, lease, backend, storage, checkpoint, and suspension identities before side effects. Reconcile creation, deletion, adoption, restore, checkpoint, suspension, and retirement without guessing ownership. Bind state and file-storage operations to opened directory objects, reject stale or colliding identities, bound inventory traversal, and fail closed when durable evidence cannot prove a safe action. The contract coordinates control-path ownership only. Provider implementations remain responsible for data-path behavior and declared capability guarantees. Assisted-by: Codex:GPT-5 Signed-off-by: Weisson <Weisson@linux.alibaba.com>
Capability: keep exact provider leases owned by lifecycle records that wait for explicit cleanup. Remove only exact owned leases from orphan inventory while leaving identity mismatches eligible for quarantine. A retained lifecycle record remains an owner until an explicit cleanup request resolves it. Assisted-by: Codex:GPT-5 Signed-off-by: Weisson <Weisson@linux.alibaba.com>
Capability: preserve the existing sandbox management schema while lifecycle state records provider write-ahead identities for crash recovery. Serialize operations through a dedicated management response type and test that persisted provider records remain absent from API responses. Assisted-by: Codex:GPT-5 Signed-off-by: Weisson <Weisson@linux.alibaba.com>
Capability: give extension authors field-level snapshot, pagination, and reconciliation invariants. Deny missing documentation in the public provider API so later public items cannot silently ship without Rustdoc. Assisted-by: Codex:GPT-5 Signed-off-by: Weisson <Weisson@linux.alibaba.com>
Keep provider composition and operational guidance in the English and Chinese user guides, with runnable examples and explicit limitations. Restore design notes, component summaries, and contributor instructions to the base revision so the documentation changes have one entry point. This documentation-only change leaves runtime behavior and source-level API documentation unchanged. Assisted-by: Codex:0.149.0 Signed-off-by: Weisson <Weisson@linux.alibaba.com>
Exercise startup rejection, opened-resource ownership, lifecycle evidence, checkpoint and suspension receipts, and capacity error handling through the public contract validators and daemon management paths. Use deterministic providers, ordinary opened files, and guest responses so the assertions run without a particular storage or runtime extension. This is test-only coverage; production behavior and APIs are unchanged. Assisted-by: Codex:0.149.0 Signed-off-by: Weisson <Weisson@linux.alibaba.com>
3bf37af to
8acb6a1
Compare
|
@codex review This PR adds build-time data-plane providers with durable lifecycle recovery, checkpoint and suspension integration, opened-resource handoff, and capability-gated capacity controls. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8acb6a191d
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if let Err(error) = instance | ||
| .advance_restore_phase(OperationPhase::RestoreBackendStarted) | ||
| .map_err(BlazeDaemonError::from) | ||
| .and_then(|_| self.persist_and_retain(instance.clone())) |
There was a problem hiding this comment.
When a provider-managed checkpoint restores a guest with a guest transport, this path proceeds from a readiness ping directly to committing the replacement lease without negotiating or invoking the newly added reseed_rng and post_restore hooks. Restoring the same captured memory more than once can therefore repeat guest RNG state and leave the real-time clock stale, risking duplicate secrets and incorrect timestamps; require the lifecycle hooks before publishing the replacement, as the provider-suspension resume path does.
Useful? React with 👍 / 👎.
| .drain_provider_capacity(DrainRequest { | ||
| scope, | ||
| operation_id: request.operation_id.unwrap_or_else(Uuid::new_v4), | ||
| }) |
There was a problem hiding this comment.
When the request omits operation_id, the server generates an ID that is returned only on success. If both internal attempts return OutcomeUnknown, the client receives an error containing no generated ID, so it cannot retry the potentially accepted drain with the contract's stable idempotency identity; another empty-body retry generates a different operation and can accumulate duplicate provider-side request records. Require the ID from the caller or persist/expose the generated ID on ambiguous outcomes.
Useful? React with 👍 / 👎.
1 participant
Why
Blaze needs an implementation-neutral way for downstream developers to
assemble a different data plane without copying daemon lifecycle logic or
changing the standard file-backed binary. Ownership, recovery, checkpoints,
hibernation, and capacity must follow a coherent contract so interrupted
operations do not leave conflicting resource ownership records.
What changed
blaze-provider-apisource contract and theblaze-provider-conformancevalidation library with a runnable minimal example.BlazeDaemonBuilderfor selecting exactly one provider when a daemonbinary is built; standard
blazedstill selects the built-in file provider.resume, capacity, and drain operations through capability-gated contracts.
reconciliation, and write-ahead recovery for interrupted or ambiguous
provider transitions.
the existing path-backed file-provider behavior.
ownership, guest lifecycle evidence, checkpoint and suspension receipts,
capacity validation, and exact HTTP error responses.
An absent lookup under a requested identity cannot erase recovery records
after the provider has returned an unrelated or invalid lease binding.
Matching bindings with invalid resource content still use normal compensation.
and an assembled provider. Active or uncertain foreign state rejects startup
before listeners open; proven-clean records are neither imported nor deleted.
including runnable examples, extension limitations, recovery guidance, and
website-compatible source links. Rust API documentation remains included.
Related issue
Closes #3073.
Contributor architecture guidance is tracked separately in #3093. Existing
design documents, component READMEs, and contributor instructions are unchanged
relative to the base branch.
User / Agent impact
Downstream developers can implement a source-level data-plane provider, run
the shared conformance checks, and assemble it with Blaze at build time. The
standard
blazedbinary keeps its existing file-backed provider and does notgain runtime provider selection or new configuration.
Risk and compatibility
The public Rust API is additive. Provider selection is fixed in the assembled
binary, startup fails if the selected provider cannot satisfy its contract,
and provider-backed lifecycle state uses an identity-specific namespace.
Both standard and provider-specific startup inspect foreign namespaces before
serving. The standard daemon still recovers its own outer-root records normally.
Responses, opened resources, inventory pages, and durable identities are
validated before use. The standard file-provider path remains the default and
does not accept provider selection from configuration or tenant requests.
This does not add runtime plugin discovery or reusable sandbox pooling. The
example file provider demonstrates the base contract; it is not a production
storage provider and does not implement the optional lifecycle extensions.
Validation
Validated the six rewritten commits on Linux x86_64 with Rust/Cargo 1.88.0,
using separately exported source trees and initially empty target directories:
3450fdbc78aa53c7c7e0c97bf5389d5f6c63e45018791586505be23bffbbc115ad45b88a8275a2bbc4186dee41c5e96b1816128ad4f01fad91ad9f9bc535b86fd9d6f43f7e2a6c82731e87e04e19fd1ffb95af634923fc73824172b5139cdfd1db3d65ca8acb6a191d23a30bc859f0deccdeb3a90a269f97Each commit also passed the default build, format check, strict all-target
Clippy, both guide example commands, and strict workspace Rustdoc. The head
additionally passed fault-injection Clippy and the documentation-test command.
The preceding five commits are unchanged.
Commands were run from
src/blazewith the lockfile unchanged:Cargo compiled test executables as an ordinary user. All emitted workspace test
executables were then run with
--test-threads=1, administrator privileges forfilesystem ownership checks, isolated temporary roots, and cleared
BLAZE_TEST_FAILPOINTSandBLAZE_TEST_FAILPOINT_FILE. Compilation could runconcurrently, but a host-side lock serialized test execution across revisions.
The default and fault-injection suites overlap and their counts must not be
added. There are currently no executable documentation tests.
The unsafe-preparation regression was checked against the previous production
code with the same test fixture: all three retention tests fail, while the
valid-binding compensation control passes. All four pass with the correction.
The standard-startup regression was also checked against the previous production
code with the same fixture: seven rejection tests fail and two normal-startup
controls pass. All nine pass with the correction. Two additional tests cover
namespace replacement or addition and records added during inspection. The
production daemon tests verify rejection before touching an existing Unix
listener, with a valid-config control that reaches listener setup.
Separately ran the opt-in
spawner::firecracker::tests::opened_resource_restore_reaches_gueston thesame head with KVM, Firecracker v1.16.0, and ordinary-file copies of compatible
restore inputs: 1 passed, 0 failed, 0 ignored. The guest returned
provider-restore-ok; no runtime process remained, the network namespaceinventory matched its initial state, and the preprovisioned root-drive target
was unchanged. This verifies one concrete opened-file restore path, not every
provider implementation or hardware failure scenario.
The paired user guides passed documentation lint and relative-link checks.
The website built in both locales at
/and/base-url-check/, and staticlink validation passed for 190 HTML files in each build. Local website checks
used Node 25.9.0 and npm 11.12.1; hosted CI uses its configured Node version.
Also passed
git diff --checkand the CI-version commitlint for all 11 commitsfrom
df8f93e61628d1f008e8cb23a55605ceae3886fcthrough the head above.Documentation and rollback
The English and Chinese user guides describe provider composition, lifecycle
operations, capability boundaries, and the runnable public examples. No design
document changes are included in the aggregate PR diff.
Existing standard deployments need no configuration migration and can revert
this PR. For a custom assembled daemon, first complete resource cleanup with
the original provider before reverting or switching to another binary. The
standard daemon does not adopt provider-specific records and refuses startup
while they remain active or uncertain. Keep the provider identity and its
compatible records together; do not rename or remove unresolved state to force
a downgrade or bypass recovery checks.