Skip to content

Commit 2d75cd1

Browse files
kvinwangclaude
andcommitted
guest-agent: document where anonymous scopes are configured
There is no built-in anonymous default, so the sample configs are the only place a host learns what to open. Fill in `[cvm.guest_agent_api]` in the shipped `vmm.toml`, in the `dstackup install` renderer, and in the two dev/test stacks with what an unconfigured 8090 used to serve: the dashboard plus the identity, version and health it publishes. Co-Authored-By: Claude Code <noreply@anthropic.com> Signed-off-by: Kevin Wang <wy721@qq.com>
1 parent 476cd6b commit 2d75cd1

7 files changed

Lines changed: 75 additions & 26 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
1414
- guest: a fresh CVM with a verity volume could take the volume for its data disk and try to LUKS-format it: volumes labelled their data partition `dstack-data`, the data disk's label. `dstack verity` now labels it `dstack-volume-data`, and the guest skips disks that carry a `dstack-volume` partition, so volumes built by older releases are safe too
1515

1616
### Changed
17-
- guest-agent: the external listener (port 8090) is no longer anonymous. Every route needs a scope, granted by a signed Ed25519 JWT or by `[cvm.guest_agent_api].anonymous_scopes` in sys-config; the gateway gets its token minted inside the CVM from the app root key. `public_logs` / `public_sysinfo` / `public_tcbinfo` are unchanged and still decide what may leave the TEE. The dashboard is filtered block by block by scope. See `docs/guest-api-auth.md`. An old gateway sees 401s and drops new guests from rotation until it is upgraded
17+
- guest-agent: the external listener (port 8090) is no longer anonymous. Every route needs a scope, granted by a signed Ed25519 JWT or by `[cvm.guest_agent_api].anonymous_scopes` in sys-config. There is no built-in anonymous default; the sample `vmm.toml` opens `guest:dashboard`, `guest:info`, `guest:version` and `guest:health`, which is what an unconfigured 8090 used to serve, while containers, logs, metrics and app-key attestation stay behind a token. The gateway gets its token minted inside the CVM from the app root key. `public_logs` / `public_sysinfo` / `public_tcbinfo` are unchanged and still decide what may leave the TEE. The dashboard is filtered block by block by scope. See `docs/guest-api-auth.md`. An old gateway sees 401s and drops new guests from rotation until it is upgraded
1818
- vmm: `qmp_socket` now defaults to `true`. Each CVM's QEMU exposes `qmp.sock` in its work dir, so the host can change disk options or compact a bloated `hda.img` (`blockdev-mirror`) without stopping the CVM. Applies to CVMs started after the change; set `qmp_socket = false` to opt out
1919

2020
## [0.6.1] - 2026-10-12

‎docs/guest-api-auth.md‎

Lines changed: 40 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -33,33 +33,53 @@ not passed to the router.
3333

3434
## Default: nothing anonymous
3535

36-
With no configuration, every scope above requires a token. The gateway still
37-
works, because it is issued one internally; the dashboard, `/metrics` and the
38-
log views stop answering to the public.
39-
40-
The app's own SDKs are unaffected: they use the internal Unix socket, which is
36+
With no `[cvm.guest_agent_api]` block, an anonymous caller is granted nothing:
37+
the dashboard, its identity block, `/metrics` and the log views all need a
38+
token. The gateway still works, because it is issued one internally. The app's
39+
own SDKs are unaffected: they use the internal Unix socket, which is
4140
unauthenticated and reachable only by the application.
4241

42+
The sample `vmm.toml` ships the block filled in with what an unconfigured 8090
43+
used to serve, so a host that copies the sample keeps the page it had:
44+
45+
| Scope | Effect |
46+
|---|---|
47+
| `guest:version` | `Worker.Version` answers |
48+
| `guest:health` | `Worker.Health` answers |
49+
| `guest:info` | the identity block renders |
50+
| `guest:dashboard` | the page loads at all |
51+
52+
Everything else is about the app's *contents* and stays behind a token even in
53+
that sample: the container table, `/metrics`, the log views and
54+
`Worker.GetAttestationForAppKey`.
55+
56+
The dashboard does not fetch `Info` unless a block would show it, so an
57+
anonymous visitor granted only `guest:dashboard` costs the TEE no attestation.
58+
4359
## Configuring anonymous access
4460

4561
Anonymous access is a host decision, made in sys-config. The VMM writes it from
4662
`[cvm.guest_agent_api]`:
4763

4864
```toml
4965
[cvm.guest_agent_api]
50-
anonymous_scopes = ["guest:dashboard", "guest:version", "guest:health"]
66+
anonymous_scopes = ["guest:dashboard", "guest:version", "guest:health", "guest:info"]
5167
```
5268

5369
The same block in the guest's `/dstack/.host-shared/.sys-config.json`:
5470

5571
```json
5672
{
5773
"guest_agent_api": {
58-
"anonymous_scopes": ["guest:dashboard", "guest:version", "guest:health"]
74+
"anonymous_scopes": ["guest:dashboard", "guest:version", "guest:health", "guest:info"]
5975
}
6076
}
6177
```
6278

79+
The list is whole, not additive, and there is no built-in default to fall back
80+
on: omitting `anonymous_scopes`, or leaving the list empty, closes the listener
81+
to anonymous callers. Removing the block entirely does the same.
82+
6383
Listing `guest:dashboard` without `guest:containers`, `guest:info`, `guest:logs`
6484
or `guest:metrics` gives an anonymous visitor the page with every block hidden.
6585
Scopes are independent: each one adds exactly its own block.
@@ -92,6 +112,9 @@ it over on registration, bound to the app id and short-lived:
92112

93113
1. The agent and `dstack-util`, both inside the CVM, derive the same Ed25519
94114
key: `derive_app_key(app_root_key, "dstack:guest-api:issuer:v1", Ed25519)`.
115+
That domain string is final: it is a wire contract between `dstack-util` and
116+
the agent, which are upgraded together but may briefly disagree, so it is
117+
never versioned again.
95118
2. `dstack-util` signs a token with `iss = "dstack:guest-api:<app_id>"`,
96119
`aud = <app_id>`, a 10-minute lifetime, and only the `guest:health` scope.
97120
3. Registration carries it to the gateway, which presents it on the v1 `Health`
@@ -135,14 +158,21 @@ host cannot alter without changing the app id.
135158

136159
## Compatibility
137160

138-
A guest with this change answers 401 to an old gateway that presents no token. The
139-
gateway counts that as unreachable and drops the instance after
161+
A guest with this change answers 401 to an old gateway that presents no token.
162+
The gateway counts that as unreachable and drops the instance after
140163
`failure_threshold` consecutive polls, so a mixed cluster rebalances away from
141164
new guests until the gateway is upgraded.
142165

143166
During the rollout, either upgrade the gateway first, or leave the host open:
144167

145168
```toml
146169
[cvm.guest_agent_api]
147-
anonymous_scopes = ["guest:version", "guest:health"]
170+
anonymous_scopes = ["guest:version", "guest:health", "guest:info", "guest:dashboard"]
148171
```
172+
173+
The reverse direction is quieter. A new guest registers with an old gateway
174+
node that does not store `agent_token`; when that record syncs to an upgraded
175+
node, the missing token means the upgraded node's health probe is refused too.
176+
The instance drops out of rotation while `order_by_stickiness` keeps routing
177+
user traffic to it. Upgrade every gateway node in the cluster before rolling
178+
guests, not just the one the VMM talks to.

‎dstack/crates/dstack-cli-core/src/config.rs‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -348,6 +348,12 @@ qemu_hotplug_off = true
348348
host_share_mode = "{host_share_mode}"
349349
qgs_port = 4050
350350
351+
# What an unconfigured 8090 serves: the dashboard, and the identity, version
352+
# and health it publishes. Containers, logs, metrics and app-key attestation
353+
# need a token. See docs/guest-api-auth.md.
354+
[cvm.guest_agent_api]
355+
anonymous_scopes = ["guest:dashboard", "guest:version", "guest:health", "guest:info"]
356+
351357
[cvm.product]
352358
sys_vendor = "dstack"
353359
product_name = "dstack"

‎dstack/dstack-types/src/lib.rs‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1255,7 +1255,7 @@ pub struct SysConfig {
12551255
#[serde(default, skip_serializing_if = "Option::is_none")]
12561256
pub mr_config: Option<String>,
12571257
/// Access control for the guest agent's external listener. Absent means
1258-
/// the code defaults: nothing anonymous, no external issuers.
1258+
/// nothing anonymous and no external issuers.
12591259
///
12601260
/// Host-supplied, and deliberately not measured: it decides who may *ask*
12611261
/// the agent, never whether data may leave the TEE. Confidentiality stays
@@ -1273,8 +1273,9 @@ pub struct GuestAgentApiConfig {
12731273
/// Scopes granted to a caller that presents no token.
12741274
///
12751275
/// Empty by default: an app that wants its dashboard on the internet has
1276-
/// to say so. This is an exposure choice, not a trust one -- the anonymous
1277-
/// caller is the same party a public 8090 already served.
1276+
/// to say so, in `vmm.toml` or in the guest's sys-config. What a host is
1277+
/// expected to open is spelled out in the sample `vmm.toml` rather than
1278+
/// here, so the shipped default stays visible where an operator sets it.
12781279
#[serde(default)]
12791280
pub anonymous_scopes: Vec<String>,
12801281
/// Issuers other than the app's own root key.

‎dstack/vmm/vmm.toml‎

Lines changed: 17 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -109,18 +109,6 @@ tdx_attestation_variant = "auto"
109109

110110
host_share_mode = "9p"
111111

112-
# Access control for the guest agent's external listener (port 8090). Absent
113-
# means nothing is anonymous: only holders of a token (the gateway, minted
114-
# inside the CVM) can call it. This is an exposure choice, not a
115-
# confidentiality one -- what may leave the TEE is still decided by the
116-
# measured public_logs / public_sysinfo / public_tcbinfo switches in
117-
# app-compose. See docs/guest-api-auth.md.
118-
# [cvm.guest_agent_api]
119-
# anonymous_scopes = ["guest:dashboard", "guest:version", "guest:health"]
120-
# [[cvm.guest_agent_api.external_issuers]]
121-
# name = "ops"
122-
# public_keys = ["<32-byte ed25519 public key, hex>"]
123-
124112
# Default preallocation for a new CVM data disk. This is a host storage
125113
# policy -- the guest cannot see it and it does not affect measurements -- and
126114
# a deploy request may pick another mode per VM.
@@ -317,6 +305,23 @@ sanitize_on_attach = false
317305
# Maximum time for all reset GPUs to become continuously VFIO-ready.
318306
sbr_timeout_ms = 10000
319307

308+
# Access control for the guest agent's external listener (port 8090). Every
309+
# route needs a scope, granted by a signed Ed25519 JWT or by the anonymous list
310+
# below. This is an exposure choice, not a confidentiality one -- what may leave
311+
# the TEE is still decided by the measured public_logs / public_sysinfo /
312+
# public_tcbinfo switches in app-compose. See docs/guest-api-auth.md.
313+
#
314+
# The list is whole, not additive, and there is no built-in anonymous default:
315+
# comment the whole block out to refuse anonymous callers entirely. The set
316+
# below is what an unconfigured 8090 used to serve -- the dashboard, and the
317+
# identity, version and health it publishes. Containers, logs, metrics and
318+
# app-key attestation need a token either way.
319+
[cvm.guest_agent_api]
320+
anonymous_scopes = ["guest:dashboard", "guest:version", "guest:health", "guest:info"]
321+
# [[cvm.guest_agent_api.external_issuers]]
322+
# name = "ops"
323+
# public_keys = ["<32-byte ed25519 public key, hex>"]
324+
320325
[gateway]
321326
base_domain = "localhost"
322327
port = 8082

‎test-suites/full-stack-compose/scripts/render-config.sh‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -216,6 +216,9 @@ exclude = []
216216
include = []
217217
allow_attach_all = false
218218
219+
[cvm.guest_agent_api]
220+
anonymous_scopes = ["guest:dashboard", "guest:version", "guest:health", "guest:info"]
221+
219222
[gateway]
220223
base_domain = "${BASE_DOMAIN}"
221224
port = ${GATEWAY1_PROXY_HOST_PORT}

‎tools/dev-stack.sh‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -261,6 +261,10 @@ kms_urls = ["https://$KMS_DOMAIN:$KMS_RPC_LISTEN_PORT"]
261261
gateway_urls = ["https://$GATEWAY_DOMAIN:$GATEWAY_RPC_LISTEN_PORT"]
262262
cid_start = $VMM_CID_POOL_START
263263
cid_pool_size = $VMM_CID_POOL_SIZE
264+
265+
[cvm.guest_agent_api]
266+
anonymous_scopes = ["guest:dashboard", "guest:version", "guest:health", "guest:info"]
267+
264268
[cvm.port_mapping]
265269
enabled = $VMM_PORT_MAPPING_ENABLED
266270
address = "127.0.0.1"

0 commit comments

Comments
 (0)