Skip to content

Commit 5a930ae

Browse files
h4x3rotabh4x3rotab
authored
Merge pull request #169 from Dstack-TEE/add-dstack-host-ssh-compose-example
feat(dev-box): SSH dev box for dstack v0.6 CVMs
2 parents f2932c6 + 6331c01 commit 5a930ae

9 files changed

Lines changed: 235 additions & 0 deletions

File tree

‎.github/workflows/build-dev-box.yml‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
name: build dev-box
2+
3+
on:
4+
push:
5+
branches: [main]
6+
paths:
7+
- 'dev-box/**'
8+
- '.github/workflows/build-dev-box.yml'
9+
pull_request:
10+
paths:
11+
- 'dev-box/**'
12+
- '.github/workflows/build-dev-box.yml'
13+
workflow_dispatch:
14+
15+
permissions:
16+
contents: read
17+
packages: write
18+
19+
jobs:
20+
build:
21+
runs-on: ubuntu-latest
22+
env:
23+
IMAGE: ghcr.io/dstack-tee/dstack-dev-box
24+
steps:
25+
- uses: actions/checkout@v4
26+
27+
- uses: docker/setup-buildx-action@v3
28+
29+
# On PRs (incl. forks) GITHUB_TOKEN lacks packages:write, so only build.
30+
- name: Log in to GHCR
31+
if: github.event_name != 'pull_request'
32+
uses: docker/login-action@v3
33+
with:
34+
registry: ghcr.io
35+
username: ${{ github.actor }}
36+
password: ${{ secrets.GITHUB_TOKEN }}
37+
38+
- name: Docker metadata
39+
id: meta
40+
uses: docker/metadata-action@v5
41+
with:
42+
images: ${{ env.IMAGE }}
43+
tags: |
44+
type=raw,value=latest,enable={{is_default_branch}}
45+
type=sha,format=short
46+
47+
- name: Build and push
48+
uses: docker/build-push-action@v6
49+
with:
50+
context: dev-box
51+
file: dev-box/Dockerfile
52+
platforms: linux/amd64
53+
push: ${{ github.event_name != 'pull_request' }}
54+
tags: ${{ steps.meta.outputs.tags }}
55+
labels: ${{ steps.meta.outputs.labels }}

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -164,6 +164,7 @@ Development and debugging tools. **Not for production.**
164164
| Example | Description |
165165
|---------|-------------|
166166
| [webshell](./webshell) | Web-based shell access for debugging |
167+
| [dev-box](./dev-box) | SSH, debugging tools, and a host shell for dstack v0.6 CVMs (replaces the removed dev OS image) |
167168
| [ssh-over-gateway](./ssh-over-gateway) | SSH tunneling through dstack gateway |
168169
| [tcp-port-forwarding](./tcp-port-forwarding) | Arbitrary TCP port forwarding |
169170

‎dev-box/Dockerfile‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
# Debian trixie matches the dstack 0.6 guest OS userland.
2+
FROM debian:trixie-slim@sha256:62bd423aeee912639ca8ee681c2b762cb3e25137e0a92b20be455898a9f034ad
3+
RUN apt-get update && apt-get install -y --no-install-recommends \
4+
openssh-server openssh-sftp-server rsync \
5+
bash-completion ca-certificates curl dnsutils file gdb git htop \
6+
iproute2 iputils-ping jq less lsof procps psmisc python3 strace \
7+
tcpdump tmux util-linux vim \
8+
docker-cli docker-compose \
9+
&& rm -rf /var/lib/apt/lists/* /etc/ssh/ssh_host_*
10+
COPY sshd_config /etc/ssh/sshd_config
11+
COPY motd /etc/motd
12+
COPY --chmod=755 host-shell /usr/local/bin/host-shell
13+
COPY --chmod=755 entrypoint.sh /usr/local/bin/entrypoint
14+
ENTRYPOINT ["/usr/local/bin/entrypoint"]

‎dev-box/README.md‎

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
# dstack Dev Box
2+
3+
dstack v0.6 no longer ships a dev OS image with SSH and debugging tools. Add this one container to your compose file to get them back: root SSH into the CVM, a full Debian toolset, the host filesystem, the host Docker daemon, and a shell in the host OS itself.
4+
5+
> [!WARNING]
6+
> This gives anyone holding an authorized key full root on the CVM: every container, the decrypted disk, and every secret in memory. It breaks the confidentiality guarantee your users rely on. The dev box is part of the attested `app-compose`, so verifiers can see it's there. Use it for development and debugging only.
7+
8+
## Usage
9+
10+
Add the `dev-box` service from [`docker-compose.yaml`](./docker-compose.yaml) next to your workload:
11+
12+
```yaml
13+
services:
14+
dev-box:
15+
image: ghcr.io/dstack-tee/dstack-dev-box:latest
16+
restart: unless-stopped
17+
privileged: true
18+
pid: host
19+
network_mode: host
20+
uts: host
21+
environment:
22+
- SSH_AUTHORIZED_KEYS=${SSH_AUTHORIZED_KEYS}
23+
volumes:
24+
- /:/host:rslave
25+
- /var/run/docker.sock:/var/run/docker.sock
26+
- /dev/pts:/dev/pts
27+
- ssh-host-keys:/etc/ssh/host-keys
28+
- home:/root
29+
30+
your-app:
31+
image: ...
32+
33+
volumes:
34+
ssh-host-keys:
35+
home:
36+
```
37+
38+
Deploy with your public key or keys in `SSH_AUTHORIZED_KEYS`, in `authorized_keys` format with one key per line. The container exits with an error if the variable is empty.
39+
40+
```bash
41+
phala deploy -n my-app -c docker-compose.yaml \
42+
--image dstack-0.6.1 \
43+
-e SSH_AUTHORIZED_KEYS="$(cat ~/.ssh/id_ed25519.pub)"
44+
```
45+
46+
On a plain dstack deployment, set `SSH_AUTHORIZED_KEYS` as an encrypted environment variable.
47+
48+
## Connect
49+
50+
`sshd` listens on port `1022`. Reach it through the dstack-gateway by adding this to `~/.ssh/config`:
51+
52+
```
53+
Host my-app
54+
User root
55+
ProxyCommand openssl s_client -quiet -connect <app-id>-1022.<gateway-domain>:443 -servername <app-id>-1022.<gateway-domain> -verify_hostname <app-id>-1022.<gateway-domain> -verify_return_error
56+
```
57+
58+
Replace `<app-id>` with your app ID and `<gateway-domain>` with the gateway base domain of the node the CVM runs on, for example `dstack-pha-prod15.phala.network`. Change `443` if your gateway uses a different port.
59+
60+
> [!NOTE]
61+
> On macOS the system `openssl` is LibreSSL, which can time out with this `ProxyCommand`. Install OpenSSL with `brew install openssl` and make sure it's first on your `PATH`.
62+
63+
## What you get
64+
65+
You land in the dev box: a Debian trixie container, the same base as the dstack 0.6 guest OS, that shares the host's PID, network, and UTS namespaces.
66+
67+
| Need | How |
68+
|------|-----|
69+
| Host processes, network, hostname | Visible directly: `ps`, `htop`, `ip`, `ss`, `tcpdump`, `strace -p` |
70+
| Host files | `/host`, for example `/host/dstack/app-compose.json` |
71+
| Containers | `docker` and `docker compose` talk to the host daemon |
72+
| systemd, journald, anything else on the host | `host-shell` opens a shell in the host OS; `host-shell CMD` runs one command there, for example `host-shell journalctl -u docker` |
73+
| File transfer | `scp`, `sftp`, and `rsync`, for example `rsync my-app:/host/tmp/dump ./` |
74+
| Port forwarding | `ssh -N -L 8080:127.0.0.1:8080 my-app` reaches any port on the CVM |
75+
76+
Also installed: `vim`, `tmux`, `git`, `curl`, `jq`, `dig`, `gdb`, `lsof`, and `python3`. Run `apt-get install` for anything else. Installed packages last until the container is recreated. `/root` is a volume, so dotfiles and history persist across restarts and updates, as does the SSH host key.
77+
78+
## How it works
79+
80+
The host rootfs is a read-only, dm-verity-protected image, so tools can't be installed on the host itself. The dev box brings its own userland and borrows the host's namespaces instead, the same design as Bottlerocket's admin container and `kubectl debug node`.
81+
82+
| Setting | Why |
83+
|---------|-----|
84+
| `privileged: true` | Full device access and capabilities for `tcpdump`, `strace`, `gdb`, and the `setns()` calls in `host-shell`. |
85+
| `pid: host`, `network_mode: host`, `uts: host` | The box sees the host's processes, network stack, and hostname. |
86+
| `/:/host:rslave` | The host filesystem, including mounts made after the box starts. |
87+
| `/var/run/docker.sock` | `docker` drives the host daemon. |
88+
| `/dev/pts:/dev/pts` | PTYs come from the host's devpts instance, so a terminal still works after `host-shell` switches to the host mount namespace. |
89+
90+
[`host-shell`](./host-shell) runs `nsenter --target 1 --all --root`, joining every namespace of host PID 1 and its root directory. Its shell is the host's own `/bin/bash`, with no dev box tools.
91+
92+
## Build your own image
93+
94+
The compose file uses `ghcr.io/dstack-tee/dstack-dev-box:latest`, built from this directory by [`build-dev-box.yml`](../.github/workflows/build-dev-box.yml). To add packages, edit the [`Dockerfile`](./Dockerfile), push the image to a registry, and change `image:`. For a quick test, [ttl.sh](https://ttl.sh) holds anonymous images for up to 24 hours:
95+
96+
```bash
97+
docker buildx build --platform linux/amd64 -t ttl.sh/my-dev-box:24h --push .
98+
```

‎dev-box/docker-compose.yaml‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
# dstack dev box: SSH access to the CVM with a full debugging toolset.
2+
#
3+
# Sessions run in this container but share the host's PID, network, and
4+
# UTS namespaces, see the host filesystem at /host, and drive the host
5+
# Docker daemon. `host-shell` drops into the host OS itself.
6+
#
7+
# This gives full root on the CVM. Development/debugging only.
8+
services:
9+
dev-box:
10+
image: ghcr.io/dstack-tee/dstack-dev-box:latest
11+
restart: unless-stopped
12+
privileged: true
13+
pid: host
14+
network_mode: host
15+
uts: host
16+
environment:
17+
# One or more public keys, newline-separated (authorized_keys format).
18+
- SSH_AUTHORIZED_KEYS=${SSH_AUTHORIZED_KEYS}
19+
volumes:
20+
- /:/host:rslave
21+
- /var/run/docker.sock:/var/run/docker.sock
22+
# Share the host's devpts so `host-shell` keeps a working terminal.
23+
- /dev/pts:/dev/pts
24+
- ssh-host-keys:/etc/ssh/host-keys
25+
- home:/root
26+
27+
volumes:
28+
ssh-host-keys:
29+
home:

‎dev-box/entrypoint.sh‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
#!/bin/sh
2+
set -eu
3+
if [ -z "${SSH_AUTHORIZED_KEYS:-}" ]; then
4+
echo "SSH_AUTHORIZED_KEYS is not set" >&2
5+
exit 1
6+
fi
7+
install -d -m 700 /root/.ssh
8+
printf '%s\n' "$SSH_AUTHORIZED_KEYS" > /root/.ssh/authorized_keys
9+
chmod 600 /root/.ssh/authorized_keys
10+
key=/etc/ssh/host-keys/ssh_host_ed25519_key
11+
[ -f "$key" ] || ssh-keygen -q -t ed25519 -N '' -f "$key"
12+
install -d -m 755 /run/sshd
13+
exec /usr/sbin/sshd -D -e

‎dev-box/host-shell‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
#!/bin/sh
2+
# Run a login shell, or the given command, directly on the host: enter
3+
# every namespace of host PID 1 and chroot to its root. --wdns resolves
4+
# the directory inside the host mount namespace.
5+
if [ $# -eq 0 ]; then
6+
exec nsenter --target 1 --all --root --wdns=/root -- /bin/bash -l
7+
fi
8+
exec nsenter --target 1 --all --root --wdns=/root -- "$@"

‎dev-box/motd‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
2+
dstack dev box: full root on this CVM. Development use only.
3+
4+
/host host root filesystem
5+
docker talks to the host Docker daemon
6+
host-shell shell in the host OS (all host namespaces)
7+
host-shell CMD run CMD on the host, e.g. host-shell journalctl -u docker
8+

‎dev-box/sshd_config‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
Port 1022
2+
HostKey /etc/ssh/host-keys/ssh_host_ed25519_key
3+
AllowUsers root
4+
PermitRootLogin prohibit-password
5+
AuthenticationMethods publickey
6+
PasswordAuthentication no
7+
KbdInteractiveAuthentication no
8+
UsePAM no
9+
Subsystem sftp internal-sftp

0 commit comments

Comments
 (0)