Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 51 additions & 0 deletions docs/deployment/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -185,6 +185,45 @@ swim:

---

## Web Dashboard (`viz`)

The runtime serves its own web dashboard: the admin ChiMod starts one HTTP
server per node, and every ChiMod that ships a `viz/` directory of
HTML/CSS/JS is mounted at `/viz/<mod_name>` with the routes its container
registered. Nothing here is collective, so each node serves the same UI
independently. See [Monitoring → Runtime Dashboard](./monitoring#runtime-dashboard)
for the pages and the REST API.

| Parameter | Default | Description |
|-----------|---------|-------------|
| `enabled` | *(see note)* | Serve the dashboard on this node. Setting this key is an explicit choice that overrides the CLI default. |
| `port` | `8080` | TCP port. `0` binds an ephemeral port (the bound port is logged). |
| `bind` | `"127.0.0.1"` | Bind address. Loopback by default: the dashboard exposes runtime internals and has no authentication, so binding `0.0.0.0` publishes them to the network — prefer an SSH tunnel. |
| `max_threads` | `16` | HTTP request threads. Thread-per-connection: a browser parks ~6 keep-alive sockets, so keep this comfortably above that. |

```yaml
viz:
enabled: true
port: 8080
bind: "127.0.0.1"
max_threads: 16
```

:::note Who gets the dashboard by default
`clio_run start` / `clio_run restart` serve the dashboard by default; an
**embedded** runtime — a unit test, an adapter, or a library user's
`CLIO_INIT` — does not, so a library user never gets an unexpected listening
socket. Setting `viz.enabled` here (or `CLIO_VIZ_ENABLE`) decides it for
both. On the command line, `--viz` / `--no-viz`, `--viz-port N`, and
`--viz-bind ADDR` do the same; naming a port or bind address implies `--viz`.
:::

A port that is already taken logs a warning and disables the dashboard; it
never fails the runtime. The dashboard is only compiled in when `Poco::Net`
is found at configure time.

---

## Compose Section

The `compose` section declaratively creates module pools at runtime startup. Each entry defines one pool.
Expand Down Expand Up @@ -868,6 +907,18 @@ engine's ranks stagger and retry their client init so they do not stampede it.
| `CLIO_INIT_ATTEMPTS` | `60` | Client-init retry attempts. |
| `CLIO_INIT_SLEEP_MS` | `3000` | **Mean** backoff between attempts; the actual sleep is uniform over `[0.5×, 1.5×]` with a per-rank seed, so same-node ranks do not all retry on the same second. Default budget: 60 × ~3 s ≈ 3 minutes. |

### Web dashboard

These override the [`viz`](#web-dashboard-viz) section.

| Variable | Default | Description |
|----------|---------|-------------|
| `CLIO_VIZ_ENABLE` | *(daemon: on, embedded: off)* | `1`/`0`. Counts as an explicit choice, so `clio_run start` will not override it. |
| `CLIO_VIZ_PORT` | `8080` | Dashboard TCP port. `0` = pick a free one. |
| `CLIO_VIZ_BIND` | `127.0.0.1` | Dashboard bind address. |
| `CLIO_VIZ_MAX_THREADS` | `16` | HTTP thread-pool size. |
| `CLIO_VIZ_PATH` | *(unset)* | `:`-separated list of viz roots (each holding one subdirectory per ChiMod) to serve pages from instead of the ones next to the loaded ChiMod libraries — edit pages against a running daemon without rebuilding. |

### Task scheduling

| Variable | Default | Description |
Expand Down
414 changes: 278 additions & 136 deletions docs/deployment/monitoring.md

Large diffs are not rendered by default.

21 changes: 21 additions & 0 deletions docs/deprecation-notes.md
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,27 @@ dynamic-loader stability. They have since been renamed too:

Downstream CMake must `find_package` / link the `clio_*` target names.

### The Python `context-visualizer` {#context-visualizer}

The standalone Flask dashboard has been removed; the runtime now serves the
dashboard itself (see [Monitoring → Runtime Dashboard](./deployment/monitoring#runtime-dashboard)).

| Then | Now |
|------|-----|
| `python -m context_visualizer [--host H] [--port P]`, `context-visualizer` console script | `clio_run start` serves it; `--viz-bind H --viz-port P` (or `viz:` in `clio.yaml`, or `CLIO_VIZ_*`) |
| `http://127.0.0.1:5000` | `http://127.0.0.1:8080` |
| `pip install iowarp-core[visualizer]`, the `flask` dependency | nothing to install — no Python involved |
| `-DCLIO_CORE_ENABLE_VISUALIZER=ON` | option removed; the dashboard builds whenever `Poco::Net` is found |
| `GET /api/node/<id>/workers`, `/api/node/<id>/system_stats`, `/api/node/<id>/bdev_stats` | `GET /api/nodes/{node}/workers`, `…/system_stats`, `…/bdevs` (`{node}` may be `local`) |
| `GET /api/system`, `/api/workers` | `GET /api/nodes/local/workers` (`/api/health` for liveness) |
| `POST /api/topology/node/<id>/shutdown` / `restart` (via SSH) | no equivalent — run `clio_run stop` / `clio_run restart` on the node |
| Pools page listing the config's `compose` section | Pools page listing the pools actually composed, with **Add Pool** and per-pool destroy |

The `context_visualizer` package, `installers/*` entries, and the pip
`[visualizer]` extra are gone; a script that still runs
`python -m context_visualizer` will fail with `No module named
context_visualizer`.

---

## Migrating a downstream project
Expand Down
176 changes: 176 additions & 0 deletions docs/getting-started/dashboard.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,176 @@
---
sidebar_position: 3
title: Dashboard
description: Open the runtime's built-in web dashboard, take a tour of its pages, and create your first pool from the browser.
---

# Dashboard

Every CLIO Runtime daemon serves a web dashboard. Start the runtime and open
[http://127.0.0.1:8080](http://127.0.0.1:8080) — there is nothing to install,
no separate process to run, and no Python involved.

This page is a tour. The full reference — configuration, the REST API, and
how to add pages for your own ChiMod — is under
[Deployment → Monitoring](../deployment/monitoring#runtime-dashboard).

## 1. Open it

Start the runtime as in the [Quick Start](./quick-start):

```bash
clio_run start &
```

The daemon logs where the dashboard is listening:

```
Viz: dashboard listening at http://127.0.0.1:8080
```

Open that address in a browser. If `:8080` is taken on your machine, pick
another port; if you are on a remote host, keep the dashboard on loopback
and tunnel to it:

```bash
# Different port
clio_run start --viz-port 9000 &

# Remote host: tunnel from your laptop, then open http://127.0.0.1:8080 locally
ssh -L 8080:127.0.0.1:8080 user@node1
```

Running in Docker? Bind the dashboard to all interfaces inside the
container and publish the port:

```yaml
services:
iowarp:
image: iowarp/deploy-cpu:latest
environment:
- CLIO_VIZ_BIND=0.0.0.0
ports:
- "9413:9413" # runtime RPC
- "8080:8080" # dashboard
command: ["clio_run", "start"]
```

:::warning No authentication
The dashboard shows runtime internals and can create and destroy pools. It
listens on loopback by default for a reason — do not bind it to a public
interface without a reverse proxy that enforces access control.
:::

:::note If nothing is listening on 8080
The dashboard is compiled into the runtime only when the build found
`Poco::Net`. If `clio_run start` logs a `Viz:` warning instead of the
green *listening* line, your build lacks it — see
[Build requirements](../deployment/monitoring#build-requirements).
:::

## 2. The tour

The navigation bar has three tabs — **Cluster**, **Pools**, **Config** — and
a connection indicator. Every page refreshes itself every couple of seconds.

### Cluster

The landing page. One card per node in the cluster with its IP, alive/dead
state, and *leader* / *this node* badges — on a laptop that is a single
card. Below it, **This node** shows live CPU and memory meters and a
workers summary: how many tasks are queued, blocked, and processed. Click a
node card (or the Workers card) to open the node page: utilization plus a
per-worker table of queue depth, blocked and periodic tasks, load, and
suspend period.

### Pools

Everything composed on this node, grouped by ChiMod. With the default
configuration you will see the DRAM block device (`clio_bdev`), the CTE
core (`clio_cte_core`) and its interposition chain (cache, indexer,
replication), the CAE, and the filesystem pool (`clio_cte_filesystem`)
that the FUSE mount drives.

Each pool is a card. Click it to open the pool's own website:

| Module | What its page shows |
|--------|---------------------|
| `clio_bdev` | One block device at a time: a capacity meter and the device's full statistics (bandwidth, latency, ops, …). |
| `clio_cte_core` | The storage-target roster — score, free space, capacity, latency, bandwidth, bytes read/written — with **register** / **unregister** buttons. |
| `clio_safe_bdev` | Array members and recovery progress, with add / replace / remove. |
| everything else | A generic pool page: identity, the scheduler's learned **task predictions** for the pool, and a box to run any of the pool's `Monitor()` queries. |

The corner **×** on a card destroys that pool (after a confirmation). The
admin pool cannot be destroyed — that is the runtime itself.

### Config

The settings the daemon **actually came up with** — after the config file,
environment variables, and CLI flags were applied — plus the route table:
every REST endpoint and static-page mount, and which ChiMod registered it.
Handy when a setting does not seem to take effect.

## 3. Create a pool from the browser

Try the **Add Pool** button on the Pools tab:

1. Pick `clio_bdev` from the module list.
2. Fill in the form — a RAM device is the quickest:
- **pool_name**: `ram::scratch`
- **pool_id**: leave the suggested free id
- **bdev_type**: `ram`
- **capacity**: `256MB`
3. Click **Validate**. Every field is checked and nothing is created; errors
come back per field.
4. Click **Create**. A new card appears under `clio_bdev`; click it to watch
the device's stats.

Modules that ship a form (bdev, safe-bdev, CTE core) get typed fields like
these. Any other module gets a **compose editor** instead — the same identity
fields plus a raw-YAML box for module parameters, so anything you can put in
`clio.yaml`'s `compose` section you can also create here.

To retire the pool, click the **×** on its card.

## 4. Watch real work

With the [FUSE mount from the Quick Start](./quick-start#3-mount-the-filesystem)
up, copy a few files into it and keep the dashboard open:

- **Cluster → This node**: the *processed* count climbs and the worker
meters move.
- **Pools → `clio_cte_core`**: the target roster's *bytes written* and
*writes* columns show the pages landing on the DRAM tier.
- **Pools → `clio_bdev`**: the block device's capacity meter and statistics
update as pages are written.

## 5. Turn it off or move it

```bash
clio_run start --no-viz # don't serve it on this node
clio_run start --viz-bind 0.0.0.0 # reachable off-box (see the warning above)
```

Or set it once in `~/.clio/clio.yaml`:

```yaml
viz:
enabled: true
port: 8080
bind: "127.0.0.1"
```

Only a daemon started with `clio_run start` serves the dashboard by default.
A program that embeds the runtime (a test, an adapter, your own
`CLIO_INIT`) does not open a port unless you set `viz.enabled` or
`CLIO_VIZ_ENABLE=1`.

## Next steps

- [Monitoring → Runtime Dashboard](../deployment/monitoring#runtime-dashboard) —
the REST API behind every page, config knobs, and the multi-node
interactive cluster
- [Web Dashboard Integration](../sdk/context-runtime/2.module_dev_guide.md#web-dashboard-integration-registerviz) —
give your own ChiMod a page and endpoints
- [Configuration → Web Dashboard](../deployment/configuration#web-dashboard-viz) —
the `viz` section and `CLIO_VIZ_*` variables
2 changes: 1 addition & 1 deletion docs/getting-started/installation.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ need:
- **HDF5** scientific data ingestion
- **ADIOS2** adapter for streaming analytics
- **Compression** backends (LibPressio, Blosc, etc.)
- **FUSE** adapter
- **FUSE** adapter on **macOS** (the Linux and Windows wheels already ship `clio_cte_fuse`)
- **Custom ChiMods** you intend to compile against the C++ headers
- **Sanitizer / debug builds** for development

Expand Down
Loading
Loading