Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -43,3 +43,6 @@ frontend/.playwright/
.idea/
.vscode/
*.swp

# Local git worktrees
.worktrees/
13 changes: 8 additions & 5 deletions demo/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,11 +69,12 @@ in the video anyway.
than falling back to an older image if a capture is missing.

Before a render, run the layout check as well, because it is the only way to see a panel
that overflows the frame without waiting half an hour for the video:
that overflows the frame or lies across another block without waiting half an hour for
the video:

```bash
npx videowright dev --port 5199 # in one shell
npm run layout # in another; exits non-zero on overflow
npm run layout # in another; exits non-zero on overflow or overlap
```

## Voice
Expand Down Expand Up @@ -108,9 +109,11 @@ still refuses to run if the recording and the written script disagree. The finis
track did **not** need either: it is one uninterrupted take. See `docs/narration.md`.

`scripts/check_scene_layout.mjs` plays the timeline in the dev server and measures the
live DOM, reporting any settled element that crosses an edge of the 1920×1080 frame.
It is worth running before a render, which takes about half an hour and is the only
other way to see a panel that overflows.
live DOM with every reveal visible. It reports any element that crosses an edge of the
1920×1080 frame, any stage whose content is taller than its row, and any two blocks that
cover the same pixels. The last two matter because a centred grid with too much content
stays inside the frame and lies across the headline above it instead. It is worth running
before a render, which takes about half an hour and is the only other way to see either.

## Screenshots

Expand Down
1 change: 0 additions & 1 deletion demo/docs/narration.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,6 @@ Each factual claim in the narration is traceable to a measurement:
| wrong-value escape rate under 1% | 0.9% (1 of 112), same document |
| better than 98% indicator precision and recall | 0.9861 / 0.9853, same document |
| about 22 ms per frame | the running application's reported per-observation latency (13–34 ms across runs) |
| AWS integration is not implemented | `GET /api/system/status` → `aws_integration: "NOT IMPLEMENTED"` |

Two claims were deliberately removed during writing because the measurement did not
support them: that the first frame was too blurred to measure (its blur score is `1.0`
Expand Down
11 changes: 6 additions & 5 deletions demo/docs/storyboard.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,12 @@ Ten scenes, 1920×1080 at 60 fps, 4:41 total. Scene order and durations match
| 2 | `problem` | 0:20 | 26.9 s | the home troubleshooting workspace as a real screenshot; four kinds of evidence it extracts | the dishwasher phone call |
| 3 | `vision` | 0:47 | 37.7 s | the real Measurement detail panel; the pipeline table; then perspective correction and the provenance rule | every measurement is real CV, not a VLM impression |
| 4 | `uncertainty` | 1:24 | 26.0 s | the angled, glaring first frame; gauge 83.7 PSI at 0.57 confidence; the quality gate's own reason string | it reports a number it does not trust |
| 5 | `active` | 1:50 | 28.2 s | the loop bar moving Reason → Investigate; all seven tool calls; then the verbatim request block and `WAITING_FOR_USER` | it asks for a better look |
| 6 | `remeasure` | 2:18 | 28.3 s | the clean second frame; gauge 87.2 PSI at 0.96, `HIGH`; the annotated evidence; the observation diff | the second look produces the measurement |
| 5 | `active` | 1:50 | 28.2 s | the loop bar moving Reason → Investigate; the names of all seven tool calls; then the reason and the verbatim request block with `WAITING_FOR_USER` | it asks for a better look |
| 6 | `remeasure` | 2:18 | 28.3 s | the clean second frame; gauge 87.2 PSI at 0.96, `HIGH`; the observation diff under the frame; then the conclusion | the second look produces the measurement |
| 7 | `diagnosis` | 2:47 | 25.1 s | the annotated panel; recorded findings; the auditability list | incident raised, every claim traceable |
| 8 | `approval` | 3:12 | 30.8 s | the incident and approval panel; `SIMULATED ACTION — REQUIRES APPROVAL`; what it will not do | it stops for a person |
| 9 | `evaluation` | 3:43 | 35.3 s | 0.47 PSI MAE, 112 published / 24 refused, 98.6% precision, 22 ms; honest failure modes; the state strip | measured, not claimed |
| 10 | `outro` | 4:18 | 23.6 s | logo, the stack, the closing promise, the AWS not-implemented line | See. Diagnose. Act. |
| 10 | `outro` | 4:18 | 23.6 s | logo, the stack, the closing promise | See. Diagnose. Act. |

## Narrative arc

Expand All @@ -35,8 +35,9 @@ replace claims with measurements, and close.
- **Real data only.** Every number on screen comes from a run recorded in
`docs/evaluation/results.md`, from the demo trace, or from the running app's own
per-observation latency. Nothing is illustrative.
- **Simulation is labelled.** The approval scene says the action is simulated and the
outro repeats that AWS integration is not implemented.
- **Simulation is labelled.** The approval scene says the action is simulated.
- **Each scene fits its frame.** A scene holds only as much as 1080 lines can show with
every reveal visible. `scripts/check_scene_layout.mjs` fails if two blocks overlap.
- **Screenshots, not mock-ups.** The framed images are the real application, captured
by `scripts/capture_app_screens.mjs`.
- **One idea per scene.** Reveals are cued to the narration by word timing, so a panel
Expand Down
Binary file modified demo/videos/sightops-demo.mp4
Binary file not shown.
24 changes: 17 additions & 7 deletions demo/videowright/components/scene.css
Original file line number Diff line number Diff line change
Expand Up @@ -129,11 +129,6 @@
align-items: center;
}

.so-scene .stage.narrow-wide {
grid-template-columns: 0.72fr 1.28fr;
align-items: center;
}

.so-scene .stage.wide-narrow {
grid-template-columns: 1.3fr 0.7fr;
align-items: center;
Expand Down Expand Up @@ -433,8 +428,13 @@
gap: 16px;
}

.so-scene .row.spaced {
margin-top: 18px;
.so-scene .row.between {
justify-content: space-between;
margin-bottom: 10px;
}

.so-scene .row.between .from {
margin-bottom: 0;
}

/* ---------- title and closing scenes ---------- */
Expand Down Expand Up @@ -522,7 +522,9 @@
font-size: 190px;
}

/* The wordmark has a line height of 1, so its descenders reach below its box. */
.so-scene.centered .lede {
margin-top: 18px;
font-size: 50px;
}

Expand Down Expand Up @@ -599,6 +601,14 @@
color: var(--color-steel);
}

/* Fills down the first column, then the second, so the order still reads top to bottom. */
.so-scene .mono-list.columns {
display: grid;
grid-template-rows: repeat(4, auto);
grid-auto-flow: column;
gap: 12px 40px;
}

/* ---------- state machine strip ---------- */

.so-scene .states {
Expand Down
59 changes: 50 additions & 9 deletions demo/videowright/scripts/check_scene_layout.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,17 @@
* for each segment it navigates straight to it, makes every reveal visible, and
* measures the live DOM.
*
* Two checks per segment:
* Four checks per segment:
* - the scene's own content must not be taller than the scene box
* (`scrollHeight` vs `clientHeight`), which catches a grid whose content is
* centred past the top and bottom edges at once;
* - every element must sit inside the scene's box, which catches a panel that
* overlaps the headline even when the scene itself is not scrolling.
* (`scrollHeight` vs `clientHeight`);
* - every element must sit inside the scene's box;
* - every element in a `.stage` must sit inside that stage's box, because content
* that leaves its row uses up the gap to the blocks above and below it;
* - no two elements may cover the same pixels unless one contains the other. This
* is the check that catches a centred grid with more content than its row: the
* grid stays inside the frame, and its ends lie across the headline above it and
* the alert below it. Two `.loop` bars in one `.loop-stack` are exempt, because
* they share a cell on purpose.
*
* Measurements are taken against the scene's own bounding box, not the viewport,
* because in the dev player the scene sits below a header. That offset does not exist
Expand All @@ -20,7 +25,7 @@
* npx videowright dev --port 5199
* node scripts/check_scene_layout.mjs
*
* Exits non-zero if any segment overflows, so it can gate a render.
* Exits non-zero if any segment overflows or overlaps, so it can gate a render.
*/

import { chromium } from "playwright";
Expand Down Expand Up @@ -89,12 +94,48 @@ for (const segment of SEGMENTS) {
overflow.push({ element: label(element), past: Math.round(past) });
}
}
for (const stage of scene.querySelectorAll(".stage")) {
const row = stage.getBoundingClientRect();
let past = 0;
for (const element of stage.querySelectorAll("*")) {
const rect = element.getBoundingClientRect();
if (rect.width < 2 || rect.height < 2) continue;
past = Math.max(past, rect.bottom - row.bottom, row.top - rect.top);
}
if (past > 1) {
overflow.push({ element: "<stage content taller than its row>", past: Math.round(past) });
}
}
overflow.sort((a, b) => b.past - a.past);

const boxes = [...scene.querySelectorAll("*")]
.map((element) => ({ element, rect: element.getBoundingClientRect() }))
.filter(({ rect }) => rect.width >= 2 && rect.height >= 2);
const overlap = [];
for (let first = 0; first < boxes.length; first++) {
for (let second = first + 1; second < boxes.length; second++) {
const a = boxes[first];
const b = boxes[second];
if (a.element.contains(b.element) || b.element.contains(a.element)) continue;
const stack = a.element.closest(".loop-stack");
if (stack && stack === b.element.closest(".loop-stack")) continue;
const across = Math.min(a.rect.right, b.rect.right) - Math.max(a.rect.left, b.rect.left);
const down = Math.min(a.rect.bottom, b.rect.bottom) - Math.max(a.rect.top, b.rect.top);
if (across > 1 && down > 1) {
overlap.push({
element: `${label(a.element)} overlaps ${label(b.element)}`,
past: Math.round(Math.min(across, down)),
});
}
}
}
overlap.sort((a, b) => b.past - a.past);

return {
sceneOverflow: Math.round(scene.scrollHeight - scene.clientHeight),
box: [Math.round(box.width), Math.round(box.height)],
overflow: overflow.slice(0, 4),
overlap: overlap.slice(0, 4),
headline:
(scene.querySelector(".headline") ?? scene.querySelector(".wordmark"))?.textContent?.trim() ??
"(none)",
Expand All @@ -103,7 +144,7 @@ for (const segment of SEGMENTS) {

const issues = [];
if (result.sceneOverflow > 1) issues.push({ element: "<scene content taller than the frame>", past: result.sceneOverflow });
issues.push(...result.overflow);
issues.push(...result.overflow, ...result.overlap);

const status = issues.length === 0 ? "ok " : "FAIL";
console.log(`${status} ${segment.padEnd(12)} ${result.box[0]}x${result.box[1]} ${result.headline}`);
Expand All @@ -116,8 +157,8 @@ for (const segment of SEGMENTS) {
await browser.close();

if (failures.length === 0) {
console.log("\nOK: every segment fits the 1920x1080 frame");
console.log("\nOK: every segment fits the 1920x1080 frame and nothing overlaps");
process.exit(0);
}
console.log(`\n${failures.length} overflow(s) across ${new Set(failures.map((f) => f.segment)).size} segment(s)`);
console.log(`\n${failures.length} layout fault(s) across ${new Set(failures.map((f) => f.segment)).size} segment(s)`);
process.exit(1);
34 changes: 14 additions & 20 deletions demo/videowright/segments/active/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,37 +6,29 @@ export default defineScene({
voiceover:
"This is the part that matters most. The agent compares image quality and measurements across observations, decides that the gauge cannot be trusted from this angle, and asks for a better photograph. The inspection pauses in a waiting state, with a specific request rather than a vague shrug: re-image the gauge square-on, filling most of the frame, with diffuse light and no specular reflection across the dial.",
html: `
<div class="so-scene">
<div class="so-scene tight">
<div>
<div class="kicker">Active perception</div>
<div class="headline wrap">It asks for a better look instead of guessing.</div>
<div class="headline small">It asks for a better look instead of guessing.</div>
</div>

<div class="stage narrow-wide" data-beat="0">
<div class="panel">
<div class="panel-title">Loop</div>
<div class="loop-stack">
${loopBar(2, 0)}
${loopBar(3, 1)}
</div>
<div class="caption">The highlight moves from Reason to Investigate when the evidence gate trips.</div>
<div class="stage" data-beat="0">
<div class="loop-stack">
${loopBar(2, 0)}
${loopBar(3, 1)}
</div>

<div class="panel">
<div class="panel-title">Tool calls, in order</div>
<ul class="mono-list">
<li>1&nbsp; inspect_panel({ image_id })</li>
<li>2&nbsp; request_new_view({ target_region: "pressure_gauge_01", instruction, reason })</li>
<div class="panel-title">Seven tool calls, in order. Calls 3 to 7 wait for a better image.</div>
<ul class="mono-list columns">
<li>1&nbsp; inspect_panel</li>
<li>2&nbsp; request_new_view</li>
<li class="dim">3&nbsp; inspect_panel</li>
<li class="dim">4&nbsp; compare_observations</li>
<li class="dim">5&nbsp; record_diagnosis</li>
<li class="dim">6&nbsp; create_incident</li>
<li class="dim">7&nbsp; request_human_approval</li>
</ul>
<div class="caption">
Seven calls, every one of them recorded with its source. Calls 3 to 7 happen
only once a better image exists.
</div>
</div>
</div>

Expand All @@ -45,9 +37,11 @@ export default defineScene({
</div>

<div class="request" data-beat="2">
<span class="from">Request from the agent</span>
<div class="row between">
<span class="from">Request from the agent</span>
<span class="chip warn"><span class="dot"></span>WAITING_FOR_USER</span>
</div>
Re-image pressure_gauge_01 square-on, filling most of the frame, with diffuse light and no specular reflection across the dial or lamp face.
<div class="row spaced"><span class="chip warn"><span class="dot"></span>WAITING_FOR_USER</span></div>
</div>
</div>
`,
Expand Down
15 changes: 8 additions & 7 deletions demo/videowright/segments/approval/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ export default defineScene({
voiceover:
"Then SightOps stops. Consequential actions require a person, so the agent requests approval and waits for it. In this demonstration the proposed action is simulated, and the interface says so plainly. Nothing is switched, no valve is turned, and no safety interlock is ever bypassed, because a troubleshooting assistant that can reach out and touch machinery is a hazard, not a product. A human decides what happens next.",
html: `
<div class="so-scene">
<div class="so-scene tight">
<div>
<div class="kicker warn">Human approval</div>
<div class="headline small">It stops before it acts.</div>
Expand All @@ -18,6 +18,7 @@ export default defineScene({
approvalUrl,
"The incident and approval panel: the proposed simulated action, an approve button and a reject button",
0,
"capped",
)}
<div class="panel warn" data-beat="0">
<div class="panel-title">Proposed action</div>
Expand All @@ -32,12 +33,12 @@ export default defineScene({

<div class="panel" data-beat="1">
<div class="panel-title">What it will not do</div>
<ul class="mono-list">
<li>switch a relay</li>
<li>turn a valve</li>
<li>bypass an interlock</li>
<li>start a motor</li>
</ul>
<div class="row">
<span class="chip">switch a relay</span>
<span class="chip">turn a valve</span>
<span class="chip">bypass an interlock</span>
<span class="chip">start a motor</span>
</div>
<div class="caption">No safety-critical actuation path exists in the system.</div>
</div>

Expand Down
3 changes: 2 additions & 1 deletion demo/videowright/segments/diagnosis/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ export default defineScene({
voiceover:
"That measurement is recorded as a diagnosis, with its evidence attached: the region ids, the values, the confidences, and the annotated image they came from. An incident is raised and a remediation is proposed. Every claim in the trace is traceable to a number that OpenCV produced, and to an image that a person can open and check for themselves.",
html: `
<div class="so-scene">
<div class="so-scene tight">
<div>
<div class="kicker">Diagnosis</div>
<div class="headline small">Every claim carries its evidence.</div>
Expand All @@ -18,6 +18,7 @@ export default defineScene({
evidence2Url,
"The annotated industrial panel: the detected regions and the measured gauge value drawn over the original photograph",
0,
"capped",
)}
<div class="panel accent" data-beat="0">
<div class="panel-title">Recorded findings</div>
Expand Down
8 changes: 3 additions & 5 deletions demo/videowright/segments/evaluation/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,10 @@ export default defineScene({
voiceover:
"The vision pipeline is measured on a reproducible fixture set with exact ground truth, across perspective, glare, blur, noise and resolution. On the gauge, the mean absolute error is under half a PSI, and fewer than one in a hundred published readings is wrong by more than a few PSI. Indicators are classified at better than 98 percent precision and recall. A frame is analysed in about 22 milliseconds on a CPU, and the agent completes every scripted task inside its bounds.",
html: `
<div class="so-scene">
<div class="so-scene tight">
<div>
<div class="kicker">Measured, not claimed</div>
<div class="headline small wrap">Reproducible evaluation, with the numbers published.</div>
<div class="headline small">Reproducible evaluation, with the numbers published.</div>
</div>

<div class="stage">
Expand All @@ -32,9 +32,7 @@ export default defineScene({
<li><span>Display lit or blank is the weakest class, and it is stated as such</span></li>
</ul>
<div class="caption">
The error, refusal and classification figures come from backend/scripts/evaluate.py
(the full table is in docs/evaluation/results.md); the frame latency is what the
running application reports per observation.
Source: backend/scripts/evaluate.py and docs/evaluation/results.md. Latency: reported by the running app.
</div>
</div>

Expand Down
4 changes: 0 additions & 4 deletions demo/videowright/segments/outro/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -27,10 +27,6 @@ export default defineScene({
<div class="promise" data-beat="1">
The agent knows what it can observe, what remains uncertain, and what needs inspecting next.
</div>

<div class="caption" data-beat="1">
AWS integration is not implemented. No AWS resource is provisioned or benchmarked.
</div>
</div>
`,
});
Loading
Loading