diff --git a/.gitignore b/.gitignore index 9ecc43a..7ba31e5 100644 --- a/.gitignore +++ b/.gitignore @@ -43,3 +43,6 @@ frontend/.playwright/ .idea/ .vscode/ *.swp + +# Local git worktrees +.worktrees/ diff --git a/demo/README.md b/demo/README.md index 9cb4700..82da8ce 100644 --- a/demo/README.md +++ b/demo/README.md @@ -69,11 +69,12 @@ in the video anyway. than falling back to an older image if a capture is missing. Before a render, run the layout check as well, because it is the only way to see a panel -that overflows the frame without waiting half an hour for the video: +that overflows the frame or lies across another block without waiting half an hour for +the video: ```bash npx videowright dev --port 5199 # in one shell -npm run layout # in another; exits non-zero on overflow +npm run layout # in another; exits non-zero on overflow or overlap ``` ## Voice @@ -108,9 +109,11 @@ still refuses to run if the recording and the written script disagree. The finis track did **not** need either: it is one uninterrupted take. See `docs/narration.md`. `scripts/check_scene_layout.mjs` plays the timeline in the dev server and measures the -live DOM, reporting any settled element that crosses an edge of the 1920×1080 frame. -It is worth running before a render, which takes about half an hour and is the only -other way to see a panel that overflows. +live DOM with every reveal visible. It reports any element that crosses an edge of the +1920×1080 frame, any stage whose content is taller than its row, and any two blocks that +cover the same pixels. The last two matter because a centred grid with too much content +stays inside the frame and lies across the headline above it instead. It is worth running +before a render, which takes about half an hour and is the only other way to see either. ## Screenshots diff --git a/demo/docs/narration.md b/demo/docs/narration.md index 6c7407d..f82328d 100644 --- a/demo/docs/narration.md +++ b/demo/docs/narration.md @@ -88,7 +88,6 @@ Each factual claim in the narration is traceable to a measurement: | wrong-value escape rate under 1% | 0.9% (1 of 112), same document | | better than 98% indicator precision and recall | 0.9861 / 0.9853, same document | | about 22 ms per frame | the running application's reported per-observation latency (13–34 ms across runs) | -| AWS integration is not implemented | `GET /api/system/status` → `aws_integration: "NOT IMPLEMENTED"` | Two claims were deliberately removed during writing because the measurement did not support them: that the first frame was too blurred to measure (its blur score is `1.0` diff --git a/demo/docs/storyboard.md b/demo/docs/storyboard.md index 40e53a7..e387f48 100644 --- a/demo/docs/storyboard.md +++ b/demo/docs/storyboard.md @@ -10,12 +10,12 @@ Ten scenes, 1920×1080 at 60 fps, 4:41 total. Scene order and durations match | 2 | `problem` | 0:20 | 26.9 s | the home troubleshooting workspace as a real screenshot; four kinds of evidence it extracts | the dishwasher phone call | | 3 | `vision` | 0:47 | 37.7 s | the real Measurement detail panel; the pipeline table; then perspective correction and the provenance rule | every measurement is real CV, not a VLM impression | | 4 | `uncertainty` | 1:24 | 26.0 s | the angled, glaring first frame; gauge 83.7 PSI at 0.57 confidence; the quality gate's own reason string | it reports a number it does not trust | -| 5 | `active` | 1:50 | 28.2 s | the loop bar moving Reason → Investigate; all seven tool calls; then the verbatim request block and `WAITING_FOR_USER` | it asks for a better look | -| 6 | `remeasure` | 2:18 | 28.3 s | the clean second frame; gauge 87.2 PSI at 0.96, `HIGH`; the annotated evidence; the observation diff | the second look produces the measurement | +| 5 | `active` | 1:50 | 28.2 s | the loop bar moving Reason → Investigate; the names of all seven tool calls; then the reason and the verbatim request block with `WAITING_FOR_USER` | it asks for a better look | +| 6 | `remeasure` | 2:18 | 28.3 s | the clean second frame; gauge 87.2 PSI at 0.96, `HIGH`; the observation diff under the frame; then the conclusion | the second look produces the measurement | | 7 | `diagnosis` | 2:47 | 25.1 s | the annotated panel; recorded findings; the auditability list | incident raised, every claim traceable | | 8 | `approval` | 3:12 | 30.8 s | the incident and approval panel; `SIMULATED ACTION — REQUIRES APPROVAL`; what it will not do | it stops for a person | | 9 | `evaluation` | 3:43 | 35.3 s | 0.47 PSI MAE, 112 published / 24 refused, 98.6% precision, 22 ms; honest failure modes; the state strip | measured, not claimed | -| 10 | `outro` | 4:18 | 23.6 s | logo, the stack, the closing promise, the AWS not-implemented line | See. Diagnose. Act. | +| 10 | `outro` | 4:18 | 23.6 s | logo, the stack, the closing promise | See. Diagnose. Act. | ## Narrative arc @@ -35,8 +35,9 @@ replace claims with measurements, and close. - **Real data only.** Every number on screen comes from a run recorded in `docs/evaluation/results.md`, from the demo trace, or from the running app's own per-observation latency. Nothing is illustrative. -- **Simulation is labelled.** The approval scene says the action is simulated and the - outro repeats that AWS integration is not implemented. +- **Simulation is labelled.** The approval scene says the action is simulated. +- **Each scene fits its frame.** A scene holds only as much as 1080 lines can show with + every reveal visible. `scripts/check_scene_layout.mjs` fails if two blocks overlap. - **Screenshots, not mock-ups.** The framed images are the real application, captured by `scripts/capture_app_screens.mjs`. - **One idea per scene.** Reveals are cued to the narration by word timing, so a panel diff --git a/demo/videos/sightops-demo.mp4 b/demo/videos/sightops-demo.mp4 index c39191c..cfd816b 100644 Binary files a/demo/videos/sightops-demo.mp4 and b/demo/videos/sightops-demo.mp4 differ diff --git a/demo/videowright/components/scene.css b/demo/videowright/components/scene.css index 853e649..26676f5 100644 --- a/demo/videowright/components/scene.css +++ b/demo/videowright/components/scene.css @@ -129,11 +129,6 @@ align-items: center; } -.so-scene .stage.narrow-wide { - grid-template-columns: 0.72fr 1.28fr; - align-items: center; -} - .so-scene .stage.wide-narrow { grid-template-columns: 1.3fr 0.7fr; align-items: center; @@ -433,8 +428,13 @@ gap: 16px; } -.so-scene .row.spaced { - margin-top: 18px; +.so-scene .row.between { + justify-content: space-between; + margin-bottom: 10px; +} + +.so-scene .row.between .from { + margin-bottom: 0; } /* ---------- title and closing scenes ---------- */ @@ -522,7 +522,9 @@ font-size: 190px; } +/* The wordmark has a line height of 1, so its descenders reach below its box. */ .so-scene.centered .lede { + margin-top: 18px; font-size: 50px; } @@ -599,6 +601,14 @@ color: var(--color-steel); } +/* Fills down the first column, then the second, so the order still reads top to bottom. */ +.so-scene .mono-list.columns { + display: grid; + grid-template-rows: repeat(4, auto); + grid-auto-flow: column; + gap: 12px 40px; +} + /* ---------- state machine strip ---------- */ .so-scene .states { diff --git a/demo/videowright/scripts/check_scene_layout.mjs b/demo/videowright/scripts/check_scene_layout.mjs index a035818..2a2823e 100644 --- a/demo/videowright/scripts/check_scene_layout.mjs +++ b/demo/videowright/scripts/check_scene_layout.mjs @@ -6,12 +6,17 @@ * for each segment it navigates straight to it, makes every reveal visible, and * measures the live DOM. * - * Two checks per segment: + * Four checks per segment: * - the scene's own content must not be taller than the scene box - * (`scrollHeight` vs `clientHeight`), which catches a grid whose content is - * centred past the top and bottom edges at once; - * - every element must sit inside the scene's box, which catches a panel that - * overlaps the headline even when the scene itself is not scrolling. + * (`scrollHeight` vs `clientHeight`); + * - every element must sit inside the scene's box; + * - every element in a `.stage` must sit inside that stage's box, because content + * that leaves its row uses up the gap to the blocks above and below it; + * - no two elements may cover the same pixels unless one contains the other. This + * is the check that catches a centred grid with more content than its row: the + * grid stays inside the frame, and its ends lie across the headline above it and + * the alert below it. Two `.loop` bars in one `.loop-stack` are exempt, because + * they share a cell on purpose. * * Measurements are taken against the scene's own bounding box, not the viewport, * because in the dev player the scene sits below a header. That offset does not exist @@ -20,7 +25,7 @@ * npx videowright dev --port 5199 * node scripts/check_scene_layout.mjs * - * Exits non-zero if any segment overflows, so it can gate a render. + * Exits non-zero if any segment overflows or overlaps, so it can gate a render. */ import { chromium } from "playwright"; @@ -89,12 +94,48 @@ for (const segment of SEGMENTS) { overflow.push({ element: label(element), past: Math.round(past) }); } } + for (const stage of scene.querySelectorAll(".stage")) { + const row = stage.getBoundingClientRect(); + let past = 0; + for (const element of stage.querySelectorAll("*")) { + const rect = element.getBoundingClientRect(); + if (rect.width < 2 || rect.height < 2) continue; + past = Math.max(past, rect.bottom - row.bottom, row.top - rect.top); + } + if (past > 1) { + overflow.push({ element: "", past: Math.round(past) }); + } + } overflow.sort((a, b) => b.past - a.past); + const boxes = [...scene.querySelectorAll("*")] + .map((element) => ({ element, rect: element.getBoundingClientRect() })) + .filter(({ rect }) => rect.width >= 2 && rect.height >= 2); + const overlap = []; + for (let first = 0; first < boxes.length; first++) { + for (let second = first + 1; second < boxes.length; second++) { + const a = boxes[first]; + const b = boxes[second]; + if (a.element.contains(b.element) || b.element.contains(a.element)) continue; + const stack = a.element.closest(".loop-stack"); + if (stack && stack === b.element.closest(".loop-stack")) continue; + const across = Math.min(a.rect.right, b.rect.right) - Math.max(a.rect.left, b.rect.left); + const down = Math.min(a.rect.bottom, b.rect.bottom) - Math.max(a.rect.top, b.rect.top); + if (across > 1 && down > 1) { + overlap.push({ + element: `${label(a.element)} overlaps ${label(b.element)}`, + past: Math.round(Math.min(across, down)), + }); + } + } + } + overlap.sort((a, b) => b.past - a.past); + return { sceneOverflow: Math.round(scene.scrollHeight - scene.clientHeight), box: [Math.round(box.width), Math.round(box.height)], overflow: overflow.slice(0, 4), + overlap: overlap.slice(0, 4), headline: (scene.querySelector(".headline") ?? scene.querySelector(".wordmark"))?.textContent?.trim() ?? "(none)", @@ -103,7 +144,7 @@ for (const segment of SEGMENTS) { const issues = []; if (result.sceneOverflow > 1) issues.push({ element: "", past: result.sceneOverflow }); - issues.push(...result.overflow); + issues.push(...result.overflow, ...result.overlap); const status = issues.length === 0 ? "ok " : "FAIL"; console.log(`${status} ${segment.padEnd(12)} ${result.box[0]}x${result.box[1]} ${result.headline}`); @@ -116,8 +157,8 @@ for (const segment of SEGMENTS) { await browser.close(); if (failures.length === 0) { - console.log("\nOK: every segment fits the 1920x1080 frame"); + console.log("\nOK: every segment fits the 1920x1080 frame and nothing overlaps"); process.exit(0); } -console.log(`\n${failures.length} overflow(s) across ${new Set(failures.map((f) => f.segment)).size} segment(s)`); +console.log(`\n${failures.length} layout fault(s) across ${new Set(failures.map((f) => f.segment)).size} segment(s)`); process.exit(1); diff --git a/demo/videowright/segments/active/index.ts b/demo/videowright/segments/active/index.ts index c0d69fc..76481fa 100644 --- a/demo/videowright/segments/active/index.ts +++ b/demo/videowright/segments/active/index.ts @@ -6,37 +6,29 @@ export default defineScene({ voiceover: "This is the part that matters most. The agent compares image quality and measurements across observations, decides that the gauge cannot be trusted from this angle, and asks for a better photograph. The inspection pauses in a waiting state, with a specific request rather than a vague shrug: re-image the gauge square-on, filling most of the frame, with diffuse light and no specular reflection across the dial.", html: ` -
+
Active perception
-
It asks for a better look instead of guessing.
+
It asks for a better look instead of guessing.
-
-
-
Loop
-
- ${loopBar(2, 0)} - ${loopBar(3, 1)} -
-
The highlight moves from Reason to Investigate when the evidence gate trips.
+
+
+ ${loopBar(2, 0)} + ${loopBar(3, 1)}
-
Tool calls, in order
-
    -
  • 1  inspect_panel({ image_id })
    • -
  • 2  request_new_view({ target_region: "pressure_gauge_01", instruction, reason })
    • +
    Seven tool calls, in order. Calls 3 to 7 wait for a better image.
    +
      +
    • 1  inspect_panel
      • +
    • 2  request_new_view
    • 3  inspect_panel
    • 4  compare_observations
    • 5  record_diagnosis
    • 6  create_incident
    • 7  request_human_approval
    -
    - Seven calls, every one of them recorded with its source. Calls 3 to 7 happen - only once a better image exists. -
@@ -45,9 +37,11 @@ export default defineScene({
- Request from the agent +
+ Request from the agent + WAITING_FOR_USER +
Re-image pressure_gauge_01 square-on, filling most of the frame, with diffuse light and no specular reflection across the dial or lamp face. -
WAITING_FOR_USER
`, diff --git a/demo/videowright/segments/approval/index.ts b/demo/videowright/segments/approval/index.ts index 664eb86..54699d4 100644 --- a/demo/videowright/segments/approval/index.ts +++ b/demo/videowright/segments/approval/index.ts @@ -7,7 +7,7 @@ export default defineScene({ voiceover: "Then SightOps stops. Consequential actions require a person, so the agent requests approval and waits for it. In this demonstration the proposed action is simulated, and the interface says so plainly. Nothing is switched, no valve is turned, and no safety interlock is ever bypassed, because a troubleshooting assistant that can reach out and touch machinery is a hazard, not a product. A human decides what happens next.", html: ` -
+
Human approval
It stops before it acts.
@@ -18,6 +18,7 @@ export default defineScene({ approvalUrl, "The incident and approval panel: the proposed simulated action, an approve button and a reject button", 0, + "capped", )}
Proposed action
@@ -32,12 +33,12 @@ export default defineScene({
What it will not do
-
    -
  • switch a relay
    • -
  • turn a valve
    • -
  • bypass an interlock
    • -
  • start a motor
    • -
+
+ switch a relay + turn a valve + bypass an interlock + start a motor +
No safety-critical actuation path exists in the system.
diff --git a/demo/videowright/segments/diagnosis/index.ts b/demo/videowright/segments/diagnosis/index.ts index e96d4a3..29fdb54 100644 --- a/demo/videowright/segments/diagnosis/index.ts +++ b/demo/videowright/segments/diagnosis/index.ts @@ -7,7 +7,7 @@ export default defineScene({ voiceover: "That measurement is recorded as a diagnosis, with its evidence attached: the region ids, the values, the confidences, and the annotated image they came from. An incident is raised and a remediation is proposed. Every claim in the trace is traceable to a number that OpenCV produced, and to an image that a person can open and check for themselves.", html: ` -
+
Diagnosis
Every claim carries its evidence.
@@ -18,6 +18,7 @@ export default defineScene({ evidence2Url, "The annotated industrial panel: the detected regions and the measured gauge value drawn over the original photograph", 0, + "capped", )}
Recorded findings
diff --git a/demo/videowright/segments/evaluation/index.ts b/demo/videowright/segments/evaluation/index.ts index fad5bf9..1571906 100644 --- a/demo/videowright/segments/evaluation/index.ts +++ b/demo/videowright/segments/evaluation/index.ts @@ -6,10 +6,10 @@ export default defineScene({ voiceover: "The vision pipeline is measured on a reproducible fixture set with exact ground truth, across perspective, glare, blur, noise and resolution. On the gauge, the mean absolute error is under half a PSI, and fewer than one in a hundred published readings is wrong by more than a few PSI. Indicators are classified at better than 98 percent precision and recall. A frame is analysed in about 22 milliseconds on a CPU, and the agent completes every scripted task inside its bounds.", html: ` -
+
Measured, not claimed
-
Reproducible evaluation, with the numbers published.
+
Reproducible evaluation, with the numbers published.
@@ -32,9 +32,7 @@ export default defineScene({
  • Display lit or blank is the weakest class, and it is stated as such
    • - The error, refusal and classification figures come from backend/scripts/evaluate.py - (the full table is in docs/evaluation/results.md); the frame latency is what the - running application reports per observation. + Source: backend/scripts/evaluate.py and docs/evaluation/results.md. Latency: reported by the running app.
    diff --git a/demo/videowright/segments/outro/index.ts b/demo/videowright/segments/outro/index.ts index 7be764a..3f27f2f 100644 --- a/demo/videowright/segments/outro/index.ts +++ b/demo/videowright/segments/outro/index.ts @@ -27,10 +27,6 @@ export default defineScene({
    The agent knows what it can observe, what remains uncertain, and what needs inspecting next.
    - -
    - AWS integration is not implemented. No AWS resource is provisioned or benchmarked. -
    `, }); diff --git a/demo/videowright/segments/remeasure/index.ts b/demo/videowright/segments/remeasure/index.ts index d18d4d8..744d329 100644 --- a/demo/videowright/segments/remeasure/index.ts +++ b/demo/videowright/segments/remeasure/index.ts @@ -10,16 +10,27 @@ export default defineScene({
    A better view
    -
    The second look is what produces the measurement.
    +
    The second look is what produces the measurement.
    - ${shot( - observation2Url, - "The SightOps workspace showing the second industrial observation: the same pump-station panel re-imaged square-on with diffuse light, and a measured gauge reading of 87.2 PSI at 0.96 confidence.", - 0, - "capped", - )} +
    + ${shot( + observation2Url, + "The SightOps workspace showing the second industrial observation: the same pump-station panel re-imaged square-on with diffuse light, and a measured gauge reading of 87.2 PSI at 0.96 confidence.", + 0, + "capped", + )} + +
    +
    compare_observations
    +
      +
    • The frame no longer needs a better view
      • +
    • Gauge confidence rose from 0.57 to 0.96
      • +
    • Measured pressure 83.7 to 87.2 PSI
      • +
    +
    +
    @@ -34,21 +45,11 @@ export default defineScene({
    -
    -
    compare_observations
    -
      -
    • The frame no longer needs a better view
      • -
    • Gauge confidence rose from 0.57 to 0.96
      • -
    • Measured pressure 83.7 to 87.2 PSI
      • -
    -
    Read from the two measurement sets, not described by a model.
    +
    + The earlier uncertainty is replaced by a measurement.
    - -
    - Observation 1: 83.7 PSI at 0.57 confidence, needing a better view. Observation 2: 87.2 PSI at 0.96. -
    `, }); diff --git a/demo/videowright/segments/vision/index.ts b/demo/videowright/segments/vision/index.ts index 9b89b1e..55ea503 100644 --- a/demo/videowright/segments/vision/index.ts +++ b/demo/videowright/segments/vision/index.ts @@ -31,11 +31,10 @@ export default defineScene({
    Measured, in the pipeline
    - ${fact("Gauge", "radial scan + shape validation")} + ${fact("Gauge", "radial scan")} ${fact("Lamp", "HSV segmentation")} ${fact("Switch", "PCA lever angle")} ${fact("Display", "stroke level")} - ${fact("Provenance", "measured", "ok")}
    @@ -44,7 +43,7 @@ export default defineScene({
    ${checkIcon} - OpenCV produces the number. The VLM's answer is labelled inferred and never overwrites it. + A VLM answer is labelled inferred. It never overwrites a measured value.
    diff --git a/docs/PROGRESS.md b/docs/PROGRESS.md index 1cd9725..4c02712 100644 --- a/docs/PROGRESS.md +++ b/docs/PROGRESS.md @@ -219,7 +219,9 @@ Latest verified runs (2026-10-09): - **Demo video** — `sync_audio.py` word check passes; video length **4:41** (16,885 frames at 60 fps, 281.42 s); 61 subtitle cues ending at 04:41.076; full decode clean; audio measured at -17.0 dB mean and -1.0 dB peak, so nothing clips. The layout checker - reports all ten segments `1920x1080 ok`. + reports all ten segments `1920x1080 ok`. That check only looked at the frame edges, and + five scenes had blocks that overlapped inside the frame; the checker now tests overlap + too, and the scenes were reduced until all ten pass. - **Live voice check** — on the running application, the default voice returns 200 `audio/mpeg` (65,663 bytes, 4.08 s) with no substitution, and requesting the library voice returns 200 with the same audio and `X-SightOps-Voice-Substituted: true`.