Redesign ForgeCAD benchmark: vision-based 3D reconstruction with agentic loop

bench/.gitignore

@@ -0,0 +1,8 @@
+# Benchmark results (generated per-run, not tracked)
+results/
+
+# Temporary scoring files
+_bench_harness.forge.js
+_bench_candidate.forge.js
+_metrics_harness.forge.js
+solution.forge.js

bench/README.md

@@ -0,0 +1,92 @@
+# ForgeCAD Benchmark — Vision-Based 3D Reconstruction
+
+An AI benchmark that measures how well language models can reproduce 3D geometry from reference images.
+
+## How it works
+
+1. **Reference model** — A `.forge.js` file defines the ground-truth 3D geometry
+2. **Preparation** — Multi-angle images are rendered and reference metrics are computed
+3. **Agentic loop** — The AI sees the reference images, writes ForgeCAD code, receives spatial scores and comparison renders, and iterates
+4. **Scoring** — The candidate model is compared to the reference using:
+   - **3D IoU** (50%) — volumetric intersection-over-union (the gold standard)
+   - **Volume match** (20%) — how close the total volume is
+   - **Bounding box match** (15%) — how close the overall dimensions are
+   - **Surface area match** (15%) — how close the surface complexity is
+
+## Quick start
+
+```bash
+# 1. Prepare a challenge from any .forge.js model
+node bench/prepare.mjs examples/cup.forge.js cup --difficulty easy
+
+# 2. Score a single solution
+node bench/score.mjs bench/challenges/cup my-solution.forge.js
+
+# 3. Run the full agentic loop with an LLM
+ANTHROPIC_API_KEY=... node bench/runner.mjs \
+  --challenge bench/challenges/cup \
+  --model claude-sonnet-4-20250514 \
+  --iterations 5
+
+# 4. Run all challenges
+ANTHROPIC_API_KEY=... node bench/runner.mjs --all --model claude-sonnet-4-20250514
+```
+
+## Directory structure
+
+```
+bench/
+  prepare.mjs            # Prepare a challenge from a reference model
+  score.mjs              # Score a candidate solution against reference
+  runner.mjs             # Agentic LLM loop with vision
+  api-reference.md       # ForgeCAD API docs given to the LLM
+  challenges/            # Benchmark challenges
+    <name>/
+      reference.forge.js # Ground-truth model
+      config.json        # Challenge metadata
+      prepared/
+        metrics.json     # Reference volume, bbox, surface area
+        views/           # Multi-angle reference renders (front, right, top, iso)
+  results/               # Run results (gitignored)
+    <run-id>/
+      summary.json
+      best-solution.forge.js
+      iterations/
+        0/
+          code.forge.js
+          score.json
+          llm-response.txt
+          views/           # Candidate renders for comparison
+```
+
+## Validators
+
+The scoring system uses spatial validators that compare the candidate mesh to the reference:
+
+| Validator | Weight | What it measures |
+|-----------|--------|------------------|
+| 3D IoU | 50% | Volumetric overlap after centering both shapes |
+| Volume | 20% | Ratio of smaller/larger volume |
+| Bounding box | 15% | Dimension similarity (orientation-independent) |
+| Surface area | 15% | Surface complexity similarity |
+
+The 3D IoU is the most meaningful metric — it directly measures how much the two shapes overlap in 3D space. A perfect cube will score poorly against a perfect sphere even if they have similar volumes.
+
+Future validators will include functional tests (e.g., does this bolt actually fit through this hole).
+
+## Adding challenges
+
+```bash
+# From an existing example
+node bench/prepare.mjs examples/bottle.forge.js bottle --difficulty hard
+
+# From a custom model
+node bench/prepare.mjs path/to/my-model.forge.js my-challenge --difficulty medium
+```
+
+The `prepare` step renders reference images (requires Chrome/Puppeteer). If rendering isn't available, you can add images manually to `bench/challenges/<name>/prepared/views/`.
+
+## Providers
+
+- `--provider anthropic` (default) — uses `ANTHROPIC_API_KEY`
+- `--provider openrouter` — uses `OPENROUTER_API_KEY`, supports any model via OpenRouter

bench/api-reference.md

@@ -0,0 +1,96 @@
+# ForgeCAD API Reference (Benchmark Subset)
+
+ForgeCAD scripts are JavaScript. The API is globally available — no imports needed.
+Scripts must `return` geometry (Shape, Assembly, or SolvedAssembly).
+
+## Primitives
+
+```javascript
+box(x, y, z, center?)           // Rectangular box. center=true → centered at origin
+cylinder(height, radius, radiusTop?, segments?, center?)  // Cylinder/cone
+sphere(radius, segments?)        // Sphere at origin
+```
+
+## Booleans
+
+```javascript
+union(a, b, ...)                 // Combine shapes
+difference(a, b, ...)            // Subtract b,c,... from a
+intersection(a, b, ...)          // Keep overlap only
+shape.subtract(other)            // Shorthand for difference
+```
+
+## Transforms
+
+```javascript
+shape.translate(x, y, z)        // Move
+shape.rotate(rx, ry, rz)        // Rotate (degrees, Euler XYZ)
+shape.scale(s)                   // Uniform scale
+shape.mirror([nx, ny, nz])      // Mirror across plane through origin
+shape.color("#hex")              // Set color
+```
+
+## Measurements
+
+```javascript
+shape.boundingBox()              // { min: [x,y,z], max: [x,y,z] }
+shape.volume()                   // mm³
+shape.surfaceArea()              // mm²
+shape.isEmpty()                  // boolean
+```
+
+## Assembly
+
+```javascript
+assembly(name)                              // Create assembly
+  .addPart(name, shape, options?)           // Add a named part
+  .addRevolute(name, parent, child, opts)   // Revolute (hinge) joint
+  .addPrismatic(name, parent, child, opts)  // Prismatic (slide) joint
+  .addFixed(name, parent, child, opts)      // Fixed connection
+  .solve(state?)                            // Solve at joint values → SolvedAssembly
+  .sweepJoint(name, from, to, steps, base?) // Sweep joint, check collisions
+  .describe()                               // Get parts/joints metadata
+
+// Joint options:
+// { axis: [x,y,z], min: deg, max: deg, default: deg,
+//   frame: Transform.identity().translate(x,y,z) }
+
+// SolvedAssembly:
+solved.getPart(name)             // Get positioned part
+solved.collisionReport()         // Check for part overlaps
+solved.minClearance(a, b, len?)  // Min gap between two parts
+solved.toGroup()                 // Convert to ShapeGroup for display
+```
+
+## Gear Coupling
+
+```javascript
+const pair = lib.gearPair({
+  pinion: { module: m, teeth: n, faceWidth: w },
+  gear:   { module: m, teeth: n, faceWidth: w },
+});
+
+assembly.addGearCoupling(drivenJoint, driverJoint, { pair })
+// Auto-computes ratio from tooth counts
+```
+
+## Transforms for Joint Frames
+
+```javascript
+Transform.identity()                    // Identity transform
+Transform.identity().translate(x, y, z) // Translation
+```
+
+## Verification (optional, for self-checks)
+
+```javascript
+verify.that(label, () => condition, message?)
+verify.equal(label, actual, expected, tolerance?)
+verify.inRange(label, value, min, max)
+verify.notColliding(label, shapeA, shapeB)
+```
+
+## Coordinate System
+
+- **Z-up** right-handed: X = left/right, Y = forward/back, Z = up/down
+- All dimensions in millimeters, angles in degrees

bench/challenges/bracket/config.json

@@ -0,0 +1,36 @@
+{
+  "name": "bracket",
+  "description": "Reproduce the bracket model from reference images",
+  "difficulty": "easy",
+  "referenceModel": "reference.forge.js",
+  "preparedAt": "2026-03-30T10:13:24.811Z",
+  "metrics": {
+    "volume": 10834.77,
+    "surfaceArea": 5452,
+    "boundingBox": {
+      "min": [
+        0,
+        0,
+        0
+      ],
+      "max": [
+        40,
+        30,
+        30
+      ],
+      "size": [
+        40,
+        30,
+        30
+      ]
+    },
+    "isEmpty": false
+  },
+  "imageSize": 512,
+  "validators": [
+    "iou",
+    "volume",
+    "bbox",
+    "surfaceArea"
+  ]
+}

bench/challenges/bracket/prepared/metrics.json

@@ -0,0 +1,22 @@
+{
+  "volume": 10834.77,
+  "surfaceArea": 5452,
+  "boundingBox": {
+    "min": [
+      0,
+      0,
+      0
+    ],
+    "max": [
+      40,
+      30,
+      30
+    ],
+    "size": [
+      40,
+      30,
+      30
+    ]
+  },
+  "isEmpty": false
+}

bench/challenges/bracket/reference.forge.js

@@ -0,0 +1,25 @@
+// Reference: L-bracket with two bolt holes
+const plateW = 40;
+const plateH = 30;
+const thickness = 5;
+const holeD = 6.5; // M6 clearance
+
+// Vertical plate (XZ plane)
+const vertical = box(plateW, thickness, plateH);
+
+// Horizontal plate (XY plane)
+const horizontal = box(plateW, plateH, thickness);
+
+// Join into L-shape
+const lShape = union(vertical, horizontal);
+
+// Hole in vertical plate (through Y axis)
+const hole1 = cylinder(thickness + 2, holeD / 2)
+  .rotate(90, 0, 0)
+  .translate(plateW / 2, thickness / 2, plateH / 2);
+
+// Hole in horizontal plate (through Z axis)
+const hole2 = cylinder(thickness + 2, holeD / 2)
+  .translate(plateW / 2, plateH / 2, thickness / 2);
+
+return lShape.subtract(hole1).subtract(hole2);

bench/challenges/cup/config.json

@@ -0,0 +1,36 @@
+{
+  "name": "cup",
+  "description": "Reproduce the cup model from reference images",
+  "difficulty": "easy",
+  "referenceModel": "reference.forge.js",
+  "preparedAt": "2026-03-30T10:13:13.773Z",
+  "metrics": {
+    "volume": 54547.91,
+    "surfaceArea": 32146.2,
+    "boundingBox": {
+      "min": [
+        -35,
+        -35,
+        0
+      ],
+      "max": [
+        35,
+        35,
+        80
+      ],
+      "size": [
+        70,
+        70,
+        80
+      ]
+    },
+    "isEmpty": false
+  },
+  "imageSize": 512,
+  "validators": [
+    "iou",
+    "volume",
+    "bbox",
+    "surfaceArea"
+  ]
+}

bench/challenges/cup/prepared/metrics.json

@@ -0,0 +1,22 @@
+{
+  "volume": 54547.91,
+  "surfaceArea": 32146.2,
+  "boundingBox": {
+    "min": [
+      -35,
+      -35,
+      0
+    ],
+    "max": [
+      35,
+      35,
+      80
+    ],
+    "size": [
+      70,
+      70,
+      80
+    ]
+  },
+  "isEmpty": false
+}

bench/challenges/cup/reference.forge.js

@@ -0,0 +1,12 @@
+// Reference: Simple tapered cup with hollow interior
+const height = 80;
+const topRadius = 35;
+const bottomRadius = 25;
+const wallThickness = 3;
+const baseThickness = 5;
+
+const outer = cylinder(height, bottomRadius, topRadius);
+const inner = cylinder(height - baseThickness, bottomRadius - wallThickness, topRadius - wallThickness)
+  .translate(0, 0, baseThickness);
+
+return outer.subtract(inner);

bench/challenges/pipe-tee/config.json

@@ -0,0 +1,36 @@
+{
+  "name": "pipe-tee",
+  "description": "Reproduce the pipe-tee model from reference images",
+  "difficulty": "medium",
+  "referenceModel": "reference.forge.js",
+  "preparedAt": "2026-03-30T10:13:30.458Z",
+  "metrics": {
+    "volume": 28338.9,
+    "surfaceArea": 19643.93,
+    "boundingBox": {
+      "min": [
+        25,
+        -15,
+        -15
+      ],
+      "max": [
+        120,
+        15,
+        40
+      ],
+      "size": [
+        95,
+        30,
+        55
+      ]
+    },
+    "isEmpty": false
+  },
+  "imageSize": 512,
+  "validators": [
+    "iou",
+    "volume",
+    "bbox",
+    "surfaceArea"
+  ]
+}