feat(evals): add deterministic VQA dataset generation

[ruthwikdasyam]

Contribution path

Closes DIM-1418

Problem

DimOS needs reproducible visual-question datasets generated from recorded camera frames.

Solution

Add dimos evals vqa generate and dimos evals vqa run workflows for deterministic multiple-choice VQA datasets. Questions are constrained by family, answers come from private Moondream evidence, and outputs include lossless PNG assets plus audit metadata.

How to Test

dimos evals vqa generate go2_short.db --image-index 100

dimos evals vqa run ~/.local/state/dimos/datasets/vqa/go2_short-frames --model gpt-4o-mini

uv run --no-sync pytest dimos/evals/vqa

AI assistance

OpenCode with GPT-5.6 Sol assisted with implementation, tests, documentation, and review.

Checklist

• I have read and approved the CLA.

[codecov]

Codecov Report

❌ Patch coverage is 89.44193% with 70 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
dimos/evals/vqa/generate.py	80.00%	29 Missing and 5 partials ⚠️
dimos/evals/vqa/suite.py	67.53%	16 Missing and 9 partials ⚠️
dimos/evals/vqa/families.py	91.54%	3 Missing and 3 partials ⚠️
dimos/evals/vqa/author.py	91.30%	1 Missing and 1 partial ⚠️
dimos/evals/vqa/primitives/moondream.py	71.42%	2 Missing ⚠️
dimos/evals/vqa/cli.py	96.00%	0 Missing and 1 partial ⚠️

@@            Coverage Diff             @@
##             main    #3488      +/-   ##
==========================================
+ Coverage   74.05%   76.11%   +2.06%     
==========================================
  Files        1283     1232      -51     
  Lines      124704   119404    -5300     
  Branches    11141    10702     -439     
==========================================
- Hits        92349    90890    -1459     
+ Misses      29493    25418    -4075     
- Partials     2862     3096     +234     

Flag	Coverage Δ
OS-ubuntu-24.04-arm	70.54% <89.44%> (+0.10%)	⬆️
OS-ubuntu-latest	72.32% <89.44%> (+0.09%)	⬆️
Py-3.10	72.31% <89.44%> (+0.09%)	⬆️
Py-3.11	72.31% <89.44%> (+0.09%)	⬆️
Py-3.12	72.31% <89.44%> (+0.10%)	⬆️
Py-3.13	72.31% <89.44%> (+0.09%)	⬆️
Py-3.14	72.31% <89.44%> (+0.09%)	⬆️
Py-3.14t	72.31% <89.44%> (+0.09%)	⬆️
SelfHosted-Large	29.78% <33.18%> (+0.07%)	⬆️
SelfHosted-Linux	35.81% <33.18%> (-0.03%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
dimos/evals/cli.py	38.46% <100.00%> (+5.12%)	⬆️
dimos/evals/vqa/test_cli.py	100.00% <100.00%> (ø)
dimos/evals/vqa/test_contracts.py	100.00% <100.00%> (ø)
dimos/models/vl/base.py	49.55% <100.00%> (ø)
dimos/evals/vqa/cli.py	96.00% <96.00%> (ø)
dimos/evals/vqa/author.py	91.30% <91.30%> (ø)
dimos/evals/vqa/primitives/moondream.py	71.42% <71.42%> (ø)
dimos/evals/vqa/families.py	91.54% <91.54%> (ø)
dimos/evals/vqa/suite.py	67.53% <67.53%> (ø)
dimos/evals/vqa/generate.py	80.00% <80.00%> (ø)

... and 83 files with indirect coverage changes

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
dimensional	🟢 Ready	View Preview	Aug 17, 2026, 5:45 PM

💡 Tip: Enable Workflows to automatically generate PRs for you.