CV-Arena

CV-Arena: An Open Benchmark for Instructional Computer Vision Problem Solving with Human-AI Collaborative Preferences

This repository contains the official code release for CV-Arena, an open benchmark for instructional computer vision problem solving evaluated under human–AI collaborative preferences. It bundles three components used to construct and evaluate the benchmark:

1. Agent — instruction-guided image-editing agent (planner + executor).
2. Score Agent — VLM-based scoring / evaluation agent that judges edits with respect to the original image and instruction.
3. Image Search — Smart Image Searching Tool used to assemble the benchmark's image pool via a cognitive-architecture-style plan / act / reflect loop.

Repository layout

opensource_code/
├── agent/                # Main agent (planner + executor)
│   └── agentic_v2_batch_2_google_offical_api.py
├── score_agent/          # Scoring / evaluation agent
│   ├── batch_process_all_models.py
│   ├── batch_process_v1.py
│   ├── analyze_results.py
│   └── ... (helpers, tests, docs)
└── image_search/         # Smart Image Searching Tool
    ├── image_tool.py
    ├── download_images.py
    ├── fix_corrupted_images.py
    ├── keywords.json
    ├── agentic_image_search/   # cognitive architecture (plan / act / reflect)
    └── scripts/                # run / status / stop helpers

Each component has its own README with detailed usage:

• agent/ — see the docstrings in agentic_v2_batch_2_google_offical_api.py.
• score_agent/README.md, score_agent/README_ALL_MODELS.md
• image_search/README.md, image_search/agentic_image_search/README.md

Installation

We recommend a Python ≥ 3.10 virtual environment per component.

# Example: image_search
cd image_search
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

agent/ and score_agent/ rely on standard libraries (openai, google-generativeai, tqdm, Pillow, etc.); install them in the same environment if you intend to run those components together.

API keys

All API keys are read from environment variables — there are no hard-coded credentials in the code. Set the following before running:

Variable	Used by
OPENAI_API_KEY	score_agent/*
GEMINI_API_KEY	agent/agentic_v2_batch_2_google_offical_api.py
FORGE_API_KEY	agent/agentic_v2_batch_2_google_offical_api.py (OpenAI-compatible endpoint)
GOOGLE_AI_API_KEY	image_search/agentic_image_search/agent.py
GOOGLE_CUSTOM_API_KEY, GOOGLE_CX	image_search/agentic_image_search/agent.py (Google Custom Search)

A template .env lives at image_search/.env.example. Copy it to image_search/.env (and analogous files for the other components) and fill in your keys.

Dataset

The CV-Arena benchmark dataset (data_with_prompt/, Expert_mode_results/, Expert_mode_results_socres/, etc.) is not included in this repository. Dataset release: TBA — download instructions, hosting URL, and license will be announced in a future update.

Once the dataset is available locally, replace the placeholder <PROJECT_ROOT> in the scripts below with the absolute path to your local checkout of the data:

• agent/agentic_v2_batch_2_google_offical_api.py (DATA_WITH_PROMPT_ROOT, OUTPUT_ROOT)
• score_agent/batch_process_all_models.py, score_agent/batch_process_v1.py
• image_search/agentic_image_search/agent.py

Quick start

After exporting your API keys and pointing the path constants at your local data, each component is a self-contained CLI:

# Score agent — process the first task as a smoke test
cd score_agent
python batch_process_v1.py --test

# Image search — populate the candidate image pool
cd image_search
python image_tool.py

See each component's README for the full set of flags.

Authors

Fangzhou Lin, Peiran Li, Lingyu Xu, Wenjing Chen, Qianwen Ge, Shuo Xing, Mingyang Wu, Xiangbo Gao, Siyuan Yang, Kazunori Yamada, Ziming Zhang, Haichong Zhang, Zhen Dong, Ming-Hsuan Yang, Zhengzhong Tu.

Citation

If you use CV-Arena in your research, please cite:

@article{lin2026cv,
  title={CV-Arena: An Open Benchmark for Instructional Computer Vision Problem Solving with Human-AI Collaborative Preferences},
  author={Lin, Fangzhou and Li, Peiran and Xu, Lingyu and Chen, Wenjing and Ge, Qianwen and Xing, Shuo and Wu, Mingyang and Gao, Xiangbo and Yang, Siyuan and Yamada, Kazunori and others},
  journal={arXiv preprint arXiv:2606.00931},
  year={2026}
}

License

This project is released under the MIT License.