Skip to content

README.md

Latest commit

 

History

History
74 lines (53 loc) · 1.27 KB

README.md

File metadata and controls

74 lines (53 loc) · 1.27 KB
Raw

Twenty Questions for Code

Setup

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Or use:

bash scripts/setup_env.sh --venv
# or
bash scripts/setup_env.sh --conda

Environment Variables

Create your local env file:

cp .env.example .env

Required in .env:

  • CODEGEN_API_KEY
  • CODEGEN_MODEL

Optional:

  • CODEGEN_BASE_URL
  • CODEGEN_TESTGEN_API_KEY
  • CODEGEN_TESTGEN_BASE_URL
  • CODEGEN_TESTGEN_MODEL

Prepare Datasets

python scripts/prepare_mbpp_variants.py --output data/mbpp_variants.jsonl
python scripts/prepare_humaneval_variants.py --output data/humaneval_variants.jsonl

Run Experiments

MBPP scored on MBPP+:

python scripts/run_experiment.py --config configs/mvp_mbpp.yaml

HumanEval scored on HumanEval+:

python scripts/run_experiment.py --config configs/mvp_humaneval.yaml --enable-humanevalplus

Gemini wrapper scored on HumanEval+:

ENABLE_HUMANEVALPLUS=1 bash scripts/run_google_humaneval_oneshot.sh 20

Summarize Results

python scripts/summarize_results.py \
  --results results/latest.jsonl \
  --output-dir results

Then look at:

  • results/summary_mbppplus_pass_at_1.csv
  • results/summary_humanevalplus_pass_at_1.csv