Refactor humaneval_infilling.py to load multiple subsets from the dataset and remove TODO comment

[Ki-Seki]

No description provided.

[Copilot]

Pull request overview

Refactors the HumanEval infilling benchmark entrypoint to load and evaluate multiple dataset subsets instead of a single split, and removes an outdated TODO about the dataset repo.

Changes:

• Load four dataset subsets (MultiLine, RandomSpan, RandomSpanLight, SingleLine) and concatenate them into a single dataset.
• Shuffle the concatenated dataset with the configured seed before evaluation.
• Remove the TODO comment about needing dataset repo repairs.

Comments suppressed due to low confidence (1)

src/gimbench/code/humaneval_infilling.py:28

• This script hard-codes split="test" while other dataset entrypoints store the split in args.dataset["split"] and pass it through to load_dataset. For consistency and configurability (and to avoid hidden behavioral changes if the desired split differs), add a split key to args.dataset and use it when loading each subset.

    args.dataset = {
        "path": "Sculpt-AI/humaneval_infilling",
        "subsets": ["MultiLine", "RandomSpan", "RandomSpanLight", "SingleLine"],
    }

    ds = concatenate_datasets(
        [load_dataset(args.dataset["path"], split="test", name=subset) for subset in args.dataset["subsets"]]
    ).shuffle(seed=args.seed)
    logger.info(f"Loaded {len(ds)} samples from dataset {args.dataset}")
    logger.info(f"Columns: {ds.column_names}")
    logger.info(f"First sample: {ds[0]}")

    conduct_eval(args, ds)

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.