Investigate persona wording in SWE-Bench prompt template

[enyst]

Summary

In the SWE-Bench user instruction template, the first user message starts with:

I have access to a python code repository in the directory /workspace/django/ . You can explore and modify files using the available tools. Consider the following issue description:

That wording appears to switch persona in a slightly odd way: the surrounding setup already frames the agent as the actor with repository/tool access, but the user message says "I have access".

Where this seems to come from

Current source appears to be:

• benchmarks/swebench/prompts/default.j2

and the same pattern also seems to exist in a few related prompt templates.

Request

Please look into:

1. whether this persona wording is intentional or accidental
2. where exactly this phrasing originated and whether it is shared across benchmark families
3. whether it has any measurable effect on model behavior

My guess is that it probably has no material effect, but I’ve been surprised by prompt phrasing details before, so it seems worth checking.

Why this is mildly concerning

Even if harmless, this may create a subtle mismatch between:

• the agent/system prompt, which frames the model as the actor using tools
• the benchmark/user prompt, which momentarily sounds like the user is the one with tool/repo access

That could matter for certain models or prompt variants, especially when doing system-prompt experiments.

This issue was created by an AI assistant (OpenHands) on behalf of the user.

[VascoSch92]

1. Whether this persona wording is intentional or accidental

It looks accidental.
A gh search code for the exact string "I have access to a python code repository" returns hits only inside the OpenHands ecosystem and its downstream forks. The likely origin is OpenHands/codescout, from which it was copied into OpenHands/benchmarks.

2. Where the phrasing originated and whether it is shared across benchmark families

It does not come from the official SWE-Bench prompt. Upstream SWE-bench/SWE-bench (swebench/inference/make_datasets/create_instance.py) defines prompt_style_2, prompt_style_3, full_file_gen, and style-2-edits-only, all of which open with "You will be provided with a partial code base and an issue statement explaining a problem to resolve." and ask for a single git apply-able patch. There is no agent/tool persona upstream at all.

3. Whether it has any measurable effect on model behavior

Needs an A/B eval. But i suppose not so much. Which models do you think are affected by that? (or can be affected?)

[0error-ob]

The persona switch seems to sit in two different framings, depending on the question being asked.

If the question is “does this prompt change individual model behavior on this benchmark?” then an A/B eval is the right test. My guess would also be that the effect is probably small for many models, though it is worth checking.

But if the question is “are scores from this harness directly comparable to scores from upstream SWE-bench?” then single-model sensitivity is only part of the issue. Two harnesses with different prompt framings are slightly different measurement configurations, even when the underlying instances are identical.

A concrete way to phrase the risk: when readers compare “OpenHands benchmark performance” and “SWE-bench leaderboard performance” side by side, they may implicitly treat the score as a property of the model alone. But the score is really a property of the model + harness + prompt + scaffold + oracle.

So the A/B answers “does this wording matter for current users?” The comparison-validity question is slightly different: “have the non-model variables been documented or bounded enough that cross-harness score comparisons are safe?”

Even if this particular persona drift turns out to be harmless, making these prompt/scaffold differences explicit would make the published scores easier to interpret.