Hi team, thanks for maintaining this benchmarks repo. it's been very helpful.
I noticed what looks like a small bug in the SWE-Bench default prompt and wanted to flag it in case it's worth fixing.
Location
https://github.com/OpenHands/benchmarks/blob/main/benchmarks/swebench/prompts/default.j2#L22-L23
The relevant lines read:
2.1 Activate the environment by running
./opt/miniconda3/etc/profile.d/conda.sh ; conda activate testbed
Issue
The path ./opt/miniconda3/... is a relative path (dot-slash), which bash resolves against the current working directory. At the time this instruction is given to the agent, the cwd is {{ instance.repo_path }}, which resolves to /workspace/[repo_name]/ (set in run_infer.py, where /testbed/. is copied into /workspace/[repo_name]/).
However, the SWE-Bench base image installs miniconda at the absolute path /opt/miniconda3/ (see https://github.com/SWE-bench/SWE-bench/blob/main/swebench/harness/dockerfiles/python.py#L25), not under the repo workspace. So running ./opt/miniconda3/etc/profile.d/conda.sh from /workspace/[repo_name]/ fails because that path doesn't exist there.
It looks like the intent may have been dot-source syntax (dot followed by a space and then /opt/...), but as written it reads as a relative-path execution.
Why it probably hasn't surfaced
The SWE-Bench base image's .bashrc already auto-activates the testbed conda env on shell startup (see https://github.com/SWE-bench/SWE-bench/blob/main/swebench/harness/dockerfiles/python.py#L45), so the env is active regardless of whether the agent successfully runs this line. The bug is effectively silent — but it may still mislead the agent or cause confusion when the command errors out.
Suggested fix
Change the line to one of the following:
source /opt/miniconda3/etc/profile.d/conda.sh ; conda activate testbed
or
. /opt/miniconda3/etc/profile.d/conda.sh ; conda activate testbed
Happy to open a PR if that would be helpful. Thanks again!
Hi team, thanks for maintaining this benchmarks repo. it's been very helpful.
I noticed what looks like a small bug in the SWE-Bench default prompt and wanted to flag it in case it's worth fixing.
Location
https://github.com/OpenHands/benchmarks/blob/main/benchmarks/swebench/prompts/default.j2#L22-L23
The relevant lines read:
Issue
The path ./opt/miniconda3/... is a relative path (dot-slash), which bash resolves against the current working directory. At the time this instruction is given to the agent, the cwd is {{ instance.repo_path }}, which resolves to /workspace/[repo_name]/ (set in run_infer.py, where /testbed/. is copied into /workspace/[repo_name]/).
However, the SWE-Bench base image installs miniconda at the absolute path /opt/miniconda3/ (see https://github.com/SWE-bench/SWE-bench/blob/main/swebench/harness/dockerfiles/python.py#L25), not under the repo workspace. So running ./opt/miniconda3/etc/profile.d/conda.sh from /workspace/[repo_name]/ fails because that path doesn't exist there.
It looks like the intent may have been dot-source syntax (dot followed by a space and then /opt/...), but as written it reads as a relative-path execution.
Why it probably hasn't surfaced
The SWE-Bench base image's .bashrc already auto-activates the testbed conda env on shell startup (see https://github.com/SWE-bench/SWE-bench/blob/main/swebench/harness/dockerfiles/python.py#L45), so the env is active regardless of whether the agent successfully runs this line. The bug is effectively silent — but it may still mislead the agent or cause confusion when the command errors out.
Suggested fix
Change the line to one of the following:
or
Happy to open a PR if that would be helpful. Thanks again!