Skip to content

fix(meta_agent): wire eval_path and iterations_left into agent instruction#32

Open
j-arndt wants to merge 1 commit into
facebookresearch:mainfrom
j-arndt:fix/meta-agent-wire-unused-params
Open

fix(meta_agent): wire eval_path and iterations_left into agent instruction#32
j-arndt wants to merge 1 commit into
facebookresearch:mainfrom
j-arndt:fix/meta-agent-wire-unused-params

Conversation

@j-arndt
Copy link
Copy Markdown

@j-arndt j-arndt commented May 4, 2026

Summary

Fixes #30: MetaAgent.forward declared and documented two
parameters — eval_path and iterations_left — that were silently dropped in
the method body. The agent received the same one-line instruction every
iteration regardless of what callers passed in.

This PR wires both parameters into the instruction string so the agent can
(a) locate prior evaluation results and read them with its existing tools,
and (b) reason about its remaining iteration budget when choosing exploration
vs. exploitation.

Why this is the right fix

The smallest possible change that actually uses the parameters as documented.
Three properties make it minimal:

  1. No new abstractions. No EvalContext, no PromptScaffold, no archive
    parser. The agent already has tools_available='all' (which includes the
    bash and edit tools); telling it where prior results live is sufficient.
  2. No coupling to archive format. This PR doesn't read or parse JSON files
    in eval_path — that would lock the framework to a specific archive
    schema and is the wrong place to make that decision. The agent reads what
    it needs at runtime via its tools.
  3. No API change. Method signature, return value, and call sites are
    unchanged. Existing callers (e.g. run_meta_agent.py) work unmodified.

Behavior change

Before — instruction was identical every iteration:

Modify any part of the codebase at `/path/to/repo`.

After — instruction includes the eval location and remaining budget when
provided:

Modify any part of the codebase at `/path/to/repo`.

Previously generated agents and their evaluation results are stored at
`/path/to/evals`. Read them before proposing modifications so you can learn
from what has already been tried and avoid repeating prior failures.

You have 7 iteration(s) remaining after this one. Budget exploration vs.
exploitation accordingly.

When eval_path is falsy or iterations_left is None, the corresponding
section is omitted — so callers who don't pass either get the original
single-line instruction unchanged.

Backward compatibility

  • API unchanged (no new arguments, no removed arguments, no type changes).
  • Return value unchanged (still implicit None).
  • Default behavior unchanged: passing iterations_left=None (the documented
    default) produces no iteration hint.
  • run_meta_agent.py and other callers require zero modifications.

Out of scope

A few related observations I noticed but deliberately did not include in this
PR to keep it focused and reviewable:

  • new_msg_history is assigned in the function body but never returned.
    Leaving as-is for a separate PR.
  • The instruction string remains a hardcoded f-string. A more flexible
    PromptScaffold abstraction would let domains customize how prior results
    are summarized; that's a larger framework-level change worth discussing on
    the issue thread before implementing.

Test plan

  • ✅ New unit tests in tests/test_meta_agent.py covering:
    • The instruction always references repo_path (regression check)
    • eval_path appears in the instruction when provided
    • eval_path is omitted from the instruction when empty/falsy
    • iterations_left appears as a numeric budget hint when provided
    • iterations_left=None (default) produces no budget hint
  • ✅ Manual sanity check: ran the modified MetaAgent.forward with a
    populated eval_path against a stub chat_with_agent and confirmed
    the agent receives the expected three-section instruction.

Files changed

  • meta_agent.py — wire both parameters into the instruction; structure as a
    list of parts joined on blank lines so the conditional sections compose
    cleanly.
  • tests/test_meta_agent.py — new file, 5 tests.

… parameters into the agent instruction; add unit tests
@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label May 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MetaAgent.forward accepts eval_path and iterations_left parameters that are never used in the body

1 participant