Repository navigation
[BUG] TrajectoryEvaluator silently drops user-supplied tools on serialization and exposes no public tools attribute #381
Description
Checks
- I have updated to the latest minor and patch version of Strands and evals
- I have checked the documentation and this is not expected behavior
- I have searched ./issues and there are no duplicates of my issue
Strands Version
1.53.0
Strands Evals Version
main @ 9b245cf (1.2.1.dev)
Python Version
3.13.14
Operating System
Amazon Linux 2023 (aarch64)
Installation Method
git clone
Steps to Reproduce
from strands_evals.evaluators import TrajectoryEvaluator
te = TrajectoryEvaluator(rubric="r", tools=["my_pkg.calculator"])
print(te.to_dict()) # no 'tools' key at all
print(hasattr(te, "tools")) # FalseExpected Behavior
Consistent with OutputEvaluator after #379: serializable user tools survive to_dict()/Experiment.to_file(), non-serializable ones are skipped with a warning, and a public evaluator.tools attribute exists so tools can be re-attached after from_file().
Actual Behavior
trajectory_evaluator.py:45-50folds user tools intoself._toolstogether with the three built-in scorer callables (exact_match_scorer,in_order_match_scorer,any_order_match_scorer), and the inheritedEvaluator.to_dict()skips_-prefixed attrs — so user-supplied tools are silently dropped from serialization, with no warning, even fully serializable module-path strings.- There is no public
toolsattribute (hasattr(te, "tools")isFalse), so the re-attach-after-loading recipe documented in fix: skip non-serializable tools in OutputEvaluator serialization #379 cannot be applied to this evaluator — which is the one feat(evaluators): allow custom tools on judge-based evaluators (Trajectory, Output, Multimodal) #324's own example used (TrajectoryEvaluator(tools=[verify_ticket_id])).
Verified on main @ 9b245cf.
Additional Context
Found during review of #379. After #379 merges, the two evaluators that both gained tools= in #324 have opposite contracts: OutputEvaluator has a public documented .tools property and persists the serializable subset with a warning; TrajectoryEvaluator has neither. Label suggestion: design / area-devx (I can't apply labels).
Possible Solution
Store user tools separately from the built-in scorers (e.g. self._user_tools), merge at evaluate() time, expose a tools property over the user subset, and mirror #379's to_dict() override. Needs a small design pass because _tools currently pre-mixes defaults with user tools.
Related Issues
Metadata
Metadata
Assignees
Labels
Type
Fields
Language
Activity