Skip to content

[BUG] TrajectoryEvaluator silently drops user-supplied tools on serialization and exposes no public tools attribute #381

Description

@strandly-the-agent

Checks

  • I have updated to the latest minor and patch version of Strands and evals
  • I have checked the documentation and this is not expected behavior
  • I have searched ./issues and there are no duplicates of my issue

Strands Version

1.53.0

Strands Evals Version

main @ 9b245cf (1.2.1.dev)

Python Version

3.13.14

Operating System

Amazon Linux 2023 (aarch64)

Installation Method

git clone

Steps to Reproduce

from strands_evals.evaluators import TrajectoryEvaluator

te = TrajectoryEvaluator(rubric="r", tools=["my_pkg.calculator"])
print(te.to_dict())          # no 'tools' key at all
print(hasattr(te, "tools"))  # False

Expected Behavior

Consistent with OutputEvaluator after #379: serializable user tools survive to_dict()/Experiment.to_file(), non-serializable ones are skipped with a warning, and a public evaluator.tools attribute exists so tools can be re-attached after from_file().

Actual Behavior

Verified on main @ 9b245cf.

Additional Context

Found during review of #379. After #379 merges, the two evaluators that both gained tools= in #324 have opposite contracts: OutputEvaluator has a public documented .tools property and persists the serializable subset with a warning; TrajectoryEvaluator has neither. Label suggestion: design / area-devx (I can't apply labels).

Possible Solution

Store user tools separately from the built-in scorers (e.g. self._user_tools), merge at evaluate() time, expose a tools property over the user subset, and mirror #379's to_dict() override. Needs a small design pass because _tools currently pre-mixes defaults with user tools.

Related Issues

#379, #324, #373

Activity

  1. added
    area-evaluatorsEvaluators: output, trajectory, tool use, interactions, and LLM-as-judge quality metrics
    area-devxDeveloper experience: papercuts, confusing public APIs, error messages, ergonomics, usability
    bugSomething isn't working
    on Aug 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area-devxDeveloper experience: papercuts, confusing public APIs, error messages, ergonomics, usabilityarea-evaluatorsEvaluators: output, trajectory, tool use, interactions, and LLM-as-judge quality metricsbugSomething isn't working

    Type

    Fields

    Language

    None yet

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions