Repository navigation
feat(server-utils): Map Mastra classifier evaluations to gen_ai.evaluate spans - #25044
Merged
Merged
Conversation
Contributor
size-limit report 📦
|
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
andreiborza
force-pushed
the
ab/mastra-classifier-evaluate-spans
branch
from
October 7, 2026 08:15
d3c4b79 to
1b15020
Compare
andreiborza
added this pull request to stack #25116
October 7, 2026 08:16
andreiborza
marked this pull request as ready for review
October 7, 2026 08:35
andreiborza
requested review from
RulaKhaled,
chargome,
isaacs,
mydea and
nicohrubec
and removed request for
a team,
isaacs and
mydea
October 7, 2026 08:35
andreiborza
force-pushed
the
ab/mastra-classifier-evaluate-spans
branch
from
October 7, 2026 09:11
1b15020 to
4c71676
Compare
![sentry[bot]](https://avatars.githubusercontent.com/in/12637?s=60&v=4){kind=small}
andreiborza
force-pushed
the
ab/mastra-classifier-evaluate-spans
branch
from
October 7, 2026 09:40
549d28a to
01f4a44
Compare
nicohrubec
reviewed
Oct 8, 2026
| import { safeChannelCallback } from '../tracing-channel'; | ||
| import * as diagnosticsChannel from '../utils/diagnosticsChannel'; | ||
|
|
||
| interface ClassifierEvaluateChannelContext { |
Member
There was a problem hiding this comment.
not specific to this PR, but I think we have an object that looks (almost) exactly like this in most or all of our integrations. we should probably refactor that at some point too
Member
Author
There was a problem hiding this comment.
Agreed, filed JS-3892 to add one shared orchestrion payload type.
| safeChannelCallback(() => { | ||
| const call = calls.get(message as object); | ||
| if (call) { | ||
| finishClassifierEvaluation(call, message as ClassifierEvaluateChannelContext); |
Member
There was a problem hiding this comment.
m/h: it seems mastra default-configures some filters that run on span events, for instance a sensitive data filter: https://mastra.ai/reference/observability/tracing/processors/sensitive-data-filter
iiuc this message object that is copied to the outputs here is stored from the original input so these filters are probably not applied yet? I am concerned this could leak sensitive data
Member
Author
There was a problem hiding this comment.
Good catch. I ended up running their processors over the data before storing it in f12213b.
I was debating wether we should reconstruct the data at all, but figured we are in the o11y business and it makes sense to do so. I'll file an issue upstream.
JPeer264
approved these changes
Oct 8, 2026
andreiborza
force-pushed
the
ab/mastra-classifier-evaluate-spans
branch
from
October 8, 2026 11:45
15ef1ec to
c3f2fa5
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit c3f2fa5. Configure here.
…ate spans Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
andreiborza
force-pushed
the
ab/mastra-classifier-evaluate-spans
branch
from
October 8, 2026 12:16
c3f2fa5 to
17acb4b
Compare
andreiborza
added a commit
that referenced
this pull request
Oct 8, 2026
## What Adds a node-mastra e2e test that runs a real Jev evaluation through a Mastra `Classifier` (via OpenRouter) and checks the `gen_ai.evaluate` span. ## Why Covers the classifier instrumentation from #25044 against the real provider, not only a mock model.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Mastra classifier evaluations (Jev calls through Mastra's
Classifier, its processors and scorers) now show up asgen_ai.evaluatespans with model, provider and token usage. When gen_ai recording is on, they also record the evaluated state, questions and answers.Why
These calls skip the AI SDK hook, and our Mastra exporter dropped their spans, so they were invisible.
Closes: #25031