Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ This integration is enabled by default when <PlatformLink to="/tracing/">tracing

The `typesafeIntegration` adds instrumentation for [`@typesafe-ai/sdk`](https://www.npmjs.com/package/@typesafe-ai/sdk) to capture <PlatformLink to="/agent-tracing/">agent tracing</PlatformLink> `gen_ai.evaluate` spans, recording model, token usage, latency, and (when enabled) evaluation inputs and answers.

If you call Jev through Vercel AI SDK's `experimental_evaluate()`, see the [Vercel AI integration](/platforms/javascript/guides/node/configuration/integrations/vercelai/#trace-jev-evaluations) instead. If you call Jev through Cloudflare Workers AI, see [Workers AI](/platforms/javascript/guides/cloudflare/agent-tracing/workers-ai/#trace-jev-evaluations).
If you call Jev through Vercel AI SDK's `experimental_evaluate()`, see the [Vercel AI integration](/platforms/javascript/guides/node/configuration/integrations/vercelai/#trace-jev-evaluations) instead. If you call Jev through Cloudflare Workers AI, see [Workers AI](/platforms/javascript/guides/cloudflare/agent-tracing/workers-ai/#trace-jev-evaluations). If you call Jev through a Mastra `Classifier`, see [Mastra Classifier Evaluations](/platforms/javascript/guides/mastra/agent-tracing/classifier-evaluations/).

```javascript
Sentry.init({
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
---
title: Mastra Classifier Evaluations
sidebar_title: Classifier Evaluations
description: "Learn how Sentry traces Mastra Classifier evaluations, such as calls to TypeSafe's Jev, as gen_ai.evaluate spans."
---

<AvailableSince version="11.6.0" />

Mastra's [`Classifier`](https://mastra.ai/reference/classifier/classifier) asks a decision model, such as TypeSafe's Jev, typed questions about your application state. These calls don't go through the Vercel AI SDK, so the Mastra integration traces them for you. This includes classifiers that your processors and scorers call. If you've followed the [Mastra Quick Start](/platforms/javascript/guides/mastra/), no additional integration or setup is needed.

```javascript {filename:src/mastra/classifiers/ticket-classifier.ts}
import { Classifier } from "@mastra/core/classifier";
import { typeSafeAi } from "@ai-sdk/typesafe-ai";

const classifier = new Classifier({
id: "ticket-classifier",
model: typeSafeAi.evaluationModel("jev-latest"),
questions: {
department: {
type: "choice",
instructions: "Which team should handle this?",
criteria: {
billing: "Charges and refunds",
technical: "Bugs and outages",
},
},
},
});

const result = await classifier.evaluate({
state: { message: "I was charged twice. Please refund the extra charge." },
});

console.log(result.answers.department.choice);
```

## What Gets Captured

Each `evaluate()` call creates a `gen_ai.evaluate` span named `evaluate <model>`, with the model, provider, latency, and token usage. If the evaluation fails, the span is marked as failed. When recording is enabled, Sentry captures `state` and `questions` as input messages and `answers` as output messages, including probabilities returned by the model.

## Control Recorded Data

Sentry runs this data through your Mastra span output processors before recording it, so Mastra's default `SensitiveDataFilter` redacts sensitive keys such as `apiKey`. Evaluation state and answers can still contain sensitive data. To stop recording them, set `genAI: { inputs: false, outputs: false }` in the <PlatformLink to="/configuration/options/#dataCollection">`dataCollection` option</PlatformLink>.

## Supported Versions

- `@sentry/node`: `>=11.6.0`
- `@mastra/core`: `>=1.69.0`

## Related Integrations

If you call Jev through `TypeSafeClient.systemOne()` or Vercel AI SDK's `experimental_evaluate()` instead, see the [TypeSafe](/platforms/javascript/guides/node/configuration/integrations/typesafe/) or [Vercel AI](/platforms/javascript/guides/node/configuration/integrations/vercelai/#trace-jev-evaluations) integration.
21 changes: 11 additions & 10 deletions includes/mastra/what-gets-captured.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,17 @@

Sentry's Mastra integration maps Mastra's span types to Sentry operations for the Agents dashboards:

| Mastra Span Type | Sentry Operation |
| -------------------- | --------------------- |
| `AGENT_RUN` | `gen_ai.invoke_agent` |
| `WORKFLOW_RUN` | `gen_ai.invoke_agent` |
| `MODEL_GENERATION` | `gen_ai.chat` |
| `TOOL_CALL` | `gen_ai.execute_tool` |
| `MCP_TOOL_CALL` | `gen_ai.execute_tool` |
| `PROVIDER_TOOL_CALL` | `gen_ai.execute_tool` |
| `CLIENT_TOOL_CALL` | `gen_ai.execute_tool` |
| `RAG_EMBEDDING` | `gen_ai.embeddings` |
| Mastra Span Type | Sentry Operation |
| ----------------------- | --------------------- |
| `AGENT_RUN` | `gen_ai.invoke_agent` |
| `WORKFLOW_RUN` | `gen_ai.invoke_agent` |
| `MODEL_GENERATION` | `gen_ai.chat` |
| `TOOL_CALL` | `gen_ai.execute_tool` |
| `MCP_TOOL_CALL` | `gen_ai.execute_tool` |
| `PROVIDER_TOOL_CALL` | `gen_ai.execute_tool` |
| `CLIENT_TOOL_CALL` | `gen_ai.execute_tool` |
| `RAG_EMBEDDING` | `gen_ai.embeddings` |
| `CLASSIFIER_EVALUATION` | `gen_ai.evaluate` |

Spans are tagged with `sentry.origin: auto.ai.mastra`.

Expand Down
Loading