Repository navigation
feat(server-utils): Record LangChain TypeSafeClassifier runs as gen_ai.evaluate spans - #25120
Conversation
…i.evaluate spans Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
size-limit report 📦
|
![sentry[bot]](https://avatars.githubusercontent.com/in/12637?s=60&v=4){kind=small}
810bc99 to
973ab3e
Compare
JPeer264
left a comment
JPeer264
left a comment
There was a problem hiding this comment.
LGTM. General food for thought. I'm pretty sure in the next months weeks more classifier from different companies will come, so maybe we find a pattern to detect classifiers based in general (not sure if that is actually possible), as startTypeSafeClassifierSpan is very much specific to typesafe.
| await classifier.invoke(new HumanMessage('My card was charged twice.')); | ||
| }); | ||
|
|
||
| await Sentry.flush(2000); |
There was a problem hiding this comment.
q: Would this scenario also work without this flush?
There was a problem hiding this comment.
I removed it to show it works yeah.
| for (const channelName of [CHANNELS.LANGCHAIN_CHAT_MODEL_INVOKE, CHANNELS.LANGCHAIN_CHAT_MODEL_STREAM]) { | ||
| diagnosticsChannel.tracingChannel<RunnableChannelContext>(channelName).start.subscribe(injectHandler); | ||
| diagnosticsChannel.tracingChannel<RunnableChannelContext>(channelName).start.subscribe(message => { | ||
| markProvidersSkipped(); |
There was a problem hiding this comment.
q: Why was this moved from outside to here? I would think this moved to not skip other providers when nothing is being registered. But if so, it would need to have tests with other providers, which isn't in this PR AFAICS.
So theoretically, haven't tried, moving this back up wouldn't fail any tests right? Or am I something?
There was a problem hiding this comment.
This moved out because both channels need the inject handler but only one of them has to suppress providers. The typesafe stuff is just a plain fetch call so nothing to suppress.
chargome
left a comment
chargome
left a comment
There was a problem hiding this comment.
Generally LGTM, my clanker found one thing that needs to be double checked.
| injectHandler(message); | ||
| const { self, arguments: args } = message as RunnableChannelContext; | ||
| recordTypeSafeClassifierState(self, args?.[0]); |
There was a problem hiding this comment.
We might silently break some setups here.
Might be best to let your clanker double check:
Both middlewares in @langchain/typesafe call the classifier with no config:
// dist/middleware/autoMode.js:79
const risk = (await classifier.invoke(state)).nouls[RISK_QUESTION_ID].noul;
// dist/middleware/modelRouter.js:104
const answer = (await classifier.invoke(latest)).choices[QUESTION_ID];
So injectHandler always synthesizes {callbacks: [sentryHandler]}, and ensureConfig overlays that per key over the config LangGraph put in AsyncLocalStorage — the inherited CallbackManager is replaced, not merged. Classifier runs inside an agent stop reaching the user's LangSmith tracer and lose their parent run id. Chat models escape this only because LangGraph passes them an explicit config.
…1/test.ts Co-authored-by: Charly Gomez <charly.gomez1310@gmail.com>
…afe-evaluate-spans # Conflicts: # dev-packages/node-integration-tests/suites/tracing/langchain/v1/instrument-with-pii.mjs
…pans' into ab/langchain-typesafe-evaluate-spans
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 99b5dad. Configure here.
## What Adds tests that check Jev calls made inside agents show up as `gen_ai.evaluate` spans. - A LangChain `createAgent` with the TypeSafe model router middleware. - A LangGraph `StateGraph` node that calls `TypeSafeClassifier` directly. ## Why These Jev calls were invisible inside agents. #25120 fixes that, and these tests keep it from regressing. Closes: #25032 --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
3 participants
What
TypeSafeClassifierruns from@langchain/typesafenow show up asgen_ai.evaluatespans with model, provider, token usage, and the state, questions and answers when gen_ai recording is on.classifier.invoke()calls are traced automatically.Why
The classifier calls Jev with
fetchdirectly, so the TypeSafe integration did not see it, and LangChain reported it as a genericinvoke_agentspan without any evaluation data.Closes: #24859