feat: Laminar tracing plugin for Hermes Agent (LAM-1511)

[laminar-coding-agent]

Summary

• New examples/hermes-plugin/ — a standalone pip package (lmnr-hermes) that plugs Laminar tracing into Hermes Agent via Hermes's native plugin hook system. No monkey-patching, no forked runtime.
• Emits a hermes.turn root span per run_conversation call with session_id / user_id / model / provider attribution, tool.<name> children (span_type=TOOL) for every tool dispatch, and subagent.<role> children for delegated subagents. Token usage, finish_reason, and tool duration_ms are attached as span attributes.
• Ships via the hermes_agent.plugins entry-point group, so pip install -e examples/hermes-plugin && hermes plugins enable lmnr-hermes is the whole install — no PyPI publish, no Hermes fork required.

Design notes

• Hook set bridged: on_session_start, pre_llm_call, post_api_request, pre_tool_call, post_tool_call, post_llm_call, on_session_end, subagent_stop. Signatures are taken directly from hermes-agent/run_agent.py (lines 9659–12899) and model_tools.py (lines 530–634).
• Thread safety: Hermes runs concurrent tool calls on a ThreadPoolExecutor. Rather than relying on OTel's thread-local current context, tool spans are parented by serializing the turn span's LaminarSpanContext and passing it as parent_span_context= — so parenting survives cross-thread hook invocations.
• Initialization is lazy-once (_ensure_initialized()), keyed off LMNR_PROJECT_API_KEY. Missing key → plugin no-ops silently; Hermes keeps working.
• Raw-provider instrumentors (OpenAI, Anthropic, Bedrock) are auto-enabled by Laminar.initialize() and nest their GenAI spans under the current Hermes turn, giving token/message-level detail for free.
• Entry-point install required adding examples/hermes-plugin to [tool.uv.workspace].members in the root pyproject.toml, mirroring how fastapi-app is wired.

Test plan

• Hermetic smoke test that replays Hermes's exact invoke_hook(...) signatures against the plugin and asserts:
  • hermes.turn / tool.terminal / subagent.researcher spans are all captured
  • tool and subagent spans parent under the turn span
  • session_id propagates to lmnr.association.properties.session_id on every span
  • hermes.model, hermes.provider, hermes.api_mode, hermes.usage.*, hermes.finish_reason, hermes.tool_duration_ms, hermes.completed attributes are set correctly
  • turn span captures the user message as input and tool result as output
• uv pip install -e examples/hermes-plugin builds the package and registers the lmnr-hermes entry point in hermes_agent.plugins (verified via importlib.metadata.entry_points().select(group="hermes_agent.plugins")).
• Existing lmnr-python test suite (uv run pytest tests/) — the 28 failures on this branch all reproduce on main (bedrock network errors + unrelated test_claude_agent/test_proxy_env.py) and are pre-existing.

🤖 Generated with Claude Code

---

Note

Medium Risk
Adds a new standalone plugin that manages OpenTelemetry span lifecycle across Hermes hook threads; concurrency/state handling and token-usage mapping could lead to incorrect/duplicated spans or leaked context if hook behavior differs in production.

Overview
Adds a new standalone examples/hermes-plugin package (lmnr-hermes) that registers via the hermes_agent.plugins entry point and emits Laminar spans for Hermes hooks.

The plugin creates a hermes.turn root span per turn, plus child tool.<name> TOOL spans, subagent.<role> spans, and explicit per-request LLM spans from pre_api_request/post_api_request (including GenAI semantic-convention token usage/caching fields for cost attribution). It includes thread-safe span bookkeeping and cleanup to handle Hermes’s multi-threaded tool execution and missing hook pairs.

Updates the root pyproject.toml uv workspace membership to include examples/hermes-plugin, and documents the new example and its tracing/port-configuration caveats in CLAUDE.md.

^{Reviewed by Cursor Bugbot for commit 3b25bce. Bugbot is set up for automated code reviews on this repo. Configure here.}

[laminar-coding-agent]

Update: emit LLM spans from pre_/post_api_request (commit b6af913)

The earlier revision of this plugin relied on Laminar's auto-enabled raw-SDK
instrumentors (OpenAI, Anthropic, Bedrock) to produce the per-call model spans
nested under each hermes.turn. That doesn't work: Hermes drives its own HTTP
client (run_agent.py ~L11880 _get_transport().normalize_response) and never
touches the anthropic / openai Python SDKs, so the raw instrumentors are
silent. Traces showed tool spans but no LLM spans, and $0 cost.

This commit has the plugin emit the LLM spans itself:

• pre_api_request opens a span (span_type="LLM", name llm.<provider>.<model>)
  parented to the turn's LaminarSpanContext (cross-thread safe).
• post_api_request closes it after setting GenAI semconv attributes. Hermes's
  CanonicalUsage fields are mapped onto the exact keys Laminar's cost
  computation reads: cache_read_tokens -> gen_ai.usage.cache_read_input_tokens,
  cache_write_tokens -> gen_ai.usage.cache_creation_input_tokens.
• _cleanup_api_spans sweeps any dangling entries in _close_turn /
  _on_session_end so a provider error between pre_ and post_ can't leak a
  span.
• Plugin also now honours LMNR_HTTP_PORT / LMNR_GRPC_PORT so a local
  Laminar dev instance (gRPC 8001, not the SDK default 8443) can be
  targeted without passing ports in the base URL (which the SDK strips).

Verification (local Laminar stack)

Ran the example Hermes task end-to-end against a local Laminar instance. Trace
ec2ec1c5-a7d8-a26b-8539-b5f9ef3cf48b shows the expected tree: hermes.turn
with three llm.anthropic.claude-opus-4-7 children interleaved with
tool.write_file / tool.terminal spans, input/output tokens filled in, and
per-call cost attributed correctly.

• Tree view with LLM spans nested under the turn span, showing token counts and cost per call: https://svwyososwvsgouxwfdlc.supabase.co/storage/v1/object/public/lmnr-coding-agent/hermes-plugin-llm-spans-tree.png
• Trace list showing the turn's total cost ($0.06395) and aggregate token usage (36108 -> 699): https://svwyososwvsgouxwfdlc.supabase.co/storage/v1/object/public/lmnr-coding-agent/hermes-plugin-trace-list-with-costs.png

CLAUDE.md has been updated with the three non-obvious findings (Hermes bypasses
anthropic/openai SDKs; CanonicalUsage -> GenAI semconv mapping; SDK port
stripping + LMNR_HTTP_PORT/LMNR_GRPC_PORT override).

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 30b8243. Configure here.}