Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ observability backends through a pluggable handler architecture.
- 🌟 **OpenInference support** - Full semantic conventions for Arize Phoenix
- 📊 **Rich metadata tracking** - Token usage, costs, tool calls, and more
- 🚀 **Built on OTP** - Supervised handlers with fault tolerance
- 🔗 **Jido integration (optional)** - Zero-code tracing for Jido composer
workflows
- 🧪 **Backend-agnostic** - Standardized event schema independent of backends

## Architecture
Expand Down Expand Up @@ -218,6 +220,35 @@ object = ReqLLM.Response.object(response)
See the [demo agent](demo/lib/demo/agent.ex) and
[ReqLLM integration guide](guides/req_llm_integration.md) for complete examples.

## Jido Integration (Optional)

For applications using [Jido](https://hexdocs.pm/jido), AgentObs provides
`AgentObs.JidoTracer` — a drop-in `Jido.Observe.Tracer` implementation that
automatically instruments all composer events with OpenTelemetry spans.

```elixir
# Add to your deps
{:jido, "~> 2.0"}

# Configure Jido to use the tracer
config :jido, :observability,
tracer: AgentObs.JidoTracer
```

That's it. All `[:jido, :composer, :agent|:llm|:tool]` events are automatically
mapped to AgentObs event types and traced with OpenInference semantic conventions.
Parent-child span nesting is preserved, so you get a full trace tree in Phoenix:

```
weather_assistant (agent)
├── gpt-4o #1 (llm)
├── get_weather (tool)
└── gpt-4o #2 (llm)
```

See the [Jido integration guide](guides/jido_integration.md) for details on
event mapping, metadata translation, and advanced usage.

## API Reference

### High-Level Instrumentation
Expand Down
188 changes: 188 additions & 0 deletions guides/jido_integration.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,188 @@
# Jido Integration Guide

AgentObs provides a drop-in tracer for [Jido](https://hexdocs.pm/jido) that
automatically instruments composer events with OpenTelemetry spans and
OpenInference semantic conventions.

## Table of Contents

- [Overview](#overview)
- [Installation](#installation)
- [Configuration](#configuration)
- [How It Works](#how-it-works)
- [Event Mapping](#event-mapping)
- [Metadata Translation](#metadata-translation)
- [Viewing Traces](#viewing-traces)
- [Advanced Usage](#advanced-usage)

## Overview

If you're using Jido's composer to orchestrate agent, LLM, and tool calls,
`AgentObs.JidoTracer` bridges those telemetry events into AgentObs. This gives
you full observability in Arize Phoenix (or any OpenTelemetry backend) with zero
manual instrumentation of your Jido workflows.

**Key benefits:**

- Zero-code instrumentation for Jido composer workflows
- Automatic parent-child span nesting
- OpenInference semantic conventions for rich visualization in Phoenix
- Graceful error handling (never crashes your application)

## Installation

Add both dependencies to `mix.exs`:

```elixir
def deps do
[
{:agent_obs, "~> 0.1.3"},
{:jido, "~> 2.0"}
]
end
```

`jido` is an **optional** dependency of `agent_obs`. The `AgentObs.JidoTracer`
module is only available when Jido is installed.

## Configuration

Point Jido's observability config at the tracer:

```elixir
# config/config.exs
config :jido, :observability,
tracer: AgentObs.JidoTracer

# Also configure AgentObs handlers as usual
config :agent_obs,
enabled: true,
handlers: [AgentObs.Handlers.Phoenix]
```

Make sure your OpenTelemetry exporter is configured to send spans to your
backend:

```elixir
# config/runtime.exs
config :opentelemetry,
span_processor: :batch,
resource: [service: [name: "my_jido_app"]]

config :opentelemetry_exporter,
otlp_protocol: :http_protobuf,
otlp_endpoint: System.get_env("ARIZE_PHOENIX_OTLP_ENDPOINT", "http://localhost:6006")
```

That's it. All Jido composer events will now appear as traced spans.

## How It Works

`AgentObs.JidoTracer` implements the `Jido.Observe.Tracer` behaviour, which
Jido calls automatically during composer execution:

1. **`span_start/2`** - Called when a composer event begins. Creates an
OpenTelemetry span with attributes translated to OpenInference conventions.
2. **`span_stop/2`** - Called when the event completes. Sets result attributes
and ends the span.
3. **`span_exception/4`** - Called on errors. Records the exception on the span
and sets error status.

Parent-child relationships are maintained automatically through OpenTelemetry
context propagation, so nested agent > LLM > tool calls appear correctly in
trace visualizers.

## Event Mapping

Jido event prefixes are classified into AgentObs event types:

| Jido Event Prefix | AgentObs Type | Span Name Example |
| -------------------------------------- | ------------- | --------------------- |
| `[:jido, :composer, :agent, ...]` | `:agent` | `"weather_assistant"` |
| `[:jido, :composer, :llm, ...]` | `:llm` | `"gpt-4o #2"` |
| `[:jido, :composer, :tool, ...]` | `:tool` | `"get_weather"` |
| `[:jido, :composer, :iteration, ...]` | `:chain` | `"iteration 3"` |
| Any other prefix | `:agent` | `"agent"` |

## Metadata Translation

The tracer automatically maps Jido metadata fields to AgentObs format:

### Agent Events

| Jido Field | AgentObs Field | Notes |
| ---------------- | -------------- | -------------------------- |
| `:query` | `:input` | Falls back to `:input` |
| `:name` | `:name` | Falls back to `:agent_module` |
| `:model` | `:model` | Optional |
| `:result` | `:output` | On stop, falls back to `:output` |

### LLM Events

| Jido Field | AgentObs Field | Notes |
| ------------------ | ------------------ | ------------------------------ |
| `:model` | `:model` | Defaults to `"unknown"` |
| `:input_messages` | `:input_messages` | Preferred |
| `:conversation` | `:input_messages` | Normalized to `%{role, content}` maps |
| `:iteration` | `:iteration` | Appended to span name |
| `:tokens` | `:tokens` | On stop |

### Tool Events

| Jido Field | AgentObs Field | Notes |
| ------------- | -------------- | ---------------------------------- |
| `:tool_name` | `:name` | Falls back to `:node_name`, `:name` |
| `:arguments` | `:arguments` | Falls back to `:params` |
| `:result` | `:result` | On stop, falls back to `:output` |

## Viewing Traces

Start a local Arize Phoenix instance:

```bash
docker run -p 6006:6006 -p 4317:4317 arizephoenix/phoenix:latest
```

Navigate to `http://localhost:6006` to see your Jido workflows as nested traces:

```
weather_assistant (agent)
├── gpt-4o #1 (llm)
├── get_weather (tool)
└── gpt-4o #2 (llm)
```

## Advanced Usage

### Combining with Manual Instrumentation

You can use `AgentObs.JidoTracer` for Jido workflows while using the
`AgentObs.trace_*` helpers for non-Jido code in the same application:

```elixir
def my_workflow(query) do
# Manual instrumentation for the outer operation
AgentObs.trace_agent("my_workflow", %{input: query}, fn ->
# Jido composer is automatically traced via JidoTracer
{:ok, result} = Jido.Composer.run(my_composer, query)
{:ok, result}
end)
end
```

### Error Handling

The tracer is designed to never crash your application. If span creation or
attribute translation fails, a warning is logged and execution continues
normally:

```
[warning] AgentObs.JidoTracer span_start failed: %RuntimeError{message: "..."}
```

## Next Steps

- **[Getting Started](getting_started.md)** - Basic AgentObs setup
- **[Instrumentation Guide](instrumentation.md)** - Manual instrumentation patterns
- **[ReqLLM Integration](req_llm_integration.md)** - ReqLLM auto-instrumentation
- **[Configuration](configuration.md)** - Advanced configuration options
5 changes: 5 additions & 0 deletions lib/agent_obs.ex
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,11 @@ defmodule AgentObs do
- `trace_llm/3` - Instruments LLM API calls
- `trace_prompt/3` - Instruments prompt template rendering

## Integrations

- `AgentObs.ReqLLM` - Automatic instrumentation for ReqLLM streaming calls
- `AgentObs.JidoTracer` - Zero-code tracing for Jido composer workflows

## Low-Level API

- `emit/2` - Emits custom telemetry events
Expand Down
49 changes: 43 additions & 6 deletions lib/agent_obs/handlers/phoenix/translator.ex
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ defmodule AgentObs.Handlers.Phoenix.Translator do
- `AGENT` - Agent loop or orchestration
- `LLM` - Large Language Model API call
- `TOOL` - Tool or function execution
- `CHAIN` - Sequence of operations (not used in AgentObs currently)
- `CHAIN` - Sequence of operations (e.g., orchestrator iteration)
- `RETRIEVER` - Vector/document retrieval (not used in AgentObs currently)

## Key Attributes
Expand All @@ -39,7 +39,7 @@ defmodule AgentObs.Handlers.Phoenix.Translator do

## Parameters

- `event_type` - One of `:agent`, `:tool`, `:llm`, `:prompt`
- `event_type` - One of `:agent`, `:tool`, `:llm`, `:chain`, `:prompt`
- `metadata` - The start metadata from AgentObs event

## Returns
Expand All @@ -65,6 +65,7 @@ defmodule AgentObs.Handlers.Phoenix.Translator do
"tool.name" => metadata.name
}
|> maybe_add("tool.description", metadata[:description])
|> maybe_add("input.value", to_json_safe(metadata[:arguments]))
|> add_tool_arguments(metadata[:arguments])
end

Expand All @@ -81,6 +82,15 @@ defmodule AgentObs.Handlers.Phoenix.Translator do
|> maybe_add("ai.model.id", metadata.model)
|> maybe_add("ai.model.provider", extract_provider(metadata.model))
|> Map.merge(flatten_input_messages(metadata[:input_messages]))
|> maybe_add("input.value", extract_llm_input_text(metadata[:input_messages]))
end

def from_start_metadata(:chain, metadata) do
%{
"openinference.span.kind" => "CHAIN"
}
|> maybe_add("metadata.iteration", metadata[:iteration])
|> maybe_add("input.value", to_json_safe(metadata[:input]))
end

def from_start_metadata(:prompt, metadata) do
Expand All @@ -97,7 +107,7 @@ defmodule AgentObs.Handlers.Phoenix.Translator do

## Parameters

- `event_type` - One of `:agent`, `:tool`, `:llm`, `:prompt`
- `event_type` - One of `:agent`, `:tool`, `:llm`, `:chain`, `:prompt`
- `metadata` - The stop metadata from AgentObs event
- `measurements` - Measurements map containing duration

Expand Down Expand Up @@ -134,7 +144,7 @@ defmodule AgentObs.Handlers.Phoenix.Translator do
|> maybe_add("llm.token_count.completion", get_in(metadata, [:tokens, :completion]))
|> maybe_add("llm.token_count.total", get_in(metadata, [:tokens, :total]))
|> maybe_add("llm.cost.total", metadata[:cost])
|> maybe_add("output.value", metadata[:finish_reason])
|> maybe_add("output.value", extract_llm_output_text(metadata[:output_messages]))
# Add gen_ai usage attributes
|> maybe_add("gen_ai.usage.input_tokens", get_in(metadata, [:tokens, :prompt]))
|> maybe_add("gen_ai.usage.output_tokens", get_in(metadata, [:tokens, :completion]))
Expand All @@ -145,6 +155,12 @@ defmodule AgentObs.Handlers.Phoenix.Translator do
|> add_duration(measurements)
end

def from_stop_metadata(:chain, metadata, measurements) do
%{}
|> maybe_add("output.value", to_json_safe(metadata[:output]))
|> add_duration(measurements)
end

def from_stop_metadata(:prompt, metadata, measurements) do
%{
"output.value" => to_json_safe(metadata[:rendered])
Expand All @@ -157,7 +173,7 @@ defmodule AgentObs.Handlers.Phoenix.Translator do

## Parameters

- `event_type` - One of `:agent`, `:tool`, `:llm`, `:prompt`
- `event_type` - One of `:agent`, `:tool`, `:llm`, `:chain`, `:prompt`
- `metadata` - The exception metadata from telemetry
- `measurements` - Measurements map containing duration

Expand All @@ -182,6 +198,27 @@ defmodule AgentObs.Handlers.Phoenix.Translator do

# Private helper functions

defp extract_llm_input_text(nil), do: nil
defp extract_llm_input_text([]), do: nil

defp extract_llm_input_text(messages) do
messages
|> Enum.reverse()
|> Enum.find(fn msg -> get_message_field(msg, :role) in ["user", :user] end)
|> case do
nil -> nil
msg -> to_json_safe(get_message_field(msg, :content))
end
end

defp extract_llm_output_text(nil), do: nil
defp extract_llm_output_text([]), do: nil

defp extract_llm_output_text([first | _]) do
content = get_message_field(first, :content)
if content, do: to_json_safe(content)
end

defp flatten_input_messages(nil), do: %{}

defp flatten_input_messages(messages) when is_list(messages) do
Expand Down Expand Up @@ -304,12 +341,12 @@ defmodule AgentObs.Handlers.Phoenix.Translator do
end)
end

defp to_json_safe(nil), do: nil
defp to_json_safe(value) when is_binary(value), do: value
defp to_json_safe(value) when is_number(value), do: value
defp to_json_safe(value) when is_boolean(value), do: value
defp to_json_safe(value) when is_atom(value), do: to_string(value)
defp to_json_safe(value) when is_map(value) or is_list(value), do: encode_json(value)
defp to_json_safe(nil), do: nil

defp encode_json(value) do
case Jason.encode(value) do
Expand Down
Loading
Loading