Skip to content

Commit ecd60da

Browse files
mschristensenmschristensen
committed
ait/features: add tool call page
Adds a page to the Messaging section that describes sending tool calls and results to users over channels. Indicates ability to build generative user interfaces or implement human in the loop workflows.
1 parent 15363db commit ecd60da

2 files changed

Lines changed: 274 additions & 0 deletions

File tree

‎src/data/nav/aitransport.ts‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,10 @@ export default {
6464
name: 'Accepting user input',
6565
link: '/docs/ai-transport/features/messaging/accepting-user-input',
6666
},
67+
{
68+
name: 'Tool calls',
69+
link: '/docs/ai-transport/features/messaging/tool-calls',
70+
},
6771
{
6872
name: 'Human-in-the-loop',
6973
link: '/docs/ai-transport/features/messaging/human-in-the-loop',

‎src/pages/docs/ai-transport/features/messaging/tool-calls.mdx‎

Lines changed: 270 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,270 @@
1+
---
2+
title: "Tool calls"
3+
meta_description: "Stream tool call execution visibility to users, enabling transparent AI interactions and generative UI experiences."
4+
meta_keywords: "tool calls, function calling, generative UI, AI transparency, tool execution, streaming JSON, realtime feedback"
5+
---
6+
7+
Modern AI models can invoke tools (also called functions) to perform specific tasks like retrieving data, performing calculations, or triggering actions. Streaming tool call information to users provides visibility into what the AI is doing, creates opportunities for rich generative UI experiences, and builds trust through transparency.
8+
9+
## What are tool calls? <a id="what"/>
10+
11+
Tool calls occur when an AI model decides to invoke a specific function or tool to accomplish a task. Rather than only returning text, the model can request to execute tools you've defined, such as fetching weather data, searching a database, or performing calculations.
12+
13+
A tool call consists of:
14+
15+
- Tool name: The identifier of the tool being invoked
16+
- Tool input: Parameters passed to the tool, often structured as JSON
17+
- Tool output: The result returned after execution
18+
19+
As an application developer, you decide how to surface tool calls to users. You may choose to display all tool calls, selectively surface specific tools or inputs/outputs, or keep tool calls entirely private.
20+
21+
Surfacing tool calls supports:
22+
23+
- Trust and transparency: Users see what actions the AI is taking, building confidence in the agent
24+
- Human-in-the-loop workflows: Expose tool calls [resolved by humans](/docs/ai-transport/features/messaging/human-in-the-loop) where users can review and approve tool execution before it happens
25+
- Generative UI: Build dynamic, contextual UI components based on the structured tool data
26+
27+
## Publishing tool calls <a id="publishing"/>
28+
29+
Publish tool call and model output messages to the channel.
30+
31+
In the example below, the `responseId` is included in the message [extras](/docs/messages#properties) to allow subscribers to correlate all messages belonging to the same response. The message [`name`](/docs/messages#properties) allows the client to distinguish between the different message types:
32+
33+
<Code>
34+
```javascript
35+
const channel = realtime.channels.get('{{RANDOM_CHANNEL_NAME}}');
36+
37+
// Example: stream returns events like:
38+
// { type: 'tool_call', name: 'get_weather', args: '{"location":"San Francisco"}', toolCallId: 'tool_123', responseId: 'resp_abc123' }
39+
// { type: 'tool_result', name: 'get_weather', result: '{"temp":72,"conditions":"sunny"}', toolCallId: 'tool_123', responseId: 'resp_abc123' }
40+
// { type: 'message', text: 'The weather in San Francisco is 72°F and sunny.', responseId: 'resp_abc123' }
41+
42+
for await (const event of stream) {
43+
if (event.type === 'tool_call') {
44+
// Publish tool call arguments
45+
await channel.publish({
46+
name: 'tool_call',
47+
data: {
48+
name: event.name,
49+
args: event.args
50+
},
51+
extras: {
52+
headers: {
53+
responseId: event.responseId,
54+
toolCallId: event.toolCallId
55+
}
56+
}
57+
});
58+
} else if (event.type === 'tool_result') {
59+
// Publish tool call results
60+
await channel.publish({
61+
name: 'tool_result',
62+
data: {
63+
name: event.name,
64+
result: event.result
65+
},
66+
extras: {
67+
headers: {
68+
responseId: event.responseId,
69+
toolCallId: event.toolCallId
70+
}
71+
}
72+
});
73+
} else if (event.type === 'message') {
74+
// Publish model output messages
75+
await channel.publish({
76+
name: 'message',
77+
data: event.text,
78+
extras: {
79+
headers: {
80+
responseId: event.responseId
81+
}
82+
}
83+
});
84+
}
85+
}
86+
```
87+
</Code>
88+
89+
<Aside data-type="note">
90+
Model APIs like OpenAI's [Responses API](https://platform.openai.com/docs/api-reference/responses) and Anthropic's [Messages API](https://platform.claude.com/docs/en/api/messages) don't include tool results in their streams - instead, you execute tools in your code and return results to the model, but the model's output doesn't echo those results back. Agent SDKs like [OpenAI Agent SDK](https://platform.openai.com/docs/guides/agents-sdk) and [Claude Agent SDK](https://platform.claude.com/docs/en/agent-sdk/overview) maintain context and surface both tool calls and results on the stream. When using model APIs directly, publish tool results to the channel separately if you want to surface them to subscribers.
91+
</Aside>
92+
93+
<Aside data-type="note">
94+
To learn how to stream individual tokens as they are generated, see the [token streaming](/docs/ai-transport/features/token-streaming) documentation.
95+
</Aside>
96+
97+
## Subscribing to tool calls <a id="subscribing"/>
98+
99+
Subscribe to tool call and model output messages on the channel.
100+
101+
In the example below, the `responseId` from the message [`extras`](/docs/api/realtime-sdk/messages#extras) is used to group tool calls and model output messages belonging to the same response. The message [`name`](/docs/messages#properties) allows the client to distinguish between the different message types:
102+
103+
<Code>
104+
```javascript
105+
const channel = realtime.channels.get('{{RANDOM_CHANNEL_NAME}}');
106+
107+
// Track responses by ID, each containing tool calls and final response
108+
const responses = new Map();
109+
110+
// Subscribe to all events on the channel
111+
await channel.subscribe((message) => {
112+
const responseId = message.extras?.headers?.responseId;
113+
114+
if (!responseId) {
115+
console.warn('Message missing responseId');
116+
return;
117+
}
118+
119+
// Initialize response object if needed
120+
if (!responses.has(responseId)) {
121+
responses.set(responseId, {
122+
toolCalls: new Map(),
123+
message: ''
124+
});
125+
}
126+
127+
const response = responses.get(responseId);
128+
129+
// Handle each message type
130+
switch (message.name) {
131+
case 'message':
132+
response.message = message.data;
133+
break;
134+
case 'tool_call':
135+
const toolCallId = message.extras?.headers?.toolCallId;
136+
response.toolCalls.set(toolCallId, {
137+
name: message.data.name,
138+
args: message.data.args
139+
});
140+
break;
141+
case 'tool_result':
142+
const resultToolCallId = message.extras?.headers?.toolCallId;
143+
const toolCall = response.toolCalls.get(resultToolCallId);
144+
if (toolCall) {
145+
toolCall.result = message.data.result;
146+
}
147+
break;
148+
}
149+
150+
// Display the tool calls and response for this turn
151+
console.log(`Response ${responseId}:`, response);
152+
});
153+
```
154+
</Code>
155+
156+
<Aside data-type="further-reading">
157+
To learn about hydrating responses from channel history, including using `rewind` or `untilAttach`, handling in-progress responses, and correlating with database records, see client hydration in the [message-per-response](/docs/ai-transport/features/token-streaming/message-per-response#hydration) and [message-per-token](/docs/ai-transport/features/token-streaming/message-per-token#hydration) documentation.
158+
</Aside>
159+
160+
## Generative UI <a id="generative-ui"/>
161+
162+
Tool calls provide structured data that can form the basis of generative UI - dynamically creating UI components based on the tool being invoked, its parameters, and the results returned. Rather than just displaying raw tool call information, you can render rich, contextual components that provide a better user experience.
163+
164+
For example, when a weather tool is invoked, instead of showing raw JSON like `{ location: 'San Francisco', temp: 72, conditions: 'sunny' }`, you can render a weather card component with icons, formatted temperature, and visual indicators:
165+
166+
<Code>
167+
```javascript
168+
const channel = realtime.channels.get('{{RANDOM_CHANNEL_NAME}}');
169+
170+
await channel.subscribe((message) => {
171+
// Render component when tool is invoked
172+
if (message.name === 'tool_call' && message.data.name === 'get_weather') {
173+
const args = JSON.parse(message.data.args);
174+
renderWeatherCard({ location: args.location, loading: true });
175+
}
176+
177+
// Update component with results
178+
if (message.name === 'tool_result' && message.data.name === 'get_weather') {
179+
const result = JSON.parse(message.data.result);
180+
renderWeatherCard(result);
181+
}
182+
});
183+
```
184+
</Code>
185+
186+
<Aside data-type="note">
187+
Tool call arguments can be streamed token by token as they are generated by the model. When implementing token-level streaming, your UI should handle parsing partial JSON gracefully to render realtime updates as the arguments stream in. To learn more about approaches to token streaming, see the [token streaming](/docs/ai-transport/features/token-streaming) documentation.
188+
</Aside>
189+
190+
## Client-side tools <a id="client-tools"/>
191+
192+
Some tools need to be executed directly on the client device rather than on the server, allowing agents to dynamically access information available on the end user's device as needed. These include tools that access device capabilities such as GPS location, camera, SMS, local files, or other native functionality.
193+
194+
Client-side tool calls follow a request-response pattern over Ably channels:
195+
196+
1. The agent publishes a tool call request to the channel.
197+
2. The client receives and executes the tool using device APIs.
198+
3. The client publishes the result back to the channel.
199+
4. The agent receives the result and continues processing.
200+
201+
<Aside data-type="further-reading">
202+
For more information about bi-directional communication patterns between agents and users, see [Accepting user input](/docs/ai-transport/features/messaging/accepting-user-input) and [Human-in-the-loop](/docs/ai-transport/features/messaging/human-in-the-loop).
203+
</Aside>
204+
205+
The client subscribes to tool call requests, executes the tool using device APIs, and publishes the result back to the channel. The `toolCallId` enables correlation between tool call requests and results:
206+
207+
<Code>
208+
```javascript
209+
const channel = realtime.channels.get('{{RANDOM_CHANNEL_NAME}}');
210+
211+
await channel.subscribe('tool_call', async (message) => {
212+
const { name, args } = message.data;
213+
const { responseId, toolCallId } = message.extras?.headers || {};
214+
215+
if (name === 'get_location') {
216+
const result = await getGeolocationPosition();
217+
await channel.publish({
218+
name: 'tool_result',
219+
data: {
220+
name: name,
221+
result: {
222+
lat: result.coords.latitude,
223+
lng: result.coords.longitude
224+
}
225+
},
226+
extras: {
227+
headers: {
228+
responseId: responseId,
229+
toolCallId: toolCallId
230+
}
231+
}
232+
});
233+
}
234+
});
235+
```
236+
</Code>
237+
238+
<Aside data-type="note">
239+
Client-side tools often require user permission to access device APIs. These permissions are managed by the device operating system, not the agent. Handle permission denials gracefully by publishing an error tool result so the AI can respond appropriately.
240+
</Aside>
241+
242+
The agent subscribes to tool results to continue processing. The `toolCallId` correlates the result back to the original request:
243+
244+
<Code>
245+
```javascript
246+
const pendingToolCalls = new Map();
247+
248+
await channel.subscribe('tool_result', (message) => {
249+
const { toolCallId, result } = message.data;
250+
const pending = pendingToolCalls.get(toolCallId);
251+
252+
if (!pending) return;
253+
254+
// Pass result back to the AI model to continue the conversation
255+
processResult(pending.responseId, toolCallId, result);
256+
257+
pendingToolCalls.delete(toolCallId);
258+
});
259+
```
260+
</Code>
261+
262+
## Human-in-the-loop workflows <a id="human-in-the-loop"/>
263+
264+
Tool calls resolved by humans are one approach to implementing human-in-the-loop workflows. When an agent encounters a tool call that needs human resolution, it publishes the tool call to the channel and waits for the human to publish the result back over the channel.
265+
266+
For example, a tool that modifies data, performs financial transactions, or accesses sensitive resources might require explicit user approval before execution. The tool call information is surfaced to the user, who can then approve or reject the action.
267+
268+
<Aside data-type="further-reading">
269+
For detailed implementation patterns and best practices for human-in-the-loop workflows, including authorization and verification strategies, see the [human-in-the-loop](/docs/ai-transport/features/messaging/human-in-the-loop) documentation.
270+
</Aside>

0 commit comments

Comments
 (0)