[codex] Fix agent GenAI columns and chat refs

[bcsherma]

Description

Fixes the Agent spans/query and chat surfaces for GenAI spans.

• Adds cache token fields to the default and chat/detail projections so cache read/write token values can be fetched back, not just used in stats histograms.
• Extends the agent span schema/query allowlist to cover persisted GenAI typed columns plus raw_span_dump when explicitly requested.
• Normalizes ref-like strings inside GenAI OTLP protobuf attributes at the external ingest boundary, so weave:///... refs are stored internally as weave-trace-internal:///... before ClickHouse insert.
• Adds focused query-builder and converter regression tests.

Root cause:

• Cache token columns and several typed GenAI fields existed in persisted spans/stats but were missing from the span list/detail projection path.
• GenAI OTLP ingest converted the request project_id and wb_user_id, but did not walk refs embedded inside protobuf AnyValue attributes. Those external refs were stored directly in span fields such as content_refs. Later, the public response adapter correctly treated external refs in an internal response as invalid and raised InvalidInternalRef, surfacing as a 500.

Testing

• uv run pytest tests/trace_server/test_trace_server_interface_util.py tests/trace_server/test_trace_server_converter.py tests/trace_server/query_builder/test_agent_query_builder.py
• Verified new GenAI OTLP ingest stores seeded content_refs / object_refs as weave-trace-internal:///... in local ClickHouse.
• Verified locally against tilt prod_weavetrace: /agents/traces/chat and /agents/conversations/chat returned 200 for newly seeded bcanfieldsherman/agents-lorem-ipsum rows.

[bcsherma]

• [codex] Fix agent GenAI columns and chat refs #6899 👈 (View in Graphite)
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

[codecov]

Codecov Report

❌ Patch coverage is 92.30769% with 2 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
.../trace_server/query_builder/agent_query_builder.py	85.71%	0 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!

[wandbot-3000]

Preview this PR with FeatureBee: https://beta.wandb.ai/?betaVersion=b2c0c8fa3bb8c877bd5ea55a2fe7ad6e20d3b6cb