Skip to content

Two $ai_generation fidelity bugs in the Claude Code session ingest #92

@mory91

Description

@mory91

Hi! Two small bugs in the Claude Code session-end ingest path that affect what PostHog sees on $ai_generation events. Both reproduce on the current main (a004aa2). Happy to PR if useful.

1. Cache token properties missing the $ai_ prefix

posthog_llma/events.pybuild_ai_generation emits:

properties = {
    ...
    "cache_read_input_tokens": cache_read_tokens,
    "cache_creation_input_tokens": cache_creation_tokens,
}

PostHog's LLM Analytics cost pipeline only reads the $ai_* namespace (docs call out $ai_cache_read_input_tokens / $ai_cache_creation_input_tokens), so cache reads/writes are dropped on the floor and $ai_total_cost_usd is computed from input+output only.

For Anthropic prompt-cached workloads the cache buckets dominate the bill, so reported cost can be ~25x lower than the actual spend. Concrete example from a real session: cache_read=150,109, cache_write=75,729, input=4, output=926 — PostHog showed ~$0.014, actual cost ~$0.343.

2. Extended thinking is emitted as {"type":"text"} instead of {"type":"thinking"}

posthog_llma/parser.py_finalize_generation concatenates thinking and text blocks into a single output_text string, and event_builder.pybuild_events wraps that as one text block:

content_blocks = []
if gen["output_text"]:
    content_blocks.append({"type": "text", "text": gen["output_text"]})

So when a Claude Code generation has both thinking and text blocks (common with extended thinking enabled), thinking content appears as plain assistant text in the PostHog UI instead of a dedicated thinking block — losing the visual distinction and breaking property queries that filter on block type. The streaming-merge work from #86 already tracks blocks by type in state["blocks_by_type"], so the typing info is right there in _finalize_generation, just collapsed before emission.

Thanks!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions