Skip to content

server: summary-stats cache key is expression structure + path strings — stale stats served when parquet bytes change under a stable path #955

Description

@paddymul

Problem

_execute_cached's docstring says the summary-stats cache key is "content-addressed on query" (buckaroo/pluggable_analysis_framework/xorq_stat_pipeline.py:192). It is not. The cache is a xorq ParquetSnapshotCache (buckaroo/server/xorq_loading.py:43-45) whose SnapshotStrategy normalizes every Read leaf by path identity only — no mtime, no size, no content (xorq caching/strategy.py:49-69); xorq documents the class itself as "source data changes do not invalidate cached results" (caching/__init__.py:150-156). The lookup is filename existence, and stored bytes are never revalidated (xorq_stat_pipeline.py:197-209).

Impact

Load the same build dir twice with the same cache_storage_path, with any parquet reachable from the expression rewritten at the same path in between — exactly what an upstream cache heal/regeneration produces. calc_key returns the identical string, the file exists, and the previous dataset's min/max/nunique/histograms render beside the new rows, counted as a cache hit with no warning. Downstream, tallyman's per-entry stat caches and expanded-build paths are stable by design, so this is live there (see buckaroo-data/tallyman#163 and its audit).

Suggested fix

Fold a content signal into the key — e.g. a digest of (path, st_size, st_mtime_ns) for every leaf Read/CachedNode the stat query reaches — or switch the stats cache to ModificationTimeStrategy/ParquetCache, which already keys on file stat. Delete the "content-addressed" docstring in the same change; the comment is what keeps the bug plausible.

Context

Found while auditing buckaroo 0.15.4 (3313534a) for the cache-soundness bug class behind buckaroo-data/tallyman#163.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions