Set retry cache TTL based on task execution time and retry backoff

[yedidyakfir]

Summary

The SignatureRetryCache currently uses a hardcoded 24-hour TTL during execution and a 5-minute short TTL on teardown. This should be made dynamic based on:

1. Task execution time — the cache should live at least as long as the task takes to execute, so retries can find it.
2. Retry backoff window — for exponential backoff, the time between retries grows. The cache TTL should account for the wait time before the next retry attempt, ensuring the cache is still available when the retry fires.

Current Behavior

# retry_cache.py
class SignatureRetryCache(AtomicRedisModel):
    Meta: ClassVar[RedisConfig] = RedisConfig(ttl=24 * 60 * 60, refresh_ttl=False)  # hardcoded 24h

async def teardown_retry_cache(state: RetryCacheState):
    await state.cache.aset_ttl(5 * 60)  # hardcoded 5 min

Desired Behavior

• Active TTL: Set based on execution_time + next_retry_backoff_delay + buffer. For example, if a task takes 10 minutes and the next retry is in 30 minutes (exponential backoff), the TTL should be at least ~45 minutes.
• Teardown TTL: Instead of a flat 5 minutes, extend it to cover the expected wait time until the next retry attempt fires. This prevents the cache from expiring before the retry can use it.
• If no retry info is available, fall back to the current defaults.

Use Cases

• Long-running tasks with exponential backoff: A task that takes 5 minutes but has a 1-hour backoff between retries would lose its cache with a 5-minute teardown TTL.
• Fast tasks with aggressive retries: A 2-second task retrying every 5 seconds doesn't need a 24-hour cache — a shorter TTL saves Redis memory.

Implementation Considerations

• The attempt_number is already available in setup_retry_cache. Hatchet may expose retry configuration (max retries, backoff strategy) that can be used to compute the expected next-retry delay.
• Consider making the TTL formula configurable via MageflowConfig so users can tune it.
• Must handle edge cases: last retry attempt (no next retry → short TTL), unknown backoff config (fall back to defaults).

Related

• Handle deleted signatures in retry cache #90 — Handle deleted signatures in retry cache (stale cache references)
• Fix: Retry cache uses workflow_id as key, causing collisions between concurrent tasks #109 — Fix retry cache key collisions between concurrent tasks

List tasks

• Compute expected next-retry delay based on attempt number and backoff strategy
• Set active cache TTL to execution_time + next_retry_delay + buffer
• Set teardown TTL to next_retry_delay + buffer instead of flat 5 minutes
• Add TTL configuration options to MageflowConfig for retry cache tuning
• Fall back to current defaults when retry/backoff info is unavailable
• Add tests for TTL calculation with various backoff strategies