Skip to content

feat(ai-service): Replace task_results global dict with Redis-backed task store#362

Closed
AugistineCreates wants to merge 1 commit into
ChainForgee:mainfrom
AugistineCreates:feat/redis-task-store
Closed

feat(ai-service): Replace task_results global dict with Redis-backed task store#362
AugistineCreates wants to merge 1 commit into
ChainForgee:mainfrom
AugistineCreates:feat/redis-task-store

Conversation

@AugistineCreates

Copy link
Copy Markdown
Contributor

Description

This PR replaces the module-level in-memory dictionary task_results in the AI Service (app/ai-service/tasks.py) with a persistent, Redis-backed task store.

Why this is necessary:

  • Resilience: A single-process in-memory dict gets destroyed whenever a pod/container restarts, losing active status tracking.
  • Scalability: In multi-process deployments, workers run in forked/different processes and cannot share status updates via a local in-memory dict, causing fork-bomb state mismatches.

Changes Made

  • Removed task_results: Deleted the task_results global dictionary.
  • Lazy Redis Client: Added lazy initialization of the Redis client (get_redis_client) targeting the configured REDIS_URL.
  • Status Persistence: Implemented tasks.set_status to write task payload states directly to Redis with a TTL of 24 hours (86400 seconds) matching Celery's result_expires.
  • Robust Retrieval: Updated tasks.get_task_status to check Celery's AsyncResult status first, and fallback to the Redis-backed store on failure or missing keys.
  • Unit & Integration Tests: Created app/ai-service/tests/test_tasks_redis.py covering:
    • Setting status to Redis with precise TTL.
    • Graceful fallback reading from Redis when Celery returns unfinished states or throws connection errors.
    • Correct E2E cross-process communication simulation (Process A writing to Redis, Process B observing status).

Acceptance Criteria Verified

  • Redis storage and 24-hour TTL enforced.
  • Reads query Celery AsyncResult first and fallback to Redis.
  • E2E behavior verified (Process A updates state, Process B reads state).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants