feat: add tenant-aware token bucket rate limiter

[onyekachi66]

Closes #659

Background & Context

Currently, single high-volume regional liquidity pools can overwhelm our processing queues (Issue #649). Because telemetry processing capacity was not strictly isolated per tenant/asset feed, an influx of requests from one region could degrade or entirely exhaust the throughput available to other regions. We needed an equitable distribution mechanism to guarantee that each configured asset feed maintains its fair share of processing bandwidth, even under load spikes from neighbors.

What Does This PR Do?

This PR introduces a thread-safe TokenBucketRateLimiter class configured to regulate request flow on a per-tenant basis.

1. Token Bucket Algorithm Implementation:

   • Added TokenBucketRateLimiter to src/queue/backpressure.py.
   • The class is parameterized by capacity (maximum burst allowed) and refill_rate (sustained token regeneration per second).
   • Tracks limits dynamically using a per-tenant dictionary protected by a threading.Lock() to ensure safe concurrent operations in asynchronous environments.
   • Computes state lazily based on time.monotonic() to evade continuous background loop overhead.
   • Provides a retry_after_s metric when requests are blocked, enabling downstream clients/queues to pause or delay processing intelligently rather than dropping data outright.

2. Equitable Tenant Distribution:

   • Limits are resolved by tenant_id uniquely, meaning exhaustion of tokens for one tenant completely shields all other tenants operating well within their limits.

3. Automated Verification:

   • Added robust pytest cases in tests/test_backpressure.py.
   • test_multitenant_token_bucket: Validates strict tenant isolation—tenant A can be fully blocked while tenant B operates normally.
   • test_token_bucket_refill: Asserts that mathematical token regeneration strictly adheres to the configured refill_rate over fractional seconds.

Impact & Expected Outcomes

• High Severity Addressed: Directly mitigates the high-impact issue where a single noisy liquidity pool could disrupt regional equity.
• Performance: The thread-safe, lock-guarded lazy evaluation algorithm introduces negligible overhead to the processing pipeline.

Testing Instructions for Reviewers

To verify this behavior locally, you can run the following test suite targeting the multitenant and refill functionality:

pytest tests/test_backpressure.py -k test_multitenant_token_bucket

All tests are confirmed green and pass locally.

Related Issues

Closes #659

[drips-wave]

@onyekachi66 Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits