[FEAT] Ordered Webhook Delivery with Dead Letter Queue

[oomokaro1]

[FEAT] Ordered Webhook Delivery with Dead Letter Queue

Priority: High

Difficulty: Hard
Estimated Effort: 3-4 days
Relevant Packages: OrbitStream_backend/, orbitstream_docs/
Labels: enhancement, infrastructure, priority:high

Requirements

1. Ordered Delivery

• Add a sequence column to webhook_deliveries table (auto-increment per session)
• When dispatching webhooks for the same session, process them sequentially
• Use a per-session queue within the main queue
• If webhook N for session X fails, block webhook N+1 for session X until N succeeds or goes to dead letter

2. Dead Letter Queue

• After 5 failed attempts, move delivery to a webhook_dead_letter table
• Dead letter entries include: all delivery attempts with timestamps and error messages
• Merchants can view dead letter entries via GET /v1/webhooks/dead-letter
• Merchants can manually retry via POST /v1/webhooks/dead-letter/:id/retry
• Merchants can dismiss via DELETE /v1/webhooks/dead-letter/:id

3. Delivery Log

• GET /v1/webhooks/deliveries — list recent deliveries (last 100, paginated)
• Filter by: event type, status (delivered/failed/pending), date range
• Each entry shows: event, status, attempts, last error, delivered_at

4. Idempotency Keys

• Each delivery gets X-OrbitStream-Delivery-Id (UUID v4)
• Each delivery gets X-OrbitStream-Timestamp (ISO 8601)
• Signature covers: delivery_id + timestamp + payload
• Document idempotency in integration guide

5. Testing

• Test ordering: webhooks for same session arrive in order
• Test dead letter: failed deliveries move to DLQ after 5 attempts
• Test manual retry: dead letter entry is retried successfully
• Test idempotency: duplicate delivery IDs are handled

[Jenola344]

Built ordered webhook delivery with DLQ and idempotency before; reviewed your current architecture ready to implement robustly with full test coverage

[Jaydbrown]

Looked through the source before jumping in. The queue is already doing a lot of what this needs — session ordering via the per-session zset and inflightSessions is solid, and the idempotency headers (X-OrbitStream-Delivery-Id, X-OrbitStream-Timestamp, HMAC signature over deliveryId.timestamp.payload) are already wired in the delivery service. No gaps there.

What's actually missing is on the delivery log side: GET /deliveries has no filtering and there's no endpoint to pull up a single delivery's attempt history. The DLQ data model is in the schema but the controller endpoints aren't there yet.

For the list endpoint I'd add event, status, after, and before query params:

const conditions = [eq(webhookDeliveries.merchantId, merchantId)];
if (event)  conditions.push(eq(webhookDeliveries.event, event));
if (status) conditions.push(eq(webhookDeliveries.status, status));
if (after)  conditions.push(gte(webhookDeliveries.createdAt, new Date(after)));
if (before) conditions.push(lte(webhookDeliveries.createdAt, new Date(before)));

And GET /deliveries/:id scoped to the merchant so a specific delivery's full attemptLog is accessible — handy when a merchant needs to debug exactly why something failed.

Happy to take this if it's still open.

[grantfox-oss]

🦊 GrantFox — @Jaydbrown has been assigned to this issue as part of the Official Campaign campaign!

Next steps:

1. Open a Pull Request referencing this issue (e.g., Closes #17)
2. Your PR will be reviewed by the OrbitStream maintainers

Good luck! Track your progress on GrantFox.