Build a bounded event deduplication engine with deterministic fingerprints

[Lakes41]

Difficulty: Advanced
Type: Feature
Recommended labels (if available in this repo): events, backend, consistency, advanced

Background

GuildPass Core may eventually consume events from multiple sources, including internal outbox processors, Stellar integrations and background jobs. Duplicate delivery is a normal possibility in distributed systems and should not automatically result in duplicate processing.

This issue introduces a standalone deduplication primitive with an in-memory implementation so it can be completed independently.

Problem

There is currently no shared Core utility for recognising recently processed event fingerprints and suppressing duplicate handling within a defined time window.

A naive set of seen IDs can grow without bound and does not define expiry semantics.

Expected Outcome

Implement a framework-independent event deduplication engine with bounded memory, explicit expiry and atomic check-and-record behaviour.

Suggested Implementation

The engine should:

• accept a deterministic event fingerprint or caller-provided event ID;
• atomically determine whether the event has already been observed;
• record new fingerprints with an expiry time;
• return a typed result describing whether processing should continue;
• support configurable retention windows;
• clean up expired entries;
• enforce a maximum retained-entry limit or another explicit memory bound;
• define eviction behaviour when capacity is reached;
• support concurrent checks against the same fingerprint safely;
• allow time injection for deterministic tests;
• avoid depending on Redis or a database.

A future Redis adapter should be possible without changing the core decision semantics.

Acceptance Criteria

• A new fingerprint is accepted for processing.
• A duplicate fingerprint inside the retention window is rejected as duplicate.
• An expired fingerprint can be accepted again according to documented semantics.
• Concurrent attempts using the same fingerprint result in at most one first-seen outcome.
• Expired entries are cleaned up.
• Memory usage is explicitly bounded.
• Capacity-eviction behaviour is deterministic or clearly documented.
• Invalid fingerprints are rejected.
• Time-dependent tests use an injectable clock or equivalent approach.
• Unit tests cover duplicate, expiry, capacity and concurrency behaviour.
• pnpm typecheck passes.
• pnpm build passes.
• pnpm test passes.
• No Redis, database or event-consumer integration is included.

Likely Affected Files/Directories

packages/event-deduplication/ or equivalent isolated package
unit tests

Independence Requirement

This issue must be independently implementable from main and must not require the transactional outbox or any event processor to exist.

[Santia2004]

Hi team! I’d love to take on this issue. I’ve previously contributed and had multiple PRs successfully merged in this repository, so I’m already well-familiar with the codebase, architecture, and workflow. Could you please assign this to me? Thanks!

[zzzbedream]

i can solve that

[guptakumarranjeet150]

Hello @maintainers,

I can tackle Build a bounded event deduplication engine with deterministic fingerprints. Proposed implementation roadmap:

• Analysis: Inspect current behavior and edge cases in Adamantine-guild/guildpass-core
• Implementation: Build Build a bounded event deduplication engine with deterministic fingerprints with clean React/Next.js components, responsive styling, and proper loading/error boundaries.
• Testing: Add component unit tests and ensure zero TypeScript/ESLint warnings.
• PR: Atomic commit with clean git hygiene and passing CI

Happy to pick this up today—please assign to me!