Ask HN: How do you handle duplicate side effects when jobs, workflows retry?

Quick context: I'm building background job automation and keep hitting this pattern:

1. Job calls external API (Stripe, SendGrid, AWS) 2. API call succeeds 3. Job crashes before recording success 4. Job retries → calls API again → duplicate

Example: process refund, send email notification, crash. Retry does both again. Customer gets duplicate refund email (or worse, duplicate refund).

I see a few approaches:

Option A: Store processed IDs in database Problem: Race between "check DB" and "call API" can still duplicate

Option B: Use API idempotency keys (Stripe supports this) Problem: Not all APIs support it (legacy systems, third-party)

Option C: Build deduplication layer that checks external system first Problem: Extra latency, extra complexity

What do you do in production? Accept some duplicates? Only use APIs with idempotency? Something else?

(I built something for Option C, but trying to understand if this is actually a common-enough problem or if I'm over-engineering.)

4 points | by shineDaPoker 6 hours ago

3 comments

  • stephenr 1 hour ago
    I think the answer is probably like most things: it depends.

    - If the external service supports idempotent operations, use that option.

    - If the external service doesn't, but has a "retrieval" feature (i.e. lookup if the thing already exists, e.g fetch refunds on a given payment), use that first.

    - If the system has neither, assess how critical it is to avoid duplicates.

  • babelfish 4 hours ago
    Use something like Temporal
  • moomoo11 5 hours ago
    You proxy those api calls yourself and have idempotency to cover you for those APIs that don’t have it. If you architect it right you won’t have more than a ms latency addition. You can avoid the race condition issues by using atomic records so if something else tries they’d see it’s in progress and exit.