Key takeaways
- Webhook delivery is commonly at least once, so consumers must expect the same notification more than once.
- A fast acknowledgement should record durable receipt before slower enrichment, routing, or notification work begins.
- Deduplicate the event and make each side effect idempotent; those are related but different controls.
- Replays, dead letters, and manual recovery need the same audit trail as the happy path.
01
The duplicate often begins as a healthy retry
A CRM sends a webhook. Your endpoint writes the contact, calls enrichment, and then times out before returning success. The sender cannot see the completed write, so it retries. If the second request creates another contact or sends the welcome message again, the failure belongs to the consumer design—not to the customer who submitted once.
HubSpot documents retries for connection failures, timeouts, and error responses, with failed notifications retried up to ten times. That behavior protects delivery. Your endpoint has to protect the business from repeated side effects.
02
Acknowledge receipt after durability, before orchestration
Verify authenticity, validate the envelope, and store the raw event or a minimal durable representation with a stable event key. Return success within the provider’s deadline, then process the event through a queue or worker. Do not hold the sender’s request open while an enrichment vendor, CRM write, or email service decides whether to respond.
- Record provider, subscription, event identifier, object identifier, event type, and occurred-at time.
- Use a unique constraint or atomic insert for the delivery key.
- Keep received, accepted, processed, suppressed, failed, and replayed states distinct.
- Store only the payload fields required for recovery and audit.
03
Make the business action idempotent too
Suppressing the same delivery identifier is not enough when two different events describe the same desired state. Upsert the CRM record by a governed identity key. Give outbound actions their own idempotency key, such as workflow plus contact plus milestone. Compare event version or occurred-at time before an older update overwrites a newer one.
Swipe to compare every column
| Layer | Key | Protected failure |
|---|---|---|
| Delivery | Provider event or subscription identifier | Exact notification arrives twice |
| Record | Governed CRM identity key | Two notifications create two people |
| Side effect | Workflow + entity + milestone | Email, task, or audience update repeats |
| State update | Object version or event time | Late event overwrites current state |
04
Practice replay without inventing a second history
Send duplicates, reversed event order, malformed payloads, slow dependencies, rate limits, and a crash after each step. A replay tool should show what will run, what is already complete, and which effects require explicit approval. Dead-letter records need an owner and expiry; an unreviewed queue is only a quieter failure.
Measure duplicate suppression, processing delay, retry count, stale-event rejection, recovery time, and customer-visible repeats. A reliable integration is not one that never sees duplicates. It is one where duplicates are ordinary input and still produce one intended outcome.
Primary sources and further reading
Use the source material to validate details against your own context and current platform configuration.
This field note follows the XenGrowth editorial policy: primary sources where available, visible limitations, material review dates, and no invented first-hand experience.
Stay with the problem
