Key takeaways
- Retry only failures that may be transient and only when the operation is safe to repeat.
- Use stable idempotency keys and external identifiers for create-like operations.
- Record attempts and outcomes so operators can distinguish delayed, failed, and duplicated work.
- Run reconciliation because a successful HTTP response does not prove the business state stayed correct.
01
A timeout does not tell you whether the write happened
The CRM may commit a record and lose the response on the way back. A client that blindly retries can create another contact, task, or opportunity. The correct response depends on whether the operation is idempotent: repeated execution must lead to the same observable state.
Google Cloud’s retry guidance separates retryable responses from idempotency and warns that retrying non-idempotent requests can create races and conflicts. Use that distinction for every integration action, not one blanket retry setting.
Swipe to compare every column
| Operation | Repeat risk | Safer pattern |
|---|---|---|
| Read record | Usually low | Retry transient failures with bounded backoff |
| Create lead | Duplicate object | Stable idempotency key or upsert by external ID |
| Increment score | Double increment | Write computed target value with version check |
| Append activity | Duplicate event | Unique event ID and duplicate constraint |
02
Give each business event a durable identity
Generate an idempotency key from the source event or store a stable unique identifier before the first attempt. Persist the request fingerprint, destination, status, response identifier, and attempt history. A process restart must recover the same key instead of inventing a new one.
Where supported, use upsert, ETags, version numbers, or conditional writes. Do not rely on fuzzy matching after the fact. Two people can share a name, and an email address can change; identity rules need an explicit source of truth and merge policy.
03
Retry politely and stop for permanent errors
Use bounded exponential backoff with jitter for timeouts, disconnects, rate limits, and server errors the destination documents as transient. Respect Retry-After. Authentication failures, invalid fields, revoked consent, and schema mismatches need correction or review rather than a faster loop.
Cap attempts and total elapsed time, then place unresolved events in a visible queue with the payload reference, reason, owner, and safe replay control. Logs should share a correlation ID across the source event, queue, worker, and destination call.
04
Reconcile the business objects on a schedule
Compare source and destination counts, identifiers, update times, key values, and terminal statuses. Sample both successes and failures. Watch for orphaned child records, events processed out of order, and records updated manually while a delayed job was waiting.
Test disconnect-after-commit, duplicate delivery, worker restart, rate limit, stale version, reordered events, and partial batch failure. Reliability is not the absence of errors; it is the ability to reach a correct, explainable state after them.
Primary sources and further reading
Use the source material to validate details against your own context and current platform configuration.
This field note follows the XenGrowth editorial policy: primary sources where available, visible limitations, material review dates, and no invented first-hand experience.
Stay with the problem
