What is an idempotency key?

A unique value the client sends with a request that changes data, such as creating a payment. The server stores the key with the result; if the same key arrives again, it returns the stored result rather than repeating the action. Many payment APIs support idempotency keys, and your own APIs should too for important operations.

How should retries be configured?

Retry only transient failures, such as timeouts and certain server errors, not validation errors. Use exponential backoff with jitter, cap the number of attempts, respect rate limit headers, and make retried operations idempotent. Aggressive retries during an outage add load and can make it worse.

What is a circuit breaker?

A pattern that stops calling a failing dependency for a while after repeated failures, returning a fallback or error quickly instead of waiting on timeouts. After a cool-down it tries again. It protects your system's capacity and gives the dependency time to recover.

Why is reconciliation needed if integrations are reliable?

Because no integration is perfect: messages can be lost, duplicated or applied out of order, and external systems can change data independently. Regular reconciliation compares records between systems, such as orders, payments and stock, and surfaces differences for correction before they become customer or financial problems.

Should we use webhooks or polling?

Use webhooks when the provider offers them and freshness matters, and verify signatures, handle duplicates and out-of-order events. Use polling when webhooks are unavailable or changes are infrequent. For important data, combine them: webhooks for speed and scheduled polling or reconciliation to catch anything missed.

Web Development6 min read

API Integration Reliability: Timeouts, Retries, Idempotency and Reconciliation

Making API integrations reliable: timeouts, retries with backoff, idempotency keys, rate limits, circuit breakers, queues, reconciliation and observability.

How do you make API integrations reliable?

Set timeouts on every call, retry only transient failures with exponential backoff and jitter, make every data-changing operation idempotent so retries cannot create duplicates, respect rate limits, use circuit breakers for failing dependencies, move non-urgent work to queues with dead-letter handling, reconcile records between systems regularly, and monitor every integration. Retry logic without idempotency can double-charge customers or amplify an outage.

This guide is for developers and architects building integrations between business systems, payment providers, ecommerce platforms and partner APIs. For the wider integration picture in Australia, see API integration for Australian businesses.

Key takeaways

Every call has a timeout appropriate to the operation.
Retries are limited, backed off, jittered and only for transient errors.
Data-changing operations use idempotency keys.
Rate limits are respected, with queuing rather than hammering.
Circuit breakers protect your system from failing dependencies.
Reconciliation catches what reliability patterns miss.

Timeouts and retries

Failure	Retry?	Notes
Connection timeout or reset	Yes, with backoff	Only if the operation is idempotent
HTTP 429 rate limited	Yes, after the indicated delay	Respect Retry-After headers
HTTP 5xx server error	Usually, limited	Some 5xx are not transient; cap attempts
HTTP 4xx validation or auth error	No	Fix the request or credentials

Idempotency

Send a unique idempotency key with each request that creates or changes data, and store results keyed by it on the server side. Payment providers document this pattern; Stripe, for example, supports idempotent requests so that retrying with the same key returns the original result (Stripe). For incoming webhooks, store processed event IDs and ignore repeats; see webhook design and payment gateway integration.

Rate limits, circuit breakers and queues

Track each partner's rate limits and pace requests below them.
Use circuit breakers so repeated failures fail fast and trigger fallbacks (martinfowler.com).
Put non-urgent work, such as syncing records or sending notifications, on queues.
Process queue messages idempotently; they may be delivered more than once.
Send repeatedly failing messages to a dead-letter queue with alerts and an owner.
Design for out-of-order messages using timestamps or version numbers.

Pro tip

Our recommendation: write down, for each integration, what happens if the other system is down for an hour. If the answer is 'we lose data' or 'customers are charged twice', fix that before adding features. See queue architecture.

Reconciliation

Schedule reconciliation jobs that compare key records between systems, such as orders in the store and the ERP, payments and payouts, stock levels, and invoices in accounting, and report differences to owners. Fix root causes, not just individual differences. Reconciliation is also your evidence that integrations work.

Observability

Give each integration request a correlation ID, log requests and outcomes without sensitive data, and track success rate, latency, retries, queue depth and dead-letter counts. Alert on sustained failures and growing backlogs, with runbooks. See application monitoring and observability.

Common mistakes

No timeouts, so threads hang on a slow partner.
Retrying payments without idempotency keys.
Unlimited immediate retries during an outage.
Dead-letter queues nobody watches.
No reconciliation, so silent failures accumulate.

Webhooks, polling or both

Webhooks push changes to you as they happen; polling asks the other system for changes on a schedule. Webhooks are faster and lighter but can be missed, duplicated or delivered out of order. Polling is simple and predictable but slower and uses more API calls.

Approach	Use when	Watch for
Webhooks	The provider supports them and freshness matters	Signature checks, duplicates, ordering, outages on your side
Polling	No webhooks, or changes are infrequent	Rate limits, 'updated since' accuracy, delay
Both	Important data must not be missed	Webhooks for speed, scheduled polling or reconciliation as a safety net

Testing integrations with third-party services

Third-party APIs fail in ways your own code does not: sandboxes differ from production, rate limits change and responses drift. Test at several levels.

Unit tests with recorded responses for parsing and mapping logic.
Contract tests that check the fields and formats you depend on, run against the provider's sandbox on a schedule.
Fault injection for timeouts, 429 and 5xx errors, duplicate webhooks and out-of-order events.
End-to-end tests in sandboxes before releases; note where sandboxes behave differently from production.
Synthetic monitoring in production: a small, safe transaction or read checked regularly with alerts.
See automated regression testing and API versioning.

Conclusion

Reliable integrations assume failure: they time out, retry carefully, never duplicate actions, slow down when partners struggle, queue what can wait, and reconcile to catch the rest. Build these patterns in from the first integration, and monitor them like any other production system.

Integrations dropping or duplicating data?

ZSpace Labs is an India-based, remote-first technology studio working with Australian and international businesses on reliable integrations and automation. If it would help to review your integration failure modes, we are happy to talk it through.

Start a Project

FAQ

Common questions.

Because a request can succeed on the server while the response is lost, for example through a timeout. If the client retries without an idempotency key, the server processes it again, creating a duplicate order, payment or record. Idempotency keys let the server recognise a repeated request and return the original result instead of acting twice.