In this guide 6 sections
A well-built integration detects the outage, queues the affected work instead of dropping it, retries within limits and reconciles with the provider once service returns. Without that design, a payment gateway outage can leave orders stalled for hours or charge customers twice the moment service comes back.
Detecting outages and telling authentication from provider failures
Consider an online marketplace seller mid-sale whose payment gateway starts failing: from inside the seller’s system, a provider outage and a broken credential look identical. Telling them apart comes first.
A provider outage needs queueing and retries, while an expired key or changed setting needs fixing now, so mistaking one for the other wastes the busiest hours. Check the provider’s status page first. A heartbeat call every minute, one lightweight request that only checks the provider answers, tells you the moment an outage starts rather than when the first customer complains. A brownout, where calls succeed one minute and time out the next, is harder to spot than a clean outage, so alert on a rising failure rate.
Send a test call with a fresh key and compare the error code against its documentation, because a rejected credential often returns a different code from an unavailable provider.
Queueing work and avoiding duplicate execution
When the gateway is unreachable, a checkout attempt should be held in a queue, marked pending and retried automatically once the provider returns, instead of failing and vanishing. The customer does not start over. Nor do they risk a second charge if the first attempt went through moments before the failure was detected, which is exactly the case a unique reference settles.
Queued work needs duplicate prevention built in. Give every payment attempt a unique reference and check it before any retry runs, because an outage is the moment duplicate charges and duplicate orders most easily slip through.
Make the queue visible to staff. A pending list that a support agent can open, showing each held order with its reference and retry count, answers a customer’s question and exposes a stuck item.
Bounded retries and communicating degraded service
Retries must stop somewhere. Use a fixed number of attempts with exponential backoff, say seconds, then a minute, then several minutes, because an endless loop hammers a struggling provider and leaves checkouts hanging. Add jitter to every delay. Otherwise every queued attempt retries at the same second, and the burst of requests when the provider returns can knock it over for a second time.
Talk to customers too. A clear message such as ‘payments are delayed, we’ll confirm shortly’ beats a generic error, because customers who assume failure often pay again by another method.
Reconciling after recovery and reviewing missed events
When the provider returns, process each queued item and check it against the provider’s record of the outage window, since some payments completed there without your system hearing. This pass closes the outage cleanly. Record the outage window exactly. The first failed call and the first successful one bound the orders to inspect, the refunds to double-check and the customers who deserve a short apology email. Hold a short postmortem within the week, listing what was queued, what was lost, which alert fired first and which change would have shortened the gap.
Skip it and you risk a customer charged for an order that never shipped, or an order shipped that was never paid for. Review the mismatch list deliberately, with a named person signing it off, and do not assume the queue alone caught every payment, refund or cancellation that overlapped the outage.
Provider changes and continuity alternatives
Some businesses stop trading when one provider fails, whether it is a payment gateway, a messaging API or a courier’s tracking feed. For those, ask whether a backup provider for the same function is worth its added complexity, especially if peak sales are exactly when an outage hurts most.
Not every integration needs a spare. A low-stakes connection rarely justifies a second provider, but anything on the critical path of payment or order confirmation deserves a deliberate answer before an outage forces one.
Test the fallback regularly. Keep each provider’s queue in its own bulkhead, so a stalled messaging queue cannot starve the checkout queue behind it for the hour that the messaging provider spends recovering. A backup provider that has never carried a payment is a hope and not a plan, so route a small slice of traffic through it on a quiet day.
Outage-to-reconciliation flow
A flowchart runs from outage detection through queueing, bounded retries and a customer-facing status message, then provider recovery, ending in a reconciliation step. That final step compares your queued records with the provider’s transaction log and catches anything that completed on one side but not the other.
Open a blank worksheet to printQuestions owners ask
Can operations continue during an outage?
Often yes, in a reduced form. Orders can still be taken and queued while a payment or messaging provider is down, provided the system was built to hold and retry that work. Full continuity depends on how central the provider is, because a courier feed can wait for an hour while a payment step cannot.
How do we avoid duplicate transactions on recovery?
Check each queued item against its unique reference before processing it, then reconcile with the provider’s record of what happened during the outage. Some attempts may have succeeded on the provider’s side even though your system never received confirmation, and only the reconciliation pass reveals them.
How long should we keep retrying?
Stop after a fixed number of attempts, then move the item to a visible exception list for a person to handle, and tell the customer where their order stands. Endless retries only hide the problem.