A customer writes on WhatsApp at 21:40 asking for a quote. The automation that normally qualifies the enquiry, creates a CRM record and sends a reply has been failing since 20:15 because the AI provider is returning errors. Nobody on the team knows. The customer waits, writes to a competitor, and the next morning the manager finds an empty queue and assumes it was a quiet evening.
That is the usual shape of an automation failure. It is rarely dramatic. It is silent, partial and noticed late, and the expensive part is often not the outage itself but what surrounds it: messages that were not processed, orders created twice, payment links sent to the same person three times, a CRM that disagrees with the payment system.
This article is a recovery plan for that situation: what breaks, how the standard protections work, what to do in the first hour, how to restore and reconcile, what European rules say, and how to judge whether paid monitoring is worth it.
Failures are normal, and the tail is long
Even the largest infrastructure providers publish post-mortems.
On 19 and 20 October 2025, Amazon Web Services had an event in its US-EAST-1 region. AWS's own summary says it started at 11:48 PM PDT on 19 October and ended at 2:20 PM PDT on 20 October, about 14.5 hours. It names the root cause as "a latent race condition in the DynamoDB DNS management system". DynamoDB error rates were elevated for about 2 hours 52 minutes, but services built on top of it, including compute launches, load balancers and Lambda, took many more hours to recover.
On 18 November 2025, Cloudflare reported core traffic failing from 11:20 UTC, largely flowing by 14:30 and all systems normal at 17:06. It states the cause was not a cyber attack: a database permissions change produced a file larger than expected, and the software reading it crashed. Users saw HTTP 5xx errors.
One older event shows what to avoid. In its summary of the 7 December 2021 event, AWS says retries from affected services added more load and kept the congestion going. A system that retries blindly can slow its own recovery.
Two conclusions follow. You cannot predict the day, but you can plan for the outage. And the end of the provider's incident is not the end of yours, because backlogs and half-finished work remain.
Where an AI automation can fail
| Layer | What breaks | What the customer or team sees | First control |
|---|---|---|---|
| Hosting and server | The machine or platform running the workflows is down or out of memory | Nothing happens; no replies, no records | Uptime check and restart procedure |
| AI provider | Rate limit, server error, overload or a spending cap | Replies slow, empty or missing | Bounded retries, fallback reply, spend alerts |
| Channel API | WhatsApp, Instagram, email or telephony stops delivering or accepting messages | Messages not arriving or not leaving | Acknowledge fast, queue, reconcile from the channel inbox |
| CRM or database | Records cannot be written or read | Duplicate or missing contacts, stale data | Write to a holding queue, replay later |
| Payments and invoicing | A request times out and is repeated | Double payment links, double charges, missing confirmations | Idempotency keys and an own ledger |
Two further failures do not look like outages: the automation runs but produces wrong output after a prompt, model or API change, or it stops triggering and raises no error because nothing ran.
What provider documentation says about errors
OpenAI's error code guide says to follow the Retry-After header on a 429 rate limit when present, and otherwise to use exponential backoff with jitter and limit the number of retries. It adds that retrying billing, spend or quota errors will not restore access.
Anthropic's errors documentation separates transient rate limits from a spend cap, where a 429 has no retry-after header and "keeps failing until access resumes". It recommends exponential backoff for 500 and 529 overload errors, and notes that an error can occur after a 200 response in a streaming call and is not handled by the standard retry mechanisms.
The pattern is shared: temporary errors deserve a bounded retry, while decisions such as a spend limit do not, and the automation should stop retrying those and alert a person.
Three protections that prevent the worst outcomes
1. Acknowledge fast, process later
Stripe's webhook documentation says to return a 2xx status before any complex logic and to process events through an asynchronous queue. Meta's webhook guidance says that if your endpoint does not return 200, Meta retries right away and then a few more times with decreasing frequency over 36 hours, after which unacknowledged notifications are dropped. Stripe retries live events for up to three days and lets you resend them manually for a limited period (15 days from the Dashboard, 30 days from its CLI). These are safety nets, not backups. A design that waits for a slow AI step before acknowledging invites timeouts and repeated deliveries, so receive the event, store it, acknowledge, and let a worker do the AI step. The order matters: return success only after the write to durable storage is confirmed. If the queue cannot be written to, return an error status so the provider retries within its window. Acknowledging an event you have not stored is how events are lost.
2. Make every action safe to repeat
An idempotent action has the same effect whether performed once or twice. Stripe's idempotent requests documentation explains that the client supplies a unique key, so a request repeated after a connection error does not create a second object. In practice the key is created once per business action, stored, and reused on every retry; a new key per attempt defeats the purpose. One detail matters for recovery: for API v1, Stripe says keys can be removed once they are at least 24 hours old, after which a reused key generates a new request. After a long outage a late retry may therefore not be protected by the provider, so keep your own ledger that says "payment link for order 4471 created" and check it before any retry. Stripe also warns that webhooks can arrive more than once and out of order, and advises logging processed event IDs.
3. Stop and alert
When an action fails after a limited number of retries it should land in a holding place, often called a dead-letter queue, where a person can see it, fix the cause and replay it. In n8n, the error handling documentation describes an error workflow that starts with an Error Trigger and runs when an execution fails, plus a Stop and Error node for failing an execution deliberately. Other platforms differ; see the comparison of n8n, Make and Zapier. An error workflow reports executions that ran and failed. By reasoning, not by a documented limit, it cannot report a trigger that stopped firing, which is why monitoring needs a second check.
The Kubera Recovery Ladder
This is a Kubera AI planning heuristic, not a validated method. It orders the actions after a failure into five rungs, each with an owner and a written trigger.
Detect. Know within minutes. Combine a heartbeat (a scheduled test message that must produce a record within a set time), alerts from the error workflow, and queue depth. A rising queue with no successful processing is the clearest sign. Contain. Pause outbound actions that carry money or promises: payment links, invoices, confirmations. Pause automatic retries that are making things worse. Store incoming items, acknowledging each only once it is safely stored, and do not act on them yet. Hold. Serve customers another way. Send a short, honest holding reply where the channel and its messaging rules allow, route new enquiries to a shared inbox or a named person, and give staff a one-page manual card. Replay. After the cause is fixed, process held items through the ledger with duplicate checks on, highest value first, in small batches. Watch the first batch before releasing the rest, so you do not trigger a second outage with your own rate limits. Reconcile. Compare systems: messages on the channel against CRM records, payments against invoices, orders against confirmations. Refund duplicates, write down what happened and name the control that would have prevented it.
The first hour: a runbook template
Adapt names and thresholds and keep it to one page.
Confirm and classify. Is it your system, your AI provider, a channel, your CRM or a payment provider? Check status pages and note the time. Contain money and promises. Pause payment links, invoices and automatic confirmations. This one step prevents the most expensive errors. Switch to manual mode. Tell the team in one message; activate the shared inbox or named responder. Capture evidence. Error messages, request IDs, timestamps, queue size. Decide wait or fail over. Use thresholds set in advance, for example no processing for 60 minutes. Ask the data question. Could personal data have been lost, altered or made unavailable? If yes or unclear, involve whoever handles data protection at once. Communicate. Tell affected customers briefly and truthfully, and tell the team when to expect the system back.
Write down who does each step, and add a human check on high-value items an AI step produced around the failure; AI agent autonomy and human-in-the-loop explains where those checks belong.
Illustrative scenario, not a specific Kubera client
A service business takes enquiries on WhatsApp (see WhatsApp Business automation in Europe). An AI step classifies each message, a workflow creates a CRM record and, for standard jobs, sends a payment link for a deposit.
At 14:00 the AI provider starts returning overload errors. Without protections, the workflow retries immediately and repeatedly, the webhook times out, WhatsApp redelivers the same messages, and each redelivery creates another CRM record and payment link. By 16:00 some customers hold three links.
With the Ladder the afternoon differs. The heartbeat misses at 14:10 and an alert reaches the operations lead. Retries stop after three attempts with backoff. Payment links are paused. New messages are stored in the queue and only then acknowledged. A holding message is sent where Meta's rules allow it: Meta's documentation says free-form replies are possible inside the 24-hour customer service window that opens when the customer writes, and outside it only pre-approved templates can be sent, to users who have opted in. Whether a reply can go out depends on the channel and the window, so customers outside it wait for a person. Staff work from the shared inbox using the manual card. At 17:30 the provider recovers; the queue is replayed through the ledger, so each message produces one record and one link. The counts match, and the review notes that Instagram had no holding step, which becomes an action item. The afternoon is still bad, but with fewer lost customers, no duplicate charges and less cleanup. These outcomes are not guaranteed.
What downtime costs and what protection is worth
This is an illustrative calculation with stated assumptions. It is not a forecast and not a Kubera price.
Assumptions. Enquiries arrive evenly across a 10-hour working day. An outage lasts 6 hours. Each enquiry is worth 60 euros in expected margin: a 20 percent conversion rate on a job with 300 euros of margin. Unprotected, 50 percent of enquiries during the outage are lost and the team spends 3 hours at 35 euros an hour on cleanup. Protected, 5 percent are lost and cleanup takes 1 hour. A monitoring and support subscription is assumed at 250 euros a month, or 3,000 euros a year.
| Enquiries per working day | Cost of one outage unprotected | Cost of one outage protected | Expected saving per outage | Similar outages a year needed to cover 3,000 euros |
|---|---|---|---|---|
| 30 | 645 euros | 89 euros | 556 euros | 5.4 |
| 60 | 1,185 euros | 143 euros | 1,042 euros | 2.9 |
| 120 | 2,265 euros | 251 euros | 2,014 euros | 1.5 |
The arithmetic: outage enquiries are daily volume times 0.6; unprotected cost is those enquiries times 50 percent times 60 euros plus 105 euros of cleanup; protected cost uses 5 percent and 35 euros.
The last column is the number of outages of this size and kind that would have to happen in a year for the savings to equal the 3,000 euros a year subscription: 3,000 divided by the saving per outage. A value of 5.4 means roughly five or six such outages a year. The savings are expected values under the assumptions above, not guaranteed amounts; a subscription lowers the chance and size of a loss and promises no particular saving. Read it honestly. At 30 enquiries a day, paid monitoring does not pay for itself on downtime alone unless outages are frequent. At 120 it does after one or two incidents. The table leaves out duplicate-charge costs and reputation, which raise the value, and shorter outages, which lower it. Compute your own break-even from four numbers: enquiries per day, value per enquiry, typical outage length and cleanup cost. See also how to measure the ROI of AI automation.
Matching protection to the cost of an hour
Ask what one hour of downtime costs in each workflow.
Low cost per hour (internal summaries, reports): alerts and a restart procedure. Medium cost per hour (lead capture, scheduling, support triage): queue, bounded retries, idempotency, holding reply, manual card. High cost per hour (payments, order intake, regulated communications): all of the above plus a tested fallback route, such as a second AI provider or a rehearsed manual path, and a named person on call.
Redundancy has its own cost: a second AI provider means two sets of prompts to test, two sets of data terms to check and two bills. Spend caps are themselves a cause of outages, so how to read an AI agent price list and how much AI automation costs bear on this.
What European rules say
This is general information, not legal advice. What applies depends on your sector, size, role and national law.
Availability is part of security under GDPR. Article 32(1) of the GDPR requires controllers and processors to implement appropriate measures, listing as appropriate the ability to ensure ongoing confidentiality, integrity, availability and resilience of processing systems and services, and the ability to restore availability and access to personal data in a timely manner after a physical or technical incident. Backups and a tested restore procedure belong in the picture wherever an automation processes personal data.
A loss of availability can be a personal data breach. Article 4(12) defines a personal data breach as a breach of security leading to the accidental or unlawful destruction, loss, alteration, unauthorised disclosure of, or access to, personal data. Depending on the circumstances, an outage that makes personal data unavailable can be a breach of security in this sense, and the absence of permanent data loss does not by itself rule it out. Where there is a breach, Article 33 requires the controller to notify the supervisory authority without undue delay and where feasible within 72 hours of becoming aware, unless the breach is unlikely to result in a risk to people, and to document every breach. Whether an outage qualifies is a case-by-case assessment.
NIS2 may apply, and size is not the only test. The NIS2 Directive applies to entities of the types listed in its Annexes I and II that are at least medium-sized. Article 2 also brings some entities in regardless of size, for example certain providers of electronic communications, trust services and DNS services, and entities whose disruption could have a significant effect on public safety or create systemic risk. It excludes some activities, and classifies in-scope entities as essential or important, with different supervision. Implementation differs by country, so check the national law. Article 21(2) lists business continuity, such as backup management and disaster recovery, and supply chain security among required measures. The reporting deadlines in Article 23 (early warning within 24 hours, incident notification within 72 hours, final report within one month) apply to significant incidents, those causing or capable of causing severe operational disruption or financial loss, or considerable damage to others. They do not apply to every outage. If you are in scope, your AI providers and automation partners are part of your supply chain.
Related data protection points are in AI data safety for European businesses.
Where this goes wrong in practice
Five patterns recur: a runbook that lives in one head (the continuity problem from founder dependency, applied to systems), unlimited retries, no holding state, alerts sent to a mailbox nobody reads, and a restore that has not been tested. Many echo why most AI projects fail: the failure is often organisational, and a named owner is the first fix.
What a monitoring and support subscription should contain
Whether you buy it from a partner such as Kubera AI or build it in house (an AI employee needs an owner and a runbook like any other, see how to build an AI employee), check that it covers specific, written things:
Monitoring of heartbeats, error rates, queue depth and spending limits, with alerts to named people. A runbook for each critical workflow, kept current. A change log of prompts, models, integrations and credentials. Provider watch: announced API changes, deprecations and pricing changes. A restore test on a set schedule, with the result written down. Post-incident reviews within an agreed time. Clear boundaries: committed response times, hours covered, what counts as an incident.
Treat any promise of zero downtime with suspicion. The credible commitments concern detection time, response time and quality of recovery. Kubera AI scopes monitoring and support in writing per project, with response commitments set out in the agreement. AI automation readiness checks whether the owners and processes exist that make any support arrangement work.
Frequently asked questions
What should I do first when my AI automation stops?
Confirm what is failing, pause anything that moves money or makes promises, and switch new enquiries to manual handling. Do not re-run failed jobs before you know they are safe to repeat.
How do I know it has stopped if nothing shows an error?
Use a heartbeat: a scheduled test input that must produce a visible result within a set time, with an alert if it does not. Error alerts catch failed runs; heartbeats catch runs that did not start.
Will I lose customer messages while my system is down?
Not necessarily. Meta retries over 36 hours and then drops unacknowledged notifications. Stripe retries live events for up to three days and allows manual resends for a limited period. Treat these as safety nets, make sure your endpoint acknowledges only what it has stored, and reconcile against the channel's own inbox after any outage.
How do I avoid duplicate payment links or charges?
Create one idempotency key per business action, store it and reuse it on every retry. Keep your own ledger of completed actions, because provider keys can expire; Stripe says v1 keys can be removed once at least 24 hours old. Keep payment steps deterministic, not decided by an AI step.
Should I retry automatically when the AI provider returns an error?
Yes for temporary errors, with limits: honour Retry-After, use exponential backoff and cap the attempts. Do not retry billing or spend-cap errors; a person must change a setting or add funds.
Do I need a second AI provider as a backup?
Only where an hour of downtime costs more than the added complexity. For most workflows a queue, a holding reply and a manual route cover most of the risk.
How long can an outage last?
Widely: about 14.5 hours for the AWS event in October 2025 and almost six hours for Cloudflare in November 2025. Plan for a range, including the recovery tail.
What are RTO and RPO?
Common planning terms. The recovery time objective is how long a workflow can be down before the cost is unacceptable. The recovery point objective is how much recent data you can afford to lose, measured in time. Set both per critical workflow.
Is an outage a GDPR data breach?
It can be. GDPR defines a breach to include accidental loss or destruction of personal data, and, depending on the circumstances, loss of availability can be a breach of security even when nothing is permanently lost. Ask the question in the first hour and take advice for your case. This is not legal advice.
Does NIS2 apply to my business?
It can, but size alone does not decide it. Sector, entity type, exceptions and national implementation all matter, so check the national law. If it applies, expect business continuity measures, and reporting at 24 hours and 72 hours for significant incidents only, not for every outage.