Outbound Wiki

Outbound stack resilience

Designing fallbacks, monitoring and recovery procedures for outages, failed syncs and changes in vendor availability.

Outbound stack resilience keeps customer state trustworthy when delivery fails, sync stalls, or a dependency disappears. Build it on clear state ownership, visible failure, controlled recovery, and a safe stop. Test the handoff into the system of record first: leads can go missing there, and that handoff matters more than the feature list that sold the capture tool.1 Use each fallback to restore state within the existing workflow.

Establish state ownership

Decide which record wins when tools disagree, then trace every read and write to it before tuning delivery.

A system of record, usually the CRM, holds the authoritative version of where the customer stands.2 Every other tool should defer to it.3

Before adding a recovery path, answer these questions:

  • Which record owns the current customer status?
  • What event is expected at each handoff?
  • Which tool is allowed to write that status?
  • What should happen when the write fails?
  • Which action can be retried safely?

Continue when each answer points to a record, an event, or a defined action. If the team cannot say which state wins, recovery work will create another disagreement instead of repairing the first one.

Monitor the delivery path

Show failure where work enters the queue and where state changes. The dashboard should help you decide whether to retry, delete, reconcile, or stop.

A delivery monitor can show upcoming deliveries, the oldest failures, attempt counts, and failure reasons. It can also let you retry or delete stuck messages.4 These signals separate a new failure from a message that has been waiting and failing for a long time.

Run a periodic monitoring job that detects prolonged synchronization failures and alerts administrators.5 The alert should identify the affected record and failure reason, so the responder can choose a recovery branch without searching across every tool.

During a review, check that you can:

  • See work that is waiting too long.
  • Tell how many attempts have occurred.
  • Identify the reason for failure.
  • Confirm that a prolonged sync failure creates an alert.
  • Take action from the failure record.

The monitor is ready when an operator can find the oldest unresolved work and understand its next action from the monitor alone.

Build the fallback path

A fallback belongs in the design when the normal event path leaves a record in an uncertain state. Keep it narrow: find the stuck state, ask the provider for current status, and reconcile the record.

Use a configurable scheduled polling job to detect records that remain in transitional status after a configurable interval, then query the provider's status API to reconcile state.6 Document the interval and status condition in the operating procedure so the team knows when the fallback takes over.

Write the reconciled result into the system of record and let downstream tools consume that state. Keep the fallback from becoming a second outbound workflow with its own customer list or competing status.

Activation covers email, sequences, and other activities that communicate with people.7 Give those activities a clear read and write path into the record, then test what happens when the record changes while a sequence is in progress.

The fallback is ready when a stuck record can return to a known state and the next outbound action follows that state.

Choose failure behavior

Choose a retry branch from three facts: whether the last action is known, whether the dependency is responding, and whether repeating the action can create a second outcome.

Retry strategies with exponential backoff and circuit breakers handle transient failures.8 Use that pattern for failures that may clear without changing the record's meaning. For uncertain writes or actions with an unclear result, use the stop or reconciliation branch and verify the state before sending anything else.

Check whether the last action completed, whether repeating it can produce a duplicate, whether the dependency is responding consistently, which record confirms recovery, and what signal allows the workflow to resume.

Every failure class should have one branch and one check that confirms the resulting state.

What not to do

Resilience breaks when the stack hides uncertainty or lets each tool make its own decision. Keep these failure patterns out of the operating procedure.

  • Do not let two systems both act as the source of truth. Resolve that conflict first; new tooling will not fix it.9
  • Do not keep attempting requests while an upstream dependency is failing. Stop for a cooldown window.10
  • Do not retry a failed action by default when the tool's idempotency guarantees do not make the retry safe. The default response should be a safe stop.11
  • Do not let activation maintain a separate private contact list, because that list can drift away from the record.12
  • Do not trust analytics and attribution when events are configured incorrectly, because the resulting view can mislead you.13

Run a failure drill from the record outward: force a delivery failure, watch the alert, exercise the fallback, and verify the final state. Record the branch that should stop, retry, or reconcile, then use that branch whenever the same failure appears.

Sources

  1. 1
    “The handoff from capture into the system of record is where leads quietly go missing, so it matters far more than the feature list that sold you the tool.”
  2. 2
    “A system of record, usually the CRM, which holds the one authoritative version of where the customer actually stands.”
  3. 3
    “Everything else in the stack should defer to it.”
  4. 4
    “You can watch all of this under Setup -> Outbound Messages (the delivery monitor), which shows next items for delivery, oldest failures, attempt counts, and failure reasons, and lets you retry or delete stuck messages.”
  5. 5
    “Monitoring Job – a periodic job that detects prolonged sync failures and alerts administrators.”
  6. 6
    “The fallback path uses a configurable scheduled polling job that detects records still in transitional status after a configurable interval and queries the provider’s status API directly to reconcile state.”
  7. 7
    “Activation, meaning email, sequences, and everything that actually communicates with people.”
  8. 8
    “Retry strategies with exponential backoff and circuit breakers handle transient failures.”
  9. 9
    “If you have two systems both behaving as though they're the source of truth, that conflict is your first problem to solve, and no amount of new tooling will paper over it.”
  10. 10
    “if upstream is failing, stop trying for a cooldown window”
  11. 11
    “The default should be safe-stop; the exception is when the tool's idempotency guarantees allow safe retry.”
  12. 12
    “It should read from and write back to the record rather than maintaining its own private contact list, because that private list inevitably drifts until you're sending a welcome series to customers who churned months ago.”
  13. 13
    “It's also where the whole system quietly lies to you if the events were set up wrong, which is a deep enough trap that we gave it its own write-up in GA4 conversion tracking.”