Escalation policy patterns: 5 templates for common incident types

$92,000 in revenue. That's what one team I worked with last year lost during forty minutes of escalation-chain delay on a Sunday payment-gateway outage. Not because of an engineering mistake. Because their alerting tool had one escalation policy for every situation — the same chain for an outage, for a post-deploy warning, for a nightly batch job failing.

This post is about why we run five separate chains in production, what specific problem each one solves, and why a single one-size-fits-all chain causes more damage than it prevents.

Five patterns we actually use

Escalation policies on most platforms degenerate into "page the on-call, then page the manager, then page the manager's manager." This works for trivial cases and breaks for everything interesting. Here are five we use in production.

1. Warning-level deploy alarms

Context: Friday, 23:40. Someone pushed a change to a payment-service integration. Error rate doubled immediately. Sergiusz, the current on-call, wakes up, opens the change log, sees a dozen commits from the last six hours. Forty minutes trying to figure out what actually shifted. Eventually calls Ilya, the author of the deployment, who rolls it back in eight minutes.

Sergiusz didn't sleep until morning. A week later something similar happened and Sergiusz started refusing night pages. Ilya could have done the same fix immediately if the page had gone to him first — he knew what he'd changed.

The deeper pattern: a deploy just landed and something went mildly wrong — error rate ticked up 2x, latency p99 doubled, a feature flag started misbehaving. Not a customer-facing outage, but an early warning where the person who shipped is the right first responder.

Pattern:

trigger: deploy-warning
chain:
  - delay: 0min
    channels: [slack, telegram_text]
    target: deployer
  - delay: 15min
    channels: [slack, telegram_text]
    target: on_call_primary
  - delay: 30min
    channels: [slack]
    target: on_call_secondary
quiet_hours: respect    # don't wake people for a warning

Key insight: the deployer is paged first, not the on-call. They know what they shipped. They probably already have the dashboard open. If they don't ack in 15 minutes, escalate to the on-call — by then it's no longer "the deployer's mess," it's an actual incident.

2. P1 customer-facing outage

Context: Customers are getting 500s. Site is down. Every minute costs revenue and trust.

Pattern:

trigger: p1
chain:
  - delay: 0min
    channels: [telegram_voip, sms, pstn]
    target: on_call_primary
  - delay: 2min
    channels: [telegram_voip, sms, pstn]
    target: on_call_primary    # retry — first attempt might have missed
  - delay: 5min
    channels: [telegram_voip, sms, pstn]
    target: on_call_secondary
  - delay: 10min
    channels: [telegram_voip, sms, pstn, slack]
    target: engineering_manager
  - delay: 20min
    channels: [telegram_voip, sms, pstn, slack]
    target: cto
quiet_hours: ignore    # P1 wakes everyone

Key insight: the retry of the primary at 2 min is critical. The first VoIP call might have been missed because the recipient was in the bathroom, in transit, or had their phone in another room. The retry costs almost nothing and dramatically improves catch rate on first-line wakes. Only escalate to secondary if BOTH primary attempts fail.

3. Security pages

Context: Something tripped a security tripwire — unusual auth pattern, vault access spike, signed-cert chain expiry warning, unauthorized IAM change.

Pattern:

trigger: security
chain:
  - delay: 0min
    channels: [telegram_voip, sms]
    target: security_on_call
  - delay: 5min
    channels: [telegram_voip, sms]
    target: ciso
parallel:
  - delay: 0min
    channels: [slack]
    target: security_team_channel    # for visibility, not for waking
quiet_hours: ignore
require_ack_from: security_on_call    # the team channel doesn't count

Key insight: parallel notification to the security team channel — visibility without requiring everyone to ack. The on-call must ack; the channel is FYI. Most platforms collapse these into one "page everyone" pattern which produces ack confusion at 03:00.

4. Batch job failures

Context: The nightly ETL didn't finish. Some non-customer-facing internal pipeline broke. Nobody is on-fire, but the data team will be angry at standup.

Pattern:

trigger: batch-failed
chain:
  - delay: 0min
    channels: [slack, email]
    target: data_engineering_team
  - delay: 4hours
    channels: [slack]
    target: data_engineering_lead
quiet_hours: defer    # if it fails at 02:00, page at 08:00 instead

Key insight: defer to business hours. Nobody needs to be woken up for a broken Airflow DAG. The platform should know that and re-route quiet-hour pages to the morning. This pattern alone saved one of our customers from a per-team mutiny — their old policy paged whoever had the data on-call regardless of severity.

5. Business-hours-only soft alerts

Context: A queue depth warning, slow query alert, disk filling up at 70%. Things that matter, but not at 04:00.

Pattern:

trigger: soft-warning
chain:
  - delay: 0min
    channels: [slack]
    target: ops_team_channel
  - delay: 30min
    channels: [slack, email]
    target: on_call_primary
quiet_hours: defer
business_hours: mon-fri 09:00-18:00 Europe/Warsaw
require_ack_from: ops_team

Key insight: defer-to-business-hours combined with slack-only first step turns a former "wake-up-for-noise" alert into a calm signal that gets handled before lunch. Most teams set this kind of alert to "page immediately" because that's the default; the cost is sleep debt and slow erosion of trust in the system.

What WardenPoint does with these

Every pattern above is declarative configuration. They live in config/escalation.php (PHP arrays) or as a Nova-edited resource (YAML-ish UI) and ship to prod as code. The audit log shows which pattern fired, who was paged, who acknowledged, when. Test fire any pattern with one button — no need to wait for a real incident to discover the chain has a typo.