16. Self-Healing Networks#

At 09:40 on a weekday, one of the Amsterdam point of presence’s ISPs started to go bad. Not down, bad. Somewhere inside that ISP’s network, well past the peering point, something began degrading the flows that crossed it. At Amsterdam’s edge nothing looked wrong: the Border Gateway Protocol (BGP) session stayed up, no BFD session dropped, the interface counters showed no loss, the running configuration matched intent exactly, and the closed loop from Chapter 15 reported no drift. By every signal the network owned, that ISP was healthy.

It was not healthy. A slice of European users reached the content over that ISP, and their 95th-percentile page load had jumped from 40 milliseconds to 130. The problem was real and it was in the path, but it lived inside the ISP, where none of the network’s own signals could see it. The first useful signal did not come from the network at all. It came from the content team’s own application dashboard, as a ticket, forty minutes in: “EU page loads are degraded, is something wrong with the network?”. An engineer eventually correlated the app symptom to that ISP, drained it by hand, and watched the page loads recover. Forty minutes of user-visible degradation, and the network had been reporting success the whole time.

The uncomfortable part is that the information was there to be had the entire forty minutes. Every one of the other twenty-nine points of presence reaches Amsterdam over the internet too, and any of them, had it been measuring, would have seen that the path over that ISP had gotten slow. Nobody was measuring from the outside, collecting those views, agreeing that the ISP was bad, and acting on it. That is what this chapter builds.

It is the second loop of Part 5, and it closes on a different thing than the first. Chapter 15 kept the configuration equal to intent. This chapter keeps the service healthy when the configuration is already correct and the network is not. It introduces the missing sense, synthetic monitoring, and the Quorum Vantage rule that makes it trustworthy, then the Dynamic Intent Layer that heals without fighting Chapter 15, and it brings in Chapter 14 at the point where the consuming team gets to set its own definition of “bad”. Section 16.1 draws the boundary, 16.2 and 16.3 build detection, 16.4 builds the cure, and 16.5 and 16.6 are about the ways it goes wrong and where to start.

16.1 When the Config Is Right but the Service Is Sick#

Chapter 15 ended on an ambiguous case it could not resolve: an ISP that is up, configured correctly, and carrying its share of users, yet performing badly. To the conformance loop that ISP is healthy, because a configuration and a RIB entry do not carry loss and latency. This chapter is the resolution of that exact case.

Keep the two questions side by side. Chapter 15 asks: did the configuration leave intent? Chapter 16 asks: did the service leave health, while the configuration stayed correct? They are different failures with different sensors. The first is a comparison against a stored value. The second is a measurement of the real world.

A brownout falls in the blind spot of both defenses a network usually has. Session-liveness detection does not trigger, because it keys on an ISP or session going down, and a brownout keeps everything up. Configuration comparison does not trigger, because nothing about the configuration changed. The ISP is doing everything it was told; it is just doing it badly. Catching that requires measuring the path, not inspecting the device, and measuring the path is a capability the network does not have until you add it.

graph LR
    PROBE[Synthetic probes<br/>full mesh]
    DETECT{Quorum<br/>degraded?}
    OVERLAY[Dynamic Intent Layer<br/>time-boxed drain overlay]
    EFF[Effective intent<br/>declared + overrides]
    DEV[(PoP<br/>devices)]
    VERIFY{Quorum<br/>recovered?}

    PROBE --> DETECT
    DETECT -->|yes, within floor + capacity| OVERLAY --> EFF --> DEV
    DEV --> PROBE
    DETECT -->|no| PROBE
    VERIFY -->|yes| OVERLAY
    DEV --> VERIFY

16.2 Synthetic Monitoring: the Missing Sense#

The missing sense is Synthetic Monitoring: active probes that generate test traffic and measure what happens to it, rather than reading counters off a device. Chapter 6 built a rich observability stack, but we treated it mostly as passive: it collects what the devices report about themselves. A device experiencing a brownout on an upstream ISP reports nothing wrong, because from its own point of view nothing is wrong. Only traffic that actually crosses the bad path feels the loss, so the only way to see a brownout is to send traffic across the path and measure it.

Two design choices decide whether the measurement is worth anything.

The first is where you measure from. A router cannot credibly measure its own uplink; if the uplink is bad, its own probes suffer the same fate and it has no baseline to compare against. So the measurement is external: every point of presence probes every other one’s ISP endpoints. Each ISP is judged from the outside, by peers that do not share its fate. Across thirty points of presence this is a large full mesh, and the size is a feature: the more independent vantage points watch an ISP, the more trustworthy the verdict about it.

You do not have to build the whole vantage-point mesh yourself. Public measurement networks like RIPE Atlas, and commercial synthetic-monitoring services such as Catchpoint or ThousandEyes that keep probes in hundreds of networks, give you vantage points off your own infrastructure. They complement your own points of presence, and they are especially good at seeing your ISPs the way a distant user’s network sees them, not just the way your other sites do.

The second is what you send. This is where a network engineer’s instinct earns its keep: a probe measures the path the probe takes, which is not automatically the path the application takes. If the network load-balances flows across an ISP’s member links by hashing the 5-tuple, a probe with one source port may ride a different member than the user traffic it is supposed to represent, and you will confidently measure a path the users never take. If the network marks traffic by DSCP and treats classes differently, an unmarked probe measures a queue the application does not use. The rule is that a probe must resemble the traffic it stands for: match the protocol, the port entropy, and the marking of the HTTPS flows it represents, or accept that you are measuring something else. And one probe flow tests one path: if traffic spreads across the members of a bundle or an equal-cost group by a per-flow hash, a single probe rides exactly one member and says nothing about the others, so covering them all takes a sweep of probe flows with enough 5-tuple variation to land on every member. A monitor that sends one flow to a many-membered link measures a fraction of it and reports the fraction as the whole. A synthetic monitor that does not look like its workload, or covers only part of it, is worse than none, because it produces confident wrong answers.

One more dimension decides what a probe represents: its address family. A dual-stack ISP carries IPv4 and IPv6 over what are often different internal paths and different peering arrangements, so the same provider can be clean on one and brownout on the other. A probe measures the family it is sent over and says nothing about the other, so the mesh runs both, the quorum is judged per family, and the drain is per family too: you may withdraw an ISP’s IPv6 while leaving its IPv4 in service, because only one of them is bad. Treating an ISP as a single health verdict across both families hides exactly the kind of one-sided degradation that is common in practice.

With trustworthy per-ISP measurements arriving from many vantage points, the question becomes when to believe them. The answer is the Quorum Vantage rule: treat an ISP as degraded only when a quorum, a configured fraction of the independent remote vantage points, agrees that it is degraded. One point of presence reporting loss to an ISP is more likely a problem at that site, or a fluke on one probe path, than a real ISP fault. A quorum of unrelated sites reporting the same degradation is a fact about the ISP. Quorum turns a noisy field of individual measurements into a single trustworthy verdict, and it is what lets the loop act without a human sanity-checking every reading. At this scale the fraction is a real dial doing real statistical work: each ISP has twenty-nine independent remote vantage points, so a simple majority already makes a single false vantage point harmless, and you can tune the fraction up for a more conservative drain or down for a faster one. It is at four sites, with three vantage points, that quorum barely means anything; at thirty it earns its keep. Apply it wherever a distributed measurement drives an automated action, because the cost of a single false vantage point is an unnecessary drain.

Quorum rests on an assumption worth guarding: that the vantage points fail independently. A shared upstream, a common transit provider or a peering point behind several of your sites, can make many vantage points agree for the wrong reason. So the loop watches for the shape of a correlated failure. If every vantage point degrades toward every ISP at one site at the same moment, the fault is almost certainly local to that site or its aggregation, not a coincidence across all its ISPs, and the correct response is to escalate rather than to drain every ISP the site has.

The probe results themselves are just another telemetry stream. They flow into the same collection, normalization, and storage pipeline from Chapter 6; the Collector does not care whether a measurement came from a device counter or a synthetic probe. At scale the mesh is the thing to watch, because a full mesh of vantage points against ISPs grows with the square of the sites: thirty points of presence probing three or four ISPs at each of the others is already on the order of a few thousand probe relationships, and the probe cardinality follows the same collection-frequency and sampling trade-offs that Chapter 11 applies to any high-volume telemetry: probe less often, sample vantage points, or introduce a hierarchy before the mesh outgrows the pipeline.

16.3 From Signal to Decision#

A quorum verdict is not yet a decision. Turning it into one takes three steps, and each guards against a way the loop could hurt more than it helps.

1. Insist on persistence. A single degraded sample, even by quorum, is a spike, not a brownout. Require the degradation to hold across several consecutive intervals before it counts. Put numbers on it: probe each ISP every 10 seconds and require four sustained intervals, and the loop catches a genuine brownout in about 40 seconds while ignoring a lone bad sample, the same hysteresis a network engineer applies to route-flap damping and for the same reason. Those two dials, the probe interval and the number of sustained intervals, are the whole tradeoff between how fast you react and how often you react to nothing: the opening story’s forty minutes becomes forty seconds, at the cost of a forty-second floor on detection.

2. Correlate the degraded ISP to what actually rides it. An ISP is not the unit anyone cares about; a service is. The loop maps the degraded ISP to the prefix sets and users it carries, and from there to the affected content-delivery traffic, so the decision is framed as “these users are riding a bad ISP”, not “an ISP has 4% loss”.

3. Emit a machine-consumable health event, the same alert-to-action payload Chapter 6 section 6.2.6.2 specified. It is structured for the next stage, not written for a human to read:

   health_event:
     isp: dc-ams isp-2
     family: ipv6
     verdict: degraded
     quorum: 24_of_29
     metrics: {loss_pct: 4.1, rtt_ms: 28, jitter_ms: 6}
     sustained_intervals: 4
     affected_prefix_sets: [cdn-eu]
     correlated_site_failure: false
     detected_at: 2026-07-18T09:44:00Z

The threshold that decides what counts as “degraded” is the interesting design question, and the answer is not the network team’s to keep. The team that runs the content service knows how much loss and latency its users can absorb before it matters; the network team does not. So the definition of “bad” belongs to the consumer, through the Consumer-Tuned Tolerance pattern: the consuming service team sets its own tolerance through a self-service API, and the platform enforces it. This is the point where Chapter 14’s treatment of automation as a product enters Part 5. The network is a service, the content team is its customer, and the drain behavior the customer gets is the Service Level Agreement (SLA).

The API should be simple, and simplicity here is also a guardrail. Rather than let a consumer type raw numbers, offer named tolerance levels, low, medium, and high sensitivity, that the platform translates into concrete thresholds (loss percentage, latency ceiling, number of sustained intervals) stored in the Source of Truth (SoT). A high-sensitivity service drains an ISP at 2% loss; a best-effort one tolerates 8%. The named levels do two jobs at once: they keep the consumer out of numbers they have no basis to choose, and they bound the input so a customer cannot set a pathological threshold that flaps the network or undercuts the Minimum-Service Floor. The product exposes choices, not dials, and the choices are all safe.

The three levels translate to concrete numbers like these, an illustrative starting point to calibrate per service rather than fixed law:

The quorum fraction is a separate dial, and it is about trust in the verdict, not the verdict’s threshold. With twenty-nine vantage points a bare majority (fifteen) already makes a single false vantage point harmless; raising the bar toward a supermajority (the 24-of-29 in the event above) trades a little detection speed for near-immunity to correlated noise. Choose the fraction by how costly an unnecessary drain is for the service: cheap to drain, take the majority and the speed; expensive, wait for the supermajority.

16.4 The Dynamic Intent Layer#

A health event says an ISP is bad and names who it hurts. Like the drift event in Chapter 15, it is an event and not a report: the Observability layer emits it, and it triggers a remediation workflow in the Orchestrator (Chapter 7), the same event-driven handoff the two loops share. Healing has to move the traffic, and how that workflow records the move is the central design decision of the chapter, because it is what determines whether this loop cooperates with Chapter 15’s loop or wars with it.

The wrong way is to reach into the device and change the Border Gateway Protocol (BGP) policy directly, out of band. Do that and the Chapter 15 conformance loop, which is still running, retrieves the running config, finds a drain that declared intent does not mention, calls it drift, and reverts it. The two loops undo each other on every cycle. The right way is the Dynamic Intent Layer: healing does not edit declared intent and does not touch devices out of band. It writes a machine-authored, time-boxed operational intent overlay that says, in the same intent model the human uses, “for now, drain dc-ams isp-2 for cdn-eu”. The network is then reconciled against effective intent, which is declared intent with the overlay applied on top. Chapter 15’s loop sees the drain as part of effective intent and enforces it instead of fighting it. The two loops compose because they read the same combined intent.

What “drain” means concretely is the ingress/egress distinction Chapter 15 drew, and here it is the whole story. Outbound, the content the point of presence sends back to its users, it commands: lower the degraded ISP’s local-preference or weight and the point of presence’s own route selection moves those responses onto its other ISPs, promptly. Inbound, the requests users send in over that ISP, it can only influence: make the ISP’s advertisement less attractive with AS-path prepending or a community the transit acts on, or withdraw the content prefix from that ISP entirely, and wait for the internet to react. So a full drain of a degraded ISP is two actions with two certainties: an immediate, reliable shift of the outbound traffic you command, and a slower, best-effort nudge of the inbound traffic you do not. Treating the inbound side as if it were as prompt and certain as the outbound is a classic way to be surprised by where users actually land.

This is the payoff of a rule Chapter 15 followed on faith: it compared against effective intent even while the overlay was always empty. Now the overlay has contents, and nothing in Chapter 15 has to change.

Three properties make the overlay safe.

• It is ordered correctly. The overlay is written first, then the drain is rendered and pushed through the very same Chapter 15 machinery: the Chapter 10 pipeline, the Chapter 11 wave and Rollback controls, and the Chapter 12 audit trail. An autonomous drain is not a shortcut around governance; it takes the same governed path a human change takes, which is part of what makes it safe to run unattended. Writing intent before changing the device closes the race where the conformance loop wakes up between the push and the overlay and briefly sees the drain as drift.

• It is bounded. The drain honors the Minimum-Service Floor, which in Chapter 15 was only data in the model and here becomes an enforced limit: the loop will not drain an ISP if doing so would drop a prefix set below its minimum number of healthy ISPs. And it is capacity-aware, which is the check a network engineer will insist on: draining a degraded-but-working ISP moves its users onto the others, and if the others cannot absorb them, the cure is worse than the disease. The headroom the loop checks against is not something the synthetic probes can see, because loss and latency say nothing about spare bandwidth; it comes from the passive utilization telemetry of Chapter 6, the interface counters and the traffic matrix, which the loop reads alongside the probe verdict. The loop drains only when the remaining ISPs have the headroom to carry what it is about to move; if they do not, it does not silently relocate the congestion, it escalates. Headroom is necessary but not sufficient, though, because ISPs are not interchangeable: a settlement-free peer routes only to its own customers, so it can absorb the prefixes it reaches and would black-hole the rest, while transit reaches the whole internet. The drain’s candidate set is the ISPs that both have the headroom and carry the affected destinations; if the only ISP with reachability is the degraded one, there is nothing safe to drain onto, and the loop escalates rather than trade a brownout for an outage.

• It is reversible on its own, and it stays measurable while drained. Draining an ISP moves user traffic off it, which raises a fair question: if nothing rides it, how does the loop know when it recovers? Because the mesh does not depend on user traffic. The probes keep hitting the ISP’s public endpoint directly, and that endpoint stays up and routed by the ISP even with the content prefix withdrawn, so the same vantage points that saw the ISP go bad keep watching it while it is out of service. When they agree, by the same quorum rule and the same persistence requirement, that it has been healthy for long enough, the overlay expires and effective intent collapses back to declared. The drain undoes itself; nobody has to remember to undo a drain six weeks later.

To make the bounded check concrete, put the model’s numbers to work. Frankfurt serves cdn-eu over three ISPs of capacity_gbps: 10 each, held to 80% safe utilization, so 8 Gbps usable apiece. Suppose cdn-eu draws 12 Gbps, evenly spread at 4 Gbps per ISP, and tier1-a is the degraded one. Draining it asks the other two to absorb 4 Gbps; each has 4 Gbps of spare (8 usable minus 4 in use), 8 Gbps between them, so the drain proceeds and each survivor settles at 6 Gbps, under its ceiling. Now suppose cdn-eu draws 21 Gbps, 7 per ISP: each survivor has just 1 Gbps of spare, 2 Gbps between them against 7 needed, so the drain would saturate them and the loop escalates instead. Same degraded ISP, opposite decision, and the utilization telemetry is what tells them apart.

The overlay is a small, self-describing record, in the same intent model as the declared layer so the loop can read both together:

overlay:
  drain: dc-ams isp-2
  prefix_set: cdn-eu
  reason: quorum-degraded
  quorum: 24_of_29
  created_at: 2026-07-18T09:44:40Z
  ttl: 30m
  restore_when: healthy for 4 intervals by quorum

Everything the three properties need is in it: what to drain and for which prefix set, why (the audit trail keeps the reason), when it was written, a time-to-live so it cannot outlive its usefulness the way an undocumented manual drain can, and the condition that ends it. Effective intent is the declared model with this record laid over the top.

Those three properties are also what let this loop act on its own at all. Chapter 12 section 12.3.4 drew the line between what automation may remediate unattended and what it must escalate: auto-remediate only when the action’s impact is bounded and reversible; otherwise alert and wait. Routing normally sits on the alert-and-wait side, and Chapter 15 kept it there. A healing drain earns the exception precisely because it is bounded by the floor and the capacity check and undoes itself on a timer, which is the same test Chapter 12 applies to any autonomous change.

Declared intent and the dynamic overlay do not have to live in the same database, and usually should not. Declared intent is slow-moving, reviewed, and versioned, which suits a Git-backed or otherwise auditable store. The overlay is fast-moving, machine-written, and short-lived, which suits an operational datastore built for rapid writes and expiries. “Source of truth” is a logical role here, not a single database: effective intent is composed at read time from both stores. This is the intended-versus-operational distinction from Chapter 4, realized as two stores with one combined view. That composition has a failure mode to design for up front: if the fast overlay store is unreachable, the loop must not silently fall back to declared intent alone, because that would drop every active drain at once and send traffic back onto the very ISPs it just moved off. The safe behavior is to hold the last-known effective intent and alert that the overlay store is down, failing static rather than failing open.

16.5 Limits and Risk#

Self-healing is the point in this book where automation starts changing production on its own, so its failure modes deserve more space than its mechanics.

The most insidious failure is that healing works. An ISP that browns out every afternoon and gets drained every afternoon looks, on the dashboard, like a loop doing its job. It is actually hiding a chronic ISP problem that belongs in a ticket to the carrier or a procurement conversation, not in an endless loop of automatic drains. The loop must count. A single ISP drained repeatedly over a window is not a healing target, it is an escalation, and the automation that quietly absorbs the pain is the automation that prevents anyone from fixing the cause.

The next failure is oscillation. Without hysteresis on both edges, drain on sustained degradation and restore on sustained recovery, an ISP hovering at the threshold flaps in and out of service, and each flap is a routing change the whole network feels. The persistence requirement from 16.3 and a deliberate hold-down before restoration are what keep a marginal ISP from becoming a metronome.

The third is blast radius, and it is why the Minimum-Service Floor and the capacity check are hard limits rather than preferences. A loop that drains without either can, in the middle of a bad internet day when several ISPs degrade at once, drain a prefix set onto nothing, or onto an ISP that promptly saturates. The floor and the capacity check are the difference between healing and outage.

Finally, healing can simply be wrong about the cause. If draining an ISP does not restore the service, the problem was not that ISP, and continuing to drain is thrashing. The loop verifies that the drain actually helped, by watching the affected service recover, and if it does not, it stops and hands the incident to a human with everything it observed. The same correlated-failure shape from 16.2, all vantage points bad at once toward one site, is another such stop condition. Self-healing that does not know when to give up is not self-healing, it is an automated way to make an incident worse.

There is one more failure mode, and it is the loop’s own blindness. The healer depends on the probe mesh and on whatever aggregates the vantage points into a quorum verdict, and both can fail. The rule is that the loop fails safe: no trustworthy data means no drain, never a drain on a guess, and a loud alert that the healer has gone blind so an operator knows the safety net is down. The probe mesh and the aggregator are themselves monitored, the same way Chapter 11 monitors the platform that runs the automation. A self-healing system that cannot tell when it has lost its own senses is more dangerous than no automation, because the operators have relaxed their watch precisely on the thing that has stopped working.

16.6 Confidence Ladder Placement#

Self-healing sits above conformance on the Confidence Ladder from Chapter 13, because it does more than keep the network equal to a stored value: it changes the network in response to the world, without asking first. That is real autonomy, and it is earned with the Automation Track Record from Chapter 11: how often the loop’s quorum verdict matched a real degradation, how often a drain actually restored the service, and how often it had to be undone or escalated.

Where to start mirrors Chapter 15: detect before you drain. The minimum viable version is the full mesh, the Quorum Vantage rule, and the health event, surfaced to the on-call engineer as a recommendation (“dc-ams isp-2 degraded by a 24-of-29 quorum, affecting cdn-eu, recommend drain”), with the drain a single confirmed action a human takes. That alone would have turned the opening story’s forty silent minutes into a two-minute alert-and-click, with no risk of the loop draining anything on its own. Only once the Track Record shows the verdicts are trustworthy and the drains are helping do you let the loop write the overlay itself, first for the least critical prefix sets, always inside the floor and the capacity check. A team that can reliably see a brownout by quorum is already most of the way to healing it.

Summary#

The Amsterdam brownout was invisible to the network because the network was looking at itself. Every device reported health while the service degraded, and the only outside views of the bad ISP, the other points of presence, were watching and saying nothing to anyone. Self-healing gives those views a voice and a verdict: synthetic probes measure each ISP from independent vantage points, the Quorum Vantage rule turns many noisy measurements into one trustworthy fact, the consumer sets what “bad” means as a product choice, and the Dynamic Intent Layer drains the ISP through a time-boxed overlay that composes with Chapter 15 instead of fighting it, inside a Minimum-Service Floor and a capacity check that keep the cure from becoming the disease. The brownout that took forty minutes and a human becomes a quorum verdict and an overlay that expires on its own.

Both loops so far run a policy a human wrote. Chapter 15 enforces the human’s configuration; Chapter 16 enforces the human’s tolerance, reacting when a threshold is crossed. Neither decides anything, and Chapter 16 in particular makes only a binary choice: drain an ISP or do not. It never asks whether shifting a fraction of the traffic, or choosing among several imperfect ISPs when none is clean, would be better. That question, deciding the plan rather than tripping a threshold, is optimization, and it is where Chapter 17 begins. Chapter 17 takes the last step, from reacting to a threshold to choosing the plan, and spends most of its length on why that step is mostly a question of limits.

References#

• Site Reliability Engineering, Beyer, Jones, Petoff, and Murphy (O’Reilly, 2016). Service level objectives and the discipline of acting on measured service health, the basis for Consumer-Tuned Tolerance and the healing decision. (Full text free online.)
• Feedback Control for Computer Systems, Philipp K. Janert (O’Reilly, 2013). Hysteresis, damping, and stability, the tools that keep the healing loop from flapping a marginal ISP.
• Internet Routing Architectures, 2nd Edition, Sam Halabi (Cisco Press, 2000). BGP path control and traffic engineering, the mechanics behind draining an ISP with local-preference, communities, and advertisement control.

💬 Found something to improve? Send feedback for this chapter