15. Closed-Loop Automation#

The content network had thirty points of presence scattered across the globe, and every one of them reached its users the same way: over the public internet, through a handful of ISPs. Usually three, sometimes four (plus a handful of local peerings). Some were global Tier 1 transit; some, in regions that offered nothing better, were regional local providers. Under normal operations, each point of presence spread its users across all of its ISPs at once, because it needed every one of them to carry the traffic it saw, and Border Gateway Protocol (BGP) decided which users rode which ISP, with the usual local-preference values and community tags.

One Tuesday at 02:00 am, a page woke the on-call engineer. Users on one of the Frankfurt point of presence’s ISPs, tier1-a, were seeing rising latency and a trickle of failed connections, and the cause was not obvious. To isolate it, and to stop the bleeding while she looked, she reached for the break-glass path: a direct, emergency change, outside the normal workflow (which starts in the source of truth and deploys through the Executor), draining tier1-a directly on the router by withdrawing its advertisement of the content prefix and steering egress off it, so its users shifted onto Frankfurt’s other two ISPs. The errors stopped at once. That confirmed tier1-a was the culprit and let her close the page and go back to sleep. The ISP’s upstream fault cleared on its own a few hours later, but no one noticed it.

The emergency drain was never undone. It had never been in the source of truth. It lived only on the router, a 2 a.m. fix that outlived the incident it was made for, and the engineer never documented it for rollback.

For six weeks, Frankfurt served all its users over two of its three ISPs without any major problem. Nothing broke. Two ISPs had the headroom for an ordinary day, and the content service kept meeting its latency targets. No alert fired, because from every dashboard Frankfurt looked healthy: sessions up, traffic flowing, no errors. The source of truth still said tier1-a was in service. In reality, the running router said otherwise. Nothing was comparing the two.

Then a live event pushed a wave of European traffic at the content network, the kind of evening the capacity had been sized for. Frankfurt should have spread that wave across three ISPs. It had two. They saturated. For twenty-five minutes, the 95th-percentile page-load time for European users climbed from 40 milliseconds to 180, and the complaints started before the on-call engineer had finished reading the first alert. The fix took two minutes: undo a six-week-old emergency change.

Nobody had made a bad change. The emergency drain was the right call at 2 a.m., and the break-glass path exists precisely so an on-call engineer can move faster than the normal workflow allows. What failed came after: the change was made outside the regular pipeline, and nothing ever reconciled the running network back to what the source of truth said should be true. A legitimate out-of-band fix, made under pressure and never brought back into intent, is exactly the divergence a closed loop exists to catch. This chapter is about closing that gap.

It builds the first and narrowest control loop in Part 5: keeping the running network equal to declared intent. It introduces the Configuration Lifecycle Pattern, models the network’s intent at two levels, and works the loop end to end using nothing but configuration and route-state comparison. It deliberately stops there. Detecting that a link is configured correctly but performing badly is a different problem that needs a sense this chapter does not yet have; Chapter 16 adds it. Sections 15.1 through 15.3 build the loop and its inputs, 15.4 and 15.5 close it with two-level drift detection and reconciliation, and 15.6 marks the boundary and the place to start.

15.1 The Closed Loop as Architecture#

Most network automation projects start as open-loop. A change is designed, validated, and deployed, and the pipeline reports success when the last device accepts the configuration. The loop ends at deployment. Whether the network still matches that intent an hour later, a week later, or after someone else touches a device, is nobody’s job. The opening story is the natural consequence of an open loop: a change that succeeded, was never checked again, and drifted in silence.

In this story the team is mature: every change normally goes through automation. In a brownfield environment just adopting automation, the potential for old-style, out-of-band changes is much higher, even once a paved road exists to roll changes out through automation. That is exactly where a closed loop pays off, because it catches the manual changes the paved road has not yet replaced.

A closed loop does not end at deployment. It treats the deployed change as a claim about the network and keeps testing that claim against reality. When reality diverges, the loop notices and acts. The stages are familiar on their own; what matters is that they form a cycle rather than a line.

The Configuration Lifecycle Pattern treats configuration as a continuous loop rather than a one-way pipeline: intent is designed, validated, and deployed, then the running network is retrieved and compared against intent, drift is detected, and reconciliation returns the network to intent. The unit of design is the loop, not any single stage. Apply it to any state that must stay true over time rather than being set once. A team with an excellent deployment pipeline but no retrieve-compare-reconcile arc has built half of the pattern, and the missing half is the half that would have caught the Frankfurt drain.

graph LR
    SOT[Source of Truth<br/>declared intent]
    REN[Pipeline<br/>render + validate]
    DEP[Executor<br/>deploy to devices]
    OBS[Collector<br/>retrieve config<br/>+ route state]
    CMP{Orchestrator<br/>compare vs intent}
    REC[Orchestrator<br/>reconcile or alert]

    SOT --> REN --> DEP --> OBS --> CMP
    CMP -->|drift| REC --> DEP
    CMP -->|match| OBS

The loop is not a new building block. It is the existing blocks from Part 2 wired into a cycle. The Source of Truth (SoT) holds intent (Chapter 4). Rendering and validation run in the pipeline (Chapter 10). The Executor deploys (Chapter 5). The Collector retrieves the running state (Chapter 6). The Orchestrator compares and decides what to do about a difference (Chapter 7). Part 5’s contribution is not any one of these; it is the closure.

15.2 Modeling the Network’s Intent#

A closed loop is only as good as the intent it compares against. If intent lives in a wiki page or an engineer’s memory, there is nothing to compare to. The loop starts in the Source of Truth (SoT), and the first design decision is what to model.

Model intent, not configuration. Chapter 4 argued this in general; the content network makes it concrete. The source of truth should not store “AS-path prepend of 3 on interface X”. It should store the decision that produced that prepend, so that the loop can tell the difference between an ISP that is intentionally deprioritized and one that drifted.

I recommend modeling that decision at two levels, the design-driven modeling from Chapter 4 section 4.2.2: a compact service-level intent that a human declares, and a device-level intent that a deterministic expansion derives from it and renders to a configuration artifact. Chapter 4 made the general case for it; the closed loop is where it earns its keep, because the loop compares the running network against the expanded device intent. Declaring decisions rather than device syntax is what keeps the model abstract and small enough to audit and the thing being compared precise enough to trust.

A service request is small. Someone wants to add a point of presence, bring up a new ISP at one, or change how a region’s users are spread across the ISPs that serve them. The information a human should have to provide is equally small:

pop:
  site: dc-fra
  isps: [tier1-a, tier1-b, local-c]
  class: standard

That is service-level intent: the smallest input that still expresses a decision. It says nothing about local-preference values, community tags, or how many ISPs must stay in service. Those follow from the class and the design.

Device-level intent is what the request expands into: the concrete, per-ISP routing policy that will be rendered to configuration.

isp_uplink:
  site: dc-fra
  isp: tier1-a
  capacity_gbps: 10
  local_address: 203.0.113.2/30
  peer_address: 203.0.113.1
  advertise: [cdn-eu]
  egress_share: equal
  export_communities: [65000:100]
prefix_set:
  name: cdn-eu
  served_by: dc-fra
  min_service_isps: 2
  tolerance: standard

One field there is worth calling out, because a network engineer looks for it first: the addressing. Every ISP uplink needs its own IP addresses, the local interface and the BGP peer, and for a public-facing content prefix these are public addresses that must be allocated, not invented. The allocations come from the IPAM system that Chapter 4’s aggregation folds into the source of truth, so the device-level intent references an assigned address rather than a hardcoded one. Addressing in the model, rather than on a spreadsheet, is also what lets the loop notice when an uplink’s addresses drift from what was assigned. The uplink’s capacity sits in the model for a related reason: the Minimum-Service Floor, and in Chapter 16 the capacity check, both reason about how much a point of presence can still carry with an ISP drained, and they can only do that if capacity is data rather than folklore.

The expansion from the first block to the second is a design-time function: a set of rules that encode how your organization builds a point of presence. Class standard means users spread evenly across all of the point of presence’s ISPs, a Minimum-Service Floor of two ISPs, and a standard tolerance band. The expansion must be deterministic and versioned: the same service input must always produce the same device intent, because the loop compares the running network against that expanded output. If the expansion is non-deterministic, every run looks like drift.

That expanded intent is not only configuration. It also states the intended operational outcome: which ISPs should carry a prefix set, and in what balance. Those are two readings of the same record. Saying cdn-eu is served over all three of Frankfurt’s ISPs at an equal share is a configuration intent, advertise the prefix on each, and an operational intent, its users should be spread across the three, at once. You do not author the expected route state separately; it falls out of the model. That is what gives 15.4 something concrete to compare the running route state against.

One property of this network shapes everything that follows, so name it now: a point of presence steers user traffic in two directions, and it controls them to different degrees:

• Outbound, the content it sends back to users, it commands: its own route selection across its ISPs decides which uplink a response leaves by, with local-preference and the like.
• Inbound, the requests users send it, it can only influence: other networks choose which ISP to reach the point of presence through, and the most it can do is make one uplink more or less attractive with prepending, communities, or selective advertisement, and wait for the internet to react.

The two directions ride the same ISPs, so a bad uplink hurts both, but the tools and the certainty differ. This chapter’s drains lean on the outbound direction, the one you command; Chapter 16 returns to the inbound side.

This loop steers which ISP a point of presence uses, not which point of presence a user reaches. In a global content network the second decision, mapping a user to a nearby point of presence, is made a level up, by DNS-based global load balancing or by anycast. That layer is the faster tool when a whole point of presence is in trouble: pull the DNS answer or withdraw the anycast route, and users re-home to the next point of presence. It is out of scope here, but it is not a different kind of system: it runs the same conform, heal, and decide loops this part describes, over its own intent and its own health signal. Draining an ISP keeps a point of presence healthy from the inside; steering users away from a point of presence is what you do when it cannot be healed at all.

Two properties of this model matter later. First, the service level is the golden path from Chapter 10: a small, safe input that most requests use, with the device detail derived rather than hand-written. Second, the model carries the Minimum-Service Floor, a hard lower bound on how many ISPs must keep carrying a prefix set. At this stage the floor is only data. It becomes an enforced guardrail in Chapter 16, when something starts draining ISPs automatically.

This is the human-declared layer of intent. It is authored deliberately, reviewed, and versioned. Chapter 16 adds a second author, a machine that writes a different kind of intent in response to what it observes. Keeping the two layers distinct starts here, with the declared layer modeled cleanly on its own.

One consequence of modeling intent this way is that most changes through the loop are data changes, not code changes, and they validate accordingly. Adding a point of presence or bringing up a new ISP edits source-of-truth data and re-runs an expansion whose templates are unchanged and already proven. That needs data validation, schema checks and referential ones: the ISP exists, its addresses are allocated, the Minimum-Service Floor still holds. It does not need a full network simulation, because no new device behavior is introduced. Simulation (Chapter 9) is reserved for changes to the templates or the expansion logic, or for high-risk config, where novel behavior is genuinely possible. Matching validation depth to what actually changed is what keeps the common case fast enough to run in a loop.

15.3 From Intent to Device, and Back#

With intent modeled, the loop renders device-level intent into configuration and pushes it. Rendering is the template step from Chapter 5: the device intent becomes the routing policy, prefix lists, and neighbor settings for each platform in its own dialect. The push and the read-back are where the loop earns the word “closed”, so they deserve a concrete mechanism.

This chapter uses NETCONF as the worked example: a protocol for structured configuration and state that uses explicit datastores and transaction semantics rather than screen-scraped commands. The push writes device intent to the candidate datastore, validates it on the device, and commits it. A confirmed commit is worth using where the platform supports it: the device applies the change but automatically rolls it back if the loop does not confirm within a timer, which bounds the blast radius of a bad change the same way the Rollback discipline in Chapter 11 does. Not every platform has confirmed-commit, so a mixed fleet needs a fallback for the ones that do not: a pre-change snapshot and an external rollback path, exactly as Chapter 11 prescribes. The whole push runs inside the pipeline from Chapter 10, with the same validation gates and wave controls as any other change.

NETCONF is one option, chosen here to keep the examples in one protocol. gRPC Network Management Interface (gNMI) fits the same loop and is often the better choice for streaming the operational state read-back; RESTCONF covers REST-oriented stacks; and CLI scraping remains the brownfield fallback for devices that expose nothing structured (Chapter 9). The loop does not care which protocol you use, only that it can both write intent and read reality.

The read-back is what most pipelines skip, and it is two reads, not one, both over the same NETCONF session that pushed the change.

The first read is the running configuration, a NETCONF get-config: retrieve the running routing policy for each ISP uplink and prefix set. This answers “was the device set up the way intent said?”

The second read is the operational route state, a NETCONF get of the device’s operational data: retrieve the actual best path the device selected for each prefix set, the effective local-preference in force, and the ISP the traffic is leaving through. This answers “did the intent actually take effect?” These are different questions, and 15.4 is built on the difference. One protocol does both jobs, which keeps the loop simple; streaming the read-back is an optimization on top of it, not a second protocol for the same task.

Retrieving state is the Collector’s job from Chapter 6. On a small network you can pull the running config and route state on a schedule, every cycle, and hand the answer to the comparison. At scale that pull gets expensive, and it is also unnecessary, because the route state can be streamed rather than polled. BGP Monitoring Protocol (BMP) exports a router’s BGP RIB to a collector as it changes, and a gRPC Network Management Interface (gNMI) subscription streams operational state the same way, both introduced in Chapter 6. Streaming turns the read-back from a periodic query into a change feed, which lowers the load and shortens the gap between a drift appearing and the loop seeing it. Either way, the comparison cadence is a scaling decision, not an afterthought: it follows the same collection-frequency and sharding trade-offs as any other telemetry in Chapter 11.

15.4 Detecting Drift at Two Levels#

Here is the core insight of the chapter: configuration applied is not intent achieved. Those are two claims, and a closed loop should verify both. So the loop runs two comparisons.

• The configuration-level comparison normalizes the running policy and diffs it against the rendered device intent. Normalization matters because a raw diff of vendor syntax is noise; the comparison happens on the normalized model from Chapter 6, so “local-preference 100” compares cleanly across platforms. A difference here is Configuration Drift: the device is no longer set up the way intent says.

• The route-state comparison diffs the actual best path the RIB selected against the path intent expects. A difference here means the intent was configured but not achieved: the traffic is not where the design put it. The intent it compares against is the operational intent from 15.2, the ISPs a prefix set should ride and in what balance, so there is a concrete expected route state to diff against, not a guess.

Return to Frankfurt to see how the two levels differ. In the declared intent, the cdn-eu prefix set is served over all three of Frankfurt’s ISPs at once, its users spread across them evenly. Six weeks ago an emergency change drained tier1-a, steering its share onto the other two, and the drain was never undone.

• At the configuration level, the loop retrieves Frankfurt’s running policy and compares it to intent. Intent advertises cdn-eu over tier1-a at an equal share; the running config has it withdrawn. Mismatch, on the very next cycle. The loop reports configuration drift on dc-fra tier1-a.
• At the route-state level, the loop retrieves where cdn-eu’s users actually flow. Intent spreads them over three ISPs; the RIB shows them on two. Mismatch. The loop reports route-state drift on cdn-eu.

Either signal alone would have caught the Frankfurt drain six weeks early. So why insist on both? Because they fail independently, and each is blind exactly where the other sees.

A configuration can match intent perfectly while the route state is wrong. A neighboring network suddenly prefers a different ISP into Frankfurt, an upstream withdraws a prefix, a next hop changes: the advertisements are exactly as declared and the users still arrive over the wrong uplink, or stop arriving at all. Configuration comparison alone reports all-clear, and only the route state sees it.

Turn it around, and a configuration can drift where route state cannot yet see it. Frankfurt’s intent includes how it should redistribute users when an ISP is drained; if that redistribution policy drifts, nothing changes today, because nothing is drained today, so the route state looks exactly as intended. The drift sits latent in the config until the next drain applies it wrong, and only the configuration comparison catches it before then.

Comparing configuration tells you the device was set up as intended. Comparing route state tells you the intent was actually achieved. You want both because each is blind exactly where the other sees, and production breaks in either gap.

There is an honest limit to the route-state comparison, and it is worth stating plainly because it is the first thing a network engineer will worry about. The intended spread is really the intended spread under nominal conditions. If one of Frankfurt’s ISPs genuinely goes down, its users redistributing across the others is not drift; it is the network doing exactly what it should. A comparison that flags every ISP failure as drift will page the on-call engineer during every real incident and get muted within a week, which is worse than having no loop at all. So the route-state comparison cannot be a blind diff against the nominal spread. It has to ask a second question first: are conditions still nominal?

Part of that answer is cheap and local, and this loop already has it, because retrieving state tells you whether each ISP session is up and its next hop reachable. But the rest of the answer, whether an ISP that is up and configured correctly is actually carrying its users well, is precisely the sense this chapter does not have. In the general case, telling a legitimate ISP failure apart from real drift needs the link-health signal that Chapter 16 introduces. Until then, the workable rule is: a missing ISP that is healthy and reachable is drift; a missing ISP whose session is down is an expected withdrawal; and the ambiguous middle, an ISP that is up but degraded, is exactly the case Chapter 16 was written to resolve. Two smaller rules keep the same comparison quiet: compare at prefix-set granularity, not per individual prefix, and treat a prefix spread across several ISPs as the normal state it is, flagging only a departure from the intended set of ISPs and their balance.

The scheduled mechanism that performs the comparison already exists: it is the continuous-compliance loop from Chapter 12 section 12.3.4. This chapter reuses it rather than rebuilding it, and adds the second comparison level. Each cycle emits a drift event: a structured record the next stage can act on without a human parsing it.

drift:
  device: dc-fra
  prefix_set: cdn-eu
  level: route_state
  expected_isps: [tier1-a, tier1-b, local-c]
  observed_isps: [tier1-b, local-c]
  tier1-a_reachable: true
  tier1-a_session: established
  detected_at: 2026-07-18T02:14:09Z

That drift event is not a log line for a human to read. The Observability layer emits it, the machine-consumable alert-to-action payload that Chapter 6 section 6.2.6.2 specified, and it triggers a remediation workflow in the Orchestrator (Chapter 7), which makes the reconcile-or-alert decision that 15.5 describes. The tier1-a_reachable and tier1-a_session fields travel with the event so that workflow can lean toward drift or toward a legitimate withdrawal before it pages anyone. Detection observes and announces; remediation decides and acts; the drift event is the handoff between them.

15.5 Reconcile, and the Feedback Loop#

A drift finding demands a decision: correct it automatically, or raise it for a human. Chapter 12 section 12.3.4 already set the rule, and it applies unchanged here: auto-remediate only when restoring intent has no forwarding-plane impact and the drift is unlikely to be a deliberate manual change; otherwise alert and wait. Routing sits on the risky side of that line. A prefix’s users shifting ISPs is exactly the kind of change a human might have made intentionally during an incident, so the default for route-affecting drift is alert-and-wait: page the on-call engineer with the device, the prefix set, the expected ISPs, and the observed ISPs, and let a person decide. Reconciliation re-renders intent and re-pushes it through the same pipeline, so a corrective action gets the same validation and Rollback safety as the original change. Because the render is Idempotency-safe, reconciling a device already in the desired state is a no-op.

That no-op is the crux of a distinction the two drift levels force. A configuration drift has an obvious cure: the running config left intent, so re-render and re-push, and it is back. A route-state drift often has no such cure, because the configuration may already match intent. If cdn-eu is missing from an ISP that is configured to carry it, and that ISP is up and healthy, re-pushing the identical config changes nothing; the users moved for a reason outside the config, a more-preferred route from a neighbor, an upstream withdrawal, a next hop that changed. The finding’s own fields point to the honest response. If the missing ISP’s session is down, this is a legitimate withdrawal, not drift, and the loop should say so and stay quiet. If the ISP is healthy and reachable and its users still left, the design is being overridden by something the intent did not anticipate, and the right move is to escalate for a human to decide whether the intent itself should change, not to re-push a config that is already correct. And if the ISP is up but degraded, that is the ambiguous middle only Chapter 16 can resolve. Reconciliation by re-push is the cure for configuration drift; route-state drift is more often a decision than a repair.

Reconciliation also has to respect physics. A routing change does not take effect the instant the commit succeeds; BGP has to reconverge, and re-verifying before it settles will show phantom drift, the loop chasing a state that is still in motion. The verify step after a reconcile waits a convergence window before it compares again, sized the way Chapter 11 section 11.1.5 sizes any reconciliation window: a small multiple of the observation interval, long enough for the change to settle, short enough to stay responsive. Confirmed-commit and the convergence wait work together, the device holding a safety net under the change while the loop gives the control plane time to catch up before it judges the result.

Auto-reconcile also has to coexist with people. A network still gets emergency changes at 2 a.m., and a loop that reverts the on-call engineer’s fix while they are still typing is the incident it was meant to prevent. The operational contract for this is the break-glass path from Chapter 11 section 11.3.7: an emergency manual change goes through a documented, audited bypass that also pauses auto-reconcile on the affected devices, and the loop does not resume enforcing intent on them until the change has been reconciled back into the Source of Truth (SoT). Without that handshake, the closed loop and the human on-call are two actors writing the same devices toward opposite goals, which is the oscillation failure mode from 15.6 wearing a person’s face.

Timing is part of the same coexistence problem. Traditional change management runs changes in approved windows, and a loop that reconciles at 14:00 on a Tuesday is making an unscheduled change no matter how correct it is. The platform draws the line by urgency. Detection runs continuously, because there is no reason not to know about drift the moment it appears. Automatic reconciliation is a change, and it is gated like one: it runs inside a maintenance window, or against a standing change ticket for that class of drift, under the same governance as Chapter 12. Healing in Chapter 16 is the deliberate exception, because it is incident response and an incident does not wait for a window, and the autonomous optimization in Chapter 17 is scheduled into windows by design. Detect always; change when the change process allows.

One design rule keeps this chapter from colliding with the next. The loop must compare and reconcile against effective intent, not against the declared layer alone. In this chapter the two are identical, because nothing else writes intent yet. But Chapter 16 introduces a machine that deliberately drains an ISP and records that drain as a second layer of intent. If the conformance loop compared only against the declared layer, it would see that healing drain as drift and revert it, and the two loops would fight. Writing the comparison against effective intent from the start, even while the dynamic layer is always empty, is what lets the two loops compose later instead of oscillating.

The loop also has parameters, and those parameters can learn. How long to wait before treating a difference as real rather than a transient mid-convergence state, how often to run the comparison, how large a reconciliation wave should be: these start as fixed values and improve from operational history. If a given drift class is transient 95% of the time and self-resolves within one polling interval, the loop can widen its wait window for that class and stop paging on it. This feedback is statistical and rule-based, tuning the loop’s own dials from its own record. It is not a model making decisions; that is Chapter 17. Here, the loop gets steadier over time by watching itself.

15.6 Boundaries, and the Confidence Ladder#

The closed loop in this chapter is powerful and narrow. It can prove that the running network matches declared intent, at both the configuration and route-state levels, and it can hold that match over time. That is exactly what the opening story needed and did not have.

It is also blind in one important way. It compares the network against intent, and intent is a set of paths and attributes, not a level of service. An ISP can match its intent perfectly, advertised and weighted exactly as the design says, carrying its share of users, and still be dropping 4% of packets because the provider behind it is having a bad hour. To this loop, that ISP is healthy: configured correctly, achieving the intended spread, no drift. Nothing in a configuration or a RIB entry reveals a brownout. Seeing it requires measuring the path, not comparing the intent, and that is the subject of Chapter 16.

The loop has failure modes of its own worth naming. It can oscillate if it fights an external actor: an engineer making manual changes while auto-reconcile reverts them, each undoing the other. Its feedback can optimize the wrong dial, widening a wait window until real drift goes unnoticed. And an over-eager reconcile can turn one drifted device into a fleet-wide reconciliation storm. Each of these is a reason to add automation gradually, not all at once.

That gradual path is the Confidence Ladder from Chapter 13, and closed-loop conformance sits on its lower rungs. The evidence that justifies climbing is the Automation Track Record from Chapter 11: how often the loop detected real drift, how often it flagged a transient as drift, and, once reconciliation is enabled, how often a reconcile succeeded without a surprise.

Before pointing the loop at production, prove it in the simulation environment from Chapter 9. Inject a known drift into the simulated network, a stray prepend, a withdrawn prefix, a lowered local-pref, and confirm the loop detects it at the right level and reports the right expected and observed values. Then inject a legitimate ISP failure, take an ISP down in the simulation, and confirm the loop does not cry drift when its users correctly redistribute across the others. A drift detector that has not been tested against both a known-bad and a known-good change is a pager waiting to cry wolf.

Where to start is the same answer the Confidence Ladder gives everywhere: detect before you reconcile. The minimum viable closed loop is the read-back and the two comparisons, with every finding going to a human and nothing corrected automatically. That single step closes the gap the opening story fell into, silent drift, with zero risk that the loop itself makes a bad change. It is complete when a drifted ISP produces an alert within one comparison cycle, at both the configuration and route-state levels. Only then, and only for the lowest-risk drift classes the Track Record has vouched for, do you turn on auto-reconcile. A team that can reliably see drift is already far ahead of the one in the opening story, even if it never automates a single correction.

Summary#

The Frankfurt drain was not a failure of automation. It was the absence of a loop. A human made a change the platform never made and the source of truth never recorded, and nothing was watching the gap between what was declared and what was running. The Configuration Lifecycle Pattern closes that loop: model intent in the source of truth at two levels, render and deploy it, retrieve the running configuration and the actual route state, compare against effective intent at both levels, and reconcile or alert. Compared against reality on a schedule, the six-week drift becomes a same-day alert.

The loop’s boundary is also its handoff. It keeps the network equal to intent, but it cannot tell that a correctly configured ISP is performing badly, because a configuration and a RIB entry do not carry loss and latency. Chapter 16 gives the loop that missing sense, and with it a second, machine-authored layer of intent that this chapter was careful to leave room for.

References#

• Feedback Control for Computer Systems, Philipp K. Janert (O’Reilly, 2013). Setpoints, stability, and hysteresis in software control loops; the design vocabulary behind the Configuration Lifecycle Pattern and its self-tuning parameters.
• Network Programmability and Automation, 2nd Edition, Matt Oswalt, Christian Adell, Scott S. Lowe, and Jason Edelman (O’Reilly, 2023). NETCONF, gNMI, and model-driven configuration and operational-state retrieval, the mechanics behind the render-push-read-back cycle in Section 15.3.
• Infrastructure as Code, 2nd Edition, Kief Morris (O’Reilly, 2021). Declarative intent, reconciliation, and drift as first-class concerns; the GitOps reconciliation model applied here to network state.

💬 Found something to improve? Send feedback for this chapter