10. Platform Engineering and CI/CD#

A network engineer at a mid-size financial services firm needed to modify a firewall rule to allow a new internal service to reach an authentication endpoint. The change itself took about ten minutes to understand and design. What followed was different. She needed to open a ticket in the ITSM system and fill out a form that asked questions her team had never agreed on. She needed to submit a change request to a separate portal, which triggered a workflow that sent an automated notification to a stale distribution list. She needed to manually run a validation script on her laptop, copy the output into a comment on the ticket, and wait for a reviewer to read it.

The reviewer was in a different timezone. He confirmed the output looked reasonable but noted that the script version she had run was not the current one; she needed to re-run it with the updated version, which lived in a shared directory whose path had changed since the documentation was written. She updated the ticket, re-ran the script, posted the new output. At that point the change was approved, but the deployment step required a separate sign-off from an on-call engineer who had operational access to the production environment. That engineer was available three days later. The whole process had taken eleven days. The firewall rule itself deployed in about two minutes.

Nobody in the story was negligent or incompetent. The ITSM ticket was required by the company’s change management policy. The reviewer was thorough. The on-call engineer had operational access because it was genuinely risky to let anyone push to production. The validation script had been written by someone who cared about correctness. Each piece existed for a real reason. The problem was that none of the pieces were properly connected. The team had CI/CD tooling, automation workflows, policies, and approval gates. What they did not have was a platform: a coherent system that wired those pieces together, reduced the manual handoffs, and gave every participant a single place to track what was happening and why.

This chapter operates on two distinct subjects that must not be conflated. The platform is the CI/CD infrastructure the platform team builds and operates: the pipeline runner, artifact store, secrets management system, and deployment engine. The network is what the platform manages: the devices, configurations, and change workflows that network engineers submit. Most sections address one subject; a few address both, and the crossover is called out explicitly at each point. When the chapter uses the word “deployment”, it refers to one of two different operations with different mechanics and different blast radius: deploying a change to network devices, or deploying a new version of a platform component.

Part 2 built the seven building blocks of the NAF Framework: a Source of Truth that models intent, an Executor that applies it, a Collector and Observability stack that tracks what happened, an Orchestrator that coordinates the sequence, a Presentation layer that exposes the interface, and a Network layer that grounds every decision in physical reality. Each block was designed to work correctly in isolation. The story reveals the gap: correct-in-isolation is not the same as usable-as-a-system. The building blocks existed. The tooling existed. The policies existed. The gap was the absence of a platform that wired them into a coherent whole.

Most network automation teams arrive at this gap through a recognizable arc: scripts that work in isolation, then pipelines that run those scripts consistently but keep them as disconnected islands, and finally a platform that wires the building blocks together into something the whole team can operate without knowing how each piece was built. This chapter is the guide for making that last transition.

A practical anchor is the five-stage build sequence in section 10.8.4. Each stage closes a specific gap, is independently deployable, and creates the condition for the next stage. Sections 10.1 through 10.7 describe what the destination requires and why. Section 10.8 provides the path to reach it incrementally, starting from wherever the team is now.

10.1 The Platform Engineering Mindset#

10.1.1 The Maturity Arc#

Most network automation journeys follow a similar pattern. A network engineer writes a script to provision VLANs. The script works. A second engineer writes a similar script for firewall rules, and a third for BGP peer configurations. The scripts accumulate, but they start to contradict each other. They assume different things about the state of the network. One is written for Python 3.8 and fails silently on 3.11. Another embeds credentials that are no longer valid. A new engineer joins the team and cannot tell which script is the canonical one, or whether any of them have been tested recently.

This is the scripts stage: individual automation that works in isolation and accumulates without coordination.

The next step most teams take is connecting their scripts to a CI/CD system. Changes go through a pipeline. Tests run automatically. The scripts become slightly more reliable because they are executed consistently. But the pipeline only runs the scripts; it does not organize them. Each workflow is still an island. The interface for submitting a change is still bespoke. The validation logic is still duplicated across pipelines. This is the pipelines stage: automation with basic consistency but no coherent structure.

Finally, the platform stage is different in kind, not just degree. The platform makes automation workflows repeatable, self-service, and usable by engineers who didn’t build it. It is not a new workflow engine or a new monitoring system. It is the paved road that connects the building blocks from Part 2 into something a whole team can operate.

The platform stage borrows its model from internal developer platforms (IDPs) in software engineering. An internal developer platform is a shared set of tools, pipelines, and services that a central team builds for other teams to use: the runtime environments, the deployment pipelines, the artifact stores, the monitoring integrations. The goal is that no team has to build that infrastructure themselves. A product team at a software company does not build its own CI system; it uses the one the platform team provides. The platform team’s job is to make the right path the easiest path, and to keep it working as product teams grow and their requirements evolve.

The same logic applies to network automation. The platform team builds the shared infrastructure that network engineers, SREs, and automation developers use when they submit a change, run a validation, or track a deployment. Depending on organizational size, this may be a dedicated team or a subset of the network automation team (more info in Chapter 13). Not everyone should need to know how the artifact store works, which pipeline runner is available, or where the simulation environment is scheduled. The platform handles it.

The progression from scripts to platform to governed system is a maturity arc, not a migration plan. Teams do not abandon the earlier stages; they build on them. A team at the platform stage still has scripts: they are now managed artifacts inside the platform, versioned and tested like any other component. A team at the governed system stage, covered in Chapter 12, still has a platform: governance adds control layers on top of it.

However, one important point is easy to miss: moving to the platform stage requires an explicit plan to decommission the earlier implementations. Scripts and ad-hoc pipelines operating alongside the platform create confusion about which path is authoritative. They split the team’s attention and accumulate the same governance and reliability problems the platform was built to solve. Platform migration completes when earlier implementations are retired, not when the platform is built. The incremental onboarding approach in section 10.8 addresses how to manage that transition without a disruptive cutover.

Where this chapter sits on that arc: the platform stage assumes that pipelines already exist and work. The work of Chapter 10 is to take those pipelines and the building blocks from Part 2 and organize them into something that functions as a coherent platform for the engineers who use it. Chapter 11 addresses what happens when that platform has to operate at scale and survive distributed failures, and Chapter 12 addresses what happens when it has to be auditable and compliant.

10.1.2 The Platform as Product for Engineers Pattern#

The platform is not infrastructure. It is a product. The Platform as Product for Engineers pattern makes that concrete: the platform team treats the platform the same way a product team treats its product. The network engineers are the customers. The CI/CD pipeline is the product. Cognitive load, self-service capability, and feedback speed are the service-level agreements.

This pattern has a direct counterpart in Chapter 14, which generalizes the product mindset to all network automation services. Chapter 10 is the first concrete application of that model: applied here to the platform itself, before it is applied to the network automation workflows the platform delivers.

The platform team cannot evaluate their work by asking “is the pipeline running?” They have to ask the same questions a product team asks: Are users adopting it? Is it reducing friction or adding it? When a user hits a problem, is the error message useful, or does it require platform team knowledge to interpret? How long does onboarding take for a new engineer? If a user gives up and reverts to a manual process, that is a product failure, not a user failure.

10.1.3 Golden Paths and Flexibility#

The platform does not lock users in. It has to provide a golden path: a well-lit, well-tested, well-documented route through the most common workflows. An engineer who wants to submit a standard firewall rule change should be able to follow the golden path and never need to ask the platform team for help. An engineer who has a legitimate reason to go off the path, perhaps because their workflow has unusual dependencies or requires a custom validation step, should be able to do so without dismantling the platform to get there (and track these deviations to adjust it).

The line between a golden path (recommended) and a hard constraint (enforced) matters. Chapter 10 handles recommendations. Chapter 12 handles enforcement. The platform team’s job at this stage is to make the recommended path attractive enough that engineers choose it voluntarily, not because there is no alternative.

A practical approach: if the golden path covers fewer than 80% of the team’s actual change requests, it is not a golden path yet. That threshold is not arbitrary: below it, more changes live outside the path than inside, and the overhead of maintaining the path exceeds the consistency it provides. It is a best-practice document for a subset of cases. Engineers who routinely work around the golden path defeat it: platform overhead without the consistency that justifies it. Measure the coverage. If it is below 80%, the gap is feedback about what the golden path is missing or wrong about, not evidence that the team is non-compliant.

10.2 Developer Experience#

10.2.1 The Cost of Friction#

The eleven-day firewall change cycle was not eleven days of idle waiting. At each step, someone was doing work: filling out forms, reviewing outputs, running scripts, approving gates. The total human time was probably a few hours spread across several people. The calendar time was eleven days because each step required a handoff, and each handoff required the next person to notice the queue, find context, and act.

Reducing that calendar time is not primarily a tooling problem. It is a workflow design problem. How does a network engineer submit, validate, and track a change without leaving the platform?

Self-service is the design goal. Every step that requires a human handoff is a step that adds calendar time and accumulates context loss. The platform should handle the handoffs automatically: intake, validation, gate, notify, deploy. The engineer’s interface to this process should be simple enough that a new team member can use it correctly on their first attempt.

10.2.2 Multiple Entry Points#

The platform must not assume a single interface. Different users have different preferences and different contexts for interaction. Chapter 8 established the Presentation layer as the block responsible for exposing automation capabilities to consumers; the surfaces described here apply that same principle to the platform’s own users, not to external operators:

• An engineer doing development work may prefer a CLI that gives immediate feedback without context-switching to a browser.
• An engineer monitoring a long-running deployment may prefer a UI with a dashboard view of what is running, what has passed, and what needs attention.
• A ticketing system or an approval workflow in another tool may need to interact with the platform via an API, submitting changes programmatically and receiving status updates without human involvement.

Designing for a single interface optimizes for one user type and creates friction for everyone else. The platform should expose the same underlying capabilities through all three surfaces: CLI, UI, and API. The interface is a presentation concern; the validation logic, the artifact pipeline, and the deployment mechanics are platform concerns that all three surfaces share.

10.2.3 Templates and Scaffolding#

One of the highest-friction moments in adopting a new platform is the first workflow. A network engineer who has never used the platform has to understand its conventions, its file structure, its required metadata, and its validation hooks before writing a single line of automation logic. If the platform doesn’t help, the engineer either learns undocumented conventions or writes non-conforming code and discovers the problem at CI time.

The solution is scaffolding: reference templates that encode the team’s conventions so engineers start with a working skeleton rather than a blank canvas. A scaffold for a new automation workflow should include the standard file structure, the required metadata fields, the hook points for platform validation, and a minimal but complete example of the workflow in use. An engineer who clones that scaffold and modifies it for their use case has followed the golden path by default.

A VLAN workflow scaffold, following the golden path for the most common campus change type, has a recognizable structure:

workflow-vlan-add/
├── workflow.yaml
├── inputs/
│   └── change.yaml         # engineer fills this in
├── templates/
│   └── vlan_add.j2         # platform-managed, do not edit
└── tests/
    └── test_vlan_render.py # unit tests, run automatically by CI

# workflow.yaml
name: vlan-add
version: "1.3"
risk_tier: medium
scope:
  unit: site
  attribute: site_code
validation:
  pre_deploy:
    - sot_integrity_check
    - drift_detection:
        attributes: [vlan_table]
  post_deploy:
    - observability_check:
        query: vlan_present
        timeout_seconds: 120

$ platform submit --workflow vlan-add --input inputs/change.yaml
job-20260315-0042
  sot_integrity_check          passed   1.2s
  drift_detection[vlan_table]  passed   3.1s
  simulation_gate              queued   est. 4m
Track: https://platform/pipeline/job-20260315-0042

The engineer has a job identifier in under 5 seconds, the first two validation results before switching context, and a URL to monitor the simulation stage. If sot_integrity_check had failed, the output names the specific missing SoT record: not “validation failed” but “device sw-bldg-b-01: not found in SoT, verify device is registered before resubmitting”. The engineer does not need to contact the platform team to understand what went wrong, and can resolve it on their own.

10.2.4 Measuring Cognitive Load#

The platform succeeds only if engineers find it useful, and the platform team needs concrete metrics to know whether they are succeeding:

• How many steps does a new engineer need to complete to submit their first change?
• How many systems do they need to log into separately?
• How long after submission does the engineer receive a result (pass or fail)?
• When a validation fails, is the error message specific enough to act on without platform team knowledge?

These are the developer experience SLAs. The right thresholds depend on organizational context, but two calibration points are useful:

• A new engineer who still needs platform team help after a full working day is encountering structural friction, not a learning curve
• A feedback loop consistently over 40 minutes trains engineers to context-switch away, return with lost context, and eventually batch changes or skip the platform for changes they consider low-risk.

Both patterns erode adoption. Measure against the team’s own baseline before the platform existed, not against a universal target.

The next diagram shows the interface boundary between what the engineer sees and what the platform does. The engineer’s view is intentionally simple: submit, track, receive. The platform’s view is where the complexity lives. That separation is the goal.

graph TB
    subgraph "Engineer View"
        E1[Submit change]
        E2[Track status]
        E3[Receive result]
    end

    subgraph "Platform View"
        P1[Normalize intake]
        P2[Validate SoT integrity]
        P3[Run tests]
        P4[Evaluate release gate]
        P5[Notify engineer]
    end

    E1 --> P1
    P1 --> P2
    P2 --> P3
    P3 --> P4
    P4 --> P5
    P5 --> E3
    E2 -.->|polls or receives push| P4

10.3 Core Building Blocks#

Three main tool categories appear throughout this chapter (and the two that follow in Part 3):

• A pipeline runner is a process scheduler that executes CI/CD pipeline stages on demand, triggered by a VCS event, an API call, or a schedule: the engine that runs your validation stages, renders your templates, and calls your deployment logic. Examples include GitLab CI, GitHub Actions, Jenkins, and AWX/AAP for Ansible-centric workflows.
• An artifact store is a versioned binary repository that holds the outputs pipeline stages produce and promote: rendered configuration files, container images, compiled policy bundles. It stores not just the artifacts but their version history and promotion state, so the platform can answer “what is currently deployed in production and when was it validated?” Examples include JFrog Artifactory or container registries such as DockerHub or the native registries provided by GitLab and GitHub.
• A secrets management system provides a secure credentials storage backend and can issue short-lived credentials on demand, scoped to a specific operation, and expires them automatically when the operation ends. Examples include HashiCorp Vault, AWS Secrets Manager, and Azure Key Vault.

These three components are infrastructure the platform team operates, not products this chapter recommends. Any system that provides these capabilities applies.

10.3.1 Version Control and Infrastructure as Code for the Platform#

Chapter 4 established version control for managing network intent. Apply the same discipline to the automation code itself: the workflows that coordinate changes, the validation scripts that gate deployments, and the policy definitions that govern what is allowed.

Everything that governs the platform’s behavior should live in Version Control System (VCS). There should be no ad-hoc changes to production pipeline runners. No configuration that exists only on a specific machine. No script that lives in someone’s home directory and is “mostly the same” as the one in the repository. The Version Control System (VCS) state is the authoritative record of what the platform should do. If the platform is behaving differently from what the repository describes, that is a bug, not a configuration choice.

The platform itself is a system: pipeline runners, artifact stores, execution environments, validation services, and observability agents. Each of these components should be managed with the same Infrastructure as Code (IaC) discipline that the network automation platform applies to the network it manages. A pipeline runner that was provisioned by hand, configured interactively, and is not described in any managed configuration is a reliability risk. If it fails, restoring it requires knowledge that may exist only in the person who originally set it up. If it needs to be reproduced in another environment for testing or disaster recovery, the reproduction requires archaeology rather than automation.

The pipeline definition itself, the stages, the gates, the approval logic, is code. It should go through review. Changes to it should be visible, reversible, and tested. The operational discipline of network automation should be applied to platform automation. Automation teams routinely automate everything except their own infrastructure.

The same discipline extends to external dependencies: role collections, package libraries, container base images, schema definitions. These are often treated as stable infrastructure maintained by someone else, safe to rely on without active management. That assumption is wrong in the same way it is wrong for application software. An unversioned dependency means the automation behaves differently depending on when it runs. Every external dependency should be pinned to a specific version, tracked in the repository, and updated through the same review process as any other platform change. Dependencies that are silently updated by the environment are dependencies that can silently change the platform’s behavior. Chapter 12 covers the supply chain security implications in detail.

10.3.2 Artifact Management#

An artifact in the automation platform is any versioned output that flows from one stage to another. Two distinct artifact categories exist and should be managed separately.

• Platform artifacts: container images that package workflow execution environments, compiled workflow definitions, pipeline runner configurations, policy binaries, and software packages (Python libraries, Ansible collections, or any versioned package the platform’s execution environment depends on). These govern how the platform itself behaves. A new version of the execution engine is a platform artifact; promoting it to production means the platform now runs differently for all workflows.

• Network change artifacts: rendered configuration files, pre-validated configuration snapshots, and deployment manifests targeting specific devices. These govern what the platform does to the network. A rendered firewall policy for site A is a network change artifact; promoting it to production means those device configurations will be applied.

Both categories share the same lifecycle model: produced by a pipeline stage, validated, promoted through environments, eventually deployed, and at some point superseded. But the validation gates differ. A platform artifact gate asks “does the new execution engine pass integration tests and backward compatibility checks?” A network change artifact gate asks “does this rendered configuration pass simulation against the target device topology?”

Without explicit artifact management, both categories accumulate without governance. The artifact store becomes a directory of outputs with no clear provenance: which pipeline run produced this? Was it validated? Against which version of the Source of Truth (SoT)? Is this the artifact currently deployed in production, or was it superseded? These questions become urgent during an incident and unanswerable without artifact management.

The platform should provide clear answers to four questions for every artifact: where is it stored, how is it versioned, who can promote it from one environment to the next, and what was its provenance (which pipeline run, against which inputs)?

Versioning follows the Semantic Versioning model: a patch increment changes something that does not affect the artifact’s output (a comment, an internal variable name); a minor increment adds new behavior without breaking existing consumers; a major increment changes existing behavior in a way that requires consumers to adapt. Without this discipline, a network engineer who pins a workflow at version 1.2 and returns six months later may be running version 1.9 without knowing what changed between them. The versioning scheme is the artifact store’s promise to its consumers; the Platform Contract pattern in section 10.8.3 extends that promise to the full platform API.

The following diagram shows the artifact lifecycle for platform and network changes, and how readiness increases as each gate is passed:

graph LR
    subgraph "Platform artifacts"
        PA1[Workflow image<br/>in registry]
        PA2[Integration-tested<br/>image]
        PA3[Staging-validated<br/>image]
        PA4[Production-released<br/>platform component]
    end

    subgraph "Network change artifacts"
        NA1[Template<br/>in VCS]
        NA2[Rendered config<br/>artifact]
        NA3[Simulation-validated<br/>artifact]
        NA4[Production-approved<br/>network change]
    end

    PA1 -->|Integration + compat tests| PA2
    PA2 -->|Staging gate| PA3
    PA3 -->|Release approval| PA4

    NA1 -->|Render + unit test| NA2
    NA2 -->|Integration + simulation gate| NA3
    NA3 -->|Release gate approval| NA4

Each arrow is a gate. A network change artifact that failed simulation should not be promotable to production-approved regardless of who attempts the promotion. A platform artifact that breaks backward compatibility should not reach the production release stage without a documented migration path for existing consumers.

10.3.3 Secrets Management#

Static credentials are the most common security failure mode in network automation platforms. An API key embedded in a pipeline config file, a service account password in an environment variable, a device password hardcoded in a workflow template: each exists independently of any workflow run. Each can be exfiltrated from the repository or the pipeline runner, and remains valid until someone manually rotates it. In a platform that runs dozens of workflow executions per day across production devices, the blast radius of a compromised static credential is the entire device fleet.

A leaked credential often lands in the VCS history and survives long after the file that held it is deleted. Run a secret-scanning tool in CI to catch this: gitleaks and TruffleHog scan both the working tree and the full commit history for exposed tokens and keys.

The alternative is a secrets management system that issues short-lived credentials on demand, scoped to a specific operation, and expires them automatically when the operation ends. Systems such as HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, or Kubernetes Secrets with external secret operators provide this capability; the choice depends on the platform’s hosting environment, but the required behavior is the same regardless of which system implements it. A workflow run that needs to connect to a device requests a credential from the secrets management system at execution time, uses it for the duration of the run, and the credential expires. The credential never appears in a configuration file, never lives in an environment variable that persists between runs, and cannot be reused after the workflow completes.

Short-lived, per-run credentials are the target. If a team cannot reach it immediately, the minimum acceptable step is to move every long-lived credential out of pipeline configuration files and into the secrets management system. A long-lived credential stored there is centrally controlled, access-audited, and rotatable from one place; the same credential hardcoded in a pipeline config is none of those. Storing long-lived secrets in the vault is the floor, not the goal.

Three properties define a correctly implemented secrets management integration for network automation:

• Per-run scoping: each workflow execution receives its own credential, issued at the moment the execution starts and expired when it ends. A credential issued for a VLAN deployment to Building B cannot be used by a concurrent workflow targeting a different site.
• Least privilege: the credential issued for a workflow carries only the permissions that workflow requires. A read-only validation workflow receives read-only credentials. A configuration deployment workflow receives write credentials scoped to the specific devices in the current wave. No workflow receives standing administrative access.
• Audit trail: every credential issuance and expiry is logged by the secrets management system, independently of the platform’s own audit log. If a credential is used outside a workflow context, that anomaly is visible.

The MVP section (10.8.4) identifies secrets management as stage 4 of the build sequence: the minimum security requirement before the platform operates against production devices. Stages 1 through 3 produce a well-organized platform; without this stage, it is not a secure one. Chapter 12 covers the access control model and audit requirements that build on this foundation.

10.4 CI/CD Architecture#

10.4.1 CI and CD Are Different Problems#

Teams often treat CI/CD as a single concept, but continuous integration and continuous delivery solve different problems and fail in different ways.

• Continuous integration asks: “Does this change work?” It validates that a proposed change is internally correct: the template renders, the tests pass, the schema is valid, the workflow does not conflict with existing definitions. A CI failure means something is wrong with the change itself. The correct response is to fix the change and re-run.

• Continuous delivery asks: “Should this change be deployed, and how?” It handles the mechanics of moving a validated artifact from the artifact store to the target environment: scheduling the deployment window, selecting the deployment strategy, managing the rollout sequence, monitoring for deployment failures, and handling rollbacks. A CD failure does not necessarily mean the change was wrong; it may mean the deployment environment was not ready, the target devices were unreachable, or the deployment strategy was misconfigured.

Conflating them produces pipelines that fail ambiguously and require platform team knowledge to diagnose. Separating them clearly means that a failure at the CI stage tells the engineer “your change has a problem”, while a failure at the CD stage tells the platform team “the deployment environment has a problem”.

10.4.2 The Pipeline as a DAG#

A network automation pipeline is a chain of dependencies. The Source of Truth (SoT) integrity check must pass before templates are rendered, because a rendered template based on a corrupt or incomplete SoT snapshot may be invalid, and the pipeline will not catch the error until later. Templates must be rendered before simulation can run, because the simulation needs a concrete artifact, not an abstract template. Simulation must pass before a deployment can proceed to production.

These dependencies form a directed acyclic graph (DAG): a structure where each step depends on one or more previous steps, no step depends on itself, and the dependencies never form a cycle. Treating the pipeline as a flat list of sequential steps throws away parallelism. If the unit tests for template rendering and the SoT integrity check are independent, they can run simultaneously. If the schema validation for three different configuration types is independent, all three can run in parallel. A pipeline designer who thinks in lists will serialize everything. One who thinks in DAGs will identify the true dependency boundaries and recover the parallelism within them.

Chapter 7 covered DAG patterns in orchestration workflows. The same principles apply here: the pipeline is itself an orchestrated workflow, and the Orchestrator patterns from Chapter 7 are directly applicable to pipeline design.

10.4.3 The GitOps Deployment Model#

GitOps is a deployment model in which the Version Control System (VCS) state is the authoritative record of what should be deployed. Any divergence between the VCS state and the deployed state is a deviation to be corrected, not a configuration choice to be preserved. No out-of-band changes to production; if a change does not go through the VCS, it did not happen. Tools that implement this model include ArgoCD and Flux for Kubernetes-hosted platform components: both continuously reconcile the running state against what the repository describes. For Ansible-centric network workflows, Git-backed job runners such as Ansible AWX or Semaphore approximate the model, executing playbooks from a versioned Git source on each qualifying commit, though they run on trigger rather than reconciling continuously.

This model is the natural fit for deploying the platform itself: pipeline definitions, IaC manifests, workflow logic, policy bundles, and execution environment images all live in VCS, all change through reviewed PRs, and all deploy automatically when a qualifying commit merges. Every pipeline runner configuration is traceable to a commit. Every policy update has a review record. Rolling back a bad platform change means reverting a commit, not reconstructing the state the platform was in before someone made an interactive change. For this class of artifact, GitOps provides strong guarantees at low operational cost, and it should be the default deployment model for every platform component from the start.

That is the platform side of the GitOps equation: VCS as the authoritative record for the infrastructure that runs the automation. The network side is more constrained.

The picture is different for network change data. Network configurations have rich internal relationships: a device belongs to a site, a site carries a set of VLANs, a VLAN is referenced by multiple access-layer switch configurations, a firewall rule references named service objects that themselves reference IP prefixes defined elsewhere. A relational Source of Truth (SoT) manages these relationships naturally: the database enforces referential integrity, queries can traverse relationships in any direction, and a single update propagates correctly to every record that depends on it. Representing the same data as flat files in a Git repository shifts that relationship management to the author of every PR: adding a new VLAN means updating the VLAN definition file, every switch configuration file that references it, and every firewall rule file that permits or denies traffic on it, in a single consistent commit. CI/CD can validate the result, but it cannot prevent the authoring complexity.

For most teams, a SoT database is the right intake path. The file-based approach works only when the data model is genuinely small and the team deliberately maintains relational consistency in files, not because a SoT database was unavailable. If you need to ask whether your data model is too complex for file-based GitOps, it is. The platform should treat a SoT record update as an equally valid trigger for the deployment pipeline as a VCS commit. In practice this is implemented through webhooks emitted by the SoT application on record save, a change-data-capture stream from the SoT’s backing database, or a polling endpoint the pipeline runner queries on a short interval; the pipeline runner accepts these events at the same intake gate as VCS push events.

The pipeline runner subscribes to VCS events and starts runs automatically. The integration is configured once at platform setup time; afterwards, every qualifying commit triggers a run without human initiation. This is what makes the GitOps model operational rather than aspirational: the VCS state is authoritative not because the team intends it to be, but because the pipeline runner enforces it automatically on every qualifying change.

10.4.4 API Design and Service Exposure#

The platform’s capabilities should be available through a clean API boundary. Other systems need to interact with it: ticketing systems that submit changes, approval workflows that gate deployments, monitoring systems that query pipeline status, reporting tools that aggregate historical data. An API boundary that is well-designed allows these integrations without requiring internal platform knowledge. Poor API boundaries force integrations to understand platform internals, creating tight coupling that makes evolution painful.

Two distinct API boundaries exist in the platform and should be designed separately:

• The external API is the interface the platform exposes to consumers: the network engineers, ticketing systems, approval workflows, and monitoring tools that submit changes, query status, and receive notifications. This API is the platform’s expression of the Presentation layer from Chapter 8, applied inward. It should be stable, versioned, and treated as a contract: consumers build integrations against it and cannot absorb breaking changes without coordination (remember, use semantic versioning!).

• The internal API is the interface between platform components: how the orchestrator enqueues tasks for workers, how workers report completion back to the scheduler, how the artifact store signals availability of a new artifact to the deployment engine, how the pipeline runner reports stage results to the monitoring system. This API is an implementation detail of the platform, not a consumer contract. It can evolve faster than the external API, but it still requires explicit schemas and versioning to prevent silent incompatibilities between platform components that are deployed independently.

A third consumer type is emerging alongside human operators and system integrations: AI agents. An AI assistant that helps a network engineer diagnose a routing issue, propose a configuration change, or validate a rollback plan needs structured access to the same platform capabilities that a ticketing system or a monitoring tool uses. The Model Context Protocol (MCP) is an open standard that defines how AI agents discover and invoke external tools and data sources. It uses JSON-RPC over stdio or HTTP: the server exposes named tools and resources, and the agent calls them. An MCP server exposed by the automation platform allows an AI agent to query pipeline status, retrieve artifact provenance, submit a change for validation, or read SoT records without requiring the agent to understand the platform’s internal architecture.

From the platform’s perspective, an MCP interface is an additional surface on the external API: it should be versioned, access-controlled, and subject to the same contract discipline as the REST or webhook interfaces. Designing the external API as a clean, well-documented boundary from the start makes MCP exposure straightforward later: the MCP server describes the existing API operations to the agent rather than requiring a separate integration. Chapter 17 examines how AI agents interact with the automation platform in the context of autonomous network operations.

The specific design of the external API, REST, event-based, webhook-driven, MCP, depends on the platform’s architecture and the consumers it serves. The protocol matters less than the boundary existing and being treated as a contract. Changes to the API that break existing consumers should go through a versioning and deprecation cycle, not be applied unilaterally because an internal team made an infrastructure decision.

10.4.5 What Can Go Wrong#

A CI/CD pipeline can fail to deliver the value it was designed for in many ways. Four failure modes appear consistently in network automation pipelines:

• The stale snapshot problem. A pipeline without ordering between the SoT integrity check and the template rendering step may render against a data snapshot that is minutes or hours old. If the SoT was updated in the window between the snapshot and the render, the rendered artifact does not reflect the current intended state. The artifact passes its own validation, is deployed, and produces a network state that differs from what the SoT describes. Nobody is alerted because the deployment succeeded.

• The bypass problem. A deployment model without main-branch protection lets engineers push to production without going through the pipeline. This is often done under time pressure, with good intentions. The result is a deployed state that has no test record, no review record, and no artifact in the artifact store. When something goes wrong later, the change is invisible to every diagnostic tool that reads the artifact store or the commit history.

  The answer to the bypass problem is not to make bypass technically impossible. It is to design a legitimate emergency path that is audited, authorized, and traceable. A network engineer who needs to apply a change at 3am, when the platform is unavailable, no reviewer is around, or the pipeline is broken, needs a way to act. If no legitimate path exists, they will find an illegitimate one, and the result is an untracked change with no audit record.

  The designed alternative is a Break-Glass Procedure procedure: direct device access outside the platform, restricted to named individuals, requiring explicit authorization (a ticket, an on-call approval, a second engineer sign-off), and mandating that every action taken is logged to a separate, tamper-evident audit trail. The break-glass path is not a shortcut; it is a controlled exception. The critical property is that it is visible: a break-glass event generates an alert, a post-incident review is expected, and the SoT is reconciled against the manual change before the next platform-driven deployment touches the affected devices. Chapter 12 defines the governance model and audit requirements for break-glass access in detail.

• The idempotency gap. A pipeline without idempotency validation may apply a change twice if re-run for any reason: a transient failure, a manual retry, a duplicate trigger. If the change is not idempotent, applying it twice produces a different result than applying it once. In network automation, that different result is a different network state, with no clear signal that something unexpected happened. Section 11.3.1 covers the design response.

• The platform component failure mode. The first three failure modes concern workflow logic. This one concerns the platform’s own infrastructure. An artifact store that is unreachable mid-promotion, a pipeline runner that crashes between the simulation gate and the deployment gate, a secrets management system that cannot issue credentials when the workflow starts: these are not workflow errors. They are platform failures. The correct design principle is fail-closed: if any platform component required for a pipeline stage is unavailable, the stage fails and the pipeline halts. A deployment that cannot confirm its artifact is correctly stored in the artifact store should not proceed. Platform component availability is a prerequisite for the pipeline’s correctness guarantees, and that dependency must be explicit in the pipeline’s failure handling rather than assumed.

10.5 Testing and Validation#

Testing is a multifaceted problem. In Part 3, we tackle three dimensions of correctness. Platform testing targets functional correctness. Does the change do what it is supposed to do? This is distinct from the resilience testing that Chapter 11 addresses (“does it survive pressure?”) and the compliance testing that Chapter 12 addresses (“is it safe and within policy?”). All three are necessary; they should be designed as separate concerns, run at appropriate stages, and interpreted with the right frame of reference.

10.5.1 Testing the Platform or the Network? Both.#

Again, there are two testing subjects: the platform has software to test, and the network has network state to validate. They require different approaches, different environments, and different success criteria:

• Testing the platform validates that the platform’s own code works correctly: templates render the right output, workflows interact correctly with the building blocks they depend on, validation rules flag the right errors. These are software tests in the traditional sense: they run in a CI pipeline, they require no network devices, and their subject is the platform’s logic.

• Testing the network is not a software test at all. It is the full automation architecture running in validation mode against a non-production environment. The same pipeline executes, the same executor applies the rendered configuration, the same observability layer collects telemetry and validates the result. The only difference from a production deployment is the target: a lab topology, an emulated environment, or a dedicated staging fleet rather than the production network. There is no separate “simulation test framework”; the test environment IS the architecture exercised against a controlled target. This distinction matters because it determines what the platform team is responsible for: they write and maintain software tests for the platform, but they configure and operate a real automation environment for network validation.

The pipeline should enforce a strict ordering between these two tracks. Software tests (unit, then integration) run first because they are fast, require no network infrastructure, and catch the class of error that makes running the full architecture against a test environment pointless. Network validation runs only after both software test levels have passed. There is no value in deploying a misconfigured template to a lab switch when a unit test would have caught the error in three seconds.

Platform software tests:

• Unit testing addresses the smallest composable unit of the automation: does this template render correctly for this input? Does this function return the expected output for a given argument? Does this validation rule correctly flag the invalid case and pass the valid one? Unit tests run fast, require no external dependencies, and fail with messages specific enough to identify the problem without a debugger. A test that takes ten seconds and requires a database connection is not a unit test; it is a slow integration test.

• Integration testing addresses whether the workflow interacts correctly with the building blocks it depends on: does the template rendering workflow correctly read from the Source of Truth (SoT) and produce an artifact the Executor can accept? Does the observability hook correctly emit a pipeline completion event to the metrics pipeline? Integration tests require the target systems to be available, which makes them slower and more environment-dependent. They run against a test instance of each building block, not against production.

Network validation:

The validation stage runs the full deployment pipeline against a test network environment. The rendered configuration is applied by the Executor to real or emulated devices of the same type as the production fleet. The Observability layer collects telemetry and the pipeline evaluates the same validation gates it would use in production: is the VLAN present in the VLAN table? Do error counters show a spike? Does the routing table reflect the intended change? This stage catches the class of problem that no software test can see: the firmware-specific error code, the vendor-specific response format, the interface behavior that is correct according to the standard but implemented differently on this device platform, as described in Chapter 9.

The test environment does not need to match production at full scale; it needs to match production in device type diversity, covering at least one device of each vendor type present in the production fleet and the most relevant roles these devices play in the network. For brownfield fleets that span multiple firmware generations on the same model, a common condition that section 11.3.5 covers in detail, include at least one device per firmware generation with documented management-plane differences, not just one per vendor. A lab of five switches spanning all three vendor types catches more vendor-specific edge cases than a lab of fifty homogeneous devices; a second switch per vendor at a different firmware generation catches the class of failures the first will miss.

Each gate in this diagram catches a different class of problem. The ordering matters: earlier gates are cheaper to fail (no deployment has started, no simulation environment has been allocated). Later gates are more expensive but catch problems that earlier gates cannot see.

graph TD
    I[Change intake<br/>SoT update / API call / policy PR]
    G1{SoT integrity<br/>check}
    GD{Drift<br/>detection}
    G2{Schema<br/>validation}
    G3{Unit tests}
    G4{Integration<br/>tests}
    G5{Simulation<br/>gate}
    G6{Release gate<br/>approval}
    D[Deployment]
    F1[Fail: fix SoT data]
    FD[Fail: investigate<br/>drift]
    F2[Fail: fix schema]
    F3[Fail: fix template<br/>or workflow logic]
    F4[Fail: fix block<br/>interactions]
    F5[Fail: fix<br/>network behavior]

    I --> G1
    G1 -->|Pass| GD
    G1 -->|Fail| F1
    GD -->|Pass| G2
    GD -->|Fail| FD
    G2 -->|Pass| G3
    G2 -->|Fail| F2
    G3 -->|Pass| G4
    G3 -->|Fail| F3
    G4 -->|Pass| G5
    G4 -->|Fail| F4
    G5 -->|Pass| G6
    G5 -->|Fail| F5
    G6 -->|Approved| D

In the next sections, we will deep dive more on them.

10.5.2 Pre-Deployment Validation Gates#

The gates described here apply to both categories of network deployment: changes to existing network configuration (modifying a firewall rule, updating a BGP community value) and new network provisioning (adding a site, onboarding a new device). The validation logic is the same; the inputs differ. For a configuration change, the SoT integrity check verifies that the affected devices are present and their current attributes are complete. For a new provisioning run, it verifies that all the data required to bootstrap the new device (site assignment, management IP, vendor credentials) is in the SoT before any deployment command is issued.

Three validation checks should run as explicit gates before any deployment proceeds (in this order):

• SoT integrity check. Are all devices referenced by the change present and complete in the Source of Truth (SoT)? Are all attributes required by the template populated? A change that references a device that no longer exists in the SoT, or a site that has not yet been added, will fail at deployment in an uninformative way. The SoT integrity check surfaces this problem at the pipeline stage, with a message that names the missing reference. For the financial services engineer’s firewall rule from this chapter’s opening, this check would verify: the target firewall device is registered in the SoT with its management IP and platform credentials, the source network object (the internal services subnet) is defined as a service record, and the authentication endpoint is registered as a known destination service. A change that references a network object renamed three weeks ago fails here, with a message naming the stale reference, not a few days later during deployment.

• Drift detection. Does the current state of the target devices match the last known good state recorded by the platform? A drifted device may respond unexpectedly to a configuration change. Deploying to a device in a drifted state risks compounding the drift or masking it further. Stop and investigate; the drift may have a legitimate cause. Do not deploy and hope idempotency covers the gap.

• Schema validation. Does the rendered artifact conform to the expected schema? This check gates the artifact store: a structurally malformed artifact should not be stored or promoted to simulation. A malformed artifact that passes schema validation has a genuine content error that simulation may catch. A malformed artifact that fails schema validation has a structural error the artifact store should reject.

In brownfield environments, applying drift detection as a global gate from day one is impractical: almost every device will have some deviation from any newly defined desired state, and a gate that blocks all of them blocks the platform from being used at all. The pragmatic approach is scoped drift detection: the gate checks only the specific attributes that the current change touches, not the device’s entire configuration state. A workflow adding a VLAN checks whether the VLAN table on the target switches matches the platform’s record for that attribute. It does not check whether the spanning tree priority, the NTP configuration, or the management ACL have drifted. Coverage grows incrementally: as each workflow is onboarded to the platform, the attributes it manages are added to the drift detection scope. Over time, the platform’s drift visibility expands workflow by workflow, without requiring a complete audit of every device before a single change can be made. This is the standard pattern for brownfield adoption: start narrow, prove the gate, extend coverage.

10.5.3 Observability Hooks in the Pipeline#

The pipeline is itself a system with observable state. How long does each stage take? What is the failure rate per stage? How deep is the queue at peak times? How long does an engineer wait from submission to first feedback?

These questions cannot be answered without instrumenting the pipeline itself. Chapter 6, section 6.2.8, covers observability of the automation system as a distinct concern from observability of the network. The platform team is responsible for implementing the instrumentation that makes the pipeline’s own health visible: stage duration, failure rate, queue depth, and time-to-feedback. Without this instrumentation, the platform team is operating their own system without the observability they prescribe for the network. This is also called Developer Experience, and it’s a key domain in any software engineering organization.

10.6 Release and Deployment Strategies#

10.6.1 Blue/Green for Platform Components#

The automation platform’s components should be releasable without disrupting in-flight operations. Deploying a new execution engine version that requires downtime affects every workflow running at that moment. A Blue/Green Deployment deployment avoids this disruption: it runs two versions simultaneously, routing new work to the new version while the old one drains and is decommissioned.

Invest in blue/green the moment any platform component has in-flight stateful operations that cannot safely be interrupted. In practice, that moment is the first time a multi-wave workflow runs in production. Before that, a scheduled maintenance window with a brief platform pause is acceptable: stateless pipelines can be interrupted and restarted without consequence. Once multi-wave workflows are running, a mid-flight wave cannot be abandoned: devices in wave 1 have already been changed, wave 2 has not started, and the workflow must complete or roll back cleanly. From that point, a platform upgrade that interrupts in-flight operations is not a maintenance event; it is an incident. Invest in blue/green before the first multi-wave workflow runs in production, not after.

The cutover moment requires explicit coordination. The platform cannot simply swap versions and assume in-flight workflows will survive the transition. Before cutover, the blue version must be drained: new workflow submissions are routed to the green version, and the blue version stops accepting new work. Workflows already running on blue are allowed to complete. Only when the blue version reports zero active workflows is it safe to decommission it.

Draining requires the platform to have an accurate count of in-flight workflow runs per component instance. This is not a trivial instrumentation requirement. An orchestrator that does not expose a real-time count of active executions cannot be safely drained; the platform team has no signal for when it is safe to proceed. The drain state itself must be observable: a dashboard metric showing active workflows on blue trending to zero, not a timer or an assumption.

For network change workflows specifically, draining is more consequential than for stateless application deployments. A workflow that has applied a change to wave 1 devices and is waiting for validation before proceeding to wave 2 is not safe to interrupt. Cutting over the orchestrator mid-wave leaves devices in a partially changed state with no workflow tracking their progression to completion or rollback. The correct behavior: a change workflow in progress must complete on the version of the orchestrator that started it. Cutover windows should be scheduled when no multi-wave workflows are active, or the blue/green switch deferred until those workflows reach a natural checkpoint.

The same strategy applies to any stateful platform component: the orchestrator, the artifact store, the observability collection agents. Platform components are not special relative to the applications they support. They should be managed with the same release discipline, with the same drain-before-decommission requirement.

10.6.2 Progressive Rollout and Platform Component Upgrades#

Progressive rollout applies to both subjects this chapter covers (i.e., platform and network), but the mechanics differ significantly between them.

Network changes: progressive wave rollout

A change targeting the full device fleet should not be applied to all devices simultaneously. The correct model is a progressive wave rollout: deploy to a small subset, validate, expand to a larger subset, validate again, and continue until the full fleet is covered. This is the same pattern that Chapter 7 describes as the structural basis for the Saga Pattern applied to network changes, and that Chapter 9 develops in the simulation context as pre-production validation before the first wave. Here, the focus is on the platform’s role in enforcing wave progression.

A VLAN deployment to 800 campus switches might structure its waves as: 5 pilot switches (one per vendor type) → 1 full building (24 switches) → 3 buildings → remaining fleet. Each wave is a workflow execution boundary: the Orchestrator applies the change to the wave, halts, runs the validation gate, and proceeds to the next wave only if validation passes.

What does wave validation look like in practice? The platform queries the Observability layer, not the devices directly. After deploying VLAN 210 to the pilot switches, the platform checks: does the Observability layer report VLAN 210 present in the VLAN table on all five target switches? Do error counters on the affected ports show no spike relative to the pre-change baseline? Do the hosts that were assigned to VLAN 210 have reachability to the gateway? These checks run automatically as a pipeline stage, but only after an explicit convergence window defined in the workflow, because networks take time to reach a consistent state and a gate that runs before convergence may certify a wave that has not stabilized. If all checks pass, the Orchestrator releases the next wave. If any fail, the deployment halts and one of two things happens: an automated rollback, or the on-call engineer receives the specific failure signal, not just “wave failed”.

This is why the Observability layer is a hard dependency for progressive rollout, not a nice-to-have. A platform that applies changes and polls devices directly for validation is coupling its deployment logic to vendor-specific data formats. A platform that applies changes and queries the Observability layer, which has already normalized telemetry from all three vendors into a common model, can run the same validation logic regardless of vendor. The validation gate is defined once per change type when the workflow is onboarded, not written ad hoc for each deployment.

Three design decisions must be made before a wave-based rollout runs:

1. How are waves sized and scoped? Site-based selection contains blast radius geographically. Vendor-stratified selection ensures each vendor is represented in early waves, catching vendor-specific edge cases before they reach the full fleet. The selection strategy should be defined in the workflow definition, not chosen at deployment time.
2. What validation defines success? Each workflow type defines its own validation gate: the specific Observability queries and thresholds that constitute “this wave succeeded”. A VLAN workflow’s gate is different from a routing policy workflow’s gate. Defining it per workflow type, upfront, is what makes automated wave progression safe.
3. What triggers an abort? A single failed device in a 5-device wave is a different signal from a single failed device in a 400-device wave. The abort threshold should be defined as a ratio, not an absolute count, and set before the deployment starts.

Before each wave deploys, the platform takes a configuration snapshot of every device in that wave’s scope. The snapshot is per-wave, not per-job: by the time wave 2 is ready to start, wave 1 devices may have received changes unrelated to the current job, and a rollback to the pre-job state would revert those alongside the job’s own changes. Taking the snapshot immediately before each wave starts means the rollback baseline is always the state that wave found on the devices, not the state the job started from.

When wave 2 fails and the abort threshold is crossed, the platform applies the pre-wave-1 snapshots automatically to all devices that received the wave 1 change. The rollback uses the same deployment infrastructure as the forward change: it applies the snapshot, runs the post-deploy observability validation gate, and confirms that each device has returned to its pre-change state before marking the rollback complete. The rollback is not a declaration; it is a verified reversal.

One specific failure mode requires explicit handling. If the rollback fails on any device because the device is unreachable or its management plane is not accepting connections, the platform halts and alerts rather than continuing to apply the rollback to the remaining wave 1 devices. A single device in a known pre-change state is a recoverable situation. A partial rollback, where some devices have been restored and others have not with no clear record of which is which, is not: it combines two different intended states without a boundary between them.

Wave-based rollout is sometimes described as a risk-reduction technique for speed: rolling out faster because blast radius is smaller. That framing is wrong, and it matters to correct it before the team builds deployment policy around it. The reason to gate wave progression on validation is not to deploy quickly. It is to deploy with earned confidence. Each wave is a trust checkpoint: the validation gate does not ask “are we moving fast enough?” It asks “do we have sufficient evidence that the change behaves correctly on this class of device, at this scope, in this part of the network?” If the gate passes, the platform has earned the right to extend the change to the next wave. If it fails, the deployment stops: not pauses, not retries without investigation, stops. A wave boundary that can be overridden under time pressure is not a gate; it is a suggestion.

Moreover, be aware that networks are distributed systems, and these take time to reach a consistent state after a change is applied. A routing policy pushed to a BGP speaker propagates to its peers over seconds or minutes depending on session count and hold timers. A validation gate that runs immediately after the configuration push may observe a state the network has not yet finished computing. The wave gate must account for this: the timeout parameters in the post-deploy observability check are not arbitrary; they reflect the expected convergence time for the change type, and shortening them to speed up the pipeline risks certifying a wave that has not actually stabilized. Build the convergence window into the workflow definition, per change type, and treat it as part of the gate specification rather than a tunable that gets reduced when the pipeline feels slow.

Every correct gate decision earns trust. That trust is what eventually justifies shorter maintenance windows, higher change frequency, and on-demand deployment. Treat the gate as optional once and the accumulation resets.

graph TD
    S[Take pre-wave snapshots<br/>for all target devices]
    W1[Deploy Wave 1]
    V1{Wave 1<br/>validation}
    W2[Deploy Wave 2]
    V2{Wave 2<br/>validation}
    AT{Abort threshold<br/>crossed?}
    W3[Deploy Wave 3...]
    RB[Auto-rollback Wave 1:<br/>apply pre-wave snapshots]
    RV{Rollback<br/>validation}
    OK[Recovery confirmed]
    RF[Halt: alert on-call<br/>device unreachable]

    S --> W1
    W1 --> V1
    V1 -->|pass| W2
    W2 --> V2
    V2 -->|pass| AT
    AT -->|no| W3
    AT -->|yes| RB
    RB --> RV
    RV -->|all devices restored| OK
    RV -->|rollback failed on device| RF

Platform component upgrades

Upgrading the execution engine, the orchestrator, or the SoT itself is a different class of change. A new version of the execution engine may change how it connects to devices. A SoT schema migration may change the data model that every workflow reads. A dependency upgrade, a new version of the automation framework or the SoT application, may introduce behavioral changes that are not visible until workflows run against production data.

The platform component upgrade path requires its own progressive strategy. For stateless components such as a new execution engine container, a canary approach routes a small fraction of incoming workflow runs to the new version before promoting it. The check is simple: does the new version’s error rate and latency match the stable version across a sufficient sample of runs? If yes, the new version replaces the old. If no, traffic is shifted back and the release is held.

For stateful upgrades, particularly SoT schema changes, the sequencing is more constrained. A SoT schema migration that is not backward-compatible must be coordinated with all workflows that read or write the affected schema fields. The migration pattern: deploy the new schema in a backward-compatible form first (add new fields alongside old ones), migrate workflows one at a time to use the new fields, then remove the old fields once no workflow depends on them. Attempting to cut over schema and workflows simultaneously in a single release is the most common cause of platform-level outages during upgrades.

The same wave discipline applies when the platform itself is distributed. A collector agent deployed per site, an observability pipeline running at each regional hub, or a Zero Touch Provisioning service operating at the edge are not a single component to upgrade in one operation: they are a fleet, and they carry the same blast radius logic as a network change targeting 800 switches. Upgrading the collector agent at all 40 sites simultaneously means that if the new version has a bug in how it parses a vendor-specific telemetry stream, every site loses observability in the same maintenance window. Applying the wave approach, pilot sites first, then a regional subset, then the full fleet, with the same gate logic that governs network changes, bounds the failure scope to the pilot wave and preserves observability at the sites not yet upgraded. The platform component upgrade is not exempt from the trust model because the components being upgraded are platform infrastructure rather than network configuration. If anything, the stakes are higher: a failed network change affects a subset of devices; a failed platform component upgrade can affect the platform’s ability to validate and recover from any subsequent change.

Chapter 11 covers the reliability impact of progressive rollout in the context of at-scale operations. Chapter 12 covers the risk containment angle: the wave boundary is a policy enforcement point as much as a reliability tool.

10.6.3 Maintenance Windows and Deployment Scheduling#

Continuous delivery does not mean continuous deployment. Most enterprise networks operate within scheduled maintenance windows: a CAB-approved time slot during which production changes are permitted. The platform’s pipeline runs whenever a change is approved, often during business hours, but the actual deployment to production devices may be constrained to a Friday night window or a pre-dawn Saturday slot.

This distinction, pipeline execution versus deployment execution, is what makes the artifact store from section 10.4 architecturally load-bearing. The pipeline produces a validated, labeled artifact. The artifact waits in the store. The deployment executes when the maintenance window opens. The artifact does not expire or become invalid because time passed; it remains exactly as validated. If the SoT changes between validation and deployment (e.g, a device is removed, a site attribute renamed) the pre-deployment drift detection gate (10.5.2) surfaces the discrepancy before any device is touched; the artifact is not re-rendered, but the gate prevents it from being applied against a state it was not built for. What changes is the deployment gate: a time-based check or a manual promotion step that prevents execution outside the approved window.

The pipeline’s validation work and the deployment execution are two separate moments, separated by the artifact store. The diagram below shows that boundary explicitly.

graph LR
    subgraph "CI/CD Pipeline"
        C1[Change intake]
        C2[Validation gates]
        C3[Release approval]
        C4[Artifact labeled<br/>production-approved]
    end

    subgraph "Artifact Store"
        A1[Artifact waits<br/>until window opens]
    end

    subgraph "Deployment"
        D1[Maintenance window<br/>opens]
        D2[Artifact retrieved<br/>and deployed]
        D3[Post-deploy<br/>validation]
    end

    C1 --> C2 --> C3 --> C4 --> A1
    A1 -->|Time gate or<br/>manual promotion| D1
    D1 --> D2 --> D3

The simpler implementation is a manual promotion action in the artifact store: a human explicitly releases the artifact for deployment when the window opens. This is the right default for most teams. The alternative, a pipeline stage that polls until the window is active and then proceeds automatically, adds scheduling logic to the pipeline and creates a long-running process that must survive overnight without interruption. That complexity is only worth adding once the volume of concurrent deployments makes manual release a bottleneck. Either way, the design separates the question “is this change correct and approved?” from the question “is now the right time to apply it?” Conflating them produces platforms that either bypass change management or block themselves waiting for windows they do not track.

The maintenance window model also clarifies what “continuous” means for network automation: continuous validation, continuous readiness, and deployment at whatever cadence the organization’s change management process allows. Teams that move from monthly maintenance windows to weekly, then nightly, then on-demand do so by building confidence in the platform’s validation gates, not by abandoning the window discipline.

10.6.4 Rollback and Failure Handling#

Rollback in network automation is more complicated than rollback in software deployment. A container deployment rolled back is the previous container version restarted. A rendered network configuration rolled back in the artifact store is an artifact in a database; the network devices are still in whatever state they were put into by the failed deployment.

The rollback path for a network change must be designed explicitly before the deployment begins. What is the previous known-good state for each target device? Is that state stored as a configuration snapshot in the artifact store, or must it be re-rendered from the Source of Truth (SoT) at rollback time? Can the rollback be applied atomically, or does it require the same staged deployment logic as the original change? What happens to devices that successfully applied the change if the rollback is triggered by a failure on a different device?

The previous known-good state is a configuration snapshot taken immediately before the deployment, stored in the artifact store with the same provenance record as the forward artifact, as described in 12.4.2. It is not re-rendered from the SoT: the SoT may have changed between the forward deployment and the rollback trigger, and rendering against the current SoT state produces the wrong baseline. The rollback applies the snapshot directly to the device, using the same deployment infrastructure as the forward change. For single-device deployments, this is atomic: the snapshot is applied and the platform validates that the device has returned to its pre-change state before closing the rollback. For multi-device deployments that are not wave-based (targeting a set of independent devices rather than a sequenced rollout), rollback applies the snapshot to each device independently and in parallel; a failure to roll back any individual device halts the operation and raises an alert rather than completing the partial rollback silently. The platform records the rollback outcome the same way it records the forward deployment: in the artifact’s provenance record, with a timestamp and outcome for each device.

I found the commit-confirm feature particularly useful: it automatically rolls back a change that is not confirmed within a set time. This simplifies rollback and gives a clear confirmation step to complete network state validation.

A platform that treats rollback as an afterthought will discover its answer to these questions under incident conditions, which is the worst time to reason about them.

10.7 Operating the Platform#

The discipline of operating a shared platform as a reliable service has a substantial body of practice in software engineering. Site Reliability Engineering (Niall Richard Murphy et al., O’Reilly, 2016) is the most directly applicable reference: its treatment of error budgets, toil elimination, and service level objectives maps cleanly onto the platform health metrics and developer feedback loops in this section. The SRE model was developed for large-scale software infrastructure, but the core principle transfers without modification: the team that operates a shared system is accountable for its reliability as a service, not just its correctness as a tool.

10.7.1 Platform Health Metrics#

A platform team that cannot observe their own platform’s health is in the same position as a network team that cannot observe their own network’s health. The metrics that matter for platform operations are different from the metrics that matter for network operations, but the principle is identical: instrument what you care about, define what healthy looks like, alert when it deviates.

These are the most relevant platform health metrics:

• Pipeline duration: how long does a complete pipeline run take, from submission to result? A pipeline that is getting slower over time is a pipeline that is losing adoption. Engineers stop waiting for feedback and start batching changes or circumventing the platform.
• Failure rate by stage: which stage fails most often? A high failure rate at the SoT integrity check stage indicates SoT data quality problems. A high failure rate at the simulation stage indicates that simulation environment coverage is not keeping pace with network changes. Each stage’s failure rate points to a different upstream problem.
• Queue depth: how many changes are waiting for pipeline capacity? A queue that grows faster than it drains indicates insufficient pipeline capacity. A queue that spikes at specific times indicates that pipeline capacity is not matched to usage patterns.
• Time-to-feedback: how long from submission to first actionable result? This is the metric most directly visible to the user and most directly correlated with platform adoption. For the campus platform in the running example, a VLAN add workflow targeting Building B’s 24 switches should return its first actionable result (the SoT integrity check outcome) within 60 seconds of submission, and a complete simulation result within 8 minutes. A pipeline that consistently exceeds 20 minutes for that scope is training engineers to disengage.

10.7.2 Developer Feedback Loops#

The time-to-feedback metric deserves specific attention because of how strongly it shapes engineer behavior.

A pipeline that returns a result in three minutes trains engineers to submit a change, look at something else for a few minutes, and return to a clear pass or fail signal. The platform is responsive. Engineers develop a natural workflow rhythm around it.

A pipeline that returns a result in forty minutes is past the threshold where most engineers hold the problem in working memory: they move on, and must reconstruct context before they can even read the result. If the result is a failure, re-acquiring context to understand the failure message takes additional time. Some engineers respond by batching changes to reduce the number of round trips; this concentrates risk and makes individual failures harder to isolate. Some engineers respond by skipping the platform for changes they are confident about; this breaks the audit trail and removes the safety gates.

Feedback speed is not a performance optimization. It is a trust signal and an adoption driver. A platform that engineers trust gives fast, useful feedback. A platform that engineers route around has failed as a product, regardless of how correctly it validates changes.

10.7.3 Multi-Team Collaboration#

As the platform matures, security teams will need to add policy gates, site reliability engineering (SRE) teams will want alerting hooks, and domain teams will need custom validation stages for their specific workflows. The platform team cannot be the only path for all of these contributions; that bottleneck scales inversely with the platform’s adoption. The sustainable model: the platform team owns the core pipeline and the golden path; other teams contribute extensions through the same review process that governs all other changes.

Organizational ownership of the pipeline definition itself requires a clear decision: who has merge rights to the pipeline configuration? The answer should be documented and enforced through access control, not left implicit. A pipeline that anyone can modify without review is a pipeline that will be modified without review under time pressure.

Not every organization starts with a dedicated platform team. Many begin with one or two engineers who own the platform alongside their regular network responsibilities. The platform engineering model described in this chapter is a target state, not an entry requirement. The discipline matters more than the headcount: a single engineer can maintain a minimal golden path, a one-pipeline CI/CD workflow, and a basic artifact store. As adoption grows and the platform demonstrates value, the team structure follows. Chapter 13 addresses the organizational and cultural conditions that make that growth possible.

10.8 Evolution and Adoption#

10.8.1 Standardization Versus Flexibility#

The platform’s golden paths represent the team’s accumulated knowledge about how to do things correctly. They encode the lessons from failed deployments, the patterns that have proven reliable, and the conventions that reduce cognitive overhead when reading a workflow someone else wrote.

Enforcing those conventions benefits from the same distinction used in Chapter 12 between guardrails and enforcement gates. A guardrail is convention: the golden path is recommended, documented, and supported. An enforcement gate is a pipeline gate that rejects configurations that violate the policy. Chapter 10 handles conventions. Chapter 12 handles enforcement.

The design question for the platform team is which conventions are important enough to enforce. Enforcing a naming convention for pipeline stages is low-value enforcement; it makes the pipeline more uniform but does not prevent failures. Enforcing a requirement that all production deployments pass simulation is high-value enforcement; it prevents a whole class of production incidents. The platform team should invest enforcement effort where it buys safety, not where it produces consistency for its own sake.

10.8.2 Migration Strategies#

Most teams adopting the platform have existing automation that was not built with the platform in mind. The temptation is to treat migration as a project: scope it, rewrite everything, and switch over on a specific date. This approach consistently fails for the same reason that “we’ll refactor everything next quarter” consistently fails in software engineering. The scope is larger than it appears, the timeline slips, and the team ends up maintaining both the old approach and a partially complete migration indefinitely.

The alternative is incremental onboarding. Identify the workflow with the highest change frequency and the highest risk: the one where failures are most costly and where the platform’s validation would provide the most immediate benefit. High frequency means the team sees results quickly: within weeks, the workflow has run enough times to expose edge cases the scaffold did not anticipate, and the team has evidence that the platform works or evidence of what needs fixing. High risk means the platform’s simulation gate and policy checks deliver genuine value rather than ceremony: if the workflow could cause a production incident when wrong, having it fail safely in simulation is a meaningful improvement over the scripts approach. Migrate that workflow first, but run it end-to-end through the platform against the simulation environment before routing production traffic through it: the goal is to confirm the platform handles this workflow type correctly before production exposes any gaps. Use it to validate the platform’s golden path, to identify the edge cases the scaffold did not anticipate, and to demonstrate to the team that the migration is achievable. Then extend to the next workflow.

Each migrated workflow makes the next one easier: the patterns are documented, the team’s familiarity with the platform increases, and the golden path evolves to accommodate real cases rather than hypothetical ones. The migration is never “done” in the sense of a completed project; it is an ongoing process of bringing more workflows into the platform’s managed surface as the team’s confidence in the platform grows.

10.8.3 The Platform Contract Pattern#

As the platform serves more teams and more workflows, the platform team faces a new category of problem: how to evolve the platform without breaking consumers who depend on its current behavior.

A platform that can change its API without notice is a platform that consumers cannot rely on. A consumer who builds a workflow against a specific version of the pipeline API, and returns six months later to find that the API has changed in a way that breaks their workflow, has experienced a service failure. The platform team may have had good reasons for the change. The consumer may have had no way to know the change was coming. The result is breakage and eroded trust.

The Platform Contract pattern addresses this: the platform team makes an explicit commitment to what the platform provides, how it behaves, and how it evolves. The contract includes a versioned API with documented semantics, a deprecation policy that announces breaking changes in advance and provides a migration path, and a timeline for when deprecated versions will stop being supported. This Platform Contract is already implicit when using commercial tools like Github Actions.

The Platform Contract applies the same service discipline that Chapter 14, section 14.2, describes for automation services generally. The platform is itself an automation service. It should be managed under the same product lifecycle that Chapter 14 describes: definition, delivery, operations, and evolution, with platform team ownership at each stage.

10.8.4 The Minimum Viable Platform#

A network engineer who finishes this chapter understands what the complete platform looks like. The question that remains is where to start building it, given a real network to keep running, a team that is learning as it goes, and no appetite for a platform that does not work until every component exists. The answer is a five-stage build sequence. Each stage closes a specific operational gap left open by the scripts approach. Each stage is independently deployable and immediately useful. Each stage creates the precondition that makes the next stage achievable.

graph TD
    S1[Stage 1<br/>VCS + CI pipeline<br/>syntax validation and unit tests]
    S2[Stage 2<br/>Artifact store<br/>versioned rendered configs]
    S3[Stage 3<br/>Simulation gate<br/>minimal lab environment]
    S4[Stage 4<br/>Secrets management<br/>per-run scoped credentials]
    S5[Stage 5<br/>Audit log<br/>provenance chain per change]

    S1 -->|Closes: scripts in home directories| S2
    S2 -->|Closes: no rollback baseline| S3
    S3 -->|Closes: untested production changes| S4
    S4 -->|Closes: static credentials in pipelines| S5
    S5 -->|Closes: unauditable change history| DONE[Governed platform]

    style S1 fill:#e8f4e8
    style S2 fill:#e8f4e8
    style S3 fill:#e8f4e8
    style S4 fill:#fff3e0
    style S5 fill:#fce4ec

1. Version Control System (VCS) for all automation code, plus a single CI pipeline that runs syntax validation and unit tests on every commit. This is the stage that closes the scripts-in-home-directories problem. Every workflow lives in a repository, every change has a history, and every execution runs in a consistent environment rather than on whoever’s laptop happens to be available. You gain the code history and repeatable execution environment that all subsequent stages require. This stage is complete when every automation workflow runs in CI and no automation code exists outside the repository: no scripts in home directories, no shared drives, no per-engineer laptops.

2. An artifact store for rendered configurations. The pipeline now stores its outputs rather than discarding them after execution. This closes the rollback gap: when a change needs to be reversed, the previous artifact exists and can be restored without re-rendering from a SoT snapshot that may have changed. It also provides the first layer of the audit trail: a versioned record of what the pipeline produced, when, and from which inputs. This stage is complete when every deployment produces an artifact with a retrievable version identifier and any past deployment can be identified by that ID.

3. A simulation gate against a minimal lab environment. Even a small number of switches in a rack, or a limited emulated topology using the same vendor device types as the production fleet, is sufficient to catch the class of errors that syntax validation cannot: template logic errors, missing SoT references, idempotency failures that only appear against a real management plane. At this stage the platform is genuinely safer than the scripts approach, not just more organized. The lab does not need to match production at full scale; it needs to match production in device type diversity. This stage is complete when no change reaches production without a recorded simulation result, pass or fail.

4. Secrets management replacing static credentials. Every workflow run receives its own short-lived credential, issued at execution time and expired automatically when the run ends. No static API keys, passwords, or tokens in pipeline configuration files or environment variables. Stages 1 through 3 are well-organized and safer than scripts; without this stage they are not secure. The static API key embedded in a pipeline configuration file is the credential that an audit will eventually find. The full credential lifecycle model is in section 12.2.2. This stage is complete when no static credential exists in any pipeline configuration, environment variable, or configuration file that a pipeline runner reads.

5. An audit log linked to change records. Every change record carries a signed provenance trail: who authorized it, which validation gates it passed, what the network state was before and after. This is the stage that transforms the platform from a deployment tool into a governance artifact: the compliance question becomes answerable from platform records rather than from a weeks-long manual reconstruction. The Provenance Chain pattern in section 12.4.2 specifies the full structure. This stage is complete when any change made in the past 90 days can be queried by device, requester, and outcome without manual correlation across multiple systems.

The complete platform described in Chapters 10 through 12 extends well beyond stage 5: policy-as-code gates, tiered risk classification, progressive rollout with canary validation, multi-region federation. Stages 1 through 4 are achievable in weeks, not months. A team operating at stage 2 is in a materially better position than a team operating on scripts, even if stage 5 is a year away. The build sequence matters because each stage is useful on its own terms, not only as a step toward the next one. The three MVP sequences across Chapters 10, 11, and 12 are unified into a single integrated build order in the Chapter 12 Summary, which deduplicates the overlapping stages and shows the full sequence in dependency order.

Summary#

This chapter traced the path from NAF building blocks that work correctly in isolation to a platform that wires them into a coherent whole: the Source of Truth becomes a pipeline gate, the Executor becomes a deployment target, the simulation environment from Chapter 9 becomes a mandatory validation stage, and every change follows the same paved path from intake to production device. The eleven-day firewall change that opened the chapter was not a failure of tools: every individual component existed. The failure was structural: the components were not wired together into a system that a network engineer could use without knowing how each piece worked internally.

That product discipline extends to every building block from Part 2. The Orchestrator’s DAG patterns from Chapter 7 govern pipeline dependency ordering. The Observability stack gains a new consumer: the pipeline itself.

Two patterns govern that product discipline. The Platform as Product for Engineers pattern frames the platform team as a product team with network engineers as customers; cognitive load, self-service capability, and feedback speed are the SLAs that determine whether the platform succeeds or fails as a product. The Platform Contract pattern frames the platform’s API and behavior as a commitment to those customers: versioned, with a documented deprecation policy, managed under the same product lifecycle that Chapter 14 applies to all automation services.

The firewall change has now passed through the full pipeline: normalized intake, SoT integrity validation, schema and simulation gates, canary scope definition, release approval. The artifact is in the store, labeled production-approved. The platform has done its job under controlled conditions.

What Chapter 11 addresses is what happens when conditions are not controlled: when forty changes arrive simultaneously, when a worker fails mid-deployment, when the SoT query that should take two seconds takes forty because a background export is competing for database connections. Building the platform is Chapter 10’s work. Making it survive is Chapter 11’s.

References#

• Team Topologies, Matthew Skelton and Manuel Pais (IT Revolution, 2019). The foundational text on organizing teams around software systems; the platform team and stream-aligned team topology maps directly to the platform-as-product model in this chapter.
• Accelerate: The Science of Lean Software and DevOps, Nicole Forsgren, Jez Humble, and Gene Kim (IT Revolution, 2018). The empirical research behind the claim that deployment frequency, lead time, and feedback speed are the leading indicators of organizational performance; the developer experience metrics in section 10.2 are grounded in this work.
• Continuous Delivery, Jez Humble and David Farley (Addison-Wesley, 2010). The original systematic treatment of CI/CD pipeline architecture; the CI-vs-CD distinction and the artifact promotion model in section 10.4 trace directly to this book.
• Platform Engineering on Kubernetes, Mauricio Salatino (Manning, 2023). A practical treatment of internal developer platforms applied to cloud-native infrastructure; useful for teams adapting the IDP model to network automation contexts.
• Site Reliability Engineering, Niall Richard Murphy, Betsy Beyer, Chris Jones, and Jennifer Petoff (O’Reilly, 2016). The foundational text on operating shared infrastructure as a reliable service; its error budget model, toil elimination framework, and SLO discipline map directly onto the platform health metrics and operational model described in section 10.7.

💬 Found something to improve? Send feedback for this chapter