Part 3: Designing for Scale and Reliability on Designing Network Automation at Scale

10 - Platform Engineering and CI/CD

Thu, 16 Jul 2026 00:00:00 +0000

10. Platform Engineering and CI/CD#

A network engineer at a mid-size financial services firm needed to modify a firewall rule to allow a new internal service to reach an authentication endpoint. The change itself took about ten minutes to understand and design. What followed was different. She needed to open a ticket in the ITSM system and fill out a form that asked questions her team had never agreed on. She needed to submit a change request to a separate portal, which triggered a workflow that sent an automated notification to a stale distribution list. She needed to manually run a validation script on her laptop, copy the output into a comment on the ticket, and wait for a reviewer to read it.

11 - Scaling and Reliability

Thu, 16 Jul 2026 00:00:00 +0000

11. Scaling and Reliability#

The firmware upgrade had been planned for three weeks.

Eight hundred campus switches, three vendors, a maintenance window from midnight to 6 AM. The automation platform was well-tested on the campus fleet. The team had run smaller upgrades through it before. Forty-eight workers ran in parallel, chosen to balance speed against device load. At midnight, the job started.

By 2:14 AM, 47 of the 48 workers had completed. Seven hundred and eighty-seven switches had been upgraded successfully. The forty-eighth worker was still running. The orchestrator showed it as active; no error had been raised. One of the devices it was targeting was not responding to any management plane query, but it had not explicitly rejected the connection either. The worker was waiting for a response that was not going to arrive.

12 - Security, Governance and Compliance

Thu, 16 Jul 2026 00:00:00 +0000

12. Security, Governance and Compliance#

The audit finding arrived on a Tuesday morning.

A PCI compliance review had examined six months of firewall change history across the organization’s network. The auditors had a straightforward request. For each of the firewall changes: who authorized it, what change ticket it was associated with, evidence that pre-deployment validation occurred, and the network state before and after.

The automation platform had made 847 changes to the firewall configuration during those six months. It had made them correctly. The validation logic had been tested and applied. The rollback procedures had worked in the two cases where they were needed. No production incident had been caused by automation during that period. By every operational metric, the platform had performed well.