Part 2: Architectural Building Blocks on Designing Network Automation at Scale

04 - Source of Truth

Sun, 15 Feb 2026 00:00:00 +0000

4. The Source of Truth#

A new service needed to go live by end of week. The change was straightforward: a new VLAN, a new subnet, updated firewall rules, and a new BGP community on the edge routers. The network engineer knew exactly what to do. What followed was four days of coordination across five systems: the IPAM tool to reserve the subnet, the CMDB to register the service, the firewall management platform to push the new policy, the router configuration system for the BGP community, and the monitoring platform to add the new thresholds. Each system had its own interface, its own data model, its own approval workflow. And because there was no shared reference, the engineer had to carry the context manually: copying values from one system to the next, hoping nothing drifted between steps.

05 - Execution

Sat, 01 Nov 2025 00:00:00 +0000

5. Execution#

The playbook had been running perfectly for months: ten access switches in the lab, two minutes end to end, clean results every time. When the team decided to roll it out to the full inventory of 800 switches, nobody expected trouble. The first 600 devices updated without issue. Then the job slowed. Then it stalled. The RADIUS server, suddenly receiving 150 simultaneous SSH authentication requests, started rejecting connections. Ansible hung on 150 devices mid-execution. An engineer killed the job.

06 - Observability

Mon, 12 Jan 2026 00:00:00 +0000

6. Observability#

A service provider’s core router had a fan module fail silently overnight. There were no alerts. The thermal management subsystem on the line cards detected rising temperatures and began throttling the forwarding ASICs to protect the hardware. Throughput on a production transit link dropped by 40%. The monitoring system saw nothing: the thermal sensors it polled were on the chassis supervisor, not on the individual line cards. The SNMP polls ran every five minutes and reported the device as healthy. The interface counters showed reduced throughput, but no threshold was configured for that pattern because the link had always been well below capacity.

07 - Orchestration

Fri, 20 Mar 2026 00:00:00 +0000

7. Orchestration#

The network team had done everything right. They had a solid Source of Truth, well-tested playbooks for every operation, and a clear runbook to execute them. On paper, deploying a new VLAN service was fully automated. In practice, it took half a day and one specific engineer.

That engineer knew the sequence. First, validate the SoT data was complete. Then run the pre-check playbook. Then review the output, look for failed devices, and decide whether to continue. Then trigger the deployment playbook. Then wait. Then run the validation playbook. Then update the ServiceNow ticket manually. If any device failed partway through, roll it back before the others noticed. She had it all in a runbook, step by step, in a shared document that nobody else had fully internalized.

08 - Presentation

Sat, 28 Mar 2026 00:00:00 +0000

8. The Presentation Layer#

The VLAN automation had been running for six weeks. The network team was proud of it. Every morning, three or four new service requests arrived from application teams, and the Orchestrator handled them without anyone on the network team touching a keyboard. The deployments worked. The switches were configured. The network was healthy.

The escalation arrived on a Thursday. The application team lead was asking why VLAN requests were taking three to five business days when the portal said “submitted.” The network team checked their queue: zero pending requests, all deployments successful. The automation had processed every request within twenty minutes of receipt. But the ServiceNow tickets still showed “In Progress,” because nobody had written the integration that would update them.

09 - The Network

Sun, 29 Mar 2026 00:00:00 +0000

9. The Network#

The VLAN automation had been running for three weeks in the lab. Three switches, one of each vendor, every workflow passing. The team felt confident. On the first production run, 23 of the 800 campus switches failed. All HPE. All running a firmware version that nobody had documented.

The playbook was checking the error response from each device after pushing the VLAN configuration. On modern HPE firmware, an already-existing VLAN returns the error code duplicate-vlan. On this older firmware version, the same condition returned vlan-exists. The playbook had been written to treat duplicate-vlan as an idempotency signal, meaning “this already exists, that is fine.” It had not been written to handle vlan-exists, so it treated that response as a failure. A third of the HPE fleet reported failure. The rollback ran cleanly. The application team’s ticket stayed open for another three hours while the network team manually audited which switches had actually been configured and which had not.