“The core routing table vanished. A major regional zone went dark. ‘Where is the backup?’ I asked. ‘I… I didn’t save it before editing,’ he stammered.”

In September 2012 at Net4 India, a junior engineer logged into our primary Cisco 7609 Core Router to apply a routine BGP neighbor description change.

Instead of pasting a single line, his terminal emulator dumped a corrupted buffer into the active SSH session.

He wiped the BGP neighbor configuration on our primary transit router.

The Crisis

Core BGP sessions dropped immediately. Over 10,000 active subnets went offline, and our NOC phones lit up with customer calls.

The engineer had not executed show running-config prior to his edits, and no manual text backup existed.

[!IMPORTANT] Never allow manual configuration edits on core infrastructure without automated version control. A single copy-paste error can wipe out hours of network routing state.

The Hero: RANCID & CVS Version Control

A month prior, I had deployed RANCID (Really Awesome New Cisco ConfIg Differ) on an internal Linux management server.

RANCID logged into all network devices via Expect scripts every hour, fetched show running-config, and committed diffs to a CVS version control repository.

I SSH’d into the RANCID management host and queried the repository:

# # Query RANCID CVS repository for pre-outage diff
# rancid-run -r core-router-01
# cvs diff -r HEAD~1 -r HEAD router.db

===================================================================
RCS file: /var/rancid/CVS/configs/core-router-01,v
retrieving revision 1.142
retrieving revision 1.143
diff -u -r1.142 -r1.143
--- core-router-01      2012/09/15 10:00:00     1.142
+++ core-router-01      2012/09/15 11:00:00     1.143
- router bgp 17465
-  neighbor 121.240.1.5 remote-as 45184
-  neighbor 121.240.1.5 route-map PEERING_IN in

The Recovery

The CVS diff clearly displayed the exact deleted BGP neighbor statements.

I copied the missing configuration blocks directly from the RANCID diff output, pasted them into the Cisco 7609 console, and executed write memory.

Within 45 seconds, BGP sessions re-established, and traffic routing recovered completely.

The Verdict

Key Takeaway

Version Control Applies to Infrastructure, Not Just Application Code.

Long before GitOps or Terraform, tools like RANCID proved that infrastructure changes must be tracked in version control. Never manage a network without automated, hourly configuration diff backups.