Sometimes I need to quickly remove one of our data centre switches from production. Typically this is done in preparation of scheduled maintenance, but it could also be necessary if I suspect that it is misbehaving in some way. Recently I stumbled across an undocumented feature in Cumulus Linux that significantly simplified this procedure.

The key is the file /cumulus/switchd/ctrl/shutdown_linkdown. This file does normally not exist, but if it is created with the contents 1, it changes the behaviour of the ASIC management daemon switchd so that instantly sets all swp<X> ports to link-down status when it is stopped. Thus the rapid removal procedure becomes:

root@cumulus:mgmt-vrf:~# echo 1 > /cumulus/switchd/ctrl/shutdown_linkdown
root@cumulus:mgmt-vrf:~# systemctl stop switchd.service

This immediately signals to all the connected devices that the switch is no longer there, allowing them to quickly re-route all their production traffic to alternate paths. However, the eth0 management interface remains up, leaving the network operator in full control.

The default behaviour is that switchd simply abandons control of the forwarding ASIC when it is stopped, essentially leaving the switch rudderless. In this state, no MAC learning can happen, new routes will not be installed, et cetera. That might be fine during a quick switchd restart, but certainly not for a more lengthy removal from the production network.

The benefit of doing it this way, as opposed to simply powering off the switch (which will also set all ports link-down), is that re-introducing it to production becomes quick and easy: simply start switchd again, and moments later everything is back up and running the way it was to begin with.

We do want keep the possibility of a quick re-introduction to production in this way, in case the removal unexpectedly caused an outage. Waiting for a powered-off switch to boot back up would take considerably longer.

There is one important caveat: if you are using MLAG, this procedure will appear as a peer link failure to the remaining switch in the MLAG cluster. To avoid an outage, only perform the procedure on the secondary switch in an MLAG cluster. You can ensure it assumes the secondary role by increasing its priority to above that of its cluster peer, like so:

root@cumulus:mgmt-vrf:~# clagctl priority 65535
root@cumulus:mgmt-vrf:~# clagctl | head -n3
The peer is alive
     Our Priority, ID, and Role: 65535 1c:ea:0b:fa:fb:fc secondary
    Peer Priority, ID, and Role: 20 1c:ea:0b:fa:fb:fd primary

At this point, it should be safe to proceed with the rapid removal procedure.

Tore Anderson

Senior Systems Consultant at Redpill Linpro

Tore works with infrastructure at Redpill Linpro. Joining us more than a decade ago as a trainee, Tore is now responsible for our network architecture and operations. Tore is passionate about IPv6 and has held a series of talks about IPv6 transition strategies and implementations.

A look at our new routers

This year we intend to upgrade all the routers in our network backbone to a brand new platform based on open networking devices from Edge-Core running Cumulus Linux. In this post - replete with pictures - we will take a close look at the new routers and the topology of our new network backbone.

Why upgrade?

Our network backbone is today based on the Juniper MX 240 routing platform. Each of them occupy 5 ... [continue reading]

Running PostgreSQL in Google Kubernetes Engine

Published on September 28, 2019

Evaluating Local DNSSEC Validators

Published on August 27, 2019