Sometimes I need to quickly remove one of our data centre switches from production. Typically this is done in preparation of scheduled maintenance, but it could also be necessary if I suspect that it is misbehaving in some way. Recently I stumbled across an undocumented feature in Cumulus Linux that significantly simplified this procedure.
The key is the file /cumulus/switchd/ctrl/shutdown_linkdown. This file does
normally not exist, but if it is created with the contents 1, it changes the
behaviour of the ASIC management daemon switchd so that instantly sets all
swp<X> ports to link-down status when it is stopped. Thus the rapid removal
procedure becomes:
root@cumulus:mgmt-vrf:~# echo 1 > /cumulus/switchd/ctrl/shutdown_linkdown
root@cumulus:mgmt-vrf:~# systemctl stop switchd.service
This immediately signals to all the connected devices that the switch is no
longer there, allowing them to quickly re-route all their production traffic to
alternate paths. However, the eth0 management interface remains up, leaving
the network operator in full control.
The default behaviour is that switchd simply abandons control of the
forwarding ASIC when it is stopped, essentially leaving the switch rudderless.
In this state, no MAC learning can happen, new routes will not be installed, et
cetera. That might be fine during a quick switchd restart, but certainly not
for a more lengthy removal from the production network.
The benefit of doing it this way, as opposed to simply powering off the switch
(which will also set all ports link-down), is that re-introducing it to
production becomes quick and easy: simply start switchd again, and moments
later everything is back up and running the way it was to begin with.
We do want keep the possibility of a quick re-introduction to production in this way, in case the removal unexpectedly caused an outage. Waiting for a powered-off switch to boot back up would take considerably longer.
There is one important caveat: if you are using MLAG, this procedure will appear as a peer link failure to the remaining switch in the MLAG cluster. To avoid an outage, only perform the procedure on the secondary switch in an MLAG cluster. You can ensure it assumes the secondary role by increasing its priority to above that of its cluster peer, like so:
root@cumulus:mgmt-vrf:~# clagctl priority 65535
root@cumulus:mgmt-vrf:~# clagctl | head -n3
The peer is alive
Our Priority, ID, and Role: 65535 1c:ea:0b:fa:fb:fc secondary
Peer Priority, ID, and Role: 20 1c:ea:0b:fa:fb:fd primary
At this point, it should be safe to proceed with the rapid removal procedure.
Update
- 2024-01-29 Updated dead links.
