Sometimes I need to quickly remove one of our data centre switches from production. Typically this is done in preparation of scheduled maintenance, but it could also be necessary if I suspect that it is misbehaving in some way. Recently I stumbled across an undocumented feature in Cumulus Linux that significantly simplified this procedure.

The key is the file /cumulus/switchd/ctrl/shutdown_linkdown. This file does normally not exist, but if it is created with the contents 1, it changes the behaviour of the ASIC management daemon switchd so that instantly sets all swp<X> ports to link-down status when it is stopped. Thus the rapid removal procedure becomes:

root@cumulus:mgmt-vrf:~# echo 1 > /cumulus/switchd/ctrl/shutdown_linkdown
root@cumulus:mgmt-vrf:~# systemctl stop switchd.service

This immediately signals to all the connected devices that the switch is no longer there, allowing them to quickly re-route all their production traffic to alternate paths. However, the eth0 management interface remains up, leaving the network operator in full control.

The default behaviour is that switchd simply abandons control of the forwarding ASIC when it is stopped, essentially leaving the switch rudderless. In this state, no MAC learning can happen, new routes will not be installed, et cetera. That might be fine during a quick switchd restart, but certainly not for a more lengthy removal from the production network.

The benefit of doing it this way, as opposed to simply powering off the switch (which will also set all ports link-down), is that re-introducing it to production becomes quick and easy: simply start switchd again, and moments later everything is back up and running the way it was to begin with.

We do want keep the possibility of a quick re-introduction to production in this way, in case the removal unexpectedly caused an outage. Waiting for a powered-off switch to boot back up would take considerably longer.

There is one important caveat: if you are using MLAG, this procedure will appear as a peer link failure to the remaining switch in the MLAG cluster. To avoid an outage, only perform the procedure on the secondary switch in an MLAG cluster. You can ensure it assumes the secondary role by increasing its priority to above that of its cluster peer, like so:

root@cumulus:mgmt-vrf:~# clagctl priority 65535
root@cumulus:mgmt-vrf:~# clagctl | head -n3
The peer is alive
     Our Priority, ID, and Role: 65535 1c:ea:0b:fa:fb:fc secondary
    Peer Priority, ID, and Role: 20 1c:ea:0b:fa:fb:fd primary

At this point, it should be safe to proceed with the rapid removal procedure.


Update

  • 2024-01-29 Updated dead links.

Tore Anderson

Senior Systems Consultant at Redpill Linpro

Tore works with infrastructure at Redpill Linpro. Joining us more than a decade ago as a trainee, Tore is now responsible for our network architecture and operations.

The irony of insecure security software

It can probably be understood from my previous blog post that if it was up to me, I’d avoid products like CrowdStrike - but every now and then I still have to install something like that. It’s not the idea of “security software” per se that I’m against, it’s the actual implementation of many of those products. This post lists up some properties that should be fulfilled for me to happy to install such a product.

Free and ... [continue reading]

Thoughts on the CrowdStrike Outage

Published on July 23, 2024

Alarms made right

Published on June 27, 2024