Sometimes I need to quickly remove one of our data centre switches from production. Typically this is done in preparation of scheduled maintenance, but it could also be necessary if I suspect that it is misbehaving in some way. Recently I stumbled across an undocumented feature in Cumulus Linux that significantly simplified this procedure.

The key is the file /cumulus/switchd/ctrl/shutdown_linkdown. This file does normally not exist, but if it is created with the contents 1, it changes the behaviour of the ASIC management daemon switchd so that instantly sets all swp<X> ports to link-down status when it is stopped. Thus the rapid removal procedure becomes:

root@cumulus:mgmt-vrf:~# echo 1 > /cumulus/switchd/ctrl/shutdown_linkdown
root@cumulus:mgmt-vrf:~# systemctl stop switchd.service

This immediately signals to all the connected devices that the switch is no longer there, allowing them to quickly re-route all their production traffic to alternate paths. However, the eth0 management interface remains up, leaving the network operator in full control.

The default behaviour is that switchd simply abandons control of the forwarding ASIC when it is stopped, essentially leaving the switch rudderless. In this state, no MAC learning can happen, new routes will not be installed, et cetera. That might be fine during a quick switchd restart, but certainly not for a more lengthy removal from the production network.

The benefit of doing it this way, as opposed to simply powering off the switch (which will also set all ports link-down), is that re-introducing it to production becomes quick and easy: simply start switchd again, and moments later everything is back up and running the way it was to begin with.

We do want keep the possibility of a quick re-introduction to production in this way, in case the removal unexpectedly caused an outage. Waiting for a powered-off switch to boot back up would take considerably longer.

There is one important caveat: if you are using MLAG, this procedure will appear as a peer link failure to the remaining switch in the MLAG cluster. To avoid an outage, only perform the procedure on the secondary switch in an MLAG cluster. You can ensure it assumes the secondary role by increasing its priority to above that of its cluster peer, like so:

root@cumulus:mgmt-vrf:~# clagctl priority 65535
root@cumulus:mgmt-vrf:~# clagctl | head -n3
The peer is alive
     Our Priority, ID, and Role: 65535 1c:ea:0b:fa:fb:fc secondary
    Peer Priority, ID, and Role: 20 1c:ea:0b:fa:fb:fd primary

At this point, it should be safe to proceed with the rapid removal procedure.

Tore Anderson

Senior Systems Consultant at Redpill Linpro

Tore works with infrastructure at Redpill Linpro. Joining us more than a decade ago as a trainee, Tore is now responsible for our network architecture and operations. Tore is passionate about IPv6 and has held a series of talks about IPv6 transition strategies and implementations.

Ansible/AWX network performance investigation

When we introduced the network configuration using Ansible and AWX at a customer, we gradually extended the configuration scope. Over time, more and more configuration got added into the configuration pool and this lead to longer and longer run-times for the playbooks.

While the job-execution got really simple by using AWX instead of the plain CLI method for Ansible, the time to finish drew heavily on that benefit.

A complete job-run over the network infrastructure took at least ... [continue reading]

FOSDEM 2020

Published on March 19, 2020

Multi-factor Authentication (MFA) for AWS CLI

Published on February 18, 2020