This post appeared originally in our sysadvent series and has been moved here following the discontinuation of the sysadvent microsite
When building a Ceph-cluster, it was important for us to plan ahead. Not only does one usually start out with a minimum of ~5 servers, but one should also expect some growth in the cluster. Running the cluster also means patching the operating system and Ceph itself, and with Ceph being a crucial infrastructure component it’s also desirable to have a proper rollback procedure.
Using CI to maintain image
We’ve grown really fond of ram-disk nodes. Using a Jenkins instance, we run a build service that will boot up a VM using an updated version of CentOS 7, connect to Puppet and provision in all the basic components that we want the server to have. This image is converted into a root file-system, and uploaded to our redundant service hosts for availability.
Covering this build process could be an entire Sysadvent calendar on its own, but this is the gist of it.
Booting it up
There’s a lovely piece of software that’s incredibly powerful for booting things on the network - and it’s called iPXE. We chain-load iPXE in the iPXE/UEFI boot process, and it lets us run a small script before we decide what image to boot. In this script, we pass information about which Puppet-environment the server belongs to, how to configure bonding, and other pieces of information that is unique to that server to the kernel command line.
We then boot up the kernel/rootfs of the image, parse /proc/cmdline
, configure networking (change a dynamic lease on a single interface to a bonded interface with a static IP address), download Puppet certificates, and then run Puppet.
Puppet puts the last pieces in place
The image will have all the basic software in place, but it may or may not have the necessary keys in place to be allowed to talk to the cluster monitors. Puppet will ensure that ceph.conf / keys are placed where they should be in a secure manner. Puppet will also take care of day to day configuration changes - such as adding new monitoring probes and tools, as we build upon and improve them.
We continuously do changes on the servers based on our best practices through Puppet, and we try to not do updates to the image unless it’s an issue of security or availability.
What about OSD state?
And this is the point I’m trying to drive home with this post: I guess you’re wondering how we know which drive is which OSD in our cluster. There’s a really simple trick that we figured out:
- For each file-system (we use XFS in production), query OSD ID by using the UUID of said file-system
- If the UUID is unknown a new OSD ID is created - but for existing OSDs, the correct ID is returned.
From that point on, we have all information we need to mount the OSDs and start the systemd-unit for it.
Something along these lines will get us up and running:
#!/bin/bash
for a in /dev/disk/by-uuid/*; do
FS=$(blkid -o value -s TYPE $a)
if [ "$FS" == "xfs" ]; then
# Get OSD id from monitors, create if missing
UUID=$(basename $a)
OSDID=$(ceph osd create $UUID)
if [ ! -e "/var/lib/ceph/osd/ceph-$OSDID" ]; then
mkdir /var/lib/ceph/osd/ceph-$OSDID
fi
if mountpoint -q /var/lib/ceph/osd/ceph-$OSDID; then
echo "Already mounted, skipping. (OSD: $OSDID, UUID: $UUID)"
else
if grep -q $UUID /etc/fstab; then
echo "Mountpoint already in fstab, not adding"
else
echo "UUID=$UUID /var/lib/ceph/osd/ceph-$OSDID xfs defaults 0 0" >> /etc/fstab
fi
mount $a /var/lib/ceph/osd/ceph-$OSDID
fi
if [ ! -e "/var/lib/ceph/osd/ceph-$OSDID/keyring" ]; then
ceph-osd -i $OSDID --mkfs --mkkey --osd-uuid $UUID
ceph auth add osd.$OSDID osd 'allow *' mon 'allow profile osd' -i /var/lib/ceph/osd/ceph-$OSDID/keyring
ceph osd crush add osd.$OSDID 1.0 host=$(facter hostname)
else
echo "Already initialized, skipping. (OSD: $OSDID, UUID: $UUID)"
fi
RUNNING=$(pgrep -f "^ceph-osd -i $OSDID\$" -c)
if [ "$RUNNING" -eq 1 ]; then
echo "Already running, not starting."
else
systemctl enable ceph-osd@${OSDID}.service
systemctl start ceph-osd@${OSDID}.service
fi
fi
done
This code has the unintended side-effect that it can also be used to easily add new OSDs to the cluster; Just insert a new disk, mkfs.xfs
, start_osds
, and it will be added to the cluster. As one would usually do, just be careful about the performance impact of the backfill-operations that adding new OSDs may have.
For journals, we use raw partitions. To ensure that journals will work across boots - even if hardware would come up in different order than before (making sda to sdb, or the other way around) - we always use /dev/disk/by-wwn/
to look up the correct partition.
What did we really achieve?
This simplifies scaling up a lot.
- When hardware has arrived, and the physical part of the job is taken care of - initializing a new node is done by booting it, formatting the drives, and starting the OSDs.
- Adding new drives is solved by formatting the drives and starting the OSDs.
It also ensures that as long as all nodes have booted the same version of the image - we can also expect them to contain the same software versions and identical configuration.
Upgrading the image is just about doing ceph set osd noout
, and perform a rolling reboot of the cluster. We don’t like what we just patched into? Perform a rollback! We keep the old image, and we know exactly what to expect from it!
The irony of insecure security software
It can probably be understood from my previous blog post that if it was up to me, I’d avoid products like CrowdStrike - but every now and then I still have to install something like that. It’s not the idea of “security software” per se that I’m against, it’s the actual implementation of many of those products. This post lists up some properties that should be fulfilled for me to happy to install such a product.