EL9's grubby sorting

So I had this el9 machine that had been sitting quietly waiting for updates for a bit too long. Blinded by courage, I went in and did a

dnf update, rebooted, and lo and behold, it worked without a hitch afterwards. Except just one small one: It did not automatically boot the newest kernel. I tried the standard things. Look at /etc/default/grub, look at the output of various grubby commands. To no avail. It simply would not pick up the latest kernel.

It is interesting that I’ve seen this on some systems, but not on all. I suspect that there is a change in grubby or kernel-install or some other script during the lifetime of el9 that made (or fixed) this bug, and it depends on if you ran a kernel-install with the bug. Or strong fluctuations in the karma field. I really don’t know.

I started to dig a bit deeper.

In the old days, we had lilo. In the not so old days, we had grub version 1. It had simple configuration file menu.lst. With grub version 2 it became more complicated. The configuration is a whole tree of files, and it is generally adviced against touching that by hand. Rather use the macros in /etc/default/grub, and let the system regenerate the files from there. Except it is even more complex than that.

el9 uses The UAPI Boot Loader Specification, or just bls for short. It is a freedesktop/systemd thingie (yeah, systemd is everywhere these days). It uses fragments of the boot configurations, generated and stored in /boot/loader/entries,

typically one for each kernel. Red Hat and derivates names these by taking the local machine’s id hash from /etc/machine-id, and the kernel version.

When the package manger installs a new kernel, the grub tools build a full configuration based on the bls fragments. Those are sorted on the filename, and not the title. Which as said, is just an id hash and a version. Now, if your machine is installed from a clone, like a VM image in a cloud environment, you may have an id left behind from the original VM image. This may mean that the new kernel may not become the index-0 kernel that typically is the default boot kernel.

So the kernel install tool will sort the bls fragments, and if the old entry from the original image has a name that comes alphabetically before your machine’s id, the old one will always end up at the top.

Bonus problem: If you remove an old kernel from an original image, typically using rpm -e or dnf remove, the entry fragments from the original image are not necessary cleaned up properly, and you may end up with a broken system with boot entries pointing to a non-installed kernel.

First check if you have old kernels that don’t match your machine id

ls -1 /boot/loader/entries/

All entries should start with the same id, as they are specific for this machine. If you have entries with a different id, rename those so they match. And remove double entries. You should not have more than one rescue conf for example.

cd /boot/loader/entries/
ls -1 | grep -v $(cat /etc/machine-id)
0510df42e58a415fa231e736f98e76b3-0-rescue.conf
0510df42e58a415fa231e736f98e76b3-5.14.0-503.16.1.el9_5.x86_64.conf

cat /etc/machine-id 
6178b622e59743e5ba7cf85105c56b4e 

test -f $(cat /etc/machine-id)-0-rescue.conf && rm 0510df42e58a415fa231e736f98e76b3-0-rescue.conf 
mv 0510df42e58a415fa231e736f98e76b3-5.14.0-503.16.1.el9_5.x86_64.conf $(cat /etc/machine-id)-5.14.0-503.16.1.el9_5.x86_64.conf

You should also check if you have dead entries in the boot loader:

awk '/^linux/  { print $2}' /boot/loader/entries/* |\
  while read i; do ls "$i" > /dev/null; done

If you get missing files, like

ls: cannot access '/boot/vmlinuz-5.14.0-503.16.1.el9_5.x86_64': No such file or directory

then remove the entry fragment file(s) pointing to that kernel.

grep -l 5.14.0-503.16.1.el9_5.x86_64 /boot/loader/entries/*
/boot/loader/entries/0510df42e58a415fa231e736f98e76b3-5.14.0-503.16.1.el9_5.x86_64.conf

rm /boot/loader/entries/0510df42e58a415fa231e736f98e76b3-5.14.0-503.16.1.el9_5.x86_64.conf

Now update grub and add with some safety measures. Edit /etc/default/grub, and add/change these options:

GRUB_DEFAULT=saved
GRUB_ENABLE_BLSCFG=true
GRUB_TIMEOUT=5
GRUB_TIMEOUT_STYLE=menu

and then apply the changes

grub2-mkconfig -o /boot/grub2/grub.cfg

If you end up with grub trying to boot an entry that does not point to an installed kernel, that will give you 5 seconds on the console to manually select another entry from the grub menu.

Check that your latest kernel is the one that is set to boot

rpm -q kernel-core| sort -V | tail -1
kernel-core-5.14.0-687.61.1.el9_8.x86_64

grubby --set-default /boot/vmlinuz-5.14.0-687.61.1.el9_8.x86_64

Run grubby again to check the result

grubby --info=ALL

Reboot and check that the system comes back up with the correct kernel

To find out about this, I had to dig down in what grub actually does. And it is a bit interesting. Grub’s component /usr/libexec/grubby/grubby-bls uses rpm-sort -c rpmnvrcmp to sort the entries in /boot/loader/entries. It may have been wiser to use rpm-sort -c vers-nvr-cmp perhaps.

Conclusion: Old boot loader entries generated before cloning an OS image, or by hand, may mess up your grub config, making it impossible to set grub to boot the newest kernel This post tries to throw a little light on the process, and how to tidy this up.

Redpill Linpro is the Open Source leader in the Nordics, helping customers with the digital transformation since back in the nineties.


Changelog

  • 2026-06-26: Posted
  • 2026-10-30: Review and tell the real story

Ingvar Hagelund

Team Lead, Application Management for Media at Redpill Linpro

Ingvar has been a system administrator at Redpill Linpro for more than 25 years. He is also a long time contributor to the Fedora and EPEL projects.

Putting Varnish CDN in front of a serverless container with OpenTofu

Serverless containers are a great way to run a small site: you push an image, you get an HTTPS URL, and you don’t pay for resources when nobody is accessing your application. What you don’t get is caching close to the client, or protection from the cold-start penalty every time a visitor arrives after the serverless container has spun down.

That is exactly what a CDN is for and an excellent choice for this is Varnish CDN which lets you ... [continue reading]

From a Luddite to a Vibe-Coder

Published on March 20, 2026