So I had this el9 machine that had been sitting quietly waiting for updates for a bit too long. Blinded by courage, I went in and did a
dnf update, rebooted, and lo and behold, it worked without a hitch
afterwards. Except just one small one: It did not automatically
boot the newest kernel. I tried the standard things. Look at
/etc/default/grub, look at the output of various grubby commands. To
no avail. It simply would not pick up the latest kernel.
It is interesting that I’ve seen this on some systems, but not on all. I suspect that there is a change in grubby or kernel-install or some other script during the lifetime of el9 that made (or fixed) this bug, and it depends on if you ran a kernel-install with the bug. Or strong fluctuations in the karma field. I really don’t know.
I started to dig a bit deeper.
In the old days, we had lilo. In the not so old days, we had grub
version 1. It had simple configuration file menu.lst. With grub
version 2 it became more complicated. The configuration is a whole
tree of files, and it is generally adviced against touching that by
hand. Rather use the macros in /etc/default/grub, and let the system
regenerate the files from there. Except it is even more complex than
that.
el9 uses The UAPI Boot Loader Specification,
or just bls for short. It is a freedesktop/systemd thingie (yeah,
systemd is everywhere these days). It uses fragments of the boot
configurations, generated and stored in /boot/loader/entries,
typically one for each kernel. Red Hat and derivates names these by
taking the local machine’s id hash from /etc/machine-id, and the
kernel version.
When the package manger installs a new kernel, the grub tools build a full configuration based on the bls fragments. Those are sorted on the filename, and not the title. Which as said, is just an id hash and a version. Now, if your machine is installed from a clone, like a VM image in a cloud environment, you may have an id left behind from the original VM image. This may mean that the new kernel may not become the index-0 kernel that typically is the default boot kernel.
So the kernel install tool will sort the bls fragments, and if the old entry from the original image has a name that comes alphabetically before your machine’s id, the old one will always end up at the top.
Bonus problem: If you remove an old kernel from an original image,
typically using rpm -e or dnf remove, the entry fragments from the
original image are not necessary cleaned up properly, and you may
end up with a broken system with boot entries pointing to a
non-installed kernel.
First check if you have old kernels that don’t match your machine id
ls -1 /boot/loader/entries/
All entries should start with the same id, as they are specific for this machine. If you have entries with a different id, rename those so they match. And remove double entries. You should not have more than one rescue conf for example.
cd /boot/loader/entries/
ls -1 | grep -v $(cat /etc/machine-id)
0510df42e58a415fa231e736f98e76b3-0-rescue.conf
0510df42e58a415fa231e736f98e76b3-5.14.0-503.16.1.el9_5.x86_64.conf
cat /etc/machine-id
6178b622e59743e5ba7cf85105c56b4e
test -f $(cat /etc/machine-id)-0-rescue.conf && rm 0510df42e58a415fa231e736f98e76b3-0-rescue.conf
mv 0510df42e58a415fa231e736f98e76b3-5.14.0-503.16.1.el9_5.x86_64.conf $(cat /etc/machine-id)-5.14.0-503.16.1.el9_5.x86_64.conf
You should also check if you have dead entries in the boot loader:
awk '/^linux/ { print $2}' /boot/loader/entries/* |\
while read i; do ls "$i" > /dev/null; done
If you get missing files, like
ls: cannot access '/boot/vmlinuz-5.14.0-503.16.1.el9_5.x86_64': No such file or directory
then remove the entry fragment file(s) pointing to that kernel.
grep -l 5.14.0-503.16.1.el9_5.x86_64 /boot/loader/entries/*
/boot/loader/entries/0510df42e58a415fa231e736f98e76b3-5.14.0-503.16.1.el9_5.x86_64.conf
rm /boot/loader/entries/0510df42e58a415fa231e736f98e76b3-5.14.0-503.16.1.el9_5.x86_64.conf
Now update grub and add with some safety measures. Edit /etc/default/grub, and add/change
these options:
GRUB_DEFAULT=saved
GRUB_ENABLE_BLSCFG=true
GRUB_TIMEOUT=5
GRUB_TIMEOUT_STYLE=menu
and then apply the changes
grub2-mkconfig -o /boot/grub2/grub.cfg
If you end up with grub trying to boot an entry that does not point to an installed kernel, that will give you 5 seconds on the console to manually select another entry from the grub menu.
Check that your latest kernel is the one that is set to boot
rpm -q kernel-core| sort -V | tail -1
kernel-core-5.14.0-687.61.1.el9_8.x86_64
grubby --set-default /boot/vmlinuz-5.14.0-687.61.1.el9_8.x86_64
Run grubby again to check the result
grubby --info=ALL
Reboot and check that the system comes back up with the correct kernel
To find out about this, I had to dig down in what grub actually
does. And it is a bit interesting. Grub’s component
/usr/libexec/grubby/grubby-bls uses rpm-sort -c rpmnvrcmp to sort
the entries in /boot/loader/entries. It may have been wiser to use
rpm-sort -c vers-nvr-cmp perhaps.
Conclusion: Old boot loader entries generated before cloning an OS image, or by hand, may mess up your grub config, making it impossible to set grub to boot the newest kernel This post tries to throw a little light on the process, and how to tidy this up.
Redpill Linpro is the Open Source leader in the Nordics, helping customers with the digital transformation since back in the nineties.
Changelog
- 2026-06-26: Posted
- 2026-10-30: Review and tell the real story
