As I was running some manual package upgrades on my Ubuntu 24.04 server, I noticed that a kernel upgrade led to a long series of kernels being listed. This was strange, as there should have been only two or three. Closer inspection revealed that the update-grub command tries to find all kernels and also looks for kernels on ZFS datasets. Given that I extensively use ZFS, I was aware of that, and normally this is the behaviour I want.
However, in this case, my home server doesn’t have root on ZFS, but on a regular MD RAID mirror with LVM. Looking at the kernels listed during the upgrade, I noticed that update-grub was digging through all ZFS datasets and their snapshots on the server, including the backups sent to my home server from various other machines, some of which do have root on ZFS. All in all this led to a huge list of kernels and a kernel upgrade process that never seemed to end.
A quick web search pointed me to this AskUbuntu.com answer, which showed that the ZFS parsing code can be found in /etc/grub.d/10_linux_zfs. In there, I found the following code block (note, for improved readability I split the long command above the done over multiple lines and added corresponding \):
# List all the dataset with a root mountpointget_root_datasets(){localpools="$(zpool list | awk '{if (NR>1) print $1}')"for p in ${pools}; dolocalrel_pool_root=$(zpool get -H altroot ${p} | awk '{print $3}')if["${rel_pool_root}" = "-"]; thenrel_pool_root="/"fi
zfs list -H -o name,canmount,mountpoint -t filesystem | \
grep -E '^'"${p}"'(\s|/[[:print:]]*\s)(on|noauto)\s'"${rel_pool_root}"'$' | \
awk '{print $1}'done}
This function first lists all the ZFS pools on the system and then (with the last, long command above the done), for each of those, returns the filesystem datasets that can be mounted.
The solution was simple: because all backups from external systems end up as datasets below the remote_backups dataset on my storage pool, simply changed the last awk statement so that it only prints datasets that don’t match remote_backups by adding $1 !~ "remote_backups":
This way, the code would still work if I ever move this system to have root on ZFS, but now update-grub skips all backups and finds only the kernels relevant for my home server :-).
The spinning disk pool on my home server uses a mirrored special device (for storing metadata and small blocks, see also this blog post at Klara Systems) based on two NVMe SSDs. Because my home server only has two M.2 slots and I wanted to have a pure SSD ZFS pool as well, I partitioned the SSDs. Each SSD has a partition for the SSD pool and one for the special device of the storage pool (which uses a mirror of spinning disks).
Note: This isn’t really a recommended production setup as you are basically hurting performance of both the special device and the SSD pool. But for my home server this works fine. For example, I use the special device’s small blocks functionality to store previews of the photo’s I store on my Nextcloud server. This makes scrolling through the Memories app’s timeline a breeze, even though the full-size photo’s are stored on the spinning disks.
Today, I noticed that the special device had filled up, and, given that there was still some unpartitioned space on the SSDs, I wondered if I could just expand the partition (using parted) used by the special device and then have the ZFS pool recognise the extra space. In the past I have expanded partition-based ZFS pools before, e.g. on after upgrading the SSD on my laptop, but I hadn’t tried this with a special device before.
After some experimentation, I can tell you: this works.
Here is how I tested this on a throw-away file-backed zpool. First create four test files: two for the actual mirror pool and two that I’ll add as a special device.
for i in{0..3} ; do truncate -s 1G file$i.raw ; done
ls -lh
total 4,0K
-rw-rw-r-- 1 lennart lennart 1,0G mrt 11 12:46 file0.raw
-rw-rw-r-- 1 lennart lennart 1,0G mrt 11 12:46 file1.raw
-rw-rw-r-- 1 lennart lennart 1,0G mrt 11 12:46 file2.raw
-rw-rw-r-- 1 lennart lennart 1,0G mrt 11 12:46 file3.raw
Create a regular mirror pool:
zpool create testpool mirror $(pwd)/file0.raw $(pwd)/file1.raw
zpool list -v testpool
I wasn’t sure whether I could just truncate the backing files for the special device to a larger size while they were part of the pool, so I detached them one by one and created new ones of 2GB, and then reattached them: