The spinning disk pool on my home server uses a mirrored special device (for storing metadata and small blocks, see also this blog post at Klara Systems) based on two NVMe SSDs. Because my home server only has two M.2 slots and I wanted to have a pure SSD ZFS pool as well, I partitioned the SSDs. Each SSD has a partition for the SSD pool and one for the special device of the storage pool (which uses a mirror of spinning disks).
Note: This isn’t really a recommended production setup as you are basically hurting performance of both the special device and the SSD pool. But for my home server this works fine. For example, I use the special device’s small blocks functionality to store previews of the photo’s I store on my Nextcloud server. This makes scrolling through the Memories app’s timeline a breeze, even though the full-size photo’s are stored on the spinning disks.
Today, I noticed that the special device had filled up, and, given that there was still some unpartitioned space on the SSDs, I wondered if I could just expand the partition (using parted) used by the special device and then have the ZFS pool recognise the extra space. In the past I have expanded partition-based ZFS pools before, e.g. on after upgrading the SSD on my laptop, but I hadn’t tried this with a special device before.
After some experimentation, I can tell you: this works.
Here is how I tested this on a throw-away file-backed zpool. First create four test files: two for the actual mirror pool and two that I’ll add as a special device.
for i in {0..3} ; do truncate -s 1G file$i.raw ; done ls -lh
total 4,0K -rw-rw-r-- 1 lennart lennart 1,0G mrt 11 12:46 file0.raw -rw-rw-r-- 1 lennart lennart 1,0G mrt 11 12:46 file1.raw -rw-rw-r-- 1 lennart lennart 1,0G mrt 11 12:46 file2.raw -rw-rw-r-- 1 lennart lennart 1,0G mrt 11 12:46 file3.raw
Create a regular mirror pool:
zpool create testpool mirror $(pwd)/file0.raw $(pwd)/file1.raw zpool list -v testpool
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
testpool 960M 146K 960M - - 0% 0% 1.00x ONLINE -
mirror-0 960M 104K 960M - - 0% 0.01% - ONLINE
/tmp/tests/file0.raw 1G - - - - - - - ONLINE
/tmp/tests/file1.raw 1G - - - - - - - ONLINE
Add the special device to the zpool:
zpool add testpool special mirror $(pwd)/file2.raw $(pwd)/file3.raw zpool list -v testpool
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
testpool 1.88G 190K 1.87G - - 0% 0% 1.00x ONLINE -
mirror-0 960M 190K 960M - - 0% 0.01% - ONLINE
/tmp/tests/file0.raw 1G - - - - - - - ONLINE
/tmp/tests/file1.raw 1G - - - - - - - ONLINE
special - - - - - - - - -
mirror-1 960M 0 960M - - 0% 0.00% - ONLINE
/tmp/tests/file2.raw 1G - - - - - - - ONLINE
/tmp/tests/file3.raw 1G - - - - - - - ONLINE
I wasn’t sure whether I could just truncate the backing files for the special device to a larger size while they were part of the pool, so I detached them one by one and created new ones of 2GB, and then reattached them:
zpool detach testpool /tmp/tests/file2.raw zpool list -v testpool truncate -s 2G file2.raw zpool attach testpool $(pwd)/file3.raw /tmp/tests/file2.raw zpool list -v testpool
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
testpool 1.88G 194K 1.87G - - 0% 0% 1.00x ONLINE -
mirror-0 960M 104K 960M - - 0% 0.01% - ONLINE
/tmp/tests/file0.raw 1G - - - - - - - ONLINE
/tmp/tests/file1.raw 1G - - - - - - - ONLINE
special - - - - - - - - -
/tmp/tests/file3.raw 1G 90K 960M - - 0% 0.00% - ONLINE
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
testpool 1.88G 278K 1.87G - - 0% 0% 1.00x ONLINE -
mirror-0 960M 97.5K 960M - - 0% 0.00% - ONLINE
/tmp/tests/file0.raw 1G - - - - - - - ONLINE
/tmp/tests/file1.raw 1G - - - - - - - ONLINE
special - - - - - - - - -
mirror-1 960M 180K 960M - - 0% 0.01% - ONLINE
/tmp/tests/file3.raw 1G - - - - - - - ONLINE
/tmp/tests/file2.raw 2G - - - - - - - ONLINE
And for the second “disk”:
zpool detach testpool /tmp/tests/file3.raw zpool list -v testpool truncate -s 2G file3.raw zpool attach testpool $(pwd)/file2.raw /tmp/tests/file3.raw zpool list -v testpool
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
testpool 1.88G 218K 1.87G - - 0% 0% 1.00x ONLINE -
mirror-0 960M 66K 960M - - 0% 0.00% - ONLINE
/tmp/tests/file0.raw 1G - - - - - - - ONLINE
/tmp/tests/file1.raw 1G - - - - - - - ONLINE
special - - - - - - - - -
/tmp/tests/file2.raw 2G 152K 960M - 1G 0% 0.01% - ONLINE
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
testpool 1.88G 296K 1.87G - - 0% 0% 1.00x ONLINE -
mirror-0 960M 54K 960M - - 0% 0.00% - ONLINE
/tmp/tests/file0.raw 1G - - - - - - - ONLINE
/tmp/tests/file1.raw 1G - - - - - - - ONLINE
special - - - - - - - - -
mirror-1 960M 242K 960M - 1G 0% 0.02% - ONLINE
/tmp/tests/file2.raw 2G - - - 1G - - - ONLINE
/tmp/tests/file3.raw 2G - - - - - - - ONLINE
And now, here comes the magic. Time to expand the pool and see of the special device will grow to 2GB:
zpool online -e testpool $(pwd)/file2.raw $(pwd)/file3.raw zpool list -v testpool
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
testpool 2.88G 226K 2.87G - - 0% 0% 1.00x ONLINE -
mirror-0 960M 48K 960M - - 0% 0.00% - ONLINE
/tmp/tests/file0.raw 1G - - - - - - - ONLINE
/tmp/tests/file1.raw 1G - - - - - - - ONLINE
special - - - - - - - - -
mirror-1 1.94G 178K 1.94G - - 0% 0.00% - ONLINE
/tmp/tests/file2.raw 2G - - - - - - - ONLINE
/tmp/tests/file3.raw 2G - - - - - - - ONLINE
Yay! It worked! So, for my actual storage pool, I ended up doing the following:
- Given that the partitions used by the special device are located at the end of the SSD, it was easy to expand them using
parted resizepart. - Run
zpool online -e storage partname-1 partname-2(so I didn’t detach/attach here).
Don’t forget to clean up the testpool:
zpool destroy testpool rm -r /tmp/tests