Lennart's weblog

Notes about open source software, computers, other stuff.

Tag: Linux (Page 1 of 7)

Expanding a partition-backed ZFS special device

11 March 2024 / / 0 Comments

The spinning disk pool on my home server uses a mirrored special device (for storing metadata and small blocks, see also this blog post at Klara Systems) based on two NVMe SSDs. Because my home server only has two M.2 slots and I wanted to have a pure SSD ZFS pool as well, I partitioned the SSDs. Each SSD has a partition for the SSD pool and one for the special device of the storage pool (which uses a mirror of spinning disks).

Note: This isn’t really a recommended production setup as you are basically hurting performance of both the special device and the SSD pool. But for my home server this works fine. For example, I use the special device’s small blocks functionality to store previews of the photo’s I store on my Nextcloud server. This makes scrolling through the Memories app’s timeline a breeze, even though the full-size photo’s are stored on the spinning disks.

Today, I noticed that the special device had filled up, and, given that there was still some unpartitioned space on the SSDs, I wondered if I could just expand the partition (using parted) used by the special device and then have the ZFS pool recognise the extra space. In the past I have expanded partition-based ZFS pools before, e.g. on after upgrading the SSD on my laptop, but I hadn’t tried this with a special device before.

After some experimentation, I can tell you: this works.

Here is how I tested this on a throw-away file-backed zpool. First create four test files: two for the actual mirror pool and two that I’ll add as a special device. 

for i in {0..3} ; do truncate -s 1G file$i.raw ; done
ls -lh

total 4,0K
-rw-rw-r-- 1 lennart lennart 1,0G mrt 11 12:46 file0.raw
-rw-rw-r-- 1 lennart lennart 1,0G mrt 11 12:46 file1.raw
-rw-rw-r-- 1 lennart lennart 1,0G mrt 11 12:46 file2.raw
-rw-rw-r-- 1 lennart lennart 1,0G mrt 11 12:46 file3.raw

Create a regular mirror pool: 

zpool create testpool mirror $(pwd)/file0.raw $(pwd)/file1.raw
zpool list -v testpool

NAME                       SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
testpool                   960M   146K   960M        -         -     0%     0%  1.00x    ONLINE  -
  mirror-0                 960M   104K   960M        -         -     0%  0.01%      -    ONLINE
    /tmp/tests/file0.raw     1G      -      -        -         -      -      -      -    ONLINE
    /tmp/tests/file1.raw     1G      -      -        -         -      -      -      -    ONLINE

Add the special device to the zpool: 

zpool add testpool special mirror $(pwd)/file2.raw $(pwd)/file3.raw
zpool list -v testpool

NAME                       SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
testpool                  1.88G   190K  1.87G        -         -     0%     0%  1.00x    ONLINE  -
  mirror-0                 960M   190K   960M        -         -     0%  0.01%      -    ONLINE
    /tmp/tests/file0.raw     1G      -      -        -         -      -      -      -    ONLINE
    /tmp/tests/file1.raw     1G      -      -        -         -      -      -      -    ONLINE
special                       -      -      -        -         -      -      -      -         -
  mirror-1                 960M      0   960M        -         -     0%  0.00%      -    ONLINE
    /tmp/tests/file2.raw     1G      -      -        -         -      -      -      -    ONLINE
    /tmp/tests/file3.raw     1G      -      -        -         -      -      -      -    ONLINE

I wasn’t sure whether I could just truncate the backing files for the special device to a larger size while they were part of the pool, so I detached them one by one and created new ones of 2GB, and then reattached them: 

zpool detach testpool /tmp/tests/file2.raw
zpool list -v testpool
truncate -s 2G file2.raw
zpool attach testpool $(pwd)/file3.raw /tmp/tests/file2.raw
zpool list -v testpool


NAME                       SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
testpool                  1.88G   194K  1.87G        -         -     0%     0%  1.00x    ONLINE  -
  mirror-0                 960M   104K   960M        -         -     0%  0.01%      -    ONLINE
    /tmp/tests/file0.raw     1G      -      -        -         -      -      -      -    ONLINE
    /tmp/tests/file1.raw     1G      -      -        -         -      -      -      -    ONLINE
special                       -      -      -        -         -      -      -      -         -
  /tmp/tests/file3.raw       1G    90K   960M        -         -     0%  0.00%      -    ONLINE
NAME                       SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
testpool                  1.88G   278K  1.87G        -         -     0%     0%  1.00x    ONLINE  -
  mirror-0                 960M  97.5K   960M        -         -     0%  0.00%      -    ONLINE
    /tmp/tests/file0.raw     1G      -      -        -         -      -      -      -    ONLINE
    /tmp/tests/file1.raw     1G      -      -        -         -      -      -      -    ONLINE
special                       -      -      -        -         -      -      -      -         -
  mirror-1                 960M   180K   960M        -         -     0%  0.01%      -    ONLINE
    /tmp/tests/file3.raw     1G      -      -        -         -      -      -      -    ONLINE
    /tmp/tests/file2.raw     2G      -      -        -         -      -      -      -    ONLINE

And for the second “disk”: 

zpool detach testpool /tmp/tests/file3.raw
zpool list -v testpool
truncate -s 2G file3.raw
zpool attach testpool $(pwd)/file2.raw /tmp/tests/file3.raw
zpool list -v testpool


NAME                       SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
testpool                  1.88G   218K  1.87G        -         -     0%     0%  1.00x    ONLINE  -
  mirror-0                 960M    66K   960M        -         -     0%  0.00%      -    ONLINE
    /tmp/tests/file0.raw     1G      -      -        -         -      -      -      -    ONLINE
    /tmp/tests/file1.raw     1G      -      -        -         -      -      -      -    ONLINE
special                       -      -      -        -         -      -      -      -         -
  /tmp/tests/file2.raw       2G   152K   960M        -        1G     0%  0.01%      -    ONLINE
NAME                       SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
testpool                  1.88G   296K  1.87G        -         -     0%     0%  1.00x    ONLINE  -
  mirror-0                 960M    54K   960M        -         -     0%  0.00%      -    ONLINE
    /tmp/tests/file0.raw     1G      -      -        -         -      -      -      -    ONLINE
    /tmp/tests/file1.raw     1G      -      -        -         -      -      -      -    ONLINE
special                       -      -      -        -         -      -      -      -         -
  mirror-1                 960M   242K   960M        -        1G     0%  0.02%      -    ONLINE
    /tmp/tests/file2.raw     2G      -      -        -        1G      -      -      -    ONLINE
    /tmp/tests/file3.raw     2G      -      -        -         -      -      -      -    ONLINE

And now, here comes the magic. Time to expand the pool and see of the special device will grow to 2GB: 

zpool online -e testpool $(pwd)/file2.raw $(pwd)/file3.raw
zpool list -v testpool

NAME                       SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
testpool                  2.88G   226K  2.87G        -         -     0%     0%  1.00x    ONLINE  -
  mirror-0                 960M    48K   960M        -         -     0%  0.00%      -    ONLINE
    /tmp/tests/file0.raw     1G      -      -        -         -      -      -      -    ONLINE
    /tmp/tests/file1.raw     1G      -      -        -         -      -      -      -    ONLINE
special                       -      -      -        -         -      -      -      -         -
  mirror-1                1.94G   178K  1.94G        -         -     0%  0.00%      -    ONLINE
    /tmp/tests/file2.raw     2G      -      -        -         -      -      -      -    ONLINE
    /tmp/tests/file3.raw     2G      -      -        -         -      -      -      -    ONLINE

Yay! It worked! So, for my actual storage pool, I ended up doing the following:

  • Given that the partitions used by the special device are located at the end of the SSD, it was easy to expand them using parted resizepart.
  • Run zpool online -e storage partname-1 partname-2 (so I didn’t detach/attach here).

Don’t forget to clean up the testpool: 

zpool destroy testpool
rm -r /tmp/tests

Related Images:

Bulk downloading and renaming of Expensify PDF reports

8 March 2024 / / 0 Comments

In my company, we have been using Expensify to manage small receipts, travel expenses, etc. Recently, however, I decided to switch to another platform that is part of the SAAS platform that our accountant uses. Even though it lacks some of the functionality provided by Expensify, having all receipts in a single location reduces the amount of time I have to spend on administrative tasks.

Every quarter Dutch companies have to file a VAT report, which meant I exported the Expensify reports to CSV files (to send to my accountant) and in PDF form, as a more “visual” backup, which lists the reported expenses, sorted in categories, and, importantly, also includes the scans on the various receipts.

As we changed accountants a couple of years ago, I wasn’t sure whether I had actually downloaded both the CSV and the PDF file for each Expensify report. Keeping records is required by Dutch law, so I decided to make sure and download all PDF files and back them up somewhere.

Unfortunately, the Expensify website doesn’t offer an option for bulk downloading of the PDF files. They do offer a kind of REST API (they call it the Integration Server), that I had played with years ago, so I decided to try that. Luckily, the credentials I had saved in my password manager still worked.

The process for downloading the PDFs consists of two steps:

  • Run a command to generate the reports, this returns the file names for the PDF files.
  • Use those names to download the PDFs

The first step took a couple of minutes to run and then listed the filenames for the PDF on stdout: 

curl -X POST 'https://integrations.expensify.com/Integration-Server/ExpensifyIntegrations' \
    -d 'requestJobDescription={
        "type":"file",
        "credentials":{
            "partnerUserID":"XXXXXXXXXX",
            "partnerUserSecret":"YYYYYYYYYY"
        },
        "onReceive":{
            "immediateResponse":["returnRandomFileName"]
        },
        "inputSettings":{
            "type":"combinedReportData",
            "filters":{
                "startDate":"2013-01-01"
            }
        },
        "outputSettings":{
            "fileExtension":"pdf",
            "includeFullPageReceiptsPdf":"true"
        }
    }' \
    --data-urlencode 'template@expensify_template.ftl'

I’m not sure what the expensify_template.ftl file does in this command, but it was necessary to create that file locally, otherwise the curl call would return an error. I simply copied the example from the sample provided in the documentation for the Expensify Integration Server. I made a copy of the long list of PDF filenames output by the above command. A typical filename would look like this: exportc992bd79-aa4a-4b04-a76a-1149194bac94-34589514.pdf. Not very descriptive… As expected (confirmed in the web UI), there were 191 file names.

Next, step two: actually downloading the files. The basic call for that is: 

curl -X POST 'https://integrations.expensify.com/Integration-Server/ExpensifyIntegrations' \
    -d 'requestJobDescription={
        "type":"download",
        "credentials":{
            "partnerUserID":"XXXXXXXXXX",
            "partnerUserSecret":"YYYYYYYYYY"
        },
        "fileName":"exportc992bd79-aa4a-4b04-a76a-1149194bac94-5803035.pdf",
        "fileSystem":"integrationServer"}
    }' \
    --data-urlencode 'template@expensify_template.ftl' --output "my_output.pdf"

So, in order to download all PDFs, I saved all file names in the file pdflist. All PDF file names are unique: 

$ wc -l pdflist
191 pdflist
$ sort pdflist| uniq | wc -l
191

Next, I used a loop to read each line from the pdflist file and fiddled a bit with the quotes so I could use the pdf variable in the Curl call and download each file: 

cat pdflist | while read pdf; do
curl -X POST 'https://integrations.expensify.com/Integration-Server/ExpensifyIntegrations' \
    -d "requestJobDescription={
        'type':'download',
        'credentials':{
            'partnerUserID':'XXXXXXXXXX',
            'partnerUserSecret':'YYYYYYYYYY'
        },
        'fileName':${pdf},
        'fileSystem':'integrationServer'}
    }" \
    --data-urlencode 'template@expensify_template.ftl' --output ${pdf}
done

This indeed gave me 191 Expensify report PDFs, with very uninformative names 😐 . To fix that I resorted to some more shell “scripting”. Every report has a title (usually something like “Small expenses 2020 Q4”) and by using the pdftotext utility, it looked like this was always on the third line of the pdftotext output. So I moved the original PDFs to a separate “archive” directory OriginalExports and ran the following to make a copy of each PDF to a new name that was equal to its title. My first attempt failed somewhat, because the number of renamed PDF files as smaller than the number of original PDFs. I guessed this would happen when two reports have the same name, and indeed, adding -i to the cp command to warn me of this showed I was right. As this was only happening for four files, I manually converted those. 

for pdf in OriginalExports/export*.pdf; do
echo ${pdf}
title=$(pdftotext ${pdf} - | head -3 | tail -1 | tr "/" "_")
cp -i ${pdf} "${title}.pdf"
done

So there I had my backup of all receipts since we started using Expensify. And if the tax office or the accountant ever want to see those receipts, I am now sure I can provide them.

Related Images:

Fixing delayed syncing with Linux Nextcloud client

30 January 2024 / / 0 Comments

Recently, I noticed that changed files were not picked up by the Nextcloud client as fast as before. As a result I sometimes missed a file (or changes in a file) on my laptop that had been created on my desktop PC.

Today, I tried to run tail and got the following message: 

tail: inotify resources exhausted
tail: inotify cannot be used, reverting to polling

This made me realise that the problem with the delayed client sync could be related to the inotify system for monitoring file changes.

It turns out there is a maximum to the number of inotify file watches: 

sysctl fs.inotify

fs.inotify.max_queued_events = 16384
fs.inotify.max_user_instances = 128
fs.inotify.max_user_watches = 524288

Given that on my desktop the Nextcloud client syncs about 235 GB with my personal Nextcloud server and about 10 GB to two servers for work (including several Git repositories), I could imagine the 65536 watches is not enough. Indeed, manually increasing that number made file syncs more or less instantaneous again: 

sysctl -n -w fs.inotify.max_user_watches=524288
sysctl fs.inotify

524288
fs.inotify.max_queued_events = 16384
fs.inotify.max_user_instances = 128
fs.inotify.max_user_watches = 524288

To make these changes permanent I added the following lines at the bottom of /etc/sysctl.conf: 

grep -A1 inotify /etc/sysctl.conf

# Increase the number of inotify watches so that the Nextcloud client
# keeps syncing without delay
fs.inotify.max_user_watches=524288

Side note: to apply the changes in the /etc/sysctl.conf file use sudo sysctl -p.

Related Images:

Don’t install Citrix’s AppProtection!

2 June 2023 / / 0 Comments

Today I returned to my desktop computer after having been away for a couple of days, during which I only used my laptop. Both machines run Ubuntu Linux, currently version 23.04.

The project I was going to work on required R, but for some reason R didn’t start. No error message, only an exit status of 1 and I was back at the shell prompt. Running R --version worked fine, but Rscript failed in the same way as regular R. I tried running R --vanilla, starting R as a different user, nothing helped.

Time to dig deeper. Make sure all Ubuntu packages are up to date. Reinstall the r-base and r-base-core packages, check whether those packages had been recently updated (no), check whether the package versions were identical to the ones on my laptop (where R worked fine). Nothing…

Maybe there is problem with a (missing) dynamic library? In the (distant) past I have had problems with that, so it is worth a shot:

$ ldd /usr/bin/Rscript
	linux-vdso.so.1 (0x00007ffcac7d4000)
	/usr/local/lib/AppProtection/libAppProtection.so (0x00007f01b9e00000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f01b9bfb000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f01ba204000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f01ba1ff000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f01ba22f000)
	libX11.so.6 => /lib/x86_64-linux-gnu/libX11.so.6 (0x00007f01ba0c1000)
	libxcb.so.1 => /lib/x86_64-linux-gnu/libxcb.so.1 (0x00007f01ba095000)
	libXi.so.6 => /lib/x86_64-linux-gnu/libXi.so.6 (0x00007f01ba081000)
	libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f01b9991000)
	libXau.so.6 => /lib/x86_64-linux-gnu/libXau.so.6 (0x00007f01ba07b000)
	libXdmcp.so.6 => /lib/x86_64-linux-gnu/libXdmcp.so.6 (0x00007f01ba073000)
	libXext.so.6 => /lib/x86_64-linux-gnu/libXext.so.6 (0x00007f01ba05c000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f01b98a8000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f01ba038000)
	libbsd.so.0 => /lib/x86_64-linux-gnu/libbsd.so.0 (0x00007f01b9893000)
	libmd.so.0 => /lib/x86_64-linux-gnu/libmd.so.0 (0x00007f01ba02b000)

That’s strange: I usually don’t have stuff installed in /usr/local. What is this AppProtection library doing there? And then it hit me: I recently (the last time I had used my desktop PC) had to install Citrix’s ICA client to do some remote desktop work for one of my clients. When I installed that package I was asked something about installing some sort of app protection. I had selected “yes”… With a feature named like that I should have known better…

Anyway, time to see if this shared library was indeed the problem. I moved the /usr/local/lib/AppProtection/ directory out of the way and tried to start R. All was fine and dandy again! Except that even an ls command now gave an error:

$ ls /usr/local/lib/AppProtection
ERROR: ld.so: object '/usr/local/lib/AppProtection/libAppProtection.so' from /etc/ld.so.preload cannot be preloaded (cannot open shared object file): ignored.
ls: cannot access '/usr/local/lib/AppProtection': No such file or directory
ERROR: ld.so: object '/usr/local/lib/AppProtection/libAppProtection.so' from /etc/ld.so.preload cannot be preloaded (cannot open shared object file): ignored.

Apparently (obviously?), something still tried to preload the library, even though it no longer existed. It turns out this was done in the file /etc/ld.so.preload:

/usr/local/lib/AppProtection/libAppProtection.so

Given that this was the only content of that file, I opted to just delete it. Finally, my system is back in a working state.

Conclusion: be more careful when installing stuff from external sources and definitely don’t install anything you don’t really need, like Citrix’s App Protection.

P.S. It turns out this was also the reason why the Mattermost app wasn’t running successfully any more.

Related Images:

LXD container snapshots, ZFS snapshots and moving containers

11 March 2023 / / 0 Comments

Here, we investigate the behaviour of LXD when moving containers between LXD cluster nodes, with a focus on various types of (filesystem) snapshots.

LXD containers can be snapshot by LXD itself, but in case one uses a ZFS storage backend, one can also use a tool like Sanoid to make snapshots of a container’s filesystem. When moving an LXD container from one LXD cluster node to another, one, of course, wants those filesystem snapshots to move along as well. Spoiler: this isn’t always the case.

Let’s create a test container on my home LXD cluster (which uses ZFS as default storage backend), starting on node wiske2: 

lxc launch ubuntu:22.04 snapmovetest --target=wiske2

Check the container is running: 

lxc list snapmovetest


+--------------+---------+-----------------------+-------------------------------------------+-----------+-----------+----------+
|     NAME     |  STATE  |         IPV4          |                   IPV6                    |   TYPE    | SNAPSHOTS | LOCATION |
+--------------+---------+-----------------------+-------------------------------------------+-----------+-----------+----------+
| snapmovetest | RUNNING | 192.168.10.158 (eth0) | 2a10:3781:782:1:216:3eff:fed5:ef48 (eth0) | CONTAINER | 0         | wiske2   |
+--------------+---------+-----------------------+-------------------------------------------+-----------+-----------+----------+

Now, let’s use LXD to create two snapshots: 

lxc snapshot snapmovetest "Test1"
sleep 10
lxc snapshot snapmovetest "Test2"

Check the snapshots have been made: 

lxc info snapmovetest | awk '$1=="Snapshots:" {toprint=1}; {if(toprint==1) {print $0}}'


Snapshots:
+-------+----------------------+------------+----------+
| NAME  |       TAKEN AT       | EXPIRES AT | STATEFUL |
+-------+----------------------+------------+----------+
| Test1 | 2023/03/11 22:22 CET |            | NO       |
+-------+----------------------+------------+----------+
| Test2 | 2023/03/11 22:22 CET |            | NO       |
+-------+----------------------+------------+----------+

At the ZFS level: 

zfs list -rtall rpool/lxd/containers/snapmovetest


NAME                                               USED  AVAIL     REFER  MOUNTPOINT
rpool/lxd/containers/snapmovetest                 24.7M   192G      748M  legacy
rpool/lxd/containers/snapmovetest@snapshot-Test1    60K      -      748M  -
rpool/lxd/containers/snapmovetest@snapshot-Test2    60K      -      748M  -

All is fine! Now, let’s move the container to node wiske3: 

lxc stop snapmovetest
lxc move snapmovetest snapmovetest --target=wiske3
lxc list snapmovetest


+--------------+---------+------+------+-----------+-----------+----------+
|     NAME     |  STATE  | IPV4 | IPV6 |   TYPE    | SNAPSHOTS | LOCATION |
+--------------+---------+------+------+-----------+-----------+----------+
| snapmovetest | STOPPED |      |      | CONTAINER | 2         | wiske3   |
+--------------+---------+------+------+-----------+-----------+----------+

Check the snapshots: 

lxc info snapmovetest | awk '$1=="Snapshots:" {toprint=1}; {if(toprint==1) {print $0}}'


Snapshots:
+-------+----------------------+------------+----------+
| NAME  |       TAKEN AT       | EXPIRES AT | STATEFUL |
+-------+----------------------+------------+----------+
| Test1 | 2023/03/11 22:22 CET |            | NO       |
+-------+----------------------+------------+----------+
| Test2 | 2023/03/11 22:22 CET |            | NO       |
+-------+----------------------+------------+----------+

At the ZFS level: 

zfs list -rtall rpool/lxd/containers/snapmovetest


NAME                                               USED  AVAIL     REFER  MOUNTPOINT
rpool/lxd/containers/snapmovetest                  749M   202G      748M  legacy
rpool/lxd/containers/snapmovetest@snapshot-Test1    60K      -      748M  -
rpool/lxd/containers/snapmovetest@snapshot-Test2    60K      -      748M  -

So far so good: snapshots taken with the native LXD toolchain get moved. Now let’s manually create a ZFS snapshot: 

zfs snapshot rpool/lxd/containers/snapmovetest@manual_zfs_snap
zfs list -rtall rpool/lxd/containers/snapmovetest


NAME                                                USED  AVAIL     REFER  MOUNTPOINT
rpool/lxd/containers/snapmovetest                   749M   202G      748M  legacy
rpool/lxd/containers/snapmovetest@snapshot-Test1     60K      -      748M  -
rpool/lxd/containers/snapmovetest@snapshot-Test2     60K      -      748M  -
rpool/lxd/containers/snapmovetest@manual_zfs_snap     0B      -      748M  -

Nove move the container back to node wiske2: 

lxc move snapmovetest snapmovetest --target=wiske2
lxc list snapmovetest


+--------------+---------+------+------+-----------+-----------+----------+
|     NAME     |  STATE  | IPV4 | IPV6 |   TYPE    | SNAPSHOTS | LOCATION |
+--------------+---------+------+------+-----------+-----------+----------+
| snapmovetest | STOPPED |      |      | CONTAINER | 2         | wiske2   |
+--------------+---------+------+------+-----------+-----------+----------+

What happened to the snapshots? 

lxc info snapmovetest | awk '$1=="Snapshots:" {toprint=1}; {if(toprint==1) {print $0}}'


Snapshots:
+-------+----------------------+------------+----------+
| NAME  |       TAKEN AT       | EXPIRES AT | STATEFUL |
+-------+----------------------+------------+----------+
| Test1 | 2023/03/11 22:22 CET |            | NO       |
+-------+----------------------+------------+----------+
| Test2 | 2023/03/11 22:22 CET |            | NO       |
+-------+----------------------+------------+----------+


zfs list -rtall rpool/lxd/containers/snapmovetest


NAME                                               USED  AVAIL     REFER  MOUNTPOINT
rpool/lxd/containers/snapmovetest                  749M   191G      748M  legacy
rpool/lxd/containers/snapmovetest@snapshot-Test1    60K      -      748M  -
rpool/lxd/containers/snapmovetest@snapshot-Test2    60K      -      748M  -

Somehow, the ZFS-level snapshot has been removed… I guess this part of the LXD manual should be written in bold (emphasis mine):

LXD assumes that it has full control over the ZFS pool and dataset. Therefore, you should never maintain any datasets or file system entities that are not owned by LXD in a ZFS pool or dataset, because LXD might delete them.

Consequently, in a LXD cluster one shouldn’t use Sanoid to make snapshots ZFS-backed LXD container filesystems. Instead, use LXD’s builtin automatic snapshot capabilities (see the snapshots.expiry and snapshots.schedule options).

Clean up: 

lxc delete snapmovetest

Related Images:

Use a script to convert Office files to PDF via Nautilus’ right-click menu

28 June 2021 / / 0 Comments

Recently, I bought a reMarkable 2, a tablet-like device with an e-ink screen that allows me to replace real paper with digital note taking, while keeping the hand-written aspect of writing notes. The device also allows me to read and annotate PDFs in a comfortable way. Given that I use RMfuse to ‘mount’ the reMarkable cloud on my computer, I normally drag-n-drop the files I want to read in GNOME Files (formerly Nautilus, GNOME’s file manager) from their location on my computer to the mounted cloud.

This works really well so far. However, I also regularly receive files in the MS Office .docx format. Often I need to make substantial changes in these documents, which I do on my laptop or computer. But sometimes I only need to read them or only put my signature at the bottom. For these cases I would open the .docx file in LibreOffice, convert it to PDF and copy it to my reMarkable. In order to speed this up, I thought it would be nice to have some way where I right-click on a .docx file in GNOME Files/Nautilus and then convert it to PDF automatically, after which I can drag-n-drop the PDF file to the mounted reMarkable cloud.

So the question was: how can I add an item to the right-click menu of Nautilus, which runs a script when I click on it. After looking around on the internet, this turned out to be quite easy. It turns out that scripts placed in ~/.local/share/nautilus/scripts/ end up in the Nautilus right-click menu under the Scripts submenu.

To do the actual conversion to PDF, I created the following script: 

#!/bin/bash
# This script converts the selected file to PDF using LibreOffice
# For general instructions on how to use Nautilus scripts, see
# https://help.ubuntu.com/community/NautilusScriptsHowto
#
# Save this script in ~/.local/share/nautilus/scripts/ and make it
# executable.

IFS_BAK=$IFS
IFS="
"

for SelectedFile in ${NAUTILUS_SCRIPT_SELECTED_FILE_PATHS}; do
    soffice \
        --nodefault \
        --nolockcheck \
        --nologo \
        --norestore \
        --nofirststartwizard \
        --convert-to pdf "${SelectedFile}"
done

IFS=$IFS_BAK

Because I wanted to be able to select multiple files, some having spaces in their names, I had to make sure the space character wasn’t used to split the NAUTILUS_SCRIPT_SELECTED_FILE_PATHS variable. That is why I temporarily replace the IFS variable with a newline.

I have only tried this on .docx files so far, but I guess it would work on presentations and spreadsheets as well.

Related Images:

Ubuntu Thunderbolt settings

Using the Lenovo Thunderbolt 3 Essential Dock with Linux

5 January 2021 / / 1 Comment

Today the Lenovo Thunderbolt 3 Essential dock I had ordered just before the new year arrived. My current laptop is a 6th Gen Lenovo Thinkpad X1 Carbon, which actually has two Tunderbolt 3 (TB3) ports. For a while the cable mess on my desk had been bothering me and a dock looked like a good way to get rid of both the clutter and the fact that I had to plug in a power cable, an HDMI cable, a USB cable and a network cable. The latter was especially tricky, because my laptop doesn’t have a dedicated ethernet port, but instead has a dongle that plugs into one of the TB3 ports. Of course, I didn’t want it to happen that I was on the road without the dongle, so double checking and making sure I had it in my bag was a regular worry.

With the TB3 Essential dock all this should be over. The dock is pretty well equipped:

  • 1 ethernet port (1Gbps)
  • 1 HDMI 2.0 port
  • 1 DisplayPort 1.4
  • 2 USB A 3.0 1.5Gbps ports
  • 2 USB C 10Gbps ports (no video support)
  • 1 3.5 mm audio jack.

The one thing that remained to be seen, of course, was whether the all those ports would work under Linux as well. I read some promising reports for other TB3-based docks from Lenovo, so I decided to order it.

After connecting the various cables to the dock came the moment supreme: I plugged in the TB3 cable to the laptop. And… The display connected to the HDMI port lit up. So far, so good! No USB functionality, however. Time to dive into the BIOS, because I remembered having seen some settings there, including some security related ones.

In the BIOS I couldn’t change the TB3 security setting because, as the help message explained, the graphics memory was set to 512MB. I’m not sure why this is an issue, but I looked up the graphics RAM setting and reduced it to 256MB. Next, I went back to the TB settings and set the security level to the first option: “No security”, followed by a reboot. Now, everything worked. That is one step in the right direction.

However, I wasn’t willing to forgo all security, so I went back to the BIOS settings and set the security level to “Secure Connection” (according to this blog post at dell.com this is security level 2 or SL2). I rebooted and indeed, no USB. So I went to Ubuntu’s Thunderbolt settings and there I had to press the ‘Unlock’ button in the top right corner, after which I could click on a button authorise the dock. After that all connections worked again. From the commandline, the boltctl utility can be used to see more information on the connected thunderbolt devices. This is the output for an unauthorised device:

$ boltctl list
? Lenovo Thunderbolt 3 Essential Dock
  ?? type:          peripheral
  ?? name:          Thunderbolt 3 Essential Dock
  ?? vendor:        Lenovo
  ?? uuid:          00b00089-417d-0801-ffff-ffffffffffff
  ?? status:        connected
  ?  ?? domain:     ca030000-0070-6f08-2382-4312b0238921
  ?  ?? authflags:  none
  ?? connected:     ma 04 jan 2021 16:26:45 UTC
  ?? stored:        ma 04 jan 2021 16:20:57 UTC
     ?? policy:     iommu
     ?? key:        no

And this is the output after clicking the “Authorise” button in the Ubuntu settings:

? boltctl list
? Lenovo Thunderbolt 3 Essential Dock
  ?? type:          peripheral
  ?? name:          Thunderbolt 3 Essential Dock
  ?? vendor:        Lenovo
  ?? uuid:          00b00089-417d-0801-ffff-ffffffffffff
  ?? status:        authorized
  ?  ?? domain:     ca030000-0070-6f08-2382-4312b0238921
  ?  ?? authflags:  none
  ?? authorized:    ma 04 jan 2021 16:30:07 UTC
  ?? connected:     ma 04 jan 2021 16:26:45 UTC
  ?? stored:        ma 04 jan 2021 16:20:57 UTC
     ?? policy:     iommu
     ?? key:        yes (new)

Nice!

However, my joy was shortlived, because after disconnecting and reconnecting the dock, the USB ports had stopped working again. The device had to be authorised again. This seemed tedious, so I set about reading more in the boltctl man page and found that there was a way to enroll a device permanently. So that is what I did. First I removed its UUID from the database:

$ boltctl forget 00b00089-417d-0801-ffff-ffffffffffff

And then added it again, this time with the auto policy:

$ boltctl enroll --policy auto 00b00089-417d-0801-ffff-ffffffffffff
 ? Lenovo Thunderbolt 3 Essential Dock
   ?? type:          peripheral
   ?? name:          Thunderbolt 3 Essential Dock
   ?? vendor:        Lenovo
   ?? uuid:          00b00089-417d-0801-ffff-ffffffffffff
   ?? dbus path:     /org/freedesktop/bolt/devices/00b00089_417d_0801_ffff_ffffffffffff
   ?? status:        authorized
   ?  ?? domain:     ca030000-0070-6f08-2382-4312b0238921
   ?  ?? parent:     ca030000-0070-6f08-2382-4312b0238921
   ?  ?? syspath:    /sys/devices/pci0000:00/0000:00:1d.0/0000:05:00.0/0000:06:00.0/0000:07:00.0/domain0/0-0/0-1
   ?  ?? authflags:  secure
   ?? authorized:    ma 04 jan 2021 22:33:08 UTC
   ?? connected:     ma 04 jan 2021 22:32:43 UTC
   ?? stored:        ma 04 jan 2021 16:20:57 UTC
      ?? policy:     auto
      ?? key:        yes

As you can see from the boltctl list output below, the policy is now set to auto (instead of the previous iommu):

$ boltctl list
 ? Lenovo Thunderbolt 3 Essential Dock
   ?? type:          peripheral
   ?? name:          Thunderbolt 3 Essential Dock
   ?? vendor:        Lenovo
   ?? uuid:          00b00089-417d-0801-ffff-ffffffffffff
   ?? status:        authorized
   ?  ?? domain:     ca030000-0070-6f08-2382-4312b0238921
   ?  ?? authflags:  secure
   ?? authorized:    ma 04 jan 2021 22:41:46 UTC
   ?? connected:     ma 04 jan 2021 22:41:45 UTC
   ?? stored:        ma 04 jan 2021 16:20:57 UTC
      ?? policy:     auto
      ?? key:        yes

Now I can disconnect and reconnect the dock without problems :-).

I also tested the 3.5mm audio port, which worked (at least for listening, I didn’t have a headset with microphone at hand). Same for the two USB A and the two USB C ports. Finally, I tested the DisplayPort and that worked too. In fact, connecting my 3440×1440 screen via both HDMI and DP to the dock worked fine. The Ubuntu display settings showed a “3 monitor” setup, two 3440×1440 screens and the latop’s own screen at 2560×1440.

So, in conclusion: the Lenovo Thunderbolt 3 Essential dock is fully supported under Ubuntu 20.04.

Thanks to this blog post at FunnelFiasco.com for pointing me to the boltctl utility!

Related Images:

Fixing Emacs tramp mode when using zsh

2 March 2016 / / 9 Comments

Today I finally took some time to fix a long-standing problem: when trying to edit a file on a remote host using Emacs tramp mode, long time-outs occurred when typing the remote file name (after hitting C-x C-f). These time-outs were so long and occurred after each key press that tramp was effectively useless.

After some digging (e.g. excluding helm as the problem source) I found this entry in the Emacs Wiki which basically told my that using zsh (the Z shell) on the remote host could be the culprit. Indeed, after adding 

[[ $TERM == "dumb" ]] && unsetopt zle && PS1='$ ' && return

at the top of my ~/.zshrc file solved the problem instantly. What this line does is simply replacing the shell prompt with a very simple one (a $ followed by a space) if the terminal is of the dumb type (which is the case for tramp).

Related Images:

Installing parted during Ubuntu installation

22 February 2016 / / 0 Comments

When installing Ubuntu (I guess a regular Debian installation won’t be any different), I sometimes would like to manually create or change partitions (by jumping to another terminal, e.g. using Alt-F2) before doing the actual install. My preferred tool for that is parted, however, on regular Ubuntu installation images (at least the server variety), parted isn’t available from the console by default.

Today I noticed that (at least on today’s daily image of Ubuntu 16.04 Xenial), a udeb file for parted is available. This is how you install it: 

udpkg -i /cdrom/pool/main/p/parted/parted-udeb_3.2-15_amd64.udeb

after which you can use parted to your heart’s content.

For more information on udebs see the Debian Installer Internals documentation.

Related Images:

Setting the console font when using an nVidia card

17 October 2015 / / 0 Comments

Even though I do most of the work I do on my workstation in a graphical desktop environment, I sometimes want or need to switch to one of the virtual terminals (consoles), for example when trying to fix a connection problem or hanging desktop environment.

Whenever I had to do this I was always bothered by the fact that the font was so large (or, the other way around, the resolution so low). What made my annoyance worse was that I knew from my early Linux days So, instead of being annoyed I decided to fix this. Thanks to the help of mchid on unix.stackexchange.com I solved in a matter of minutes. The tricky part for me was to realise I am using an nVidia graphics card, which means things are just a little bit different than normally.

Just in case StackExchange ever goes down or this answer gets lost I will reproduce it below.

For newer Debian & Ubuntu distros using nvidia, I had to do the following: First, edit /etc/default/grub. Change the following line: 

#GRUB_GFXMODE=640x480

to this: 

GRUB_GFXMODE=1920x1200
GRUB_GFXPAYLOAD_LINUX=keep

replacing 1280×800 with the desired resolution.

Then: 

echo "echo FRAMEBUFFER=y" | sudo tee /etc/initramfs-tools/conf.d/splash
sudo update-initramfs -u
sudo update-grub

To simply change the font size, you can do so using the following command: 

sudo dpkg-reconfigure console-setup

Related Images:

« Older posts

Archives

© 2024 Lennart's weblog

Theme by Anders Norén — Up ↑