Lennart's weblog

Open source, computers, Africa and other more (or less) interesting stuff.

Tag: Ubuntu (page 1 of 4)

Installing parted during Ubuntu installation

When installing Ubuntu (I guess a regular Debian installation won’t be any different), I sometimes would like to manually create or change partitions (by jumping to another terminal, e.g. using Alt-F2) before doing the actual install. My preferred tool for that is parted, however, on regular Ubuntu installation images (at least the server variety), parted isn’t available from the console by default.

Today I noticed that (at least on today’s daily image of Ubuntu 16.04 Xenial), a udeb file for parted is available. This is how you install it:

udpkg -i /cdrom/pool/main/p/parted/parted-udeb_3.2-15_amd64.udeb

after which you can use parted to your heart’s content.

For more information on udebs see the Debian Installer Internals documentation.

Setting the console font when using an nVidia card

Even though I do most of the work I do on my workstation in a graphical desktop environment, I sometimes want or need to switch to one of the virtual terminals (consoles), for example when trying to fix a connection problem or hanging desktop environment.

Whenever I had to do this I was always bothered by the fact that the font was so large (or, the other way around, the resolution so low). What made my annoyance worse was that I knew from my early Linux days So, instead of being annoyed I decided to fix this. Thanks to the help of mchid on unix.stackexchange.com I solved in a matter of minutes. The tricky part for me was to realise I am using an nVidia graphics card, which means things are just a little bit different than normally.

Just in case StackExchange ever goes down or this answer gets lost I will reproduce it below.

For newer Debian & Ubuntu distros using nvidia, I had to do the following: First, edit /etc/default/grub. Change the following line:

#GRUB_GFXMODE=640x480

to this:

GRUB_GFXMODE=1920x1200
GRUB_GFXPAYLOAD_LINUX=keep

replacing 1280×800 with the desired resolution.

Then:

echo "echo FRAMEBUFFER=y" | sudo tee /etc/initramfs-tools/conf.d/splash
sudo update-initramfs -u
sudo update-grub

To simply change the font size, you can do so using the following command:

sudo dpkg-reconfigure console-setup

Moving annual backups from an external disk with Ext4 to an external disk with ZFS

For a few years I have used the Christmas holidays to create a full
backup of my /home on an external hard disk. For that I used a
Bash script around rsync that uses hard links to keep the used disk
space under control. Each backup was saved in a directory named with
the date of the backup. POSIX ACLs were also backed up.

Since last year’s backup I have moved to ZFS (using ZFS on Linux
with Ubuntu 14.04
) as filesystem for /home (and others). Since ZFS
makes checksums of data and metadata it has the possibility to
detect corrupted files (and if the data is redundant it can also fix
them). This is a feature I’d like to have for my backups as
well: I’d rather know it when corruption occurs than live in
ignorance.

So the plan is to move the old backups from the external disk to the
ZFS pool in my server. and instead of using hard links I’ll transfer
the backups in order from old to new to the ZFS pool making a
snapshot for each. Additionally I will also turn on compression
(using the lz4 algorithm). Once that is done I will reformat the
external drive and create a ZFS pool called “JaarlijkseBackupPool” on
it (jaarlijks means annual in Dutch).

The old situation

In the current/old situation, this is how much disk space is used
on the external disk (with and without taking the hard links into
account):

$ sudo du -csh /mnt/JaarlijkseBackups/*
102G    /mnt/JaarlijkseBackups/2010-11-28
121G    /mnt/JaarlijkseBackups/2013-02-04
101G    /mnt/JaarlijkseBackups/2013-12-23
324G    total
$ sudo du -clsh /mnt/JaarlijkseBackups/*
102G    /mnt/JaarlijkseBackups/2010-11-28
193G    /mnt/JaarlijkseBackups/2013-02-04
255G    /mnt/JaarlijkseBackups/2013-12-23
549G    total

Copying the data from the Ext4 disk to a temporary ZFS filesystem on my server

The ZFS pool in my server is called storage. In order to save the
POSIX ACLs of the Ext4 system, they need to be enabled when
creating the ZFS filesystem as well. Setting xattr=sa means the
ACLS are stored more efficiently (although this option is not
compatible with other ZFS implementations at this time, so if I
would try to import the ZFS pool in FreeBSD for example, that
information would be inaccessible).

$ zfs create storage/JaarlijkseBackupsOrganized \
      -o compression=lz4 \
      -o acltype=posixacl \
      -o xattr=sa
$ sudo rsync -ahPAXHS --numeric-ids \
     /storage/JaarlijkseBackups/2010-11-28/ \
     /storage/JaarlijkseBackupsOrganized
$ zfs snapshot storage/JaarlijkseBackupsOrganized@2010-11-28

Followed by the same for the same rsync and zfs snapshot
commands for the other two dates.
Once that is finished, this is the status of that ZFS FS:

$ zfs list -r -t all storage/JaarlijkseBackupsOrganized
NAME                                            USED  AVAIL  REFER  MOUNTPOINT
storage/JaarlijkseBackupsOrganized              275G   438G   272G  /storage/JaarlijkseBackupsOrganized
storage/JaarlijkseBackupsOrganized@2010-11-28  1,03G      -  88,9G  -
storage/JaarlijkseBackupsOrganized@2013-02-04  2,33G      -   196G  -
storage/JaarlijkseBackupsOrganized@2013-12-23      0      -   272G  -
$ zfs get -r -t all compressratio storage/JaarlijkseBackupsOrganized
NAME                                           PROPERTY       VALUE  SOURCE
storage/JaarlijkseBackupsOrganized             compressratio  1.13x  -
storage/JaarlijkseBackupsOrganized@2010-11-28  compressratio  1.19x  -
storage/JaarlijkseBackupsOrganized@2013-02-04  compressratio  1.14x  -
storage/JaarlijkseBackupsOrganized@2013-12-23  compressratio  1.12x  -

Partitioning the external disk

The external disk is as 1TB Samsung SATA 3Gbps SpinPoint F2 EcoGreen disk
(type HD103SI, serial number: S1VSJD6ZB02657). The disk uses 512B
sectors:

sudo hdparm -I /dev/sdf |grep Sector
     Logical/Physical Sector size:           512 bytes

Before using it with ZFS, it needs to be partitioned. I used
parted:

$ parted /dev/sdf
GNU Parted 2.3
Using /dev/sdf
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) p
Model: ATA SAMSUNG HD103SI (scsi)
Disk /dev/sdf: 1000GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  1000GB  1000GB  primary  ext4

(parted) mklabel
New disk label type? gpt
(parted) u
Unit?  [compact]? MB
(parted) p
Model: ATA SAMSUNG HD103SI (scsi)
Disk /dev/sdf: 1000205MB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start  End  Size  File system  Name  Flags

(parted) mkpart
Partition name?  []? JaarlijkseBackups-HD103SI-S1VSJD6ZB02657
File system type?  [ext2]? zfs
Start? 1M
End? 1000204M
(parted) p
Model: ATA SAMSUNG HD103SI (scsi)
Disk /dev/sdf: 1000205MB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End        Size       File system  Name                                  Flags
 1      1,05MB  1000204MB  1000203MB  ext4         JaarlijkseBackups-HD103SI-S1VSJD6ZB0

(parted) q

This removes the old partition table and creates a new GPT
partition table (which allows naming partitions). Next I set the
units to MB so I can leave 1MB at the beginning and end of the
partition (can be helpful when importing this pool in
e.g. FreeBSD). The disk also shows up in /dev/disk/by=partlabel
now.

Creating the new ZFS pool

$ zpool create -o ashift=9 JaarlijkseBackupPool \
    /dev/disk/by-partlabel/JaarlijkseBackups-HD103SI-S1VSJD6ZB0
$ zpool status JaarlijkseBackupPool
  pool: JaarlijkseBackupPool
 state: ONLINE
  scan: none requested
config:

        NAME                                    STATE     READ WRITE CKSUM
        JaarlijkseBackupPool                    ONLINE       0     0     0
          JaarlijkseBackups-HD103SI-S1VSJD6ZB0  ONLINE       0     0     0

errors: No known data errors

Migrating the data

Now that the new ZFS pool and filesystem are all in place, it is
time to move the backups to the new place, starting with the oldest
backup. The -R option also make sure the attributes like
compression and xattr are transferred to the new FS. The
following commands send each snapshot to the new pool (the -n
option of zfs receive is for doing a dry run, just to show how it
works). After the first snapshot is sent, the other two are sent
using the -i option to zfs send so that only the incremental
differences between the snapshots are sent.

$ zfs send -vR storage/JaarlijkseBackupsOrganized@2010-11-28 | \
      zfs receive -Fvu JaarlijkseBackupPool/oldRsyncBackups
$ zfs send -vR -i storage/JaarlijkseBackupsOrganized@2010-11-28 \
    storage/JaarlijkseBackupsOrganized@2013-02-04 | \
    zfs receive -Fvu JaarlijkseBackupPool/oldRsyncBackups
$ zfs send -vR -i storage/JaarlijkseBackupsOrganized@2013-02-04 \
      storage/JaarlijkseBackupsOrganized@2013-12-23 | \
      zfs receive -Fvu JaarlijkseBackupPool/oldRsyncBackups -n
send from @2013-02-04 to storage/JaarlijkseBackupsOrganized@2013-12-23 estimated size is 84,3G
total estimated size is 84,3G
TIME        SENT   SNAPSHOT
would receive incremental stream of storage/JaarlijkseBackupsOrganized@2013-12-23 into JaarlijkseBackupPool@2013-12-23
14:09:16   4,22M   storage/JaarlijkseBackupsOrganized@2013-12-23
14:09:17   8,46M   storage/JaarlijkseBackupsOrganized@2013-12-23
14:09:18   18,4M   storage/JaarlijkseBackupsOrganized@2013-12-23
14:09:19   24,8M   storage/JaarlijkseBackupsOrganized@2013-12-23
^C
$ zfs send -vR -i  storage/JaarlijkseBackupsOrganized@2013-02-04 \
      storage/JaarlijkseBackupsOrganized@2013-12-23 | \
      zfs receive -Fvu JaarlijkseBackupPool/oldRsyncBackups

Add this year’s backup

At first I tried to add the new backups also to the
oldRsyncBackups FS, but that didn’t work (at least not with an
incremental backup), so I ended up making a new backup. The extra
cost in disk space is not a real problem. Disk space is rather
cheap and the current configuration will last me at least one more
year. So after creating a snapshot called 2014-12-26 of my
/home I ran:

   $ zfs send -v  storage/home@2014-12-26 | \
      zfs receive -Fu JaarlijkseBackupPool/home
$ zfs list -r -t all JaarlijkseBackupPool
NAME                                              USED  AVAIL  REFER  MOUNTPOINT
JaarlijkseBackupPool                              581G   332G    30K  /JaarlijkseBackupPool
JaarlijkseBackupPool/home                         311G   332G   311G  /JaarlijkseBackupPool/home
JaarlijkseBackupPool/home@2014-12-26             51,2M      -   311G  -
JaarlijkseBackupPool/oldRsyncBackups              271G   332G   267G  /JaarlijkseBackupPool/oldRsyncBackups
JaarlijkseBackupPool/oldRsyncBackups@2010-11-28   974M      -  87,1G  -
JaarlijkseBackupPool/oldRsyncBackups@2013-02-04  2,23G      -   193G  -
JaarlijkseBackupPool/oldRsyncBackups@2013-12-23      0      -   267G  -
$ zfs get -r compressratio JaarlijkseBackupPool
NAME                                             PROPERTY       VALUE  SOURCE
JaarlijkseBackupPool                             compressratio  1.15x  -
JaarlijkseBackupPool/home                        compressratio  1.17x  -
JaarlijkseBackupPool/home@2014-12-26             compressratio  1.17x  -
JaarlijkseBackupPool/oldRsyncBackups             compressratio  1.13x  -
JaarlijkseBackupPool/oldRsyncBackups@2010-11-28  compressratio  1.19x  -
JaarlijkseBackupPool/oldRsyncBackups@2013-02-04  compressratio  1.14x  -
JaarlijkseBackupPool/oldRsyncBackups@2013-12-23  compressratio  1.12x  -

Finishing up

In order to be able to disconnect the external drive without
damaging the filesystems use

zpool export JaarlijkseBackupPool

Later, the drive/pool can be imported using the zpool import
command.

Now that the migration is done, the intermediate filesystem
(including the snapshots) can also be removed:

zfs destroy -r storage/JaarlijkseBackupsOrganized

For reference: the old rsync script

#!/bin/sh
#
# Time-stamp: <2013-02-04 16:48:31 (root)>
# This scripts helps me create my annual backups to an external hard
# disk. The script uses rsync's hard link option to make hard links to
# the previous backups for files that haven't changed. It makes the
# backup based on an LVM snapshot it creates of the LV that contains
# the /home partition.
# This script needs to be run as root.
 
today=`date +%F`
olddate="2013-02-04"
 
srcdir="/mnt/backupsrc/"
destdir="/mnt/backupdest/JaarlijkseBackups/$today"
prevdir="/mnt/backupdest/JaarlijkseBackups/$olddate"
 
# LVM options
VG=raid5vg
LV=home
 
# rstnc options
options="-ahPAXHS --numeric-ids"
exclusions="--exclude 'lost+found/'"
#  --exclude '*/.thumbnails'"
# exclusions="$exclusions --exclude '*/.gvfs/'"
# exclusions="$exclusions --exclude '*/.cache/' --exclude '**/Cache'"
# exclusions="$exclusions --exclude '*/.recycle/'"
 
# Check to see if the previous backup directory exists
if [ ! -d $prevdir ]; then
    echo "Error: The directory with the previous back up ($prevdir) doesn't exist" 1>&2
    exit 1
fi
 
# Make a snapshot of the home LV that we can backup
lvcreate -L15G -s -n snap$LV /dev/$VG/$LV
mount /dev/$VG/snap$LV $srcdir
 
 
# Start the backup, first a dry-run, then the full one
rsynccommand="rsync $options $exclusions --link-dest=$prevdir $srcdir $destdir"
 
$rsynccommand -n
 
# Wait for user input
echo "This was a dry run. Press a key to continue with the real stuff or"
echo "hit Ctrl-c to abort."
read dummy
 
$rsynccommand

Enabling external commands in the Nagios web interface

After an upgrade of one of my Ubuntu server that runs Nagios, I ran into the following error message (again…) when I tried to issue a command from the web interface:

Error: Could not stat() command file ‘/var/lib/nagios3/rw/nagios.cmd’!

This post by Barry O’Donovan shows very nicely how this problem can/should be fixed in Ubuntu. Much cleaner than chmod/chown-ing the directories myself. Thanks Barry!

Fixing font errors when running qmon on a remote server

Sun Grid Engine (SGE) is a bath queue system that can be used to distribute computation intensive tasks across one or more servers/CPUs. SGE has a graphical configuration utility called qmon, but when you start it on a remote machine (using SSH), you may end up with errors like this:

Warning: Cannot convert string "-adobe-courier-medium-r-*--14-*-*-*-m-*-*-*" to type FontStruct
Warning: Cannot convert string "-adobe-courier-bold-r-*--14-*-*-*-m-*-*-*" to type FontStruct
Warning: Cannot convert string "-adobe-courier-medium-r-*--12-*-*-*-m-*-*-*" to type FontStruct
X Error of failed request:  BadName (named color or font does not exist)
  Major opcode of failed request:  45 (X_OpenFont)
  Serial number of failed request:  329
  Current serial number in output stream:  340

The warnings are not really a problem, but the error is. It can be solved by running the following on the client (i.e. your local) machine (assuming it runs Debian or Ubuntu):

sudo apt-get install xfonts-75dpi
xset +fp /usr/share/fonts/X11/75dpi
xset fp rehash

Fixing backlight control with Ubuntu on my ThinkPad T440s

Some time ago I bought a ThinkPad T440s for work. It’s an amazing machine! Before that I used a ThinkPad X121, which served me very well on my daily commute. This machine was getting a bit old, and given that my new job (more about which in a later post) also requires me to have a better machine with more screen real estate, it was high time to upgrade.

Ubuntu (13.10 and 14.04) runs well on the T440s, only two things didn’t work as expected:

  • The WWAN interface (mobile internet, from Ericsson) seems to connect when I select it in the network manager, but the adaptor seems to disappear almost immediately after that. A few seconds it appears again. [edit 20140514]I just found out that it’s working, probably this was fixed in Ubuntu 14.04[/edit]
  • The screen’s backlight brightness can be reduced/increase using the Fn-F5 and Fn-F6 keys, but only in a weird way: several key presses are needed for one unit of decrease/increase.

This last bug can be fixed by booting with the following kernel argument:

  acpi_backlight=vendor

Simply add this to the GRUB_CMDLINE_LINUX_DEFAULT line in /etc/default/grub:

  GRUB_CMDLINE_LINUX_DEFAULT="acpi_backlight=vendor quiet splash"

and run sudo update-grub. Reboot and you will be able to change the backlight brightness in finer steps.

I found this solution somewhere on the internet a few weeks ago, wrote it down, but can’t remember anymore what the original URL was. My apologies.

Fixing colours in git output after upgrading to Ubuntu 14.04

After upgrading my Ubuntu 13.10 installation to 14.04, I noticed that the output of several git commands (e.g. git diff and git log) didn’t show colours as they used to, but showed ESC[ ANSI codes instead.
A quick internet search lead to this post on unix.stackexchange.com where the LESS environment variable was ‘blamed’. Indeed, I have my LESS variable (re-) defined in my .bashrc and .zshrc files.

The solution was to add -R to the environment variable, which allows raw control characters to be displayed. I now have the environment variable defined as:

LESS='--quiet -X -F -R'

Using SSH for WordPress updates via the web interface

Note to self: if you want to be able to upgrade WordPress via the web interface using SSH, you need to have the libssh2-php package installed on Debian/Ubuntu Linux.

Don’t forget to restart Apache after installing the package. Reload the WordPress admin pages and you’ll see the SSH option added.

Fixing the compose key in Emacs on Ubuntu 13.10

After upgrading to Ubuntu 13.10 some time ago I noticed that my compose key (i.e. the key that you press followed by e.g. c and , to create a ç) didn’t work anymore in Emacs. I found two bug reports on this issue [1, 2], both effectively suggesting the same solution: start Emacs like this

XMODIFIERS=@im=none emacs

So I added the following line to my ~/.zshrc and ~/.bashrc files:

alias emacs='XMODIFIERS=@im=none emacs'

(of course somewhere before I set my EDITOR variable). I also changed the command in the launcher I use (GLX-Dock/Cairo-Dock).
This doesn’t make it a system-wide (or even user-wide) fix, e.g. the Firefox add-on “It’s all text” doesn’t get it, but it covers most of my use cases.

ProbABEL v0.4.2 released

During the Christmas holidays I released a new version of ProbABEL (v0.4.2). The official release announcement can be found here. ProbABEL is a toolset that allows running GWAS (Genome-Wide Association Studies) in a fast and efficient manner. It implements regression using the linear, logistic or Cox proportional hazards models.

This version is mostly a bug fix release. The most important user-visible change is the fact that the ‘official’ name for the wrapper script that runs a GWAS over a range of chromosomes is now called probabel instead of probabel.pl. This change was induced by my attempts to get ProbABEL packaged in the Debian Linux repositories. One of the warnings that occurred during the package creation process was a Lintian warning that said that scripts with ‘language extensions’ are not allowed. There are several reasons for that, but the one I found most compelling was the fact that the user shouldn’t be concerned with the programming/scripting language we used to write it in. Moreover, being ‘agnostic’ in this matter also allows us to write such a script in a different language.
Of course, we have left the original name in place (via a symlink) in order not to disrupt any current pipelines. If the user runs the script with the old name a warning appears, urging him/her to start using the new name and that the old name will be deprecated in the future.

In the mean time, ProbABEL v0.4.1 has been accepted in Debian (unstable) and as of today it is also available in Debian ‘testing’. Lots of thanks to the Debian Med team that helped me a lot in preparing the .deb package. Note that the package has been split up in probabel (architecture-dependent files) and probabel-examples (with architecture independent files: the examples). See the Debian Package Tracking System page for ProbABEL for more details of the package.

From Debian the package has trickled down to Ubuntu as well (Launchpad page here), so it will be available by default in the next Ubuntu release (14.04, a.k.a. Trusty Tahr).

Older posts

© 2018 Lennart's weblog

Theme by Anders NorenUp ↑