Lennart's weblog

Notes about open source software, computers, other stuff.

Page 4 of 14

Using rsync to backup a ZFS file system to a remote Synology Diskstation

Some time ago I moved from using LVM to using ZFS on my home server. This meant I also had to change the backup script I used to make backups on a remote Synology Diskstation. Below is the updated script. I also updated it such that it now needs a single command line argument: the hostname of the Diskstation to backup to (because I now have two Diskstations at different locations). If you want to run this script from cron you should set up key-based SSH login (see also here and here).

#!/bin/bash
#
# This script makes a backup of my home dirs to a Synology DiskStation at
# another location. I use ZFS for my /home, so I make a snapshot first and
# backup from there.
#
# This script requires that the first command line argument is the
# host name of the remote backup server (the Synology NAS). It also
# assumes that the location of the backups is the same on each
# remote backup server.
#
# Time-stamp: <2014-10-27 11:35:39 (L.C. Karssen)>
# This script it licensed under the GNU GPLv3.
 
set -u
 
if [ ${#} -lt 1 ]; then
    echo -n "ERROR: Please specify a host name as first command" 1>&2
    echo " line option" 1>&2
    exit -1
fi
 
###############################
# Some settings
###############################
# Options for the remote (Synology) backup destination
DESTHOST=$1
DESTUSER=root
DESTPATH=/volume1/Backups/
DEST=${DESTUSER}@${DESTHOST}:${DESTPATH}
 
# Options for the client (the data to be backed up)
# ZFS options
ZFS_POOL=storage
ZFS_DATASET=home
ZFS_SNAPSHOT=rsync_snapshot
SNAPDIR="/home/.zfs/snapshot/$ZFS_SNAPSHOT"
 
# Backup source path. Don't forget to have trailing / otherwise
# rsync's --delete option won't work
SRC=${SNAPDIR}/
 
# rsync options
OPTIONS="--delete -azvhHSP --numeric-ids --stats"
OPTIONS="$OPTIONS --timeout=60 --delete-excluded"
OPTIONS="$OPTIONS --skip-compress=gz/jpg/mp[34]/7z/bz2/ace/avi/deb/gpg/iso/jpeg/lz/lzma/lzo/mov/ogg/png/rar/CR2/JPG/MOV"
EXCLUSIONS="--exclude lost+found --exclude .thumbnails --exclude .gvfs"
EXCLUSIONS="$EXCLUSIONS --exclude .cache --exclude Cache"
EXCLUSIONS="$EXCLUSIONS --exclude .local/share/Trash"
EXCLUSIONS="$EXCLUSIONS --exclude home/lennart/tmp/Downloads/*.iso"
EXCLUSIONS="$EXCLUSIONS --exclude home/lennart/.recycle"
EXCLUSIONS="$EXCLUSIONS --exclude _dev_dvb_adapter0_Philips_TDA10023_DVB*"
 
 
 
###############################
# The real work
###############################
 
# Create the ZFS snapshot
if [ -d $SNAPDIR ]; then
    # If the directory exists, another backup process may be running
    echo "Directory $SNAPDIR already exists! Is another backup still running?"
    exit -1
else
    # Let's make snapshots
    zfs snapshot $ZFS_POOL/$ZFS_DATASET@$ZFS_SNAPSHOT
fi
 
 
# Do the actual backup
rsync -e 'ssh' $OPTIONS $EXCLUSIONS $SRC $DEST
 
# Remove the ZFS snapshot
if [ -d $SNAPDIR ]; then
    zfs destroy $ZFS_POOL/$ZFS_DATASET@$ZFS_SNAPSHOT
else
    echo "$SNAPDIR does not exist!" 1>&2
    exit 2
fi
 
exit 0

Related Images:

Multiple accounts on an SSH server: managing key files

I’ve got several domains hosted at the same hosting company, and the company provides SSH access for each of them with a different user name, but with the same SSH server address. As I’m using key-based login to the server (see also my post here) I ran into the following problem: How do I set up my SSH config file such that it knows which key to use for which user name?

It turns out that the solution is easy (thanks Kelvin!): if you use the %r variable in the ~/.ssh/config file it contains the user name which you used when logging in. Similarly, the %h contains the host name you used on the command line. So all I needed to do was to create entries like this:

Host ssh.myhoster.com
     IdentityFile ~/.ssh/hosting-%r.key

and make sure that the corresponding key files are named hosting-domain1.key, hosting-domain2.key, etc. and then log in using a command like ssh domain1@ssh.myhoster.com.

Related Images:

Hiding columns in LaTeX export of org-mode tables

Today I was working on an Emacs org-mode document that I wanted to export to PDF. The document contained several tables and for the PDF export I wanted to hide one of the columns in the table. Of course I could have removed the column in the org source, but since I might need it in the future that wasn’t really an option.

Searching the internet I came across this e-mail discussion on the org-mode mailing list, where radio tables were suggested. I briefly tried to get that working, but it seems that this is more of an option if you are working in e.g. a LaTeX document and want to use org-style formatted tables.

So I tried another search, this time on how to hide columns in LaTeX, having the idea in mind that I could then use that to fix the org-mode export. Thanks to question on tex.stackexchange.com I came up with the followin solution:

First add the following lines at the top of the org file, after the regular org-mode header (if you have one):

#+LATEX_HEADER: \usepackage{array}
#+LATEX_HEADER: \newcolumntype{H}{>{\setbox0=\hbox\bgroup}c<{\egroup}@{}}

This defines a new column type with the name H (for ‘hidden’). Next, just before the table, simply add an #+ATTR_LATEX: attribute (see the org-mode manual):

#+ATTR_LATEX: :align lHl
| col 1 | to be hidden | col3   |
|-------+--------------+--------|
|     1 | secret       | info 1 |
|     2 | private      | info 2 |
|     3 | hidden       | info 3 |

When you export this to PDF (via C-c C-e lo) the table in the PDF only contains the first and last column.

Related Images:

SSH with several keys: fix “Too many authentication failures” error

Yesterday I created an SSH key for a new machine. Today I try to log in to a different machine, one that actually doesn’t use keys, and I got the error mentioned in the title. It turns out SSH offers all available keys by default, so I ran out of login attempts before I noticed it.

The solution is simple: Add

IdentitiesOnly yes

to your ~/.ssh/config file.

A more detailed explanation can be found here.

Related Images:

Fixing the crackling noise when turning the volume knob of an old amplifier

Last weekend I dragged an old JVC A-G90 amplifier from the attic to my study/work room. I was planning on putting it on sale, but while playing with it I decided to connect my computer to it and see whether it could replace my Logitech Z-680 (see below). A quick inventory of my computer use showed me that I didn’t really need 5.1 sound, so going to stereo wouldn’t be a big loss.

So, I hooked up the amplifier to the PC and a set of Klipsch RB-51 bookshelf speakers for a test. It was great! The only downside were the crackling noises I heard when turning the volume knob. These were one of the reasons why it ended up in the attic anyway. I seem to remember that, several years ago, I used a bit of contact spray to fix this issue. For a while. A bit of internet searching led me to the following hint: “Just turn the amplifier off and turn the knob for 20 to 100 times, this usually makes the crackling noise go away, since it is usually caused by dust in the pot meter”. And it worked!

So now I have a good-working amp and excellent sound (I found another set of Klipsch RB-51’s on sale), even at low volumes. Much better than with the Z-680. I also dug up the JVC KD-WR90 cassette player and the JVC T-E50L tuner, as well as the (not matching because too wide) JVC XL-V231 CD player, so now the old stereo set that I know from my childhood is restored in all its glory :-).

What I should do, of course, is compare this amp to the Marantz NR-1601 and SR-6006 I own as well. Just to see how they compare. Maybe something for a cold winter evening.

P.S. Speaker placement

One note about speaker placement: the speakers are very close to me (i.e. less than 1 meter). At first I placed the right speaker at ear level and the left one 15 cm lower. It seemed that the left speaker had lost most of its high-frequency spectrum. Only after raising it to the same height did I get proper balanced sound. Maybe this specific to the Klipsch speakers with their horn tweeter, maybe not. Anyway, something to keep in mind!

P.P.S. The Z-680 hiss problem

The problem with the Z-680 was that it has a low level of hiss/noise coming from the speakers. The hiss wasn’t noticeable when playing games or listening to music, but when the speakers were idle the noise was very noticeable and distracting. Even when the set was muted it was present. This turns out to be an unfixable problem (there was a partial fix with a firmware update) and several discussion threads about it can be found on the internet.

When I used the set in a home theatre setup I never noticed the hiss, but ever since I used it as my computer speaker set (i.e. sitting closer to the speakers), it annoyed me greatly. Even more so since I started working from home, most of the time sitting behind the computer without music, but still with the speakers on in order to notice e.g. Skype calls coming in.

Related Images:

Enabling external commands in the Nagios web interface

After an upgrade of one of my Ubuntu server that runs Nagios, I ran into the following error message (again…) when I tried to issue a command from the web interface:

Error: Could not stat() command file ‘/var/lib/nagios3/rw/nagios.cmd’!

This post by Barry O’Donovan shows very nicely how this problem can/should be fixed in Ubuntu. Much cleaner than chmod/chown-ing the directories myself. Thanks Barry!

Related Images:

Fixing font errors when running qmon on a remote server

Sun Grid Engine (SGE) is a bath queue system that can be used to distribute computation intensive tasks across one or more servers/CPUs. SGE has a graphical configuration utility called qmon, but when you start it on a remote machine (using SSH), you may end up with errors like this:

Warning: Cannot convert string "-adobe-courier-medium-r-*--14-*-*-*-m-*-*-*" to type FontStruct
Warning: Cannot convert string "-adobe-courier-bold-r-*--14-*-*-*-m-*-*-*" to type FontStruct
Warning: Cannot convert string "-adobe-courier-medium-r-*--12-*-*-*-m-*-*-*" to type FontStruct
X Error of failed request:  BadName (named color or font does not exist)
  Major opcode of failed request:  45 (X_OpenFont)
  Serial number of failed request:  329
  Current serial number in output stream:  340

The warnings are not really a problem, but the error is. It can be solved by running the following on the client (i.e. your local) machine (assuming it runs Debian or Ubuntu):

sudo apt-get install xfonts-75dpi
xset +fp /usr/share/fonts/X11/75dpi
xset fp rehash

Related Images:

Using ‘expect’ to distribute files among users

I’m currently teaching at the Summmer School in Statistical Omics in Split, Croatia. A great experience!

Because of the computations involved in the project work, we have access to a server. However, since the machine is part of a university cluster, I haven’t been given full root permissions (in fact, I’m only allowed to use sudo to install packages).

Now, the problem I had to solve was that I needed to distribute a certain file (.Renviron) to each student’s home directory. Normally I’d use sudo to do that, but the admin hadn’t allowed me to use cp via sudo. Furtunately, I had a list of user names and passwords for the students (because I had to distribute those), so I thought I’d use su - to change to each student’s account and copy the file, something along the lines of

echo PASSWORD | su -

and then loop over each account. Unfortunately, while testing the script I found out it wouldn’t work since su complained:

su: must be run from a terminal

Then I remembered the expect tool, which executes commands based on what it ‘sees’ on the command line. In this case I wanted it to enter the password at su‘s prompt. This is the expect script I came up with, it accepts two command line arguments, the user name and the password:

#!/usr/bin/expect -f
 
set user [lindex $argv 0]
set pass [lindex $argv 1]
 
spawn su - $user
expect "Password: "
send "$pass\r"
expect "$ "
send "cp -i /common/WORK/school/lennart/.Renviron .\r"
expect "$ "
send "ls -l .Renviron\r"
expect "$ "
send "exit\r"

The script was wrapped in the Bash script that I had already written:

#!/bin/bash
#
# This script is used to copy files from this directory to the
# home directories of the users listed in $USERFILE.
 
USERFILE=accounts.txt
SRCFILE=/common/WORK/school/lennart/.Renviron
 
while read user passw; do
    ./copy_file_to_users.expect $user $passw
done < $USERFILE

Related Images:

Changing the default mode of the Emacs scratch buffer

After starting Emacs you end up in the *scratch* buffer (assuming you’ve disabled the startup message in your .emacs file). The *scratch* can be used for writing down notes and some Lisp experiments (since it uses the Emacs Lisp major mode by default).

Now, I’m not very much of a Lisp programmer, but I do use Org-mode a lot. Consequently, I found myself changing the buffer’s major mode to org-mode regularly. And Emacs wouldn’t be Emacs if you couldn’t change this to a default. So, thanks to Bozhidar Batsov over at Emacs Redux, I’ve added the following lines to my Emacs configuration file:

;; Set the default mode of the scratch buffer to Org
(setq initial-major-mode 'org-mode)
;; and change the message accordingly
(setq initial-scratch-message "\
# This buffer is for notes you don't want to save. You can use
# org-mode markup (and all Org's goodness) to organise the notes.
# If you want to create a file, visit that file with C-x C-f,
# then enter the text in that file's own buffer.
 
")

Related Images:

« Older posts Newer posts »

© 2024 Lennart's weblog

Theme by Anders NorénUp ↑