Lennart's weblog

Notes about open source software, computers, other stuff.

Tag: Debian (Page 2 of 3)

Converting from bzr to git

6 June 2013 / / 0 Comments

I’m in the process of moving several of my projects that used Bazaar (bzr) for revision control to Git. Converting a repository from bzr to git is very easy when using the fastimport package. In a Debian-based distribution run the following command to install the package (don’t be fooled by its name, it also contains the fastexport option):

sudo aptitude install bzr-fastimport

The go into the directory that contains your bzr repo and run:

git init
bzr fast-export `pwd` | git fast-import

You can now check a few things, e.g. running git log to see whether the change log was imported correctly. This is also the moment to move the content of your .bzrignore file to a .gitignore file.

If all is well, let’s clean up:

rm -r .bzr 
git reset HEAD

Thanks to Ron DuPlain for his post here, from which I got most of this info.

Related Images:

Solving “RTNETLINK answers: File exists” when running ifup

28 March 2013 / / 11 Comments

On a server with multiple network cards I tried to configure the eth3 interface by editing /etc/network/interfaces (this was an Ubuntu 12.04 machine).

This was the contents of /etc/networking/interfaces:

# The loopback network interface
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
        address xxx.yyy.zzz.mmm
        netmask 255.255.255.0
        gateway xxx.yyy.zzz.1
        dns-nameservers xxx.yyy.zzz.aaa xxx.yyy.zzz.bbb
        dns-search mydomain.nl

auto eth3
iface eth3 inet static
        address 192.168.4.1
        netmask 255.255.255.0
        gateway 192.168.4.1

When I tried to bring the interface up I got an error message:

$ ifup eth3
RTNETLINK answers: File exists
Failed to bring up eth3.

It took me a while to figure it out, but the problem was the gw line in the eth3 entry. Of course you can only have one default gateway in your setup. I missed this because I was also trying to add routes to networks behind the machine on the other end of eth3.
In the end, removing the gw line in the eth3 entry solved the problem.

My final /etc/networking/interfaces looks like this:

# The loopback network interface
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
        address xxx.yyy.zzz.mmm
        netmask 255.255.255.0
        gateway xxx.yyy.zzz.1
        dns-nameservers xxx.yyy.zzz.aaa xxx.yyy.zzz.bbb
        dns-search mydomain.nl

auto eth3
iface eth3 inet static
        address 192.168.4.1
        netmask 255.255.255.0
        post-up /sbin/route add -net 192.168.1.0 netmask 255.255.255.0 gw 192.168.4.250
        post-up /sbin/route add -net 192.168.2.0 netmask 255.255.255.0 gw 192.168.4.250
        post-up /sbin/route add -net 192.168.3.0 netmask 255.255.255.0 gw 192.168.4.250
        post-down /sbin/route del -net 192.168.1.0 netmask 255.255.255.0
        post-down /sbin/route del -net 192.168.2.0 netmask 255.255.255.0
        post-down /sbin/route del -net 192.168.3.0 netmask 255.255.255.0

Update 2013-08-19: Removed network entries as per Ville’s suggestion.

Related Images:

Pairing a device with a Logitech unifying receiver in Linux

8 February 2013 / / 2 Comments

My girlfriend’s keyboard and mouse stopped working some time ago. It turned out that her Logitech unifying receiver (a small USB dongle for keyboard and mouse) was a bit broken, only when twisted in a certain way it would work. So, I called Logitech, explained the situation and they offered to send us a replacement for free. Well done Logitech support!

Now, since we both use Linux as our main OS, the question was how to pair the mouse and keyboard with the new receiver. Logitech provides a piece of Windows software, but nothing for Linux. It turns out it’s not that difficult and you can find various little C programmes that do it for you. I tried Travis Reeder’s solution and it worked like a charm on my Ubuntu 12.04 machine.

These are the steps I took.
First I switched off the keybord and the mouse, then ran the following:

$ git clone https://github.com/treeder/logitech_unifier.git
Cloning into 'logitech_unifier'...
remote: Counting objects: 35, done.
remote: Compressing objects: 100% (26/26), done.
remote: Total 35 (delta 11), reused 33 (delta 9)
Unpacking objects: 100% (35/35), done.
$ cd logitech_unifier/
$ ./autopair.sh 
Logitech Unified Reciever unify binary not compiled, attemping compilation
Logitech Unified Reciever unify binary was successfully compiled
Auto-discovering Logitech Unified Reciever
Logitech Unified Reciever found on /dev/hidraw0!
Turn off the device you wish to pair and then press enter
[sudo] password for lennart: 
The receiver is ready to pair a new device.
Switch your device on to pair it.

I ran the autopair.sh script twice, once for the mouse and once for the keyboard.

Thanks Travis!

Related Images:

Comparing rsnapshot and obnam for scheduled large backups

4 January 2013 / / 4 Comments

Introduction

The home directories of the servers I administer at work total about 6.5TB of data. The home directories are stored on a file server (using ext4 partitions) and served to the other server over NFSv3 with a bonded 1Gbps LAN link.

As you all know backups are a good idea but how to implement a backup strategy for this kind of data? We decided quite early that using tapes as backup medium was out of the question. We simply can’t afford them given the amount of disk space we need. Moreover, tapes usually require operator involvement and neither me nor my colleague feels like going to the data centre every week. Our idea was to back up to another server with enough disk space in a different part of the data centre. For off-site backups we can always make an annual (maybe monthly) backup either on tape at SurfSARA/BigGrid or on a remote server.

Before implementing a given strategy several things need to be known and tested. The major questions we wanted to have an answer to were:

  1. How often do we want to backup the data? Daily snapshots? Weekly? Monthly?
  2. How many of the backups mentioned above do we want to keep? And for how long?
  3. In order to answer these questions (given a roughly fixed amount of backup space) we need to know
    • How much data changes per night/week/etc.
    • How much duplication is there in the data? How many people store the same file (or blocks, if you go for block-level deduplication)?
  4. Is NFS/network speed a limiting factor when running the backups?
  5. Can the tool preserve additional file system attributes like POSIX ACLS?

Candidates

After looking around the web and looking back at my own experiences I came up with three possible candidates. Each of them allows for backup rotation and preserves Posix ACLs (so points 1 and 5 above have been taken care of).

  1. Bacula: enterprise-level backup application that I’ve used in combination with tapes in the past. Easily supports multiple clients, tape robots, etc. No deduplication. All metadata etc. are stored in a (MySQL) database, so restoring takes some effort (and don’t forget to make a backup of the database as well!).
  2. rsnapshot: based on rsync, makes snapshots using hard links. Easy to restore, because files are simply copied to the backup medium.
  3. rdiff-backup: similar to rsnapshot, but doesn’t allow for removal of intermediate backups after a given time interval. Consequently it was the first candidate to fall of my list.
  4. Obnam: a young tool that promises block level data deduplication. Stores backed up data in its own file format. Tools for browsing those archives are not really well developed yet.

Tests

Because I already had quite some experience with Bacula but none with the other two candidates (although I use rsync a lot) I decided to start a test run with Obnam, followed by a run with rsnapshot. These are the results:

Obnam

After backing up /home completely (which took several days!), a new run, several days later took (timing by the Linux time command):

Backed up 3443706 files, uploaded 94.0 GiB in 127h48m49s at 214.2 KiB/s average speed830 files; 1.24 GiB (0 B/s)

real    7668m56.628s
user    4767m16.132s
sys     162m48.739s

From the obname log file:

2012-11-17 12:41:34 INFO VFS: baseurl=/home read=0 written=0
2012-11-21 23:09:36 INFO VFS: baseurl=/backups/backup_home read=2727031576964 written=150015706142
2012-11-21 23:09:36 INFO Backup performance statistics:
2012-11-21 23:09:36 INFO * files found: 3443706
2012-11-21 23:09:36 INFO * uploaded data: 100915247663 bytes (93.9846482715 GiB)
2012-11-21 23:09:36 INFO * duration: 460128.627629 s
2012-11-21 23:09:36 INFO * average speed: 214.179341663 KiB/s
2012-11-21 23:09:36 INFO Backup finished.
2012-11-21 23:09:36 INFO Obnam ends
2012-11-21 23:09:36 INFO obnam version 1.2 ends normally

So: ~5 days for backing up ~100 GB of changed data… Load was not high on the machines, neither in terms of CPU, nor in terms of RAM. Disk usage in /backups/backup_home was 5.7T, disk usage of /home was 6.6T, so there is some dedup, it seems.

rsnapshot

A full backup of /home to (according to the log file):

[27/Nov/2012:12:55:31] /usr/bin/rsnapshot daily: started
[27/Nov/2012:12:55:31] echo 17632 > /var/run/rsnapshot.pid
[27/Nov/2012:12:55:31] mkdir -m 0700 -p /backups/backup_home_rsnapshot/
[27/Nov/2012:12:55:31] mkdir -m 0755 -p /backups/backup_home_rsnapshot/daily.0/
[27/Nov/2012:12:55:31] /usr/bin/rsync -a --delete --numeric-ids --relative --delete-excluded /home /backups/backup_home_rsnapshot/daily.0/localhost/
[28/Nov/2012:23:16:16] touch /backups/backup_home_rsnapshot/daily.0/
[28/Nov/2012:23:16:16] rm -f /var/run/rsnapshot.pid
[28/Nov/2012:23:16:16] /usr/bin/rsnapshot daily: completed successfully

So: ~1.5 days for a full backup of 6.3TB. An incremental backup a
day later took:

[29/Nov/2012:13:10:21] /usr/bin/rsnapshot daily: started
[29/Nov/2012:13:10:21] echo 20359 > /var/run/rsnapshot.pid
[29/Nov/2012:13:10:21] mv /backups/backup_home_rsnapshot/daily.0/ /backups/backup_home_rsnapshot/daily.1/
[29/Nov/2012:13:10:21] mkdir -m 0755 -p /backups/backup_home_rsnapshot/daily.0/
[29/Nov/2012:13:10:21] /usr/bin/rsync -a –delete –numeric-ids –relative –delete-excluded –link-dest=/backups/backup_home_rsnapshot/daily.1/localhost/ /home /backups/backup_home_rsnapshot/daily.0/localhost/
[29/Nov/2012:13:25:09] touch /backups/backup_home_rsnapshot/daily.0/
[29/Nov/2012:13:25:09] rm -f /var/run/rsnapshot.pid
[29/Nov/2012:13:25:09] /usr/bin/rsnapshot daily: completed successfully

So: 15 minutes… and the changed data amounted to 21GB.

This gave me a clear winner: rsnapshot! Not only is it very fast, but given its simple way of storing data restoring a backup of any file is quickly done.

We now also have answers to our questions: Our daily changing volume is of the order of ~ 100GB, there isn’t much data that can be deduplicated. We also monitored the network usage and, depending on the server load it can be limiting, but since a daily differential backup takes only 15-30 minutes that isn’t a problem.
For a remote backup sever that was connected with a 100Mbps line we did see that the initial backup took a very long time. We should try to get a faster connection to that machine.

The future

The next challenge we face is how to back up some of the large data sets we have/produce. These include aligned BAM files of next-generation sequencing data, VCF files of the same data, results from genomic imputations (both as gzip-ed text files and as binary files in DatABEL format). This also totals several TB. Luckily these files usually don’t change on a daily basis.

Related Images:

ProbABEL v0.3.0 released

4 January 2013 / / 0 Comments

On New Year’s day I released version 0.3.0 of ProbABEL, almost two months after the previous release.

This update contains a few small bug fixes, but the most important feature of this new release is that thanks to the work of Maarten Kooyman we have a four to five-fold speed increase for the types of GWAS we run at work. In his e-mail to the GenABEL developers list he explains what he did to achieve this. The take-home-message of it is that you should always look for a suitable library for important tasks of any program you write. The old ProbABEL was based on a self-written matrix class that handled things like matrix multiplication and matrix subsetting. In the new release we make use of the Eigen C++ template library, maintained and developed by people who know much more about fast implementations of linear algebra than we do.

For those of you running Ubuntu Linux (or one of its derivatives and probably also Debian) I have set up the GenABEL PPA (personal package archive) where you can download and install the ProbABEL .deb package and stay up to date with future updates.
ProbABEL is also available for MS Windows, although we don’t have much experience running it on that platform.

Development of ProbABEL (and other members of the GenABEL suite) takes place on this R-forge page. If you are in search of an open source project to contribute to, feel free to contact us!

User support for the GenABEL suite can be found at our forum.

Related Images:

The Raspberry Pi runs ProbABEL

21 November 2012 / / 1 Comment

One of the first things I tried on my Raspberry Pi was to compile ProbABEL and see if it runs. Since the Raspberry Pi has an ARM processor I wasn’t sure whether our code was portable to it. Apparently it is! Compiling ProbABEL (r.1027 from SVN) took 30 minutes (single threaded of course) compared to 34 seconds on my Desktop (4 threads on an Intel Core i3 processor), but hey, it worked :-). Surprisingly it also passed all the checks in make check.

Once I hook up some more storage to device I will try to run ProbABEL on some real data. It will be interesting to see how much time it takes to run a linear regression on e.g. chromosome 22 of HapMap3 imputated data for a few hundred samples…

Will the Raspberry Pi be the next platform for GWAS ;-)?

Related Images:

Enter the Raspberry Pi!

21 November 2012 / / 1 Comment

Two weeks ago I received a Raspberry Pi! The Raspberry Pi is a small computer based not on a “regular” Intel or AMD x86 processor, but on an ARM processor (similar to the ones used in smartphones etc.). The one I ordered is a model B (with ethernet) and 512MB RAM.

The idea behind this nifty little computer is to provide kids with a low-cost but fully functional computer with which they could start learning more about programming. I’m not sure if this goal will be widely met, but for me it worked ;-). Having this little machine (with its case it measures roughly 10 x 6 x 2.5 cm) in my hands and installing Raspbian Linux on an SD card and looking at the terminal as it booted reminded me of the times when I first played with Slackware Linux on a 486. Of course Raspbian (well, Linux in general) is much more advanced than Slackware 7.0 back in 1999/2000 but the not too stellar performance of the graphical desktop is somewhat comparable.

Apart from playing around with it I’m not sure yet what I’m going to use it for. A domotica hub? A small web sserver? Use DosBox to play old games (from even before the 486 era)? We’ll see!

By the way, I order mine on Thursday Novermber 8th and on the Tuesday after that the package landed on my doorstep. Amazing after hearing about people waiting for months for their orders to be shipped. I order mine from New IT. It probably cost a little bit more, but who cares :-).

Related Images:

ProbABEL 0.2.2 released

21 November 2012 / / 1 Comment

On November 7th I released version 0.2.2 of ProbABEL, a set of programs that allow scientists (usually geneticists and epidemiologists) to run Genome-wide association studies (GWAS) in a fast and efficient way, even on machines with low amounts of RAM.

ProbABEL is part of the GenABEL suite, wich is a set of open source package for statistical genomics. Its main developer is Yurii Aulchenko, my former supervisor at the Erasmus Medical Centre.

This update contains a few small bug fixes and an update of the probabel.pl wrapper script that enables the use of chunked imputation output files as input. For more detailed changes, check the announcement.
For those of you running Ubuntu Linux (or one of its derivatives and probably also Debian) I have set up the GenABEL PPA (personal package archive) where you can download and install the ProbABEL .deb package and stay up to date with future updates.
ProbABEL is also available for MS Windows, although we don’t have much experience running it on that platform.

Development of ProbABEL (and other members of the GenABEL suite) takes place on this R-forge page. If you are in search of an open source project to contribute to, feel free to contact us!

User support for the GenABEL suite can be found at our forum.

Related Images:

Enabling middle mouse button emulation in Ubuntu 12.10

3 November 2012 / / 0 Comments

A few days ago I installed Ubuntu 12.10 on a 2009 laptop (a Dell Latitude D820). This laptop has a trackpoint (the little nib between the G, B and H keys on the keyboard that functions as a mouse). The trackpoint has its own set of mouse buttons, but, unlike my present Thinkpad, it doesn’t have a middle mouse button.
From my early Linux years I remembered that you could use a simultaneous click on the left and right buttons of a mouse to emulate the click of a middle mouse button. This option had to be set in the Xorg configuration file. My guess was that by now this was no longer needed, but I couldn’t find the appropriate option anywhere in Ubuntu’s settings. It’s quite a shame to see that Gnome/Unity/others(?) remove settings like that.
Anyway, looking around the web I found the solution. The middle mouse butten emulation can be enabled in the Gnome configuration by running the following in a terminal:

gsettings set org.gnome.settings-daemon.peripherals.mouse middle-button-enabled true

It worked immediately!

As an aside, I also had to install the packages nvidia-current and nvidia-settings in order to use the nVidia drivers. Without them using the Unity interface was dead slow, even though the laptop has an nVidia Geforce Go 7400.

Related Images:

Ubuntu 12.04: LaTeX siunitx package and the correct font in beamer

20 May 2012 / / 0 Comments

When writing scientific text I normally use the siunitx package to typeset numbers (using \num{}) and numbers with units (e.g. \SI{4.0}{TB}) in LaTeX.
Something that bothered me for a while was the fact that when making presentations in Beamer, numbers and units would be typeset in a serif font, whereas beamer uses sans-serif fonts.
The solution is actually documented in the manual, but what to me a while to figure out was that in Ubuntu 12.04 (and therefore in Debian as well, I presume) the version of the siunitx package is lagging compared to what is available on CTAN. Ubuntu 12.04 still has version 1.3a of the siunitx package, which was released in September 2009. For this version the solution is to add the following to the preamble of your document:

\sisetup{obeyfamily=false,mathrm=mathsf,textrm=sffamily}

Related Images:

« Older posts Newer posts »

Archives

© 2024 Lennart's weblog

Theme by Anders Norén — Up ↑