As I discussed here, git allows you to commit only some of the changes you made to a given file. If you are working in Emacs you probably already know the wonders of Magit. In order to do the same partial committing of a file you can simply open magit-status and go to the file you’re interested in. This will highlight the changed parts of the text. With your cursor in the changed block you’d like to commit simply press
Page 3 of 14
This morning version 0.9-6 of the DatABEL R package was published on CRAN. This is only a minor update that consists of a few small changes and one bug fix. See the official announcement for more information.
DatABEL is an R package that allows users to access files with large matrices (of several gigabytes or more in size) in a fast and efficient manner. The package is mainly used for genome-wide association analyses using e.g. ProbABEL or OmicABEL.
Related Images:
The New Yorker has a nice article about the GNU manifesto, which turned thirty earlier this month.
It nicely summarises what lead RMS to publish the manifesto and start the Free Software Foundation and also briefly explains the difference between Free Software and Open Source software.
Related Images:
If you only want to commit some of the changes to a file in a Git repository, use
git add --patch your_changed_file |
This will interactively ask you which lines to keep:
$ git add --patch .emacs diff --git a/.emacs b/.emacs index d903495..5a0eb9e 100644 --- a/.emacs +++ b/.emacs @@ -69,9 +69,9 @@ ;;; Make better buffer names when opening files with the same name -(when (autoload 'uniquify "uniquify" "uniquify" t) +(when (require 'uniquify nil 'noerror) (setq uniquify-buffer-name-style 'post-forward-angle-brackets) - ) +) Stage this hunk [y,n,q,a,d,/,K,j,J,g,s,e,?]? |
Source and more information on StackOverflow.com.
Related Images:
A quick note to self: I wanted to find out what Subversion version was run on R-forge, which I access via SSH. This is how to do it:
$ ssh username@svn.r-forge.r-project.org svnserve --version svnserve, version 1.6.17 (r1128011) compiled Nov 20 2011, 01:10:33 Copyright (C) 2000-2009 CollabNet. Subversion is open source software, see http://subversion.apache.org/ This product includes software developed by CollabNet (http://www.Collab.Net/). The following repository back-end (FS) modules are available: * fs_base : Module for working with a Berkeley DB repository. * fs_fs : Module for working with a plain file (FSFS) repository. Cyrus SASL authentication is available.
Related Images:
For a few years I have used the Christmas holidays to create a full
backup of my /home on an external hard disk. For that I used a
Bash script around rsync that uses hard links to keep the used disk
space under control. Each backup was saved in a directory named with
the date of the backup. POSIX ACLs were also backed up.
Since last year’s backup I have moved to ZFS (using ZFS on Linux
with Ubuntu 14.04) as filesystem for /home (and others). Since ZFS
makes checksums of data and metadata it has the possibility to
detect corrupted files (and if the data is redundant it can also fix
them). This is a feature I’d like to have for my backups as
well: I’d rather know it when corruption occurs than live in
ignorance.
So the plan is to move the old backups from the external disk to the
ZFS pool in my server. and instead of using hard links I’ll transfer
the backups in order from old to new to the ZFS pool making a
snapshot for each. Additionally I will also turn on compression
(using the lz4 algorithm). Once that is done I will reformat the
external drive and create a ZFS pool called “JaarlijkseBackupPool” on
it (jaarlijks means annual in Dutch).
The old situation
In the current/old situation, this is how much disk space is used
on the external disk (with and without taking the hard links into
account):
$ sudo du -csh /mnt/JaarlijkseBackups/* 102G /mnt/JaarlijkseBackups/2010-11-28 121G /mnt/JaarlijkseBackups/2013-02-04 101G /mnt/JaarlijkseBackups/2013-12-23 324G total $ sudo du -clsh /mnt/JaarlijkseBackups/* 102G /mnt/JaarlijkseBackups/2010-11-28 193G /mnt/JaarlijkseBackups/2013-02-04 255G /mnt/JaarlijkseBackups/2013-12-23 549G total
Copying the data from the Ext4 disk to a temporary ZFS filesystem on my server
The ZFS pool in my server is called storage. In order to save the
POSIX ACLs of the Ext4 system, they need to be enabled when
creating the ZFS filesystem as well. Setting xattr=sa means the
ACLS are stored more efficiently (although this option is not
compatible with other ZFS implementations at this time, so if I
would try to import the ZFS pool in FreeBSD for example, that
information would be inaccessible).
$ zfs create storage/JaarlijkseBackupsOrganized \
-o compression=lz4 \
-o acltype=posixacl \
-o xattr=sa
$ sudo rsync -ahPAXHS --numeric-ids \
/storage/JaarlijkseBackups/2010-11-28/ \
/storage/JaarlijkseBackupsOrganized
$ zfs snapshot storage/JaarlijkseBackupsOrganized@2010-11-28
Followed by the same for the same rsync and zfs snapshot
commands for the other two dates.
Once that is finished, this is the status of that ZFS FS:
$ zfs list -r -t all storage/JaarlijkseBackupsOrganized NAME USED AVAIL REFER MOUNTPOINT storage/JaarlijkseBackupsOrganized 275G 438G 272G /storage/JaarlijkseBackupsOrganized storage/JaarlijkseBackupsOrganized@2010-11-28 1,03G - 88,9G - storage/JaarlijkseBackupsOrganized@2013-02-04 2,33G - 196G - storage/JaarlijkseBackupsOrganized@2013-12-23 0 - 272G - $ zfs get -r -t all compressratio storage/JaarlijkseBackupsOrganized NAME PROPERTY VALUE SOURCE storage/JaarlijkseBackupsOrganized compressratio 1.13x - storage/JaarlijkseBackupsOrganized@2010-11-28 compressratio 1.19x - storage/JaarlijkseBackupsOrganized@2013-02-04 compressratio 1.14x - storage/JaarlijkseBackupsOrganized@2013-12-23 compressratio 1.12x -
Partitioning the external disk
The external disk is as 1TB Samsung SATA 3Gbps SpinPoint F2 EcoGreen disk
(type HD103SI, serial number: S1VSJD6ZB02657). The disk uses 512B
sectors:
sudo hdparm -I /dev/sdf |grep Sector
Logical/Physical Sector size: 512 bytes
Before using it with ZFS, it needs to be partitioned. I used
parted:
$ parted /dev/sdf GNU Parted 2.3 Using /dev/sdf Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) p Model: ATA SAMSUNG HD103SI (scsi) Disk /dev/sdf: 1000GB Sector size (logical/physical): 512B/512B Partition Table: msdos Number Start End Size Type File system Flags 1 1049kB 1000GB 1000GB primary ext4 (parted) mklabel New disk label type? gpt (parted) u Unit? [compact]? MB (parted) p Model: ATA SAMSUNG HD103SI (scsi) Disk /dev/sdf: 1000205MB Sector size (logical/physical): 512B/512B Partition Table: gpt Number Start End Size File system Name Flags (parted) mkpart Partition name? []? JaarlijkseBackups-HD103SI-S1VSJD6ZB02657 File system type? [ext2]? zfs Start? 1M End? 1000204M (parted) p Model: ATA SAMSUNG HD103SI (scsi) Disk /dev/sdf: 1000205MB Sector size (logical/physical): 512B/512B Partition Table: gpt Number Start End Size File system Name Flags 1 1,05MB 1000204MB 1000203MB ext4 JaarlijkseBackups-HD103SI-S1VSJD6ZB0 (parted) q
This removes the old partition table and creates a new GPT
partition table (which allows naming partitions). Next I set the
units to MB so I can leave 1MB at the beginning and end of the
partition (can be helpful when importing this pool in
e.g. FreeBSD). The disk also shows up in /dev/disk/by=partlabel
now.
Creating the new ZFS pool
$ zpool create -o ashift=9 JaarlijkseBackupPool \
/dev/disk/by-partlabel/JaarlijkseBackups-HD103SI-S1VSJD6ZB0
$ zpool status JaarlijkseBackupPool
pool: JaarlijkseBackupPool
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
JaarlijkseBackupPool ONLINE 0 0 0
JaarlijkseBackups-HD103SI-S1VSJD6ZB0 ONLINE 0 0 0
errors: No known data errors
Migrating the data
Now that the new ZFS pool and filesystem are all in place, it is
time to move the backups to the new place, starting with the oldest
backup. The -R option also make sure the attributes like
compression and xattr are transferred to the new FS. The
following commands send each snapshot to the new pool (the -n
option of zfs receive is for doing a dry run, just to show how it
works). After the first snapshot is sent, the other two are sent
using the -i option to zfs send so that only the incremental
differences between the snapshots are sent.
$ zfs send -vR storage/JaarlijkseBackupsOrganized@2010-11-28 | \
zfs receive -Fvu JaarlijkseBackupPool/oldRsyncBackups
$ zfs send -vR -i storage/JaarlijkseBackupsOrganized@2010-11-28 \
storage/JaarlijkseBackupsOrganized@2013-02-04 | \
zfs receive -Fvu JaarlijkseBackupPool/oldRsyncBackups
$ zfs send -vR -i storage/JaarlijkseBackupsOrganized@2013-02-04 \
storage/JaarlijkseBackupsOrganized@2013-12-23 | \
zfs receive -Fvu JaarlijkseBackupPool/oldRsyncBackups -n
send from @2013-02-04 to storage/JaarlijkseBackupsOrganized@2013-12-23 estimated size is 84,3G
total estimated size is 84,3G
TIME SENT SNAPSHOT
would receive incremental stream of storage/JaarlijkseBackupsOrganized@2013-12-23 into JaarlijkseBackupPool@2013-12-23
14:09:16 4,22M storage/JaarlijkseBackupsOrganized@2013-12-23
14:09:17 8,46M storage/JaarlijkseBackupsOrganized@2013-12-23
14:09:18 18,4M storage/JaarlijkseBackupsOrganized@2013-12-23
14:09:19 24,8M storage/JaarlijkseBackupsOrganized@2013-12-23
^C
$ zfs send -vR -i storage/JaarlijkseBackupsOrganized@2013-02-04 \
storage/JaarlijkseBackupsOrganized@2013-12-23 | \
zfs receive -Fvu JaarlijkseBackupPool/oldRsyncBackups
Add this year’s backup
At first I tried to add the new backups also to the
oldRsyncBackups FS, but that didn’t work (at least not with an
incremental backup), so I ended up making a new backup. The extra
cost in disk space is not a real problem. Disk space is rather
cheap and the current configuration will last me at least one more
year. So after creating a snapshot called 2014-12-26 of my
/home I ran:
$ zfs send -v storage/home@2014-12-26 | \
zfs receive -Fu JaarlijkseBackupPool/home
$ zfs list -r -t all JaarlijkseBackupPool
NAME USED AVAIL REFER MOUNTPOINT
JaarlijkseBackupPool 581G 332G 30K /JaarlijkseBackupPool
JaarlijkseBackupPool/home 311G 332G 311G /JaarlijkseBackupPool/home
JaarlijkseBackupPool/home@2014-12-26 51,2M - 311G -
JaarlijkseBackupPool/oldRsyncBackups 271G 332G 267G /JaarlijkseBackupPool/oldRsyncBackups
JaarlijkseBackupPool/oldRsyncBackups@2010-11-28 974M - 87,1G -
JaarlijkseBackupPool/oldRsyncBackups@2013-02-04 2,23G - 193G -
JaarlijkseBackupPool/oldRsyncBackups@2013-12-23 0 - 267G -
$ zfs get -r compressratio JaarlijkseBackupPool
NAME PROPERTY VALUE SOURCE
JaarlijkseBackupPool compressratio 1.15x -
JaarlijkseBackupPool/home compressratio 1.17x -
JaarlijkseBackupPool/home@2014-12-26 compressratio 1.17x -
JaarlijkseBackupPool/oldRsyncBackups compressratio 1.13x -
JaarlijkseBackupPool/oldRsyncBackups@2010-11-28 compressratio 1.19x -
JaarlijkseBackupPool/oldRsyncBackups@2013-02-04 compressratio 1.14x -
JaarlijkseBackupPool/oldRsyncBackups@2013-12-23 compressratio 1.12x -
Finishing up
In order to be able to disconnect the external drive without
damaging the filesystems use
zpool export JaarlijkseBackupPool
Later, the drive/pool can be imported using the zpool import
command.
Now that the migration is done, the intermediate filesystem
(including the snapshots) can also be removed:
zfs destroy -r storage/JaarlijkseBackupsOrganized
For reference: the old rsync script
#!/bin/sh # # Time-stamp: <2013-02-04 16:48:31 (root)> # This scripts helps me create my annual backups to an external hard # disk. The script uses rsync's hard link option to make hard links to # the previous backups for files that haven't changed. It makes the # backup based on an LVM snapshot it creates of the LV that contains # the /home partition. # This script needs to be run as root. today=`date +%F` olddate="2013-02-04" srcdir="/mnt/backupsrc/" destdir="/mnt/backupdest/JaarlijkseBackups/$today" prevdir="/mnt/backupdest/JaarlijkseBackups/$olddate" # LVM options VG=raid5vg LV=home # rstnc options options="-ahPAXHS --numeric-ids" exclusions="--exclude 'lost+found/'" # --exclude '*/.thumbnails'" # exclusions="$exclusions --exclude '*/.gvfs/'" # exclusions="$exclusions --exclude '*/.cache/' --exclude '**/Cache'" # exclusions="$exclusions --exclude '*/.recycle/'" # Check to see if the previous backup directory exists if [ ! -d $prevdir ]; then echo "Error: The directory with the previous back up ($prevdir) doesn't exist" 1>&2 exit 1 fi # Make a snapshot of the home LV that we can backup lvcreate -L15G -s -n snap$LV /dev/$VG/$LV mount /dev/$VG/snap$LV $srcdir # Start the backup, first a dry-run, then the full one rsynccommand="rsync $options $exclusions --link-dest=$prevdir $srcdir $destdir" $rsynccommand -n # Wait for user input echo "This was a dry run. Press a key to continue with the real stuff or" echo "hit Ctrl-c to abort." read dummy $rsynccommand |
Related Images:
It was quite a long time in the making and then a bunch of other stuff came in between, but I finally managed to release v0.4.4 of ProbABEL!
ProbABEL is a toolset for doing fast, memory (RAM) efficient genome-wide regression tests.
This is a bugfix release, but a major one for those who use the Cox proportional hazards regression module. Thanks to some of our users on the GenABEL forum, a serious bug leading to way to many NaN’s in the output was discovered, fixed and tested. This is one of the best examples of community collaboration I have seen in the GenABEL project.
Another bug fixed in this release is one that caused a failed install on MacOS X and FreeBSD. Again a bug reported on the forum by one of our users. Great work!
Uploads to Debian and the Ubuntu PPA are coming ASAP.
Now, let’s get ready for a new feature release, which will include p-value calculation (a long-standing feature request) and major speed-ups (implemented by former colleague Maarten Kooyman). Time to get to work ;-)!
Related Images:
Some time ago I moved from using LVM to using ZFS on my home server. This meant I also had to change the backup script I used to make backups on a remote Synology Diskstation. Below is the updated script. I also updated it such that it now needs a single command line argument: the hostname of the Diskstation to backup to (because I now have two Diskstations at different locations). If you want to run this script from cron you should set up key-based SSH login (see also here and here).
#!/bin/bash # # This script makes a backup of my home dirs to a Synology DiskStation at # another location. I use ZFS for my /home, so I make a snapshot first and # backup from there. # # This script requires that the first command line argument is the # host name of the remote backup server (the Synology NAS). It also # assumes that the location of the backups is the same on each # remote backup server. # # Time-stamp: <2014-10-27 11:35:39 (L.C. Karssen)> # This script it licensed under the GNU GPLv3. set -u if [ ${#} -lt 1 ]; then echo -n "ERROR: Please specify a host name as first command" 1>&2 echo " line option" 1>&2 exit -1 fi ############################### # Some settings ############################### # Options for the remote (Synology) backup destination DESTHOST=$1 DESTUSER=root DESTPATH=/volume1/Backups/ DEST=${DESTUSER}@${DESTHOST}:${DESTPATH} # Options for the client (the data to be backed up) # ZFS options ZFS_POOL=storage ZFS_DATASET=home ZFS_SNAPSHOT=rsync_snapshot SNAPDIR="/home/.zfs/snapshot/$ZFS_SNAPSHOT" # Backup source path. Don't forget to have trailing / otherwise # rsync's --delete option won't work SRC=${SNAPDIR}/ # rsync options OPTIONS="--delete -azvhHSP --numeric-ids --stats" OPTIONS="$OPTIONS --timeout=60 --delete-excluded" OPTIONS="$OPTIONS --skip-compress=gz/jpg/mp[34]/7z/bz2/ace/avi/deb/gpg/iso/jpeg/lz/lzma/lzo/mov/ogg/png/rar/CR2/JPG/MOV" EXCLUSIONS="--exclude lost+found --exclude .thumbnails --exclude .gvfs" EXCLUSIONS="$EXCLUSIONS --exclude .cache --exclude Cache" EXCLUSIONS="$EXCLUSIONS --exclude .local/share/Trash" EXCLUSIONS="$EXCLUSIONS --exclude home/lennart/tmp/Downloads/*.iso" EXCLUSIONS="$EXCLUSIONS --exclude home/lennart/.recycle" EXCLUSIONS="$EXCLUSIONS --exclude _dev_dvb_adapter0_Philips_TDA10023_DVB*" ############################### # The real work ############################### # Create the ZFS snapshot if [ -d $SNAPDIR ]; then # If the directory exists, another backup process may be running echo "Directory $SNAPDIR already exists! Is another backup still running?" exit -1 else # Let's make snapshots zfs snapshot $ZFS_POOL/$ZFS_DATASET@$ZFS_SNAPSHOT fi # Do the actual backup rsync -e 'ssh' $OPTIONS $EXCLUSIONS $SRC $DEST # Remove the ZFS snapshot if [ -d $SNAPDIR ]; then zfs destroy $ZFS_POOL/$ZFS_DATASET@$ZFS_SNAPSHOT else echo "$SNAPDIR does not exist!" 1>&2 exit 2 fi exit 0 |
Related Images:
I’ve got several domains hosted at the same hosting company, and the company provides SSH access for each of them with a different user name, but with the same SSH server address. As I’m using key-based login to the server (see also my post here) I ran into the following problem: How do I set up my SSH config file such that it knows which key to use for which user name?
It turns out that the solution is easy (thanks Kelvin!): if you use the %r variable in the ~/.ssh/config file it contains the user name which you used when logging in. Similarly, the %h contains the host name you used on the command line. So all I needed to do was to create entries like this:
Host ssh.myhoster.com
IdentityFile ~/.ssh/hosting-%r.key
and make sure that the corresponding key files are named hosting-domain1.key, hosting-domain2.key, etc. and then log in using a command like ssh domain1@ssh.myhoster.com.
Related Images:
Today I was working on an Emacs org-mode document that I wanted to export to PDF. The document contained several tables and for the PDF export I wanted to hide one of the columns in the table. Of course I could have removed the column in the org source, but since I might need it in the future that wasn’t really an option.
Searching the internet I came across this e-mail discussion on the org-mode mailing list, where radio tables were suggested. I briefly tried to get that working, but it seems that this is more of an option if you are working in e.g. a LaTeX document and want to use org-style formatted tables.
So I tried another search, this time on how to hide columns in LaTeX, having the idea in mind that I could then use that to fix the org-mode export. Thanks to question on tex.stackexchange.com I came up with the followin solution:
First add the following lines at the top of the org file, after the regular org-mode header (if you have one):
#+LATEX_HEADER: \usepackage{array}
#+LATEX_HEADER: \newcolumntype{H}{>{\setbox0=\hbox\bgroup}c<{\egroup}@{}} |
This defines a new column type with the name H (for ‘hidden’). Next, just before the table, simply add an #+ATTR_LATEX: attribute (see the org-mode manual):
#+ATTR_LATEX: :align lHl | col 1 | to be hidden | col3 | |-------+--------------+--------| | 1 | secret | info 1 | | 2 | private | info 2 | | 3 | hidden | info 3 | |
When you export this to PDF (via C-c C-e lo) the table in the PDF only contains the first and last column.