Notes about open source software, computers, other stuff.

Author: LCK (Page 2 of 13)

A physicist on the move...

Converting a PDF file to a PNG file

Although ideally I try to create and use PDF files whenever I create something that is to be printed or even used on screen. It’s a universal, well defined open format. The added advantage is that is works well with vector graphics, something you really want for high quality material. However, sometimes external circumstances force you to deviate from your preferences. Yesterday I needed to import a PDF image into a LibreOffice Impress presentation. Unfortunately Impress can’t import PDF images. So I decided to convert my PDF image to a PNG file. My first idea was to use the convert utility:

convert file.pdf file.png

However, the default resolution was too low (and can be corrected using the -r option), but more importantly, I had some colour problems that I didn’t know how to correct.

I quickly looked around on the web and found this answer on superuser.com that pointed me to the pdftoppm tool. That worked brilliantly:

pdftoppm -r 1200 -png file.pdf > file.png

Configuring Org2blog

Yesterday I installed Org2blog, which allows me to write my blog posts in Emacs org-mode and push them to my WordPress blog from within Emacs. So far I like it a lot! One less reason to leave Emacs :-), and hopefully also a reason to blog more often. Other good things about keeping your blog posts in Emacs are:

  • You can simply export them to e.g. PDF. In my current setup it’s a easy as adding the line

    #+LATEX_CLASS: lckartcl
    

    somewhere at the top of the file (before the actual text of the post starts) to tell org-mode that it should use my personal LaTeX export style, followed by C-c C-e l o and a nicely formatted PDF of my blog post pops up.

  • You keep all your blog posts in plain text format, so if you would decide to change to a different blogging platform, uploading the old posts should be fairly easy.

Org2blog’s GitHub page mentions C-c p as prefix key for Org2blog’s functions, but in my case this prefix is already used by Projectile, and looking in Org2blog’s Customize Group I noticed that C-c M-p is an alternative prefix, so I’m using that to get the following functionality:

C-c M-p p publish buffer
C-c M-p P post buffer as page and publish
C-c M-p d post buffer as draft
C-c M-p D post buffer as page draft
C-c M-p t complete category

This is the Org2blog configuration in my .emacs file (note that I’m using John Wiegley’s use-package macro):

;;;;;;;;;;;;;;;;;;;;
;; Configure Org2blog, which allows me to write blog posts in org-mode
;; and then push them to my WordPress blog.
(use-package org2blog
  :config
  (require 'org2blog-autoloads)
  (setq org2blog/wp-blog-alist
        '(("blog.karssen.org"
           :url "https://blog.karssen.org/xmlrpc.php"
           :username "xxxxxx"
           :default-title "New blog post"
           :default-categories "Linux"
           :tags-as-categories nil)))
  )

Upgrading to Org-mode 8.3 via the package repository: fixing an error

Today I tried to upgrade Emacs Org-mode to version 8.3. I used the regular package upgrade, but got the following error:

Invalid function: org-babel-header-args-safe-fn

Unfortunately, Irreal’s advice to byte-compile ob-R.el (twice) didn’t work out for me (by the way: thanks Planet Emacsen for aggregating so many useful posts!).

Browsing through some discussions on the emacs-orgmode mailing list it seemed that the error was due to org-mode being loaded while reinstalling the package. So I did the following:

  • I started emacs without loading my personal settings: emacs -Q
  • Next I ran the following code from my .emacs file in the scratch buffer (M-x eval-region) to set up the package manager:

    (require 'package)
    (package-initialize)
    ;; Add the original Emacs Lisp Package Archive
    (add-to-list 'package-archives '("melpa" . "http://melpa.org/packages/") t)
    ;; Add the user-contributed repository
    (add-to-list 'package-archives
                 '("marmalade" . "http://marmalade-repo.org/packages/"))
    
  • And finally I used the package manager to remove and then install the latest org package.

    Now all is fine again! 🙂

    And by the way: this is my first blog post using Org2blog!

Experimenting with Google’s AdSense

I don’t think this blog has any regular readers (especially since I’m not posting very regularly), but if they would exist they would have noticed two prominent changes in the last two days:

  1. the WordPress theme for the site has changed, and
  2. this blog now has several advertisements from Google’s AdSense programme.

Obviously, the two changes are tied together. From almost the first day of this blog’s existence I had been using the Carrington theme, and although I still like it a lot (including the fact that it has two columns on the left), its appearance on mobile devices was sub-par. To fix this I looked around for a theme with “responsive design” and the current one looks quite nice both on my desktop machines and on my phone.

Changing themes had been on my list for quite some time, but the reason that I took the time to actually do it was because of my idea to play around a bit with Google’s AdSense program. Apparently, Google likes it if a site looks well on all platforms. The main reason to add ads to this blog was to experiment a bit and simply to see if this is a viable way of recouping (some of) the costs associated with hosting this blog. I’ve got a decent number of monthly views (at least I think it’s decent 🙂 and it’s definitely more than I expected when I started out) so why not give it a try. Moreover, since this is my personal, low profile site, it can also give me an idea if it’s worth having advertisements on some of the community sites that I run.

So, all in all, I think the theme change is definitely a good one, and about the ads, we’ll see. Maybe it works out, maybe it doesn’t. All in all I hope they don’t interfere too much with normal reading of the site.

Using Magit to commit only some of the changes in a file

As I discussed here, git allows you to commit only some of the changes you made to a given file. If you are working in Emacs you probably already know the wonders of Magit. In order to do the same partial committing of a file you can simply open magit-status and go to the file you’re interested in. This will highlight the changed parts of the text. With your cursor in the changed block you’d like to commit simply press s and that change will be staged. If this is all you want press c to commit and you’re done!

Source

DatABEL v0.9-6 has been published on CRAN

This morning version 0.9-6 of the DatABEL R package was published on CRAN. This is only a minor update that consists of a few small changes and one bug fix. See the official announcement for more information.

DatABEL is an R package that allows users to access files with large matrices (of several gigabytes or more in size) in a fast and efficient manner. The package is mainly used for genome-wide association analyses using e.g. ProbABEL or OmicABEL.

Git: commit only some of the changes in a file

If you only want to commit some of the changes to a file in a Git repository, use

git add --patch your_changed_file

This will interactively ask you which lines to keep:

$ git add --patch .emacs
diff --git a/.emacs b/.emacs
index d903495..5a0eb9e 100644
--- a/.emacs
+++ b/.emacs
@@ -69,9 +69,9 @@
 
 ;;; Make better buffer names when opening files with the same name
-(when (autoload 'uniquify "uniquify" "uniquify" t)
+(when (require 'uniquify nil 'noerror)
   (setq uniquify-buffer-name-style 'post-forward-angle-brackets)
-  )
+)
 
Stage this hunk [y,n,q,a,d,/,K,j,J,g,s,e,?]?

Source and more information on StackOverflow.com.

Getting the version of a remote SVN repository via SSH

A quick note to self: I wanted to find out what Subversion version was run on R-forge, which I access via SSH. This is how to do it:

$ ssh username@svn.r-forge.r-project.org svnserve --version
svnserve, version 1.6.17 (r1128011)
   compiled Nov 20 2011, 01:10:33

Copyright (C) 2000-2009 CollabNet.
Subversion is open source software, see http://subversion.apache.org/
This product includes software developed by CollabNet (http://www.Collab.Net/).

The following repository back-end (FS) modules are available:

* fs_base : Module for working with a Berkeley DB repository.
* fs_fs : Module for working with a plain file (FSFS) repository.

Cyrus SASL authentication is available.

Moving annual backups from an external disk with Ext4 to an external disk with ZFS

For a few years I have used the Christmas holidays to create a full
backup of my /home on an external hard disk. For that I used a
Bash script around rsync that uses hard links to keep the used disk
space under control. Each backup was saved in a directory named with
the date of the backup. POSIX ACLs were also backed up.

Since last year’s backup I have moved to ZFS (using ZFS on Linux
with Ubuntu 14.04
) as filesystem for /home (and others). Since ZFS
makes checksums of data and metadata it has the possibility to
detect corrupted files (and if the data is redundant it can also fix
them). This is a feature I’d like to have for my backups as
well: I’d rather know it when corruption occurs than live in
ignorance.

So the plan is to move the old backups from the external disk to the
ZFS pool in my server. and instead of using hard links I’ll transfer
the backups in order from old to new to the ZFS pool making a
snapshot for each. Additionally I will also turn on compression
(using the lz4 algorithm). Once that is done I will reformat the
external drive and create a ZFS pool called “JaarlijkseBackupPool” on
it (jaarlijks means annual in Dutch).

The old situation

In the current/old situation, this is how much disk space is used
on the external disk (with and without taking the hard links into
account):

$ sudo du -csh /mnt/JaarlijkseBackups/*
102G    /mnt/JaarlijkseBackups/2010-11-28
121G    /mnt/JaarlijkseBackups/2013-02-04
101G    /mnt/JaarlijkseBackups/2013-12-23
324G    total
$ sudo du -clsh /mnt/JaarlijkseBackups/*
102G    /mnt/JaarlijkseBackups/2010-11-28
193G    /mnt/JaarlijkseBackups/2013-02-04
255G    /mnt/JaarlijkseBackups/2013-12-23
549G    total

Copying the data from the Ext4 disk to a temporary ZFS filesystem on my server

The ZFS pool in my server is called storage. In order to save the
POSIX ACLs of the Ext4 system, they need to be enabled when
creating the ZFS filesystem as well. Setting xattr=sa means the
ACLS are stored more efficiently (although this option is not
compatible with other ZFS implementations at this time, so if I
would try to import the ZFS pool in FreeBSD for example, that
information would be inaccessible).

$ zfs create storage/JaarlijkseBackupsOrganized \
      -o compression=lz4 \
      -o acltype=posixacl \
      -o xattr=sa
$ sudo rsync -ahPAXHS --numeric-ids \
     /storage/JaarlijkseBackups/2010-11-28/ \
     /storage/JaarlijkseBackupsOrganized
$ zfs snapshot storage/JaarlijkseBackupsOrganized@2010-11-28

Followed by the same for the same rsync and zfs snapshot
commands for the other two dates.
Once that is finished, this is the status of that ZFS FS:

$ zfs list -r -t all storage/JaarlijkseBackupsOrganized
NAME                                            USED  AVAIL  REFER  MOUNTPOINT
storage/JaarlijkseBackupsOrganized              275G   438G   272G  /storage/JaarlijkseBackupsOrganized
storage/JaarlijkseBackupsOrganized@2010-11-28  1,03G      -  88,9G  -
storage/JaarlijkseBackupsOrganized@2013-02-04  2,33G      -   196G  -
storage/JaarlijkseBackupsOrganized@2013-12-23      0      -   272G  -
$ zfs get -r -t all compressratio storage/JaarlijkseBackupsOrganized
NAME                                           PROPERTY       VALUE  SOURCE
storage/JaarlijkseBackupsOrganized             compressratio  1.13x  -
storage/JaarlijkseBackupsOrganized@2010-11-28  compressratio  1.19x  -
storage/JaarlijkseBackupsOrganized@2013-02-04  compressratio  1.14x  -
storage/JaarlijkseBackupsOrganized@2013-12-23  compressratio  1.12x  -

Partitioning the external disk

The external disk is as 1TB Samsung SATA 3Gbps SpinPoint F2 EcoGreen disk
(type HD103SI, serial number: S1VSJD6ZB02657). The disk uses 512B
sectors:

sudo hdparm -I /dev/sdf |grep Sector
     Logical/Physical Sector size:           512 bytes

Before using it with ZFS, it needs to be partitioned. I used
parted:

$ parted /dev/sdf
GNU Parted 2.3
Using /dev/sdf
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) p
Model: ATA SAMSUNG HD103SI (scsi)
Disk /dev/sdf: 1000GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  1000GB  1000GB  primary  ext4

(parted) mklabel
New disk label type? gpt
(parted) u
Unit?  [compact]? MB
(parted) p
Model: ATA SAMSUNG HD103SI (scsi)
Disk /dev/sdf: 1000205MB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start  End  Size  File system  Name  Flags

(parted) mkpart
Partition name?  []? JaarlijkseBackups-HD103SI-S1VSJD6ZB02657
File system type?  [ext2]? zfs
Start? 1M
End? 1000204M
(parted) p
Model: ATA SAMSUNG HD103SI (scsi)
Disk /dev/sdf: 1000205MB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End        Size       File system  Name                                  Flags
 1      1,05MB  1000204MB  1000203MB  ext4         JaarlijkseBackups-HD103SI-S1VSJD6ZB0

(parted) q

This removes the old partition table and creates a new GPT
partition table (which allows naming partitions). Next I set the
units to MB so I can leave 1MB at the beginning and end of the
partition (can be helpful when importing this pool in
e.g. FreeBSD). The disk also shows up in /dev/disk/by=partlabel
now.

Creating the new ZFS pool

$ zpool create -o ashift=9 JaarlijkseBackupPool \
    /dev/disk/by-partlabel/JaarlijkseBackups-HD103SI-S1VSJD6ZB0
$ zpool status JaarlijkseBackupPool
  pool: JaarlijkseBackupPool
 state: ONLINE
  scan: none requested
config:

        NAME                                    STATE     READ WRITE CKSUM
        JaarlijkseBackupPool                    ONLINE       0     0     0
          JaarlijkseBackups-HD103SI-S1VSJD6ZB0  ONLINE       0     0     0

errors: No known data errors

Migrating the data

Now that the new ZFS pool and filesystem are all in place, it is
time to move the backups to the new place, starting with the oldest
backup. The -R option also make sure the attributes like
compression and xattr are transferred to the new FS. The
following commands send each snapshot to the new pool (the -n
option of zfs receive is for doing a dry run, just to show how it
works). After the first snapshot is sent, the other two are sent
using the -i option to zfs send so that only the incremental
differences between the snapshots are sent.

$ zfs send -vR storage/JaarlijkseBackupsOrganized@2010-11-28 | \
      zfs receive -Fvu JaarlijkseBackupPool/oldRsyncBackups
$ zfs send -vR -i storage/JaarlijkseBackupsOrganized@2010-11-28 \
    storage/JaarlijkseBackupsOrganized@2013-02-04 | \
    zfs receive -Fvu JaarlijkseBackupPool/oldRsyncBackups
$ zfs send -vR -i storage/JaarlijkseBackupsOrganized@2013-02-04 \
      storage/JaarlijkseBackupsOrganized@2013-12-23 | \
      zfs receive -Fvu JaarlijkseBackupPool/oldRsyncBackups -n
send from @2013-02-04 to storage/JaarlijkseBackupsOrganized@2013-12-23 estimated size is 84,3G
total estimated size is 84,3G
TIME        SENT   SNAPSHOT
would receive incremental stream of storage/JaarlijkseBackupsOrganized@2013-12-23 into JaarlijkseBackupPool@2013-12-23
14:09:16   4,22M   storage/JaarlijkseBackupsOrganized@2013-12-23
14:09:17   8,46M   storage/JaarlijkseBackupsOrganized@2013-12-23
14:09:18   18,4M   storage/JaarlijkseBackupsOrganized@2013-12-23
14:09:19   24,8M   storage/JaarlijkseBackupsOrganized@2013-12-23
^C
$ zfs send -vR -i  storage/JaarlijkseBackupsOrganized@2013-02-04 \
      storage/JaarlijkseBackupsOrganized@2013-12-23 | \
      zfs receive -Fvu JaarlijkseBackupPool/oldRsyncBackups

Add this year’s backup

At first I tried to add the new backups also to the
oldRsyncBackups FS, but that didn’t work (at least not with an
incremental backup), so I ended up making a new backup. The extra
cost in disk space is not a real problem. Disk space is rather
cheap and the current configuration will last me at least one more
year. So after creating a snapshot called 2014-12-26 of my
/home I ran:

   $ zfs send -v  storage/home@2014-12-26 | \
      zfs receive -Fu JaarlijkseBackupPool/home
$ zfs list -r -t all JaarlijkseBackupPool
NAME                                              USED  AVAIL  REFER  MOUNTPOINT
JaarlijkseBackupPool                              581G   332G    30K  /JaarlijkseBackupPool
JaarlijkseBackupPool/home                         311G   332G   311G  /JaarlijkseBackupPool/home
JaarlijkseBackupPool/home@2014-12-26             51,2M      -   311G  -
JaarlijkseBackupPool/oldRsyncBackups              271G   332G   267G  /JaarlijkseBackupPool/oldRsyncBackups
JaarlijkseBackupPool/oldRsyncBackups@2010-11-28   974M      -  87,1G  -
JaarlijkseBackupPool/oldRsyncBackups@2013-02-04  2,23G      -   193G  -
JaarlijkseBackupPool/oldRsyncBackups@2013-12-23      0      -   267G  -
$ zfs get -r compressratio JaarlijkseBackupPool
NAME                                             PROPERTY       VALUE  SOURCE
JaarlijkseBackupPool                             compressratio  1.15x  -
JaarlijkseBackupPool/home                        compressratio  1.17x  -
JaarlijkseBackupPool/home@2014-12-26             compressratio  1.17x  -
JaarlijkseBackupPool/oldRsyncBackups             compressratio  1.13x  -
JaarlijkseBackupPool/oldRsyncBackups@2010-11-28  compressratio  1.19x  -
JaarlijkseBackupPool/oldRsyncBackups@2013-02-04  compressratio  1.14x  -
JaarlijkseBackupPool/oldRsyncBackups@2013-12-23  compressratio  1.12x  -

Finishing up

In order to be able to disconnect the external drive without
damaging the filesystems use

zpool export JaarlijkseBackupPool

Later, the drive/pool can be imported using the zpool import
command.

Now that the migration is done, the intermediate filesystem
(including the snapshots) can also be removed:

zfs destroy -r storage/JaarlijkseBackupsOrganized

For reference: the old rsync script

#!/bin/sh
#
# Time-stamp: <2013-02-04 16:48:31 (root)>
# This scripts helps me create my annual backups to an external hard
# disk. The script uses rsync's hard link option to make hard links to
# the previous backups for files that haven't changed. It makes the
# backup based on an LVM snapshot it creates of the LV that contains
# the /home partition.
# This script needs to be run as root.
 
today=`date +%F`
olddate="2013-02-04"
 
srcdir="/mnt/backupsrc/"
destdir="/mnt/backupdest/JaarlijkseBackups/$today"
prevdir="/mnt/backupdest/JaarlijkseBackups/$olddate"
 
# LVM options
VG=raid5vg
LV=home
 
# rstnc options
options="-ahPAXHS --numeric-ids"
exclusions="--exclude 'lost+found/'"
#  --exclude '*/.thumbnails'"
# exclusions="$exclusions --exclude '*/.gvfs/'"
# exclusions="$exclusions --exclude '*/.cache/' --exclude '**/Cache'"
# exclusions="$exclusions --exclude '*/.recycle/'"
 
# Check to see if the previous backup directory exists
if [ ! -d $prevdir ]; then
    echo "Error: The directory with the previous back up ($prevdir) doesn't exist" 1>&2
    exit 1
fi
 
# Make a snapshot of the home LV that we can backup
lvcreate -L15G -s -n snap$LV /dev/$VG/$LV
mount /dev/$VG/snap$LV $srcdir
 
 
# Start the backup, first a dry-run, then the full one
rsynccommand="rsync $options $exclusions --link-dest=$prevdir $srcdir $destdir"
 
$rsynccommand -n
 
# Wait for user input
echo "This was a dry run. Press a key to continue with the real stuff or"
echo "hit Ctrl-c to abort."
read dummy
 
$rsynccommand
« Older posts Newer posts »

© 2022 Lennart's weblog

Theme by Anders NorenUp ↑