The package contains the following:

• a USB adapter (called the Stick)
• a Circle+, the master adaptor that keeps track of the other devices in the network
• a Circle, the regular members of the Plugwise network

The Circles communicate to each other using the ZigBee protocol in the 2.4GHz range. According to the documentation, the range of each Circle is about 5m.

Unfortunately the Source software only runs on windows. Luckily some people have already analysed the protocol and written some software to control the Plugwise devices (see links below).

First steps

Plugging the USB dongle in gives the following output in /var/log/syslog:

Nov 19 12:20:37 barabas kernel: [  182.855742] usb 1-1.6.1.1.3: new full speed USB device number 14 using ehci_hcd
Nov 19 12:20:37 barabas mtp-probe: checking bus 1, device 14: "/sys/devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.6/1-1.6.1/1-1.6.1.1/1-1.6.1.1.3"
Nov 19 12:20:37 barabas mtp-probe: bus: 1, device: 14 was not an MTP device
Nov 19 12:20:37 barabas kernel: [  183.169370] usbcore: registered new interface driver usbserial
Nov 19 12:20:37 barabas kernel: [  183.169389] USB Serial support registered for generic
Nov 19 12:20:37 barabas kernel: [  183.169431] usbcore: registered new interface driver usbserial_generic
Nov 19 12:20:37 barabas kernel: [  183.169434] usbserial: USB Serial Driver core
Nov 19 12:20:37 barabas kernel: [  183.171310] USB Serial support registered for FTDI USB Serial Device
Nov 19 12:20:37 barabas kernel: [  183.171552] ftdi_sio 1-1.6.1.1.3:1.0: FTDI USB Serial Device converter detected
Nov 19 12:20:37 barabas kernel: [  183.171588] usb 1-1.6.1.1.3: Detected FT232RL
Nov 19 12:20:37 barabas kernel: [  183.171591] usb 1-1.6.1.1.3: Number of endpoints 2
Nov 19 12:20:37 barabas kernel: [  183.171595] usb 1-1.6.1.1.3: Endpoint 1 MaxPacketSize 64
Nov 19 12:20:37 barabas kernel: [  183.171598] usb 1-1.6.1.1.3: Endpoint 2 MaxPacketSize 64
Nov 19 12:20:37 barabas kernel: [  183.171602] usb 1-1.6.1.1.3: Setting MaxPacketSize 64
Nov 19 12:20:37 barabas kernel: [  183.171975] usb 1-1.6.1.1.3: FTDI USB Serial Device converter now attached to ttyUSB0
Nov 19 12:20:37 barabas kernel: [  183.171998] usbcore: registered new interface driver ftdi_sio
Nov 19 12:20:37 barabas kernel: [  183.172002] ftdi_sio: v1.6.0:USB FTDI Serial Converters Driver
Nov 19 12:20:37 barabas modem-manager[901]: <info>  (ttyUSB0) opening serial port...
Nov 19 12:20:49 barabas modem-manager[901]: <info>  (ttyUSB0) closing serial port...
Nov 19 12:20:49 barabas modem-manager[901]: <info>  (ttyUSB0) serial port closed
Nov 19 12:20:49 barabas modem-manager[901]: <info>  (ttyUSB0) opening serial port...
Nov 19 12:20:55 barabas modem-manager[901]: <info>  (ttyUSB0) closing serial port...
Nov 19 12:20:55 barabas modem-manager[901]: <info>  (ttyUSB0) serial port closed

lsusb gives:

Bus 001 Device 014: ID 0403:6001 Future Technology Devices International, Ltd FT232 USB-Serial (UART) IC

I couldn’t get the pairing to work under Linux (with the PlugwiseOnLinux scripts), Even though I corrected the MAC address in the python code. I then tried it in Windows where I failed at first also. After resetting the Circle+ and the Circle (removing/inserting them into the power outlet with 3 second intervals, as mentioned in the FAW on the plugwise website) I managed to pair the Circles. Looking back, I think I didn’t wait long enough for the pairing to work under Linux. During the trials in Windows I noticed that the pairing can take up to about 5 minutes…

Back in Linux I used python-plugwise (see links below) to turn the Circles on and off, e.g. this is how I turn my Circle+ off (note that I am a member of the dialout group, which is needed to communicate with /dev/ttyUSB0):

$ plugwise_util -d /dev/ttyUSB0 -m 000D6F0000B1C117 -s off

This is what I want! The only minor downside of python-plugwise is that it depends on the crcmod python library, which apparently is not package for Debian/Ubuntu. So installing using the python-setup framework as mentioned in the README is necessary.

Reading out the current power usage of my Circle works also:

$ plugwise_util -d /dev/ttyUSB0 -m 000D6F0000B85134 -p
power usage: 2.27W

So, no that it works, what am I going to do with the Plugwise modules? I’m going to use them in my backup scripts to switch the power to my external hard drives.

Making a .deb

I used checkinstall to make a package of python-plugwise. In a working directory, first check out the source code of python-plugwise using mercurial, as mentioned on the web site:

$ hg clone https://bitbucket.org/hadara/python-plugwise

Then run checkinstall and don’t forget to fill in the details correctly. For example, the package name is ‘python’ by default, which you definitely don’t want, since that would overwrite Ubuntu’s default ‘python’ package. Also make sure that you remove the crcmod python library if you installed python-plugwise before, otherwise it won’t get packaged. The output below shows the final values, after I changed them.

$ sudo checkinstall -D python setup.py install
 
checkinstall 1.6.2, Copyright 2009 Felipe Eduardo Sanchez Diaz Duran
	      This software is released under the GNU GPL.
 
 
 
*****************************************
**** Debian package creation selected ***
*****************************************
 
This package will be built according to these values:
 
0 -  Maintainer: [ lennart@karssen.org ]
1 -  Summary: [ python-plugwise is used to control the Plugwise power switches as well as read out information on power usage. ]
2 -  Name:    [ python-plugwise ]
3 -  Version: [ 0.2-hg-20111120 ]
4 -  Release: [ 1 ]
5 -  License: [ GPL ]
6 -  Group:   [ checkinstall ]
7 -  Architecture: [ amd64 ]
8 -  Source location: [ python-plugwise ]
9 -  Alternate source location: [  ]
10 - Requires: [ python ]
11 - Provides: [ python-plugwise ]
12 - Conflicts: [  ]
13 - Replaces: [  ]
 
Enter a number to change any of them or press ENTER to continue:

You can check the contents of the package to make sure the crcmod
files are included using dpkg:

$ dpkg --contents python-plugwise_0.2-hg-20111120-1_amd64.deb

An idea for later: make an SNMP module that calls plugwise_util to get the power usage so that I can monitor the power usage of a device using Cacti.

Links

• The software I use: python-plugwise
• The blog of Maarten Damen, the guy that seems to have been the first one to analyse the plugwise protocol
• Another blog, also with info on the protocol
• PlugwiseOnLinux, a set of python scripts for pairing Plugwise and the Circles. Did not work with me at first, didn’t retry later
• PlugWeb, a PHP interface, for those wanting to control their Plugwise network via a web interface
• According to its website DomotiGa can also control Plugwise devices, but I haven’t tried it (yet)

Loggerhead is a webfrontend for Bazaar (usually abbreviated as bzr) repositories. Bazaar is a so-called distributed version control system. So, if you have one or more bzr repositories you can use Loggerhead to look at the files, read the change logs and see the differences between revisions from within your web browser.

The main purpose of this post is to document the steps needed to configure Loggerhead and Apache to work together to publish your bzr repos on the web. The need for this post arose when I tried to get this setup to work and found that there isn’t a lot of documentation on how to get this done and most of it is out of date. The folowing steps were performed on a Linux server with Ubuntu 11.04 installed.

Basic Loggerhead configuration

First, let’s install Loggerhead:

$ aptitude install loggerhead

Although the package is called loggerhead, the actual binary that is run is called serve-branches. The package provides start and stop scripts for the service (/etc/init.d/loggerhead), but to start successfully the file /etc/serve-branches.conf needs to exist. Older documentation I found on the web refers to the file /etc/loggerhead.conf, but that file has become obsolete.

The serve-branches.conf file contains three lines:

served_branches=/home/bzr
prefix=
port=8080

Here, the line served_branches points to the directory under which you store your bzr repositories. Each repo needs to be stored in its own directory. So in this example all the repos are in subdirectories of /home/bzr/.

You have to make sure that loggerhead can read the files in that directory. Loggerhead runs as the loggerhead user but I made the directories readable and accessible by all users:

$ chmod -R a+rx /home/bzr/

If you now start Loggerhead:

$ service start loggerhead

you should be able to visit http://localhost:8080 in your browser and see your repositories.

How to publish your branch to this shared repository?

Now that our repository browser is set up, how do we publish our branches to it so that there actually is something to browse through? Here is how you publish your branch to the server, assuming that you are in a directory that contains a branch and want to publish it as myTests:

$ bzr push --create-prefix sftp://username@server.yourdomain.com/home/bzr/myTests

As you probably suspected, the --create-prefix option is only necessary the first time you push your branch. Note that we are using sftp here. Loggerhead itself doesn’t allow writes to the published repos. So, every user that want to push his/her changes to this repository needs to have sftp access to the /home/bzr directory. I solved that problem by adding all people that need to be able to push changes to a Linux group called vcs (for Version Control Systems) and then set the primary group of /home/bzr/ to vcs as well as giving group write permissions to this directory:

$ ls -ld /home/bzr/
drwxrwxr-x 4 root vcs 4096 2011-08-16 23:10 /home/bzr/

Adding Apache to the mix

In my case I already have a web server (Apache) running on port 80. Since I’d rather not open yet another port (8080 in this case) on my router, I wanted to use Apache to hand over the requests for bzr page to Loggerhead. For that I needed to install the following packages:

$ aptitude install python-pastedeploy

Next, I needed to change the contents of the /etc/serve-branches.conf file to this:

served_branches=/home/bzr
prefix=/bzr
port=8080

The prefix indicates the location in the URL where Apache will serve the repos. In this case that will be http://server.yourdomain.com/bzr/.

And finally I needed to configure Apache. First, make sure that the proxy and proxy-http modules are loaded:

$ a2enmod proxy proxy_http

Next, create a file /etc/apache/conf.d/sites-available/loggerhead with the following contents:

# Configuration for browsing of Bazaar repos. Make sure loggerhead is running.
<Location "/bzr/">
    ProxyPass http://127.0.0.1:8080/
    ProxyPassReverse http://127.0.0.1:8080/
</Location>

Note that Loggerhead and Apache run on the same host, that’s why I set the IP to 127.0.0.1.

Finally it’s time to enable the site and restart Apache:

$ a2ensite loggerhead
$ service apache2 restart

Now it should be possible to browse your repos at http://server.yourdomain.com/bzr/. Note the final /, it’s important.

Securing access with an LDAP connection

I have stored all my Unix user and group information in an LDAP server. To make sure that only people in the Unix group vcs are allowed access to the loggerhead pages, change the Apache configuration file loggerhead to the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

# Configuration for browsing of Bazaar repos. Make sure loggerhead is running.
<Location "/bzr/">
    ProxyPass http://127.0.0.1:8080/
    ProxyPassReverse http://127.0.0.1:8080/
 
    # LDAP authentication
    AuthType Basic
    AuthName "Karssen.org VCS users"
    AuthBasicProvider ldap
    AuthLDAPURL "ldap://ldap.yourdomain.com/ou=Users,dc=yourdomain,dc=com?uid"
    AuthLDAPGroupAttribute memberUid
    AuthLDAPGroupAttributeIsDN off
    Order Allow,Deny
    Allow From All
    Require ldap-group cn=vcs,ou=Groups,dc=yourdomain,dc=com
</Location>

Lines 11 and 12 are needed because the vcs group is not an LDAP group. I store my Unix (POSIX) groups in a separate OU in the LDAP tree (see line 15).
Don’t forget to restart Apache after making these changes.

References

• Bazaar documentation on Apache integration
• Launchpad bug 563118 describing the unexpected change from loggerhead.conf to serve-branches.conf
• Apache, LDAP and groups

Note that all these instructions were performed as root.

The puppet master

Gaffel will be puppet master. I’ve added a DNS entry for puppet.karssen.org that points to gaffel. This installs the client and the Puppet master:

$ aptitude install puppet puppetmaster

The main configuration of server and client can be found in /etc/puppet/puppet.conf. We’ll leave it at the default for now. The file /etc/puppet/manifests/sites.pp contains options that apply to the whole site. Let’s make it and add the following contents:

import "nodes"
 
# The filebucket is for backups. Originals of files that Puppet modifies
# get stored here.
filebucket { main: server => 'puppet.karssen.org' }
File  { backup => main}
 
# Set the default $PATH$ for executing commands on node systems.
Exec { path => "/usr/bin:/usr/sbin:/bin:/sbin:" }

The file /etc/puppet/manifests/nodes.pp defines the nodes/clients that will be managed by puppet as well as what configuration will be applied to them, so-called roles. For now, let’s make a quick example:

node common {
	include packages
}
 
node lambik inherits common {
	include ntp::client
}

Both the ‘packages’ and the ‘ntp’ modules still need to be defined. Let’s do that now.

Modules are collections of puppet code (known as manifests) and related files that are used for client configuration. Modules are stored in /etc/puppet/modules/.
Let’s start with the ntp example. First make the necessary directory structure:

$ mkdir -p /etc/puppet/modules/ntp/{manifests,files,templates}

Every modules needs a file init.pp that declares the class. It can also include other files. The files and templates directories are used to store files that need to be copied to the node or templates to make such files, respectively. We’ll come across some examples of both. This is the init.pp file for the ntp role (/etc/puppet/modules/ntp/manifests/init.pp):

class ntp::client {
	 package { "ntp":
    		 ensure => installed,
	 }
 
	 service { "ntp_client":
		 name       => "ntp"
    		 ensure     => running,
#		 hasstatus  => true,
		 hasrestart => true,
		 require    => Package["ntp"],
	 }
}

Here we indicate that the NTP service must be running and that it’s init script (in /etc/init.d) accepts the status and restart options. Lastly in the require line we note that before this manifest can be applied we must make sure that the package ntp has been installed. This is necessary, because the order in which the two directives are executed is not necessarily the order in
which they appear in the manifest.

The # in from of the hasstatus attribute is because of a bug inthe puppet version (2.6.4) shipped with Ubuntu 11.04. See http://projects.puppetlabs.com/issues/5610 for the bug report. In version 2.6.7 it is supposedly fixed.

In our nodes.pp file we also mentioned a packages class. In this class we list all the packages that we want to have installed on the node. Let’s make the packages module. First create the necessary directories:

$ mkdir -p /etc/puppet/modules/packages/{manifests,files,templates}

Add the file /etc/puppet/modules/packages/manifests/init.pp:

class packages {
	 $base_packages = [
	 "openssh-server",
	 "nfs-common",
	 "etckeeper",
	 "htop",
	 "iotop",
	 "iftop",
	 ]
 
	 $editor_packages = [
	 "emacs",
	 "emacs-goodies-el",
	 "elscreen",
	 ]
 
	 $all_packages = [
	 $base_packages,
	 $editor_packages,
	 ]
 
	 package { $all_packages:
	      ensure => installed,
	 }
}

Here I’ve defined three variables (beginning with a $ sign), one for base packages, one for editor-related packages and one called $all_packages that incorporates them both. Finally, I tell puppet to ensure they are all installed.

Setting up a client

As a test client I’m using lambik, one of my MythTV frontends.

$ aptitude install puppet

To make sure that puppet starts by default on system startup edit the file /etc/default/puppet and set START to yes:

# Defaults for puppet - sourced by /etc/init.d/puppet
 
# Start puppet on boot?
START=yes
 
# Startup options
DAEMON_OPTS=""

Now edit /etc/puppet/puppet.conf (on the client) and add the FQDN of the puppet master server to the [main] section:

[main]
logdir=/var/log/puppet
vardir=/var/lib/puppet
ssldir=/var/lib/puppet/ssl
rundir=/var/run/puppet
factpath=$vardir/lib/facter
templatedir=$confdir/templates
prerun_command=/etc/puppet/etckeeper-commit-pre
postrun_command=/etc/puppet/etckeeper-commit-post
server = puppet.karssen.org
 
[master]
# These are needed when the puppetmaster is run by passenger
# and can safely be removed if webrick is used.
ssl_client_header = SSL_CLIENT_S_DN
ssl_client_verify_header = SSL_CLIENT_VERIFY

Setting up secure communication between master and nodes and first test run

Puppet uses SSL certificates to set up a secure connection between master and nodes. Before you can apply any changes to the client, certificates need to be exchanged and signed. First, tell the client to connect to the puppet master:

$ puppetd --test
info: Creating a new SSL key for lambik.karssen.org
warning: peer certificate won't be verified in this SSL session
info: Caching certificate for ca
warning: peer certificate won't be verified in this SSL session
warning: peer certificate won't be verified in this SSL session
info: Creating a new SSL certificate request for lambik.karssen.org
info: Certificate Request fingerprint (md5): 1D:A3:3A:4A:A6:DA:D6:C8:96:F4:D4:7E:52:F4:12:1D
warning: peer certificate won't be verified in this SSL session
warning: peer certificate won't be verified in this SSL session
warning: peer certificate won't be verified in this SSL session
Exiting; no certificate found and waitforcert is disabled

On the puppet master we can now sign the certificate:

$ puppetca -l
lambik.karssen.org
$ puppetca -s lambik.karssen.org
notice: Signed certificate request for lambik.karssen.org
notice: Removing file Puppet::SSL::CertificateRequest lambik.karssen.org at '/var/lib/puppet/ssl/ca/requests/lambik.karssen.org.pem'

On the client we can now rerun puppetd:

root@lambik:~# puppetd --test
info: Caching catalog for lambik.karssen.org
info: Applying configuration version '1311930908'
notice: /Stage[main]/Packages/Package[iotop]/ensure: ensure changed 'purged' to 'present'
notice: /Stage[main]/Packages/Package[iftop]/ensure: ensure changed 'purged' to 'present'
notice: /Stage[main]/Ntp/Package[ntp]/ensure: ensure changed 'purged' to 'present'
notice: /Stage[main]/Packages/Package[emacs-goodies-el]/ensure: ensure changed 'purged' to 'present'
notice: /Stage[main]/Packages/Package[htop]/ensure: ensure changed 'purged' to 'present'
info: Creating state file /var/lib/puppet/state/state.yaml
notice: Finished catalog run in 78.43 seconds

If all went well, we can now start the puppet client daemon to keep our system under puppet control:

$ service puppet start

Adding (configuration) files to the roles

Since I run my own NTP server (ntp.karssen.org, only accessible from inside my LAN), the NTP configuration file (/etc/ntp.conf) must be changed. Of course, we want Puppet to take care of this. The ntp.conf file I want to distribute to all nodes has the following contents (note that the only change is the name of the server and commenting the restrict lines):

# /etc/ntp.conf, configuration for ntpd; see ntp.conf(5) for help
 
driftfile /var/lib/ntp/ntp.drift
 
 
# Enable this if you want statistics to be logged.
#statsdir /var/log/ntpstats/
 
statistics loopstats peerstats clockstats
filegen loopstats file loopstats type day enable
filegen peerstats file peerstats type day enable
filegen clockstats file clockstats type day enable
 
# Specify one or more NTP servers.
 
# Use servers from the NTP Pool Project. Approved by Ubuntu Technical Board
# on 2011-02-08 (LP: #104525). See http://www.pool.ntp.org/join.html for
# more information.
server ntp.karssen.org
 
# Use Ubuntu's ntp server as a fallback.
server ntp.ubuntu.com
 
# Access control configuration; see /usr/share/doc/ntp-doc/html/accopt.html for
# details.  The web page <http://support.ntp.org/bin/view/Support/AccessRestrict
ions>
# might also be helpful.
#
# Note that "restrict" applies to both servers and clients, so a configuration
# that might be intended to block requests from certain clients could also end
# up blocking replies from your own upstream servers.
 
# By default, exchange time with everybody, but don't allow configuration.
#restrict -4 default kod notrap nomodify nopeer noquery
#restrict -6 default kod notrap nomodify nopeer noquery
 
# Local users may interrogate the ntp server more closely.
restrict 127.0.0.1
restrict ::1
 
# Clients from this (example!) subnet have unlimited access, but only if
# cryptographically authenticated.
#restrict 192.168.123.0 mask 255.255.255.0 notrust
 
 
# If you want to provide time to your local subnet, change the next line.
# (Again, the address is an example only.)
#broadcast 192.168.123.255
 
# If you want to listen to time broadcasts on your local subnet, de-comment the
# next lines.  Please do this only if you trust everybody on the network!
#disable auth
#broadcastclient

Save this file in /etc/puppet/modules/ntp/files (on the puppet master). Now edit the manifest for the ntp role (/etc/puppet/modules/ntp/manifest/init.pp) to add the file section and a subscribe command:

class ntp::client {
	 package { "ntp":
		      ensure => installed,
	 }
 
	 service { "ntp_client":
	      name       => "ntp",
	      ensure     => running,
#	      hasstatus  => true,
	      hasrestart => true,
	      require    => Package["ntp"],
	      subscribe  => File["ntp_client_config"],
	 }
 
	 file { "ntp_client_config":
		   path => "/etc/ntp.conf",
	   owner   => root,
	   group   => root,
	   mode    => 644,
	   source  => "puppet:///ntp/ntp.conf",
	   require => Package["ntp"],
	 }
}

The URL specified in the source line automatically looks in the right place (as mentioned just above) for the file. Because we don’t want to wait for puppet to automatically pass on this configuration, let’s run it by hand:

root@lambik:~# puppetd --test
info: Caching catalog for lambik.karssen.org
info: Applying configuration version '1311936811'
--- /etc/ntp.conf	2011-06-17 07:59:54.000000000 +0200
+++ /tmp/puppet-file20110729-12128-1h3fupz-0	2011-07-29 12:53:33.279622938 +0200
@@ -16,16 +16,14 @@
 # Use servers from the NTP Pool Project. Approved by Ubuntu Technical Board
 # on 2011-02-08 (LP: #104525). See http://www.pool.ntp.org/join.html for
 # more information.
-server 0.ubuntu.pool.ntp.org
-server 1.ubuntu.pool.ntp.org
-server 2.ubuntu.pool.ntp.org
-server 3.ubuntu.pool.ntp.org
+server ntp.karssen.org
 
 # Use Ubuntu's ntp server as a fallback.
 server ntp.ubuntu.com
 
@@ -33,8 +31,8 @@
 # up blocking replies from your own upstream servers.
 
 # By default, exchange time with everybody, but don't allow configuration.
-restrict -4 default kod notrap nomodify nopeer noquery
-restrict -6 default kod notrap nomodify nopeer noquery
+#restrict -4 default kod notrap nomodify nopeer noquery
+#restrict -6 default kod notrap nomodify nopeer noquery
 
 # Local users may interrogate the ntp server more closely.
 restrict 127.0.0.1
info: FileBucket adding /etc/ntp.conf as {md5}32280703a4ba7aa1148c48895097ed07
info: /Stage[main]/Ntp::Client/File[ntp_client_config]: Filebucketed /etc/ntp.conf to main with sum 32280703a4ba7aa1148c48895097ed07
notice: /Stage[main]/Ntp::Client/File[ntp_client_config]/content: content changed '{md5}32280703a4ba7aa1148c48895097ed07' to '{md5}0d1b81c95bab1f6b08eb27dfaeb18bb5'
info: /Stage[main]/Ntp::Client/File[ntp_client_config]: Scheduling refresh of Service[ntp_client]
notice: /Stage[main]/Ntp::Client/Service[ntp_client]: Triggered 'refresh' from 1 events
notice: Finished catalog run in 3.06 seconds

Setting NFS mounts in /etc/fstab

On my clients I want to mount several NFS shares. Let’s create the directories for the nfs_mounts module (on the puppet master of course):

$ mkdir -p /etc/puppet/modules/nfs_mounts/{manifests,files,templates}

Next, let’s edit the manifest (/etc/puppet/modules/nfs_mounts/manifests/init.pp):

class nfs_mounts {
	 # Create the shared folder unless it already exists
	 exec { "/bin/mkdir -p /var/sharedtmp/":
		   unless => "/usr/bin/test -d /var/sharedtmp/",
	 }
 
	 mount { "/var/sharedtmp/":
	    atboot  => true,
	    ensure  => mounted,
	    device  => "nfs.karssen.org:/var/sharedtmp",
	    fstype  => "nfs",
	    options => "vers=3",
	    require => Package["nfs-common"],
	 }
}

This should make the /var/sharedtmp directory and mount it. Note that I mention the nfs_common package in a require line. This package was defined in the packages module (in the $base_packages variable. Now let’s add this module to the nodes.pp file:

node common {
  include packages
}
 
node lambik inherits common {
	include ntp::client
	include nfs_mounts
}

Since I’ve got more than a single NFS mount, let’s extend the previous example and use a defined resource. Change the file /etc/puppet/modules/nfs_mounts/manifests/init.pp as follows:

define nfs_mount(
	  $location,
	  $server  = "nfs.karssen.org",
	  $options = "vers=3",
	  $fstype  = "nfs"
) {
  file {"$location":
	  ensure => directory,
  }
 
  mount { "$location":
  	atboot  => true,
	ensure  => mounted,
	device  => "${server}:${location}",
	fstype  => "$fstype",
	options => "$options",
	require => [ Package["nfs-common"], File["$location"] ],
  }
}
 
class nfs_mounts {
 
			 nfs_mount { "/home":
		 	   location => "/home",
		 }
 
			 nfs_mount { "/var/sharedtmp":
		 	    location => "/var/sharedtmp",
 		 }
 
			 nfs_mount { "/var/video":
	 	    	    location => "/var/video",
 		 }
 
			 nfs_mount { "/var/music":
	 	    	    location => "/var/music",
 		 }
}

Here we first define a resource called nfs_mount, which can accept various parameters, all of which have a default value, except $location. Secondly we ensure that this location is a directory and then we define how it should be mounted. In the subsequent class definition we use this nfs_mount resource several times to mount the various NFS shares.
Note that it would have been easier if the definition of nfs_mount would have started with

define nfs_mount (
	  $location = $name,

because then the invocations of nfs_mount in the class would not
need the location => line. Unfortunately this doesn’t work. It’s
a known bug that has been fixed in version 2.6.5
(http://projects.puppetlabs.com/issues/5061).

Links

• Article by Jes Fraser in the July 2011 edition of Linux Journal. A lot of this info was taken from this article.
• A blog
• The Puppet documentation
• The Puppet language guide

In the Unix/Linux world many programs are compiled from source in three steps:

./configure
make
make install

Usually the necessary files for this have been created using the autotools. The goal of the first step is to create a so-called Makefile that contains instructions on how to compile and install the files (as done in the two subsequent make steps.

Some software packages, however, include a ready-made Makefile that, in addition, doesn’t accept the environment variable DESTDIR. This last point is what makes packaging the application into a .deb file a bit tricky. The reason for this is that the package build scripts want to install the files of your application in a temporary directory and not into system-wide directories like /usr/bin/ etc. during the packing process. As such, packaging does not require root privileges.

At work we use many programs and tool sets developed by ourselves and other scientists. I know from my own experience that setting up autotools for your program is not trivial. Actually, for lack of time I’ve never successfully done it and for most of the rather simple programs that I’ve written setting up a complete autoconf/automake environment seems a bit overkill. I usually ended up writing a simple Makefile that compiles to code and installs it (usually in /usr/local/bin).

Merlin by Abecasis et al. is a great piece of software developed at the University of Michigan. However, as you may have expected by now, its Makefile does not accept the DESTDIR variable, instead running make tells you that in order to install in a different directory you’ll have to run

make INSTALLDIR=/some/other/directory

Therefore, all quick and dirty .deb recipes one finds on the Internet do not work without some adaptations. So here is what I did to make a .deb of it. It won’t be a full tutorial on how to do packaging, see the references at the end of this post for that. I’ll assume here that you have your build environment set up (e.g. the build-essential and fakeroot packages, as well as some others).

tar -xzf merlin-1.1.2.tar.gz
cd merlin-1.1.2
dh_make --single --email youremail@address --file ../merlin-1.1.2.tar.gz

Now the basic files are ready. Apart from the untarred source files the files needed for Debian packaging have also been created (in merlin-1.1.2/debian).

Time to make the necessary changes. First, since the Makefile included with merlin does not accept the DESTDIR variable that the Debian packaging system uses we’ll patch the Makefile in such a way that it works (I tried to fix this in the debian/control file, but in the end adapting the Makefile was much easier). I do this by changing the line

INSTALLDIR=/usr/local/bin

to

# default installation directory
ifeq ($(DESTDIR),"")
    INSTALLDIR=/usr/local/bin/
else
    INSTALLDIR=$(DESTDIR)/usr/bin/
endif

Let’s do some polishing of the package. I don’t want to make the perfect package, but adding a bit of text to the debian/control file make a lot of difference. This is what it looked like after my edits:

Source: merlin
Section: science
Priority: extra
Maintainer: Lennart C. Karssen <youremail@address>
Build-Depends: debhelper (>= 7)
Standards-Version: 3.8.3
Homepage: http://www.sph.umich.edu/csg/abecasis/merlin/index.html
 
Package: merlin
Architecture: any
Depends: ${shlibs:Depends}, ${misc:Depends}
Description: Package for fast pedigree analysis
 MERLIN uses sparse trees to represent gene flow in pedigrees
 and is one of the fastest pedigree analysis packages around
 (Abecasis et al, 2002).

Also editing the file debian/changelog is a good idea, especially since I changed the source code (remember the Makefile?). This is what I wrote:

merlin (1.1.2-1) unstable; urgency=low
 
  * Initial release
  * Adjusted Makefile to make DESTDIR work.
 
 -- Lennart C. Karssen <youremail@address>  Tue, 05 Apr 2011 12:04:21 +0200

Officially you should edit the debian/copyright file as well, but since the merlin licence doesn’t allow distribution of the source or the binaries I didn’t bother.

To finally build the package run

dpkg-buildpackage -rfakeroot -us -uc

This creates a .deb file in the directory where you started. As a final touch you can check your package for errors with

lintian ../merlin_1.1.2-1_amd64.deb

References:

• http://www.debian.org/doc/maint-guide
• http://www.debian-administration.org/articles/337
• http://www.webupd8.org/2010/01/how-to-create-deb-package-ubuntu-debian.html

The good thing for me is that I can now make off-site backups of my home directories. I configured the DS211j to accept SSH connections so that I can log into it (as user admin or root). I used the web interface to create a directory for my backups (which appeared to be /volume1/BackupLennart after logging in with SSH).

After making a hole in her firewall that allowed me to connect to the DS211j, I created a backup script in /etc/cron.daily with the following contents:

#!/bin/bash
#
# This script makes a backup of my home dirs to a Synology DiskStation at
# another location. I use LVM for my /home, so I make a snapshot first and
# backup from there.
#
# Time-stamp: <2011-02-06 21:30:14 (lennart)>
 
###############################
# Some settings
###############################
 
# LVM options
VG=raidvg01
LV=home
MNTDIR=/mnt/home_rsync_snapshot/
 
# rsync options
DEST=root@remote-machine.example.com:/volume1/BackupLennart/
SRC=${MNTDIR}/*
OPTIONS="-e ssh --delete --progress -azvhHS --numeric-ids --delete-excluded "
EXCLUSIONS="--exclude lost+found --exclude .thumbnails --exclude .gvfs --exclude .cache --exclude Cache"
 
 
 
###############################
# The real work
###############################
 
# Create the LVM snapshot
if [ -d $MNTDIR ]; then
    # If the snapshot directory exists, another backup process may be
    # running
    echo "$MNTDIR already exists! Another backup still running?"
    exit -1
else
    # Let's make snapshots
    mkdir -p $MNTDIR
    lvcreate -L5G -s -n snap$LV /dev/$VG/$LV
    mount /dev/$VG/snap$LV $MNTDIR
fi
 
 
# Do the actual backup
rsync $OPTIONS $EXCLUSIONS $SRC $DEST
 
# Remove the LVM snapshot
if [ -d $MNTDIR ]; then
    umount /dev/$VG/snap$LV
    lvremove -f /dev/$VG/snap$LV
    rmdir $MNTDIR
else
    echo "$MNTDIR does not exist!"
    exit -1
fi

Let’s walk through it: in the first section I configure several variables. Since I use LVM on my server, I can use it to make a snapshot of my /home partition. The LVM volume group I use is called ‘raidvg01′. Withing that VG my /home partition resides in a logical volume called ‘home’. The variable MNTDIR is the place where I mount the LVM snapshot of ‘home’.

The rsync options are quite straight forward. Check the rsync man page to find out what they mean. Note that I used the --numeric-ids option because the DS211j doesn’t have the same users as my server and this way all ownerships will still be correct if I ever need to restore from this backup.

In the section called “The real work” I first create the MNTDIR directory. Subsequently I create the LVM snapshot and mount it. After this the rsync backup can be run and finally I unmount the snapshot and remove it, followed by the removal of the MNTDIR.

Since the script is placed in /etc/cron.daily it will be executed every day. Since we use SSH to connect to the remote DS211j I set up SSH key access without a password. This Debian howto will tell you how to set that up.

The only thing missing in this setup is that the backups are not stored in an encrypted form on the remote NAS, but for now this is good enough. I can’t wait until the network bandwidth on both sides of this backup connection get so fast (and affordable) that I can easily sync my music as well. Right now uploads are so slow that I hardly dare to include those. I know that I shouldn’t complain since the Netherlands has one of the highest broadband penetrations in the world, but, hey, don’t you just always want a little more, just like Oliver Twist?

At work I recently implemented disk quota on our server. While trying to setup /etc/warnquota.conf I noticed the example lines at the bottom that showed how to configure warnquota to look up e-mail addresses in an LDAP directory. This was exactly what I needed, because we store our user’s e-mail address in our LDAP tree. Without this feature warnquota would try to send its warning mails to user@our-server.example.com instead of (or even other addresses for guests that only visit us for a few weeks). The lines in /etc/warnquota.conf were:

LDAP_MAIL = true
LDAP_HOST = ldap.example.com
LDAP_PORT = 389
LDAP_BASEDN = ou=Users,dc=example,dc=com
LDAP_SEARCH_ATTRIBUTE = uid
LDAP_MAIL_ATTRIBUTE = mail
LDAP_DEFAULT_MAIL_DOMAIN = example.com

So, after saving the file I tested it by running warnquota -s (as root, and I also made sure I reduced my own quota so I would be the one getting an e-mail warning).

Unfortunately warnquota spitted out some errors:

warnquota: Error in config file (line 65), ignoring
warnquota: Error in config file (line 66), ignoring
warnquota: Error in config file (line 67), ignoring
warnquota: Error in config file (line 68), ignoring
warnquota: Error in config file (line 69), ignoring
warnquota: Error in config file (line 70), ignoring
warnquota: Warning: Mailer exitted abnormally.

These were the line numbers with the LDAP options above . Google pointed me to an old bug in Fedora that was marked as resolved. I also found out that the quota tools should be compiled with LDAP support for this to work. To be sure that it was actually possible I configured warnquota on my home server that runs Ubuntu 10.04 and also uses LDAP. There, it all worked as expected.

So, my next step was clear: make my own RPM package for quota. The one installed by CentOS 5.4 is quota-3.13-1.2.5.el5. These are the steps I took:

• Enable the CentOS source repository by creating the file etc/yum.repos.d/CentOS-Source.repo with this contents:

  [centos-src]
  name=CentOS $releasever - $basearch - Source
  baseurl=http://mirror.centos.org/centos/$releasever/os/SRPMS/
  gpgcheck=1
  gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5

  The run yum update and check that the new repository is listed.

• Install the yum-utils and rpmdevtools packages: sudo yum install yum-utils rpmdevtools.
• Set up a directory to do your build work in. I created the directory ~/tmp/pkgtest.
• Run rpmdev-setuptree to create the required sub directories.
• Set the basic build configuration by creating the file ~/.rpmmacros with the following contents:

  # Path to top of build area
  %_topdir    /home/lennart/tmp/pkgtest

• Go into the SRPMS directory and download the source package: yumdownloader --source quota
• In the top level directory run rpm -i SRPMS/quota-3.13-1.2.5.el5.src.rpm to unpack the package.
• The SPECS directory now contains the .spec file that contains the build instructions. The SOURCES directory contains the source files and patches from Red Hat. In a temporary directory I untar-ed the quotatools source tar.gz file and ran ./configure --help to find out which option I should add to the spec file in order to enable LDAP lookup. The option was: --enable-ldapmail=yes. The set of configure lines in the spec file now looked like this:

  %build
  %configure \
  	--with-ext2direct=no --enable-rootsbin --enable-ldapmail=yes
  make

  In the spec file I also added a changelog entry:

  * Mon Oct 18 2010 Lennart Karssen <lennart@karssen.org 1:3.13-1.2.6
  - Added --enable-ldapmail=try to the ./configure line to enable LDAP
    for looking up mail addresses. (Resolves Red Hat Bugzilla 133207,
    it is marked as resolved there, but apparently was reintroduced.)

  And I also bumped the build version number at the top of the file (the Release: line). Finally, I added openldap-devel to the BuildPreReq line (of course I ran into a compilation error first and then installed the openldap-devel package ).

• Now it’s time to build the package. In the SPEC directory run: rpmbuild -bb quota.spec and wait. The RPM package is created in the RPMS directory.
• Install the package: sudo rpm -Uvh RPMS/x86_64/quota-3.13-1.2.6.x86_64.rpm (if you didn’t bump the package version number the --replacepkgs must be added to ‘upgrade’ to the same version).

And that was it! The package installed cleanly and a test run of warnquota -s was successful.

Today I implemented Nagios event handlers in this setup. The idea behind an event handler is the following: If e.g. a service goes down, Nagios should try to solve this problem itself before notifying the administrator (me). It should, in this case, simply try to restart the service.

The Nagios documentation [1] describes how to do this for a service that runs on the same machine as the Nagios service. In my case, however, the services are running on the to real servers. To me it seemed logical to use NRPE to execute the necessary commands on the remote hosts (since NRPE was already running on those machines anyway).
In order to adapt the scheme from the Nagios docs to work on remote servers as well three things need to be done:

• The command that is executed by the event handler script should be changed to use NRPE
• On the remote machine the nagios user (under which the NRPE service is running) should be given some sudo rights so that it is actually allowed to start a service.
• The NRPE configuration on the remote machine should of course be changed to include the new command(s) for starting services.

So here we go! First, the Nagios configuration on the management host. In the service definition file I added one line for the event handler to each service. The definition of one service now looks like this (the last line was added):

define service {
       use                      generic-service
       hostgroup_name           sge-exec-servers
       service_description      SGE execd
       check_command            check_nrpe_1arg!check_sge_execd
       notification_interval    0 ; set > 0 if you want to be renotified
       event_handler            restart-service!sge-execd
}

Next, the restart-service command must be defined. I did that in a file that I called /etc/nagios3/conf.d/event-handlers.cfg:

define command {
       command_name     restart-service
       command_line     /etc/nagios3/conf.d/event_handler_script.sh $SERVICESTATE$ $SERVICESTATETYPE $ $SERVICEATTEMPT$ $HOSTADDRESS$ $ARG1$ $SERVICEDESC$
}

The variable $ARG1$ here is the name of the service that needs to be restarted. In this example it is sge-execd from the event_handler line in the service definition. The $HOSTADDRESS will be used in the event handler script to send the right host name to NRPE.
The event_handler_script.sh referenced here is almost identical to the one in the Nagios documentation. As mentioned in the plan above, I changed it slightly so that it uses NRPE.

#!/bin/sh                                                                                            
#
# Event handler script for restarting the nrpe server on the local machine
# Taken from the Nagios documentation and
# http://www.techadre.com/sites/techadre.com/files/event_handler_script_0.txt
# Adapted by L.C. Karssen
# Time-stamp: <2010-09-14 15:24:33 (root)>
#
# Note: This script will only restart the nrpe server if the service is
#       retried 3 times (in a "soft" state) or if the web service somehow
#       manages to fall into a "hard" error state.
#
 
date=`date`
 
# What state is the NRPE service in?
case "$1" in
OK)
        # The service just came back up, so don't do anything...
        ;;
WARNING)
        # We don't really care about warning states, since the service is probably still running...
        ;;
UNKNOWN)
        # We don't know what might be causing an unknown error, so don't do anything...
        ;;
CRITICAL)
        # Aha!  The BLAH service appears to have a problem - perhaps we should restart the server...
 
        # Is this a "soft" or a "hard" state?
        case "$2" in
 
        # We're in a "soft" state, meaning that Nagios is in the middle of retrying the
        # check before it turns into a "hard" state and contacts get notified...
        SOFT)
                # What check attempt are we on?  We don't want to restart the web server on the firs\
t
                # check, because it may just be a fluke!
                case "$3" in
 
                # Wait until the check has been tried 3 times before restarting the web server.
                # If the check fails on the 4th time (after we restart the web server), the state
                # type will turn to "hard" and contacts will be notified of the problem.
                # Hopefully this will restart the web server successfully, so the 4th check will
                # result in a "soft" recovery.  If that happens no one gets notified because we
                # fixed the problem!
                3)
                        echo -n "Restarting service $6 (3rd soft critical state)...\n"
                        # Call NRPE to restart the service on the remote machine
                        /usr/lib/nagios/plugins/check_nrpe -H $4 -c restart-$5
                        echo "$date - restart $6 - SOFT"  >> /tmp/eventhandlers
                        ;;
                        esac
                ;;
 
        # The service somehow managed to turn into a hard error without getting fixed.
        # It should have been restarted by the code above, but for some reason it didn't.
        # Let's give it one last try, shall we?
        # Note: Contacts have already been notified of a problem with the service at this
        # point (unless you disabled notifications for this service)
        HARD)
                case "$3" in
 
                4)
                        echo -n "Restarting $6 service...\n"
                        # Call the init script to restart the NRPE server
                        echo "$date - restart $6 - HARD"  >> /tmp/eventhandlers
                        /usr/lib/nagios/plugins/check_nrpe -H $4 -c restart-$5
                        ;;
                        esac
                ;;
        esac
        ;;
esac
exit 0

Now Nagios can be restarted and should continue its work as usual. Time to make the changes on the remote hosts.

First, we’ll grant the necessary sudo rights to the nagios user. Run visudo and add these lines:

## Allow NRPE to restart sevices
User_Alias NAGIOS = nagios,nagcmd
Cmnd_Alias NAGIOSCOMMANDS = /usr/sbin/service
Defaults:NAGIOS !requiretty
NAGIOS    ALL=(ALL)    NOPASSWD: NAGIOSCOMMANDS

And finally add the required lines in the NRPE config file (/etc/nagios/nrep.cfg):

command[restart-sge-execd]=/usr/bin/sudo /usr/sbin/service gridengine-exec start

Restart the NRPE daemon and it should all work. Test it by manually stopping the service.

[1] Nagios documentation on Event Handlers
[2] Two blog posts that describe a similar set up. I used these as a starting point for my own set up.

I took quite some time to look around for a laptop like this, and this Thinkpad seems to be the only one that satisfies my minimum requirements:
- Matte screen; no glossy screens for me, I’ve already got a mirror in my bathroom .
- Trackpoint; yep, that’s the red dot in between the G, H, and B keys.
- A processor that was more powerful than Intel’s Atom
- A decent keyboard, because for me, using Linux means using the command line and Emacs a lot.

After several weeks of use I’ve found only one drawback to this machine: it’s processor is not that efficient. It uses quite some power and therefore gets a bit hot. As a result the fan runs a lot (even though it’s not that audible) and battery life is not too good. I’m getting approximately 2 to 3 hours out of it if I reduce the screen brightness and turn wifi off. That could have been better (maybe Lenovo should have used an Intel CULV processor?), but it’s not too much of a limitation. But this came at no surprise, most reviews on the web mention it.

After opening the box I quickly made an image of the Windows partitions that were on it and then proceeded to install Ubuntu 10.04 on it. Most of the hardware was recognised by the 2.6.32 kernel included with Ubuntu’s 10.04 release. However, as several blogs (see links below) pointed out there are a few bumps, e.g. with suspend and resume, or the wireless chip that is able to connect, but doesn’t want to send or receive data. The bumps were smoothed out by installing a newer kernel (2.6.35-12-generic) from the Ubuntu kernel PPA. The 2.6.35 kernel is the one that will be used in the next Ubuntu release and the PPA contains packages that make this kernel run in the present release as well. With that kernel, suspend and hibernate run well, as well as most Fn function keys. In fact, the only one that doesn’t seem to work is Fn+F3 for microphone mute. I had to turn on the bluetooth module in Windows before it showed up in Ubuntu (as noted by several blogs). At the moment, the things that don’t work correctly are:
- The microphone doesn’t record (neither in the sound recorder, nor when using Skype). Sometimes it shows some activity if the mic-volume slider is moved to about 25%, but I couldn’t get that to work reliably.
- The combined mic/headphone jack doesn’t mute the speakers if a pair headphones is plugged in (neither is any sound heard through the headphones).
Maybe a newer ALSA release in the upcomming Ubuntu 10.10 will remedy these problems.

I was pleasantly surprised by the fact that using the open source radeon driver (installed by default) for the AMD/ATI graphics card worked out of the box, including Compiz 3D desktop fancy stuff. The VGA out also worked perfectly when I hooked it up to my Sony Bravia TV. Xorg’s RandR detected it and I could choose between an extended desktop or a clone setup.

As I already mentioned, I’m a trackpoint user, so I wanted to disable the touchpad, especially since the two buttons for it are located at the front edge of the laptop and are easily pressed when the device sits on your lap and you’ve got your knees pulled up.
Secondly I enabled wheel emulation for the trackpoint. Now, if I click and hold the middle ‘mouse’ button and push the trackpoint in a certain direction it acts as a scroll wheel. To achieve this I created the file /usr/lib/X11/xorg.conf.d/20-thinkpad.conf (EDIT: for Ubuntu 10.10 this file should be located in /usr/share/X11/xorf.conf.d/) with the following contents:

Section "InputClass"
	Identifier "Trackpoint Wheel Emulation"
	MatchProduct "Trackpoint"
	MatchDevicePath "/dev/input/dev*"
	Driver "evdev"
	Option "EmulateWheel" "true"
	Option "EmulateWheelButton" "2"
	Option "Emulate3Buttons" "3"
	Option "XAxisMapping" "6 7"
	Option "YAxisMapping" "4 5"
EndSection

All in all I’m very happy with the X100e. It’s a small but sturdy laptop with an excellent screen and a great keyboard.

Some links:
- An excellent review of the Lenovo Thinkpad X100e
- A recent review at AnandTech
- Ubuntu kernel PPA
- ThinkWiki page for the X100e, has lots of info on running Linux on this laptop.
- A blog about installing Ubuntu Linux on the X100e, the problems mentioned in that post and its comments have now been solved (if you install the 2.6.35 kernel from the PPA). I tried the gpointing-device-settings package for some time (to get the trackpoint scroll functionality to work), but its settings didn’t survive across reboots or even after hibernating, so I removed it again.

The trackball features four buttons: two large ones for left and right-clicking and to smaller ones that work as back and forward buttons in Firefox, for example.

After plugging it into my PC it was instantly recognised by X (I’m using Ubuntu 10.04). There’s no middle mouse button, but that can be emulated by clicking the left and right mouse buttons at the same time (something I’ve been use to on older laptops, and, well, even from the time that some of the mouses I owned only had two buttons). However, I did miss my scroll wheel. A quick search on the Internet brought me to Rob Meerman’s website where he explains a lot about the Trackman and how it works in X. He even has a special section on Ubuntu 10.04. In short it comes down to these commands:

xinput set-int-prop "Logitech USB Trackball" "Evdev Wheel Emulation Button" 8 8
xinput set-int-prop "Logitech USB Trackball" "Evdev Wheel Emulation" 8 1

Unfortunately the changes made by these commands are not persistent across reboots. I’ll try to fix that later.

EDIT: To add middle mouse button emulation and horizontal scrolling (thanks to rejistania below) run:

xinput set-int-prop "Logitech USB Trackball" "Evdev Middle Button Emulation" 8 1
xinput set-prop "Logitech USB Trackball" "Evdev Wheel Emulation Axes" 6 7 4 5

END EDIT

Regarding the use of a trackball compared to an ordinary mouse my experiences so far have been very positive. It didn’t take me a lot of time to get used to it. Also precision placement of the pointer doesn’t seem to be more difficult that with a regular mouse. So for now my wireless Logitech mouse can take a holiday . The nicest think about the trackball is the fact that you don’t have to move the whole device. So it’s less ‘weight lifting’. Also, the fact that the ball (in combination with the small button) is the scroll wheel, makes for a relatively heavy wheel without much friction, so scrolling large distances can simply be done by giving the ball a good spin. Nice!