Monthly Archives: March 2011

Migrate KVM Disk Access from IDE to Virtio

This article assumes a KVM guest that runs from raw or qcow2 devices using emulated ide access. Since some time now, full virtualisation for block devices is available, facilitating the benefit of paravirtualisation.

Benchmarked in `bonnie++`, the difference looks like this, revealing a way better performance of `virtio`:

IDE:
~~~
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
2G 856 94 33126 7 30392 7 4727 92 739753 70 300.4 20
Latency 51096us 2473ms 972ms 5240us 1516us 382ms
Version 1.96 ------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 35 0 +++++ +++ 46 0 34 0 +++++ +++ 44 0
Latency 338ms 808us 2136ms 341ms 155us 2016ms
~~~

Virtio:
~~~
Version 1.96 ------Sequential Output------ --Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
2G 732 91 58767 8 72183 11 4026 94 1477698 80 321.6 9
Latency 16751us 1159ms 492ms 4155us 624us 422ms
Version 1.96 ------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 34 0 +++++ +++ 46 0 34 0 +++++ +++ 45 0
Latency 240ms 790us 1502ms 296ms 167us 1450ms
~~~

So in order to make the switch from from `ide` to `virtio`, the following steps need to be taken:

Run `virsh edit `. From there, edit the config file and adjust all lines of
~~~

~~~
so they look like this
~~~

~~~
Furthermore, remove all `

` lines so that `libvirt` can regenerate them appropriately.

Inside the guest, edit `/etc/fstab` and replace all occurrences of `/dev/sdX` with /dev/vdX`.

That's it, now shutdown the machine and start it with an `virsh start ` (just a reboot inside the started VM won't work).

Settin up a custom Weave Sync Server for Firefox 4

 

Introduction

Starting with Firefox 4 it is really easy to keep your different Firefox browser setups at home, on your mobile or at work in sync. The Mozilla Corporation even hosts a central server for you so basically the setup can't be more easy.

Yet, if you're a little paranoid and you don't want to store your precious browser data on someoneelse's servers, you might wish for your own sync server. There are two posibilities to setup your own server: the [Weave Minial Server](http://tobyelliott.wordpress.com/) and the full blown one.

Weave Minimal Server's development has been stopped now and is deprecated since shortly after FF4's relase. Therefore, this article documents the setup of the full server.

The chosen install location will be `/var/www/weave`. The article assumes a Mysql server for authentication installed locally. Further, a functional Apache 2 server is assumed with a fully configured SSL setup. The documentation was tested using an Ubuntu 10.10 setup.

Package Retrieval and Storage

A working installation of a full Weave server consists of of two packages: the Weave Sync Server and the Weave Registration Server. You may download and unpack the archives like this:

mkdir -p /var/www/weave/weave-synchronisation \
/var/www/weave/weave-registration
cd /var/www/weave && touch index.html
wget http://hg.mozilla.org/services/sync-server/archive/tip.tar.gz
tar xzf tip.tar.gz -C weave-synchronisation --strip=1 && rm tip.tar.gz
wget http://hg.mozilla.org/services/reg-server/archive/tip.tar.gz
tar xzf tip.tar.gz -C weave-registration --strip=1 && rm tip.tar.gz

In case the download links are deprecated, go and look for the according packages at their respective root directories: and .

**Note as of Oct 29, 2011:** the links are deprecated as the whole sync-server gets an overhaul. The old version is still running smooth though, so the updated links are: and .

Database Setup

Create a new database called `weave` containing the following tables:

~~~
CREATE TABLE `collections` (
`userid` int(11) NOT NULL,
`collectionid` smallint(6) NOT NULL,
`name` varchar(32) NOT NULL,
PRIMARY KEY (`userid`,`collectionid`),
KEY `nameindex` (`userid`,`name`)
) ENGINE=InnoDB;
~~~

~~~
CREATE TABLE `wbo` (
`username` int(11) NOT NULL,
`collection` smallint(6) NOT NULL default '0',
`id` varbinary(64) NOT NULL default '',
`parentid` varbinary(64) default NULL,
`predecessorid` varbinary(64) default NULL,
`sortindex` int(11) default NULL,
`modified` bigint(20) default NULL,
`payload` longtext,
`payload_size` int(11) default NULL,
`ttl` int(11) default '2100000000',
PRIMARY KEY (`username`,`collection`,`id`),
KEY `parentindex` (`username`,`collection`,`parentid`),
KEY `modified` (`username`,`collection`,`modified`),
KEY `weightindex` (`username`,`collection`,`sortindex`),
KEY `predecessorindex` (`username`,`collection`,`predecessorid`),
KEY `size_index` (`username`,`payload_size`)
) ENGINE=InnoDB;
~~~

~~~
create table users (
id int(11) NOT NULL PRIMARY KEY auto_increment,
username varchar(32),
password_hash varbinary(128),
email varbinary(64),
status tinyint(4) default '1',
alert text,
reset varbinary(32) default null,
reset_expiration datetime
) engine=InnoDB;
~~~

Now, create a user named `weave` in `mysql` with `data` permissions on the newly created `Weave` database. Remember the user's password for later.

Config Adjustments on Sync and Reg Server

To customise the configs of reg and sync portion of the full Weave server, two files have to be created before being edited:

cd /var/www/weave/weave-synchronisation/1.1 && \
cp default_constants.php.dist default_constants.php
cd /var/www/weave/weave-registration/1.0 && \
cp weave_user_constants.php.dist weave_user_constants.php

Adjustment of the Synchronisation Server

Now, edit `/var/www/weave/weave-synchronisation/1.1/default_constants.php`. First, adjust the mysql user's database password according to the previous step. Chanve `weave` to your own password there in the **two** line that say:

if (!defined('WEAVE_MYSQL_STORE_READ_PASS')) { define('WEAVE_MYSQL_STORE_READ_PASS', 'weave'); }

Next, adjust the size of maximum payload in the following line. I recommend just adding a zero to `262144`:

if (!defined('WEAVE_PAYLOAD_MAX_SIZE')) { define('WEAVE_PAYLOAD_MAX_SIZE', 262144); } #256K

Now, uncomment the following line and insert a random character string where its supposed to:

#if (!defined('WEAVE_SHA_SALT')) { define('WEAVE_SHA_SALT', 'salt goes here'); }

You might also want to adjust the default quota from 5 MB to maybe 20 MB. Uncomment this line and adjust it:

#if (!defined('WEAVE_QUOTA')) { define('WEAVE_QUOTA', 5120000); } #5M

Adjustment of the Registration Server

The configuration file for the registration server is located at `/var/www/weave/weave-registration/1.0/weave_user_constants.php`. The following lines need to be customised. First, insert the `mysql` password by replacing the string `weave`:

if (!defined('WEAVE_MYSQL_AUTH_PASS')) { define('WEAVE_MYSQL_AUTH_PASS', 'weave'); }

Next, uncomment the salt line as before and insert the same character string as before for the sync server:

#if (!defined('WEAVE_SHA_SALT')) { define('WEAVE_SHA_SALT', 'salt goes here'); }

If you like to have user registration protected by captcha (I assume you do), get an account at and get yourself a global key. Adjust the following three line to suit your needs:

#if (!defined('WEAVE_REGISTER_USE_CAPTCHA')) { define('WEAVE_REGISTER_USE_CAPTCHA', 0); }
#if (!defined('RECAPTCHA_PUBLIC_KEY')) { define('RECAPTCHA_PUBLIC_KEY', '6LfWcwUAAAAAABnmLyhmgddYeJGdiRlo2MWSOpAl'); }
#if (!defined('RECAPTCHA_PRIVATE_KEY')) { define('RECAPTCHA_PRIVATE_KEY', '6LfWcwUAAAAAAHpjpBNSaxwLVQXQIG-S0Y6IG38O '); }

Apache2 Setup

This article assumes that you have your Apache 2 webserver SSL-secured and that you want the Weave server to be located at `https://yourdomain.tld/weave/`. Therefore, the following lines need to be inserted into your Apache 2 config (in Ubuntu that is `/etc/apache2/sites-enabled/default-ssl`):

# ----- weave-server -----------------------------------------------------------------

Options Indexes FollowSymLinks
AllowOverride none
Order allow,deny
Allow from all

Alias /weave/1.0 /var/www/weave/weave-synchronisation/1.1/index.php
Alias /weave/1.1 /var/www/weave/weave-synchronisation/1.1/index.php
Alias /weave/1.2 /var/www/weave/weave-synchronisation/1.1/index.php

Alias /weave/user/1.0 /var/www/weave/weave-registration/1.0/index.php
Alias /weave/user/1 /var/www/weave/weave-registration/1.0/index.php
Alias /weave/user /var/www/weave/weave-registration/1.0/index.php
Alias /weave/misc/1.0/captcha_html /var/www/weave/weave-registration/1.0/captcha.php
Alias /weave/misc/1/captcha_html /var/www/weave/weave-registration/1.0/captcha.php

Alias /weave /var/www/weave
# ----- /weave-server ----------------------------------------------------------------

Run

service apache2 reload

to make the changes work.

Client Setup

To use your brand new Weave server in your shiny Firefox 4, choose `Set Up Sync...` from the `Tools` menu. There, you will need to create a new account and enter your server address (`https://yourdomain.tld/weave/`) in the server field after choosing the use of a custom server over Firefox Sync Server.

GnuDIP setup on Ubuntu 10.10

 

*[GnuDIP]: GNU Dynamic IP

Introduction

This article briefly describes the setup of a dynamic dns-server, very similar to those commercial ones from dyndns.org and others. It will need access to the domain's zone file on the service provider's nameserver and will offer to manage dyndns-services using own domain names.

A specialty of the approach here is that a subdomain named `ddns` is added to `yourdomain.tld`. This results in dynamic names like `home.ddns.yourdomain.tld`, so that e.g. `www.yourdomain.tld` is still being managed by the nameservers of the service provider, as they usually offer a more failsafe service level.

Prerequisites

This article assumes a running web-server for the GnuDIP's webfrontend as well as a mysql-server of version > 5. This documentation won't explain their setup in further detail.

The following Ubuntu-packages need to be installed:

~~~
aptitude install bind9 apache2 libapache2-mod-perl2 \
mysql-server dnsutils xinetd wget
~~~

Initially, the following lines have to be added to the zone file of the domain which supposedly managed by the service provider (not locally):

~~~
ddns IN NS ns1.ddns
ns1.ddns IN A {IP_of_the_ddns-server}
~~~

GnuDIP setup

This documentation stores the GnuDIP files in `/opt/gnudip`. Unpacking the GnuDIP sources, the archive actually stores the documentation in its root-folder, which is not very elegant. Therefore, in this article the doc-files go into a subfolder called `doc` whereas everything else will reside in GnuDIP's root folder `/opt/gnudip`.

Create a new directory for GnuDIP:
~~~
mkdir -p /opt/gnudip
~~~

Download the source package:
~~~
(cd /opt/gnudip && \
wget http://gnudip2.sourceforge.net/gnudip-www/src/gnudip-2.3.5.tar.gz)
~~~

Unpack the archive in the recently created directory:
~~~
(cd /opt/gnudip && \
tar xzf /opt/gnudip/gnudip-2.3.5.tar.gz -C /opt/gnudip --strip 2)
~~~

To me, the directory structure of the source package is a bit weird. That's why I reorder a little:
~~~
mkdir /opt/gnudip/doc
tar xzf /opt/gnudip/gnudip-2.3.5.tar.gz -C /opt/gnudip/doc --strip 1
rm -r /opt/gnudip/doc/gnudip
~~~

In `/opt/gnudip/etc/` the new dns-update-keys have to be created for later use with `bind9` nameserver:
~~~
(cd /opt/gnudip/etc && \
dnssec-keygen -r /dev/urandom -a hmac-md5 -b 128 -n HOST gnudip-key)
~~~

This results in two files called `/opt/gnudip/etc/Kgnudip-key.+157+{random_id}.key` and `/opt/gnudip/etc/Kgnudip-key.+157+{random_id}.private`.

GnuDIP's configuration needs to know about the new `private`-file, so the following line in `/opt/gnudip/etc/gnudip.conf` has to be adjusted accordingly:
~~~
nsupdate = -k /opt/gnudip/etc/Kgnudip-key.+157+{random_id}.private
~~~

Database preparation

Given an adequately configured mysql-server, the file `/opt/gnudip/doc/gnudip.mysql` needs to be edited in order to become compatible with recent versions (5.*) of mysql-server:
~~~
(cd /opt/gnudip/doc && \
sed -i "s/\(.*\)default '0'\(.*\)auto_increment,/\1\2auto_increment,/" gnudip.mysql)
~~~

Now, the password for the GnuDIP database has to be set in `/opt/gnudip/doc/gnudip.mysql`. The relevant parts are at:
~~~
grant select, insert, update, delete on gnudip2.to gnudip identified by '{your_password}';
grant select, insert, update, delete on gnudip2.to gnudip@localhost identified by '{your_password}';
~~~
The same password has to be set in `/opt/gnudip/etc/gnudip.conf` at the line containing:
~~~
gnudipdatabase = {your_password}
~~~

Now, the mysql database and tables are created:
~~~
(cd /opt/gnudip/doc && mysql -p < gnudip.mysql) ~~~

Configuration of bind9 nameserver

The update-keys created earlier in GnuDIP's etc-directory have to introduced to `bind9`. Therefore, the key portion in `/opt/gnudip/etc/Kgnudip-key.+157+{random_id}.key` (cryptic string at the end of the file, assuming `8hmy3QYvyfLb8ZB3D5MHsf==` here) is extracted and saved as `/etc/bind/gnudip-key`:
~~~
key gnudip-key {
algorithm hmac-md5;
// the TSIG key
secret "8hmy3QYvyfLb8ZB3D5MHsf==";
};
~~~

A new zone-file for the dynamic domain has be created at `/etc/bind/db.ddns{yourdomain.tld}`. Beware not to mess with the dots:
~~~~
$TTL 86400 ; default TTL (1 day)
@ IN SOA ns1.ddns.{yourdomain.tld}. root.{yourdomain.tld}. (
0 ; serial
3600 ; refresh (1 hour)
1800 ; retry (30 minutes)
604800 ; expire (1 week)
0 ; TTL for NACK-s (0 seconds)
)
IN NS ns1.ddns.{yourdomain.tld}.
ns1 IN A {your_ip_address}
~~~~

Now, the key-file and the zone-file have to be introduced to `bind9`. Therefore the following lines have to be added to the end of `/etc/bind/named.conf`:

~~~
// include definition of GnuDIP update key
include "/etc/bind/gnudip-key";

// define GnuDIP dynamic DNS zone
zone "ddns.{yourdomain.tld}" in {
type master;
file "/etc/bind/db.ddns.{yourdomain.tld}";
allow-query { any; };
update-policy { grant gnudip-key subdomain ddns.{yourdomain.tld}; };
};
~~~

In order to create new journal-files for the dynamic domain, `bind9` will need write access to its configuration directory:
~~~
chown bind /etc/bind
~~~

The nameserver now needs a restart:
~~~
service bind9 restart
~~~

A short test is useful in order to proof the correct implementation until now. First, the connection with the new ddns-nameserver is tested:

~~~
$ dig test.ddns.{yourdomain.tld}

; <<>> DiG 9.6.0-APPLE-P2 <<>> test.ddns.{yourdomain.tld}
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 58181 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0 ;; QUESTION SECTION: ;test.ddns.{yourdomain.tld}. IN A ;; AUTHORITY SECTION: ddns.{yourdomain.tld}. 0 IN SOA ns1.ddns.{yourdomain.tld}. root.{yourdomain.tld}. 19 3600 1800 604800 0 ;; Query time: 11 msec ;; SERVER: 10.10.10.254#53(10.10.10.254) ;; WHEN: Sun Mar 6 16:38:22 2011 ;; MSG SIZE rcvd: 85 ~~~ Now, the update function is tested: ~~~ $ /usr/bin/nsupdate -v -k /opt/gnudip/etc/Kgnudip-key.+157+{random_id}.private > update add test.ddns.{yourdomain.tld}. 0 A 127.0.0.1
>
> $
~~~

Running the first command again should now resolve `test.ddns.{yourdomain.tld}` to `127.0.0.1`.

Configuration of Apache webserver

In this scenario, `Apache 2` is used to serve the webfrontend for the GnuDIP admin as well as for its users. Also, access to web-application is supposed to run over a ssl-secured connection.

Activate ssl in Ubuntu's `Apache` (load module, activate ssl, restart daemon):
~~~
a2enmod ssl
a2ensite default-ssl
service apache2 restart
~~~

The following lines have to be added to `/etc/apache2/sites-enabled/default-ssl`:
~~~
RedirectMatch ^/gnudip(\/*)$ https://ns1.ddns.{yourdomain.tld}/gnudip/cgi-bin/gnudip.cgi
Alias /gnudip/html/ /opt/gnudip/html/

Options Indexes
ReadmeName .README
HeaderName .HEADER
RemoveHandler .pl
RemoveType .pl
AddType text/plain .pl

ScriptAlias /gnudip/cgi-bin/ /opt/gnudip/cgi-bin/
~~~

`Apache` has to have access to the files above:
~~~
chown -R www-data:www-data /opt/gnudip
~~~

The initial admin-user is created using the following script:
~~~
/opt/gnudip/sbin/gdipadmin.pl -u {admin_name} {admin_password}
~~~

This user can now connect to GnuDIP's webfrontend available at:
~~~
https://ns1.ddns.{yourdomain.tld}/gnudip
~~~

Certainly, it is useful to reconfigure `ns1.ddns.{yourdomain.tld}` to something more memorable. The configuration options there are self explanatory and are not further documented at this point.

`Apache` needs a restart:
~~~
service apache2 restart
~~~

Configuration of xinetd

To be able to use update clients like e.g. `ez-ipupdate` a GnuDIP server on port 3495/tcp has be be set up. This is done by following these steps:

Introduction of port 3495 to the system is done by adding this line to `/etc/services`:
~~~
gnudip 3495/tcp
~~~

The file `/etc/xinetd.d/gnudip` has to be created with this content:
~~~
service gnudip
{
flags = REUSE
socket_type = stream
protocol = tcp
wait = no
user = www-data
server = /opt/gnudip/sbin/gdipinet.pl
bind = 0.0.0.0
}
~~~

`xinetd` needs a restart now:
~~~
service xinetd restart
~~~

Client configuration

A valid update by `ez-ipupdate` may use the following command where [`dyndns_name` == `username`]:

~~~
ez-ipupdate -i {external_interface} -h {dyndns_name}.ddns.{yourdomain.tld} -S gnudip \
-u {username}:{password} -s {address_of_ddns-server}
~~~

Aside from the aforementioned client software GnuDIP comes with command line-based clients of its own in `/opt/gnudip/client`.

An elegant way of updating the dynamic dns entry is GnuDIP's "Auto URL"-function which offers a web-based update-method with an auth-cookie. Using this, a bookmark in a webbrowser can offer a shortcut to a quick IP-update. Also, commandline based browsers like `w3m` or `elinks` can be used to automatically update IPs where no ip-update-client is available.

Simple activation of SSL for Apache2 in Ubuntu 10.10

Just a few simple steps are necessary in order to activate ssl on apache-daemon in Ubuntu 10.10:

Load module:

a2enmod ssl

Activate ssl site config:

a2ensite default-ssl

Restart apache:

service apache2 restart

Adjust the Number of Loop Devices

If you work intensively with virtualisation, vpns and other stuff that makes use of loopback-devices, the default number of 8 in distros like Debian and Ubuntu is pretty low. This article explains how to increase its number:

In /etc/modules change

loop

to

loop max_loop=64

After that, run the following command:

for i in $(seq 0 63); do
mknod -m0660 /dev/loop$i b 7 $i
chown root.disk /dev/loop$i
done

Using vmbuilder with libvirt

In order to use `vmbuilder` (in my case `python-vm-builder 0.12.4`) to generate fully functional Ubuntu kvm-images, the following adjustments to the libvirt-template of vmbuilder have to be made in order to work.

First, make a dir-copy of `/etc/vmbuilder/libvirt/` to a location that suits you and use it by handing over the `--templates`-option to vmbuilder.

The following lines of `libvirt/libvirtxml.tmpl` have to be changed:

#if $mac

#end if

has to be moved just in front of

~~~

~~~

The line

~~~

~~~

has to become

~~~

~~~

Add the following lines right before the closing `devices`-tag: