Skip to contentSkip to conversation

automount 4.1.3: unmount timeout not working

DISCUSSION Pavel Troller 4 participants 12 messages Jan 22, 2005 – Jan 29, 2005 2 messages not in the archive Kernel automounter

lore
Share & tools
lore
#0 lore raw

Hello to all the list members,
I decided to write to this list after a lot of time spent by trying to
find an answer for my question, but I didn't find any so I'm trying the luck here.
I'm using automount 4.1.3 with kernel 2.6.10, but the problem appeared
formerly, with 2.6.7, and it's not directly related to the kernel, because it didn't appear with a kernel change, but with some other change in my system or setup, which I cannot track down.
The problem itself is that the autoumount doesn't unmount automatically after
the timeout expires. Of course the mounted volume is not used anymore. Manual umnounting with umount /mnt/volume (by root) works. Forced unmounting by sending SIGUSR1 to automount works too.
When a verbose/debug mode is used, there is no message indicating that
a timeout expired. It simply seems that the timeout is not triggered at all.

I'm inserting an excerpt of the system log made by automount in debug mode:

Jan 22 22:26:51 arcus automount[16106]: starting automounter version 4.1.3, path = /mnt, maptype = file, mapname = /etc/autofs/auto.mnt
Jan 22 22:26:51 arcus automount[16106]: mount(bind): bind_works = 1
Jan 22 22:26:51 arcus automount[16106]: using kernel protocol version 4.05
Jan 22 22:26:51 arcus automount[16106]: using timeout 2 seconds; freq 1 secs
Jan 22 22:26:52 arcus automount[16106]: sig 14 switching from 1 to 2
Jan 22 22:26:52 arcus automount[16106]: get_pkt: state 1, next 2
Jan 22 22:26:52 arcus automount[16106]: st_expire(): state = 1
Jan 22 22:26:52 arcus automount[16106]: expire_proc: exp_proc=16118
Jan 22 22:26:52 arcus automount[16106]: handle_child: got pid 16118, sig 0 (0), stat 0
Jan 22 22:26:52 arcus automount[16106]: sigchld: exp 16118 finished, switching from 1 to 1
Jan 22 22:26:52 arcus automount[16106]: get_pkt: state 2, next 1
Jan 22 22:26:52 arcus automount[16106]: st_ready(): state = 2
Jan 22 22:27:10 arcus automount[16106]: handle_packet: type = 0
Jan 22 22:27:10 arcus automount[16106]: handle_packet_missing: token 24, name udfcd
Jan 22 22:27:10 arcus automount[16106]: attempting to mount entry /mnt/udfcd
Jan 22 22:27:10 arcus automount[16146]: lookup(file): udfcd -> -fstype=udf,rw,noatime,gid=32,umask=002^I:/dev/pktcdvd/0
[Details about parsing map files and output from packet writing driver removed]
Jan 22 22:27:14 arcus kernel: UDF-fs INFO UDF 0.9.8.1 (2004/29/09) Mounting volume 'LinuxUDF', timestamp 2004/11/15 17:24 (103c)
Jan 22 22:27:14 arcus automount[16146]: mount(generic): mounted /dev/pktcdvd/0 type udf on /mnt/udfcd
Jan 22 22:27:14 arcus automount[16106]: handle_child: got pid 16146, sig 0 (0), stat 0
Jan 22 22:27:14 arcus automount[16106]: sig_child: found pending iop pid 16146: signalled 0 (sig 0), exit status 0
Jan 22 22:27:14 arcus automount[16106]: send_ready: token=24

The above log was made by starting the automounter and entering "ls /mnt/udfcd". 2 seconds after these messages, another ones indicating expiration of the mount shoud appear, but they don't, and the volume is still mounted. Now I'm killing with SIGUSR1:

Jan 22 22:45:09 arcus automount[16106]: sig 10 switching from 1 to 3
Jan 22 22:45:09 arcus automount[16106]: get_pkt: state 1, next 3
Jan 22 22:45:09 arcus automount[16106]: st_prune(): state = 1
Jan 22 22:45:09 arcus automount[16106]: expire_proc: exp_proc=17400
Jan 22 22:45:09 arcus automount[16106]: handle_packet: type = 2
Jan 22 22:45:09 arcus automount[16106]: handle_packet_expire_multi: token 25, name udfcd
Jan 22 22:45:09 arcus automount[17401]: expiring path /mnt/udfcd
Jan 22 22:45:09 arcus automount[17401]: umount_multi: path=/mnt/udfcd incl=1
Jan 22 22:45:09 arcus automount[17401]: umount_multi: unmounting dir=/mnt/udfcd
Jan 22 22:45:13 arcus automount[17401]: rm_unwanted: /mnt/udfcd
Jan 22 22:45:13 arcus automount[17401]: expired /mnt/udfcd
Jan 22 22:45:13 arcus automount[16106]: handle_child: got pid 17401, sig 0 (0), stat 0
Jan 22 22:45:13 arcus automount[16106]: sig_child: found pending iop pid 17401: signalled 0 (sig 0), exit status 0
Jan 22 22:45:13 arcus automount[16106]: send_ready: token=25
Jan 22 22:45:13 arcus automount[16106]: handle_child: got pid 17400, sig 0 (0), stat 0
Jan 22 22:45:13 arcus automount[16106]: sigchld: exp 17400 finished, switching from 3 to 1
Jan 22 22:45:13 arcus automount[16106]: get_pkt: state 3, next 1
Jan 22 22:45:13 arcus automount[16106]: st_ready(): state = 3

I'm really unhappy with this, automount worked for a long time for me and now such a problem... Any help would be greatly appreciated! And please, Cc: me, I'm not subscribed to the list. Many thanks in advance!

With regards, Pavel Troller

#1 lore raw
Quoted 31 lines · Pavel Troller

Hello to all the list members,
I decided to write to this list after a lot of time spent by trying to
find an answer for my question, but I didn't find any so I'm trying the luck here.
I'm using automount 4.1.3 with kernel 2.6.10, but the problem appeared
formerly, with 2.6.7, and it's not directly related to the kernel, because it didn't appear with a kernel change, but with some other change in my system or setup, which I cannot track down.
The problem itself is that the autoumount doesn't unmount automatically after
the timeout expires. Of course the mounted volume is not used anymore. Manual umnounting with umount /mnt/volume (by root) works. Forced unmounting by sending SIGUSR1 to automount works too.
When a verbose/debug mode is used, there is no message indicating that
a timeout expired. It simply seems that the timeout is not triggered at all.

I'm inserting an excerpt of the system log made by automount in debug mode:

Jan 22 22:26:51 arcus automount[16106]: starting automounter version 4.1.3, path = /mnt, maptype = file, mapname = /etc/autofs/auto.mnt
Jan 22 22:26:51 arcus automount[16106]: mount(bind): bind_works = 1
Jan 22 22:26:51 arcus automount[16106]: using kernel protocol version 4.05
Jan 22 22:26:51 arcus automount[16106]: using timeout 2 seconds; freq 1 secs
Jan 22 22:26:52 arcus automount[16106]: sig 14 switching from 1 to 2
Jan 22 22:26:52 arcus automount[16106]: get_pkt: state 1, next 2
Jan 22 22:26:52 arcus automount[16106]: st_expire(): state = 1
Jan 22 22:26:52 arcus automount[16106]: expire_proc: exp_proc=16118
Jan 22 22:26:52 arcus automount[16106]: handle_child: got pid 16118, sig 0 (0), stat 0
Jan 22 22:26:52 arcus automount[16106]: sigchld: exp 16118 finished, switching from 1 to 1
Jan 22 22:26:52 arcus automount[16106]: get_pkt: state 2, next 1
Jan 22 22:26:52 arcus automount[16106]: st_ready(): state = 2
Jan 22 22:27:10 arcus automount[16106]: handle_packet: type = 0
Jan 22 22:27:10 arcus automount[16106]: handle_packet_missing: token 24, name udfcd
Quoted 9 lines
Jan 22 22:27:10 arcus automount[16106]: attempting to mount entry /mnt/udfcd
Jan 22 22:27:10 arcus automount[16146]: lookup(file): udfcd -> -fstype=udf,rw,noatime,gid=32,umask=002^I:/dev/pktcdvd/0
[Details about parsing map files and output from packet writing driver removed]
Jan 22 22:27:14 arcus kernel: UDF-fs INFO UDF 0.9.8.1 (2004/29/09) Mounting volume 'LinuxUDF', timestamp 2004/11/15 17:24 (103c)
Jan 22 22:27:14 arcus automount[16146]: mount(generic): mounted /dev/pktcdvd/0 type udf on /mnt/udfcd
Jan 22 22:27:14 arcus automount[16106]: handle_child: got pid 16146, sig 0 (0), stat 0
Jan 22 22:27:14 arcus automount[16106]: sig_child: found pending iop pid 16146: signalled 0 (sig 0), exit status 0
Jan 22 22:27:14 arcus automount[16106]: send_ready: token=24

But you leave out the crucial log trace following the mount?

Can we see more of what follows here please.

Ian

#2 lore raw
Quoted 6 lines

But you leave out the crucial log trace following the mount?

Can we see more of what follows here please.

Ian

Hello Ian,
OK, I'm sending the full system log during the mount/umount... I don't know
which part You are interested in, but I don't see anything special...
BTW using the UDF packet mode is just an example, umount timeout doesn't
work on vfat floppies or iso9660 cd's as well...
With regards, Pavel

... automount startup log not included, it' identical to the one sent earlier [ls /mnt/udfcd issued]

Jan 26 20:33:32 arcus automount[1989]: handle_packet: type = 0
Jan 26 20:33:32 arcus automount[1989]: handle_packet_missing: token 3, name udfcd
Jan 26 20:33:32 arcus automount[1989]: attempting to mount entry /mnt/udfcd
Jan 26 20:33:32 arcus automount[3411]: lookup(file): udfcd -> -fstype=udf,rw,noatime,gid=32,umask=002^I:/dev/pktcdvd/0
Jan 26 20:33:32 arcus automount[3411]: parse(sun): expanded entry: -fstype=udf,rw,noatime,gid=32,umask=002^I:/dev/pktcdvd/0
Jan 26 20:33:32 arcus automount[3411]: parse(sun): dequote("fstype=udf,rw,noatime,gid=32,umask=002") -> fstype=udf,rw,noatime,gid=32,umask=002
Jan 26 20:33:32 arcus automount[3411]: parse(sun): gathered options: fstype=udf,rw,noatime,gid=32,umask=002
Jan 26 20:33:32 arcus automount[3411]: parse(sun): dequote("/dev/pktcdvd/0") -> /dev/pktcdvd/0
Jan 26 20:33:32 arcus automount[3411]: parse(sun): core of entry: options=fstype=udf,rw,noatime,gid=32,umask=002, loc=/dev/pktcdvd/0Jan 26 20:33:32 arcus automount[3411]: parse(sun): mounting root /mnt, mountpoint udfcd, what /dev/pktcdvd/0, fstype udf, options rw,noatime,gid=32,umask=002
Jan 26 20:33:32 arcus automount[3411]: do_mount /dev/pktcdvd/0 /mnt/udfcd type udf options rw,noatime,gid=32,umask=002 using module
generic
Jan 26 20:33:32 arcus automount[3411]: mount(generic): calling mkdir_path /mnt/udfcd
Jan 26 20:33:32 arcus automount[3411]: mount(generic): calling mount -t udf -s -o rw,noatime,gid=32,umask=002 /dev/pktcdvd/0 /mnt/udfcd
Jan 26 20:33:32 arcus /sbin/hotplug: no runnable /etc/hotplug/module.agent is installed
Jan 26 20:33:32 arcus /sbin/hotplug: no runnable /etc/hotplug/module.agent is installed
Jan 26 20:33:32 arcus kernel: pktcdvd: inserted media is CD-RW
Jan 26 20:33:33 arcus kernel: pktcdvd: Fixed packets, 32 blocks, Mode-2 disc
Jan 26 20:33:33 arcus kernel: pktcdvd: Max. media speed: 16
Jan 26 20:33:33 arcus kernel: pktcdvd: write speed 16x
Jan 26 20:33:33 arcus kernel: pktcdvd: 590526kB available on disc
Jan 26 20:33:36 arcus kernel: UDF-fs INFO UDF 0.9.8.1 (2004/29/09) Mounting volume 'LinuxUDF', timestamp 2004/11/15 17:24 (103c)
Jan 26 20:33:36 arcus automount[3411]: mount(generic): mounted /dev/pktcdvd/0 type udf on /mnt/udfcd
Jan 26 20:33:36 arcus automount[1989]: handle_child: got pid 3411, sig 0 (0), stat 0
Jan 26 20:33:36 arcus automount[1989]: sig_child: found pending iop pid 3411: signalled 0 (sig 0), exit status 0
Jan 26 20:33:36 arcus automount[1989]: send_ready: token=3
[Nothing more logged for the mount action]
[umount had to occur 2 seconds later]
Jan 26 20:35:19 arcus login[18647]: ROOT LOGIN on `vc/1'
[root is killing automount with USR1] (also contained in previous mail)
Jan 26 20:35:33 arcus automount[1989]: sig 10 switching from 1 to 3
Jan 26 20:35:33 arcus automount[1989]: get_pkt: state 1, next 3
Jan 26 20:35:33 arcus automount[1989]: st_prune(): state = 1
Jan 26 20:35:33 arcus automount[1989]: expire_proc: exp_proc=3574
Jan 26 20:35:33 arcus automount[1989]: handle_packet: type = 2
Jan 26 20:35:33 arcus automount[1989]: handle_packet_expire_multi: token 4, name udfcd
Jan 26 20:35:33 arcus automount[3575]: expiring path /mnt/udfcd
Jan 26 20:35:33 arcus automount[3575]: umount_multi: path=/mnt/udfcd incl=1
Jan 26 20:35:33 arcus automount[3575]: umount_multi: unmounting dir=/mnt/udfcd
#3 lore raw
Quoted 9 lines · Pavel Troller
Quoted 6 lines

But you leave out the crucial log trace following the mount?

Can we see more of what follows here please.

Ian

Hello Ian,
OK, I'm sending the full system log during the mount/umount... I don't know
which part You are interested in, but I don't see anything special...

Maybe because what I'm looking for is not there.
So this is probably the cause of the probem.

I think that what is happening is that for the short timeouts you are using the expire timer is being set to 0, disabling expiration.

I'll have a look but does this work if you set the timeout to 4 seconds or above?

Ian

#4 lore raw
Quoted 13 lines · Pavel Troller
Quoted 11 lines
Quoted 7 lines

But you leave out the crucial log trace following the mount?

Can we see more of what follows here please.

Ian

Hello Ian,
OK, I'm sending the full system log during the mount/umount... I don't know
which part You are interested in, but I don't see anything special...

Quoted 9 lines

Maybe because what I'm looking for is not there.
So this is probably the cause of the probem.

I think that what is happening is that for the short timeouts you are using the expire timer is being set to 0, disabling expiration.

I'll have a look but does this work if you set the timeout to 4 seconds or above?

Hi Ian,
I increased the timeout to 10 seconds for the test. No change has been
observed, still no auto umount.
I would like to remember that it worked perfectly and then stopped working
probably after upgrading some part of this system. I'm using exactly the same setup on other computers with no problems. Timeout is set to 3 there, it was 3 here too, but I tried to change the value after it ceased to work. So definitely, it's not the cause.
The only bigger change I remember I did was upgrading bash to 3.0. But I
cannot imagine how could changing a shell cause such a problem. I even tried to put the old bash back and test it and it didn't work too.
Please tell me where to put some debug texts to be logged or so, I can
modify the source and cooperate with finding the bug.
With regards, Pavel

#5 lore raw
Quoted 22 lines · Pavel Troller
Quoted 13 lines · Pavel Troller
Quoted 11 lines
Quoted 7 lines

But you leave out the crucial log trace following the mount?

Can we see more of what follows here please.

Ian

Hello Ian,
OK, I'm sending the full system log during the mount/umount... I don't know
which part You are interested in, but I don't see anything special...

Quoted 9 lines

Maybe because what I'm looking for is not there.
So this is probably the cause of the probem.

I think that what is happening is that for the short timeouts you are using the expire timer is being set to 0, disabling expiration.

I'll have a look but does this work if you set the timeout to 4 seconds or above?

Quoted 13 lines

Hi Ian,
I increased the timeout to 10 seconds for the test. No change has been
observed, still no auto umount.
I would like to remember that it worked perfectly and then stopped working
probably after upgrading some part of this system. I'm using exactly the same setup on other computers with no problems. Timeout is set to 3 there, it was 3 here too, but I tried to change the value after it ceased to work. So definitely, it's not the cause.
The only bigger change I remember I did was upgrading bash to 3.0. But I
cannot imagine how could changing a shell cause such a problem. I even tried to put the old bash back and test it and it didn't work too.
Please tell me where to put some debug texts to be logged or so, I can
modify the source and cooperate with finding the bug.

What I'm looking for is log entries after the mount happens.
Where you see the alarm firing and expire checks happening.
Basically, you don't have any expiration happening (obviously) so the alarm timer must be set to 0. We need to establish why.

So using the --debug should give you that.
Include the log from each test please and I'll put together a patch to log some extra stuff.

I'll go back through you mails to check what version your using.

Ian

#6 lore raw
Quoted 5 lines

What I'm looking for is log entries after the mount happens. Where you see the alarm firing and expire checks happening. Basically, you don't have any expiration happening (obviously) so the alarm timer must be set to 0. We need to establish why.

Quoted 7 lines

So using the --debug should give you that.
Include the log from each test please and I'll put together a patch to log some extra stuff.

I'll go back through you mails to check what version your using.

Ian

Hi Ian!
Thanks for Your great care, I hope that we'll fix it soon! To save your time:

root@arcus:~# automount --version
Linux automount version 4.1.3
  And my /etc/init.d/autofs already contains
#       The automount options may be added here
#       e.g. amoptions='-t 15'
amoptions='--timeout 3 -d'
  so I think that the debug flag is already on.
                                      With regards, Pavel
#7 lore raw
Quoted 15 lines · Pavel Troller
Quoted 5 lines

What I'm looking for is log entries after the mount happens. Where you see the alarm firing and expire checks happening. Basically, you don't have any expiration happening (obviously) so the alarm timer must be set to 0. We need to establish why.

Quoted 7 lines

So using the --debug should give you that.
Include the log from each test please and I'll put together a patch to log some extra stuff.

I'll go back through you mails to check what version your using.

Ian

Hi Ian!
Thanks for Your great care, I hope that we'll fix it soon! To save your time:

Quoted 9 lines
root@arcus:~# automount --version
Linux automount version 4.1.3
 And my /etc/init.d/autofs already contains
#       The automount options may be added here
#       e.g. amoptions='-t 15'
amoptions='--timeout 3 -d'
 so I think that the debug flag is already on.
                                     With regards, Pavel

OK. How about the log with a timeout of 10.

Ian

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

#10 lore raw
Quoted 18 lines · Pavel Troller
Quoted 8 lines
Quoted 5 lines

OK. How about the log with a timeout of 10.

Ian

OK, here is it. Classic iso cd used instead of udf. Sending just to You not to spam the list.

Can you try this patch please.

--- autofs-4.1.3-michael/daemon/automount.c.sig-fix 2005-01-19
20:11:09.000000000 +0800
+++ autofs-4.1.3-michael/daemon/automount.c 2005-01-19
20:12:02.000000000 +0800

Review line 927

Quoted 9 lines
(quoted diff) +1 -1
@@ -920,8 +920,8 @@
 		return 1;
 
 	case EXP_STARTED:
-		sigprocmask(SIG_SETMASK, &ready_sigs, NULL);
 		ap.state = ST_EXPIRE;
+		sigprocmask(SIG_SETMASK, &ready_sigs, NULL);
 		return 0;
 	}
 	return 1;

Hi Ian!
It's unbelievable: IT WORKS!!!!
In 3 sec intervals, the following now appears in the log:

Jan 29 13:19:28 arcus automount[13950]: sig 14 switching from 1 to 2
Jan 29 13:19:28 arcus automount[13950]: get_pkt: state 1, next 2
Jan 29 13:19:28 arcus automount[13950]: st_expire(): state = 1
Jan 29 13:19:28 arcus automount[13950]: expire_proc: exp_proc=14094
Jan 29 13:19:28 arcus automount[13950]: handle_child: got pid 14094, sig 0 (0), stat 0
Jan 29 13:19:28 arcus automount[13950]: sigchld: exp 14094 finished, switching from 2 to 1
Jan 29 13:19:28 arcus automount[13950]: get_pkt: state 2, next 1
Jan 29 13:19:28 arcus automount[13950]: st_ready(): state = 2

And if there is a filesystem to umount, it's umounted in a flash! I even changed the timeout to my favourite 3 seconds and it works too! I can't explain it to myself: How such a simple change in the sequence of
the code can have such a big impact ?
And why it works to most of people ? Really strange :-).
But never mind, thank You very much! It's really great to have the automount
functional again!
With neverending thanks, Pavel

#11 lore raw
Quoted 2 lines · Pavel Troller

Hi Ian!
It's unbelievable: IT WORKS!!!!

It took me a while to pick it first time.

Your the second win with this patch.
What I don't understand is why all of a sudden it starts happening.

Quoted 11 lines

In 3 sec intervals, the following now appears in the log:

Jan 29 13:19:28 arcus automount[13950]: sig 14 switching from 1 to 2
Jan 29 13:19:28 arcus automount[13950]: get_pkt: state 1, next 2
Jan 29 13:19:28 arcus automount[13950]: st_expire(): state = 1
Jan 29 13:19:28 arcus automount[13950]: expire_proc: exp_proc=14094
Jan 29 13:19:28 arcus automount[13950]: handle_child: got pid 14094, sig 0 (0), stat 0
Jan 29 13:19:28 arcus automount[13950]: sigchld: exp 14094 finished, switching from 2 to 1
Jan 29 13:19:28 arcus automount[13950]: get_pkt: state 2, next 1
Jan 29 13:19:28 arcus automount[13950]: st_ready(): state = 2
Quoted 4 lines

And if there is a filesystem to umount, it's umounted in a flash! I even changed the timeout to my favourite 3 seconds and it works too! I can't explain it to myself: How such a simple change in the sequence of
the code can have such a big impact ?

It's a classic race. I think the state is ending up not being set correctly as signals are being unlocked to early. Consequently, automount doesn't pass through the right places to set the alarm for the expire.

Ian

#12 lore raw
In reply to Pavel Troller
Quoted 8 lines · Pavel Troller

Hello to all the list members,
I decided to write to this list after a lot of time spent by trying to
find an answer for my question, but I didn't find any so I'm trying the luck here.
I'm using automount 4.1.3 with kernel 2.6.10, but the problem appeared
formerly, with 2.6.7, and it's not directly related to the kernel, because it didn't appear with a kernel change, but with some other change in my system or setup, which I cannot track down.

I also see this (Debian Sarge, installation done some weeks ago, package version is 4.1.3-4). WIth an older installation (another Sarge), still using 3.9.99-4.0.0pre10-16, this doesn't show up.

Quoted 2 lines

The problem itself is that the autoumount doesn't unmount automatically after
the timeout expires.

An upgrade to 4.1.3-8 apparently cured the problem; I don't know how close the Debian package maintainer Steiner Gundersson works with autofs people, and whether he's introduced some patches that have not been in the previous version (and may have found their way into 4.1.4). Sure, Debian Sarge is called "testing", that's what I'm doing, it's not "stable" yet (but there's still hope), but I'd like to see a close cooperation between the mainstream maintainers and the maintainers of Linux distros...

Cheers,
steffen

Signature · 4 lines
Steffen Grunewald * * * Merlin cluster admin (http://pandora.aei.mpg.de)
Albert-Einstein-Institut (MPI Gravitationsphysik, http://www.aei.mpg.de)
       Science Park Golm, Am Mühlenberg 1, 14476 Potsdam, Germany
e-mail: steffen.grunewald(*)aei.mpg.de * +49-331-567-{fon:7233,fax:7298}
#13 lore raw
Quoted 3 lines · Steffen Grunewald

I also see this (Debian Sarge, installation done some weeks ago, package version is 4.1.3-4). WIth an older installation (another Sarge), still using 3.9.99-4.0.0pre10-16, this doesn't show up.

4.1.3-4 is old, there has bene _lots_ of changes since then. The latest version in sarge is 4.1.3-8; 4.1.3-9 (latest in sid) is broken due to a build problem, and only fixes a minor issue anyhow.

Quoted 4 lines

An upgrade to 4.1.3-8 apparently cured the problem; I don't know how close the Debian package maintainer Steiner Gundersson works with autofs people, and whether he's introduced some patches that have not been in the previous version (and may have found their way into 4.1.4).

I'm at least here on the list, and we're currently trying to sync most of the Debian packages into 4.1.4 (upstream has already done most of them); see the other discussion today. :-)

/* Steinar */

Signature · 1 line
Homepage: http://www.sesse.net/