Skip to contentSkip to conversation

bug and fun with XFS: unable to handle kernel NULL pointer dereference

DISCUSSION Michael Monnerie 2 participants 5 messages Jul 25, 2010 – Jul 26, 2010

lore
Share & tools
lore

Michael Monnerie

author
#0 lore raw

I just enjoy an obviously broken XFS filesystem. It was a running server, which I planned to migrate so I did "rsync -aHAX / otherhost::rsyncmodule", and experienced a "killed". At that time I thought it was a one time mistake, so restarted rsync, but Murphy made it get killed again.

So I looked into dmesg, just to find this: It's the log of all messages, so maybe twice the same, I copy everything for reference. See attachment "xfs-bug.dmesg.txt".

I started to look, and quickly found a funny problem: Once I mount that partition, I cannot unmount it again:

# mount /disks/work/
# umount /disks/work/
umount: /disks/work: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))

So I rebooted without mounting that partition, and

# xfs_repair -n /dev/xvda2 [VERSION:3.1.2]
xfs_repair: /lib64/libuuid.so.1: no version information available 
(required by xfs_repair)                                                                                                                                          
Phase 1 - find and verify superblock...                                                                                                                                                                                             
Phase 2 - using internal log                                                                                                                                                                                                        
        - scan filesystem freespace and inode maps...                                                                                                                                                                               
        - found root inode chunk
Phase 3 - for each AG...
        - scan (but don't clear) agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 1
local inode 8636461 attr too small (size = 0, min size = 4)
bad attribute fork in inode 8636461, would clear attr fork
would have cleared inode 8636461
        - agno = 2
        - agno = 3
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - check for inodes claiming duplicate blocks...
        - agno = 0
        - agno = 1
        - agno = 3
local inode 8636461 attr too small (size = 0, min size = 4)
bad attribute fork in inode 8636461, would clear attr fork
would have cleared inode 8636461
        - agno = 2
No modify flag set, skipping phase 5
Phase 6 - check inode connectivity...
        - traversing filesystem ...
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
Phase 7 - verify link counts...
No modify flag set, skipping filesystem flush and exiting.

xfs_repair corrected it painlessly, and everything fine again. Just wanted to report that a simple mount works and an immediate umount fails. Maybe this could be fixed, would make repair simpler.

Signature · 13 lines
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services
http://proteger.at [gesprochen: Prot-e-schee]
Tel: 0660 / 415 65 31

****** Aktuelles Radiointerview! ******
http://www.it-podcast.at/aktuelle-sendung.html

// Wir haben im Moment zwei Häuser zu verkaufen:
// http://zmi.at/langegg/
// http://zmi.at/haus2009/
BUG: unable to handle kernel NULL pointer dereference at 0000000000000002
IP: [<ffffffffa0066e0e>] xfs_attr_shortform_getvalue+0x24/0xda [xfs]
PGD 692a5067 PUD 69f6c067 PMD 0 
Oops: 0000 [1] SMP 
last sysfs file: /sys/devices/virtual/net/lo/type
CPU 1 
Modules linked in: binfmt_misc nfs lockd nfs_acl sunrpc ipv6 fuse loop dm_mod rtc_core rtc_lib xennet mptspi mptscsih mptbase scsi_transport_spi scsi_mod xfs reiserfs thermal_sys hwmon xenblk cdrom
Supported: Yes
Pid: 1809, comm: syslog-ng Not tainted 2.6.27.48-0.1-xen #1
RIP: e030:[<ffffffffa0066e0e>]  [<ffffffffa0066e0e>] xfs_attr_shortform_getvalue+0x24/0xda [xfs]
RSP: e02b:ffff880068b63cd8  EFLAGS: 00010296
RAX: 0000000000000000 RBX: 0000000000000004 RCX: 000000007c7bfd5e
RDX: 000000007c7bc727 RSI: 0000000000000002 RDI: ffff880068b63d18
RBP: ffff880068b63d18 R08: 0000000000002008 R09: ffffffffa00d81b0
R10: ffff880068b63e68 R11: ffffffff80562f0c R12: ffff880068b63dd8
R13: 0000000000000000 R14: 0000000000002008 R15: ffff880068b63e3c
FS:  00007fae927a76f0(0000) GS:ffff8800014d5140(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process syslog-ng (pid: 1809, threadinfo ffff880068b62000, task ffff88006a3ce4c0)
Stack:  ffff880069b50600 ffff880069b50600 ffff880069b50600 ffff880068b63dd8
 0000000000000000 0000000000002008 ffff880068b63e3c ffffffffa0065b08
 ffffffff80562f15 000000000000000a 0000000000000000 0000200800000000
Call Trace:
 [<ffffffffa0065b08>] xfs_attr_fetch+0x87/0xd4 [xfs]
 [<ffffffffa0065bd6>] xfs_attr_get+0x81/0xa3 [xfs]
 [<ffffffffa00aea3f>] xfs_xattr_secure_get+0x41/0x4e [xfs]
 [<ffffffff802bb7c2>] generic_getxattr+0x62/0x66
 [<ffffffff803066db>] cap_inode_need_killpriv+0x2d/0x3b
 [<ffffffff802b3ca9>] fnotify_change+0xa6/0x334
 [<ffffffff8029d6cb>] chown_common+0x81/0x9a
 [<ffffffff8029d74b>] sys_fchown+0x67/0x91
 [<ffffffff8020b418>] system_call_fastpath+0x16/0x1b
 [<00007fae91ccfc07>] 0x7fae91ccfc07


Code: 41 5d 41 5e 41 5f c3 41 57 41 56 41 55 45 31 ed 41 54 55 48 89 fd 53 48 83 ec 08 48 8b 47 30 48 8b 40 58 48 8b 40 18 48 8d 58 04 <44> 0f b6 78 02 e9 92 00 00 00 44 0f b6 23 44 3b 65 08 75 77 48 
RIP  [<ffffffffa0066e0e>] xfs_attr_shortform_getvalue+0x24/0xda [xfs]
 RSP <ffff880068b63cd8>
CR2: 0000000000000002
---[ end trace 0b5d9a76acf21c38 ]---
BUG: unable to handle kernel NULL pointer dereference at 0000000000000002
IP: [<ffffffffa0066e0e>] xfs_attr_shortform_getvalue+0x24/0xda [xfs]
PGD 27932067 PUD 35bf2067 PMD 0 
Oops: 0000 [2] SMP 
last sysfs file: /sys/devices/xen/vif-1/modalias
CPU 0 
Modules linked in: binfmt_misc nfs lockd nfs_acl sunrpc ipv6 fuse loop dm_mod rtc_core rtc_lib xennet mptspi mptscsih mptbase scsi_transport_spi scsi_mod xfs reiserfs thermal_sys hwmon xenblk cdrom
Supported: Yes
Pid: 5866, comm: rsync Tainted: G      D   2.6.27.48-0.1-xen #1
RIP: e030:[<ffffffffa0066e0e>]  [<ffffffffa0066e0e>] xfs_attr_shortform_getvalue+0x24/0xda [xfs]
RSP: e02b:ffff880068b71be8  EFLAGS: 00010296
RAX: 0000000000000000 RBX: 0000000000000004 RCX: 00000000275b19c4
RDX: 0000000008d26645 RSI: 00000000ffffffff RDI: ffff880068b71c28
RBP: ffff880068b71c28 R08: 0000000000000002 R09: ffffffffa00d81d0
R10: ffff880068b71e08 R11: ffff880068b71e08 R12: ffff880068b71ce8
R13: 0000000000000000 R14: 0000000000000002 R15: ffff880068b71d44
FS:  00007f44789eb6f0(0000) GS:ffffffff80763080(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process rsync (pid: 5866, threadinfo ffff880068b70000, task ffff880027b30280)
Stack:  0000000000000000 ffff880069b50600 ffff880069b50600 ffff880068b71ce8
 ffff8800636e1ea8 0000000000000002 ffff880068b71d44 ffffffffa0065b08
 ffffffffa00b06d4 000000000000000c ffff8800636e1ea8 0000000200000130
Call Trace:
 [<ffffffffa0065b08>] xfs_attr_fetch+0x87/0xd4 [xfs]
 [<ffffffffa0065bd6>] xfs_attr_get+0x81/0xa3 [xfs]
 [<ffffffffa005ce96>] xfs_acl_get_attr+0x46/0x63 [xfs]
 [<ffffffffa005d613>] xfs_acl_vget+0x6b/0xec [xfs]
 [<ffffffff802bb7c2>] generic_getxattr+0x62/0x66
 [<ffffffff802bc09d>] vfs_getxattr+0xaf/0xc1
 [<ffffffff802bc155>] getxattr+0xa6/0x107
 [<ffffffff802bc2e1>] sys_getxattr+0x4c/0x67
 [<ffffffff8020b418>] system_call_fastpath+0x16/0x1b
 [<00007f4477f2ed79>] 0x7f4477f2ed79


Code: 41 5d 41 5e 41 5f c3 41 57 41 56 41 55 45 31 ed 41 54 55 48 89 fd 53 48 83 ec 08 48 8b 47 30 48 8b 40 58 48 8b 40 18 48 8d 58 04 <44> 0f b6 78 02 e9 92 00 00 00 44 0f b6 23 44 3b 65 08 75 77 48 
RIP  [<ffffffffa0066e0e>] xfs_attr_shortform_getvalue+0x24/0xda [xfs]
 RSP <ffff880068b71be8>
CR2: 0000000000000002
---[ end trace 0b5d9a76acf21c38 ]---
BUG: unable to handle kernel NULL pointer dereference at 0000000000000002
IP: [<ffffffffa0066e0e>] xfs_attr_shortform_getvalue+0x24/0xda [xfs]
PGD 6203b067 PUD 13a8c067 PMD 0 
Oops: 0000 [3] SMP 
last sysfs file: /sys/devices/xen/vif-1/modalias
CPU 0 
Modules linked in: binfmt_misc nfs lockd nfs_acl sunrpc ipv6 fuse loop dm_mod rtc_core rtc_lib xennet mptspi mptscsih mptbase scsi_transport_spi scsi_mod xfs reiserfs thermal_sys hwmon xenblk cdrom
Supported: Yes
Pid: 6042, comm: rsync Tainted: G      D   2.6.27.48-0.1-xen #1
RIP: e030:[<ffffffffa0066e0e>]  [<ffffffffa0066e0e>] xfs_attr_shortform_getvalue+0x24/0xda [xfs]
RSP: e02b:ffff880027b99be8  EFLAGS: 00010296
RAX: 0000000000000000 RBX: 0000000000000004 RCX: 00000000275b19c4
RDX: 0000000008d26645 RSI: 00000000ffffffff RDI: ffff880027b99c28
RBP: ffff880027b99c28 R08: 0000000000000002 R09: ffffffffa00d81d0
R10: ffff880027b99e08 R11: ffff880027b99e08 R12: ffff880027b99ce8
R13: 0000000000000000 R14: 0000000000000002 R15: ffff880027b99d44
FS:  00007fb4d45fc6f0(0000) GS:ffffffff80763080(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process rsync (pid: 6042, threadinfo ffff880027b98000, task ffff880027be6040)
Stack:  0000000000000001 ffff880069b50600 ffff880069b50600 ffff880027b99ce8
 ffff8800636e1068 0000000000000002 ffff880027b99d44 ffffffffa0065b08
 ffffffffa00b06d4 000000000000000c ffff8800636e1068 0000000200000130
Call Trace:
 [<ffffffffa0065b08>] xfs_attr_fetch+0x87/0xd4 [xfs]
 [<ffffffffa0065bd6>] xfs_attr_get+0x81/0xa3 [xfs]
 [<ffffffffa005ce96>] xfs_acl_get_attr+0x46/0x63 [xfs]
 [<ffffffffa005d613>] xfs_acl_vget+0x6b/0xec [xfs]
 [<ffffffff802bb7c2>] generic_getxattr+0x62/0x66
 [<ffffffff802bc09d>] vfs_getxattr+0xaf/0xc1
 [<ffffffff802bc155>] getxattr+0xa6/0x107
 [<ffffffff802bc2e1>] sys_getxattr+0x4c/0x67
 [<ffffffff8020b418>] system_call_fastpath+0x16/0x1b
 [<00007fb4d3b3fd79>] 0x7fb4d3b3fd79


Code: 41 5d 41 5e 41 5f c3 41 57 41 56 41 55 45 31 ed 41 54 55 48 89 fd 53 48 83 ec 08 48 8b 47 30 48 8b 40 58 48 8b 40 18 48 8d 58 04 <44> 0f b6 78 02 e9 92 00 00 00 44 0f b6 23 44 3b 65 08 75 77 48 
RIP  [<ffffffffa0066e0e>] xfs_attr_shortform_getvalue+0x24/0xda [xfs]
 RSP <ffff880027b99be8>
CR2: 0000000000000002
---[ end trace 0b5d9a76acf21c38 ]---
BUG: unable to handle kernel NULL pointer dereference at 0000000000000002
IP: [<ffffffffa0066e0e>] xfs_attr_shortform_getvalue+0x24/0xda [xfs]
PGD 68f43067 PUD 6921f067 PMD 0 
Oops: 0000 [4] SMP 
last sysfs file: /sys/devices/xen/vif-1/modalias
CPU 0 
Modules linked in: binfmt_misc nfs lockd nfs_acl sunrpc ipv6 fuse loop dm_mod rtc_core rtc_lib xennet mptspi mptscsih mptbase scsi_transport_spi scsi_mod xfs reiserfs thermal_sys hwmon xenblk cdrom
Supported: Yes
Pid: 6084, comm: rsync Tainted: G      D   2.6.27.48-0.1-xen #1
RIP: e030:[<ffffffffa0066e0e>]  [<ffffffffa0066e0e>] xfs_attr_shortform_getvalue+0x24/0xda [xfs]
RSP: e02b:ffff88006a381be8  EFLAGS: 00010296
RAX: 0000000000000000 RBX: 0000000000000004 RCX: 00000000275b19c4
RDX: 0000000008d26645 RSI: 00000000ffffffff RDI: ffff88006a381c28
RBP: ffff88006a381c28 R08: 0000000000000002 R09: ffffffffa00d81d0
R10: ffff88006a381e08 R11: ffff88006a381e08 R12: ffff88006a381ce8
R13: 0000000000000000 R14: 0000000000000002 R15: ffff88006a381d44
FS:  00007f23683d46f0(0000) GS:ffffffff80763080(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process rsync (pid: 6084, threadinfo ffff88006a380000, task ffff880027be6500)
Stack:  ffff880035a7ab40 ffff880069b50600 ffff880069b50600 ffff88006a381ce8
 ffff8800636e1d78 0000000000000002 ffff88006a381d44 ffffffffa0065b08
 ffffffffa00b06d4 000000000000000c ffff8800636e1d78 0000000200000130
Call Trace:
 [<ffffffffa0065b08>] xfs_attr_fetch+0x87/0xd4 [xfs]
 [<ffffffffa0065bd6>] xfs_attr_get+0x81/0xa3 [xfs]
 [<ffffffffa005ce96>] xfs_acl_get_attr+0x46/0x63 [xfs]
 [<ffffffffa005d613>] xfs_acl_vget+0x6b/0xec [xfs]
 [<ffffffff802bb7c2>] generic_getxattr+0x62/0x66
 [<ffffffff802bc09d>] vfs_getxattr+0xaf/0xc1
 [<ffffffff802bc155>] getxattr+0xa6/0x107
 [<ffffffff802bc2e1>] sys_getxattr+0x4c/0x67
 [<ffffffff8020b418>] system_call_fastpath+0x16/0x1b
 [<00007f2367917d79>] 0x7f2367917d79


Code: 41 5d 41 5e 41 5f c3 41 57 41 56 41 55 45 31 ed 41 54 55 48 89 fd 53 48 83 ec 08 48 8b 47 30 48 8b 40 58 48 8b 40 18 48 8d 58 04 <44> 0f b6 78 02 e9 92 00 00 00 44 0f b6 23 44 3b 65 08 75 77 48 
RIP  [<ffffffffa0066e0e>] xfs_attr_shortform_getvalue+0x24/0xda [xfs]
 RSP <ffff88006a381be8>
CR2: 0000000000000002
---[ end trace 0b5d9a76acf21c38 ]---

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#1 lore raw
Quoted 9 lines · Michael Monnerie

I just enjoy an obviously broken XFS filesystem. It was a running server, which I planned to migrate so I did "rsync -aHAX / otherhost::rsyncmodule", and experienced a "killed". At that time I thought it was a one time mistake, so restarted rsync, but Murphy made it get killed again.

So I looked into dmesg, just to find this: It's the log of all messages, so maybe twice the same, I copy everything for reference. See attachment "xfs-bug.dmesg.txt".

The first occurrence is:

Quoted 1 line

Pid: 1809, comm: syslog-ng Not tainted 2.6.27.48-0.1-xen #1

That's an old kernel, and doesn't seem related to the rsync triggered problem, even though it is the same oops signature.

Quoted 8 lines

I started to look, and quickly found a funny problem: Once I mount that partition, I cannot unmount it again:

# mount /disks/work/
# umount /disks/work/
umount: /disks/work: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))

Some other process has taken a reference to the fs, I'd say. And if that process triggered an oops, then you'd see this.

Quoted 17 lines

So I rebooted without mounting that partition, and

# xfs_repair -n /dev/xvda2 [VERSION:3.1.2]
xfs_repair: /lib64/libuuid.so.1: no version information available 
(required by xfs_repair)                                                                                                                                          
Phase 1 - find and verify superblock...                                                                                                                                                                                             
Phase 2 - using internal log                                                                                                                                                                                                        
        - scan filesystem freespace and inode maps...                                                                                                                                                                               
        - found root inode chunk
Phase 3 - for each AG...
        - scan (but don't clear) agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
        - agno = 1
local inode 8636461 attr too small (size = 0, min size = 4)
bad attribute fork in inode 8636461, would clear attr fork
would have cleared inode 8636461

Corrupt attribute fork - matches with the oops signatures. I'd definitely consider upgrading your kernel as a first step...

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
author
#2 lore raw

Re: permanent kernel upgrading (was: bug and fun with XFS: unable to handle kernel NULL pointer dereference)

Quoted 8 lines · On Montag, 26. Juli 2010 Dave Chinner wrote:
Quoted 5 lines
# mount /disks/work/
# umount /disks/work/
umount: /disks/work: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))

Some other process has taken a reference to the fs, I'd say. And if that process triggered an oops, then you'd see this.

This is definitely improbable ;-). I stopped all processes, except sshd and mingetty (for the console processes), and "lsof" reported no open files. I'd swear it must be a bug, but as you said 2.6.27 (openSUSE 11.1) is just too old to take care.

Quoted 2 lines

Corrupt attribute fork - matches with the oops signatures. I'd definitely consider upgrading your kernel as a first step...

That's why I wrote "which I planned to migrate". I was just in the process of doing so.

This brings me to another question: openSUSE 11.1 is, despite being old, still maintained, so I was under the impression that I'd get updates for problems. But I already received "upgrade your kernel" the last time for my openSUSE 11.2 with it's kernel vmlinuz-2.6.31.12, which by the time was the latest release (in the meantime openSUSE 11.3 got released). So is this just a generic recommendation of developers who love the bleeding-edge stuff and know how many problems they fixed since 2.6.27 ("yak, that old stuff, how could it ever run?"), or is it known that not enough fixes go into "maintained" kernels by distributions?

I have tons of servers to maintain, and can't compile kernels on my own anymore (I did this when I was younger), so I used openSUSE and I am in the process of replacing servers to SLES, which should receive much better services but still kernel upgrading is not supported on such platforms. So what is the best practice that one should choose when running "long time services" where you can't upgrade all the time?

Just curious. :-)

Signature · 13 lines
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services
http://proteger.at [gesprochen: Prot-e-schee]
Tel: 0660 / 415 65 31

****** Aktuelles Radiointerview! ******
http://www.it-podcast.at/aktuelle-sendung.html

// Wir haben im Moment zwei Häuser zu verkaufen:
// http://zmi.at/langegg/
// http://zmi.at/haus2009/

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#3 lore raw

Re: permanent kernel upgrading (was: bug and fun with XFS: unable to handle kernel NULL pointer dereference)

Quoted 3 lines · Michael Monnerie

This brings me to another question: openSUSE 11.1 is, despite being old, still maintained, so I was under the impression that I'd get updates for problems.

Most subsystems are not directly "maintained" by the distro maintainers. They _hope_ the upstream developers backport all their bug fixes to "long term stable" kernels like 2.6.27 and just merge those backports in. The reality is that after one or two releases, upstream has moved on and the long-term stable kernels don't see very many fixes from upstream...

Quoted 3 lines

But I already received "upgrade your kernel" the last time for my openSUSE 11.2 with it's kernel vmlinuz-2.6.31.12, which by the time was the latest release (in the meantime openSUSE 11.3 got released).

Yup, that's the distro maintainer's way of saying "we don't actively support that subsystem".

Quoted 4 lines

So is this just a generic recommendation of developers who love the bleeding-edge stuff and know how many problems they fixed since 2.6.27 ("yak, that old stuff, how could it ever run?"), or is it known that not enough fixes go into "maintained" kernels by distributions?

I think it is more of a reflection on where resources are available. Most free distros do very little in the way of maintenance except for security issues because they don't have the knowledge or resources to triage and fix most subsystems.

However, upstream resources are limited. If we identify a problem as a fixed bug, then it is up to the distro maintainers to backport it, not us. Distro maintenance is not our problem or responsisbility because no upstream developer has the bandwidth to be able to supply fixes to everyone, nor are they likely to want to beause it's mostly boring, unrewarding work.

That said, many upstream developers are also distro maintainers and as a result those distros tend to keep up to date with all needed fixes for those subsystems.

Quoted 6 lines

I have tons of servers to maintain, and can't compile kernels on my own anymore (I did this when I was younger), so I used openSUSE and I am in the process of replacing servers to SLES, which should receive much better services but still kernel upgrading is not supported on such platforms. So what is the best practice that one should choose when running "long time services" where you can't upgrade all the time?

Purchase a support contract for an enterprise distro that actively supports the features you require....

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
author
#4 lore raw

Re: permanent kernel upgrading (was: bug and fun with XFS: unable to handle kernel NULL pointer dereference)

Quoted 2 lines · On Montag, 26. Juli 2010 Dave Chinner wrote:

Purchase a support contract for an enterprise distro that actively supports the features you require....

I just installed SLES 11 SP1, has kernel 2.6.32.13-0.4, hope that's enough :-)

Signature · 13 lines
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services
http://proteger.at [gesprochen: Prot-e-schee]
Tel: 0660 / 415 65 31

****** Aktuelles Radiointerview! ******
http://www.it-podcast.at/aktuelle-sendung.html

// Wir haben im Moment zwei Häuser zu verkaufen:
// http://zmi.at/langegg/
// http://zmi.at/haus2009/

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs