3.15-rc5 btrfs send/receive corruption errors?

DISCUSSION Marc MERLIN 3 participants 11 messages May 10, 2014 – May 19, 2014 Btrfs file system

lore

Share & tools

lore

Marc MERLIN

author

2014-05-10 22:07 UTC #0 lore raw

Howdy,

While moving data back to a brand new btrfs FS I had just created (with 3.14 tools and under 3.15), I got this:

legolas:/mnt/btrfs_pool2# for i in tmp_ro.20140507_09:00:31 root_ro.20140507_10:00:20 usr_ro.20140507_09:00:41 var_ro.20140507_09:00:58 home_ro.20140507_10:00:01; do btrfs send $i | btrfs receive /mnt/btrfs_pool1/; done
At subvol tmp_ro.20140507_09:00:31
At subvol tmp_ro.20140507_09:00:31
At subvol root_ro.20140507_10:00:20
At subvol root_ro.20140507_10:00:20
At subvol usr_ro.20140507_09:00:41
At subvol usr_ro.20140507_09:00:41
At subvol var_ro.20140507_09:00:58
At subvol var_ro.20140507_09:00:58
ERROR: chown www/Q����Q��� failed. No such file or directory
At subvol home_ro.20140507_10:00:01
At subvol home_ro.20140507_10:00:01
ERROR: chmod merlin/.config/google-chrome-mysetup/���� failed. No such file or directory
legolas:/mnt/btrfs_pool2#

legolas:/mnt/btrfs_pool2/var_ro.20140507_09:00:58/www# ls
shows no file that looks like Q^W^F^D����Q^W^F^D���

Same deal with merlin/.config/google-chrome-mysetup/

However, I see:

legolas:/mnt/btrfs_pool1/home_ro.20140507_10:00:01# l
total 16
drwxr-xr-x 1 root   root    20 May 10 09:09 ./
drwxr-xr-x 1 root   root   244 May 10 07:50 ../
drwx------ 1 root   root     0 May 10 09:09 ?0$?/
drwxr-xr-x 1 merlin merlin 198 May  7 06:21 merlin/
=> corruption

and

legolas:/mnt/btrfs_pool1/var_ro.20140507_09:00:58# l
total 24
(...)
drwx------ 1 root root  276 May 10 09:07 o1346374-23116-0/
drwx------ 1 root root   38 May 10 09:07 o671296-2268-0/
drwx------ 1 root root   48 May 10 09:07 o676276-2269-0/
drwx------ 1 root root   24 May 10 09:07 o676661-2269-0/
drwx------ 1 root root   72 May 10 09:07 o677466-2270-0/
drwx------ 1 root root  278 May 10 09:07 o718029-2271-0/
drwx------ 1 root root   96 May 10 09:07 o952662-7468-0/
drwxr-xr-x 1 root root    0 Jan 29  2012 opt/
drwx------ 1 root root    0 May 10 09:08 ?Q???????Q???????Q???????/
=> corruption too

I could believe that btrfs has an issue on the source drive, but ls -l looks clean on the source, and clearly looks corrupted on the destination after send/receive.

Any clues?

I'll try reverting to 3.14 if I have enough time this morning to see if the error happens again.

Thanks,
Marc

Signature · 4 lines

"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Marc MERLIN

author

2014-05-10 22:47 UTC #1 lore raw

Quoted 25 lines · Marc MERLIN

Howdy,

While moving data back to a brand new btrfs FS I had just created (with 3.14 tools and under 3.15), I got this:

legolas:/mnt/btrfs_pool2# for i in tmp_ro.20140507_09:00:31 root_ro.20140507_10:00:20 usr_ro.20140507_09:00:41 var_ro.20140507_09:00:58 home_ro.20140507_10:00:01; do btrfs send $i | btrfs receive /mnt/btrfs_pool1/; done
At subvol tmp_ro.20140507_09:00:31
At subvol tmp_ro.20140507_09:00:31
At subvol root_ro.20140507_10:00:20
At subvol root_ro.20140507_10:00:20
At subvol usr_ro.20140507_09:00:41
At subvol usr_ro.20140507_09:00:41
At subvol var_ro.20140507_09:00:58
At subvol var_ro.20140507_09:00:58
ERROR: chown www/Q����Q��� failed. No such file or directory
At subvol home_ro.20140507_10:00:01
At subvol home_ro.20140507_10:00:01
ERROR: chmod merlin/.config/google-chrome-mysetup/���� failed. No such file or directory
legolas:/mnt/btrfs_pool2#

legolas:/mnt/btrfs_pool2/var_ro.20140507_09:00:58/www# ls
shows no file that looks like Q^W^F^D����Q^W^F^D���

Same deal with merlin/.config/google-chrome-mysetup/

Quoted 8 lines

However, I see:

legolas:/mnt/btrfs_pool1/home_ro.20140507_10:00:01# l
total 16
drwxr-xr-x 1 root   root    20 May 10 09:09 ./
drwxr-xr-x 1 root   root   244 May 10 07:50 ../
drwx------ 1 root   root     0 May 10 09:09 ?0$?/
drwxr-xr-x 1 merlin merlin 198 May  7 06:21 merlin/
=> corruption

I tried with 3.14.3 and it went further, however it died with

legolas:/mnt/btrfs_pool2# btrfs send  home_ro.20140507_10:00:01 | btrfs receive /mnt/btrfs_pool1/
At subvol home_ro.20140507_10:00:01
At subvol home_ro.20140507_10:00:01
ERROR: send ioctl failed with -5: Input/output error
ERROR: unexpected EOF in stream.

I'll look up -5 later when I have time, but I guess there is a problem on the source that is causing copies to fail with both kernels?

Marc

Signature · 4 lines

"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Chris Murphy

2014-05-11 00:07 UTC #2 lore raw

Quoted 36 lines · Marc MERLIN

Quoted 25 lines · Marc MERLIN

Howdy,

While moving data back to a brand new btrfs FS I had just created (with 3.14 tools and under 3.15), I got this:

legolas:/mnt/btrfs_pool2# for i in tmp_ro.20140507_09:00:31 root_ro.20140507_10:00:20 usr_ro.20140507_09:00:41 var_ro.20140507_09:00:58 home_ro.20140507_10:00:01; do btrfs send $i | btrfs receive /mnt/btrfs_pool1/; done
At subvol tmp_ro.20140507_09:00:31
At subvol tmp_ro.20140507_09:00:31
At subvol root_ro.20140507_10:00:20
At subvol root_ro.20140507_10:00:20
At subvol usr_ro.20140507_09:00:41
At subvol usr_ro.20140507_09:00:41
At subvol var_ro.20140507_09:00:58
At subvol var_ro.20140507_09:00:58
ERROR: chown www/Q����Q��� failed. No such file or directory
At subvol home_ro.20140507_10:00:01
At subvol home_ro.20140507_10:00:01
ERROR: chmod merlin/.config/google-chrome-mysetup/���� failed. No such file or directory
legolas:/mnt/btrfs_pool2#

legolas:/mnt/btrfs_pool2/var_ro.20140507_09:00:58/www# ls
shows no file that looks like Q^W^F^D����Q^W^F^D���

Same deal with merlin/.config/google-chrome-mysetup/

Quoted 8 lines

However, I see:

legolas:/mnt/btrfs_pool1/home_ro.20140507_10:00:01# l
total 16
drwxr-xr-x 1 root   root    20 May 10 09:09 ./
drwxr-xr-x 1 root   root   244 May 10 07:50 ../
drwx------ 1 root   root     0 May 10 09:09 ?0$?/
drwxr-xr-x 1 merlin merlin 198 May  7 06:21 merlin/
=> corruption

I tried with 3.14.3 and it went further, however it died with

Quoted 8 lines

legolas:/mnt/btrfs_pool2# btrfs send  home_ro.20140507_10:00:01 | btrfs receive /mnt/btrfs_pool1/
At subvol home_ro.20140507_10:00:01
At subvol home_ro.20140507_10:00:01
ERROR: send ioctl failed with -5: Input/output error
ERROR: unexpected EOF in stream.
I'll look up -5 later when I have time, but I guess there is a problem on the source that is causing copies to fail with both kernels?\

I think the issue might be there are still problems and changing code on the receive side (btrfs-progs). The kernel side code responsible for send is probably working correctly.

Chris Murphy

Marc MERLIN

author

2014-05-11 00:17 UTC #3 lore raw

Quoted 1 line · Chris Murphy

I think the issue might be there are still problems and changing code on the receive side (btrfs-progs). The kernel side code responsible for send is probably working correctly.

Mmmh, good point, I didn't consider that.
Do you think I should downgrade btrfs-progs back down from 3.14 to 3.12?

Marc

Signature · 4 lines

"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Chris Murphy

2014-05-11 00:50 UTC #4 lore raw

Quoted 5 lines · Marc MERLIN

Quoted 1 line · Chris Murphy
I think the issue might be there are still problems and changing code on the receive side (btrfs-progs). The kernel side code responsible for send is probably working correctly.
Mmmh, good point, I didn't consider that.
Do you think I should downgrade btrfs-progs back down from 3.14 to 3.12?

If anything, David's btrfs-integration branch, but I'm not sure what its status is at the moment compared to 3.14.

Chris Murphy

Marc MERLIN

author

2014-05-11 02:30 UTC #5 lore raw

Re: 3.15-rc5 btrfs send/receive corruption errors? Does scrub warn of silent corruption?

In reply to Chris Murphy

Quoted 20 lines · Marc MERLIN

Quoted 9 lines · Marc MERLIN
I tried with 3.14.3 and it went further, however it died with
legolas:/mnt/btrfs_pool2# btrfs send  home_ro.20140507_10:00:01 | btrfs receive /mnt/btrfs_pool1/
At subvol home_ro.20140507_10:00:01
At subvol home_ro.20140507_10:00:01
ERROR: send ioctl failed with -5: Input/output error
ERROR: unexpected EOF in stream.
I'll look up -5 later when I have time, but I guess there is a problem on the source that is causing copies to fail with both kernels?
This brings me back to the earlier question:
When my other FS died, scrub ran ok just earlier.
Now, having 2 btrfs sends (not incremental, full) fail with 2 kernels would indicate that something might be wrong on the source filesystem.
Yet, last night's scrub ran fine too:

Quoted 8 lines

Quoted 4 lines · Anacron
/etc/cron.daily/btrfs-scrub:
scrub device /dev/mapper/cryptroot (id 1) done
scrub started at Fri May 9 06:09:14 2014 and finished after 19153 seconds total bytes scrubbed: 646.15GiB with 0 errors
So, does scrub actually make sure everything on my filesystem is sane, or can it miss some kinds of corruptions?

I'm thinking that there must be some corruption on btrfs_pool2, because:

Quoted 1 line · Chris Murphy

I think the issue might be there are still problems and changing code on the receive side (btrfs-progs). The kernel side code responsible for send is probably working correctly.

I tried
3.15 and progs 3.12
3.15 and progs 3.14
3.14 and progs 3.12
3.14 and progs 3.14

All 4 failed to run

legolas:/mnt/btrfs_pool2# btrfs send home_ro.20140507_10:00:01 | btrfs receive /mnt/btrfs_pool1/

3.14 gives:
ERROR: send ioctl failed with -5: Input/output error
ERROR: unexpected EOF in stream.

3.15 gives a more detailled message how it can't rename some corrupted filename (regardless of whether I use progs 3.12 or 3.14).

So, since scrub tells me my FS is ok, is scrub right, or do I have a file pattern that is causing a bug in btrfs send ?

Thanks,
Marc

Signature · 4 lines

"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Marc MERLIN

author

2014-05-12 23:26 UTC #6 lore raw

Re: 3.15-rc5 btrfs send/receive corruption errors? Does scrub warn of silent corruption?

In reply to Marc MERLIN

Quoted 11 lines · Marc MERLIN

Quoted 9 lines · Marc MERLIN
I tried with 3.14.3 and it went further, however it died with
legolas:/mnt/btrfs_pool2# btrfs send  home_ro.20140507_10:00:01 | btrfs receive /mnt/btrfs_pool1/
At subvol home_ro.20140507_10:00:01
At subvol home_ro.20140507_10:00:01
ERROR: send ioctl failed with -5: Input/output error
ERROR: unexpected EOF in stream.
I'll look up -5 later when I have time, but I guess there is a problem on the source that is causing copies to fail with both kernels?

Quoted 6 lines

This brings me back to the earlier question:
When my other FS died, scrub ran ok just earlier.
Now, having 2 btrfs sends (not incremental, full) fail with 2 kernels would indicate that something might be wrong on the source filesystem.

So I copied the entire FS back to the SSD using rsync instead of btrfs send/receive and as far as I can tell all the data got over fine.

Is there anything you'd like from the subvolumes on the source that btrfs cannot process and that I'm going to delete so that I can start syncing back from the SSD to the HDD?

Thanks,
Marc

Signature · 4 lines

"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Filipe David Manana

2014-05-13 20:11 UTC #7 lore raw

Re: 3.15-rc5 btrfs send/receive corruption errors? Does scrub warn of silent corruption?

Quoted 19 lines · Marc MERLIN

Quoted 11 lines · Marc MERLIN
Quoted 9 lines · Marc MERLIN
I tried with 3.14.3 and it went further, however it died with
legolas:/mnt/btrfs_pool2# btrfs send  home_ro.20140507_10:00:01 | btrfs receive /mnt/btrfs_pool1/
At subvol home_ro.20140507_10:00:01
At subvol home_ro.20140507_10:00:01
ERROR: send ioctl failed with -5: Input/output error
ERROR: unexpected EOF in stream.
I'll look up -5 later when I have time, but I guess there is a problem on the source that is causing copies to fail with both kernels?
Quoted 6 lines
This brings me back to the earlier question:
When my other FS died, scrub ran ok just earlier.
Now, having 2 btrfs sends (not incremental, full) fail with 2 kernels would indicate that something might be wrong on the source filesystem.

Quoted 6 lines

So I copied the entire FS back to the SSD using rsync instead of btrfs send/receive and as far as I can tell all the data got over fine.
Is there anything you'd like from the subvolumes on the source that btrfs cannot process and that I'm going to delete so that I can start syncing back from the SSD to the HDD?

For the issue you had with send sending weird path names, I just found a case that leads to it (or a crash or some other weird stuff):

https://patchwork.kernel.org/patch/4170401/

But you really need to be using a lot of hard links and deleting them, so maybe it's caused by something else.

thanks

Quoted 8 lines

Thanks,
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | PGP 1024R/763BE901

Signature · 5 lines

Filipe David Manana,

"Reasonable men adapt themselves to the world.
 Unreasonable men adapt the world to themselves.
 That's why all progress depends on unreasonable men."

Marc MERLIN

author

2014-05-14 13:26 UTC #8 lore raw

Re: 3.15-rc5 btrfs send/receive corruption errors? Does scrub warn of silent corruption?

Quoted 11 lines · Filipe David Manana

Quoted 3 lines
Is there anything you'd like from the subvolumes on the source that btrfs cannot process and that I'm going to delete so that I can start syncing back from the SSD to the HDD?
For the issue you had with send sending weird path names, I just found a case that leads to it (or a crash or some other weird stuff):
https://patchwork.kernel.org/patch/4170401/
But you really need to be using a lot of hard links and deleting them, so maybe it's caused by something else.

Unfortunately, even with your patch, I see get

legolas:/mnt/btrfs_pool2# btrfs send home_ro.20140507_10\:00\:01 | btrfs receive /mnt/btrfs_pool1/
At subvol home_ro.20140507_10:00:01
At subvol home_ro.20140507_10:00:01
ERROR: chown merlin/.config/google-chrome-mysetup/��� failed. No such file or directory

I just ran btrfsck and I see nothing majorly wrong with the source filesystem:

legolas:~# btrfsck /dev/mapper/disk2 2>&1 |tee /tmp/fsck
checking extents
checking free space cache
checking fs roots
root 22504 inode 1926322 errors 400, nbytes wrong
Checking filesystem on /dev/mapper/disk2
UUID: 6afd4707-876c-46d6-9de2-21c4085b7bed
free space inode generation (0) did not match free space cache generation (78684)
free space inode generation (0) did not match free space cache generation (75988)
free space inode generation (0) did not match free space cache generation (76193)
free space inode generation (0) did not match free space cache generation (28818)
free space inode generation (0) did not match free space cache generation (28818)
free space inode generation (0) did not match free space cache generation (33187)
free space inode generation (0) did not match free space cache generation (31543)
free space inode generation (0) did not match free space cache generation (16710)
found 283033724420 bytes used err is 1
total csum bytes: 663653972
total tree bytes: 7333687296
total fs tree bytes: 5844262912
total extent tree bytes: 631451648
btree space waste bytes: 1497868045
file data blocks allocated: 1081231372288
 referenced 807338209280
Btrfs v3.14.1

To be clear, I do not need this to work, this is a snapshot I'm going to delete anyway, but if there is anything you'd like me to try or capture for you to help with improving the code, please let me know.

Marc

Signature · 4 lines

"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Marc MERLIN

author

2014-05-17 22:23 UTC #9 lore raw

Re: 3.15-rc5 btrfs send/receive corruption errors? Does scrub warn of silent corruption?

Before I delete this and start over, anything else you'd like from it?

Also, it the 3.15rc5 deadlock I had, did not occur again. So, I think it may well have been related to my doing a big send/receive.
See "3.15rc5 deadlock" thread.

Marc

Quoted 63 lines · Marc MERLIN

Quoted 11 lines · Filipe David Manana
Quoted 3 lines
Is there anything you'd like from the subvolumes on the source that btrfs cannot process and that I'm going to delete so that I can start syncing back from the SSD to the HDD?
For the issue you had with send sending weird path names, I just found a case that leads to it (or a crash or some other weird stuff):
https://patchwork.kernel.org/patch/4170401/
But you really need to be using a lot of hard links and deleting them, so maybe it's caused by something else.
Unfortunately, even with your patch, I see get
legolas:/mnt/btrfs_pool2# btrfs send home_ro.20140507_10\:00\:01 | btrfs receive /mnt/btrfs_pool1/
At subvol home_ro.20140507_10:00:01
At subvol home_ro.20140507_10:00:01
ERROR: chown merlin/.config/google-chrome-mysetup/��� failed. No such file or directory
I just ran btrfsck and I see nothing majorly wrong with the source filesystem:
legolas:~# btrfsck /dev/mapper/disk2 2>&1 |tee /tmp/fsck
checking extents
checking free space cache
checking fs roots
root 22504 inode 1926322 errors 400, nbytes wrong
Checking filesystem on /dev/mapper/disk2
UUID: 6afd4707-876c-46d6-9de2-21c4085b7bed
free space inode generation (0) did not match free space cache generation (78684)
free space inode generation (0) did not match free space cache generation (75988)
free space inode generation (0) did not match free space cache generation (76193)
free space inode generation (0) did not match free space cache generation (28818)
free space inode generation (0) did not match free space cache generation (28818)
free space inode generation (0) did not match free space cache generation (33187)
free space inode generation (0) did not match free space cache generation (31543)
free space inode generation (0) did not match free space cache generation (16710)
found 283033724420 bytes used err is 1
total csum bytes: 663653972
total tree bytes: 7333687296
total fs tree bytes: 5844262912
total extent tree bytes: 631451648
btree space waste bytes: 1497868045
file data blocks allocated: 1081231372288
 referenced 807338209280
Btrfs v3.14.1
To be clear, I do not need this to work, this is a snapshot I'm going to delete anyway, but if there is anything you'd like me to try or capture for you to help with improving the code, please let me know.
Marc
Signature · 8 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Signature · 4 lines

"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Filipe David Manana

2014-05-19 12:54 UTC #10 lore raw

Re: 3.15-rc5 btrfs send/receive corruption errors? Does scrub warn of silent corruption?

Quoted 1 line · Marc MERLIN

Before I delete this and start over, anything else you'd like from it?

Can you create an image of the fs with btrfs-image
(https://btrfs.wiki.kernel.org/index.php/Btrfs-image) and uploaded it somewhere (or send it to me directly) to see if it's easy to reproduce?

thanks

Quoted 78 lines

Also, it the 3.15rc5 deadlock I had, did not occur again. So, I think it may well have been related to my doing a big send/receive.
See "3.15rc5 deadlock" thread.
Marc
Quoted 52 lines · Marc MERLIN
Quoted 11 lines · Filipe David Manana
Quoted 3 lines
Is there anything you'd like from the subvolumes on the source that btrfs cannot process and that I'm going to delete so that I can start syncing back from the SSD to the HDD?
For the issue you had with send sending weird path names, I just found a case that leads to it (or a crash or some other weird stuff):
https://patchwork.kernel.org/patch/4170401/
But you really need to be using a lot of hard links and deleting them, so maybe it's caused by something else.
Unfortunately, even with your patch, I see get
legolas:/mnt/btrfs_pool2# btrfs send home_ro.20140507_10\:00\:01 | btrfs receive /mnt/btrfs_pool1/
At subvol home_ro.20140507_10:00:01
At subvol home_ro.20140507_10:00:01
ERROR: chown merlin/.config/google-chrome-mysetup/ ��� failed. No such file or directory
I just ran btrfsck and I see nothing majorly wrong with the source filesystem:
legolas:~# btrfsck /dev/mapper/disk2 2>&1 |tee /tmp/fsck
checking extents
checking free space cache
checking fs roots
root 22504 inode 1926322 errors 400, nbytes wrong
Checking filesystem on /dev/mapper/disk2
UUID: 6afd4707-876c-46d6-9de2-21c4085b7bed
free space inode generation (0) did not match free space cache generation (78684)
free space inode generation (0) did not match free space cache generation (75988)
free space inode generation (0) did not match free space cache generation (76193)
free space inode generation (0) did not match free space cache generation (28818)
free space inode generation (0) did not match free space cache generation (28818)
free space inode generation (0) did not match free space cache generation (33187)
free space inode generation (0) did not match free space cache generation (31543)
free space inode generation (0) did not match free space cache generation (16710)
found 283033724420 bytes used err is 1
total csum bytes: 663653972
total tree bytes: 7333687296
total fs tree bytes: 5844262912
total extent tree bytes: 631451648
btree space waste bytes: 1497868045
file data blocks allocated: 1081231372288
 referenced 807338209280
Btrfs v3.14.1
To be clear, I do not need this to work, this is a snapshot I'm going to delete anyway, but if there is anything you'd like me to try or capture for you to help with improving the code, please let me know.
Quoted 11 lines
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | PGP 1024R/763BE901
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R. Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | PGP 1024R/763BE901

Signature · 5 lines

Filipe David Manana,

"Reasonable men adapt themselves to the world.
 Unreasonable men adapt the world to themselves.
 That's why all progress depends on unreasonable men."