ext2 filesystem corruptions back from dead? 2.4.0-test11
I just got these while doing many compiles on my box ....
Nov 23 00:40:06 viper kernel: EXT2-fs warning (device ide0(3,3)): ext2_unlink: Deleting nonexistent file (622295), 0 Nov 23 00:40:06 viper kernel: = 1 Nov 23 00:40:06 viper kernel: EXT2-fs error (device ide0(3,3)): ext2_free_blocks: Freeing blocks not in datazone - block = 540028982, count = 1 Nov 23 00:40:06 viper kernel: EXT2-fs error (device ide0(3,3)): ext2_free_blocks: Freeing blocks not in datazone - block = 540024880, count = 1 Nov 23 00:40:06 viper kernel: EXT2-fs error (device ide0(3,3)):
ext2_free_blocks: Freeing blocks not in datazone - block = 170926128, count = 1
What else should I provide?
[mhaque@viper mhaque]$ uname -a
Linux viper.haque.net 2.4.0-test11 #6 Sun Nov 19 22:17:33 EST 2000 i686 unknown
Nov 16 19:03:15 viper kernel: Uniform Multi-Platform E-IDE driver Revision: 6.31 Nov 16 19:03:15 viper kernel: ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx Nov 16 19:03:15 viper kernel: PIIX4: IDE controller on PCI bus 00 dev 39 Nov 16 19:03:15 viper kernel: PIIX4: chipset revision 1 Nov 16 19:03:15 viper kernel: PIIX4: not 100%% native mode: will probe irqs later Nov 16 19:03:15 viper kernel: ide0: BM-DMA at 0xf000-0xf007, BIOS settings: hda:DMA, hdb:DMA Nov 16 19:03:15 viper kernel: ide1: BM-DMA at 0xf008-0xf00f, BIOS
settings: hdc:DMA, hdd:DMA
Nov 16 19:03:15 viper kernel: hda: IBM-DJNA-371350, ATA DISK drive Nov 16 19:03:15 viper kernel: hdb: CREATIVEDVD-ROM DVD2240E 12/24/97, ATAPI CDROM drive Nov 16 19:03:15 viper kernel: hdc: Maxtor 82160D2, ATA DISK drive Nov 16 19:03:15 viper kernel: hdd: IBM-DTLA-307045, ATA DISK drive
Signature · 12 lines
=====================================================================
Mohammad A. Haque http://www.haque.net/
[email protected]
"Alcohol and calculus don't mix. Project Lead
Don't drink and derive." --Unknown http://wm.themes.org/
[email protected]
=====================================================================
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/Quoted 12 lines · Mohammad A. Haque
I just got these while doing many compiles on my box ....
Nov 23 00:40:06 viper kernel: EXT2-fs warning (device ide0(3,3)): ext2_unlink: Deleting nonexistent file (622295), 0 Nov 23 00:40:06 viper kernel: = 1 Nov 23 00:40:06 viper kernel: EXT2-fs error (device ide0(3,3)): ext2_free_blocks: Freeing blocks not in datazone - block = 540028982, count = 1 Nov 23 00:40:06 viper kernel: EXT2-fs error (device ide0(3,3)): ext2_free_blocks: Freeing blocks not in datazone - block = 540024880, count = 1 Nov 23 00:40:06 viper kernel: EXT2-fs error (device ide0(3,3)):
Quoted 2 lines
ext2_free_blocks: Freeing blocks not in datazone - block = 170926128, count = 1
I'm not sure where the nonexistent file comes from. According to the printf statement, you're trying to unlink a file with no links, so it would be interesting to see if 622295 is a valid inode number (it should be, or there would have been more error messages). Doing
dumpe2fs -h /dev/hda3
may help to find out where this bogus inode came from.
These block numbers decode to ASCII data:
540028982 = 0x20303036 = " 336"
540024880 = 0x20302030 = " 3 3"
170926128 = 0x0a302030 = "\n3 3"
There were problems like this quite a while ago (block numbers that are really ASCII data)... I can't recall what the problem turned out to be at that time.
I would suggest a full fsck to start with (you have probably already done so). If you haven't done a full fsck on this filesystem in a long time, there is a chance the corruption was from the old kernel bug.
Cheers, Andreas
Signature · 7 lines
Andreas Dilger \ "If a man ate a pound of pasta and a pound of antipasto,
\ would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/ -- Dogbert
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/Quoted 4 lines
540028982 =
0x20303036= " 336"
540024880 =0x20302030= " 3 3"
170926128 =0x0a302030= "\n3 3"
These should be:
540028982 = 0x20303036 = " 006"
540024880 = 0x20302030 = " 0 0"
170926128 = 0x0a302030 = "\n0 0"
/Pär-Ola Nilsson
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
Quoted 1 line · Neil Brown
Oh, good. It's not just me and Tigran then.
You have it all backwards. It would be good if it were just you and Tigran. Unfortunately it also hits me.
(I am reorganizing my disks, copying large trees from one place to the other. Always doing a diff -r between old and new before removing the old version. Yesterday I had a diff -r showing that the old version was corrupted and the new was OK. Of course a second look showed that the old version also was OK, the corruption must have been in the buffer cache, not on disk.)
Andries
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
Quoted 7 lines · Guest section DW
Quoted 1 line · Neil Brown
Oh, good. It's not just me and Tigran then.
You have it all backwards. It would be good if it were just you and Tigran. Unfortunately it also hits me.
Quoted 7 lines
(I am reorganizing my disks, copying large trees from one place to the other. Always doing a diff -r between old and new before removing the old version. Yesterday I had a diff -r showing that the old version was corrupted and the new was OK. Of course a second look showed that the old version also was OK, the corruption must have been in the buffer cache, not on disk.)
Are these disks IDE disks by any chance?
Ion
Signature · 6 lines
It is better to keep your mouth shut and be thought a fool,
than to open it and remove all doubt.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/Quoted 9 lines · Ion Badulescu
Quoted 7 lines
(I am reorganizing my disks, copying large trees from one place to the other. Always doing a diff -r between old and new before removing the old version. Yesterday I had a diff -r showing that the old version was corrupted and the new was OK. Of course a second look showed that the old version also was OK, the corruption must have been in the buffer cache, not on disk.)
Are these disks IDE disks by any chance?
Yes.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
Yep. Unless of course they are SCSI with an identity crisis =P
Quoted 5 lines · Ion Badulescu
Are these disks IDE disks by any chance?
Ion
Signature · 12 lines
=====================================================================
Mohammad A. Haque http://www.haque.net/
[email protected]
"Alcohol and calculus don't mix. Project Lead
Don't drink and derive." --Unknown http://wm.themes.org/
[email protected]
=====================================================================
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/Quoted 6 lines · Ion Badulescu
Quoted 4 lines
Yesterday I had a diff -r showing that the old version was corrupted and the new was OK. Of course a second look showed that the old version also was OK, the corruption must have been in the buffer cache, not on disk.)
Are these disks IDE disks by any chance?
What the F*** does that have to do with the price of eggs in china, heh? Just maybe if you could follow a thread, you would see that that Alex Viro has pointed out that changes in the FS layer as dorked things.
Since there have been not kernel changes to the driver that effect the code since 2.4.0-test5 or test6 and it now randomly shows up after five or six revisions out from the change, and the changes were chipset only.
Please make your point.
Andre Hedrick
Linux ATA Development
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
Quoted 3 lines · Andre Hedrick
What the F*** does that have to do with the price of eggs in china, heh? Just maybe if you could follow a thread, you would see that that Alex Viro has pointed out that changes in the FS layer as dorked things.
?
If you have a l-k feed from future - please share. I'm not saying that fs/* is not the source of that stuff, but I sure as hell had not said that it is. I simply don't know yet.
Quoted 3 lines
Since there have been not kernel changes to the driver that effect the code since 2.4.0-test5 or test6 and it now randomly shows up after five or six revisions out from the change, and the changes were chipset only.
generic_unplug_device() was changed more or less recently. I doubt that it is relevant, but...
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
Quoted 10 lines · Alexander Viro
Quoted 3 lines · Andre Hedrick
What the F*** does that have to do with the price of eggs in china, heh? Just maybe if you could follow a thread, you would see that that Alex Viro has pointed out that changes in the FS layer as dorked things.
?
If you have a l-k feed from future - please share. I'm not saying that
Date: Thu, 23 Nov 2000 04:37:21 -0500 (EST)
Quoted 2 lines
fs/*is not the source of that stuff, but I sure as hell had not said that it is. I simply don't know yet.
You were pointing out changes to reproduce the effect.
Quoted 6 lines
Quoted 3 lines
Since there have been not kernel changes to the driver that effect the code since 2.4.0-test5 or test6 and it now randomly shows up after five or six revisions out from the change, and the changes were chipset only.
generic_unplug_device()was changed more or less recently. I doubt that it is relevant, but...
Cool, the issue was that I get tried of people blaming the ATA subsystem for things that it does not do or has control over. Basically, I kill bogus threads that try to tag me with an old problem of the past that was a hardware issue.
Given the latest stats that more than 90% of the linux install base is hinged on me getting the low-level engine core correct, I go on benders when cheap shots are take across the bow.
Now the only issue that is even on the radar map is a potential 1GB cross copy execution where I have a single report that md5sums do not match. I have yet to reproduce it even with the identical hardware sent to me.
I questioned _A_ about this and there may be a case that is OS independent which is more important to me than other. This is a non-fixable for 6->12 months, until I kick some tail in the standards committee meetings over this point. If it is a reality, Linux and Microsoft will join as the OS's represented there and force the change. Only because there are potentical side-bars that NT is effected also in these rare cases.
Cheers,
Andre Hedrick
Linux ATA Development
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
On Thu, 23 Nov 2000, Andre Hedrick wrote:
[I wrote]
Quoted 9 lines
Quoted 2 lines
?
If you have a l-k feed from future - please share. I'm not saying thatDate: Thu, 23 Nov 2000 04:37:21 -0500 (EST)
Quoted 2 lines
fs/*is not the source of that stuff, but I sure as hell had not said that it is. I simply don't know yet.You were pointing out changes to reproduce the effect.
Erm... Since then the problem had been reproduced on the patched tree, so we apparently have something else. Behaviour on disk/quota overflow is a separate story - even with fixes for that problem stays.
Quoted 11 lines
Quoted 6 lines
Quoted 3 lines
Since there have been not kernel changes to the driver that effect the code since 2.4.0-test5 or test6 and it now randomly shows up after five or six revisions out from the change, and the changes were chipset only.
generic_unplug_device()was changed more or less recently. I doubt that it is relevant, but...Cool, the issue was that I get tried of people blaming the ATA subsystem for things that it does not do or has control over. Basically, I kill bogus threads that try to tag me with an old problem of the past that was a hardware issue.
<shrug> I don't see any attempts to tag you (or ATA subsystem, for that matter) in that thread. And thread is hardly bogus... I agree that changes in drivers/ide/* are very unlikely to be the source of that, but information of that kind can help to weed out some of the changes in ll_rw_blk.c.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
Quoted 2 lines · Alexander Viro
<shrug> I don't see any attempts to tag you (or ATA subsystem, for that matter) in that thread. And thread is hardly bogus... I agree that changes in
We agree that the "thread" is valid, trust that point.
There was a quick pointed question that present, "Is it an IDE disk?" to paraphase the statement.
Quoted 2 lines
drivers/ide/*are very unlikely to be the source of that, but information of that kind can help to weed out some of the changes inll_rw_blk.c.
What may be even more helpful is when I get arround to making an option, for some outstanding patches for 2.5, that would allow for user-space pattern pushing through the driver that gets properly inserted in to the list/buffer-head to make it pass through the block layer. This kind of testing will allow for nibble level tracing through everything, I hope.
Cheers,
Andre Hedrick
CTO Timpanogas Research Group
EVP Linux Development, TRG
Linux ATA Development
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
Quoted 5 lines · Andre Hedrick
Since there have been not kernel changes to the driver that effect the code since 2.4.0-test5 or test6 and it now randomly shows up after five or six revisions out from the change, and the changes were chipset only.
Please make your point.
My point is simple: I'm trying to see if there is a pattern. I've had filesystems corrupted with 2.2.18 + the backported IDE driver. Other people have had filesystems corrupted with 2.4.0 + the same IDE driver. If *all* people seeing f/s corruption have IDE disks and *none* of them have SCSI, there might be something worth looking into. It might as well be pure coincidence.
What's especially bothering me is the fact that I've seen the IDE driver choke on DMA or something, and then continue on with life, while serving *bad* *data* to the upper layers. Even if there were real problems with the DMA transfers (which is not the case, 2.2.18pre without the IDE patch runs flawlessly), a driver should never ever serve bad blocks to the f/s layer. Locking up the machine completely, like some SCSI low-level drivers do, is much better.
************************************************************************
So I'm asking the same question, to all those who have seen unexplained filesystem corruption with 2.4.0: are you using IDE drives? If the answer is yes, can you check the logs and see if, at *any* point before the corruption occurred, the IDE driver choked and disabled DMA for *any* of your disks?
************************************************************************
Even if 90% of the installed base is IDE and 10% is SCSI, in terms of how heavily the hardware is being stressed the advantage of IDE over SCSI is definitely not 9:1.
And Andre, don't take this personally. We're just trying to save our precious data here, nothing more. :-) If something comes out of this inquiry, it might just give you a lead.
Thanks,
Ion
Signature · 7 lines
It is better to keep your mouth shut and be thought a fool,
than to open it and remove all doubt.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/Quoted 1 line · Mohammad A. Haque
Yep. Unless of course they are SCSI with an identity crisis =P
Ok. Are there any IDE-related errors in your logs prior to getting the f/s corruption? They could be relevant no matter how much time passed between them and the first signs of corruption.
Are your drives running with UDMA transfers enabled?
Thanks,
Ion
Signature · 7 lines
It is better to keep your mouth shut and be thought a fool,
than to open it and remove all doubt.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/Quoted 5 lines · Ion Badulescu
So I'm asking the same question, to all those who have seen unexplained filesystem corruption with 2.4.0: are you using IDE drives? If the answer is yes, can you check the logs and see if, at *any* point before the corruption occurred, the IDE driver choked and disabled DMA for *any* of your disks?
I have both IDE and SCSI drives in my machine, but have only seen corruption on the SCSI drives. That doesn't mean that the problem only exists on the SCSI drives - they IDE ones are not frequently written to. I have disabled DMA myself on all my IDE drives because if I enable it, the IDE driver always chokes the first time they are anything like hammered (well, it always used to - I haven't actually tried it recently).
Signature · 9 lines
Mike Ricketts <[email protected]> Phone: +44 7968 381810 Humility is the first of the virtues -- for other people. -- Oliver Wendell Holmes - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] Please read the FAQ at http://www.tux.org/lkml/
I have seen ext2 filesystem corruption both on SCSI and IDE drives.
Tigran
Quoted 8 lines · Mike Ricketts
Quoted 5 lines · Ion Badulescu
So I'm asking the same question, to all those who have seen unexplained filesystem corruption with 2.4.0: are you using IDE drives? If the answer is yes, can you check the logs and see if, at *any* point before the corruption occurred, the IDE driver choked and disabled DMA for *any* of your disks?
Quoted 8 lines
I have both IDE and SCSI drives in my machine, but have only seen corruption on the SCSI drives. That doesn't mean that the problem only exists on the SCSI drives - they IDE ones are not frequently written to. I have disabled DMA myself on all my IDE drives because if I enable it, the IDE driver always chokes the first time they are anything like hammered (well, it always used to - I haven't actually tried it recently).
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
Quoted 1 line · Ion Badulescu
Ok. Are there any IDE-related errors in your logs
Once, after a reboot:
Nov 22 17:25:50 mette kernel: hdf: status error: status=0x58 { DriveReady SeekComplete DataRequest }
Nov 22 17:25:50 mette kernel: hdf: drive not ready for command
Nov 22 17:25:50 mette kernel: hdf: status timeout: status=0xd0 { Busy }
Nov 22 17:25:50 mette kernel: hdf: drive not ready for command
Nov 22 17:25:52 mette kernel: ide2: reset: success(But I described the situation where the data on disk was correct and the date in core was not - almost certainly this is not an IDE problem.)
Andries
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
I get the following followingon every reboot once.
Nov 23 01:14:37 viper kernel: hdb: drive_cmd: status=0x51 { DriveReady
SeekComplete Error }
Nov 23 01:14:37 viper kernel: hdb: drive_cmd: error=0x04
Nov 23 01:14:37 viper kernel: hdb: drive_cmd: status=0x51 { DriveReady
SeekComplete Error }
Nov 23 01:14:37 viper kernel: hdb: drive_cmd: error=0x04hdb is my DVD drive. But other than that I haven't seen any other ide related errors.
I found these two lines nested in between alot of other messages that I missed before.
Nov 23 00:35:11 viper kernel: EXT2-fs error (device ide0(3,3)): ext2_free_blocks: bit already cleared for block 147021 Nov 23 00:35:11 viper kernel: EXT2-fs error (device ide0(3,3)):
ext2_free_blocks: bit already cleared for block 147021
Then I get these ....
Nov 23 00:40:06 viper kernel: EXT2-fs warning (device ide0(3,3)): ext2_unlink: Deleting nonexistent file (622295), 0 Nov 23 00:40:06 viper kernel: = 1 Nov 23 00:40:06 viper kernel: EXT2-fs error (device ide0(3,3)): ext2_free_blocks: Freeing blocks not in datazone - block = 540028982, count = 1 Nov 23 00:40:06 viper kernel: EXT2-fs error (device ide0(3,3)):
ext2_free_blocks: Freeing blocks not in datazone - block = 540024880, count = 1
[mhaque@viper mhaque]$ sudo hdparm -iv /dev/hda
/dev/hda:
multcount = 16 (on) I/O support = 3 (32-bit w/sync) unmaskirq = 0 (off) using_dma = 1 (on) keepsettings = 1 (on) nowerr = 0 (off) readonly = 0 (off)
readahead = 128 (on)
geometry = 1650/255/63, sectors = 26520480, start = 0
Model=IBM-DJNA-371350, FwRev=J76OA30K, SerialNo=GM0GMFE4929 Config={ HardSect NotMFM HdSw>15uSec Fixed DTR>10Mbs } RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=34 BuffType=DualPortCache, BuffSize=1966kB, MaxMultSect=16, MultSect=16 CurCHS=16383/16/63, CurSects=-66060037, LBA=yes, LBAsects=26520480 IORDY=on/off, tPIO={min:240,w/IORDY:120}, tDMA={min:120,rec:120} PIO modes: pio0 pio1 pio2 pio3 pio4
DMA modes: mdma0 mdma1 mdma2 udma0 udma1 *udma2 udma3 udma4
[mhaque@viper mhaque]$ sudo hdparm -iv /dev/hdb
/dev/hdb:
HDIO_GET_MULTCOUNT failed: Invalid argument
I/O support = 3 (32-bit w/sync) unmaskirq = 0 (off) using_dma = 1 (on) keepsettings = 1 (on)
HDIO_GET_NOWERR failed: Invalid argument
readonly = 1 (on)
readahead = 128 (on)
HDIO_GETGEO failed: Invalid argument
Model=CREATIVEDVD-ROM DVD2240E 12/24/97, FwRev=1.7A, SerialNo= Config={ Fixed Removeable DTR<=5Mbs DTR>10Mbs nonMagnetic } RawCHS=0/0/0, TrkSize=0, SectSize=0, ECCbytes=0 BuffType=unknown, BuffSize=0kB, MaxMultSect=0
(maybe): CurCHS=0/0/0, CurSects=0, LBA=yes, LBAsects=0 IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:150} PIO modes: pio0 pio1 pio2 pio4
DMA modes: sdma0 sdma1 sdma2 sdma? mdma0 mdma1 *mdma2
Quoted 13 lines · Ion Badulescu
Ok. Are there any IDE-related errors in your logs prior to getting the f/s corruption? They could be relevant no matter how much time passed between them and the first signs of corruption.
Are your drives running with UDMA transfers enabled?
Thanks,
Ion--
It is better to keep your mouth shut and be thought a fool,
than to open it and remove all doubt.
Signature · 12 lines
=====================================================================
Mohammad A. Haque http://www.haque.net/
[email protected]
"Alcohol and calculus don't mix. Project Lead
Don't drink and derive." --Unknown http://wm.themes.org/
[email protected]
=====================================================================
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/In article <[email protected]>,
Quoted 3 lines · Guest section DW
(But I described the situation where the data on disk was correct and the date in core was not - almost certainly this is not an IDE problem.)
Ehh.. It only means that it would have been a read failure instead of a write failure.
Linus
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
Quoted 8 lines · Mike Ricketts
Quoted 5 lines · Ion Badulescu
So I'm asking the same question, to all those who have seen unexplained filesystem corruption with 2.4.0: are you using IDE drives? If the answer is yes, can you check the logs and see if, at *any* point before the corruption occurred, the IDE driver choked and disabled DMA for *any* of your disks?
Quoted 6 lines
I have both IDE and SCSI drives in my machine, but have only seen corruption on the SCSI drives. That doesn't mean that the problem only exists on the SCSI drives - they IDE ones are not frequently written to. I have disabled DMA myself on all my IDE drives because if I enable it, the IDE driver always chokes the first time they are anything like hammered (well, it always used to - I haven't actually tried it recently).
This is the kind of data point that is needed.
A possible storage class independent problem.
More important, what was the first kernel you began to notice this problem. Next, I need you to enable the DMA engine in ATA to verify that is happening on both classes.
Cheers,
Andre Hedrick
CTO Timpanogas Research Group
EVP Linux Development, TRG
Linux ATA Development
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/