Skip to contentSkip to conversation

strange ext3 corruption problem on 2.6.x

DISCUSSION Marc Lehmann 3 participants 3 messages Mar 13, 2004

lore
Share & tools
lore
#0 lore raw

I use lvm-over-raid5 and get these messages once a day (requiring a reboot afterwards):

EXT3-fs error (device dm-0): ext3_readdir: bad entry in directory #4804801: directory entry across blocks - offset=0, inode=0, rec_len=50000, name_len=152
Aborting journal on device dm-0.
add_dirent_to_buf: aborting transaction: Journal has aborted in __ext3_journal_get_write_access<2>EXT3-fs error (device dm-0) in add_dirent_to _buf: Journal has aborted
EXT3-fs error (device dm-0) in ext3_writeback_writepage: IO failure
EXT3-fs error (device dm-0) in ext3_writeback_writepage: IO failure
ext3_abort called.
EXT3-fs abort (device dm-0): ext3_journal_start: Detected aborted journal
Remounting filesystem read-only
EXT3-fs error (device dm-0) in start_transaction: Journal has aborted
EXT3-fs error (device dm-0) in ext3_delete_inode: Journal has aborted
EXT3-fs error (device dm-0) in ext3_create: Journal has aborted
EXT3-fs error (device dm-0): ext3_readdir: bad entry in directory #4804801: directory entry across blocks - offset=0, inode=0, rec_len=50000, name_len=152
EXT3-fs error (device dm-0): ext3_readdir: bad entry in directory #4804801: directory entry across blocks - offset=0, inode=0, rec_len=50000, name_len=152
EXT3-fs error (device dm-0): ext3_readdir: bad entry in directory #4804801: directory entry across blocks - offset=0, inode=0, rec_len=50000, name_len=152

e2fsck after rebooting shows no errors and nothing to fix, and in fact, in this very incident my home directory was missing, after rebooting it was there again, so so far this doesn't look like on-disk data corruption.

About my configuration:

5 IDE disks were combined into one raid5, with lvm on top. Theer are two lvs on the raid, one formatted with ext3 and one with reiserfs. the array was not degraded and not rebuilding. Data throughput under 2.6 is much lower than under 2.4, though (and 2.6 takes enourmous amounts of cpu for reading from the raid5 array), but this issue is probably a seperate problem.

Both partitions currently undergo heavy filesystem activity, mainly untar'ing big tars with lots of medium-sized files (e.g. 10gb of jpeg files, or cvs directories).

Reiserfs so far never gave a problem, neither did ext3 filesystems on normal harddisk partitions (although the latter ones were never under write stress like the partitions on the lv partitions).

There are no other kernel messages between mounting the volume and the problem.

I can use this machine for many hours under no stress without any problems.

I had these problems on 2.6.3 and 2.6.4, other 2.6. kernels have not been tested.

Using 2.4 on the same machine (lvm1) doesn't show any problems (the machine is a dual P-III 1ghz).

Summary: the ext3 partition regularly gives me these problems (about once per day), while reiserfs on the same device does not. Neither of them make problems under 2.4.

Hope that helps,

Signature · 7 lines
      -----==-                                             |
      ----==-- _                                           |
      ---==---(_)__  __ ____  __       Marc Lehmann      +--
      --==---/ / _ \/ // /\ \/ /       [email protected]      |e|
      -=====/_/_//_/\_,_/ /_/\_\       XX11-RIPE         --+
    The choice of a GNU generation                       |
                                                         |
#1 lore raw
Quoted 7 lines · Marc Lehmann

I use lvm-over-raid5 and get these messages once a day (requiring a reboot afterwards):

EXT3-fs error (device dm-0): ext3_readdir: bad entry in directory #4804801: directory entry across blocks - offset=0, inode=0, rec_len=50000, name_len=152
Aborting journal on device dm-0.

(and fsck comes up clean)

There have been earlier reports of this. Too many for it to be some random
glitch. We've had similar reports in 2.4, usually with raid5.

I'm fairly confident in ext3 - it's hard to think of an ext3-level bug which wouldn't have 10x as many reports from non-md users. But perhaps some timing unique to the MD layer is triggering some ext3 bug.

Joe, Neil: have you spotted reports like this? Any suggestions as to how to track it down a bit?

#2 lore raw
Quoted 14 lines · Andrew Morton
Quoted 7 lines · Marc Lehmann

I use lvm-over-raid5 and get these messages once a day (requiring a reboot afterwards):

EXT3-fs error (device dm-0): ext3_readdir: bad entry in directory #4804801: directory entry across blocks - offset=0, inode=0, rec_len=50000, name_len=152
Aborting journal on device dm-0.

(and fsck comes up clean)

There have been earlier reports of this. Too many for it to be some random
glitch. We've had similar reports in 2.4, usually with raid5.

Quoted 6 lines

I'm fairly confident in ext3 - it's hard to think of an ext3-level bug which wouldn't have 10x as many reports from non-md users. But perhaps some timing unique to the MD layer is triggering some ext3 bug.

Joe, Neil: have you spotted reports like this? Any suggestions as to how to track it down a bit?

I, too, have been experiencing this with ext3 on top of lvm on top of raid5. I also have a dual-proc machine.

It seems to be some sort of race condition because it is triggered by multiple disk-io intensive processes using the same volume. Many mornings, when I first login to this machine which runs all of the time, I find that one or more of the volumes is mounted read-only. Sometimes e2fsck shows errors and sometimes it doesn't.