EXT3 filesystem corruptions on AoE, RAID and LVM?
Hi all,
I'm maintaining two clusters, with machines running a mix between Debian Stable with Etch-kernels to have AoE (ATA over Ethernet support). Machines in these clusters "export" their harddisks using AoE, and one machine in the cluster imports those using the kernel "aoe"-module. On top of those imported devices, multiple RAID5-arrays are created, and LVM is running on top of RAID, ext3 on the LVM LV.
After a few days, I get EXT3-errors. like this:
Quoted 2 lines
Quoted 2 lines
EXT3-fs: mounted filesystem with ordered data mode. EXT3-fs error (device loop0):
ext3_free_blocks_sb: bit already
cleared for block 412186
Quoted 3 lines
Quoted 3 lines
Aborting journal on device loop0.
EXT3-fs error (device loop0) inext3_free_blocks_sb: Journal has aborted EXT3-fs error (device loop0) inext3_reserve_inode_write: Journal has
aborted
Quoted 2 lines
Quoted 2 lines
EXT3-fs error (device loop0) in
ext3_truncate: Journal has aborted EXT3-fs error (device loop0) inext3_reserve_inode_write: Journal has
aborted
Quoted 2 lines
Quoted 2 lines
EXT3-fs error (device loop0) in
ext3_orphan_del: Journal has aborted EXT3-fs error (device loop0) inext3_reserve_inode_write: Journal has
aborted
Quoted 3 lines
Quoted 3 lines
EXT3-fs error (device loop0) in
ext3_delete_inode: Journal has aborted__journal_remove_journal_head: freeingb_committed_data__journal_remove_journal_head: freeingb_committed_data
(...)
Quoted 3 lines
Quoted 3 lines
__journal_remove_journal_head: freeingb_committed_dataext3_abortcalled.
EXT3-fs error (device loop0):ext3_journal_start_sb: Detected aborted
journal
Quoted 2 lines
Quoted 2 lines
Remounting filesystem read-only
__journal_remove_journal_head: freeingb_committed_data
FSCK'ing the filesystem fixes those errors, but after a few days (or weeks, depending on the fs load) the corruptions appear again. I might be worth telling you that there are no other suspicious messages in my logs.
I saw some other discussions on the mailinglist, but I don't think their related to my problems. I don't know if I need to file a bug on this, neither do I know which details you need to help me solve this problem. So for now I just want to here your thoughts. FYI:
Kernel information for cluster 1:
Quoted 2 lines
Quoted 2 lines
root@infinity:~# uname -a Linux infinity 2.6.17-2-686 #1 SMP Wed Sep 13 16:34:10 UTC 2006 i686
GNU/Linux
And cluster 2:
Quoted 2 lines
Quoted 2 lines
dust:~# uname -a Linux dust 2.6.18-3-686 #1 SMP Thu Nov 23 20:49:23 UTC 2006 i686
GNU/Linux
Note that these are not vanilla kernels, but Debian kernels. However, AFAIK there are no Debian-specific patches to AoE, ext3, LVM or RAID.
Thanks for your replies!
Best regards,
-- Bas van Schaik
Hi all,
Sorry for being this brutal and replying to my own post, but is there anyone out here who might be able to help me with this problem? Or at least help me with debugging it? Today I've discovered one other type of inconsistency:
Quoted 2 lines
# ls -la mydir -r-S--s-wt 1 2245250426 3397732635 32768 1929-11-11 16:54 mydir
Obviously, there are no UID 2245250426 and GID 3397732635 on my filesystem, and this file certainly isn't created or modified on the 11th of November 1929. I cannot remove the directory, I get a "permission denied" (being root!). Of course, I assume an fsck will solve this problem (probably by just removing this entry from the filesystem and attaching it to lost+found), but I keep checking my filesystem about every five days now! And that means another 5 hours of downtime with this special type of "cluster"...
Thanks in advance for all your thoughts!
-- Bas
Quoted 88 lines · Bas van Schaik
Hi all,
I'm maintaining two clusters, with machines running a mix between Debian Stable with Etch-kernels to have AoE (ATA over Ethernet support). Machines in these clusters "export" their harddisks using AoE, and one machine in the cluster imports those using the kernel "aoe"-module. On top of those imported devices, multiple RAID5-arrays are created, and LVM is running on top of RAID, ext3 on the LVM LV.
After a few days, I get EXT3-errors. like this:
Quoted 3 lines
Quoted 3 lines
EXT3-fs: mounted filesystem with ordered data mode. EXT3-fs error (device loop0):
ext3_free_blocks_sb: bit alreadycleared for block 412186
Quoted 4 lines
Quoted 4 lines
Aborting journal on device loop0.
EXT3-fs error (device loop0) inext3_free_blocks_sb: Journal has aborted EXT3-fs error (device loop0) inext3_reserve_inode_write: Journal hasaborted
Quoted 3 lines
Quoted 3 lines
EXT3-fs error (device loop0) in
ext3_truncate: Journal has aborted EXT3-fs error (device loop0) inext3_reserve_inode_write: Journal hasaborted
Quoted 3 lines
Quoted 3 lines
EXT3-fs error (device loop0) in
ext3_orphan_del: Journal has aborted EXT3-fs error (device loop0) inext3_reserve_inode_write: Journal hasaborted
Quoted 4 lines
Quoted 4 lines
EXT3-fs error (device loop0) in
ext3_delete_inode: Journal has aborted__journal_remove_journal_head: freeingb_committed_data__journal_remove_journal_head: freeingb_committed_data(...)
Quoted 4 lines
Quoted 4 lines
__journal_remove_journal_head: freeingb_committed_dataext3_abortcalled.
EXT3-fs error (device loop0):ext3_journal_start_sb: Detected abortedjournal
Quoted 3 lines
Quoted 3 lines
Remounting filesystem read-only
__journal_remove_journal_head: freeingb_committed_dataFSCK'ing the filesystem fixes those errors, but after a few days (or weeks, depending on the fs load) the corruptions appear again. I might be worth telling you that there are no other suspicious messages in my logs.
I saw some other discussions on the mailinglist, but I don't think their related to my problems. I don't know if I need to file a bug on this, neither do I know which details you need to help me solve this problem. So for now I just want to here your thoughts. FYI:
Kernel information for cluster 1:
Quoted 3 lines
Quoted 3 lines
root@infinity:~# uname -a Linux infinity 2.6.17-2-686 #1 SMP Wed Sep 13 16:34:10 UTC 2006 i686GNU/Linux
And cluster 2:
Quoted 3 lines
Quoted 3 lines
dust:~# uname -a Linux dust 2.6.18-3-686 #1 SMP Thu Nov 23 20:49:23 UTC 2006 i686GNU/Linux
Note that these are not vanilla kernels, but Debian kernels. However, AFAIK there are no Debian-specific patches to AoE, ext3, LVM or RAID.
Thanks for your replies!
Best regards,
-- Bas van Schaik
-
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html