Bug in ext3
I recently compiled support for ext3 into the kernel (2.4.15-pre4) and booted that kernel onto a system that didn't have any ext3 partitions. On boot I got these messages:
JBD: no valid journal superblock found
JBD: no valid journal superblock found
EXT3-fs: error loading journal.
This was when it mounted the root filesystem. After several minutes of disk activity on /, I got this:
EXT2-fs error (device sd(8,20)): ext2_free_blocks: Freeing blocks not in datazone - block = 4294965248, count = 6872
Then it was mounted ro. Nothing happened to any of the other partitions (including the 250gig RAID partition, just the root partition. I am assuming that even though it was not an ext3 partition, that the ext3 driver took control of it. I tried fsck and remount,rw, but the problem kept occuring (although with a different message about when removing a file that the bit was already cleared).
Rebuilding without ext3 solves the symptoms, but of course not the real problem.
Ben
Signature · 4 lines
.----------=======-=-======-=========-----------=====------------=-=-----. / Ben Collins -- Debian GNU/Linux \ ` [email protected] -- [email protected] -- [email protected] ' `---=========------=======-------------=-=-----=-===-======-------=--=---'
Quoted 9 lines · Ben Collins
I recently compiled support for ext3 into the kernel (2.4.15-pre4) and booted that kernel onto a system that didn't have any ext3 partitions. On boot I got these messages:
JBD: no valid journal superblock found
JBD: no valid journal superblock found
EXT3-fs: error loading journal.
It sounds like the superblock claims to be an ext3 fs, but something has scrogged the journal file.
e2fsck should have removed the journal in this situation, with the message "*** ext3 journal has been deleted - filesystem is now ext2 only ***".
Please send the output of dumpe2fs, and of `fsck -fy'.
Probably you can repair it by booting with init=/bin/sh' and running tune2fs -O ^has-journal' agains the fs, and then fscking it.
Quoted 27 lines · Andreas Dilger
Quoted 9 lines · Ben Collins
I wont say that I am absolutely 100% sure that ext3 was never tried on this filesystem. I am pretty certain, but I'm guessing it doesn't really make much difference at this point. Your scenario of the corruption makes sense. I'll see if I can test your patch at some point (but I most likely cannot).
Filesystem features:
has_journalfiletypesparse_super
...
Journal inode: 48What I'm thinking happened here is at some point long ago (maybe before the server was put into production, who knows) someone tested ext3 on it. When they were done, they deleted the .journal file (inode #48) but e2fsck didn't clean up the superblock journal fields, and it went unnoticed until now.
The other alternative is that you have some sort of random single-bit data corruption going on (the journal inode is also a single bit set, 48 =
0x30, but a different bit thanhas_journal, =0x0004).In any case, with e2fsprogs 1.18 (and probably _only_ that version) it doesn't complain about
has_journal, but it also doesn't check if the journal is bad and clean it up. When you try to start with an ext3-aware kernel, it conspires to corrupt inode 48 when it tries to unload the journal, even when it knows the journal is bad.
Quoted 3 lines
What would be interesting to correlate is what inode 48 is (probably a directory, or you wouldn't have noticed it at all), with the corruption problems you are having while ext3 is loaded.
48 /usr/lib/perl5/5.005/File/Copy.pm
Since this file is pretty small, I can only assume that it overwrote some adjacent files. There is some corruption in this file (luckily in the comment area :) starting at the 25th byte, and extending 12 bytes in length. Here's the values from hexedit:
00 00 00 01 00 00 00 00 00 00 00 00
Signature · 4 lines
.----------=======-=-======-=========-----------=====------------=-=-----. / Ben Collins -- Debian GNU/Linux \ ` [email protected] -- [email protected] -- [email protected] ' `---=========------=======-------------=-=-----=-===-======-------=--=---'
Quoted 5 lines · Ben Collins
Quoted 3 lines · Andreas Dilger
What would be interesting to correlate is what inode 48 is (probably a directory, or you wouldn't have noticed it at all), with the corruption problems you are having while ext3 is loaded.
Quoted 8 lines
48
/usr/lib/perl5/5.005/File/Copy.pmSince this file is pretty small, I can only assume that it overwrote some adjacent files. There is some corruption in this file (luckily in the comment area :) starting at the 25th byte, and extending 12 bytes in length. Here's the values from hexedit:
00 00 00 01 00 00 00 00 00 00 00 00
No, this would be the only expected corruption - there are 3 32-bit fields that that get written to disk in journal_update_superblock(), and these are consistent with that.
That means the source of the other corruption is unknown.
Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/
Quoted 11 lines · Andrew Morton
Quoted 9 lines · Ben Collins
I recently compiled support for ext3 into the kernel (2.4.15-pre4) and booted that kernel onto a system that didn't have any ext3 partitions. On boot I got these messages:
JBD: no valid journal superblock found
JBD: no valid journal superblock found
EXT3-fs: error loading journal.
Quoted 8 lines
It sounds like the superblock claims to be an ext3 fs, but something has scrogged the journal file.
e2fsck should have removed the journal in this situation, with the message "*** ext3 journal has been deleted - filesystem is now ext2 only ***".
Please send the output of dumpe2fs, and of `fsck -fy'.
No, it has always been an ext2 filesystem, and never was ext3. Fsck shows no errors. The point being that I do _not_ want my root filesystem to be ext3, but I do want ext3 built into the kernel. That case should not cause a problem like I have seen.
Ben
Signature · 4 lines
.----------=======-=-======-=========-----------=====------------=-=-----. / Ben Collins -- Debian GNU/Linux \ ` [email protected] -- [email protected] -- [email protected] ' `---=========------=======-------------=-=-----=-===-======-------=--=---'
Quoted 13 lines · Andreas Dilger
Quoted 12 lines · Ben Collins
Quoted 3 lines · Andreas Dilger
What would be interesting to correlate is what inode 48 is (probably a directory, or you wouldn't have noticed it at all), with the corruption problems you are having while ext3 is loaded.
48
/usr/lib/perl5/5.005/File/Copy.pmSince this file is pretty small, I can only assume that it overwrote some adjacent files. There is some corruption in this file (luckily in the comment area :) starting at the 25th byte, and extending 12 bytes in length. Here's the values from hexedit:
Quoted 7 lines
Quoted 1 line
00 00 00 01 00 00 00 00 00 00 00 00No, this would be the only expected corruption - there are 3 32-bit fields that that get written to disk in
journal_update_superblock(), and these are consistent with that.That means the source of the other corruption is unknown.
The "other" corruption only occured while booted with the ext3-enabled kernel. They haven't appeared under the non-ext3 kernel at all. Even after it got mounted read-only, performing an fsck, and remounting read-write, it would reoccur over and over. So this "other" corruption doesn't even sound like it can be caused by the scenario you described (which sounds like a one shot problem).
Ben
Signature · 4 lines
.----------=======-=-======-=========-----------=====------------=-=-----. / Ben Collins -- Debian GNU/Linux \ ` [email protected] -- [email protected] -- [email protected] ' `---=========------=======-------------=-=-----=-===-======-------=--=---'
Quoted 8 lines · Ben Collins
Quoted 2 lines
Please send the output of dumpe2fs, and of `fsck -fy'.
No, it has always been an ext2 filesystem, and never was ext3. Fsck shows no errors. The point being that I do _not_ want my root filesystem to be ext3, but I do want ext3 built into the kernel. That case should not cause a problem like I have seen.
ext3 thinks that the filesystem's superblock has the EXT3_FEATURE_COMPAT_HAS_JOURNAL bit set in the s_feature_compat field of the on-disk superblock.
It's probable that that bit _is_ set. ext2 will never notice it.
Please: the dumpe2fs output?
-
Quoted 20 lines · Ben Collins
Quoted 19 lines · Andreas Dilger
Quoted 5 lines · Ben Collins
Quoted 3 lines · Andreas Dilger
What would be interesting to correlate is what inode 48 is (probably a directory, or you wouldn't have noticed it at all), with the corruption problems you are having while ext3 is loaded.
Quoted 8 lines
48
/usr/lib/perl5/5.005/File/Copy.pmSince this file is pretty small, I can only assume that it overwrote some adjacent files. There is some corruption in this file (luckily in the comment area :) starting at the 25th byte, and extending 12 bytes in length. Here's the values from hexedit:
00 00 00 01 00 00 00 00 00 00 00 00No, this would be the only expected corruption - there are 3 32-bit fields that that get written to disk in
journal_update_superblock(), and these are consistent with that.
Quoted 8 lines
Quoted 1 line
That means the source of the other corruption is unknown.
The "other" corruption only occured while booted with the ext3-enabled kernel. They haven't appeared under the non-ext3 kernel at all. Even after it got mounted read-only, performing an fsck, and remounting read-write, it would reoccur over and over. So this "other" corruption doesn't even sound like it can be caused by the scenario you described (which sounds like a one shot problem).
Ben, just for kicks, try ext3 with 2.4.14 instead of 2.4.15-pre4.
Have you been able to reproduce this on another machine?
Also, post the output of lspci -vv.
Mike
Quoted 9 lines · Ben Collins
Quoted 1 line · Andreas Dilger
That means the source of the other corruption is unknown.
The "other" corruption only occured while booted with the ext3-enabled kernel. They haven't appeared under the non-ext3 kernel at all. Even after it got mounted read-only, performing an fsck, and remounting read-write, it would reoccur over and over. So this "other" corruption doesn't even sound like it can be caused by the scenario you described (which sounds like a one shot problem).
I'm not saying it wasn't caused by ext3, I'm just saying that it is unknown.
The problem I described _would_ happen each time the filesystem was tried to be mounted as ext3, but it would only happen to the single file given as the journal inode (i.e. inode #48 in your case).
You say that remounting the root fs read-only, running fsck, and then remounting read-write would cause the filesystem to be corrupted? In that case, the problem _has_ to be in ext2, because the root filesystem was mounted as ext2, so there is _no_ way that ext3 could corrupt it again (the two filesystem codes are completely separate).
Note also, that if you have a serious corruption in the filesystem, and you run e2fsck on it while it is mounted (read only, presumably), then you need to reboot the system after e2fsck is done. Otherwise, the kernel cache may not be consistent with what is on disk, and anything that gets written to the disk again from the kernel has the possibility of corrupting the filesystem.
Normally, this doesn't happen, but it is possible. With the directories in the page cache (since 2.4.5 or so), the block device and the directory pages are not coherent. This makes it much more important to reboot if e2fsck makes major changes to the filesystem. Regular files have been in the page cache for a long time, but e2fsck does not change the contents of files, so it is not normally an issue.
This _might_ be worked around by e2fsck calling an ioctl to flush the kernel caches, but that would probably be complex, and given the complexity of system startup scripts today it is hard to limit the number of affected directories/processes. An alternative would be to put e2fsck into the initrd, so that root can be checked before it is mounted, but that is also a change to the way systems are started.
There _may_ also be some issues with the block devices in page cache (i.e. disconnect between inode tables or bitmaps between the block device user (e2fsck) and what the kernel sees, but that would be an outright bug.
Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/
Quoted 10 lines · Andrew Morton
Quoted 8 lines · Ben Collins
Quoted 2 lines
Please send the output of dumpe2fs, and of `fsck -fy'.
No, it has always been an ext2 filesystem, and never was ext3. Fsck shows no errors. The point being that I do _not_ want my root filesystem to be ext3, but I do want ext3 built into the kernel. That case should not cause a problem like I have seen.
Quoted 7 lines
ext3 thinks that the filesystem's superblock has the
EXT3_FEATURE_COMPAT_HAS_JOURNALbit set in thes_feature_compatfield of the on-disk superblock.It's probable that that bit _is_ set. ext2 will never notice it.
Please: the dumpe2fs output?
Seems it does have the field set. I guess the bug is then that if there is no journal, then it shoudl fail to mount it, so ext2 will take over. Is there any reason to mount a partition as ext3 if there is no journal to be found?
Filesystem volume name: <none> Last mounted on: <not available> Filesystem UUID: <none> Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: has_journal filetype sparse_super Filesystem state: not clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 1015808 Block count: 2028288 Reserved block count: 101414 Free blocks: 372624 Free inodes: 690438 First block: 0 Block size: 4096 Fragment size: 4096 Blocks per group: 32768 Fragments per group: 32768 Inodes per group: 16384 Inode blocks per group: 512 Last mount time: Thu Nov 15 10:07:12 2001 Last write time: Thu Nov 15 15:55:23 2001 Mount count: 2 Maximum mount count: 20 Last checked: Thu Nov 15 08:48:40 2001 Check interval: 15552000 (6 months) Next check after: Tue May 14 09:48:40 2002 Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 128
Signature · 4 lines
.----------=======-=-======-=========-----------=====------------=-=-----. / Ben Collins -- Debian GNU/Linux \ ` [email protected] -- [email protected] -- [email protected] ' `---=========------=======-------------=-=-----=-===-======-------=--=---'
Quoted 30 lines · Ben Collins
Seems it does have the field set. I guess the bug is then that if there is no journal, then it shoudl fail to mount it, so ext2 will take over. Is there any reason to mount a partition as ext3 if there is no journal to be found?
Filesystem volume name: <none> Last mounted on: <not available> Filesystem UUID: <none> Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: has_journal filetype sparse_super Filesystem state: not clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 1015808 Block count: 2028288 Reserved block count: 101414 Free blocks: 372624 Free inodes: 690438 First block: 0 Block size: 4096 Fragment size: 4096 Blocks per group: 32768 Fragments per group: 32768 Inodes per group: 16384 Inode blocks per group: 512 Last mount time: Thu Nov 15 10:07:12 2001 Last write time: Thu Nov 15 15:55:23 2001 Mount count: 2
Quoted 8 lines
Maximum mount count: 20 Last checked: Thu Nov 15 08:48:40 2001 Check interval: 15552000 (6 months) Next check after: Tue May 14 09:48:40 2002 Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 128
Are you running a current version of e2fsprogs? 1.25?
If you are, then this indicates that the filesystem has has_journal set, but it doesn't have a journal inode. That is certainly something which e2fsck should detect and fix. This may be a fsck bug.
You should be able to fix this with `tune2fs -O ^has-journal' on the unmounted or readonly fs.
Quoted 5 lines · Ben Collins
Quoted 4 lines · Andrew Morton
ext3 thinks that the filesystem's superblock has the
EXT3_FEATURE_COMPAT_HAS_JOURNALbit set in thes_feature_compatfield of the on-disk superblock.
Quoted 8 lines
Quoted 3 lines
It's probable that that bit _is_ set. ext2 will never notice it.
Please: the dumpe2fs output?
Seems it does have the field set. I guess the bug is then that if there is no journal, then it shoudl fail to mount it, so ext2 will take over. Is there any reason to mount a partition as ext3 if there is no journal to be found?
It _does_ fail to mount the filesystem as ext3, and the ext2 code properly mounts it. You can see this because the error message you got (in your previous posting said "EXT2-fs: ..." so the error came from ext2.
Please run e2fsck (1.25) to clear this up. It may be that you have other corruption in your filesystem. If you are sure you _never_ tried ext3 on this filesystem before, yet the has_journal bit is set, this could be an indication of memory or cable problems.
Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/
Quoted 9 lines · Andrew Morton
Are you running a current version of e2fsprogs? 1.25?
If you are, then this indicates that the filesystem has
has_journalset, but it doesn't have a journal inode. That is certainly something which e2fsck should detect and fix. This may be a fsck bug.You should be able to fix this with `tune2fs -O ^has-journal' on the unmounted or readonly fs.
Actually, it's 1.18. I'll upgrade e2fsprogs. At the same time, would it not be prudent to make ext3 fail to mount if it cannot setup the journal? If it decides to keep going in the event that there is no journal, it should not break like it did.
Ben
Signature · 4 lines
.----------=======-=-======-=========-----------=====------------=-=-----. / Ben Collins -- Debian GNU/Linux \ ` [email protected] -- [email protected] -- [email protected] ' `---=========------=======-------------=-=-----=-===-======-------=--=---'
Quoted 15 lines · Andreas Dilger
Quoted 5 lines · Ben Collins
Quoted 4 lines · Andrew Morton
ext3 thinks that the filesystem's superblock has the
EXT3_FEATURE_COMPAT_HAS_JOURNALbit set in thes_feature_compatfield of the on-disk superblock.Quoted 8 lines
Quoted 3 lines
It's probable that that bit _is_ set. ext2 will never notice it.
Please: the dumpe2fs output?
Seems it does have the field set. I guess the bug is then that if there is no journal, then it shoudl fail to mount it, so ext2 will take over. Is there any reason to mount a partition as ext3 if there is no journal to be found?
Quoted 8 lines
It _does_ fail to mount the filesystem as ext3, and the ext2 code properly mounts it. You can see this because the error message you got (in your previous posting said "EXT2-fs: ..." so the error came from ext2.
Please run e2fsck (1.25) to clear this up. It may be that you have other corruption in your filesystem. If you are sure you _never_ tried ext3 on this filesystem before, yet the
has_journalbit is set, this could be an indication of memory or cable problems.
Uh, something corrupted it. Believe me, there is no other corruption. I've reverted to a non-ext3 kernel, and after a day of serious IO, no problems have shown. So something is wrong, and it isn't my filesystem (the erroneous flag needs to be cleared, yes, but the fact remains that there is a problem in this case).
Ben
Signature · 4 lines
.----------=======-=-======-=========-----------=====------------=-=-----. / Ben Collins -- Debian GNU/Linux \ ` [email protected] -- [email protected] -- [email protected] ' `---=========------=======-------------=-=-----=-===-======-------=--=---'
Quoted 25 lines · Ben Collins
Quoted 15 lines · Andreas Dilger
Quoted 5 lines · Ben Collins
Quoted 4 lines · Andrew Morton
ext3 thinks that the filesystem's superblock has the
EXT3_FEATURE_COMPAT_HAS_JOURNALbit set in thes_feature_compatfield of the on-disk superblock.Quoted 8 lines
Quoted 3 lines
It's probable that that bit _is_ set. ext2 will never notice it.
Please: the dumpe2fs output?
Seems it does have the field set. I guess the bug is then that if there is no journal, then it shoudl fail to mount it, so ext2 will take over. Is there any reason to mount a partition as ext3 if there is no journal to be found?
Quoted 8 lines
It _does_ fail to mount the filesystem as ext3, and the ext2 code properly mounts it. You can see this because the error message you got (in your previous posting said "EXT2-fs: ..." so the error came from ext2.
Please run e2fsck (1.25) to clear this up. It may be that you have other corruption in your filesystem. If you are sure you _never_ tried ext3 on this filesystem before, yet the
has_journalbit is set, this could be an indication of memory or cable problems.
Quoted 5 lines
Uh, something corrupted it. Believe me, there is no other corruption. I've reverted to a non-ext3 kernel, and after a day of serious IO, no problems have shown. So something is wrong, and it isn't my filesystem (the erroneous flag needs to be cleared, yes, but the fact remains that there is a problem in this case).
Yes, true.
Can you try again with a journal in this FS with ext3?
Quoted 12 lines · Mike Fedyk
Quoted 11 lines
Quoted 5 lines
Please run e2fsck (1.25) to clear this up. It may be that you have other corruption in your filesystem. If you are sure you _never_ tried ext3 on this filesystem before, yet the
has_journalbit is set, this could be an indication of memory or cable problems.Uh, something corrupted it. Believe me, there is no other corruption. I've reverted to a non-ext3 kernel, and after a day of serious IO, no problems have shown. So something is wrong, and it isn't my filesystem (the erroneous flag needs to be cleared, yes, but the fact remains that there is a problem in this case).
Quoted 3 lines
Yes, true.
Can you try again with a journal in this FS with ext3?
Can't really do that. This is a production system. I will however try to reproduce it on another system. If I can do that, I'll try your suggestion.
Ben
Signature · 4 lines
.----------=======-=-======-=========-----------=====------------=-=-----. / Ben Collins -- Debian GNU/Linux \ ` [email protected] -- [email protected] -- [email protected] ' `---=========------=======-------------=-=-----=-===-======-------=--=---'
Quoted 11 lines · Ben Collins
Quoted 4 lines · Andreas Dilger
Please run e2fsck (1.25) to clear this up. It may be that you have other corruption in your filesystem. If you are sure you _never_ tried ext3 on this filesystem before, yet the
has_journalbit is set, this could be an indication of memory or cable problems.Uh, something corrupted it. Believe me, there is no other corruption. I've reverted to a non-ext3 kernel, and after a day of serious IO, no problems have shown. So something is wrong, and it isn't my filesystem (the erroneous flag needs to be cleared, yes, but the fact remains that there is a problem in this case).
I don't disagree that something corrupted it, but it is hard to tell from here what it could be. Looking at ext3_read_super(), it is pretty much a read-only path, except journal recovery. If, for some reason, you had an old, unrecovered ext3 journal in the fs, it is possible that recovering from it would corrupt your fs by writing old data into the fs.
This _shouldn't_ happen with newer kernels, but with old 2.2 ext3 code this was a possibility. Also, with old e2fsck code (1.18 was right at the very beginning when ext3 support was being added) it is possible that it didn't fail because of the has_journal flag, but it wasn't smart enough to detect and remove an old corrupt journal. I'm not saying this is a likely scenario either, but we don't have much to go on.
Are you _absolutely_ sure that there was never any ext3 testing done a long time ago on this fs? Looking at it closer, it seems unlikely that random corruption could cause something like the above scenario, because there are many checks before we get to journal recovery.
Looking at the ext3_read_super() path, we call ext3_load_journal(), which calls journal_wipe() and journal_load(), which both call journal_load()->load_superblock()->journal_get_superblock(), which gives us the two "JBD: no valid journal superblock found" messages, and return errors before doing anything else. Then we get the message "EXT3-fs: error loading journal." and return without mounting the fs.
Hmm, there is a possibility that journal_destroy() calling journal_update_superblock() scribbling data into the first block of the old "journal". Stephen, Andrew, we need to exit from the journal_get_superblock() with j_sb_buffer = NULL, and then check for this in journal_destroy() so we don't call journal_update_superblock(). How does the below patch look?
Ben, unfortunately, the dumpe2fs 1.18 output doesn't show the journal fields. When you upgrade to e2fsprogs 1.25, can you run dumpe2fs -h <dev> again and send the output, so we can see what is all set? It appears that at least the "has_journal" flag and a s_journal_inum are set for it to get as far as it did. The new e2fsck will clean that up properly.
Cheers, Andreas
=========================================================================
fs/jbd/journal.c +11 -4
@@ -900,7 +900,7 @@if (sb->s_header.h_magic !=htonl (JFS_MAGIC_NUMBER ) || sb->s_blocksize !=htonl (journal->j_blocksize)) {printk (KERN_WARNING "JBD: no valid journal superblock found\n" ); -return -EINVAL ; + goto error; }switch (ntohl (sb->s_header.h_blocktype)) {
@@ -912,17 +912,22 @@break ;default :printk (KERN_WARNING "JBD: unrecognised superblock format ID\n" ); -return -EINVAL ; + goto error; }if (ntohl (sb->s_maxlen) < journal->j_maxlen) journal->j_maxlen =ntohl (sb->s_maxlen);else if (ntohl (sb->s_maxlen) > journal->j_maxlen) {printk (KERN_WARNING "JBD: journal file too short\n" ); -return -EINVAL ; + goto error; }return 0 ; + +error: +brelse (bh); + journal->j_sb_buffer =NULL ; +return -EINVAL ; }/*
@@ -1027,7 +1032,8 @@/* We can now mark the journal as empty. */ journal->j_tail =0 ; journal->j_tail_sequence = ++journal->j_transaction_sequence; -journal_update_superblock (journal,1 ); +if (journal->j_sb_buffer) +journal_update_superblock (journal,1 );if (journal->j_inode)iput (journal->j_inode);
@@ -1716,7 +1710,6 @@journal_destroy_caches (); } +MODULE_LICENSE ("GPL" );module_init (journal_init);module_exit (journal_exit);
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/
Quoted 19 lines · Andreas Dilger
Quoted 11 lines · Ben Collins
Quoted 4 lines · Andreas Dilger
Please run e2fsck (1.25) to clear this up. It may be that you have other corruption in your filesystem. If you are sure you _never_ tried ext3 on this filesystem before, yet the
has_journalbit is set, this could be an indication of memory or cable problems.Uh, something corrupted it. Believe me, there is no other corruption. I've reverted to a non-ext3 kernel, and after a day of serious IO, no problems have shown. So something is wrong, and it isn't my filesystem (the erroneous flag needs to be cleared, yes, but the fact remains that there is a problem in this case).
I don't disagree that something corrupted it, but it is hard to tell from here what it could be. Looking at
ext3_read_super(), it is pretty much a read-only path, except journal recovery. If, for some reason, you had an old, unrecovered ext3 journal in the fs, it is possible that recovering from it would corrupt your fs by writing old data into the fs.
Quoted 6 lines
This _shouldn't_ happen with newer kernels, but with old 2.2 ext3 code this was a possibility. Also, with old e2fsck code (1.18 was right at the very beginning when ext3 support was being added) it is possible that it didn't fail because of the
has_journalflag, but it wasn't smart enough to detect and remove an old corrupt journal. I'm not saying this is a likely scenario either, but we don't have much to go on.
I wont say that I am absolutely 100% sure that ext3 was never tried on this filesystem. I am pretty certain, but I'm guessing it doesn't really make much difference at this point. Your scenario of the corruption makes sense. I'll see if I can test your patch at some point (but I most likely cannot).
Filesystem volume name: <none> Last mounted on: <not available> Filesystem UUID: <none> Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: has_journal filetype sparse_super Filesystem state: not clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 1015808 Block count: 2028288 Reserved block count: 101414 Free blocks: 368490 Free inodes: 688732 First block: 0 Block size: 4096 Fragment size: 4096 Blocks per group: 32768 Fragments per group: 32768 Inodes per group: 16384 Inode blocks per group: 512 Last mount time: Thu Nov 15 10:07:12 2001 Last write time: Thu Nov 15 18:29:55 2001 Mount count: 2 Maximum mount count: 20 Last checked: Thu Nov 15 08:48:40 2001 Check interval: 15552000 (6 months) Next check after: Tue May 14 09:48:40 2002 Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 128 Journal UUID: <none> Journal inode: 48 Journal device: 0x0000 First orphan inode: 0
Signature · 4 lines
.----------=======-=-======-=========-----------=====------------=-=-----. / Ben Collins -- Debian GNU/Linux \ ` [email protected] -- [email protected] -- [email protected] ' `---=========------=======-------------=-=-----=-===-======-------=--=---'
Quoted 9 lines · Ben Collins
I wont say that I am absolutely 100% sure that ext3 was never tried on this filesystem. I am pretty certain, but I'm guessing it doesn't really make much difference at this point. Your scenario of the corruption makes sense. I'll see if I can test your patch at some point (but I most likely cannot).
Filesystem features:
has_journalfiletypesparse_super
...
Journal inode: 48
What I'm thinking happened here is at some point long ago (maybe before the server was put into production, who knows) someone tested ext3 on it. When they were done, they deleted the .journal file (inode #48) but e2fsck didn't clean up the superblock journal fields, and it went unnoticed until now.
The other alternative is that you have some sort of random single-bit data corruption going on (the journal inode is also a single bit set, 48 = 0x30, but a different bit than has_journal, = 0x0004).
In any case, with e2fsprogs 1.18 (and probably _only_ that version) it doesn't complain about has_journal, but it also doesn't check if the journal is bad and clean it up. When you try to start with an ext3-aware kernel, it conspires to corrupt inode 48 when it tries to unload the journal, even when it knows the journal is bad.
What would be interesting to correlate is what inode 48 is (probably a directory, or you wouldn't have noticed it at all), with the corruption problems you are having while ext3 is loaded.
Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/
Hi,
Quoted 3 lines · Ben Collins
Seems it does have the field set. I guess the bug is then that if there is no journal, then it shoudl fail to mount it, so ext2 will take over.
It _did_ fail to mount it, and ext2 _did_ take over: that's why, when ext2 found something else wrong with the filesystem, the errors looked like
EXT2-fs error (device sd(8,20)): ext2_free_blocks: Freeing blocks not in datazone - block = 4294965248, count = 6872
which were marked as ext2 warnings, not ext3 warnings.
Cheers,
Stephen
Hi,
Quoted 5 lines · Andreas Dilger
I don't disagree that something corrupted it, but it is hard to tell from here what it could be. Looking at
ext3_read_super(), it is pretty much a read-only path, except journal recovery. If, for some reason, you had an old, unrecovered ext3 journal in the fs, it is possible that recovering from it would corrupt your fs by writing old data into the fs.
Except that can only happen once we have found a journal to recover, and the error that ext3 spat out indicated that it couldn't find a journal at all and was giving up.
Quoted 6 lines
Looking at the
ext3_read_super()path, we callext3_load_journal(), which callsjournal_wipe()andjournal_load(), which both call journal_load()->load_superblock()->journal_get_superblock(), which gives us the two "JBD: no valid journal superblock found" messages, and return errors before doing anything else. Then we get the message "EXT3-fs: error loading journal." and return without mounting the fs.
Indeed.
Quoted 6 lines
Hmm, there is a possibility that
journal_destroy()callingjournal_update_superblock()scribbling data into the first block of the old "journal". Stephen, Andrew, we need to exit from thejournal_get_superblock()withj_sb_buffer= NULL, and then check for this injournal_destroy()so we don't calljournal_update_superblock(). How does the below patch look?
Looks OK. I've done a slightly better version which catches a couple of extra cases but it's basically the same solution. I've also added a tiny patch to prevent a failed journal_wipe() from being followed by a journal_load() attempt, so we don't get the same error twice.
Patch below.
Cheers,
Stephen
Index: fs/ext3/super.c
===================================================================
RCS file: /cvsroot/gkernel/ext3/fs/ext3/super.c,v
retrieving revision 1.34.2.11
ext3/super.c +4 -3
@@ -1350,7 +1350,7 @@journal_t *journal;int journal_inum =le32_to_cpu (es->s_journal_inum);int journal_dev =le32_to_cpu (es->s_journal_dev); -int err; +int err =0 ;int really_read_only; really_read_only =is_read_only (sb->s_dev);
@@ -1400,9 +1400,10 @@ }if (!EXT3_HAS_INCOMPAT_FEATURE (sb,EXT3_FEATURE_INCOMPAT_RECOVER )) -journal_wipe (journal, !really_read_only); + err =journal_wipe (journal, !really_read_only); +if (!err) + err =journal_load (journal); - err =journal_load (journal);if (err) {printk (KERN_ERR "EXT3-fs: error loading journal.\n" );journal_destroy (journal);
Index: fs/jbd/journal.c
===================================================================
RCS file: /cvsroot/gkernel/ext3/fs/jbd/journal.c,v retrieving revision 1.49.2.6
jbd/journal.c +29 -7
@@ -773,6 +773,18 @@return journal; } +/* +* If the journal init or create aborts, we need to mark the journal +* superblock as being NULL to prevent the journal destroy from writing +* back a bogus superblock. +*/ +static void journal_fail_superblock (journal_t *journal) +{ +struct buffer_head *bh = journal->j_sb_buffer; +brelse (bh); + journal->j_sb_buffer =NULL ; +} +/* * Given a journal_t structure, initialise the various fields for * startup of a new journaling session. We use this both when creating
@@ -826,6 +838,7 @@if (journal->j_maxlen <JFS_MIN_JOURNAL_BLOCKS ) {printk (KERN_ERR "Journal length (%d blocks) too short.\n" , journal->j_maxlen); +journal_fail_superblock (journal);return -EINVAL ; }
@@ -916,7 +929,8 @@ {struct buffer_head *bh;journal_superblock_t *sb; - +int err = -EIO ; + bh = journal->j_sb_buffer;J_ASSERT (bh !=NULL );
@@ -926,16 +940,18 @@if (!buffer_uptodate (bh)) {printk (KERN_ERR "JBD: IO error reading journal superblock\n" ); -return -EIO ; + goto out; } } sb = journal->j_superblock; + err = -EINVAL ; +if (sb->s_header.h_magic !=htonl (JFS_MAGIC_NUMBER ) || sb->s_blocksize !=htonl (journal->j_blocksize)) {printk (KERN_WARNING "JBD: no valid journal superblock found\n" ); -return -EINVAL ; + goto out; }switch (ntohl (sb->s_header.h_blocktype)) {
@@ -947,17 +963,21 @@break ;default :printk (KERN_WARNING "JBD: unrecognised superblock format ID\n" ); -return -EINVAL ; + goto out; }if (ntohl (sb->s_maxlen) < journal->j_maxlen) journal->j_maxlen =ntohl (sb->s_maxlen);else if (ntohl (sb->s_maxlen) > journal->j_maxlen) {printk (KERN_WARNING "JBD: journal file too short\n" ); -return -EINVAL ; + goto out; }return 0 ; + +out: +journal_fail_superblock (journal); +return err; }/*
@@ -1062,7 +1082,10 @@/* We can now mark the journal as empty. */ journal->j_tail =0 ; journal->j_tail_sequence = ++journal->j_transaction_sequence; -journal_update_superblock (journal,1 ); +if (journal->j_sb_buffer) { +journal_update_superblock (journal,1 ); +brelse (journal->j_sb_buffer); + }if (journal->j_inode)iput (journal->j_inode);
@@ -1070,7 +1093,6 @@journal_destroy_revoke (journal);unlock_journal (journal); -brelse (journal->j_sb_buffer);kfree (journal);MOD_DEC_USE_COUNT ; }
Hi,
Quoted 4 lines · Stephen C. Tweedie
Looks OK. I've done a slightly better version which catches a couple of extra cases but it's basically the same solution. I've also added a tiny patch to prevent a failed
journal_wipe()from being followed by ajournal_load()attempt, so we don't get the same error twice.
This definitely fixes that error path: I just get one, clean error now, and no corruption of the file that was masquerading as the journal. It doesn't properly release the journal inode, though, so we oops on a later ext2 mount as we think we already have the (ext3) inode in cache. Fix to follow.
Ben, thanks for this --- this level of corrupt journal is something that hasn't been tested in this way before.
--Stephen