2.4.0-test11 ext2 fs corruption
Quoted 36 lines · Alexander Viro
Quoted 26 lines · Petr Vandrovec
Hi Al,
during weekend I was uncompressing XFree (Debian's 4.0.1-7) at home,
with 2.4.0-test11 running on Celeron 300A, 128MB RAM, SMP kernel on up. It failed to compilelbxproxy/di/main.c. After some investigation I found that they were overwritten by some source font data. fsck did not reveal any croslinked clusters, nothing. Filesystem itself uses 4KB clusters.Today I found some spare time and investigated it further. There is
same data contents in:
programs/lbxproxy/di/init.c0-8720 fonts/bdf/75dpi/lubR24.bdf 0x5000-0x7210lbxfuncs.c 0x0000-0x0EC0 lubR24.bdf 0x8000-0x8EC0 0x0EC1-0x0FFF zero 0x1000-0x5ABC lutBS08.bdf 0x0000-0x4ABC 0x5ABD-0x5FFF zero 0x6000-0x92C1 lutBS10.bdf 0x0000-0x32C1 lbxutil.c 0x0000-0x1E27 lutBS10.bdf 0x4000-0x5E27 0x1E28-0x1FFF zero 0x2000-0x3452 lutBS12.bdf 0x0000-0x1452 main.c 0-4614 lutBS12.bdf 0x2000-0x3206 options.c 0x0000-0x222E lutBS12.bdf 0x4000-0x622E 0x222F-0x2FFF zero 0x3000-0x4E30 lutBS14.bdf 0x0000-0x1E30 pm.c 0-11706 lutBS14.bdf 0x2000-0x4DA8 (blocks 722433-722459) (blocks 558899-~558927)Quoted 7 lines
Other files are intouch. As you can see, somewhat disk blocks ended somewhere else than they should in addition to correct place. I also found that data after end of file in di/*.c files are not cleared, so maybe that ide driver did a mistake? But I was not able to find how to convert either block address, or LBA adress, or CHS address (drive uses 839/240/63, but I hope that it runs in LBA) to get 558899 from 722433 or vice versa.
Quoted 3 lines
Erm... Do you mean that you've got a 1-1 correspondence in data between these two ranges? Then it looks like something way below the fs level... Weird. Could you verify it with dd?
Yes, it is identical copy. But I do not think that hdd can write same data into two places with one command...
vana:/# dd if=/dev/hdd1 bs=4096 count=27 skip=722433 | md5sum 27+0 records in 27+0 records out 613de4a7ea664ce34b2a9ec8203de0f4 vana:/# dd if=/dev/hdd1 bs=4096 count=27 skip=558899 | md5sum 27+0 records in 27+0 records out 613de4a7ea664ce34b2a9ec8203de0f4 vana:/#
I found match by searching of contents of init.c in other files.
It is just these 27 blocks; blocks before and after range differs.
Best regards,
Petr Vandrovec [email protected]
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
From: "Petr Vandrovec" <[email protected]>
Date: Tue, 28 Nov 2000 21:10:36 MET-1
Yes, it is identical copy. But I do not think that hdd can write same data into two places with one command...
Petr, did the af_inet.c assertions get triggered on this same machine?
If yes, you seem to have some crazy kernel data corruptions going on, and whatever it is would seem to be the cause of both these problems you are reporting.
Later,
David S. Miller
[email protected]
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
Quoted 6 lines · Petr Vandrovec
Quoted 2 lines
two ranges? Then it looks like something way below the fs level... Weird. Could you verify it with dd?
Yes, it is identical copy. But I do not think that hdd can write same data into two places with one command...
Quoted 9 lines
vana:/# dd if=/dev/hdd1 bs=4096 count=27 skip=722433 | md5sum 27+0 records in 27+0 records out 613de4a7ea664ce34b2a9ec8203de0f4 vana:/# dd if=/dev/hdd1 bs=4096 count=27 skip=558899 | md5sum 27+0 records in 27+0 records out 613de4a7ea664ce34b2a9ec8203de0f4 vana:/#
Bloody hell... OK, let's see. Both ranges are covered by multiple files and are way larger than one page. I.e. anything on pagecache level is extremely unlikely - pages are not searched by physical location on disk. And I really doubt that it's ext2_get_block() - we would have to get a systematic error (constant offset), then read the data in for no good reason, then forget the page->buffers, then get the right values fro ext2_get_block(), leave the data unmodified _and_ write it.
It almost looks like a request in queue got fscked up retaining the ->bh from one of the previous (also coalesced) requests and having correct ->sector. Weird.
Linus, Andrea - any ideas? Situation looks so: after massive file creation a range of disk with the data from new files (many new files) got duplicated over another range - one with the data from older files (also many of them). 27 blocks, block size == 4Kb. No intersection between inodes, fsck is happy with fs, just a data ending up in two places on disk. No warnings from IDE or ext2 drivers.
Kernel: test11 built with 2.95.2, so gcc bug may very well be there. However, I really wonder what could trigger it in ll_rw_blk.c - 5:1 that shit had hit the fan there.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
Quoted 38 lines · Alexander Viro
Quoted 6 lines · Petr Vandrovec
Quoted 2 lines
two ranges? Then it looks like something way below the fs level... Weird. Could you verify it with dd?
Yes, it is identical copy. But I do not think that hdd can write same data into two places with one command...
Quoted 9 lines
vana:/# dd if=/dev/hdd1 bs=4096 count=27 skip=722433 | md5sum 27+0 records in 27+0 records out 613de4a7ea664ce34b2a9ec8203de0f4 vana:/# dd if=/dev/hdd1 bs=4096 count=27 skip=558899 | md5sum 27+0 records in 27+0 records out 613de4a7ea664ce34b2a9ec8203de0f4 vana:/#Bloody hell... OK, let's see. Both ranges are covered by multiple files and are way larger than one page. I.e. anything on pagecache level is extremely unlikely - pages are not searched by physical location on disk. And I really doubt that it's
ext2_get_block()- we would have to get a systematic error (constant offset), then read the data in for no good reason, then forget thepage->buffers, then get the right values froext2_get_block(), leave the data unmodified _and_ write it.It almost looks like a request in queue got fscked up retaining the ->bh from one of the previous (also coalesced) requests and having correct ->sector. Weird.
Linus, Andrea - any ideas? Situation looks so: after massive file creation a range of disk with the data from new files (many new files) got duplicated over another range - one with the data from older files (also many of them). 27 blocks, block size == 4Kb. No intersection between inodes, fsck is happy with fs, just a data ending up in two places on disk. No warnings from IDE or ext2 drivers.
Quoted 3 lines
Kernel: test11 built with 2.95.2, so gcc bug may very well be there. However, I really wonder what could trigger it in
ll_rw_blk.c- 5:1 that shit had hit the fan there.
I picked up a bug in ialloc.c back in February, for which I submitted a poorly constructed patch (for which I was privately and properly flamed; as I recall my subsequent attempts to post an improved version failed for various reasons which may or may not include ORBS). Anyway, the basic idea is clear:
http://marc.theaimsgroup.com/?l=linux-kernel&m=95162877201890&w=2
I'll make a proper patch out of this if you like. This *could* cause the effect we're seeing here.
--
Daniel
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
Quoted 1 line · Alexander Viro
Bloody hell...
I don't know if this is the bug he's got, in fact I doubt it, but it's a bug and it needs fixing. The problem is, ext2_get_group_desc effectively returns two results; one of them is being assigned from on conditional paths and the other isn't. This bug will cause - on very rare occasions - the wrong group descriptor block to be marked dirty, and changes might be lost. I think what we'd see as a result is wrong block, inode and directory counts.
The fix below is kind of gross. The way I really want to do the fix is to remove one parameter from ext2_get_group_desc and thereby get rid of the troublesome side effect for good, but that kind of change isn't compatible with 'code freeze'.
(linux.2.4.0-test11)
ext2/ialloc.c renamed from ext2/ialloc.c.old +13 -9
@@ -260,7 +260,7 @@ {struct super_block * sb;struct buffer_head * bh; -struct buffer_head * bh2; +struct buffer_head * bh2, * tmpbh2;int i, j, avefreei;struct inode * inode;int bitmap_nr;
@@ -293,10 +293,11 @@/* I am not yet convinced that this next bit is necessary. i = dir->u.ext2_i.i_block_group; for (j = 0; j < sb->u.ext2_sb.s_groups_count; j++) { -tmp = ext2_get_group_desc (sb, i, &bh2); +tmp = ext2_get_group_desc (sb, i, &tmpbh2); if (tmp && (le16_to_cpu(tmp->bg_used_dirs_count) << 8) < le16_to_cpu(tmp->bg_free_inodes_count)) { +bh2 = tmpbh2; gdp = tmp; break; }
@@ -306,7 +307,7 @@*/ if (!gdp) {for (j =0 ; j < sb->u.ext2_sb.s_groups_count; j++) { - tmp =ext2_get_group_desc (sb, j, &bh2); + tmp =ext2_get_group_desc (sb, j, &tmpbh2);if (tmp &&le16_to_cpu (tmp->bg_free_inodes_count) &&le16_to_cpu (tmp->bg_free_inodes_count) >= avefreei) {
@@ -314,6 +315,7 @@ (le16_to_cpu(tmp->bg_free_blocks_count) > le16_to_cpu(gdp->bg_free_blocks_count))) { i = j; + bh2 = tmpbh2; gdp = tmp; } }
@@ -326,11 +328,11 @@* Try to place the inode in its parent directory */ i = dir->u.ext2_i.i_block_group; - tmp =ext2_get_group_desc (sb, i, &bh2); -if (tmp &&le16_to_cpu (tmp->bg_free_inodes_count)) + tmp =ext2_get_group_desc (sb, i, &tmpbh2); +if (tmp &&le16_to_cpu (tmp->bg_free_inodes_count)) { + bh2 = tmpbh2; gdp = tmp; -else - { + }else {/* * Use a quadratic hash to find a group with a * free inode
@@ -339,9 +341,10 @@ i += j;if (i >= sb->u.ext2_sb.s_groups_count) i -= sb->u.ext2_sb.s_groups_count; - tmp =ext2_get_group_desc (sb, i, &bh2); + tmp =ext2_get_group_desc (sb, i, &tmpbh2);if (tmp &&le16_to_cpu (tmp->bg_free_inodes_count)) { + bh2 = tmpbh2; gdp = tmp;break ; }
@@ -355,9 +358,10 @@for (j =2 ; j < sb->u.ext2_sb.s_groups_count; j++) {if (++i >= sb->u.ext2_sb.s_groups_count) i =0 ; - tmp =ext2_get_group_desc (sb, i, &bh2); + tmp =ext2_get_group_desc (sb, i, &tmpbh2);if (tmp &&le16_to_cpu (tmp->bg_free_inodes_count)) { + bh2 = tmpbh2; gdp = tmp;break ; }
--
Daniel
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
Quoted 11 lines · Daniel Phillips
Quoted 1 line · Alexander Viro
Bloody hell...
I don't know if this is the bug he's got, in fact I doubt it, but it's a bug and it needs fixing. The problem is,
ext2_get_group_desceffectively returns two results; one of them is being assigned from on conditional paths and the other isn't. This bug will cause - on very rare occasions - the wrong group descriptor block to be marked dirty, and changes might be lost. I think what we'd see as a result is wrong block, inode and directory counts.
Quoted 4 lines
The fix below is kind of gross. The way I really want to do the fix is to remove one parameter from
ext2_get_group_descand thereby get rid of the troublesome side effect for good, but that kind of change isn't compatible with 'code freeze'.
Yes, it is. Moreover, correct solution is slightly different and changes ext2_get_group_desc() semantics. Wait until tomorrow, OK?
However, it's not the source of reported problems. We clearly have b0rken data in bitmaps themselves.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/