corruption
I did again a large test comparing two identical trees. Found again corruption, and, upon inspection, the disk files did not differ - this is in-core corruption only.
A few days ago:
diff -r /c2/linux/linux-2.4.0-test10/linux/include/asm-sparc/ecc.h /g1/linux/li\nux-2.4.0-test10/linux/include/asm-sparc/ecc.h
80,83c80,95
< #define ECC_FADDR0_CACHE 0x00000800 < #define ECC_FADDR0_SIZE 0x00000700 < #define ECC_FADDR0_TYPE 0x000000f0 < #define ECC_FADDR0_PADDR 0x0000000f
---
Quoted 8 lines
#define ECC_FADDR0_Ccount << RATIO_SCALE_LOG;
if (db->bytes_out != 0)
{
new_ratio /= db->bytes_out;
}
if (new_ratio < db->ratio || new_ratio < 1 * RATIO_SCALE)
{Quoted 8 lines
bsd_clear (db);
return 1;
}
db->ratio = new_ratio;
}
}
return 0;
}Here the corruption starts precisely 3072 bytes into the file (which lives on a filesystem with 1024-byte blocks).
But the tail is a fragment of drivers/isdn/isdn_bsdcomp.c starting at an offset of 7168 bytes.
The former lives on blocks 6373895 6373896 6373897 6373898 6373899, the other on blocks 2475568...2475579,2475616...2475628.
Today:
diff -r /g1/linux/linux-2.4.0-test11vanilla/linux/net/sched/sch_cbq.c /c2/linux\
/linux-2.4.0-test11vanilla/linux/net/sched/sch_cbq.c
2000c2000,2115
< cbq_destr\201XM^@\202XM^@^@^@^@^@^@^@^@^@^@^@...
(lots of nulls)
with corruption starting at an offset of 47104=46*1024 bytes. Don't know where the corruption part is from.
Andries
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
Quoted 4 lines · On Wed, 29 Nov 2000 [email protected] wrote:
I did again a large test comparing two identical trees. Found again corruption, and, upon inspection, the disk files did not differ - this is in-core corruption only.
Ok. It definitely looks like the 1kB thing has become broken somehow.
The fact that it is in-core only doesn't mean that much - it could still easily be just problems at read-time, and if you have an IDE disk I would strongly suggest you try out the patch that Jens Axboe posted, re-initializing the "head" pointer when doing a re-merge.
That said, the VM/ext2 angle should definitely be looked at too. Nothing has really changed there in some time - can you give a rough estimate on when you suspect you started seeing it? Ie is it new to one of the test11 pre-kernels, or does it happen so occasionally that you can't tell whether it happened much earlier too?
Linus
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
Quoted 10 lines · Linus Torvalds
The fact that it is in-core only doesn't mean that much - it could still easily be just problems at read-time, and if you have an IDE disk I would strongly suggest you try out the patch that Jens Axboe posted, re-initializing the "head" pointer when doing a re-merge.
That said, the VM/ext2 angle should definitely be looked at too. Nothing has really changed there in some time - can you give a rough estimate on when you suspect you started seeing it? Ie is it new to one of the test11 pre-kernels, or does it happen so occasionally that you can't tell whether it happened much earlier too?
Problem fixed by Jens' patch had been there since March, so if it's a mix of __make_request() screwing up and something else... Urgh.
I'ld really like to see details on the box with ext2 corruption on SCSI. Tigran, IIRC you had it on SCSI boxen, right? Could you send me relevant part of logs?
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
Quoted 5 lines · Alexander Viro
I'ld really like to see details on the box with ext2 corruption on SCSI. Tigran, IIRC you had it on SCSI boxen, right? Could you send me relevant part of logs?
I definitely did have this very corruption on a 4xXeon SCSI-only box. But the bad news is that I reinstalled redhat7 on it immediately after this happened so I don't have the logs. _However_, I don't need that particular root filesystem there anymore (since more disks arrive today and I'm rearranging stuff) so I'll try and corrupt it for you right now. Using test12-pre3, unless you have better suggestions on what to do to help.
Regards,
Tigran
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
Quoted 8 lines · Tigran Aivazian
Quoted 5 lines · Alexander Viro
I'ld really like to see details on the box with ext2 corruption on SCSI. Tigran, IIRC you had it on SCSI boxen, right? Could you send me relevant part of logs?
I definitely did have this very corruption on a 4xXeon SCSI-only box. But
"This" as in "range of blocks duplicated onto another range", "random crap in indirect blocks" or both?
Quoted 5 lines
the bad news is that I reinstalled redhat7 on it immediately after this happened so I don't have the logs. _However_, I don't need that particular root filesystem there anymore (since more disks arrive today and I'm rearranging stuff) so I'll try and corrupt it for you right now. Using test12-pre3, unless you have better suggestions on what to do to help.
Could you look for duplicates too?
TIA,
Al
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
Quoted 1 line · Alexander Viro
Could you look for duplicates too?
will do. One useful finding so far -- trying simultaneous mke2fs /dev/sdX1 for X = {b,c,d,e,f} deadlocks the machine dead (and without kdb such death was in vain). (each disk is 37G, RAM is 6G)
I know this is offtopic for this thread but not for this list.
I am continuing to pursue the corruption. Nothing yet.
Regards,
Tigran
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
Quoted 3 lines · Alexander Viro
Problem fixed by Jens' patch had been there since March, so if it's a mix of
__make_request()screwing up and something else... Urgh.
No, the bug really got introduced in test11 due to the request merging stuff.
The patch may _look_ like it fixed a generic problem that has been there forever, but we didn't actually need the spinlock for initializing "head" at all. It's initialized to a constant offset within the unchaning request queue, so we can happily do it outside the spinlock.
The reason the initialization was moved inside the spinlock was really just that it had to be re-initialized for the case where we re-did the merge, so it had to be moved down to inside the loop - and it just happens to happen inside the spinlock now.
So the spinlock protection was never relevant to the bug - forgetting to re-initialize a variable when a straight-line code was turned into a loop was the bug.
Quoted 3 lines
I'ld really like to see details on the box with ext2 corruption on SCSI. Tigran, IIRC you had it on SCSI boxen, right? Could you send me relevant part of logs?
I suspect that Tigran may have seen other instability (of which we had lots back when he saw it), and that the current rash is for the IDE problem only.
Which is not to say that there might not be SCSI issues or other issues too, but I'm also not convinced that the SCSI thing might not just be a red herring at this point.
Linus
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
Quoted 4 lines · Linus Torvalds
Quoted 1 line · Alexander Viro
Quoted 8 lines
Quoted 2 lines
Problem fixed by Jens' patch had been there since March, so if it's a mix of
__make_request()screwing up and something else... Urgh.No, the bug really got introduced in test11 due to the request merging stuff.
The patch may _look_ like it fixed a generic problem that has been there forever, but we didn't actually need the spinlock for initializing "head"
Sure.
Quoted 2 lines
at all. It's initialized to a constant offset within the unchaning request queue, so we can happily do it outside the spinlock.
Actually, I was not thinking about spinlock. What I missed was the fact that again: was quite recent. My apologies...
Quoted 11 lines
Quoted 3 lines
I'ld really like to see details on the box with ext2 corruption on SCSI. Tigran, IIRC you had it on SCSI boxen, right? Could you send me relevant part of logs?
I suspect that Tigran may have seen other instability (of which we had lots back when he saw it), and that the current rash is for the IDE problem only.
Which is not to say that there might not be SCSI issues or other issues too, but I'm also not convinced that the SCSI thing might not just be a red herring at this point.
There are two quite distinct patterns: duplicated range vs. crap in metadata. The former looks like a bug caught by Jens. The latter (especially in bitmaps) seems to be older[1] and independent from elevator stuff. _That_ may be a fs/buffer.c or fs/ext2/* bug. The former definitely lives below the fs/buffer.c level.
[1] "older" may mean "shared with 2.2" here - ISTR bug reports looking like that and IIRC they were never resolved. BTW, if you know some searchable l-k archive... DN sucks coprolites through the straw these days ;-/
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
Re: searchable l-k archive
Quoted 2 lines · Al Viro
BTW, if you know some searchable l-k archive... DN sucks coprolites through the straw these days ;-/
There are archives of pretty much all of the linux- and oss-development related mailing lists at:
For l-k in particular, I use
http://marc.theaimsgroup.com/?l=linux-kernel&r=1&w=4
Cheers, Andreas
Signature · 7 lines
Andreas Dilger \ "If a man ate a pound of pasta and a pound of antipasto,
\ would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/ -- Dogbert
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/Quoted 1 line · Alexander Viro
that again: was quite recent. My apologies...
Never mind, strict patch reading was obviously misleading in this case.
Quoted 2 lines
[1] "older" may mean "shared with 2.2" here - ISTR bug reports looking like that and IIRC they were never resolved. [..]
I don't recall any report after 2.2.1x series but I might have missed them. Previous reports may be due the bugs that got fixed there.
Andrea
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
Quoted 1 line · Alexander Viro
Problem fixed by Jens' patch had been there since March, so if it's a
No, it's there only since Jens fixed the request merging bug in test11 or so.
With previous kernel the head pointer couldn't change so that change was unnecessary and initializing it outside the critical section was a micro scalability optimization :).
Andrea
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
Quoted 9 lines · Andrea Arcangeli
Quoted 1 line · Alexander Viro
Problem fixed by Jens' patch had been there since March, so if it's a
No, it's there only since Jens fixed the request merging bug in test11 or so.
With previous kernel the head pointer couldn't change so that change was unnecessary and initializing it outside the critical section was a micro scalability optimization :).
To be honest, I have a big problem with micro optimisations that prevent the big optimisations from happening.
Would it be an idea to explicitly comment such dangerous micro optimisations so people implementing the big optimisations later on won't run into nasty surprises?
regards,
Rik
--
Hollywood goes for world dumbination,
Trailer at 11. http://www.surriel.com/
http://www.conectiva.com/ http://distro.conectiva.com.br/
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
Quoted 6 lines · Rik van Riel
To be honest, I have a big problem with micro optimisations that prevent the big optimisations from happening.
Would it be an idea to explicitly comment such dangerous micro optimisations so people implementing the big optimisations later on won't run into nasty surprises?
Did you read the code we're talking about?
Andrea
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/
Quoted 9 lines · Andrea Arcangeli
Quoted 6 lines · Rik van Riel
To be honest, I have a big problem with micro optimisations that prevent the big optimisations from happening.
Would it be an idea to explicitly comment such dangerous micro optimisations so people implementing the big optimisations later on won't run into nasty surprises?
Did you read the code we're talking about?
This particular piece of code may be a bad example of my "complaint", but I guess we can just as easily take something like shrink_mmap() as our example ...
regards,
Rik
--
Hollywood goes for world dumbination,
Trailer at 11. http://www.surriel.com/
http://www.conectiva.com/ http://distro.conectiva.com.br/
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
Please read the FAQ at http://www.tux.org/lkml/