filesystem corruption (ReiserFS, 2.6.6): regions replaced by \000 bytes
Hi folks.
I'm afraid this bug-report will be rather worthless as it is, because the bug has proven remarkable elusive and has defeated all my attempts to track it down to a precise test case or set of circumstances. But since it seems important, I thought it might be worth a post anyway. Any help is appreciated in clarifying the circumstances which trigger the problem, or generally in making this report more useful.
The bottom line: I've experienced file corruption, of the following
nature: consecutive regions (all, it seems, aligned on 256-byte boundaries, and typically around 1kb or 2kb in length) of seemingly random files are replaced by null bytes. The filesystem is ReiserFS (but as nearly all my filesystems are ReiserFS anyway, I cannot conclusively say that the bug is indeed in ReiserFS). The problem has occurred with kernel versions Debian-packaged 2.6.6-1-686-smp and home-compiled 2.6.6-mm2 (SMP also). The hardware is an Intel bi-PII450 (with 256MB RAM) using aic7xxx as low-level disk driver; and I have good reasons to think that the hardware is sound (and the memory banks in particular). Distribution is Debian Sarge (with a few unstable packages as well). System load is moderate. The affected files were typically corrupted during operation of Debian "apt-get":
either while updating the apt-cache (which became corrupted) or while extracting packages (which randomly corrupted newly extracted files).
That's about all I can say for sure. I have tried to reproduce the problem by stress-testing the filesystem (creating large numbers of small files, or small numbers of large files, containing ARC4 streams produced by <URL: ftp://quatramaran.ens.fr/pub/madore/misc/arc4gen.c > and checking them afterward against the same procedural generator) -- but to no avail: even by trying heavily concurrent access I have not been able to reproduce a single occurrence of the bug). Maybe apt-get has a specific way of writing files (but I can't really think how; might it use mmap() in some way? that doesn't sound plausible) which makes it trigger the bug.
I also have a UP box (Intel PIII600 with 384MB RAM and IDE disks) which, even though it has a nearly identical setup and is used in a roughly equal way, has not experienced any kind of corruption; so maybe the bug is SMP-specific (which would explain it going more or less unnoticed) -- on the other hand, a friend of mine has mentioned having observed similar problems on a UP box with 2.6.5 installed under very heavy load on ReiserFS, but I can't say more here. I'm really sorry that all this is very fuzzy.
Any suggestions (either on how to fix the problem or to work around it, or on how to reproduce it experimentally) are welcome. A friendly pat on the back would also be welcome. :-)
Signature · 3 lines
David A. Madore
([email protected],
http://www.eleves.ens.fr:8080/home/madore/ )Quoted 9 lines · David Madore
Hi folks.
I'm afraid this bug-report will be rather worthless as it is, because the bug has proven remarkable elusive and has defeated all my attempts to track it down to a precise test case or set of circumstances. But since it seems important, I thought it might be worth a post anyway. Any help is appreciated in clarifying the circumstances which trigger the problem, or generally in making this report more useful.
Quoted 4 lines
The bottom line: I've experienced file corruption, of the following
nature: consecutive regions (all, it seems, aligned on 256-byte boundaries, and typically around 1kb or 2kb in length) of seemingly random files are replaced by null bytes.
The good news is that we tracked this one down recently. 2.6.7-rc1 shouldn't do this anymore.
-chris
Hi,
Uh, seems I've experienced this problem too. But since there were disk-full condition and at the same time electricity went away ... I thought I had really bad luck (needed to apt-get --reinstall whole debian).
Glad to hear it's found and fixed now.
Lenar
Quoted 13 lines · Chris Mason
The good news is that we tracked this one down recently. 2.6.7-rc1 shouldn't do this anymore.
(quoted diff) +0
-2-chris -To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 7 lines · On May-28 2004, Fri, 08:46 -0400 Chris Mason <[email protected]> wrote:
Quoted 4 lines
The bottom line: I've experienced file corruption, of the following
nature: consecutive regions (all, it seems, aligned on 256-byte boundaries, and typically around 1kb or 2kb in length) of seemingly random files are replaced by null bytes.The good news is that we tracked this one down recently. 2.6.7-rc1 shouldn't do this anymore.
So did this only affect SMP machines?
Signature · 1 line
Tomas Szepe <[email protected]>
Quoted 8 lines · Tomas Szepe
Quoted 5 lines · On May-28 2004, Fri, 08:46 -0400 Chris Mason <[email protected]> wrote:
Quoted 4 lines
The bottom line: I've experienced file corruption, of the following
nature: consecutive regions (all, it seems, aligned on 256-byte boundaries, and typically around 1kb or 2kb in length) of seemingly random files are replaced by null bytes.
Quoted 4 lines
Quoted 2 lines
The good news is that we tracked this one down recently. 2.6.7-rc1 shouldn't do this anymore.
So did this only affect SMP machines?
No, if you slept in the right spot you could hit it on UP.
-chris
Quoted 9 lines · Chris Mason
Quoted 8 lines · Tomas Szepe
On May-28 2004, Fri, 08:46 -0400
Quoted 5 lines · Chris Mason
Quoted 4 lines
The bottom line: I've experienced file corruption, of the following nature: consecutive regions (all, it seems, aligned on 256-byte boundaries, and typically around 1kb or 2kb in length) of seemingly random files are replaced by null bytes.
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
The good news is that we tracked this one down recently. 2.6.7-rc1 shouldn't do this anymore.
So did this only affect SMP machines?
No, if you slept in the right spot you could hit it on UP.
I saw this once when using 2.6.6, it was messing up the filesystem structures as well (ext2 & ext3), replacing mostly with nulls, some with random letters and numbers, for 4-6 character lengths, and not on any nice boundaries. Since I stopped trying to use the ATI framebuffer driver (this is on a 21164A alpha, 164LX motherboard, ATI Mach64 CT video), it seems to have stopped. Also, I noticed that the framebuffer driver didn't work so well.
2.6.7-rc1 seems to have a compile error for alpha as well, I'll post that in a separate email.
Pat
Signature · 3 lines
PLUG Vice President -- http://plug.purdue.org The Computer Refuge -- http://computer-refuge.org Purdue University Research Computing -- http://www.itap.purdue.edu/rcs
Quoted 17 lines · Pat
Quoted 9 lines · Chris Mason
Quoted 8 lines · Tomas Szepe
On May-28 2004, Fri, 08:46 -0400
Quoted 5 lines · Chris Mason
Quoted 4 lines
The bottom line: I've experienced file corruption, of the following nature: consecutive regions (all, it seems, aligned on 256-byte boundaries, and typically around 1kb or 2kb in length) of seemingly random files are replaced by null bytes.
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
The good news is that we tracked this one down recently. 2.6.7-rc1 shouldn't do this anymore.
So did this only affect SMP machines?
No, if you slept in the right spot you could hit it on UP.
Quoted 8 lines
I saw this once when using 2.6.6, it was messing up the filesystem structures as well (ext2 & ext3), replacing mostly with nulls, some with random letters and numbers, for 4-6 character lengths, and not on any nice boundaries. Since I stopped trying to use the ATI framebuffer driver (this is on a 21164A alpha, 164LX motherboard, ATI Mach64 CT video), it seems to have stopped. Also, I noticed that the framebuffer driver didn't work so well.
This is different. The reiserfs bug can only trigger data corruptions
in reiserfs, and won't trigger metadata problems.
-chris
Quoted 9 lines · On May-28 2004, Fri, 12:29 -0400 Chris Mason <[email protected]> wrote:
Quoted 8 lines · Tomas Szepe
Quoted 5 lines · On May-28 2004, Fri, 08:46 -0400 Chris Mason <[email protected]> wrote:
Quoted 4 lines
The bottom line: I've experienced file corruption, of the following
nature: consecutive regions (all, it seems, aligned on 256-byte boundaries, and typically around 1kb or 2kb in length) of seemingly random files are replaced by null bytes.
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
The good news is that we tracked this one down recently. 2.6.7-rc1 shouldn't do this anymore.
So did this only affect SMP machines?
No, if you slept in the right spot you could hit it on UP.
Uh oh. Any idea about when the bug was introduced?
Signature · 1 line
Tomas Szepe <[email protected]>
On Fri, 2004-05-28 at 12:45, Tomas Szepe wrote:
[ reiserfs data corruption bug ]
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
So did this only affect SMP machines?
No, if you slept in the right spot you could hit it on UP.
Uh oh. Any idea about when the bug was introduced?
2.6.6, with the reiserfs data=ordered patches.
-chris
Quoted 12 lines · Tomas Szepe
Quoted 9 lines · On May-28 2004, Fri, 12:29 -0400 Chris Mason <[email protected]> wrote:
Quoted 8 lines · Tomas Szepe
Quoted 5 lines · On May-28 2004, Fri, 08:46 -0400 Chris Mason <[email protected]> wrote:
Quoted 4 lines
The bottom line: I've experienced file corruption, of the following
nature: consecutive regions (all, it seems, aligned on 256-byte boundaries, and typically around 1kb or 2kb in length) of seemingly random files are replaced by null bytes.
Quoted 9 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
The good news is that we tracked this one down recently. 2.6.7-rc1 shouldn't do this anymore.
So did this only affect SMP machines?
No, if you slept in the right spot you could hit it on UP.
Uh oh. Any idea about when the bug was introduced?
As far as I know, I was the first to publicly complain about the bug, first to Bitmover, (I was hitting it when using bk) who then figured out that it was a kernel bug, hence the long "1352 NUL bytes at end of page" thread.
I first noticed the bug around April 15, doing almost nightly kernel updates and builds. The bug was rather hard to hit reliably though, so it could have been there for some time earlier.
Steven
Quoted 11 lines · Tomas Szepe
Quoted 6 lines · On May-28 2004, Fri, 08:46 -0400 Chris Mason <[email protected]> wrote:
Quoted 6 lines
The bottom line: I've experienced file corruption, of the following
nature: consecutive regions (all, it seems, aligned on 256-byte boundaries, and typically around 1kb or 2kb in length) of seemingly random files are replaced by null bytes.
Quoted 9 lines
Quoted 4 lines
The good news is that we tracked this one down recently. 2.6.7-rc1 shouldn't do this anymore.
So did this only affect SMP machines?
No, it's UP here. And I think it happened first with 2.6.6-rc2-mm2.
Lenar
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
I would like to see ReiserFS V3 enter a feature freeze real soon.
While I like and appreciate the data journaling stuff, and I think it should go in, real soon now I think we should avoid adding new features to V3. Let the mission critical server folks have a reiserfs version that only gets bug fixes added to it, and let V4 be for those who want excitement.
Are there any things which Chris and Jeff think should go in besides data journaling/ordering and bitmap algorithm changes?
Also, I would like to see some serious benchmarks of the bitmap algorithm changes before they go in. They seem nice in theory, and some users liked them for their uses, but that does not make a serious scientific study. Such a study has a high chance of making them even better.;-)
zam, I view you as the block allocator maintainer, please review that bitmap code from Chris.
Chris and Jeff, can you propose a benchmarking plan for the bitmap code?
Hans
Quoted 29 lines · Chris Mason
Quoted 8 lines · Vladimir Saveliev
Quoted 5 lines
The good news is that we tracked this one down recently. 2.6.7-rc1 shouldn't do this anymore.
Just out of curiosity, what was it?
It was a bug in
reiserfs_file_writecaused by the data=ordered changes. I was droppingi_sembefore changingi_size, and the result was that writepage was zeroing out bytes it thought was past the end of the file.See the recent thread with "1352 NUL bytes at the end of a page?" in the subject. The funny part was that we started getting bug reports for this on the suse kernel just a few days before Steven Cole reported it, so I was able to overlap some of the bug hunting/testing.
-chris
Re: I would like to see ReiserFS V3 enter a feature freeze real soon.
Quoted 9 lines · Hans Reiser
While I like and appreciate the data journaling stuff, and I think it should go in, real soon now I think we should avoid adding new features to V3. Let the mission critical server folks have a reiserfs version that only gets bug fixes added to it, and let V4 be for those who want excitement.
Are there any things which Chris and Jeff think should go in besides data journaling/ordering and bitmap algorithm changes?
We've got io error fault tolerance that needs to go in after the barrier code has stabilized. I can't promise that I'll never making another change in there, but my goal is to keep them to a minimum.
Quoted 6 lines
Also, I would like to see some serious benchmarks of the bitmap algorithm changes before they go in. They seem nice in theory, and some users liked them for their uses, but that does not make a serious scientific study. Such a study has a high chance of making them even better.;-)
Some benchmarks have been posted on reiserfs-list, but I'd love to coordinate with you on getting some mongo numbers. I can spout off a long list of places where the code does better then the original v3 allocator, but that's because those were the ones I was trying to fix ;-)
Quoted 4 lines
zam, I view you as the block allocator maintainer, please review that bitmap code from Chris.
Chris and Jeff, can you propose a benchmarking plan for the bitmap code?
A good start would be to just rebenchmark against v4.
-chris
Re: I would like to see ReiserFS V3 enter a feature freeze real soon.
Quoted 5 lines · Chris Mason
Quoted 2 lines · Hans Reiser
While I like and appreciate the data journaling stuff, and I think it should go in, real soon now I think we should avoid adding new features to V3. Let the mission critical server folks have a reiserfs version that only gets bug fixes added to it, and let V4 be for those who want excitement.
Quoted 8 lines
Quoted 4 lines
Are there any things which Chris and Jeff think should go in besides data journaling/ordering and bitmap algorithm changes?
We've got io error fault tolerance that needs to go in after the barrier code has stabilized.
Ok, this qualifies as bug fixing or close enough.;-)
Quoted 5 lines
I can't promise that I'll never making another change in there, but my goal is to keep them to a minimum.
Quoted 8 lines
Quoted 4 lines
Also, I would like to see some serious benchmarks of the bitmap algorithm changes before they go in. They seem nice in theory, and some users liked them for their uses, but that does not make a serious scientific study. Such a study has a high chance of making them even better.;-)
Some benchmarks have been posted on reiserfs-list, but I'd love to coordinate with you on getting some mongo numbers.
Ok.
Quoted 7 lines
I can spout off a
long list of places where the code does better then the original v3 allocator, but that's because those were the ones I was trying to fix ;-)
Quoted 9 lines
Quoted 5 lines
zam, I view you as the block allocator maintainer, please review that bitmap code from Chris.
Chris and Jeff, can you propose a benchmarking plan for the bitmap code?
A good start would be to just rebenchmark against v4.
V4 performance is not at a stable point at the moment I think, I have not been monitoring things closely due to trying to earn bucks consulting, and performance did not get tested every week, but there have been reports of performance decreasing and no reports of anyone investigating it, so I need to....
Elena, please compose a URL consisting of past benchmarks of various V4 snapshots and send it to me. (I did not read the last one you sent, sorry about that, so include the contents of that one also).
If the objective is to determine if the algorithm is good, then we should test it with only the algorithm in question changed.
I would be quite happy to add the algorithm to V4 (or Chris and Jeff can do that) and test it on vs. off.
Zam, place that on your todo list and send the todo list to me....
Quoted 7 lines
-chris
Re: I would like to see ReiserFS V3 enter a feature freeze real soon.
Quoted 19 lines · Hans Reiser
Quoted 14 lines
I can't promise that I'll never making another change in there, but my goal is to keep them to a minimum.
Quoted 8 lines
Also, I would like to see some serious benchmarks of the bitmap algorithm changes before they go in. They seem nice in theory, and some users liked them for their uses, but that does not make a serious scientific study. Such a study has a high chance of making them even better.;-)
Quoted 3 lines
Some benchmarks have been posted on reiserfs-list, but I'd love to coordinate with you on getting some mongo numbers.
Ok.
Quoted 9 lines
Quoted 3 lines
A good start would be to just rebenchmark against v4.
V4 performance is not at a stable point at the moment I think, I have not been monitoring things closely due to trying to earn bucks consulting, and performance did not get tested every week, but there have been reports of performance decreasing and no reports of anyone investigating it, so I need to....
Sure, since v4 is being done again -mm right now (right?) you can just benchmark against a few of the new options. mount -o alloc=skip_busy will give you the old allocator.
Quoted 9 lines
Elena, please compose a URL consisting of past benchmarks of various V4 snapshots and send it to me. (I did not read the last one you sent, sorry about that, so include the contents of that one also).
If the objective is to determine if the algorithm is good, then we should test it with only the algorithm in question changed.
I would be quite happy to add the algorithm to V4 (or Chris and Jeff can do that) and test it on vs. off.
The algorithm has a few key components, but v4 doesn't need most of it. The part to inherit packing localities down the chain would be most interesting in v4.
The rest approximates things v4 should already be good at, like grouping some of the metadata near the data blocks.
-chris