2.5.12 severe ext3 filesystem corruption warning!

I gave the 2.5.12 kernel a shot on my workstation tonight and found an *extremely* serious ext3 filesystem corrupting behavior.

The only files that were mangled were files that had been modified while running, so it looks like this is an issue with finding the contents of modified files to write, not with random data dropping onto the disk.

That said, however, the contents were rather random -- blocks from .overview were placed into a number of other files, chunks of .newsrc.eld written to the gkrellm configuration file and the like.

Unmodified files, however, don't seem to have been touched. This can't be perfect, of course, as something I never look at may have been destroyed, but it seems to be reliably.

The system is a mobile P4, 512MB RAM, IDE disk. Only one filesystem seems badly effected, my home directory, which is ext3 and fully data-journaled.

I couldn't find corruption on the root filesystem[1] but there isn't much of that which is actively written at runtime. Nothing notable in /var seems broken, thankfully, so no panic. :)

Anyway, I don't know if anyone else is seeing the problems but this is a first worrying datapoint with the kernel...

Daniel

Footnotes:
[1] Contains everything else, including /usr and /var

20+ years as a vegetarian and the guy who steals my credit card
orders $6,000 worth of chicken parts: proof that the most powerful
force in the universe is Irony.
        -- David Weinberger, _JOHO_ (2000-03-20)

A few things..

Are your other filesystems using journalled data as well?

Are you sure that all kernel files were recompiled? If, for example, you had some 2.5.11 objects in the link, that would be bad.

Do you know whether the bad data is actually on-disk, or could it be just in-RAM? ie: was the data still bad after a reboot?

What blocksize is that filesystem using? The output of `dumpe2fs -h /dev/whatever' will tell you this.

Can you please force an fsck against that filesystem, see what it says?

Thanks.

-

[snip]

Andrew, judging by the filenames he'd mentioned, I suspect that he runs innd. I.e. one of the very few programs heavily using truncate(). And no, it doesn't promise anything good - last time we had crap in truncate/mmap interaction it was a hell to fix.

I suspect that you had screwed the truncate exclusion warranties up. If _any_ IO happens in the area currently manipulated by ->truncate() - you are screwed and results would look pretty much like the things mentioned in bug report.

OK, thanks. That's something to go on.

The only change which comes to mind is discard_buffer() - it's no longer clearing BH_Uptodate. Because for the partial page at the end of the mapping, buffers outside i_size were zeroed in truncate_partial_page(). But with (presumed) 4k blocksize, it can't be that.

ext3_writepage() can hold a reference against the buffers after the page has come unlocked, so a try_to_free in the right window will fail, which will leave the page floating about on the page LRU, not mapped into any address_space, clean, not uptodate, but with uptodate buffers. Nobody but the VM should ever find that page, but it does make more sense to mark that buffer not uptodate. This would only happen on super-heavy SMP loads.

So hmm.

There's a small SMP-only race in truncate_list_pages:
the test for PageWriteback happens against an unlocked page. There's a three-instruction window where writeback could start up. That won't be it, but I'll fix that.

-

[...]

Oh, it's a laptop, so SMP is right out of the picture. ;)
Daniel

Many of my favorite shamans are rock stars. They probably don't even know
they're shamans but they know how to get to ecstasy and back, and how to take
others with them. They may not have a license, but they know how how to drive.
        -- Gabrielle Roth, _Maps to Ecstasy_

Yes:

] mount
/dev/discs/disc0/part6 on / type ext3 (rw,data=journal)
none on /proc type proc (rw)
none on /proc/bus/usb type usbdevfs (rw)
/dev/discs/disc0/part7 on /home/daniel type ext3 (rw,data=journal)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
anu.rimspace.net:/music on /music type nfs (rw,noexec,nosuid,nodev,rsize=8192,wsize=8192,nfsvers=3,bg,soft,addr=210.23.138.21)

Yes. The kernel was build with clean, oldconfig, dep and then building the image and modules.

I only found it after the reboot; being the cautious sort I rebooted as soon as it showed odd data issues, back to the known stable 2.5.6 and did a full fsck run. That showed no issues, which was nice, but I found the data corruption.

2k blocks, as I found that 4k was giving me over 40% wasted space last time I bothered to check. I have most of the files in this filesystem being ~2K in size.

Full output of the command is attached below.

I did. The filesystem itself was clean -- not even a warning. Just data corruption, not metadata.

Not quite. I found the corruption in three places:

* my XEmacs and Gnus mail spool.
* gkrellm configuration files.
* galeon/mozilla bookmarks and preferences.

XEmacs uses the mode (O_WRONLY | O_TRUNC) when it writes out files, so everything that was written there would have been truncated before output. There shouldn't have been any mmap() of the files, though.

It's also worth noting that XEmacs would fsync() the file handle after writing the content, for what that's worth.

I found corruption in the galeon bookmarks file, which seemed to start writing XML half way through the actual data and to add a block of around 2K NULL bytes half way through the content.

I also found corruption in the mozilla prefs.js file from the same application[1]. That was simply a truncated file -- no NULL bytes or anything, just a file that cut off half way through a single expression like:

user_pref("wallet.capture

This /may/ be the logical break between two write(2) calls or a partly completed write, though.

Ick. Er, good luck. I am quite happy to provide more information and even to install a scratch disk and try to get it to fail the same way if you wish.

I can't use the same kernel build, though, as I removed the binary kernel. I can build an equivalent one, of course, but it's not quite the same. Sorry about that -- I did it before thinking about it. :/

Let me know if there is any more information that would be useful, other than the corrupted files. I killed those, also, without thinking. :(

Daniel

Footnotes:
[1] Galeon is a GNOME front-end for Mozilla, in case you didn't know.

The English have the most rigid code of immorality in the world.
        -- Malcom Bradbury, _Eating People is Wrong_

Daniel, this is good stuff. Thanks.

So all of those files were written to by their application during the failed session.

Question on the table is: did the kernel write the wrong stuff into those files, or did it forget to write the right stuff?

Is the incorrect content within those files recognisable at all? If so, what is it?

That sounds like metadata corruption. Are you sure that the file doesn't have a chunk of invisible nulls in it, after the text? Because if it got chopped off in this manner, e2fsck should have detected something.

It would be good if you had the time to do that.

-

Yes. Files that they accessed in a read-only fashion, and untouched files, seem entirely unharmed.

Wrong stuff.

Yes. There are three things that I noted as incorrect in the files.

Many of them, but not all, featured large blocks of NULL (0) bytes, on the order of 2K worth of them. No exacting counts available but they did fill a page in an 80x30 terminal, generally.

Several of the files featured the "right" content, just in the wrong location. For example, the galeon 'bookmarks.xbel' file started about a third of the way into it's correct content.

Only one of the files seemed to be gratuitously truncated, the mozilla prefs.js file.

The remaining file content seemed to be randomly swizzled. I found part of a .overview file in .newsrc.eld, for example, and part of the .newsrc.eld elsewhere. Also, 'active' got written to a mail folder...

There were *no* cases where the file content was unrecognizable with the exception of the NULL bytes. In *every* case the content was from a *completely* different file.

This included files in different directories getting content swapped, so a file in ~ got content from ~/Mail/<group>/, so it's not just within the same directory.

I can't think of a single verifiable case where a file got the wrong data from another process, though. That's not a "didn't", just a "can't remember seeing", though, so don't take that as gospel.

[...]

Yes. I used an editor that displays the NULL byte correctly in files to view it when looking for damage and there was *no* NULL content in this file.

None of the NULL bytes were at the end of the file, always in the middle of the data.

Yup. I carefully watched the forced fsck and no errors. This is:
] fsck.ext3 -V
e2fsck 1.27 (8-Mar-2002)

	Using EXT2FS Library version 1.27, 8-Mar-2002

Sure. I will have a run at reproducing it in an hour or two, then post the results to the list under this thread, with a direct CC. :)

Daniel

Nobody objects to a woman being a good writer or sculptor or geneticist if at
the same time she manages to be a good wife, good mother, good-looking,
well-groomed and unaggressive.
        -- Leslie M. McIntyre