File system testing
Hello all,
We're running into some possible data corruption issues with some log files stored on a JFFS2 partition.
Our current configuration is admittedly a bit unusual. We've got a 2.6.11 kernel on a PowerPC processor. However, we ported the JFFS2 and MTD code from a 2.6.21 kernel. As you might guess, there were some other issues that prevented us from just moving to a recent kernel, though we're still considering that.
To start with, however, I've got some general questions on testing the system.
We've got a script that will write a series of files to the filesystem, and then read them back, checking the contents.
The script will then unmount the filesystem, and re-mount it. Sometimes, when reading the files again, we'll see a problem. For instance, seeing nulls (0x00) instead of the expected characters.
We'll also see at least one message indicating that a node has invalid size (would go over the end of the erase block) and after that some "magic bitmask 0x1985 not found" messages.
We then modified the script to sleep after writing each file, that seemed to help, and the test ran longer without seeing a data corruption problem. Though we did eventually see one after a long time.
So, questions:
Is this a good way to test the filesystem in general?
Is it possible to push the filesystem too hard, by reading a writing a lot of data all at once? Will the reads block the writes from being completed? Or block the erasing dirty eraseblocks?
Thanks,
James Graves
Quoted 10 lines · James Graves
We've got a script that will write a series of files to the filesystem, and then read them back, checking the contents.
The script will then unmount the filesystem, and re-mount it. Sometimes, when reading the files again, we'll see a problem. For instance, seeing nulls (
0x00) instead of the expected characters.We'll also see at least one message indicating that a node has invalid size (would go over the end of the erase block) and after that some "magic bitmask
0x1985not found" messages.
Hm. That's very bad. Can you reproduce with CONFIG_JFFS2_FS_DEBUG=1 and a serial console?
Quoted 7 lines
We then modified the script to sleep after writing each file, that seemed to help, and the test ran longer without seeing a data corruption problem. Though we did eventually see one after a long time.
So, questions:
Is this a good way to test the filesystem in general?
Yes. We have a test in mtd-utils.git which does something very similar.
Quoted 3 lines
Is it possible to push the filesystem too hard, by reading a writing a lot of data all at once? Will the reads block the writes from being completed? Or block the erasing dirty eraseblocks?
No, none of that should happen.
Signature · 1 line
dwmw2
Thanks for the reply David.
Turns out to apparently be a hardware problem.
Data written to the last 512 of flash memory (of a 32Mbyte flash part) sometimes doesn't work. Sometimes, even though the programming apparently worked, you'll just get back 0xff anyway. The rest of the flash memory works fine, and we were able to run our filesystem exerciser for over 12 hours solid, with no errors, once we excluded that part of the flash memory.
So, obviously, any block at the end of memory was getting corrupted.
I'm expecting there's some kind of memory mapping conflict, but that'll take further investigation.
----------------------------------------------------------------
What's funny is that we'd previously done some informal testing of the raw partition. Reading and writing images to the /dev/mtdblock3 device directly.
And it seemed to be working fine. But, as it turns out, we were only testing approximately 32Mbytes... and not _exactly_ 32Mbytes.
Let that be a lesson to you all!
Thanks again,
James Graves
Quoted 3 lines · James Graves
Thanks for the reply David.
Turns out to apparently be a hardware problem.
Good. I was hoping it might be :)
Signature · 1 line
dwmw2