JFFS2: file contents in case of data CRC error
I have a JFFS2 partition on 32MByte NAND device.
When reading a specific file - see below for details - JFFS2 reports a Data CRC error but function nand_correct_data never returns -1, so I assume ECC algorithm is able to correct errors. However the file is not equal to the original one that has been written to NAND. In fact, in the middle of the file, I see a 4-kByte "hole" where all bytes are 0. Anybody can help me about understanding if this is the expected behaviour of JFFS2?
Thanks in advance.
bash-2.05# mount -t jffs2 -o ro /dev/mtdblock0 /mnt/nand0mtdblock_open
ok
bash-2.05# cp -v /mnt/nand0/f.img /tmp/jffs2_get_inode_nodes(): Data CRC failed on node at 0x01d55544: Read 0x80b8997f, calculated 0x57f0ea8e /mnt/nand0/f.img' -> /tmp/f.img'
Signature · 5 lines
llandre DAVE Electronics System House - R&D Department web: http://www.dave.eu email: r&[email protected]
Hi
Could you please give information about how you can get this error? I need to know use case.
Quoted 8 lines
I have a JFFS2 partition on 32MByte NAND device.
When reading a specific file - see below for details - JFFS2 reports a Data CRC error but functionnand_correct_datanever returns -1, so I assume ECC algorithm is able to correct errors.
However the file is not equal to the original one that has been writtento NAND. In fact, in the middle of the file, I see a 4-kByte "hole" where all bytes are 0.
It is Ok. NAND error correction could return that data is correct even if it is not correct. It finds the issues with guarantee only in case of 1 or 2 bits errors.
Quoted 5 lines
Anybody can help me about understanding if this is the expected behaviour of JFFS2?
Thanks in advance.
Quoted 7 lines
bash-2.05# mount -t jffs2 -o ro
/dev/mtdblock0/mnt/nand0mtdblock_open
ok
bash-2.05# cp -v/mnt/nand0/f.img/tmp/jffs2_get_inode_nodes(): Data CRC failed on node at0x01d55544: Read0x80b8997f, calculated0x57f0ea8e/mnt/nand0/f.img' ->/tmp/f.img'
This is a bad error! How obtained you this? What you did before getting this?
Alexey
--------------------------------------------------------------------
Closed Joint Stock Company Intel A/O
Registered legal address: 125252, Moscow, Russian Federation, Chapayevsky Per, 14.
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
Alexey, Matt and Jamie,
thank you all for the precious information.
Quoted 2 lines
Could you please give information about how you can get this error? I need to know use case.
The issue has been reported by a customer working on PPC405EP-based system running linux 2.4.20.
The device under test has run a temperature cycle test. The temperature has been varied in the whole industrial range (-40/+85 deg C). After completing the test, the files stored on NAND have been read back and compared to the original ones - written on NAND before starting the temperature test - and the difference in the file f.img previously described arose.
Quoted 5 lines
Quoted 1 line
to NAND. In fact, in the middle of the file, I see a 4-kByte "hole" where all bytes are 0.
It is Ok. NAND error correction could return that data is correct even if it is not correct. It finds the issues with guarantee only in case of 1 or 2 bits errors.
About this point I agree with catboat (http://article.gmane.org/gmane.linux.drivers.mtd/19463). In my opinion in this case JFFS2 should report an error code != 0 in order to warn the userland that something did not work perfectly.
Quoted 9 lines
Quoted 6 lines
bash-2.05# mount -t jffs2 -o ro
/dev/mtdblock0/mnt/nand0mtdblock_open
ok
bash-2.05# cp -v/mnt/nand0/f.img/tmp/jffs2_get_inode_nodes(): Data CRC failed on node at0x01d55544: Read0x80b8997f, calculated0x57f0ea8e/mnt/nand0/f.img' ->/tmp/f.img'This is a bad error! How obtained you this? What you did before getting this?
See previous description.
Signature · 5 lines
llandre DAVE Electronics System House - R&D Department web: http://www.dave.eu email: r&[email protected]
Hi
Quoted 5 lines
Quoted 2 lines
Could you please give information about how you can get this error? I need to know use case.
The issue has been reported by a customer working on PPC405EP-based system running linux 2.4.20.
The device under test has run a temperature cycle test. The
temperature
Quoted 5 lines
has been varied in the whole industrial range (-40/+85 deg C). After completing the test, the files stored on NAND have been read back and compared to the original ones - written on NAND before starting the temperature test - and the difference in the file f.img previously described arose.
Ahh. The temperature tests. That explains a lot.
Probably device worked improperly in the test but test did not turn off device when NAND become unworkable. As result JFFS2 did not detect the problem of broken CRC on mount but did it on attempt to read the broken data and printed error message.
So JFFS2 performed scan on read - found CRC error in node -> JFFS2 marked node to be obsolete
-> As result hole in the file has been formed. -> You read 0x0 in the middle of the file.
I think nobody considered such tests before. I am not sure if any file system may be reliable in such a case with temperature tests.
Unfortunately I have no idea how this issue could be solved :).
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
to NAND. In fact, in the middle of the file, I see a 4-kByte "hole" where all bytes are 0.
It is Ok. NAND error correction could return that data is correct
even
Quoted 1 line
Quoted 1 line
if it is not correct. It finds the issues with guarantee only in case
of
Quoted 3 lines
Quoted 1 line
1 or 2 bits errors.
About this point I agree with catboat
(http://article.gmane.org/gmane.linux.drivers.mtd/19463). In my
opinion
Quoted 1 line
in this case JFFS2 should report an error code != 0 in order to warn
the
Quoted 2 lines
userland that something did not work perfectly.
--------------------------------------------------------------------
Closed Joint Stock Company Intel A/O
Registered legal address: 125252, Moscow, Russian Federation, Chapayevsky Per, 14.
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
Quoted 9 lines · Alexey
So JFFS2 performed scan on read - found CRC error in node -> JFFS2 marked node to be obsolete
-> As result hole in the file has been formed. -> You read0x0in the middle of the file.I think nobody considered such tests before. I am not sure if any file system may be reliable in such a case with temperature tests.
Unfortunately I have no idea how this issue could be solved :).
If the corruption makes it impossible to detect the node corresponding to a block, then indeed how can it be solved? If the bit flips happen in the node header, not the data, there really is no way to know that some data is lost from the right block.
I'm thinking the only way to detect this with high reliability is to store summaries of the existence of blocks in another part of storage, with checksums and serial numbers - like some of the latest disk filesystems begin to. Then missing nodes are detectable and can translate to I/O errors in userspace.
The other way, is for apps themselves to store checksums of their files. I had to do this, because we were getting occasional zeroed blocks from JFFS2, and that happened in the middle of executables, so we had apps which would run and occasionally crash or go wrong because part of their code contained zeroes.
To avoid random wrong behaviour, I checksummed executable programs before running them. It does seem it would be better if the filesystem offered better integrity or error guarantees.
-- Jamie
Quoted 4 lines
I'm thinking the only way to detect this with high reliability is to store summaries of the existence of blocks in another part of storage, with checksums and serial numbers - like some of the latest disk filesystems begin to.
Can you name these file systems?
Signature · 5 lines
llandre DAVE Electronics System House - R&D Department web: http://www.dave.eu email: r&[email protected]
Quoted 5 lines · llandre
Quoted 4 lines
I'm thinking the only way to detect this with high reliability is to store summaries of the existence of blocks in another part of storage, with checksums and serial numbers - like some of the latest disk filesystems begin to.
Can you name these file systems?
ZFS, HammerFS and Btrfs, I think.
-- Jamie
Quoted 4 lines
Ahh. The temperature tests. That explains a lot. Probably device worked improperly in the test but test did not turn off device when NAND become unworkable. As result JFFS2 did not detect the
problem of broken CRC on mount but did it on attempt to read the broken
data and printed error message.
I was told by the customer that before starting the temperature test several files have been copied on NAND. During the temperature test the processor never accessed these files on NAND. Once the test has been completed, the files on NAND have been read back and compared to the original ones.
Quoted 2 lines
So JFFS2 performed scan on read - found CRC error in node -> JFFS2 marked node to be obsolete
Sure. This is the code snippet in function jffs2_get_inode_nodes:
if (crc != je32_to_cpu(node.i.data_crc)) {
printk(KERN_NOTICE "jffs2_get_inode_nodes(): Data CRC failed on node
at 0x%08x: Read 0x%08x, calculated 0x%08x\n",
ref_offset(ref), je32_to_cpu(node.i.data_crc), crc); jffs2_mark_node_obsolete(c, ref); spin_lock(&c->erase_completion_lock); continue; ...
Quoted 2 lines
-> As result hole in the file has been formed.
-> You read0x0in the middle of the file.
I can't find the function that, in case of obsolete node, zeroes the chunk of data passed to user space. Can you?
Quoted 2 lines
I think nobody considered such tests before. I am not sure if any file system may be reliable in such a case with temperature tests.
Hmm ... I don't think the environmental condition may influence JFFS2 behaviour. The NAND used in the test is industrial grade so it is guaranteed by manufacturer that it works in -40/+85 degC range. JFFS2 clearly does not anything about environment temperature. On the other hand I'm wondering if temperature variation caused multiple bit flipping in one NAND block that MTD was not able to correct and this led to the problem we are talking about.
Signature · 5 lines
llandre DAVE Electronics System House - R&D Department web: http://www.dave.eu email: r&[email protected]
Hi,
Quoted 4 lines
Quoted 2 lines
-> As result hole in the file has been formed.
-> You read0x0in the middle of the file.I can't find the function that, in case of obsolete node, zeroes the chunk of data passed to user space. Can you?
See jffs2_read_inode_range() function.
Quoted 1 line
Quoted 1 line
I think nobody considered such tests before. I am not sure if any
file
Quoted 6 lines
Quoted 1 line
system may be reliable in such a case with temperature tests.
Hmm ... I don't think the environmental condition may influence JFFS2 behaviour. The NAND used in the test is industrial grade so it is guaranteed by manufacturer that it works in -40/+85 degC range. JFFS2 clearly does not anything about environment temperature. On the other hand I'm wondering if temperature variation caused multiple bit
flipping
Quoted 3 lines
in one NAND block that MTD was not able to correct and this led to the
problem we are talking about.
I guess the problem is related to multiple bit flipping caused by temperature.
Thanks,
Alexey
Jffss2_write_super erasing all the blocks
In reply to llandreHi,
I have a question about JFFS2 fs. In my setup, I am running JFFS2 on AMD Spansion flash (NOR flash). What I noticed was that whenever I erase a file from my flash, I see that jffs2_write_super function gets called which in turn calls jffs2_erase_pending_blocks(0) which goes ahead and deletes all the pending blocks. While this erase is going on super block is locked and we can not perform any other operation with JFFS2.
Can someone please explain the logic behind this or if there is something I am missing here?
Thanks,
Nikhil
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of llandre
Sent: Thursday, January 31, 2008 8:54 AM
To: [email protected]
Subject: JFFS2: file contents in case of data CRC error
I have a JFFS2 partition on 32MByte NAND device.
When reading a specific file - see below for details - JFFS2 reports a Data CRC error but function nand_correct_data never returns -1, so I assume ECC algorithm is able to correct errors.
However the file is not equal to the original one that has been written to NAND. In fact, in the middle of the file, I see a 4-kByte "hole" where all bytes are 0.
Anybody can help me about understanding if this is the expected behaviour of JFFS2?
Thanks in advance.
bash-2.05# mount -t jffs2 -o ro /dev/mtdblock0 /mnt/nand0 mtdblock_open ok bash-2.05# cp -v /mnt/nand0/f.img /tmp/ jffs2_get_inode_nodes(): Data CRC failed on node at 0x01d55544: Read 0x80b8997f, calculated 0x57f0ea8e /mnt/nand0/f.img' -> /tmp/f.img'
--
llandre
DAVE Electronics System House - R&D Department
web: http://www.dave.eu email: r&[email protected]
______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/
SV: Jffss2_write_super erasing all the blocks
This is a long standing issue that I tried to address fairly recently. Please search the archive and you should find some posts/patch from me about this.
Jocke
Quoted 61 lines
-----Ursprungligt meddelande-----
Från: [email protected] [mailto:[email protected]] För Nikhil Bansal (nikbansa)
Skickat: den 31 januari 2008 20:22
Till: [email protected]
Ämne:Jffss2_write_supererasing all the blocksHi,
I have a question about JFFS2 fs. In my setup, I am running JFFS2 on AMD Spansion flash (NOR flash). What I noticed was that whenever I erase a file from my flash, I see that
jffs2_write_superfunction gets called which in turn callsjffs2_erase_pending_blocks(0)which goes ahead and deletes all the pending blocks. While this erase is going on super block is locked and we can not perform any other operation with JFFS2.Can someone please explain the logic behind this or if there is something I am missing here?
Thanks,
Nikhil-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of llandre
Sent: Thursday, January 31, 2008 8:54 AM
To: [email protected]
Subject: JFFS2: file contents in case of data CRC errorI have a JFFS2 partition on 32MByte NAND device.
When reading a specific file - see below for details - JFFS2 reports a Data CRC error but functionnand_correct_datanever returns -1, so I assume ECC algorithm is able to correct errors.
However the file is not equal to the original one that has been written to NAND. In fact, in the middle of the file, I see a 4-kByte "hole" where all bytes are 0.
Anybody can help me about understanding if this is the expected behaviour of JFFS2?
Thanks in advance.bash-2.05# mount -t jffs2 -o ro
/dev/mtdblock0/mnt/nand0mtdblock_openok bash-2.05# cp -v/mnt/nand0/f.img/tmp/jffs2_get_inode_nodes(): Data CRC failed on node at0x01d55544: Read0x80b8997f, calculated0x57f0ea8e/mnt/nand0/f.img' ->/tmp/f.img'--
llandreDAVE Electronics System House - R&D Department
web: http://www.dave.eu email: r&[email protected]______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/
Quoted 10 lines · llandre
I have a JFFS2 partition on 32MByte NAND device.
When reading a specific file - see below for details - JFFS2 reports a Data CRC error but functionnand_correct_datanever returns -1, so I assume ECC algorithm is able to correct errors.
However the file is not equal to the original one that has been written to NAND. In fact, in the middle of the file, I see a 4-kByte "hole" where all bytes are 0.
Anybody can help me about understanding if this is the expected behaviour of JFFS2?
Thanks in advance.
I saw exactly this problem on an s3c2440 board back in October. It turned out that the ECC correction code in the s3c2410 driver was incorrect in a couple of ways, resulting in corrupt data being passed up to JFFS2. I'm not sure, but I think what happened from there is that JFFS2 failed to decompress a block and instead of returning an error it left the page zeroed. You can use jffs2dump to correlate the offset reported in the CRC error with the problematic inode. Check the thread "bit flip" in October 2007.
What platform is this on?
Matt
Quoted 12 lines · Matt Reimer
Quoted 10 lines · llandre
I have a JFFS2 partition on 32MByte NAND device.
When reading a specific file - see below for details - JFFS2 reports a Data CRC error but functionnand_correct_datanever returns -1, so I assume ECC algorithm is able to correct errors.
However the file is not equal to the original one that has been written to NAND. In fact, in the middle of the file, I see a 4-kByte "hole" where all bytes are 0.
Anybody can help me about understanding if this is the expected behaviour of JFFS2?
Thanks in advance.
Quoted 8 lines
I saw exactly this problem on an s3c2440 board back in October. It turned out that the ECC correction code in the s3c2410 driver was incorrect in a couple of ways, resulting in corrupt data being passed up to JFFS2. I'm not sure, but I think what happened from there is that JFFS2 failed to decompress a block and instead of returning an error it left the page zeroed. You can use jffs2dump to correlate the offset reported in the CRC error with the problematic inode. Check the thread "bit flip" in October 2007.
I've seen similar things with cfi_cmdset_0002 on NOR flash, with Linux 2.4.26. Occasional bit flips on reading flash, resulting in zero blocks in the read file from JFFS2 - no errors reported to the application.
I have reason to think our board timings were ever so slightly out of spec, and haven't seen it in newer board revisions.
-- Jamie
Quoted 4 lines
I've seen similar things with
cfi_cmdset_0002on NOR flash, with Linux 2.4.26. Occasional bit flips on reading flash, resulting in zero blocks in the read file from JFFS2 - no errors reported to the application.
Have you found the JFFS2 portion of code that "zeroes" the block before passing it to the userland?
Regards,
llandre
DAVE Electronics System House - R&D Department
web: http://www.dave.eu email: r&[email protected]
Quoted 5 lines
up to JFFS2. I'm not sure, but I think what happened from there is that JFFS2 failed to decompress a block and instead of returning an error it left the page zeroed. You can use jffs2dump to correlate the offset reported in the CRC error with the problematic inode. Check the thread "bit flip" in October 2007.
Thanks a lot for this advice. The system is based on linux 2.4.20 and ELDK 2.0.2 so the jffs2dump is not available out-of-the-box. I'll try to build recent mtd-utils package.
Quoted 1 line
What platform is this on?
PowerPC 405EP.
Regards,
llandre
DAVE Electronics System House - R&D Department
web: http://www.dave.eu email: r&[email protected]
Quoted 6 lines
Quoted 5 lines
up to JFFS2. I'm not sure, but I think what happened from there is that JFFS2 failed to decompress a block and instead of returning an error it left the page zeroed. You can use jffs2dump to correlate the offset reported in the CRC error with the problematic inode. Check the thread "bit flip" in October 2007.
Thanks a lot for this advice. The system is based on linux 2.4.20 and ELDK 2.0.2 so the jffs2dump is not available out-of-the-box. I'll try to build recent mtd-utils package.
Matt,
I managed to built jffs2dump. Here a snippet of the dump it generated:
...
Inode node at 0x01d0fc84, totlen 0x00000378, #ino 22,
version 8, isize 26065, csize 820, dsize 1489, offset
24576
Wrong bitmask at 0x01d0fffc, 0x0000
Inode node at 0x01d10000, totlen 0x00000307, #ino 20,
version 356, isize 1318912, csize 707, dsize 3118, offset 1315794
Inode node at 0x01d10308, totlen 0x000003b0, #ino 20,
version 357, isize 1323008, csize 876, dsize 4096, offset 1318912
...
Inode node at 0x01d9bdd0, totlen 0x0000006a, #ino 10,
version 2, isize 38, csize 38, dsize 38, offset
0
Inode node at 0x01d9be3c, totlen 0x00000044, #ino 20,
version 1, isize 0, csize 0, dsize 0, offset
0
Dirent node at 0x01d9be80, totlen 0x0000003f, #pino 4,
version 4, #ino 20, nsize 23, name f.img
Inode node at 0x01d9bec0, totlen 0x0000013a, #ino 20,
version 2, isize 1049, csize 246, dsize 1049, offset
0
Wrong bitmask at 0x01d9bffc, 0x0000
Inode node at 0x01d9c000, totlen 0x00000550, #ino 18, version 230, isize 757760, csize 1292, dsize 1292, offset 756468
...
IIUC the inode number of file f.img is 20 so I have to look for the lines including "#ino 20" to understand where the file physically is stored on NAND. One of these line is for example:
Inode node at 0x01d10000, totlen 0x00000307, #ino 20, version
356, isize 1318912, csize 707, dsize 3118, offset 1315794
Then I looked at the binary dump of whole NAND device I created as follows:
nanddump /dev/mtd0 nand.bin 0 33554432
However around the offset 0x01d10000 in the file nand.bin all bytes are 0xFF so it seems there are no data in that area. What offset in the file correspond to the physical address 0x01d10000 ?
Signature · 5 lines
llandre DAVE Electronics System House - R&D Department web: http://www.dave.eu email: r&[email protected]