MTD:unstable bit issues?

My hardware: vcs452 nandflash
my kernel:2.6.16(I know it is too old:(,but this problem seems not relevant to the kernel version)

test steps:
1.copy a 100M file to the nand flash partition from remote client 2.use fsync to flush the mem to flash
3.use md5 to check whether the original file is different from the copied one
4.repeat power-cuts test for 5 times, when booting up, it will check the md5 again.
5.do step1 again

after a whole night test,(power-cut for 100 times), the step 4 failed. 1)there are some bytes changed:
-----------------------------------------------------
-original-file D0 40 E8 3E 00 00 BA 1A D0 40 0A 00 43 00 00 00 +file-in-flash D0 40 0F 41 00 00 BA 1A D0 40 0A 00 43 00 00 00
-----------------------------------------------------
2)a 4k bytes hole with all 0 in the flash file.
-----------------------------------------------------
-original-file DD AF C8 EE 57 EB 84 CE A1 7F B1 38 E7 22 51 2F
- 94 AE 77 5B 48 EA 61 8C 09 BB C5 74 6E B1 87 D2
- A9 32 8D 2A E7 7F D2 C3 DB 1A 92 E7 66 7C B8 4E
......

+file-in-flash 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
+ 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
+ 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
......
-----------------------------------------------------

I search google and find it might be relevant to "unstable bits issue"(bit-flip) in nand.
I wonder is the bit-flip the possible reason to cause such a 4k all-zero hole?
Thanks for any suggestion.

B.R.
Jia

Hello Jia:

Hi Thomas
Yes, we use jffs2.
How did u solve your bug? :)

Hello Jia:

Unfortunately, we didn't solve it. Some day, it was simply gone; I have no idea what had changed. I couldn't reproduce it anymore, so I couldn't do any further analysis.

Anyway, most likely this is not a NAND issue or MTD issue, but a JFFS2 issue. And I would certainly want to hear about it if you manage to find and fix it ...

Best regards,
Thomas

Hi Thomas
till now, we have no evidence that it is a jffs2.
we only observed a ubi error msg(not 100% sure it caused this hole):

Oct 15 20:09:10 (none) kernel: UBI error: ubi_io_write error -5 while writing 2048 bytes to PEB 1663:20480, written 0 bytes

Besides, it only appeared in power-cut stress cut, so I suspect it is relevant to "unstable bits issue". from google information, this issue appears in both nand and nor flash, but more frequently in nand.

Unstable bits issue does not give you tons of zeroes. I really think that this is either a JFFS2 bug - fsync does not work, or your test scripts have a bug - you power-cut before fsync or you do fsync, then write more to this file, then power-cut.

Best Regards,
Artem Bityutskiy

I've seen it a number of times with JFFS2 with parallel NOR, on ancient kernels (2.4.26-uc0). Typically the symptom would be an executable crashing, and a 4k hole would be causing that.

We always put it down to a faulty board with poor signal and/or timings to the NOR, as it "seemed" to happen on particular boards more, and simply removed those boards from service. However testing wasn't very systematic.

If JFFS2 sees a corrupt block (detected by CRC), it simply discards that block from the file data as if it's never been written, making a hole. An I/O error would be much nicer than corrupt data, but a hole is what we get.

It might have been caused by faulty boards but in a different way:
Some of our boards crashed every few weeks because the manufacturer's tolerances between CPU and DRAM were too tight. After a very long time in the field (years), we learned to underclock those slightly. Maybe, occasional DRAM corruption was also causing occasional NOR corruption as a side effect (e.g. writing/reading data that didn't match the CRC during JFFS2 GC), leading to 4k holes in previously good files (bad but understandable); or maybe occasional crashes caused it (not acceptable and shouldn't happen). But I am not sure the two issues were correlated anyway.

-- Jamie

I do not think it is related to unstable bits. It may be related to JFFS2 having a bug and not flushing the write-buffer on fsync, so fsync does not work. Check that.

Best Regards,
Artem Bityutskiy