Testing generic empty page bit flips recovery
I am trying to follow up on this discussion from this patch set (https://patchwork.ozlabs.org/patch/539059/) which suggested that Michael instead test the generic bitflips recovery that is implemented by Boris "mtd: nand: properly handle bitflips in erased pages" patchset
(http://lists.infradead.org/pipermail/linux-mtd/2015-September/061617.html).
I would like to test Boris patchset but first I need to recreate the error that his patch is fixing.
The error that the patchset is attempting to fix isn't something I have ever encountered before. Currently I am trying to reproduce this issue on a TI K2E evm that uses the davinci nand driver. I flashed the nand's file-system partition with a ubi filesystem and the board is currently set to boot using the file-system on the nand. After about 60 secs I cut the power from the board and boot the board again. What I would expect is that the board will eventually fail to mount the ubi filesystem but currently the board has ran for over 24 hours and powered on and off over 1400 times and its still mounting the file-system perfectly fine.
Any suggestions on a test case that I can use to force the empty page bit flips error?
Hi Franklin,
Quoted 21 lines · On Wed, 30 Dec 2015 08:10:20 -0600 "Franklin S Cooper Jr." <[email protected]>…
I am trying to follow up on this discussion from this patch set (https://patchwork.ozlabs.org/patch/539059/) which suggested that Michael instead test the generic bitflips recovery that is implemented by Boris "mtd: nand: properly handle bitflips in erased pages" patchset
(http://lists.infradead.org/pipermail/linux-mtd/2015-September/061617.html).
I would like to test Boris patchset but first I need to recreate the error that his patch is fixing.The error that the patchset is attempting to fix isn't something I have ever encountered before. Currently I am trying to reproduce this issue on a TI K2E evm that uses the davinci nand driver. I flashed the nand's file-system partition with a ubi filesystem and the board is currently set to boot using the file-system on the nand. After about 60 secs I cut the power from the board and boot the board again. What I would expect is that the board will eventually fail to mount the ubi filesystem but currently the board has ran for over 24 hours and powered on and off over 1400 times and its still mounting the file-system perfectly fine.
Quoted 4 lines
Any suggestions on a test case that I can use to force the empty page bit flips error?
The davinci driver seems to support raw accesses, so you can try to apply this patch [1] against the mtd-utils tree (not sure it still applies cleany, but it should work with mtd-utils-1.5.1), and use the nandflipbits tool:
# flash_erase /dev/mtdX <offset> 1 # nandflipbits /dev/mtdX 1@<offset> # nanddump -f /tmp/dump -s <offset> -l <page-size> /dev/mtdX
Without the patch, nanddump should complain about uncorrectable errors, and if you hexdump /dev/dump you should see the bitflip.
If nanddump does not complain after applying my patch, then it means it fixes the "bitflips in erased pages" bug.
Best Regards,
Boris
[1]http://lists.infradead.org/pipermail/linux-mtd/2014-November/056634.html
Signature · 3 lines
Boris Brezillon, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com
Quoted 44 lines · Boris Brezillon
Hi Franklin,
Quoted 21 lines · On Wed, 30 Dec 2015 08:10:20 -0600 "Franklin S Cooper Jr." <[email protected]>…
I am trying to follow up on this discussion from this patch set (https://patchwork.ozlabs.org/patch/539059/) which suggested that Michael instead test the generic bitflips recovery that is implemented by Boris "mtd: nand: properly handle bitflips in erased pages" patchset
(http://lists.infradead.org/pipermail/linux-mtd/2015-September/061617.html).
I would like to test Boris patchset but first I need to recreate the error that his patch is fixing.The error that the patchset is attempting to fix isn't something I have ever encountered before. Currently I am trying to reproduce this issue on a TI K2E evm that uses the davinci nand driver. I flashed the nand's file-system partition with a ubi filesystem and the board is currently set to boot using the file-system on the nand. After about 60 secs I cut the power from the board and boot the board again. What I would expect is that the board will eventually fail to mount the ubi filesystem but currently the board has ran for over 24 hours and powered on and off over 1400 times and its still mounting the file-system perfectly fine.
Quoted 4 lines
Any suggestions on a test case that I can use to force the empty page bit flips error?
The davinci driver seems to support raw accesses, so you can try to apply this patch [1] against the mtd-utils tree (not sure it still applies cleany, but it should work with mtd-utils-1.5.1), and use the nandflipbits tool:
# flash_erase /dev/mtdX <offset> 1 # nandflipbits /dev/mtdX 1@<offset> # nanddump -f /tmp/dump -s <offset> -l <page-size> /dev/mtdXWithout the patch, nanddump should complain about uncorrectable errors, and if you hexdump
/dev/dumpyou should see the bitflip.
If nanddump does not complain after applying my patch, then it means it fixes the "bitflips in erased pages" bug.
Quoted 5 lines
Best Regards,
Boris
[1]http://lists.infradead.org/pipermail/linux-mtd/2014-November/056634.html
Hi Boris,
Thanks for the quick reply. I built mtd-utils with your patch and ran the suggested commands on a 4.1 based kernel without your kernel patchset and I didn't see your expected output. The 4.1 based kernel hasn't had any changes to davinci_nand or nand subsystem that would address this bitflip error.
I'm currently going to attempt to run the same test on the latest mainline.
Here is the output I received when I ran your suggested commands on the 4.1 based kernel.Any
root@k2e-evm:~# ./flash_erase /dev/mtd4 4096 1 Erasing 128 Kibyte @ 0 -- 100 % complete root@k2e-evm:~# ./nandflipbits /dev/mtd4 1@4096 root@k2e-evm:~# ./nanddump -f /tmp/dump -s 4096 -l 2048 /dev/mtd4 ECC failed: 0 ECC corrected: 0 Number of bad blocks: 0 Number of bbt blocks: 4 Block size 131072, page size 2048, OOB size 64 root@k2e-evm:~# hexdump /tmp/dump 0000000 fffd ffff ffff ffff ffff ffff ffff ffff 0000010 ffff ffff ffff ffff ffff ffff ffff ffff * 0000800
Any thoughts on why I'm not seeing the expected error?
Quoted 65 lines · On Wed, 30 Dec 2015 09:33:52 -0600 "Franklin S Cooper Jr." <[email protected]>…
Quoted 44 lines · Boris Brezillon
Hi Franklin,
Quoted 21 lines · On Wed, 30 Dec 2015 08:10:20 -0600 "Franklin S Cooper Jr." <[email protected]>…
I am trying to follow up on this discussion from this patch set (https://patchwork.ozlabs.org/patch/539059/) which suggested that Michael instead test the generic bitflips recovery that is implemented by Boris "mtd: nand: properly handle bitflips in erased pages" patchset
(http://lists.infradead.org/pipermail/linux-mtd/2015-September/061617.html).
I would like to test Boris patchset but first I need to recreate the error that his patch is fixing.The error that the patchset is attempting to fix isn't something I have ever encountered before. Currently I am trying to reproduce this issue on a TI K2E evm that uses the davinci nand driver. I flashed the nand's file-system partition with a ubi filesystem and the board is currently set to boot using the file-system on the nand. After about 60 secs I cut the power from the board and boot the board again. What I would expect is that the board will eventually fail to mount the ubi filesystem but currently the board has ran for over 24 hours and powered on and off over 1400 times and its still mounting the file-system perfectly fine.
Quoted 4 lines
Any suggestions on a test case that I can use to force the empty page bit flips error?
The davinci driver seems to support raw accesses, so you can try to apply this patch [1] against the mtd-utils tree (not sure it still applies cleany, but it should work with mtd-utils-1.5.1), and use the nandflipbits tool:
# flash_erase /dev/mtdX <offset> 1 # nandflipbits /dev/mtdX 1@<offset> # nanddump -f /tmp/dump -s <offset> -l <page-size> /dev/mtdXWithout the patch, nanddump should complain about uncorrectable errors, and if you hexdump
/dev/dumpyou should see the bitflip.
If nanddump does not complain after applying my patch, then it means it fixes the "bitflips in erased pages" bug.Quoted 5 lines
Best Regards,
Boris
[1]http://lists.infradead.org/pipermail/linux-mtd/2014-November/056634.html
Hi Boris,
Thanks for the quick reply. I built mtd-utils with your patch and ran the suggested commands on a 4.1 based kernel without your kernel patchset and I didn't see your expected output. The 4.1 based kernel hasn't had any changes to
davinci_nandor nand subsystem that would address this bitflip error.I'm currently going to attempt to run the same test on the latest mainline.
Quoted 6 lines
Here is the output I received when I ran your suggested commands on the 4.1 based kernel.Any
root@k2e-evm:~# ./flash_erase /dev/mtd4 4096 1 Erasing 128 Kibyte @ 0 -- 100 % complete root@k2e-evm:~# ./nandflipbits /dev/mtd4 1@4096 root@k2e-evm:~# ./nanddump -f /tmp/dump -s 4096 -l 2048
You should probably use a block aligned offset (in your case a block is 128k), but that's not the problem here.
Quoted 8 lines
/dev/mtd4
ECC failed: 0
ECC corrected: 0
Number of bad blocks: 0
Number of bbt blocks: 4
Block size 131072, page size 2048, OOB size 64root@k2e-evm:~# hexdump /tmp/dump 0000000 fffd ffff ffff ffff ffff ffff ffff ffff
^
The bitflip is here.
Quoted 6 lines
0000010 ffff ffff ffff ffff ffff ffff ffff ffff
*
0000800Any thoughts on why I'm not seeing the expected error?
Is ecc4bit mode really selected (ti,davinci-ecc-bits = 4 in your DT node)?
You can add a trace there [1] to check that.
[1]http://lxr.free-electrons.com/source/drivers/mtd/nand/davinci_nand.c?v=4.1#L706
Signature · 3 lines
Boris Brezillon, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com
Quoted 83 lines · On Wed, 30 Dec 2015 09:33:52 -0600 "Franklin S Cooper Jr." <[email protected]>…
Quoted 44 lines · Boris Brezillon
Hi Franklin,
Quoted 21 lines · On Wed, 30 Dec 2015 08:10:20 -0600 "Franklin S Cooper Jr." <[email protected]>…
I am trying to follow up on this discussion from this patch set (https://patchwork.ozlabs.org/patch/539059/) which suggested that Michael instead test the generic bitflips recovery that is implemented by Boris "mtd: nand: properly handle bitflips in erased pages" patchset
(http://lists.infradead.org/pipermail/linux-mtd/2015-September/061617.html).
I would like to test Boris patchset but first I need to recreate the error that his patch is fixing.The error that the patchset is attempting to fix isn't something I have ever encountered before. Currently I am trying to reproduce this issue on a TI K2E evm that uses the davinci nand driver. I flashed the nand's file-system partition with a ubi filesystem and the board is currently set to boot using the file-system on the nand. After about 60 secs I cut the power from the board and boot the board again. What I would expect is that the board will eventually fail to mount the ubi filesystem but currently the board has ran for over 24 hours and powered on and off over 1400 times and its still mounting the file-system perfectly fine.
Quoted 4 lines
Any suggestions on a test case that I can use to force the empty page bit flips error?
The davinci driver seems to support raw accesses, so you can try to apply this patch [1] against the mtd-utils tree (not sure it still applies cleany, but it should work with mtd-utils-1.5.1), and use the nandflipbits tool:
# flash_erase /dev/mtdX <offset> 1 # nandflipbits /dev/mtdX 1@<offset> # nanddump -f /tmp/dump -s <offset> -l <page-size> /dev/mtdXWithout the patch, nanddump should complain about uncorrectable errors, and if you hexdump
/dev/dumpyou should see the bitflip.
If nanddump does not complain after applying my patch, then it means it fixes the "bitflips in erased pages" bug.Quoted 5 lines
Best Regards,
Boris
[1]http://lists.infradead.org/pipermail/linux-mtd/2014-November/056634.html
Hi Boris,
Thanks for the quick reply. I built mtd-utils with your patch and ran the suggested commands on a 4.1 based kernel without your kernel patchset and I didn't see your expected output. The 4.1 based kernel hasn't had any changes to
davinci_nandor nand subsystem that would address this bitflip error.I'm currently going to attempt to run the same test on the latest mainline.
Here is the output I received when I ran your suggested commands on the 4.1 based kernel.Any
root@k2e-evm:~# ./flash_erase /dev/mtd4 4096 1 Erasing 128 Kibyte @ 0 -- 100 % complete root@k2e-evm:~# ./nandflipbits /dev/mtd4 1@4096 root@k2e-evm:~# ./nanddump -f /tmp/dump -s 4096 -l 2048 /dev/mtd4 ECC failed: 0 ECC corrected: 0 Number of bad blocks: 0 Number of bbt blocks: 4 Block size 131072, page size 2048, OOB size 64 root@k2e-evm:~# hexdump /tmp/dump 0000000 fffd ffff ffff ffff ffff ffff ffff ffff 0000010 ffff ffff ffff ffff ffff ffff ffff ffff * 0000800
Quoted 2 lines
Any thoughts on why I'm not seeing the expected error?
Oh, actually this behavior is explained in the commit message:
"Currently empty page bit flips are not corrected and report 0 errors."
Which explains why you're seeing the bitflip in the dump, but nothing reported by the MTD layer.
After applying my patch, the bitflip should simply disappear. You can then try to generate more bitflips than the engine can actually fix (nandflipbits /dev/mtd4 1@0:5@0:49@0:98@0:132@0) and check that MTD reports an uncorrectable error.
Signature · 3 lines
Boris Brezillon, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com
Quoted 90 lines · Boris Brezillon
Quoted 81 lines · On Wed, 30 Dec 2015 09:33:52 -0600 "Franklin S Cooper Jr." <[email protected]>…
Quoted 44 lines · Boris Brezillon
Hi Franklin,
Quoted 21 lines · On Wed, 30 Dec 2015 08:10:20 -0600 "Franklin S Cooper Jr." <[email protected]>…
I am trying to follow up on this discussion from this patch set (https://patchwork.ozlabs.org/patch/539059/) which suggested that Michael instead test the generic bitflips recovery that is implemented by Boris "mtd: nand: properly handle bitflips in erased pages" patchset
(http://lists.infradead.org/pipermail/linux-mtd/2015-September/061617.html).
I would like to test Boris patchset but first I need to recreate the error that his patch is fixing.The error that the patchset is attempting to fix isn't something I have ever encountered before. Currently I am trying to reproduce this issue on a TI K2E evm that uses the davinci nand driver. I flashed the nand's file-system partition with a ubi filesystem and the board is currently set to boot using the file-system on the nand. After about 60 secs I cut the power from the board and boot the board again. What I would expect is that the board will eventually fail to mount the ubi filesystem but currently the board has ran for over 24 hours and powered on and off over 1400 times and its still mounting the file-system perfectly fine.
Quoted 4 lines
Any suggestions on a test case that I can use to force the empty page bit flips error?
The davinci driver seems to support raw accesses, so you can try to apply this patch [1] against the mtd-utils tree (not sure it still applies cleany, but it should work with mtd-utils-1.5.1), and use the nandflipbits tool:
# flash_erase /dev/mtdX <offset> 1 # nandflipbits /dev/mtdX 1@<offset> # nanddump -f /tmp/dump -s <offset> -l <page-size> /dev/mtdXWithout the patch, nanddump should complain about uncorrectable errors, and if you hexdump
/dev/dumpyou should see the bitflip.
If nanddump does not complain after applying my patch, then it means it fixes the "bitflips in erased pages" bug.Quoted 5 lines
Best Regards,
Boris
[1]http://lists.infradead.org/pipermail/linux-mtd/2014-November/056634.html
Hi Boris,
Thanks for the quick reply. I built mtd-utils with your patch and ran the suggested commands on a 4.1 based kernel without your kernel patchset and I didn't see your expected output. The 4.1 based kernel hasn't had any changes to
davinci_nandor nand subsystem that would address this bitflip error.I'm currently going to attempt to run the same test on the latest mainline.
Here is the output I received when I ran your suggested commands on the 4.1 based kernel.Any
root@k2e-evm:~# ./flash_erase /dev/mtd4 4096 1 Erasing 128 Kibyte @ 0 -- 100 % complete root@k2e-evm:~# ./nandflipbits /dev/mtd4 1@4096 root@k2e-evm:~# ./nanddump -f /tmp/dump -s 4096 -l 2048 /dev/mtd4 ECC failed: 0 ECC corrected: 0 Number of bad blocks: 0 Number of bbt blocks: 4 Block size 131072, page size 2048, OOB size 64 root@k2e-evm:~# hexdump /tmp/dump 0000000 fffd ffff ffff ffff ffff ffff ffff ffff 0000010 ffff ffff ffff ffff ffff ffff ffff ffff * 0000800Quoted 2 lines
Any thoughts on why I'm not seeing the expected error?
Oh, actually this behavior is explained in the commit message:
"Currently empty page bit flips are not corrected and report 0 errors."
Quoted 7 lines
Which explains why you're seeing the bitflip in the dump, but nothing reported by the MTD layer.
After applying my patch, the bitflip should simply disappear. You can then try to generate more bitflips than the engine can actually fix (nandflipbits
/dev/mtd41@0:5@0:49@0:98@0:132@0) and check that MTD reports an uncorrectable error.
I verified that I am indeed using ecc4bit mode.
I attempted to run the series of nandflipsbits as you suggested but I get "invalid bit description" error from the utility. Some reason I can only use the nandflipsbits utility for bits 1-7. Anything higher and I get the "Invalid bit description" error.
On the latest master commit I ran nandflipsbits for bits 1-7 at address 0. However, I still didn't receive any error from nanddump although I do see the flip bits from the hexdump /tmp/dump output.
I then applied your patchset ontop of the latest mainline and ran nandflipsbits for bits 1-7 at address 0. I get the below output which seems to be correct.
root@k2e-evm:~# ./nandflipbits /dev/mtd4 1@0 root@k2e-evm:~# ./nandflipbits /dev/mtd4 2@0 root@k2e-evm:~# ./nandflipbits /dev/mtd4 3@0 root@k2e-evm:~# ./nandflipbits /dev/mtd4 4@0 root@k2e-evm:~# ./nandflipbits /dev/mtd4 5@0 root@k2e-evm:~# ./nandflipbits /dev/mtd4 6@0 root@k2e-evm:~# ./nandflipbits /dev/mtd4 7@0 root@k2e-evm:~# ./nanddump -f /tmp/dump -s 0 -l 2048 /dev/mtd4
ECC failed: 1
ECC corrected: 18
Number of bad blocks: 0
Number of bbt blocks: 4
Block size 131072, page size 2048, OOB size 64 Dumping data starting at 0x00000000 and ending at 0x00000800...
ECC: 4 corrected bitflip(s) at offset 0x00000000
root@k2e-evm:~# hexdump /tmp/dump 0000000 ffff ffff ffff ffff ffff ffff ffff ffff * 0000800
One thing that confuses me is if I repeatedly call nanddump I continue to get the "ECC: 4 corrected bitflips" message and the "ECC corrected" count increases by 4 each time. If these bits are being corrected which is apparent from looking at the output of nanddump shouldn't sequential calls indicate that no bitflips needed to be corrected since it was corrected previously?
Quoted 9 lines · Franklin S Cooper Jr.
One thing that confuses me is if I repeatedly call nanddump I continue to get the "ECC: 4 corrected bitflips" message and the "ECC corrected" count increases by 4 each time. If these bits are being corrected which is apparent from looking at the output of nanddump shouldn't sequential calls indicate that no bitflips needed to be corrected since it was corrected previously?
Hi Franklin,
I'm making a guess at the source of your confusion, but I've had to answer this to repeated colleagues recently, so I'll give it a try.
I think you're expecting the "correction" to actually be written and fixed on-flash. This is not the case.
Bitflip corrections are applied as the flash data is read. Every time. So once a bit is flipped on the NAND, it will always stay that way on the physical device until erased and rewritten. Which doesn't happen for something as minor as a single bit-flip. So from that point on, it will always read and correct the flip and report it to the kernel log.
UBI will move the data once the threshold gets hit, but up until that point it will continue to read the same bitflip and correct it.
Hope that helps.
- Steve
Quoted 23 lines · Steve deRosier
Quoted 8 lines · Franklin S Cooper Jr.
One thing that confuses me is if I repeatedly call nanddump I continue to get the "ECC: 4 corrected bitflips" message and the "ECC corrected" count increases by 4 each time. If these bits are being corrected which is apparent from looking at the output of nanddump shouldn't sequential calls indicate that no bitflips needed to be corrected since it was corrected previously?
Hi Franklin,
I'm making a guess at the source of your confusion, but I've had to answer this to repeated colleagues recently, so I'll give it a try.
I think you're expecting the "correction" to actually be written and fixed on-flash. This is not the case.
Bitflip corrections are applied as the flash data is read. Every time. So once a bit is flipped on the NAND, it will always stay that way on the physical device until erased and rewritten. Which doesn't happen for something as minor as a single bit-flip. So from that point on, it will always read and correct the flip and report it to the kernel log.
Quoted 6 lines
UBI will move the data once the threshold gets hit, but up until that point it will continue to read the same bitflip and correct it.
Hope that helps.
- Steve
Steve,
That makes perfect sense. Thanks for explaining.
Quoted 101 lines · On Wed, 30 Dec 2015 10:40:49 -0600 "Franklin S Cooper Jr." <[email protected]>…
Quoted 90 lines · Boris Brezillon
Quoted 81 lines · On Wed, 30 Dec 2015 09:33:52 -0600 "Franklin S Cooper Jr." <[email protected]>…
Quoted 44 lines · Boris Brezillon
Hi Franklin,
Quoted 21 lines · On Wed, 30 Dec 2015 08:10:20 -0600 "Franklin S Cooper Jr." <[email protected]>…
I am trying to follow up on this discussion from this patch set (https://patchwork.ozlabs.org/patch/539059/) which suggested that Michael instead test the generic bitflips recovery that is implemented by Boris "mtd: nand: properly handle bitflips in erased pages" patchset
(http://lists.infradead.org/pipermail/linux-mtd/2015-September/061617.html).
I would like to test Boris patchset but first I need to recreate the error that his patch is fixing.The error that the patchset is attempting to fix isn't something I have ever encountered before. Currently I am trying to reproduce this issue on a TI K2E evm that uses the davinci nand driver. I flashed the nand's file-system partition with a ubi filesystem and the board is currently set to boot using the file-system on the nand. After about 60 secs I cut the power from the board and boot the board again. What I would expect is that the board will eventually fail to mount the ubi filesystem but currently the board has ran for over 24 hours and powered on and off over 1400 times and its still mounting the file-system perfectly fine.
Quoted 4 lines
Any suggestions on a test case that I can use to force the empty page bit flips error?
The davinci driver seems to support raw accesses, so you can try to apply this patch [1] against the mtd-utils tree (not sure it still applies cleany, but it should work with mtd-utils-1.5.1), and use the nandflipbits tool:
# flash_erase /dev/mtdX <offset> 1 # nandflipbits /dev/mtdX 1@<offset> # nanddump -f /tmp/dump -s <offset> -l <page-size> /dev/mtdXWithout the patch, nanddump should complain about uncorrectable errors, and if you hexdump
/dev/dumpyou should see the bitflip.
If nanddump does not complain after applying my patch, then it means it fixes the "bitflips in erased pages" bug.Quoted 5 lines
Best Regards,
Boris
[1]http://lists.infradead.org/pipermail/linux-mtd/2014-November/056634.html
Hi Boris,
Thanks for the quick reply. I built mtd-utils with your patch and ran the suggested commands on a 4.1 based kernel without your kernel patchset and I didn't see your expected output. The 4.1 based kernel hasn't had any changes to
davinci_nandor nand subsystem that would address this bitflip error.I'm currently going to attempt to run the same test on the latest mainline.
Here is the output I received when I ran your suggested commands on the 4.1 based kernel.Any
root@k2e-evm:~# ./flash_erase /dev/mtd4 4096 1 Erasing 128 Kibyte @ 0 -- 100 % complete root@k2e-evm:~# ./nandflipbits /dev/mtd4 1@4096 root@k2e-evm:~# ./nanddump -f /tmp/dump -s 4096 -l 2048 /dev/mtd4 ECC failed: 0 ECC corrected: 0 Number of bad blocks: 0 Number of bbt blocks: 4 Block size 131072, page size 2048, OOB size 64 root@k2e-evm:~# hexdump /tmp/dump 0000000 fffd ffff ffff ffff ffff ffff ffff ffff 0000010 ffff ffff ffff ffff ffff ffff ffff ffff * 0000800Quoted 2 lines
Any thoughts on why I'm not seeing the expected error?
Oh, actually this behavior is explained in the commit message:
"Currently empty page bit flips are not corrected and report 0 errors."
Quoted 7 lines
Which explains why you're seeing the bitflip in the dump, but nothing reported by the MTD layer.
After applying my patch, the bitflip should simply disappear. You can then try to generate more bitflips than the engine can actually fix (nandflipbits
/dev/mtd41@0:5@0:49@0:98@0:132@0) and check that MTD reports an uncorrectable error.
Quoted 7 lines
I verified that I am indeed using ecc4bit mode.
I attempted to run the series of nandflipsbits as you suggested but I get "invalid bit description" error from the utility. Some reason I can only use the nandflipsbits utility for bits 1-7. Anything higher and I get the "Invalid bit description" error.
Indeed. I developed that tool a long time ago and didn't remember that the bit field is encoding the bit offset within a byte. This command should work.
nandflipbits /dev/mtd4 1@0:5@0:7@30:3@46:5@47
Quoted 5 lines
On the latest master commit I ran nandflipsbits for bits 1-7 at address 0. However, I still didn't receive any error from nanddump although I do see the flip bits from the hexdump
/tmp/dumpoutput.
How many of them do you see?
Quoted 22 lines
I then applied your patchset ontop of the latest mainline and ran nandflipsbits for bits 1-7 at address 0. I get the below output which seems to be correct.
root@k2e-evm:~# ./nandflipbits /dev/mtd4 1@0 root@k2e-evm:~# ./nandflipbits /dev/mtd4 2@0 root@k2e-evm:~# ./nandflipbits /dev/mtd4 3@0 root@k2e-evm:~# ./nandflipbits /dev/mtd4 4@0 root@k2e-evm:~# ./nandflipbits /dev/mtd4 5@0 root@k2e-evm:~# ./nandflipbits /dev/mtd4 6@0 root@k2e-evm:~# ./nandflipbits /dev/mtd4 7@0 root@k2e-evm:~# ./nanddump -f /tmp/dump -s 0 -l 2048 /dev/mtd4ECC failed: 1
ECC corrected: 18
Number of bad blocks: 0
Number of bbt blocks: 4
Block size 131072, page size 2048, OOB size 64 Dumping data starting at0x00000000and ending at0x00000800...
ECC: 4 corrected bitflip(s) at offset0x00000000
Quoted 4 lines
root@k2e-evm:~# hexdump /tmp/dump 0000000 ffff ffff ffff ffff ffff ffff ffff ffff * 0000800
Hm, that's weird. You should get an ECC failure since the ECC strength is only 4bits/512byte and you 8 bits have been flipped.
Quoted 8 lines
One thing that confuses me is if I repeatedly call nanddump I continue to get the "ECC: 4 corrected bitflips" message and the "ECC corrected" count increases by 4 each time. If these bits are being corrected which is apparent from looking at the output of nanddump shouldn't sequential calls indicate that no bitflips needed to be corrected since it was corrected previously?
Nope, they're corrected on the fly and only in RAM, so each time you read the page, you'll have to fix the bitflips until you erase and rewrite the faulty block.
Signature · 3 lines
Boris Brezillon, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com
Quoted 155 lines · Boris Brezillon
Quoted 98 lines · On Wed, 30 Dec 2015 10:40:49 -0600 "Franklin S Cooper Jr." <[email protected]>…
Quoted 89 lines · Boris Brezillon
Quoted 80 lines · On Wed, 30 Dec 2015 09:33:52 -0600 "Franklin S Cooper Jr." <[email protected]>…
Quoted 44 lines · Boris Brezillon
Hi Franklin,
Quoted 21 lines · On Wed, 30 Dec 2015 08:10:20 -0600 "Franklin S Cooper Jr." <[email protected]>…
I am trying to follow up on this discussion from this patch set (https://patchwork.ozlabs.org/patch/539059/) which suggested that Michael instead test the generic bitflips recovery that is implemented by Boris "mtd: nand: properly handle bitflips in erased pages" patchset
(http://lists.infradead.org/pipermail/linux-mtd/2015-September/061617.html).
I would like to test Boris patchset but first I need to recreate the error that his patch is fixing.The error that the patchset is attempting to fix isn't something I have ever encountered before. Currently I am trying to reproduce this issue on a TI K2E evm that uses the davinci nand driver. I flashed the nand's file-system partition with a ubi filesystem and the board is currently set to boot using the file-system on the nand. After about 60 secs I cut the power from the board and boot the board again. What I would expect is that the board will eventually fail to mount the ubi filesystem but currently the board has ran for over 24 hours and powered on and off over 1400 times and its still mounting the file-system perfectly fine.
Quoted 4 lines
Any suggestions on a test case that I can use to force the empty page bit flips error?
The davinci driver seems to support raw accesses, so you can try to apply this patch [1] against the mtd-utils tree (not sure it still applies cleany, but it should work with mtd-utils-1.5.1), and use the nandflipbits tool:
# flash_erase /dev/mtdX <offset> 1 # nandflipbits /dev/mtdX 1@<offset> # nanddump -f /tmp/dump -s <offset> -l <page-size> /dev/mtdXWithout the patch, nanddump should complain about uncorrectable errors, and if you hexdump
/dev/dumpyou should see the bitflip.
If nanddump does not complain after applying my patch, then it means it fixes the "bitflips in erased pages" bug.Quoted 5 lines
Best Regards,
Boris
[1]http://lists.infradead.org/pipermail/linux-mtd/2014-November/056634.html
Hi Boris,
Thanks for the quick reply. I built mtd-utils with your patch and ran the suggested commands on a 4.1 based kernel without your kernel patchset and I didn't see your expected output. The 4.1 based kernel hasn't had any changes to
davinci_nandor nand subsystem that would address this bitflip error.I'm currently going to attempt to run the same test on the latest mainline.
Here is the output I received when I ran your suggested commands on the 4.1 based kernel.Any
root@k2e-evm:~# ./flash_erase /dev/mtd4 4096 1 Erasing 128 Kibyte @ 0 -- 100 % complete root@k2e-evm:~# ./nandflipbits /dev/mtd4 1@4096 root@k2e-evm:~# ./nanddump -f /tmp/dump -s 4096 -l 2048 /dev/mtd4 ECC failed: 0 ECC corrected: 0 Number of bad blocks: 0 Number of bbt blocks: 4 Block size 131072, page size 2048, OOB size 64 root@k2e-evm:~# hexdump /tmp/dump 0000000 fffd ffff ffff ffff ffff ffff ffff ffff 0000010 ffff ffff ffff ffff ffff ffff ffff ffff * 0000800Quoted 2 lines
Any thoughts on why I'm not seeing the expected error?
Oh, actually this behavior is explained in the commit message:
"Currently empty page bit flips are not corrected and report 0 errors."
Quoted 7 lines
Which explains why you're seeing the bitflip in the dump, but nothing reported by the MTD layer.
After applying my patch, the bitflip should simply disappear. You can then try to generate more bitflips than the engine can actually fix (nandflipbits
/dev/mtd41@0:5@0:49@0:98@0:132@0) and check that MTD reports an uncorrectable error.Quoted 7 lines
I verified that I am indeed using ecc4bit mode.
I attempted to run the series of nandflipsbits as you suggested but I get "invalid bit description" error from the utility. Some reason I can only use the nandflipsbits utility for bits 1-7. Anything higher and I get the "Invalid bit description" error.
Indeed. I developed that tool a long time ago and didn't remember that the bit field is encoding the bit offset within a byte. This command should work.
nandflipbits
/dev/mtd41@0:5@0:7@30:3@46:5@47Quoted 4 lines
On the latest master commit I ran nandflipsbits for bits 1-7 at address 0. However, I still didn't receive any error from nanddump although I do see the flip bits from the hexdump
/tmp/dumpoutput.How many of them do you see?
Quoted 21 lines
I then applied your patchset ontop of the latest mainline and ran nandflipsbits for bits 1-7 at address 0. I get the below output which seems to be correct.
root@k2e-evm:~# ./nandflipbits /dev/mtd4 1@0 root@k2e-evm:~# ./nandflipbits /dev/mtd4 2@0 root@k2e-evm:~# ./nandflipbits /dev/mtd4 3@0 root@k2e-evm:~# ./nandflipbits /dev/mtd4 4@0 root@k2e-evm:~# ./nandflipbits /dev/mtd4 5@0 root@k2e-evm:~# ./nandflipbits /dev/mtd4 6@0 root@k2e-evm:~# ./nandflipbits /dev/mtd4 7@0 root@k2e-evm:~# ./nanddump -f /tmp/dump -s 0 -l 2048 /dev/mtd4ECC failed: 1
ECC corrected: 18
Number of bad blocks: 0
Number of bbt blocks: 4
Block size 131072, page size 2048, OOB size 64 Dumping data starting at0x00000000and ending at0x00000800...
ECC: 4 corrected bitflip(s) at offset0x00000000Quoted 4 lines
root@k2e-evm:~# hexdump /tmp/dump 0000000 ffff ffff ffff ffff ffff ffff ffff ffff * 0000800Hm, that's weird. You should get an ECC failure since the ECC strength is only 4bits/512byte and you 8 bits have been flipped.
Quoted 7 lines
One thing that confuses me is if I repeatedly call nanddump I continue to get the "ECC: 4 corrected bitflips" message and the "ECC corrected" count increases by 4 each time. If these bits are being corrected which is apparent from looking at the output of nanddump shouldn't sequential calls indicate that no bitflips needed to be corrected since it was corrected previously?
Quoted 5 lines
Nope, they're corrected on the fly and only in RAM, so each time you read the page, you'll have to fix the bitflips until you erase and rewrite the faulty block.
Hi Boris,
Here is the entire output that should answer your questions.
In the log I am running the following commands:flash_erase /dev/mtd4 0 0
./nanddump -f /tmp/dump -s 0 -l 2048 /dev/mtd4
hexdump /tmp/dump
./nandflipbits /dev/mtd4 1@0:5@0:7@30:3@46:5@47
./nanddump -f /tmp/dump -s 0 -l 2048 /dev/mtd4
hexdump /tmp/dump
Output on mainline kernel without bitflip correction patches:
http://pastebin.com/MgBVxALR
Output on mainline kernel with bitflip correction patches:
http://pastebin.com/NdKv0NhV
Some reason I'm only getting 1 bit being corrected when using the bitflip correction patches. Comparing my logs from before to now the only difference I'm seeing is that ECC failed is increasing but ECC corrected isn't changing.
Quoted 179 lines · On Wed, 30 Dec 2015 11:45:38 -0600 "Franklin S Cooper Jr." <[email protected]>…
Quoted 155 lines · Boris Brezillon
Quoted 98 lines · On Wed, 30 Dec 2015 10:40:49 -0600 "Franklin S Cooper Jr." <[email protected]>…
Quoted 89 lines · Boris Brezillon
Quoted 80 lines · On Wed, 30 Dec 2015 09:33:52 -0600 "Franklin S Cooper Jr." <[email protected]>…
Quoted 44 lines · Boris Brezillon
Hi Franklin,
Quoted 21 lines · On Wed, 30 Dec 2015 08:10:20 -0600 "Franklin S Cooper Jr." <[email protected]>…
I am trying to follow up on this discussion from this patch set (https://patchwork.ozlabs.org/patch/539059/) which suggested that Michael instead test the generic bitflips recovery that is implemented by Boris "mtd: nand: properly handle bitflips in erased pages" patchset
(http://lists.infradead.org/pipermail/linux-mtd/2015-September/061617.html).
I would like to test Boris patchset but first I need to recreate the error that his patch is fixing.The error that the patchset is attempting to fix isn't something I have ever encountered before. Currently I am trying to reproduce this issue on a TI K2E evm that uses the davinci nand driver. I flashed the nand's file-system partition with a ubi filesystem and the board is currently set to boot using the file-system on the nand. After about 60 secs I cut the power from the board and boot the board again. What I would expect is that the board will eventually fail to mount the ubi filesystem but currently the board has ran for over 24 hours and powered on and off over 1400 times and its still mounting the file-system perfectly fine.
Quoted 4 lines
Any suggestions on a test case that I can use to force the empty page bit flips error?
The davinci driver seems to support raw accesses, so you can try to apply this patch [1] against the mtd-utils tree (not sure it still applies cleany, but it should work with mtd-utils-1.5.1), and use the nandflipbits tool:
# flash_erase /dev/mtdX <offset> 1 # nandflipbits /dev/mtdX 1@<offset> # nanddump -f /tmp/dump -s <offset> -l <page-size> /dev/mtdXWithout the patch, nanddump should complain about uncorrectable errors, and if you hexdump
/dev/dumpyou should see the bitflip.
If nanddump does not complain after applying my patch, then it means it fixes the "bitflips in erased pages" bug.Quoted 5 lines
Best Regards,
Boris
[1]http://lists.infradead.org/pipermail/linux-mtd/2014-November/056634.html
Hi Boris,
Thanks for the quick reply. I built mtd-utils with your patch and ran the suggested commands on a 4.1 based kernel without your kernel patchset and I didn't see your expected output. The 4.1 based kernel hasn't had any changes to
davinci_nandor nand subsystem that would address this bitflip error.I'm currently going to attempt to run the same test on the latest mainline.
Here is the output I received when I ran your suggested commands on the 4.1 based kernel.Any
root@k2e-evm:~# ./flash_erase /dev/mtd4 4096 1 Erasing 128 Kibyte @ 0 -- 100 % complete root@k2e-evm:~# ./nandflipbits /dev/mtd4 1@4096 root@k2e-evm:~# ./nanddump -f /tmp/dump -s 4096 -l 2048 /dev/mtd4 ECC failed: 0 ECC corrected: 0 Number of bad blocks: 0 Number of bbt blocks: 4 Block size 131072, page size 2048, OOB size 64 root@k2e-evm:~# hexdump /tmp/dump 0000000 fffd ffff ffff ffff ffff ffff ffff ffff 0000010 ffff ffff ffff ffff ffff ffff ffff ffff * 0000800Quoted 2 lines
Any thoughts on why I'm not seeing the expected error?
Oh, actually this behavior is explained in the commit message:
"Currently empty page bit flips are not corrected and report 0 errors."
Quoted 7 lines
Which explains why you're seeing the bitflip in the dump, but nothing reported by the MTD layer.
After applying my patch, the bitflip should simply disappear. You can then try to generate more bitflips than the engine can actually fix (nandflipbits
/dev/mtd41@0:5@0:49@0:98@0:132@0) and check that MTD reports an uncorrectable error.Quoted 7 lines
I verified that I am indeed using ecc4bit mode.
I attempted to run the series of nandflipsbits as you suggested but I get "invalid bit description" error from the utility. Some reason I can only use the nandflipsbits utility for bits 1-7. Anything higher and I get the "Invalid bit description" error.
Indeed. I developed that tool a long time ago and didn't remember that the bit field is encoding the bit offset within a byte. This command should work.
nandflipbits
/dev/mtd41@0:5@0:7@30:3@46:5@47Quoted 4 lines
On the latest master commit I ran nandflipsbits for bits 1-7 at address 0. However, I still didn't receive any error from nanddump although I do see the flip bits from the hexdump
/tmp/dumpoutput.How many of them do you see?
Quoted 21 lines
I then applied your patchset ontop of the latest mainline and ran nandflipsbits for bits 1-7 at address 0. I get the below output which seems to be correct.
root@k2e-evm:~# ./nandflipbits /dev/mtd4 1@0 root@k2e-evm:~# ./nandflipbits /dev/mtd4 2@0 root@k2e-evm:~# ./nandflipbits /dev/mtd4 3@0 root@k2e-evm:~# ./nandflipbits /dev/mtd4 4@0 root@k2e-evm:~# ./nandflipbits /dev/mtd4 5@0 root@k2e-evm:~# ./nandflipbits /dev/mtd4 6@0 root@k2e-evm:~# ./nandflipbits /dev/mtd4 7@0 root@k2e-evm:~# ./nanddump -f /tmp/dump -s 0 -l 2048 /dev/mtd4ECC failed: 1
ECC corrected: 18
Number of bad blocks: 0
Number of bbt blocks: 4
Block size 131072, page size 2048, OOB size 64 Dumping data starting at0x00000000and ending at0x00000800...
ECC: 4 corrected bitflip(s) at offset0x00000000Quoted 4 lines
root@k2e-evm:~# hexdump /tmp/dump 0000000 ffff ffff ffff ffff ffff ffff ffff ffff * 0000800Hm, that's weird. You should get an ECC failure since the ECC strength is only 4bits/512byte and you 8 bits have been flipped.
Quoted 7 lines
One thing that confuses me is if I repeatedly call nanddump I continue to get the "ECC: 4 corrected bitflips" message and the "ECC corrected" count increases by 4 each time. If these bits are being corrected which is apparent from looking at the output of nanddump shouldn't sequential calls indicate that no bitflips needed to be corrected since it was corrected previously?
Quoted 5 lines
Nope, they're corrected on the fly and only in RAM, so each time you read the page, you'll have to fix the bitflips until you erase and rewrite the faulty block.
Hi Boris,
Here is the entire output that should answer your questions.
In the log I am running the following commands:
flash_erase/dev/mtd40 0
./nanddump -f/tmp/dump-s 0 -l 2048/dev/mtd4
hexdump/tmp/dump
./nandflipbits/dev/mtd41@0:5@0:7@30:3@46:5@47
./nanddump -f/tmp/dump-s 0 -l 2048/dev/mtd4
hexdump/tmp/dumpOutput on mainline kernel without bitflip correction patches:
http://pastebin.com/MgBVxALR
Quoted 8 lines
Output on mainline kernel with bitflip correction patches:
http://pastebin.com/NdKv0NhVSome reason I'm only getting 1 bit being corrected when using the bitflip correction patches. Comparing my logs from before to now the only difference I'm seeing is that ECC failed is increasing but ECC corrected isn't changing.
That's what I was expecting: your ECC engine is only fixing 4bits/512byte, which is why the bitflip in erased page correction fail when you have more than 4 bits flipped in a given 512byte block.
Now try to flip only 4 bits instead of 5:
./nandflipbits /dev/mtd4 1@0:5@0:7@30:3@46
Signature · 3 lines
Boris Brezillon, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com
Quoted 187 lines · Boris Brezillon
Quoted 176 lines · On Wed, 30 Dec 2015 11:45:38 -0600 "Franklin S Cooper Jr." <[email protected]>…
Quoted 154 lines · Boris Brezillon
Quoted 97 lines · On Wed, 30 Dec 2015 10:40:49 -0600 "Franklin S Cooper Jr." <[email protected]>…
Quoted 89 lines · Boris Brezillon
Quoted 80 lines · On Wed, 30 Dec 2015 09:33:52 -0600 "Franklin S Cooper Jr." <[email protected]>…
Quoted 44 lines · Boris Brezillon
Hi Franklin,
Quoted 21 lines · On Wed, 30 Dec 2015 08:10:20 -0600 "Franklin S Cooper Jr." <[email protected]>…
I am trying to follow up on this discussion from this patch set (https://patchwork.ozlabs.org/patch/539059/) which suggested that Michael instead test the generic bitflips recovery that is implemented by Boris "mtd: nand: properly handle bitflips in erased pages" patchset
(http://lists.infradead.org/pipermail/linux-mtd/2015-September/061617.html).
I would like to test Boris patchset but first I need to recreate the error that his patch is fixing.The error that the patchset is attempting to fix isn't something I have ever encountered before. Currently I am trying to reproduce this issue on a TI K2E evm that uses the davinci nand driver. I flashed the nand's file-system partition with a ubi filesystem and the board is currently set to boot using the file-system on the nand. After about 60 secs I cut the power from the board and boot the board again. What I would expect is that the board will eventually fail to mount the ubi filesystem but currently the board has ran for over 24 hours and powered on and off over 1400 times and its still mounting the file-system perfectly fine.
Quoted 4 lines
Any suggestions on a test case that I can use to force the empty page bit flips error?
The davinci driver seems to support raw accesses, so you can try to apply this patch [1] against the mtd-utils tree (not sure it still applies cleany, but it should work with mtd-utils-1.5.1), and use the nandflipbits tool:
# flash_erase /dev/mtdX <offset> 1 # nandflipbits /dev/mtdX 1@<offset> # nanddump -f /tmp/dump -s <offset> -l <page-size> /dev/mtdXWithout the patch, nanddump should complain about uncorrectable errors, and if you hexdump
/dev/dumpyou should see the bitflip.
If nanddump does not complain after applying my patch, then it means it fixes the "bitflips in erased pages" bug.Quoted 5 lines
Best Regards,
Boris
[1]http://lists.infradead.org/pipermail/linux-mtd/2014-November/056634.html
Hi Boris,
Thanks for the quick reply. I built mtd-utils with your patch and ran the suggested commands on a 4.1 based kernel without your kernel patchset and I didn't see your expected output. The 4.1 based kernel hasn't had any changes to
davinci_nandor nand subsystem that would address this bitflip error.I'm currently going to attempt to run the same test on the latest mainline.
Here is the output I received when I ran your suggested commands on the 4.1 based kernel.Any
root@k2e-evm:~# ./flash_erase /dev/mtd4 4096 1 Erasing 128 Kibyte @ 0 -- 100 % complete root@k2e-evm:~# ./nandflipbits /dev/mtd4 1@4096 root@k2e-evm:~# ./nanddump -f /tmp/dump -s 4096 -l 2048 /dev/mtd4 ECC failed: 0 ECC corrected: 0 Number of bad blocks: 0 Number of bbt blocks: 4 Block size 131072, page size 2048, OOB size 64 root@k2e-evm:~# hexdump /tmp/dump 0000000 fffd ffff ffff ffff ffff ffff ffff ffff 0000010 ffff ffff ffff ffff ffff ffff ffff ffff * 0000800Quoted 2 lines
Any thoughts on why I'm not seeing the expected error?
Oh, actually this behavior is explained in the commit message:
"Currently empty page bit flips are not corrected and report 0 errors."
Quoted 7 lines
Which explains why you're seeing the bitflip in the dump, but nothing reported by the MTD layer.
After applying my patch, the bitflip should simply disappear. You can then try to generate more bitflips than the engine can actually fix (nandflipbits
/dev/mtd41@0:5@0:49@0:98@0:132@0) and check that MTD reports an uncorrectable error.Quoted 7 lines
I verified that I am indeed using ecc4bit mode.
I attempted to run the series of nandflipsbits as you suggested but I get "invalid bit description" error from the utility. Some reason I can only use the nandflipsbits utility for bits 1-7. Anything higher and I get the "Invalid bit description" error.
Indeed. I developed that tool a long time ago and didn't remember that the bit field is encoding the bit offset within a byte. This command should work.
nandflipbits
/dev/mtd41@0:5@0:7@30:3@46:5@47Quoted 4 lines
On the latest master commit I ran nandflipsbits for bits 1-7 at address 0. However, I still didn't receive any error from nanddump although I do see the flip bits from the hexdump
/tmp/dumpoutput.How many of them do you see?
Quoted 21 lines
I then applied your patchset ontop of the latest mainline and ran nandflipsbits for bits 1-7 at address 0. I get the below output which seems to be correct.
root@k2e-evm:~# ./nandflipbits /dev/mtd4 1@0 root@k2e-evm:~# ./nandflipbits /dev/mtd4 2@0 root@k2e-evm:~# ./nandflipbits /dev/mtd4 3@0 root@k2e-evm:~# ./nandflipbits /dev/mtd4 4@0 root@k2e-evm:~# ./nandflipbits /dev/mtd4 5@0 root@k2e-evm:~# ./nandflipbits /dev/mtd4 6@0 root@k2e-evm:~# ./nandflipbits /dev/mtd4 7@0 root@k2e-evm:~# ./nanddump -f /tmp/dump -s 0 -l 2048 /dev/mtd4ECC failed: 1
ECC corrected: 18
Number of bad blocks: 0
Number of bbt blocks: 4
Block size 131072, page size 2048, OOB size 64 Dumping data starting at0x00000000and ending at0x00000800...
ECC: 4 corrected bitflip(s) at offset0x00000000Quoted 4 lines
root@k2e-evm:~# hexdump /tmp/dump 0000000 ffff ffff ffff ffff ffff ffff ffff ffff * 0000800Hm, that's weird. You should get an ECC failure since the ECC strength is only 4bits/512byte and you 8 bits have been flipped.
Quoted 7 lines
One thing that confuses me is if I repeatedly call nanddump I continue to get the "ECC: 4 corrected bitflips" message and the "ECC corrected" count increases by 4 each time. If these bits are being corrected which is apparent from looking at the output of nanddump shouldn't sequential calls indicate that no bitflips needed to be corrected since it was corrected previously?
Quoted 5 lines
Nope, they're corrected on the fly and only in RAM, so each time you read the page, you'll have to fix the bitflips until you erase and rewrite the faulty block.
Hi Boris,
Here is the entire output that should answer your questions.
In the log I am running the following commands:
flash_erase/dev/mtd40 0
./nanddump -f/tmp/dump-s 0 -l 2048/dev/mtd4
hexdump/tmp/dump
./nandflipbits/dev/mtd41@0:5@0:7@30:3@46:5@47
./nanddump -f/tmp/dump-s 0 -l 2048/dev/mtd4
hexdump/tmp/dumpOutput on mainline kernel without bitflip correction patches:
http://pastebin.com/MgBVxALRQuoted 8 lines
Output on mainline kernel with bitflip correction patches:
http://pastebin.com/NdKv0NhVSome reason I'm only getting 1 bit being corrected when using the bitflip correction patches. Comparing my logs from before to now the only difference I'm seeing is that ECC failed is increasing but ECC corrected isn't changing.
Quoted 7 lines
That's what I was expecting: your ECC engine is only fixing 4bits/512byte, which is why the bitflip in erased page correction fail when you have more than 4 bits flipped in a given 512byte block.
Now try to flip only 4 bits instead of 5:
./nandflipbits
/dev/mtd41@0:5@0:7@30:3@46
Here is the output:
root@k2e-evm:~/# ./flash_erase /dev/mtd4 0 1 root@k2e-evm:~/# ./nanddump -f /tmp/dump -s 0 -l 2048 /dev/mtd4 ECC failed: 5 ECC corrected: 0 Number of bad blocks: 0 Number of bbt blocks: 4 Block size 131072, page size 2048, OOB size 64 Dumping data starting at 0x00000000 and ending at 0x00000800... root@k2e-evm:~/# hexdump /tmp/dump 0000000 ffff ffff ffff ffff ffff ffff ffff ffff * 0000800 root@k2e-evm:~/# ./nandflipbits /dev/mtd4 1@0:5@0:7@30:3@46 root@k2e-evm:~/# ./nanddump -f /tmp/dump -s 0 -l 2048 /dev/mtd4 ECC failed: 5 ECC corrected: 0 Number of bad blocks: 0 Number of bbt blocks: 4 Block size 131072, page size 2048, OOB size 64 Dumping data starting at 0x00000000 and ending at 0x00000800... ECC: 4 corrected bitflip(s) at offset 0x00000000 root@k2e-evm:~/# hexdump /tmp/dump 0000000 ffff ffff ffff ffff ffff ffff ffff ffff * 0000800
Running nanddump again shows that 4 bits were corrected. So it seems like things are working as expected.
It seems like patches 2-5 from your patchset weren't pulled in because you and Brian wanted more testing on other platforms. If your going to submit a rev 4 please feel free to CC me so I can test the patches out and add a Tested-by. If not feel free to add a Tested-by for your current rev 3 patchset or if you can bounce those emails my way I can add it myself. Which ever approach you prefer.
Thank you for your help and let me know if there is any further test you would like me to run.
Quoted 3 lines
Quoted 225 lines · On Wed, 30 Dec 2015 12:07:44 -0600 "Franklin S Cooper Jr." <[email protected]>…
Quoted 187 lines · Boris Brezillon
Quoted 176 lines · On Wed, 30 Dec 2015 11:45:38 -0600 "Franklin S Cooper Jr." <[email protected]>…
Quoted 154 lines · Boris Brezillon
Quoted 97 lines · On Wed, 30 Dec 2015 10:40:49 -0600 "Franklin S Cooper Jr." <[email protected]>…
Quoted 89 lines · Boris Brezillon
Quoted 80 lines · On Wed, 30 Dec 2015 09:33:52 -0600 "Franklin S Cooper Jr." <[email protected]>…
Quoted 44 lines · Boris Brezillon
Hi Franklin,
Quoted 21 lines · On Wed, 30 Dec 2015 08:10:20 -0600 "Franklin S Cooper Jr." <[email protected]>…
I am trying to follow up on this discussion from this patch set (https://patchwork.ozlabs.org/patch/539059/) which suggested that Michael instead test the generic bitflips recovery that is implemented by Boris "mtd: nand: properly handle bitflips in erased pages" patchset
(http://lists.infradead.org/pipermail/linux-mtd/2015-September/061617.html).
I would like to test Boris patchset but first I need to recreate the error that his patch is fixing.The error that the patchset is attempting to fix isn't something I have ever encountered before. Currently I am trying to reproduce this issue on a TI K2E evm that uses the davinci nand driver. I flashed the nand's file-system partition with a ubi filesystem and the board is currently set to boot using the file-system on the nand. After about 60 secs I cut the power from the board and boot the board again. What I would expect is that the board will eventually fail to mount the ubi filesystem but currently the board has ran for over 24 hours and powered on and off over 1400 times and its still mounting the file-system perfectly fine.
Quoted 4 lines
Any suggestions on a test case that I can use to force the empty page bit flips error?
The davinci driver seems to support raw accesses, so you can try to apply this patch [1] against the mtd-utils tree (not sure it still applies cleany, but it should work with mtd-utils-1.5.1), and use the nandflipbits tool:
# flash_erase /dev/mtdX <offset> 1 # nandflipbits /dev/mtdX 1@<offset> # nanddump -f /tmp/dump -s <offset> -l <page-size> /dev/mtdXWithout the patch, nanddump should complain about uncorrectable errors, and if you hexdump
/dev/dumpyou should see the bitflip.
If nanddump does not complain after applying my patch, then it means it fixes the "bitflips in erased pages" bug.Quoted 5 lines
Best Regards,
Boris
[1]http://lists.infradead.org/pipermail/linux-mtd/2014-November/056634.html
Hi Boris,
Thanks for the quick reply. I built mtd-utils with your patch and ran the suggested commands on a 4.1 based kernel without your kernel patchset and I didn't see your expected output. The 4.1 based kernel hasn't had any changes to
davinci_nandor nand subsystem that would address this bitflip error.I'm currently going to attempt to run the same test on the latest mainline.
Here is the output I received when I ran your suggested commands on the 4.1 based kernel.Any
root@k2e-evm:~# ./flash_erase /dev/mtd4 4096 1 Erasing 128 Kibyte @ 0 -- 100 % complete root@k2e-evm:~# ./nandflipbits /dev/mtd4 1@4096 root@k2e-evm:~# ./nanddump -f /tmp/dump -s 4096 -l 2048 /dev/mtd4 ECC failed: 0 ECC corrected: 0 Number of bad blocks: 0 Number of bbt blocks: 4 Block size 131072, page size 2048, OOB size 64 root@k2e-evm:~# hexdump /tmp/dump 0000000 fffd ffff ffff ffff ffff ffff ffff ffff 0000010 ffff ffff ffff ffff ffff ffff ffff ffff * 0000800Quoted 2 lines
Any thoughts on why I'm not seeing the expected error?
Oh, actually this behavior is explained in the commit message:
"Currently empty page bit flips are not corrected and report 0 errors."
Quoted 7 lines
Which explains why you're seeing the bitflip in the dump, but nothing reported by the MTD layer.
After applying my patch, the bitflip should simply disappear. You can then try to generate more bitflips than the engine can actually fix (nandflipbits
/dev/mtd41@0:5@0:49@0:98@0:132@0) and check that MTD reports an uncorrectable error.Quoted 7 lines
I verified that I am indeed using ecc4bit mode.
I attempted to run the series of nandflipsbits as you suggested but I get "invalid bit description" error from the utility. Some reason I can only use the nandflipsbits utility for bits 1-7. Anything higher and I get the "Invalid bit description" error.
Indeed. I developed that tool a long time ago and didn't remember that the bit field is encoding the bit offset within a byte. This command should work.
nandflipbits
/dev/mtd41@0:5@0:7@30:3@46:5@47Quoted 4 lines
On the latest master commit I ran nandflipsbits for bits 1-7 at address 0. However, I still didn't receive any error from nanddump although I do see the flip bits from the hexdump
/tmp/dumpoutput.How many of them do you see?
Quoted 21 lines
I then applied your patchset ontop of the latest mainline and ran nandflipsbits for bits 1-7 at address 0. I get the below output which seems to be correct.
root@k2e-evm:~# ./nandflipbits /dev/mtd4 1@0 root@k2e-evm:~# ./nandflipbits /dev/mtd4 2@0 root@k2e-evm:~# ./nandflipbits /dev/mtd4 3@0 root@k2e-evm:~# ./nandflipbits /dev/mtd4 4@0 root@k2e-evm:~# ./nandflipbits /dev/mtd4 5@0 root@k2e-evm:~# ./nandflipbits /dev/mtd4 6@0 root@k2e-evm:~# ./nandflipbits /dev/mtd4 7@0 root@k2e-evm:~# ./nanddump -f /tmp/dump -s 0 -l 2048 /dev/mtd4ECC failed: 1
ECC corrected: 18
Number of bad blocks: 0
Number of bbt blocks: 4
Block size 131072, page size 2048, OOB size 64 Dumping data starting at0x00000000and ending at0x00000800...
ECC: 4 corrected bitflip(s) at offset0x00000000Quoted 4 lines
root@k2e-evm:~# hexdump /tmp/dump 0000000 ffff ffff ffff ffff ffff ffff ffff ffff * 0000800Hm, that's weird. You should get an ECC failure since the ECC strength is only 4bits/512byte and you 8 bits have been flipped.
Quoted 7 lines
One thing that confuses me is if I repeatedly call nanddump I continue to get the "ECC: 4 corrected bitflips" message and the "ECC corrected" count increases by 4 each time. If these bits are being corrected which is apparent from looking at the output of nanddump shouldn't sequential calls indicate that no bitflips needed to be corrected since it was corrected previously?
Quoted 5 lines
Nope, they're corrected on the fly and only in RAM, so each time you read the page, you'll have to fix the bitflips until you erase and rewrite the faulty block.
Hi Boris,
Here is the entire output that should answer your questions.
In the log I am running the following commands:
flash_erase/dev/mtd40 0
./nanddump -f/tmp/dump-s 0 -l 2048/dev/mtd4
hexdump/tmp/dump
./nandflipbits/dev/mtd41@0:5@0:7@30:3@46:5@47
./nanddump -f/tmp/dump-s 0 -l 2048/dev/mtd4
hexdump/tmp/dumpOutput on mainline kernel without bitflip correction patches:
http://pastebin.com/MgBVxALRQuoted 8 lines
Output on mainline kernel with bitflip correction patches:
http://pastebin.com/NdKv0NhVSome reason I'm only getting 1 bit being corrected when using the bitflip correction patches. Comparing my logs from before to now the only difference I'm seeing is that ECC failed is increasing but ECC corrected isn't changing.
Quoted 7 lines
That's what I was expecting: your ECC engine is only fixing 4bits/512byte, which is why the bitflip in erased page correction fail when you have more than 4 bits flipped in a given 512byte block.
Now try to flip only 4 bits instead of 5:
./nandflipbits
/dev/mtd41@0:5@0:7@30:3@46Here is the output:
root@k2e-evm:~/# ./flash_erase /dev/mtd4 0 1 root@k2e-evm:~/# ./nanddump -f /tmp/dump -s 0 -l 2048 /dev/mtd4 ECC failed: 5 ECC corrected: 0 Number of bad blocks: 0 Number of bbt blocks: 4 Block size 131072, page size 2048, OOB size 64 Dumping data starting at 0x00000000 and ending at 0x00000800... root@k2e-evm:~/# hexdump /tmp/dump 0000000 ffff ffff ffff ffff ffff ffff ffff ffff * 0000800 root@k2e-evm:~/# ./nandflipbits /dev/mtd4 1@0:5@0:7@30:3@46 root@k2e-evm:~/# ./nanddump -f /tmp/dump -s 0 -l 2048 /dev/mtd4 ECC failed: 5 ECC corrected: 0 Number of bad blocks: 0 Number of bbt blocks: 4 Block size 131072, page size 2048, OOB size 64 Dumping data starting at 0x00000000 and ending at 0x00000800... ECC: 4 corrected bitflip(s) at offset 0x00000000 root@k2e-evm:~/# hexdump /tmp/dump 0000000 ffff ffff ffff ffff ffff ffff ffff ffff * 0000800
Quoted 7 lines
Running nanddump again shows that 4 bits were corrected. So it seems like things are working as expected.
It seems like patches 2-5 from your patchset weren't pulled in because you and Brian wanted more testing on other platforms. If your going to submit a rev 4 please feel free to CC me so I can test the patches out and add a Tested-by.
Just sent a v4. Feel free to test it and add your
Tested-by/Acked-by/Reviewed-by.
Signature · 3 lines
Boris Brezillon, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com