ubi on MLC nand flash
Hi everyone,
I recently started to do serious testing of UBI on the diskonchip G4 MLC nand driver I'm finishing up. I started with the io_basic ubi test in mtd-utils. What I find is that, after a few minutes, enough PEBs are marked as bad to exhaust the reserve PEB pool, UBI switches to r/o mode, and the test fails. The reason is that - on this device at least - bit flips seem to be persistent; i.e., you will get e.g. 1 bit flip every time you read a certain page. Consequently, when the bit flip occurs and the PEB gets scrubbed, the torture test fails because the bit flip reoccurs, and the PEB is marked bad.
I expected that eventually I might have to dig into the "program disturb", "read-disturb" or "paired pages" MLC issues, but the problem seems more fundamental. My general impression is that UBI is too unforgiving for this device. The ecc can correct up to 4 bit flips, so 1 bit flip seems to not be a big deal. I'm new to UBI so this is not a critique or a proposal, I'm just hoping some experts can offer some advice or opinions. The obvious remedy is to set a higher threshold for marking a PEB as bad, say 2 or 3 bit flips.
Thanks,
Mike
Quoted 10 lines · Mike Dunn
Hi everyone,
I recently started to do serious testing of UBI on the diskonchip G4 MLC nand driver I'm finishing up. I started with the
io_basicubi test in mtd-utils. What I find is that, after a few minutes, enough PEBs are marked as bad to exhaust the reserve PEB pool, UBI switches to r/o mode, and the test fails. The reason is that - on this device at least - bit flips seem to be persistent; i.e., you will get e.g. 1 bit flip every time you read a certain page. Consequently, when the bit flip occurs and the PEB gets scrubbed, the torture test fails because the bit flip reoccurs, and the PEB is marked bad.
Hi Mike,
I had the same results on recent (34 nm) SLC devices.
Quoted 7 lines
I expected that eventually I might have to dig into the "program disturb", "read-disturb" or "paired pages" MLC issues, but the problem seems more fundamental. My general impression is that UBI is too unforgiving for this device. The ecc can correct up to 4 bit flips, so 1 bit flip seems to not be a big deal. I'm new to UBI so this is not a critique or a proposal, I'm just hoping some experts can offer some advice or opinions. The obvious remedy is to set a higher threshold for marking a PEB as bad, say 2 or 3 bit flips.
I discussed the matter with a nand manufacturer a while ago; the information I could get (for SLC devices, not MLC) can be summarized as follows:
1. A block should be marked bad if a number of bitflips greater than what ecc is able to correct has been detected after erase/program; or if the operation failed with a status error
2. If the maximum number of correctable bitflips is reached during a read operation, data should be relocated to another block, without marking the block as bad
I could not get definitive information about the handling of persistent bitflips, apart from the fact that they are expected and should not cause a block to be marked as bad (as long as the ecc capability is not exceeded). Most nand datasheets I had in my hands are also vague on the subject; they lack a precise error handling strategy description for multi-bitflip devices.
Point 2 above seems reasonable as long as bitflips are reversible (i.e. cancelled by an erase operation); but what if the maximum number of correctable errors is reached during a read, those errors being caused by persistent bitflips ? Should the block be considered bad (IMHO it should be scrubbed then marked bad), or should data be simply relocated ?
When I asked the latter question to a nand manufacturer, his recommendation was (quoting):
"(...) not to mark the block bad (because the error is correctable), and to keep a copy of critical data in another location as backup" (!).
I suggest the following strategy:
Upon reading, when errors are detected (and corrected by ecc):
- if (nb of errors < ecc capability (*)) then no scrubbing, do nothing
- if (nb of errors == ecc capability (*)) then
- scrub block, then torture it and compute nb of persistent bitflips
- if (nb of persistent errors < ecc capability (*)) then block is OK
- if (nb of persistent errors == ecc capability (*)) then mark block as bad [because a single additional bitflip (e.g. a read disturb) would cause data loss]
(*) In order to improve reliability, thresholds can be used instead of max ecc capability.
I'm interested to hear opinions from mtd users/nand experts on the subject; I know that at least a few of us had to implement ecc thresholds recently. And UBI/mtd should be modified to support this (IIRC Artem was pushing in that direction a while ago).
BR,
--
Ivan
Hi and thanks again, Ivan,
Quoted 38 lines · Ivan Djelic
Quoted 10 lines · Mike Dunn
Hi everyone,
I recently started to do serious testing of UBI on the diskonchip G4 MLC nand driver I'm finishing up. I started with the
io_basicubi test in mtd-utils. What I find is that, after a few minutes, enough PEBs are marked as bad to exhaust the reserve PEB pool, UBI switches to r/o mode, and the test fails. The reason is that - on this device at least - bit flips seem to be persistent; i.e., you will get e.g. 1 bit flip every time you read a certain page. Consequently, when the bit flip occurs and the PEB gets scrubbed, the torture test fails because the bit flip reoccurs, and the PEB is marked bad.Hi Mike,
I had the same results on recent (34 nm) SLC devices.Quoted 7 lines
I expected that eventually I might have to dig into the "program disturb", "read-disturb" or "paired pages" MLC issues, but the problem seems more fundamental. My general impression is that UBI is too unforgiving for this device. The ecc can correct up to 4 bit flips, so 1 bit flip seems to not be a big deal. I'm new to UBI so this is not a critique or a proposal, I'm just hoping some experts can offer some advice or opinions. The obvious remedy is to set a higher threshold for marking a PEB as bad, say 2 or 3 bit flips.
I discussed the matter with a nand manufacturer a while ago; the information I could get (for SLC devices, not MLC) can be summarized as follows:
1. A block should be marked bad if a number of bitflips greater than what ecc is able to correct has been detected after erase/program; or if the operation failed with a status error
2. If the maximum number of correctable bitflips is reached during a read operation, data should be relocated to another block, without marking the block as bad
I could not get definitive information about the handling of persistent bitflips, apart from the fact that they are expected and should not cause a block to be marked as bad (as long as the ecc capability is not exceeded). Most nand datasheets I had in my hands are also vague on the subject; they lack a precise error handling strategy description for multi-bitflip devices.
Quoted 9 lines
Point 2 above seems reasonable as long as bitflips are reversible (i.e. cancelled by an erase operation); but what if the maximum number of correctable errors is reached during a read, those errors being caused by persistent bitflips ? Should the block be considered bad (IMHO it should be scrubbed then marked bad), or should data be simply relocated ?
When I asked the latter question to a nand manufacturer, his recommendation was (quoting):
"(...) not to mark the block bad (because the error is correctable), and to keep a copy of critical data in another location as backup" (!).
Assign each block a reliability rating, and data a criticality rating? UBI and
file I/O would get pretty bureaucratic. :-)
Quoted 11 lines
I suggest the following strategy:
Upon reading, when errors are detected (and corrected by ecc):
- if (nb of errors < ecc capability (*)) then no scrubbing, do nothing
- if (nb of errors == ecc capability (*)) then
- scrub block, then torture it and compute nb of persistent bitflips
- if (nb of persistent errors < ecc capability (*)) then block is OK
- if (nb of persistent errors == ecc capability (*)) then mark block as bad [because a single additional bitflip (e.g. a read disturb) would cause data loss]
Quoted 2 lines
(*) In order to improve reliability, thresholds can be used instead of max ecc capability.
One wrinkle is that the torture test is performed over the entire erase block, not just the page(s) with the correctible error(s). So the biflip stats are cumulative over the entire block, and may not even occur on the same page. The current UBI policy for the torture test is that *any* bitflips on *any* page following the erasure causes the block to be marked bad.
Another complication is that there's currently no way to accurately determine in the UBI code the number of bitflips the read operation caused. Currently the occurrence of bitflips (one or more) is determined by the return code from the mtd subsystem, which has exclusive access to the device during the read operation. Just checking the ecc_stats field in the mtd_info structure could include errors in read operations performed by other processes.
Quoted 6 lines
I'm interested to hear opinions from mtd users/nand experts on the subject; I know that at least a few of us had to implement ecc thresholds recently. And UBI/mtd should be modified to support this (IIRC Artem was pushing in that direction a while ago).
Thanks,
Mike
Quoted 4 lines · Mike Dunn
Hi everyone,
I recently started to do serious testing of UBI on the diskonchip G4 MLC nand driver I'm finishing up.
Sounds promising - we've got another serious mtd community memeber! :-)
Quoted 1 line
I started with the
io_basicubi test in mtd-utils.
Makes sense to exclude UBIFS and test UBI directly first indeed.
Quoted 2 lines
What I find is that, after a few minutes, enough PEBs are marked as bad to exhaust the reserve PEB pool
I guess you can make it larger, the default 1% is just something which was good enough for our super-robust OneNAND flash.
Also, for MLC you probably want a smaller WL threshold, I heard that modern MLCs have ereaseblock liftimes smaller than 10000 erase-cycles. So the default 4096 might be too big.
Quoted 5 lines
, UBI switches to r/o mode, and the test fails. The
reason is that - on this device at least - bit flips seem to be persistent; i.e., you will get e.g. 1 bit flip every time you read a certain page. Consequently, when the bit flip occurs and the PEB gets scrubbed, the torture test fails because the bit flip reoccurs, and the PEB is marked bad.
A quick hack you can do to go further in your investigations without being block by this issue is to hack your driver and make it to just not return -EUCLEAN in case of 1 bit flip or may be even 2. Then you can see ahead what else happens to UBI.
WRT the real solution - I agree with Ivan - see his e-mail, and I'll send some comments on that.
Quoted 7 lines
I expected that eventually I might have to dig into the "program disturb", "read-disturb" or "paired pages" MLC issues, but the problem seems more fundamental. My general impression is that UBI is too unforgiving for this device. The ecc can correct up to 4 bit flips, so 1 bit flip seems to not be a big deal. I'm new to UBI so this is not a critique or a proposal, I'm just hoping some experts can offer some advice or opinions. The obvious remedy is to set a higher threshold for marking a PEB as bad, say 2 or 3 bit flips.
You are right - UBI is too unforgiving. But this should be fixable, it just needs a brave knight to do the job :-)
Artem.
Quoted 21 lines · Mike Dunn
Quoted 11 lines
I suggest the following strategy:
Upon reading, when errors are detected (and corrected by ecc):
- if (nb of errors < ecc capability (*)) then no scrubbing, do nothing
- if (nb of errors == ecc capability (*)) then
- scrub block, then torture it and compute nb of persistent bitflips
- if (nb of persistent errors < ecc capability (*)) then block is OK
- if (nb of persistent errors == ecc capability (*)) then mark block as bad [because a single additional bitflip (e.g. a read disturb) would cause data loss]Quoted 2 lines
(*) In order to improve reliability, thresholds can be used instead of max ecc capability.
One wrinkle is that the torture test is performed over the entire erase block, not just the page(s) with the correctible error(s). So the biflip stats are cumulative over the entire block, and may not even occur on the same page. The current UBI policy for the torture test is that *any* bitflips on *any* page following the erasure causes the block to be marked bad.
Quoted 6 lines
Another complication is that there's currently no way to accurately determine in the UBI code the number of bitflips the read operation caused. Currently the occurrence of bitflips (one or more) is determined by the return code from the mtd subsystem, which has exclusive access to the device during the read operation. Just checking the
ecc_statsfield in themtd_infostructure could include errors in read operations performed by other processes.
What about something like this.
1. MTD knows flash's ECC strength (driver sets it)
2. MTD sets the scrub level = ECC strength by default
3. MTD can expose the scrub level and ECC strength via sysfs and make the scrub level sysfs file writable, so the user can vary it between 1 and ECC strength.
4. MTD just does not report -EUCEAN if the ECC correction order is less than the scrub level.
Then you do not need to change UBI at all.
WRT blank pages, I guess MTD can gain some internal smartness as well - the driver can report to the NAND base that a blank page was read, and the ECC correction order, then NAND base will make the decision about reporting -EUCLEAN and setting the buffer to all 0xFFs.
Also, it sounds like this may require re-working the current MTD interface and turn all these function pointers (mtd->read(), etc) into normal functions (mtd_read()) which will allow inserting additional logic at various levels.
WRT ecc_stats - IMHO, it is useless and rudimentary thing and could be just killed...
Artem.
Hi Artem, thanks for having a look.
Quoted 6 lines · Artem Bityutskiy
Quoted 2 lines · Mike Dunn
What I find is that, after a few minutes, enough PEBs are marked as bad to exhaust the reserve PEB pool
I guess you can make it larger, the default 1% is just something which was good enough for our super-robust OneNAND flash.
This device only has 512 erase blocks (256K each), and typically fewer usable for a filesystem, so I hate to expend more for overhead.
Quoted 3 lines
Also, for MLC you probably want a smaller WL threshold, I heard that modern MLCs have ereaseblock liftimes smaller than 10000 erase-cycles. So the default 4096 might be too big.
Thanks. Did see the comment in ubi/Kconfig.
Quoted 9 lines
Quoted 5 lines
, UBI switches to r/o mode, and the test fails. The
reason is that - on this device at least - bit flips seem to be persistent; i.e., you will get e.g. 1 bit flip every time you read a certain page. Consequently, when the bit flip occurs and the PEB gets scrubbed, the torture test fails because the bit flip reoccurs, and the PEB is marked bad.A quick hack you can do to go further in your investigations without being block by this issue is to hack your driver and make it to just not return
-EUCLEANin case of 1 bit flip or may be even 2. Then you can see ahead what else happens to UBI.
Already done as a temporary work-around. But -EUCLEAN is actually returned from nand_base.c
Mike
Quoted 37 lines · Artem Bityutskiy
Quoted 20 lines · Mike Dunn
Quoted 11 lines
I suggest the following strategy:
Upon reading, when errors are detected (and corrected by ecc):
- if (nb of errors < ecc capability (*)) then no scrubbing, do nothing
- if (nb of errors == ecc capability (*)) then
- scrub block, then torture it and compute nb of persistent bitflips
- if (nb of persistent errors < ecc capability (*)) then block is OK
- if (nb of persistent errors == ecc capability (*)) then mark block as bad [because a single additional bitflip (e.g. a read disturb) would cause data loss]Quoted 2 lines
(*) In order to improve reliability, thresholds can be used instead of max ecc capability.
One wrinkle is that the torture test is performed over the entire erase block, not just the page(s) with the correctible error(s). So the biflip stats are cumulative over the entire block, and may not even occur on the same page. The current UBI policy for the torture test is that *any* bitflips on *any* page following the erasure causes the block to be marked bad.
Quoted 6 lines
Another complication is that there's currently no way to accurately determine in the UBI code the number of bitflips the read operation caused. Currently the occurrence of bitflips (one or more) is determined by the return code from the mtd subsystem, which has exclusive access to the device during the read operation. Just checking the
ecc_statsfield in themtd_infostructure could include errors in read operations performed by other processes.What about something like this.
1. MTD knows flash's ECC strength (driver sets it)
2. MTD sets the scrub level = ECC strength by default
3. MTD can expose the scrub level and ECC strength via sysfs and make the scrub level sysfs file writable, so the user can vary it between 1 and ECC strength.
4. MTD just does not report-EUCEANif the ECC correction order is less than the scrub level.
Quoted 1 line
Then you do not need to change UBI at all.
That sounds reasonable, but the changes seem broadly consequential.
Quoted 4 lines
WRT blank pages, I guess MTD can gain some internal smartness as well - the driver can report to the NAND base that a blank page was read, and the ECC correction order, then NAND base will make the decision about reporting
-EUCLEANand setting the buffer to all 0xFFs.
I haven't yet surveyed the other drivers regarding ecc and blank page reading. I assumed that ecc was disregarded for blank pages, but probably some drivers are more thoughtful about it than I originally was.
Quoted 4 lines
Also, it sounds like this may require re-working the current MTD interface and turn all these function pointers (
mtd->read(), etc) into normal functions (mtd_read()) which will allow inserting additional logic at various levels.
Oofa. What have I gotten myself into? I don't have all those devices on which to test the changes, and I'd hate to break a driver. But you're right. Both mtd and nand interfaces would have to change to provide a mechanism for returning an error count (corrected or uncorrected) to some yet-to-be-implemented mtd infrastructure code. Drivers that don't use the NAND interface currently return -EUCLEAN directly to the higher layer (e.g. UBI). For drivers using the nand interface, nand_base.c handles it.
Quoted 2 lines
WRT
ecc_stats- IMHO, it is useless and rudimentary thing and could be just killed...
Some userspace mtd-utils for nand currently use it, though.
I'm able to at least look into making these changes if you want to go ahead. My motivation is to get a robust ubifs on my diskonchip G4.
Thanks,
Mike
Quoted 2 lines · Mike Dunn
Oofa. What have I gotten myself into? I don't have all those devices on which to test the changes, and I'd hate to break a driver.
Well, you do not have to test on all devices. If you do small incremental reviewable changes, CC the right people, and test on nandsim and mtdram plus your device, this should be good enough. Those who are interested will review/test. Those who do not care - will fix their drivers after the breakage. Of course the goal is to not break anything, but my point is that myriads of divers which is impossible to test should not prevent the progress :-)
Quoted 1 line
Some userspace mtd-utils for nand currently use it, though.
Well, not sure if the usage makes much sense with modern flashes. Anyway, if you do not need to change this - don't touch it :-)
Quoted 2 lines
I'm able to at least look into making these changes if you want to go ahead. My motivation is to get a robust ubifs on my diskonchip G4.
Sounds good, send nice incremental patches :-)
Signature · 2 lines
Best Regards, Artem Bityutskiy
Quoted 3 lines · Artem Bityutskiy
Sounds good, send nice incremental patches :-)
Which git tree should I use?
Thanks,
Mike
Quoted 7 lines · Mike Dunn
Quoted 3 lines · Artem Bityutskiy
Sounds good, send nice incremental patches :-)
Which git tree should I use?
I guess the l2-mtd-2.6.git tree since it contains the latest MTD work. Although it is currently based on 3.2-rc1 which may be a bit unstable. If this does not work, we can create something based on 3.1 release plus the latest MTD patches for you, I suppose.
Signature · 2 lines
Best Regards, Artem Bityutskiy