MLC support for UBI/UBIFS
Richard Weinberger
maintainerDear MTD folks,
As some of you might have noticed, the last few months Boris and I have been heavily working on MLC support. Our first approaches targeted UBIFS since we thought MLC NAND's most annoying feature, namely the paired pages constraint, can be best addressed there. It turned out that UBIFS is the wrong layer and given the massive changes required to make it work, we had to go back to the drawing board, multiple times.
Finally we came up with an approach which is a) rather simple and b) working. This approach operates on the UBI level and we call it "LEB consolidation", it is also known as "write lower and merge".
The main problem with paired pages is that an interrupted write to a high page will also damage the content of its partner, the low page. Some MLC NANDs offer a safe SLC mode where only low pages are exposed and prevent data loss due to interrupted writes can happen. The obvious down side is that you lose 50% of your NAND's storage capacity.
While SLC mode is clearly not an option for us, the "LEB consolidation" approach makes partial use of it. On MLC, UBI will operate on PEBs in two modes, software emulated SLC and raw mode.
By default, LEBs are written in software emulated SLC mode. Hence, only lower pages are written, such that interrupted writes won't damage already written data. If UBI notices that it has no longer enough writable LEBs, it tries to consolidate two full LEBs. Full means that the last page has been written and therefore no further writes are expected. This works since you can program an erase block only in linear manner and on MLC the number of programs (NOP) is in general 1. This also means that there are no subpages.
LEB consolidation works more or less like UBI's atomic LEB change feature. But instead of atomically exchanging the data of a LEB, it atomically moves the data of two LEBs into another LEB in raw mode.
The pages of the source LEBs are read in SLC mode, on the consolidated LEB both low and high pages are used since raw mode is used.
Consolidated LEBs are read-only by design. Especially as we merge only full LEBs. This also implies that a PEB can host one or two LEBs.
If one LEB of a consolidated LEB pair is unmapped the other LEB is marked as candidate for consolidation again.
If a PEB hosts consolidated LEBs it will carry two successive volume headers right after the erase counter header. Currently the relationship between PEBs and LEBs is 1:1, with MLC it is 1:2. Which means that UBI will expose two times as many LEBs as PEBs. This complicates UBI a bit and userspace interfacing needs some massaging since in many places the number of LEBs and PEBs were considered to be the same, and the two different concepts where kind of interchangeable. With MLC this matters and makes a difference.
As UBI is only allowed to consolidate full LEBs, a user on top of UBI should try to fill LEBs as much as possible before mapping a new LEB. If half of all available LEBs are mapped but not filled UBI will refuse to map more since all available PEBs are full and there is no empty PEB to atomically consolidate two LEBs into. Luckily, UBIFS behaves sane and does not map new LEBs before previously mapped LEBs aren't full.
Adding MLC support in UBI requires changes in the on-flash layout, as now two volume headers can exist. The original plan was just raising UBI version from 1 to 2 but this turned out as almost impossible as UBI userspace tools are not designed for backwards compatibility.
They refuse to work when UBI version is not 1. Instead of doing a flag day release we stay with version 1 (for ever) and add a new feature flag to UBI which indicates which extra features the implementation supports. 0x1 will be "basic UBI version 1" support, and 0x2 "LEB consolidation".
Our implementation is being tested on nandsim (with some hacks to have paired pages) and real MLC NAND from Hynix, Micron and Toshiba.
So far it seems to work and no major issues have been spotted.
Of course it survives UBI tests too.
We'll soon start sending patches. Some patches are already on the list and/or need an update.
For full MLC NAND support patch series for NAND core, UBI and mtd-utils will be sent.
The current (not ready for mainline) code can be found here:git://git.infradead.org/users/rw/ubi-mlc.git wip
Feedback and tests reports are very welcome!
We'd like to thank Next Thing co. for funding this work! :-)
Thanks,
//richard
Hi, Richard
Do you need our Micron 70s and 80s MLC NAND sample?
Current code only enable Hynix MLC NAND, can I send patch about this our MLC NAND paired page distance to you? I already tested them on your codes.
Yours Faithfully
BeanHuo
Cell :+86 183 2156 9399
Quoted 90 lines
-----Original Message-----
From: Richard Weinberger [mailto:[email protected]]
Sent: Thursday, April 28, 2016 10:29 PM
To: [email protected]
Cc: Alexander Kaplan <[email protected]>; Boris Brezillon <[email protected]>; David Gstir <[email protected]>; David Oberhollenzer <[email protected]>; Bean Huo 霍斌斌 (beanhuo) <[email protected]>; Artem Bityutskiy <[email protected]>
Subject: [ANNOUNCE] MLC support for UBI/UBIFSDear MTD folks,
As some of you might have noticed, the last few months Boris and I have been heavily working on MLC support. Our first approaches targeted UBIFS since we thought MLC NAND's most annoying feature, namely the paired pages constraint, can be best addressed there. It turned out that UBIFS is the wrong layer and given the massive changes required to make it work, we had to go back to the drawing board, multiple times.
Finally we came up with an approach which is a) rather simple and b) working. This approach operates on the UBI level and we call it "LEB consolidation", it is also known as "write lower and merge".The main problem with paired pages is that an interrupted write to a high page will also damage the content of its partner, the low page. Some MLC NANDs offer a safe SLC mode where only low pages are exposed and prevent data loss due to interrupted writes can happen. The obvious down side is that you lose 50% of your NAND's storage capacity.
While SLC mode is clearly not an option for us, the "LEB consolidation" approach makes partial use of it. On MLC, UBI will operate on PEBs in two modes, software emulated SLC and raw mode.
By default, LEBs are written in software emulated SLC mode. Hence, only lower pages are written, such that interrupted writes won't damage already written data. If UBI notices that it has no longer enough writable LEBs, it tries to consolidate two full LEBs. Full means that the last page has been written and therefore no further writes are expected. This works since you can program an erase block only in linear manner and on MLC the number of programs (NOP) is in general 1. This also means that there are no subpages. LEB consolidation works more or less like UBI's atomic LEB change feature. But instead of atomically exchanging the data of a LEB, it atomically moves the data of two LEBs into another LEB in raw mode.
The pages of the source LEBs are read in SLC mode, on the consolidated LEB both low and high pages are used since raw mode is used. Consolidated LEBs are read-only by design. Especially as we merge only full LEBs.
This also implies that a PEB can host one or two LEBs.
If one LEB of a consolidated LEB pair is unmapped the other LEB is marked as candidate for consolidation again.
If a PEB hosts consolidated LEBs it will carry two successive volume headers right after the erase counter header. Currently the relationship between PEBs and LEBs is 1:1, with MLC it is 1:2. Which means that UBI will expose two times as many LEBs as PEBs. This complicates UBI a bit and userspace interfacing needs some massaging since in many places the number of LEBs and PEBs were considered to be the same, and the two different concepts where kind of interchangeable.
With MLC this matters and makes a difference.
As UBI is only allowed to consolidate full LEBs, a user on top of UBI should try to fill LEBs as much as possible before mapping a new LEB. If half of all available LEBs are mapped but not filled UBI will refuse to map more since all available PEBs are full and there is no empty PEB to atomically consolidate two LEBs into. Luckily, UBIFS behaves sane and does not map new LEBs before previously mapped LEBs aren't full.Adding MLC support in UBI requires changes in the on-flash layout, as now two volume headers can exist. The original plan was just raising UBI version from 1 to 2 but this turned out as almost impossible as UBI userspace tools are not designed for backwards compatibility.
They refuse to work when UBI version is not 1. Instead of doing a flag day release we stay with version 1 (for ever) and add a new feature flag to UBI which indicates which extra features the implementation supports.0x1will be "basic UBI version 1" support, and0x2"LEB consolidation".Our implementation is being tested on nandsim (with some hacks to have paired pages) and real MLC NAND from Hynix, Micron and Toshiba. So far it seems to work and no major issues have been spotted. Of course it survives UBI tests too.
We'll soon start sending patches. Some patches are already on the list and/or need an update.
For full MLC NAND support patch series for NAND core, UBI and mtd-utils will be sent.The current (not ready for mainline) code can be found here:
git://git.infradead.org/users/rw/ubi-mlc.gitwipFeedback and tests reports are very welcome!
We'd like to thank Next Thing co. for funding this work! :-)
Thanks,
//richard
Richard Weinberger
maintainerBean,
Am 04.05.2016 um 09:40 schrieb Bean Huo 霍斌斌 (beanhuo):
Quoted 4 lines
Hi, Richard
Do you need our Micron 70s and 80s MLC NAND sample?
Current code only enable Hynix MLC NAND, can I send patch about this our MLC NAND paired page distance to you? I already tested them on your codes.
Patches are always welcome. :-)
Does your MLC have a mapping other than dist3 or dist6
as implemented in Boris' "[PATCH 2/4] mtd: nand: implement two pairing scheme" patch?
Thanks,
//richard
Hi, Richard
Yes , different with "dis6 and dis3", but I think this is not a perfect
Solution on paired-page distance detect. Because different vendor and different series NAND with different such paired-page distance function. as a result, this table will be bigger and bigger. I now want to do if we can probe this information from DTS.
Yours Faithfully
BeanHuo
Cell :+86 183 2156 9399
Quoted 28 lines
-----Original Message-----
From: Richard Weinberger [mailto:[email protected]]
Sent: Wednesday, May 04, 2016 4:27 PM
To: Bean Huo 霍斌斌 (beanhuo) <[email protected]>; [email protected]
Cc: Alexander Kaplan <[email protected]>; Boris Brezillon <[email protected]>; David Gstir <[email protected]>; David Oberhollenzer <[email protected]>; Artem Bityutskiy <[email protected]>
Subject: Re: [ANNOUNCE] MLC support for UBI/UBIFSBean,
Am 04.05.2016 um 09:40 schrieb Bean Huo 霍斌斌 (beanhuo):
Quoted 4 lines
Hi, Richard
Do you need our Micron 70s and 80s MLC NAND sample?
Current code only enable Hynix MLC NAND, can I send patch about this our MLC NAND paired page distance to you? I already tested them on yourcodes.
Patches are always welcome. :-)
Does your MLC have a mapping other than dist3 or dist6 as implemented in Boris' "[PATCH 2/4] mtd: nand: implement two pairing scheme" patch?
Thanks,
//richard
Quoted 3 lines · On Wed, 4 May 2016 08:55:12 +0000 Bean Huo 霍斌斌 (beanhuo) <[email protected]>…
Hi, Richard
Yes , different with "dis6 and dis3", but I think this is not a perfect
Solution on paired-page distance detect. Because different vendor and different series NAND with different such paired-page distance function. as a result, this table will be bigger and bigger. I now want to do if we can probe this information from DTS.
Sorry but I'm clearly opposed to that: the NAND infrastructure provides a way to uniquely identify the part number, let's not add extra information in the DT if it's not needed.
Just giving a simple use case where putting NAND chip details in the DT is a bad idea: some board manufacturers choose 2 or 3 equivalent NANDs coming from different vendors (and those NANDs may have different pairing scheme). With your solution this means having one DT per NAND chip, which is not easy to deal with.
For the "the table will keep growing and become unmaintainable" argument, I agree. My plan is to provide a per-vendor ->init() hook so that each vendor can implement a simpler solution to detect/assign the pairing scheme (I'm pretty sure this is correlated to the NAND technology...), or other vendor specific stuff (read-retry, HW SLC mode, ...).
Regarding your initial statement, are you sure your NAND chip does not use one of the already defined scheme. I had a look at several NAND datasheets (including Micron ones), and all of them were using either distance 3 or distance 6.
Can you share more information about those different pairing scheme?
Thanks,
Boris
Signature · 3 lines
Boris Brezillon, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com
Richard Weinberger
maintainerBean,
Am 04.05.2016 um 10:55 schrieb Bean Huo 霍斌斌 (beanhuo):
Quoted 3 lines
Hi, Richard
Yes , different with "dis6 and dis3", but I think this is not a perfect
Solution on paired-page distance detect. Because different vendor and different series NAND with different such paired-page distance function. as a result, this table will be bigger and bigger. I now want to do if we can probe this information from DTS.
Not sure if I understand you correctly.
Is it not possible to describe your pairing scheme in an algorithmic way? Did you check Boris' paired pages patches for MTD?
Please try to implement it.
Having the pairing table in DT is IMHO not a good thing.
Thanks,
//richard
Quoted 23 lines · On Wed, 4 May 2016 11:17:35 +0200 Boris Brezillon…
Quoted 3 lines · On Wed, 4 May 2016 08:55:12 +0000 Bean Huo 霍斌斌 (beanhuo) <[email protected]>…
Hi, Richard
Yes , different with "dis6 and dis3", but I think this is not a perfect
Solution on paired-page distance detect. Because different vendor and different series NAND with different such paired-page distance function. as a result, this table will be bigger and bigger. I now want to do if we can probe this information from DTS.Sorry but I'm clearly opposed to that: the NAND infrastructure provides a way to uniquely identify the part number, let's not add extra information in the DT if it's not needed.
Just giving a simple use case where putting NAND chip details in the DT is a bad idea: some board manufacturers choose 2 or 3 equivalent NANDs coming from different vendors (and those NANDs may have different pairing scheme). With your solution this means having one DT per NAND chip, which is not easy to deal with.For the "the table will keep growing and become unmaintainable" argument, I agree. My plan is to provide a per-vendor ->init() hook so that each vendor can implement a simpler solution to detect/assign the pairing scheme (I'm pretty sure this is correlated to the NAND technology...), or other vendor specific stuff (read-retry, HW SLC mode, ...).
Quoted 5 lines
Regarding your initial statement, are you sure your NAND chip does not use one of the already defined scheme. I had a look at several NAND datasheets (including Micron ones), and all of them were using either distance 3 or distance 6.
Can you share more information about those different pairing scheme?
Okay, I had a closer look at your "driver:mtd:ubi:add new bakvol module in ubi layer" patch and some Micron datasheet, and it seems your L7X NANDs are using the "standard" distance 6 scheme, but L8X are actually using a different scheme.
This being said, I don't see any problem implementing this new scheme (the question is, is it really used by other vendors or not).
Signature · 3 lines
Boris Brezillon, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com