UBI/UBIFS: dealing with MLC's paired pages
Hello,
I'm currently working on the paired pages problem we have on MLC chips. I remember discussing it with Artem earlier this year when I was preparing my talk for ELC.
I now have some time I can spend working on this problem and I started looking at how this can be solved.
First let's take a look at the UBI layer.
There's one basic thing we have to care about: protecting UBI metadata. There are two kind of metadata:
1/ those stored at the beginning of each erase block (EC and VID
headers)
2/ those stored in specific volumes (layout and fastmap volumes)
We don't have to worry about #2 since those are written using atomic update, and atomic updates are immune to this paired page corruption problem (either the whole write is valid, or none of it is valid).
This leaves problem #1.
For this case, Artem suggested to duplicate the EC header in the VID header so that if page 0 is corrupted we can recover the EC info from page 1 (which will contain both VID and EC info).
Doing that is fine for dealing with EC header corruption, since, AFAIK, none of the NAND vendors are pairing page 0 with page 1. Still remains the VID header corruption problem. Do prevent that we still have several solutions:
a/ skip the page paired with the VID header. This is doable and can be
hidden from UBI users, but it also means that we're loosing another page for metadata (not a negligible overhead)
b/ storing VID info (PEB <-> LEB association) somewhere else. Fastmap
seems the right place to put that in, since fastmap is already storing those information for almost all blocks. Still we would have to modify fastmap a bit to store information about all erase blocks and not only those that are not part of the fastmap pool. Also, updating that in real-time would require using a log approach, instead of the atomic update currently used by fastmap when it runs out of PEBs in it's free PEB pool. Note that the log approach does not have to be applied to all fastmap data (we just need it for the PEB <-> LEB info).
Another off-topic note regarding the suggested log approach: we could also use it to log which PEB was last written/erased, and use that to handle the unstable bits issue.
c/ (also suggested by Artem) delay VID write until we have enough data
to write on the LEB, and thus guarantee that it cannot be corrupted (at least by programming on the paired page ;-)) anymore. Doing that would also require logging data to be written on those LEBs somewhere, not to mention the impact of copying the data twice (once in the log, and then when we have enough data, in the real block).
I don't have any strong opinion about which solution is the best, also I'm maybe missing other aspects or better solutions, so feel free to comment on that and share your thoughts.
That's all for the UBI layer. We will likely need new functions (and new fields in existing structures) to help UBI users deal with MLC
NANDs: for example a field exposing the storage type or a function helping users skip one (or several) blocks to secure the data they have written so far. Anyway, those are things we can discuss after deciding which approach we want to take.
Now, let's talk about the UBIFS layer. We are facing pretty much the same problem in there: we need to protect the data we have already written from time to time.
AFAIU (correct me if I'm wrong), data should be secure when we sync the file system, or commit the UBIFS journal (feel free to correct me if I'm not using the right terms in my explanation).
As explained earlier, the only way to secure data is to skip some pages (those that are paired with the already written ones).
I see two approaches here (there might be more):
1/ do not skip any pages until we are asked to secure the data, and
then skip as much pages as needed to ensure nobody can ever corrupt the data. With this approach you can loose a non negligible amount of space. For example, with this paired pages scheme [1], if you only write page on page 2 and want to secure your data, you'll have to skip pages 3 to 8.
2/ use the NAND in 'SLC mode' (AKA only write on half the pages in a
block). With this solution you always loose half the NAND capacity, but in case of small writes, it's still more efficient than #1. Of course using that solution is not acceptable, because you'll only be able to use half the NAND capacity, but the plan is to use it in conjunction with the GC, so that from time to time UBIFS data chunks/nodes can be put in a single erase block without skipping half the pages.
Note that currently the GC does not work this way: it tries to collect chunks one by one and write them to the journal to free a dirty LEB. What we would need here is a way to collect enough data to fill an entire block and after that release the LEBs that where previously using half the LEB capacity.
Of course both of those solutions implies marking the skipped regions as dirty so that the GC can account for the padded space. For #1 we should probably also use padding nodes to reflect how much space is lost on the media, though I'm not sure how this can be done. For #2, we may have to differentiate 'full' and 'half' LEBs in the LPT.
Anyway, all the above are just some ideas I had or suggestions I got from other people and I wanted to share. I'm open to any new suggestions, because none of the proposed solutions are easy to implement.
Best Regards,
Boris
P.S.: Note that I'm not discussing the WP solution on purpose: I'd like
to have a solution that is completely HW independent.
[1]https://www.olimex.com/Products/Components/IC/H27UBG8T2BTR/resources/H27UBG8T2BTR.pdf,
chapter 6.1. Paired Page Address Information
Signature · 3 lines
Boris Brezillon, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com
Quoted 6 lines · Boris Brezillon
Hello,
I'm currently working on the paired pages problem we have on MLC chips.
I remember discussing it with Artem earlier this year when I was preparing my talk for ELC.
Hi Boris,
excellent summary, very structured. I won't generate any new idea now, just an implementation tactics suggestion.
For an implementation, I'd started with a power cut emulator which emulates paired pages. I'd probably do it in UBI, may be lower. I'd also write a good UBI power-cut test application. And then I'd start playing with various implementation approaches. I'd use the test-driven approach.
Artem.
Hi Artem,
Quoted 8 lines · On Thu, 17 Sep 2015 18:20:39 +0300 Artem Bityutskiy <[email protected]> wrote:
Quoted 6 lines · Boris Brezillon
Hello,
I'm currently working on the paired pages problem we have on MLC chips.
I remember discussing it with Artem earlier this year when I was preparing my talk for ELC.
Quoted 4 lines
Hi Boris,
excellent summary, very structured. I won't generate any new idea now, just an implementation tactics suggestion.
I'm taking that too :-).
Quoted 3 lines
For an implementation, I'd started with a power cut emulator which emulates paired pages. I'd probably do it in UBI, may be lower.
I actually these kind of emulation in nandsim, though I currently generate a kernel Oops when the emulated power-cut occurs, which is not really easy to use (still both paired pages are corrupted on the file used by nandsim to store NAND data).
I'm considering changing the behavior to return -EROFS instead of BUG(), but I'm still not sure upper layers are expecting this error...
I know that using an emulation layer is the only way to go if we want to test the implementation, but I managed to generate those paired pages problem manually by launching a reset in the middle of a page program operation. Even this 'page program interruption' code is still hacky, I think we'll be able to 'easily' validate the solution in real world use cases when it's ready.
Quoted 2 lines
I'd
also write a good UBI power-cut test application.
Not sure what you mean by a UBI power-cut application?
Quoted 2 lines
And then I'd start
playing with various implementation approaches.
Yep, that was the plan, I was hoping you could help me exclude some of them, but I guess testing all of them is the only way to find the best one :-/.
Quoted 2 lines
I'd use the test-driven
approach.
Hm, yep I guess that's the only way to test as much cases as possible, but even with that I doubt I'll be able to think of all the cases that could happen in real world.
Thanks for the feedback.
Best Regards,
Boris
Signature · 3 lines
Boris Brezillon, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com
Richard Weinberger
maintainerBoris,
Am 17.09.2015 um 17:46 schrieb Boris Brezillon:
Quoted 4 lines
Quoted 2 lines
I'd
also write a good UBI power-cut test application.Not sure what you mean by a UBI power-cut application?
UBI has a mechanism so emulate a power-cut. Userspace
can trigger it. I assume Artem meant that we could extend the mechanism to emulate paired page related issues in UBI.
Quoted 7 lines
Quoted 2 lines
And then I'd start
playing with various implementation approaches.Yep, that was the plan, I was hoping you could help me exclude some of them, but I guess testing all of them is the only way to find the best one :-/.
Quoted 6 lines
Quoted 2 lines
I'd use the test-driven
approach.Hm, yep I guess that's the only way to test as much cases as possible, but even with that I doubt I'll be able to think of all the cases that could happen in real world.
Yeah, the crucial point is that we have to emulate paired pages very good. Testing using emulation is nice but we need bare metal tests too. I have one board with MLC NAND, I'll happily wear it do death. B-)
Thanks,
//richard
Dear all,
Il 17/09/2015 18:47, Richard Weinberger ha scritto:
Quoted 16 lines
Boris,
Am 17.09.2015 um 17:46 schrieb Boris Brezillon:
Quoted 3 lines
Quoted 2 lines
I'd
also write a good UBI power-cut test application.Not sure what you mean by a UBI power-cut application?
UBI has a mechanism so emulate a power-cut. Userspace
can trigger it. I assume Artem meant that we could extend the mechanism to emulate paired page related issues in UBI.Quoted 6 lines
Quoted 2 lines
And then I'd start
playing with various implementation approaches.Yep, that was the plan, I was hoping you could help me exclude some of them, but I guess testing all of them is the only way to find the best one :-/.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
I'd use the test-driven
approach.Hm, yep I guess that's the only way to test as much cases as possible, but even with that I doubt I'll be able to think of all the cases that could happen in real world.
Yeah, the crucial point is that we have to emulate paired pages very good. Testing using emulation is nice but we need bare metal tests too. I have one board with MLC NAND, I'll happily wear it do death. B-)
I think Boris has the same board somewhere ;-)
I perfectly understand the reason why using nandsim (and powercut simulator in general) but, AFAIK, the powercut problem is hard to "simulate" because the main issue is when the device see a loss of power in the middle of an operation (page write or block erase)
I think that the best approach for bare metal test is something like the following:
- connect a real powercut device (a simple relais that cut the main power supply driven by a GPIO)
- drive this device inside the MTD code (probably with random delay after issuing a NAND command)
I think that I (as DAVE) can provide this kind of hardware, with an easy plug-in connector on our hostboard (if those are the one that Richard speak about). Please let me know if you're interesting in it, if so I'll forward this request to our hardware guys and give you an official confirm.
While running this kind of test, I would also increase CPU load, to reduce bypass capacitor intrusion (which may lead to wrong result in a generic case)
Kind Regards,
Signature · 3 lines
Andrea SCIAN DAVE Embedded Systems
Hi Andrea,
Quoted 22 lines · On Fri, 18 Sep 2015 09:17:02 +0200 Andrea Scian <[email protected]> wrote:
Dear all,
Il 17/09/2015 18:47, Richard Weinberger ha scritto:
Quoted 18 lines
Boris,
Am 17.09.2015 um 17:46 schrieb Boris Brezillon:
Quoted 3 lines
Quoted 2 lines
I'd
also write a good UBI power-cut test application.Not sure what you mean by a UBI power-cut application?
UBI has a mechanism so emulate a power-cut. Userspace
can trigger it. I assume Artem meant that we could extend the mechanism to emulate paired page related issues in UBI.Quoted 8 lines
Quoted 2 lines
And then I'd start
playing with various implementation approaches.Yep, that was the plan, I was hoping you could help me exclude some of them, but I guess testing all of them is the only way to find the best one :-/.
Quoted 2 lines
I'd use the test-driven
approach.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Hm, yep I guess that's the only way to test as much cases as possible, but even with that I doubt I'll be able to think of all the cases that could happen in real world.
Yeah, the crucial point is that we have to emulate paired pages very good. Testing using emulation is nice but we need bare metal tests too. I have one board with MLC NAND, I'll happily wear it do death. B-)
I think Boris has the same board somewhere ;-)
Yep :-).
Quoted 5 lines
I perfectly understand the reason why using nandsim (and powercut simulator in general) but, AFAIK, the powercut problem is hard to "simulate" because the main issue is when the device see a loss of power in the middle of an operation (page write or block erase)
Well, it can be easily simulated in nandsim. Here is a dirty hack [1] doing that. Of course my implementation is far from perfect, and a lot of things are hardcoded (like the paired pages scheme), but I'm pretty sure it is able to emulate the behavior of a power cut when a specific page in block is accessed.
The other reason we want to simulate it is because we need to test what happens if a corruption happens at specific places: corruption of UBI EC, VID and payload data. This means that we need to be able to simulate a powercut when a specific page (relatively to a block) is accessed.
Quoted 7 lines
I think that the best approach for bare metal test is something like the following:
- connect a real powercut device (a simple relais that cut the main power supply driven by a GPIO)
- drive this device inside the MTD code (probably with random delay after issuing a NAND command)
Hm, it's seems like a complicated infrastructure. All you need to trigger corruptions in paired pages is to interrupt the program operation in the middle, and this can be done by simply sending a reset command while it's taking place (I tested that method, and if I reset the chip after tPROG / 2 it always corrupts both paired pages).
Quoted 10 lines
I think that I (as DAVE) can provide this kind of hardware, with an easy plug-in connector on our hostboard (if those are the one that Richard speak about).
Please let me know if you're interesting in it, if so I'll forward this request to our hardware guys and give you an official confirm.While running this kind of test, I would also increase CPU load, to reduce bypass capacitor intrusion (which may lead to wrong result in a generic case)
Of course, real world tests are welcome, but I don't think we can rely on them while developing the solution.
Anyway, thanks for the proposition.
Best Regards,
Boris
[1]http://code.bulix.org/73xjfn-88945
Signature · 3 lines
Boris Brezillon, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com
Hi Andrea,
Quoted 5 lines · Andrea Scian
I perfectly understand the reason why using nandsim (and powercut simulator in general) but, AFAIK, the powercut problem is hard to "simulate" because the main issue is when the device see a loss of power
in the middle of an operation (page write or block erase)
This is right, and no doubts real power cuts testing is the most important thing.
However, at the beginning, it is very hard to develop if you do not have a quick way to verify your ideas. Simulation is exactly for this - to make the first reliable draft. Once that work, you go to the second stage - real HW testing.
Real HW testing requires a real power cycle, no guarantees power cut happens at the right moment, so you may spend hours emulating just one paired-page case. Compare this to just running a script, and it emulates you 100 paired-page cases during 10 minutes. And you can emulate it easily at the interesting places, not just during the main data writes.
So, to recap, I suggest emulation to make the first draft, and then start heavy real testing to shape the final solution.
Artem.
Quoted 4 lines · Artem Bityutskiy
Real HW testing requires a real power cycle, no guarantees power cut happens at the right moment, so you may spend hours emulating just one
paired-page case.
Sorry, I meant reproducing just one paired-page case.
Signature · 12 lines
Best Regards, Artem Bityutskiy --------------------------------------------------------------------- Intel Finland Oy Registered Address: PL 281, 00181 Helsinki Business Identity Code: 0357606 - 4 Domiciled in Helsinki This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
Boris, Artem,
thanks to both of you for you detailed description. I'll follow this development, for sure I'll learn a lot :-)
Kind Regards,
Signature · 5 lines
Andrea SCIAN DAVE Embedded Systems Il 18/09/2015 09:54, Artem Bityutskiy ha scritto:
Quoted 29 lines
Hi Andrea,
Quoted 5 lines · Andrea Scian
I perfectly understand the reason why using nandsim (and powercut simulator in general) but, AFAIK, the powercut problem is hard to "simulate" because the main issue is when the device see a loss of power
in the middle of an operation (page write or block erase)This is right, and no doubts real power cuts testing is the most important thing.
However, at the beginning, it is very hard to develop if you do not have a quick way to verify your ideas. Simulation is exactly for this - to make the first reliable draft. Once that work, you go to the second stage - real HW testing.
Real HW testing requires a real power cycle, no guarantees power cut happens at the right moment, so you may spend hours emulating just one paired-page case. Compare this to just running a script, and it emulates you 100 paired-page cases during 10 minutes. And you can emulate it easily at the interesting places, not just during the main data writes.
So, to recap, I suggest emulation to make the first draft, and then start heavy real testing to shape the final solution.
Artem.
Hello
Actually, we are working on the paired pages problem too. We have on MLC chips and developed a hardware power control board to simulate the real power loss cycle.
We have many ideas to share with you.
1. emulating the paired-page case
HW: Develop a power control daughter board to control the power supply to NAND, including voltage/ramp control.
SW: Add a module in NAND controller, utilize power board to shut down NAND power when programming paired upper page.
This is easy for us to reproduce paired-page case, in order to guarantee the power loss moment, we add use FPGA logic
to control the power board and detect the status of NAND.
2. EC/VID header corruption
As Boris's excellent summary mentioned, "duplicate the EC header in the VID header", I also believe this is a good solution to protect EC, and we are doing this and testing on MLC.
For VID header, I think skip pages will waste too many capacity, and SLC mode conjugation with GC will make PE cycling higher.
We are developing another solution to store VID info into other page's OOB area in its own block, because UBI does not use OOB and ECC code always not use all OOB area.
We are still developing and testing these solutions to protect EC and VID on MLC.
All the above is my limited work on paired pages, and I am open to any new suggestions and cooperation.
-----Original Message-----
From: linux-mtd [mailto:[email protected]] On Behalf Of Andrea Scian
Sent: Friday, September 18, 2015 5:38 PM
To: [email protected]; Richard Weinberger; Boris Brezillon
Cc: Iwo Mergler; Jeff Lauruhn (jlauruhn); Qi Wang 王起 "(qiwang)"; [email protected]; Brian Norris; David Woodhouse
Subject: Re: UBI/UBIFS: dealing with MLC's paired pages
Boris, Artem,
thanks to both of you for you detailed description. I'll follow this development, for sure I'll learn a lot :-)
Kind Regards,
Signature · 5 lines
Andrea SCIAN DAVE Embedded Systems Il 18/09/2015 09:54, Artem Bityutskiy ha scritto:
Quoted 23 lines
Hi Andrea,
Quoted 4 lines · Andrea Scian
I perfectly understand the reason why using nandsim (and powercut simulator in general) but, AFAIK, the powercut problem is hard to "simulate" because the main issue is when the device see a loss of power in the middle of an operation (page write or block erase)
This is right, and no doubts real power cuts testing is the most important thing.
However, at the beginning, it is very hard to develop if you do not have a quick way to verify your ideas. Simulation is exactly for this
- to make the first reliable draft. Once that work, you go to the second stage - real HW testing.Real HW testing requires a real power cycle, no guarantees power cut happens at the right moment, so you may spend hours emulating just one paired-page case. Compare this to just running a script, and it emulates you 100 paired-page cases during 10 minutes. And you can emulate it easily at the interesting places, not just during the main data writes.
Quoted 5 lines
So, to recap, I suggest emulation to make the first draft, and then start heavy real testing to shape the final solution.
Artem.
______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/
Richard Weinberger
maintainerHi!
Am 24.09.2015 um 03:57 schrieb Karl Zhang 张双锣 (karlzhang):
Quoted 2 lines
We are developing another solution to store VID info into other page's OOB area in its own block, because UBI does not use OOB and ECC code always not use all OOB area.
sorry, I really detest this idea. Not using the OOB area is one of the design principles behind UBI. We've have learned from JFFS and YAFFS that using OOB is problematic.
I'd give it up only if nothing else is applicable.
Thanks,
//richard
Hi Karl,
Quoted 11 lines · On Thu, 24 Sep 2015 01:57:41 +0000 Karl Zhang 张双锣 (karlzhang)…
Hello
Actually, we are working on the paired pages problem too. We have on MLC chips and developed a hardware power control board to simulate the real power loss cycle.
We have many ideas to share with you.
1. emulating the paired-page case
HW: Develop a power control daughter board to control the power supply to NAND, including voltage/ramp control.
SW: Add a module in NAND controller, utilize power board to shut down NAND power when programming paired upper page.
Even the SW solution sounds like a HW solution to me :-). When I say SW emulation, I mean something that doesn't require a reboot or power-off operation.
Quoted 3 lines
This is easy for us to reproduce paired-page case, in order to guarantee the power loss moment, we add use FPGA logic
to control the power board and detect the status of NAND.
That's not so easy for me ;-).
Anyway, as I answered to Andrea, testing on real HW is definitely necessary, but doing it while evaluating the different options is not as efficient as emulating paired pages behavior.
Quoted 11 lines
2. EC/VID header corruption
As Boris's excellent summary mentioned, "duplicate the EC header in the VID header", I also believe this is a good solution to protect EC, and we are doing this and testing on MLC.For VID header, I think skip pages will waste too many capacity, and SLC mode conjugation with GC will make PE cycling higher.
We are developing another solution to store VID info into other page's OOB area in its own block, because UBI does not use OOB and ECC code always not use all OOB area.
Hm, using the OOB area to do that is not such a good idea IMO, and I see at least 2 reasons:
1/ You're supposing that you'll always have enough space to store the VID info (the header is currently taking 64 bytes, even if we could compress it by removing the padding), and this is not necessarily true (particularly with some NAND controllers which are allowing as much space as possible for ECC bytes).
2/ Most of the time ECC bytes are not protected (and even if some controllers are able to protect a few of them, we're not sure to have 64 bytes of protected OOB bytes per page), which means you're writing something that can be corrupted by bitflips. I know that the header is protected by a CRC, but that won't help recovering the data, just let you know when the header is corruption.
To summarize, you'll have to duplicate the VID info in all pages, and you're not guaranteed to have a valid one (even if, the more pages you write the less chance you have to get all of them corrupted)
Regarding the fact that you'll have to at least loose/reserve one page to protect the VID info, I actually had another (crazy?) idea (which I didn't expose in my previous mail, because I thought it would be a major change in the UBI design):
How about completely getting rid of the VID header and relying on the fastmap database (not the current one, but something close to it) + a UBI journal logging the different changes (PEB erase, LEB map, ...). This way we should be able to recover all the information (even the EC ones) even after a power cut. Of course, still remains the problem of the fastmap volume corruption.
Quoted 3 lines
We are still developing and testing these solutions to protect EC and VID on MLC.
Okay, let us know about the results. Also, can you share the code publicly, or is this something you want to keep private until you have a stable version?
Quoted 3 lines
All the above is my limited work on paired pages, and I am open to any new suggestions and cooperation.
Thanks for sharing your ideas.
Best Regards,
Boris
Signature · 3 lines
Boris Brezillon, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com
Hi,
Quoted 19 lines · Boris Brezillon
Quoted 9 lines
2. EC/VID header corruption
As Boris's excellent summary mentioned, "duplicate the EC header in the VID header", I also believe this is a good solution to protect EC, and we are doing this and testing on MLC.For VID header, I think skip pages will waste too many capacity, and SLC mode conjugation with GC will make PE cycling higher.
We are developing another solution to store VID info into other page's OOB area in its own block, because UBI does not use OOB and ECC code always not use all OOB area.
Hm, using the OOB area to do that is not such a good idea IMO, and I see at least 2 reasons:
1/ You're supposing that you'll always have enough space to store the VID info (the header is currently taking 64 bytes, even if we could compress it by removing the padding), and this is not necessarily true (particularly with some NAND controllers which are allowing as much space as possible for ECC bytes).
Quoted 6 lines
2/ Most of the time ECC bytes are not protected (and even if some controllers are able to protect a few of them, we're not sure to have 64 bytes of protected OOB bytes per page), which means you're writing something that can be corrupted by bitflips. I know that the header is protected by a CRC, but that won't help recovering the data, just let you know when the header is corruption.
And 3:
Only NAND provides an OOB area. Other flash devices like parallel or SPI NOR don't. And we definitely want to continue supporting platforms with such flash devices and UBI (and UBIFS).
Thanks,
Stefan
Richard Weinberger
maintainerHi!
Am 17.09.2015 um 15:22 schrieb Boris Brezillon:
Quoted 43 lines
Hello,
I'm currently working on the paired pages problem we have on MLC chips. I remember discussing it with Artem earlier this year when I was preparing my talk for ELC.
I now have some time I can spend working on this problem and I started looking at how this can be solved.
First let's take a look at the UBI layer.
There's one basic thing we have to care about: protecting UBI metadata. There are two kind of metadata:
1/ those stored at the beginning of each erase block (EC and VID
headers)
2/ those stored in specific volumes (layout and fastmap volumes)We don't have to worry about #2 since those are written using atomic update, and atomic updates are immune to this paired page corruption problem (either the whole write is valid, or none of it is valid).
This leaves problem #1.
For this case, Artem suggested to duplicate the EC header in the VID header so that if page 0 is corrupted we can recover the EC info from page 1 (which will contain both VID and EC info).
Doing that is fine for dealing with EC header corruption, since, AFAIK, none of the NAND vendors are pairing page 0 with page 1. Still remains the VID header corruption problem. Do prevent that we still have several solutions:
a/ skip the page paired with the VID header. This is doable and can be
hidden from UBI users, but it also means that we're loosing another page for metadata (not a negligible overhead)
b/ storing VID info (PEB <-> LEB association) somewhere else. Fastmap
seems the right place to put that in, since fastmap is already storing those information for almost all blocks. Still we would have to modify fastmap a bit to store information about all erase blocks and not only those that are not part of the fastmap pool. Also, updating that in real-time would require using a log approach, instead of the atomic update currently used by fastmap when it runs out of PEBs in it's free PEB pool. Note that the log approach does not have to be applied to all fastmap data (we just need it for the PEB <-> LEB info).
Another off-topic note regarding the suggested log approach: we could also use it to log which PEB was last written/erased, and use
Quoted 8 lines
that to handle the unstable bits issue.
c/ (also suggested by Artem) delay VID write until we have enough data
to write on the LEB, and thus guarantee that it cannot be corrupted (at least by programming on the paired page ;-)) anymore. Doing that would also require logging data to be written on those LEBs somewhere, not to mention the impact of copying the data twice (once in the log, and then when we have enough data, in the real block).
Let's start with UBI, as soon it is stable on MLC NAND we can focus on UBIFS.
Solution a) sounds very promising to me as the can be implemented easily and loosing another page for meta data is IMHO acceptable on MLC. Especially as MLC NANDs are anyways bigger and cheaper than SLC.
b) is tricky as fastmap follows the design principle that UBI can fall back to a full scan if the fastmap is corrupted or a self check fails. If the ability to full scan suddenly depends on fastmap it can become messy.
In terms of computer science c) is the most elegant solution but converting UBI to a log based "block layer" is not trivial and as you wrote the write overhead is not negligible.
So, I'd vote for a) and see how well it does in our powercut tests. :-)
Thanks,
//richard
Hi Richard,
Quoted 55 lines · On Tue, 29 Sep 2015 13:19:01 +0200 Richard Weinberger <[email protected]> wrote:
Hi!
Am 17.09.2015 um 15:22 schrieb Boris Brezillon:
Quoted 43 lines
Hello,
I'm currently working on the paired pages problem we have on MLC chips. I remember discussing it with Artem earlier this year when I was preparing my talk for ELC.
I now have some time I can spend working on this problem and I started looking at how this can be solved.
First let's take a look at the UBI layer.
There's one basic thing we have to care about: protecting UBI metadata. There are two kind of metadata:
1/ those stored at the beginning of each erase block (EC and VID
headers)
2/ those stored in specific volumes (layout and fastmap volumes)We don't have to worry about #2 since those are written using atomic update, and atomic updates are immune to this paired page corruption problem (either the whole write is valid, or none of it is valid).
This leaves problem #1.
For this case, Artem suggested to duplicate the EC header in the VID header so that if page 0 is corrupted we can recover the EC info from page 1 (which will contain both VID and EC info).
Doing that is fine for dealing with EC header corruption, since, AFAIK, none of the NAND vendors are pairing page 0 with page 1. Still remains the VID header corruption problem. Do prevent that we still have several solutions:
a/ skip the page paired with the VID header. This is doable and can be
hidden from UBI users, but it also means that we're loosing another page for metadata (not a negligible overhead)
b/ storing VID info (PEB <-> LEB association) somewhere else. Fastmap
seems the right place to put that in, since fastmap is already storing those information for almost all blocks. Still we would have to modify fastmap a bit to store information about all erase blocks and not only those that are not part of the fastmap pool. Also, updating that in real-time would require using a log approach, instead of the atomic update currently used by fastmap when it runs out of PEBs in it's free PEB pool. Note that the log approach does not have to be applied to all fastmap data (we just need it for the PEB <-> LEB info).
Another off-topic note regarding the suggested log approach: we could also use it to log which PEB was last written/erased, and useQuoted 8 lines
that to handle the unstable bits issue.
c/ (also suggested by Artem) delay VID write until we have enough data
to write on the LEB, and thus guarantee that it cannot be corrupted (at least by programming on the paired page ;-)) anymore. Doing that would also require logging data to be written on those LEBs somewhere, not to mention the impact of copying the data twice (once in the log, and then when we have enough data, in the real block).
Quoted 2 lines
Let's start with UBI, as soon it is stable on MLC NAND we can focus on UBIFS.
I wish it was that simple, but the decision we take at the UBI layer will most likely impact the choices we'll have at the UBIFS layer. So yes, focusing on the UBI layer for the implementation sounds sensible, but I think we have to carefully think about the solution we want to test first, and what are the impact on the UBIFS implementation.
Quoted 4 lines
Solution a) sounds very promising to me as the can be implemented easily and loosing another page for meta data is IMHO acceptable on MLC. Especially as MLC NANDs are anyways bigger and cheaper than SLC.
Yes, solution a) is definitely the simplest one (and probably the one I'll try first). Regarding the overhead, we go from 2/number_of_pages_per_block to 4/number_of_pages_per_block (not counting the overhead of internal volumes, since they should be pretty much the same in both cases), so wouldn't say it's negligible even for MLCs. But I agree that having a reliable solution at the cost of more overhead can be a good match for our first implementation.
Quoted 5 lines
b) is tricky as fastmap follows the design principle that UBI can fall back to a full scan if the fastmap is corrupted or a self check fails. If the ability to full scan suddenly depends on fastmap it can become messy.
We are only talking about paired pages corruption here, so I hope both information will not be corrupted at the same time: VID header should be valid unless a power-cut occurred while writing on the page paired with the VID header one, which means fastmap volume should still be valid (unless we are experiencing data-retention issues, which can be true for the SLC case too). And even if the LEB and fastmap information are corrupted, we should be able to reconstruct it and discard the LEB with the corrupted VID header.
Anyway, this approach is way more complicated to implement, and I reserve it as a "going further" topic ;-).
Quoted 4 lines
In terms of computer science c) is the most elegant solution but converting UBI to a log based "block layer" is not trivial and as you wrote the write overhead is not negligible.
Hm, I don't know if it's the most elegant solution (I'm still concerned by the write overhead caused by the extra copy step, though solution b) as some overhead too), but I agree that implementing that one is not trivial.
Quoted 2 lines
So, I'd vote for a) and see how well it does in our powercut tests. :-)
a) it is. I'll focus on that solution first.
Thanks for the advices.
Best Regards,
Boris
Signature · 3 lines
Boris Brezillon, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com
Hi,
Here is a quick status update of my progress and a few questions to UBI/UBIFS experts.
Quoted 52 lines · On Thu, 17 Sep 2015 15:22:40 +0200 Boris Brezillon…
Hello,
I'm currently working on the paired pages problem we have on MLC chips. I remember discussing it with Artem earlier this year when I was preparing my talk for ELC.
I now have some time I can spend working on this problem and I started looking at how this can be solved.
First let's take a look at the UBI layer.
There's one basic thing we have to care about: protecting UBI metadata. There are two kind of metadata:
1/ those stored at the beginning of each erase block (EC and VID
headers)
2/ those stored in specific volumes (layout and fastmap volumes)We don't have to worry about #2 since those are written using atomic update, and atomic updates are immune to this paired page corruption problem (either the whole write is valid, or none of it is valid).
This leaves problem #1.
For this case, Artem suggested to duplicate the EC header in the VID header so that if page 0 is corrupted we can recover the EC info from page 1 (which will contain both VID and EC info).
Doing that is fine for dealing with EC header corruption, since, AFAIK, none of the NAND vendors are pairing page 0 with page 1. Still remains the VID header corruption problem. Do prevent that we still have several solutions:
a/ skip the page paired with the VID header. This is doable and can be
hidden from UBI users, but it also means that we're loosing another page for metadata (not a negligible overhead)
b/ storing VID info (PEB <-> LEB association) somewhere else. Fastmap
seems the right place to put that in, since fastmap is already storing those information for almost all blocks. Still we would have to modify fastmap a bit to store information about all erase blocks and not only those that are not part of the fastmap pool. Also, updating that in real-time would require using a log approach, instead of the atomic update currently used by fastmap when it runs out of PEBs in it's free PEB pool. Note that the log approach does not have to be applied to all fastmap data (we just need it for the PEB <-> LEB info).
Another off-topic note regarding the suggested log approach: we could also use it to log which PEB was last written/erased, and use that to handle the unstable bits issue.
c/ (also suggested by Artem) delay VID write until we have enough data
to write on the LEB, and thus guarantee that it cannot be corrupted (at least by programming on the paired page ;-)) anymore. Doing that would also require logging data to be written on those LEBs somewhere, not to mention the impact of copying the data twice (once in the log, and then when we have enough data, in the real block).
Quoted 3 lines
I don't have any strong opinion about which solution is the best, also I'm maybe missing other aspects or better solutions, so feel free to comment on that and share your thoughts.
I decided to go for the simplest solution (but I can't promise I won't change my mind if this approach appears to be wrong), which is either using a LEB is MLC or SLC mode. In SLC modes, only the first page of each pair is used, which completely address the paired pages problem. For now the SLC mode logic is hidden in the MTD/NAND layers which are providing functions to write/read in SLC mode.
Thanks to this differentiation, UBI is now exposing two kind of LEBs:
- the secure (small) LEBS (those accessed in SLC mode)
- the unsecure (big) LEBS (those accessed in MLC mode)
The secure LEBs are marked as such with a flag in the VID header, which allows tracking secure/unsecure LEBs and controlling the maximum size a UBI user can read/write from/to a LEB.
This approach assume LEB 0 and 1 are never paired together (which AFAICT is always true), because VID is stored on page 1 and we need the secure_flag information to know how to access the LEB (SLC or MLC mode). Of course I expose a few new helpers in the kernel API, and we'll probably have to do it for the ioctl interface too if this approach is validated.
That's all I got for the UBI layer.
Richard, Artem, any feedback so far?
Quoted 38 lines
That's all for the UBI layer. We will likely need new functions (and new fields in existing structures) to help UBI users deal with MLC
NANDs: for example a field exposing the storage type or a function helping users skip one (or several) blocks to secure the data they have written so far. Anyway, those are things we can discuss after deciding which approach we want to take.Now, let's talk about the UBIFS layer. We are facing pretty much the same problem in there: we need to protect the data we have already written from time to time.
AFAIU (correct me if I'm wrong), data should be secure when we sync the file system, or commit the UBIFS journal (feel free to correct me if I'm not using the right terms in my explanation).
As explained earlier, the only way to secure data is to skip some pages (those that are paired with the already written ones).I see two approaches here (there might be more):
1/ do not skip any pages until we are asked to secure the data, and
then skip as much pages as needed to ensure nobody can ever corrupt the data. With this approach you can loose a non negligible amount of space. For example, with this paired pages scheme [1], if you only write page on page 2 and want to secure your data, you'll have to skip pages 3 to 8.
2/ use the NAND in 'SLC mode' (AKA only write on half the pages in a
block). With this solution you always loose half the NAND capacity, but in case of small writes, it's still more efficient than #1. Of course using that solution is not acceptable, because you'll only be able to use half the NAND capacity, but the plan is to use it in conjunction with the GC, so that from time to time UBIFS data chunks/nodes can be put in a single erase block without skipping half the pages.
Note that currently the GC does not work this way: it tries to collect chunks one by one and write them to the journal to free a dirty LEB. What we would need here is a way to collect enough data to fill an entire block and after that release the LEBs that where previously using half the LEB capacity.
Quoted 5 lines
Of course both of those solutions implies marking the skipped regions as dirty so that the GC can account for the padded space. For #1 we should probably also use padding nodes to reflect how much space is lost on the media, though I'm not sure how this can be done. For #2, we may have to differentiate 'full' and 'half' LEBs in the LPT.
If you followed my un/secure LEB approach described above, you probably know that we don't have much solutions for the UBIFS layer.
My idea here is to use a garbage collection mechanism which will consolidate data LEBs (LEBs containing valid data nodes). By default all LEBs are used in secure (SLC) mode, which makes the UBIFS layer reliable. From time to time the consolidation GC will choose a few secure LEBs and move their nodes to an unsecure LEB. The idea is to fill the entire unsecure LEB, so that we never write on it afterwards, thus preventing any paired page corruption. Once this copy is finished we can release/unmap the secure LEBs we have consolidated (after adding a bud node to reference the unsecure LEB of course).
Here are a few details about the implementation I started to develop (questions will come after ;-)).
I added a new category (called LPROPS_FULL) to track the LEBs that are almost full (lp->dirty + lp->free < leb_size / 4), so that we can easily consolidate 2 to 3 full LEBs into a single unsecure LEB. The consolidation is done by filling as much nodes as possible into an unsecure LEB, and after a single pass, this should results in at least one freed LEB freed: the consolidation moves nodes from at least 2 secure LEBs into a single one, so you're freeing 2 LEBs but need to keep one for the next consolidation iteration, hence the single LEB freed.
Now comes the questions to the UBIFS experts:
- should I create a new journal head to do what's described above? AFAICT I can't use the GC head, because the GC can still do it's job in parallel of the consolidation-GC, and the GC LEB might already be filled with some data nodes, right?
I thought about using the data head, but again, it might already point to a partially filled data LEB.
I added a journal head called BIG_DATA_HEAD, but I'm not sure this is acceptable, so let me know what you think about that?
- when should we run the consolidation-GC? After the standard GC pass, when this one didn't make any progress, or should we launch it as soon as we have enough full LEBs to fill an unsecure LEB? The second solution might have a small impact on performances of an empty FS (below the half capacity size), but ITOH, it will scale better when the FS size exceed this limit (no need to run the GC each time we want to write new data).
- I still need to understand the races between TNC and GC, since I'm pretty sure I'll face the same kind of problems with the consolidation-GC. Can someone explains that to me, or should I dig further into the code :-)?
I'm pretty sure I forgot a lot of problematics here, also note that my implementation is not finished yet, so this consolidation-GC concept has not been validated. If you see anything that could defeat this approach, please let me know so that I can adjust my development.
Thanks.
Best Regards,
Boris
Signature · 3 lines
Boris Brezillon, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com
Richard Weinberger
maintainerBoris,
Am 23.10.2015 um 10:14 schrieb Boris Brezillon:
Quoted 61 lines
Quoted 50 lines
I'm currently working on the paired pages problem we have on MLC chips. I remember discussing it with Artem earlier this year when I was preparing my talk for ELC.
I now have some time I can spend working on this problem and I started looking at how this can be solved.
First let's take a look at the UBI layer.
There's one basic thing we have to care about: protecting UBI metadata. There are two kind of metadata:
1/ those stored at the beginning of each erase block (EC and VID
headers)
2/ those stored in specific volumes (layout and fastmap volumes)We don't have to worry about #2 since those are written using atomic update, and atomic updates are immune to this paired page corruption problem (either the whole write is valid, or none of it is valid).
This leaves problem #1.
For this case, Artem suggested to duplicate the EC header in the VID header so that if page 0 is corrupted we can recover the EC info from page 1 (which will contain both VID and EC info).
Doing that is fine for dealing with EC header corruption, since, AFAIK, none of the NAND vendors are pairing page 0 with page 1. Still remains the VID header corruption problem. Do prevent that we still have several solutions:
a/ skip the page paired with the VID header. This is doable and can be
hidden from UBI users, but it also means that we're loosing another page for metadata (not a negligible overhead)
b/ storing VID info (PEB <-> LEB association) somewhere else. Fastmap
seems the right place to put that in, since fastmap is already storing those information for almost all blocks. Still we would have to modify fastmap a bit to store information about all erase blocks and not only those that are not part of the fastmap pool. Also, updating that in real-time would require using a log approach, instead of the atomic update currently used by fastmap when it runs out of PEBs in it's free PEB pool. Note that the log approach does not have to be applied to all fastmap data (we just need it for the PEB <-> LEB info).
Another off-topic note regarding the suggested log approach: we could also use it to log which PEB was last written/erased, and use that to handle the unstable bits issue.
c/ (also suggested by Artem) delay VID write until we have enough data
to write on the LEB, and thus guarantee that it cannot be corrupted (at least by programming on the paired page ;-)) anymore. Doing that would also require logging data to be written on those LEBs somewhere, not to mention the impact of copying the data twice (once in the log, and then when we have enough data, in the real block).Quoted 3 lines
I don't have any strong opinion about which solution is the best, also I'm maybe missing other aspects or better solutions, so feel free to comment on that and share your thoughts.
I decided to go for the simplest solution (but I can't promise I won't change my mind if this approach appears to be wrong), which is either using a LEB is MLC or SLC mode. In SLC modes, only the first page of each pair is used, which completely address the paired pages problem. For now the SLC mode logic is hidden in the MTD/NAND layers which are providing functions to write/read in SLC mode.
Quoted 8 lines
Thanks to this differentiation, UBI is now exposing two kind of LEBs:
- the secure (small) LEBS (those accessed in SLC mode)
- the unsecure (big) LEBS (those accessed in MLC mode)The secure LEBs are marked as such with a flag in the VID header, which allows tracking secure/unsecure LEBs and controlling the maximum size a UBI user can read/write from/to a LEB.
This approach assume LEB 0 and 1 are never paired together (which
You mean page 0 and 1?
Quoted 8 lines
AFAICT is always true), because VID is stored on page 1 and we need the
secure_flaginformation to know how to access the LEB (SLC or MLC mode). Of course I expose a few new helpers in the kernel API, and we'll probably have to do it for the ioctl interface too if this approach is validated.That's all I got for the UBI layer.
Richard, Artem, any feedback so far?
Changing the on-flash format of UBI is a rather big thing. If it needs to be done I'm fine with it but we have to give our best to change it only once. :-)
Quoted 70 lines
Quoted 38 lines
That's all for the UBI layer. We will likely need new functions (and new fields in existing structures) to help UBI users deal with MLC
NANDs: for example a field exposing the storage type or a function helping users skip one (or several) blocks to secure the data they have written so far. Anyway, those are things we can discuss after deciding which approach we want to take.Now, let's talk about the UBIFS layer. We are facing pretty much the same problem in there: we need to protect the data we have already written from time to time.
AFAIU (correct me if I'm wrong), data should be secure when we sync the file system, or commit the UBIFS journal (feel free to correct me if I'm not using the right terms in my explanation).
As explained earlier, the only way to secure data is to skip some pages (those that are paired with the already written ones).I see two approaches here (there might be more):
1/ do not skip any pages until we are asked to secure the data, and
then skip as much pages as needed to ensure nobody can ever corrupt the data. With this approach you can loose a non negligible amount of space. For example, with this paired pages scheme [1], if you only write page on page 2 and want to secure your data, you'll have to skip pages 3 to 8.
2/ use the NAND in 'SLC mode' (AKA only write on half the pages in a
block). With this solution you always loose half the NAND capacity, but in case of small writes, it's still more efficient than #1. Of course using that solution is not acceptable, because you'll only be able to use half the NAND capacity, but the plan is to use it in conjunction with the GC, so that from time to time UBIFS data chunks/nodes can be put in a single erase block without skipping half the pages.
Note that currently the GC does not work this way: it tries to collect chunks one by one and write them to the journal to free a dirty LEB. What we would need here is a way to collect enough data to fill an entire block and after that release the LEBs that where previously using half the LEB capacity.Quoted 5 lines
Of course both of those solutions implies marking the skipped regions as dirty so that the GC can account for the padded space. For #1 we should probably also use padding nodes to reflect how much space is lost on the media, though I'm not sure how this can be done. For #2, we may have to differentiate 'full' and 'half' LEBs in the LPT.
If you followed my un/secure LEB approach described above, you probably know that we don't have much solutions for the UBIFS layer.
My idea here is to use a garbage collection mechanism which will consolidate data LEBs (LEBs containing valid data nodes). By default all LEBs are used in secure (SLC) mode, which makes the UBIFS layer reliable. From time to time the consolidation GC will choose a few secure LEBs and move their nodes to an unsecure LEB. The idea is to fill the entire unsecure LEB, so that we never write on it afterwards, thus preventing any paired page corruption. Once this copy is finished we can release/unmap the secure LEBs we have consolidated (after adding a bud node to reference the unsecure LEB of course).
Here are a few details about the implementation I started to develop (questions will come after ;-)).
I added a new category (calledLPROPS_FULL) to track the LEBs that are almost full (lp->dirty+lp->free<leb_size/ 4), so that we can easily consolidate 2 to 3 full LEBs into a single unsecure LEB. The consolidation is done by filling as much nodes as possible into an unsecure LEB, and after a single pass, this should results in at least one freed LEB freed: the consolidation moves nodes from at least 2 secure LEBs into a single one, so you're freeing 2 LEBs but need to keep one for the next consolidation iteration, hence the single LEB freed.
Quoted 9 lines
Now comes the questions to the UBIFS experts:
- should I create a new journal head to do what's described above? AFAICT I can't use the GC head, because the GC can still do it's job in parallel of the consolidation-GC, and the GC LEB might already be filled with some data nodes, right?
I thought about using the data head, but again, it might already point to a partially filled data LEB.
I added a journal head calledBIG_DATA_HEAD, but I'm not sure this is acceptable, so let me know what you think about that?
I'd vote for a new head.
If it turns out to be similar enough to another head we can still merge it to that head.
Quoted 7 lines
- when should we run the consolidation-GC? After the standard GC pass, when this one didn't make any progress, or should we launch it as soon as we have enough full LEBs to fill an unsecure LEB? The second solution might have a small impact on performances of an empty FS (below the half capacity size), but ITOH, it will scale better when the FS size exceed this limit (no need to run the GC each time we want to write new data).
I'd go for a hybrid approach.
Run the consolidation-GC if standard GC was unable to produce free space and if more than X small LEBs are full.
Quoted 4 lines
- I still need to understand the races between TNC and GC, since I'm pretty sure I'll face the same kind of problems with the consolidation-GC. Can someone explains that to me, or should I dig further into the code :-)?
Not sure if I understand this questions correctly.
What you need for sure is i) a way to find out whether a LEB can be packed and ii) lock it while packing.
Quoted 4 lines
I'm pretty sure I forgot a lot of problematics here, also note that my implementation is not finished yet, so this consolidation-GC concept has not been validated. If you see anything that could defeat this approach, please let me know so that I can adjust my development.
Please share your patches as soon as possible. Just mark them as RFC (really flaky code). I'll happily test them on my MLC boards and review them.
Thanks,
//richard
Hi Richard,
Quoted 68 lines · On Tue, 27 Oct 2015 21:16:28 +0100 Richard Weinberger <[email protected]> wrote:
Boris,
Am 23.10.2015 um 10:14 schrieb Boris Brezillon:
Quoted 65 lines
Quoted 50 lines
I'm currently working on the paired pages problem we have on MLC chips. I remember discussing it with Artem earlier this year when I was preparing my talk for ELC.
I now have some time I can spend working on this problem and I started looking at how this can be solved.
First let's take a look at the UBI layer.
There's one basic thing we have to care about: protecting UBI metadata. There are two kind of metadata:
1/ those stored at the beginning of each erase block (EC and VID
headers)
2/ those stored in specific volumes (layout and fastmap volumes)We don't have to worry about #2 since those are written using atomic update, and atomic updates are immune to this paired page corruption problem (either the whole write is valid, or none of it is valid).
This leaves problem #1.
For this case, Artem suggested to duplicate the EC header in the VID header so that if page 0 is corrupted we can recover the EC info from page 1 (which will contain both VID and EC info).
Doing that is fine for dealing with EC header corruption, since, AFAIK, none of the NAND vendors are pairing page 0 with page 1. Still remains the VID header corruption problem. Do prevent that we still have several solutions:
a/ skip the page paired with the VID header. This is doable and can be
hidden from UBI users, but it also means that we're loosing another page for metadata (not a negligible overhead)
b/ storing VID info (PEB <-> LEB association) somewhere else. Fastmap
seems the right place to put that in, since fastmap is already storing those information for almost all blocks. Still we would have to modify fastmap a bit to store information about all erase blocks and not only those that are not part of the fastmap pool. Also, updating that in real-time would require using a log approach, instead of the atomic update currently used by fastmap when it runs out of PEBs in it's free PEB pool. Note that the log approach does not have to be applied to all fastmap data (we just need it for the PEB <-> LEB info).
Another off-topic note regarding the suggested log approach: we could also use it to log which PEB was last written/erased, and use that to handle the unstable bits issue.
c/ (also suggested by Artem) delay VID write until we have enough data
to write on the LEB, and thus guarantee that it cannot be corrupted (at least by programming on the paired page ;-)) anymore. Doing that would also require logging data to be written on those LEBs somewhere, not to mention the impact of copying the data twice (once in the log, and then when we have enough data, in the real block).Quoted 3 lines
I don't have any strong opinion about which solution is the best, also I'm maybe missing other aspects or better solutions, so feel free to comment on that and share your thoughts.
I decided to go for the simplest solution (but I can't promise I won't change my mind if this approach appears to be wrong), which is either using a LEB is MLC or SLC mode. In SLC modes, only the first page of each pair is used, which completely address the paired pages problem. For now the SLC mode logic is hidden in the MTD/NAND layers which are providing functions to write/read in SLC mode.
Thanks to this differentiation, UBI is now exposing two kind of LEBs:
- the secure (small) LEBS (those accessed in SLC mode)
- the unsecure (big) LEBS (those accessed in MLC mode)
Quoted 6 lines
Quoted 4 lines
The secure LEBs are marked as such with a flag in the VID header, which allows tracking secure/unsecure LEBs and controlling the maximum size a UBI user can read/write from/to a LEB.
This approach assume LEB 0 and 1 are never paired together (whichYou mean page 0 and 1?
Yes.
Quoted 7 lines
Quoted 7 lines
AFAICT is always true), because VID is stored on page 1 and we need the
secure_flaginformation to know how to access the LEB (SLC or MLC mode). Of course I expose a few new helpers in the kernel API, and we'll probably have to do it for the ioctl interface too if this approach is validated.
Quoted 6 lines
Quoted 2 lines
That's all I got for the UBI layer.
Richard, Artem, any feedback so far?Changing the on-flash format of UBI is a rather big thing. If it needs to be done I'm fine with it but we have to give our best to change it only once. :-)
Yes, I know that, and I don't pretend I chose the right solution ;-), any other suggestions to avoid changing the on-flash format?
Note that I only added a new flag, and this flag is only set when you map a LEB in SLC mode, which is not the default case, which in turn means you'll be able to attach to an existing UBI partition. Of course the reverse is not true, once you've started using the secure LEB feature you can't attach this image with an UBI implementation that does not support this feature.
Quoted 81 lines
Quoted 71 lines
Quoted 39 lines
That's all for the UBI layer. We will likely need new functions (and new fields in existing structures) to help UBI users deal with MLC
NANDs: for example a field exposing the storage type or a function helping users skip one (or several) blocks to secure the data they have written so far. Anyway, those are things we can discuss after deciding which approach we want to take.Now, let's talk about the UBIFS layer. We are facing pretty much the same problem in there: we need to protect the data we have already written from time to time.
AFAIU (correct me if I'm wrong), data should be secure when we sync the file system, or commit the UBIFS journal (feel free to correct me if I'm not using the right terms in my explanation).
As explained earlier, the only way to secure data is to skip some pages (those that are paired with the already written ones).I see two approaches here (there might be more):
1/ do not skip any pages until we are asked to secure the data, and
then skip as much pages as needed to ensure nobody can ever corrupt the data. With this approach you can loose a non negligible amount of space. For example, with this paired pages scheme [1], if you only write page on page 2 and want to secure your data, you'll have to skip pages 3 to 8.
2/ use the NAND in 'SLC mode' (AKA only write on half the pages in a
block). With this solution you always loose half the NAND capacity, but in case of small writes, it's still more efficient than #1. Of course using that solution is not acceptable, because you'll only be able to use half the NAND capacity, but the plan is to use it in conjunction with the GC, so that from time to time UBIFS data chunks/nodes can be put in a single erase block without skipping half the pages.
Note that currently the GC does not work this way: it tries to collect chunks one by one and write them to the journal to free a dirty LEB. What we would need here is a way to collect enough data to fill an entire block and after that release the LEBs that where previously using half the LEB capacity.Quoted 5 lines
Of course both of those solutions implies marking the skipped regions as dirty so that the GC can account for the padded space. For #1 we should probably also use padding nodes to reflect how much space is lost on the media, though I'm not sure how this can be done. For #2, we may have to differentiate 'full' and 'half' LEBs in the LPT.
If you followed my un/secure LEB approach described above, you probably know that we don't have much solutions for the UBIFS layer.
My idea here is to use a garbage collection mechanism which will consolidate data LEBs (LEBs containing valid data nodes). By default all LEBs are used in secure (SLC) mode, which makes the UBIFS layer reliable. From time to time the consolidation GC will choose a few secure LEBs and move their nodes to an unsecure LEB. The idea is to fill the entire unsecure LEB, so that we never write on it afterwards, thus preventing any paired page corruption. Once this copy is finished we can release/unmap the secure LEBs we have consolidated (after adding a bud node to reference the unsecure LEB of course).
Here are a few details about the implementation I started to develop (questions will come after ;-)).
I added a new category (calledLPROPS_FULL) to track the LEBs that are almost full (lp->dirty+lp->free<leb_size/ 4), so that we can easily consolidate 2 to 3 full LEBs into a single unsecure LEB. The consolidation is done by filling as much nodes as possible into an unsecure LEB, and after a single pass, this should results in at least one freed LEB freed: the consolidation moves nodes from at least 2 secure LEBs into a single one, so you're freeing 2 LEBs but need to keep one for the next consolidation iteration, hence the single LEB freed.Quoted 9 lines
Now comes the questions to the UBIFS experts:
- should I create a new journal head to do what's described above? AFAICT I can't use the GC head, because the GC can still do it's job in parallel of the consolidation-GC, and the GC LEB might already be filled with some data nodes, right?
I thought about using the data head, but again, it might already point to a partially filled data LEB.
I added a journal head calledBIG_DATA_HEAD, but I'm not sure this is acceptable, so let me know what you think about that?
Quoted 3 lines
I'd vote for a new head.
If it turns out to be similar enough to another head we can still merge it to that head.
Yep, that's what I chose too. Actually, AFAIU, if we want the standard and consolidation GC to work concurrently we need to add a new journal head anyway.
Quoted 9 lines
Quoted 8 lines
- when should we run the consolidation-GC? After the standard GC pass, when this one didn't make any progress, or should we launch it as soon as we have enough full LEBs to fill an unsecure LEB? The second solution might have a small impact on performances of an empty FS (below the half capacity size), but ITOH, it will scale better when the FS size exceed this limit (no need to run the GC each time we want to write new data).
Quoted 3 lines
I'd go for a hybrid approach.
Run the consolidation-GC if standard GC was unable to produce free space and if more than X small LEBs are full.
That's probably the best solution indeed.
Quoted 10 lines
Quoted 5 lines
- I still need to understand the races between TNC and GC, since I'm pretty sure I'll face the same kind of problems with the consolidation-GC. Can someone explains that to me, or should I dig further into the code :-)?
Not sure if I understand this questions correctly.
What you need for sure is i) a way to find out whether a LEB can be packed and ii) lock it while packing.
Hm, locking the whole TNC while we are consolidating several LEBs seems a bit extreme (writing a whole unsecure LEB can take a non-negligible amount of time). I think we can do this consolidation without taking the TNC lock by first writing all the nodes on the new LEB without updating the TNC, and once the unsecure LEB is filled update the TNC in one go (that's what I'm trying to do here [1]).
Quoted 8 lines
Quoted 5 lines
I'm pretty sure I forgot a lot of problematics here, also note that my implementation is not finished yet, so this consolidation-GC concept has not been validated. If you see anything that could defeat this approach, please let me know so that I can adjust my development.
Please share your patches as soon as possible. Just mark them as RFC (really flaky code). I'll happily test them on my MLC boards and review them.
I can share the code (actually it's already on my github repo [2]), but it's not even tested so don't expect to make it work on your board ;-).
Thanks for your first suggestions.
Best Regards,
Boris
[1]https://github.com/bbrezillon/linux-sunxi/blob/23cb262f1c73d24b2a52f41f91fb4c6c1305e8e7/fs/ubifs/gc.c#L739 [2]https://github.com/bbrezillon/linux-sunxi/tree/mlc-wip
Signature · 3 lines
Boris Brezillon, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com
Quoted 99 lines · Boris Brezillon
Hi Richard,
Quoted 68 lines · On Tue, 27 Oct 2015 21:16:28 +0100 Richard Weinberger <[email protected]> wrote:
Boris,
Am 23.10.2015 um 10:14 schrieb Boris Brezillon:
Quoted 65 lines
Quoted 50 lines
I'm currently working on the paired pages problem we have on MLC chips. I remember discussing it with Artem earlier this year when I was preparing my talk for ELC.
I now have some time I can spend working on this problem and I started looking at how this can be solved.
First let's take a look at the UBI layer.
There's one basic thing we have to care about: protecting UBI metadata. There are two kind of metadata:
1/ those stored at the beginning of each erase block (EC and VID
headers)
2/ those stored in specific volumes (layout and fastmap volumes)We don't have to worry about #2 since those are written using atomic update, and atomic updates are immune to this paired page corruption problem (either the whole write is valid, or none of it is valid).
This leaves problem #1.
For this case, Artem suggested to duplicate the EC header in the VID header so that if page 0 is corrupted we can recover the EC info from page 1 (which will contain both VID and EC info).
Doing that is fine for dealing with EC header corruption, since, AFAIK, none of the NAND vendors are pairing page 0 with page 1. Still remains the VID header corruption problem. Do prevent that we still have several solutions:
a/ skip the page paired with the VID header. This is doable and can be
hidden from UBI users, but it also means that we're loosing another page for metadata (not a negligible overhead)
b/ storing VID info (PEB <-> LEB association) somewhere else. Fastmap
seems the right place to put that in, since fastmap is already storing those information for almost all blocks. Still we would have to modify fastmap a bit to store information about all erase blocks and not only those that are not part of the fastmap pool. Also, updating that in real-time would require using a log approach, instead of the atomic update currently used by fastmap when it runs out of PEBs in it's free PEB pool. Note that the log approach does not have to be applied to all fastmap data (we just need it for the PEB <-> LEB info).
Another off-topic note regarding the suggested log approach: we could also use it to log which PEB was last written/erased, and use that to handle the unstable bits issue.
c/ (also suggested by Artem) delay VID write until we have enough data
to write on the LEB, and thus guarantee that it cannot be corrupted (at least by programming on the paired page ;-)) anymore. Doing that would also require logging data to be written on those LEBs somewhere, not to mention the impact of copying the data twice (once in the log, and then when we have enough data, in the real block).Quoted 3 lines
I don't have any strong opinion about which solution is the best, also I'm maybe missing other aspects or better solutions, so feel free to comment on that and share your thoughts.
I decided to go for the simplest solution (but I can't promise I won't change my mind if this approach appears to be wrong), which is either using a LEB is MLC or SLC mode. In SLC modes, only the first page of each pair is used, which completely address the paired pages problem. For now the SLC mode logic is hidden in the MTD/NAND layers which are providing functions to write/read in SLC mode.
Thanks to this differentiation, UBI is now exposing two kind of LEBs:
- the secure (small) LEBS (those accessed in SLC mode)
- the unsecure (big) LEBS (those accessed in MLC mode)Quoted 6 lines
Quoted 4 lines
The secure LEBs are marked as such with a flag in the VID header, which allows tracking secure/unsecure LEBs and controlling the maximum size a UBI user can read/write from/to a LEB.
This approach assume LEB 0 and 1 are never paired together (whichYou mean page 0 and 1?
Yes.
Quoted 7 lines
Quoted 7 lines
AFAICT is always true), because VID is stored on page 1 and we need the
secure_flaginformation to know how to access the LEB (SLC or MLC mode). Of course I expose a few new helpers in the kernel API, and we'll probably have to do it for the ioctl interface too if this approach is validated.Quoted 6 lines
Quoted 2 lines
That's all I got for the UBI layer.
Richard, Artem, any feedback so far?Changing the on-flash format of UBI is a rather big thing. If it needs to be done I'm fine with it but we have to give our best to change it only once. :-)
Yes, I know that, and I don't pretend I chose the right solution ;-), any other suggestions to avoid changing the on-flash format?
Quoted 6 lines
Note that I only added a new flag, and this flag is only set when you map a LEB in SLC mode, which is not the default case, which in turn means you'll be able to attach to an existing UBI partition. Of course the reverse is not true, once you've started using the secure LEB feature you can't attach this image with an UBI implementation that does not support this feature.
Isn't a secure LEB just a plain LEB with half pages unused? Since you only write secure LEBs normally and unsecure LEBs only in garbage collector and you can tell secure LEB by the layout of used pages there isn't really need for special marking AFAICFT
It might be a good idea to not allow mounting a flash which is supposed to be protected against page corruption with a driver that does not support that protection.
On the other hand, if backwards compatibility is desired and the information can be stored without introducing a new flag it might be a good idea to allow that as well.
Thanks
Michal
Quoted 108 lines · On Wed, 28 Oct 2015 11:44:49 +0100 Michal Suchanek <[email protected]> wrote:
Quoted 99 lines · Boris Brezillon
Hi Richard,
Quoted 68 lines · On Tue, 27 Oct 2015 21:16:28 +0100 Richard Weinberger <[email protected]> wrote:
Boris,
Am 23.10.2015 um 10:14 schrieb Boris Brezillon:
Quoted 65 lines
Quoted 50 lines
I'm currently working on the paired pages problem we have on MLC chips. I remember discussing it with Artem earlier this year when I was preparing my talk for ELC.
I now have some time I can spend working on this problem and I started looking at how this can be solved.
First let's take a look at the UBI layer.
There's one basic thing we have to care about: protecting UBI metadata. There are two kind of metadata:
1/ those stored at the beginning of each erase block (EC and VID
headers)
2/ those stored in specific volumes (layout and fastmap volumes)We don't have to worry about #2 since those are written using atomic update, and atomic updates are immune to this paired page corruption problem (either the whole write is valid, or none of it is valid).
This leaves problem #1.
For this case, Artem suggested to duplicate the EC header in the VID header so that if page 0 is corrupted we can recover the EC info from page 1 (which will contain both VID and EC info).
Doing that is fine for dealing with EC header corruption, since, AFAIK, none of the NAND vendors are pairing page 0 with page 1. Still remains the VID header corruption problem. Do prevent that we still have several solutions:
a/ skip the page paired with the VID header. This is doable and can be
hidden from UBI users, but it also means that we're loosing another page for metadata (not a negligible overhead)
b/ storing VID info (PEB <-> LEB association) somewhere else. Fastmap
seems the right place to put that in, since fastmap is already storing those information for almost all blocks. Still we would have to modify fastmap a bit to store information about all erase blocks and not only those that are not part of the fastmap pool. Also, updating that in real-time would require using a log approach, instead of the atomic update currently used by fastmap when it runs out of PEBs in it's free PEB pool. Note that the log approach does not have to be applied to all fastmap data (we just need it for the PEB <-> LEB info).
Another off-topic note regarding the suggested log approach: we could also use it to log which PEB was last written/erased, and use that to handle the unstable bits issue.
c/ (also suggested by Artem) delay VID write until we have enough data
to write on the LEB, and thus guarantee that it cannot be corrupted (at least by programming on the paired page ;-)) anymore. Doing that would also require logging data to be written on those LEBs somewhere, not to mention the impact of copying the data twice (once in the log, and then when we have enough data, in the real block).Quoted 3 lines
I don't have any strong opinion about which solution is the best, also I'm maybe missing other aspects or better solutions, so feel free to comment on that and share your thoughts.
I decided to go for the simplest solution (but I can't promise I won't change my mind if this approach appears to be wrong), which is either using a LEB is MLC or SLC mode. In SLC modes, only the first page of each pair is used, which completely address the paired pages problem. For now the SLC mode logic is hidden in the MTD/NAND layers which are providing functions to write/read in SLC mode.
Thanks to this differentiation, UBI is now exposing two kind of LEBs:
- the secure (small) LEBS (those accessed in SLC mode)
- the unsecure (big) LEBS (those accessed in MLC mode)Quoted 6 lines
Quoted 4 lines
The secure LEBs are marked as such with a flag in the VID header, which allows tracking secure/unsecure LEBs and controlling the maximum size a UBI user can read/write from/to a LEB.
This approach assume LEB 0 and 1 are never paired together (whichYou mean page 0 and 1?
Yes.
Quoted 7 lines
Quoted 7 lines
AFAICT is always true), because VID is stored on page 1 and we need the
secure_flaginformation to know how to access the LEB (SLC or MLC mode). Of course I expose a few new helpers in the kernel API, and we'll probably have to do it for the ioctl interface too if this approach is validated.Quoted 6 lines
Quoted 2 lines
That's all I got for the UBI layer.
Richard, Artem, any feedback so far?Changing the on-flash format of UBI is a rather big thing. If it needs to be done I'm fine with it but we have to give our best to change it only once. :-)
Yes, I know that, and I don't pretend I chose the right solution ;-), any other suggestions to avoid changing the on-flash format?
Quoted 6 lines
Note that I only added a new flag, and this flag is only set when you map a LEB in SLC mode, which is not the default case, which in turn means you'll be able to attach to an existing UBI partition. Of course the reverse is not true, once you've started using the secure LEB feature you can't attach this image with an UBI implementation that does not support this feature.
Quoted 4 lines
Isn't a secure LEB just a plain LEB with half pages unused? Since you only write secure LEBs normally and unsecure LEBs only in garbage collector and you can tell secure LEB by the layout of used pages there isn't really need for special marking AFAICFT
This implies scanning several pages per block to determine which type of LEB is in use, which will drastically increase the attach time. The whole point of this flag is to avoid scanning anything else but the EC and VID headers (or the fastmap LEBs if fastmap is in use).
Quoted 4 lines
It might be a good idea to not allow mounting a flash which is supposed to be protected against page corruption with a driver that does not support that protection.
That can be done by incrementing the UBI_VERSION value...
Quoted 4 lines
On the other hand, if backwards compatibility is desired and the information can be stored without introducing a new flag it might be a good idea to allow that as well.
... but I agree that we should avoid breaking the backward compatibility if that's possible.
Signature · 3 lines
Boris Brezillon, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com
Quoted 116 lines · Boris Brezillon
Quoted 108 lines · On Wed, 28 Oct 2015 11:44:49 +0100 Michal Suchanek <[email protected]> wrote:
Quoted 99 lines · Boris Brezillon
Hi Richard,
Quoted 68 lines · On Tue, 27 Oct 2015 21:16:28 +0100 Richard Weinberger <[email protected]> wrote:
Boris,
Am 23.10.2015 um 10:14 schrieb Boris Brezillon:
Quoted 65 lines
Quoted 50 lines
I'm currently working on the paired pages problem we have on MLC chips. I remember discussing it with Artem earlier this year when I was preparing my talk for ELC.
I now have some time I can spend working on this problem and I started looking at how this can be solved.
First let's take a look at the UBI layer.
There's one basic thing we have to care about: protecting UBI metadata. There are two kind of metadata:
1/ those stored at the beginning of each erase block (EC and VID
headers)
2/ those stored in specific volumes (layout and fastmap volumes)We don't have to worry about #2 since those are written using atomic update, and atomic updates are immune to this paired page corruption problem (either the whole write is valid, or none of it is valid).
This leaves problem #1.
For this case, Artem suggested to duplicate the EC header in the VID header so that if page 0 is corrupted we can recover the EC info from page 1 (which will contain both VID and EC info).
Doing that is fine for dealing with EC header corruption, since, AFAIK, none of the NAND vendors are pairing page 0 with page 1. Still remains the VID header corruption problem. Do prevent that we still have several solutions:
a/ skip the page paired with the VID header. This is doable and can be
hidden from UBI users, but it also means that we're loosing another page for metadata (not a negligible overhead)
b/ storing VID info (PEB <-> LEB association) somewhere else. Fastmap
seems the right place to put that in, since fastmap is already storing those information for almost all blocks. Still we would have to modify fastmap a bit to store information about all erase blocks and not only those that are not part of the fastmap pool. Also, updating that in real-time would require using a log approach, instead of the atomic update currently used by fastmap when it runs out of PEBs in it's free PEB pool. Note that the log approach does not have to be applied to all fastmap data (we just need it for the PEB <-> LEB info).
Another off-topic note regarding the suggested log approach: we could also use it to log which PEB was last written/erased, and use that to handle the unstable bits issue.
c/ (also suggested by Artem) delay VID write until we have enough data
to write on the LEB, and thus guarantee that it cannot be corrupted (at least by programming on the paired page ;-)) anymore. Doing that would also require logging data to be written on those LEBs somewhere, not to mention the impact of copying the data twice (once in the log, and then when we have enough data, in the real block).Quoted 3 lines
I don't have any strong opinion about which solution is the best, also I'm maybe missing other aspects or better solutions, so feel free to comment on that and share your thoughts.
I decided to go for the simplest solution (but I can't promise I won't change my mind if this approach appears to be wrong), which is either using a LEB is MLC or SLC mode. In SLC modes, only the first page of each pair is used, which completely address the paired pages problem. For now the SLC mode logic is hidden in the MTD/NAND layers which are providing functions to write/read in SLC mode.
Thanks to this differentiation, UBI is now exposing two kind of LEBs:
- the secure (small) LEBS (those accessed in SLC mode)
- the unsecure (big) LEBS (those accessed in MLC mode)Quoted 6 lines
Quoted 4 lines
The secure LEBs are marked as such with a flag in the VID header, which allows tracking secure/unsecure LEBs and controlling the maximum size a UBI user can read/write from/to a LEB.
This approach assume LEB 0 and 1 are never paired together (whichYou mean page 0 and 1?
Yes.
Quoted 7 lines
Quoted 7 lines
AFAICT is always true), because VID is stored on page 1 and we need the
secure_flaginformation to know how to access the LEB (SLC or MLC mode). Of course I expose a few new helpers in the kernel API, and we'll probably have to do it for the ioctl interface too if this approach is validated.Quoted 6 lines
Quoted 2 lines
That's all I got for the UBI layer.
Richard, Artem, any feedback so far?Changing the on-flash format of UBI is a rather big thing. If it needs to be done I'm fine with it but we have to give our best to change it only once. :-)
Yes, I know that, and I don't pretend I chose the right solution ;-), any other suggestions to avoid changing the on-flash format?
Quoted 6 lines
Note that I only added a new flag, and this flag is only set when you map a LEB in SLC mode, which is not the default case, which in turn means you'll be able to attach to an existing UBI partition. Of course the reverse is not true, once you've started using the secure LEB feature you can't attach this image with an UBI implementation that does not support this feature.
Quoted 4 lines
Isn't a secure LEB just a plain LEB with half pages unused? Since you only write secure LEBs normally and unsecure LEBs only in garbage collector and you can tell secure LEB by the layout of used pages there isn't really need for special marking AFAICFT
Quoted 4 lines
This implies scanning several pages per block to determine which type of LEB is in use, which will drastically increase the attach time. The whole point of this flag is to avoid scanning anything else but the EC and VID headers (or the fastmap LEBs if fastmap is in use).
Why do you need to scan anything more than you would normally?
You assume that any blocks already written are written correctly and you will write any new blocks securely and perform garbage collection to condense blocks which have unused pages that cannot be written securely or were already used and have stale data. The current data format supposedly already allows determining which pages are in use and which not without extra scanning so there is no more work to be done. The only issue I see is that blocks which are writable by current driver cannot be written securely in some cases. You have to deal with that while upgrading a filesystem from old format anyway so not changing the format will just use the upgrade code path more.
Thanks
Michal
Quoted 7 lines · Boris Brezillon
I decided to go for the simplest solution (but I can't promise I won't
change my mind if this approach appears to be wrong), which is either using a LEB is MLC or SLC mode. In SLC modes, only the first page of each pair is used, which completely address the paired pages problem. For now the SLC mode logic is hidden in the MTD/NAND layers which are providing functions to write/read in SLC mode.
Most of the writes go through the journalling subsystem.
There are some non-journal writes, related to internal meta-date management, like from other subsystems: log, the master node, LPT, index, GC.
In case of journal subsystem, in MLC mode you just skip pages every time the "flush write-buffer" API call is used.
In LPT subsystem, you invent a custom solution, skip pages as needed.
In master - probably nothing needs to be done, since we have 2 copies.
Index, GC - data also goes via journal, so the journal subsystem solution will probably cover it.
Quoted 3 lines
Thanks to this differentiation, UBI is now exposing two kind of LEBs:
- the secure (small) LEBS (those accessed in SLC mode)
- the unsecure (big) LEBS (those accessed in MLC mode)
Is this really necessary? Feels like a bit of over-complication to the UBI layer.
Can UBI care about itself WRT MLC safeness, and let UBIFS care about itself?
Hi Artem,
Don't take the following answer as a try to teach you how UBI/UBIFS work or should work with MLC NANDs. I still listen to your suggestions, but when I had a look at how this "skip pages on demand" approach could be implemented I realized it was not so simple.
Also, if you don't mind I'd like to finish my consolidation-GC implementation before trying a new approach, which don't mean I won't consider the "skip pages on demand" one.
Quoted 18 lines · On Wed, 28 Oct 2015 14:24:45 +0200 Artem Bityutskiy <[email protected]> wrote:
Quoted 7 lines · Boris Brezillon
I decided to go for the simplest solution (but I can't promise I won't
change my mind if this approach appears to be wrong), which is either using a LEB is MLC or SLC mode. In SLC modes, only the first page of each pair is used, which completely address the paired pages problem. For now the SLC mode logic is hidden in the MTD/NAND layers which are providing functions to write/read in SLC mode.Most of the writes go through the journalling subsystem.
There are some non-journal writes, related to internal meta-date management, like from other subsystems: log, the master node, LPT, index, GC.
In case of journal subsystem, in MLC mode you just skip pages every time the "flush write-buffer" API call is used.
Quoted 6 lines
In LPT subsystem, you invent a custom solution, skip pages as needed.
In master - probably nothing needs to be done, since we have 2 copies.
Index, GC - data also goes via journal, so the journal subsystem solution will probably cover it.
For the general concept I agree that it should probably work, but here are my concerns (maybe you'll prove me wrong ;-)):
1/ will you ever be able to use a full LEB without skipping any pages? I mean, when use the "skip pages on demand" you can easily have more than half the page in your LEB skipped, because when you write only on one page, you'll have to skip between 3 to 8 pages (it depends on the pairing scheme). I'll try to run gather some statistics to see how often wbuf are synced to see if that's a real problem.
The consolidation approach has the advantage of being able to consolidate existing LEBs to completely fill them, but the consolidation stuff could probably work with the "skip pages on demand".
2/ skipping pages on demand is not as easy as only writing on lower pages of each pair. As you might know, when skipping pages to secure your data, you'll also have to skip some lower pages so that you end up with an offset to a memory region that can be contiguously written to, and when you skip those lower pages, you have to write on it, because NAND chips require that the lower page of each pair be programmed before the higher one (ignoring this will just render some pages unreliable).
3/ UBIFS is really picky when it comes to corrupted nodes detection, and there are a few cases where it refuses to mount the FS when a corrupted node is detected. One of this case is when the corrupted page (filled with one or several nodes) is filled with non-ff data, which is likely to happen with MLC NANDs (paired pages are not contiguous). We discussed about relaxing this policy a few weeks ago, but what should we do when such a corruption is detected? Drop all nodes with a sequence higher or equal to the last valid node on the LEB?
Note that with the consolidation-GC approach we don't have this problem because the consolidate LEB is added to journal after it has been completely filled with data, and marked as full (->free = 0) so that nobody can reclaim it to write data on it.
Quoted 8 lines
Quoted 5 lines
Thanks to this differentiation, UBI is now exposing two kind of LEBs:
- the secure (small) LEBS (those accessed in SLC mode)
- the unsecure (big) LEBS (those accessed in MLC mode)Is this really necessary? Feels like a bit of over-complication to the UBI layer.
Hm, it's actually not so complicated: SLC mode is implemented by the NAND layer and UBI is just using MTD functions to access the NAND in SLC mode. I'm more concerned by the on-flash format changes problem raised by Richard.
Quoted 4 lines
Can UBI care about itself WRT MLC safeness, and let UBIFS care about itself?
Sorry but I don't agree here. By exposing the secure LEB concept, UBI does not specifically care about UBIFS, it just provides a way for all UBI users to address the problem brought by paired pages in a generic way.
Maybe the secure LEB approach is wrong, but in the end UBI will expose other functions to handle those paired pages problems (ubi_secure_data() to skip pages for example), and this layering (NAND/MTD/UBI/UBIFS) is IMO the only sane way to let each layer handle what it's supposed to handle and let the upper layers use the new features to mitigate the problems.
So, no matter which solution is chosen, it will impact the UBI, MTD, and NAND layers.
Best Regards,
Boris
Signature · 3 lines
Boris Brezillon, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com
Quoted 10 lines · On Fri, 30 Oct 2015 09:15:21 +0100 Boris Brezillon…
2/ skipping pages on demand is not as easy as only writing on lower pages of each pair. As you might know, when skipping pages to secure your data, you'll also have to skip some lower pages so that you end up with an offset to a memory region that can be contiguously written to, and when you skip those lower pages, you have to write on it, because NAND chips require that the lower page of each pair be programmed before the higher one (ignoring this will just render some pages unreliable).
Quoted 4 lines
3/ UBIFS is really picky when it comes to corrupted nodes detection, and there are a few cases where it refuses to mount the FS when a corrupted node is detected. One of this case is when the corrupted page (filled with one or several nodes) is filled with non-ff data,
I meant, "Once of this case is when the corrupted page is followed by a page filled with non-ff data"
Quoted 9 lines
which is likely to happen with MLC NANDs (paired pages are not contiguous). We discussed about relaxing this policy a few weeks ago, but what should we do when such a corruption is detected? Drop all nodes with a sequence higher or equal to the last valid node on the LEB?
Note that with the consolidation-GC approach we don't have this problem because the consolidate LEB is added to journal after it has been completely filled with data, and marked as full (->free = 0) so that nobody can reclaim it to write data on it.
Signature · 3 lines
Boris Brezillon, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com
Quoted 11 lines
Hi Artem,
Don't take the following answer as a try to teach you how UBI/UBIFS work or should work with MLC NANDs. I still listen to your suggestions, but when I had a look at how this "skip pages on demand" approach could be implemented I realized it was not so simple.
Also, if you don't mind I'd like to finish my consolidation-GC implementation before trying a new approach, which don't mean I won't consider the "skip pages on demand" one.
"Skip page" is really a tough solution to code in UBI. Generally, I know that skip page implement in FTL layer to solve MLC paired page issue, that is easy to code.
Is it necessary to implement in Linux? I don't know.
Quoted 100 lines
Quoted 18 lines · On Wed, 28 Oct 2015 14:24:45 +0200 Artem Bityutskiy <[email protected]> wrote:
Quoted 7 lines · Boris Brezillon
I decided to go for the simplest solution (but I can't promise I won't change my mind if this approach appears to be wrong), which is either using a LEB is MLC or SLC mode. In SLC modes, only the first page of each pair is used, which completely address the paired pages problem.
For now the SLC mode logic is hidden in the MTD/NAND layers which are providing functions to write/read in SLC mode.Most of the writes go through the journalling subsystem.
There are some non-journal writes, related to internal meta-date management, like from other subsystems: log, the master node, LPT, index, GC.
In case of journal subsystem, in MLC mode you just skip pages every time the "flush write-buffer" API call is used.
Quoted 6 lines
In LPT subsystem, you invent a custom solution, skip pages as needed.
In master - probably nothing needs to be done, since we have 2 copies.
Index, GC - data also goes via journal, so the journal subsystem solution will probably cover it.
For the general concept I agree that it should probably work, but here are my concerns (maybe you'll prove me wrong ;-)):
1/ will you ever be able to use a full LEB without skipping any pages? I mean, when use the "skip pages on demand" you can easily have more than half the page in your LEB skipped, because when you write only on one page, you'll have to skip between 3 to 8 pages (it depends on the pairing scheme). I'll try to run gather some statistics to see how often wbuf are synced to see if that's a real problem.
The consolidation approach has the advantage of being able to consolidate existing LEBs to completely fill them, but the consolidation stuff could probably work with the "skip pages on demand".2/ skipping pages on demand is not as easy as only writing on lower pages of each pair. As you might know, when skipping pages to secure your data, you'll also have to skip some lower pages so that you end up with an offset to a memory region that can be contiguously written to, and when you skip those lower pages, you have to write on it, because NAND chips require that the lower page of each pair be programmed before the higher one (ignoring this will just render some pages unreliable).
3/ UBIFS is really picky when it comes to corrupted nodes detection, and there are a few cases where it refuses to mount the FS when a corrupted node is detected. One of this case is when the corrupted page (filled with one or several nodes) is filled with non-ff data, which is likely to happen with MLC NANDs (paired pages are not contiguous). We discussed about relaxing this policy a few weeks ago, but what should we do when such a corruption is detected? Drop all nodes with a sequence higher or equal to the last valid node on the LEB?
Note that with the consolidation-GC approach we don't have this problem because the consolidate LEB is added to journal after it has been completely filled with data, and marked as full (->free = 0) so that nobody can reclaim it to write data on it.Quoted 8 lines
Quoted 5 lines
Thanks to this differentiation, UBI is now exposing two kind of LEBs:
- the secure (small) LEBS (those accessed in SLC mode)
- the unsecure (big) LEBS (those accessed in MLC mode)Is this really necessary? Feels like a bit of over-complication to the UBI layer.
Hm, it's actually not so complicated: SLC mode is implemented by the NAND layer and UBI is just using MTD functions to access the NAND in SLC mode. I'm more concerned by the on-flash format changes problem raised by Richard.
Quoted 4 lines
Can UBI care about itself WRT MLC safeness, and let UBIFS care about itself?
Sorry but I don't agree here. By exposing the secure LEB concept, UBI does not specifically care about UBIFS, it just provides a way for all UBI users to address the problem brought by paired pages in a generic way.
Maybe the secure LEB approach is wrong, but in the end UBI will expose other functions to handle those paired pages problems
(ubi_secure_data()to skip pages for example), and this layering (NAND/MTD/UBI/UBIFS) is IMO the only sane way to let each layer handle what it's supposed to handle and let the upper layers use the new features to mitigate the problems.
So, no matter which solution is chosen, it will impact the UBI, MTD, and NAND layers.Best Regards,
Boris
--
Boris Brezillon, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com
Quoted 8 lines · Boris Brezillon
Hi Artem,
Don't take the following answer as a try to teach you how UBI/UBIFS work
or should work with MLC NANDs. I still listen to your suggestions, but
when I had a look at how this "skip pages on demand" approach could be implemented I realized it was not so simple.
Sure.
Could you verify my understanding please.
You realized that "skip on demand" is not easy, and you suggest that we simply write all the data twice - first time we skip pages, and then we garbage collect everything. At the end, roughly speaking, we trade off half of the IO speed, power, and NAND lifetime.
About secure LEBs - do you suggest UBI exposes 2 different LEB sizes at the same time - secure and unsecure, or you it could be only in one of the modes.
Thanks.
Quoted 10 lines · On Fri, 30 Oct 2015 11:08:10 +0200 Artem Bityutskiy <[email protected]> wrote:
Quoted 8 lines · Boris Brezillon
Hi Artem,
Don't take the following answer as a try to teach you how UBI/UBIFS work
or should work with MLC NANDs. I still listen to your suggestions, but
when I had a look at how this "skip pages on demand" approach could be implemented I realized it was not so simple.
Quoted 8 lines
Sure.
Could you verify my understanding please.
You realized that "skip on demand" is not easy, and you suggest that we simply write all the data twice - first time we skip pages, and then we garbage collect everything. At the end, roughly speaking, we trade off half of the IO speed, power, and NAND lifetime.
That will be pretty much the same with the "skip on demand" approach, because you'll probably loose a lot of space when syncing the wbuf. Remember that you have to skip between 3 and 8 pages, so if your buffers are regularly synced (either manually of by the timer), you'll increase the dirty space in those LEBs, and in the end you'll just rely on the regular GC to collect those partially written LEB. Except that the regular GC will in turn loose some space when syncing its wbuf.
Moreover, the standard GC only takes place when you can't find a free LEB anymore, which will probably happen when you reach something close to half the partition size in case of MLC chips (it may be a bit higher if you managed to occupy more than half of each LEB capacity). This means that your FS will become slower when you reach this limit, though maybe this can be addressed by triggering the GC before we run out of free LEBs.
Quoted 4 lines
About secure LEBs - do you suggest UBI exposes 2 different LEB sizes at the same time - secure and unsecure, or you it could be only in one of the modes.
A given LEB can only be in secure or unsecure mode, but a UBI volume can expose both unsecure and secure LEBs, and those LEBs have different sizes.
The secure/unsecure mode is chosen when mapping the LEB, and the LEB stays in this mode until it's unmapped.
Best Regards,
Boris
Signature · 3 lines
Boris Brezillon, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com
Quoted 20 lines · Boris Brezillon
Quoted 17 lines · On Fri, 30 Oct 2015 11:08:10 +0200 Artem Bityutskiy <[email protected]> wrote:
Quoted 11 lines · Boris Brezillon
Hi Artem,
Don't take the following answer as a try to teach you how UBI/UBIFS
work
or should work with MLC NANDs. I still listen to your suggestions,
but
when I had a look at how this "skip pages on demand" approach could
be implemented I realized it was not so simple.Sure.
Could you verify my understanding please.
Quoted 7 lines
Quoted 7 lines
You realized that "skip on demand" is not easy, and you suggest that we
simply write all the data twice - first time we skip pages, and then we
garbage collect everything. At the end, roughly speaking, we trade off
half of the IO speed, power, and NAND lifetime.
So I guess the answer is generally "yes", right? I just want to be clear about the trade-off.
Quoted 2 lines
That will be pretty much the same with the "skip on demand" approach, because you'll probably loose a lot of space when syncing the wbuf.
Write buffer is designed to optimized space usage. Instead of wasting the rest of the NAND page, we wait for more data to arrive and put it to the same NAND page with the previous piece of data.
This suggests that we do not sync it too often, or at least that the efforts were taken not to do this.
Off the top of my head, we sync the write-buffer (AKA wbuf) in these cases:
1. Journal commit, which happens once in a while, depends on journal size.
2. User-initiated sync, like fsync(), sync(), remount, etc.
3. Write-buffer timer, which fires when there were no writes withing certain interval, like 5 seconds. The time can be tuned.
4. Other situations like the end of GC, etc - these are related to meta -data management.
Now, imagine you writing a lot of data, like uncompressing a big tarball, or compressing, or just backing up your /home. In this situation you have a continuous flow of data from VFS to UBIFS.
UBIFS will keep writing the data to the journal, and there won't be any wbuf syncs. The syncs will happen only on journal commit. So you end up with LEBs full of data and not requiring any GC.
But yes, if we are talking about, say, an idle system, which occasionally writes something, there will be a wbuf sync after every write.
So in the "I need all your capacity" kind of situations where IO speed matters, and there are a lot of data written - we'd be optimal, no double writes.
In the "I am mostly idle" type of situations we'll do double writes.
SIGLUNCH, colleagues waiting, sorry, I guess I wrote enough :-)
Quoted 6 lines
A given LEB can only be in secure or unsecure mode, but a UBI volume can expose both unsecure and secure LEBs, and those LEBs have different
sizes.
The secure/unsecure mode is chosen when mapping the LEB, and the LEB stays in this mode until it's unmapped.
This is not going to be a little value add to UBI, this is going to be a big change in my opinion. If UBIFS ends up using this - may worth the effort. Otherwise, I'd argue that this would need an important customer to be worth the effort.
Quoted 52 lines · Artem Bityutskiy
Quoted 20 lines · Boris Brezillon
Quoted 17 lines · On Fri, 30 Oct 2015 11:08:10 +0200 Artem Bityutskiy <[email protected]> wrote:
Quoted 11 lines · Boris Brezillon
Hi Artem,
Don't take the following answer as a try to teach you how UBI/UBIFS
work
or should work with MLC NANDs. I still listen to your suggestions,
but
when I had a look at how this "skip pages on demand" approach could
be implemented I realized it was not so simple.Sure.
Could you verify my understanding please.
Quoted 7 lines
Quoted 7 lines
You realized that "skip on demand" is not easy, and you suggest that we
simply write all the data twice - first time we skip pages, and then we
garbage collect everything. At the end, roughly speaking, we trade off
half of the IO speed, power, and NAND lifetime.So I guess the answer is generally "yes", right? I just want to be clear about the trade-off.
Quoted 2 lines
That will be pretty much the same with the "skip on demand" approach, because you'll probably loose a lot of space when syncing the wbuf.
Write buffer is designed to optimized space usage. Instead of wasting the rest of the NAND page, we wait for more data to arrive and put it to the same NAND page with the previous piece of data.
This suggests that we do not sync it too often, or at least that the efforts were taken not to do this.
Off the top of my head, we sync the write-buffer (AKA wbuf) in these cases:
1. Journal commit, which happens once in a while, depends on journal size.
2. User-initiated sync, likefsync(),sync(), remount, etc.
3. Write-buffer timer, which fires when there were no writes withing certain interval, like 5 seconds. The time can be tuned.
4. Other situations like the end of GC, etc - these are related to meta -data management.
Quoted 7 lines
Now, imagine you writing a lot of data, like uncompressing a big tarball, or compressing, or just backing up your /home. In this situation you have a continuous flow of data from VFS to UBIFS.
UBIFS will keep writing the data to the journal, and there won't be any wbuf syncs. The syncs will happen only on journal commit. So you end up with LEBs full of data and not requiring any GC.
Actually, since there is no guarantee that the data ever gets written (or error reported in case it cannot) unless you fsync() every file before close() any sane uncompressor, backup program, etc. will fsync() every file written regardless of its size. So if your home has a lot of configuration files and sources this will not be nice continuous stream of data in many cases.
IIRC scp(1) does not fsync() potentially resulting in large amounts of data getting silently lost. For that reason and others it should be renamed to iscp.
Thanks
Michal
Quoted 8 lines · Michal Suchanek
Actually, since there is no guarantee that the data ever gets written (or error reported in case it cannot) unless you
fsync()every file beforeclose()any sane uncompressor, backup program, etc. willfsync()every file written regardless of its size. So if your home has
a lot of configuration files and sources this will not be nice continuous stream of data in many cases.
Quoted 4 lines
IIRC scp(1) does not
fsync()potentially resulting in large amounts of
data getting silently lost. For that reason and others it should be renamed to iscp.
This is true. If you get more or less steady stream of incoming writes without syncs in-between, you do not need to sync the write-buffer, you do not need to skip pages to cover the MLC paired pages. In these situations we do not have to end up with double writing.
If there are no writes for some time, it is good idea to sync. VFS will write-back by timeout, UBIFS will flush write-buffer by timeout.
Quoted 8 lines · Boris Brezillon
Moreover, the standard GC only takes place when you can't find a free LEB anymore, which will probably happen when you reach something close
to half the partition size in case of MLC chips (it may be a bit higher if you managed to occupy more than half of each LEB capacity). This means that your FS will become slower when you reach this limit, though maybe this can be addressed by triggering the GC before we run out of free LEBs.
Right. I'd call it a detail. But the big picture is - if you have to GC all the data you write, you write twice. When exactly you do the second write is a detail - sometimes it is deferred, it is in background etc, sometimes right away - you have to GC older data before being able to write new data.
Now, by no means I am criticizing you or your decisions, you are doing great job. I am more like summarizing and trying to give you some food for thoughts. :-)
Richard Weinberger
maintainerAm 30.10.2015 um 12:43 schrieb Artem Bityutskiy:
Quoted 10 lines · Boris Brezillon
Quoted 8 lines
Moreover, the standard GC only takes place when you can't find a free LEB anymore, which will probably happen when you reach something close
to half the partition size in case of MLC chips (it may be a bit higher if you managed to occupy more than half of each LEB capacity). This means that your FS will become slower when you reach this limit, though maybe this can be addressed by triggering the GC before we run out of free LEBs.
Quoted 5 lines
Right. I'd call it a detail. But the big picture is - if you have to GC all the data you write, you write twice. When exactly you do the second write is a detail - sometimes it is deferred, it is in background etc, sometimes right away - you have to GC older data before being able to write new data.
That is a valid concern.
But to me the idea sounds promising and is worth a try. We will stress test it and figure how much the actual overhead is.
Thanks,
//richard
Quoted 3 lines · Richard Weinberger
That is a valid concern.
But to me the idea sounds promising and is worth a try. We will stress test it and figure how much the actual overhead is.
Well, for me the question of "do we double-write or try to do a better job" is more of a fundamental question, not a concern.
Right now I personally do not share the opinion that doing a better job is hard, and double write is easy. I may be wrong though. So for me it does not look like - "hey, we'll just write twice, it is worth a try" is a good starting point.
And then "hey, we did not try to do a better job, so we write twice, let's have this upstream" is a strong position.
Quoted 2 lines · Artem Bityutskiy
And then "hey, we did not try to do a better job, so we write twice, let's have this upstream" is a strong position.
I meant NOT a strong position here, sorry.
Signature · 12 lines
Best Regards, Artem Bityutskiy --------------------------------------------------------------------- Intel Finland Oy Registered Address: PL 281, 00181 Helsinki Business Identity Code: 0357606 - 4 Domiciled in Helsinki This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
Quoted 10 lines · On Fri, 30 Oct 2015 13:43:15 +0200 Artem Bityutskiy <[email protected]> wrote:
Quoted 8 lines · Boris Brezillon
Moreover, the standard GC only takes place when you can't find a free LEB anymore, which will probably happen when you reach something close
to half the partition size in case of MLC chips (it may be a bit higher if you managed to occupy more than half of each LEB capacity). This means that your FS will become slower when you reach this limit, though maybe this can be addressed by triggering the GC before we run out of free LEBs.
Quoted 5 lines
Right. I'd call it a detail. But the big picture is - if you have to GC all the data you write, you write twice. When exactly you do the second write is a detail - sometimes it is deferred, it is in background etc, sometimes right away - you have to GC older data before being able to write new data.
You're right, but it makes a big difference when all your writes are taking longer because you need to run the GC to retrieve a free LEB, and this is probably what's gonna happen when your FS raises ~1/2 its maximum size. Doing it in background (collecting a few valid nodes on each GC step and letting user operations take place between each of these step) should help mitigating this problem.
Quoted 4 lines
Now, by no means I am criticizing you or your decisions, you are doing great job. I am more like summarizing and trying to give you some food for thoughts. :-)
No problem, I don't take it personally. I actually think arguing on technical stuff is a good way to find the best solution ;-).
Signature · 3 lines
Boris Brezillon, Free Electrons Embedded Linux and Kernel engineering http://free-electrons.com
Quoted 7 lines · Boris Brezillon
You're right, but it makes a big difference when all your writes are taking longer because you need to run the GC to retrieve a free LEB, and
this is probably what's gonna happen when your FS raises ~1/2 its maximum size. Doing it in background (collecting a few valid nodes on each GC step and letting user operations take place between each of these step) should help mitigating this problem.
It makes difference, yes. However, again, the worst case scenario is that whenever I need to write, I have do GC, because I am "punished" by previous writes. The worst case scenario is twice as slow write.
Guaranteed twice as fast wear is the other implication.
Increased power consumption is another one. Not every embedded system will find the "you have to do a lot of job in background" UBIFS feature attractive.
Anyway, could you spend a bit more time trying to provide convincing arguments that doing "skip on demand" is hard, or does not gain anything. You expressed this opinion, but so far it did not look 100% convincing.
Thanks!
Quoted 8 lines · Boris Brezillon
1/ do not skip any pages until we are asked to secure the data, and
then skip as much pages as needed to ensure nobody can ever
corrupt
the data. With this approach you can loose a non negligible amount of space. For example, with this paired pages scheme [1], if you only write page on page 2 and want to secure your data, you'll
have
to skip pages 3 to 8.
This sounds like a the right way to go to me.