UBIFS: recovery of master node
Dear all,
I'm trying to understand how UBIFS recovers master node from a corrupted flash (e.g. after power cut during one of the two of master node or due flash corruption itself).
IIUC (please correct me if I'm wrong) UBIFS store two copies of master node, in LEB1 and LEB2 (LEB0 is reserved for superblock), ref. http://www.linux-mtd.infradead.org/doc/ubifs_whitepaper.pdf.
Inside ubifs_recover_master_node() try to read first LEB1 and, only in case of get_master_node() does NOT return an error, it tries to read LEB2 (ref. fs/ubifs/recovery.c) I'm working with a 3.10 class kernel, but I've found nearly the same code on mainline.
On my test-bed, the recovery fails because get_master_node(LEB1) fails (exactly here http://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/tree/fs/ubifs/recovery.c#n184)
I'm hacking around the code to see what's really happening under the wood, but I'm a UBIFS newbie I would like to ask:
1) why if get_master_node(LEB1) fails we don't ALWAYS look at get_master_node(LEB2)? I think we should try to read LEB2 even if something really bad happens to LEB1.. or not?
2) if I bypass the get_master_node(LEB1) return value, I found that get_master_node(LEB2) fails too, for the same reason of LEB1 (see above). IIUC we check about empty space because master node pages get written without being erased every time, but I'm still studying this topic ;-)
If I bypass that check too, I can mount UBIFS and everything inside the FS is there but, of course, I'm sure I'm doing something that may be wrong..
WDYT?
Thanks in advance and kind regards,
Signature · 3 lines
Andrea SCIAN DAVE Embedded Systems
Andrea,
Quoted 22 lines · Andrea Scian
Dear all,
I'm trying to understand how UBIFS recovers master node from a corrupted flash (e.g. after power cut during one of the two of master node or due flash corruption itself).
IIUC (please correct me if I'm wrong) UBIFS store two copies of master node, in LEB1 and LEB2 (LEB0 is reserved for superblock), ref.
http://www.linux-mtd.infradead.org/doc/ubifs_whitepaper.pdf.
Insideubifs_recover_master_node()try to read first LEB1 and, only in case ofget_master_node()does NOT return an error, it tries to read LEB2 (ref.fs/ubifs/recovery.c)
I'm working with a 3.10 class kernel, but I've found nearly the same code on mainline.On my test-bed, the recovery fails because
get_master_node(LEB1)fails (exactly here
http://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/tree/fs/ubifs/recovery.c#n184)
I'm hacking around the code to see what's really happening under the wood, but I'm a UBIFS newbie I would like to ask:
Quoted 3 lines
1) why if
get_master_node(LEB1)fails we don't ALWAYS look atget_master_node(LEB2)? I think we should try to read LEB2 even if something really bad happens to LEB1.. or not?
AFAIK the idea was that only upon plausible errors the second LEB will be used. If reading LEB1 fails due to an internal MTD error UBIFS gives up.
Quoted 9 lines
2) if I bypass the
get_master_node(LEB1)return value, I found thatget_master_node(LEB2)fails too, for the same reason of LEB1 (see above). IIUC we check about empty space because master node pages get written without being erased every time, but I'm still studying this topic ;-)If I bypass that check too, I can mount UBIFS and everything inside the FS is there but, of course, I'm sure I'm doing something that may be wrong..
WDYT?
So, you're facing bitflips on empty space?
Is this MLC NAND?
Signature · 2 lines
Thanks, //richard
Dear Richard,
Il 16/07/2015 17:29, Richard Weinberger ha scritto:
Quoted 24 lines
Andrea,
Quoted 21 lines · Andrea Scian
Dear all,
I'm trying to understand how UBIFS recovers master node from a corrupted flash (e.g. after power cut during one of the two of master node or due flash corruption itself).
IIUC (please correct me if I'm wrong) UBIFS store two copies of master node, in LEB1 and LEB2 (LEB0 is reserved for superblock), ref.
http://www.linux-mtd.infradead.org/doc/ubifs_whitepaper.pdf.
Insideubifs_recover_master_node()try to read first LEB1 and, only in case ofget_master_node()does NOT return an error, it tries to read LEB2 (ref.fs/ubifs/recovery.c)
I'm working with a 3.10 class kernel, but I've found nearly the same code on mainline.On my test-bed, the recovery fails because
get_master_node(LEB1)fails (exactly here
http://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/tree/fs/ubifs/recovery.c#n184)
I'm hacking around the code to see what's really happening under the wood, but I'm a UBIFS newbie I would like to ask:
Quoted 5 lines
Quoted 3 lines
1) why if
get_master_node(LEB1)fails we don't ALWAYS look atget_master_node(LEB2)? I think we should try to read LEB2 even if something really bad happens to LEB1.. or not?AFAIK the idea was that only upon plausible errors the second LEB will be used. If reading LEB1 fails due to an internal MTD error UBIFS gives up.
Understood, I think you already told me that UBI/UBIFS assume that empty flash always stay empty
However, to me it seems this is a heavy limitation, if we have another master node, why don't we always try to use it? Again, I'm just trying to understand the recovery code and try to improve it (if possible ;-) )
Quoted 10 lines
Quoted 9 lines
2) if I bypass the
get_master_node(LEB1)return value, I found thatget_master_node(LEB2)fails too, for the same reason of LEB1 (see above). IIUC we check about empty space because master node pages get written without being erased every time, but I'm still studying this topic ;-)If I bypass that check too, I can mount UBIFS and everything inside the FS is there but, of course, I'm sure I'm doing something that may be wrong..
WDYT?
So, you're facing bitflips on empty space?
Yes, and this seems the same behavior I saw previously regarding factory bad block marker (ref http://lists.infradead.org/pipermail/linux-mtd/2015-March/058151.html)
Quoted 1 line
Is this MLC NAND?
Yes, I'm doing some heavy stress test on it
Kind Regards,
Signature · 3 lines
Andrea SCIAN DAVE Embedded Systems
Il 16/07/2015 17:29, Richard Weinberger ha scritto:
Quoted 10 lines
Andrea,
Quoted 5 lines · Andrea Scian
If I bypass that check too, I can mount UBIFS and everything inside the FS is there but, of course, I'm sure I'm doing something that may be wrong..
WDYT?
So, you're facing bitflips on empty space?
Another UBI/UBIFS "implementation" question: are there some other places, apart from get_master_node(), where UBIFS check empty space corruption and fails badly if something wrong?
TIA,
Signature · 3 lines
Andrea SCIAN DAVE Embedded Systems
Richard Weinberger
maintainerAm 17.07.2015 um 08:58 schrieb Andrea Scian:
Quoted 5 lines
Il 16/07/2015 17:29, Richard Weinberger ha scritto:
Quoted 4 lines
Andrea,
Quoted 1 line · Andrea Scian
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
If I bypass that check too, I can mount UBIFS and everything inside the FS is there but, of course, I'm sure I'm doing something that may be wrong..
WDYT?
So, you're facing bitflips on empty space?
Another UBI/UBIFS "implementation" question: are there some other places, apart from
get_master_node(), where UBIFS check empty space corruption and fails badly if something wrong?
Having non-corrupted empty space is a fundamental requirement of UBIFS. If you patch it out you'll hurt UBIFS's ability to recover from a power cut. Someone tried to do so already.
I know, cheap modern NAND, especially MLC seems to show bitflips also on empty pages. Not all NAND controllers can deal with that and will just return an uncorrectable ECC error upon reading.
IMHO the right place to deal with that is MTD core.
Please search the archives, Brian posted some patches some time ago.
Thanks,
//richard
Dear Richard,
Il 17/07/2015 09:24, Richard Weinberger ha scritto:
Quoted 13 lines
Am 17.07.2015 um 08:58 schrieb Andrea Scian:
Quoted 12 lines
Il 16/07/2015 17:29, Richard Weinberger ha scritto:
Quoted 10 lines
Andrea,
Quoted 5 lines · Andrea Scian
If I bypass that check too, I can mount UBIFS and everything inside the FS is there but, of course, I'm sure I'm doing something that may be wrong..
WDYT?
So, you're facing bitflips on empty space?
Quoted 8 lines
Quoted 4 lines
Another UBI/UBIFS "implementation" question: are there some other places, apart from
get_master_node(),
where UBIFS check empty space corruption and fails
badly if something wrong?Having non-corrupted empty space is a fundamental requirement of UBIFS. If you patch it out you'll hurt UBIFS's ability to recover from a power cut. Someone tried to do so already.
Thanks, this are the internals of UBIFS I'm not aware of, and for this I'm asking the experts :-)
Quoted 3 lines
I know, cheap modern NAND, especially MLC seems to show bitflips also on empty pages. Not all NAND controllers can deal with that and will just return an uncorrectable ECC error upon reading.
Is the any NAND controller able to do so? ;-)
Quoted 1 line
IMHO the right place to deal with that is MTD core.
I agree with you, however I'm handling it at lowest level, inside the NAND controller.
I know that having this code into the MTD NAND layer will allow us to have a "controller independent" implementation, however MTD see only a bigger picture: for example MTD sees only a NAND page (4k in my case) while the NAND controller usually apply ECC on smaller (1k in my case) section and here we have the right threshold to apply (ecc_strength or something a bit smaller if you prefer).
Finding the right threshold is, IMHO, the real trick.
Quoted 1 line
Please search the archives, Brian posted some patches some time ago.
Thanks for point this out.
I've found some patches from one year ago from Huang Shijie (@freescale) which looks very close to mine. However I cannot see any inclusion in the mainline.
I'll continue my research.
Thanks and kind regards,
Signature · 3 lines
Andrea SCIAN DAVE Embedded Systems
Richard Weinberger
maintainerAm 17.07.2015 um 10:04 schrieb Andrea Scian:
Quoted 26 lines
Dear Richard,
Il 17/07/2015 09:24, Richard Weinberger ha scritto:
Quoted 13 lines
Am 17.07.2015 um 08:58 schrieb Andrea Scian:
Quoted 12 lines
Il 16/07/2015 17:29, Richard Weinberger ha scritto:
Quoted 10 lines
Andrea,
Quoted 5 lines · Andrea Scian
If I bypass that check too, I can mount UBIFS and everything inside the FS is there but, of course, I'm sure I'm doing something that may be wrong..
WDYT?
So, you're facing bitflips on empty space?
Quoted 8 lines
Quoted 4 lines
Another UBI/UBIFS "implementation" question: are there some other places, apart from
get_master_node(),
where UBIFS check empty space corruption and fails
badly if something wrong?Having non-corrupted empty space is a fundamental requirement of UBIFS. If you patch it out you'll hurt UBIFS's ability to recover from a power cut. Someone tried to do so already.
Quoted 7 lines
Thanks, this are the internals of UBIFS I'm not aware of, and for this I'm asking the experts :-)
Quoted 3 lines
I know, cheap modern NAND, especially MLC seems to show bitflips also on empty pages. Not all NAND controllers can deal with that and will just return an uncorrectable ECC error upon reading.
Is the any NAND controller able to do so? ;-)
TBH, I don't know. :-)
Quoted 8 lines
Quoted 1 line
IMHO the right place to deal with that is MTD core.
I agree with you, however I'm handling it at lowest level, inside the NAND controller.
I know that having this code into the MTD NAND layer will allow us to have a "controller independent" implementation, however MTD see only a bigger picture: for example MTD sees only a NAND page (4k in my case) while the NAND controller usually apply ECC on smaller (1k in my case) section and here we have the right threshold to apply (ecc_strengthor something a bit smaller if you prefer).Finding the right threshold is, IMHO, the real trick.
Another problem is that on some controllers ECC simply does not work for empty pages. So, you'll get an uncorrectable ECC error upon the first bitflip.
Quoted 2 lines
Thanks for point this out.
I've found some patches from one year ago from Huang Shijie (@freescale) which looks very close to mine. However I cannot see any inclusion in the mainline.
None of the proposed solutions went mainline so far. :(
Thanks,
//richard
2015-07-17 10:10 GMT+02:00 Richard Weinberger <[email protected]>:
Quoted 29 lines
Am 17.07.2015 um 10:04 schrieb Andrea Scian:
Quoted 28 lines
Dear Richard,
Il 17/07/2015 09:24, Richard Weinberger ha scritto:
Quoted 13 lines
Am 17.07.2015 um 08:58 schrieb Andrea Scian:
Quoted 12 lines
Il 16/07/2015 17:29, Richard Weinberger ha scritto:
Quoted 10 lines
Andrea,
Quoted 5 lines · Andrea Scian
If I bypass that check too, I can mount UBIFS and everything inside the FS is there but, of course, I'm sure I'm doing something that may be wrong..
WDYT?
So, you're facing bitflips on empty space?
Quoted 8 lines
Quoted 4 lines
Another UBI/UBIFS "implementation" question: are there some other places, apart from
get_master_node(),
where UBIFS check empty space corruption and fails
badly if something wrong?Having non-corrupted empty space is a fundamental requirement of UBIFS. If you patch it out you'll hurt UBIFS's ability to recover from a power cut. Someone tried to do so already.
Thanks, this are the internals of UBIFS I'm not aware of, and for this I'm asking the experts :-)
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
I know, cheap modern NAND, especially MLC seems to show bitflips also on empty pages. Not all NAND controllers can deal with that and will just return an uncorrectable ECC error upon reading.
Is the any NAND controller able to do so? ;-)
TBH, I don't know. :-)
The atmel nand controllers since SAMA5 can do that (and that quite easy to do) :
All they have to do is set an XOR value at the end of the ECC computation so that ECC(blank_page)==0xFFFFFFFFF...
(maybe it's more complicated that that, but that's the idea).
As far as I've seen, the actual implementation (in atmel_nand for chips >SAMA5) is buggy :
http://lxr.free-electrons.com/source/drivers/mtd/nand/atmel_nand.c#L857
If the ECC is all FF, the page is considered to be a blank page (but it's not memset-ed to 0)
and a bitflip in the ecc area is not handled either.
I didn't have time to rebound on Brian's patch proposal yet (
http://patchwork.ozlabs.org/patch/328994/ ) but it's clear that we have to do something to address that.
At least, UBIFS is screaming out loud when a blank page is not blank ! others FS may just write corrupted data...
Richard.
Quoted 14 lines · Richard Weinberger
Am 17.07.2015 um 08:58 schrieb Andrea Scian:
Quoted 13 lines
Il 16/07/2015 17:29, Richard Weinberger ha scritto:
Quoted 12 lines
Andrea,
Quoted 7 lines · Andrea Scian
If I bypass that check too, I can mount UBIFS and everything inside the FS
is there but, of course, I'm sure I'm doing something that may be wrong..WDYT?
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
So, you're facing bitflips on empty space?
Another UBI/UBIFS "implementation" question: are there some other places, apart from
get_master_node(), where UBIFS check empty space corruption and fails badly if something wrong?Having non-corrupted empty space is a fundamental requirement of UBIFS.
I am not sure it is that fundamental. What UBIFS needs is to distinguish between used and usused flash areas. It does this by comparing agains 0xFFs. Simple, worked fine in the past.
If the space is empty, UBIFS assumes it can write to it. UBIFS is being paranoid and also verifies that entire empty space contains all 0xFFs. Also very simple, worked fine in the past.
Now if you have "corrupted empty space" (i.e., you cannot write to it), this does not have to be the end of the world, you can clean it up by doing the "atomic LEB change" operation. It is just not implemented, but it could be done - matter of engineer-hours spent.
Artem.
Richard Weinberger
maintainerAm 17.07.2015 um 13:38 schrieb Artem Bityutskiy:
Quoted 21 lines · Richard Weinberger
Quoted 20 lines
Am 17.07.2015 um 08:58 schrieb Andrea Scian:
Quoted 18 lines
Il 16/07/2015 17:29, Richard Weinberger ha scritto:
Quoted 5 lines
Andrea,
Quoted 1 line · Andrea Scian
Quoted 8 lines
Quoted 6 lines
If I bypass that check too, I can mount UBIFS and everything inside the FS
is there but, of course, I'm sure I'm doing something that may be wrong..WDYT?
So, you're facing bitflips on empty space?
Another UBI/UBIFS "implementation" question: are there some other places, apart from
get_master_node(), where UBIFS check empty space corruption and fails badly if something wrong?
Quoted 6 lines
Quoted 2 lines
Having non-corrupted empty space is a fundamental requirement of UBIFS.
I am not sure it is that fundamental. What UBIFS needs is to distinguish between used and usused flash areas. It does this by comparing agains 0xFFs. Simple, worked fine in the past.
Okay. This good to know. I thought it is much more fundamental. :-)
Quoted 8 lines
If the space is empty, UBIFS assumes it can write to it. UBIFS is being paranoid and also verifies that entire empty space contains all 0xFFs. Also very simple, worked fine in the past.
Now if you have "corrupted empty space" (i.e., you cannot write to it), this does not have to be the end of the world, you can clean it up by doing the "atomic LEB change" operation. It is just not implemented, but it could be done - matter of engineer-hours spent.
Thanks for pointing this out, Artem!
Thanks,
//richard