UBI: Make MTD_UBI_FASTMAP non-experimental
Subsystems (2)
RFC PATCH 1/1 UBI: Make MTD_UBI_FASTMAP non-experimental
MTD_UBI_FASTMAP has been set as experimental since it was merged back in 2012.
There hasn't been much change in the format,
so we can consider the feature stable and start being careful about breaking the format.
(This is somewhat of a pre-requisite for anyone actually using the feature in the real world and depending on it)
Drop the experimental note and the warning text about the on-flash format not being finalized.
--- drivers/mtd/ubi/Kconfig | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)
drivers/mtd/ubi/Kconfig +1 -4
@@ -57,12 +57,9 @@ config MTD_UBI_BEB_LIMIT Leave the default value if unsure. config MTD_UBI_FASTMAP - bool "UBI Fastmap (Experimental feature)" + bool "UBI Fastmap" default n help - Important: this feature is experimental so far and the on-flash - format for fastmap may change in the next kernel versions - Fastmap is a mechanism which allows attaching an UBI device in nearly constant time. Instead of scanning the whole MTD device it only has to locate a checkpoint (called fastmap) on the device.
Signature · 6 lines
2.1.4
/^JN - Jesper Nilsson
--
Jesper Nilsson -- [email protected]Richard Weinberger
maintainerJesper,
Am 29.03.2017 um 17:38 schrieb Jesper Nilsson:
Quoted 11 lines
MTD_UBI_FASTMAPhas been set as experimental since it was merged back in 2012.There hasn't been much change in the format,
so we can consider the feature stable and start being careful about breaking the format.
(This is somewhat of a pre-requisite for anyone actually using the feature in the real world and depending on it)Drop the experimental note and the warning text about the on-flash format not being finalized.
I fully agree, we can drop this note. But we have to add another one.
While Fastmap is a nice feature to speed-up the attach time it comes with a cost. It makes UBI less robust. I saw issues on NAND chips which misbehaved slightly where UBI was able to recover when using a full scan but not when Fastmap was used. The UBI full scan code is paranoid and can sort out problems very early, with Fastmap enabled you lose this valuable property.
So, users should enable Fastmap only when they absolutely need a very fast attach time and be very sure that the NAND works as expected.
Thanks,
//richard
Quoted 24 lines · Richard Weinberger
Jesper,
Am 29.03.2017 um 17:38 schrieb Jesper Nilsson:
Quoted 11 lines
MTD_UBI_FASTMAPhas been set as experimental since it was merged back in 2012.There hasn't been much change in the format,
so we can consider the feature stable and start being careful about breaking the format.
(This is somewhat of a pre-requisite for anyone actually using the feature in the real world and depending on it)Drop the experimental note and the warning text about the on-flash format not being finalized.
I fully agree, we can drop this note. But we have to add another one.
While Fastmap is a nice feature to speed-up the attach time it comes with a cost. It makes UBI less robust. I saw issues on NAND chips which misbehaved slightly where UBI was able to recover when using a full scan but not when Fastmap was used. The UBI full scan code is paranoid and can sort out problems very early, with Fastmap enabled you lose this valuable property.
Quoted 3 lines
So, users should enable Fastmap only when they absolutely need a very fast attach time and be very sure that the NAND works as expected.
So we should document this with a big fat warning and set fastmap to default=n ?
Signature · 2 lines
Best regards, Marek Vasut
Hi Richard, Marek,
Quoted 25 lines · Marek Vasut
Quoted 24 lines · Richard Weinberger
Jesper,
Am 29.03.2017 um 17:38 schrieb Jesper Nilsson:
Quoted 11 lines
MTD_UBI_FASTMAPhas been set as experimental since it was merged back in 2012.There hasn't been much change in the format,
so we can consider the feature stable and start being careful about breaking the format.
(This is somewhat of a pre-requisite for anyone actually using the feature in the real world and depending on it)Drop the experimental note and the warning text about the on-flash format not being finalized.
I fully agree, we can drop this note. But we have to add another one.
While Fastmap is a nice feature to speed-up the attach time it comes with a cost. It makes UBI less robust. I saw issues on NAND chips which misbehaved slightly where UBI was able to recover when using a full scan but not when Fastmap was used. The UBI full scan code is paranoid and can sort out problems very early, with Fastmap enabled you lose this valuable property.
Quoted 6 lines
Quoted 3 lines
So, users should enable Fastmap only when they absolutely need a very fast attach time and be very sure that the NAND works as expected.
So we should document this with a big fat warning and set fastmap to default=n ?
Does this sound reasonable?
Note that this feature makes UBI less robust, since Fastmap does not scan the full flash, which might lead to problems on misbehaving NAND chips. Only enable this if the speedup in attach is really important and you can be sure that the NAND works as expected.
Quoted 2 lines
Best regards,
Marek Vasut
/^JN - Jesper Nilsson
Signature · 1 line
Jesper Nilsson -- [email protected]
Richard Weinberger
maintainerJesper,
Am 30.03.2017 um 19:39 schrieb Jesper Nilsson:
Quoted 8 lines
Quoted 2 lines
So we should document this with a big fat warning and set fastmap to default=n ?
Does this sound reasonable?
Note that this feature makes UBI less robust, since Fastmap does not scan the full flash, which might lead to problems on misbehaving NAND chips. Only enable this if the speedup in attach is really important and
I'm not a native English speaker, but shouldn't this be "...if speedup of the attach time is important ..."
Quoted 1 line
you can be sure that the NAND works as expected.
Looks fine!
Thanks,
//richard
PATCH v2 1/1 UBI: Make MTD_UBI_FASTMAP non-experimental
MTD_UBI_FASTMAP has been set as experimental since it was merged back in 2012.
There hasn't been much change in the format,
so we can consider the feature stable and start being careful about breaking the format.
(This is somewhat of a pre-requisite for anyone actually using the feature in the real world and depending on it)
Drop the experimental note and the warning text about the on-flash format not being finalized, but add a brief warning that Fastmap actually makes UBI less robust.
---
Changes in v2:
- Add warning that Fastmap making UBI less robust
drivers/mtd/ubi/Kconfig | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
drivers/mtd/ubi/Kconfig +5 -4
@@ -57,12 +57,9 @@ config MTD_UBI_BEB_LIMIT Leave the default value if unsure. config MTD_UBI_FASTMAP - bool "UBI Fastmap (Experimental feature)" + bool "UBI Fastmap" default n help - Important: this feature is experimental so far and the on-flash - format for fastmap may change in the next kernel versions - Fastmap is a mechanism which allows attaching an UBI device in nearly constant time. Instead of scanning the whole MTD device it only has to locate a checkpoint (called fastmap) on the device.
@@ -74,6 +71,10 @@ config MTD_UBI_FASTMAP images are still usable with UBI implementations without fastmap support. On typical flash devices the whole fastmap fits into one PEB. UBI will reserve PEBs to hold two fastmaps. + Note that this feature makes UBI less robust, since Fastmap does not scan + the full flash, which might lead to problems on misbehaving NAND chips. + Only enable this if speedup of the attach time is really important + and you can be sure that the NAND works as expected. If in doubt, say "N".
Signature · 6 lines
2.1.4
/^JN - Jesper Nilsson
--
Jesper Nilsson -- [email protected]Hi Richard,
Quoted 12 lines · Richard Weinberger
Jesper,
Am 30.03.2017 um 19:39 schrieb Jesper Nilsson:
Quoted 8 lines
Quoted 2 lines
So we should document this with a big fat warning and set fastmap to default=n ?
Does this sound reasonable?
Note that this feature makes UBI less robust, since Fastmap does not scan the full flash, which might lead to problems on misbehaving NAND chips. Only enable this if the speedup in attach is really important and
Quoted 6 lines
I'm not a native English speaker, but shouldn't this be "...if speedup of the attach time is important ..."
Quoted 1 line
you can be sure that the NAND works as expected.
Looks fine!
As you saw I resent the patch with this formulation added.
However, after thinking about it (and with input from some coworkers), could we pinpoint the failure case a bit more here?
What is the exact problem behaviour on NAND chips that we're worried about, and in which case will UBI be less robust if we don't scan the full flash?
My first reaction was that this was a natural conclusion, but if the NAND flash is failing, we should either be in the case that the FASTMAP is corrupted or that the original data is corrupted. Both should be found by current implementation. Or am I missing additional failure cases here?
I getting a bit worried about using the feature at all, even if it seems to work right now...
Quoted 2 lines
Thanks,
//richard
/^JN - Jesper Nilsson
Signature · 1 line
Jesper Nilsson -- [email protected]
Hi Richard,
I'm still worried about this failure case, do we really believe that the flash could fail in such a way that the fastmap is corrupted in an undetectable way?
If we do detect corruption we should be no worse off than earlier since we should ignore the fastmap, IIRC.
Could you please elaborate on the problem you were thinking about?
Right now I'm hesitant to use fastmap in any production code, even if it works with my current hardware, since there is no guarantee that the flash chips won't get replaced with a second source option down the line...
/Jesper
Quoted 37 lines · Jesper Nilsson
Hi Richard,
Quoted 12 lines · Richard Weinberger
Jesper,
Am 30.03.2017 um 19:39 schrieb Jesper Nilsson:
Quoted 8 lines
Quoted 2 lines
So we should document this with a big fat warning and set fastmap to default=n ?
Does this sound reasonable?
Note that this feature makes UBI less robust, since Fastmap does not scan the full flash, which might lead to problems on misbehaving NAND chips. Only enable this if the speedup in attach is really important and
Quoted 6 lines
I'm not a native English speaker, but shouldn't this be "...if speedup of the attach time is important ..."
Quoted 1 line
you can be sure that the NAND works as expected.
Looks fine!
As you saw I resent the patch with this formulation added.
However, after thinking about it (and with input from some coworkers), could we pinpoint the failure case a bit more here?
What is the exact problem behaviour on NAND chips that we're worried about, and in which case will UBI be less robust if we don't scan the full flash?
My first reaction was that this was a natural conclusion, but if the NAND flash is failing, we should either be in the case that the FASTMAP is corrupted or that the original data is corrupted. Both should be found by current implementation. Or am I missing additional failure cases here?
Quoted 9 lines
I getting a bit worried about using the feature at all, even if it seems to work right now...
Quoted 2 lines
Thanks,
//richard/^JN - Jesper Nilsson
Signature · 1 line
Jesper Nilsson -- [email protected]
/^JN - Jesper Nilsson
Signature · 1 line
Jesper Nilsson -- [email protected]
Richard Weinberger
maintainerJesper,
Am 09.05.2017 um 09:46 schrieb Jesper Nilsson:
Quoted 8 lines
Hi Richard,
I'm still worried about this failure case, do we really believe that the flash could fail in such a way that the fastmap is corrupted in an undetectable way?
If we do detect corruption we should be no worse off than earlier since we should ignore the fastmap, IIRC.
In a perfect world, yes.
Quoted 2 lines
Could you please elaborate on the problem you were thinking about?
e.g.
commit 74f2c6e9a47cf4e508198c8594626cc82906a13d
Author: Richard Weinberger <[email protected]> Date: Tue Jun 14 10:12:17 2016 +0200
ubi: Be more paranoid while seaching for the most recent Fastmap
Since PEB erasure is asynchornous it can happen that there is more than one Fastmap on the MTD. This is fine because the attach logic will pick the Fastmap data structure with the highest sequence number.
On a not so well configured MTD stack spurious ECC errors are common. Causes can be different, bad hardware, wrong operating modes, etc... If the most current Fastmap renders bad due to ECC errors UBI might pick an older Fastmap to attach from.
While this can only happen on an anyway broken setup it will show completely different sympthoms and makes finding the root cause much more difficult.
So, be debug friendly and fall back to scanning mode of we're facing an ECC error while scanning for Fastmap.
Quoted 4 lines
Right now I'm hesitant to use fastmap in any production code, even if it works with my current hardware, since there is no guarantee that the flash chips won't get replaced with a second source option down the line...
Fastmap is an aggressive optimization and makes finding issues much harder.
Thanks,
//richard