suspect UBIFS async operations causing issues during reboot

We are doing reboot testing with UBIFS on the 3.10 kernel with a new chipset we are working on.

Over 1000's of reboots we eventually find that the NAND has uncorrectable ECC errors reported on a random page when it is mounted.

We have found the problem is that a NAND erase operation is in progress when the reboot occurs. Since the NAND is in the middle of the erase operation the page is mostly FF with some random bits not erased when the reboot occurs.

We suspect the problem is the asynchronous nature of the UBIFS operations. Perhaps the small write buffer that can take 3-5 seconds to be written or some other operation occuring in UBI/UBIFS? I don't think the shutdown of the filesystem is dealing with all the threads properly.

Quoted 11 lines · Scott Branden

We are doing reboot testing with UBIFS on the 3.10 kernel with a new chipset we are working on.

Over 1000's of reboots we eventually find that the NAND has uncorrectable ECC errors reported on a random page when it is mounted.

We have found the problem is that a NAND erase operation is in progress when the reboot occurs. Since the NAND is in the middle of the erase operation the page is mostly FF with some random bits not erased when the reboot occurs.

Quoted 4 lines

We suspect the problem is the asynchronous nature of the UBIFS operations. Perhaps the small write buffer that can take 3-5 seconds to be written or some other operation occuring in UBI/UBIFS? I don't think the shutdown of the filesystem is dealing with all the threads properly.

And what about powercuts?

UBI/UBIFS was designed to survive powercuts.

If your NAND shows strange issues even after a clean reboot something nasty is going on. Does your driver pass all UBI/MTD test?

Hi Richard,

Thanks for the feedback. Comments inline.

Quoted 17 lines · Richard Weinberger

Quoted 15 lines · Scott Branden

We are doing reboot testing with UBIFS on the 3.10 kernel with a new chipset we are working on.

Over 1000's of reboots we eventually find that the NAND has uncorrectable ECC errors reported on a random page when it is mounted.

We have found the problem is that a NAND erase operation is in progress when the reboot occurs. Since the NAND is in the middle of the erase operation the page is mostly FF with some random bits not erased when the reboot occurs.

We suspect the problem is the asynchronous nature of the UBIFS operations. Perhaps the small write buffer that can take 3-5 seconds to be written or some other operation occuring in UBI/UBIFS? I don't think the shutdown of the filesystem is dealing with all the threads properly.

Quoted 1 line

And what about powercuts?

powercuts would exhibit the exact same behaviour as we are observing: the erase is interrupted by loss of power so the NAND block being erased would be in a partially erased state. powercuts have little to do with the reboot sequence I am describing.

Quoted 1 line

UBI/UBIFS was designed to survive powercuts.

Yes, this does not cause UBIFS to fail to survive the powercut. It does cause blocks to not be erased properly.

The block that didn't finish to erase is uncorrectable on next boot-up:

[    1.330000] UBI: attaching mtd7 to ubi0
[    2.000000] iproc_nand 18046000.nand: uncorrectable error at 0x18700000

This issue is this blocks shouldn't be corrupted in the first place if UBI/UBIFS shut downs properly.

Quoted 3 lines

If your NAND shows strange issues even after a clean reboot something nasty is going on. Does your driver pass all UBI/MTD test?

We are in the process of running the MTD tests. But this appears to have nothing to do with a buggy driver or not. The NAND driver will do what it is told to do. If it is told to erase a block it will erase a block. It can't control if the system reboots in the middle of this operation?

This appears to be a UBI/UBIFS issue. UBI/UBIFS operations are still going on after the filesystem in unmounted. The shutdown process completes and a reboot happens. My guess is these operations are due to the asynchronous threads of UBI/UBIFS not being handled properly during the shutdown process?

I have found other people have reported unexplained flash corruption. We back ported this to the 3.10 kernel which solved most of the flash corruption issues:
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/fs/super.c?id=807612db2f9940b9fa6deaef054eb16d51bd3e00

This only remaining flash corruption issue is due to the described issue of reboot happening in the middle of an erase cycle.

Hi!

Am 05.11.2014 um 18:56 schrieb Scott Branden:

Quoted 22 lines

Hi Richard,

Thanks for the feedback. Comments inline.

Quoted 17 lines · Richard Weinberger

Quoted 15 lines · Scott Branden

We are doing reboot testing with UBIFS on the 3.10 kernel with a new chipset we are working on.

Over 1000's of reboots we eventually find that the NAND has uncorrectable ECC errors reported on a random page when it is mounted.

We have found the problem is that a NAND erase operation is in progress when the reboot occurs. Since the NAND is in the middle of the erase operation the page is mostly FF with some random bits not erased when the reboot occurs.

We suspect the problem is the asynchronous nature of the UBIFS operations. Perhaps the small write buffer that can take 3-5 seconds to be written or some other operation occuring in UBI/UBIFS? I don't think the shutdown of the filesystem is dealing with all the threads properly.

Quoted 6 lines

Quoted 1 line

And what about powercuts?

powercuts would exhibit the exact same behaviour as we are observing: the erase is interrupted by loss of power so the NAND block being erased would be in a partially erased state. powercuts have little to do with the reboot sequence I am describing.

Quoted 1 line

UBI/UBIFS was designed to survive powercuts.

Yes, this does not cause UBIFS to fail to survive the powercut. It does cause blocks to not be erased properly.

Makes sense.

Quoted 16 lines

The block that didn't finish to erase is uncorrectable on next boot-up:

[    1.330000] UBI: attaching mtd7 to ubi0
[    2.000000] iproc_nand 18046000.nand: uncorrectable error at 0x18700000

This issue is this blocks shouldn't be corrupted in the first place if UBI/UBIFS shut downs properly.

Quoted 3 lines

If your NAND shows strange issues even after a clean reboot something nasty is going on. Does your driver pass all UBI/MTD test?

We are in the process of running the MTD tests. But this appears to have nothing to do with a buggy driver or not. The NAND driver will do what it is told to do. If it is told to erase a block it will erase a block. It can't control if the system reboots in the middle of this operation?

This appears to be a UBI/UBIFS issue. UBI/UBIFS operations are still going on after the filesystem in unmounted. The shutdown process completes and a reboot happens. My guess is these operations are due to the asynchronous threads of UBI/UBIFS not being handled properly during the shutdown process?

Quoted 4 lines

I have found other people have reported unexplained flash corruption. We back ported this to the 3.10 kernel which solved most of the flash corruption issues:
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/fs/super.c?id=807612db2f9940b9fa6deaef054eb16d51bd3e00

This only remaining flash corruption issue is due to the described issue of reboot happening in the middle of an erase cycle.

You can verify your hypothesis easily. Add a printk() to ubi_detach_mtd_dev(). This function shuts down UBI and also the background thread which does all erase work.

Thanks,
//richard

Quoted 77 lines · Richard Weinberger

Hi!

Am 05.11.2014 um 18:56 schrieb Scott Branden:

Quoted 37 lines

Hi Richard,

Thanks for the feedback. Comments inline.

Quoted 25 lines · Richard Weinberger

Quoted 15 lines · Scott Branden

We are doing reboot testing with UBIFS on the 3.10 kernel with a new chipset
we are working on.

Over 1000's of reboots we eventually find that the NAND has uncorrectable
ECC errors reported on a random page when it is mounted.

We have found the problem is that a NAND erase operation is in progress when
the reboot occurs. Since the NAND is in the middle of the erase operation
the page is mostly FF with some random bits not erased when the reboot occurs.

Quoted 7 lines

We suspect the problem is the asynchronous nature of the UBIFS operations.
Perhaps the small write buffer that can take 3-5 seconds to be written or
some other operation occuring in UBI/UBIFS? I don't think the shutdown of
the filesystem is dealing with all the threads properly.

Quoted 1 line

And what about powercuts?

powercuts would exhibit the exact same behaviour as we are observing:
the erase is interrupted by loss of power so the NAND block being erased would be in a partially erased
state. powercuts have little to do with the reboot sequence I am describing.

Quoted 3 lines

Quoted 1 line

UBI/UBIFS was designed to survive powercuts.

Yes, this does not cause UBIFS to fail to survive the powercut. It does cause blocks to not be erased properly.

Makes sense.

Quoted 31 lines

The block that didn't finish to erase is uncorrectable on next boot-up:

[    1.330000] UBI: attaching mtd7 to ubi0
[    2.000000] iproc_nand 18046000.nand: uncorrectable error at

0x18700000

This issue is this blocks shouldn't be corrupted in the first place if UBI/UBIFS shut downs properly.

Quoted 4 lines

If your NAND shows strange issues even after a clean reboot something nasty is
going on. Does your driver pass all UBI/MTD test?

We are in the process of running the MTD tests. But this appears to have nothing to do with a buggy driver or not. The NAND driver will do what it is told to do. If it is told
to erase a block it will erase a block. It can't control if the system reboots in the middle of this operation?

This appears to be a UBI/UBIFS issue. UBI/UBIFS operations are still going on after the filesystem in unmounted. The shutdown process completes and a reboot happens. My guess is
these operations are due to the asynchronous threads of UBI/UBIFS not being handled properly during the shutdown process?

I have found other people have reported unexplained flash corruption. We back ported this to the 3.10 kernel which solved most of the flash corruption issues:
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/fs/super.c?id=807612db2f9940b9fa6deaef054eb16d51bd3e00

Quoted 7 lines

Quoted 2 lines

This only remaining flash corruption issue is due to the described issue of reboot happening in the middle of an erase cycle.

You can verify your hypothesis easily. Add a printk() to ubi_detach_mtd_dev(). This function shuts down UBI and also the background thread which does
all erase work.

Hi Richard,

The printk never happens.

I only find ubi_detach_mtd_dev can be called by ubi_exit. But ubi_exit
is only called if it is a module...

static void __exit ubi_exit(void)

{
     int i;

     for (i = 0; i < UBI_MAX_DEVICES; i++)
         if (ubi_devices[i]) {
             mutex_lock(&ubi_devices_mutex);
             ubi_detach_mtd_dev(ubi_devices[i]->ubi_num, 1);
             mutex_unlock(&ubi_devices_mutex);
         }
     ubi_debugfs_exit();
     kmem_cache_destroy(ubi_wl_entry_slab);
     misc_deregister(&ubi_ctrl_cdev);
     class_remove_file(ubi_class, &ubi_version);
     class_destroy(ubi_class);
}

module_exit(ubi_exit);

Quoted 4 lines

Thanks,
//richard

It looks like the erase happening in the middle of reboot was uncovered in 2009 and never addressed properly?

https://lkml.org/lkml/2009/6/9/16
https://lkml.org/lkml/2010/2/12/144

Was there a proper resolution to this issue?

Am 06.11.2014 um 22:56 schrieb Scott Branden:

Quoted 5 lines

It looks like the erase happening in the middle of reboot was uncovered in 2009 and never addressed properly?

https://lkml.org/lkml/2009/6/9/16
https://lkml.org/lkml/2010/2/12/144

Quoted 1 line

Was there a proper resolution to this issue?

Did you read the threads you've posted?

There two answers:
https://lkml.org/lkml/2010/2/12/143

https://lkml.org/lkml/2010/2/12/144

Thanks,
//richard

Quoted 6 lines

Am 06.11.2014 um 22:56 schrieb Scott Branden:

Quoted 5 lines

It looks like the erase happening in the middle of reboot was uncovered in 2009 and never addressed properly?

https://lkml.org/lkml/2009/6/9/16
https://lkml.org/lkml/2010/2/12/144

Quoted 6 lines

Quoted 1 line

Was there a proper resolution to this issue?

Did you read the threads you've posted?

There two answers:
https://lkml.org/lkml/2010/2/12/143

Yes, there is no hardware solution to a reset happening in the middle of an erase operation to NAND.

Am 07.11.2014 um 18:31 schrieb Scott Branden:

Quoted 9 lines · Richard Weinberger

Quoted 8 lines

Am 06.11.2014 um 22:56 schrieb Scott Branden:

Quoted 6 lines

It looks like the erase happening in the middle of reboot was uncovered in 2009 and never addressed properly?

https://lkml.org/lkml/2009/6/9/16
https://lkml.org/lkml/2010/2/12/144

Was there a proper resolution to this issue?

Quoted 5 lines

Quoted 4 lines

Did you read the threads you've posted?

There two answers:
https://lkml.org/lkml/2010/2/12/143

Yes, there is no hardware solution to a reset happening in the middle of an erase operation to NAND.

Well, I agree with David that anything we do in software will only hide the real problem or trim down the window.

Thanks,
//richard

Quoted 2 lines · Richard Weinberger

Well, I agree with David that anything we do in software will only hide the real problem or trim down the window.

Hi Richard,

Currently the NAND does not shut down in a clean manner for a reboot operation. This is due to the asynchronous ubi_thread make flash erase calls. unmount is done properly in ubi already and cleanly shuts down. reboot is not done in a clean manner as there is no reboot_notifier to handle the situation.

This is not hiding a real problem. It is just shutting down ubi properly rather than pulling the power from it in the middle of operations.

In addition to this - a reboot_notifier needs to be added at the mtd level to shut it down properly as well.

This is not trimming down a window. It is having the drivers shut down properly so they do not look like a power failure to the NAND device.

There is no solution to the power failure - it will corrupt pages in the middle of erasure. And you do handle this in UBI/UBIFS. But why corrupt other erase pages unnecessarily when all that needs to be done is shut down the drivers properly. I don't know what you are agreeing with David with? It is not making a window smaller. It is changing the functionality so that the UBI and MTD drivers are shut down cleanly in reboot situations. Right now, they are not shut down at all in these situations.

Quoted 22 lines · Richard Weinberger

Quoted 2 lines

Well, I agree with David that anything we do in software will only hide the real problem or trim down the window.

Hi Richard,

Currently the NAND does not shut down in a clean manner for a reboot operation. This is due to the asynchronous ubi_thread make flash erase calls. unmount is done properly in ubi already and cleanly shuts down. reboot is not done in a clean manner as there is no reboot_notifier to handle the situation.

This is not hiding a real problem. It is just shutting down ubi properly rather than pulling the power from it in the middle of operations.

In addition to this - a reboot_notifier needs to be added at the mtd level to shut it down properly as well.

This is not trimming down a window. It is having the drivers shut down properly so they do not look like a power failure to the NAND device.

Quoted 8 lines

There is no solution to the power failure - it will corrupt pages in the middle of erasure. And you do handle this in UBI/UBIFS. But why corrupt other erase pages unnecessarily when all that needs to be done is shut down the drivers properly. I don't know what you are agreeing with David with? It is not making a window smaller. It is changing the functionality so that the UBI and MTD drivers are shut down cleanly in reboot situations. Right now, they are not shut down at all in these situations.

I agree with Scott's statements. While it's fine to talk about how all layers (from bootloader to UBIFS) should be able to handle a power cut in the midst of an erase, that does *not* mean that we should intentionally deny the chance to shut down cleanly.

AFAICT, Scott's not trying to work around any unsound reset behaviors (in UBIFS or in his bootloader); he's just trying to shut things down gracefully, just as we would try to terminate processes, sync file systems, etc., rather than just cutting power on reboot.

If there is a solution which makes Artem and David happy, I'm perfectly fine. :)

Quoted 22 lines · Scott Branden

Quoted 2 lines · Richard Weinberger

Well, I agree with David that anything we do in software will only hide the real problem or trim down the window.

Hi Richard,

Currently the NAND does not shut down in a clean manner for a reboot operation. This is due to the asynchronous ubi_thread make flash erase calls. unmount is done properly in ubi already and cleanly shuts down. reboot is not done in a clean manner as there is no reboot_notifier to handle the situation.

This is not hiding a real problem. It is just shutting down ubi properly rather than pulling the power from it in the middle of operations.

In addition to this - a reboot_notifier needs to be added at the mtd level to shut it down properly as well.

This is not trimming down a window. It is having the drivers shut down properly so they do not look like a power failure to the NAND device.

Quoted 8 lines

There is no solution to the power failure - it will corrupt pages in the middle of erasure. And you do handle this in UBI/UBIFS. But why corrupt other erase pages unnecessarily when all that needs to be done is shut down the drivers properly. I don't know what you are agreeing with David with? It is not making a window smaller. It is changing the functionality so that the UBI and MTD drivers are shut down cleanly in reboot situations. Right now, they are not shut down at all in these situations.

I agree with Scott's statements. While it's fine to talk about how all layers (from bootloader to UBIFS) should be able to handle a power cut in the midst of an erase, that does *not* mean that we should intentionally deny the chance to shut down cleanly.

AFAICT, Scott's not trying to work around any unsound reset behaviors (in UBIFS or in his bootloader); he's just trying to shut things down gracefully, just as we would try to terminate processes, sync file systems, etc., rather than just cutting power on reboot.

Brian

Am 07.11.2014 um 18:31 schrieb Scott Branden:

Quoted 9 lines · Richard Weinberger

Quoted 8 lines

Am 06.11.2014 um 22:56 schrieb Scott Branden:

Quoted 6 lines

It looks like the erase happening in the middle of reboot was uncovered in 2009 and never addressed properly?

https://lkml.org/lkml/2009/6/9/16
https://lkml.org/lkml/2010/2/12/144

Was there a proper resolution to this issue?

Quoted 5 lines

Quoted 4 lines

Did you read the threads you've posted?

There two answers:
https://lkml.org/lkml/2010/2/12/143

Yes, there is no hardware solution to a reset happening in the middle of an erase operation to NAND.

Well, I agree with David that anything we do in software will only hide the real problem or trim down the window.

Quoted 11 lines

Am 07.11.2014 um 18:31 schrieb Scott Branden:

Quoted 9 lines · Richard Weinberger

Quoted 8 lines

Am 06.11.2014 um 22:56 schrieb Scott Branden:

Quoted 6 lines

It looks like the erase happening in the middle of reboot was uncovered in 2009 and never addressed properly?

https://lkml.org/lkml/2009/6/9/16
https://lkml.org/lkml/2010/2/12/144

Was there a proper resolution to this issue?

Quoted 8 lines

Quoted 5 lines

Quoted 4 lines

Did you read the threads you've posted?

There two answers:
https://lkml.org/lkml/2010/2/12/143

Yes, there is no hardware solution to a reset happening in the middle of an erase operation to NAND.

Well, I agree with David that anything we do in software will only hide the real problem or trim down the window.

There's something I don't understand here. It could be (and probably will prove to be) my lack of knowledge on the detailed workings of UBI.

Back in jffs2 days, erased blocks were so indicated by writing a 'cleanmarker' pattern to the OOB area. Thus, when scanning the flash, if a block was encountered which appeared erased but lacked the cleanmarker, it was re-erased just in case the previous erase was interrupted and therefore did not leave the bits in a properly erased state.

With ubifs, cleanmarkers are not used (partly because MLC flashes wouldn't support two writes to the OOB area: one for the cleanmarker and one for the ECC), but there _is_ a header at the start of each PEB. Thus the same situation really holds, if a (seemingly) erased PEB is encountered with no EC header, it could be considered the leftover of an unfinished erase operation. I don't know for a fact if (or how) UBI does this though.

Of course, and interrupted erase operation could leave a block in a seemingly un-erased state, i.e. the data appears intact (but may not be). But in that case the block would already be superseded by another block (i.e. any potential data would have already been copied to another block with the header infoinvalidating the old one). So in this case the block would go on an erase list at some point because it is no longer valid.

Since interrupted erase seems to be of so much a concern I've obviously missed something above. But I can't figure out what.

The only thing that seems relevant among the links above is

https://lkml.org/lkml/2010/2/12/144

which indicates that half-erased blocks might cause problems with certain boot loaders, but again, that's a problem with the bootloader, not UBI.

Quoted 77 lines · Richard Weinberger

Hi!

Am 05.11.2014 um 18:56 schrieb Scott Branden:

Quoted 37 lines

Hi Richard,

Thanks for the feedback. Comments inline.

Quoted 25 lines · Richard Weinberger

Quoted 15 lines · Scott Branden

We are doing reboot testing with UBIFS on the 3.10 kernel with a new chipset
we are working on.

Over 1000's of reboots we eventually find that the NAND has uncorrectable
ECC errors reported on a random page when it is mounted.

We have found the problem is that a NAND erase operation is in progress when
the reboot occurs. Since the NAND is in the middle of the erase operation
the page is mostly FF with some random bits not erased when the reboot occurs.

Quoted 7 lines

We suspect the problem is the asynchronous nature of the UBIFS operations.
Perhaps the small write buffer that can take 3-5 seconds to be written or
some other operation occuring in UBI/UBIFS? I don't think the shutdown of
the filesystem is dealing with all the threads properly.

Quoted 1 line

And what about powercuts?

powercuts would exhibit the exact same behaviour as we are observing:
the erase is interrupted by loss of power so the NAND block being erased would be in a partially erased
state. powercuts have little to do with the reboot sequence I am describing.

Quoted 3 lines

Quoted 1 line

UBI/UBIFS was designed to survive powercuts.

Yes, this does not cause UBIFS to fail to survive the powercut. It does cause blocks to not be erased properly.

Makes sense.

Quoted 31 lines

The block that didn't finish to erase is uncorrectable on next boot-up:

[    1.330000] UBI: attaching mtd7 to ubi0
[    2.000000] iproc_nand 18046000.nand: uncorrectable error at

0x18700000

This issue is this blocks shouldn't be corrupted in the first place if UBI/UBIFS shut downs properly.

Quoted 4 lines

If your NAND shows strange issues even after a clean reboot something nasty is
going on. Does your driver pass all UBI/MTD test?

We are in the process of running the MTD tests. But this appears to have nothing to do with a buggy driver or not. The NAND driver will do what it is told to do. If it is told
to erase a block it will erase a block. It can't control if the system reboots in the middle of this operation?

This appears to be a UBI/UBIFS issue. UBI/UBIFS operations are still going on after the filesystem in unmounted. The shutdown process completes and a reboot happens. My guess is
these operations are due to the asynchronous threads of UBI/UBIFS not being handled properly during the shutdown process?

I have found other people have reported unexplained flash corruption. We back ported this to the 3.10 kernel which solved most of the flash corruption issues:
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/fs/super.c?id=807612db2f9940b9fa6deaef054eb16d51bd3e00

Quoted 7 lines

Quoted 2 lines

This only remaining flash corruption issue is due to the described issue of reboot happening in the middle of an erase cycle.

You can verify your hypothesis easily. Add a printk() to ubi_detach_mtd_dev(). This function shuts down UBI and also the background thread which does
all erase work.

Hi Richard,

The printk never happens.

I only find ubi_detach_mtd_dev can be called by ubi_exit. But ubi_exit
is only called if it is a module...

static void __exit ubi_exit(void)

{
     int i;

     for (i = 0; i < UBI_MAX_DEVICES; i++)
         if (ubi_devices[i]) {
             mutex_lock(&ubi_devices_mutex);
             ubi_detach_mtd_dev(ubi_devices[i]->ubi_num, 1);
             mutex_unlock(&ubi_devices_mutex);
         }
     ubi_debugfs_exit();
     kmem_cache_destroy(ubi_wl_entry_slab);
     misc_deregister(&ubi_ctrl_cdev);
     class_remove_file(ubi_class, &ubi_version);
     class_destroy(ubi_class);
}

module_exit(ubi_exit);

Quoted 1 line

Quoted 3 lines

Thanks,
//richard

______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/

Over 1000's of reboots we eventually find that the NAND has uncorrectable ECC errors reported on a random page when it is mounted.

We suspect the problem is the asynchronous nature of the UBIFS operations. Perhaps the small write buffer that can take 3-5 seconds to be written or some other operation occuring in UBI/UBIFS? I don't think the shutdown of the filesystem is dealing with all the threads properly.

<REBOOT happens here with NAND ERASE COMMAND in progress corrupting 0x18700000 NAND Addresses!> Corrupted NAND only happens when erase operation in progress when restarting system happens.

Hi Scott,

sorry for late reply, but better later than never.

Quoted 2 lines · Scott Branden

Over 1000's of reboots we eventually find that the NAND has uncorrectable ECC errors reported on a random page when it is mounted.

How do you find the uncorrectable errors? Do you scan the entire NAND chip after you boot up? Or do you read all files stored in the UBIFS file-system, or you do not do anything special, just mount and notice ECC error messages in dmesg? Does UBIFS fail to mount?

What is the time-window where power cut may lead to problems in your NAND. And how these problems are seen by the software? I mean, what happens to the data? Can it become "mostly OK", except of one or few pages with too many bit-flips? I understand that during erase all 0 bits "become 1s", but not instanteneously, so in case of an interrupt they may read as 1 or 0 randomly. But the bits which were 1s - nothing happens, they stay to be 1s?

Quoted 5 lines

We suspect the problem is the asynchronous nature of the UBIFS operations. Perhaps the small write buffer that can take 3-5 seconds to be written or some other operation occuring in UBI/UBIFS? I don't think the shutdown of the filesystem is dealing with all the threads properly.

Yes, writes are asynchronous. There is the write-buffer of the NAND page size, and there is Linux write-back, which flushes dirty data in background (standard stuff for all file-systems)

Quoted 3 lines

<REBOOT happens here with NAND ERASE COMMAND in progress corrupting 0x18700000 NAND Addresses!> Corrupted NAND only happens when erase operation in progress when restarting system happens.

I acknowledge that there may be problems with interrupted erase. We saw them in case of NOR, where erase is very slow and it is easy to interrupt it. We never saw this for NAND, but I may well imagine that this may be an issue in case of NAND.

For NOR, we mitigated the issue by "invalidating" the PEB before erasing. Check the 'nor_erase_prepare()' function in 'drivers/mtd/ubi/io.c' and its commentaries.

The first thing you may try is - add a similar quick hack to UBI and invalidate the first NAND page or the first 2 NAND pages (depends on whether you use sub-pages or not).

You can just write all zeroes. The point is to corrupt data, so that the subsequent read results in a CRC check failure.

See what happens.

Some general notes.

In general, if UBI or UBIFS decided to erase an LEB, the data in there are not longer needed. E.g., when GC of UBIFS moves all the valid data to another PEB, the older PEB is not needed, it is scheduled for erasure. The erasure happens asynchronously. If you have a power cut, and the PEB erase operation was interrupted, and you end up with a PEB which is "mostly fine", son next time you mount UBIFS it may start reading from it (e.g., if this was a journal PEB), and get errors.

Now, my point is that this should not be a fundamental problem for UBIFS. This should be fixable. It may need good UBIFS knowledge to fix, and time, though.

One way to deal with this is to emulate erase interruptions at UBI level. Similarly how we implemented the power cut testing infrastructure in UBIFS.

On the other hand, if you can invalidate the PEB before you start erasing, this should just solve the problem. So I'd start with this, and see what happens. You may have more than one type of issues, so fixing the erase interrupt issue this way quickly may let you exlculde this type of problems. And generally, I am not opposed to this solution in upstream too, if it works for everyone.