Skip to contentSkip to conversation

Flashing UBIFS image to gluebi mtd

DISCUSSION Konstantin Tokarev 3 participants 8 messages Oct 31, 2013 – Nov 1, 2013 Memory technology devices (MTD)

lore
Share & tools
lore
#0 lore raw

Hi all,

I'm migrating rootfs of device from SquashFS on top of gluebi to UBIFS. However, old version of firmware upgrade program writes UBIFS image to gluebi mtd device instead of using recommended volume update procedure. While it works for SquashFS images, writing UBIFS image leads to the next errors messages in random moments:

UBI error: ubi_io_read: error -77 while reading 76 bytes from PEB 769:26000, read 0 bytes UBIFS error (pid 383): ubifs_read_node: cannot read node 2 from LEB 347:21904, error -5 UBIFS warning (pid 383): ubifs_ro_mode: switched to read-only mode, error -5

However, when device is rebooted first time after upgrade, UBIFS mounts successfully.

Do you have any idea what is going on and how to destinguish "broken" UBIFS volume from correctly flashed via ubi volume device?

Signature · 2 lines
Regards,
Konstantin
#1 lore raw

Hi Konstantin,

Quoted 2 lines · Konstantin Tokarev

I'm migrating rootfs of device from SquashFS on top of gluebi to UBIFS.

Sounds like you could use ubiblock for that setup, which is suitable for squashfs, and completely independent of ubifs (thus less cpu and memory hungry and easier to setup).

You can find some patches floating around, but the *latest* work is here:

http://git.free-electrons.com/users/ezequiel-garcia/linux/log/?h=ubiblock-v4-cache-optional

And you'll need a patch for mtd-utils as well:

http://git.free-electrons.com/users/ezequiel-garcia/mtd-utils

Feel free to use it and report any issues. However, be very careful as the driver is a bit experimental yet.

Others are already using it, so it should be more or less fine!

Signature · 3 lines
Ezequiel García, Free Electrons
Embedded Linux, Kernel and Android Engineering
http://free-electrons.com
#2 lore raw

31.10.2013, 19:08, "Ezequiel Garcia" <[email protected]>:

Quoted 9 lines

Hi Konstantin,

Quoted 1 line · Konstantin Tokarev

 I'm migrating rootfs of device from SquashFS on top of gluebi to UBIFS.

Sounds like you could use ubiblock for that setup, which is suitable for squashfs, and completely independent of ubifs (thus less cpu and memory hungry and easier to setup).

Could you elaborate this a bit more? Do you mean that block device emultaion layer on top of UBI + FS on top of that can work faster than UBIFS on top of UBI?

Btw, I'm more interested in finding out in what way my UBIFS volume can be broken in given scenario and how can I detect it (to reflash once again, using UBI-aware updater this time).

Signature · 2 lines
Regards,
Konstantin
#3 lore raw
Quoted 9 lines · Konstantin Tokarev

31.10.2013, 19:08, "Ezequiel Garcia" <[email protected]>:

Quoted 6 lines

Hi Konstantin,

Quoted 1 line · Konstantin Tokarev

 I'm migrating rootfs of device from SquashFS on top of gluebi to UBIFS.

Quoted 8 lines
Quoted 3 lines

Sounds like you could use ubiblock for that setup, which is suitable for squashfs, and completely independent of ubifs (thus less cpu and memory hungry and easier to setup).

Could you elaborate this a bit more? Do you mean that block device emultaion layer on top of UBI + FS on top of that can work faster than UBIFS on top of UBI?

Given you provided no details, I'm not sure why you're migrating from squashfs (which is a read-only, block oriented fs) to ubifs (which supports r/w on MTD). But anyway, let me do this homework for you :-)

So, you had squashfs on top of gluebi, right? I guess you had this:

mtd (plus your NAND driver) -> ubi -> gluebi/mtd -> mtdblock -> squashfs

On the other side, using ubiblock to emulate the block device you'd end up with this:

mtd (plus your NAND driver) -> ubi -> ubiblock -> squashfs

Which is less memory (cpu?) hungry than the first alternative. The latest git branch also contained a non-cached option which has shown some good behavior, given squashfs does its own caching.

But of course, this is _NOT_ a replacement for UBIFS, and using ubiblock for R/W is strictly discouraged.

And here's the (already ancient) discussion:

http://lists.infradead.org/pipermail/linux-mtd/2012-November/045011.html

And the v2:
http://lists.infradead.org/pipermail/linux-mtd/2012-December/045274.html

I'll post the latest work as soon as I find some time, but you're free to pursue your own research.

Hope it helps and best of lucks!

Signature · 3 lines
Ezequiel García, Free Electrons
Embedded Linux, Kernel and Android Engineering
http://free-electrons.com
#4 lore raw
In reply to Konstantin Tokarev
Quoted 13 lines · Konstantin Tokarev

31.10.2013, 19:08, "Ezequiel Garcia" <[email protected]>:

Quoted 9 lines

Hi Konstantin,

Quoted 1 line · Konstantin Tokarev

 I'm migrating rootfs of device from SquashFS on top of gluebi to UBIFS.

Sounds like you could use ubiblock for that setup, which is suitable for squashfs, and completely independent of ubifs (thus less cpu and memory hungry and easier to setup).

Quoted 7 lines

Could you elaborate this a bit more? Do you mean that block device emultaion layer on top of UBI + FS on top of that can work faster than UBIFS on top of UBI?

Btw, I'm more interested in finding out in what way my UBIFS volume can be broken in given scenario and how can I detect it (to reflash once again, using UBI-aware updater this time).

You didn't mention how you overwrote your UBI partition with new data. dd ? nandwrite ? I'm getting the impression your problem is based on the fact that when updating the firmware, you are using update firmware which is not UBI aware, and you don't get UBI-awareness on the system until you've done the update. A chicken-and-egg-situation. One solution in that case would be to have an intermediate firmware which was identical to what was in the device to start with but that had an UBI-aware upgrade application.

Also I don't think you mentioned exactly your old setup. I'm assuming squashfs on top of mtd, with no UBI involved, right?

/Ricard

Signature · 3 lines
Ricard Wolf Wanderlöf                           ricardw(at)axis.com
Axis Communications AB, Lund, Sweden            www.axis.com
Phone +46 46 272 2016                           Fax +46 46 13 61 30
#5 lore raw

01.11.2013, 11:48, "Ricard Wanderlof" <[email protected]>:

Quoted 14 lines · Konstantin Tokarev
Quoted 13 lines

 31.10.2013, 19:08, "Ezequiel Garcia" <[email protected]>:

Quoted 7 lines

 Hi Konstantin,

Quoted 1 line · Konstantin Tokarev

  I'm migrating rootfs of device from SquashFS on top of gluebi to UBIFS.

 Sounds like you could use ubiblock for that setup, which is suitable for squashfs, and completely independent of ubifs (thus less cpu and memory hungry and easier to setup).

 Could you elaborate this a bit more? Do you mean that block device emultaion layer on top of UBI + FS on top of that can work faster than UBIFS on top of UBI?

Quoted 6 lines
Quoted 3 lines

 Btw, I'm more interested in finding out in what way my UBIFS volume can be broken in given scenario and how can I detect it (to reflash once again, using UBI-aware updater this time).

You didn't mention how you overwrote your UBI partition with new data. dd ? nandwrite ?

I write data to mtd device file (gluebi-based in this case) using write() function, one mtd eraseblock at a time. Effectively it's the same as dd.

Quoted 4 lines

I'm getting the impression your problem is based on the fact that when updating the firmware, you are using update firmware which is not UBI aware, and you don't get UBI-awareness on the system until you've done the update.

Exactly.

Quoted 3 lines

A chicken-and-egg-situation. One solution in that case
would be to have an intermediate firmware which was identical to what was in the device to start with but that had an UBI-aware upgrade application.

Not a solution, because end user of device might skip this intermediate firmware accidentally.

It would be perfectly OK if rootfs failed to mount after update because it led to reflash once again with the same firmware. It would be possible to do any new consistency check, because initramfs software had already been upgraded.

The problem is that incorrectly flashed UBIFS mounts without error, all content is readable and has valid md5sums, and file write and mv operations also succeed.

Quoted 3 lines

Also I don't think you mentioned exactly your old setup. I'm assuming squashfs on top of mtd, with no UBI involved, right?

No, on top of UBI (to handle bad blocks and wear leveling).

Signature · 2 lines
Regards,
Konstantin
#6 lore raw
Quoted 9 lines · Konstantin Tokarev
Quoted 6 lines
Quoted 3 lines

 Btw, I'm more interested in finding out in what way my UBIFS volume can be broken in given scenario and how can I detect it (to reflash once again, using UBI-aware updater this time).

You didn't mention how you overwrote your UBI partition with new data. dd ? nandwrite ?

I write data to mtd device file (gluebi-based in this case) using write() function, one mtd eraseblock at a time. Effectively it's the same as dd.

How do you handle potential bad blocks in the flash ?

Quoted 4 lines
Quoted 2 lines

Also I don't think you mentioned exactly your old setup. I'm assuming squashfs on top of mtd, with no UBI involved, right?

No, on top of UBI (to handle bad blocks and wear leveling).

Sorry, my bad, reading your post again I see that you explicitly stated squashfs on top of gluebi ...

Quoted 6 lines
Quoted 4 lines

I'm getting the impression your problem is based on the fact that when updating the firmware, you are using update firmware which is not UBI aware, and you don't get UBI-awareness on the system until you've done the update.

Exactly.

... which means that you've been running UBI all the time, but not ubifs. (?)

Quoted 6 lines
Quoted 3 lines

A chicken-and-egg-situation. One solution in that case
would be to have an intermediate firmware which was identical to what was in the device to start with but that had an UBI-aware upgrade application.

Not a solution, because end user of device might skip this intermediate firmware accidentally.

I was afraid of that ... :-)

Quoted 9 lines

It would be perfectly OK if rootfs failed to mount after update because it led to reflash once again with the same firmware. It would be possible to do any new consistency check, because initramfs software had already been upgraded.

The problem is that incorrectly flashed UBIFS mounts without error, all content is readable and has valid md5sums, and file write and mv operations also succeed.

But in your original post, the following line was present in the log:

UBIFS warning (pid 383): ubifs_ro_mode: switched to read-only mode, error -5

indicating that writing had been switched off due to the errors detected. Of course, this might be arbitrary depending on exactly which errors happened to occur at that particular boot instance.

/Ricard

Signature · 3 lines
Ricard Wolf Wanderlöf                           ricardw(at)axis.com
Axis Communications AB, Lund, Sweden            www.axis.com
Phone +46 46 272 2016                           Fax +46 46 13 61 30
#7 lore raw

01.11.2013, 13:43, "Ricard Wanderlof" <[email protected]>:

Quoted 9 lines · Konstantin Tokarev
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

   Btw, I'm more interested in finding out in what way my UBIFS volume can be broken in given scenario and how can I detect it (to reflash once again, using UBI-aware updater this time).

  You didn't mention how you overwrote your UBI partition with new data. dd ? nandwrite ?

  I write data to mtd device file (gluebi-based in this case) using write() function, one mtd eraseblock at a time. Effectively it's the same as dd.

 How do you handle potential bad blocks in the flash ?

Bad blocks are detected using ioctl on mtd device (MEMGETBADBLOCK) and skipped, though AFAIU gluebi mtd cannot have bad blocks.

Quoted 5 lines
Quoted 3 lines
Quoted 2 lines

  Also I don't think you mentioned exactly your old setup. I'm assuming squashfs on top of mtd, with no UBI involved, right?

  No, on top of UBI (to handle bad blocks and wear leveling).

 Sorry, my bad, reading your post again I see that you explicitly stated squashfs on top of gluebi ...

Quoted 7 lines
Quoted 5 lines
Quoted 4 lines

  I'm getting the impression your problem is based on the fact that when updating the firmware, you are using update firmware which is not UBI aware, and you don't get UBI-awareness on the system until you've done the update.

  Exactly.

 ... which means that you've been running UBI all the time, but not ubifs. (?)

Yes.

I guess that we should have written SquashFS images using UBI volume update procedure as well, to UBI device instead of gluebi MTD. Code was written to write filesystems like JFFS2 to raw MTD's and just reused here without second thought.

Quoted 14 lines
Quoted 5 lines
Quoted 3 lines

  A chicken-and-egg-situation. One solution in that case
  would be to have an intermediate firmware which was identical to what was in the device to start with but that had an UBI-aware upgrade application.

  Not a solution, because end user of device might skip this intermediate firmware accidentally.

 I was afraid of that ... :-)

Quoted 8 lines

  It would be perfectly OK if rootfs failed to mount after update because it led to reflash once again with the same firmware. It would be possible to do any new consistency check, because initramfs software had already been upgraded.

  The problem is that incorrectly flashed UBIFS mounts without error, all content is readable and has valid md5sums, and file write and mv operations also succeed.

Quoted 8 lines

 But in your original post, the following line was present in the log:

 UBIFS warning (pid 383): ubifs_ro_mode: switched to read-only mode, error -5

 indicating that writing had been switched off due to the errors detected. Of course, this might be arbitrary depending on exactly which errors happened to occur at that particular boot instance.

I've read code of gluebi_write in gluebi.c, vol_cdev_ioctl in cdev.c, and ubi_start_update in upd.c - looks like the only difference between correct update and what is done in my case is that volume is not wiped out before writing new volume image here.

(I know that raw mtd on NAND should be erased as well, but this is not done for gluebi MTD because our code erases MTD only if MTD type == MTD_NANDFLASH)

Signature · 2 lines
Regards,
Konstantin