Skip to contentSkip to conversation

dfu: dfu and UBI Volumes

U-Boot Heiko Schocher 6 participants 35 messages May 24, 2013 – May 29, 2013

lore
Share & tools
lore
#0 lore raw

Hello,

just digging in DFU support in U-Boot for an upcoming board support based on an AM335x. This board support uses for example a rootfs in an UBI Volume on a NAND flash, and this should be updated with dfu ...

How To do this? Current state on this board is to erase the rootfs mtd partition with a nand erase and write the new image using dfu_nand.c ... which calls in the end nand_write ... which is ... lets say ... not the prefered way on an UBI volume ...

How to solve this? Any ideas?

bye,
Heiko

Signature · 2 lines
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
#1 lore raw

Hi Heiko,

Quoted 6 lines · Heiko Schocher

Hello,

just digging in DFU support in U-Boot for an upcoming board support based on an AM335x. This board support uses for example a rootfs in an UBI Volume on a NAND flash, and this should be updated with dfu ...

Quoted 7 lines

How To do this? Current state on this board is to erase the rootfs mtd partition with a nand erase and write the new image using dfu_nand.c ... which calls in the end nand_write ... which is ... lets say ... not the prefered way on an UBI volume ...

How to solve this? Any ideas?

Well, what would you like ideally to do? Why is nand_write not ideal for a UBI volume.

Note that dfu will skip over the bad blocks...

Quoted 5 lines

bye,
Heiko

Signature · 2 lines
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany

Regards

-- Pantelis

#2 lore raw
Quoted 15 lines · Pantelis Antoniou

Hi Heiko,

Quoted 11 lines · Heiko Schocher

Hello,

just digging in DFU support in U-Boot for an upcoming board support based on an AM335x. This board support uses for example a rootfs in an UBI Volume on a NAND flash, and this should be updated with dfu ...

How To do this? Current state on this board is to erase the rootfs mtd partition with a nand erase and write the new image using dfu_nand.c ... which calls in the end nand_write ... which is ... lets say ... not the prefered way on an UBI volume ...

Quoted 6 lines
Quoted 1 line

How to solve this? Any ideas?

Well, what would you like ideally to do? Why is nand_write not ideal for a UBI volume.

Note that dfu will skip over the bad blocks...

Presumably because they want to replace say ubi0:rootfs (and leave ubi0:user-data and ubi0:u-boot-env and so forth alone) rather than write in a new ubi container of everything.

I would suggest that, so long as our existing UBI infrastructure allows this, you add a new method, dfu_ubi which takes care of programming things. This shouldn't be too bad to write as I've heard the existing infrastucture was easily expanded for SPI (and patches are pending a little more clean up prior to posting).

Signature · 8 lines
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20130524/5bd80b11/attachment.pgp>
#3 lore raw

Hello Tom,

Am 24.05.2013 19:12, schrieb Tom Rini:

Quoted 27 lines · Pantelis Antoniou
Quoted 15 lines

Hi Heiko,

Quoted 11 lines · Heiko Schocher

Hello,

just digging in DFU support in U-Boot for an upcoming board support based on an AM335x. This board support uses for example a rootfs in an UBI Volume on a NAND flash, and this should be updated with dfu ...

How To do this? Current state on this board is to erase the rootfs mtd partition with a nand erase and write the new image using dfu_nand.c ... which calls in the end nand_write ... which is ... lets say ... not the prefered way on an UBI volume ...

Quoted 6 lines
Quoted 1 line

How to solve this? Any ideas?

Well, what would you like ideally to do? Why is nand_write not ideal for a UBI volume.

Note that dfu will skip over the bad blocks...

Presumably because they want to replace say ubi0:rootfs (and leave ubi0:user-data and ubi0:u-boot-env and so forth alone) rather than write in a new ubi container of everything.

Quoted 5 lines

I would suggest that, so long as our existing UBI infrastructure allows this, you add a new method, dfu_ubi which takes care of programming things. This shouldn't be too bad to write as I've heard the existing infrastucture was easily expanded for SPI (and patches are pending a little more clean up prior to posting).

This sounds easy ... but they have also raw nand partitions, for example spl partitions on one nand flash ... for which dfu_nand.c fits perfectly ... is it possible to use dfu_nand.c and another dfu_xxx.c at the same time?

I would say no, if I understand the code right:
- starting dfu with "dfu interface={nand or mmc} ..."

and after that, you can not switch anymore between nand or mmc, right?

You must press Ctrl-C to end dfu and start it again for switching to another interface ...

Or should I look to add ubi support to drivers/dfu/dfu_nand.c?

Something like:
Add in dfu_fill_entity_nand a "else if (!strcmp(st, "ubipart")) {" (Did not looked deeper in this, if this is a possible way... and
thinking about it ... it is not the correct way, ubi should be seperate, because it maybe runs also on nor flash ...)

So, the best way would be, to switch in "dfu nand" mode between subinterfaces ... like "raw", "part", "ubi", ...

Ah, looking in drivers/dfu/dfu_mmc.c, they use dfu->layout for switching between DFU_RAW_ADDR, DFU_FS_FAT, DFU_FS_EXT4...

After all ... should we add a DFU_UBI and add this to drivers/dfu/dfu_nand.c?

bye,
Heiko

Signature · 2 lines
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
#4 lore raw

Hi Heiko,

Quoted 37 lines

Hello Tom,

Am 24.05.2013 19:12, schrieb Tom Rini:

Quoted 28 lines · Pantelis Antoniou
Quoted 16 lines

Hi Heiko,

Quoted 12 lines · Heiko Schocher

Hello,

just digging in DFU support in U-Boot for an upcoming board support based on an AM335x. This board support uses for example a rootfs in an UBI Volume on a NAND flash, and this should be updated with dfu ...

How To do this? Current state on this board is to erase the rootfs mtd partition with a nand erase and write the new image using dfu_nand.c ... which calls in the end nand_write ... which is ... lets say ... not the prefered way on an UBI volume ...

Quoted 6 lines
Quoted 1 line

How to solve this? Any ideas?

Well, what would you like ideally to do? Why is nand_write not ideal for a UBI volume.

Note that dfu will skip over the bad blocks...

Presumably because they want to replace say ubi0:rootfs (and leave ubi0:user-data and ubi0:u-boot-env and so forth alone) rather than write in a new ubi container of everything.

Quoted 5 lines

I would suggest that, so long as our existing UBI infrastructure allows this, you add a new method, dfu_ubi which takes care of programming things. This shouldn't be too bad to write as I've heard the existing infrastucture was easily expanded for SPI (and patches are pending a little more clean up prior to posting).

Quoted 4 lines

This sounds easy ... but they have also raw nand partitions, for example spl partitions on one nand flash ... for which dfu_nand.c fits perfectly ... is it possible to use dfu_nand.c and another dfu_xxx.c at the same time?

I'm not so familiar with nand devices handling, but in my opinion we shall create dfu_ubi.c file.

I think that, nand part of dfu handling shall be separated from ubi, even if UBI itself is layed on nand.

However, Tom and Pantelis shall give their opinion, since they spent a lot of time on forcing dfu_nand.c to work.

Quoted 28 lines

I would say no, if I understand the code right:
- starting dfu with "dfu interface={nand or mmc} ..."

and after that, you can not switch anymore between nand or mmc, right?

You must press Ctrl-C to end dfu and start it again for switching to another interface ...

Or should I look to add ubi support to drivers/dfu/dfu_nand.c?

Something like:
Add in dfu_fill_entity_nand a "else if (!strcmp(st, "ubipart")) {" (Did not looked deeper in this, if this is a possible way... and
thinking about it ... it is not the correct way, ubi should be seperate, because it maybe runs also on nor flash ...)

So, the best way would be, to switch in "dfu nand" mode between subinterfaces ... like "raw", "part", "ubi", ...

Ah, looking in drivers/dfu/dfu_mmc.c, they use dfu->layout for switching between DFU_RAW_ADDR, DFU_FS_FAT, DFU_FS_EXT4...

After all ... should we add a DFU_UBI and add this to drivers/dfu/dfu_nand.c?

bye,
Heiko

Signature · 5 lines
Best regards,

Lukasz Majewski

Samsung R&D Poland (SRPOL) | Linux Platform Group
#5 lore raw

Hello Lukasz,

Am 27.05.2013 09:02, schrieb Lukasz Majewski:

Quoted 44 lines

Hi Heiko,

Quoted 37 lines

Hello Tom,

Am 24.05.2013 19:12, schrieb Tom Rini:

Quoted 28 lines · Pantelis Antoniou
Quoted 16 lines

Hi Heiko,

Quoted 12 lines · Heiko Schocher

Hello,

just digging in DFU support in U-Boot for an upcoming board support based on an AM335x. This board support uses for example a rootfs in an UBI Volume on a NAND flash, and this should be updated with dfu ...

How To do this? Current state on this board is to erase the rootfs mtd partition with a nand erase and write the new image using dfu_nand.c ... which calls in the end nand_write ... which is ... lets say ... not the prefered way on an UBI volume ...

Quoted 6 lines
Quoted 1 line

How to solve this? Any ideas?

Well, what would you like ideally to do? Why is nand_write not ideal for a UBI volume.

Note that dfu will skip over the bad blocks...

Presumably because they want to replace say ubi0:rootfs (and leave ubi0:user-data and ubi0:u-boot-env and so forth alone) rather than write in a new ubi container of everything.

Quoted 5 lines

I would suggest that, so long as our existing UBI infrastructure allows this, you add a new method, dfu_ubi which takes care of programming things. This shouldn't be too bad to write as I've heard the existing infrastucture was easily expanded for SPI (and patches are pending a little more clean up prior to posting).

Quoted 4 lines

This sounds easy ... but they have also raw nand partitions, for example spl partitions on one nand flash ... for which dfu_nand.c fits perfectly ... is it possible to use dfu_nand.c and another dfu_xxx.c at the same time?

Quoted 5 lines

I'm not so familiar with nand devices handling, but in my opinion we shall create dfu_ubi.c file.

I think that, nand part of dfu handling shall be separated from ubi, even if UBI itself is layed on nand.

Yes, I tend also to this .. but not as a separte "dfu interface .." "dfu nand .." should stay, and each partition should know, if it is a raw nand, or UBI, or jffs2?,... as like in dfu_mmc.c ... but this should not be hardcoded in every dfu_xxx.c, instead it should be something like a subinterface ... if it is possible.

Quoted 2 lines

However, Tom and Pantelis shall give their opinion, since they spent a lot of time on forcing dfu_nand.c to work.

Yep, that would be great, as I am a dfu beginner ;-)

bye,
Heiko

Signature · 2 lines
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
#6 lore raw

Hi Guys,

Quoted 50 lines · Heiko Schocher

Hello Lukasz,

Am 27.05.2013 09:02, schrieb Lukasz Majewski:

Quoted 47 lines

Hi Heiko,

Quoted 37 lines

Hello Tom,

Am 24.05.2013 19:12, schrieb Tom Rini:

Quoted 28 lines · Pantelis Antoniou
Quoted 16 lines

Hi Heiko,

Quoted 12 lines · Heiko Schocher

Hello,

just digging in DFU support in U-Boot for an upcoming board support based on an AM335x. This board support uses for example a rootfs in an UBI Volume on a NAND flash, and this should be updated with dfu ...

How To do this? Current state on this board is to erase the rootfs mtd partition with a nand erase and write the new image using dfu_nand.c ... which calls in the end nand_write ... which is ... lets say ... not the prefered way on an UBI volume ...

Quoted 6 lines
Quoted 1 line

How to solve this? Any ideas?

Well, what would you like ideally to do? Why is nand_write not ideal for a UBI volume.

Note that dfu will skip over the bad blocks...

Presumably because they want to replace say ubi0:rootfs (and leave ubi0:user-data and ubi0:u-boot-env and so forth alone) rather than write in a new ubi container of everything.

Quoted 5 lines

I would suggest that, so long as our existing UBI infrastructure allows this, you add a new method, dfu_ubi which takes care of programming things. This shouldn't be too bad to write as I've heard the existing infrastucture was easily expanded for SPI (and patches are pending a little more clean up prior to posting).

Quoted 4 lines

This sounds easy ... but they have also raw nand partitions, for example spl partitions on one nand flash ... for which dfu_nand.c fits perfectly ... is it possible to use dfu_nand.c and another dfu_xxx.c at the same time?

I'm not so familiar with nand devices handling, but in my opinion we shall create dfu_ubi.c file.

Quoted 9 lines
Quoted 2 lines

I think that, nand part of dfu handling shall be separated from ubi, even if UBI itself is layed on nand.

Yes, I tend also to this .. but not as a separte "dfu interface .." "dfu nand .." should stay, and each partition should know, if it is a raw nand, or UBI, or jffs2?,... as like in dfu_mmc.c ... but this should not be hardcoded in every dfu_xxx.c, instead it should be something like a subinterface ... if it is possible.

I'm not completely up to speed with UBI, but dfu_ubi seems to be the way to go for me.

Looks like it's simple enough; erase (but don't step over the wear counters) , write (but skip over the wear counters).

I don't know how smart you have to be with UBI version; be very careful when the binary format of UBI changes.

Quoted 10 lines
Quoted 2 lines

However, Tom and Pantelis shall give their opinion, since they spent a lot of time on forcing dfu_nand.c to work.

Yep, that would be great, as I am a dfu beginner ;-)

bye,
Heiko

Signature · 2 lines
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany

Regards

-- Pantelis

#7 lore raw

Hello,

Am 27.05.2013 09:35, schrieb Pantelis Antoniou:

Quoted 64 lines

Hi Guys,

Quoted 50 lines · Heiko Schocher

Hello Lukasz,

Am 27.05.2013 09:02, schrieb Lukasz Majewski:

Quoted 47 lines

Hi Heiko,

Quoted 37 lines

Hello Tom,

Am 24.05.2013 19:12, schrieb Tom Rini:

Quoted 28 lines · Pantelis Antoniou
Quoted 16 lines

Hi Heiko,

Quoted 12 lines · Heiko Schocher

Hello,

just digging in DFU support in U-Boot for an upcoming board support based on an AM335x. This board support uses for example a rootfs in an UBI Volume on a NAND flash, and this should be updated with dfu ...

How To do this? Current state on this board is to erase the rootfs mtd partition with a nand erase and write the new image using dfu_nand.c ... which calls in the end nand_write ... which is ... lets say ... not the prefered way on an UBI volume ...

Quoted 6 lines
Quoted 1 line

How to solve this? Any ideas?

Well, what would you like ideally to do? Why is nand_write not ideal for a UBI volume.

Note that dfu will skip over the bad blocks...

Presumably because they want to replace say ubi0:rootfs (and leave ubi0:user-data and ubi0:u-boot-env and so forth alone) rather than write in a new ubi container of everything.

Quoted 5 lines

I would suggest that, so long as our existing UBI infrastructure allows this, you add a new method, dfu_ubi which takes care of programming things. This shouldn't be too bad to write as I've heard the existing infrastucture was easily expanded for SPI (and patches are pending a little more clean up prior to posting).

Quoted 4 lines

This sounds easy ... but they have also raw nand partitions, for example spl partitions on one nand flash ... for which dfu_nand.c fits perfectly ... is it possible to use dfu_nand.c and another dfu_xxx.c at the same time?

I'm not so familiar with nand devices handling, but in my opinion we shall create dfu_ubi.c file.

Quoted 9 lines
Quoted 2 lines

I think that, nand part of dfu handling shall be separated from ubi, even if UBI itself is layed on nand.

Yes, I tend also to this .. but not as a separte "dfu interface .." "dfu nand .." should stay, and each partition should know, if it is a raw nand, or UBI, or jffs2?,... as like in dfu_mmc.c ... but this should not be hardcoded in every dfu_xxx.c, instead it should be something like a subinterface ... if it is possible.

Quoted 2 lines

I'm not completely up to speed with UBI, but dfu_ubi seems to be the way to go for me.

But how to handle a raw nand partition and a ubi partition on one nand?

If ubi is a new dfu interface, somebody must start dfu on u-boot with "dfu nand .." or "dfu ubi .." dependent on which partition has to be updated ... before using dfu-util on the host side ... and start dfu-util for the correct partition...

This seems not really userfriendly to me ... if I have to use the u-boot shell, why using dfu-util on a host pc instead executing tftp and update my images within u-boot?

Is ubi really a "interface" as nand or mmc ... ?

Quoted 2 lines

Looks like it's simple enough; erase (but don't step over the wear counters) , write (but skip over the wear counters).

Yep, or load the complete image in ram, and write it with "ubi write ..."

Quoted 2 lines

I don't know how smart you have to be with UBI version; be very careful when the binary format of UBI changes.

Thanks!

bye,
Heiko

Signature · 2 lines
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
#8 lore raw

Dear Heiko Schocher,

Quoted 10 lines · In message <[email protected]> you wrote:

But how to handle a raw nand partition and a ubi partition on one nand?

If ubi is a new dfu interface, somebody must start dfu on u-boot with "dfu nand .." or "dfu ubi .." dependent on which partition has to be updated ... before using dfu-util on the host side ... and start dfu-util for the correct partition...

This seems not really userfriendly to me ... if I have to use the

Indeed, this makes no sense and breaks the whole concept of DFU to be able to download a sinlge firmware image with one, simple command.

Quoted 1 line

Is ubi really a "interface" as nand or mmc ... ?

No, it is not. It could be considered a "partition type" at best.

Quoted 4 lines
Quoted 2 lines

Looks like it's simple enough; erase (but don't step over the wear counters) , write (but skip over the wear counters).

Yep, or load the complete image in ram, and write it with "ubi write ..."

Not "or". When dealing with UBI volumes, then "ubi write" (or the equivalent C API) is the way to go.

Best regards,

Wolfgang Denk

Signature · 5 lines
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
All a hacker needs is a tight PUSHJ, a loose pair of UUOs, and a warm
place to shift.
#9 lore raw

Hi

Quoted 21 lines · Wolfgang Denk

Dear Heiko Schocher,

Quoted 10 lines · In message <[email protected]> you wrote:

But how to handle a raw nand partition and a ubi partition on one nand?

If ubi is a new dfu interface, somebody must start dfu on u-boot with "dfu nand .." or "dfu ubi .." dependent on which partition has to be updated ... before using dfu-util on the host side ... and start dfu-util for the correct partition...

This seems not really userfriendly to me ... if I have to use the

Indeed, this makes no sense and breaks the whole concept of DFU to be able to download a sinlge firmware image with one, simple command.

Quoted 1 line

Is ubi really a "interface" as nand or mmc ... ?

No, it is not. It could be considered a "partition type" at best.

Quoted 9 lines
Quoted 4 lines
Quoted 2 lines

Looks like it's simple enough; erase (but don't step over the wear counters) , write (but skip over the wear counters).

Yep, or load the complete image in ram, and write it with "ubi write ..."

Not "or". When dealing with UBI volumes, then "ubi write" (or the equivalent C API) is the way to go.

I pretty much agree. UBI looks like it's partition type.

BTW, the whole point of DFU is not to store your image in RAM at all. There are very few systems that have that much RAM.

Quoted 10 lines

Best regards,

Wolfgang Denk

Signature · 5 lines
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
All a hacker needs is a tight PUSHJ, a loose pair of UUOs, and a warm
place to shift.

Regards

-- Pantelis

#10 lore raw
Quoted 35 lines · Pantelis Antoniou

Hi

Quoted 21 lines · Wolfgang Denk

Dear Heiko Schocher,

Quoted 10 lines · In message <[email protected]> you wrote:

But how to handle a raw nand partition and a ubi partition on one nand?

If ubi is a new dfu interface, somebody must start dfu on u-boot with "dfu nand .." or "dfu ubi .." dependent on which partition has to be updated ... before using dfu-util on the host side ... and start dfu-util for the correct partition...

This seems not really userfriendly to me ... if I have to use the

Indeed, this makes no sense and breaks the whole concept of DFU to be able to download a sinlge firmware image with one, simple command.

Quoted 1 line

Is ubi really a "interface" as nand or mmc ... ?

No, it is not. It could be considered a "partition type" at best.

Quoted 9 lines
Quoted 4 lines
Quoted 2 lines

Looks like it's simple enough; erase (but don't step over the wear counters) , write (but skip over the wear counters).

Yep, or load the complete image in ram, and write it with "ubi write ..."

Not "or". When dealing with UBI volumes, then "ubi write" (or the equivalent C API) is the way to go.

Quoted 4 lines

I pretty much agree. UBI looks like it's partition type.

BTW, the whole point of DFU is not to store your image in RAM at all. There are very few systems that have that much RAM.

This _may_ be the hard part for UBI. When doing raw block writes for NAND/MMC, we're able to write them out quickly and thus support images larger than RAM. But for filesystems we don't support that notion in general for write and so limit ourselves to 8MiB or so files. Fine for the most part, but not fine for UBI. It's possible that we can support this on UBI easier than we can on filesystems, but I just don't know.

Signature · 8 lines
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20130527/2f384bf4/attachment.pgp>
#11 lore raw

Dear Tom,

Quoted 5 lines · In message <20130527204127.GY17119@bill-the-cat> you wrote:

This _may_ be the hard part for UBI. When doing raw block writes for NAND/MMC, we're able to write them out quickly and thus support images larger than RAM. But for filesystems we don't support that notion in general for write and so limit ourselves to 8MiB or so files. Fine for

Where exactly is this 8 MB limit coming into play?

Quoted 2 lines

the most part, but not fine for UBI. It's possible that we can support this on UBI easier than we can on filesystems, but I just don't know.

I thought the only size limitation for images we can load is available system RAM? Is this not the case?

Best regards,

Wolfgang Denk

Signature · 6 lines
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
PoB = "Prisoner of Bill" -- those held captive, unwillingly or other-
wise, by the contemptible Microsoft monopoly.
         -- Tom Christiansen in <[email protected]>
#12 lore raw
Quoted 10 lines · Wolfgang Denk

Dear Tom,

Quoted 5 lines · In message <20130527204127.GY17119@bill-the-cat> you wrote:

This _may_ be the hard part for UBI. When doing raw block writes for NAND/MMC, we're able to write them out quickly and thus support images larger than RAM. But for filesystems we don't support that notion in general for write and so limit ourselves to 8MiB or so files. Fine for

Where exactly is this 8 MB limit coming into play?

In buffering the data. We cannot write a chunk of a file to a filesystem and then append to it, we don't have the API today.

Quoted 5 lines
Quoted 2 lines

the most part, but not fine for UBI. It's possible that we can support this on UBI easier than we can on filesystems, but I just don't know.

I thought the only size limitation for images we can load is available system RAM? Is this not the case?

It is the case for raw media and not the case for filesystems. If it will be the case for UBI depends on what API we have available to us in the UBI layer.

Signature · 8 lines
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20130527/ccb6cb98/attachment.pgp>
#13 lore raw

Dear Tom,

Quoted 5 lines · In message <20130527233735.GZ17119@bill-the-cat> you wrote:
Quoted 2 lines

Where exactly is this 8 MB limit coming into play?

In buffering the data. We cannot write a chunk of a file to a filesystem and then append to it, we don't have the API today.

Sorry, I still don't get it. Assuming I have a GiB of RAM, why can I not load a 256 MiB file to RAM, and then write it to a file system?

I have definitely dealt with images and files bigger than 8 MiB in thepast, so I really don't see where any buffer problem could be.

Best regards,

Wolfgang Denk

Signature · 5 lines
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
No question is too silly to ask. Of course, some  questions  are  too
silly to to answer...  - L. Wall & R. L. Schwartz, _Programming Perl_
#14 lore raw
Quoted 6 lines · Wolfgang Denk

Dear Tom,

Quoted 3 lines · In message <20130527233735.GZ17119@bill-the-cat> you wrote:
Quoted 2 lines

Where exactly is this 8 MB limit coming into play?

Quoted 8 lines
Quoted 2 lines

In buffering the data. We cannot write a chunk of a file to a filesystem and then append to it, we don't have the API today.

Sorry, I still don't get it. Assuming I have a GiB of RAM, why can I not load a 256 MiB file to RAM, and then write it to a file system?

I have definitely dealt with images and files bigger than 8 MiB in thepast, so I really don't see where any buffer problem could be.

I thought I might not have been clear about where this limit comes from, after I sent the email. The problem we have, and this is only for writing to a filesystem (_not_ writing of a filesystem) is that we do not have the API for appending to files, only create/overwrite. So we must read the whole file into memory, and then write it out. The DFU protocol doesn't have (I would swear anyhow) a part where it says "I'm about to send you a blob of X bytes", so we cannot know at the start how much data is coming our way.

Today we "solve" this with a statically defined CONFIG_SYS_DFU_MAX_FILE_SIZE. Looking at things again, I think this is buggy right now in that we need to also whack DFU_DATA_BUF_SIZE to also be that same value. Going forward, we may be able to switch this to (and both of these are off the top of my head) a getenv to see how much space to malloc, or just making it a malloc and adding some compile-time check to ensure that the malloc area is at least as big as CONFIG_SYS_DFU_MAX_FILE_SIZE.

Signature · 8 lines
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20130528/4a2e0ec7/attachment.pgp>
#15 lore raw

Hi Tom,

Quoted 25 lines · Tom Rini
Quoted 6 lines · Wolfgang Denk

Dear Tom,

Quoted 3 lines · In message <20130527233735.GZ17119@bill-the-cat> you wrote:
Quoted 2 lines

Where exactly is this 8 MB limit coming into play?

Quoted 8 lines
Quoted 2 lines

In buffering the data. We cannot write a chunk of a file to a filesystem and then append to it, we don't have the API today.

Sorry, I still don't get it. Assuming I have a GiB of RAM, why can I not load a 256 MiB file to RAM, and then write it to a file system?

I have definitely dealt with images and files bigger than 8 MiB in thepast, so I really don't see where any buffer problem could be.

I thought I might not have been clear about where this limit comes from, after I sent the email. The problem we have, and this is only for writing to a filesystem (_not_ writing of a filesystem) is that we do not have the API for appending to files, only create/overwrite. So we must read the whole file into memory, and then write it out. The DFU protocol doesn't have (I would swear anyhow) a part where it says "I'm about to send you a blob of X bytes", so we cannot know at the start how much data is coming our way.

Quoted 9 lines

Today we "solve" this with a statically defined CONFIG_SYS_DFU_MAX_FILE_SIZE. Looking at things again, I think this is buggy right now in that we need to also whack DFU_DATA_BUF_SIZE to also be that same value. Going forward, we may be able to switch this to (and both of these are off the top of my head) a getenv to see how much space to malloc, or just making it a malloc and adding some compile-time check to ensure that the malloc area is at least as big as CONFIG_SYS_DFU_MAX_FILE_SIZE.

Correct, the DFU protocol doesn't have a method to inform you before hand about the size of the transfer about to happen.

The only possible solution I see at this point is to have an environment variable, i.e. dfubuf that controls the size of the buffer.

Upon start of a dfu transfer we can allocate the buffer, and do our thing.

Quoted 2 lines
Signature · 1 line
Tom

Regards

-- Pantelis

#16 lore raw

Dear Pantelis Antoniou,

Quoted 39 lines · Pantelis Antoniou

Hi Tom,

Quoted 25 lines · Tom Rini
Quoted 6 lines · Wolfgang Denk

Dear Tom,

Quoted 3 lines · In message <20130527233735.GZ17119@bill-the-cat> you wrote:
Quoted 2 lines

Where exactly is this 8 MB limit coming into play?

Quoted 8 lines
Quoted 2 lines

In buffering the data. We cannot write a chunk of a file to a filesystem and then append to it, we don't have the API today.

Sorry, I still don't get it. Assuming I have a GiB of RAM, why can I not load a 256 MiB file to RAM, and then write it to a file system?

I have definitely dealt with images and files bigger than 8 MiB in thepast, so I really don't see where any buffer problem could be.

I thought I might not have been clear about where this limit comes from, after I sent the email. The problem we have, and this is only for writing to a filesystem (_not_ writing of a filesystem) is that we do not have the API for appending to files, only create/overwrite. So we must read the whole file into memory, and then write it out. The DFU protocol doesn't have (I would swear anyhow) a part where it says "I'm about to send you a blob of X bytes", so we cannot know at the start how much data is coming our way.

Quoted 9 lines

Today we "solve" this with a statically defined CONFIG_SYS_DFU_MAX_FILE_SIZE. Looking at things again, I think this is buggy right now in that we need to also whack DFU_DATA_BUF_SIZE to also be that same value. Going forward, we may be able to switch this to (and both of these are off the top of my head) a getenv to see how much space to malloc, or just making it a malloc and adding some compile-time check to ensure that the malloc area is at least as big as CONFIG_SYS_DFU_MAX_FILE_SIZE.

Quoted 8 lines

Correct, the DFU protocol doesn't have a method to inform you before hand about the size of the transfer about to happen.

The only possible solution I see at this point is to have an environment variable, i.e. dfubuf that controls the size of the buffer.

Upon start of a dfu transfer we can allocate the buffer, and do our thing.

I don't know the details of the DFU implementation in U-Boot, but the specification leaves the choice between programming the firmware on-the-fly during the download, and later during the manifestation phase (or a mix of both). Hence, there is not need for a global firmware buffer if U-Boot goes for the on-the-fly programming strategy. The only buffer constraint would be wTransferSize (chosen by U-Boot for the control endpoint) in that case. See "7. Manifestation Phase" on page 26 here:
http://www.usb.org/developers/devclass_docs/DFU_1.1.pdf

Of course this can't yet apply to writing files on file systems since the current API in U-Boot misses the append feature, but this could be applied to program raw memory partitions, including UBI images.

Best regards,
Beno?t

#17 lore raw

Hi Beno?t

Quoted 51 lines · Beno?t Th?baudeau

Dear Pantelis Antoniou,

Quoted 39 lines · Pantelis Antoniou

Hi Tom,

Quoted 25 lines · Tom Rini
Quoted 6 lines · Wolfgang Denk

Dear Tom,

Quoted 3 lines · In message <20130527233735.GZ17119@bill-the-cat> you wrote:
Quoted 2 lines

Where exactly is this 8 MB limit coming into play?

Quoted 8 lines
Quoted 2 lines

In buffering the data. We cannot write a chunk of a file to a filesystem and then append to it, we don't have the API today.

Sorry, I still don't get it. Assuming I have a GiB of RAM, why can I not load a 256 MiB file to RAM, and then write it to a file system?

I have definitely dealt with images and files bigger than 8 MiB in thepast, so I really don't see where any buffer problem could be.

I thought I might not have been clear about where this limit comes from, after I sent the email. The problem we have, and this is only for writing to a filesystem (_not_ writing of a filesystem) is that we do not have the API for appending to files, only create/overwrite. So we must read the whole file into memory, and then write it out. The DFU protocol doesn't have (I would swear anyhow) a part where it says "I'm about to send you a blob of X bytes", so we cannot know at the start how much data is coming our way.

Quoted 9 lines

Today we "solve" this with a statically defined CONFIG_SYS_DFU_MAX_FILE_SIZE. Looking at things again, I think this is buggy right now in that we need to also whack DFU_DATA_BUF_SIZE to also be that same value. Going forward, we may be able to switch this to (and both of these are off the top of my head) a getenv to see how much space to malloc, or just making it a malloc and adding some compile-time check to ensure that the malloc area is at least as big as CONFIG_SYS_DFU_MAX_FILE_SIZE.

Quoted 8 lines

Correct, the DFU protocol doesn't have a method to inform you before hand about the size of the transfer about to happen.

The only possible solution I see at this point is to have an environment variable, i.e. dfubuf that controls the size of the buffer.

Upon start of a dfu transfer we can allocate the buffer, and do our thing.

Quoted 9 lines

I don't know the details of the DFU implementation in U-Boot, but the specification leaves the choice between programming the firmware on-the-fly during the download, and later during the manifestation phase (or a mix of both). Hence, there is not need for a global firmware buffer if U-Boot goes for the on-the-fly programming strategy. The only buffer constraint would be wTransferSize (chosen by U-Boot for the control endpoint) in that case. See "7. Manifestation Phase" on page 26 here:
http://www.usb.org/developers/devclass_docs/DFU_1.1.pdf

The problem is not DFU TBH, it's that since we don't have an option to append to a file, we have to have the whole file transferred in RAM and written in one go. The raw medium dfu methods in u-boot don't have a problem.

Quoted 4 lines

Of course this can't yet apply to writing files on file systems since the current API in U-Boot misses the append feature, but this could be applied to program raw memory partitions, including UBI images.

It already happens for raw memory partitions, it's the UBI images being discussed.

Quoted 2 lines

Best regards,
Beno?t

Regards

-- Pantelis

#18 lore raw

Hi Pantelis,

Quoted 70 lines · Pantelis Antoniou

Hi Beno?t

Quoted 51 lines · Beno?t Th?baudeau

Dear Pantelis Antoniou,

Quoted 39 lines · Pantelis Antoniou

Hi Tom,

Quoted 25 lines · Tom Rini
Quoted 6 lines · Wolfgang Denk

Dear Tom,

Quoted 3 lines · In message <20130527233735.GZ17119@bill-the-cat> you wrote:
Quoted 2 lines

Where exactly is this 8 MB limit coming into play?

Quoted 8 lines
Quoted 2 lines

In buffering the data. We cannot write a chunk of a file to a filesystem and then append to it, we don't have the API today.

Sorry, I still don't get it. Assuming I have a GiB of RAM, why can I not load a 256 MiB file to RAM, and then write it to a file system?

I have definitely dealt with images and files bigger than 8 MiB in thepast, so I really don't see where any buffer problem could be.

I thought I might not have been clear about where this limit comes from, after I sent the email. The problem we have, and this is only for writing to a filesystem (_not_ writing of a filesystem) is that we do not have the API for appending to files, only create/overwrite. So we must read the whole file into memory, and then write it out. The DFU protocol doesn't have (I would swear anyhow) a part where it says "I'm about to send you a blob of X bytes", so we cannot know at the start how much data is coming our way.

Quoted 9 lines

Today we "solve" this with a statically defined CONFIG_SYS_DFU_MAX_FILE_SIZE. Looking at things again, I think this is buggy right now in that we need to also whack DFU_DATA_BUF_SIZE to also be that same value. Going forward, we may be able to switch this to (and both of these are off the top of my head) a getenv to see how much space to malloc, or just making it a malloc and adding some compile-time check to ensure that the malloc area is at least as big as CONFIG_SYS_DFU_MAX_FILE_SIZE.

Quoted 8 lines

Correct, the DFU protocol doesn't have a method to inform you before hand about the size of the transfer about to happen.

The only possible solution I see at this point is to have an environment variable, i.e. dfubuf that controls the size of the buffer.

Upon start of a dfu transfer we can allocate the buffer, and do our thing.

Quoted 10 lines

I don't know the details of the DFU implementation in U-Boot, but the specification leaves the choice between programming the firmware on-the-fly during the download, and later during the manifestation phase (or a mix of both). Hence, there is not need for a global firmware buffer if U-Boot goes for
the on-the-fly programming strategy. The only buffer constraint would be wTransferSize (chosen by U-Boot for the control endpoint) in that case. See "7. Manifestation Phase" on page 26 here:
http://www.usb.org/developers/devclass_docs/DFU_1.1.pdf

The problem is not DFU TBH, it's that since we don't have an option to append to a file, we have to have the whole file transferred in RAM and written in one go. The raw medium dfu methods in u-boot don't have a problem.

Quoted 8 lines
Quoted 5 lines

Of course this can't yet apply to writing files on file systems since the current API in U-Boot misses the append feature, but this could be applied to
program raw memory partitions, including UBI images.

It already happens for raw memory partitions, it's the UBI images being discussed.

But what does appending to a file has to do with programming a UBI image, which is a memory partition containing a whole file system? This is what I don't get in this discussion. Is it because of a restriction of the DFU API in U-Boot?

Best regards,
Beno?t

#19 lore raw

Hi

Quoted 79 lines · Beno?t Th?baudeau

Hi Pantelis,

Quoted 76 lines · Pantelis Antoniou

Hi Beno?t

Quoted 51 lines · Beno?t Th?baudeau

Dear Pantelis Antoniou,

Quoted 39 lines · Pantelis Antoniou

Hi Tom,

Quoted 25 lines · Tom Rini
Quoted 6 lines · Wolfgang Denk

Dear Tom,

Quoted 3 lines · In message <20130527233735.GZ17119@bill-the-cat> you wrote:
Quoted 2 lines

Where exactly is this 8 MB limit coming into play?

Quoted 8 lines
Quoted 2 lines

In buffering the data. We cannot write a chunk of a file to a filesystem and then append to it, we don't have the API today.

Sorry, I still don't get it. Assuming I have a GiB of RAM, why can I not load a 256 MiB file to RAM, and then write it to a file system?

I have definitely dealt with images and files bigger than 8 MiB in thepast, so I really don't see where any buffer problem could be.

I thought I might not have been clear about where this limit comes from, after I sent the email. The problem we have, and this is only for writing to a filesystem (_not_ writing of a filesystem) is that we do not have the API for appending to files, only create/overwrite. So we must read the whole file into memory, and then write it out. The DFU protocol doesn't have (I would swear anyhow) a part where it says "I'm about to send you a blob of X bytes", so we cannot know at the start how much data is coming our way.

Quoted 9 lines

Today we "solve" this with a statically defined CONFIG_SYS_DFU_MAX_FILE_SIZE. Looking at things again, I think this is buggy right now in that we need to also whack DFU_DATA_BUF_SIZE to also be that same value. Going forward, we may be able to switch this to (and both of these are off the top of my head) a getenv to see how much space to malloc, or just making it a malloc and adding some compile-time check to ensure that the malloc area is at least as big as CONFIG_SYS_DFU_MAX_FILE_SIZE.

Quoted 8 lines

Correct, the DFU protocol doesn't have a method to inform you before hand about the size of the transfer about to happen.

The only possible solution I see at this point is to have an environment variable, i.e. dfubuf that controls the size of the buffer.

Upon start of a dfu transfer we can allocate the buffer, and do our thing.

Quoted 10 lines

I don't know the details of the DFU implementation in U-Boot, but the specification leaves the choice between programming the firmware on-the-fly during the download, and later during the manifestation phase (or a mix of both). Hence, there is not need for a global firmware buffer if U-Boot goes for
the on-the-fly programming strategy. The only buffer constraint would be wTransferSize (chosen by U-Boot for the control endpoint) in that case. See "7. Manifestation Phase" on page 26 here:
http://www.usb.org/developers/devclass_docs/DFU_1.1.pdf

The problem is not DFU TBH, it's that since we don't have an option to append to a file, we have to have the whole file transferred in RAM and written in one go. The raw medium dfu methods in u-boot don't have a problem.

Quoted 5 lines

Of course this can't yet apply to writing files on file systems since the current API in U-Boot misses the append feature, but this could be applied to
program raw memory partitions, including UBI images.

Quoted 7 lines
Quoted 2 lines

It already happens for raw memory partitions, it's the UBI images being discussed.

But what does appending to a file has to do with programming a UBI image, which is a memory partition containing a whole file system? This is what I don't get in this discussion. Is it because of a restriction of the DFU API in U-Boot?

Don't expect a discussion on a mailing list to stay on topic for long :) We sort of drifted from UBI to the fixed sized buffer.

Quoted 2 lines

Best regards,
Beno?t

Regards

-- Pantelis

#20 lore raw
In reply to Benoît Thébaudeau

On Tue, May 28, 2013 at 06:31:41PM +0200, Beno??t Th??baudeau wrote:

[snip]

Quoted 3 lines

Of course this can't yet apply to writing files on file systems since the current API in U-Boot misses the append feature, but this could be applied to program raw memory partitions, including UBI images.

Correct. We can write a whole UBI image, today, of NAND size, regardless of DDR size. But modifying UBI volumes (so UBIFS or your kernel in UBI or ...) will depend on what the UBI API provides us today. Modifying files on UBIFS (say replacing the kernel that's in UBIFS rather than a UBI volume itself) is another wrinkle, due to a lack of filesystem append.

Signature · 8 lines
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20130528/849ae00d/attachment.pgp>
#21 lore raw

Dear Tom,

Quoted 7 lines · In message <20130528172309.GF5829@bill-the-cat> you wrote:
Quoted 4 lines

Of course this can't yet apply to writing files on file systems since the current API in U-Boot misses the append feature, but this could be applied to program raw memory partitions, including UBI images.

Correct. We can write a whole UBI image, today, of NAND size, regardless of DDR size. But modifying UBI volumes (so UBIFS or your

I don't think so. To write a UBI volume on an existing UBI device, you would use the "ubi write" command. This translates into a call of ubi_volume_write(char *volume, void *buf, size_t size) which means the size must be known before you start writing; as far as I can tell ubi_volume_write() does not support incremental write operations of individual "parts" of a volume.

Best regards,

Wolfgang Denk

Signature · 6 lines
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
Its always easier short term to pee in the pond
than install a toilet - it's just not a good long term plan.
          - Alan Cox in <[email protected]>
#22 lore raw
Quoted 11 lines · Wolfgang Denk

Dear Tom,

Quoted 7 lines · In message <20130528172309.GF5829@bill-the-cat> you wrote:
Quoted 4 lines

Of course this can't yet apply to writing files on file systems since the current API in U-Boot misses the append feature, but this could be applied to program raw memory partitions, including UBI images.

Correct. We can write a whole UBI image, today, of NAND size, regardless of DDR size. But modifying UBI volumes (so UBIFS or your

Quoted 6 lines

I don't think so. To write a UBI volume on an existing UBI device, you would use the "ubi write" command. This translates into a call of ubi_volume_write(char *volume, void *buf, size_t size) which means the size must be known before you start writing; as far as I can tell ubi_volume_write() does not support incremental write operations of individual "parts" of a volume.

OK. A very quick read of ubi_volume_write leads into the guts of the write being in drivers/mtd/ubi/upd.c and it feels like you could actually do the volume write in chunks with ubi_start_update() and ubi_more_update_data() (and maybe some other housekeeping bits). It might take a bit more work since it looks like looks like both functions rely on knowing the size at the start, but that's just to make sure the size will fit in the volume.

Signature · 8 lines
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20130528/7edf2c92/attachment.pgp>
#23 lore raw

Hello Tom,

Am 28.05.2013 23:16, schrieb Tom Rini:

Quoted 19 lines · Wolfgang Denk
Quoted 11 lines

Dear Tom,

Quoted 7 lines · In message <20130528172309.GF5829@bill-the-cat> you wrote:
Quoted 4 lines

Of course this can't yet apply to writing files on file systems since the current API in U-Boot misses the append feature, but this could be applied to program raw memory partitions, including UBI images.

Correct. We can write a whole UBI image, today, of NAND size, regardless of DDR size. But modifying UBI volumes (so UBIFS or your

Quoted 6 lines

I don't think so. To write a UBI volume on an existing UBI device, you would use the "ubi write" command. This translates into a call of ubi_volume_write(char *volume, void *buf, size_t size) which means the size must be known before you start writing; as far as I can tell ubi_volume_write() does not support incremental write operations of individual "parts" of a volume.

Quoted 7 lines

OK. A very quick read of ubi_volume_write leads into the guts of the write being in drivers/mtd/ubi/upd.c and it feels like you could actually do the volume write in chunks with ubi_start_update() and ubi_more_update_data() (and maybe some other housekeeping bits). It might take a bit more work since it looks like looks like both functions rely on knowing the size at the start, but that's just to make sure the size will fit in the volume.

Yes, I think so too ... seems some work, but not unsolveable ...

Hmm.. is it possible to get the filesize over dfu on startup? (I hope so, as it makes no sense to transfer a file, if it does not fit in the partition ... )

bye,
Heiko

Signature · 2 lines
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
#24 lore raw
Quoted 30 lines · Heiko Schocher

Hello Tom,

Am 28.05.2013 23:16, schrieb Tom Rini:

Quoted 19 lines · Wolfgang Denk
Quoted 11 lines

Dear Tom,

Quoted 7 lines · In message <20130528172309.GF5829@bill-the-cat> you wrote:
Quoted 4 lines

Of course this can't yet apply to writing files on file systems since the current API in U-Boot misses the append feature, but this could be applied to program raw memory partitions, including UBI images.

Correct. We can write a whole UBI image, today, of NAND size, regardless of DDR size. But modifying UBI volumes (so UBIFS or your

Quoted 6 lines

I don't think so. To write a UBI volume on an existing UBI device, you would use the "ubi write" command. This translates into a call of ubi_volume_write(char *volume, void *buf, size_t size) which means the size must be known before you start writing; as far as I can tell ubi_volume_write() does not support incremental write operations of individual "parts" of a volume.

Quoted 7 lines

OK. A very quick read of ubi_volume_write leads into the guts of the write being in drivers/mtd/ubi/upd.c and it feels like you could actually do the volume write in chunks with ubi_start_update() and ubi_more_update_data() (and maybe some other housekeeping bits). It might take a bit more work since it looks like looks like both functions rely on knowing the size at the start, but that's just to make sure the size will fit in the volume.

Quoted 5 lines

Yes, I think so too ... seems some work, but not unsolveable ...

Hmm.. is it possible to get the filesize over dfu on startup? (I hope so, as it makes no sense to transfer a file, if it does not fit in the partition ... )

There is not, but that's OK. Even if you think you had the room for the whole image you could find a new bad block and not fit, so it's just behaving like when we do a raw NAND write, get the data chunk, make sure we think we still have room, write if we do, error if we don't.

Signature · 8 lines
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20130529/7a06f8da/attachment.pgp>
#25 lore raw
In reply to Wolfgang Denk

Hello Wolfgang,

Am 27.05.2013 23:25, schrieb Wolfgang Denk:

Quoted 10 lines

Dear Tom,

Quoted 5 lines · In message <20130527204127.GY17119@bill-the-cat> you wrote:

This _may_ be the hard part for UBI. When doing raw block writes for NAND/MMC, we're able to write them out quickly and thus support images larger than RAM. But for filesystems we don't support that notion in general for write and so limit ourselves to 8MiB or so files. Fine for

Where exactly is this 8 MB limit coming into play?

You find this in drivers/dfu/dfu.c:

static unsigned char __aligned(CONFIG_SYS_CACHELINE_SIZE)

                                   dfu_buf[DFU_DATA_BUF_SIZE];

I had just a patch in my workqueue, where I made this value at least configurable ... but this limits not the transfer file size, see below...

Quoted 5 lines
Quoted 2 lines

the most part, but not fine for UBI. It's possible that we can support this on UBI easier than we can on filesystems, but I just don't know.

I thought the only size limitation for images we can load is available system RAM? Is this not the case?

As I read in code: Raw nand partitions fill the above buffer, if it is full -> flush to nand, and go on ... so no limit for them.

drivers/dfu/dfu_mmc.c use (another?, why?) buffer:

static unsigned char __aligned(CONFIG_SYS_CACHELINE_SIZE)

                                dfu_file_buf[CONFIG_SYS_DFU_MAX_FILE_SIZE];

and use this buffer for not raw partitions ... and this buffer gets flushed, only if the complete file is transfered, as the README states:

CONFIG_SYS_DFU_MAX_FILE_SIZE
When updating files rather than the raw storage device, we use a static buffer to copy the file into and then write the buffer once we've been given the whole file. Define this to the maximum filesize (in bytes) for the buffer. Default is 4 MiB if undefined.

bye,
Heiko

Signature · 2 lines
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
#26 lore raw

Dear Heiko,

Quoted 7 lines · In message <[email protected]> you wrote:
Quoted 2 lines

Where exactly is this 8 MB limit coming into play?

You find this in drivers/dfu/dfu.c:

static unsigned char __aligned(CONFIG_SYS_CACHELINE_SIZE)

                                   dfu_buf[DFU_DATA_BUF_SIZE];

Ah, so it is a DFU restriction!

Well, this is a design problem that needs to be fixed.

Quoted 2 lines

As I read in code: Raw nand partitions fill the above buffer, if it is full -> flush to nand, and go on ... so no limit for them.

So this boils down to the question whether there is some "incremental ubi write" method.

Hello UBI experts?

Quoted 4 lines

drivers/dfu/dfu_mmc.c use (another?, why?) buffer:

static unsigned char __aligned(CONFIG_SYS_CACHELINE_SIZE)

                                dfu_file_buf[CONFIG_SYS_DFU_MAX_FILE_SIZE];

...

Quoted 10 lines

and use this buffer for not raw partitions ... and this buffer gets flushed, only if the complete file is transfered, as the README states:

CONFIG_SYS_DFU_MAX_FILE_SIZE
When updating files rather than the raw storage device, we use a static buffer to copy the file into and then write the buffer once we've been given the whole file. Define this to the maximum filesize (in bytes) for the buffer. Default is 4 MiB if undefined.

This makes very little sense to me. Why do we need another (and even smaller) buffer when we already have one?

Best regards,

Wolfgang Denk

Signature · 5 lines
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
"We don't have to protect the environment -- the Second Coming is  at
hand."                                                   - James Watt
#27 lore raw
Quoted 12 lines · Wolfgang Denk

Dear Heiko,

Quoted 7 lines · In message <[email protected]> you wrote:
Quoted 2 lines

Where exactly is this 8 MB limit coming into play?

You find this in drivers/dfu/dfu.c:

static unsigned char __aligned(CONFIG_SYS_CACHELINE_SIZE)

                                   dfu_buf[DFU_DATA_BUF_SIZE];

Ah, so it is a DFU restriction!

[snip]

Quoted 16 lines
Quoted 4 lines

drivers/dfu/dfu_mmc.c use (another?, why?) buffer:

static unsigned char __aligned(CONFIG_SYS_CACHELINE_SIZE)

                                dfu_file_buf[CONFIG_SYS_DFU_MAX_FILE_SIZE];

...

Quoted 10 lines

and use this buffer for not raw partitions ... and this buffer gets flushed, only if the complete file is transfered, as the README states:

CONFIG_SYS_DFU_MAX_FILE_SIZE
When updating files rather than the raw storage device, we use a static buffer to copy the file into and then write the buffer once we've been given the whole file. Define this to the maximum filesize (in bytes) for the buffer. Default is 4 MiB if undefined.

Quoted 2 lines

This makes very little sense to me. Why do we need another (and even smaller) buffer when we already have one?

Per my other email, the intention and implementation didn't quite match-up. The intent of the README should be (but isn't) reflected) in the code. And perhaps we can come up with something better than a big static allocation. Perhaps.

Signature · 8 lines
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20130528/9ead52ad/attachment.pgp>
#28 lore raw
In reply to Tom Rini

Hello Tom,

Am 27.05.2013 22:41, schrieb Tom Rini:

Quoted 8 lines · Pantelis Antoniou
Quoted 7 lines

Hi

Quoted 3 lines · Wolfgang Denk

Dear Heiko Schocher,

In message <[email protected]> you wrote:

[...]

Quoted 11 lines
Quoted 4 lines

I pretty much agree. UBI looks like it's partition type.

BTW, the whole point of DFU is not to store your image in RAM at all. There are very few systems that have that much RAM.

This _may_ be the hard part for UBI. When doing raw block writes for NAND/MMC, we're able to write them out quickly and thus support images larger than RAM. But for filesystems we don't support that notion in general for write and so limit ourselves to 8MiB or so files. Fine for the most part, but not fine for UBI. It's possible that we can support this on UBI easier than we can on filesystems, but I just don't know.

I also do not know this at the moment :-(

Maybe a UBI expert can help here?

bye,
Heiko

Signature · 2 lines
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
#29 lore raw
In reply to Heiko Schocher

On Sun, May 26, 2013 at 09:09:22AM +0200, Heiko Schocher wrote:

[snip]

Quoted 5 lines

Ah, looking in drivers/dfu/dfu_mmc.c, they use dfu->layout for switching between DFU_RAW_ADDR, DFU_FS_FAT, DFU_FS_EXT4...

After all ... should we add a DFU_UBI and add this to drivers/dfu/dfu_nand.c?

Yes, I think we should be able to adapt dfu_nand to support raw (current) and UBI (which will need a little further handling so you can update per UBI container). In MMC (and there's examples in trats and am335x_evm) we say <name> ext4/fat/part/raw device#/start_blk part#/end_blk. I would imagine, but testing and implementation might show a better way, we do UBI as <name> ubi ubiN volume-name, ie:
rootfs ubi ubi0 rootfs
user ubi ubi0 user-data
and so forth, and augment dfu_nand.c with UBI knowledge, ala dfu_mmc and fat/ext knowledge.

Signature · 8 lines
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20130527/32a73a89/attachment.pgp>
#30 lore raw

Hello Tom,

Am 27.05.2013 21:21, schrieb Tom Rini:

Quoted 9 lines · Heiko Schocher

[snip]

Quoted 5 lines

Ah, looking in drivers/dfu/dfu_mmc.c, they use dfu->layout for switching between DFU_RAW_ADDR, DFU_FS_FAT, DFU_FS_EXT4...

After all ... should we add a DFU_UBI and add this to drivers/dfu/dfu_nand.c?

Quoted 10 lines

Yes, I think we should be able to adapt dfu_nand to support raw (current) and UBI (which will need a little further handling so you can update per UBI container). In MMC (and there's examples in trats and am335x_evm) we say <name> ext4/fat/part/raw device#/start_blk part#/end_blk. I would imagine, but testing and implementation might show a better way, we do UBI as <name> ubi ubiN volume-name, ie:
rootfs ubi ubi0 rootfs
user ubi ubi0 user-data
and so forth, and augment dfu_nand.c with UBI knowledge, ala dfu_mmc and fat/ext knowledge.

Yes, I think, thats the way to go ...

bye,
Heiko

Signature · 2 lines
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
#31 lore raw

Dear Heiko,

Quoted 9 lines · In message <[email protected]> you wrote:
Quoted 7 lines

I would imagine, but testing and implementation might show a better way, we do UBI as <name> ubi ubiN volume-name, ie:
rootfs ubi ubi0 rootfs
user ubi ubi0 user-data
and so forth, and augment dfu_nand.c with UBI knowledge, ala dfu_mmc and fat/ext knowledge.

Yes, I think, thats the way to go ...

I doubt that dfu_nand.c is the right place for this. What if I start using DFU on NOR flash? The decisions which device type (NAND, MMC, NOR, USB mass storage, ...) to habdle on one side, and which partition type / image type (raw, UBI volume, file, ...) on the other side are fully orthogonal to each other. They should be handled by independent code, and not one of them repeated for all implementations of the other.

Best regards,

Wolfgang Denk

Signature · 4 lines
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
A mouse is an elephant built by the Japanese.
#32 lore raw

Hello Wolfgang,

Am 28.05.2013 07:58, schrieb Wolfgang Denk:

Quoted 13 lines

Dear Heiko,

Quoted 9 lines · In message <[email protected]> you wrote:
Quoted 7 lines

I would imagine, but testing and implementation might show a better way, we do UBI as <name> ubi ubiN volume-name, ie:
rootfs ubi ubi0 rootfs
user ubi ubi0 user-data
and so forth, and augment dfu_nand.c with UBI knowledge, ala dfu_mmc and fat/ext knowledge.

Yes, I think, thats the way to go ...

Quoted 7 lines

I doubt that dfu_nand.c is the right place for this. What if I start using DFU on NOR flash? The decisions which device type (NAND, MMC, NOR, USB mass storage, ...) to habdle on one side, and which partition type / image type (raw, UBI volume, file, ...) on the other side are fully orthogonal to each other. They should be handled by independent code, and not one of them repeated for all implementations of the other.

Yes, exactly ...

bye,
Heiko

Signature · 2 lines
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
#33 lore raw
Quoted 24 lines · Heiko Schocher

Hello Wolfgang,

Am 28.05.2013 07:58, schrieb Wolfgang Denk:

Quoted 13 lines

Dear Heiko,

Quoted 9 lines · In message <[email protected]> you wrote:
Quoted 7 lines

I would imagine, but testing and implementation might show a better way, we do UBI as <name> ubi ubiN volume-name, ie:
rootfs ubi ubi0 rootfs
user ubi ubi0 user-data
and so forth, and augment dfu_nand.c with UBI knowledge, ala dfu_mmc and fat/ext knowledge.

Yes, I think, thats the way to go ...

Quoted 7 lines

I doubt that dfu_nand.c is the right place for this. What if I start using DFU on NOR flash? The decisions which device type (NAND, MMC, NOR, USB mass storage, ...) to habdle on one side, and which partition type / image type (raw, UBI volume, file, ...) on the other side are fully orthogonal to each other. They should be handled by independent code, and not one of them repeated for all implementations of the other.

Quoted 1 line

Yes, exactly ...

We can see about what re-org makes sense once we've got another implementation here. We might be able to share the filesystem-based write code, but that in part might cause some screams since it'll depend on our continued (ab)use of run_command. The device-specific code does
end up being the device-specific API part. Unless we're adding ubifs
in addition to ubi volume support to DFU, we don't have real shared filesystem stuff, yet.

Signature · 8 lines
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20130528/6494f8a1/attachment.pgp>
#34 lore raw
In reply to Pantelis Antoniou

Dear Pantelis Antoniou,

Quoted 10 lines · In message <[email protected]> you…
Quoted 7 lines

How To do this? Current state on this board is to erase the rootfs mtd partition with a nand erase and write the new image using dfu_nand.c ... which calls in the end nand_write ... which is ... lets say ... not the prefered way on an UBI volume ...

How to solve this? Any ideas?

Well, what would you like ideally to do? Why is nand_write not ideal for a UBI volume.

The key problem is that nand_erase will destroy all UBI wear levelling and erase counters. When working with UBI, you NEVER want to do this.

See for example [1] for reference.

Quoted 1 line

Note that dfu will skip over the bad blocks...

Which is even worse here.

DFU should be able to deal with UBI volumes, and using proper access routines (ubi write) to write the new volume(s). But never ever should any low level erase of the underlying flash device be needed nor used.

[1] http://www.linux-mtd.infradead.org/doc/ubi.html#L_format

Best regards,

Wolfgang Denk

Signature · 4 lines
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
What's the sound a name makes when it's dropped?