LZMA support

U-Boot-Users Luigi 'Comio' Mantellini 8 participants 14 messages Mar 28, 2008 – Apr 6, 2008

lore

Share & tools

lore

Luigi 'Comio' Mantellini

author

2008-03-28 17:18 UTC #1 lore raw

[U-Boot-Users] LZMA support (patch)

Hi everyone,

this patch adds the lzma support with the following constraints:

- it doesn't install the LZMA SDK files. I provide the import_lzma.sh
script to import the needs file from a lzma sdk release
- the imported lzma sdk files are _not touched_
- the lzma support is enabled defining the CONFIG_LZMA define
- I haven't tested this patch
- ?Use this patch at your own risk

The patch creates:

- the lib_lzma directory with the import_lzmasdk.sh script and a simple
front-end to LzmaDecode() fuction
- the include/lzma directory with some include files to use in the
U-Boot code

modifies:

- common/do_bootm.c file
- common/image.c file
- include/image.h file

Please read the lib_lzma/README.txt file.

Luigi

Quoted 37 lines · Luigi 'Comio' Mantellini

Hi people,
I have a stupid question: why does u-boot not support the LZMA algorithm?
(Anyway, I have a self-made and untested patch that introduces this compression algorithm).
Thanks a lot.
luigi
Industrie Dial Face S.p.A.
Luigi Mantellini
R&D - Software
Industrie Dial Face S.p.A.
Via Canzo, 4
20068 Peschiera Borromeo (MI), Italy
Tel.:   +39 02 5167 2813
Fax:    +39 02 5167 2459
E-mail: luigi.mantellini at idf-hit.com
GPG fingerprint: 3DD1 7B71 FBDF 6376
1B4A
                 B003 175F E979 907E
1650
-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace. It's the best place to buy or sell services for just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
_______________________________________________
U-Boot-Users mailing list
U-Boot-Users at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/u-boot-users

Signature · 20 lines

     ______       Luigi Mantellini
   .'______'.     R&D - Software
  (.'      '.)    Industrie Dial Face S.p.A.
  ( :=----=: )    Via Canzo, 4
  ('.______.')    20068 Peschiera Borromeo (MI), Italy
   '.______.'     Tel.: +39 02 5167 2813
                  Fax:  +39 02 5167 2459
Ind.  Dial Face   Email: luigi.mantellini at idf-hit.com
www.idf-hit.com   GPG fingerprint: 3DD1 7B71 FBDF 6376 1B4A
                                   B003 175F E979 907E 1650



-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-LZMA-support-for-kernel-images.patch
Type: application/mbox
Size: 15652 bytes
Desc: not available
Url : http://lists.denx.de/pipermail/u-boot/attachments/20080328/1cb1fe9e/attachment.bin

Jerry Van Baren

2008-03-28 17:31 UTC #2 lore raw

[U-Boot-Users] LZMA support (patch)

Quoted 20 lines · Luigi 'Comio' Mantellini

Hi everyone,
this patch adds the lzma support with the following constraints:
- it doesn't install the LZMA SDK files. I provide the import_lzma.sh
script to import the needs file from a lzma sdk release
- the imported lzma sdk files are _not touched_
- the lzma support is enabled defining the CONFIG_LZMA define
- I haven't tested this patch
- ?Use this patch at your own risk
The patch creates:
- the lib_lzma directory with the import_lzmasdk.sh script and a simple
front-end to LzmaDecode() fuction
- the include/lzma directory with some include files to use in the
U-Boot code
modifies:

Quoted 7 lines

- common/do_bootm.c file
- common/image.c file
- include/image.h file
Please read the lib_lzma/README.txt file.
Luigi

Hi Luigi,

I have a stupid question in return: why do we want lzma (7zip) added to u-boot? The linux/uboot world is dominated by gzip. I realize 7zip has advantages over gzip albeit incremental, not revolutionary IIRC.

I am not aware of proposals or implementations of kernels compressed with lzma (could well be my ignorance). Adding more code for minimal benefits is a hard sell (we already have plenty of bloat :-O).

Best regards,
gvb

Hoi-Ho Chan

2008-03-28 17:56 UTC #3 lore raw

[U-Boot-Users] LZMA support (patch)

Quoted 35 lines · Jerry Van Baren

Quoted 20 lines · Luigi 'Comio' Mantellini
Hi everyone,
this patch adds the lzma support with the following constraints:
- it doesn't install the LZMA SDK files. I provide the import_lzma.sh
script to import the needs file from a lzma sdk release
- the imported lzma sdk files are _not touched_
- the lzma support is enabled defining the CONFIG_LZMA define
- I haven't tested this patch
- ?Use this patch at your own risk
The patch creates:
- the lib_lzma directory with the import_lzmasdk.sh script and a simple
front-end to LzmaDecode() fuction
- the include/lzma directory with some include files to use in the
U-Boot code
modifies:
Quoted 7 lines
- common/do_bootm.c file
- common/image.c file
- include/image.h file
Please read the lib_lzma/README.txt file.
Luigi
Hi Luigi,
I have a stupid question in return: why do we want lzma (7zip) added to u-boot? The linux/uboot world is dominated by gzip. I realize 7zip has advantages over gzip albeit incremental, not revolutionary IIRC.

Quoted 8 lines

I am not aware of proposals or implementations of kernels compressed with lzma (could well be my ignorance). Adding more code for minimal benefits is a hard sell (we already have plenty of bloat :-O).
Best regards,
gvb

Would be great if it's being used for squashfs-lzma support.

Thanks
Donald

Luigi 'Comio' Mantellini

author

2008-03-28 18:06 UTC #4 lore raw

[U-Boot-Users] LZMA support (patch)

In reply to Jerry Van Baren

Hi Jerry,

see inline comments.

Quoted 27 lines · Jerry Van Baren

Quoted 26 lines · Luigi 'Comio' Mantellini
Hi everyone,
this patch adds the lzma support with the following constraints:
- it doesn't install the LZMA SDK files. I provide the import_lzma.sh
script to import the needs file from a lzma sdk release
- the imported lzma sdk files are _not touched_
- the lzma support is enabled defining the CONFIG_LZMA define
- I haven't tested this patch
- ?Use this patch at your own risk
The patch creates:
- the lib_lzma directory with the import_lzmasdk.sh script and a simple
front-end to LzmaDecode() fuction
- the include/lzma directory with some include files to use in the
U-Boot code
modifies:
- common/do_bootm.c file
- common/image.c file
- include/image.h file
Please read the lib_lzma/README.txt file.

Quoted 8 lines

Quoted 1 line
Luigi
Hi Luigi,
I have a stupid question in return: why do we want lzma (7zip) added to u-boot? The linux/uboot world is dominated by gzip. I realize 7zip has advantages over gzip albeit incremental, not revolutionary IIRC.

LZMA usually performs better than gzip in terms compression ratio. My application has a small flash memory and I need to keep two (2) kernel images...

From these observations born my need to save as much memory as possible.

Best regards,

luigi

Quoted 6 lines

I am not aware of proposals or implementations of kernels compressed with lzma (could well be my ignorance). Adding more code for minimal benefits is a hard sell (we already have plenty of bloat :-O).
Best regards,
gvb

Signature · 10 lines

     ______       Luigi Mantellini
   .'______'.     R&D - Software
  (.'      '.)    Industrie Dial Face S.p.A.
  ( :=----=: )    Via Canzo, 4
  ('.______.')    20068 Peschiera Borromeo (MI), Italy
   '.______.'     Tel.: +39 02 5167 2813
                  Fax:  +39 02 5167 2459
Ind.  Dial Face   Email: luigi.mantellini at idf-hit.com
www.idf-hit.com   GPG fingerprint: 3DD1 7B71 FBDF 6376 1B4A
                                   B003 175F E979 907E 1650

Wolfgang Denk

2008-03-28 19:13 UTC #5 lore raw

[U-Boot-Users] LZMA support (patch)

In reply to Luigi 'Comio' Mantellini

Quoted 5 lines · In message <1206724722.23627.28.camel@localhost> you wrote:

this patch adds the lzma support with the following constraints:
- it doesn't install the LZMA SDK files. I provide the import_lzma.sh
script to import the needs file from a lzma sdk release

Sorry, but I will not accept this.

U-Boot has always been self-contained, and I don't intend to change this.

Please make sure that a patch contains all files that belong to it.

Also, please explain what this new code is useful for. So far I don;t see any real-life use cases ?

Best regards,

Wolfgang Denk

Signature · 4 lines

DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
"I like your game but we have to change the rules."

Luigi 'Comio' Mantellini

author

2008-03-28 21:26 UTC #6 lore raw

[U-Boot-Users] LZMA support (patch)

Quoted 9 lines

Quoted 7 lines · In message <1206724722.23627.28.camel@localhost> you wrote:
this patch adds the lzma support with the following constraints:
- it doesn't install the LZMA SDK files. I provide the
import_lzma.sh
script to import the needs file from a lzma sdk release

Quoted 5 lines

Sorry, but I will not accept this.
U-Boot has always been self-contained, and I don't intend to change
this.

Running the contained script the directory will be populated with all need files (LzmaTypes.h, LzmaDecode.*). Anyway, I can resubmit the patch including all files. The LZMA SDK is released under LGPL with exceptions but I don't understand if the license is compatible with the u-boot's license. For this reason I omitted the sources from the SDK.

Quoted 9 lines

Please make sure that a patch contains all files that belong to it.
Also, please explain what this new code is useful for. So far I don;t
see any real-life use cases ?

LMZA performs better than LZ on binary files. I will switch to LZMA to save flash memory space on my appliance. Of course, the LZMA is slower than LZ.

Anyway, if the LZMA SDK license is compatible with U-boot , I think that LZMA should be chosen by firmware developer when the flash memory space is a critical resource.

My 2 Cents,

luigi

Quoted 13 lines

Best regards,
Wolfgang Denk
--
DENX Software Engineering GmbH, MD: Wolfgang Denk & Detlev
Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email:
wd at denx.de
"I like your game but we have to change the rules."

Wolfgang Denk

2008-03-28 22:33 UTC #7 lore raw

[U-Boot-Users] LZMA support (patch)

Quoted 7 lines · In message <[email protected]> you…

Running the contained script the directory will be populated with all need files (LzmaTypes.h, LzmaDecode.*). Anyway, I can resubmit the patch including all files. The LZMA SDK is released under LGPL with exceptions but I don't understand if the license is compatible with the u-boot's license. For this reason I omitted the sources from the SDK.

This makes no sense to me.

Either the license is compatible, and the code can be included, or it is not, then it cannot. In the latter case, the whole patch makes no sense.

Please check for compatibility of licenses berfore resubmitting.

Quoted 3 lines

LMZA performs better than LZ on binary files. I will switch to LZMA to save flash memory space on my appliance. Of course, the LZMA is slower than LZ.

But so far there is no code (board support) in U-Boot to use this feature, right? And it's not used by the Linux kernel either?

And your patch does not even enable support for it in the mkimage tool, so you cannot even create images that use the feature?

How exactly is this code supposed to be used?

Quoted 3 lines

Anyway, if the LZMA SDK license is compatible with U-boot , I think that LZMA should be chosen by firmware developer when the flash memory space is a critical resource.

Frankly, if you are so short with flash you have a h/w dsign problem.

Best regards,

Wolfgang Denk

Signature · 4 lines

DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
core error - bus dumped

Ulf Samuelsson

2008-03-28 23:02 UTC #8 lore raw

[U-Boot-Users] LZMA support (patch)

Quoted 7 lines

Quoted 6 lines
Running the contained script the directory will be populated with all need files (LzmaTypes.h, LzmaDecode.*). Anyway, I can resubmit the patch including all files. The LZMA SDK is released under LGPL with exceptions but I don't understand if the license is compatible with the u-boot's license. For this reason I omitted the sources from the SDK.

Quoted 6 lines

This makes no sense to me.
Either the license is compatible, and the code can be included, or it is not, then it cannot. In the latter case, the whole patch makes no sense.

I think there should be no problem from a license point of view.

The home page says that you can use unmodified source in your own code without having to release it to the public.

Since U-boot is open source to start with, this cannot be a problem.

It goes on and says that if the LZMA lib is modified, then the resulting code has to be released as open source. Again, no problem for U-boot, since it is open source to start with.

There are also commercial licenses available, but that is not applicable since noone signed a commerical license.

Quoted 9 lines

Please check for compatibility of licenses berfore resubmitting.
Quoted 3 lines
LMZA performs better than LZ on binary files. I will switch to LZMA to save flash memory space on my appliance. Of course, the LZMA is slower than LZ.
But so far there is no code (board support) in U-Boot to use this feature, right? And it's not used by the Linux kernel either?

There are kernel patches for LZMA compression available. Can be selected at least in buildroot.

Quoted 3 lines

And your patch does not even enable support for it in the mkimage tool, so you cannot even create images that use the feature?

Maybe the patches for Linux creates an image???
Never saw any need to use it myself.

Quoted 8 lines

How exactly is this code supposed to be used?
Quoted 3 lines
Anyway, if the LZMA SDK license is compatible with U-boot , I think that LZMA should be chosen by firmware developer when the flash memory space is a critical resource.
Frankly, if you are so short with flash you have a h/w design problem.

Lots of applications are cost sensitive, and shaving 50 cents here, 50 cents there could be important.
Hard to judge without knowing the details.

Quoted 4 lines

Best regards,
Wolfgang Denk

Best Regards
Ulf Samuelsson

Luigi 'Comio' Mantellini

author

2008-03-28 23:24 UTC #9 lore raw

[U-Boot-Users] LZMA support (patch)

In reply to Wolfgang Denk

Hi Wolfgang,

Quoted 12 lines

In message <[email protected]>
Quoted 8 lines · you
Running the contained script the directory will be populated with all need files (LzmaTypes.h, LzmaDecode.*). Anyway, I can
resubmit the patch including all files. The LZMA SDK is released under LGPL with exceptions but I don't understand if the license is compatible with the u-boot's license. For this reason I omitted the sources from the SDK.

Quoted 1 line

This makes no sense to me.

I'm not expert in legalese questions.

Quoted 9 lines

Either the license is compatible, and the code can be included, or it
is not, then it cannot. In the latter case, the whole patch makes no
sense.
Please check for compatibility of licenses berfore resubmitting.

I think that the license is compatible (see the
http://www.7-zip.org/sdk.html). I just share my code with community...

Quoted 11 lines

Quoted 6 lines
LMZA performs better than LZ on binary files. I will switch to
LZMA to save flash memory space on my appliance. Of course, the
LZMA is slower than LZ.
But so far there is no code (board support) in U-Boot to use
this
feature, right? And it's not used by the Linux kernel either?

The code is used (eventually) by do_bootm routine to uncompress a lzma-ed kernel image. The patch provides the opportune image.c/cmd_bootm.c changes for this.

Quoted 4 lines

And your patch does not even enable support for it in the mkimage
tool, so you cannot even create images that use the feature?

mkimage doesn't compress anything! As well as gzipped images, the lzma-ed images are created by an external tool (7z?):
mkimage adds just a simple identification header to identify the used compression method. The proposed patch permits to mkimage to set the opportune compression type (id=3) in the uImage to recognize the lzma images.

Quoted 10 lines

How exactly is this code supposed to be used?
Quoted 4 lines
Anyway, if the LZMA SDK license is compatible with U-boot , I think that LZMA should be chosen by firmware developer when the
flash memory space is a critical resource.
Frankly, if you are so short with flash you have a h/w dsign problem.

The project constraints require a cheap flash memory to reduce the final cost (and prize) of the device. This is not a design problem...

luigi

Quoted 13 lines

Best regards,
Wolfgang Denk
--
DENX Software Engineering GmbH, MD: Wolfgang Denk & Detlev
Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email:
wd at denx.de
core error - bus dumped

Mike Frysinger

2008-03-29 02:13 UTC #10 lore raw

[U-Boot-Users] LZMA support (patch)

In reply to Wolfgang Denk

Quoted 5 lines · Wolfgang Denk

But so far there is no code (board support) in U-Boot to use this feature, right? And it's not used by the Linux kernel either?
And your patch does not even enable support for it in the mkimage tool, so you cannot even create images that use the feature?

ive seen patches floating around to add lzma support akin to gzip/bzip2 in the uImage format. makes sense to me, and some people are very interested in shaving off flash space in return for slightly longer boot time. linux kernel is irrelevant here. someone else doing this means i dont have to at some point ;).
-mike
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 827 bytes
Desc: This is a digitally signed message part.
Url : http://lists.denx.de/pipermail/u-boot/attachments/20080328/1d8c1860/attachment.pgp

Luigi 'Comio' Mantellini

author

2008-03-29 20:21 UTC #11 lore raw

[U-Boot-Users] LZMA support (patch)

Hi All,

I will rework the patch in order to include all files from LZMA SDK and some fix. I'll send the patch on Tuesday (I'm in vacation). These are the implemented features:

- LZMA SDK 4.57 Decode code (vanilla)
- A Buff2Buff wrapper to decode the image (LZMA_Alone format)
- Fixes for mkimage (option -C lzma) and do_bootm
- small footprint and small impact on the u-boot sources.

To use the LZMA algorithm: just #define CONFIG_LZMA in your board config file.

Ciao.

luigi

Quoted 20 lines · Wolfgang Denk

Quoted 7 lines
But so far there is no code (board support) in U-Boot to use this
feature, right? And it's not used by the Linux kernel either?
And your patch does not even enable support for it in the mkimage
tool, so you cannot even create images that use the feature?
ive seen patches floating around to add lzma support akin to gzip/bzip2 in the
uImage format. makes sense to me, and some people are very interested in
shaving off flash space in return for slightly longer boot time. linux
kernel is irrelevant here. someone else doing this means i dont have to at
some point ;).
-mike

Jean-Christophe PLAGNIOL-VILLARD

2008-04-01 06:39 UTC #12 lore raw

[U-Boot-Users] LZMA support (patch)

Quoted 6 lines · Luigi 'Comio' Mantellini

Hi All,
I will rework the patch in order to include all files from LZMA SDK and some fix. I'll send the patch on Tuesday (I'm in vacation). These are the implemented features:

Quoted 7 lines

- LZMA SDK 4.57 Decode code (vanilla)
- A Buff2Buff wrapper to decode the image (LZMA_Alone format)
- Fixes for mkimage (option -C lzma) and do_bootm
- small footprint and small impact on the u-boot sources.
To use the LZMA algorithm: just #define CONFIG_LZMA in your board config file.

Please update the README

Best Regards,
J.

Harald Welte

2008-04-06 10:01 UTC #13 lore raw

[U-Boot-Users] LZMA support (patch)

In reply to Wolfgang Denk

Hi Wolfgang, Luigi,

Quoted 3 lines · Wolfgang Denk

Either the license is compatible, and the code can be included, or it is not, then it cannot. In the latter case, the whole patch makes no sense.

Given my expertise with license compliance, I volunteer to look into the legal issues, to the point that if I have any doubt I will consult with my legal counsel on that subject.

Quoted 6 lines

Quoted 3 lines
LMZA performs better than LZ on binary files. I will switch to LZMA to save flash memory space on my appliance. Of course, the LZMA is slower than LZ.
But so far there is no code (board support) in U-Boot to use this feature, right? And it's not used by the Linux kernel either?

As for board support / LZMA: I don't know. But I'm pretty sure that I've seen LZMA compressed u-boot images before. LZMA is really popular with cheap consumer-grade embedded linux equipment, particularly DSL CPE and Wifi Routers. Most of them don't use u-boot, but only use it for the kernel image and the flash filesystem. They're using quite simple out-of-tree patches for that, though.

I think this has come up on lkml before, and Linus didn't like it very much. Probably related to the fact that many core kernel hackers look at embedded as sort of a special case, when in reality I am quite sure that the total quantity of embedded systems running Linux is significantly larger than the number of Linux servers + desktops together...

Quoted 2 lines

And your patch does not even enable support for it in the mkimage tool, so you cannot even create images that use the feature?

That has obviously to be fixed.

Quoted 1 line

How exactly is this code supposed to be used?

some documentation would certainly be fine, yes.

Quoted 5 lines

Quoted 3 lines
Anyway, if the LZMA SDK license is compatible with U-boot , I think that LZMA should be chosen by firmware developer when the flash memory space is a critical resource.
Frankly, if you are so short with flash you have a h/w dsign problem.

Yes and no. I tend to agree, but I also realize that there are economic incentives to keep the bom cost as low as possible, especially in large-quantity consumer equipment...

You can just as well claim: If you're using suboptimal compression and wasting flash space, you have a software design problem ;)

I think it makes a lot of sense to add LZMA support to u-boot, but obviously in a clean, consistent, documented way. It doesn't hurt, will not be compiled unless explicitly selected, and the API is zlib like, i.e. the code changes it requires are non-intrusive.

Just my thoughs...

--
- Harald Welte <[email protected]> http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."

                                                  (ETSI EN 300 175-7 Ch. A6)

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://lists.denx.de/pipermail/u-boot/attachments/20080406/432514cb/attachment.pgp