Skip to contentSkip to conversation

use host compressor for rootfs

PATCH New Philippe Reynes 4 participants 10 messages Apr 6, 2014 – Apr 12, 2014 VFS

lore
Share & tools
lore

Philippe Reynes

author
#0 lore raw

cover use host compressor for rootfs

When the rootfs is compressed with lzma, lzo or xz, we use the binary compiled by buildroot. But with gzip or bzip, we use the tools of the machine. This serie change this behaviour for gzip and bzip2, it add a dependancy on host-<compressor> and use it to compress the rootfs.

Philippe Reynes (2):
rootfs-gzip: use host-gzip instead of local gzip
rootfs-bzip: use host-bzip2 instead of local bzip2

 fs/common.mk           |    6 ++++--
 package/bzip2/bzip2.mk |    2 ++
 package/gzip/gzip.mk   |    3 +++
 3 files changed, 9 insertions(+), 2 deletions(-)
Signature · 1 line
1.7.4.4
author
#1 lore raw

PATCH 1/2 rootfs-gzip: use host-gzip instead of local gzip

Signed-off-by Philippe Reynes <[email protected]>
---
 fs/common.mk         |    3 ++-
 package/gzip/gzip.mk |    3 +++
 2 files changed, 5 insertions(+), 1 deletions(-)
fs/common.mk +2 -1
index d95c26b..57bcd2d 100644
@@ -41,8 +41,9 @@ ROOTFS_$(2)_DEPENDENCIES += host-fakeroot host-makedevs \
 	$(if $(PACKAGES_USERS),host-mkpasswd)
 
 ifeq ($$(BR2_TARGET_ROOTFS_$(2)_GZIP),y)
+ROOTFS_$(2)_DEPENDENCIES += host-gzip
 ROOTFS_$(2)_COMPRESS_EXT = .gz
-ROOTFS_$(2)_COMPRESS_CMD = gzip -9 -c
+ROOTFS_$(2)_COMPRESS_CMD = $$(GZIP) -9 -c
 endif
 ifeq ($$(BR2_TARGET_ROOTFS_$(2)_BZIP2),y)
 ROOTFS_$(2)_COMPRESS_EXT = .bz2
package/gzip/gzip.mk +3 -0
index e1a7bef..ced6485 100644
@@ -11,3 +11,6 @@ GZIP_LICENSE = GPLv3+
 GZIP_LICENSE_FILES = COPYING
 
 $(eval $(autotools-package))
+$(eval $(host-autotools-package))
+
+GZIP = $(HOST_DIR)/usr/bin/gzip
Signature · 1 line
1.7.4.4
author
#2 lore raw

PATCH 2/2 rootfs-bzip: use host-bzip2 instead of local bzip2

In reply to Philippe Reynes
Signed-off-by Philippe Reynes <[email protected]>
---
 fs/common.mk           |    3 ++-
 package/bzip2/bzip2.mk |    2 ++
 2 files changed, 4 insertions(+), 1 deletions(-)
fs/common.mk +2 -1
index 57bcd2d..8e11b46 100644
@@ -46,8 +46,9 @@ ROOTFS_$(2)_COMPRESS_EXT = .gz
 ROOTFS_$(2)_COMPRESS_CMD = $$(GZIP) -9 -c
 endif
 ifeq ($$(BR2_TARGET_ROOTFS_$(2)_BZIP2),y)
+ROOTFS_$(2)_DEPENDENCIES += host-bzip2
 ROOTFS_$(2)_COMPRESS_EXT = .bz2
-ROOTFS_$(2)_COMPRESS_CMD = bzip2 -9 -c
+ROOTFS_$(2)_COMPRESS_CMD = $$(BZIP2) -9 -c
 endif
 ifeq ($$(BR2_TARGET_ROOTFS_$(2)_LZMA),y)
 ROOTFS_$(2)_DEPENDENCIES += host-lzma
package/bzip2/bzip2.mk +2 -0
index fcddb55..469f72b 100644
@@ -65,3 +65,5 @@ endef
 
 $(eval $(generic-package))
 $(eval $(host-generic-package))
+
+BZIP2 = $(HOST_DIR)/usr/bin/bzip2
Signature · 1 line
1.7.4.4
#3 lore raw

cover use host compressor for rootfs

In reply to Philippe Reynes

Hi Philippe,

Quoted 6 lines · Philippe Reynes

When the rootfs is compressed with lzma, lzo or xz, we use the binary compiled by buildroot. But with gzip or bzip, we use the tools of the machine. This serie change this behaviour for gzip and bzip2, it add a dependancy on host-<compressor> and use it to compress the rootfs.

gzip and bzip2 are mandatory dependencies of buildroot
(http://nightly.buildroot.org/manual.html#requirement-mandatory). These packages are shipped by default with virtually all general purpose distros. What problem are you trying to fix besides consistency?

baruch

Quoted 11 lines

Philippe Reynes (2):
rootfs-gzip: use host-gzip instead of local gzip
rootfs-bzip: use host-bzip2 instead of local bzip2

 fs/common.mk           |    6 ++++--
 package/bzip2/bzip2.mk |    2 ++
 package/gzip/gzip.mk   |    3 +++
 3 files changed, 9 insertions(+), 2 deletions(-)
Signature · 1 line
1.7.4.4
Signature · 3 lines
     http://baruch.siach.name/blog/                  ~. .~   Tk Open Systems
=}------------------------------------------------ooO--U--Ooo------------{=
   - baruch at tkos.co.il - tel: +972.2.679.5364, http://www.tkos.co.il -
author
#4 lore raw

cover use host compressor for rootfs

Hi Baruch,

I've missed this "mandatory dependancy".
I suppose that this dependancy is usefull to uncompress tarball. But I think that the compilation of firmware should be done with tools compiled by buildroot.

If lzma is used to compress the rootfs, then lzma is compile by builtoot. If gzip is used to compress the rootfs, then we use local gzip. It's not very "coherent".
And using compiled binary with buildroot increase chance to regenerate firmware on (very) different machine.

And I agree with you, it's not a "real" issue, it's more a "nice to have".

Regards,
Philippe

Le Dimanche 6 avril 2014 9h59, Baruch Siach <[email protected]> a ?crit :

Hi Philippe,

Quoted 6 lines · Philippe Reynes

When the rootfs is compressed with lzma, lzo or xz, we use the binary compiled by buildroot. But with gzip or bzip, we use the tools of the machine. This serie change this behaviour for gzip and bzip2, it add a dependancy on host-<compressor> and use it to compress the rootfs.

gzip and bzip2 are mandatory dependencies of buildroot
(http://nightly.buildroot.org/manual.html#requirement-mandatory). These packages are shipped by default with virtually all general purpose distros. What problem are you trying to fix besides consistency?

baruch

Quoted 11 lines

Philippe Reynes (2):
? rootfs-gzip: use host-gzip instead of local gzip ? rootfs-bzip: use host-bzip2 instead of local bzip2

? fs/common.mk? ? ? ? ?  |? ? 6 ++++--
? package/bzip2/bzip2.mk |? ? 2 ++
? package/gzip/gzip.mk?  |? ? 3 +++

? 3 files changed, 9 insertions(+), 2 deletions(-)

Signature · 1 line
1.7.4.4
Signature · 7 lines
? ? http://baruch.siach.name/blog/ ? ? ? ? ? ? ? ?  ~. .~?  Tk Open Systems
=}------------------------------------------------ooO--U--Ooo------------{=
?  - baruch at tkos.co.il - tel: +972.2.679.5364, http://www.tkos.co.il 
-
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20140406/9ef8f90e/attachment.html>
#5 lore raw

cover use host compressor for rootfs

Philippe Reynes <[email protected]> schreef:

Quoted 7 lines

Hi Baruch,

I've missed this "mandatory dependancy".
I suppose that this dependancy is usefull to uncompress tarball. But I think that the compilation of firmware should be done with tools compiled by buildroot.

Quoted 7 lines

If lzma is used to compress the rootfs, then lzma is compile by builtoot. If gzip is used to compress the rootfs, then we use local gzip. It's not very "coherent".
And using compiled binary with buildroot increase chance to regenerate firmware on (very) different machine.

And I agree with you, it's not a "real" issue, it's more a "nice to have".

The problem is that the build time is increased without good reason.
If there is a problem with a specific version of gzip/bzip2, we could always use the suitable-host-package mechanism as we do for tar. In fact, you could start using it for lzma too and avoid building host-lzma if it's already present...

Best regards,
Thomas

author
#6 lore raw

cover use host compressor for rootfs

Hi Thomas,

I've experienced a weird behavior with lzma.
We are using lzma to compress initramfs.

On an old buildroot where the local lzma was used.

It was working fine on some machine, but it fails on others. The fails was a "not enought memory" at boot time. Of course, it was also an old kernel. I never understood why using local lzma raise this issue.

After we add a dependency to "host-lzma",
it works everywhere.

So using host-<compressor> fix this issue, and
now we have the same behavior on all desktop.

This serie is based on : if it could happen on

lzma, it could happen on gzip/bzip2.

Regards,
Philippe

Le Dimanche 6 avril 2014 11h32, Thomas De Schampheleire <[email protected]> a ?crit :

Philippe Reynes <[email protected]> schreef:

Quoted 7 lines

Hi Baruch,

I've missed this "mandatory dependancy".
I suppose that this dependancy is usefull to uncompress tarball. But I think that the compilation of firmware should be done with tools compiled by buildroot.

Quoted 7 lines

If lzma is used to compress the rootfs, then lzma is compile by builtoot. If gzip is used to compress the rootfs, then we use local gzip. It's not very "coherent".
And using compiled binary with buildroot increase chance to regenerate firmware on (very) different machine.

And I agree with you, it's not a "real" issue, it's more a "nice to have".

The problem is that the build time is increased without good reason.
If there is a problem with a specific version of gzip/bzip2, we could always use the suitable-host-package mechanism as we do for tar. In fact, you could start using it for lzma too and avoid building host-lzma if it's already present...

Best regards,
Thomas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20140406/7000f352/attachment-0001.html>

#7 lore raw

cover use host compressor for rootfs

In reply to Philippe Reynes

Dear Philippe Reynes,

Quoted 5 lines · Philippe Reynes

I've missed this "mandatory dependancy".
I suppose that this dependancy is usefull to uncompress tarball. But I think that the compilation of firmware should be done with tools compiled by buildroot.

Quoted 7 lines

If lzma is used to compress the rootfs, then lzma is compile by builtoot. If gzip is used to compress the rootfs, then we use local gzip. It's not very "coherent".
And using compiled binary with buildroot increase chance to regenerate firmware on (very) different machine.

And I agree with you, it's not a "real" issue, it's more a "nice to have".

Well, I guess there are two important things here:

1) Buildroot tries to avoid building native utilities needlessly. So when possible, we try to use the tools available on the machine, in order to keep the overall build time as reasonable as possible.

2) gzip and bzip2 are widely available tools, which haven't caused any compatibility issue in years. On the other hands, tools like lzma, xz and so on are a lot less widely available. They may not even be available at all on old Linux distributions that some companies are still using.

Both of these points combined explain why we handle gzip/bzip2 differently than lzma/xz.

Best regards,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
#8 lore raw

cover use host compressor for rootfs

Hi Thomas, Thomas,

Quoted 21 lines · Thomas Petazzoni
Quoted 5 lines · Philippe Reynes

I've missed this "mandatory dependancy".
I suppose that this dependancy is usefull to uncompress tarball. But I think that the compilation of firmware should be done with tools compiled by buildroot.

Quoted 7 lines

If lzma is used to compress the rootfs, then lzma is compile by builtoot. If gzip is used to compress the rootfs, then we use local gzip. It's not very "coherent".
And using compiled binary with buildroot increase chance to regenerate firmware on (very) different machine.

And I agree with you, it's not a "real" issue, it's more a "nice to have".

Well, I guess there are two important things here:

1) Buildroot tries to avoid building native utilities needlessly. So when possible, we try to use the tools available on the machine, in order to keep the overall build time as reasonable as possible.

Quoted 8 lines

2) gzip and bzip2 are widely available tools, which haven't caused any compatibility issue in years. On the other hands, tools like lzma, xz and so on are a lot less widely available. They may not even be available at all on old Linux distributions that some companies are still using.

Both of these points combined explain why we handle gzip/bzip2 differently than lzma/xz.

Since commit d1f325f554cab (xzcat: treat as host prerequisite and build if needed) xz is always built when the host does not have it. Can we remove host-xz from the dependencies of HOST_SQUASHFS and ROOTFS_*_XZ?

baruch

Signature · 3 lines
     http://baruch.siach.name/blog/                  ~. .~   Tk Open Systems
=}------------------------------------------------ooO--U--Ooo------------{=
   - baruch at tkos.co.il - tel: +972.2.679.5364, http://www.tkos.co.il -
#9 lore raw

cover use host compressor for rootfs

In reply to Philippe Reynes

Dear Philippe Reynes,

Quoted 10 lines · Philippe Reynes

When the rootfs is compressed with lzma, lzo or xz, we use the binary compiled by buildroot. But with gzip or bzip, we use the tools of the machine. This serie change this behaviour for gzip and bzip2, it add a dependancy on host-<compressor> and use it to compress the rootfs.

Philippe Reynes (2):
rootfs-gzip: use host-gzip instead of local gzip
rootfs-bzip: use host-bzip2 instead of local bzip2

Since the general reception about these patches has been fairly negative, I've marked them as Rejected in patchwork.

Thanks for your contribution!

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com