image_types: use pigz to create .gz files

PATCH In review Ross Burton 5 participants 8 messages Apr 10, 2018 – Apr 13, 2018

Share & tools

Ross Burton

author

2018-04-10 14:35 UTC #0 lore raw

PATCH 1/1 image_types: use pigz to create .gz files

Since pigz is no longer a drop-in replacement for gzip (oe-core 1624b7b) the image creation has been using gzip instead of pigz, despite still depending on pigz-native. Fix this by invoking pigz explicitly.

Signed-off-by Ross Burton <[email protected]>

---
 meta/classes/image_types.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

meta/classes/image_types.bbclass +1 -1

index e872ae20982..00a00d318f4 100644
@@ -281,7 +281,7 @@ COMPRESSIONTYPES ?= ""
 
 CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip sum md5sum sha1sum sha224sum sha256sum sha384sum sha512sum bmap u-boot vmdk vdi qcow2 ${COMPRESSIONTYPES}"
 CONVERSION_CMD_lzma = "lzma -k -f -7 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
-CONVERSION_CMD_gz = "gzip -f -9 -n -c ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"
+CONVERSION_CMD_gz = "pigz -f -9 -n -c ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"
 CONVERSION_CMD_bz2 = "pbzip2 -f -k ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
 CONVERSION_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} ${XZ_THREADS} --check=${XZ_INTEGRITY_CHECK} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.xz"
 CONVERSION_CMD_lz4 = "lz4 -9 -z -l ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.lz4"

Signature · 1 line

2.11.0

Martin Jansa

2018-04-11 10:53 UTC #1 lore raw

What about using pigz for sstate-cache archives?

Quoted 43 lines · Ross Burton

Since pigz is no longer a drop-in replacement for gzip (oe-core 1624b7b) the
image creation has been using gzip instead of pigz, despite still depending on
pigz-native. Fix this by invoking pigz explicitly.
Signed-off-by Ross Burton <[email protected]>
---
 meta/classes/image_types.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/image_types. renamed from meta/classes/image_types.bbclass +0 -0
bbclass
index e872ae20982..00a00d318f4 100644
meta/classes/image_types.bbclass +0 -0
@@ -281,7 +281,7 @@ COMPRESSIONTYPES ?= ""
 
 CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip sum md5sum sha1sum
sha224sum sha256sum sha384sum sha512sum bmap u-boot vmdk vdi qcow2 ${COMPRESSIONTYPES}"
CONVERSION_CMD_lzma = "lzma -k -f -7 ${IMAGE_NAME}${IMAGE_NAME_
SUFFIX}.${type}"
-CONVERSION_CMD_gz = "gzip -f -9 -n -c ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}
Quoted 1 line
${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"
+CONVERSION_CMD_gz = "pigz -f -9 -n -c ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}
Quoted 1 line
${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"
CONVERSION_CMD_bz2 = "pbzip2 -f -k ${IMAGE_NAME}${IMAGE_NAME_
SUFFIX}.${type}"
CONVERSION_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} ${XZ_THREADS}
--check=${XZ_INTEGRITY_CHECK} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.xz"
CONVERSION_CMD_lz4 = "lz4 -9 -z -l ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}
${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.lz4"
--
2.11.0
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Burton, Ross

author

2018-04-11 13:36 UTC #2 lore raw

Wouldn't that introduce a fun circular dependency if pigz-native needs to be built?

Ross

Quoted 51 lines · Martin Jansa

What about using pigz for sstate-cache archives?
Quoted 46 lines · Ross Burton
Since pigz is no longer a drop-in replacement for gzip (oe-core 1624b7b) the
image creation has been using gzip instead of pigz, despite still depending on
pigz-native. Fix this by invoking pigz explicitly.
Signed-off-by Ross Burton <[email protected]>
---
 meta/classes/image_types.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/image_types.bbclass +0 -0
index e872ae20982..00a00d318f4 100644
@@ -281,7 +281,7 @@ COMPRESSIONTYPES ?= ""
 
 CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip sum md5sum sha1sum
sha224sum sha256sum sha384sum sha512sum bmap u-boot vmdk vdi qcow2 ${COMPRESSIONTYPES}"
CONVERSION_CMD_lzma = "lzma -k -f -7
${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
-CONVERSION_CMD_gz = "gzip -f -9 -n -c ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz" +CONVERSION_CMD_gz = "pigz -f -9 -n -c ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"
CONVERSION_CMD_bz2 = "pbzip2 -f -k
${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
CONVERSION_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} ${XZ_THREADS}
--check=${XZ_INTEGRITY_CHECK} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.xz"
CONVERSION_CMD_lz4 = "lz4 -9 -z -l
${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.lz4"
--
2.11.0
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Khem Raj

2018-04-11 16:04 UTC #3 lore raw

Quoted 3 lines · Ross

Wouldn't that introduce a fun circular dependency if pigz-native needs to be built?

interesting idea certainly, if it ends up with improvements in build time, we could
add pigz to host dependencies.

Quoted 59 lines

Ross
Quoted 44 lines · Martin Jansa
What about using pigz for sstate-cache archives?
Quoted 41 lines · Ross Burton
Since pigz is no longer a drop-in replacement for gzip (oe-core 1624b7b) the
image creation has been using gzip instead of pigz, despite still depending on
pigz-native. Fix this by invoking pigz explicitly.
Signed-off-by Ross Burton <[email protected]>
---
 meta/classes/image_types.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/image_types.bbclass +0 -0
index e872ae20982..00a00d318f4 100644
@@ -281,7 +281,7 @@ COMPRESSIONTYPES ?= ""
 
 CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip sum md5sum sha1sum
sha224sum sha256sum sha384sum sha512sum bmap u-boot vmdk vdi qcow2 ${COMPRESSIONTYPES}"
CONVERSION_CMD_lzma = "lzma -k -f -7
${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
-CONVERSION_CMD_gz = "gzip -f -9 -n -c ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz" +CONVERSION_CMD_gz = "pigz -f -9 -n -c ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"
CONVERSION_CMD_bz2 = "pbzip2 -f -k
${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
CONVERSION_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} ${XZ_THREADS}
--check=${XZ_INTEGRITY_CHECK} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.xz"
CONVERSION_CMD_lz4 = "lz4 -9 -z -l
${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.lz4"
--
2.11.0
Quoted 7 lines
Quoted 5 lines
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Burton, Ross

author

2018-04-11 16:07 UTC #4 lore raw

Woud be interesting to see a benchmark of a full build from scratch with host gzip vs host pigz...

Quoted 68 lines · Khem Raj

Quoted 3 lines · Ross
Wouldn't that introduce a fun circular dependency if pigz-native needs to be built?
interesting idea certainly, if it ends up with improvements in build time, we could
add pigz to host dependencies.
Quoted 59 lines
Ross
Quoted 44 lines · Martin Jansa
What about using pigz for sstate-cache archives?
Quoted 41 lines · Ross Burton
Since pigz is no longer a drop-in replacement for gzip (oe-core 1624b7b) the
image creation has been using gzip instead of pigz, despite still depending on
pigz-native. Fix this by invoking pigz explicitly.
Signed-off-by Ross Burton <[email protected]>
---
 meta/classes/image_types.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/image_types.bbclass +0 -0
index e872ae20982..00a00d318f4 100644
@@ -281,7 +281,7 @@ COMPRESSIONTYPES ?= ""
 
 CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip sum md5sum sha1sum
sha224sum sha256sum sha384sum sha512sum bmap u-boot vmdk vdi qcow2 ${COMPRESSIONTYPES}"
CONVERSION_CMD_lzma = "lzma -k -f -7
${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
-CONVERSION_CMD_gz = "gzip -f -9 -n -c ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz" +CONVERSION_CMD_gz = "pigz -f -9 -n -c ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"
CONVERSION_CMD_bz2 = "pbzip2 -f -k
${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
CONVERSION_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} ${XZ_THREADS}
--check=${XZ_INTEGRITY_CHECK} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.xz"
CONVERSION_CMD_lz4 = "lz4 -9 -z -l
${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.lz4"
--
2.11.0
Quoted 7 lines
Quoted 5 lines
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Denys Dmytriyenko

2018-04-12 01:19 UTC #5 lore raw

Quoted 2 lines · Ross

Woud be interesting to see a benchmark of a full build from scratch with host gzip vs host pigz...

Is pigz so widely used now to be available in every Linux distro like gzip?

Quoted 74 lines · Khem Raj

Quoted 68 lines · Ross
Quoted 3 lines
Wouldn't that introduce a fun circular dependency if pigz-native needs to be built?
interesting idea certainly, if it ends up with improvements in build time, we could
add pigz to host dependencies.
Quoted 59 lines
Ross
Quoted 44 lines · Martin Jansa
What about using pigz for sstate-cache archives?
Quoted 41 lines · Ross Burton
Since pigz is no longer a drop-in replacement for gzip (oe-core 1624b7b) the
image creation has been using gzip instead of pigz, despite still depending on
pigz-native. Fix this by invoking pigz explicitly.
Signed-off-by Ross Burton <[email protected]>
---
 meta/classes/image_types.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/image_types.bbclass +0 -0
index e872ae20982..00a00d318f4 100644
@@ -281,7 +281,7 @@ COMPRESSIONTYPES ?= ""
 
 CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip sum md5sum sha1sum
sha224sum sha256sum sha384sum sha512sum bmap u-boot vmdk vdi qcow2 ${COMPRESSIONTYPES}"
CONVERSION_CMD_lzma = "lzma -k -f -7
${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
-CONVERSION_CMD_gz = "gzip -f -9 -n -c ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz" +CONVERSION_CMD_gz = "pigz -f -9 -n -c ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"
CONVERSION_CMD_bz2 = "pbzip2 -f -k
${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
CONVERSION_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} ${XZ_THREADS}
--check=${XZ_INTEGRITY_CHECK} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.xz"
CONVERSION_CMD_lz4 = "lz4 -9 -z -l
${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.lz4"
--
2.11.0
Quoted 7 lines
Quoted 5 lines
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Signature · 4 lines
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Khem Raj

2018-04-12 01:34 UTC #6 lore raw

Quoted 6 lines · Denys Dmytriyenko

Quoted 2 lines · Ross
Woud be interesting to see a benchmark of a full build from scratch with host gzip vs host pigz...
Is pigz so widely used now to be available in every Linux distro like gzip?

Tested distros do seem to provide it as old as ubuntu 14.04 and centos 6 xz is another option for sstate archives the extra time it requires might be compensated with threaded run in addition the archives will be smaller

Quoted 75 lines

Quoted 75 lines · Khem Raj
Quoted 5 lines · Ross
Quoted 3 lines
Wouldn't that introduce a fun circular dependency if pigz-native needs to be built?
Quoted 63 lines
interesting idea certainly, if it ends up with improvements in build time, we could
add pigz to host dependencies.
Quoted 59 lines
Ross
Quoted 44 lines · Martin Jansa
What about using pigz for sstate-cache archives?
Quoted 41 lines · Ross Burton
Since pigz is no longer a drop-in replacement for gzip (oe-core 1624b7b) the
image creation has been using gzip instead of pigz, despite still depending on
pigz-native. Fix this by invoking pigz explicitly.
Signed-off-by Ross Burton <[email protected]>
---
 meta/classes/image_types.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/image_types.bbclass +0 -0
index e872ae20982..00a00d318f4 100644
@@ -281,7 +281,7 @@ COMPRESSIONTYPES ?= ""
 
 CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip sum md5sum sha1sum
sha224sum sha256sum sha384sum sha512sum bmap u-boot vmdk vdi qcow2 ${COMPRESSIONTYPES}"
CONVERSION_CMD_lzma = "lzma -k -f -7
${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
-CONVERSION_CMD_gz = "gzip -f -9 -n -c ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz" +CONVERSION_CMD_gz = "pigz -f -9 -n -c ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"
CONVERSION_CMD_bz2 = "pbzip2 -f -k
${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
CONVERSION_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} ${XZ_THREADS}
--check=${XZ_INTEGRITY_CHECK} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.xz"
CONVERSION_CMD_lz4 = "lz4 -9 -z -l
${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.lz4"
--
2.11.0
Quoted 7 lines
Quoted 5 lines
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Richard Purdie

2018-04-13 12:07 UTC #7 lore raw

In reply to Burton, Ross

Quoted 3 lines · Ross

Wouldn't that introduce a fun circular dependency if pigz-native needs
to be built?

For the sstate code we could probably make it use pigz if present.

We could also only depend on pigz-native for target, or encode the actual pigz-native dependencies.

So there may be a few options if its worth the performance win.

Cheers,

Richard