Skip to contentSkip to conversation

zstd: move recipe to oe-core from meta-oe

PATCH In review Stefan Agner 4 participants 6 messages Nov 20, 2019 – Aug 17, 2020 Zstd

lore
Share & tools
lore
#0 lore raw

PATCH v5 1/2 zstd: move recipe to oe-core from meta-oe

From: Stefan Agner <[email protected]>

Zstd is a very fast compression algorithm and has gained quite some popularity recently. An upcoming patch allows to compress rootfs using Zstd. Moving Zstd to oe-core allows this new conversion formats to be automatically tested by the oe-selftest imagefeatures.ImageFeatures.test_image_fstypes test.

Furthermore there are three packages in oe-core which allow to enable Zstd support through packageconfig currently.

---
 meta/conf/distro/include/maintainers.inc |  1 +
 meta/recipes-extended/zstd/zstd_1.4.4.bb | 35 ++++++++++++++++++++++++
 2 files changed, 36 insertions(+)
 create mode 100644 meta/recipes-extended/zstd/zstd_1.4.4.bb
meta/conf/distro/include/maintainers.inc +1 -0
index d85e5b697f..dff7ce28f8 100644
@@ -774,3 +774,4 @@ RECIPE_MAINTAINER_pn-xwininfo = "Armin Kuster <[email protected]>"
 RECIPE_MAINTAINER_pn-xz = "Denys Dmytriyenko <[email protected]>"
 RECIPE_MAINTAINER_pn-zip = "Denys Dmytriyenko <[email protected]>"
 RECIPE_MAINTAINER_pn-zlib = "Denys Dmytriyenko <[email protected]>"
+RECIPE_MAINTAINER_pn-zstd = "Alex Kiernan <[email protected]>"
meta/recipes-extended/zstd/zstd_1.4.4.bb new file +35 -0
new file mode 100644
index 0000000000..eb201f4139
@@ -0,0 +1,35 @@
+SUMMARY = "Zstandard - Fast real-time compression algorithm"
+DESCRIPTION = "Zstandard is a fast lossless compression algorithm, targeting \
+real-time compression scenarios at zlib-level and better compression ratios. \
+It's backed by a very fast entropy stage, provided by Huff0 and FSE library."
+HOMEPAGE = "http://www.zstd.net/"
+SECTION = "console/utils"
+
+LICENSE = "BSD-3-Clause & GPLv2"
+LIC_FILES_CHKSUM = "file://LICENSE;md5=c7f0b161edbe52f5f345a3d1311d0b32 \
+                    file://COPYING;md5=39bba7d2cf0ba1036f2a6e2be52fe3f0"
+
+SRC_URI = "git://github.com/facebook/zstd.git;nobranch=1"
+
+SRCREV = "10f0e6993f9d2f682da6d04aa2385b7d53cbb4ee"
+UPSTREAM_CHECK_GITTAGREGEX = "v(?P<pver>\d+(\.\d+)+)"
+
+S = "${WORKDIR}/git"
+
+PACKAGECONFIG ??= ""
+PACKAGECONFIG[lz4] = "HAVE_LZ4=1,HAVE_LZ4=0,lz4"
+PACKAGECONFIG[lzma] = "HAVE_LZMA=1,HAVE_LZMA=0,xz"
+PACKAGECONFIG[zlib] = "HAVE_ZLIB=1,HAVE_ZLIB=0,zlib"
+
+# See programs/README.md for how to use this
+ZSTD_LEGACY_SUPPORT ??= "4"
+
+do_compile () {
+    oe_runmake ${PACKAGECONFIG_CONFARGS} ZSTD_LEGACY_SUPPORT=${ZSTD_LEGACY_SUPPORT}
+}
+
+do_install () {
+    oe_runmake install 'DESTDIR=${D}'
+}
+
+BBCLASSEXTEND = "native nativesdk"
Signature · 1 line
2.17.1
#1 lore raw

PATCH v5 2/2 image_types: add Zstandard conversion support

From: Stefan Agner <[email protected]>

Add Zstandard (or just Zstd) compression support. This allows to create Zstd compressed tarballs by using tar.zst as IMAGE_FSTYPES.

---
 meta/classes/image_types.bbclass | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
meta/classes/image_types.bbclass +6 -2
index 2eeffbb366..de00492932 100644
@@ -59,6 +59,8 @@ XZ_INTEGRITY_CHECK ?= "crc32"
 
 ZIP_COMPRESSION_LEVEL ?= "-9"
 
+ZSTD_COMPRESSION_LEVEL ?= "-3"
+
 JFFS2_SUM_EXTRA_ARGS ?= ""
 IMAGE_CMD_jffs2 = "mkfs.jffs2 --root=${IMAGE_ROOTFS} --faketime --output=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.jffs2 ${EXTRA_IMAGECMD}"
 
@@ -269,7 +271,7 @@ IMAGE_TYPES = " \
     hddimg \
     squashfs squashfs-xz squashfs-lzo squashfs-lz4 \
     ubi ubifs multiubi \
-    tar tar.gz tar.bz2 tar.xz tar.lz4 \
+    tar tar.gz tar.bz2 tar.xz tar.lz4 tar.zst \
     cpio cpio.gz cpio.xz cpio.lzma cpio.lz4 \
     wic wic.gz wic.bz2 wic.lzma \
     container \
@@ -282,7 +284,7 @@ IMAGE_TYPES = " \
 # CONVERSION_CMD/DEPENDS.
 COMPRESSIONTYPES ?= ""
 
-CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip sum md5sum sha1sum sha224sum sha256sum sha384sum sha512sum bmap u-boot vmdk vdi qcow2 base64 ${COMPRESSIONTYPES}"
+CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip zst sum md5sum sha1sum sha224sum sha256sum sha384sum sha512sum bmap u-boot vmdk vdi qcow2 base64 ${COMPRESSIONTYPES}"
 CONVERSION_CMD_lzma = "lzma -k -f -7 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
 CONVERSION_CMD_gz = "gzip -f -9 -n -c --rsyncable ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"
 CONVERSION_CMD_bz2 = "pbzip2 -f -k ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
@@ -290,6 +292,7 @@ CONVERSION_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} ${XZ_DEFAULTS} --check=
 CONVERSION_CMD_lz4 = "lz4 -9 -z -l ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.lz4"
 CONVERSION_CMD_lzo = "lzop -9 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
 CONVERSION_CMD_zip = "zip ${ZIP_COMPRESSION_LEVEL} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.zip ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
+CONVERSION_CMD_zst = "zstd -f -k -T0 -c ${ZSTD_COMPRESSION_LEVEL} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.zst"
 CONVERSION_CMD_sum = "sumtool -i ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} -o ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sum ${JFFS2_SUM_EXTRA_ARGS}"
 CONVERSION_CMD_md5sum = "md5sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.md5sum"
 CONVERSION_CMD_sha1sum = "sha1sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sha1sum"
@@ -310,6 +313,7 @@ CONVERSION_DEPENDS_xz = "xz-native"
 CONVERSION_DEPENDS_lz4 = "lz4-native"
 CONVERSION_DEPENDS_lzo = "lzop-native"
 CONVERSION_DEPENDS_zip = "zip-native"
+CONVERSION_DEPENDS_zst = "zstd-native"
 CONVERSION_DEPENDS_sum = "mtd-utils-native"
 CONVERSION_DEPENDS_bmap = "bmap-tools-native"
 CONVERSION_DEPENDS_u-boot = "u-boot-tools-native"
Signature · 1 line
2.17.1
#2 lore raw

Re: [PATCH v5 2/2] image_types: add Zstandard conversion support

Hi,

I am a bit confused on which direction we are going here. It seems that this second patch now got merged, but not the first one (instead disabling testing tar.zstd). I am fine with that, but in this case we need to readd zstd to meta-oe (since there it already had been removed, probably accidentally).

--
Stefan

Quoted 78 lines · Stefan Agner

From: Stefan Agner <[email protected]>

Add Zstandard (or just Zstd) compression support. This allows to create Zstd compressed tarballs by using tar.zst as IMAGE_FSTYPES.

---
 meta/classes/image_types.bbclass | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
meta/classes/image_types.bbclass +2 -1
index 2eeffbb366..de00492932 100644
@@ -59,6 +59,8 @@ XZ_INTEGRITY_CHECK ?= "crc32"
 
 ZIP_COMPRESSION_LEVEL ?= "-9"
 
+ZSTD_COMPRESSION_LEVEL ?= "-3"
+
 JFFS2_SUM_EXTRA_ARGS ?= ""
 IMAGE_CMD_jffs2 = "mkfs.jffs2 --root=${IMAGE_ROOTFS} --faketime
--output=${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.jffs2

${EXTRA_IMAGECMD}"

meta/classes/image_types.bbclass +1 -2
@@ -269,7 +271,7 @@ IMAGE_TYPES = " \
     hddimg \
     squashfs squashfs-xz squashfs-lzo squashfs-lz4 \
     ubi ubifs multiubi \
-    tar tar.gz tar.bz2 tar.xz tar.lz4 \
+    tar tar.gz tar.bz2 tar.xz tar.lz4 tar.zst \
     cpio cpio.gz cpio.xz cpio.lzma cpio.lz4 \
     wic wic.gz wic.bz2 wic.lzma \
     container \
@@ -282,7 +284,7 @@ IMAGE_TYPES = " \
 # CONVERSION_CMD/DEPENDS.
 COMPRESSIONTYPES ?= ""
 
-CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip sum md5sum sha1sum

sha224sum sha256sum sha384sum sha512sum bmap u-boot vmdk vdi qcow2 base64 ${COMPRESSIONTYPES}"
+CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip zst sum md5sum sha1sum sha224sum sha256sum sha384sum sha512sum bmap u-boot vmdk vdi qcow2 base64 ${COMPRESSIONTYPES}"
CONVERSION_CMD_lzma = "lzma -k -f -7 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}" CONVERSION_CMD_gz = "gzip -f -9 -n -c --rsyncable
${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"
CONVERSION_CMD_bz2 = "pbzip2 -f -k ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"

meta/classes/image_types.bbclass +0 -0
@@ -290,6 +292,7 @@ CONVERSION_CMD_xz = "xz -f -k -c

${XZ_COMPRESSION_LEVEL} ${XZ_DEFAULTS} --check=
CONVERSION_CMD_lz4 = "lz4 -9 -z -l
${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.lz4"
CONVERSION_CMD_lzo = "lzop -9 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}" CONVERSION_CMD_zip = "zip ${ZIP_COMPRESSION_LEVEL}
${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.zip ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}" +CONVERSION_CMD_zst = "zstd -f -k -T0 -c ${ZSTD_COMPRESSION_LEVEL} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.zst"
CONVERSION_CMD_sum = "sumtool -i
${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} -o ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sum ${JFFS2_SUM_EXTRA_ARGS}"
CONVERSION_CMD_md5sum = "md5sum
${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.md5sum"
CONVERSION_CMD_sha1sum = "sha1sum
${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sha1sum"

meta/classes/image_types.bbclass +1 -0
@@ -310,6 +313,7 @@ CONVERSION_DEPENDS_xz = "xz-native"
 CONVERSION_DEPENDS_lz4 = "lz4-native"
 CONVERSION_DEPENDS_lzo = "lzop-native"
 CONVERSION_DEPENDS_zip = "zip-native"
+CONVERSION_DEPENDS_zst = "zstd-native"
 CONVERSION_DEPENDS_sum = "mtd-utils-native"
 CONVERSION_DEPENDS_bmap = "bmap-tools-native"
 CONVERSION_DEPENDS_u-boot = "u-boot-tools-native"
#3 lore raw

Re: [PATCH v5 2/2] image_types: add Zstandard conversion support

Quoted 5 lines · Stefan Agner

I am a bit confused on which direction we are going here. It seems that this second patch now got merged, but not the first one (instead disabling testing tar.zstd). I am fine with that, but in this case we need to readd zstd to meta-oe (since there it already had been removed, probably accidentally).

I queued in my staging branch the image type definition (which is agnostic of where zstd is) and the selftest tweak to skip it from the tests because they're the uncontentious part of the series. We can add the image type without the recipe itself moving.

Personally I'm leaning towards keeping zstd out of core *for now* but I struggle to have a strong opinion either way.

If zstd has already been removed from meta-oe then yes, it should be reverted. Lesson is to never send a removal until the recipe has actually moved! :)

Ross

#4 lore raw

Re: [OE-core] [PATCH v5 2/2] image_types: add Zstandard conversion support

Quoted 4 lines · Ross Burton

...
Personally I'm leaning towards keeping zstd out of core *for now* but I struggle to have a strong opinion either way.
...

Is there a chance the "for now" is over?

zstd is becoming more and more popular on the target side, and right now cmake is already building an own copy also on the native side.

Quoted 1 line

Ross

Thanks
Adrian

#5 lore raw

Re: [OE-core] [PATCH v5 2/2] image_types: add Zstandard conversion support

Quoted 7 lines · Adrian Bunk
Quoted 5 lines · Ross Burton

...
Personally I'm leaning towards keeping zstd out of core *for now* but I
struggle to have a strong opinion either way.
...

Quoted 5 lines

Is there a chance the "for now" is over?

zstd is becoming more and more popular on the target side, and right now cmake is already building an own copy also on the native side.

Ross is away, I'm leaning towards adding it...

Cheers,

Richard