Skip to contentSkip to conversation

opkg: enable zstd support

PATCH In review Etienne Cordonnier 4 participants 12 messages Sep 13, 2022 – Sep 28, 2022 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 opkg: enable zstd support

This allows the use of zstd for opkg packages by using OPKGBUILDCMD:
OPKGBUILDCMD = "opkg-build -Z zstd"

---
 meta/recipes-devtools/opkg/opkg_0.6.0.bb | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
meta/recipes-devtools/opkg/opkg_0.6.0.bb +2 -1
index 7b351e8123..e38d9d6f3f 100644
@@ -30,7 +30,7 @@ inherit autotools pkgconfig ptest
 target_localstatedir := "${localstatedir}"
 OPKGLIBDIR ??= "${target_localstatedir}/lib"
 
-PACKAGECONFIG ??= "libsolv"
+PACKAGECONFIG ??= "libsolv zstd"
 
 PACKAGECONFIG[gpg] = "--enable-gpg,--disable-gpg,\
     gnupg gpgme libgpg-error,\
@@ -39,6 +39,7 @@ PACKAGECONFIG[gpg] = "--enable-gpg,--disable-gpg,\
 PACKAGECONFIG[curl] = "--enable-curl,--disable-curl,curl"
 PACKAGECONFIG[ssl-curl] = "--enable-ssl-curl,--disable-ssl-curl,curl openssl"
 PACKAGECONFIG[sha256] = "--enable-sha256,--disable-sha256"
+PACKAGECONFIG[zstd] = "--enable-zstd,--disable-zstd,zstd"
 PACKAGECONFIG[libsolv] = "--with-libsolv,--without-libsolv,libsolv"
 
 EXTRA_OECONF:class-native = "--localstatedir=/${@os.path.relpath('${localstatedir}', '${STAGING_DIR_NATIVE}')} --sysconfdir=/${@os.path.relpath('${sysconfdir}', '${STAGING_DIR_NATIVE}')}"
Signature · 1 line
2.36.1.vfs.0.0
#1 lore raw

ACK from me - apart from enabling zstd by default.

Quoted 14 lines · Etienne Cordonnier via lists.openembedded.org

This allows the use of zstd for opkg packages by using OPKGBUILDCMD:
OPKGBUILDCMD = "opkg-build -Z zstd"

---
  meta/recipes-devtools/opkg/opkg_0.6.0.bb | 3 ++-
  1 file changed, 2 insertions(+), 1 deletion(-)
meta/recipes-devtools/opkg/opkg_0.6.0.bb +0 -0
index 7b351e8123..e38d9d6f3f 100644

meta/recipes-devtools/opkg/opkg_0.6.0.bb inherit autotools pkgconfig ptest · line 33

Quoted 5 lines
meta/recipes-devtools/opkg/opkg_0.6.0.bb +1 -1
@@ -30,7 +30,7 @@ inherit autotools pkgconfig ptest
  target_localstatedir := "${localstatedir}"
  OPKGLIBDIR ??= "${target_localstatedir}/lib"
 
-PACKAGECONFIG ??= "libsolv"
+PACKAGECONFIG ??= "libsolv zstd"

Building in zstd support by default is a little suspect to me.

Unless I'm mistaken, OE-core will only build xz-compressed IPKs by default. So zstd support would be unnecessary for a distro integrator who just uses upstream OE-core.

For distros which use zstd compression in their packages, I think it would be more appropriate to overwrite the opkg PACKAGECONFIG in a .bbappend.

Is there something I'm not considering here?

Quoted 20 lines
meta/recipes-devtools/opkg/opkg_0.6.0.bb +1 -1
  PACKAGECONFIG[gpg] = "--enable-gpg,--disable-gpg,\
      gnupg gpgme libgpg-error,\
@@ -39,6 +39,7 @@ PACKAGECONFIG[gpg] = "--enable-gpg,--disable-gpg,\
  PACKAGECONFIG[curl] = "--enable-curl,--disable-curl,curl"
  PACKAGECONFIG[ssl-curl] = "--enable-ssl-curl,--disable-ssl-curl,curl openssl"
  PACKAGECONFIG[sha256] = "--enable-sha256,--disable-sha256"
+PACKAGECONFIG[zstd] = "--enable-zstd,--disable-zstd,zstd"
  PACKAGECONFIG[libsolv] = "--with-libsolv,--without-libsolv,libsolv"
 
  EXTRA_OECONF:class-native = "--localstatedir=/${@os.path.relpath('${localstatedir}', '${STAGING_DIR_NATIVE}')} --sysconfdir=/${@os.path.relpath('${sysconfdir}', '${STAGING_DIR_NATIVE}')}"
 
-=-=-=-=-=-=-=-=-=-=-=-

Links: You receive all messages sent to this group.
View/Reply Online (#170575): https://urldefense.com/v3/__https://lists.openembedded.org/g/openembedded-core/message/170575__;!!FbZ0ZwI3Qg!pogpLkfpLwPy4zdzhTDEFkMT6eNuynfXzyaNvTq4OK74eYha04b285qGN5bk1t7aQmJAqnSBddI6-TD8FLOtzgtG_5PjNUviztKz$
Mute This Topic: https://urldefense.com/v3/__https://lists.openembedded.org/mt/93654146/3616788__;!!FbZ0ZwI3Qg!pogpLkfpLwPy4zdzhTDEFkMT6eNuynfXzyaNvTq4OK74eYha04b285qGN5bk1t7aQmJAqnSBddI6-TD8FLOtzgtG_5PjNVdoKO7n$
Group Owner: [email protected]
Unsubscribe: https://urldefense.com/v3/__https://lists.openembedded.org/g/openembedded-core/unsub__;!!FbZ0ZwI3Qg!pogpLkfpLwPy4zdzhTDEFkMT6eNuynfXzyaNvTq4OK74eYha04b285qGN5bk1t7aQmJAqnSBddI6-TD8FLOtzgtG_5PjNfNktoah$ [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

Signature · 5 lines
Alex Stewart
Software Engineer - NI Real-Time OS
NI (National Instruments)

[email protected]
#2 lore raw
Quoted 26 lines · Alex Stewart

ACK from me - apart from enabling zstd by default.

Quoted 14 lines · Etienne Cordonnier via lists.openembedded.org

This allows the use of zstd for opkg packages by using OPKGBUILDCMD:
OPKGBUILDCMD = "opkg-build -Z zstd"

---
  meta/recipes-devtools/opkg/opkg_0.6.0.bb | 3 ++-
  1 file changed, 2 insertions(+), 1 deletion(-)
meta/recipes-devtools/opkg/opkg_0.6.0.bb +0 -0
index 7b351e8123..e38d9d6f3f 100644

meta/recipes-devtools/opkg/opkg_0.6.0.bb inherit autotools pkgconfig ptest · line 33

Quoted 5 lines
meta/recipes-devtools/opkg/opkg_0.6.0.bb +1 -1
@@ -30,7 +30,7 @@ inherit autotools pkgconfig ptest
  target_localstatedir := "${localstatedir}"
  OPKGLIBDIR ??= "${target_localstatedir}/lib"
 
-PACKAGECONFIG ??= "libsolv"
+PACKAGECONFIG ??= "libsolv zstd"

Building in zstd support by default is a little suspect to me.

Quoted 8 lines

Unless I'm mistaken, OE-core will only build xz-compressed IPKs by default. So zstd support would be unnecessary for a distro integrator who just uses upstream OE-core.

For distros which use zstd compression in their packages, I think it would be more appropriate to overwrite the opkg PACKAGECONFIG in a .bbappend.

This is perhaps fine. I do wonder if there is some performance comparison data between xz and zstd compressed ipks
with opkg, it might help users on making this choice and also if we should consider using
zstd by default at some point or not.

Quoted 33 lines

Is there something I'm not considering here?

Quoted 7 lines
meta/recipes-devtools/opkg/opkg_0.6.0.bb +0 -0
  PACKAGECONFIG[gpg] = "--enable-gpg,--disable-gpg,\
      gnupg gpgme libgpg-error,\

meta/recipes-devtools/opkg/opkg_0.6.0.bb PACKAGECONFIG[gpg] = "--enable-gpg,--disable-gpg,\ · line 45

Quoted 7 lines
meta/recipes-devtools/opkg/opkg_0.6.0.bb +1 -0
@@ -39,6 +39,7 @@ PACKAGECONFIG[gpg] = "--enable-gpg,--disable-gpg,\
  PACKAGECONFIG[curl] = "--enable-curl,--disable-curl,curl"
  PACKAGECONFIG[ssl-curl] = "--enable-ssl-curl,--disable-ssl-curl,curl openssl"
  PACKAGECONFIG[sha256] = "--enable-sha256,--disable-sha256"
+PACKAGECONFIG[zstd] = "--enable-zstd,--disable-zstd,zstd"
  PACKAGECONFIG[libsolv] = "--with-libsolv,--without-libsolv,libsolv"
 
  EXTRA_OECONF:class-native = "--localstatedir=/${@os.path.relpath('${localstatedir}', '${STAGING_DIR_NATIVE}')} --sysconfdir=/${@os.path.relpath('${sysconfdir}', '${STAGING_DIR_NATIVE}')}"

--
Alex Stewart
Software Engineer - NI Real-Time OS
NI (National Instruments)

[email protected]

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#170608): https://lists.openembedded.org/g/openembedded-core/message/170608
Mute This Topic: https://lists.openembedded.org/mt/93654146/1997914
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

A message that is not in the archive (sent off-list or not yet received): CANOoYsMVb9YpfwjTNHv4kSARM15_3T34CA-Ekr=t7P5F0XO4bA@mail.gmail.com

#4 lore raw
Quoted 113 lines · Alex Feinman

I do have some numbers. When I was selling this change internally, I did a comparison on our internal build.

Combined write IPK times (Σ t do_package_write_ipk)

xz 162m 35s

gz 52m 13s

zstd 33m 49s

Compression rate for zstd was closer to xz than to gz but not as good as xz. For systems that have to cache packages on the device with limited storage xz might be a better option, but for the bulk of projects zstd is the best choice

Additionally, zstd offers much faster decompression than xz so the rootfs build step that includes unpacking all of the ipks, takes 3m 58s with xz and 2m 44s with zstd.

One other thing of note - if your build includes debug packages, some may be quite large. E.g. one of our components produces a 2.2 GB debug package (uncompressed). On large files xz requires a disproportionally large amount of time resulting in 15 minutes needed to simply write ipk for the abovementioned packages, whereas zstd took about 45 sec. For frequent tasks like bitbaking a single package this translates in a lot of saved time.

Bottom line - I think making xz a default package compressor was not entirely thought through. gzip or zstd is what the default should be.

One final note: I could not find a reasonable explanation for why opkg-tools require code changes to support a different compressor. BSD tar and GNU tar both can easily accept compressors that they have no idea about (via -I option) because all of them provide a unified command line interface for use in pipes. If this were done similar to tar, we could have used any compressor we wanted, including the multithreaded versions (zstdmt)

Quoted 88 lines · Khem Raj
Quoted 7 lines · Alex Stewart

ACK from me - apart from enabling zstd by default.

Quoted 3 lines · Etienne Cordonnier via lists.openembedded.org

This allows the use of zstd for opkg packages by using OPKGBUILDCMD:
OPKGBUILDCMD = "opkg-build -Z zstd"

Quoted 7 lines

Review

Quoted 7 lines
---
  meta/recipes-devtools/opkg/opkg_0.6.0.bb | 3 ++-
  1 file changed, 2 insertions(+), 1 deletion(-)
(quoted diff) +0 -0

b/meta/recipes-devtools/opkg/opkg_0.6.0.bb

Quoted 12 lines

meta/recipes-devtools/opkg/opkg_0.6.0.bb inherit autotools pkgconfig ptest · line 33

Quoted 9 lines

index 7b351e8123..e38d9d6f3f 100644

meta/recipes-devtools/opkg/opkg_0.6.0.bb +1 -1
@@ -30,7 +30,7 @@ inherit autotools pkgconfig ptest
  target_localstatedir := "${localstatedir}"
  OPKGLIBDIR ??= "${target_localstatedir}/lib"
 
-PACKAGECONFIG ??= "libsolv"
+PACKAGECONFIG ??= "libsolv zstd"

Building in zstd support by default is a little suspect to me.

Quoted 8 lines

Unless I'm mistaken, OE-core will only build xz-compressed IPKs by default. So zstd support would be unnecessary for a distro integrator who just uses upstream OE-core.

For distros which use zstd compression in their packages, I think it would be more appropriate to overwrite the opkg PACKAGECONFIG in a .bbappend.

This is perhaps fine. I do wonder if there is some performance comparison data between xz and zstd compressed ipks
with opkg, it might help users on making this choice and also if we should consider using
zstd by default at some point or not.

Quoted 8 lines

Is there something I'm not considering here?

meta/recipes-devtools/opkg/opkg_0.6.0.bb PACKAGECONFIG[gpg] = "--enable-gpg,--disable-gpg,\ · line 40

Quoted 6 lines
meta/recipes-devtools/opkg/opkg_0.6.0.bb +0 -0
  PACKAGECONFIG[gpg] = "--enable-gpg,--disable-gpg,\
      gnupg gpgme libgpg-error,\
@@ -39,6 +39,7 @@ PACKAGECONFIG[gpg] = "--enable-gpg,--disable-gpg,\
  PACKAGECONFIG[curl] = "--enable-curl,--disable-curl,curl"
  PACKAGECONFIG[ssl-curl] =

"--enable-ssl-curl,--disable-ssl-curl,curl openssl"

Quoted 5 lines

meta/recipes-devtools/opkg/opkg_0.6.0.bb PACKAGECONFIG[gpg] = "--enable-gpg,--disable-gpg,\

Quoted 5 lines
meta/recipes-devtools/opkg/opkg_0.6.0.bb +1 -0
  PACKAGECONFIG[sha256] = "--enable-sha256,--disable-sha256"
+PACKAGECONFIG[zstd] = "--enable-zstd,--disable-zstd,zstd"
  PACKAGECONFIG[libsolv] = "--with-libsolv,--without-libsolv,libsolv"
 
  EXTRA_OECONF:class-native =

"--localstatedir=/${@os.path.relpath('${localstatedir}', '${STAGING_DIR_NATIVE}')} --sysconfdir=/${@os.path.relpath('${sysconfdir}', '${STAGING_DIR_NATIVE}')}"

Quoted 9 lines
Quoted 3 lines

--
Alex Stewart
Software Engineer - NI Real-Time OS
NI (National Instruments)

Quoted 6 lines

[email protected]

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#170608):

https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_170608&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=_VWADuvDS3QdSEYWTnh8wNH0lOxVHty18QZynvOtpcU&e=

Quoted 1 line

Mute This Topic:

https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_93654146_1997914&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=QncdL2I_pCDsz7AMbsTgStzC7bt5V2x_E_gIblrtjZY&e=

Quoted 2 lines

Group Owner: [email protected] Unsubscribe:

https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=PxAE_3yGzbSZ21-LHz_1r3gd2_STPoZcrYDeu9vd_GM&e=
[[email protected]]

Quoted 2 lines

-=-=-=-=-=-=-=-=-=-=-=-

#5 lore raw
In reply to a message not in the archive

Thanks for sharing, really appreciate it. How close were compression rates in numbers ? I think some users might be sensitive to the archive sizes, if its something
with in acceptable range, I do agree, zstd would be a better default these days.

Quoted 99 lines · Alex Feinman

I do have some numbers. When I was selling this change internally, I did a comparison on our internal build.

Combined write IPK times (Σ t do_package_write_ipk)

xz 162m 35s

gz 52m 13s

zstd 33m 49s

Compression rate for zstd was closer to xz than to gz but not as good as xz. For systems that have to cache packages on the device with limited storage xz might be a better option, but for the bulk of projects zstd is the best choice

Additionally, zstd offers much faster decompression than xz so the rootfs build step that includes unpacking all of the ipks, takes 3m 58s with xz and 2m 44s with zstd.

One other thing of note - if your build includes debug packages, some may be quite large. E.g. one of our components produces a 2.2 GB debug package (uncompressed). On large files xz requires a disproportionally large amount of time resulting in 15 minutes needed to simply write ipk for the abovementioned packages, whereas zstd took about 45 sec. For frequent tasks like bitbaking a single package this translates in a lot of saved time.

Bottom line - I think making xz a default package compressor was not entirely thought through. gzip or zstd is what the default should be.

One final note: I could not find a reasonable explanation for why opkg-tools require code changes to support a different compressor. BSD tar and GNU tar both can easily accept compressors that they have no idea about (via -I option) because all of them provide a unified command line interface for use in pipes. If this were done similar to tar, we could have used any compressor we wanted, including the multithreaded versions (zstdmt)

Quoted 76 lines · Khem Raj
Quoted 26 lines · Alex Stewart

ACK from me - apart from enabling zstd by default.

Quoted 14 lines · Etienne Cordonnier via lists.openembedded.org

This allows the use of zstd for opkg packages by using OPKGBUILDCMD:
OPKGBUILDCMD = "opkg-build -Z zstd"

---
  meta/recipes-devtools/opkg/opkg_0.6.0.bb | 3 ++-
  1 file changed, 2 insertions(+), 1 deletion(-)
meta/recipes-devtools/opkg/opkg_0.6.0.bb +0 -0
index 7b351e8123..e38d9d6f3f 100644

meta/recipes-devtools/opkg/opkg_0.6.0.bb inherit autotools pkgconfig ptest · line 33

Quoted 5 lines
meta/recipes-devtools/opkg/opkg_0.6.0.bb +1 -1
@@ -30,7 +30,7 @@ inherit autotools pkgconfig ptest
  target_localstatedir := "${localstatedir}"
  OPKGLIBDIR ??= "${target_localstatedir}/lib"
 
-PACKAGECONFIG ??= "libsolv"
+PACKAGECONFIG ??= "libsolv zstd"

Building in zstd support by default is a little suspect to me.

Quoted 8 lines

Unless I'm mistaken, OE-core will only build xz-compressed IPKs by default. So zstd support would be unnecessary for a distro integrator who just uses upstream OE-core.

For distros which use zstd compression in their packages, I think it would be more appropriate to overwrite the opkg PACKAGECONFIG in a .bbappend.

This is perhaps fine. I do wonder if there is some performance comparison data between xz and zstd compressed ipks
with opkg, it might help users on making this choice and also if we should consider using
zstd by default at some point or not.

Quoted 33 lines

Is there something I'm not considering here?

Quoted 7 lines
meta/recipes-devtools/opkg/opkg_0.6.0.bb +0 -0
  PACKAGECONFIG[gpg] = "--enable-gpg,--disable-gpg,\
      gnupg gpgme libgpg-error,\

meta/recipes-devtools/opkg/opkg_0.6.0.bb PACKAGECONFIG[gpg] = "--enable-gpg,--disable-gpg,\ · line 45

Quoted 7 lines
meta/recipes-devtools/opkg/opkg_0.6.0.bb +1 -0
@@ -39,6 +39,7 @@ PACKAGECONFIG[gpg] = "--enable-gpg,--disable-gpg,\
  PACKAGECONFIG[curl] = "--enable-curl,--disable-curl,curl"
  PACKAGECONFIG[ssl-curl] = "--enable-ssl-curl,--disable-ssl-curl,curl openssl"
  PACKAGECONFIG[sha256] = "--enable-sha256,--disable-sha256"
+PACKAGECONFIG[zstd] = "--enable-zstd,--disable-zstd,zstd"
  PACKAGECONFIG[libsolv] = "--with-libsolv,--without-libsolv,libsolv"
 
  EXTRA_OECONF:class-native = "--localstatedir=/${@os.path.relpath('${localstatedir}', '${STAGING_DIR_NATIVE}')} --sysconfdir=/${@os.path.relpath('${sysconfdir}', '${STAGING_DIR_NATIVE}')}"

--
Alex Stewart
Software Engineer - NI Real-Time OS
NI (National Instruments)

[email protected]

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#170608): https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_170608&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=_VWADuvDS3QdSEYWTnh8wNH0lOxVHty18QZynvOtpcU&e=
Mute This Topic: https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_93654146_1997914&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=QncdL2I_pCDsz7AMbsTgStzC7bt5V2x_E_gIblrtjZY&e=
Group Owner: [email protected]
Unsubscribe: https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=PxAE_3yGzbSZ21-LHz_1r3gd2_STPoZcrYDeu9vd_GM&e= [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#6 lore raw
In reply to a message not in the archive
Quoted 8 lines · Alex Feinman

I do have some numbers. When I was selling this change internally, I did a comparison on our internal build.
Combined write IPK times (Σ t do_package_write_ipk)

xz         162m 35s
gz         52m 13s
zstd       33m 49s

Compression rate for zstd was closer to xz than to gz but not as good as xz. For systems that have to cache packages on the device with limited storage xz might be a better option, but for the bulk of projects zstd is the best choice
Additionally, zstd offers much faster decompression than xz so the rootfs build step that includes unpacking all of the ipks, takes 3m 58s with xz and 2m 44s with zstd.
One other thing of note - if your build includes debug packages, some may be quite large. E.g. one of our components produces a 2.2 GB debug package (uncompressed). On large files xz requires a disproportionally large amount of time resulting in 15 minutes needed to simply write ipk for the abovementioned packages, whereas zstd took about 45 sec. For frequent tasks like bitbaking a single package this translates in a lot of saved time.

Those are certainly compelling performance improvements. Assuming that the final data-segment size is within 5%-ish of xz, then I would agree with the rest of the thread that it should probably be the contemporary default.

And if we make it the default compressor for OE IPKs, then obviously my criticism in the original PR is satisfied.

Quoted 1 line

Bottom line - I think making xz a default package compressor was not entirely thought through. gzip or zstd is what the default should be.

ZStandard support was only added to opkg last September [1]. Before that, xz was the new hotness that replaced gzip. :)

[1] https://git.yoctoproject.org/opkg/commit/?id=5dead419e94bce2e6b743ad786c1daec0e1aa294

Quoted 1 line

One final note: I could not find a reasonable explanation for why opkg-tools require code changes to support a different compressor. BSD tar and GNU tar both can easily accept compressors that they have no idea about (via -I option) because all of them provide a unified command line interface for use in pipes. If this were done similar to tar, we could have used any compressor we wanted, including the multithreaded versions (zstdmt)

Well, presumably IPK creation tools can only support the matrix of compression algorithms which your opkg binary can decompress. I suppose someone could try to implement a plugable compression module system for opkg. But given that nearly everyone uses opkg in an embedded context, I'm not sure it would get much use.

Quoted 108 lines · Khem Raj
    On Tue, Sep 13, 2022 at 12:19 PM Alex Stewart
    <[email protected]> wrote:
    >
    > ACK from me - apart from enabling zstd by default.
    >
    > On 9/13/22 07:37, Etienne Cordonnier via lists.openembedded.org
    <https://urldefense.com/v3/__http://lists.openembedded.org__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_bxrNjvo$>
    wrote:
    > > This allows the use of zstd for opkg packages by using
    OPKGBUILDCMD:
    > > OPKGBUILDCMD = "opkg-build -Z zstd"
    > >
    > > Signed-off-by: Alex Feinman <[email protected]>
    > > Signed-off-by: Etienne Cordonnier <[email protected]>
    > > ---
    > >   meta/recipes-devtools/opkg/opkg_0.6.0.bb
    <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$>
    | 3 ++-
    > >   1 file changed, 2 insertions(+), 1 deletion(-)
    > >
    > > diff --git a/meta/recipes-devtools/opkg/opkg_0.6.0.bb
    <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$>
    b/meta/recipes-devtools/opkg/opkg_0.6.0.bb
    <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$>
    > > index 7b351e8123..e38d9d6f3f 100644
    > > --- a/meta/recipes-devtools/opkg/opkg_0.6.0.bb
    <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$>
    > > +++ b/meta/recipes-devtools/opkg/opkg_0.6.0.bb
    <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$>
    > > @@ -30,7 +30,7 @@ inherit autotools pkgconfig ptest
    > >   target_localstatedir := "${localstatedir}"
    > >   OPKGLIBDIR ??= "${target_localstatedir}/lib"
    > >
    > > -PACKAGECONFIG ??= "libsolv"
    > > +PACKAGECONFIG ??= "libsolv zstd"
    >
    > Building in zstd support by default is a little suspect to me.
    >
    > Unless I'm mistaken, OE-core will only build xz-compressed IPKs by
    > default. So zstd support would be unnecessary for a distro
    integrator
    > who just uses upstream OE-core.
    >
    > For distros which use zstd compression in their packages, I think it
    > would be more appropriate to overwrite the opkg PACKAGECONFIG in a
    > .bbappend.
    >

    This is perhaps fine. I do wonder if there is some performance
    comparison data between xz and zstd compressed ipks
    with opkg, it might help users on making this choice and also if we
    should consider using
    zstd by default at some point or not.

    > Is there something I'm not considering here?
    >
    > >
    > >   PACKAGECONFIG[gpg] = "--enable-gpg,--disable-gpg,\
    > >       gnupg gpgme libgpg-error,\
    > > @@ -39,6 +39,7 @@ PACKAGECONFIG[gpg] =
    "--enable-gpg,--disable-gpg,\
    > >   PACKAGECONFIG[curl] = "--enable-curl,--disable-curl,curl"
    > >   PACKAGECONFIG[ssl-curl] =
    "--enable-ssl-curl,--disable-ssl-curl,curl openssl"
    > >   PACKAGECONFIG[sha256] = "--enable-sha256,--disable-sha256"
    > > +PACKAGECONFIG[zstd] = "--enable-zstd,--disable-zstd,zstd"
    > >   PACKAGECONFIG[libsolv] =
    "--with-libsolv,--without-libsolv,libsolv"
    > >
    > >   EXTRA_OECONF:class-native =
    "--localstatedir=/${@os.path.relpath('${localstatedir}',
    '${STAGING_DIR_NATIVE}')}
    --sysconfdir=/${@os.path.relpath('${sysconfdir}',
    '${STAGING_DIR_NATIVE}')}"
    > >
    > >
    > >
    >
    > --
    > Alex Stewart
    > Software Engineer - NI Real-Time OS
    > NI (National Instruments)
    >
    > [email protected]
    >
    >
    > -=-=-=-=-=-=-=-=-=-=-=-
    > Links: You receive all messages sent to this group.
    > View/Reply Online (#170608):
    https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_170608&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=_VWADuvDS3QdSEYWTnh8wNH0lOxVHty18QZynvOtpcU&e=
    <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_170608&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=_VWADuvDS3QdSEYWTnh8wNH0lOxVHty18QZynvOtpcU&e=>

    > Mute This Topic:
    https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_93654146_1997914&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=QncdL2I_pCDsz7AMbsTgStzC7bt5V2x_E_gIblrtjZY&e=
    <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_93654146_1997914&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=QncdL2I_pCDsz7AMbsTgStzC7bt5V2x_E_gIblrtjZY&e=>

    > Group Owner: [email protected]
    <mailto:openembedded-core%[email protected]>
    > Unsubscribe:
    https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=PxAE_3yGzbSZ21-LHz_1r3gd2_STPoZcrYDeu9vd_GM&e=
    <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=PxAE_3yGzbSZ21-LHz_1r3gd2_STPoZcrYDeu9vd_GM&e=>
     [[email protected]]
    > -=-=-=-=-=-=-=-=-=-=-=-
    >
Signature · 5 lines
Alex Stewart
Software Engineer - NI Real-Time OS
NI (National Instruments)

[email protected]
#7 lore raw

I ran a build of core-image-full-cmdline using xz and zstd, using pre-populated downloads and sstate-cache directories but with empty tmp directory. Here are the numbers:
zstd:
bitbake core-image-full-cmdline took 2m52.768s (real), the resulting directory tmp/deploy/ipk is 1.6GB big.
xz:
bitbake core-image-full-cmdline took 4m14.214s (real), the resulting directory tmp/deploy/ipk/ is 996M big

So the build with xz is 47% slower (254/172) than with zstd for this "incremental build" use-case.

See the 5 biggest packages, the difference in compression-ratio increases with big files:
build$ ls -lh tmp-zstd/deploy/ipk/core2-64/ --sort=size | head -n5 total 1.6G
-rw-r--r-- 3 ecordonnier ecordonnier 331M Sep 14 10:39 gcc-dbg_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 264M Sep 14 10:39 openssl-dbg_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 113M Sep 14 10:39 openssl-ptest_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 47M Sep 14 10:39
binutils-dbg_2.39-r0_core2-64.ipk
build$ ls -lh tmp-xz/deploy/ipk/core2-64/ --sort=size | head -n5 total 963M
-rw-r--r-- 3 ecordonnier ecordonnier 214M Sep 14 10:44 gcc-dbg_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 193M Sep 14 10:44 openssl-dbg_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 35M Sep 14 10:44
openssl-ptest_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 32M Sep 14 10:44
binutils-dbg_2.39-r0_core2-64.ipk
...

I think for use-cases where the size of the ipk packages matters, xz is better. For use-cases where it does not matter (ipk packages not deployed), build-time matters more than compression-ratio and zstd is better.

Regarding the enablement of zstd in opkg per default, I'll send a new version of the patch without this line.
My thinking for enabling zstd per default in opkg was that zstd is already enabled per default in libarchive's PACKAGECONFIG, and disabling zstd in opkg's PACKAGECONFIG removes only a few lines of code from opkg (opkg uses libarchive for doing the actual compression/decompression).

Quoted 196 lines · Alex Stewart
Quoted 6 lines · Alex Feinman

I do have some numbers. When I was selling this change internally, I did a comparison on our internal build.
Combined write IPK times (Σ t do_package_write_ipk)

xz         162m 35s
gz         52m 13s
zstd       33m 49s
Quoted 14 lines

Compression rate for zstd was closer to xz than to gz but not as good as xz. For systems that have to cache packages on the device with limited storage xz might be a better option, but for the bulk of projects zstd is the best choice
Additionally, zstd offers much faster decompression than xz so the rootfs build step that includes unpacking all of the ipks, takes 3m 58s with xz and 2m 44s with zstd.
One other thing of note - if your build includes debug packages, some may be quite large. E.g. one of our components produces a 2.2 GB debug package (uncompressed). On large files xz requires a disproportionally large amount of time resulting in 15 minutes needed to simply write ipk for the abovementioned packages, whereas zstd took about 45 sec. For frequent tasks like bitbaking a single package this translates in a lot of saved time.

Those are certainly compelling performance improvements. Assuming that the final data-segment size is within 5%-ish of xz, then I would agree with the rest of the thread that it should probably be the contemporary default.

And if we make it the default compressor for OE IPKs, then obviously my criticism in the original PR is satisfied.

Quoted 2 lines

Bottom line - I think making xz a default package compressor was not entirely thought through. gzip or zstd is what the default should be.

ZStandard support was only added to opkg last September [1]. Before that, xz was the new hotness that replaced gzip. :)

[1]

https://urldefense.proofpoint.com/v2/url?u=https-3A__git.yoctoproject.org_opkg_commit_-3Fid-3D5dead419e94bce2e6b743ad786c1daec0e1aa294&d=DwIDaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsyDLWdbBqarUzFxz3aALck&m=VG_fgpCGq8Zgu73KQP1wTtb1D1TZftNpXj8jGnouPtGIBYyLIZbG8F-85POUcVN7&s=Mwoq2kkjfZhto6J5OomduJ5Rhyg_oSe-dkjeltE4Ls8&e=

Quoted 7 lines

One final note: I could not find a reasonable explanation for why opkg-tools require code changes to support a different compressor. BSD tar and GNU tar both can easily accept compressors that they have no idea about (via -I option) because all of them provide a unified command line interface for use in pipes. If this were done similar to tar, we could have used any compressor we wanted, including the multithreaded versions (zstdmt)

Well, presumably IPK creation tools can only support the matrix of compression algorithms which your opkg binary can decompress. I suppose someone could try to implement a plugable compression module system for opkg. But given that nearly everyone uses opkg in an embedded context, I'm not sure it would get much use.

Quoted 10 lines · Khem Raj
    On Tue, Sep 13, 2022 at 12:19 PM Alex Stewart
    <[email protected]> wrote:
    >
    > ACK from me - apart from enabling zstd by default.
    >
    > On 9/13/22 07:37, Etienne Cordonnier via lists.openembedded.org
    <

https://urldefense.com/v3/__http://lists.openembedded.org__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_bxrNjvo$

Quoted 11 lines

wrote:
> > This allows the use of zstd for opkg packages by using OPKGBUILDCMD:
> > OPKGBUILDCMD = "opkg-build -Z zstd"
> >
> > Signed-off-by: Alex Feinman <[email protected]> > > Signed-off-by: Etienne Cordonnier <[email protected]> > > ---
> > meta/recipes-devtools/opkg/opkg_0.6.0.bb
<

https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$

Quoted 6 lines
    | 3 ++-
    > >   1 file changed, 2 insertions(+), 1 deletion(-)
    > >
    > > diff --git a/meta/recipes-devtools/opkg/opkg_0.6.0.bb
    <

https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$

Quoted 3 lines
    b/meta/recipes-devtools/opkg/opkg_0.6.0.bb
    <

https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$

Quoted 4 lines
Quoted 3 lines
Quoted 3 lines

index 7b351e8123..e38d9d6f3f 100644
--- a/meta/recipes-devtools/opkg/opkg_0.6.0.bb

    <

https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$

Quoted 3 lines
Quoted 2 lines
Quoted 2 lines

+++ b/meta/recipes-devtools/opkg/opkg_0.6.0.bb

    <

https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$

Quoted 10 lines
Quoted 10 lines

meta/recipes-devtools/opkg/opkg_0.6.0.bb inherit autotools pkgconfig ptest · line 33

Quoted 7 lines
meta/recipes-devtools/opkg/opkg_0.6.0.bb +1 -1
@@ -30,7 +30,7 @@ inherit autotools pkgconfig ptest
  target_localstatedir := "${localstatedir}"
  OPKGLIBDIR ??= "${target_localstatedir}/lib"
 
-PACKAGECONFIG ??= "libsolv"
+PACKAGECONFIG ??= "libsolv zstd"

Building in zstd support by default is a little suspect to me.

Quoted 6 lines
Quoted 2 lines

Unless I'm mistaken, OE-core will only build xz-compressed IPKs by default. So zstd support would be unnecessary for a distro

integrator
> who just uses upstream OE-core.
>
> For distros which use zstd compression in their packages, I think

it

Quoted 37 lines
Quoted 3 lines

would be more appropriate to overwrite the opkg PACKAGECONFIG in a .bbappend.

    This is perhaps fine. I do wonder if there is some performance
    comparison data between xz and zstd compressed ipks
    with opkg, it might help users on making this choice and also if we
    should consider using
    zstd by default at some point or not.

    > Is there something I'm not considering here?
    >
    > >
    > >   PACKAGECONFIG[gpg] = "--enable-gpg,--disable-gpg,\
    > >       gnupg gpgme libgpg-error,\
    > > @@ -39,6 +39,7 @@ PACKAGECONFIG[gpg] =
    "--enable-gpg,--disable-gpg,\
    > >   PACKAGECONFIG[curl] = "--enable-curl,--disable-curl,curl"
    > >   PACKAGECONFIG[ssl-curl] =
    "--enable-ssl-curl,--disable-ssl-curl,curl openssl"
    > >   PACKAGECONFIG[sha256] = "--enable-sha256,--disable-sha256"
    > > +PACKAGECONFIG[zstd] = "--enable-zstd,--disable-zstd,zstd"
    > >   PACKAGECONFIG[libsolv] =
    "--with-libsolv,--without-libsolv,libsolv"
    > >
    > >   EXTRA_OECONF:class-native =
    "--localstatedir=/${@os.path.relpath('${localstatedir}',
    '${STAGING_DIR_NATIVE}')}
    --sysconfdir=/${@os.path.relpath('${sysconfdir}',
    '${STAGING_DIR_NATIVE}')}"
    > >
    > >
    > >
    >
    > --
    > Alex Stewart
    > Software Engineer - NI Real-Time OS
Quoted 9 lines
    > NI (National Instruments)
    >
    > [email protected]
    >
    >
    > -=-=-=-=-=-=-=-=-=-=-=-
    > Links: You receive all messages sent to this group.
    > View/Reply Online (#170608):

https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_170608&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=_VWADuvDS3QdSEYWTnh8wNH0lOxVHty18QZynvOtpcU&e=

Quoted 1 line
    <

https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_170608&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=_VWADuvDS3QdSEYWTnh8wNH0lOxVHty18QZynvOtpcU&e=

Quoted 4 lines
Quoted 3 lines

Mute This Topic:

https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_93654146_1997914&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=QncdL2I_pCDsz7AMbsTgStzC7bt5V2x_E_gIblrtjZY&e=

Quoted 1 line
    <

https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_93654146_1997914&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=QncdL2I_pCDsz7AMbsTgStzC7bt5V2x_E_gIblrtjZY&e=

Quoted 6 lines
Quoted 3 lines

Group Owner: [email protected]

<mailto:openembedded-core%[email protected]> > Unsubscribe:

https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=PxAE_3yGzbSZ21-LHz_1r3gd2_STPoZcrYDeu9vd_GM&e=

Quoted 1 line
    <

https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=PxAE_3yGzbSZ21-LHz_1r3gd2_STPoZcrYDeu9vd_GM&e=

Quoted 5 lines
     [[email protected]]
    > -=-=-=-=-=-=-=-=-=-=-=-
    >

--
Alex Stewart
Software Engineer - NI Real-Time OS
NI (National Instruments)

[email protected]

#8 lore raw

Also note that zstd's default compression level is 3 per default (from a 1 to 19 scale). I did not test other compression levels.

Quoted 246 lines · Etienne Cordonnier

I ran a build of core-image-full-cmdline using xz and zstd, using pre-populated downloads and sstate-cache directories but with empty tmp directory. Here are the numbers:
zstd:
bitbake core-image-full-cmdline took 2m52.768s (real), the resulting directory tmp/deploy/ipk is 1.6GB big.
xz:
bitbake core-image-full-cmdline took 4m14.214s (real), the resulting directory tmp/deploy/ipk/ is 996M big

So the build with xz is 47% slower (254/172) than with zstd for this "incremental build" use-case.

See the 5 biggest packages, the difference in compression-ratio increases with big files:
build$ ls -lh tmp-zstd/deploy/ipk/core2-64/ --sort=size | head -n5 total 1.6G
-rw-r--r-- 3 ecordonnier ecordonnier 331M Sep 14 10:39 gcc-dbg_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 264M Sep 14 10:39 openssl-dbg_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 113M Sep 14 10:39 openssl-ptest_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 47M Sep 14 10:39
binutils-dbg_2.39-r0_core2-64.ipk
build$ ls -lh tmp-xz/deploy/ipk/core2-64/ --sort=size | head -n5 total 963M
-rw-r--r-- 3 ecordonnier ecordonnier 214M Sep 14 10:44 gcc-dbg_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 193M Sep 14 10:44 openssl-dbg_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 35M Sep 14 10:44
openssl-ptest_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 32M Sep 14 10:44
binutils-dbg_2.39-r0_core2-64.ipk
...

I think for use-cases where the size of the ipk packages matters, xz is better. For use-cases where it does not matter (ipk packages not deployed), build-time matters more than compression-ratio and zstd is better.

Regarding the enablement of zstd in opkg per default, I'll send a new version of the patch without this line.
My thinking for enabling zstd per default in opkg was that zstd is already enabled per default in libarchive's PACKAGECONFIG, and disabling zstd in opkg's PACKAGECONFIG removes only a few lines of code from opkg (opkg uses libarchive for doing the actual compression/decompression).

Quoted 196 lines · Alex Stewart
Quoted 6 lines · Alex Feinman

I do have some numbers. When I was selling this change internally, I did a comparison on our internal build.
Combined write IPK times (Σ t do_package_write_ipk)

xz         162m 35s
gz         52m 13s
zstd       33m 49s
Quoted 14 lines

Compression rate for zstd was closer to xz than to gz but not as good as xz. For systems that have to cache packages on the device with limited storage xz might be a better option, but for the bulk of projects zstd is the best choice
Additionally, zstd offers much faster decompression than xz so the rootfs build step that includes unpacking all of the ipks, takes 3m 58s with xz and 2m 44s with zstd.
One other thing of note - if your build includes debug packages, some may be quite large. E.g. one of our components produces a 2.2 GB debug package (uncompressed). On large files xz requires a disproportionally large amount of time resulting in 15 minutes needed to simply write ipk for the abovementioned packages, whereas zstd took about 45 sec. For frequent tasks like bitbaking a single package this translates in a lot of saved time.

Those are certainly compelling performance improvements. Assuming that the final data-segment size is within 5%-ish of xz, then I would agree with the rest of the thread that it should probably be the contemporary default.

And if we make it the default compressor for OE IPKs, then obviously my criticism in the original PR is satisfied.

Quoted 2 lines

Bottom line - I think making xz a default package compressor was not entirely thought through. gzip or zstd is what the default should be.

ZStandard support was only added to opkg last September [1]. Before that, xz was the new hotness that replaced gzip. :)

[1]

https://urldefense.proofpoint.com/v2/url?u=https-3A__git.yoctoproject.org_opkg_commit_-3Fid-3D5dead419e94bce2e6b743ad786c1daec0e1aa294&d=DwIDaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsyDLWdbBqarUzFxz3aALck&m=VG_fgpCGq8Zgu73KQP1wTtb1D1TZftNpXj8jGnouPtGIBYyLIZbG8F-85POUcVN7&s=Mwoq2kkjfZhto6J5OomduJ5Rhyg_oSe-dkjeltE4Ls8&e=

Quoted 7 lines

One final note: I could not find a reasonable explanation for why opkg-tools require code changes to support a different compressor. BSD tar and GNU tar both can easily accept compressors that they have no idea about (via -I option) because all of them provide a unified command line interface for use in pipes. If this were done similar to tar, we could have used any compressor we wanted, including the multithreaded versions (zstdmt)

Well, presumably IPK creation tools can only support the matrix of compression algorithms which your opkg binary can decompress. I suppose someone could try to implement a plugable compression module system for opkg. But given that nearly everyone uses opkg in an embedded context, I'm not sure it would get much use.

Quoted 10 lines · Khem Raj
    On Tue, Sep 13, 2022 at 12:19 PM Alex Stewart
    <[email protected]> wrote:
    >
    > ACK from me - apart from enabling zstd by default.
    >
    > On 9/13/22 07:37, Etienne Cordonnier via lists.openembedded.org
    <

https://urldefense.com/v3/__http://lists.openembedded.org__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_bxrNjvo$

Quoted 11 lines

wrote:
> > This allows the use of zstd for opkg packages by using OPKGBUILDCMD:
> > OPKGBUILDCMD = "opkg-build -Z zstd"
> >
> > Signed-off-by: Alex Feinman <[email protected]> > > Signed-off-by: Etienne Cordonnier <[email protected]> > > ---
> > meta/recipes-devtools/opkg/opkg_0.6.0.bb
<

https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$

Quoted 6 lines
    | 3 ++-
    > >   1 file changed, 2 insertions(+), 1 deletion(-)
    > >
    > > diff --git a/meta/recipes-devtools/opkg/opkg_0.6.0.bb
    <

https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$

Quoted 3 lines
    b/meta/recipes-devtools/opkg/opkg_0.6.0.bb
    <

https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$

Quoted 4 lines
Quoted 3 lines
Quoted 3 lines

index 7b351e8123..e38d9d6f3f 100644
--- a/meta/recipes-devtools/opkg/opkg_0.6.0.bb

    <

https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$

Quoted 3 lines
Quoted 2 lines
Quoted 2 lines

+++ b/meta/recipes-devtools/opkg/opkg_0.6.0.bb

    <

https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$

Quoted 10 lines
Quoted 10 lines

meta/recipes-devtools/opkg/opkg_0.6.0.bb inherit autotools pkgconfig ptest · line 33

Quoted 7 lines
meta/recipes-devtools/opkg/opkg_0.6.0.bb +1 -1
@@ -30,7 +30,7 @@ inherit autotools pkgconfig ptest
  target_localstatedir := "${localstatedir}"
  OPKGLIBDIR ??= "${target_localstatedir}/lib"
 
-PACKAGECONFIG ??= "libsolv"
+PACKAGECONFIG ??= "libsolv zstd"

Building in zstd support by default is a little suspect to me.

Quoted 6 lines
Quoted 2 lines

Unless I'm mistaken, OE-core will only build xz-compressed IPKs by default. So zstd support would be unnecessary for a distro

integrator
> who just uses upstream OE-core.
>
> For distros which use zstd compression in their packages, I think

it

Quoted 37 lines
Quoted 3 lines

would be more appropriate to overwrite the opkg PACKAGECONFIG in a .bbappend.

    This is perhaps fine. I do wonder if there is some performance
    comparison data between xz and zstd compressed ipks
    with opkg, it might help users on making this choice and also if we
    should consider using
    zstd by default at some point or not.

    > Is there something I'm not considering here?
    >
    > >
    > >   PACKAGECONFIG[gpg] = "--enable-gpg,--disable-gpg,\
    > >       gnupg gpgme libgpg-error,\
    > > @@ -39,6 +39,7 @@ PACKAGECONFIG[gpg] =
    "--enable-gpg,--disable-gpg,\
    > >   PACKAGECONFIG[curl] = "--enable-curl,--disable-curl,curl"
    > >   PACKAGECONFIG[ssl-curl] =
    "--enable-ssl-curl,--disable-ssl-curl,curl openssl"
    > >   PACKAGECONFIG[sha256] = "--enable-sha256,--disable-sha256"
    > > +PACKAGECONFIG[zstd] = "--enable-zstd,--disable-zstd,zstd"
    > >   PACKAGECONFIG[libsolv] =
    "--with-libsolv,--without-libsolv,libsolv"
    > >
    > >   EXTRA_OECONF:class-native =
    "--localstatedir=/${@os.path.relpath('${localstatedir}',
    '${STAGING_DIR_NATIVE}')}
    --sysconfdir=/${@os.path.relpath('${sysconfdir}',
    '${STAGING_DIR_NATIVE}')}"
    > >
    > >
    > >
    >
    > --
    > Alex Stewart
    > Software Engineer - NI Real-Time OS
Quoted 9 lines
    > NI (National Instruments)
    >
    > [email protected]
    >
    >
    > -=-=-=-=-=-=-=-=-=-=-=-
    > Links: You receive all messages sent to this group.
    > View/Reply Online (#170608):

https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_170608&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=_VWADuvDS3QdSEYWTnh8wNH0lOxVHty18QZynvOtpcU&e=

Quoted 1 line
    <

https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_170608&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=_VWADuvDS3QdSEYWTnh8wNH0lOxVHty18QZynvOtpcU&e=

Quoted 4 lines
Quoted 3 lines

Mute This Topic:

https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_93654146_1997914&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=QncdL2I_pCDsz7AMbsTgStzC7bt5V2x_E_gIblrtjZY&e=

Quoted 1 line
    <

https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_93654146_1997914&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=QncdL2I_pCDsz7AMbsTgStzC7bt5V2x_E_gIblrtjZY&e=

Quoted 6 lines
Quoted 3 lines

Group Owner: [email protected]

<mailto:openembedded-core%[email protected]> > Unsubscribe:

https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=PxAE_3yGzbSZ21-LHz_1r3gd2_STPoZcrYDeu9vd_GM&e=

Quoted 1 line
    <

https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=PxAE_3yGzbSZ21-LHz_1r3gd2_STPoZcrYDeu9vd_GM&e=

Quoted 5 lines
     [[email protected]]
    > -=-=-=-=-=-=-=-=-=-=-=-
    >

--
Alex Stewart
Software Engineer - NI Real-Time OS
NI (National Instruments)

[email protected]

#9 lore raw

Thanks for checking.

I'd be interested to know if setting a higher compression level for zstd can get us to a similar compression ratio to xz. If so, then I think it could be some real value to distro maintainers to be able to *tune* their compression.

That's not blocking for your new PR though.

Quoted 297 lines · Etienne Cordonnier

Also note that zstd's default compression level is 3 per default (from a 1 to 19 scale). I did not test other compression levels.

On Wed, Sep 14, 2022 at 11:58 AM Etienne Cordonnier <[email protected]> wrote:

I ran a build of core-image-full-cmdline using xz and zstd, using pre-populated downloads and sstate-cache directories but with empty tmp directory. Here are the numbers:
zstd:
bitbake core-image-full-cmdline took 2m52.768s (real), the resulting directory tmp/deploy/ipk is 1.6GB big.
xz:
bitbake core-image-full-cmdline took 4m14.214s (real), the resulting directory tmp/deploy/ipk/ is 996M big

So the build with xz is 47% slower (254/172) than with zstd for this "incremental build" use-case.

See the 5 biggest packages, the difference in compression-ratio increases with big files:
build$ ls -lh tmp-zstd/deploy/ipk/core2-64/ --sort=size | head -n5 total 1.6G
-rw-r--r-- 3 ecordonnier ecordonnier  331M Sep 14 10:39 gcc-dbg_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier  264M Sep 14 10:39 openssl-dbg_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier  113M Sep 14 10:39 openssl-ptest_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier   47M Sep 14 10:39 binutils-dbg_2.39-r0_core2-64.ipk
build$ ls -lh tmp-xz/deploy/ipk/core2-64/ --sort=size | head -n5 total 963M
-rw-r--r-- 3 ecordonnier ecordonnier  214M Sep 14 10:44 gcc-dbg_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier  193M Sep 14 10:44 openssl-dbg_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier   35M Sep 14 10:44 openssl-ptest_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier   32M Sep 14 10:44 binutils-dbg_2.39-r0_core2-64.ipk
...

I think for use-cases where the size of the ipk packages matters, xz is better. For use-cases where it does not matter (ipk packages not deployed), build-time matters more than compression-ratio and zstd is better.

Regarding the enablement of zstd in opkg per default, I'll send a new version of the patch without this line.
My thinking for enabling zstd per default in opkg was that zstd is already enabled per default in libarchive's PACKAGECONFIG, and disabling zstd in opkg's PACKAGECONFIG removes only a few lines of code from opkg (opkg uses libarchive for doing the actual compression/decompression).

On Tue, Sep 13, 2022 at 11:57 PM Alex Stewart
<[email protected]> wrote:

On 9/13/22 15:20, Alex Feinman wrote:
> I do have some numbers. When I was selling this change internally, I
> did a comparison on our internal build.
>            Combined write IPK times (Σ t do_package_write_ipk) > xz         162m 35s
> gz         52m 13s
> zstd       33m 49s
> Compression rate for zstd was closer to xz than to gz but not as good
> as xz. For systems that have to cache packages on the device with
> limited storage xz might be a better option, but for the bulk of
> projects zstd is the best choice
> Additionally, zstd offers much faster decompression than xz so the
> rootfs build step that includes unpacking all of the ipks, takes 3m
> 58s with xz and 2m 44s with zstd.
> One other thing of note - if your build includes debug packages, some
> may be quite large. E.g. one of our components produces a 2.2 GB debug
> package (uncompressed). On large files xz requires a disproportionally
> large amount of time resulting in 15 minutes needed to simply write
> ipk for the abovementioned packages, whereas zstd took about 45 sec.
> For frequent tasks like bitbaking a single package this translates in
> a lot of saved time.

Those are certainly compelling performance improvements. Assuming that
the final data-segment size is within 5%-ish of xz, then I would agree
with the rest of the thread that it should probably be the contemporary
default.

And if we make it the default compressor for OE IPKs, then obviously my
criticism in the original PR is satisfied.

> Bottom line - I think making xz a default package compressor was not
> entirely thought through. gzip or zstd is what the default should be.

ZStandard support was only added to opkg last September [1]. Before
that, xz was the new hotness that replaced gzip. :)

[1]
https://urldefense.proofpoint.com/v2/url?u=https-3A__git.yoctoproject.org_opkg_commit_-3Fid-3D5dead419e94bce2e6b743ad786c1daec0e1aa294&d=DwIDaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsyDLWdbBqarUzFxz3aALck&m=VG_fgpCGq8Zgu73KQP1wTtb1D1TZftNpXj8jGnouPtGIBYyLIZbG8F-85POUcVN7&s=Mwoq2kkjfZhto6J5OomduJ5Rhyg_oSe-dkjeltE4Ls8&e=
<https://urldefense.proofpoint.com/v2/url?u=https-3A__git.yoctoproject.org_opkg_commit_-3Fid-3D5dead419e94bce2e6b743ad786c1daec0e1aa294&d=DwIDaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsyDLWdbBqarUzFxz3aALck&m=VG_fgpCGq8Zgu73KQP1wTtb1D1TZftNpXj8jGnouPtGIBYyLIZbG8F-85POUcVN7&s=Mwoq2kkjfZhto6J5OomduJ5Rhyg_oSe-dkjeltE4Ls8&e=>

> One final note: I could not find a reasonable explanation for why
> opkg-tools require code changes to support a different compressor. BSD
> tar and GNU tar both can easily accept compressors that they have no
> idea about (via -I option) because all of them provide a unified
> command line interface for use in pipes. If this were done similar to
> tar, we could have used any compressor we wanted, including the > multithreaded versions (zstdmt)

Well, presumably IPK creation tools can only support the matrix of
compression algorithms which your opkg binary can decompress. I suppose
someone could try to implement a plugable compression module system for
opkg. But given that nearly everyone uses opkg in an embedded context,
I'm not sure it would get much use.

>
> On Tue, Sep 13, 2022 at 12:43 PM Khem Raj <[email protected]> wrote:
>
>     On Tue, Sep 13, 2022 at 12:19 PM Alex Stewart
>     <[email protected]> wrote:
>     >
>     > ACK from me - apart from enabling zstd by default. >     >
>     > On 9/13/22 07:37, Etienne Cordonnier via lists.openembedded.org
<https://urldefense.com/v3/__http://lists.openembedded.org__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFhOF1_vg$>
>
 <https://urldefense.com/v3/__http://lists.openembedded.org__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_bxrNjvo$>
>     wrote:
>     > > This allows the use of zstd for opkg packages by using >     OPKGBUILDCMD:
>     > > OPKGBUILDCMD = "opkg-build -Z zstd"
>     > >
>     > > Signed-off-by: Alex Feinman <[email protected]> >     > > Signed-off-by: Etienne Cordonnier <[email protected]> >     > > ---
>     > >   meta/recipes-devtools/opkg/opkg_0.6.0.bb
<https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$>
>
 <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$>
>     | 3 ++-
>     > >   1 file changed, 2 insertions(+), 1 deletion(-) >     > >
>     > > diff --git a/meta/recipes-devtools/opkg/opkg_0.6.0.bb
<https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$>
>
 <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$>
>     b/meta/recipes-devtools/opkg/opkg_0.6.0.bb
<https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$>
>
 <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$>
>     > > index 7b351e8123..e38d9d6f3f 100644
>     > > --- a/meta/recipes-devtools/opkg/opkg_0.6.0.bb
<https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$>
>
 <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$>
>     > > +++ b/meta/recipes-devtools/opkg/opkg_0.6.0.bb
<https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$>
>
 <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$>
>     > > @@ -30,7 +30,7 @@ inherit autotools pkgconfig ptest >     > >   target_localstatedir := "${localstatedir}" >     > >   OPKGLIBDIR ??= "${target_localstatedir}/lib" >     > >
>     > > -PACKAGECONFIG ??= "libsolv"
>     > > +PACKAGECONFIG ??= "libsolv zstd"
>     >
>     > Building in zstd support by default is a little suspect to me.
>     >
>     > Unless I'm mistaken, OE-core will only build xz-compressed IPKs by
>     > default. So zstd support would be unnecessary for a distro >     integrator
>     > who just uses upstream OE-core.
>     >
>     > For distros which use zstd compression in their packages, I think it
>     > would be more appropriate to overwrite the opkg PACKAGECONFIG in a
>     > .bbappend.
>     >
>
>     This is perhaps fine. I do wonder if there is some performance
>     comparison data between xz and zstd compressed ipks >     with opkg, it might help users on making this choice and also if we
>     should consider using
>     zstd by default at some point or not.
>
>     > Is there something I'm not considering here?
>     >
>     > >
>     > >   PACKAGECONFIG[gpg] = "--enable-gpg,--disable-gpg,\
>     > >       gnupg gpgme libgpg-error,\
>     > > @@ -39,6 +39,7 @@ PACKAGECONFIG[gpg] =
>     "--enable-gpg,--disable-gpg,\
>     > >   PACKAGECONFIG[curl] = "--enable-curl,--disable-curl,curl"
>     > >   PACKAGECONFIG[ssl-curl] =
>     "--enable-ssl-curl,--disable-ssl-curl,curl openssl" >     > >   PACKAGECONFIG[sha256] = "--enable-sha256,--disable-sha256"
>     > > +PACKAGECONFIG[zstd] = "--enable-zstd,--disable-zstd,zstd"
>     > >   PACKAGECONFIG[libsolv] =
>     "--with-libsolv,--without-libsolv,libsolv"
>     > >
>     > >   EXTRA_OECONF:class-native =
>  "--localstatedir=/${@os.path.relpath('${localstatedir}', >     '${STAGING_DIR_NATIVE}')}
>     --sysconfdir=/${@os.path.relpath('${sysconfdir}', >     '${STAGING_DIR_NATIVE}')}"
>     > >
>     > >
>     > >
>     >
>     > --
>     > Alex Stewart
>     > Software Engineer - NI Real-Time OS
>     > NI (National Instruments)
>     >
>     > [email protected]
>     >
>     >
>     > -=-=-=-=-=-=-=-=-=-=-=-
>     > Links: You receive all messages sent to this group. >     > View/Reply Online (#170608):
>
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_170608&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=_VWADuvDS3QdSEYWTnh8wNH0lOxVHty18QZynvOtpcU&e=
<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_170608&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=_VWADuvDS3QdSEYWTnh8wNH0lOxVHty18QZynvOtpcU&e=>
>
 <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_170608&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=_VWADuvDS3QdSEYWTnh8wNH0lOxVHty18QZynvOtpcU&e=
<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_170608&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=_VWADuvDS3QdSEYWTnh8wNH0lOxVHty18QZynvOtpcU&e=>>
>
>     > Mute This Topic:
>
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_93654146_1997914&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=QncdL2I_pCDsz7AMbsTgStzC7bt5V2x_E_gIblrtjZY&e=
<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_93654146_1997914&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=QncdL2I_pCDsz7AMbsTgStzC7bt5V2x_E_gIblrtjZY&e=>
>
 <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_93654146_1997914&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=QncdL2I_pCDsz7AMbsTgStzC7bt5V2x_E_gIblrtjZY&e=
<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_93654146_1997914&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=QncdL2I_pCDsz7AMbsTgStzC7bt5V2x_E_gIblrtjZY&e=>>
>
>     > Group Owner:
[email protected] <mailto:openembedded-core%[email protected]> >     <mailto:openembedded-core%[email protected] <mailto:openembedded-core%[email protected]>> >     > Unsubscribe:
>
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=PxAE_3yGzbSZ21-LHz_1r3gd2_STPoZcrYDeu9vd_GM&e=
<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=PxAE_3yGzbSZ21-LHz_1r3gd2_STPoZcrYDeu9vd_GM&e=>
>
 <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=PxAE_3yGzbSZ21-LHz_1r3gd2_STPoZcrYDeu9vd_GM&e=
<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=PxAE_3yGzbSZ21-LHz_1r3gd2_STPoZcrYDeu9vd_GM&e=>>
>      [[email protected]]
>     > -=-=-=-=-=-=-=-=-=-=-=-
>     >
>

--
Alex Stewart
Software Engineer - NI Real-Time OS
NI (National Instruments)

[email protected]

Signature · 5 lines
Alex Stewart
Software Engineer - NI Real-Time OS
NI (National Instruments)

[email protected]
#10 lore raw
Quoted 7 lines · Alex Stewart

Thanks for checking.

I'd be interested to know if setting a higher compression level for zstd can get us to a similar compression ratio to xz. If so, then I think it could be some real value to distro maintainers to be able to *tune* their compression.

yeah it will be interesting to say try something like level 9 but I think times might regress with that but it might be good to know the balance and perhaps suggest size mode and performance mode of zstd instead of xz

Quoted 312 lines

That's not blocking for your new PR though.

Quoted 290 lines · Etienne Cordonnier

Also note that zstd's default compression level is 3 per default (from a 1 to 19 scale). I did not test other compression levels.

On Wed, Sep 14, 2022 at 11:58 AM Etienne Cordonnier <[email protected]> wrote:

    I ran a build of core-image-full-cmdline using xz and zstd, using
    pre-populated downloads and sstate-cache directories but with
    empty tmp directory. Here are the numbers:
    zstd:
    bitbake core-image-full-cmdline took 2m52.768s (real), the
    resulting directory tmp/deploy/ipk is 1.6GB big.
    xz:
    bitbake core-image-full-cmdline took 4m14.214s (real), the
    resulting directory tmp/deploy/ipk/ is 996M big

    So the build with xz is 47% slower (254/172) than with zstd for
    this "incremental build" use-case.

    See the 5 biggest packages, the difference in compression-ratio
    increases with big files:
    build$ ls -lh tmp-zstd/deploy/ipk/core2-64/ --sort=size | head -n5
    total 1.6G
    -rw-r--r-- 3 ecordonnier ecordonnier  331M Sep 14 10:39
    gcc-dbg_12.2.0-r0_core2-64.ipk
    -rw-r--r-- 3 ecordonnier ecordonnier  264M Sep 14 10:39
    openssl-dbg_3.0.5-r0_core2-64.ipk
    -rw-r--r-- 3 ecordonnier ecordonnier  113M Sep 14 10:39
    openssl-ptest_3.0.5-r0_core2-64.ipk
    -rw-r--r-- 3 ecordonnier ecordonnier   47M Sep 14 10:39
    binutils-dbg_2.39-r0_core2-64.ipk
    build$ ls -lh tmp-xz/deploy/ipk/core2-64/ --sort=size | head -n5
    total 963M
    -rw-r--r-- 3 ecordonnier ecordonnier  214M Sep 14 10:44
    gcc-dbg_12.2.0-r0_core2-64.ipk
    -rw-r--r-- 3 ecordonnier ecordonnier  193M Sep 14 10:44
    openssl-dbg_3.0.5-r0_core2-64.ipk
    -rw-r--r-- 3 ecordonnier ecordonnier   35M Sep 14 10:44
    openssl-ptest_3.0.5-r0_core2-64.ipk
    -rw-r--r-- 3 ecordonnier ecordonnier   32M Sep 14 10:44
    binutils-dbg_2.39-r0_core2-64.ipk
    ...

    I think for use-cases where the size of the ipk packages matters,
    xz is better. For use-cases where it does not matter (ipk packages
    not deployed), build-time matters more than compression-ratio and
    zstd is better.

    Regarding the enablement of zstd in opkg per default, I'll send a
    new version of the patch without this line.
    My thinking for enabling zstd per default in opkg was that zstd is
    already enabled per default in libarchive's PACKAGECONFIG, and
    disabling zstd in opkg's PACKAGECONFIG removes only a few lines of
    code from opkg (opkg uses libarchive for doing the actual
    compression/decompression).

    On Tue, Sep 13, 2022 at 11:57 PM Alex Stewart
    <[email protected]> wrote:



        On 9/13/22 15:20, Alex Feinman wrote:
        > I do have some numbers. When I was selling this change
        internally, I
        > did a comparison on our internal build.
        >            Combined write IPK times (Σ t do_package_write_ipk)
        > xz         162m 35s
        > gz         52m 13s
        > zstd       33m 49s
        > Compression rate for zstd was closer to xz than to gz but
        not as good
        > as xz. For systems that have to cache packages on the device
        with
        > limited storage xz might be a better option, but for the
        bulk of
        > projects zstd is the best choice
        > Additionally, zstd offers much faster decompression than xz
        so the
        > rootfs build step that includes unpacking all of the ipks,
        takes 3m
        > 58s with xz and 2m 44s with zstd.
        > One other thing of note - if your build includes debug
        packages, some
        > may be quite large. E.g. one of our components produces a
        2.2 GB debug
        > package (uncompressed). On large files xz requires a
        disproportionally
        > large amount of time resulting in 15 minutes needed to
        simply write
        > ipk for the abovementioned packages, whereas zstd took about
        45 sec.
        > For frequent tasks like bitbaking a single package this
        translates in
        > a lot of saved time.

        Those are certainly compelling performance improvements.
        Assuming that
        the final data-segment size is within 5%-ish of xz, then I
        would agree
        with the rest of the thread that it should probably be the
        contemporary
        default.

        And if we make it the default compressor for OE IPKs, then
        obviously my
        criticism in the original PR is satisfied.

        > Bottom line - I think making xz a default package compressor
        was not
        > entirely thought through. gzip or zstd is what the default
        should be.

        ZStandard support was only added to opkg last September [1].
        Before
        that, xz was the new hotness that replaced gzip. :)

        [1]
        https://urldefense.proofpoint.com/v2/url?u=https-3A__git.yoctoproject.org_opkg_commit_-3Fid-3D5dead419e94bce2e6b743ad786c1daec0e1aa294&d=DwIDaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsyDLWdbBqarUzFxz3aALck&m=VG_fgpCGq8Zgu73KQP1wTtb1D1TZftNpXj8jGnouPtGIBYyLIZbG8F-85POUcVN7&s=Mwoq2kkjfZhto6J5OomduJ5Rhyg_oSe-dkjeltE4Ls8&e=
        <https://urldefense.proofpoint.com/v2/url?u=https-3A__git.yoctoproject.org_opkg_commit_-3Fid-3D5dead419e94bce2e6b743ad786c1daec0e1aa294&d=DwIDaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsyDLWdbBqarUzFxz3aALck&m=VG_fgpCGq8Zgu73KQP1wTtb1D1TZftNpXj8jGnouPtGIBYyLIZbG8F-85POUcVN7&s=Mwoq2kkjfZhto6J5OomduJ5Rhyg_oSe-dkjeltE4Ls8&e=>


        > One final note: I could not find a reasonable explanation
        for why
        > opkg-tools require code changes to support a different
        compressor. BSD
        > tar and GNU tar both can easily accept compressors that they
        have no
        > idea about (via -I option) because all of them provide a
        unified
        > command line interface for use in pipes. If this were done
        similar to
        > tar, we could have used any compressor we wanted, including the
        > multithreaded versions (zstdmt)

        Well, presumably IPK creation tools can only support the
        matrix of
        compression algorithms which your opkg binary can decompress.
        I suppose
        someone could try to implement a plugable compression module
        system for
        opkg. But given that nearly everyone uses opkg in an embedded
        context,
        I'm not sure it would get much use.

        >
        > On Tue, Sep 13, 2022 at 12:43 PM Khem Raj
        <[email protected]> wrote:
        >
        >     On Tue, Sep 13, 2022 at 12:19 PM Alex Stewart
        >     <[email protected]> wrote:
        >     >
        >     > ACK from me - apart from enabling zstd by default.
        >     >
        >     > On 9/13/22 07:37, Etienne Cordonnier via
        lists.openembedded.org
        <https://urldefense.com/v3/__http://lists.openembedded.org__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFhOF1_vg$>
        >
         <https://urldefense.com/v3/__http://lists.openembedded.org__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_bxrNjvo$>
        >     wrote:
        >     > > This allows the use of zstd for opkg packages by using
        >     OPKGBUILDCMD:
        >     > > OPKGBUILDCMD = "opkg-build -Z zstd"
        >     > >
        >     > > Signed-off-by: Alex Feinman <[email protected]>
        >     > > Signed-off-by: Etienne Cordonnier <[email protected]>
        >     > > ---
        >     > >   meta/recipes-devtools/opkg/opkg_0.6.0.bb
        <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$>
        >
         <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$>
        >     | 3 ++-
        >     > >   1 file changed, 2 insertions(+), 1 deletion(-)
        >     > >
        >     > > diff --git
        a/meta/recipes-devtools/opkg/opkg_0.6.0.bb
        <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$>
        >
         <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$>
        >     b/meta/recipes-devtools/opkg/opkg_0.6.0.bb
        <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$>
        >
         <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$>
        >     > > index 7b351e8123..e38d9d6f3f 100644
        >     > > --- a/meta/recipes-devtools/opkg/opkg_0.6.0.bb
        <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$>
        >
         <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$>
        >     > > +++ b/meta/recipes-devtools/opkg/opkg_0.6.0.bb
        <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$>
        >
         <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$>
        >     > > @@ -30,7 +30,7 @@ inherit autotools pkgconfig ptest
        >     > >   target_localstatedir := "${localstatedir}"
        >     > >   OPKGLIBDIR ??= "${target_localstatedir}/lib"
        >     > >
        >     > > -PACKAGECONFIG ??= "libsolv"
        >     > > +PACKAGECONFIG ??= "libsolv zstd"
        >     >
        >     > Building in zstd support by default is a little
        suspect to me.
        >     >
        >     > Unless I'm mistaken, OE-core will only build
        xz-compressed IPKs by
        >     > default. So zstd support would be unnecessary for a distro
        >     integrator
        >     > who just uses upstream OE-core.
        >     >
        >     > For distros which use zstd compression in their
        packages, I think it
        >     > would be more appropriate to overwrite the opkg
        PACKAGECONFIG in a
        >     > .bbappend.
        >     >
        >
        >     This is perhaps fine. I do wonder if there is some
        performance
        >     comparison data between xz and zstd compressed ipks
        >     with opkg, it might help users on making this choice and
        also if we
        >     should consider using
        >     zstd by default at some point or not.
        >
        >     > Is there something I'm not considering here?
        >     >
        >     > >
        >     > >   PACKAGECONFIG[gpg] = "--enable-gpg,--disable-gpg,\
        >     > >       gnupg gpgme libgpg-error,\
        >     > > @@ -39,6 +39,7 @@ PACKAGECONFIG[gpg] =
        >     "--enable-gpg,--disable-gpg,\
        >     > >   PACKAGECONFIG[curl] =
        "--enable-curl,--disable-curl,curl"
        >     > >   PACKAGECONFIG[ssl-curl] =
        >     "--enable-ssl-curl,--disable-ssl-curl,curl openssl"
        >     > >   PACKAGECONFIG[sha256] =
        "--enable-sha256,--disable-sha256"
        >     > > +PACKAGECONFIG[zstd] =
        "--enable-zstd,--disable-zstd,zstd"
        >     > >   PACKAGECONFIG[libsolv] =
        >     "--with-libsolv,--without-libsolv,libsolv"
        >     > >
        >     > >   EXTRA_OECONF:class-native =
        >  "--localstatedir=/${@os.path.relpath('${localstatedir}',
        >     '${STAGING_DIR_NATIVE}')}
        >     --sysconfdir=/${@os.path.relpath('${sysconfdir}',
        >     '${STAGING_DIR_NATIVE}')}"
        >     > >
        >     > >
        >     > >
        >     >
        >     > --
        >     > Alex Stewart
        >     > Software Engineer - NI Real-Time OS
        >     > NI (National Instruments)
        >     >
        >     > [email protected]
        >     >
        >     >
        >     > -=-=-=-=-=-=-=-=-=-=-=-
        >     > Links: You receive all messages sent to this group.
        >     > View/Reply Online (#170608):
        >
        https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_170608&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=_VWADuvDS3QdSEYWTnh8wNH0lOxVHty18QZynvOtpcU&e=
        <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_170608&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=_VWADuvDS3QdSEYWTnh8wNH0lOxVHty18QZynvOtpcU&e=>
        >
         <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_170608&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=_VWADuvDS3QdSEYWTnh8wNH0lOxVHty18QZynvOtpcU&e=
        <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_170608&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=_VWADuvDS3QdSEYWTnh8wNH0lOxVHty18QZynvOtpcU&e=>>
        >
        >     > Mute This Topic:
        >
        https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_93654146_1997914&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=QncdL2I_pCDsz7AMbsTgStzC7bt5V2x_E_gIblrtjZY&e=
        <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_93654146_1997914&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=QncdL2I_pCDsz7AMbsTgStzC7bt5V2x_E_gIblrtjZY&e=>
        >
         <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_93654146_1997914&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=QncdL2I_pCDsz7AMbsTgStzC7bt5V2x_E_gIblrtjZY&e=
        <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_93654146_1997914&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=QncdL2I_pCDsz7AMbsTgStzC7bt5V2x_E_gIblrtjZY&e=>>
        >
        >     > Group Owner:
        [email protected]
        <mailto:openembedded-core%[email protected]>
        >     <mailto:openembedded-core%[email protected]
        <mailto:openembedded-core%[email protected]>>
        >     > Unsubscribe:
        >
        https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=PxAE_3yGzbSZ21-LHz_1r3gd2_STPoZcrYDeu9vd_GM&e=
        <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=PxAE_3yGzbSZ21-LHz_1r3gd2_STPoZcrYDeu9vd_GM&e=>
        >
         <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=PxAE_3yGzbSZ21-LHz_1r3gd2_STPoZcrYDeu9vd_GM&e=
        <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=PxAE_3yGzbSZ21-LHz_1r3gd2_STPoZcrYDeu9vd_GM&e=>>
        >      [[email protected]]
        >     > -=-=-=-=-=-=-=-=-=-=-=-
        >     >
Quoted 9 lines
        >

        --
        Alex Stewart
        Software Engineer - NI Real-Time OS
        NI (National Instruments)

        [email protected]

--
Alex Stewart
Software Engineer - NI Real-Time OS
NI (National Instruments)

[email protected]

#11 lore raw

I tested it a bit more today. I used the standard poky's local.conf and added those lines. You can see that at zstd level 9 there is still a significant difference in the compression ratio with xz for gcc-dbg which is a big file. At zstd level 19, gcc-dbg is 241MB big instead of 214MB with xz (12.6% more).

PACKAGE_CLASSES = "package_ipk"
ZSTD_DEFAULTS = "-T0 -19" PACKAGECONFIG:append:pn-opkg-native = " zstd" OPKGBUILDCMD = "opkg-build -Z zstd -a ${ZSTD_DEFAULTS}"

Zstd at level 9:

build$ ls -lh tmp-zstd-level9/deploy/ipk/core2-64/ --sort=size | head -n10 total 1.5G
-rw-r--r-- 3 ecordonnier ecordonnier 302M Sep 28 18:33 gcc-dbg_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 249M Sep 28 18:33 openssl-dbg_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 106M Sep 28 18:33 openssl-ptest_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 44M Sep 28 18:33
binutils-dbg_2.39-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 26M Sep 28 18:33
perl-ptest_5.36.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 23M Sep 28 18:33
gcc_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 15M Sep 28 18:32
elfutils-ptest_0.187-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 15M Sep 28 18:33
gcc-src_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 13M Sep 28 18:33
coreutils-dbg_9.1-r0_core2-64.ipk

zstd at level 19:

build$ ls -lh tmp-zstd-level19/deploy/ipk/core2-64/ --sort=size | head -n10 total 1.1G
-rw-r--r-- 3 ecordonnier ecordonnier 241M Sep 28 18:04 gcc-dbg_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 213M Sep 28 18:03 openssl-dbg_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 36M Sep 28 17:58
binutils-dbg_2.39-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 30M Sep 28 18:00
openssl-ptest_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 22M Sep 28 17:59
perl-ptest_5.36.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 20M Sep 28 17:58
gcc_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 14M Sep 28 17:57
elfutils-ptest_0.187-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 12M Sep 28 17:58
gcc-src_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 12M Sep 28 17:57
g++_12.2.0-r0_core2-64.ipk

xz at level 9 (the default):

build$ ls -lh tmp-xz/deploy/ipk/core2-64/ --sort=size | head -n10 total 963M
-rw-r--r-- 3 ecordonnier ecordonnier 214M Sep 14 10:44 gcc-dbg_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 193M Sep 14 10:44 openssl-dbg_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 35M Sep 14 10:44
openssl-ptest_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 32M Sep 14 10:44
binutils-dbg_2.39-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 20M Sep 14 10:44
perl-ptest_5.36.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 19M Sep 14 10:44
gcc_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 13M Sep 14 10:44
elfutils-ptest_0.187-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 12M Sep 14 10:44
gcc-src_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 11M Sep 14 10:44
g++_12.2.0-r0_core2-64.ipk

Quoted 378 lines · Khem Raj
Quoted 7 lines · Alex Stewart

Thanks for checking.

I'd be interested to know if setting a higher compression level for zstd can get us to a similar compression ratio to xz. If so, then I think it could be some real value to distro maintainers to be able to *tune* their compression.

yeah it will be interesting to say try something like level 9 but I think times
might regress with that but it might be good to know the balance and perhaps
suggest size mode and performance mode of zstd instead of xz

Quoted 71 lines

That's not blocking for your new PR though.

Quoted 66 lines · Etienne Cordonnier

Also note that zstd's default compression level is 3 per default (from a 1 to 19 scale). I did not test other compression levels.

On Wed, Sep 14, 2022 at 11:58 AM Etienne Cordonnier <[email protected]> wrote:

I ran a build of core-image-full-cmdline using xz and zstd, using pre-populated downloads and sstate-cache directories but with empty tmp directory. Here are the numbers:
zstd:
bitbake core-image-full-cmdline took 2m52.768s (real), the resulting directory tmp/deploy/ipk is 1.6GB big.
xz:
bitbake core-image-full-cmdline took 4m14.214s (real), the resulting directory tmp/deploy/ipk/ is 996M big

So the build with xz is 47% slower (254/172) than with zstd for this "incremental build" use-case.

See the 5 biggest packages, the difference in compression-ratio increases with big files:
build$ ls -lh tmp-zstd/deploy/ipk/core2-64/ --sort=size | head -n5 total 1.6G
-rw-r--r-- 3 ecordonnier ecordonnier 331M Sep 14 10:39 gcc-dbg_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 264M Sep 14 10:39 openssl-dbg_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 113M Sep 14 10:39 openssl-ptest_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 47M Sep 14 10:39
binutils-dbg_2.39-r0_core2-64.ipk
build$ ls -lh tmp-xz/deploy/ipk/core2-64/ --sort=size | head -n5 total 963M
-rw-r--r-- 3 ecordonnier ecordonnier 214M Sep 14 10:44 gcc-dbg_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 193M Sep 14 10:44 openssl-dbg_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 35M Sep 14 10:44
openssl-ptest_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier 32M Sep 14 10:44
binutils-dbg_2.39-r0_core2-64.ipk
...

I think for use-cases where the size of the ipk packages matters, xz is better. For use-cases where it does not matter (ipk packages not deployed), build-time matters more than compression-ratio and zstd is better.

Regarding the enablement of zstd in opkg per default, I'll send a new version of the patch without this line.
My thinking for enabling zstd per default in opkg was that zstd is already enabled per default in libarchive's PACKAGECONFIG, and disabling zstd in opkg's PACKAGECONFIG removes only a few lines of code from opkg (opkg uses libarchive for doing the actual compression/decompression).

On Tue, Sep 13, 2022 at 11:57 PM Alex Stewart
<[email protected]> wrote:

On 9/13/22 15:20, Alex Feinman wrote:
> I do have some numbers. When I was selling this change internally, I
> did a comparison on our internal build.
> Combined write IPK times (Σ t

do_package_write_ipk)

Quoted 46 lines
Quoted 46 lines
        > xz         162m 35s
        > gz         52m 13s
        > zstd       33m 49s

> Compression rate for zstd was closer to xz than to gz but not as good
> as xz. For systems that have to cache packages on the device with
> limited storage xz might be a better option, but for the bulk of
> projects zstd is the best choice
> Additionally, zstd offers much faster decompression than xz so the
> rootfs build step that includes unpacking all of the ipks, takes 3m
> 58s with xz and 2m 44s with zstd.
> One other thing of note - if your build includes debug packages, some
> may be quite large. E.g. one of our components produces a 2.2 GB debug
> package (uncompressed). On large files xz requires a disproportionally
> large amount of time resulting in 15 minutes needed to simply write
> ipk for the abovementioned packages, whereas zstd took about 45 sec.
> For frequent tasks like bitbaking a single package this translates in
> a lot of saved time.

Those are certainly compelling performance improvements. Assuming that
the final data-segment size is within 5%-ish of xz, then I would agree
with the rest of the thread that it should probably be the contemporary
default.

And if we make it the default compressor for OE IPKs, then obviously my
criticism in the original PR is satisfied.

> Bottom line - I think making xz a default package compressor was not
> entirely thought through. gzip or zstd is what the default should be.

Quoted 6 lines
Quoted 6 lines

ZStandard support was only added to opkg last September [1]. Before
that, xz was the new hotness that replaced gzip. :)

[1]

https://urldefense.proofpoint.com/v2/url?u=https-3A__git.yoctoproject.org_opkg_commit_-3Fid-3D5dead419e94bce2e6b743ad786c1daec0e1aa294&d=DwIDaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsyDLWdbBqarUzFxz3aALck&m=VG_fgpCGq8Zgu73KQP1wTtb1D1TZftNpXj8jGnouPtGIBYyLIZbG8F-85POUcVN7&s=Mwoq2kkjfZhto6J5OomduJ5Rhyg_oSe-dkjeltE4Ls8&e=

Quoted 1 line
Quoted 1 line
        <

https://urldefense.proofpoint.com/v2/url?u=https-3A__git.yoctoproject.org_opkg_commit_-3Fid-3D5dead419e94bce2e6b743ad786c1daec0e1aa294&d=DwIDaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsyDLWdbBqarUzFxz3aALck&m=VG_fgpCGq8Zgu73KQP1wTtb1D1TZftNpXj8jGnouPtGIBYyLIZbG8F-85POUcVN7&s=Mwoq2kkjfZhto6J5OomduJ5Rhyg_oSe-dkjeltE4Ls8&e=

Quoted 14 lines
Quoted 14 lines

> One final note: I could not find a reasonable explanation for why
> opkg-tools require code changes to support a different compressor. BSD
> tar and GNU tar both can easily accept compressors that they have no
> idea about (via -I option) because all of them provide a unified
> command line interface for use in pipes. If this were done similar to
> tar, we could have used any compressor we wanted, including

the

Quoted 12 lines
Quoted 12 lines

> multithreaded versions (zstdmt)

Well, presumably IPK creation tools can only support the matrix of
compression algorithms which your opkg binary can decompress. I suppose
someone could try to implement a plugable compression module system for
opkg. But given that nearly everyone uses opkg in an embedded context,
I'm not sure it would get much use.

Quoted 12 lines
Quoted 4 lines
        >
        > On Tue, Sep 13, 2022 at 12:43 PM Khem Raj
        <[email protected]> wrote:
        >
Quoted 8 lines
        >     On Tue, Sep 13, 2022 at 12:19 PM Alex Stewart
        >     <[email protected]> wrote:
        >     >
        >     > ACK from me - apart from enabling zstd by default.
        >     >
        >     > On 9/13/22 07:37, Etienne Cordonnier via
        lists.openembedded.org
        <

https://urldefense.com/v3/__http://lists.openembedded.org__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFhOF1_vg$

Quoted 3 lines
Quoted 3 lines
        >
         <

https://urldefense.com/v3/__http://lists.openembedded.org__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_bxrNjvo$

Quoted 3 lines
Quoted 3 lines

> wrote:
> > > This allows the use of zstd for opkg packages by

using

Quoted 5 lines
Quoted 5 lines
        >     OPKGBUILDCMD:
        >     > > OPKGBUILDCMD = "opkg-build -Z zstd"
        >     > >
        >     > > Signed-off-by: Alex Feinman <[email protected]>
        >     > > Signed-off-by: Etienne Cordonnier <

[email protected]>

Quoted 3 lines
Quoted 3 lines
        >     > > ---
        >     > >   meta/recipes-devtools/opkg/opkg_0.6.0.bb
        <

https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$

Quoted 3 lines
Quoted 3 lines
        >
         <

https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$

Quoted 7 lines
Quoted 7 lines
        >     | 3 ++-
        >     > >   1 file changed, 2 insertions(+), 1 deletion(-)
        >     > >
        >     > > diff --git
        a/meta/recipes-devtools/opkg/opkg_0.6.0.bb
        <

https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$

Quoted 3 lines
Quoted 3 lines
        >
         <

https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$

Quoted 3 lines
Quoted 3 lines
        >     b/meta/recipes-devtools/opkg/opkg_0.6.0.bb
        <

https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$

Quoted 3 lines
Quoted 3 lines
        >
         <

https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$

Quoted 4 lines
Quoted 4 lines
        >     > > index 7b351e8123..e38d9d6f3f 100644
        >     > > --- a/meta/recipes-devtools/opkg/opkg_0.6.0.bb
        <

https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$

Quoted 3 lines
Quoted 3 lines
        >
         <

https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$

Quoted 3 lines
Quoted 3 lines
        >     > > +++ b/meta/recipes-devtools/opkg/opkg_0.6.0.bb
        <

https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$

Quoted 3 lines
Quoted 3 lines
        >
         <

https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$

Quoted 14 lines
Quoted 9 lines
        >     > > @@ -30,7 +30,7 @@ inherit autotools pkgconfig ptest
        >     > >   target_localstatedir := "${localstatedir}"
        >     > >   OPKGLIBDIR ??= "${target_localstatedir}/lib"
        >     > >
        >     > > -PACKAGECONFIG ??= "libsolv"
        >     > > +PACKAGECONFIG ??= "libsolv zstd"
        >     >
        >     > Building in zstd support by default is a little
Quoted 5 lines

suspect to me.
> >
> > Unless I'm mistaken, OE-core will only build
xz-compressed IPKs by
> > default. So zstd support would be unnecessary for a

distro

Quoted 50 lines
Quoted 50 lines
        >     integrator
        >     > who just uses upstream OE-core.
        >     >
        >     > For distros which use zstd compression in their
        packages, I think it
        >     > would be more appropriate to overwrite the opkg
        PACKAGECONFIG in a
        >     > .bbappend.
        >     >
        >
        >     This is perhaps fine. I do wonder if there is some
        performance
        >     comparison data between xz and zstd compressed ipks
        >     with opkg, it might help users on making this choice and
        also if we
        >     should consider using
        >     zstd by default at some point or not.
        >
        >     > Is there something I'm not considering here?
        >     >
        >     > >
        >     > >   PACKAGECONFIG[gpg] = "--enable-gpg,--disable-gpg,\
        >     > >       gnupg gpgme libgpg-error,\
        >     > > @@ -39,6 +39,7 @@ PACKAGECONFIG[gpg] =
        >     "--enable-gpg,--disable-gpg,\
        >     > >   PACKAGECONFIG[curl] =
        "--enable-curl,--disable-curl,curl"
        >     > >   PACKAGECONFIG[ssl-curl] =
        >     "--enable-ssl-curl,--disable-ssl-curl,curl openssl"
        >     > >   PACKAGECONFIG[sha256] =
        "--enable-sha256,--disable-sha256"
        >     > > +PACKAGECONFIG[zstd] =
        "--enable-zstd,--disable-zstd,zstd"
        >     > >   PACKAGECONFIG[libsolv] =
        >     "--with-libsolv,--without-libsolv,libsolv"
        >     > >
        >     > >   EXTRA_OECONF:class-native =
        >  "--localstatedir=/${@os.path.relpath('${localstatedir}',
        >     '${STAGING_DIR_NATIVE}')}
        >     --sysconfdir=/${@os.path.relpath('${sysconfdir}',
        >     '${STAGING_DIR_NATIVE}')}"
        >     > >
        >     > >
        >     > >
        >     >
        >     > --
        >     > Alex Stewart
        >     > Software Engineer - NI Real-Time OS
        >     > NI (National Instruments)
        >     >
Quoted 8 lines
Quoted 8 lines
        >     > [email protected]
        >     >
        >     >
        >     > -=-=-=-=-=-=-=-=-=-=-=-
        >     > Links: You receive all messages sent to this group.
        >     > View/Reply Online (#170608):
        >

https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_170608&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=_VWADuvDS3QdSEYWTnh8wNH0lOxVHty18QZynvOtpcU&e=

Quoted 1 line
Quoted 1 line
        <

https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_170608&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=_VWADuvDS3QdSEYWTnh8wNH0lOxVHty18QZynvOtpcU&e=

Quoted 3 lines
Quoted 3 lines
        >
         <

https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_170608&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=_VWADuvDS3QdSEYWTnh8wNH0lOxVHty18QZynvOtpcU&e=

Quoted 1 line
Quoted 1 line
        <

https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_170608&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=_VWADuvDS3QdSEYWTnh8wNH0lOxVHty18QZynvOtpcU&e=

Quoted 5 lines
Quoted 5 lines
        >
        >     > Mute This Topic:
        >

https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_93654146_1997914&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=QncdL2I_pCDsz7AMbsTgStzC7bt5V2x_E_gIblrtjZY&e=

Quoted 1 line
Quoted 1 line
        <

https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_93654146_1997914&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=QncdL2I_pCDsz7AMbsTgStzC7bt5V2x_E_gIblrtjZY&e=

Quoted 3 lines
Quoted 3 lines
        >
         <

https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_93654146_1997914&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=QncdL2I_pCDsz7AMbsTgStzC7bt5V2x_E_gIblrtjZY&e=

Quoted 1 line
Quoted 1 line
        <

https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_93654146_1997914&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=QncdL2I_pCDsz7AMbsTgStzC7bt5V2x_E_gIblrtjZY&e=

Quoted 10 lines
Quoted 10 lines

>
> > Group Owner:
[email protected] <mailto:openembedded-core%[email protected]>
> <mailto:openembedded-core%[email protected]
<mailto:openembedded-core%[email protected]>>
> > Unsubscribe:
>

https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=PxAE_3yGzbSZ21-LHz_1r3gd2_STPoZcrYDeu9vd_GM&e=

Quoted 1 line
Quoted 1 line
        <

https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=PxAE_3yGzbSZ21-LHz_1r3gd2_STPoZcrYDeu9vd_GM&e=

Quoted 3 lines
Quoted 3 lines
        >
         <

https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=PxAE_3yGzbSZ21-LHz_1r3gd2_STPoZcrYDeu9vd_GM&e=

Quoted 1 line
Quoted 1 line
        <

https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=PxAE_3yGzbSZ21-LHz_1r3gd2_STPoZcrYDeu9vd_GM&e=

Quoted 21 lines
Quoted 13 lines
        >      [[email protected]]
        >     > -=-=-=-=-=-=-=-=-=-=-=-
        >     >
        >

        --
        Alex Stewart
        Software Engineer - NI Real-Time OS
        NI (National Instruments)

        [email protected]

--
Alex Stewart
Software Engineer - NI Real-Time OS
NI (National Instruments)

[email protected]

#12 lore raw

Thanks for doing the research.

What was the build time on the zstd L19 compression vs. xz? Was there still an improvement?

Quoted 97 lines · Etienne Cordonnier

I tested it a bit more today. I used the standard poky's local.conf and added those lines. You can see that at zstd level 9 there is still a significant difference in the compression ratio with xz for gcc-dbg which is a big file. At zstd level 19, gcc-dbg is 241MB big instead of 214MB with xz (12.6% more).

PACKAGE_CLASSES = "package_ipk"
ZSTD_DEFAULTS = "-T0 -19" PACKAGECONFIG:append:pn-opkg-native = " zstd" OPKGBUILDCMD = "opkg-build -Z zstd -a ${ZSTD_DEFAULTS}"

Zstd at level 9:

build$ ls -lh tmp-zstd-level9/deploy/ipk/core2-64/  --sort=size | head -n10
total 1.5G
-rw-r--r-- 3 ecordonnier ecordonnier  302M Sep 28 18:33 gcc-dbg_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier  249M Sep 28 18:33 openssl-dbg_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier  106M Sep 28 18:33 openssl-ptest_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier   44M Sep 28 18:33 binutils-dbg_2.39-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier   26M Sep 28 18:33 perl-ptest_5.36.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier   23M Sep 28 18:33 gcc_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier   15M Sep 28 18:32 elfutils-ptest_0.187-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier   15M Sep 28 18:33 gcc-src_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier   13M Sep 28 18:33 coreutils-dbg_9.1-r0_core2-64.ipk

zstd at level 19:

build$ ls -lh tmp-zstd-level19/deploy/ipk/core2-64/  --sort=size | head -n10 total 1.1G
-rw-r--r-- 3 ecordonnier ecordonnier  241M Sep 28 18:04 gcc-dbg_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier  213M Sep 28 18:03 openssl-dbg_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier   36M Sep 28 17:58 binutils-dbg_2.39-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier   30M Sep 28 18:00 openssl-ptest_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier   22M Sep 28 17:59 perl-ptest_5.36.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier   20M Sep 28 17:58 gcc_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier   14M Sep 28 17:57 elfutils-ptest_0.187-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier   12M Sep 28 17:58 gcc-src_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier   12M Sep 28 17:57 g++_12.2.0-r0_core2-64.ipk

xz at level 9 (the default):

build$ ls -lh tmp-xz/deploy/ipk/core2-64/  --sort=size | head -n10 total 963M
-rw-r--r-- 3 ecordonnier ecordonnier  214M Sep 14 10:44 gcc-dbg_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier  193M Sep 14 10:44 openssl-dbg_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier   35M Sep 14 10:44 openssl-ptest_3.0.5-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier   32M Sep 14 10:44 binutils-dbg_2.39-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier   20M Sep 14 10:44 perl-ptest_5.36.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier   19M Sep 14 10:44 gcc_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier   13M Sep 14 10:44 elfutils-ptest_0.187-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier   12M Sep 14 10:44 gcc-src_12.2.0-r0_core2-64.ipk
-rw-r--r-- 3 ecordonnier ecordonnier   11M Sep 14 10:44 g++_12.2.0-r0_core2-64.ipk

On Wed, Sep 14, 2022 at 5:42 PM Khem Raj <[email protected]> wrote:

On Wed, Sep 14, 2022 at 8:37 AM Alex Stewart <[email protected]> wrote:
>
> Thanks for checking.
>
> I'd be interested to know if setting a higher compression level for zstd
> can get us to a similar compression ratio to xz. If so, then I think it
> could be some real value to distro maintainers to be able to *tune* > their compression.

yeah it will be interesting to say try something like level 9 but I think times
might regress with that but it might be good to know the balance and perhaps
suggest size mode and performance mode of zstd instead of xz

Signature · 5 lines
Alex Stewart
Software Engineer - NI Real-Time OS
NI (National Instruments)

[email protected]