Skip to contentSkip to conversation

bitbake.conf: Add lz4c, pzstd and zstd

PATCH In review Joshua Watt 8 participants 14 messages Jul 14, 2021 – Aug 19, 2021

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 bitbake.conf: Add lz4c, pzstd and zstd

Adds the command line compression tools which are needed by bitbake

---
 meta/conf/bitbake.conf | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
meta/conf/bitbake.conf +3 -3
index f451ba6a47..3801db7d42 100644
@@ -484,12 +484,12 @@ HOSTTOOLS += " \
     [ ar as awk basename bash bzip2 cat chgrp chmod chown chrpath cmp comm cp cpio \
     cpp cut date dd diff diffstat dirname du echo egrep env expand expr false \
     fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip gzip \
-    head hostname iconv id install ld ldd ln ls make md5sum mkdir mkfifo mknod \
+    head hostname iconv id install ld ldd ln ls lz4c make md5sum mkdir mkfifo mknod \
     mktemp mv nm objcopy objdump od patch perl pr printf pwd \
-    python3 ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \
+    python3 pzstd ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \
     sha1sum sha224sum sha256sum sha384sum sha512sum \
     sleep sort split stat strings strip tail tar tee test touch tr true uname \
-    uniq wc wget which xargs \
+    uniq wc wget which xargs zstd \
 "
 
 # Tools needed to run testimage runtime image testing
Signature · 1 line
2.32.0
#1 lore raw

While I welcome the transition to zstd, I think the commit message needs to be more descriptive. Needed by bitbake where and how? Is there some patchset upcoming?

Alex

Quoted 41 lines · Joshua Watt

Adds the command line compression tools which are needed by bitbake

---
 meta/conf/bitbake.conf | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
meta/conf/bitbake.conf +2 -2
index f451ba6a47..3801db7d42 100644
@@ -484,12 +484,12 @@ HOSTTOOLS += " \
     [ ar as awk basename bash bzip2 cat chgrp chmod chown chrpath cmp
 comm cp cpio \
     cpp cut date dd diff diffstat dirname du echo egrep env expand expr
 false \
     fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip
 gzip \
-    head hostname iconv id install ld ldd ln ls make md5sum mkdir mkfifo
 mknod \
+    head hostname iconv id install ld ldd ln ls lz4c make md5sum mkdir
 mkfifo mknod \
     mktemp mv nm objcopy objdump od patch perl pr printf pwd \
-    python3 ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \
+    python3 pzstd ranlib readelf readlink realpath rm rmdir rpcgen sed
 seq sh \

sha1sum sha224sum sha256sum sha384sum sha512sum \
sleep sort split stat strings strip tail tar tee test touch tr true
uname \

meta/conf/bitbake.conf +1 -2
-    uniq wc wget which xargs \
+    uniq wc wget which xargs zstd \
 "
 
 # Tools needed to run testimage runtime image testing
--

2.32.0

#2 lore raw
Quoted 1 line · Alexander Kanavin

While I welcome the transition to zstd, I think the commit message needs to be more descriptive. Needed by bitbake where and how? Is there some patchset upcoming?

Ya, this is necessary for the patch: https://lists.openembedded.org/g/bitbake-devel/message/12385, which isn't merged yet.

Quoted 47 lines

Alex

On Wed, 14 Jul 2021 at 15:42, Joshua Watt <[email protected] <mailto:[email protected]>> wrote:

    Adds the command line compression tools which are needed by bitbake

    Signed-off-by: Joshua Watt <[email protected]
    <mailto:[email protected]>>
    ---
     meta/conf/bitbake.conf | 6 +++---
     1 file changed, 3 insertions(+), 3 deletions(-)

    diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
    index f451ba6a47..3801db7d42 100644
    --- a/meta/conf/bitbake.conf
    +++ b/meta/conf/bitbake.conf
    @@ -484,12 +484,12 @@ HOSTTOOLS += " \
         [ ar as awk basename bash bzip2 cat chgrp chmod chown chrpath
    cmp comm cp cpio \
         cpp cut date dd diff diffstat dirname du echo egrep env
    expand expr false \
         fgrep file find flock g++ gawk gcc getconf getopt git grep
    gunzip gzip \
    -    head hostname iconv id install ld ldd ln ls make md5sum mkdir
    mkfifo mknod \
    +    head hostname iconv id install ld ldd ln ls lz4c make md5sum
    mkdir mkfifo mknod \
         mktemp mv nm objcopy objdump od patch perl pr printf pwd \
    -    python3 ranlib readelf readlink realpath rm rmdir rpcgen sed
    seq sh \
    +    python3 pzstd ranlib readelf readlink realpath rm rmdir
    rpcgen sed seq sh \
         sha1sum sha224sum sha256sum sha384sum sha512sum \
         sleep sort split stat strings strip tail tar tee test touch
    tr true uname \
    -    uniq wc wget which xargs \
    +    uniq wc wget which xargs zstd \
     "

     # Tools needed to run testimage runtime image testing
    -- 
    2.32.0
#3 lore raw

Right, but that still only sets up the infra and tests :) What will use it?

Alex

Quoted 58 lines · Joshua Watt

On 7/14/21 8:45 AM, Alexander Kanavin wrote:

While I welcome the transition to zstd, I think the commit message needs to be more descriptive. Needed by bitbake where and how? Is there some patchset upcoming?

Ya, this is necessary for the patch:
https://lists.openembedded.org/g/bitbake-devel/message/12385, which isn't merged yet.

Alex

Quoted 41 lines · Joshua Watt

Adds the command line compression tools which are needed by bitbake

---
 meta/conf/bitbake.conf | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
meta/conf/bitbake.conf +2 -2
index f451ba6a47..3801db7d42 100644
@@ -484,12 +484,12 @@ HOSTTOOLS += " \
     [ ar as awk basename bash bzip2 cat chgrp chmod chown chrpath cmp
 comm cp cpio \
     cpp cut date dd diff diffstat dirname du echo egrep env expand expr
 false \
     fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip
 gzip \
-    head hostname iconv id install ld ldd ln ls make md5sum mkdir mkfifo
 mknod \
+    head hostname iconv id install ld ldd ln ls lz4c make md5sum mkdir
 mkfifo mknod \
     mktemp mv nm objcopy objdump od patch perl pr printf pwd \
-    python3 ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \
+    python3 pzstd ranlib readelf readlink realpath rm rmdir rpcgen sed
 seq sh \

sha1sum sha224sum sha256sum sha384sum sha512sum \
sleep sort split stat strings strip tail tar tee test touch tr true
uname \

meta/conf/bitbake.conf +1 -2
-    uniq wc wget which xargs \
+    uniq wc wget which xargs zstd \
 "
 
 # Tools needed to run testimage runtime image testing
--

2.32.0

#4 lore raw
In reply to Joshua Watt

Hello,

Quoted 14 lines · Joshua Watt

Adds the command line compression tools which are needed by bitbake

---
 meta/conf/bitbake.conf | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
meta/conf/bitbake.conf +0 -0
index f451ba6a47..3801db7d42 100644
@@ -484,12 +484,12 @@ HOSTTOOLS += " \
     [ ar as awk basename bash bzip2 cat chgrp chmod chown chrpath cmp comm cp cpio \
     cpp cut date dd diff diffstat dirname du echo egrep env expand expr false \

meta/conf/bitbake.conf HOSTTOOLS += " \ · line 492

Quoted 10 lines
meta/conf/bitbake.conf +3 -3
     fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip gzip \
-    head hostname iconv id install ld ldd ln ls make md5sum mkdir mkfifo mknod \
+    head hostname iconv id install ld ldd ln ls lz4c make md5sum mkdir mkfifo mknod \
     mktemp mv nm objcopy objdump od patch perl pr printf pwd \
-    python3 ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \
+    python3 pzstd ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \
     sha1sum sha224sum sha256sum sha384sum sha512sum \
     sleep sort split stat strings strip tail tar tee test touch tr true uname \
-    uniq wc wget which xargs \
+    uniq wc wget which xargs zstd \

zstd seems to hurt reproducibility:

AssertionError: The following deb packages are missing or different and not in exclusion list: /home/pokybuild/yocto-worker/reproducible-ubuntu/build/build-st/reproducibleB/tmp/deploy/deb/./core2-64/elfutils-ptest_0.185-r0_amd64.deb /home/pokybuild/yocto-worker/reproducible-ubuntu/build/build-st/reproducibleB/tmp/deploy/deb/./core2-64/man-db-dbg_2.9.4-r0_amd64.deb /home/pokybuild/yocto-worker/reproducible-ubuntu/build/build-st/reproducibleB/tmp/deploy/deb/./core2-64/man-db-src_2.9.4-r0_amd64.deb /home/pokybuild/yocto-worker/reproducible-ubuntu/build/build-st/reproducibleB/tmp/deploy/deb/./core2-64/man-db_2.9.4-r0_amd64.deb
The following ipk packages are missing or different and not in exclusion list: /home/pokybuild/yocto-worker/reproducible-ubuntu/build/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/elfutils-ptest_0.185-r0_core2-64.ipk /home/pokybuild/yocto-worker/reproducible-ubuntu/build/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/man-db-dbg_2.9.4-r0_core2-64.ipk /home/pokybuild/yocto-worker/reproducible-ubuntu/build/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/man-db-src_2.9.4-r0_core2-64.ipk /home/pokybuild/yocto-worker/reproducible-ubuntu/build/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/man-db_2.9.4-r0_core2-64.ipk
The following rpm packages are missing or different and not in exclusion list: /home/pokybuild/yocto-worker/reproducible-ubuntu/build/build-st/reproducibleB/tmp/deploy/rpm/./core2_64/elfutils-ptest-0.185-r0.core2_64.rpm /home/pokybuild/yocto-worker/reproducible-ubuntu/build/build-st/reproducibleB/tmp/deploy/rpm/./core2_64/man-db-2.9.4-r0.core2_64.rpm /home/pokybuild/yocto-worker/reproducible-ubuntu/build/build-st/reproducibleB/tmp/deploy/rpm/./core2_64/man-db-dbg-2.9.4-r0.core2_64.rpm /home/pokybuild/yocto-worker/reproducible-ubuntu/build/build-st/reproducibleB/tmp/deploy/rpm/./core2_64/man-db-src-2.9.4-r0.core2_64.rpm

diffoscope output here:

https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20210814-qe8khapn/packages/diff-html/

I would think this was expected...

Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#5 lore raw
In reply to Joshua Watt
Quoted 17 lines · Joshua Watt

Adds the command line compression tools which are needed by bitbake

---
 meta/conf/bitbake.conf | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
meta/conf/bitbake.conf +0 -1
index f451ba6a47..3801db7d42 100644
@@ -484,12 +484,12 @@ HOSTTOOLS += " \
     [ ar as awk basename bash bzip2 cat chgrp chmod chown chrpath cmp comm cp cpio \
     cpp cut date dd diff diffstat dirname du echo egrep env expand expr false \
     fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip gzip \
-    head hostname iconv id install ld ldd ln ls make md5sum mkdir mkfifo mknod \

meta/conf/bitbake.conf HOSTTOOLS += " \ · line 494

Quoted 10 lines
meta/conf/bitbake.conf +3 -2
+    head hostname iconv id install ld ldd ln ls lz4c make md5sum mkdir mkfifo mknod \
     mktemp mv nm objcopy objdump od patch perl pr printf pwd \
-    python3 ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \
+    python3 pzstd ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \
     sha1sum sha224sum sha256sum sha384sum sha512sum \
     sleep sort split stat strings strip tail tar tee test touch tr true uname \
-    uniq wc wget which xargs \
+    uniq wc wget which xargs zstd \
 "
 

this should also be reflecting on the distro prerequisites for supported distros.

Quoted 7 lines
meta/conf/bitbake.conf +0 -1
 # Tools needed to run testimage runtime image testing
--

2.32.0

#6 lore raw
In reply to Joshua Watt
Quoted 17 lines · Joshua Watt

Adds the command line compression tools which are needed by bitbake

---
 meta/conf/bitbake.conf | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
meta/conf/bitbake.conf +0 -1
index f451ba6a47..3801db7d42 100644
@@ -484,12 +484,12 @@ HOSTTOOLS += " \
     [ ar as awk basename bash bzip2 cat chgrp chmod chown chrpath cmp comm cp cpio \
     cpp cut date dd diff diffstat dirname du echo egrep env expand expr false \
     fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip gzip \
-    head hostname iconv id install ld ldd ln ls make md5sum mkdir mkfifo mknod \

meta/conf/bitbake.conf HOSTTOOLS += " \ · line 494

Quoted 10 lines
meta/conf/bitbake.conf +3 -2
+    head hostname iconv id install ld ldd ln ls lz4c make md5sum mkdir mkfifo mknod \
     mktemp mv nm objcopy objdump od patch perl pr printf pwd \
-    python3 ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \
+    python3 pzstd ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \
     sha1sum sha224sum sha256sum sha384sum sha512sum \
     sleep sort split stat strings strip tail tar tee test touch tr true uname \
-    uniq wc wget which xargs \
+    uniq wc wget which xargs zstd \
 "
 

on ubuntu 18.04 we will need

sudo apt install zstd liblz4-tool

Quoted 7 lines
meta/conf/bitbake.conf +0 -1
 # Tools needed to run testimage runtime image testing
--

2.32.0

#7 lore raw
Quoted 31 lines · Khem Raj
Quoted 17 lines · Joshua Watt

Adds the command line compression tools which are needed by bitbake

---
  meta/conf/bitbake.conf | 6 +++---
  1 file changed, 3 insertions(+), 3 deletions(-)
meta/conf/bitbake.conf +0 -1
index f451ba6a47..3801db7d42 100644
@@ -484,12 +484,12 @@ HOSTTOOLS += " \
      [ ar as awk basename bash bzip2 cat chgrp chmod chown chrpath cmp comm cp cpio \
      cpp cut date dd diff diffstat dirname du echo egrep env expand expr false \
      fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip gzip \
-    head hostname iconv id install ld ldd ln ls make md5sum mkdir mkfifo mknod \

meta/conf/bitbake.conf HOSTTOOLS += " \ · line 494

Quoted 10 lines
meta/conf/bitbake.conf +3 -2
+    head hostname iconv id install ld ldd ln ls lz4c make md5sum mkdir mkfifo mknod \
      mktemp mv nm objcopy objdump od patch perl pr printf pwd \
-    python3 ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \
+    python3 pzstd ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \
      sha1sum sha224sum sha256sum sha384sum sha512sum \
      sleep sort split stat strings strip tail tar tee test touch tr true uname \
-    uniq wc wget which xargs \
+    uniq wc wget which xargs zstd \
  "
 

on ubuntu 18.04 we will need

Quoted 1 line

sudo apt install zstd liblz4-tool

Michael,

Can you update the docs for this change?

http://docs.yoctoproject.org/ref-manual/system-requirements.html?highlight=bzip2

I don't know what the package names are for non-debian distros.

This must have been done for the YP AB cluster but maybe that's done manually?

../Randy

Quoted 12 lines
Quoted 12 lines
meta/conf/bitbake.conf +0 -1
  # Tools needed to run testimage runtime image testing
--

2.32.0

Signature · 2 lines
# Randy MacLeod
# Wind River Linux
#8 lore raw

Hi,

I kind of missed out on this change... but I got caught up by reality hard this morning :-(.

I currently don't see a reason why zstd is required on the host - only systemd is having a hard dependency against zstd atm, while the other lately added references are just optional PACKAGECONFIGs (and there are plenty of them which have loose ends - in poky and meta-oe)

So the question I want to raise: why a hard dependency? wouldn't have been HOSTTOOLS_NONFATAL enough?

I mean this is the second hard cut in the project within just weeks and this time it was host related, which is even harder to fix in a timely manner in a corporate environment (basically rolling out changes to all dev installations isn't something I fancy on a Wednesday morning :-( )

None of the commit messages somehow mentions, why we all of a sudden need that on the host, instead of building it from sources (which should be possible judging from the latest master sources).

And on that note: that the documentation wasn't updated in the same changeset is very unfortunate (not even the crops container where updated)

/rant

Konrad

Quoted 70 lines · Randy MacLeod
Quoted 35 lines · Khem Raj
Quoted 29 lines · Joshua Watt

Adds the command line compression tools which are needed by bitbake

---
  meta/conf/bitbake.conf | 6 +++---

  1 file changed, 3 insertions(+), 3 deletions(-)
meta/conf/bitbake.conf +3 -3
index f451ba6a47..3801db7d42 100644
@@ -484,12 +484,12 @@ HOSTTOOLS += " \
       [ ar as awk basename bash bzip2 cat chgrp chmod chown chrpath cmp comm cp cpio \
       cpp cut date dd diff diffstat dirname du echo egrep env expand expr false \
       fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip gzip \
-    head hostname iconv id install ld ldd ln ls make md5sum mkdir 
 mkfifo mknod \
+    head hostname iconv id install ld ldd ln ls lz4c make md5sum 
 mkdir mkfifo mknod \
       mktemp mv nm objcopy objdump od patch perl pr printf pwd \
-    python3 ranlib readelf readlink realpath rm rmdir rpcgen sed seq 
 sh \
+    python3 pzstd ranlib readelf readlink realpath rm rmdir rpcgen 
 sed seq sh \
       sha1sum sha224sum sha256sum sha384sum sha512sum \
       sleep sort split stat strings strip tail tar tee test touch tr true uname \
-    uniq wc wget which xargs \
+    uniq wc wget which xargs zstd \
Quoted 2 lines

  "

on ubuntu 18.04 we will need

Quoted 1 line

sudo apt install zstd liblz4-tool

Michael,

Can you update the docs for this change?

http://docs.yoctoproject.org/ref-manual/system-requirements.html?highlight=bzip2

I don't know what the package names are for non-debian distros.

This must have been done for the YP AB cluster but maybe that's done manually?

../Randy

Quoted 12 lines
Quoted 12 lines

  # Tools needed to run testimage runtime image testing

Signature · 1 line
2.32.0
#9 lore raw

Hi Konrad,

Quoted 6 lines · Konrad Weihmann

I mean this is the second hard cut in the project within just weeks and this time it was host related, which is even harder to fix in a timely manner in a corporate environment (basically rolling out changes to all dev installations isn't something I fancy on a Wednesday morning :-( )

The choice to follow bleeding edge, means bleeding at some points in time.

Why do you follow master on all dev installations if you are not able to match (slightly) disruptive changes?

I would recommend to prevalidate changes on one instance before spreading across all (inflexible) dev installations anyway.

I did not follow the rationale closely, but I think it was related to CMake dependencies as well.

Regards,

Leon.

#10 lore raw
In reply to Konrad Weihmann
Quoted 16 lines · Konrad Weihmann

I kind of missed out on this change... but I got caught up by reality hard this morning :-(.

I currently don't see a reason why zstd is required on the host - only systemd is having a hard dependency against zstd atm, while the other lately added references are just optional PACKAGECONFIGs (and there are plenty of them which have loose ends - in poky and meta-oe)

So the question I want to raise: why a hard dependency? wouldn't have been HOSTTOOLS_NONFATAL enough?

I mean this is the second hard cut in the project within just weeks and this time it was host related, which is even harder to fix in a timely manner in a corporate environment (basically rolling out changes to all dev installations isn't something I fancy on a Wednesday morning :-( )

Quoted 6 lines

None of the commit messages somehow mentions, why we all of a sudden need that on the host, instead of building it from sources (which should be possible judging from the latest master sources).

And on that note: that the documentation wasn't updated in the same changeset is very unfortunate (not even the crops container where updated)

Sorry the change is a bit of a pain. It has been in master-next for a while and we did have to jump through a number of hoops as a project to enable this (e.g. adding to buildtools and our autobuilder infrastructure).

The reason we're doing this is that we need to use compression within bitbake itself for some of our output files. We've not done this in the past as it is a pain to have the host prerequisites however there are multiple places we can potentially benefit, such as compressed pkgdata and compressed siginfo files. It is also possible we may be able to store a compressed parsed datastore.

The siginfo files and datastore could mean we can improve debugging of a number of currently hard to solve problems our users face.

Without this we're unlikely to be able to move forward in these areas. If we do it, we open up new development opportunities and potentially improved usability. We can't "just build it" as that would mean we can't use it within bitbake itself.

Sorry this isn't well described anywhere, it is mainly as we can't yet "guarantee" that this will do these things. We do know it won't happen unless we try though.

Master *is* a development branch, not a stable release so things can change.

Cheers,

Richard

#11 lore raw
Quoted 28 lines · Richard Purdie
Quoted 16 lines · Konrad Weihmann

I kind of missed out on this change... but I got caught up by reality hard this morning :-(.

I currently don't see a reason why zstd is required on the host - only systemd is having a hard dependency against zstd atm, while the other lately added references are just optional PACKAGECONFIGs (and there are plenty of them which have loose ends - in poky and meta-oe)

So the question I want to raise: why a hard dependency? wouldn't have been HOSTTOOLS_NONFATAL enough?

I mean this is the second hard cut in the project within just weeks and this time it was host related, which is even harder to fix in a timely manner in a corporate environment (basically rolling out changes to all dev installations isn't something I fancy on a Wednesday morning :-( )

Quoted 6 lines

None of the commit messages somehow mentions, why we all of a sudden need that on the host, instead of building it from sources (which should be possible judging from the latest master sources).

And on that note: that the documentation wasn't updated in the same changeset is very unfortunate (not even the crops container where updated)

Sorry the change is a bit of a pain. It has been in master-next for a while and we did have to jump through a number of hoops as a project to enable this (e.g. adding to buildtools and our autobuilder infrastructure).

Quoted 6 lines

The reason we're doing this is that we need to use compression within bitbake itself for some of our output files. We've not done this in the past as it is a pain to have the host prerequisites however there are multiple places we can potentially benefit, such as compressed pkgdata and compressed siginfo files. It is also possible we may be able to store a compressed parsed datastore.

Thanks Richard, that's explains why it's needed

Quoted 11 lines

The siginfo files and datastore could mean we can improve debugging of a number of currently hard to solve problems our users face.

Without this we're unlikely to be able to move forward in these areas. If we do it, we open up new development opportunities and potentially improved usability. We can't "just build it" as that would mean we can't use it within bitbake itself.

Sorry this isn't well described anywhere, it is mainly as we can't yet "guarantee" that this will do these things. We do know it won't happen unless we try though.

Agree - btw maybe a good opportunity to insist on better commit messages ;-)- just out of interest: is there any guideline, maybe it's worth to at least link to a commit message guide at https://www.openembedded.org/wiki/How_to_submit_a_patch_to_OpenEmbedded or here https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines.
And slightly off-topic, but maybe also worth to do some automation, as I've seen multiple patches lately, that don't 100% follow the patch guideline in place.

Quoted 2 lines

Master *is* a development branch, not a stable release so things can change.

I've been brave enough to enforce a multi-branch development (just to keep the bar low when jumping from release to release), maybe I take that as a personal learning :)

Quoted 5 lines

Cheers,

Richard

#12 lore raw
Quoted 62 lines · Konrad Weihmann
Quoted 17 lines · Richard Purdie
Quoted 16 lines · Konrad Weihmann

I kind of missed out on this change... but I got caught up by reality hard this morning :-(.

I currently don't see a reason why zstd is required on the host - only systemd is having a hard dependency against zstd atm, while the other lately added references are just optional PACKAGECONFIGs (and there are plenty of them which have loose ends - in poky and meta-oe)

So the question I want to raise: why a hard dependency? wouldn't have been HOSTTOOLS_NONFATAL enough?

I mean this is the second hard cut in the project within just weeks and this time it was host related, which is even harder to fix in a timely manner in a corporate environment (basically rolling out changes to all dev installations isn't something I fancy on a Wednesday morning :-( )

Quoted 6 lines
Quoted 6 lines

None of the commit messages somehow mentions, why we all of a sudden need that on the host, instead of building it from sources (which should be possible judging from the latest master sources).

And on that note: that the documentation wasn't updated in the same changeset is very unfortunate (not even the crops container where

updated)

Quoted 2 lines

Sorry the change is a bit of a pain. It has been in master-next for a

while

Quoted 1 line

and we did have to jump through a number of hoops as a project to enable

this

Quoted 3 lines

(e.g. adding to buildtools and our autobuilder infrastructure).

The reason we're doing this is that we need to use compression within

bitbake

Quoted 1 line

itself for some of our output files. We've not done this in the past as

it is

Quoted 1 line

a pain to have the host prerequisites however there are multiple places

we can

Quoted 1 line

potentially benefit, such as compressed pkgdata and compressed siginfo

files.

Quoted 1 line

It is also possible we may be able to store a compressed parsed

datastore.

Quoted 1 line

Thanks Richard, that's explains why it's needed

Quoted 4 lines

The siginfo files and datastore could mean we can improve debugging of a number of currently hard to solve problems our users face.

Without this we're unlikely to be able to move forward in these areas.

If we

Quoted 2 lines

do it, we open up new development opportunities and potentially improved usability. We can't "just build it" as that would mean we can't use it

within

Quoted 5 lines

bitbake itself.

Sorry this isn't well described anywhere, it is mainly as we can't yet "guarantee" that this will do these things. We do know it won't happen unless we try though.

Quoted 9 lines

Agree - btw maybe a good opportunity to insist on better commit messages ;-)- just out of interest: is there any guideline, maybe it's worth to at least link to a commit message guide at
https://www.openembedded.org/wiki/How_to_submit_a_patch_to_OpenEmbedded
or here https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines.
And slightly off-topic, but maybe also worth to do some automation, as I've seen multiple patches lately, that don't 100% follow the patch guideline in place.

Sorry, that was my fault. I was preoccupied and didn't fix it when it was pointed out earlier.

FWIW, I do have some WIP changes that use this functionality that will hopefully land soon so it will be used :)

Quoted 17 lines
Quoted 2 lines

Master *is* a development branch, not a stable release so things can

change.

I've been brave enough to enforce a multi-branch development (just to keep the bar low when jumping from release to release), maybe I take that as a personal learning :)

Quoted 5 lines

Cheers,

Richard

#13 lore raw
In reply to Konrad Weihmann
Quoted 51 lines · Konrad Weihmann
Quoted 28 lines · Richard Purdie
Quoted 16 lines · Konrad Weihmann

I kind of missed out on this change... but I got caught up by reality hard this morning :-(.

I currently don't see a reason why zstd is required on the host - only systemd is having a hard dependency against zstd atm, while the other lately added references are just optional PACKAGECONFIGs (and there are plenty of them which have loose ends - in poky and meta-oe)

So the question I want to raise: why a hard dependency? wouldn't have been HOSTTOOLS_NONFATAL enough?

I mean this is the second hard cut in the project within just weeks and this time it was host related, which is even harder to fix in a timely manner in a corporate environment (basically rolling out changes to all dev installations isn't something I fancy on a Wednesday morning :-( )

Quoted 6 lines

None of the commit messages somehow mentions, why we all of a sudden need that on the host, instead of building it from sources (which should be possible judging from the latest master sources).

And on that note: that the documentation wasn't updated in the same changeset is very unfortunate (not even the crops container where updated)

Sorry the change is a bit of a pain. It has been in master-next for a while and we did have to jump through a number of hoops as a project to enable this (e.g. adding to buildtools and our autobuilder infrastructure).

Quoted 6 lines

The reason we're doing this is that we need to use compression within bitbake itself for some of our output files. We've not done this in the past as it is a pain to have the host prerequisites however there are multiple places we can potentially benefit, such as compressed pkgdata and compressed siginfo files. It is also possible we may be able to store a compressed parsed datastore.

Thanks Richard, that's explains why it's needed

Quoted 11 lines

The siginfo files and datastore could mean we can improve debugging of a number of currently hard to solve problems our users face.

Without this we're unlikely to be able to move forward in these areas. If we do it, we open up new development opportunities and potentially improved usability. We can't "just build it" as that would mean we can't use it within bitbake itself.

Sorry this isn't well described anywhere, it is mainly as we can't yet "guarantee" that this will do these things. We do know it won't happen unless we try though.

Quoted 8 lines

Agree - btw maybe a good opportunity to insist on better commit messages ;-)- just out of interest: is there any guideline, maybe it's worth to at least link to a commit message guide at
https://www.openembedded.org/wiki/How_to_submit_a_patch_to_OpenEmbedded
or here https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines.
And slightly off-topic, but maybe also worth to do some automation, as I've seen multiple patches lately, that don't 100% follow the patch guideline in place.

We had automation to test some of these things. It broke and we simply don't have the maintainers to look after it and keep it working, it is simply too much for Michael and I to manage to hold together.

I'm doing the best I can but I can't do this alone. Until we get more people stepping up to help with things like the automated tests, we'll have to manage with what I can do.

FWIW I had to spend a ton of time fixing buildtools, getting the workers to have the right binaries and so on to merge this change. I tried to get the docs updated but I didn't micromanage it and it didn't happen as yet. I'm aware of it, I don't know if I'll just have to go and fix that next too.

So yes, I agree nice commit messages would be nice. Please feel free to help review patches.

Cheers,

Richard