bitbake.conf: Add lz4c, pzstd and zstd
PATCH 1/1 bitbake.conf: Add lz4c, pzstd and zstd
Adds the command line compression tools which are needed by bitbake
--- meta/conf/bitbake.conf | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
meta/conf/bitbake.conf +3 -3
@@ -484,12 +484,12 @@ HOSTTOOLS += " \ [ ar as awk basename bash bzip2 cat chgrp chmod chown chrpath cmp comm cp cpio \ cpp cut date dd diff diffstat dirname du echo egrep env expand expr false \ fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip gzip \ - head hostname iconv id install ld ldd ln ls make md5sum mkdir mkfifo mknod \ + head hostname iconv id install ld ldd ln ls lz4c make md5sum mkdir mkfifo mknod \ mktemp mv nm objcopy objdump od patch perl pr printf pwd \ - python3 ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \ + python3 pzstd ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \ sha1sum sha224sum sha256sum sha384sum sha512sum \ sleep sort split stat strings strip tail tar tee test touch tr true uname \ - uniq wc wget which xargs \ + uniq wc wget which xargs zstd \ " # Tools needed to run testimage runtime image testing
Signature · 1 line
2.32.0
While I welcome the transition to zstd, I think the commit message needs to be more descriptive. Needed by bitbake where and how? Is there some patchset upcoming?
Alex
Quoted 41 lines · Joshua Watt
Adds the command line compression tools which are needed by bitbake
Signed-off-by Joshua Watt <[email protected]>--- meta/conf/bitbake.conf | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)meta/conf/bitbake.conf +2
-2@@ -484,12 +484,12 @@ HOSTTOOLS += " \ [ ar as awk basename bash bzip2 cat chgrp chmod chown chrpath cmp comm cp cpio \ cpp cut date dd diff diffstat dirname du echo egrep env expand expr false \ fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip gzip \ - head hostname iconv id install ld ldd ln ls make md5sum mkdir mkfifo mknod \ + head hostname iconv id install ld ldd ln ls lz4c make md5sum mkdir mkfifo mknod \ mktemp mv nm objcopy objdump od patch perl pr printf pwd \ - python3 ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \ + python3 pzstd ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \sha1sum sha224sum sha256sum sha384sum sha512sum \
sleep sort split stat strings strip tail tar tee test touch tr true
uname \meta/conf/bitbake.conf +1
-2- uniq wc wget which xargs \ + uniq wc wget which xargs zstd \ " # Tools needed to run testimage runtime image testing --2.32.0
Quoted 1 line · Alexander Kanavin
While I welcome the transition to zstd, I think the commit message needs to be more descriptive. Needed by bitbake where and how? Is there some patchset upcoming?
Ya, this is necessary for the patch: https://lists.openembedded.org/g/bitbake-devel/message/12385, which isn't merged yet.
Quoted 47 lines
Alex
On Wed, 14 Jul 2021 at 15:42, Joshua Watt <[email protected] <mailto:[email protected]>> wrote:
Adds the command line compression tools which are needed by bitbake Signed-off-by: Joshua Watt <[email protected] <mailto:[email protected]>> --- meta/conf/bitbake.conf | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf index f451ba6a47..3801db7d42 100644 --- a/meta/conf/bitbake.conf +++ b/meta/conf/bitbake.conf @@ -484,12 +484,12 @@ HOSTTOOLS += " \ [ ar as awk basename bash bzip2 cat chgrp chmod chown chrpath cmp comm cp cpio \ cpp cut date dd diff diffstat dirname du echo egrep env expand expr false \ fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip gzip \ - head hostname iconv id install ld ldd ln ls make md5sum mkdir mkfifo mknod \ + head hostname iconv id install ld ldd ln ls lz4c make md5sum mkdir mkfifo mknod \ mktemp mv nm objcopy objdump od patch perl pr printf pwd \ - python3 ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \ + python3 pzstd ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \ sha1sum sha224sum sha256sum sha384sum sha512sum \ sleep sort split stat strings strip tail tar tee test touch tr true uname \ - uniq wc wget which xargs \ + uniq wc wget which xargs zstd \ " # Tools needed to run testimage runtime image testing -- 2.32.0
Right, but that still only sets up the infra and tests :) What will use it?
Alex
Quoted 58 lines · Joshua Watt
On 7/14/21 8:45 AM, Alexander Kanavin wrote:
While I welcome the transition to zstd, I think the commit message needs to be more descriptive. Needed by bitbake where and how? Is there some patchset upcoming?
Ya, this is necessary for the patch:
https://lists.openembedded.org/g/bitbake-devel/message/12385, which isn't merged yet.Alex
Quoted 41 lines · Joshua Watt
Adds the command line compression tools which are needed by bitbake
Signed-off-by Joshua Watt <[email protected]>--- meta/conf/bitbake.conf | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)meta/conf/bitbake.conf +2
-2@@ -484,12 +484,12 @@ HOSTTOOLS += " \ [ ar as awk basename bash bzip2 cat chgrp chmod chown chrpath cmp comm cp cpio \ cpp cut date dd diff diffstat dirname du echo egrep env expand expr false \ fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip gzip \ - head hostname iconv id install ld ldd ln ls make md5sum mkdir mkfifo mknod \ + head hostname iconv id install ld ldd ln ls lz4c make md5sum mkdir mkfifo mknod \ mktemp mv nm objcopy objdump od patch perl pr printf pwd \ - python3 ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \ + python3 pzstd ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \sha1sum sha224sum sha256sum sha384sum sha512sum \
sleep sort split stat strings strip tail tar tee test touch tr true
uname \meta/conf/bitbake.conf +1
-2- uniq wc wget which xargs \ + uniq wc wget which xargs zstd \ " # Tools needed to run testimage runtime image testing --2.32.0
Hello,
Quoted 14 lines · Joshua Watt
Adds the command line compression tools which are needed by bitbake
Signed-off-by Joshua Watt <[email protected]>--- meta/conf/bitbake.conf | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)meta/conf/bitbake.conf +0
-0@@ -484,12 +484,12 @@ HOSTTOOLS += " \ [ ar as awk basename bash bzip2 cat chgrp chmod chown chrpath cmp comm cp cpio \ cpp cut date dd diff diffstat dirname du echo egrep env expand expr false \
meta/conf/bitbake.conf HOSTTOOLS += " \ · line 492
Quoted 10 lines
meta/conf/bitbake.conf +3
-3fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip gzip \ - head hostname iconv id install ld ldd ln ls make md5sum mkdir mkfifo mknod \ + head hostname iconv id install ld ldd ln ls lz4c make md5sum mkdir mkfifo mknod \ mktemp mv nm objcopy objdump od patch perl pr printf pwd \ - python3 ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \ + python3 pzstd ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \ sha1sum sha224sum sha256sum sha384sum sha512sum \ sleep sort split stat strings strip tail tar tee test touch tr true uname \ - uniq wc wget which xargs \ + uniq wc wget which xargs zstd \
zstd seems to hurt reproducibility:
AssertionError: The following deb packages are missing or different and not in exclusion list: /home/pokybuild/yocto-worker/reproducible-ubuntu/build/build-st/reproducibleB/tmp/deploy/deb/./core2-64/elfutils-ptest_0.185-r0_amd64.deb /home/pokybuild/yocto-worker/reproducible-ubuntu/build/build-st/reproducibleB/tmp/deploy/deb/./core2-64/man-db-dbg_2.9.4-r0_amd64.deb /home/pokybuild/yocto-worker/reproducible-ubuntu/build/build-st/reproducibleB/tmp/deploy/deb/./core2-64/man-db-src_2.9.4-r0_amd64.deb /home/pokybuild/yocto-worker/reproducible-ubuntu/build/build-st/reproducibleB/tmp/deploy/deb/./core2-64/man-db_2.9.4-r0_amd64.deb
The following ipk packages are missing or different and not in exclusion list: /home/pokybuild/yocto-worker/reproducible-ubuntu/build/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/elfutils-ptest_0.185-r0_core2-64.ipk /home/pokybuild/yocto-worker/reproducible-ubuntu/build/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/man-db-dbg_2.9.4-r0_core2-64.ipk /home/pokybuild/yocto-worker/reproducible-ubuntu/build/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/man-db-src_2.9.4-r0_core2-64.ipk /home/pokybuild/yocto-worker/reproducible-ubuntu/build/build-st/reproducibleB/tmp/deploy/ipk/./core2-64/man-db_2.9.4-r0_core2-64.ipk
The following rpm packages are missing or different and not in exclusion list: /home/pokybuild/yocto-worker/reproducible-ubuntu/build/build-st/reproducibleB/tmp/deploy/rpm/./core2_64/elfutils-ptest-0.185-r0.core2_64.rpm /home/pokybuild/yocto-worker/reproducible-ubuntu/build/build-st/reproducibleB/tmp/deploy/rpm/./core2_64/man-db-2.9.4-r0.core2_64.rpm /home/pokybuild/yocto-worker/reproducible-ubuntu/build/build-st/reproducibleB/tmp/deploy/rpm/./core2_64/man-db-dbg-2.9.4-r0.core2_64.rpm /home/pokybuild/yocto-worker/reproducible-ubuntu/build/build-st/reproducibleB/tmp/deploy/rpm/./core2_64/man-db-src-2.9.4-r0.core2_64.rpm
diffoscope output here:
https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20210814-qe8khapn/packages/diff-html/
I would think this was expected...
Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Quoted 17 lines · Joshua Watt
Adds the command line compression tools which are needed by bitbake
Signed-off-by Joshua Watt <[email protected]>--- meta/conf/bitbake.conf | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)meta/conf/bitbake.conf +0
-1@@ -484,12 +484,12 @@ HOSTTOOLS += " \ [ ar as awk basename bash bzip2 cat chgrp chmod chown chrpath cmp comm cp cpio \ cpp cut date dd diff diffstat dirname du echo egrep env expand expr false \ fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip gzip \ - head hostname iconv id install ld ldd ln ls make md5sum mkdir mkfifo mknod \
meta/conf/bitbake.conf HOSTTOOLS += " \ · line 494
Quoted 10 lines
meta/conf/bitbake.conf +3
-2+ head hostname iconv id install ld ldd ln ls lz4c make md5sum mkdir mkfifo mknod \ mktemp mv nm objcopy objdump od patch perl pr printf pwd \ - python3 ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \ + python3 pzstd ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \ sha1sum sha224sum sha256sum sha384sum sha512sum \ sleep sort split stat strings strip tail tar tee test touch tr true uname \ - uniq wc wget which xargs \ + uniq wc wget which xargs zstd \ "
this should also be reflecting on the distro prerequisites for supported distros.
Quoted 7 lines
meta/conf/bitbake.conf +0
-1# Tools needed to run testimage runtime image testing --2.32.0
Quoted 17 lines · Joshua Watt
Adds the command line compression tools which are needed by bitbake
Signed-off-by Joshua Watt <[email protected]>--- meta/conf/bitbake.conf | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)meta/conf/bitbake.conf +0
-1@@ -484,12 +484,12 @@ HOSTTOOLS += " \ [ ar as awk basename bash bzip2 cat chgrp chmod chown chrpath cmp comm cp cpio \ cpp cut date dd diff diffstat dirname du echo egrep env expand expr false \ fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip gzip \ - head hostname iconv id install ld ldd ln ls make md5sum mkdir mkfifo mknod \
meta/conf/bitbake.conf HOSTTOOLS += " \ · line 494
Quoted 10 lines
meta/conf/bitbake.conf +3
-2+ head hostname iconv id install ld ldd ln ls lz4c make md5sum mkdir mkfifo mknod \ mktemp mv nm objcopy objdump od patch perl pr printf pwd \ - python3 ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \ + python3 pzstd ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \ sha1sum sha224sum sha256sum sha384sum sha512sum \ sleep sort split stat strings strip tail tar tee test touch tr true uname \ - uniq wc wget which xargs \ + uniq wc wget which xargs zstd \ "
on ubuntu 18.04 we will need
sudo apt install zstd liblz4-tool
Quoted 7 lines
meta/conf/bitbake.conf +0
-1# Tools needed to run testimage runtime image testing --2.32.0
Quoted 31 lines · Khem Raj
Quoted 17 lines · Joshua Watt
Adds the command line compression tools which are needed by bitbake
Signed-off-by Joshua Watt <[email protected]>--- meta/conf/bitbake.conf | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)meta/conf/bitbake.conf +0
-1@@ -484,12 +484,12 @@ HOSTTOOLS += " \ [ ar as awk basename bash bzip2 cat chgrp chmod chown chrpath cmp comm cp cpio \ cpp cut date dd diff diffstat dirname du echo egrep env expand expr false \ fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip gzip \ - head hostname iconv id install ld ldd ln ls make md5sum mkdir mkfifo mknod \meta/conf/bitbake.conf HOSTTOOLS += " \ · line 494
Quoted 10 lines
meta/conf/bitbake.conf +3
-2+ head hostname iconv id install ld ldd ln ls lz4c make md5sum mkdir mkfifo mknod \ mktemp mv nm objcopy objdump od patch perl pr printf pwd \ - python3 ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \ + python3 pzstd ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \ sha1sum sha224sum sha256sum sha384sum sha512sum \ sleep sort split stat strings strip tail tar tee test touch tr true uname \ - uniq wc wget which xargs \ + uniq wc wget which xargs zstd \ "on ubuntu 18.04 we will need
Quoted 1 line
sudo apt install zstd liblz4-tool
Michael,
Can you update the docs for this change?
http://docs.yoctoproject.org/ref-manual/system-requirements.html?highlight=bzip2
I don't know what the package names are for non-debian distros.
This must have been done for the YP AB cluster but maybe that's done manually?
../Randy
Quoted 12 lines
Quoted 12 lines
meta/conf/bitbake.conf +0
-1# Tools needed to run testimage runtime image testing --2.32.0
Signature · 2 lines
# Randy MacLeod # Wind River Linux
Hi,
I kind of missed out on this change... but I got caught up by reality hard this morning :-(.
I currently don't see a reason why zstd is required on the host - only systemd is having a hard dependency against zstd atm, while the other lately added references are just optional PACKAGECONFIGs (and there are plenty of them which have loose ends - in poky and meta-oe)
So the question I want to raise: why a hard dependency? wouldn't have been HOSTTOOLS_NONFATAL enough?
I mean this is the second hard cut in the project within just weeks and this time it was host related, which is even harder to fix in a timely manner in a corporate environment (basically rolling out changes to all dev installations isn't something I fancy on a Wednesday morning :-( )
None of the commit messages somehow mentions, why we all of a sudden need that on the host, instead of building it from sources (which should be possible judging from the latest master sources).
And on that note: that the documentation wasn't updated in the same changeset is very unfortunate (not even the crops container where updated)
/rant
Konrad
Quoted 70 lines · Randy MacLeod
Quoted 35 lines · Khem Raj
Quoted 29 lines · Joshua Watt
Adds the command line compression tools which are needed by bitbake
Signed-off-by Joshua Watt <[email protected]>---
meta/conf/bitbake.conf| 6 +++---1 file changed, 3 insertions(+), 3 deletions(-)meta/conf/bitbake.conf +3
-3@@ -484,12 +484,12 @@ HOSTTOOLS += " \ [ ar as awk basename bash bzip2 cat chgrp chmod chown chrpath cmp comm cp cpio \ cpp cut date dd diff diffstat dirname du echo egrep env expand expr false \ fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip gzip \ - head hostname iconv id install ld ldd ln ls make md5sum mkdir mkfifo mknod \ + head hostname iconv id install ld ldd ln ls lz4c make md5sum mkdir mkfifo mknod \ mktemp mv nm objcopy objdump od patch perl pr printf pwd \ - python3 ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \ + python3 pzstd ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \ sha1sum sha224sum sha256sum sha384sum sha512sum \ sleep sort split stat strings strip tail tar tee test touch tr true uname \ - uniq wc wget which xargs \ + uniq wc wget which xargs zstd \Quoted 2 lines
"
on ubuntu 18.04 we will need
Quoted 1 line
sudo apt install zstd liblz4-tool
Michael,
Can you update the docs for this change?
http://docs.yoctoproject.org/ref-manual/system-requirements.html?highlight=bzip2
I don't know what the package names are for non-debian distros.
This must have been done for the YP AB cluster but maybe that's done manually?
../Randy
Quoted 12 lines
Quoted 12 lines
# Tools needed to run testimage runtime image testing
Signature · 1 line
2.32.0
Hi Konrad,
Quoted 6 lines · Konrad Weihmann
I mean this is the second hard cut in the project within just weeks and this time it was host related, which is even harder to fix in a timely manner in a corporate environment (basically rolling out changes to all dev installations isn't something I fancy on a Wednesday morning :-( )
The choice to follow bleeding edge, means bleeding at some points in time.
Why do you follow master on all dev installations if you are not able to match (slightly) disruptive changes?
I would recommend to prevalidate changes on one instance before spreading across all (inflexible) dev installations anyway.
I did not follow the rationale closely, but I think it was related to CMake dependencies as well.
Regards,
Leon.
Quoted 16 lines · Konrad Weihmann
I kind of missed out on this change... but I got caught up by reality hard this morning :-(.
I currently don't see a reason why zstd is required on the host - only systemd is having a hard dependency against zstd atm, while the other lately added references are just optional PACKAGECONFIGs (and there are plenty of them which have loose ends - in poky and meta-oe)
So the question I want to raise: why a hard dependency? wouldn't have been
HOSTTOOLS_NONFATALenough?I mean this is the second hard cut in the project within just weeks and this time it was host related, which is even harder to fix in a timely manner in a corporate environment (basically rolling out changes to all dev installations isn't something I fancy on a Wednesday morning :-( )
Quoted 6 lines
None of the commit messages somehow mentions, why we all of a sudden need that on the host, instead of building it from sources (which should be possible judging from the latest master sources).
And on that note: that the documentation wasn't updated in the same changeset is very unfortunate (not even the crops container where updated)
Sorry the change is a bit of a pain. It has been in master-next for a while and we did have to jump through a number of hoops as a project to enable this (e.g. adding to buildtools and our autobuilder infrastructure).
The reason we're doing this is that we need to use compression within bitbake itself for some of our output files. We've not done this in the past as it is a pain to have the host prerequisites however there are multiple places we can potentially benefit, such as compressed pkgdata and compressed siginfo files. It is also possible we may be able to store a compressed parsed datastore.
The siginfo files and datastore could mean we can improve debugging of a number of currently hard to solve problems our users face.
Without this we're unlikely to be able to move forward in these areas. If we do it, we open up new development opportunities and potentially improved usability. We can't "just build it" as that would mean we can't use it within bitbake itself.
Sorry this isn't well described anywhere, it is mainly as we can't yet "guarantee" that this will do these things. We do know it won't happen unless we try though.
Master *is* a development branch, not a stable release so things can change.
Cheers,
Richard
Quoted 28 lines · Richard Purdie
Quoted 16 lines · Konrad Weihmann
I kind of missed out on this change... but I got caught up by reality hard this morning :-(.
I currently don't see a reason why zstd is required on the host - only systemd is having a hard dependency against zstd atm, while the other lately added references are just optional PACKAGECONFIGs (and there are plenty of them which have loose ends - in poky and meta-oe)
So the question I want to raise: why a hard dependency? wouldn't have been
HOSTTOOLS_NONFATALenough?I mean this is the second hard cut in the project within just weeks and this time it was host related, which is even harder to fix in a timely manner in a corporate environment (basically rolling out changes to all dev installations isn't something I fancy on a Wednesday morning :-( )
Quoted 6 lines
None of the commit messages somehow mentions, why we all of a sudden need that on the host, instead of building it from sources (which should be possible judging from the latest master sources).
And on that note: that the documentation wasn't updated in the same changeset is very unfortunate (not even the crops container where updated)
Sorry the change is a bit of a pain. It has been in master-next for a while and we did have to jump through a number of hoops as a project to enable this (e.g. adding to buildtools and our autobuilder infrastructure).
Quoted 6 lines
The reason we're doing this is that we need to use compression within bitbake itself for some of our output files. We've not done this in the past as it is a pain to have the host prerequisites however there are multiple places we can potentially benefit, such as compressed pkgdata and compressed siginfo files. It is also possible we may be able to store a compressed parsed datastore.
Thanks Richard, that's explains why it's needed
Quoted 11 lines
The siginfo files and datastore could mean we can improve debugging of a number of currently hard to solve problems our users face.
Without this we're unlikely to be able to move forward in these areas. If we do it, we open up new development opportunities and potentially improved usability. We can't "just build it" as that would mean we can't use it within bitbake itself.
Sorry this isn't well described anywhere, it is mainly as we can't yet "guarantee" that this will do these things. We do know it won't happen unless we try though.
Agree - btw maybe a good opportunity to insist on better commit messages ;-)- just out of interest: is there any guideline, maybe it's worth to at least link to a commit message guide at https://www.openembedded.org/wiki/How_to_submit_a_patch_to_OpenEmbedded or here https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines.
And slightly off-topic, but maybe also worth to do some automation, as I've seen multiple patches lately, that don't 100% follow the patch guideline in place.
Quoted 2 lines
Master *is* a development branch, not a stable release so things can change.
I've been brave enough to enforce a multi-branch development (just to keep the bar low when jumping from release to release), maybe I take that as a personal learning :)
Quoted 5 lines
Cheers,
Richard
Quoted 62 lines · Konrad Weihmann
Quoted 17 lines · Richard Purdie
Quoted 16 lines · Konrad Weihmann
I kind of missed out on this change... but I got caught up by reality hard this morning :-(.
I currently don't see a reason why zstd is required on the host - only systemd is having a hard dependency against zstd atm, while the other lately added references are just optional PACKAGECONFIGs (and there are plenty of them which have loose ends - in poky and meta-oe)
So the question I want to raise: why a hard dependency? wouldn't have been
HOSTTOOLS_NONFATALenough?I mean this is the second hard cut in the project within just weeks and this time it was host related, which is even harder to fix in a timely manner in a corporate environment (basically rolling out changes to all dev installations isn't something I fancy on a Wednesday morning :-( )
Quoted 6 lines
Quoted 6 lines
None of the commit messages somehow mentions, why we all of a sudden need that on the host, instead of building it from sources (which should be possible judging from the latest master sources).
And on that note: that the documentation wasn't updated in the same changeset is very unfortunate (not even the crops container where
updated)
Quoted 2 lines
Sorry the change is a bit of a pain. It has been in master-next for a
while
Quoted 1 line
and we did have to jump through a number of hoops as a project to enable
this
Quoted 3 lines
(e.g. adding to buildtools and our autobuilder infrastructure).
The reason we're doing this is that we need to use compression within
bitbake
Quoted 1 line
itself for some of our output files. We've not done this in the past as
it is
Quoted 1 line
a pain to have the host prerequisites however there are multiple places
we can
Quoted 1 line
potentially benefit, such as compressed pkgdata and compressed siginfo
files.
Quoted 1 line
It is also possible we may be able to store a compressed parsed
datastore.
Quoted 1 line
Thanks Richard, that's explains why it's needed
Quoted 4 lines
The siginfo files and datastore could mean we can improve debugging of a number of currently hard to solve problems our users face.
Without this we're unlikely to be able to move forward in these areas.
If we
Quoted 2 lines
do it, we open up new development opportunities and potentially improved usability. We can't "just build it" as that would mean we can't use it
within
Quoted 5 lines
bitbake itself.
Sorry this isn't well described anywhere, it is mainly as we can't yet "guarantee" that this will do these things. We do know it won't happen unless we try though.
Quoted 9 lines
Agree - btw maybe a good opportunity to insist on better commit messages ;-)- just out of interest: is there any guideline, maybe it's worth to at least link to a commit message guide at
https://www.openembedded.org/wiki/How_to_submit_a_patch_to_OpenEmbedded
or here https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines.
And slightly off-topic, but maybe also worth to do some automation, as I've seen multiple patches lately, that don't 100% follow the patch guideline in place.
Sorry, that was my fault. I was preoccupied and didn't fix it when it was pointed out earlier.
FWIW, I do have some WIP changes that use this functionality that will hopefully land soon so it will be used :)
Quoted 17 lines
Quoted 2 lines
Master *is* a development branch, not a stable release so things can
change.
I've been brave enough to enforce a multi-branch development (just to keep the bar low when jumping from release to release), maybe I take that as a personal learning :)
Quoted 5 lines
Cheers,
Richard
Quoted 51 lines · Konrad Weihmann
Quoted 28 lines · Richard Purdie
Quoted 16 lines · Konrad Weihmann
I kind of missed out on this change... but I got caught up by reality hard this morning :-(.
I currently don't see a reason why zstd is required on the host - only systemd is having a hard dependency against zstd atm, while the other lately added references are just optional PACKAGECONFIGs (and there are plenty of them which have loose ends - in poky and meta-oe)
So the question I want to raise: why a hard dependency? wouldn't have been
HOSTTOOLS_NONFATALenough?I mean this is the second hard cut in the project within just weeks and this time it was host related, which is even harder to fix in a timely manner in a corporate environment (basically rolling out changes to all dev installations isn't something I fancy on a Wednesday morning :-( )
Quoted 6 lines
None of the commit messages somehow mentions, why we all of a sudden need that on the host, instead of building it from sources (which should be possible judging from the latest master sources).
And on that note: that the documentation wasn't updated in the same changeset is very unfortunate (not even the crops container where updated)
Sorry the change is a bit of a pain. It has been in master-next for a while and we did have to jump through a number of hoops as a project to enable this (e.g. adding to buildtools and our autobuilder infrastructure).
Quoted 6 lines
The reason we're doing this is that we need to use compression within bitbake itself for some of our output files. We've not done this in the past as it is a pain to have the host prerequisites however there are multiple places we can potentially benefit, such as compressed pkgdata and compressed siginfo files. It is also possible we may be able to store a compressed parsed datastore.
Thanks Richard, that's explains why it's needed
Quoted 11 lines
The siginfo files and datastore could mean we can improve debugging of a number of currently hard to solve problems our users face.
Without this we're unlikely to be able to move forward in these areas. If we do it, we open up new development opportunities and potentially improved usability. We can't "just build it" as that would mean we can't use it within bitbake itself.
Sorry this isn't well described anywhere, it is mainly as we can't yet "guarantee" that this will do these things. We do know it won't happen unless we try though.
Quoted 8 lines
Agree - btw maybe a good opportunity to insist on better commit messages ;-)- just out of interest: is there any guideline, maybe it's worth to at least link to a commit message guide at
https://www.openembedded.org/wiki/How_to_submit_a_patch_to_OpenEmbedded
or here https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines.
And slightly off-topic, but maybe also worth to do some automation, as I've seen multiple patches lately, that don't 100% follow the patch guideline in place.
We had automation to test some of these things. It broke and we simply don't have the maintainers to look after it and keep it working, it is simply too much for Michael and I to manage to hold together.
I'm doing the best I can but I can't do this alone. Until we get more people stepping up to help with things like the automated tests, we'll have to manage with what I can do.
FWIW I had to spend a ton of time fixing buildtools, getting the workers to have the right binaries and so on to merge this change. I tried to get the docs updated but I didn't micromanage it and it didn't happen as yet. I'm aware of it, I don't know if I'll just have to go and fix that next too.
So yes, I agree nice commit messages would be nice. Please feel free to help review patches.
Cheers,
Richard