Skip to contentSkip to conversation

bitbake.conf: require lz4 instead of lz4c

PATCH In review Justin Bronder 6 participants 14 messages Nov 5, 2024 – Dec 10, 2024

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 bitbake.conf: require lz4 instead of lz4c

With bitbake commit 'bb: compress: use lz4 instead of lz4c' we require lz4, not lz4c [1]. Going as far back as Ubuntu 16.04/Fedora 25, they are packaged together so it should be safe to simply adjust the requirement from lz4c to lz4.

Note that the two are largely the same, but upstream has considered lz4c deprecated since at least 2018 [2].

1. https://git.openembedded.org/bitbake/commit/?id=907472034b344e4eb73cfd43059a413469f52e1c
2. https://github.com/lz4/lz4/pull/553

---
 meta/conf/bitbake.conf | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/conf/bitbake.conf +1 -1
index eda505c861..14e761e2e7 100644
@@ -517,7 +517,7 @@ HOSTTOOLS += " \
     [ ar as awk basename bash bunzip2 bzip2 cat chgrp chmod chown chrpath cmp comm cp cpio \
     cpp cut date dd diff diffstat dirname du echo egrep env expand expr false \
     fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip gzip \
-    head hostname iconv id install ld ldd ln ls lz4c make md5sum mkdir mkfifo mknod \
+    head hostname iconv id install ld ldd ln ls lz4 make md5sum mkdir mkfifo mknod \
     mktemp mv nm objcopy objdump od patch perl pr printf pwd \
     python3 pzstd ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \
     sha1sum sha224sum sha256sum sha384sum sha512sum \
Signature · 1 line
2.46.1
#1 lore raw
Quoted 22 lines · Justin Bronder via lists.openembedded.org

With bitbake commit 'bb: compress: use lz4 instead of lz4c' we require lz4, not lz4c [1].  Going as far back as Ubuntu 16.04/Fedora 25, they are packaged together so it should be safe to simply adjust the requirement from lz4c to lz4.

Note that the two are largely the same, but upstream has considered lz4c deprecated since at least 2018 [2].

1. https://git.openembedded.org/bitbake/commit/?id=907472034b344e4eb73cfd43059a413469f52e1c
2. https://github.com/lz4/lz4/pull/553

---
 meta/conf/bitbake.conf | 2 +-

 1 file changed, 1 insertion(+), 1 deletion(-)
meta/conf/bitbake.conf +0 -0
index eda505c861..14e761e2e7 100644

meta/conf/bitbake.conf HOSTTOOLS += " \ · line 523

Quoted 8 lines
meta/conf/bitbake.conf +1 -1
@@ -517,7 +517,7 @@ HOSTTOOLS += " \
      [ ar as awk basename bash bunzip2 bzip2 cat chgrp chmod chown chrpath cmp comm cp cpio \
      cpp cut date dd diff diffstat dirname du echo egrep env expand expr false \
      fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip gzip \
-    head hostname iconv id install ld ldd ln ls lz4c make md5sum mkdir mkfifo mknod \
+    head hostname iconv id install ld ldd ln ls lz4 make md5sum mkdir mkfifo mknod \
      mktemp mv nm objcopy objdump od patch perl pr printf pwd \
      python3 pzstd ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \
      sha1sum sha224sum sha256sum sha384sum sha512sum \

This caused a failure in barebox:

https://valkyrie.yoctoproject.org/#/builders/23/builds/409/steps/14/logs/stdio

Cheers,

Richard

#2 lore raw
Quoted 34 lines · Richard Purdie
Quoted 22 lines · Justin Bronder via lists.openembedded.org

With bitbake commit 'bb: compress: use lz4 instead of lz4c' we require lz4, not lz4c [1].� Going as far back as Ubuntu 16.04/Fedora 25, they are packaged together so it should be safe to simply adjust the requirement from lz4c to lz4.

Note that the two are largely the same, but upstream has considered lz4c deprecated since at least 2018 [2].

1. https://git.openembedded.org/bitbake/commit/?id=907472034b344e4eb73cfd43059a413469f52e1c
2. https://github.com/lz4/lz4/pull/553

---
�meta/conf/bitbake.conf | 2 +-
�1 file changed, 1 insertion(+), 1 deletion(-)

meta/conf/bitbake.conf +0 -0
index eda505c861..14e761e2e7 100644

meta/conf/bitbake.conf HOSTTOOLS += " \ · line 523

Quoted 8 lines
meta/conf/bitbake.conf +1 -1
@@ -517,7 +517,7 @@ HOSTTOOLS += " \
 ���� [ ar as awk basename bash bunzip2 bzip2 cat chgrp chmod chown chrpath cmp comm cp cpio \
 ���� cpp cut date dd diff diffstat dirname du echo egrep env expand expr false \
 ���� fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip gzip \
-��� head hostname iconv id install ld ldd ln ls lz4c make md5sum mkdir mkfifo mknod \
+��� head hostname iconv id install ld ldd ln ls lz4 make md5sum mkdir mkfifo mknod \
 ���� mktemp mv nm objcopy objdump od patch perl pr printf pwd \
 ���� python3 pzstd ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \
 ���� sha1sum sha224sum sha256sum sha384sum sha512sum \

This caused a failure in barebox:

Quoted 2 lines

https://valkyrie.yoctoproject.org/#/builders/23/builds/409/steps/14/logs/stdio

I see, sorry about that.

What would be the recommended path forward?

- Add lz4c to HOSTTOOLS_NONFATAL so it's still symlinked. This would fix barebox and any other recipes that still depend on lz4c on distributions where lz4c is still packaged but leave others without an early warning system.
- Submit patches to barebox and others to use lz4 instead of lz4c. This is probably more correct but will leave Tumbleweed unsupported by default with other distros to presumably follow.
- Convince Tumbleweed (and others) to continue packaging lz4c.

Signature · 1 line
Justin Bronder
#3 lore raw
Quoted 53 lines · Justin Bronder
Quoted 45 lines · Richard Purdie
Quoted 31 lines · Justin Bronder via lists.openembedded.org

With bitbake commit 'bb: compress: use lz4 instead of lz4c' we require
lz4, not lz4c [1].  Going as far back as Ubuntu 16.04/Fedora 25, they
are packaged together so it should be safe to simply adjust the requirement from lz4c to lz4.

Note that the two are largely the same, but upstream has considered lz4c
deprecated since at least 2018 [2].

1.
https://git.openembedded.org/bitbake/commit/?id=907472034b344e4eb73cfd43059a413469f52e1c
2. https://github.com/lz4/lz4/pull/553

---
 meta/conf/bitbake.conf | 2 +-

 1 file changed, 1 insertion(+), 1 deletion(-)
meta/conf/bitbake.conf +0 -0
index eda505c861..14e761e2e7 100644
@@ -517,7 +517,7 @@ HOSTTOOLS += " \

     [ ar as awk basename bash bunzip2 bzip2 cat chgrp chmod chown chrpath cmp comm cp cpio \
     cpp cut date dd diff diffstat dirname du echo egrep env expand expr false \
     fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip gzip \

Quoted 9 lines

-    head hostname iconv id install ld ldd ln ls lz4c make md5sum mkdir mkfifo mknod \
+    head hostname iconv id install ld ldd ln ls lz4 make md5sum mkdir mkfifo mknod \
     mktemp mv nm objcopy objdump od patch perl pr printf pwd \
     python3 pzstd ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \
     sha1sum sha224sum sha256sum sha384sum sha512sum \

This caused a failure in barebox:

Quoted 2 lines

https://valkyrie.yoctoproject.org/#/builders/23/builds/409/steps/14/logs/stdio

I see, sorry about that.

What would be the recommended path forward?

Quoted 7 lines

- Add lz4c to HOSTTOOLS_NONFATAL so it's still symlinked.  This would fix barebox and any other recipes that still depend on lz4c on distributions where lz4c is still packaged but leave others without an early warning system.
- Submit patches to barebox and others to use lz4 instead of lz4c. This is probably more correct but will leave Tumbleweed unsupported by default with other distros to presumably follow.

barebox only recently merged so I think it should be fine to see if we can switch barebox to use lz4. How would that leave Tumbleweed supported?

Cheers,

Richard

#4 lore raw
Quoted 62 lines · Richard Purdie
Quoted 53 lines · Justin Bronder
Quoted 45 lines · Richard Purdie
Quoted 31 lines · Justin Bronder via lists.openembedded.org

With bitbake commit 'bb: compress: use lz4 instead of lz4c' we require
lz4, not lz4c [1].� Going as far back as Ubuntu 16.04/Fedora 25, they
are packaged together so it should be safe to simply adjust the requirement from lz4c to lz4.

Note that the two are largely the same, but upstream has considered lz4c
deprecated since at least 2018 [2].

1.
https://git.openembedded.org/bitbake/commit/?id=907472034b344e4eb73cfd43059a413469f52e1c
2. https://github.com/lz4/lz4/pull/553

---
�meta/conf/bitbake.conf | 2 +-
�1 file changed, 1 insertion(+), 1 deletion(-)

meta/conf/bitbake.conf +0 -0
index eda505c861..14e761e2e7 100644
@@ -517,7 +517,7 @@ HOSTTOOLS += " \

���� [ ar as awk basename bash bunzip2 bzip2 cat chgrp chmod chown chrpath cmp comm cp cpio \
���� cpp cut date dd diff diffstat dirname du echo egrep env expand expr false \
���� fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip gzip \

Quoted 9 lines

-��� head hostname iconv id install ld ldd ln ls lz4c make md5sum mkdir mkfifo mknod \
+��� head hostname iconv id install ld ldd ln ls lz4 make md5sum mkdir mkfifo mknod \
���� mktemp mv nm objcopy objdump od patch perl pr printf pwd \
���� python3 pzstd ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \
���� sha1sum sha224sum sha256sum sha384sum sha512sum \

This caused a failure in barebox:

Quoted 2 lines

https://valkyrie.yoctoproject.org/#/builders/23/builds/409/steps/14/logs/stdio

I see, sorry about that.

What would be the recommended path forward?

Quoted 7 lines

- Add lz4c to HOSTTOOLS_NONFATAL so it's still symlinked.� This would fix barebox and any other recipes that still depend on lz4c on distributions where lz4c is still packaged but leave others without an early warning system.
- Submit patches to barebox and others to use lz4 instead of lz4c.� This is probably more correct but will leave Tumbleweed unsupported by default with other distros to presumably follow.

Quoted 3 lines

barebox only recently merged so I think it should be fine to see if we can switch barebox to use lz4. How would that leave Tumbleweed supported?

Ok, I've submitted a quite simple patch to barebox upstream and I'll resubmit this along with that patch. Pending other recipes requiring lz4c, that will fix things for Tumbleweed.

Signature · 1 line
Justin Bronder
#5 lore raw
In reply to Justin Bronder

Can we rather make the affected recipe depend on the native tool rather than the host one? And pls file a ticket with barebox upstream or maybe the maintainer can respond right here.

Alex

Quoted 76 lines · Justin Bronder via lists.openembedded.org
Quoted 1 line · Richard Purdie

On Tue, 2024-11-05 at 10:37 -0500, Justin Bronder via

Quoted 6 lines · lists.openembedded.org
Quoted 6 lines

With bitbake commit 'bb: compress: use lz4 instead of lz4c' we require lz4, not lz4c [1]. Going as far back as Ubuntu 16.04/Fedora 25, they are packaged together so it should be safe to simply adjust the requirement from lz4c to lz4.

Note that the two are largely the same, but upstream has considered

lz4c

Quoted 3 lines
Quoted 3 lines

deprecated since at least 2018 [2].

1.

https://git.openembedded.org/bitbake/commit/?id=907472034b344e4eb73cfd43059a413469f52e1c

Quoted 7 lines
Quoted 7 lines

2. https://github.com/lz4/lz4/pull/553

---
 meta/conf/bitbake.conf | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 6 lines

meta/conf/bitbake.conf HOSTTOOLS += " \ · line 517

Quoted 6 lines
meta/conf/bitbake.conf +0 -0
index eda505c861..14e761e2e7 100644
@@ -517,7 +517,7 @@ HOSTTOOLS += " \
     [ ar as awk basename bash bunzip2 bzip2 cat chgrp chmod chown

chrpath cmp comm cp cpio \

Quoted 1 line

meta/conf/bitbake.conf HOSTTOOLS += " \ · line 518

Quoted 1 line
meta/conf/bitbake.conf +0 -0
     cpp cut date dd diff diffstat dirname du echo egrep env expand

expr false \

Quoted 1 line

meta/conf/bitbake.conf HOSTTOOLS += " \ · line 519

Quoted 1 line
meta/conf/bitbake.conf +0 -0
     fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip

gzip \

Quoted 1 line

meta/conf/bitbake.conf HOSTTOOLS += " \ · line 520

Quoted 1 line
meta/conf/bitbake.conf +0 -1
-    head hostname iconv id install ld ldd ln ls lz4c make md5sum

mkdir mkfifo mknod \

Quoted 1 line

meta/conf/bitbake.conf HOSTTOOLS += " \ · line 520

Quoted 1 line
meta/conf/bitbake.conf +1 -0
+    head hostname iconv id install ld ldd ln ls lz4 make md5sum mkdir

mkfifo mknod \

Quoted 2 lines

meta/conf/bitbake.conf HOSTTOOLS += " \ · line 522

Quoted 2 lines
meta/conf/bitbake.conf +0 -0
     mktemp mv nm objcopy objdump od patch perl pr printf pwd \
     python3 pzstd ranlib readelf readlink realpath rm rmdir rpcgen

sed seq sh \

Quoted 6 lines

meta/conf/bitbake.conf HOSTTOOLS += " \ · line 523

Quoted 2 lines
meta/conf/bitbake.conf +0 -0
     sha1sum sha224sum sha256sum sha384sum sha512sum \

This caused a failure in barebox:

https://valkyrie.yoctoproject.org/#/builders/23/builds/409/steps/14/logs/stdio

Quoted 1 line

I see, sorry about that.

What would be the recommended path forward?

- Add lz4c to HOSTTOOLS_NONFATAL so it's still symlinked. This would fix barebox and any other recipes that still depend on lz4c on distributions where lz4c is still packaged but leave others without an early warning system.
- Submit patches to barebox and others to use lz4 instead of lz4c. This is probably more correct but will leave Tumbleweed unsupported by default with other distros to presumably follow.
- Convince Tumbleweed (and others) to continue packaging lz4c.

--
Justin Bronder

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#206790):
https://lists.openembedded.org/g/openembedded-core/message/206790
Mute This Topic: https://lists.openembedded.org/mt/109427109/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#6 lore raw

Hi Alex,

Am Mittwoch, dem 06.11.2024 um 17:03 +0100 schrieb Alexander Kanavin:

Quoted 1 line

Can we rather make the affected recipe depend on the native tool rather than the host one?

we could and I agree this would be the more correct way.
The only concern I have is that this would require adding dependencies for all potential compression methods currently supported by barebox (lz4, lzo, gzip, xz).

Or I have to add some more pseudo-PACKAGECONFIG options and enable only lz4 by default. However, I fear that this would quickly result in accidentally using the host tools again. Not sure if there is local way of filtering out allowed host tools?..

Quoted 1 line

And pls file a ticket with barebox upstream or maybe the maintainer can respond right here.

Thankfully, Justin already contributed the fix to barebox upstream in the meantime and as far as I can see, it was quickly merged and should thus be part of the November release.

Regards, Enrico

Quoted 56 lines

Alex

Quoted 52 lines · Justin Bronder via lists.openembedded.org
Quoted 35 lines · Richard Purdie
Quoted 22 lines · Justin Bronder via lists.openembedded.org

With bitbake commit 'bb: compress: use lz4 instead of lz4c' we require lz4, not lz4c [1].  Going as far back as Ubuntu 16.04/Fedora 25, they are packaged together so it should be safe to simply adjust the requirement from lz4c to lz4.

Note that the two are largely the same, but upstream has considered lz4c deprecated since at least 2018 [2].

1. https://git.openembedded.org/bitbake/commit/?id=907472034b344e4eb73cfd43059a413469f52e1c
2. https://github.com/lz4/lz4/pull/553

---
 meta/conf/bitbake.conf | 2 +-

 1 file changed, 1 insertion(+), 1 deletion(-)
meta/conf/bitbake.conf +0 -0
index eda505c861..14e761e2e7 100644
@@ -517,7 +517,7 @@ HOSTTOOLS += " \

     [ ar as awk basename bash bunzip2 bzip2 cat chgrp chmod chown chrpath cmp comm cp cpio

Quoted 9 lines

\
     cpp cut date dd diff diffstat dirname du echo egrep env expand expr false \
     fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip gzip \
-    head hostname iconv id install ld ldd ln ls lz4c make md5sum mkdir mkfifo mknod \
+    head hostname iconv id install ld ldd ln ls lz4 make md5sum mkdir mkfifo mknod \
     mktemp mv nm objcopy objdump od patch perl pr printf pwd \
     python3 pzstd ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \
     sha1sum sha224sum sha256sum sha384sum sha512sum \

This caused a failure in barebox:

Quoted 2 lines

https://valkyrie.yoctoproject.org/#/builders/23/builds/409/steps/14/logs/stdio

I see, sorry about that.

What would be the recommended path forward?

- Add lz4c to HOSTTOOLS_NONFATAL so it's still symlinked.  This would fix barebox and any other recipes that still depend on lz4c on distributions where lz4c is still packaged but leave others without an early warning system.
- Submit patches to barebox and others to use lz4 instead of lz4c.  This is probably more correct but will leave Tumbleweed unsupported by default with other distros to presumably follow.
- Convince Tumbleweed (and others) to continue packaging lz4c.

Signature · 4 lines
Pengutronix e.K.                           | Enrico Jörns                |
Embedded Linux Consulting & Support        | https://www.pengutronix.de/ |
Steuerwalder Str. 21                       | Phone: +49-5121-206917-180  |
31137 Hildesheim, Germany                  | Fax:   +49-5121-206917-9    |
#7 lore raw
Quoted 2 lines · Enrico Jörns

Thankfully, Justin already contributed the fix to barebox upstream in the meantime and as far as I can see, it was quickly merged and should thus be part of the November release.

I'd say you could resend the above patch with a barebox backport.

Alex

#8 lore raw

Am Dienstag, dem 12.11.2024 um 14:34 +0100 schrieb Alexander Kanavin:

Quoted 6 lines · Enrico Jörns
Quoted 3 lines

Thankfully, Justin already contributed the fix to barebox upstream in the meantime and as far as I
can see, it was quickly merged and should thus be part of the November release.

I'd say you could resend the above patch with a barebox backport.

To clarify: Justin already did this in a v2 and the patch is in oe-core upstream.

I read this as meaning that you no longer consider a native lz4 dependency necessary.

Thanks and regards, Enrico

Quoted 2 lines

Alex

Signature · 4 lines
Pengutronix e.K.                           | Enrico Jörns                |
Embedded Linux Consulting & Support        | https://www.pengutronix.de/ |
Steuerwalder Str. 21                       | Phone: +49-5121-206917-180  |
31137 Hildesheim, Germany                  | Fax:   +49-5121-206917-9    |
#10 lore raw
Quoted 7 lines · Gu�ni M�r Gilbert via lists.openembedded.org

Hi I'm seeing a build error on latest master when building Linux 6.10 due to missing lz4c tool.

I see it's still used even in Linux 6.12: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/Makefile?h=v6.12.1#n511

On my build host, lz4c is symlinked to lz4. But I suspect the issue is lz4c was removed from HOSTTOOLS.

Anyone else seeing this problem?

Yes, I can reproduce it when setting CONFIG_KERNEL_LZ4. I've sent a patch upstream. If they are receptive I'll submit it for linux-yocto. But perhaps in the meantime we should add lz4c to HOSTTOOLS_NONFATAL?

https://lore.kernel.org/linux-kbuild/[email protected]/T/#u

Signature · 1 line
Justin Bronder
#11 lore raw

Hi Justin, thanks for submitting a patch upstream. I can confirm I have this in my defconfig: “CONFIG_KERNEL_LZ4=y”

#12 lore raw
In reply to Justin Bronder
Quoted 11 lines · Justin Bronder
Quoted 7 lines · Guðni Már Gilbert via lists.openembedded.org

Hi I'm seeing a build error on latest master when building Linux 6.10 due to missing lz4c tool.

I see it's still used even in Linux 6.12: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/Makefile?h=v6.12.1#n511

On my build host, lz4c is symlinked to lz4. But I suspect the issue is lz4c was removed from HOSTTOOLS.

Anyone else seeing this problem?

Yes, I can reproduce it when setting CONFIG_KERNEL_LZ4. I've sent a patch upstream. If they are receptive I'll submit it for linux-yocto.

Ok, this got merged to 6.13-rc kernel series, but it's now broken for all the downstream BSPs and older kernel versions that use LZ4, besides linux-yocto. Sigh...

Quoted 1 line

But perhaps in the meantime we should add lz4c to HOSTTOOLS_NONFATAL?

Was there a patch submitted to have both lz4 and lz4c listed in HOSTTOOLS* (at least temporatily until this gets propagated everywhere)?

Signature · 1 line
Denys
#13 lore raw
Quoted 18 lines · Denys Dmytriyenko
Quoted 11 lines · Justin Bronder
Quoted 7 lines · Gu�ni M�r Gilbert via lists.openembedded.org

Hi I'm seeing a build error on latest master when building Linux 6.10 due to missing lz4c tool.

I see it's still used even in Linux 6.12: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/Makefile?h=v6.12.1#n511

On my build host, lz4c is symlinked to lz4. But I suspect the issue is lz4c was removed from HOSTTOOLS.

Anyone else seeing this problem?

Yes, I can reproduce it when setting CONFIG_KERNEL_LZ4. I've sent a patch upstream. If they are receptive I'll submit it for linux-yocto.

Ok, this got merged to 6.13-rc kernel series, but it's now broken for all the downstream BSPs and older kernel versions that use LZ4, besides linux-yocto. Sigh...

Quoted 4 lines
Quoted 1 line

But perhaps in the meantime we should add lz4c to HOSTTOOLS_NONFATAL?

Was there a patch submitted to have both lz4 and lz4c listed in HOSTTOOLS* (at least temporatily until this gets propagated everywhere)?

I sent one just now. Apologies for the annoyances this has caused. No good deed goes unpunished ;)

https://lore.kernel.org/openembedded-core/[email protected]/T/#u

Signature · 1 line
Justin Bronder