Skip to contentSkip to conversation

package/lzma-alone: new package

PATCH Applied Rafał Miłecki 3 participants 4 messages Mar 8, 2022 – Mar 10, 2022

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 package/lzma-alone: new package

From: Rafał Miłecki <[email protected]>

This new package provides "lzma_alone" host binary based on the original LZMA SDK. It provides few extra options when compared to the LZMA Utils / XZ Utils project "lzma" binary (already packaged as the "lzma").

This packaging schema (LZMA SDK with lzma -> lzma_alone rename) follows Debian's solution. Please note that Debian also uses LZMA SDK for the base "lzma" tool which may be considered for Buildroot too - as an independent change. Similar packaging is also used by Ubuntu & Arch.

lzma_alone is a requirement for preparing firmware images for some Broadcom based home routers with a picky CFE bootloader. It has limited LZMA support and building compatible images requires specifying dictionary size and lc/lp/pb LZMA values manually.

Version 9.22 is used as it's the last release using .tar.bz2 format. The same version is used by Debian.

---
Hi, this is V2 of my following old attempt:

[PATCH] package/lzma-alone: new package
https://patchwork.ozlabs.org/project/buildroot/patch/[email protected]/

It doesn't seem like I'm going to get V1 properly reviewed / commented so let me try with V2 with changes I think were expected.

---
 package/Config.in.host             |  1 +
 package/lzma-alone/Config.in.host  | 18 ++++++++++++++++++
 package/lzma-alone/lzma-alone.hash |  1 +
 package/lzma-alone/lzma-alone.mk   | 22 ++++++++++++++++++++++
 4 files changed, 42 insertions(+)
 create mode 100644 package/lzma-alone/Config.in.host
 create mode 100644 package/lzma-alone/lzma-alone.hash
 create mode 100644 package/lzma-alone/lzma-alone.mk
package/Config.in.host +1 -0
index e5fbb07fc2..cd1c34b5a3 100644
@@ -50,6 +50,7 @@ menu "Host utilities"
 	source "package/lld/Config.in.host"
 	source "package/lpc3250loader/Config.in.host"
 	source "package/lttng-babeltrace/Config.in.host"
+	source "package/lzma-alone/Config.in.host"
 	source "package/mender-artifact/Config.in.host"
 	source "package/meson-tools/Config.in.host"
 	source "package/mfgtools/Config.in.host"
package/lzma-alone/Config.in.host new file +18 -0
new file mode 100644
index 0000000000..053b869635
@@ -0,0 +1,18 @@
+config BR2_PACKAGE_HOST_LZMA_ALONE
+	bool "host lzma-alone"
+	help
+	  LZMA has two well-known command-line tools:
+	  1. lzma from LZMA SDK
+	  2. lzma from XZ Utils (previously LZMA Utils)
+
+	  This package provides the former one which:
+	  1. Gives more control over compression parameters
+	  2. Gets named lzma_alone to allow coexistence
+
+	  This tool may be used to prepare firmwares for bootloaders
+	  with limited LZMA support.
+	  E.g. Broadcom's CFE bootloader for MIPS BCM47xx devices is
+	  known to fail to boot images using LZMA compression with
+	  dictionary.
+
+	  https://www.7-zip.org/sdk.html
package/lzma-alone/lzma-alone.hash new file +1 -0
new file mode 100644
index 0000000000..39759c4e3b
@@ -0,0 +1 @@
+sha256	9aade84f229fb25f7aef39d8866b375fe6d35a9e18098d7cd86a99e294902944	lzma922.tar.bz2
package/lzma-alone/lzma-alone.mk new file +22 -0
new file mode 100644
index 0000000000..6258bb4a95
@@ -0,0 +1,22 @@
+################################################################################
+#
+# lzma-alone
+#
+################################################################################
+
+LZMA_ALONE_VERSION = 9.22
+LZMA_ALONE_SITE = https://sourceforge.net/projects/sevenzip/files/LZMA%20SDK
+LZMA_ALONE_SOURCE = lzma922.tar.bz2
+LZMA_ALONE_STRIP_COMPONENTS = 0
+LZMA_ALONE_LICENSE = LGPL-2.1-or-later, BSD-3-Clause
+
+define HOST_LZMA_ALONE_BUILD_CMDS
+	$(HOST_MAKE_ENV) $(HOST_CONFIGURE_OPTS) $(MAKE) -C $(@D)/C/Util/Lzma -f makefile.gcc
+	$(HOST_MAKE_ENV) $(HOST_CONFIGURE_OPTS) $(MAKE) -C $(@D)/CPP/7zip/Bundles/LzmaCon -f makefile.gcc
+endef
+
+define HOST_LZMA_ALONE_INSTALL_CMDS
+	$(INSTALL) -D -m 0755 $(@D)/CPP/7zip/Bundles/LzmaCon/lzma $(HOST_DIR)/bin/lzma_alone
+endef
+
+$(eval $(host-generic-package))
Signature · 6 lines
2.34.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#1 lore raw

Rafał, All,

On 2022-03-08 19:35 +0100, Rafał Miłecki spake thusly:

Quoted 11 lines

From: Rafał Miłecki <[email protected]>

This new package provides "lzma_alone" host binary based on the original LZMA SDK. It provides few extra options when compared to the LZMA Utils / XZ Utils project "lzma" binary (already packaged as the "lzma").

This packaging schema (LZMA SDK with lzma -> lzma_alone rename) follows Debian's solution. Please note that Debian also uses LZMA SDK for the base "lzma" tool which may be considered for Buildroot too - as an independent change. Similar packaging is also used by Ubuntu & Arch.

Quoted 6 lines

lzma_alone is a requirement for preparing firmware images for some Broadcom based home routers with a picky CFE bootloader. It has limited LZMA support and building compatible images requires specifying dictionary size and lc/lp/pb LZMA values manually.

Version 9.22 is used as it's the last release using .tar.bz2 format.

We could introduce a host variant of p7zip, so that we could automatically add that to FOO_EXTRACT_DEPENDENCIES if the host does not already have 7za (or 7zr), like we do for lzip archives for example.

Quoted 5 lines

The
same version is used by Debian.

---

[--SNIP--]

package/lzma-alone/lzma-alone.hash line 1

Quoted 7 lines
package/lzma-alone/lzma-alone.hash new file +1 -0
new file mode 100644
index 0000000000..39759c4e3b
@@ -0,0 +1 @@
+sha256	9aade84f229fb25f7aef39d8866b375fe6d35a9e18098d7cd86a99e294902944	lzma922.tar.bz2

Fields must be separated by two spaces; fixed.

I also added the sha1 that is provided int he SF page, and approipriate comments.

Quoted 7 lines
package/lzma-alone/lzma-alone.mk new file +1 -0
new file mode 100644
index 0000000000..6258bb4a95
@@ -0,0 +1,22 @@
+################################################################################

package/lzma-alone/lzma-alone.mk line 11

Quoted 10 lines
package/lzma-alone/lzma-alone.mk new file +10 -0
+#
+# lzma-alone
+#
+################################################################################
+
+LZMA_ALONE_VERSION = 9.22
+LZMA_ALONE_SITE = https://sourceforge.net/projects/sevenzip/files/LZMA%20SDK
+LZMA_ALONE_SOURCE = lzma922.tar.bz2
+LZMA_ALONE_STRIP_COMPONENTS = 0
+LZMA_ALONE_LICENSE = LGPL-2.1-or-later, BSD-3-Clause

The license stated in lzma.txt (at the root of the source tree) states the "LZMA SDK is written and placed in the public domain by Igor Pavlov."

The 'GPL' term only appears in that file, and BSD appears nowhere. Also, the usual words in the BSD or GPL licenses do not appear in the source at all: 'redistribution', 'provided', 'disclaimer'.

WikiPedia seems to agree that it is Public Domain since 2008 and version 4.62:

    https://en.wikipedia.org/wiki/Lempel%E2%80%93Ziv%E2%80%93Markov_chain_algorithm#7-Zip_reference_implementation

Fixed.

package/lzma-alone/lzma-alone.mk line 15

Quoted 3 lines
package/lzma-alone/lzma-alone.mk new file +3 -0
+define HOST_LZMA_ALONE_BUILD_CMDS
+	$(HOST_MAKE_ENV) $(HOST_CONFIGURE_OPTS) $(MAKE) -C $(@D)/C/Util/Lzma -f makefile.gcc
+	$(HOST_MAKE_ENV) $(HOST_CONFIGURE_OPTS) $(MAKE) -C $(@D)/CPP/7zip/Bundles/LzmaCon -f makefile.gcc

Lines too long, I've split them.

package/lzma-alone/lzma-alone.mk line 19

Quoted 4 lines
package/lzma-alone/lzma-alone.mk new file +4 -0
+endef
+
+define HOST_LZMA_ALONE_INSTALL_CMDS
+	$(INSTALL) -D -m 0755 $(@D)/CPP/7zip/Bundles/LzmaCon/lzma $(HOST_DIR)/bin/lzma_alone

Ditto.

Applied to master, thanks.

Could you see at using a more recent version, with the p7zip host variant I suggested above, please?

Regards,
Yann E. MORIN.

Quoted 10 lines
package/lzma-alone/lzma-alone.mk new file +3 -0
+endef
+
+$(eval $(host-generic-package))
Signature · 6 lines
2.34.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#2 lore raw
Quoted 17 lines · Yann E. MORIN

On 2022-03-08 19:35 +0100, Rafał Miłecki spake thusly:

Quoted 16 lines

From: Rafał Miłecki <[email protected]>

This new package provides "lzma_alone" host binary based on the original LZMA SDK. It provides few extra options when compared to the LZMA Utils / XZ Utils project "lzma" binary (already packaged as the "lzma").

This packaging schema (LZMA SDK with lzma -> lzma_alone rename) follows Debian's solution. Please note that Debian also uses LZMA SDK for the base "lzma" tool which may be considered for Buildroot too - as an independent change. Similar packaging is also used by Ubuntu & Arch.

lzma_alone is a requirement for preparing firmware images for some Broadcom based home routers with a picky CFE bootloader. It has limited LZMA support and building compatible images requires specifying dictionary size and lc/lp/pb LZMA values manually.

Quoted 5 lines
Quoted 1 line

Version 9.22 is used as it's the last release using .tar.bz2 format.

We could introduce a host variant of p7zip, so that we could automatically add that to FOO_EXTRACT_DEPENDENCIES if the host does not already have 7za (or 7zr), like we do for lzip archives for example.

I can see 7z format support is no less messy than LZMA tools.

It seems that 7-Zip project didn't provide command-line tool initially and that resulted in developing p7zip. It seems it has changed though.

So we have two projects:

1. p7zip
* Provides "7z" and "7za"
* Last updated in 2016
* Suffers from security issues
* Requires patching (11 Debian patches, 2 for CVEs)

2. 7-Zip
* Provides "7zr" (documented as "7za")
* Last updated in 2021
* Messy source
* Doesn't even compile on some systems:
../../../C/Util/7z/7zMain.c:363:5: error: initializer element is not constant

     (const UInt64)60 * 60 * 24 * (89 + 365 * (kUnixTimeStartYear - kFileTimeStartYear));
     ^

Which one of above we should try to package?

Quoted 1 line

Applied to master, thanks.

Huge thanks for your help.
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

#3 lore raw
Quoted 29 lines · Rafał Miłecki
Quoted 17 lines · Yann E. MORIN

On 2022-03-08 19:35 +0100, Rafał Miłecki spake thusly:

Quoted 16 lines

From: Rafał Miłecki <[email protected]>

This new package provides "lzma_alone" host binary based on the original LZMA SDK. It provides few extra options when compared to the LZMA Utils / XZ Utils project "lzma" binary (already packaged as the "lzma").

This packaging schema (LZMA SDK with lzma -> lzma_alone rename) follows Debian's solution. Please note that Debian also uses LZMA SDK for the base "lzma" tool which may be considered for Buildroot too - as an independent change. Similar packaging is also used by Ubuntu & Arch.

lzma_alone is a requirement for preparing firmware images for some Broadcom based home routers with a picky CFE bootloader. It has limited LZMA support and building compatible images requires specifying dictionary size and lc/lp/pb LZMA values manually.

Quoted 5 lines
Quoted 1 line

Version 9.22 is used as it's the last release using .tar.bz2 format.

We could introduce a host variant of p7zip, so that we could automatically add that to FOO_EXTRACT_DEPENDENCIES if the host does not already have 7za (or 7zr), like we do for lzip archives for example.

I can see 7z format support is no less messy than LZMA tools.

It seems that 7-Zip project didn't provide command-line tool initially and that resulted in developing p7zip. It seems it has changed though.

Quoted 7 lines

So we have two projects:

1. p7zip
* Provides "7z" and "7za"
* Last updated in 2016
* Suffers from security issues
* Requires patching (11 Debian patches, 2 for CVEs)

Last commit in the fork that we use in package/p7zip [1] is from December '21,
so not that bad I think.

Regards,
Arnout

[1] https://github.com/jinfeihan57/p7zip

Quoted 14 lines

2. 7-Zip
* Provides "7zr" (documented as "7za")
* Last updated in 2021
* Messy source
* Doesn't even compile on some systems:
../../../C/Util/7z/7zMain.c:363:5: error: initializer element is not constant
     (const UInt64)60 * 60 * 24 * (89 + 365 * (kUnixTimeStartYear -
kFileTimeStartYear));
     ^

Which one of above we should try to package?

Quoted 7 lines
Quoted 1 line

Applied to master, thanks.

Huge thanks for your help.
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot