Skip to contentSkip to conversation

rpm2cpio.sh: update to the last 4.x version

PATCH Superseded [email protected] 3 participants 8 messages Jun 15, 2023 – Jun 19, 2023

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/2 rpm2cpio.sh: update to the last 4.x version

From: Alberto Planas <[email protected]>

openSUSE RPMs are compressing the RPM payload using zstd, that correspond to the magic ID 0x28, 0xb5, 0x2f.

This patch update the script to the last version from the rpm project, and add support to this compression format, and extract the cpio payload using the "unzstd" binary.

---
 scripts/rpm2cpio.sh | 30 ++++++++++++++++++++----------
 1 file changed, 20 insertions(+), 10 deletions(-)
 mode change 100755 => 100644 scripts/rpm2cpio.sh
scripts/rpm2cpio.sh +20 -10
old mode 100755
new mode 100644
index 2034373fe4..8199b43784
@@ -7,7 +7,7 @@ fatal() {
 }
 
 pkg="$1"
-[ -n "$pkg" -a -e "$pkg" ] ||
+[ -n "$pkg" ] && [ -e "$pkg" ] ||
 	fatal "No package supplied"
 
 _dd() {
@@ -16,14 +16,23 @@ _dd() {
 }
 
 calcsize() {
+
+	case "$(_dd $1 bs=4 count=1 | tr -d '\0')" in
+		"$(printf '\216\255\350')"*) ;; # '\x8e\xad\xe8'
+		*) fatal "File doesn't look like rpm: $pkg" ;;
+	esac
+
 	offset=$(($1 + 8))
 
 	local i b b0 b1 b2 b3 b4 b5 b6 b7
 
 	i=0
 	while [ $i -lt 8 ]; do
- 		b=$(_dd $(($offset + $i)) bs=1 count=1; echo X)
- 		b=${b%X}
+		# add . to not loose \n
+		# strip \0 as it gets dropped with warning otherwise
+		b="$(_dd $(($offset + $i)) bs=1 count=1 | tr -d '\0' ; echo .)"
+		b=${b%.}    # strip . again
+
 		[ -z "$b" ] &&
 			b="0" ||
 			b="$(exec printf '%u\n' "'$b")"
@@ -35,7 +44,7 @@ calcsize() {
 	offset=$(($offset + $rsize))
 }
 
-case "$(_dd 0 bs=8 count=1)" in
+case "$(_dd 0 bs=4 count=1 | tr -d '\0')" in
 	"$(printf '\355\253\356\333')"*) ;; # '\xed\xab\xee\xdb'
 	*) fatal "File doesn't look like rpm: $pkg" ;;
 esac
@@ -46,10 +55,11 @@ sigsize=$rsize
 calcsize $(($offset + (8 - ($sigsize % 8)) % 8))
 hdrsize=$rsize
 
-case "$(_dd $offset bs=3 count=1)" in
-	"$(printf '\102\132')"*) _dd $offset | bzip2 -d ;; # '\x42\x5a'
-	"$(printf '\037\213')"*) _dd $offset | gunzip  ;; # '\x1f\x8b'
-	"$(printf '\375\067')"*) _dd $offset | xzcat   ;; # '\xfd\x37'
-	"$(printf '\135\000')"*) _dd $offset | unlzma  ;; # '\x5d\x00'
-	*) fatal "Unrecognized rpm file: $pkg" ;;
+case "$(_dd $offset bs=2 count=1 | tr -d '\0')" in
+	"$(printf '\102\132')") _dd $offset | bunzip2 ;; # '\x42\x5a'
+	"$(printf '\037\213')") _dd $offset | gunzip  ;; # '\x1f\x8b'
+	"$(printf '\375\067')") _dd $offset | xzcat   ;; # '\xfd\x37'
+	"$(printf '\135')") _dd $offset | unlzma      ;; # '\x5d\x00'
+	"$(printf '\050\265')") _dd $offset | unzstd  ;; # '\x28\xb5'
+	*) fatal "Unrecognized payload compression format in rpm file: $pkg" ;;
 esac
Signature · 1 line
2.41.0
#1 lore raw

PATCH v2 2/2 bitbake.conf: add unzstd in HOSTTOOLS

From: Alberto Planas <[email protected]>

rpm2cpio.sh can make calls to unzstd to uncompress the RPM payload that conform the cpio file.

zstd is already part of HOSTTOOLS, as a link to the system installed zstd.

This patch add unzstd in HOSTOOLS list as a non-optional binary, so is available to rpm2cpio.sh when it is required.

---
 meta/conf/bitbake.conf | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/conf/bitbake.conf +1 -1
index e28df7478f..702881144e 100644
@@ -524,7 +524,7 @@ HOSTTOOLS += " \
     python3 pzstd ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \
     sha1sum sha224sum sha256sum sha384sum sha512sum \
     sleep sort split stat strings strip tail tar tee test touch tr true uname \
-    uniq wc wget which xargs zstd \
+    uniq unzstd wc wget which xargs zstd \
 "
 
 # Tools needed to run testimage runtime image testing
Signature · 1 line
2.41.0
#2 lore raw

Re: [OE-core] [PATCH v2 2/2] bitbake.conf: add unzstd in HOSTTOOLS

Quoted 21 lines · Alberto Planas via lists.openembedded.org

From: Alberto Planas <[email protected]>

rpm2cpio.sh can make calls to unzstd to uncompress the RPM payload that conform the cpio file.

zstd is already part of HOSTTOOLS, as a link to the system installed zstd.

This patch add unzstd in HOSTOOLS list as a non-optional binary, so is available to rpm2cpio.sh when it is required.

---
 meta/conf/bitbake.conf | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/conf/bitbake.conf +0 -0
index e28df7478f..702881144e 100644

meta/conf/bitbake.conf HOSTTOOLS += " \ · line 530

Quoted 8 lines
meta/conf/bitbake.conf +1 -1
@@ -524,7 +524,7 @@ HOSTTOOLS += " \
     python3 pzstd ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \
     sha1sum sha224sum sha256sum sha384sum sha512sum \
     sleep sort split stat strings strip tail tar tee test touch tr true uname \
-    uniq wc wget which xargs zstd \
+    uniq unzstd wc wget which xargs zstd \
 "
 
 # Tools needed to run testimage runtime image testing

Couldn't we just use zstd -d in the script? They're always the same binary anyway as far as I understand it...

Cheers,

Richard

#3 lore raw

Re: [OE-core] [PATCH v2 2/2] bitbake.conf: add unzstd in HOSTTOOLS

Quoted 4 lines

From: Richard Purdie <[email protected]>

Couldn't we just use zstd -d in the script? They're always the same binary anyway as far as I understand it...

Yes, this should work. I decided to include the second patch to avoid changing the rpm2cpio.sh code, that is a verbatim copy of the upstream script (minus the added comment from the old version, that reference where to find the script)

If this is the preferred way I can do a v3 that drop this patch and changes rpm2cpio.sh.

#4 lore raw

Re: [OE-core] [PATCH v2 2/2] bitbake.conf: add unzstd in HOSTTOOLS

Quoted 5 lines · Alberto Planas Dominguez
Quoted 4 lines

From: Richard Purdie <[email protected]>

Couldn't we just use zstd -d in the script? They're always the same binary anyway as far as I understand it...

Quoted 7 lines

Yes, this should work. I decided to include the second patch to avoid changing the rpm2cpio.sh code, that is a verbatim copy of the upstream script (minus the added comment from the old version, that reference where to find the script)

If this is the preferred way I can do a v3 that drop this patch and changes rpm2cpio.sh.

I guess we can stay in sync then, I was just hoping not to add more to HOSTTOOLS. Thanks for the context.

Cheers,

Richard

#5 lore raw

Re: [OE-core] [PATCH v2 2/2] bitbake.conf: add unzstd in HOSTTOOLS

I am seeing some race condition see

https://errors.yoctoproject.org/Errors/Details/708923/

Quoted 30 lines · Richard Purdie
Quoted 5 lines · Alberto Planas Dominguez
Quoted 4 lines

From: Richard Purdie <[email protected]>

Couldn't we just use zstd -d in the script? They're always the same binary anyway as far as I understand it...

Quoted 7 lines

Yes, this should work. I decided to include the second patch to avoid changing the rpm2cpio.sh code, that is a verbatim copy of the upstream script (minus the added comment from the old version, that reference where to find the script)

If this is the preferred way I can do a v3 that drop this patch and changes rpm2cpio.sh.

I guess we can stay in sync then, I was just hoping not to add more to HOSTTOOLS. Thanks for the context.

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#182861): https://lists.openembedded.org/g/openembedded-core/message/182861
Mute This Topic: https://lists.openembedded.org/mt/99546093/1997914
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#6 lore raw
In reply to [email protected]

I am seeing some race condition see

https://errors.yoctoproject.org/Errors/Details/708923/

Quoted 96 lines · Alberto Planas via lists.openembedded.org

From: Alberto Planas <[email protected]>

openSUSE RPMs are compressing the RPM payload using zstd, that correspond to the magic ID 0x28, 0xb5, 0x2f.

This patch update the script to the last version from the rpm project, and add support to this compression format, and extract the cpio payload using the "unzstd" binary.

---
 scripts/rpm2cpio.sh | 30 ++++++++++++++++++++----------
 1 file changed, 20 insertions(+), 10 deletions(-)
 mode change 100755 => 100644 scripts/rpm2cpio.sh
scripts/rpm2cpio.sh +20 -11
old mode 100755
new mode 100644
index 2034373fe4..8199b43784
@@ -7,7 +7,7 @@ fatal() {
 }
 
 pkg="$1"
-[ -n "$pkg" -a -e "$pkg" ] ||
+[ -n "$pkg" ] && [ -e "$pkg" ] ||
        fatal "No package supplied"
 
 _dd() {
@@ -16,14 +16,23 @@ _dd() {
 }
 
 calcsize() {
+
+       case "$(_dd $1 bs=4 count=1 | tr -d '\0')" in
+               "$(printf '\216\255\350')"*) ;; # '\x8e\xad\xe8'
+               *) fatal "File doesn't look like rpm: $pkg" ;;
+       esac
+
        offset=$(($1 + 8))
 
        local i b b0 b1 b2 b3 b4 b5 b6 b7
 
        i=0
        while [ $i -lt 8 ]; do
-               b=$(_dd $(($offset + $i)) bs=1 count=1; echo X)
-               b=${b%X}
+               # add . to not loose \n
+               # strip \0 as it gets dropped with warning otherwise
+               b="$(_dd $(($offset + $i)) bs=1 count=1 | tr -d '\0' ; echo .)"
+               b=${b%.}    # strip . again
+
                [ -z "$b" ] &&
                        b="0" ||
                        b="$(exec printf '%u\n' "'$b")"
@@ -35,7 +44,7 @@ calcsize() {
        offset=$(($offset + $rsize))
 }
 
-case "$(_dd 0 bs=8 count=1)" in
+case "$(_dd 0 bs=4 count=1 | tr -d '\0')" in
        "$(printf '\355\253\356\333')"*) ;; # '\xed\xab\xee\xdb'
        *) fatal "File doesn't look like rpm: $pkg" ;;
 esac
@@ -46,10 +55,11 @@ sigsize=$rsize
 calcsize $(($offset + (8 - ($sigsize % 8)) % 8))
 hdrsize=$rsize
 
-case "$(_dd $offset bs=3 count=1)" in
-       "$(printf '\102\132')"*) _dd $offset | bzip2 -d ;; # '\x42\x5a'
-       "$(printf '\037\213')"*) _dd $offset | gunzip  ;; # '\x1f\x8b'
-       "$(printf '\375\067')"*) _dd $offset | xzcat   ;; # '\xfd\x37'
-       "$(printf '\135\000')"*) _dd $offset | unlzma  ;; # '\x5d\x00'
-       *) fatal "Unrecognized rpm file: $pkg" ;;
+case "$(_dd $offset bs=2 count=1 | tr -d '\0')" in
+       "$(printf '\102\132')") _dd $offset | bunzip2 ;; # '\x42\x5a'
+       "$(printf '\037\213')") _dd $offset | gunzip  ;; # '\x1f\x8b'
+       "$(printf '\375\067')") _dd $offset | xzcat   ;; # '\xfd\x37'
+       "$(printf '\135')") _dd $offset | unlzma      ;; # '\x5d\x00'
+       "$(printf '\050\265')") _dd $offset | unzstd  ;; # '\x28\xb5'
+       *) fatal "Unrecognized payload compression format in rpm file: $pkg" ;;
 esac
--

2.41.0

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#182844): https://lists.openembedded.org/g/openembedded-core/message/182844
Mute This Topic: https://lists.openembedded.org/mt/99546092/1997914
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-