Skip to contentSkip to conversation

xz: Add ptest support

PATCH In review Xu, Chi 4 participants 5 messages Jan 26, 2024 – Feb 5, 2024

lore
Share & tools
lore

Xu, Chi

author
#0 lore raw

PATCH v3 1/1 xz: Add ptest support

There are two types of cases: executables and POSIX shell scripts. All test cases PASS.
Add xz-ptest to PTESTS_FAST because test duration less than 30s on qemux86-64.

root@qemux86-64:~# ptest-runner xz
START: ptest-runner
2024-01-26T03:32
BEGIN: /usr/lib/xz/ptest
=== test_bcj_exact_size.c ===
PASS: test_exact_size
PASS: test_empty_block
---
=== END ===
...
=== test_vli.c ===
PASS: test_lzma_vli_size
PASS: test_lzma_vli_encode
PASS: test_lzma_vli_decode
---
=== END ===
PASS: test_files.sh
PASS: test_scripts.sh
PASS: test_compress.sh
DURATION: 18
END: /usr/lib/xz/ptest
2024-01-26T03:33
STOP: ptest-runner
TOTAL: 1 FAIL: 0
---
 .../distro/include/ptest-packagelists.inc     |  1 +
 meta/recipes-extended/xz/xz/run-ptest         | 26 +++++++++++++++++
 meta/recipes-extended/xz/xz_5.4.5.bb          | 28 +++++++++++++++++--
 3 files changed, 53 insertions(+), 2 deletions(-)
 create mode 100644 meta/recipes-extended/xz/xz/run-ptest
meta/conf/distro/include/ptest-packagelists.inc +1 -0
index a1643c5548..68a3f63a6a 100644
@@ -77,6 +77,7 @@ PTESTS_FAST = "\
     sed \
     slang \
     wayland \
+    xz \
     zlib \
     libexif \
 "
meta/recipes-extended/xz/xz/run-ptest new file +26 -0
new file mode 100644
index 0000000000..cc8ba92016
@@ -0,0 +1,26 @@
+#!/bin/bash
+
+ptestdir=$(dirname "$(readlink -f "$0")")
+cd "$ptestdir"/tests || exit
+
+# executables test cases
+tests_exec=$(file $(ls test_*) | grep "ELF" | awk -F: '{print $1}')
+for i in ${tests_exec}; do
+    ./$i
+done
+
+# POSIX shell script test cases
+export srcdir=${ptestdir}/tests
+test_shell="test_files.sh test_scripts.sh test_compress.sh"
+for j in ${test_shell}; do
+    if [ $j == "test_compress.sh" ]; then
+        ./$j create_compress_files
+    else
+        ./$j
+    fi
+    if [ $? -eq 0 ]; then
+        echo "PASS: $j"
+    else
+        echo "FAIL: $j"
+    fi
+done
meta/recipes-extended/xz/xz_5.4.5.bb +26 -2
index 09eade8be0..0f0849c3e5 100644
@@ -24,13 +24,15 @@ LIC_FILES_CHKSUM = "file://COPYING;md5=c8ea84ebe7b93cce676b54355dc6b2c0 \
                     file://lib/getopt.c;endline=23;md5=2069b0ee710572c03bb3114e4532cd84 \
                     "
 
-SRC_URI = "https://tukaani.org/xz/xz-${PV}.tar.gz"
+SRC_URI = "https://tukaani.org/xz/xz-${PV}.tar.gz \
+           ${@bb.utils.contains('PTEST_ENABLED', '1', 'file://run-ptest', '', d)} \
+          "
 SRC_URI[sha256sum] = "135c90b934aee8fbc0d467de87a05cb70d627da36abe518c357a873709e5b7d6"
 UPSTREAM_CHECK_REGEX = "xz-(?P<pver>\d+(\.\d+)+)\.tar"
 
 CACHED_CONFIGUREVARS += "gl_cv_posix_shell=/bin/sh"
 
-inherit autotools gettext
+inherit autotools gettext ptest
 
 PACKAGES =+ "liblzma"
 
@@ -42,3 +44,25 @@ ALTERNATIVE:${PN} = "xz xzcat unxz \
                      lzma lzcat unlzma"
 
 BBCLASSEXTEND = "native nativesdk"
+
+RDEPENDS:${PN}-ptest += "bash file"
+
+do_compile_ptest() {
+        oe_runmake check TESTS=
+}
+
+do_install_ptest () {
+    install -d ${D}${PTEST_PATH}/tests
+    find ${B}/tests/.libs -type f -executable -exec cp {} ${D}${PTEST_PATH}/tests \;
+    cp ${B}/config.h ${D}${PTEST_PATH}
+    for i in files xzgrep_expected_output test_files.sh test_scripts.sh test_compress.sh; do
+        cp -r ${S}/tests/$i ${D}${PTEST_PATH}/tests
+    done
+    mkdir -p ${D}${PTEST_PATH}/src/xz
+    ln -s ${bindir}/xz ${D}${PTEST_PATH}/src/xz/xz
+    mkdir -p ${D}${PTEST_PATH}/src/xzdec
+    ln -s ${bindir}/xzdec ${D}${PTEST_PATH}/src/xzdec/xzdec
+    mkdir -p ${D}${PTEST_PATH}/src/scripts
+    ln -s ${bindir}/xzdiff ${D}${PTEST_PATH}/src/scripts/xzdiff
+    ln -s ${bindir}/xzgrep ${D}${PTEST_PATH}/src/scripts/xzgrep
+}
Signature · 1 line
2.25.1
#1 lore raw

Hello,

I've had a reproducibility failure:

https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20240203-fdpg888c/packages/diff-html/

Quoted 143 lines · Chi Xu

There are two types of cases: executables and POSIX shell scripts. All test cases PASS.
Add xz-ptest to PTESTS_FAST because test duration less than 30s on qemux86-64.

root@qemux86-64:~# ptest-runner xz
START: ptest-runner
2024-01-26T03:32
BEGIN: /usr/lib/xz/ptest
=== test_bcj_exact_size.c ===
PASS: test_exact_size
PASS: test_empty_block
---
=== END ===
...
=== test_vli.c ===
PASS: test_lzma_vli_size
PASS: test_lzma_vli_encode
PASS: test_lzma_vli_decode
---
=== END ===
PASS: test_files.sh
PASS: test_scripts.sh
PASS: test_compress.sh
DURATION: 18
END: /usr/lib/xz/ptest
2024-01-26T03:33
STOP: ptest-runner
TOTAL: 1 FAIL: 0
---
 .../distro/include/ptest-packagelists.inc     |  1 +
 meta/recipes-extended/xz/xz/run-ptest         | 26 +++++++++++++++++
 meta/recipes-extended/xz/xz_5.4.5.bb          | 28 +++++++++++++++++--
 3 files changed, 53 insertions(+), 2 deletions(-)
 create mode 100644 meta/recipes-extended/xz/xz/run-ptest
meta/conf/distro/include/ptest-packagelists.inc +1 -0
index a1643c5548..68a3f63a6a 100644
@@ -77,6 +77,7 @@ PTESTS_FAST = "\
     sed \
     slang \
     wayland \
+    xz \
     zlib \
     libexif \
 "
meta/recipes-extended/xz/xz/run-ptest new file +26 -0
new file mode 100644
index 0000000000..cc8ba92016
@@ -0,0 +1,26 @@
+#!/bin/bash
+
+ptestdir=$(dirname "$(readlink -f "$0")")
+cd "$ptestdir"/tests || exit
+
+# executables test cases
+tests_exec=$(file $(ls test_*) | grep "ELF" | awk -F: '{print $1}')
+for i in ${tests_exec}; do
+    ./$i
+done
+
+# POSIX shell script test cases
+export srcdir=${ptestdir}/tests
+test_shell="test_files.sh test_scripts.sh test_compress.sh"
+for j in ${test_shell}; do
+    if [ $j == "test_compress.sh" ]; then
+        ./$j create_compress_files
+    else
+        ./$j
+    fi
+    if [ $? -eq 0 ]; then
+        echo "PASS: $j"
+    else
+        echo "FAIL: $j"
+    fi
+done
meta/recipes-extended/xz/xz_5.4.5.bb +26 -2
index 09eade8be0..0f0849c3e5 100644
@@ -24,13 +24,15 @@ LIC_FILES_CHKSUM = "file://COPYING;md5=c8ea84ebe7b93cce676b54355dc6b2c0 \
                     file://lib/getopt.c;endline=23;md5=2069b0ee710572c03bb3114e4532cd84 \
                     "
 
-SRC_URI = "https://tukaani.org/xz/xz-${PV}.tar.gz"
+SRC_URI = "https://tukaani.org/xz/xz-${PV}.tar.gz \
+           ${@bb.utils.contains('PTEST_ENABLED', '1', 'file://run-ptest', '', d)} \
+          "
 SRC_URI[sha256sum] = "135c90b934aee8fbc0d467de87a05cb70d627da36abe518c357a873709e5b7d6"
 UPSTREAM_CHECK_REGEX = "xz-(?P<pver>\d+(\.\d+)+)\.tar"
 
 CACHED_CONFIGUREVARS += "gl_cv_posix_shell=/bin/sh"
 
-inherit autotools gettext
+inherit autotools gettext ptest
 
 PACKAGES =+ "liblzma"
 
@@ -42,3 +44,25 @@ ALTERNATIVE:${PN} = "xz xzcat unxz \
                      lzma lzcat unlzma"
 
 BBCLASSEXTEND = "native nativesdk"
+
+RDEPENDS:${PN}-ptest += "bash file"
+
+do_compile_ptest() {
+        oe_runmake check TESTS=
+}
+
+do_install_ptest () {
+    install -d ${D}${PTEST_PATH}/tests
+    find ${B}/tests/.libs -type f -executable -exec cp {} ${D}${PTEST_PATH}/tests \;
+    cp ${B}/config.h ${D}${PTEST_PATH}
+    for i in files xzgrep_expected_output test_files.sh test_scripts.sh test_compress.sh; do
+        cp -r ${S}/tests/$i ${D}${PTEST_PATH}/tests
+    done
+    mkdir -p ${D}${PTEST_PATH}/src/xz
+    ln -s ${bindir}/xz ${D}${PTEST_PATH}/src/xz/xz
+    mkdir -p ${D}${PTEST_PATH}/src/xzdec
+    ln -s ${bindir}/xzdec ${D}${PTEST_PATH}/src/xzdec/xzdec
+    mkdir -p ${D}${PTEST_PATH}/src/scripts
+    ln -s ${bindir}/xzdiff ${D}${PTEST_PATH}/src/scripts/xzdiff
+    ln -s ${bindir}/xzgrep ${D}${PTEST_PATH}/src/scripts/xzgrep
+}
Signature · 11 lines
2.25.1



-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#194345): https://lists.openembedded.org/g/openembedded-core/message/194345
Mute This Topic: https://lists.openembedded.org/mt/103971280/3617179
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Signature · 3 lines
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
author
#2 lore raw

Hello,

I'm checking this issue, but can not reproduce it on my host now. I tried to build and rebuild xz-ptest on ubuntu 20.04 and ubuntu 22.04, xz-ptest-5.4.5-r0.core2_64.rpm are same.
Do you have any suggestions?

Ubuntu 20.04

root@localhost:/root$ git clone git://git.yoctoproject.org/poky
root@localhost:/root/poky$ cd poky
root@localhost:/root/poky$ source oe-init-build-env
root@localhost:/root/poky/build$ bitbake xz
root@localhost:/root/poky/build$ ls -l tmp/work/core2-64-poky-linux/xz/5.4.5/deploy-rpms/core2_64/xz-ptest-5.4.5-r0.core2_64.rpm
-rw-r--r-- 2 root root 162489 Feb  5 13:27 tmp/work/core2-64-poky-linux/xz/5.4.5/deploy-rpms/core2_64/xz-ptest-5.4.5-r0.core2_64.rpm
root@localhost:/root/poky/build$ md5sum tmp/work/core2-64-poky-linux/xz/5.4.5/deploy-rpms/core2_64/xz-ptest-5.4.5-r0.core2_64.rpm
b26fd0014d1c23f6e3d9bf8704137b82  tmp/work/core2-64-poky-linux/xz/5.4.5/deploy-rpms/core2_64/xz-ptest-5.4.5-r0.core2_64.rpm
root@localhost:/root/poky/build$ bitbake -c cleansstate xz
root@localhost:/root/poky/build$ ls -l tmp/work/core2-64-poky-linux/xz/5.4.5/deploy-rpms/core2_64/xz-ptest-5.4.5-r0.core2_64.rpm
-rw-r--r-- 2 root root 162489 Feb  5 13:32 tmp/work/core2-64-poky-linux/xz/5.4.5/deploy-rpms/core2_64/xz-ptest-5.4.5-r0.core2_64.rpm
root@localhost:/root/poky/build$ md5sum tmp/work/core2-64-poky-linux/xz/5.4.5/deploy-rpms/core2_64/xz-ptest-5.4.5-r0.core2_64.rpm
b26fd0014d1c23f6e3d9bf8704137b82  tmp/work/core2-64-poky-linux/xz/5.4.5/deploy-rpms/core2_64/xz-ptest-5.4.5-r0.core2_64.rpm

Ubuntu 22.04

root@localhost:/root/poky/build$ ls -l tmp/work/core2-64-poky-linux/xz/5.4.5/deploy-rpms/core2_64/xz-ptest-5.4.5-r0.core2_64.rpm
-rw-r--r-- 2 root root 162489 Feb  5 13:56 tmp/work/core2-64-poky-linux/xz/5.4.5/deploy-rpms/core2_64/xz-ptest-5.4.5-r0.core2_64.rpm
root@localhost:/root/poky/build$ md5sum tmp/work/core2-64-poky-linux/xz/5.4.5/deploy-rpms/core2_64/xz-ptest-5.4.5-r0.core2_64.rpm
b26fd0014d1c23f6e3d9bf8704137b82  tmp/work/core2-64-poky-linux/xz/5.4.5/deploy-rpms/core2_64/xz-ptest-5.4.5-r0.core2_64.rpm
root@localhost:/root/poky/build$ bitbake -c cleansstate xz
root@localhost:/root/poky/build$ ls -l tmp/work/core2-64-poky-linux/xz/5.4.5/deploy-rpms/core2_64/xz-ptest-5.4.5-r0.core2_64.rpm
-rw-r--r-- 2 root root 162489 Feb  5 13:59 tmp/work/core2-64-poky-linux/xz/5.4.5/deploy-rpms/core2_64/xz-ptest-5.4.5-r0.core2_64.rpm
root@localhost:/root/poky/build$ md5sum tmp/work/core2-64-poky-linux/xz/5.4.5/deploy-rpms/core2_64/xz-ptest-5.4.5-r0.core2_64.rpm
b26fd0014d1c23f6e3d9bf8704137b82  tmp/work/core2-64-poky-linux/xz/5.4.5/deploy-rpms/core2_64/xz-ptest-5.4.5-r0.core2_64.rpm

Thanks,
Chi

Quoted 179 lines

-----Original Message-----
From: Alexandre Belloni <[email protected]>
Sent: Sunday, February 4, 2024 5:23 AM
To: Xu, Chi <[email protected]>
Cc: [email protected]
Subject: Re: [OE-core] [PATCH v3] xz: Add ptest support

CAUTION: This email comes from a non Wind River email account! Do not click links or open attachments unless you recognize the sender and know the content is safe.

Hello,

I've had a reproducibility failure:

https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20240203-
fdpg888c/packages/diff-html/

meta/recipes-extended/xz/xz_5.4.5.bb LIC_FILES_CHKSUM =

Quoted 90 lines · Chi Xu

There are two types of cases: executables and POSIX shell scripts. All test cases PASS.
Add xz-ptest to PTESTS_FAST because test duration less than 30s on qemux86-64.

root@qemux86-64:~# ptest-runner xz
START: ptest-runner
2024-01-26T03:32
BEGIN: /usr/lib/xz/ptest
=== test_bcj_exact_size.c ===
PASS: test_exact_size
PASS: test_empty_block
---
=== END ===
...
=== test_vli.c ===
PASS: test_lzma_vli_size
PASS: test_lzma_vli_encode
PASS: test_lzma_vli_decode
---
=== END ===
PASS: test_files.sh
PASS: test_scripts.sh
PASS: test_compress.sh
DURATION: 18
END: /usr/lib/xz/ptest
2024-01-26T03:33
STOP: ptest-runner
TOTAL: 1 FAIL: 0
---
 .../distro/include/ptest-packagelists.inc     |  1 +
 meta/recipes-extended/xz/xz/run-ptest         | 26 +++++++++++++++++
 meta/recipes-extended/xz/xz_5.4.5.bb          | 28 +++++++++++++++++--
 3 files changed, 53 insertions(+), 2 deletions(-)  create mode 100644

meta/recipes-extended/xz/xz/run-ptest

meta/conf/distro/include/ptest-packagelists.inc +1 -0
index a1643c5548..68a3f63a6a 100644
@@ -77,6 +77,7 @@ PTESTS_FAST = "\
     sed \
     slang \
     wayland \
+    xz \
     zlib \
     libexif \
 "
meta/recipes-extended/xz/xz/run-ptest new file +26 -0
new file mode 100644
index 0000000000..cc8ba92016
@@ -0,0 +1,26 @@
+#!/bin/bash
+
+ptestdir=$(dirname "$(readlink -f "$0")") cd "$ptestdir"/tests ||
+exit
+
+# executables test cases
+tests_exec=$(file $(ls test_*) | grep "ELF" | awk -F: '{print $1}')
+for i in ${tests_exec}; do
+    ./$i
+done
+
+# POSIX shell script test cases
+export srcdir=${ptestdir}/tests
+test_shell="test_files.sh test_scripts.sh test_compress.sh"
+for j in ${test_shell}; do
+    if [ $j == "test_compress.sh" ]; then
+        ./$j create_compress_files
+    else
+        ./$j
+    fi
+    if [ $? -eq 0 ]; then
+        echo "PASS: $j"
+    else
+        echo "FAIL: $j"
+    fi
+done
meta/recipes-extended/xz/xz_5.4.5.bb +0 -0
index 09eade8be0..0f0849c3e5 100644
@@ -24,13 +24,15 @@ LIC_FILES_CHKSUM =

"file://COPYING;md5=c8ea84ebe7b93cce676b54355dc6b2c0 \

Quoted 1 line

file://lib/getopt.c;endline=23;md5=2069b0ee710572c03bb3114e4532cd84 \

meta/recipes-extended/xz/xz_5.4.5.bb LIC_FILES_CHKSUM = "file://COPYING;md5=c8ea84ebe7b93cce676b54355dc6b2c0 \

Quoted 7 lines
                     "
meta/recipes-extended/xz/xz_5.4.5.bb +3 -1
-SRC_URI = "https://tukaani.org/xz/xz-${PV}.tar.gz"
+SRC_URI = "https://tukaani.org/xz/xz-${PV}.tar.gz \
+           ${@bb.utils.contains('PTEST_ENABLED', '1', 'file://run-ptest', '', d)} \
+          "
 SRC_URI[sha256sum] =

"135c90b934aee8fbc0d467de87a05cb70d627da36abe518c357a873709e5b7d 6"

Quoted 13 lines

UPSTREAM_CHECK_REGEX = "xz-(?P<pver>\d+(\.\d+)+)\.tar"

CACHED_CONFIGUREVARS += "gl_cv_posix_shell=/bin/sh"

meta/recipes-extended/xz/xz_5.4.5.bb +1 -1
-inherit autotools gettext
+inherit autotools gettext ptest
 
 PACKAGES =+ "liblzma"
@@ -42,3 +44,25 @@ ALTERNATIVE:${PN} = "xz xzcat unxz \
                      lzma lzcat unlzma"
 
 BBCLASSEXTEND = "native nativesdk"

meta/recipes-extended/xz/xz_5.4.5.bb ALTERNATIVE:${PN} = "xz xzcat unxz \ · line 56

Quoted 10 lines
meta/recipes-extended/xz/xz_5.4.5.bb +10 -0
+
+RDEPENDS:${PN}-ptest += "bash file"
+
+do_compile_ptest() {
+        oe_runmake check TESTS=
+}
+
+do_install_ptest () {
+    install -d ${D}${PTEST_PATH}/tests
+    find ${B}/tests/.libs -type f -executable -exec cp {}

${D}${PTEST_PATH}/tests \;

meta/recipes-extended/xz/xz_5.4.5.bb ALTERNATIVE:${PN} = "xz xzcat unxz \ · line 58

Quoted 2 lines
meta/recipes-extended/xz/xz_5.4.5.bb +2 -0
+    cp ${B}/config.h ${D}${PTEST_PATH}
+    for i in files xzgrep_expected_output test_files.sh test_scripts.sh

test_compress.sh; do

Quoted 15 lines
meta/recipes-extended/xz/xz_5.4.5.bb +10 -1
+        cp -r ${S}/tests/$i ${D}${PTEST_PATH}/tests
+    done
+    mkdir -p ${D}${PTEST_PATH}/src/xz
+    ln -s ${bindir}/xz ${D}${PTEST_PATH}/src/xz/xz
+    mkdir -p ${D}${PTEST_PATH}/src/xzdec
+    ln -s ${bindir}/xzdec ${D}${PTEST_PATH}/src/xzdec/xzdec
+    mkdir -p ${D}${PTEST_PATH}/src/scripts
+    ln -s ${bindir}/xzdiff ${D}${PTEST_PATH}/src/scripts/xzdiff
+    ln -s ${bindir}/xzgrep ${D}${PTEST_PATH}/src/scripts/xzgrep
+}
--

2.25.1

Quoted 7 lines

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#194345):
https://lists.openembedded.org/g/openembedded-core/message/194345
Mute This Topic: https://lists.openembedded.org/mt/103971280/3617179
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-

core/unsub

Quoted 3 lines

[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

--
Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com

#3 lore raw

Hi,

FYI, since the reproducibility problem only impact the RPM package, it might be related to the problem noticed by Richard last night on IRC:
[11:35:32 pm] <RP> kanavin: It looks like it runs file on the files within the archive but for some reason it isn't sorting that in the header, so probably an unsorted directory listing somewhere
[11:58:34 pm] <RP> kanavin: looking at package_rpm and it's os.walk, there is no sorting in there at all. I suspect that leads to non-determinstic ordering of the files sections in the spec files and I suspect that in turn leads rpm to reorder headers
(https://libera.irclog.whitequark.org/yocto/2024-02-04#35724802;)

Quoted 238 lines · Chi Xu

Hello,

I'm checking this issue, but can not reproduce it on my host now. I tried to build and rebuild xz-ptest on ubuntu 20.04 and ubuntu 22.04, xz-ptest-5.4.5-r0.core2_64.rpm are same.
Do you have any suggestions?

Ubuntu 20.04

root@localhost:/root$ git clone git://git.yoctoproject.org/poky
root@localhost:/root/poky$ cd poky
root@localhost:/root/poky$ source oe-init-build-env
root@localhost:/root/poky/build$ bitbake xz
root@localhost:/root/poky/build$ ls -l
tmp/work/core2-64-poky-linux/xz/5.4.5/deploy-rpms/core2_64/xz-ptest-5.4.5-r0.core2_64.rpm
-rw-r--r-- 2 root root 162489 Feb  5 13:27
tmp/work/core2-64-poky-linux/xz/5.4.5/deploy-rpms/core2_64/xz-ptest-5.4.5-r0.core2_64.rpm
root@localhost:/root/poky/build$ md5sum
tmp/work/core2-64-poky-linux/xz/5.4.5/deploy-rpms/core2_64/xz-ptest-5.4.5-r0.core2_64.rpm
b26fd0014d1c23f6e3d9bf8704137b82
tmp/work/core2-64-poky-linux/xz/5.4.5/deploy-rpms/core2_64/xz-ptest-5.4.5-r0.core2_64.rpm
root@localhost:/root/poky/build$ bitbake -c cleansstate xz
root@localhost:/root/poky/build$ ls -l
tmp/work/core2-64-poky-linux/xz/5.4.5/deploy-rpms/core2_64/xz-ptest-5.4.5-r0.core2_64.rpm
-rw-r--r-- 2 root root 162489 Feb  5 13:32
tmp/work/core2-64-poky-linux/xz/5.4.5/deploy-rpms/core2_64/xz-ptest-5.4.5-r0.core2_64.rpm
root@localhost:/root/poky/build$ md5sum
tmp/work/core2-64-poky-linux/xz/5.4.5/deploy-rpms/core2_64/xz-ptest-5.4.5-r0.core2_64.rpm
b26fd0014d1c23f6e3d9bf8704137b82
tmp/work/core2-64-poky-linux/xz/5.4.5/deploy-rpms/core2_64/xz-ptest-5.4.5-r0.core2_64.rpm

Ubuntu 22.04

root@localhost:/root/poky/build$ ls -l
tmp/work/core2-64-poky-linux/xz/5.4.5/deploy-rpms/core2_64/xz-ptest-5.4.5-r0.core2_64.rpm
-rw-r--r-- 2 root root 162489 Feb  5 13:56
tmp/work/core2-64-poky-linux/xz/5.4.5/deploy-rpms/core2_64/xz-ptest-5.4.5-r0.core2_64.rpm
root@localhost:/root/poky/build$ md5sum
tmp/work/core2-64-poky-linux/xz/5.4.5/deploy-rpms/core2_64/xz-ptest-5.4.5-r0.core2_64.rpm
b26fd0014d1c23f6e3d9bf8704137b82
tmp/work/core2-64-poky-linux/xz/5.4.5/deploy-rpms/core2_64/xz-ptest-5.4.5-r0.core2_64.rpm
root@localhost:/root/poky/build$ bitbake -c cleansstate xz
root@localhost:/root/poky/build$ ls -l
tmp/work/core2-64-poky-linux/xz/5.4.5/deploy-rpms/core2_64/xz-ptest-5.4.5-r0.core2_64.rpm
-rw-r--r-- 2 root root 162489 Feb  5 13:59
tmp/work/core2-64-poky-linux/xz/5.4.5/deploy-rpms/core2_64/xz-ptest-5.4.5-r0.core2_64.rpm
root@localhost:/root/poky/build$ md5sum
tmp/work/core2-64-poky-linux/xz/5.4.5/deploy-rpms/core2_64/xz-ptest-5.4.5-r0.core2_64.rpm
b26fd0014d1c23f6e3d9bf8704137b82
tmp/work/core2-64-poky-linux/xz/5.4.5/deploy-rpms/core2_64/xz-ptest-5.4.5-r0.core2_64.rpm

Thanks,
Chi

Quoted 9 lines

-----Original Message-----
From: Alexandre Belloni <[email protected]>
Sent: Sunday, February 4, 2024 5:23 AM
To: Xu, Chi <[email protected]>
Cc: [email protected]
Subject: Re: [OE-core] [PATCH v3] xz: Add ptest support

CAUTION: This email comes from a non Wind River email account! Do not click links or open attachments unless you recognize the sender

and

Quoted 45 lines

know the content is safe.

Hello,

I've had a reproducibility failure:

https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20240203-
fdpg888c/packages/diff-html/

Quoted 35 lines · Chi Xu

There are two types of cases: executables and POSIX shell scripts. All test cases PASS.
Add xz-ptest to PTESTS_FAST because test duration less than 30s on qemux86-64.

root@qemux86-64:~# ptest-runner xz
START: ptest-runner
2024-01-26T03:32
BEGIN: /usr/lib/xz/ptest
=== test_bcj_exact_size.c ===
PASS: test_exact_size
PASS: test_empty_block
---
=== END ===
...
=== test_vli.c ===
PASS: test_lzma_vli_size
PASS: test_lzma_vli_encode
PASS: test_lzma_vli_decode
---
=== END ===
PASS: test_files.sh
PASS: test_scripts.sh
PASS: test_compress.sh
DURATION: 18
END: /usr/lib/xz/ptest
2024-01-26T03:33
STOP: ptest-runner
TOTAL: 1 FAIL: 0
---
 .../distro/include/ptest-packagelists.inc     |  1 +
 meta/recipes-extended/xz/xz/run-ptest         | 26 +++++++++++++++++
 meta/recipes-extended/xz/xz_5.4.5.bb          | 28

+++++++++++++++++--

Quoted 57 lines

meta/recipes-extended/xz/xz_5.4.5.bb LIC_FILES_CHKSUM =

Quoted 55 lines
 3 files changed, 53 insertions(+), 2 deletions(-)  create mode 100644

meta/recipes-extended/xz/xz/run-ptest

meta/conf/distro/include/ptest-packagelists.inc +1 -0
index a1643c5548..68a3f63a6a 100644
@@ -77,6 +77,7 @@ PTESTS_FAST = "\
     sed \
     slang \
     wayland \
+    xz \
     zlib \
     libexif \
 "
meta/recipes-extended/xz/xz/run-ptest new file +26 -0
new file mode 100644
index 0000000000..cc8ba92016
@@ -0,0 +1,26 @@
+#!/bin/bash
+
+ptestdir=$(dirname "$(readlink -f "$0")") cd "$ptestdir"/tests ||
+exit
+
+# executables test cases
+tests_exec=$(file $(ls test_*) | grep "ELF" | awk -F: '{print $1}')
+for i in ${tests_exec}; do
+    ./$i
+done
+
+# POSIX shell script test cases
+export srcdir=${ptestdir}/tests
+test_shell="test_files.sh test_scripts.sh test_compress.sh"
+for j in ${test_shell}; do
+    if [ $j == "test_compress.sh" ]; then
+        ./$j create_compress_files
+    else
+        ./$j
+    fi
+    if [ $? -eq 0 ]; then
+        echo "PASS: $j"
+    else
+        echo "FAIL: $j"
+    fi
+done
meta/recipes-extended/xz/xz_5.4.5.bb +0 -0
index 09eade8be0..0f0849c3e5 100644
@@ -24,13 +24,15 @@ LIC_FILES_CHKSUM =

"file://COPYING;md5=c8ea84ebe7b93cce676b54355dc6b2c0 \

Quoted 1 line
Quoted 6 lines

file://lib/getopt.c;endline=23;md5=2069b0ee710572c03bb3114e4532cd84 \

meta/recipes-extended/xz/xz_5.4.5.bb LIC_FILES_CHKSUM = "file://COPYING;md5=c8ea84ebe7b93cce676b54355dc6b2c0 \

Quoted 5 lines
                     "
meta/recipes-extended/xz/xz_5.4.5.bb +2 -1
-SRC_URI = "https://tukaani.org/xz/xz-${PV}.tar.gz"
+SRC_URI = "https://tukaani.org/xz/xz-${PV}.tar.gz \
+           ${@bb.utils.contains('PTEST_ENABLED', '1',

'file://run-ptest', '', d)} \

Quoted 16 lines
Quoted 2 lines

+ "
SRC_URI[sha256sum] =

"135c90b934aee8fbc0d467de87a05cb70d627da36abe518c357a873709e5b7d 6"

Quoted 12 lines

UPSTREAM_CHECK_REGEX = "xz-(?P<pver>\d+(\.\d+)+)\.tar"

CACHED_CONFIGUREVARS += "gl_cv_posix_shell=/bin/sh"

meta/recipes-extended/xz/xz_5.4.5.bb +1 -1
-inherit autotools gettext
+inherit autotools gettext ptest
 
 PACKAGES =+ "liblzma"
@@ -42,3 +44,25 @@ ALTERNATIVE:${PN} = "xz xzcat unxz \
                      lzma lzcat unlzma"
 
Quoted 14 lines

meta/recipes-extended/xz/xz_5.4.5.bb ALTERNATIVE:${PN} = "xz xzcat unxz \

Quoted 11 lines
meta/recipes-extended/xz/xz_5.4.5.bb +10 -0
 BBCLASSEXTEND = "native nativesdk"
+
+RDEPENDS:${PN}-ptest += "bash file"
+
+do_compile_ptest() {
+        oe_runmake check TESTS=
+}
+
+do_install_ptest () {
+    install -d ${D}${PTEST_PATH}/tests
+    find ${B}/tests/.libs -type f -executable -exec cp {}

${D}${PTEST_PATH}/tests \;

Quoted 2 lines
meta/recipes-extended/xz/xz_5.4.5.bb +2 -0
+    cp ${B}/config.h ${D}${PTEST_PATH}
+    for i in files xzgrep_expected_output test_files.sh

test_scripts.sh

Quoted 20 lines

test_compress.sh; do

Quoted 11 lines
meta/recipes-extended/xz/xz_5.4.5.bb +10 -1
+        cp -r ${S}/tests/$i ${D}${PTEST_PATH}/tests
+    done
+    mkdir -p ${D}${PTEST_PATH}/src/xz
+    ln -s ${bindir}/xz ${D}${PTEST_PATH}/src/xz/xz
+    mkdir -p ${D}${PTEST_PATH}/src/xzdec
+    ln -s ${bindir}/xzdec ${D}${PTEST_PATH}/src/xzdec/xzdec
+    mkdir -p ${D}${PTEST_PATH}/src/scripts
+    ln -s ${bindir}/xzdiff ${D}${PTEST_PATH}/src/scripts/xzdiff
+    ln -s ${bindir}/xzgrep ${D}${PTEST_PATH}/src/scripts/xzgrep
+}
--
Quoted 6 lines

2.25.1

Quoted 3 lines

--
Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#194916):
https://lists.openembedded.org/g/openembedded-core/message/194916
Mute This Topic: https://lists.openembedded.org/mt/103971280/4316185
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

Signature · 2 lines
Yoann Congal
Smile ECS - Tech expert
#4 lore raw

I just sent two patches for this.

Alex

Quoted 233 lines · Yoann Congal

Hi,

FYI, since the reproducibility problem only impact the RPM package, it might be related to the problem noticed by Richard last night on IRC:
[11:35:32 pm] <RP> kanavin: It looks like it runs file on the files within the archive but for some reason it isn't sorting that in the header, so probably an unsorted directory listing somewhere
[11:58:34 pm] <RP> kanavin: looking at package_rpm and it's os.walk, there is no sorting in there at all. I suspect that leads to non-determinstic ordering of the files sections in the spec files and I suspect that in turn leads rpm to reorder headers
(https://libera.irclog.whitequark.org/yocto/2024-02-04#35724802;)

Quoted 34 lines · Chi Xu

Hello,

I'm checking this issue, but can not reproduce it on my host now. I tried to build and rebuild xz-ptest on ubuntu 20.04 and ubuntu 22.04, xz-ptest-5.4.5-r0.core2_64.rpm are same.
Do you have any suggestions?

Ubuntu 20.04

root@localhost:/root$ git clone git://git.yoctoproject.org/poky
root@localhost:/root/poky$ cd poky
root@localhost:/root/poky$ source oe-init-build-env
root@localhost:/root/poky/build$ bitbake xz
root@localhost:/root/poky/build$ ls -l tmp/work/core2-64-poky-linux/xz/5.4.5/deploy-rpms/core2_64/xz-ptest-5.4.5-r0.core2_64.rpm
-rw-r--r-- 2 root root 162489 Feb  5 13:27 tmp/work/core2-64-poky-linux/xz/5.4.5/deploy-rpms/core2_64/xz-ptest-5.4.5-r0.core2_64.rpm
root@localhost:/root/poky/build$ md5sum tmp/work/core2-64-poky-linux/xz/5.4.5/deploy-rpms/core2_64/xz-ptest-5.4.5-r0.core2_64.rpm
b26fd0014d1c23f6e3d9bf8704137b82  tmp/work/core2-64-poky-linux/xz/5.4.5/deploy-rpms/core2_64/xz-ptest-5.4.5-r0.core2_64.rpm
root@localhost:/root/poky/build$ bitbake -c cleansstate xz
root@localhost:/root/poky/build$ ls -l tmp/work/core2-64-poky-linux/xz/5.4.5/deploy-rpms/core2_64/xz-ptest-5.4.5-r0.core2_64.rpm
-rw-r--r-- 2 root root 162489 Feb  5 13:32 tmp/work/core2-64-poky-linux/xz/5.4.5/deploy-rpms/core2_64/xz-ptest-5.4.5-r0.core2_64.rpm
root@localhost:/root/poky/build$ md5sum tmp/work/core2-64-poky-linux/xz/5.4.5/deploy-rpms/core2_64/xz-ptest-5.4.5-r0.core2_64.rpm
b26fd0014d1c23f6e3d9bf8704137b82  tmp/work/core2-64-poky-linux/xz/5.4.5/deploy-rpms/core2_64/xz-ptest-5.4.5-r0.core2_64.rpm

Ubuntu 22.04

root@localhost:/root/poky/build$ ls -l tmp/work/core2-64-poky-linux/xz/5.4.5/deploy-rpms/core2_64/xz-ptest-5.4.5-r0.core2_64.rpm
-rw-r--r-- 2 root root 162489 Feb  5 13:56 tmp/work/core2-64-poky-linux/xz/5.4.5/deploy-rpms/core2_64/xz-ptest-5.4.5-r0.core2_64.rpm
root@localhost:/root/poky/build$ md5sum tmp/work/core2-64-poky-linux/xz/5.4.5/deploy-rpms/core2_64/xz-ptest-5.4.5-r0.core2_64.rpm
b26fd0014d1c23f6e3d9bf8704137b82  tmp/work/core2-64-poky-linux/xz/5.4.5/deploy-rpms/core2_64/xz-ptest-5.4.5-r0.core2_64.rpm
root@localhost:/root/poky/build$ bitbake -c cleansstate xz
root@localhost:/root/poky/build$ ls -l tmp/work/core2-64-poky-linux/xz/5.4.5/deploy-rpms/core2_64/xz-ptest-5.4.5-r0.core2_64.rpm
-rw-r--r-- 2 root root 162489 Feb  5 13:59 tmp/work/core2-64-poky-linux/xz/5.4.5/deploy-rpms/core2_64/xz-ptest-5.4.5-r0.core2_64.rpm
root@localhost:/root/poky/build$ md5sum tmp/work/core2-64-poky-linux/xz/5.4.5/deploy-rpms/core2_64/xz-ptest-5.4.5-r0.core2_64.rpm
b26fd0014d1c23f6e3d9bf8704137b82  tmp/work/core2-64-poky-linux/xz/5.4.5/deploy-rpms/core2_64/xz-ptest-5.4.5-r0.core2_64.rpm
Quoted 176 lines

Thanks,
Chi

Quoted 170 lines

-----Original Message-----
From: Alexandre Belloni <[email protected]>
Sent: Sunday, February 4, 2024 5:23 AM
To: Xu, Chi <[email protected]>
Cc: [email protected]
Subject: Re: [OE-core] [PATCH v3] xz: Add ptest support

CAUTION: This email comes from a non Wind River email account! Do not click links or open attachments unless you recognize the sender and know the content is safe.

Hello,

I've had a reproducibility failure:

https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20240203-
fdpg888c/packages/diff-html/

meta/recipes-extended/xz/xz_5.4.5.bb LIC_FILES_CHKSUM =

Quoted 90 lines · Chi Xu

There are two types of cases: executables and POSIX shell scripts. All test cases PASS.
Add xz-ptest to PTESTS_FAST because test duration less than 30s on qemux86-64.

root@qemux86-64:~# ptest-runner xz
START: ptest-runner
2024-01-26T03:32
BEGIN: /usr/lib/xz/ptest
=== test_bcj_exact_size.c ===
PASS: test_exact_size
PASS: test_empty_block
---
=== END ===
...
=== test_vli.c ===
PASS: test_lzma_vli_size
PASS: test_lzma_vli_encode
PASS: test_lzma_vli_decode
---
=== END ===
PASS: test_files.sh
PASS: test_scripts.sh
PASS: test_compress.sh
DURATION: 18
END: /usr/lib/xz/ptest
2024-01-26T03:33
STOP: ptest-runner
TOTAL: 1 FAIL: 0
---
 .../distro/include/ptest-packagelists.inc     |  1 +
 meta/recipes-extended/xz/xz/run-ptest         | 26 +++++++++++++++++
 meta/recipes-extended/xz/xz_5.4.5.bb          | 28 +++++++++++++++++--
 3 files changed, 53 insertions(+), 2 deletions(-)  create mode 100644

meta/recipes-extended/xz/xz/run-ptest

meta/conf/distro/include/ptest-packagelists.inc +1 -0
index a1643c5548..68a3f63a6a 100644
@@ -77,6 +77,7 @@ PTESTS_FAST = "\
     sed \
     slang \
     wayland \
+    xz \
     zlib \
     libexif \
 "
meta/recipes-extended/xz/xz/run-ptest new file +26 -0
new file mode 100644
index 0000000000..cc8ba92016
@@ -0,0 +1,26 @@
+#!/bin/bash
+
+ptestdir=$(dirname "$(readlink -f "$0")") cd "$ptestdir"/tests ||
+exit
+
+# executables test cases
+tests_exec=$(file $(ls test_*) | grep "ELF" | awk -F: '{print $1}')
+for i in ${tests_exec}; do
+    ./$i
+done
+
+# POSIX shell script test cases
+export srcdir=${ptestdir}/tests
+test_shell="test_files.sh test_scripts.sh test_compress.sh"
+for j in ${test_shell}; do
+    if [ $j == "test_compress.sh" ]; then
+        ./$j create_compress_files
+    else
+        ./$j
+    fi
+    if [ $? -eq 0 ]; then
+        echo "PASS: $j"
+    else
+        echo "FAIL: $j"
+    fi
+done
meta/recipes-extended/xz/xz_5.4.5.bb +0 -0
index 09eade8be0..0f0849c3e5 100644
@@ -24,13 +24,15 @@ LIC_FILES_CHKSUM =

"file://COPYING;md5=c8ea84ebe7b93cce676b54355dc6b2c0 \

Quoted 1 line

file://lib/getopt.c;endline=23;md5=2069b0ee710572c03bb3114e4532cd84 \

meta/recipes-extended/xz/xz_5.4.5.bb LIC_FILES_CHKSUM = "file://COPYING;md5=c8ea84ebe7b93cce676b54355dc6b2c0 \

Quoted 7 lines
                     "
meta/recipes-extended/xz/xz_5.4.5.bb +3 -1
-SRC_URI = "https://tukaani.org/xz/xz-${PV}.tar.gz"
+SRC_URI = "https://tukaani.org/xz/xz-${PV}.tar.gz \
+           ${@bb.utils.contains('PTEST_ENABLED', '1', 'file://run-ptest', '', d)} \
+          "
 SRC_URI[sha256sum] =

"135c90b934aee8fbc0d467de87a05cb70d627da36abe518c357a873709e5b7d 6"

Quoted 13 lines

UPSTREAM_CHECK_REGEX = "xz-(?P<pver>\d+(\.\d+)+)\.tar"

CACHED_CONFIGUREVARS += "gl_cv_posix_shell=/bin/sh"

meta/recipes-extended/xz/xz_5.4.5.bb +1 -1
-inherit autotools gettext
+inherit autotools gettext ptest
 
 PACKAGES =+ "liblzma"
@@ -42,3 +44,25 @@ ALTERNATIVE:${PN} = "xz xzcat unxz \
                      lzma lzcat unlzma"
 
 BBCLASSEXTEND = "native nativesdk"

meta/recipes-extended/xz/xz_5.4.5.bb ALTERNATIVE:${PN} = "xz xzcat unxz \ · line 56

Quoted 10 lines
meta/recipes-extended/xz/xz_5.4.5.bb +10 -0
+
+RDEPENDS:${PN}-ptest += "bash file"
+
+do_compile_ptest() {
+        oe_runmake check TESTS=
+}
+
+do_install_ptest () {
+    install -d ${D}${PTEST_PATH}/tests
+    find ${B}/tests/.libs -type f -executable -exec cp {}

${D}${PTEST_PATH}/tests \;

meta/recipes-extended/xz/xz_5.4.5.bb ALTERNATIVE:${PN} = "xz xzcat unxz \ · line 58

Quoted 2 lines
meta/recipes-extended/xz/xz_5.4.5.bb +2 -0
+    cp ${B}/config.h ${D}${PTEST_PATH}
+    for i in files xzgrep_expected_output test_files.sh test_scripts.sh

test_compress.sh; do

Quoted 11 lines
meta/recipes-extended/xz/xz_5.4.5.bb +10 -1
+        cp -r ${S}/tests/$i ${D}${PTEST_PATH}/tests
+    done
+    mkdir -p ${D}${PTEST_PATH}/src/xz
+    ln -s ${bindir}/xz ${D}${PTEST_PATH}/src/xz/xz
+    mkdir -p ${D}${PTEST_PATH}/src/xzdec
+    ln -s ${bindir}/xzdec ${D}${PTEST_PATH}/src/xzdec/xzdec
+    mkdir -p ${D}${PTEST_PATH}/src/scripts
+    ln -s ${bindir}/xzdiff ${D}${PTEST_PATH}/src/scripts/xzdiff
+    ln -s ${bindir}/xzgrep ${D}${PTEST_PATH}/src/scripts/xzgrep
+}
--
Quoted 6 lines

2.25.1

--
Alexandre Belloni, co-owner and COO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com

--
Yoann Congal
Smile ECS - Tech expert

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#194917): https://lists.openembedded.org/g/openembedded-core/message/194917
Mute This Topic: https://lists.openembedded.org/mt/103971280/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-