Enable swupdate in reproducible check
A message that is not in the archive (sent off-list or not yet received): [email protected]
v3 cover Enable swupdate in reproducible check
From: venkata pyla <[email protected]>
Hi,
The following patch series enables swupdate in reproducible check CI, and also it include fixes for the reproducible build failures #68 and #69.
with this patch series the reproducible builds are getting passed in amd64 and arm64 architectures considering following artifacts.
- vmlinux
- initrd
- rootfs tarball
- squashfs image
- swu file
armhf is not reproducible yet due to this issue #70 - WIP
filesystem images and disk partition tables are yet to add to the reproducible check - WIP.
v3 changes:
- fix indentation issue in squashfs.bbclass (dd241fbd)
- Use bookworm for reproducible checks
- Update isar refernce to bring reproducible fixes
v2 changes:
- Include swupdate feature in base image itself instead of separate targets
- Added commit to fix file timestamps are not reproducible in squashfs image and swu file.
venkata pyla (8):
.reproducible-check-ci.yml: Include swupdate artifacts in reproducible
check
.reproducible-check-ci.yml: Drop 'base' in job names .reproducible-check-ci.yml: Include only files required for RB check repro-tests.sh: Enable html output in diffoscope squashfs.bbclass: Fix file timestamps are not reproducible in squashfs
image
swupdate.bbclass: Fix file timestamps are not reproducible in swu file .reproducible-check-ci.yml: Enable reproudicble check on bookworm kas-cip.yml: Update isar reference
.reproducible-check-ci.yml | 37 +++++++++++++++++++++++++------------ classes/squashfs.bbclass | 6 ++++++ classes/swupdate.bbclass | 10 +++++++++- kas-cip.yml | 2 +- scripts/repro-tests.sh | 2 ++ 5 files changed, 43 insertions(+), 14 deletions(-)
Signature · 1 line
2.20.1
Jan Kiszka
AppliedQuoted 45 lines · On 07.07.23 16:15, [email protected] wrote:
From: venkata pyla <[email protected]>
Hi,
The following patch series enables swupdate in reproducible check CI, and also it include fixes for the reproducible build failures #68 and #69.
with this patch series the reproducible builds are getting passed in amd64 and arm64 architectures considering following artifacts.
- vmlinux
- initrd
- rootfs tarball
- squashfs image
- swu filearmhf is not reproducible yet due to this issue #70 - WIP
filesystem images and disk partition tables are yet to add to the reproducible check - WIP.
v3 changes:
- fix indentation issue in squashfs.bbclass (dd241fbd)
- Use bookworm for reproducible checks
- Update isar refernce to bring reproducible fixesv2 changes:
- Include swupdate feature in base image itself instead of separate targets
- Added commit to fix file timestamps are not reproducible in squashfs image and swu file.venkata pyla (8):
.reproducible-check-ci.yml: Include swupdate artifacts in reproducible
check
.reproducible-check-ci.yml: Drop 'base' in job names .reproducible-check-ci.yml: Include only files required for RB checkrepro-tests.sh: Enable html output in diffoscope squashfs.bbclass: Fix file timestamps are not reproducible in squashfs
image
swupdate.bbclass: Fix file timestamps are not reproducible in swu file .reproducible-check-ci.yml: Enable reproudicble check on bookworm kas-cip.yml: Update isar reference
Quoted 7 lines
.reproducible-check-ci.yml | 37 +++++++++++++++++++++++++------------ classes/squashfs.bbclass | 6 ++++++ classes/swupdate.bbclass | 10 +++++++++- kas-cip.yml | 2 +- scripts/repro-tests.sh | 2 ++ 5 files changed, 43 insertions(+), 14 deletions(-)
Thanks, applied to next.
Jan
Signature · 2 lines
Siemens AG, Technology Competence Center Embedded Linux
PATCH v3 1/8 .reproducible-check-ci.yml: Include swupdate artifacts in reproducible check
In reply to a message not in the archiveFrom: venkata pyla <[email protected]>
Till now minimal images are verified against the reproducible build checks, now extend the verification to swupdate feature.
--- .reproducible-check-ci.yml | 6 ++++++ 1 file changed, 6 insertions(+)
.reproducible-check-ci.yml +6 -0
@@ -69,18 +69,24 @@ build:qemu-amd64-base-repro-build: - .repro-build variables: target: qemu-amd64 + extension: ebg-swu + wic_targz: enable build:qemu-arm64-base-repro-build: extends: - .repro-build variables: target: qemu-arm64 + extension: ebg-swu + wic_targz: enable build:qemu-arm-base-repro-build: extends: - .repro-build variables: target: qemu-arm + extension: ebg-swu + wic_targz: enable # repro build test test:qemu-amd64-base-repro-test:
Signature · 1 line
2.20.1
PATCH v3 2/8 .reproducible-check-ci.yml: Drop 'base' in job names
In reply to a message not in the archiveFrom: venkata pyla <[email protected]>
Now the reproducible check is not restricted to minimal(base) image but also it include other features like swupdate, secureboot, and security extensions, so drop the name 'base' in the job names to avoid confusion.
--- .reproducible-check-ci.yml | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-)
.reproducible-check-ci.yml +9 -9
@@ -64,7 +64,7 @@ # repro build -build:qemu-amd64-base-repro-build: +build:qemu-amd64-repro-build: extends: - .repro-build variables:
@@ -72,7 +72,7 @@ build:qemu-amd64-base-repro-build: extension: ebg-swu wic_targz: enable -build:qemu-arm64-base-repro-build: +build:qemu-arm64-repro-build: extends: - .repro-build variables:
@@ -80,7 +80,7 @@ build:qemu-arm64-base-repro-build: extension: ebg-swu wic_targz: enable -build:qemu-arm-base-repro-build: +build:qemu-arm-repro-build: extends: - .repro-build variables:
@@ -89,26 +89,26 @@ build:qemu-arm-base-repro-build: wic_targz: enable # repro build test -test:qemu-amd64-base-repro-test: +test:qemu-amd64-repro-test: extends: - .repro-test variables: target: qemu-amd64 dependencies: - - build:qemu-amd64-base-repro-build + - build:qemu-amd64-repro-build -test:qemu-arm64-base-repro-test: +test:qemu-arm64-repro-test: extends: - .repro-test variables: target: qemu-arm64 dependencies: - - build:qemu-arm64-base-repro-build + - build:qemu-arm64-repro-build -test:qemu-arm-base-repro-test: +test:qemu-arm-repro-test: extends: - .repro-test variables: target: qemu-arm dependencies: - - build:qemu-arm-base-repro-build + - build:qemu-arm-repro-build
Signature · 1 line
2.20.1
PATCH v3 3/8 .reproducible-check-ci.yml: Include only files required for RB check
In reply to a message not in the archiveFrom: venkata pyla <[email protected]>
artifacts folder contains many other files like dpkg_status, manifest, etc, which are not verified in reproducible check, so include only the files are used in reproducible check.
--- .reproducible-check-ci.yml | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
.reproducible-check-ci.yml +6 -2
@@ -39,8 +39,12 @@ expire_in: 1 day paths: - scripts/repro-tests.sh - - image1 - - image2 + - image*/*initrd.img + - image*/*-vmlinu* + - image*/*.tar.gz + - image*/*.squashfs + - image*/*.swu + - image*/linux.efi .repro-test: image:
Signature · 1 line
2.20.1
PATCH v3 4/8 repro-tests.sh: Enable html output in diffoscope
In reply to a message not in the archiveFrom: venkata pyla <[email protected]>
diffoscope generates output in html format that will be helpful to easily view the non-reproducible differences in the browser.
--- .reproducible-check-ci.yml | 3 ++- scripts/repro-tests.sh | 2 ++ 2 files changed, 4 insertions(+), 1 deletion(-)
.reproducible-check-ci.yml +2 -1
@@ -64,7 +64,8 @@ when: always expire_in: 1 day paths: - - "./*diffoscope_output.txt" + - "diffoscope_output" + - "./*diffoscope_output.*" # repro build
scripts/repro-tests.sh +2 -0
@@ -78,6 +78,8 @@ res=0 for file in "$@"; do if [ -f "${artifacts1}/${file}" ] && [ -f "${artifacts1}/${file}" ]; then if $DIFFOSCOPE --text "${file}.diffoscope_output.txt" \ + --html-dir diffoscope_output \ + --html "${file}.diffoscope_output.html" \ "${artifacts1}/${file}" \ "${artifacts2}/${file}" > /dev/null 2>&1; then echo "${file}: ${GREEN}Reproducible${NC}" | tee -a diffoscope_output.txt
Signature · 1 line
2.20.1
PATCH v3 5/8 squashfs.bbclass: Fix file timestamps are not reproducible in squashfs image
In reply to a message not in the archiveFrom: venkata pyla <[email protected]>
The generated squashfs image contains non-reproducible file timestamps, so set with value in SOURCE_DATE_EPOCH variable if it is set.
--- classes/squashfs.bbclass | 6 ++++++ 1 file changed, 6 insertions(+)
classes/squashfs.bbclass +6 -0
@@ -31,6 +31,12 @@ SQUASHFS_MEMLIMIT ?= "${@int(get_free_mem() * 3/4)}" SQUASHFS_CREATION_LIMITS = "-mem ${SQUASHFS_MEMLIMIT} -processors ${SQUASHFS_THREADS}" python __anonymous() { + # Set file timestamps for reproducible builds + source_date_epoch = d.getVar('SOURCE_DATE_EPOCH') + if source_date_epoch: + args = " -fstime {time}".format(time=source_date_epoch) + d.appendVar('SQUASHFS_CREATION_ARGS', args) + exclude_directories = d.getVar('SQUASHFS_EXCLUDE_DIRS').split() if len(exclude_directories) == 0: return
Signature · 1 line
2.20.1
PATCH v3 6/8 swupdate.bbclass: Fix file timestamps are not reproducible in swu file
In reply to a message not in the archiveFrom: venkata pyla <[email protected]>
The generated .swu file is not reproducible due to its file contents are copied with its build time stamps, so set all file timestamps to SOURCE_DATE_EPOCH value if it is set.
Closes #69
--- classes/swupdate.bbclass | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
classes/swupdate.bbclass +9 -1
@@ -115,6 +115,10 @@ IMAGE_CMD:swu() { done cd "${PP_WORK}/swu" for file in "${SWU_DESCRIPTION_FILE}" ${SWU_ADDITIONAL_FILES}; do + # Set file timestamps for reproducible builds + if [ -n "${SOURCE_DATE_EPOCH}" ]; then + touch -d@"${SOURCE_DATE_EPOCH}" "$file" + fi echo "$file" if [ -n "$sign" -a "${SWU_DESCRIPTION_FILE}" = "$file" ]; then if [ "${SWU_SIGNATURE_TYPE}" = "rsa" ]; then
@@ -129,9 +133,13 @@ IMAGE_CMD:swu() { -inkey "${PP_WORK}/dev.key" \ -outform DER -nosmimecap -binary fi + # Set file timestamps for reproducible builds + if [ -n "${SOURCE_DATE_EPOCH}" ]; then + touch -d@"${SOURCE_DATE_EPOCH}" "$file.${SWU_SIGNATURE_EXT}" + fi echo "$file.${SWU_SIGNATURE_EXT}" fi - done | cpio -ovL -H crc > "${SWU_BUILDCHROOT_IMAGE_FILE}"' + done | cpio -ovL --reproducible -H crc > "${SWU_BUILDCHROOT_IMAGE_FILE}"' } python do_check_swu_partition_uuids() {
Signature · 1 line
2.20.1
PATCH v3 7/8 .reproducible-check-ci.yml: Enable reproudicble check on bookworm
In reply to a message not in the archiveFrom: venkata pyla <[email protected]>
Generating squashfs image reproducibly is enabled from bookworm version.
--- .reproducible-check-ci.yml | 2 ++ 1 file changed, 2 insertions(+)
.reproducible-check-ci.yml +2 -0
@@ -19,6 +19,7 @@ deploy_kernelci: disable artifact_suffix: ".tar.gz" base_yaml: "kas-cip.yml:kas/board/${target}.yml" + release: bookworm stage: build tags: - large
@@ -56,6 +57,7 @@ - schedules variables: GIT_STRATEGY: none + release: bookworm before_script: - apt update && DEBIAN_FRONTEND=noninteractive apt install -y diffoscope script:
Signature · 1 line
2.20.1
PATCH v3 8/8 kas-cip.yml: Update isar reference
In reply to a message not in the archiveFrom: venkata pyla <[email protected]>
It brings the fix for arm linux files are not build reproducibly
--- kas-cip.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kas-cip.yml +1 -1
@@ -22,7 +22,7 @@ repos: isar: url: https://github.com/ilbers/isar.git - refspec: f95f69d3b83cc28976ef81eeefb5ec5dd17e544f + refspec: 478f3c324f50eca0d082cd1c8f23ef50e28071fb layers: meta:
Signature · 1 line
2.20.1