Skip to contentSkip to conversation

Enable swupdate in reproducible check

PATCH Superseded [email protected] 2 participants 11 messages Jul 3, 2023 1 message not in the archive

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

[email protected]

author
#1 lore raw

v2 cover Enable swupdate in reproducible check

From: venkata pyla <[email protected]>

Hi,

The following patch series enables swupdate in reproducible check CI, and also it include fixes for the reproducible build failures with swupdate feature.

venkata pyla (6):
.reproducible-check-ci.yml: Include swupdate artifacts in reproducible
check
.reproducible-check-ci.yml: Drop 'base' in job names .reproducible-check-ci.yml: Include only files required for RB check repro-tests.sh: Enable html output in diffoscope squashfs.bbclass: Fix file timestamps are not reproducible in squashfs
image
swupdate.bbclass: Fix file timestamps are not reproducible in swu file

 .reproducible-check-ci.yml | 35 +++++++++++++++++++++++------------
 classes/squashfs.bbclass   |  6 ++++++
 classes/swupdate.bbclass   | 10 +++++++++-
 scripts/repro-tests.sh     |  2 ++
 4 files changed, 40 insertions(+), 13 deletions(-)
Signature · 1 line
2.20.1
author
#2 lore raw

PATCH v2 1/6 .reproducible-check-ci.yml: Include swupdate artifacts in reproducible check

In reply to a message not in the archive

From: venkata pyla <[email protected]>

Till now minimal images are verified against the reproducible build checks, now extend the verification to swupdate feature.

Signed-off-by venkata pyla <[email protected]>
---
 .reproducible-check-ci.yml | 6 ++++++
 1 file changed, 6 insertions(+)
.reproducible-check-ci.yml +6 -0
index 6eb82ea..021d4ef 100644
@@ -71,18 +71,24 @@ build:qemu-amd64-base-repro-build:
     - .repro-build
   variables:
     target: qemu-amd64
+    extension: ebg-swu
+    wic_targz: enable
 
 build:qemu-arm64-base-repro-build:
   extends:
     - .repro-build
   variables:
     target: qemu-arm64
+    extension: ebg-swu
+    wic_targz: enable
 
 build:qemu-arm-base-repro-build:
   extends:
     - .repro-build
   variables:
     target: qemu-arm
+    extension: ebg-swu
+    wic_targz: enable
 
 # repro build test
 test:qemu-amd64-base-repro-test:
Signature · 1 line
2.20.1
author
#3 lore raw

PATCH v2 2/6 .reproducible-check-ci.yml: Drop 'base' in job names

In reply to a message not in the archive

From: venkata pyla <[email protected]>

Now the reproducible check is not restricted to minimal(base) image but also it include other features like swupdate, secureboot, and security extensions, so drop the name 'base' in the job names to avoid confusion.

Signed-off-by venkata pyla <[email protected]>
---
 .reproducible-check-ci.yml | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)
.reproducible-check-ci.yml +9 -9
index 021d4ef..f19bd2c 100644
@@ -66,7 +66,7 @@
 
 
 # repro build
-build:qemu-amd64-base-repro-build:
+build:qemu-amd64-repro-build:
   extends:
     - .repro-build
   variables:

@@ -74,7 +74,7 @@ build:qemu-amd64-base-repro-build:
     extension: ebg-swu
     wic_targz: enable
 
-build:qemu-arm64-base-repro-build:
+build:qemu-arm64-repro-build:
   extends:
     - .repro-build
   variables:

@@ -82,7 +82,7 @@ build:qemu-arm64-base-repro-build:
     extension: ebg-swu
     wic_targz: enable
 
-build:qemu-arm-base-repro-build:
+build:qemu-arm-repro-build:
   extends:
     - .repro-build
   variables:

@@ -91,26 +91,26 @@ build:qemu-arm-base-repro-build:
     wic_targz: enable
 
 # repro build test
-test:qemu-amd64-base-repro-test:
+test:qemu-amd64-repro-test:
   extends:
     - .repro-test
   variables:
     target: qemu-amd64
   dependencies:
-    - build:qemu-amd64-base-repro-build
+    - build:qemu-amd64-repro-build
 
-test:qemu-arm64-base-repro-test:
+test:qemu-arm64-repro-test:
   extends:
     - .repro-test
   variables:
     target: qemu-arm64
   dependencies:
-    - build:qemu-arm64-base-repro-build
+    - build:qemu-arm64-repro-build
 
-test:qemu-arm-base-repro-test:
+test:qemu-arm-repro-test:
   extends:
     - .repro-test
   variables:
     target: qemu-arm
   dependencies:
-    - build:qemu-arm-base-repro-build
+    - build:qemu-arm-repro-build
Signature · 1 line
2.20.1
author
#4 lore raw

PATCH v2 3/6 .reproducible-check-ci.yml: Include only files required for RB check

In reply to a message not in the archive

From: venkata pyla <[email protected]>

artifacts folder contains many other files like dpkg_status, manifest, etc, which are not verified in reproducible check, so include only the files are used in reproducible check.

Signed-off-by venkata pyla <[email protected]>
---
 .reproducible-check-ci.yml | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
.reproducible-check-ci.yml +6 -2
index f19bd2c..88644a7 100644
@@ -40,8 +40,12 @@
     expire_in: 1 day
     paths:
       - scripts/repro-tests.sh
-      - image1
-      - image2
+      - image*/*initrd.img
+      - image*/*-vmlinu*
+      - image*/*.tar.gz
+      - image*/*.squashfs
+      - image*/*.swu
+      - image*/linux.efi
 
 .repro-test:
   image:
Signature · 1 line
2.20.1
author
#5 lore raw

PATCH v2 4/6 repro-tests.sh: Enable html output in diffoscope

In reply to a message not in the archive

From: venkata pyla <[email protected]>

diffoscope generates output in html format that will be helpful to easily view the non-reproducible differences in the browser.

Signed-off-by venkata pyla <[email protected]>
---
 .reproducible-check-ci.yml | 3 ++-
 scripts/repro-tests.sh     | 2 ++
 2 files changed, 4 insertions(+), 1 deletion(-)
.reproducible-check-ci.yml +2 -1
index 88644a7..1f59eba 100644
@@ -66,7 +66,8 @@
     when: always
     expire_in: 1 day
     paths:
-      - "./*diffoscope_output.txt"
+      - "diffoscope_output"
+      - "./*diffoscope_output.*"
 
 
 # repro build
scripts/repro-tests.sh +2 -0
index b37c15e..94de950 100755
@@ -78,6 +78,8 @@ res=0
 for file in "$@"; do
 	if [ -f "${artifacts1}/${file}" ] && [ -f "${artifacts1}/${file}" ]; then
 		if $DIFFOSCOPE --text "${file}.diffoscope_output.txt" \
+			--html-dir diffoscope_output \
+			--html "${file}.diffoscope_output.html" \
 			"${artifacts1}/${file}" \
 			"${artifacts2}/${file}" > /dev/null 2>&1; then
 			echo "${file}: ${GREEN}Reproducible${NC}" | tee -a diffoscope_output.txt
Signature · 1 line
2.20.1
author
#6 lore raw

PATCH v2 5/6 squashfs.bbclass: Fix file timestamps are not reproducible in squashfs image

In reply to a message not in the archive

From: venkata pyla <[email protected]>

The generated squashfs image contains non-reproducible file timestamps, so set with value in SOURCE_DATE_EPOCH variable if it is set.

Closes #68
Signed-off-by venkata pyla <[email protected]>
---
 classes/squashfs.bbclass | 6 ++++++
 1 file changed, 6 insertions(+)
classes/squashfs.bbclass +6 -0
index b39be0c..469cdc5 100644
@@ -31,6 +31,12 @@ SQUASHFS_MEMLIMIT ?= "${@int(get_free_mem() * 3/4)}"
 SQUASHFS_CREATION_LIMITS = "-mem ${SQUASHFS_MEMLIMIT} -processors ${SQUASHFS_THREADS}"
 
 python __anonymous() {
+    # Set file timestamps for reproducible builds
+    source_date_epoch = d.getVar('SOURCE_DATE_EPOCH')
+    if source_date_epoch:
+       args = " -fstime {time}".format(time=source_date_epoch)
+    d.appendVar('SQUASHFS_CREATION_ARGS', args)
+
     exclude_directories = d.getVar('SQUASHFS_EXCLUDE_DIRS').split()
     if len(exclude_directories) == 0:
         return
Signature · 1 line
2.20.1
#7 lore raw

Re: [isar-cip-core v2 5/6] squashfs.bbclass: Fix file timestamps are not reproducible in squashfs image

Quoted 17 lines · On 03.07.23 08:22, [email protected] wrote:

From: venkata pyla <[email protected]>

The generated squashfs image contains non-reproducible file timestamps, so set with value in SOURCE_DATE_EPOCH variable if it is set.

Closes #68
Signed-off-by venkata pyla <[email protected]>
---
 classes/squashfs.bbclass | 6 ++++++
 1 file changed, 6 insertions(+)
classes/squashfs.bbclass +0 -0
index b39be0c..469cdc5 100644

classes/squashfs.bbclass SQUASHFS_MEMLIMIT ?= "${@int(get_free_mem() * 3/4)}" · line 38

Quoted 8 lines
classes/squashfs.bbclass +5 -0
@@ -31,6 +31,12 @@ SQUASHFS_MEMLIMIT ?= "${@int(get_free_mem() * 3/4)}"
 SQUASHFS_CREATION_LIMITS = "-mem ${SQUASHFS_MEMLIMIT} -processors ${SQUASHFS_THREADS}"
 
 python __anonymous() {
+    # Set file timestamps for reproducible builds
+    source_date_epoch = d.getVar('SOURCE_DATE_EPOCH')
+    if source_date_epoch:
+       args = " -fstime {time}".format(time=source_date_epoch)
+    d.appendVar('SQUASHFS_CREATION_ARGS', args)

Wrong indention of this line - args is undefined if source_date_epoch is None.

classes/squashfs.bbclass SQUASHFS_MEMLIMIT ?= "${@int(get_free_mem() * 3/4)}" · line 42

Quoted 4 lines
classes/squashfs.bbclass +1 -0
+
     exclude_directories = d.getVar('SQUASHFS_EXCLUDE_DIRS').split()
     if len(exclude_directories) == 0:
         return

Jan

Signature · 2 lines
Siemens AG, Technology
Competence Center Embedded Linux
author
#8 lore raw

RE: [cip-dev] [isar-cip-core v2 5/6] squashfs.bbclass: Fix file timestamps are not reproducible in squashfs image

Quoted 36 lines

-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Jan Kiszka via lists.cip-project.org
Sent: Monday, July 3, 2023 4:21 PM
To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) <Venkata.Pyla@toshiba- tsip.com>; [email protected]
Cc: dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba- tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ ○ＤＩＧ□ＭＰＳ○ＭＰ４) <[email protected]>
Subject: Re: [cip-dev] [isar-cip-core v2 5/6] squashfs.bbclass: Fix file timestamps are not reproducible in squashfs image

classes/squashfs.bbclass SQUASHFS_MEMLIMIT ?= "${@int(get_free_mem() *

Quoted 17 lines · On 03.07.23 08:22, [email protected] wrote:

From: venkata pyla <[email protected]>

The generated squashfs image contains non-reproducible file timestamps, so set with value in SOURCE_DATE_EPOCH variable if it is set.

Closes #68
Signed-off-by venkata pyla <[email protected]>
---
 classes/squashfs.bbclass | 6 ++++++
 1 file changed, 6 insertions(+)
classes/squashfs.bbclass index renamed from classes/squashfs.bbclass +0 -0

b39be0c..469cdc5 100644

classes/squashfs.bbclass +0 -0
@@ -31,6 +31,12 @@ SQUASHFS_MEMLIMIT ?= "${@int(get_free_mem() *

3/4)}"

classes/squashfs.bbclass SQUASHFS_MEMLIMIT ?= "${@int(get_free_mem() * · line 31

Quoted 1 line
classes/squashfs.bbclass +0 -0
 SQUASHFS_CREATION_LIMITS = "-mem ${SQUASHFS_MEMLIMIT} -processors

${SQUASHFS_THREADS}"

Quoted 1 line
Quoted 8 lines

classes/squashfs.bbclass SQUASHFS_MEMLIMIT ?= "${@int(get_free_mem() * 3/4)}" · line 38

Quoted 6 lines

python __anonymous() {

classes/squashfs.bbclass +5 -0
+    # Set file timestamps for reproducible builds
+    source_date_epoch = d.getVar('SOURCE_DATE_EPOCH')
+    if source_date_epoch:
+       args = " -fstime {time}".format(time=source_date_epoch)
+    d.appendVar('SQUASHFS_CREATION_ARGS', args)

Wrong indention of this line - args is undefined if source_date_epoch is None.

Thanks, the assignment should be inside the if loop, I will correct it in v3

Quoted 11 lines
Quoted 5 lines

+
exclude_directories = d.getVar('SQUASHFS_EXCLUDE_DIRS').split() if len(exclude_directories) == 0:
return

Jan

--
Siemens AG, Technology
Competence Center Embedded Linux

author
#9 lore raw

PATCH v2 6/6 swupdate.bbclass: Fix file timestamps are not reproducible in swu file

In reply to a message not in the archive

From: venkata pyla <[email protected]>

The generated .swu file is not reproducible due to its file contents are copied with its build time stamps, so set all file timestamps to SOURCE_DATE_EPOCH value if it is set.

Closes #69

Signed-off-by venkata pyla <[email protected]>
---
 classes/swupdate.bbclass | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
classes/swupdate.bbclass +9 -1
index ac59c00..3d2b5f0 100644
@@ -115,6 +115,10 @@ IMAGE_CMD:swu() {
         done
         cd "${PP_WORK}/swu"
         for file in "${SWU_DESCRIPTION_FILE}" ${SWU_ADDITIONAL_FILES}; do
+            # Set file timestamps for reproducible builds
+            if [ -n "${SOURCE_DATE_EPOCH}" ]; then
+                touch -d@"${SOURCE_DATE_EPOCH}" "$file"
+            fi
             echo "$file"
             if [ -n "$sign" -a "${SWU_DESCRIPTION_FILE}" = "$file" ]; then
                 if [ "${SWU_SIGNATURE_TYPE}" = "rsa" ]; then

@@ -129,9 +133,13 @@ IMAGE_CMD:swu() {
                         -inkey "${PP_WORK}/dev.key" \
                         -outform DER -nosmimecap -binary
                 fi
+                # Set file timestamps for reproducible builds
+                if [ -n "${SOURCE_DATE_EPOCH}" ]; then
+                    touch -d@"${SOURCE_DATE_EPOCH}" "$file.${SWU_SIGNATURE_EXT}"
+                fi
                 echo "$file.${SWU_SIGNATURE_EXT}"
            fi
-        done | cpio -ovL -H crc > "${SWU_BUILDCHROOT_IMAGE_FILE}"'
+        done | cpio -ovL --reproducible -H crc > "${SWU_BUILDCHROOT_IMAGE_FILE}"'
 }
 
 python do_check_swu_partition_uuids() {
Signature · 1 line
2.20.1
#10 lore raw

Re: [isar-cip-core v2 6/6] swupdate.bbclass: Fix file timestamps are not reproducible in swu file

Quoted 18 lines · On 03.07.23 08:22, [email protected] wrote:

From: venkata pyla <[email protected]>

The generated .swu file is not reproducible due to its file contents are copied with its build time stamps, so set all file timestamps to SOURCE_DATE_EPOCH value if it is set.

Closes #69

Signed-off-by venkata pyla <[email protected]>
---
 classes/swupdate.bbclass | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
classes/swupdate.bbclass +0 -0
index ac59c00..3d2b5f0 100644

classes/swupdate.bbclass IMAGE_CMD:swu() { · line 121

Quoted 7 lines
classes/swupdate.bbclass +4 -0
@@ -115,6 +115,10 @@ IMAGE_CMD:swu() {
         done
         cd "${PP_WORK}/swu"
         for file in "${SWU_DESCRIPTION_FILE}" ${SWU_ADDITIONAL_FILES}; do
+            # Set file timestamps for reproducible builds
+            if [ -n "${SOURCE_DATE_EPOCH}" ]; then
+                touch -d@"${SOURCE_DATE_EPOCH}" "$file"
+            fi

Does this work for SWU_ADDITIONAL_FILES as well? If I read the code correctly, those are just symbolic links.

Jan

Quoted 18 lines
classes/swupdate.bbclass +5 -1
             echo "$file"
             if [ -n "$sign" -a "${SWU_DESCRIPTION_FILE}" = "$file" ]; then
                 if [ "${SWU_SIGNATURE_TYPE}" = "rsa" ]; then

@@ -129,9 +133,13 @@ IMAGE_CMD:swu() {
                         -inkey "${PP_WORK}/dev.key" \
                         -outform DER -nosmimecap -binary
                 fi
+                # Set file timestamps for reproducible builds
+                if [ -n "${SOURCE_DATE_EPOCH}" ]; then
+                    touch -d@"${SOURCE_DATE_EPOCH}" "$file.${SWU_SIGNATURE_EXT}"
+                fi
                 echo "$file.${SWU_SIGNATURE_EXT}"
            fi
-        done | cpio -ovL -H crc > "${SWU_BUILDCHROOT_IMAGE_FILE}"'
+        done | cpio -ovL --reproducible -H crc > "${SWU_BUILDCHROOT_IMAGE_FILE}"'
 }
 
 python do_check_swu_partition_uuids() {
Signature · 2 lines
Siemens AG, Technology
Competence Center Embedded Linux
author
#11 lore raw

RE: [cip-dev] [isar-cip-core v2 6/6] swupdate.bbclass: Fix file timestamps are not reproducible in swu file

Quoted 28 lines

-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Jan Kiszka via lists.cip-project.org
Sent: Monday, July 3, 2023 4:26 PM
To: pyla venkata(ＴＳＩＰ TMIEC ODG Porting) <Venkata.Pyla@toshiba- tsip.com>; [email protected]
Cc: dinesh kumar(ＴＳＩＰ TMIEC ODG Porting) <dinesh.kumar@toshiba- tsip.com>; hayashi kazuhiro(林 和宏 ＤＭＥ ○ＤＩＧ□ＭＰＳ○ＭＰ４) <[email protected]>
Subject: Re: [cip-dev] [isar-cip-core v2 6/6] swupdate.bbclass: Fix file timestamps are not reproducible in swu file

Quoted 13 lines · On 03.07.23 08:22, [email protected] wrote:

From: venkata pyla <[email protected]>

The generated .swu file is not reproducible due to its file contents are copied with its build time stamps, so set all file timestamps to SOURCE_DATE_EPOCH value if it is set.

Closes #69

Signed-off-by venkata pyla <[email protected]>
---
 classes/swupdate.bbclass | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
Quoted 16 lines

classes/swupdate.bbclass IMAGE_CMD:swu() { · line 121

Quoted 13 lines
classes/swupdate.bbclass index renamed from classes/swupdate.bbclass +0 -0

ac59c00..3d2b5f0 100644

classes/swupdate.bbclass +0 -0
@@ -115,6 +115,10 @@ IMAGE_CMD:swu() {
         done
         cd "${PP_WORK}/swu"
         for file in "${SWU_DESCRIPTION_FILE}"

${SWU_ADDITIONAL_FILES}; do

classes/swupdate.bbclass +4 -0
+            # Set file timestamps for reproducible builds
+            if [ -n "${SOURCE_DATE_EPOCH}" ]; then
+                touch -d@"${SOURCE_DATE_EPOCH}" "$file"
+            fi

Does this work for SWU_ADDITIONAL_FILES as well? If I read the code correctly, those are just symbolic links.

Yes, it works because the touch command (without -h option) changes the original file timestamp instead of the symbolic link file.

Quoted 27 lines

Jan

classes/swupdate.bbclass IMAGE_CMD:swu() {

Quoted 10 lines
             echo "$file"
             if [ -n "$sign" -a "${SWU_DESCRIPTION_FILE}" = "$file" ]; then
                 if [ "${SWU_SIGNATURE_TYPE}" = "rsa" ]; then @@

-129,9 +133,13 @@ IMAGE_CMD:swu() {
-inkey "${PP_WORK}/dev.key" \
-outform DER -nosmimecap -binary
fi

classes/swupdate.bbclass +3 -0
+                # Set file timestamps for reproducible builds
+                if [ -n "${SOURCE_DATE_EPOCH}" ]; then
+                    touch -d@"${SOURCE_DATE_EPOCH}"

"$file.${SWU_SIGNATURE_EXT}"

classes/swupdate.bbclass IMAGE_CMD:swu() {

Quoted 5 lines
classes/swupdate.bbclass +2 -1
+                fi
                 echo "$file.${SWU_SIGNATURE_EXT}"
            fi
-        done | cpio -ovL -H crc > "${SWU_BUILDCHROOT_IMAGE_FILE}"'
+        done | cpio -ovL --reproducible -H crc >

"${SWU_BUILDCHROOT_IMAGE_FILE}"'

Quoted 3 lines

}

python do_check_swu_partition_uuids() {

--
Siemens AG, Technology
Competence Center Embedded Linux