Skip to contentSkip to conversation

classes/kernel: Generate reproducible kernel module tarball

PATCH Superseded Joshua Watt 2 participants 5 messages May 12, 2020

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 classes/kernel: Generate reproducible kernel module tarball

If reproducible binaries are requested, generate the kernel modules tarball in a reproducible way. Namely, sort the tarball contents, clamp the mtime to the source date epoch, compress the archive with a consistent compression level (9), and don't include the name and timestamp in the compressed file.

[YOCTO #12719]

---
 meta/classes/kernel.bbclass | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
meta/classes/kernel.bbclass +9 -1
index a724645466..c124645643 100644
@@ -691,7 +691,15 @@ kernel_do_deploy() {
 
 	if [ ${MODULE_TARBALL_DEPLOY} = "1" ] && (grep -q -i -e '^CONFIG_MODULES=y$' .config); then
 		mkdir -p ${D}${root_prefix}/lib
-		tar -cvzf $deployDir/modules-${MODULE_TARBALL_NAME}.tgz -C ${D}${root_prefix} lib
+		if [ "${BUILD_REPRODUCIBLE_BINARIES}" = "1" ]; then
+			TAR_ARGS="--sort=name --clamp-mtime --mtime=@${SOURCE_DATE_EPOCH}"
+			GZIP_ARGS="-9n"
+		else
+			TAR_ARGS=""
+			GZIP_ARGS=""
+		fi
+		tar $TAR_ARGS -cv -C ${D}${root_prefix} lib | gzip $GZIP_ARGS > $deployDir/modules-${MODULE_TARBALL_NAME}.tgz
+
 		ln -sf modules-${MODULE_TARBALL_NAME}.tgz $deployDir/modules-${MODULE_TARBALL_LINK_NAME}.tgz
 	fi
 
Signature · 1 line
2.17.1
#1 lore raw
Quoted 23 lines · Joshua Watt

If reproducible binaries are requested, generate the kernel modules tarball in a reproducible way. Namely, sort the tarball contents, clamp the mtime to the source date epoch, compress the archive with a consistent compression level (9), and don't include the name and timestamp in the compressed file.

[YOCTO #12719]

---
 meta/classes/kernel.bbclass | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
meta/classes/kernel.bbclass +1 -1
index a724645466..c124645643 100644
@@ -691,7 +691,15 @@ kernel_do_deploy() {
 
 	if [ ${MODULE_TARBALL_DEPLOY} = "1" ] && (grep -q -i -e '^CONFIG_MODULES=y$' .config); then
 		mkdir -p ${D}${root_prefix}/lib
-		tar -cvzf $deployDir/modules-${MODULE_TARBALL_NAME}.tgz -C ${D}${root_prefix} lib
+		if [ "${BUILD_REPRODUCIBLE_BINARIES}" = "1" ]; then

meta/classes/kernel.bbclass kernel_do_deploy() { · line 704

Quoted 10 lines
meta/classes/kernel.bbclass +8 -0
+			TAR_ARGS="--sort=name --clamp-mtime --mtime=@${SOURCE_DATE_EPOCH}"
+			GZIP_ARGS="-9n"
+		else
+			TAR_ARGS=""
+			GZIP_ARGS=""
+		fi
+		tar $TAR_ARGS -cv -C ${D}${root_prefix} lib | gzip $GZIP_ARGS > $deployDir/modules-${MODULE_TARBALL_NAME}.tgz
+
 		ln -sf modules-${MODULE_TARBALL_NAME}.tgz $deployDir/modules-${MODULE_TARBALL_LINK_NAME}.tgz
 	fi

Do we need to make this conditional or could we just do it by default?

Cheers,

Richard

#2 lore raw
Quoted 20 lines · Richard Purdie
Quoted 19 lines · Joshua Watt

If reproducible binaries are requested, generate the kernel modules tarball in a reproducible way. Namely, sort the tarball contents, clamp the mtime to the source date epoch, compress the archive with a consistent compression level (9), and don't include the name and timestamp in the compressed file.

[YOCTO #12719]

---
  meta/classes/kernel.bbclass | 10 +++++++++-
  1 file changed, 9 insertions(+), 1 deletion(-)
meta/classes/kernel.bbclass +0 -0
index a724645466..c124645643 100644
@@ -691,7 +691,15 @@ kernel_do_deploy() {
 
Quoted 15 lines
Quoted 4 lines
meta/classes/kernel.bbclass +1 -1
  	if [ ${MODULE_TARBALL_DEPLOY} = "1" ] && (grep -q -i -e '^CONFIG_MODULES=y$' .config); then
  		mkdir -p ${D}${root_prefix}/lib
-		tar -cvzf $deployDir/modules-${MODULE_TARBALL_NAME}.tgz -C ${D}${root_prefix} lib
+		if [ "${BUILD_REPRODUCIBLE_BINARIES}" = "1" ]; then

meta/classes/kernel.bbclass kernel_do_deploy() { · line 704

Quoted 10 lines
meta/classes/kernel.bbclass +8 -0
+			TAR_ARGS="--sort=name --clamp-mtime --mtime=@${SOURCE_DATE_EPOCH}"
+			GZIP_ARGS="-9n"
+		else
+			TAR_ARGS=""
+			GZIP_ARGS=""
+		fi
+		tar $TAR_ARGS -cv -C ${D}${root_prefix} lib | gzip $GZIP_ARGS > $deployDir/modules-${MODULE_TARBALL_NAME}.tgz
+
  		ln -sf modules-${MODULE_TARBALL_NAME}.tgz $deployDir/modules-${MODULE_TARBALL_LINK_NAME}.tgz
  	fi

Do we need to make this conditional or could we just do it by default?

I don't know if we can rely on SOURCE_DATE_EPOCH being set unless reproducible_build.bbclass is inherited, but at a minimum we can always use the gzip args.

Quoted 5 lines

Cheers,

Richard

#3 lore raw
Quoted 37 lines · Joshua Watt
Quoted 20 lines · Richard Purdie
Quoted 19 lines · Joshua Watt

If reproducible binaries are requested, generate the kernel modules tarball in a reproducible way. Namely, sort the tarball contents, clamp the mtime to the source date epoch, compress the archive with a consistent compression level (9), and don't include the name and timestamp in the compressed file.

[YOCTO #12719]

---
  meta/classes/kernel.bbclass | 10 +++++++++-
  1 file changed, 9 insertions(+), 1 deletion(-)
meta/classes/kernel.bbclass +0 -0
index a724645466..c124645643 100644
@@ -691,7 +691,15 @@ kernel_do_deploy() {
 
Quoted 15 lines
Quoted 4 lines
meta/classes/kernel.bbclass +1 -1
  	if [ ${MODULE_TARBALL_DEPLOY} = "1" ] && (grep -q -i -e '^CONFIG_MODULES=y$' .config); then
  		mkdir -p ${D}${root_prefix}/lib
-		tar -cvzf $deployDir/modules-${MODULE_TARBALL_NAME}.tgz -C ${D}${root_prefix} lib
+		if [ "${BUILD_REPRODUCIBLE_BINARIES}" = "1" ]; then

meta/classes/kernel.bbclass kernel_do_deploy() { · line 704

Quoted 10 lines
meta/classes/kernel.bbclass +8 -0
+			TAR_ARGS="--sort=name --clamp-mtime --mtime=@${SOURCE_DATE_EPOCH}"
+			GZIP_ARGS="-9n"
+		else
+			TAR_ARGS=""
+			GZIP_ARGS=""
+		fi
+		tar $TAR_ARGS -cv -C ${D}${root_prefix} lib | gzip $GZIP_ARGS > $deployDir/modules-${MODULE_TARBALL_NAME}.tgz
+
  		ln -sf modules-${MODULE_TARBALL_NAME}.tgz $deployDir/modules-${MODULE_TARBALL_LINK_NAME}.tgz
  	fi

Do we need to make this conditional or could we just do it by default?

Quoted 3 lines

I don't know if we can rely on SOURCE_DATE_EPOCH being set unless reproducible_build.bbclass is inherited, but at a minimum we can always use the gzip args.

Good point, I assumed we had a default for some reason.

I think the patch will be more obvious if its conditional on SOURCE_DATE_EPOCH and we just use the other options unconditionally.

I wonder if we should just move build_reproducibility_simple.bbclass into bitbake.conf?

Cheers,

Richard

#4 lore raw

PATCH v2 1/1 classes/kernel: Generate reproducible kernel module tarball

In reply to Joshua Watt

If reproducible binaries are requested, generate the kernel modules tarball in a reproducible way. Namely, sort the tarball contents, clamp the mtime to the source date epoch, compress the archive with a consistent compression level (9), and don't include the name and timestamp in the compressed file.

[YOCTO #12719]

---
 meta/classes/kernel.bbclass | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
meta/classes/kernel.bbclass +7 -1
index a724645466..6846d6040d 100644
@@ -691,7 +691,13 @@ kernel_do_deploy() {
 
 	if [ ${MODULE_TARBALL_DEPLOY} = "1" ] && (grep -q -i -e '^CONFIG_MODULES=y$' .config); then
 		mkdir -p ${D}${root_prefix}/lib
-		tar -cvzf $deployDir/modules-${MODULE_TARBALL_NAME}.tgz -C ${D}${root_prefix} lib
+		if [ -n "${SOURCE_DATE_EPOCH}" ]; then
+			TAR_ARGS="--sort=name --clamp-mtime --mtime=@${SOURCE_DATE_EPOCH}"
+		else
+			TAR_ARGS=""
+		fi
+		tar $TAR_ARGS -cv -C ${D}${root_prefix} lib | gzip -9n > $deployDir/modules-${MODULE_TARBALL_NAME}.tgz
+
 		ln -sf modules-${MODULE_TARBALL_NAME}.tgz $deployDir/modules-${MODULE_TARBALL_LINK_NAME}.tgz
 	fi
 
Signature · 1 line
2.17.1