Skip to contentSkip to conversation

fs/tar: explicitly set extended header values to ensure binary reproducibility

PATCH Superseded Atharva Lele 4 participants 6 messages Jul 2, 2019 – Jul 5, 2019

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 fs/tar: explicitly set extended header values to ensure binary reproducibility

Since we use --xattrs-include='*' to include all extended attributes, tar creates a PAX formatted archive. The archive metadata captures atime and ctime of files. To fix this, GNU recommends that we pass this added argument to tar to create binary reproducible packages. Setting of mtime is handled in fs/common.mk using touch on all files.

Diffoscope output pre-change: https://gitlab.com/snippets/1871111
Diffoscope output after change is blank i.e. binary reproducibile rootfs is created.

GNU Recommendation: https://www.gnu.org/software/tar/manual/tar.html#SEC147

---
Changes v1 -> v2:
- Added comment about mtime handling

---
 fs/tar/tar.mk | 4 ++++
 1 file changed, 4 insertions(+)
fs/tar/tar.mk +4 -0
index 4c6327ace8..f6d11baa84 100644
@@ -8,6 +8,10 @@ TAR_OPTS := $(call qstrip,$(BR2_TARGET_ROOTFS_TAR_OPTIONS))
 
 ROOTFS_TAR_DEPENDENCIES = $(BR2_TAR_HOST_DEPENDENCY)
 
+ifeq ($(BR2_REPRODUCIBLE),y)
+TAR_OPTS += --pax-option=exthdr.name=%d/PaxHeaders/%f,atime:=0,ctime:=0
+endif
+
 define ROOTFS_TAR_CMD
 	(cd $(TARGET_DIR); find -print0 | LC_ALL=C sort -z | \
 		tar $(TAR_OPTS) -cf $@ --null --xattrs-include='*' --no-recursion -T - --numeric-owner)
Signature · 1 line
2.20.1
#1 lore raw

PATCH v2 1/1 fs/tar: explicitly set extended header values to ensure binary reproducibility

Atharva,

Quoted 7 lines · Atharva Lele

Since we use --xattrs-include='*' to include all extended attributes, tar creates a PAX formatted archive. The archive metadata captures atime and ctime of files. To fix this, GNU recommends that we pass this added argument to tar to create binary reproducible packages. Setting of mtime is handled in fs/common.mk using touch on all files.

Quoted 7 lines

Diffoscope output pre-change: https://gitlab.com/snippets/1871111
Diffoscope output after change is blank i.e. binary reproducibile rootfs is created.

GNU Recommendation: https://www.gnu.org/software/tar/manual/tar.html#SEC147

Quoted 29 lines

---
Changes v1 -> v2:
- Added comment about mtime handling

---
 fs/tar/tar.mk | 4 ++++
 1 file changed, 4 insertions(+)
fs/tar/tar.mk +4 -1
index 4c6327ace8..f6d11baa84 100644
@@ -8,6 +8,10 @@ TAR_OPTS := $(call qstrip,$(BR2_TARGET_ROOTFS_TAR_OPTIONS))
 
 ROOTFS_TAR_DEPENDENCIES = $(BR2_TAR_HOST_DEPENDENCY)
 
+ifeq ($(BR2_REPRODUCIBLE),y)
+TAR_OPTS += --pax-option=exthdr.name=%d/PaxHeaders/%f,atime:=0,ctime:=0
+endif
+
 define ROOTFS_TAR_CMD
        (cd $(TARGET_DIR); find -print0 | LC_ALL=C sort -z | \
                tar $(TAR_OPTS) -cf $@ --null --xattrs-include='*' --no-recursion -T - --numeric-owner)
--

2.20.1

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Signature · 23 lines
Matthew Weber | Pr. Software Engineer | Commercial Avionics

COLLINS AEROSPACE

400 Collins Road NE, Cedar Rapids, Iowa 52498, USA

Tel: +1 319 295 7349 | FAX: +1 319 263 6099

matthew.weber at collins.com | collinsaerospace.com



CONFIDENTIALITY WARNING: This message may contain proprietary and/or
privileged information of Collins Aerospace and its affiliated
companies. If you are not the intended recipient, please 1) Do not
disclose, copy, distribute or use this message or its contents. 2)
Advise the sender by return email. 3) Delete all copies (including all
attachments) from your computer. Your cooperation is greatly
appreciated.


Any export restricted material should be shared using my
matthew.weber at corp.rockwellcollins.com address.
#2 lore raw

PATCH v2 1/1 fs/tar: explicitly set extended header values to ensure binary reproducibility

In reply to Atharva Lele
Quoted 27 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Atharva" == Atharva Lele <[email protected]> writes:

Since we use --xattrs-include='*' to include all extended attributes, tar creates a PAX formatted archive. The archive metadata captures atime and ctime of files. To fix this, GNU recommends that we pass this added argument to tar to create binary reproducible packages. Setting of mtime is handled in fs/common.mk using touch on all files.

Diffoscope output pre-change: https://gitlab.com/snippets/1871111
Diffoscope output after change is blank i.e. binary reproducibile rootfs is created.

GNU Recommendation: https://www.gnu.org/software/tar/manual/tar.html#SEC147

---
Changes v1 -> v2:
- Added comment about mtime handling

---
 fs/tar/tar.mk | 4 ++++
 1 file changed, 4 insertions(+)
fs/tar/tar.mk +0 -0
index 4c6327ace8..f6d11baa84 100644

fs/tar/tar.mk TAR_OPTS := $(call qstrip,$(BR2_TARGET_ROOTFS_TAR_OPTIONS)) · line 13

Quoted 6 lines
fs/tar/tar.mk +3 -0
@@ -8,6 +8,10 @@ TAR_OPTS := $(call qstrip,$(BR2_TARGET_ROOTFS_TAR_OPTIONS))
 
 ROOTFS_TAR_DEPENDENCIES = $(BR2_TAR_HOST_DEPENDENCY)
 
+ifeq ($(BR2_REPRODUCIBLE),y)
+TAR_OPTS += --pax-option=exthdr.name=%d/PaxHeaders/%f,atime:=0,ctime:=0
+endif

It would be good to add a short comment saying something like

# don't store atime/ctime

Is there any use case for ever storing these in the rootfs, or should we just do this fix unconditionally?

Signature · 1 line
Bye, Peter Korsgaard
#3 lore raw

PATCH v2 1/1 fs/tar: explicitly set extended header values to ensure binary reproducibility

Quoted 27 lines · IST Peter Korsgaard
Quoted 27 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Atharva" == Atharva Lele <[email protected]> writes:

Since we use --xattrs-include='*' to include all extended attributes, tar creates a PAX formatted archive. The archive metadata captures atime and ctime of files. To fix this, GNU recommends that we pass this added argument to tar to create binary reproducible packages. Setting of mtime is handled in fs/common.mk using touch on all files.

Diffoscope output pre-change: https://gitlab.com/snippets/1871111
Diffoscope output after change is blank i.e. binary reproducibile rootfs is created.

GNU Recommendation:
https://www.gnu.org/software/tar/manual/tar.html#SEC147

 fs/tar/tar.mk | 4 ++++
 1 file changed, 4 insertions(+)
fs/tar/tar.mk +0 -0
index 4c6327ace8..f6d11baa84 100644
@@ -8,6 +8,10 @@ TAR_OPTS := $(call

qstrip,$(BR2_TARGET_ROOTFS_TAR_OPTIONS)) >
ROOTFS_TAR_DEPENDENCIES = $(BR2_TAR_HOST_DEPENDENCY)

Quoted 8 lines

fs/tar/tar.mk TAR_OPTS := $(call qstrip,$(BR2_TARGET_ROOTFS_TAR_OPTIONS)) · line 13

Quoted 3 lines
fs/tar/tar.mk +3 -0
+ifeq ($(BR2_REPRODUCIBLE),y)
+TAR_OPTS += --pax-option=exthdr.name=%d/PaxHeaders/%f,atime:=0,ctime:=0
+endif

It would be good to add a short comment saying something like

# don't store atime/ctime

Definitely. Will do in a v3.

Quoted 2 lines

Is there any use case for ever storing these in the rootfs, or should we just do this fix unconditionally?

Personally, I can't think of a reason why we'd want to store atime/ctime in the rootfs. I think they got included when we added --xattrs-include='*'

Arnout, Matt and Yann: any thoughts on that?

Signature · 2 lines
Regards,
Atharva Lele
#4 lore raw

PATCH v2 1/1 fs/tar: explicitly set extended header values to ensure binary reproducibility

Quoted 1 line · Atharva Lele

On Thursday, July 4, 2019 11:58:21 AM IST Peter Korsgaard wrote:

[snip]

Quoted 5 lines
Quoted 2 lines

Is there any use case for ever storing these in the rootfs, or should we just do this fix unconditionally?

Personally, I can't think of a reason why we'd want to store atime/ctime in the rootfs.

ctime is pretty useless in a tarball because when extracting the ctime will be
set to extraction time anyway (tested with -xpf as root).

There might be some use for atime, but atime is already set by the "touch"
command that preceeds the tarball creation, so it doesn't make much of a difference.

So yes, we probably don't need to make this depend on BR2_REPRODUCIBLE.

Regards,
Arnout

Quoted 4 lines

I think they got included when we added --xattrs-include='*'

Arnout, Matt and Yann: any thoughts on that?

#5 lore raw

PATCH v2 1/1 fs/tar: explicitly set extended header values to ensure binary reproducibility

Quoted 7 lines · IST Arnout Vandecappelle
Quoted 1 line · Atharva Lele

On Thursday, July 4, 2019 11:58:21 AM IST Peter Korsgaard wrote:

[snip]

Quoted 3 lines
Quoted 2 lines

Is there any use case for ever storing these in the rootfs, or should we just do this fix unconditionally?

Quoted 6 lines
Quoted 3 lines

Personally, I can't think of a reason why we'd want to store atime/ctime in
the rootfs.

ctime is pretty useless in a tarball because when extracting the ctime will
be set to extraction time anyway (tested with -xpf as root).

Makes sense. Thanks for the explanation.

Quoted 3 lines

There might be some use for atime, but atime is already set by the "touch"
command that preceeds the tarball creation, so it doesn't make much of a difference.

Fair enough. One question, does atime change again when we access the file when tar-ing it?

Quoted 1 line

So yes, we probably don't need to make this depend on BR2_REPRODUCIBLE.

I agree.

Signature · 2 lines
Regards,
Atharva Lele