fs/tar: explicitly set extended header values to ensure binary reproducibility
PATCH v2 1/1 fs/tar: explicitly set extended header values to ensure binary reproducibility
Since we use --xattrs-include='*' to include all extended attributes, tar creates a PAX formatted archive. The archive metadata captures atime and ctime of files. To fix this, GNU recommends that we pass this added argument to tar to create binary reproducible packages. Setting of mtime is handled in fs/common.mk using touch on all files.
Diffoscope output pre-change: https://gitlab.com/snippets/1871111
Diffoscope output after change is blank i.e. binary reproducibile rootfs is created.
GNU Recommendation: https://www.gnu.org/software/tar/manual/tar.html#SEC147
---
Changes v1 -> v2:
- Added comment about mtime handling
--- fs/tar/tar.mk | 4 ++++ 1 file changed, 4 insertions(+)
fs/tar/tar.mk +4 -0
@@ -8,6 +8,10 @@ TAR_OPTS := $(call qstrip,$(BR2_TARGET_ROOTFS_TAR_OPTIONS)) ROOTFS_TAR_DEPENDENCIES = $(BR2_TAR_HOST_DEPENDENCY) +ifeq ($(BR2_REPRODUCIBLE),y) +TAR_OPTS += --pax-option=exthdr.name=%d/PaxHeaders/%f,atime:=0,ctime:=0 +endif + define ROOTFS_TAR_CMD (cd $(TARGET_DIR); find -print0 | LC_ALL=C sort -z | \ tar $(TAR_OPTS) -cf $@ --null --xattrs-include='*' --no-recursion -T - --numeric-owner)
Signature · 1 line
2.20.1
PATCH v2 1/1 fs/tar: explicitly set extended header values to ensure binary reproducibility
Atharva,
Quoted 7 lines · Atharva Lele
Since we use --xattrs-include='*' to include all extended attributes, tar creates a PAX formatted archive. The archive metadata captures atime and ctime of files. To fix this, GNU recommends that we pass this added argument to tar to create binary reproducible packages. Setting of mtime is handled in
fs/common.mkusing touch on all files.
Quoted 7 lines
Diffoscope output pre-change: https://gitlab.com/snippets/1871111
Diffoscope output after change is blank i.e. binary reproducibile rootfs is created.GNU Recommendation: https://www.gnu.org/software/tar/manual/tar.html#SEC147
Signed-off-by Atharva Lele <[email protected]>
Quoted 29 lines
---
Changes v1 -> v2:
- Added comment about mtime handling--- fs/tar/tar.mk | 4 ++++ 1 file changed, 4 insertions(+)fs/tar/tar.mk +4
-1@@ -8,6 +8,10 @@ TAR_OPTS := $(call qstrip,$(BR2_TARGET_ROOTFS_TAR_OPTIONS)) ROOTFS_TAR_DEPENDENCIES = $(BR2_TAR_HOST_DEPENDENCY) +ifeq ($(BR2_REPRODUCIBLE),y) +TAR_OPTS += --pax-option=exthdr.name=%d/PaxHeaders/%f,atime:=0,ctime:=0 +endif + define ROOTFS_TAR_CMD (cd $(TARGET_DIR); find -print0 | LC_ALL=C sort -z | \ tar $(TAR_OPTS) -cf $@ --null --xattrs-include='*' --no-recursion -T - --numeric-owner) --2.20.1
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 23 lines
Matthew Weber | Pr. Software Engineer | Commercial Avionics COLLINS AEROSPACE 400 Collins Road NE, Cedar Rapids, Iowa 52498, USA Tel: +1 319 295 7349 | FAX: +1 319 263 6099 matthew.weber at collins.com | collinsaerospace.com CONFIDENTIALITY WARNING: This message may contain proprietary and/or privileged information of Collins Aerospace and its affiliated companies. If you are not the intended recipient, please 1) Do not disclose, copy, distribute or use this message or its contents. 2) Advise the sender by return email. 3) Delete all copies (including all attachments) from your computer. Your cooperation is greatly appreciated. Any export restricted material should be shared using my matthew.weber at corp.rockwellcollins.com address.
PATCH v2 1/1 fs/tar: explicitly set extended header values to ensure binary reproducibility
In reply to Atharva LeleQuoted 27 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Atharva" == Atharva Lele <[email protected]> writes:
Since we use --xattrs-include='*' to include all extended attributes, tar creates a PAX formatted archive. The archive metadata captures atime and ctime of files. To fix this, GNU recommends that we pass this added argument to tar to create binary reproducible packages. Setting of mtime is handled in
fs/common.mkusing touch on all files.Diffoscope output pre-change: https://gitlab.com/snippets/1871111
Diffoscope output after change is blank i.e. binary reproducibile rootfs is created.GNU Recommendation: https://www.gnu.org/software/tar/manual/tar.html#SEC147
Signed-off-by Atharva Lele <[email protected]>---
Changes v1 -> v2:
- Added comment about mtime handling--- fs/tar/tar.mk | 4 ++++ 1 file changed, 4 insertions(+)fs/tar/tar.mk +0
-0
fs/tar/tar.mk TAR_OPTS := $(call qstrip,$(BR2_TARGET_ROOTFS_TAR_OPTIONS)) · line 13
Quoted 6 lines
fs/tar/tar.mk +3
-0@@ -8,6 +8,10 @@ TAR_OPTS := $(call qstrip,$(BR2_TARGET_ROOTFS_TAR_OPTIONS)) ROOTFS_TAR_DEPENDENCIES = $(BR2_TAR_HOST_DEPENDENCY) +ifeq ($(BR2_REPRODUCIBLE),y) +TAR_OPTS += --pax-option=exthdr.name=%d/PaxHeaders/%f,atime:=0,ctime:=0 +endif
It would be good to add a short comment saying something like
# don't store atime/ctime
Is there any use case for ever storing these in the rootfs, or should we just do this fix unconditionally?
Signature · 1 line
Bye, Peter Korsgaard
PATCH v2 1/1 fs/tar: explicitly set extended header values to ensure binary reproducibility
Quoted 27 lines · IST Peter Korsgaard
Quoted 27 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Atharva" == Atharva Lele <[email protected]> writes:
Since we use --xattrs-include='*' to include all extended attributes, tar creates a PAX formatted archive. The archive metadata captures atime and ctime of files. To fix this, GNU recommends that we pass this added argument to tar to create binary reproducible packages. Setting of mtime is handled in
fs/common.mkusing touch on all files.Diffoscope output pre-change: https://gitlab.com/snippets/1871111
Diffoscope output after change is blank i.e. binary reproducibile rootfs is created.GNU Recommendation:
https://www.gnu.org/software/tar/manual/tar.html#SEC147Signed-off-by Atharva Lele <[email protected]>fs/tar/tar.mk | 4 ++++ 1 file changed, 4 insertions(+)fs/tar/tar.mk +0
-0@@ -8,6 +8,10 @@ TAR_OPTS := $(callqstrip,$(BR2_TARGET_ROOTFS_TAR_OPTIONS)) >
ROOTFS_TAR_DEPENDENCIES= $(BR2_TAR_HOST_DEPENDENCY)
Quoted 8 lines
fs/tar/tar.mk TAR_OPTS := $(call qstrip,$(BR2_TARGET_ROOTFS_TAR_OPTIONS)) · line 13
Quoted 3 lines
fs/tar/tar.mk +3
-0+ifeq ($(BR2_REPRODUCIBLE),y) +TAR_OPTS += --pax-option=exthdr.name=%d/PaxHeaders/%f,atime:=0,ctime:=0 +endifIt would be good to add a short comment saying something like
# don't store atime/ctime
Definitely. Will do in a v3.
Quoted 2 lines
Is there any use case for ever storing these in the rootfs, or should we just do this fix unconditionally?
Personally, I can't think of a reason why we'd want to store atime/ctime in the rootfs. I think they got included when we added --xattrs-include='*'
Arnout, Matt and Yann: any thoughts on that?
Signature · 2 lines
Regards, Atharva Lele
PATCH v2 1/1 fs/tar: explicitly set extended header values to ensure binary reproducibility
Quoted 1 line · Atharva Lele
On Thursday, July 4, 2019 11:58:21 AM IST Peter Korsgaard wrote:
[snip]
Quoted 5 lines
Quoted 2 lines
Is there any use case for ever storing these in the rootfs, or should we just do this fix unconditionally?
Personally, I can't think of a reason why we'd want to store atime/ctime in the rootfs.
ctime is pretty useless in a tarball because when extracting the ctime will be
set to extraction time anyway (tested with -xpf as root).
There might be some use for atime, but atime is already set by the "touch"
command that preceeds the tarball creation, so it doesn't make much of a difference.
So yes, we probably don't need to make this depend on BR2_REPRODUCIBLE.
Regards,
Arnout
Quoted 4 lines
I think they got included when we added --xattrs-include='*'
Arnout, Matt and Yann: any thoughts on that?
PATCH v2 1/1 fs/tar: explicitly set extended header values to ensure binary reproducibility
Quoted 7 lines · IST Arnout Vandecappelle
Quoted 1 line · Atharva Lele
On Thursday, July 4, 2019 11:58:21 AM IST Peter Korsgaard wrote:
[snip]
Quoted 3 lines
Quoted 2 lines
Is there any use case for ever storing these in the rootfs, or should we just do this fix unconditionally?
Quoted 6 lines
Quoted 3 lines
Personally, I can't think of a reason why we'd want to store atime/ctime in
the rootfs.ctime is pretty useless in a tarball because when extracting the ctime will
be set to extraction time anyway (tested with -xpf as root).
Makes sense. Thanks for the explanation.
Quoted 3 lines
There might be some use for atime, but atime is already set by the "touch"
command that preceeds the tarball creation, so it doesn't make much of a difference.
Fair enough. One question, does atime change again when we access the file when tar-ing it?
Quoted 1 line
So yes, we probably don't need to make this depend on
BR2_REPRODUCIBLE.
I agree.
Signature · 2 lines
Regards, Atharva Lele