Skip to contentSkip to conversation

fs/cpio: make cpio rootfs reproducible

PATCH New Atharva Lele 3 participants 3 messages Jul 12, 2019 – Jul 14, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 fs/cpio: make cpio rootfs reproducible

Pass the recommended argument in the CPIO manual to make cpio archives reproducible.

Reference: https://www.gnu.org/software/cpio/manual/cpio.html#Copy_002dpass-mode

Pre-patch diffoscope output: https://gitlab.com/snippets/1874745
Post-patch: https://gitlab.com/snippets/1874746

We can see that post-patch, the archive related differences are removed. The differences are arising from utils/bin/getconf. This will have to be investigated further. However, that is unrelated to cpio.

---
 fs/cpio/cpio.mk | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
fs/cpio/cpio.mk +5 -1
index 1f0cafe62f..4847f56d53 100644
@@ -29,8 +29,12 @@ endif # BR2_ROOTFS_DEVICE_CREATION_STATIC
 
 ROOTFS_CPIO_PRE_GEN_HOOKS += ROOTFS_CPIO_ADD_INIT
 
+ifeq ($(BR2_REPRODUCIBLE), y)
+CPIO_OPTS = --reproducible
+endif
+
 define ROOTFS_CPIO_CMD
-	cd $(TARGET_DIR) && find . | cpio --quiet -o -H newc > $@
+	cd $(TARGET_DIR) && find . | cpio $(CPIO_OPTS) --quiet -o -H newc > $@
 endef
 
 ifeq ($(BR2_TARGET_ROOTFS_CPIO_UIMAGE),y)
Signature · 1 line
2.22.0
#1 lore raw

PATCH 1/1 fs/cpio: make cpio rootfs reproducible

Hi Atharva,

Quoted 22 lines · Atharva Lele

Pass the recommended argument in the CPIO manual to make cpio archives reproducible.

Reference: https://www.gnu.org/software/cpio/manual/cpio.html#Copy_002dpass-mode

Pre-patch diffoscope output: https://gitlab.com/snippets/1874745
Post-patch: https://gitlab.com/snippets/1874746

We can see that post-patch, the archive related differences are removed. The differences are arising from utils/bin/getconf. This will have to be investigated further. However, that is unrelated to cpio.

---
 fs/cpio/cpio.mk | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
fs/cpio/cpio.mk +0 -0
index 1f0cafe62f..4847f56d53 100644

fs/cpio/cpio.mk endif # BR2_ROOTFS_DEVICE_CREATION_STATIC · line 33

Quoted 5 lines
fs/cpio/cpio.mk +2 -0
@@ -29,8 +29,12 @@ endif # BR2_ROOTFS_DEVICE_CREATION_STATIC
 
 ROOTFS_CPIO_PRE_GEN_HOOKS += ROOTFS_CPIO_ADD_INIT
 
+ifeq ($(BR2_REPRODUCIBLE), y)
+CPIO_OPTS = --reproducible

Unfortunately, this option was only introduced in cpio 2.12 (2015). Older
distros, including Debian oldstable and Ubuntu 2016.04, both of which are still being maintained, have an earlier cpio version (2.11) so don't have this option.

Therefore, I think we have to check for a suitable system cpio and build
host-cpio if it is not suitable. Or maybe, for reproducible, always build host-cpio?

Yann, what do you think?

Otherwise, the patch looks good. But I've marked it as Changes Requested in
pathchwork.

Regards,
Arnout

Quoted 9 lines
fs/cpio/cpio.mk +3 -1
+endif
+
 define ROOTFS_CPIO_CMD
-	cd $(TARGET_DIR) && find . | cpio --quiet -o -H newc > $@
+	cd $(TARGET_DIR) && find . | cpio $(CPIO_OPTS) --quiet -o -H newc > $@
 endef
 
 ifeq ($(BR2_TARGET_ROOTFS_CPIO_UIMAGE),y)
#2 lore raw

PATCH 1/1 fs/cpio: make cpio rootfs reproducible

Arnout, Atharva, All,

On 2019-07-14 00:34 +0200, Arnout Vandecappelle spake thusly:

Quoted 3 lines · Atharva Lele
Quoted 2 lines

Pass the recommended argument in the CPIO manual to make cpio archives reproducible.

[--SNIP--]

Quoted 10 lines

fs/cpio/cpio.mk endif # BR2_ROOTFS_DEVICE_CREATION_STATIC · line 33

Quoted 2 lines
fs/cpio/cpio.mk +2 -0
+ifeq ($(BR2_REPRODUCIBLE), y)
+CPIO_OPTS = --reproducible

Unfortunately, this option was only introduced in cpio 2.12 (2015). Older
distros, including Debian oldstable and Ubuntu 2016.04, both of which are still being maintained, have an earlier cpio version (2.11) so don't have this option.

Therefore, I think we have to check for a suitable system cpio and build
host-cpio if it is not suitable. Or maybe, for reproducible, always build host-cpio?

If we are reproducible, then always build our cpio version.

We currently do not have a host-cpio package, but it is relatively trivial to add (patch pending). Yet, I tried and it adds about 25s to the build, somewhat comparable to the overhead for host-tar...

If someone gets sufficiently annoyed by the overhead, they'll provide a patch to switch to checking for a suitable cpio.

But in any case, I think that people who do care about reproducibility, do care more about it than they do about time.

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'