Skip to contentSkip to conversation

fs/cpio: sort 'find' utility output (reproducible builds)

PATCH New Yurii Monakov 3 participants 4 messages Jun 29, 2020 – Jul 16, 2020

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 fs/cpio: sort 'find' utility output (reproducible builds)

Order of find utility results are not sorted and depend on items attributes, so cpio produces different results across machines.

This patch sorts find output before piping to cpio to make builds reproducible.

---
 fs/cpio/cpio.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/cpio/cpio.mk +1 -1
index 28a435dd5e..73837ddecb 100644
@@ -37,7 +37,7 @@ ROOTFS_CPIO_OPTS += --reproducible
 endif
 
 define ROOTFS_CPIO_CMD
-	cd $(TARGET_DIR) && find . | cpio $(ROOTFS_CPIO_OPTS) --quiet -o -H newc > $@
+	cd $(TARGET_DIR) && find . | sort | cpio $(ROOTFS_CPIO_OPTS) --quiet -o -H newc > $@
 endef
 
 ifeq ($(BR2_TARGET_ROOTFS_CPIO_UIMAGE),y)
Signature · 1 line
2.17.1
#1 lore raw

PATCH 1/1 fs/cpio: sort 'find' utility output (reproducible builds)

Yurii, All,

On 2020-06-29 15:37 +0300, Yurii Monakov spake thusly:

Quoted 7 lines

Order of find utility results are not sorted and depend on items attributes, so cpio produces different results across machines.

This patch sorts find output before piping to cpio to make builds reproducible.

Thanks for the ptatch, but see below...

Quoted 9 lines
---
 fs/cpio/cpio.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/cpio/cpio.mk +0 -0
index 28a435dd5e..73837ddecb 100644

fs/cpio/cpio.mk ROOTFS_CPIO_OPTS += --reproducible · line 40

Quoted 5 lines
fs/cpio/cpio.mk +1 -1
@@ -37,7 +37,7 @@ ROOTFS_CPIO_OPTS += --reproducible
 endif
 
 define ROOTFS_CPIO_CMD
-	cd $(TARGET_DIR) && find . | cpio $(ROOTFS_CPIO_OPTS) --quiet -o -H newc > $@
+	cd $(TARGET_DIR) && find . | sort | cpio $(ROOTFS_CPIO_OPTS) --quiet -o -H newc > $@

sort order depends on the locale, so you also have to enforce the locale to guarantee reproducibility; this is usually done with:

    ... |LC_ALL=C sort |...

So I've fixed that, wrapped the line that is now too long, and reworded the commit log slightly.

Applied to master, thanks.

Regards,
Yann E. MORIN.

Quoted 10 lines
fs/cpio/cpio.mk +0 -0
 endef
 
 ifeq ($(BR2_TARGET_ROOTFS_CPIO_UIMAGE),y)
Signature · 6 lines
2.17.1

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#2 lore raw

PATCH 1/1 fs/cpio: sort 'find' utility output (reproducible builds)

Yann,

Quoted 2 lines

So I've fixed that, wrapped the line that is now too long, and reworded the commit log slightly.

Thanks for fixing my patch. I'm pretty new to reproducibility problems, so I was not aware of locale issues.

Best Regards,
Yurii Monakov

#3 lore raw

PATCH 1/1 fs/cpio: sort 'find' utility output (reproducible builds)

In reply to Yurii Monakov
Quoted 9 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Yurii" == Yurii Monakov <[email protected]> writes:

Order of find utility results are not sorted and depend on items attributes, so cpio produces different results across machines.

This patch sorts find output before piping to cpio to make builds reproducible.

Committed to 2020.02.x and 2020.05.x, thanks.

Signature · 1 line
Bye, Peter Korsgaard