Skip to contentSkip to conversation

package/cpio: add host version

PATCH Superseded Atharva Lele 2 participants 6 messages Jul 14, 2019 – Jul 16, 2019

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/2 package/cpio: add host version

From: "Yann E. MORIN" <[email protected]>

The latest cpio has a --reproducible option, which may come handy when we try to, well, be reproducible...

[Atharva: Made change pointed out by Arnout: remove HOST_CPIO_CONF_OPTS]
Cc Arnout Vandecappelle <[email protected]>
Cc Atharva Lele <[email protected]>

---
Changes v1 -> v2:
- Add patch from Yann to my series

---
 package/cpio/cpio.mk | 1 +
 1 file changed, 1 insertion(+)
package/cpio/cpio.mk +1 -0
index 9a09279572..cc40838840 100644
@@ -17,3 +17,4 @@ CPIO_DEPENDENCIES += argp-standalone
 endif
 
 $(eval $(autotools-package))
+$(eval $(host-autotools-package))
Signature · 1 line
2.22.0
#1 lore raw

PATCH 2/2 fs/cpio: make cpio rootfs reproducible

Pass the recommended argument in the CPIO manual to make cpio archives reproducible.

Reference: https://www.gnu.org/software/cpio/manual/cpio.html#Copy_002dpass-mode

Pre-patch diffoscope output: https://gitlab.com/snippets/1874745
Post-patch: https://gitlab.com/snippets/1874746

We can see that post-patch, the archive related differences are removed. The differences are arising from utils/bin/getconf. This will have to be investigated further. However, that is unrelated to cpio.

---
Changes v1 -> v2:
- Added comment
- Build host-cpio if we are reproducible, reason is --reproducible option does not exist prior to cpio 2.12. (suggested by Yann, Arnout)

---
 fs/cpio/cpio.mk | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
fs/cpio/cpio.mk +9 -1
index 1f0cafe62f..521418f950 100644
@@ -29,8 +29,16 @@ endif # BR2_ROOTFS_DEVICE_CREATION_STATIC
 
 ROOTFS_CPIO_PRE_GEN_HOOKS += ROOTFS_CPIO_ADD_INIT
 
+# --repoducible option was not introduced until cpio v2.12. Some distros
+# like Ubuntu 16.04 or Debian oldstable which package v2.11. Hence build
+# host-cpio.
+ifeq ($(BR2_REPRODUCIBLE), y)
+ROOTFS_CPIO_DEPENDENCIES += host-cpio
+CPIO_OPTS = --reproducible
+endif
+
 define ROOTFS_CPIO_CMD
-	cd $(TARGET_DIR) && find . | cpio --quiet -o -H newc > $@
+	cd $(TARGET_DIR) && find . | cpio $(CPIO_OPTS) --quiet -o -H newc > $@
 endef
 
 ifeq ($(BR2_TARGET_ROOTFS_CPIO_UIMAGE),y)
Signature · 1 line
2.22.0
#2 lore raw

PATCH 2/2 fs/cpio: make cpio rootfs reproducible

Atharva, All,

On 2019-07-14 21:53 +0530, Atharva Lele spake thusly:

Quoted 27 lines

Pass the recommended argument in the CPIO manual to make cpio archives reproducible.

Reference: https://www.gnu.org/software/cpio/manual/cpio.html#Copy_002dpass-mode

Pre-patch diffoscope output: https://gitlab.com/snippets/1874745
Post-patch: https://gitlab.com/snippets/1874746

We can see that post-patch, the archive related differences are removed. The differences are arising from utils/bin/getconf. This will have to be investigated further. However, that is unrelated to cpio.

---
Changes v1 -> v2:
- Added comment
- Build host-cpio if we are reproducible, reason is --reproducible option does not exist prior to cpio 2.12. (suggested by Yann, Arnout)

---
 fs/cpio/cpio.mk | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
fs/cpio/cpio.mk +0 -0
index 1f0cafe62f..521418f950 100644

fs/cpio/cpio.mk endif # BR2_ROOTFS_DEVICE_CREATION_STATIC · line 37

Quoted 9 lines
fs/cpio/cpio.mk +6 -0
@@ -29,8 +29,16 @@ endif # BR2_ROOTFS_DEVICE_CREATION_STATIC
 
 ROOTFS_CPIO_PRE_GEN_HOOKS += ROOTFS_CPIO_ADD_INIT
 
+# --repoducible option was not introduced until cpio v2.12. Some distros
+# like Ubuntu 16.04 or Debian oldstable which package v2.11. Hence build
+# host-cpio.
+ifeq ($(BR2_REPRODUCIBLE), y)
+ROOTFS_CPIO_DEPENDENCIES += host-cpio
+CPIO_OPTS = --reproducible

Having a pure assignment in a condition is risky for the future. If we were to later add new unconditional options, we would add them before this part (as is customary to do: uncoditional stuff goes before conditional stuff). In this case, the assignment here would override the uncoditional options, which is not nice.

Instead, I think we prefer an append-assignment in conditional blocks, even if the variable is not already defined.

Arnout, Thomas, can you confirm/infirm my opinion? ;-)

Otherwise:

Regards,
Yann E. MORIN.

Quoted 15 lines
fs/cpio/cpio.mk +3 -1
+endif
+
 define ROOTFS_CPIO_CMD
-	cd $(TARGET_DIR) && find . | cpio --quiet -o -H newc > $@
+	cd $(TARGET_DIR) && find . | cpio $(CPIO_OPTS) --quiet -o -H newc > $@
 endef
 
 ifeq ($(BR2_TARGET_ROOTFS_CPIO_UIMAGE),y)
Signature · 6 lines
2.22.0

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#3 lore raw

PATCH 2/2 fs/cpio: make cpio rootfs reproducible

Atharva, All,

On 2019-07-14 18:47 +0200, Yann E. MORIN spake thusly:

Quoted 3 lines

On 2019-07-14 21:53 +0530, Atharva Lele spake thusly:

Quoted 2 lines

Pass the recommended argument in the CPIO manual to make cpio archives reproducible.

[--SNIP--]

Quoted 6 lines

fs/cpio/cpio.mk endif # BR2_ROOTFS_DEVICE_CREATION_STATIC · line 37

Quoted 6 lines
fs/cpio/cpio.mk +6 -0
+# --repoducible option was not introduced until cpio v2.12. Some distros
+# like Ubuntu 16.04 or Debian oldstable which package v2.11. Hence build
+# host-cpio.
+ifeq ($(BR2_REPRODUCIBLE), y)
+ROOTFS_CPIO_DEPENDENCIES += host-cpio
+CPIO_OPTS = --reproducible

In addition to my previous comments: the variable shall be prefixed with ROOTFS_CPIO_, not just with CPIO_, otherwise it could clash with variables from the cpio package itself.

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#4 lore raw

PATCH 2/2 fs/cpio: make cpio rootfs reproducible

Yann, all:

Replying to both of Yann's emails in this one:

Quoted 8 lines · IST Yann E. MORIN

Having a pure assignment in a condition is risky for the future. If we were to later add new unconditional options, we would add them before this part (as is customary to do: uncoditional stuff goes before conditional stuff). In this case, the assignment here would override the uncoditional options, which is not nice.

Instead, I think we prefer an append-assignment in conditional blocks, even if the variable is not already defined.

Ah yes that makes sense. I'll change it to an append-assignment in v2. Thanks!

Quoted 9 lines · IST Yann E. MORIN

Atharva, All,

On 2019-07-14 18:47 +0200, Yann E. MORIN spake thusly:

Quoted 3 lines

On 2019-07-14 21:53 +0530, Atharva Lele spake thusly:

Quoted 2 lines

Pass the recommended argument in the CPIO manual to make cpio archives reproducible.

[--SNIP--]

Quoted 12 lines
Quoted 8 lines

fs/cpio/cpio.mk endif # BR2_ROOTFS_DEVICE_CREATION_STATIC · line 37

Quoted 8 lines

+# --repoducible option was not introduced until cpio v2.12. Some distros
+# like Ubuntu 16.04 or Debian oldstable which package v2.11. Hence build

fs/cpio/cpio.mk +4 -0
+# host-cpio.
+ifeq ($(BR2_REPRODUCIBLE), y)
+ROOTFS_CPIO_DEPENDENCIES += host-cpio
+CPIO_OPTS = --reproducible

In addition to my previous comments: the variable shall be prefixed with ROOTFS_CPIO_, not just with CPIO_, otherwise it could clash with variables from the cpio package itself.

Agreed, fixing it in v2.

Quoted 2 lines

Regards,
Yann E. MORIN.

Thanks for the review!

Regards,
Atharva Lele

#5 lore raw

PATCH 1/2 package/cpio: add host version

In reply to Atharva Lele

Atharva, All,

On 2019-07-14 21:53 +0530, Atharva Lele spake thusly:

Quoted 8 lines

From: "Yann E. MORIN" <[email protected]>

The latest cpio has a --reproducible option, which may come handy when we try to, well, be reproducible...

[Atharva: Made change pointed out by Arnout: remove HOST_CPIO_CONF_OPTS]

Actually, this is not very intereesting, because it does not say what was in HOST_CPIO_CONF_OPTS. What we need to know is that we do not need to force --bindir. So, I'd have written:

    [Atharva: don't force --bindir, as noticed by Arnout]
Quoted 3 lines
Cc Arnout Vandecappelle <[email protected]>
Cc Atharva Lele <[email protected]>

Except for the minor nit above:

Thanks! :-)

Regards,
Yann E. MORIN.

Quoted 23 lines

---
Changes v1 -> v2:
- Add patch from Yann to my series

---
 package/cpio/cpio.mk | 1 +
 1 file changed, 1 insertion(+)
package/cpio/cpio.mk +1 -0
index 9a09279572..cc40838840 100644
@@ -17,3 +17,4 @@ CPIO_DEPENDENCIES += argp-standalone
 endif
 
 $(eval $(autotools-package))
+$(eval $(host-autotools-package))
Signature · 6 lines
2.22.0

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'