support building from read-only source tree

PATCH New Nathan Lynch 3 participants 10 messages Jun 18, 2012 – May 30, 2013 Kbuild

Share & tools

Nathan Lynch

author

2012-06-18 15:02 UTC #0 lore raw

cover support building from read-only source tree

There are several places in Buildroot where files are copied from the source tree to the output directory and then modified. The most obvious case is copying a .config or equivalent to the build directory and then modifying it with sed, as with uClibc or busybox.

This works fine when the source file has write permissions; this is preserved in the destination copy. This is the behavior in the common case of running from a git repository or a release tarball.

However, when the source file is read-only, the destination copy is also not writable and such fixups fail. This occurs when the the Buildroot source tree is being hosted in a Perforce repository. Perforce uses local file permissions to track whether a file is being edited and checks out all regular files read-only by design.

The following changes were enough to get qemu_mips_malta_defconfig to build and run successfully. I submit these patches for discussion and illustration of the effort required to support this use case.

Nathan Lynch (4):
uClibc: ensure .oldconfig is writable ensure target fs is writable
busybox: ensure $(BUSYBOX_BUILD_CONFIG) is writable
linux: ensure buildroot_defconfig is writable

 Makefile                   |    1 +
 linux/linux.mk             |    3 ++-
 package/busybox/busybox.mk |    3 ++-
 toolchain/uClibc/uclibc.mk |    1 +
 4 files changed, 6 insertions(+), 2 deletions(-)

Signature · 1 line

1.7.10.2

Nathan Lynch

author

2012-06-18 15:02 UTC #1 lore raw

PATCH 1/4 uClibc: ensure .oldconfig is writable

If $(UCLIBC_CONFIG_FILE) is read-only, the sed fixups to .oldconfig fail.

Signed-off-by Nathan Lynch <[email protected]>

---
 toolchain/uClibc/uclibc.mk |    1 +
 1 file changed, 1 insertion(+)

toolchain/uClibc/uclibc.mk +1 -0

index 0728135..f41e6b9 100644
@@ -101,6 +101,7 @@ endif
 # Some targets may wish to provide their own UCLIBC_CONFIG_FILE...
 $(UCLIBC_DIR)/.oldconfig: $(UCLIBC_DIR)/.patched $(UCLIBC_CONFIG_FILE)
 	cp -f $(UCLIBC_CONFIG_FILE) $(UCLIBC_DIR)/.oldconfig
+	chmod u+w $(UCLIBC_DIR)/.oldconfig
 	$(SED) 's,^CROSS_COMPILER_PREFIX=.*,CROSS_COMPILER_PREFIX="$(TARGET_CROSS)",g' \
 		-e 's,# TARGET_$(UCLIBC_TARGET_ARCH) is not set,TARGET_$(UCLIBC_TARGET_ARCH)=y,g' \
 		-e 's,^TARGET_ARCH=".*",TARGET_ARCH=\"$(UCLIBC_TARGET_ARCH)\",g' \

Signature · 1 line

1.7.10.2

Nathan Lynch

author

2012-06-18 15:02 UTC #2 lore raw

PATCH 2/4 ensure target fs is writable

In reply to Nathan Lynch

If the source target skeleton is read-only, modifications to the output target (such as creating /etc/hostname and /etc/issue) fail.

Signed-off-by Nathan Lynch <[email protected]>

---
 Makefile |    1 +
 1 file changed, 1 insertion(+)

Makefile +1 -0

index 9a9634d..38276cd 100644
@@ -378,6 +378,7 @@ endif
 	if ! [ -d "$(TARGET_DIR)/bin" ]; then \
 		if [ -d "$(TARGET_SKELETON)" ]; then \
 			cp -fa $(TARGET_SKELETON)/* $(TARGET_DIR)/; \
+			chmod -R u+w $(TARGET_DIR)/; \
 		fi; \
 	fi
 	-find $(TARGET_DIR) -type d -name CVS -print0 -o -name .svn -print0 | xargs -0 rm -rf

Signature · 1 line

1.7.10.2

Arnout Vandecappelle

2012-06-20 23:12 UTC #3 lore raw

PATCH 2/4 ensure target fs is writable

Quoted 13 lines · Nathan Lynch

If the source target skeleton is read-only, modifications to the output target (such as creating /etc/hostname and /etc/issue) fail.
Signed-off-by Nathan Lynch<[email protected]>
---
  Makefile |    1 +
  1 file changed, 1 insertion(+)
Makefile +0 -0
index 9a9634d..38276cd 100644

Makefile endif · line 381

Quoted 4 lines

Makefile +1 -0

@@ -378,6 +378,7 @@ endif
  	if ! [ -d "$(TARGET_DIR)/bin" ]; then \
  		if [ -d "$(TARGET_SKELETON)" ]; then \
  			cp -fa $(TARGET_SKELETON)/* $(TARGET_DIR)/; \
+			chmod -R u+w $(TARGET_DIR)/; \

Maybe it's safer to do this only on directories, e.g. with
rsync -a --chmod=Du+w $(TARGET_SKELETON)/ $(TARGET_DIR)/ (untested)

Regards,
Arnout

Signature · 6 lines

Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 +32-16-286540
Essensium/Mind                                     http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

Nathan Lynch

author

2012-06-18 15:02 UTC #4 lore raw

PATCH 3/4 busybox: ensure $(BUSYBOX_BUILD_CONFIG) is writable

In reply to Nathan Lynch

If $(BUSYBOX_CONFIG_FILE) is read-only, the sed fixups to $(BUSYBOX_BUILD_CONFIG) fail.

Signed-off-by Nathan Lynch <[email protected]>

---
 package/busybox/busybox.mk |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

package/busybox/busybox.mk +2 -1

index 1d988f8..50a612f 100644
@@ -115,7 +115,8 @@ endef
 endif
 
 define BUSYBOX_COPY_CONFIG
-	cp -f $(BUSYBOX_CONFIG_FILE) $(BUSYBOX_BUILD_CONFIG)
+	cp -f $(BUSYBOX_CONFIG_FILE) $(BUSYBOX_BUILD_CONFIG) && \
+		chmod u+w $(BUSYBOX_BUILD_CONFIG)
 endef
 
 # Disable shadow passwords support if unsupported by the C library

Signature · 1 line

1.7.10.2

Arnout Vandecappelle

2012-06-20 23:13 UTC #5 lore raw

PATCH 3/4 busybox: ensure $(BUSYBOX_BUILD_CONFIG) is writable

Quoted 13 lines · Nathan Lynch

If $(BUSYBOX_CONFIG_FILE) is read-only, the sed fixups to $(BUSYBOX_BUILD_CONFIG) fail.
Signed-off-by Nathan Lynch<[email protected]>
---
  package/busybox/busybox.mk |    3 ++-
  1 file changed, 2 insertions(+), 1 deletion(-)
package/busybox/busybox.mk +0 -0
index 1d988f8..50a612f 100644

package/busybox/busybox.mk endef · line 119

Quoted 6 lines

package/busybox/busybox.mk +2 -1

@@ -115,7 +115,8 @@ endef
  endif
 
  define BUSYBOX_COPY_CONFIG
-	cp -f $(BUSYBOX_CONFIG_FILE) $(BUSYBOX_BUILD_CONFIG)
+	cp -f $(BUSYBOX_CONFIG_FILE) $(BUSYBOX_BUILD_CONFIG)&&  \
+		chmod u+w $(BUSYBOX_BUILD_CONFIG)

Just put it on two lines, without the && \ connector.

Regards,
Arnout

Signature · 6 lines

Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 +32-16-286540
Essensium/Mind                                     http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

Nathan Lynch

author

2012-06-18 15:02 UTC #6 lore raw

PATCH 4/4 linux: ensure buildroot_defconfig is writable

In reply to Nathan Lynch

If $(KERNEL_SOURCE_CONFIG) is read-only, the rm of $(KERNEL_ARCH_PATH)/configs/buildroot_defconfig will prompt the user. Make it writable and use rm -f for good measure.

Signed-off-by Nathan Lynch <[email protected]>

---
 linux/linux.mk |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

linux/linux.mk +2 -1

index f165dda..211d054 100644
@@ -138,8 +138,9 @@ endif
 
 define LINUX_CONFIGURE_CMDS
 	cp $(KERNEL_SOURCE_CONFIG) $(KERNEL_ARCH_PATH)/configs/buildroot_defconfig
+	chmod u+w $(KERNEL_ARCH_PATH)/configs/buildroot_defconfig
 	$(TARGET_MAKE_ENV) $(MAKE1) $(LINUX_MAKE_FLAGS) -C $(@D) buildroot_defconfig
-	rm $(KERNEL_ARCH_PATH)/configs/buildroot_defconfig
+	rm -f $(KERNEL_ARCH_PATH)/configs/buildroot_defconfig
 	$(if $(BR2_ARM_EABI),
 		$(call KCONFIG_ENABLE_OPT,CONFIG_AEABI,$(@D)/.config),
 		$(call KCONFIG_DISABLE_OPT,CONFIG_AEABI,$(@D)/.config))

Signature · 1 line

1.7.10.2

Yann E. MORIN

2013-05-26 14:45 UTC #7 lore raw

cover support building from read-only source tree

In reply to Nathan Lynch

Nathan, All,

On 2012-06-18 10:02 -0500, Nathan Lynch spake thusly:

Quoted 15 lines

There are several places in Buildroot where files are copied from the source tree to the output directory and then modified. The most obvious case is copying a .config or equivalent to the build directory and then modifying it with sed, as with uClibc or busybox.
This works fine when the source file has write permissions; this is preserved in the destination copy. This is the behavior in the common case of running from a git repository or a release tarball.
However, when the source file is read-only, the destination copy is also not writable and such fixups fail. This occurs when the the Buildroot source tree is being hosted in a Perforce repository. Perforce uses local file permissions to track whether a file is being edited and checks out all regular files read-only by design.

Quoted 3 lines

The following changes were enough to get qemu_mips_malta_defconfig to build and run successfully. I submit these patches for discussion and illustration of the effort required to support this use case.

Unfortunately, those patches mostly no longer apply, since there have been some code churn (reoridering) in the meantime.

I'll take whatever I can, and adapt the remaining.

Regards,
Yann E. MORIN.

Signature · 6 lines

.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

Yann E. MORIN

2013-05-26 16:36 UTC #8 lore raw

cover support building from read-only source tree

Nathan, All,

On 2013-05-26 16:45 +0200, Yann E. MORIN spake thusly:

Quoted 10 lines

On 2012-06-18 10:02 -0500, Nathan Lynch spake thusly:
Quoted 4 lines
There are several places in Buildroot where files are copied from the source tree to the output directory and then modified. The most obvious case is copying a .config or equivalent to the build directory and then modifying it with sed, as with uClibc or busybox.
Unfortunately, those patches mostly no longer apply, since there have been some code churn (reoridering) in the meantime.
I'll take whatever I can, and adapt the remaining.

I've added them to my patchwork branch:
https://www.gitorious.org/buildroot/buildroot/commits/yem-patchwork-fixes

Thank you!

Regards,
Yann E. MORIN.

Signature · 6 lines

.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

Nathan Lynch

author

2013-05-30 20:36 UTC #9 lore raw

cover support building from read-only source tree

Quoted 12 lines · Yann E. MORIN

Nathan, All,
On 2013-05-26 16:45 +0200, Yann E. MORIN spake thusly:
Quoted 9 lines
On 2012-06-18 10:02 -0500, Nathan Lynch spake thusly:
Quoted 4 lines
There are several places in Buildroot where files are copied from the source tree to the output directory and then modified. The most obvious case is copying a .config or equivalent to the build directory and then modifying it with sed, as with uClibc or busybox.
Unfortunately, those patches mostly no longer apply, since there have been some code churn (reoridering) in the meantime.

Quoted 6 lines

Quoted 1 line
I'll take whatever I can, and adapt the remaining.
I've added them to my patchwork branch:
https://www.gitorious.org/buildroot/buildroot/commits/yem-patchwork-fixes
Thank you!

Thank you for picking these up. I haven't been sufficiently motivated to pursue these (obviously); hopefully they will be useful to others.