Makefile: improve $(TARGET_DIR) (re)creation

PATCH Superseded Bjørn Forsman 2 participants 12 messages Feb 17, 2011 – Mar 3, 2011 Kbuild

lore

Share & tools

lore

Bjørn Forsman

author

2011-02-17 23:44 UTC #0 lore raw

PATCH v2 1/2 Makefile: improve $(TARGET_DIR) (re)creation

This patch fixes the following use case:

1. Build, hack, ...
2. Oh no, $(TARGET_DIR) is no longer in sync with $(TARGET_SKELETON) and/or the packages selected in menuconfig. If only there was a way to *easily* rebuild $(TARGET_DIR) so it is in sync...

With this patch, just remove $(TARGET_DIR) and Buildroot will recreate it.

Main changes:
* Always copy $(TARGET_SKELETON) to $(TARGET_DIR).
* If $(TARGET_DIR) needs to be created (i.e. if the directory is missing), remove stamp files so all currently selected packages will be reinstalled.
* If $(TARGET_DIR)/lib/modules/ is missing, try to install kernel modules.

Also:
* remove code touching .fakeroot.00000 file (not needed)
* remove code trying to remove CVS and .svn stuff from target skeleton (not needed)

Signed-off-by Bj?rn Forsman <[email protected]>

---
Changes in v2:
* remove gcc stamp file so libgcc_s.so.1 will be copied to TARGET_DIR
* make sure kernel modules get reinstalled

 Makefile       |   26 +++++++++++---------------
 linux/linux.mk |    6 ++++--
 2 files changed, 15 insertions(+), 17 deletions(-)

Makefile +10 ~~-14~~

index d954178..4d41e98 100644
@@ -364,7 +364,7 @@ $(TARGETS_ALL): __real_tgt_%: $(BASE_TARGETS) %
 dirs: $(DL_DIR) $(TOOLCHAIN_DIR) $(BUILD_DIR) $(STAGING_DIR) $(TARGET_DIR) \
 	$(HOST_DIR) $(BR2_DEPENDS_DIR) $(BINARIES_DIR) $(STAMP_DIR)
 
-$(BASE_TARGETS): dirs $(O)/toolchainfile.cmake
+$(BASE_TARGETS): dirs $(O)/toolchainfile.cmake copy-target-skeleton
 
 $(BUILD_DIR)/buildroot-config/auto.conf: $(CONFIG_DIR)/.config
 	$(MAKE) $(EXTRAMAKEARGS) silentoldconfig

@@ -390,9 +390,10 @@ $(O)/toolchainfile.cmake:
 	" > $@
 
 .PHONY: all world dirs clean distclean source outputmakefile \
+	copy-target-skeleton \
 	$(BASE_TARGETS) $(TARGETS) $(TARGETS_ALL) \
 	$(TARGETS_CLEAN) $(TARGETS_DIRCLEAN) $(TARGETS_SOURCE) \
-	$(DL_DIR) $(TOOLCHAIN_DIR) $(BUILD_DIR) $(STAGING_DIR) $(TARGET_DIR) \
+	$(DL_DIR) $(TOOLCHAIN_DIR) $(BUILD_DIR) $(STAGING_DIR) \
 	$(HOST_DIR) $(BR2_DEPENDS_DIR) $(BINARIES_DIR) $(STAMP_DIR)
 
 #############################################################

@@ -416,19 +417,14 @@ ifeq ($(BR2_ROOTFS_SKELETON_CUSTOM),y)
 TARGET_SKELETON=$(BR2_ROOTFS_SKELETON_CUSTOM_PATH)
 endif
 
-$(BUILD_DIR)/.root:
-	mkdir -p $(TARGET_DIR)
-	if ! [ -d "$(TARGET_DIR)/bin" ]; then \
-		if [ -d "$(TARGET_SKELETON)" ]; then \
-			cp -fa $(TARGET_SKELETON)/* $(TARGET_DIR)/; \
-		fi; \
-		touch $(STAGING_DIR)/.fakeroot.00000; \
-	fi
-	-find $(TARGET_DIR) -type d -name CVS -print0 -o -name .svn -print0 | xargs -0 rm -rf
-	-find $(TARGET_DIR) -type f \( -name .empty -o -name '*~' \) -print0 | xargs -0 rm -rf
-	touch $@
+copy-target-skeleton: $(TARGET_DIR)
+	cp -ua $(TARGET_SKELETON)/* $(TARGET_DIR)/ && \
+	find $(TARGET_DIR) -type f \( -name .empty -o -name '*~' \) -print0 | xargs -0 rm -rf
 
-$(TARGET_DIR): $(BUILD_DIR)/.root
+$(TARGET_DIR):
+	mkdir $(TARGET_DIR)
+	rm -f $(BUILD_DIR)/*/.stamp_target_installed
+	rm -f $(STAMP_DIR)/gcc_libs_target_installed
 
 erase-fakeroots:
 	rm -f $(BUILD_DIR)/.fakeroot*

linux/linux.mk +4 -2

index 9076fb9..0ad8c0f 100644
@@ -147,6 +147,9 @@ $(LINUX26_DIR)/.stamp_compiled: $(LINUX26_DIR)/.stamp_configured $(LINUX26_DIR)/
 $(LINUX26_DIR)/.stamp_installed: $(LINUX26_DIR)/.stamp_compiled
 	@$(call MESSAGE,"Installing kernel")
 	cp $(LINUX26_IMAGE_PATH) $(BINARIES_DIR)
+	$(Q)touch $@
+
+$(TARGET_DIR)/lib/modules:
 	# Install modules and remove symbolic links pointing to build
 	# directories, not relevant on the target
 	@if [ $(shell grep -c "CONFIG_MODULES=y" $(LINUX26_DIR)/.config) != 0 ] ; then 	\

@@ -155,9 +158,8 @@ $(LINUX26_DIR)/.stamp_installed: $(LINUX26_DIR)/.stamp_compiled
 		rm -f $(TARGET_DIR)/lib/modules/$(LINUX26_VERSION_PROBED)/build ;	\
 		rm -f $(TARGET_DIR)/lib/modules/$(LINUX26_VERSION_PROBED)/source ;	\
 	fi
-	$(Q)touch $@
 
-linux linux26: host-module-init-tools $(LINUX26_DEPENDENCIES) $(LINUX26_DIR)/.stamp_installed
+linux linux26: host-module-init-tools $(LINUX26_DEPENDENCIES) $(LINUX26_DIR)/.stamp_installed $(TARGET_DIR)/lib/modules
 
 linux-menuconfig linux-xconfig linux-gconfig linux-nconfig linux26-menuconfig linux26-xconfig linux26-gconfig linux26-nconfig: dirs $(LINUX26_DIR)/.stamp_configured
 	$(MAKE) $(LINUX26_MAKE_FLAGS) -C $(LINUX26_DIR) \

Signature · 1 line

1.7.1

Bjørn Forsman

author

2011-02-21 17:02 UTC #1 lore raw

PATCH v2 1/2 Makefile: improve $(TARGET_DIR) (re)creation

2011/2/18 Bj?rn Forsman <[email protected]>:

Quoted 24 lines

This patch fixes the following use case:
?1. Build, hack, ...
?2. Oh no, $(TARGET_DIR) is no longer in sync with $(TARGET_SKELETON) ? ?and/or the packages selected in menuconfig. If only there was a way ? ?to *easily* rebuild $(TARGET_DIR) so it is in sync...
With this patch, just remove $(TARGET_DIR) and Buildroot will recreate it.
Main changes:
?* Always copy $(TARGET_SKELETON) to $(TARGET_DIR).
?* If $(TARGET_DIR) needs to be created (i.e. if the directory is ? missing), remove stamp files so all currently selected packages will ? be reinstalled.
?* If $(TARGET_DIR)/lib/modules/ is missing, try to install kernel ? modules.
Also:
?* remove code touching .fakeroot.00000 file (not needed)
?* remove code trying to remove CVS and .svn stuff from target skeleton ? (not needed)
Signed-off-by Bj?rn Forsman <[email protected]>

Quoted 8 lines

---
Changes in v2:
?* remove gcc stamp file so libgcc_s.so.1 will be copied to TARGET_DIR ?* make sure kernel modules get reinstalled
?Makefile ? ? ? | ? 26 +++++++++++--------------- ?linux/linux.mk | ? ?6 ++++--
?2 files changed, 15 insertions(+), 17 deletions(-)

Ping?

Bjørn Forsman

author

2011-03-01 16:03 UTC #2 lore raw

PATCH v2 1/2 Makefile: improve $(TARGET_DIR) (re)creation

2011/2/21 Bj?rn Forsman <[email protected]>:

Quoted 30 lines

2011/2/18 Bj?rn Forsman <[email protected]>:
Quoted 29 lines
This patch fixes the following use case:
?1. Build, hack, ...
?2. Oh no, $(TARGET_DIR) is no longer in sync with $(TARGET_SKELETON) ? ?and/or the packages selected in menuconfig. If only there was a way ? ?to *easily* rebuild $(TARGET_DIR) so it is in sync...
With this patch, just remove $(TARGET_DIR) and Buildroot will recreate it.
Main changes:
?* Always copy $(TARGET_SKELETON) to $(TARGET_DIR).
?* If $(TARGET_DIR) needs to be created (i.e. if the directory is ? missing), remove stamp files so all currently selected packages will ? be reinstalled.
?* If $(TARGET_DIR)/lib/modules/ is missing, try to install kernel ? modules.
Also:
?* remove code touching .fakeroot.00000 file (not needed)
?* remove code trying to remove CVS and .svn stuff from target skeleton ? (not needed)
Signed-off-by Bj?rn Forsman <[email protected]>
---
Changes in v2:
?* remove gcc stamp file so libgcc_s.so.1 will be copied to TARGET_DIR ?* make sure kernel modules get reinstalled

Quoted 5 lines

Quoted 3 lines
?Makefile ? ? ? | ? 26 +++++++++++--------------- ?linux/linux.mk | ? ?6 ++++--
?2 files changed, 15 insertions(+), 17 deletions(-)
Ping?

Ping ping :-)

Now that the new cycle has opened, any comments on this?

I'd really like to get this functionality in mainline :-)

Best regards,
Bj?rn Forsman

Thomas Petazzoni

2011-03-01 16:20 UTC #3 lore raw

PATCH v2 1/2 Makefile: improve $(TARGET_DIR) (re)creation

Quoted 5 lines · On Tue, 1 Mar 2011 17:03:22 +0100 Bj?rn Forsman <[email protected]> wrote:

Ping ping :-)
Now that the new cycle has opened, any comments on this?
I'd really like to get this functionality in mainline :-)

I think the functionality is great, but I'd prefer to see it integrated on top of the packaging work started by Lionel rather than coded as proposed in the current patch. I am also not sure that rebuilding the $(TARGET_DIR) without rebuilding the $(STAGING_DIR) is a good thing.

However, it's true that the packaging work started by Lionel is not ready for inclusion yet, so maybe we should merge this as is, and then update it once we have the proper packaging mechanism working.

Regards,

Thomas

Signature · 4 lines

Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

Bjørn Forsman

author

2011-03-01 17:16 UTC #4 lore raw

PATCH v2 1/2 Makefile: improve $(TARGET_DIR) (re)creation

Quoted 5 lines · Thomas Petazzoni

Quoted 2 lines · On Tue, 1 Mar 2011 17:03:22 +0100 Bj?rn Forsman <[email protected]> wrote:
Ping ping :-)

Quoted 7 lines

Quoted 3 lines
Now that the new cycle has opened, any comments on this?
I'd really like to get this functionality in mainline :-)
I think the functionality is great, but I'd prefer to see it integrated on top of the packaging work started by Lionel rather than coded as proposed in the current patch.

I see.

Quoted 2 lines

I am also not sure that rebuilding the
$(TARGET_DIR) without rebuilding the $(STAGING_DIR) is a good thing.

TARGET_DIR depends on STAGING_DIR, and not the other way around, so I don't see why STAGING_DIR should be rebuilt. Am I missing something?

Quoted 3 lines

However, it's true that the packaging work started by Lionel is not ready for inclusion yet, so maybe we should merge this as is, and then update it once we have the proper packaging mechanism working.

Yes, please :-)

Quoted 8 lines

Regards,
Thomas
--
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

Bjørn Forsman

author

2011-03-03 18:22 UTC #5 lore raw

PATCH v2 1/2 Makefile: improve $(TARGET_DIR) (re)creation

In reply to Bjørn Forsman

2011/2/18 Bj?rn Forsman <[email protected]>:

Quoted 6 lines

This patch fixes the following use case:
?1. Build, hack, ...
?2. Oh no, $(TARGET_DIR) is no longer in sync with $(TARGET_SKELETON) ? ?and/or the packages selected in menuconfig. If only there was a way ? ?to *easily* rebuild $(TARGET_DIR) so it is in sync...

There is a bug in this patch: Qt isn't completely reinstalled (libs are, but not demos/examples). I will send an updated patch shortly.

Best regards,
Bj?rn Forsman

Bjørn Forsman

author

2011-02-17 23:44 UTC #6 lore raw

PATCH v2 2/2 linux.mk: depend on actual kernel image instead of stamp file

In reply to Bjørn Forsman

Use the real output file (kernel image) as target instead of a stamp file. This way Buildroot cannot be tricked into thinking that the kernel image is installed when it's not.

Signed-off-by Bj?rn Forsman <[email protected]>

---
 linux/linux.mk |    8 ++++----
 1 files changed, 4 insertions(+), 4 deletions(-)

linux/linux.mk +4 -4

index 0ad8c0f..5855658 100644
@@ -144,7 +144,7 @@ $(LINUX26_DIR)/.stamp_compiled: $(LINUX26_DIR)/.stamp_configured $(LINUX26_DIR)/
 	$(Q)touch $@
 
 # Installation
-$(LINUX26_DIR)/.stamp_installed: $(LINUX26_DIR)/.stamp_compiled
+$(BINARIES_DIR)/$(LINUX26_IMAGE_NAME): $(LINUX26_DIR)/.stamp_compiled
 	@$(call MESSAGE,"Installing kernel")
 	cp $(LINUX26_IMAGE_PATH) $(BINARIES_DIR)
 	$(Q)touch $@

@@ -153,13 +153,13 @@ $(TARGET_DIR)/lib/modules:
 	# Install modules and remove symbolic links pointing to build
 	# directories, not relevant on the target
 	@if [ $(shell grep -c "CONFIG_MODULES=y" $(LINUX26_DIR)/.config) != 0 ] ; then 	\
-		$(TARGET_MAKE_ENV) $(MAKE1) $(LINUX26_MAKE_FLAGS) -C $(@D) 		\
+		$(TARGET_MAKE_ENV) $(MAKE1) $(LINUX26_MAKE_FLAGS) -C $(LINUX26_DIR)	\
 			DEPMOD="$(HOST_DIR)/usr/sbin/depmod" modules_install ;		\
 		rm -f $(TARGET_DIR)/lib/modules/$(LINUX26_VERSION_PROBED)/build ;	\
 		rm -f $(TARGET_DIR)/lib/modules/$(LINUX26_VERSION_PROBED)/source ;	\
 	fi
 
-linux linux26: host-module-init-tools $(LINUX26_DEPENDENCIES) $(LINUX26_DIR)/.stamp_installed $(TARGET_DIR)/lib/modules
+linux linux26: host-module-init-tools $(LINUX26_DEPENDENCIES) $(BINARIES_DIR)/$(LINUX26_IMAGE_NAME) $(TARGET_DIR)/lib/modules
 
 linux-menuconfig linux-xconfig linux-gconfig linux-nconfig linux26-menuconfig linux26-xconfig linux26-gconfig linux26-nconfig: dirs $(LINUX26_DIR)/.stamp_configured
 	$(MAKE) $(LINUX26_MAKE_FLAGS) -C $(LINUX26_DIR) \

@@ -167,7 +167,7 @@ linux-menuconfig linux-xconfig linux-gconfig linux-nconfig linux26-menuconfig li
 
 # Support for rebuilding the kernel after the initramfs file list has
 # been generated in $(BINARIES_DIR)/rootfs.initramfs.
-$(LINUX26_DIR)/.stamp_initramfs_rebuilt: $(LINUX26_DIR)/.stamp_installed $(BINARIES_DIR)/rootfs.initramfs
+$(LINUX26_DIR)/.stamp_initramfs_rebuilt: $(BINARIES_DIR)/$(LINUX26_IMAGE_NAME) $(BINARIES_DIR)/rootfs.initramfs
 	@$(call MESSAGE,"Rebuilding kernel with initramfs")
 	# Remove the previously generated initramfs which was empty,
 	# to make sure the kernel will actually regenerate it.

Signature · 1 line

1.7.1

Bjørn Forsman

author

2011-02-21 17:02 UTC #7 lore raw

PATCH v2 2/2 linux.mk: depend on actual kernel image instead of stamp file

2011/2/18 Bj?rn Forsman <[email protected]>:

Quoted 8 lines

Use the real output file (kernel image) as target instead of a stamp file. This way Buildroot cannot be tricked into thinking that the kernel image is installed when it's not.
Signed-off-by Bj?rn Forsman <[email protected]>
---
?linux/linux.mk | ? ?8 ++++----
?1 files changed, 4 insertions(+), 4 deletions(-)

Ping?

Bjørn Forsman

author

2011-03-01 16:06 UTC #8 lore raw

PATCH v2 2/2 linux.mk: depend on actual kernel image instead of stamp file

2011/2/21 Bj?rn Forsman <[email protected]>:

Quoted 11 lines

2011/2/18 Bj?rn Forsman <[email protected]>:
Quoted 8 lines
Use the real output file (kernel image) as target instead of a stamp file. This way Buildroot cannot be tricked into thinking that the kernel image is installed when it's not.
Signed-off-by Bj?rn Forsman <[email protected]>
---
?linux/linux.mk | ? ?8 ++++----
?1 files changed, 4 insertions(+), 4 deletions(-)
Ping?

Ping ping :-)

Any comments?

Best regards,
Bj?rn Forsman

Thomas Petazzoni

2011-03-01 16:23 UTC #9 lore raw

PATCH v2 2/2 linux.mk: depend on actual kernel image instead of stamp file

In reply to Bjørn Forsman

Hello,

Quoted 3 lines · On Fri, 18 Feb 2011 00:44:54 +0100 Bj?rn Forsman <[email protected]>…

Use the real output file (kernel image) as target instead of a stamp file. This way Buildroot cannot be tricked into thinking that the kernel image is installed when it's not.

Well, we switched *all* packages from dependencies on installed files to dependencies on stamp files when migrating them from hand-coded makefiles to the package infrastructures, so I don't think we should do the opposite for the Linux kernel package.

But maybe we need to provide for each package (kernel included) a clean and nice mechanism to delete their stamp files so that they get reinstalled at the next make invocation.

Regardless of the solution chosen, I think any change in this direction should come with a clear update in the documentation. I don't want to see dozens of different hacks^Wmechanisms to do partial rebuilds being added all over the place without any coherency nor documentation.

Regards!

Thomas

Signature · 4 lines

Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

Bjørn Forsman

author

2011-03-01 18:54 UTC #10 lore raw

PATCH v2 2/2 linux.mk: depend on actual kernel image instead of stamp file

Quoted 5 lines · Thomas Petazzoni

Hello,
On Fri, 18 Feb 2011 00:44:54 +0100
Bj?rn Forsman <[email protected]> wrote:

Quoted 8 lines

Quoted 3 lines
Use the real output file (kernel image) as target instead of a stamp file. This way Buildroot cannot be tricked into thinking that the kernel image is installed when it's not.
Well, we switched *all* packages from dependencies on installed files to dependencies on stamp files when migrating them from hand-coded makefiles to the package infrastructures, so I don't think we should do the opposite for the Linux kernel package.

Yes, I know. But I think that the Linux kernel image is special. It is one file. It is easy to track. Using stamp files for *generic packages* makes sense because it is difficult to keep track of all the files they may install.

Here is how Buildroot currently behave:

1. build and see that output/images/ contain rootfs.tar and zImage.
2. remove output/images/ and rebuild
3. output/images/ now only has rootfs.tar, no zImage!

That is just confusing behaviour and I want to fix it.

Quoted 3 lines

But maybe we need to provide for each package (kernel included) a clean and nice mechanism to delete their stamp files so that they get reinstalled at the next make invocation.

That is possible. But wouldn't you then have to track dependencies yourself? I'd rather "rm -rf output/target/ output/images/" and have Buildroot rebuild these dirs automatically than issuing some "make pkg1-clean-stamp ... pkgN-clean-stamp".

Or did I misunderstand you idea?

Quoted 4 lines

Regardless of the solution chosen, I think any change in this direction should come with a clear update in the documentation. I don't want to see dozens of different hacks^Wmechanisms to do partial rebuilds being added all over the place without any coherency nor documentation.

I wasn't thinking that big :-) This patch is just a small usability fix IMO, so I don't think is warrants a doc update.

Quoted 8 lines

Regards!
Thomas
--
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

Bjørn Forsman

author

2011-03-01 20:06 UTC #11 lore raw

PATCH v2 2/2 linux.mk: depend on actual kernel image instead of stamp file

In reply to Thomas Petazzoni

Hi Thomas,

I forgot to say thank you for your feedback in my previous mail. So, thank you for your feedback :-)

Also, I have an idea for a patch that provide almost the same functionality as this one -- without removing stamp files (in the patch): If output/images/ needs to be created, remove all stamp files from packages that install into output/images/.

It is basically the same method as patch 1 in this series. What do you think?

Right now I'm not sure which method I like best, but I figured I should post it anyway.

Best regards,
Bj?rn Forsman