Force rebuild of local site packages and theirs local dependencies

The purpose of this patch is to rebuild local site packages and theirs local dependencies if the source has changed. Therefore, if the source of a package has changed, it will be rebuild if you type make or make <pkg>. Likewise, if a package has a library dependency which is local site package too and you type make <pkg>, the library will be rebuild if the source has been modified.

This behavior is pretty useful if you use buildroot with many of your own packages. Especially if you share these packages with a developer team through a version control system like git.

---
 package/pkg-generic.mk | 4 ++++
 1 file changed, 4 insertions(+)

index d5b29f0..929a31c 100644
@@ -509,6 +509,10 @@ ifeq ($$(filter $(1),$$(DEPENDENCIES_HOST_PREREQ)),)
 $$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies
 endif
 
+ifeq ($$($(2)_SITE_METHOD),local)
+$$($(2)_TARGET_RSYNC): $(1)-clean-for-rebuild
+endif
+
 ifeq ($$($(2)_OVERRIDE_SRCDIR),)
 # In the normal case (no package override), the sequence of steps is
 #  source, by downloading

2.1.4

Hi Anthony,

This is pretty much my Buildroot setup/work flow.

I'm not sure I follow. Let's say I have a package called test that contains two files, main.c and test.c and depends on libtest. If I apply the patch, modify test main.c, and execute 'make test', I get:

(master)$ make test
make -C /home/dkehn/buildroot/src-master O=/home/dkehn/buildroot/src-master/output//. test rm -f /home/dkehn/buildroot/src-master/output//build/libtest-1.0.0/.stamp_rsynced rm -f /home/dkehn/buildroot/src-master/output//build/libtest-1.0.0/.stamp_staging_installed rm -f /home/dkehn/buildroot/src-master/output//build/libtest-1.0.0/.stamp_target_installed rm -f /home/dkehn/buildroot/src-master/output//build/libtest-1.0.0/.stamp_images_installed rm -f /home/dkehn/buildroot/src-master/output//build/libtest-1.0.0/.stamp_host_installed rm -f /home/dkehn/buildroot/src-master/output//build/libtest-1.0.0/.stamp_built

rsync -au --exclude .svn --exclude .git --exclude .hg --exclude .bzr --exclude CVS doug/package/libtest/src/ /home/dkehn/buildroot/src-master/output//build/libtest-1.0.0

rm -f /home/dkehn/buildroot/src-master/output//build/test-1.0.0/.stamp_rsynced rm -f /home/dkehn/buildroot/src-master/output//build/test-1.0.0/.stamp_staging_installed rm -f /home/dkehn/buildroot/src-master/output//build/test-1.0.0/.stamp_target_installed rm -f /home/dkehn/buildroot/src-master/output//build/test-1.0.0/.stamp_images_installed rm -f /home/dkehn/buildroot/src-master/output//build/test-1.0.0/.stamp_host_installed rm -f /home/dkehn/buildroot/src-master/output//build/test-1.0.0/.stamp_built

rsync -au --exclude .svn --exclude .git --exclude .hg --exclude .bzr --exclude CVS doug/package/test/src/ /home/dkehn/buildroot/src-master/output//build/test-1.0.0

Nothing was built. I would think it desirable, at a minimum, for test to be rebuilt. If I then execute
make test-rebuild
both main.c and test.c are rebuilt even though only main.c changed. I don't think this is desirable.

On the other hand, using 'make test-rebuild' works as expected. However, I did notice one oddity. Let's say I modify libtest.c and then execute 'make test-rebuild'.

(master)$ make test-rebuild
make -C /home/dkehn/buildroot/src-master O=/home/dkehn/buildroot/src-master/output//. test-rebuild rm -f /home/dkehn/buildroot/src-master/output//build/test-1.0.0/.stamp_rsynced
rm -f /home/dkehn/buildroot/src-master/output//build/test-1.0.0/.stamp_staging_installed rm -f /home/dkehn/buildroot/src-master/output//build/test-1.0.0/.stamp_target_installed rm -f /home/dkehn/buildroot/src-master/output//build/test-1.0.0/.stamp_images_installed rm -f /home/dkehn/buildroot/src-master/output//build/test-1.0.0/.stamp_host_installed rm -f /home/dkehn/buildroot/src-master/output//build/test-1.0.0/.stamp_built
rm -f /home/dkehn/buildroot/src-master/output//build/libtest-1.0.0/.stamp_rsynced rm -f /home/dkehn/buildroot/src-master/output//build/libtest-1.0.0/.stamp_staging_installed rm -f /home/dkehn/buildroot/src-master/output//build/libtest-1.0.0/.stamp_target_installed rm -f /home/dkehn/buildroot/src-master/output//build/libtest-1.0.0/.stamp_images_installed rm -f /home/dkehn/buildroot/src-master/output//build/libtest-1.0.0/.stamp_host_installed rm -f /home/dkehn/buildroot/src-master/output//build/libtest-1.0.0/.stamp_built

rsync -au --exclude .svn --exclude .git --exclude .hg --exclude .bzr --exclude CVS doug/package/libtest/src/ /home/dkehn/buildroot/src-master/output//build/libtest-1.0.0

rsync -au --exclude .svn --exclude .git --exclude .hg --exclude .bzr --exclude CVS doug/package/test/src/ /home/dkehn/buildroot/src-master/output//build/test-1.0.0

make[3]: Nothing to be done for 'all'.

I was expecting libtest.c in libtest to be rebuilt and all of test to be rebuilt. A second execution of 'make test-rebuild' does perform the expected steps. ...interesting.

I apologize if I've mucked this whole thing up. These were just my observations for an interesting patch that followed a similar work flow.

Regards,
...doug

Hi doug,

Indeed. There is a big problem with my patch. I believed the job was done and rebuild the package and these dependencies if change was found but it doesn't. I'm disappointed. I pushed the patch too quickly.

I'm trying to find a new way to implement my needs.

Regards,
Anthony.

Dear Anthony Viallard,

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

Hi Thomas,

The main thing is this not convenient. I have about 30 packages and we are a team. I don't want to check after each git pull the packages which have been modified and call make <pkg>-rebuild for each of them (and be aware about dependencies, for example calling libraries rebuild before because the <pkg>-rebuild target doesn't rebuild the dependencies).

My patch is a quick workaround to do my needs. Moreover, like doug claims, the patch doesn't work. There is an issue. I will try to find why.

If there was a "make rebuild" target and if the "make <pkg>-rebuild" was able to rebuild all the dependencies too, it will satisfy my needs.

Regards,
Anthony.

An important thing is we don't increase the version of the package after a change. But it's not very convenient to increase the version after each small changes...

Regards,
Anthony.

Hi Antony,

Here is what I have done to support the workflow, I believe, you want. I don't think... this is something that Buildroot needs to support natively; but, maybe it could/should be documented somewhere.

I create a custom directory structure, let's call it 'mycustom', that mimics Buildroot's. The mycustom directory contains proprietary bits to be incorporated into the Buildroot environment while keeping them separate from Buildroot. I set BR2_EXTERNAL=mycustom when executing the initial make to create the output directory and specify the default configuration.

<Buildroot>/mycustom/Config.in contains:
comment "MyCustom"

menu "Libraries"
source "$BR2_EXTERNAL/package/libmycustom/Config.in" endmenu

menu "Applications"
source "$BR2_EXTERNAL/package/mycustomapp/Config.in" endmenu

In <Buildroot>/mycustom/external.mk, I add a mycustom-rebuild target to rebuild all of my custom packages.

        mycustom-rebuild:
            @ sed -e '1,/#.MyCustom/d' \
                  -ne 's/^BR2_PACKAGE_\(.*\)=y/\1/p' \
                  $(BR2_CONFIG) | \
                    tr '[:upper:]' '[:lower:]' | \
                    sed -e 's/_/-/' | \
                    while read t; do \
                            $(MAKE1) -C $(BASE_DIR) O=$(O) \
                                    $${t}-dirclean $${t}-rebuild || exit 1; \
                    done
            $(MAKE) -C $(BASE_DIR) O=$(O) target-post-image

Presently, Buildroot puts all user provided options at the end of BR2_CONFIG so it's sufficient to extract all BR2_PACKAGE_* definitions after the the Config.in comment "# MyCustom". Each extracted BR2_CONFIG definition becomes the make target to execute after BR2_PACKAGE_ is stripped from the entry, it is changed to lower case, and '_' is changed to '-'. For example, BR2_PACKAGE_LIBMYCUSTOM is transformed to libmycustom. libmycustom is then used to create the libmycustom-dirclean and libmycustom-rebuild targets. Lastly, the target-post-image target is executed to regenerate the root file system(s).

With this in place, after a 'git pull', I can simply execute 'make mycustom-rebuild' to only rebuild all of my custom stuff.

As you can see there are clearly some personal preferences in the mycustom-rebuild target that others may not want. This is why I don't think this is something Buildroot needs to support natively. Anyway, maybe this will help you implement the function you want.

Regards,
...doug

Hi doug,

Thanks for the hint. I will try to do the same thing on my side. But I think the recipe will be different for me. I don't want to rebuild all the packages each time but rebuild only if there is a change in the source code.

Regards,
Anthony.

Hello Anthony,

On Monday 22 June 2015 12:15:14 Viallard Anthony wrote:
[..]

Does this process would work?

rm build/*/.stamp_built
make

J?r?me Pouiller, Sysmic
Embedded Linux specialist
http://www.sysmic.fr

Hi J?rome,

Yes. It can be a solution. I already did this way to rebuild package in the old time when there wasn't the *-rebuild target command:

rm -rf output/build/clearsilver-0.10.5/.stamp_built && rm -rf output/build/clearsilver-0.10.5/.stamp_configured

But I prefer an integrated mechanism in the makefile which make the think more reliable and transparent :)

Regards,
Anthony.

When I need very specific action for a project (generally 'run', 'check', 'flash', etc...), I write a fragment of Makefile and a declare it in BR2_PACKAGE_OVERRIDE_FILE.

Therefore, you may add a local.mk file with:

local-rebuild:
rm build/*/.stamp_built

...and maybe (not tested):

$(PACKAGES): local-rebuild

(Someday, I will post a patch to be able to display those local targets in 'make help' output)

J?r?me Pouiller, Sysmic
Embedded Linux specialist
http://www.sysmic.fr