build: minimize rebuilds

PATCH In review Sam Ravnborg 2 participants 8 messages Dec 26, 2017 – Jan 6, 2018 Kbuild

Share & tools

Sam Ravnborg

author

2017-12-26 18:32 UTC #0 lore raw

cover build: minimize rebuilds

Following patchset minimizes re-builds.
With the setup I use at the moment where I run build tests on 100+ arm related boards I see that the build time is reduced to half - for an otherwise clean build with no changes.
In other words - the wall clock time for a simple "try to build all boards" where nothing has changed goes from ~2:35 minutes to 1:21 minutes

The patches has been tested in my local setup - and no missing rebuilds was seen.

Sam Ravnborg (4):
build: fix that defaultenv is always rebuilt
build: fix that passwd.h is always built
build: fix that the bbenv file is always rebuilt
build: fix that LZO file is always rebuilt

 common/Makefile        | 28 +++++++++++++++-------------
 defaultenv/Makefile    |  4 ++--
 scripts/Kbuild.include | 12 +++++++-----
 scripts/Makefile.lib   |  2 +-
 4 files changed, 25 insertions(+), 21 deletions(-)

_______________________________________________
barebox mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/barebox

Sam Ravnborg

author

2017-12-26 18:35 UTC #1 lore raw

PATCH 1/4 build: fix that defaultenv is always rebuilt

From fcffd0176b3394cc664d9bf5b31b053253323bfb Mon Sep 17 00:00:00 2001
From: Sam Ravnborg <[email protected]>
Date: Tue, 26 Dec 2017 16:47:16 +0100
Subject: [PATCH 1/4] build: fix that defaultenv is always rebuilt

Using following pattern:

	target: prerequisite FORCE
		$(call cmd,foo)

is almost always wrong. Due to FORCE target will be rebuilt using cmd_foo no mater the timestamps.

The following pattern:

	target: prerequisite FORCE
        	$(call if_changed,foo)

Will call cmd_foo only if:
- target is not updated
- or command to built target has changed

But to make this work target must be added to the list af targets built.
This can be done by assigning target to extra-y or targets.

Use the latter pattern for defaultenv to avoid rebuilding the environmentt for each build.

Signed-off-by Sam Ravnborg <[email protected]>

---
 defaultenv/Makefile | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

defaultenv/Makefile +2 -2

index 33e0eceef..f313b04e8 100644
@@ -14,7 +14,7 @@ quiet_cmd_env_default = ENV     $@
 cmd_env_default = ($(srctree)/scripts/genenv $(srctree) $(objtree) $@ $(CONFIG_DEFAULT_ENVIRONMENT_PATH))
 
 $(obj)/barebox_default_env: FORCE
-	$(call cmd,env_default)
+	$(call if_changed,env_default)
 
 quiet_cmd_env_h = ENVH    $@
 cmd_env_h = cat $< | (cd $(obj) && $(objtree)/scripts/bin2c "__aligned(4) default_environment") > $@; \

@@ -27,4 +27,4 @@ quiet_cmd_env_zero = ENVZ    $@
 cmd_env_zero = ($(objtree)/scripts/bareboxenv -z $(CONFIG_DEFAULT_ENVIRONMENT_PATH) $@)
 
 $(obj)/barebox_zero_env: FORCE
-	$(call cmd,env_zero)
+	$(call if_changed,env_zero)

Signature · 7 lines

2.12.0


_______________________________________________
barebox mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/barebox

Sam Ravnborg

author

2017-12-26 18:36 UTC #2 lore raw

PATCH 2/4 build: fix that passwd.h is always built

In reply to Sam Ravnborg

From 48fe20e2bf2249b2f89d96c9787e0b489c015054 Mon Sep 17 00:00:00 2001
From: Sam Ravnborg <[email protected]>
Date: Tue, 26 Dec 2017 18:02:17 +0100
Subject: [PATCH 2/4] build: fix that passwd.h is always built

Use the kbuild provided support for generated
files to avoid that passwd.h is always generated
thus triggering further re-builds

Signed-off-by Sam Ravnborg <[email protected]>

---
 common/Makefile | 28 +++++++++++++++-------------
 1 file changed, 15 insertions(+), 13 deletions(-)

common/Makefile +15 ~~-13~~

index 8cd0ab300..5351ef0f7 100644
@@ -63,27 +63,29 @@ obj-$(CONFIG_BAREBOX_UPDATE_IMX_NAND_FCB) += imx-bbu-nand-fcb.o
 obj-$(CONFIG_CONSOLE_RATP)	+= ratp.o
 obj-$(CONFIG_BOOT)		+= boot.o
 
-quiet_cmd_pwd_h = PWDH    $@
 ifdef CONFIG_PASSWORD
-ifneq ($(CONFIG_PASSWORD_DEFAULT),"")
-PASSWD_FILE := $(shell cd $(srctree); find $(CONFIG_PASSWORD_DEFAULT) -type f)
-cmd_pwd_h = echo -n "static const char default_passwd[] = \"" > $@; \
-	cat $< | tr -d '\n' >> $@; \
-	echo "\";" >> $@
 
-include/generated/passwd.h: $(PASSWD_FILE)
-	$(call if_changed,pwd_h)
+ifeq ($(CONFIG_PASSWORD_DEFAULT),"")
+define filechk_passwd
+	echo "static const char default_passwd[] = \"\";"
+endef
 else
-cmd_pwd_h = echo "static const char default_passwd[] = \"\";" > $@
-
-include/generated/passwd.h: FORCE
-	$(call if_changed,pwd_h)
+define filechk_passwd
+(							\
+echo -n "static const char default_passwd[] = \"";	\
+cat $(shell find $(srctree) -name $(CONFIG_PASSWORD_DEFAULT) -type f) | tr -d '\n';\
+echo "\";"						\
+)
+endef
 endif
 
+include/generated/passwd.h: $(srctree)/$(src)/Makefile FORCE
+	$(call filechk,passwd)
+
 targets += include/generated/passwd.h
 
 $(obj)/password.o: include/generated/passwd.h
-endif
+endif # CONFIG_PASSWORD
 
 # dependencies on generated files need to be listed explicitly
 $(obj)/version.o: include/generated/compile.h

Signature · 7 lines

2.12.0


_______________________________________________
barebox mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/barebox

Sam Ravnborg

author

2017-12-26 18:36 UTC #3 lore raw

PATCH 3/4 build: fix that the bbenv file is always rebuilt

In reply to Sam Ravnborg

From ac9ca6505d5b887c351117d9c033c8a76cc77125 Mon Sep 17 00:00:00 2001
From: Sam Ravnborg <[email protected]>
Date: Tue, 26 Dec 2017 18:05:14 +0100
Subject: [PATCH 3/4] build: fix that the bbenv file is always rebuilt

Use if_changed in rule for bbenv file. This avoids re-builds.
The target is already assigned to extra-y - so the kbuild logic will work as expected.

Signed-off-by Sam Ravnborg <[email protected]>

---
 scripts/Makefile.lib | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

scripts/Makefile.lib +1 -1

index b084bacab..d202a7299 100644
@@ -292,7 +292,7 @@ quiet_cmd_env = ENV     $@
 cmd_env=$(srctree)/scripts/genenv $(srctree) $(objtree) $@ $<
 
 %.bbenv$(DEFAULT_COMPRESSION_SUFFIX): % FORCE
-	$(call cmd,env)
+	$(call if_changed,env)
 
 # Bzip2
 # ---------------------------------------------------------------------------

Signature · 7 lines

2.12.0


_______________________________________________
barebox mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/barebox

Sam Ravnborg

author

2017-12-26 18:37 UTC #4 lore raw

PATCH 4/4 build: fix that LZO file is always rebuilt

In reply to Sam Ravnborg

From e398a00f84db33ea5ae7f6ee12c54511ef7a94fc Mon Sep 17 00:00:00 2001
From: Sam Ravnborg <[email protected]>
Date: Tue, 26 Dec 2017 18:09:35 +0100
Subject: [PATCH 4/4] build: fix that LZO file is always rebuilt

Port the make-cmd from linux kernel.
with the updated version $$ is porperly escaped, thus the LZO rule works as intended.
And we avoid rebuilds when not required

Signed-off-by Sam Ravnborg <[email protected]>

---
 scripts/Kbuild.include | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

scripts/Kbuild.include +7 -5

index 310dcdc94..fb3fa88fb 100644
@@ -205,11 +205,13 @@ else
 arg-check = $(if $(strip $(cmd_$@)),,1)
 endif
 
-# >'< substitution is for echo to work,
-# >$< substitution to preserve $ when reloading .cmd file
-# note: when using inline perl scripts [perl -e '...$$t=1;...']
-# in $(cmd_xxx) double $$ your perl vars
-make-cmd = $(subst \\,\\\\,$(subst \#,\\\#,$(subst $$,$$$$,$(call escsq,$(cmd_$(1))))))
+# Replace >$< with >$$< to preserve $ when reloading the .cmd file
+# (needed for make)
+# Replace >#< with >\#< to avoid starting a comment in the .cmd file
+# (needed for make)
+# Replace >'< with >'\''< to be able to enclose the whole string in '...'
+# (needed for the shell)
+make-cmd = $(call escsq,$(subst \#,\\\#,$(subst $$,$$$$,$(cmd_$(1)))))
 
 # Find any prerequisites that is newer than target or that does not exist.
 # PHONY targets skipped in both cases.

Signature · 7 lines

2.12.0


_______________________________________________
barebox mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/barebox

Sascha Hauer

2018-01-05 10:50 UTC #5 lore raw

Re: [PATCH 4/4] build: fix that LZO file is always rebuilt

Hi Sam,

Quoted 26 lines · Sam Ravnborg

From e398a00f84db33ea5ae7f6ee12c54511ef7a94fc Mon Sep 17 00:00:00 2001
From: Sam Ravnborg <[email protected]>
Date: Tue, 26 Dec 2017 18:09:35 +0100
Subject: [PATCH 4/4] build: fix that LZO file is always rebuilt
Port the make-cmd from linux kernel.
with the updated version $$ is porperly escaped, thus the LZO rule works as intended.
And we avoid rebuilds when not required
Signed-off-by Sam Ravnborg <[email protected]>
---
 scripts/Kbuild.include | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)
scripts/Kbuild.include +0 -3
index 310dcdc94..fb3fa88fb 100644
@@ -205,11 +205,13 @@ else
 arg-check = $(if $(strip $(cmd_$@)),,1)
 endif
 
-# >'< substitution is for echo to work,
-# >$< substitution to preserve $ when reloading .cmd file
-# note: when using inline perl scripts [perl -e '...$$t=1;...']

scripts/Kbuild.include else · line 215

Quoted 10 lines

scripts/Kbuild.include +7 -2

-# in $(cmd_xxx) double $$ your perl vars
-make-cmd = $(subst \\,\\\\,$(subst \#,\\\#,$(subst $$,$$$$,$(call escsq,$(cmd_$(1))))))
+# Replace >$< with >$$< to preserve $ when reloading the .cmd file
+# (needed for make)
+# Replace >#< with >\#< to avoid starting a comment in the .cmd file
+# (needed for make)
+# Replace >'< with >'\''< to be able to enclose the whole string in '...'
+# (needed for the shell)
+make-cmd = $(call escsq,$(subst \#,\\\#,$(subst $$,$$$$,$(cmd_$(1)))))

This doesn't work for me. Without this patch I do not see any rebuilds of lzo files, but with this patch I do.

Seems you are using another shell, dash vs. bash? Porting the second hunk of the kernel commit 164f0d2efaaef83 fixes it for me.

Sascha

Signature · 9 lines

Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

_______________________________________________
barebox mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/barebox

Sascha Hauer

2018-01-05 14:57 UTC #6 lore raw

Re: [PATCH 4/4] build: fix that LZO file is always rebuilt

Quoted 43 lines · Sascha Hauer

Hi Sam,
Quoted 26 lines · Sam Ravnborg
From e398a00f84db33ea5ae7f6ee12c54511ef7a94fc Mon Sep 17 00:00:00 2001
From: Sam Ravnborg <[email protected]>
Date: Tue, 26 Dec 2017 18:09:35 +0100
Subject: [PATCH 4/4] build: fix that LZO file is always rebuilt
Port the make-cmd from linux kernel.
with the updated version $$ is porperly escaped, thus the LZO rule works as intended.
And we avoid rebuilds when not required
Signed-off-by Sam Ravnborg <[email protected]>
---
 scripts/Kbuild.include | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)
scripts/Kbuild.include +0 -3
index 310dcdc94..fb3fa88fb 100644
@@ -205,11 +205,13 @@ else
 arg-check = $(if $(strip $(cmd_$@)),,1)
 endif
 
-# >'< substitution is for echo to work,
-# >$< substitution to preserve $ when reloading .cmd file
-# note: when using inline perl scripts [perl -e '...$$t=1;...']
scripts/Kbuild.include else · line 215
Quoted 10 lines
scripts/Kbuild.include +7 -2
-# in $(cmd_xxx) double $$ your perl vars
-make-cmd = $(subst \\,\\\\,$(subst \#,\\\#,$(subst $$,$$$$,$(call escsq,$(cmd_$(1))))))
+# Replace >$< with >$$< to preserve $ when reloading the .cmd file
+# (needed for make)
+# Replace >#< with >\#< to avoid starting a comment in the .cmd file
+# (needed for make)
+# Replace >'< with >'\''< to be able to enclose the whole string in '...'
+# (needed for the shell)
+make-cmd = $(call escsq,$(subst \#,\\\#,$(subst $$,$$$$,$(cmd_$(1)))))
 
This doesn't work for me. Without this patch I do not see any rebuilds of lzo files, but with this patch I do.

Quoted 2 lines

Seems you are using another shell, dash vs. bash? Porting the second hunk of the kernel commit 164f0d2efaaef83 fixes it for me.

Ok, applied with the missing hunk from 164f0d2efaaef83:

@@ -220,7 +222,7 @@ any-prereq = $(filter-out $(PHONY),$?) $(filter-out $(PHONY) $(wildcard $^),$^)

 if_changed = $(if $(strip $(any-prereq) $(arg-check)),                       \
        @set -e;                                                             \
        $(echo-cmd) $(cmd_$(1));                                             \

- echo 'cmd_$@ := $(make-cmd)' > $(dot-target).cmd)
+ printf '%s\n' 'cmd_$@ := $(make-cmd)' > $(dot-target).cmd)

Please let me know if I have overlooked something.

Sascha

Signature · 9 lines

Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

_______________________________________________
barebox mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/barebox

Sam Ravnborg

author

2018-01-06 07:35 UTC #7 lore raw

Re: [PATCH 4/4] build: fix that LZO file is always rebuilt

Hi Sascha.

Quoted 9 lines

Quoted 9 lines
scripts/Kbuild.include else · line 214
Quoted 3 lines
scripts/Kbuild.include +2 -0
+# (needed for the shell)
+make-cmd = $(call escsq,$(subst \#,\\\#,$(subst $$,$$$$,$(cmd_$(1)))))
This doesn't work for me. Without this patch I do not see any rebuilds of lzo files, but with this patch I do.
Seems you are using another shell, dash vs. bash? Porting the second hunk of the kernel commit 164f0d2efaaef83 fixes it for me.

$ sh --version
GNU bash, version 4.2.46(1)-release (x86_64-redhat-linux-gnu)

This is a CentOS based system.

Review any-prereq = $(filter-out $(PHONY),$?) $(filter-out $(PHONY) $(wildcard $^),$^) · line 226

Quoted 10 lines

Ok, applied with the missing hunk from 164f0d2efaaef83:

(quoted diff) +1 -1

@@ -220,7 +222,7 @@ any-prereq = $(filter-out $(PHONY),$?) $(filter-out $(PHONY) $(wildcard $^),$^)
 if_changed = $(if $(strip $(any-prereq) $(arg-check)),                       \
        @set -e;                                                             \
        $(echo-cmd) $(cmd_$(1));                                             \
-       echo 'cmd_$@ := $(make-cmd)' > $(dot-target).cmd)
+       printf '%s\n' 'cmd_$@ := $(make-cmd)' > $(dot-target).cmd)

The above snip have no impact on my system - which is why I left it out. To try to do minimal changes only.
I should have mentioned this - as well as the specific reference to the kernel commit.

but in other words - tested with the above change. OK.

Sam

_______________________________________________
barebox mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/barebox