Qt5 parallel build of qmake
Hi all,
I've put a few people in Cc that seem to be active users of Qt5.
I noticed during a test build that during the configure step, qmake is built
but not in parallel. I think it should be possible to enable parallel build for qmake by passing MAKE="$(MAKE)" in the environment during configuration. Would someone be interested in checking that out?
Ideally, you should also test with different make versions because they
sometimes throw other stuff in the MAKE variable which may screw things up.
Regards,
Arnout
Hi Arnout, All,
Il 18/04/2019 00:16, Arnout Vandecappelle ha scritto:
Quoted 8 lines
Hi all,
I've put a few people in Cc that seem to be active users of Qt5.
I noticed during a test build that during the configure step, qmake is built
but not in parallel. I think it should be possible to enable parallel build for qmake by passing MAKE="$(MAKE)" in the environment during configuration. Would someone be interested in checking that out?
I am interested. I'll try to have it working, let's see if I'm able.
Quoted 2 lines
Ideally, you should also test with different make versions because they
sometimes throw other stuff in the MAKE variable which may screw things up.
How much back should I go with make versions? I mean, now I have 4.1, maybe back to 3.82? Or more?
Best regards
Signature · 10 lines
Giulio Benetti CTO MICRONOVA SRL Sede: Via A. Niedda 3 - 35010 Vigonza (PD) Tel. 049/8931563 - Fax 049/8931346 Cod.Fiscale - P.IVA 02663420285 Capitale Sociale ? 26.000 i.v. Iscritta al Reg. Imprese di Padova N. 02663420285 Numero R.E.A. 258642
Quoted 3 lines
Regards,
Arnout
Quoted 12 lines · Giulio Benetti
Hi Arnout, All,
Il 18/04/2019 00:16, Arnout Vandecappelle ha scritto:
Quoted 8 lines
? Hi all,
? I've put a few people in Cc that seem to be active users of Qt5.
? I noticed during a test build that during the configure step, qmake is built but not in parallel. I think it should be possible to enable parallel build for qmake by passing MAKE="$(MAKE)" in the environment during configuration. Would someone be interested in checking that out?
Quoted 8 lines
I am interested. I'll try to have it working, let's see if I'm able.
Quoted 2 lines
? Ideally, you should also test with different make versions because they sometimes throw other stuff in the MAKE variable which may screw things up.
How much back should I go with make versions?
I mean, now I have 4.1, maybe back to 3.82?
Or more?
Just 4.1 and 3.82 should be fine.
Regards,
Arnout
Hello Arnout, All,
Il 21/04/2019 09:08, Arnout Vandecappelle ha scritto:
Quoted 14 lines · Giulio Benetti
Quoted 11 lines
Hi Arnout, All,
Il 18/04/2019 00:16, Arnout Vandecappelle ha scritto:
Quoted 8 lines
? Hi all,
? I've put a few people in Cc that seem to be active users of Qt5.
? I noticed during a test build that during the configure step, qmake is built
but not in parallel. I think it should be possible to enable parallel build for qmake by passing MAKE="$(MAKE)" in the environment during configuration. Would someone be interested in checking that out?
I've found out which is the problem and where is its origin.
On qt5base(and every other package using MAKEFLAGS) this:
MAKEFLAGS="$(MAKEFLAGS) -j$(PARALLEL_JOBS)" expands in this:
MAKEFLAGS=" --no-print-directory -- -j5"
so -j5 is ignored, then is not built in parallel.
In:
https://git.buildroot.net/buildroot/tree/support/scripts/mkmakefile#n33
basic Makefile is created with a dashed argument(-C to pass Buildroot Root Makefile as argument), but then also O=... option is passed.
Here is a snippet of my Makefile file created by mkmakefile script:
...
MAKEARGS := -C /home/giuliobenetti/git/buildroot MAKEARGS += O=$(if $(patsubst /%,,$(makedir)),$(CURDIR)/)$(patsubst %/,%,$(makedir))
MAKEFLAGS += --no-print-directory
.PHONY: _all $(MAKECMDGOALS)
all := $(filter-out Makefile,$(MAKECMDGOALS))
_all:
echo Root MAKEARGS=$(MAKEARGS) echo Root MAKEFLAGS=$(MAKEFLAGS) $(Q)umask 0022 && $(MAKE) $(MAKEARGS) $(all)
...
Doing lot of tries I've found out(I didn't know) that "make", when evaluates $(MAKEARGS) it works as expected and gives:
Root MAKEARGS=-C /home/giuliobenetti/git/buildroot O=/home/giuliobenetti/git/build-br/.
BUT, when printing MAKEFLAGS with echo, it gives:
Root MAKEFLAGS= --no-print-directory --
qt5base and every other package using $(MAKEFLAGS) under a *recipe* only will always evaluate $(MAKEFLAGS) as "--".
Indeed if you try to use $(info $(MAKEFLAGS)) it will be different, without showing the 2 dashes.
Going deeper I've found that those 2 dashes are appended only if you pass to MAKEARGS some variable like "O=/home/..."
I mean, if you remove "O=/home/..." then MAKEFLAGS is "--no-print-directory" only.
The point is that MAKEARGS expects Makefile arguments only and not Variable as I understand. When you add variables, a "--" is appended between Make flags and Variables.
One possible solution I thought is cleaning MAKEFLAGS from those isolated 2 dashes.
Before I thought to pass directly O=/home/.. when calling $(MAKE), but the problems comes out again, since in sub-makefile, that will be a MAKEARGS again.
I don't see any different way to deny Make to emit "--".
This impacts MAKEFLAGS variable everywhere when evaluated under recipe like "echo $(MAKEFLAGS)" and it works in this way for both make-4.1 and make-3.82 too.
What do you think?
Happy easter and kind regards!
Signature · 10 lines
Giulio Benetti CTO MICRONOVA SRL Sede: Via A. Niedda 3 - 35010 Vigonza (PD) Tel. 049/8931563 - Fax 049/8931346 Cod.Fiscale - P.IVA 02663420285 Capitale Sociale ? 26.000 i.v. Iscritta al Reg. Imprese di Padova N. 02663420285 Numero R.E.A. 258642
Quoted 14 lines
Quoted 8 lines
I am interested. I'll try to have it working, let's see if I'm able.
Quoted 2 lines
? Ideally, you should also test with different make versions because they
sometimes throw other stuff in the MAKE variable which may screw things up.How much back should I go with make versions?
I mean, now I have 4.1, maybe back to 3.82?
Or more?Just 4.1 and 3.82 should be fine.
Regards,
Arnout
Quoted 20 lines · Giulio Benetti
Hello Arnout, All,
Il 21/04/2019 09:08, Arnout Vandecappelle ha scritto:
Quoted 14 lines · Giulio Benetti
Quoted 11 lines
Hi Arnout, All,
Il 18/04/2019 00:16, Arnout Vandecappelle ha scritto:
Quoted 8 lines
?? Hi all,
?? I've put a few people in Cc that seem to be active users of Qt5.
?? I noticed during a test build that during the configure step, qmake is built but not in parallel. I think it should be possible to enable parallel build for qmake by passing MAKE="$(MAKE)" in the environment during configuration. Would someone be interested in checking that out?
I've found out which is the problem and where is its origin.
Quoted 8 lines
On qt5base(and every other package using MAKEFLAGS) this:
MAKEFLAGS="$(MAKEFLAGS) -j$(PARALLEL_JOBS)" expands in this:
MAKEFLAGS=" --no-print-directory -- -j5"
so -j5 is ignored, then is not built in parallel.In:
https://git.buildroot.net/buildroot/tree/support/scripts/mkmakefile#n33
Note that I observed the same issue without out-of-tree build, and anyway if I
do out-of-tree build I always use explicit O= rather than the generated makefile. So the problem is not (only) due to mkmakefile.
Quoted 9 lines
basic
Makefileis created with a dashed argument(-C to pass Buildroot RootMakefileas argument), but then also O=... option is passed.Here is a snippet of my
Makefilefile created by mkmakefile script:
...
MAKEARGS := -C/home/giuliobenetti/git/buildrootMAKEARGS += O=$(if $(patsubst /%,,$(makedir)),$(CURDIR)/)$(patsubst %/,%,$(makedir))
Note that MAKEARGS is not a special variable for make...
Quoted 2 lines
MAKEFLAGS += --no-print-directory
But MAKEFLAGS is. However, the MAKEFLAGS variable is always rebuilt by the new
make process by combining the environment variable MAKEFLAGS with the command line options that are passed to it.
Quoted 15 lines
.PHONY:
_all$(MAKECMDGOALS)all??? := $(filter-out Makefile,$(MAKECMDGOALS))
_all:
????echo Root MAKEARGS=$(MAKEARGS)
????echo Root MAKEFLAGS=$(MAKEFLAGS)
????$(Q)umask 0022 && $(MAKE) $(MAKEARGS) $(all)
...Doing lot of tries I've found out(I didn't know) that "make", when evaluates $(MAKEARGS) it works as expected and gives:
Root MAKEARGS=-C/home/giuliobenetti/git/buildrootO=/home/giuliobenetti/git/build-br/.
Quoted 7 lines
BUT, when printing MAKEFLAGS with echo, it gives:
Root MAKEFLAGS= --no-print-directory --qt5base and every other package using $(MAKEFLAGS) under a *recipe* only will always evaluate $(MAKEFLAGS) as "--".
Indeed if you try to use $(info $(MAKEFLAGS)) it will be different, without showing the 2 dashes.
The difference between echo and $(info) is that echo is in a subshell and info
is not. So apparently, make doesn't just manipulate $(MAKEFLAGS) when it starts, but also when it forks a subprocess.
Quoted 3 lines
Going deeper I've found that those 2 dashes are appended only if you pass to MAKEARGS some variable like "O=/home/..."
I mean, if you remove "O=/home/..." then MAKEFLAGS is "--no-print-directory" only.
That is ...weird... because you do the echo in the outer make, which doesn't
know that MAKEARGS has any relation with MAKEFLAGS...
Quoted 3 lines
The point is that MAKEARGS expects
Makefilearguments only and not Variable as I understand. When you add variables, a "--" is appended between Make flags and Variables.
Yes, when make starts, it reworks things so that the variables go into
MAKEOVERRIDES.
Quoted 1 line
One possible solution I thought is cleaning MAKEFLAGS from those isolated 2 dashes.
I don't think that that's possible, because they're not really there.
Quoted 9 lines
Before I thought to pass directly O=/home/.. when calling $(MAKE), but the problems comes out again, since in sub-makefile, that will be a MAKEARGS again.
I don't see any different way to deny Make to emit "--".This impacts MAKEFLAGS variable everywhere when evaluated under recipe like "echo $(MAKEFLAGS)" and it works in this way for both make-4.1 and make-3.82 too.
What do you think?
What about swapping the -j and the MAKEFLAGS in qt5base? I.e.
MAKEFLAGS="-j$(PARALLEL_JOBS) $(MAKEFLAGS)"
Regards,
Arnout
Quoted 2 lines
Happy easter and kind regards!
Hello Arnout,
Il 22/04/2019 09:56, Arnout Vandecappelle ha scritto:
Quoted 74 lines · Giulio Benetti
Quoted 20 lines
Hello Arnout, All,
Il 21/04/2019 09:08, Arnout Vandecappelle ha scritto:
Quoted 14 lines · Giulio Benetti
Quoted 11 lines
Hi Arnout, All,
Il 18/04/2019 00:16, Arnout Vandecappelle ha scritto:
Quoted 8 lines
?? Hi all,
?? I've put a few people in Cc that seem to be active users of Qt5.
?? I noticed during a test build that during the configure step, qmake is built
but not in parallel. I think it should be possible to enable parallel build for qmake by passing MAKE="$(MAKE)" in the environment during configuration. Would someone be interested in checking that out?I've found out which is the problem and where is its origin.
Quoted 8 lines
On qt5base(and every other package using MAKEFLAGS) this:
MAKEFLAGS="$(MAKEFLAGS) -j$(PARALLEL_JOBS)" expands in this:
MAKEFLAGS=" --no-print-directory -- -j5"
so -j5 is ignored, then is not built in parallel.In:
https://git.buildroot.net/buildroot/tree/support/scripts/mkmakefile#n33Note that I observed the same issue without out-of-tree build, and anyway if I
do out-of-tree build I always use explicit O= rather than the generated makefile. So the problem is not (only) due to mkmakefile.Quoted 9 lines
basic
Makefileis created with a dashed argument(-C to pass Buildroot RootMakefileas argument), but then also O=... option is passed.Here is a snippet of my
Makefilefile created by mkmakefile script:
...
MAKEARGS := -C/home/giuliobenetti/git/buildrootMAKEARGS += O=$(if $(patsubst /%,,$(makedir)),$(CURDIR)/)$(patsubst %/,%,$(makedir))Note that MAKEARGS is not a special variable for make...
Quoted 2 lines
MAKEFLAGS += --no-print-directory
But MAKEFLAGS is. However, the MAKEFLAGS variable is always rebuilt by the new
make process by combining the environment variable MAKEFLAGS with the command line options that are passed to it.Quoted 18 lines
.PHONY:
_all$(MAKECMDGOALS)all??? := $(filter-out Makefile,$(MAKECMDGOALS))
_all:
????echo Root MAKEARGS=$(MAKEARGS)
????echo Root MAKEFLAGS=$(MAKEFLAGS)
????$(Q)umask 0022 && $(MAKE) $(MAKEARGS) $(all)
...Doing lot of tries I've found out(I didn't know) that "make", when evaluates $(MAKEARGS) it works as expected and gives:
Root MAKEARGS=-C/home/giuliobenetti/git/buildrootO=/home/giuliobenetti/git/build-br/.BUT, when printing MAKEFLAGS with echo, it gives:
Root MAKEFLAGS= --no-print-directory --
Quoted 8 lines
Quoted 4 lines
qt5base and every other package using $(MAKEFLAGS) under a *recipe* only will always evaluate $(MAKEFLAGS) as "--".
Indeed if you try to use $(info $(MAKEFLAGS)) it will be different, without showing the 2 dashes.The difference between echo and $(info) is that echo is in a subshell and info
is not. So apparently, make doesn't just manipulate $(MAKEFLAGS) when it starts, but also when it forks a subprocess.
Yes, this is what I've noticed.
Quoted 7 lines
Quoted 4 lines
Going deeper I've found that those 2 dashes are appended only if you pass to MAKEARGS some variable like "O=/home/..."
I mean, if you remove "O=/home/..." then MAKEFLAGS is "--no-print-directory" only.That is ...weird... because you do the echo in the outer make, which doesn't
know that MAKEARGS has any relation with MAKEFLAGS...
Right, the point is that MAKEARGS is passed to $(MAKE) as:
$(MAKE) $(MAKEARGS) $(all)
Quoted 18 lines
Quoted 4 lines
The point is that MAKEARGS expects
Makefilearguments only and not Variable as I understand. When you add variables, a "--" is appended between Make flags and Variables.Yes, when make starts, it reworks things so that the variables go into
MAKEOVERRIDES.Quoted 1 line
One possible solution I thought is cleaning MAKEFLAGS from those isolated 2 dashes.
I don't think that that's possible, because they're not really there.
Quoted 5 lines
Before I thought to pass directly O=/home/.. when calling $(MAKE), but the problems comes out again, since in sub-makefile, that will be a MAKEARGS again.
I don't see any different way to deny Make to emit "--".
Quoted 8 lines
Quoted 4 lines
This impacts MAKEFLAGS variable everywhere when evaluated under recipe like "echo $(MAKEFLAGS)" and it works in this way for both make-4.1 and make-3.82 too.
What do you think?
What about swapping the -j and the MAKEFLAGS in qt5base? I.e.
MAKEFLAGS="-j$(PARALLEL_JOBS) $(MAKEFLAGS)"
That was my first idea, then trying to go deeper it took me out of the track.
Yes, this is the best solution.
In the beginning I thought it was Makefile introducing some "--" dirt, instead it is "make" that behaves that way when passing variables. Also, MAKEFLAGS is not used very frequently.
So I go with swapping MAKEFLAGS and PARALLEL_JOBS.
Thank you
Best regards
Signature · 10 lines
Giulio Benetti CTO MICRONOVA SRL Sede: Via A. Niedda 3 - 35010 Vigonza (PD) Tel. 049/8931563 - Fax 049/8931346 Cod.Fiscale - P.IVA 02663420285 Capitale Sociale ? 26.000 i.v. Iscritta al Reg. Imprese di Padova N. 02663420285 Numero R.E.A. 258642
Quoted 4 lines
Regards,
ArnoutQuoted 2 lines
Happy easter and kind regards!
PATCH 1/1 package/qt5base: fix qmake parallel build
When configuring qt5base, qmake is built, but it's not built in parallel mode. This is due to MAKEFLAGS having 2 dashes on its tail, so this:
MAKEFLAGS="$(MAKEFLAGS) -j$(PARALLEL_JOBS)"
expands in this(i.e. 5 njobs):
MAKEFLAGS="--no-print-directory -- -j5"
and -j5 gets ignored due to "--" preceeding -j5.
Double dashes are part of $(MAKEFLAGS) only when evaluated by shell.
Swap $(MAKEFLAGS) and -j$(PARALLEL_JOBS) to avoid having "--" before -j$(PARALLEL_JOBS), this way -j$(PARALLEL_JOBS) won't be ignored by ./configure.
--- package/qt5/qt5base/qt5base.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
package/qt5/qt5base/qt5base.mk +1 -1
@@ -324,7 +324,7 @@ define QT5BASE_CONFIGURE_CMDS (cd $(@D); \ $(TARGET_MAKE_ENV) \ PKG_CONFIG="$(PKG_CONFIG_HOST_BINARY)" \ - MAKEFLAGS="$(MAKEFLAGS) -j$(PARALLEL_JOBS)" \ + MAKEFLAGS="-j$(PARALLEL_JOBS) $(MAKEFLAGS)" \ ./configure \ -v \ -prefix /usr \
Signature · 1 line
2.17.1
PATCH 1/1 package/qt5base: fix qmake parallel build
Hi
Quoted 9 lines · Giulio Benetti
When configuring qt5base, qmake is built, but it's not built in parallel mode. This is due to MAKEFLAGS having 2 dashes on its tail, so this:
MAKEFLAGS="$(MAKEFLAGS) -j$(PARALLEL_JOBS)"
expands in this(i.e. 5 njobs):
MAKEFLAGS="--no-print-directory -- -j5"
and -j5 gets ignored due to "--" preceeding -j5.
Double dashes are part of $(MAKEFLAGS) only when evaluated by shell.
Quoted 6 lines
Swap $(MAKEFLAGS) and -j$(PARALLEL_JOBS) to avoid having "--" before -j$(PARALLEL_JOBS), this way -j$(PARALLEL_JOBS) won't be ignored by ./configure.
Signed-off-by Giulio Benetti <[email protected]>---
Thank you
Michael
Quoted 23 lines
package/qt5/qt5base/qt5base.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)package/qt5/qt5base/qt5base.mk +1
-2@@ -324,7 +324,7 @@ define QT5BASE_CONFIGURE_CMDS (cd $(@D); \ $(TARGET_MAKE_ENV) \ PKG_CONFIG="$(PKG_CONFIG_HOST_BINARY)" \ - MAKEFLAGS="$(MAKEFLAGS) -j$(PARALLEL_JOBS)" \ + MAKEFLAGS="-j$(PARALLEL_JOBS) $(MAKEFLAGS)" \ ./configure \ -v \ -prefix /usr \ --2.17.1
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 4 lines
| Michael Nazzareno Trimarchi Amarula Solutions BV | | COO - Founder Cruquiuskade 47 | | +31(0)851119172 Amsterdam 1018 AM NL | | [`as] http://www.amarulasolutions.com |
Thomas Petazzoni
AppliedPATCH 1/1 package/qt5base: fix qmake parallel build
In reply to Giulio BenettiQuoted 8 lines · On Mon, 22 Apr 2019 11:58:42 +0200 Giulio Benetti…
When configuring qt5base, qmake is built, but it's not built in parallel mode. This is due to MAKEFLAGS having 2 dashes on its tail, so this:
MAKEFLAGS="$(MAKEFLAGS) -j$(PARALLEL_JOBS)"
expands in this(i.e. 5 njobs):
MAKEFLAGS="--no-print-directory -- -j5"
and -j5 gets ignored due to "--" preceeding -j5.
Double dashes are part of $(MAKEFLAGS) only when evaluated by shell.
Quoted 8 lines
Swap $(MAKEFLAGS) and -j$(PARALLEL_JOBS) to avoid having "--" before -j$(PARALLEL_JOBS), this way -j$(PARALLEL_JOBS) won't be ignored by ./configure.
Signed-off-by Giulio Benetti <[email protected]>--- package/qt5/qt5base/qt5base.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Applied to master, thanks.
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com