bundles/pycaml/: force pycaml preparation to be done serially
PATCH 1/1 bundles/pycaml/: force pycaml preparation to be done serially
This fixes parallel compilation when using 32 threads on a 32-core system:
The error:
make[6]: Entering directory '/home/mcgrof/coccinelle/bundles/pycaml' /usr/bin/ocamlopt.opt -I chemoelectric-pycaml-8614105 -cclib -lpycaml_stubs -ccopt -I/usr/include/python2.7 -ccopt -I/usr/include/x86_64-linux-gnu/python2.7 -cclib -lpython2.7 -c chemoelectric-pycaml-8614105/pycaml.ml -o chemoelectric-pycaml-8614105/pycaml.cmx
make[6]: *** No rule to make target 'chemoelectric-pycaml-8614105/pycaml.a', needed by 'all-opt-build-targets'. Stop.
make[6]: *** Waiting for unfinished jobs....
make[6]: Leaving directory '/home/mcgrof/coccinelle/bundles/pycaml' ../Makefile.bundles:33: recipe for target 'all.opt' failed make[5]: *** [all.opt] Error 2
make[5]: Leaving directory '/home/mcgrof/coccinelle/bundles/pycaml' Makefile:186: recipe for target '/home/mcgrof/coccinelle/bundles/pycaml/.opt' failed make[4]: *** [/home/mcgrof/coccinelle/bundles/pycaml/.opt] Error 2 make[4]: Leaving directory '/home/mcgrof/coccinelle' Makefile:180: recipe for target 'subdirs.opt' failed make[3]: *** [subdirs.opt] Error 1
make[3]: Leaving directory '/home/mcgrof/coccinelle' Makefile:163: recipe for target 'opt-compil' failed make[2]: *** [opt-compil] Error 2
make[2]: Leaving directory '/home/mcgrof/coccinelle' Makefile:143: recipe for target 'all-release' failed make[1]: *** [all-release] Error 2
make[1]: Leaving directory '/home/mcgrof/coccinelle' Makefile:121: recipe for target 'all' failed
make: *** [all] Error 2
--- bundles/pycaml/Makefile | 2 ++ 1 file changed, 2 insertions(+)
bundles/pycaml/Makefile +2 -0
@@ -17,6 +17,8 @@ export PYMAJOR=$(shell echo ${PYTHON_VERSION} | sed -e 's/\..*//') include ../Makefile.bundles +.NOTPARALLEL: .prepare + .prepare: $(MARKER) rm -f "$(SRC_DIR)/pycaml.mli" $(CC) -E -w -D PYMAJOR$(PYMAJOR) -xc pycaml.pp.ml >$(SRC_DIR)/pycaml.ml
Signature · 1 line
2.8.1
PATCH 1/1 bundles/pycaml/: force pycaml preparation to be done serially
Quoted 45 lines · Luis R. Rodriguez
This fixes parallel compilation when using 32 threads on a 32-core system:
The error:
make[6]: Entering directory '
/home/mcgrof/coccinelle/bundles/pycaml'/usr/bin/ocamlopt.opt-I chemoelectric-pycaml-8614105 -cclib(quoted diff) +0
-2-lpycaml_stubs -ccopt -I/usr/include/python2.7 -ccopt -I/usr/include/x86_64-linux-gnu/python2.7 -cclib -lpython2.7 -cchemoelectric-pycaml-8614105/pycaml.ml -o chemoelectric-pycaml-8614105/pycaml.cmx
make[6]: *** No rule to make target 'chemoelectric-pycaml-8614105/pycaml.a', needed by 'all-opt-build-targets'. Stop.
make[6]: *** Waiting for unfinished jobs....
make[6]: Leaving directory '/home/mcgrof/coccinelle/bundles/pycaml' ../Makefile.bundles:33: recipe for target 'all.opt' failed make[5]: *** [all.opt] Error 2
make[5]: Leaving directory '/home/mcgrof/coccinelle/bundles/pycaml' Makefile:186: recipe for target '/home/mcgrof/coccinelle/bundles/pycaml/.opt' failed make[4]: *** [/home/mcgrof/coccinelle/bundles/pycaml/.opt] Error 2 make[4]: Leaving directory '/home/mcgrof/coccinelle' Makefile:180: recipe for target 'subdirs.opt' failed make[3]: *** [subdirs.opt] Error 1
make[3]: Leaving directory '/home/mcgrof/coccinelle' Makefile:163: recipe for target 'opt-compil' failed make[2]: *** [opt-compil] Error 2
make[2]: Leaving directory '/home/mcgrof/coccinelle' Makefile:143: recipe for target 'all-release' failed make[1]: *** [all-release] Error 2
make[1]: Leaving directory '/home/mcgrof/coccinelle' Makefile:121: recipe for target 'all' failed
make: *** [all] Error 2Signed-off-by Luis R. Rodriguez <[email protected]>--- bundles/pycaml/Makefile | 2 ++ 1 file changed, 2 insertions(+)bundles/pycaml/Makefile +0
-0
bundles/pycaml/Makefile export PYMAJOR=$(shell echo ${PYTHON_VERSION} | sed -e 's/\..*//') · line 24
Quoted 8 lines
bundles/pycaml/Makefile +2
-0@@ -17,6 +17,8 @@ export PYMAJOR=$(shell echo ${PYTHON_VERSION} | sed -e 's/\..*//') include ../Makefile.bundles +.NOTPARALLEL: .prepare + .prepare: $(MARKER) rm -f "$(SRC_DIR)/pycaml.mli" $(CC) -E -w -D PYMAJOR$(PYMAJOR) -xc pycaml.pp.ml >$(SRC_DIR)/pycaml.ml
Thanks for the patch. I'm not sure that it will address all possible issues, but if it is an improvement, then ok.
julia
PATCH 1/1 bundles/pycaml/: force pycaml preparation to be done serially
Quoted 52 lines · Julia Lawall
Quoted 48 lines · Luis R. Rodriguez
This fixes parallel compilation when using 32 threads on a 32-core system:
The error:
make[6]: Entering directory '
/home/mcgrof/coccinelle/bundles/pycaml'/usr/bin/ocamlopt.opt-I chemoelectric-pycaml-8614105 -cclib(quoted diff) +0
-2-lpycaml_stubs -ccopt -I/usr/include/python2.7 -ccopt -I/usr/include/x86_64-linux-gnu/python2.7 -cclib -lpython2.7 -cchemoelectric-pycaml-8614105/pycaml.ml -o chemoelectric-pycaml-8614105/pycaml.cmx
make[6]: *** No rule to make target 'chemoelectric-pycaml-8614105/pycaml.a', needed by 'all-opt-build-targets'. Stop.
make[6]: *** Waiting for unfinished jobs....
make[6]: Leaving directory '/home/mcgrof/coccinelle/bundles/pycaml' ../Makefile.bundles:33: recipe for target 'all.opt' failed make[5]: *** [all.opt] Error 2
make[5]: Leaving directory '/home/mcgrof/coccinelle/bundles/pycaml' Makefile:186: recipe for target '/home/mcgrof/coccinelle/bundles/pycaml/.opt' failed make[4]: *** [/home/mcgrof/coccinelle/bundles/pycaml/.opt] Error 2 make[4]: Leaving directory '/home/mcgrof/coccinelle' Makefile:180: recipe for target 'subdirs.opt' failed make[3]: *** [subdirs.opt] Error 1
make[3]: Leaving directory '/home/mcgrof/coccinelle' Makefile:163: recipe for target 'opt-compil' failed make[2]: *** [opt-compil] Error 2
make[2]: Leaving directory '/home/mcgrof/coccinelle' Makefile:143: recipe for target 'all-release' failed make[1]: *** [all-release] Error 2
make[1]: Leaving directory '/home/mcgrof/coccinelle' Makefile:121: recipe for target 'all' failed
make: *** [all] Error 2Signed-off-by Luis R. Rodriguez <[email protected]>--- bundles/pycaml/Makefile | 2 ++ 1 file changed, 2 insertions(+)bundles/pycaml/Makefile +0
-0@@ -17,6 +17,8 @@ export PYMAJOR=$(shell echo ${PYTHON_VERSION} | sed -e 's/\..*//') include ../Makefile.bundles
Quoted 8 lines
bundles/pycaml/Makefile export PYMAJOR=$(shell echo ${PYTHON_VERSION} | sed -e 's/\..*//') · line 24
Quoted 5 lines
bundles/pycaml/Makefile +2
-0+.NOTPARALLEL: .prepare + .prepare: $(MARKER) rm -f "$(SRC_DIR)/pycaml.mli" $(CC) -E -w -D PYMAJOR$(PYMAJOR) -xc pycaml.pp.ml >$(SRC_DIR)/pycaml.mlThanks for the patch. I'm not sure that it will address all possible issues,
Agreed.
Quoted 1 line
but if it is an improvement, then ok.
It makes parallel building work with a minimal change. The bigger hammer is to not trust any bundles and add +.NOTPARALLEL: all to the top level bundles/Makefile, that was my first approach, however it didn't seem like a good idea either as then you'd slow the bundle build generally.
Luis
PATCH 1/1 bundles/pycaml/: force pycaml preparation to be done serially
Quoted 64 lines · Luis R. Rodriguez
Quoted 52 lines · Julia Lawall
Quoted 48 lines · Luis R. Rodriguez
This fixes parallel compilation when using 32 threads on a 32-core system:
The error:
make[6]: Entering directory '
/home/mcgrof/coccinelle/bundles/pycaml'/usr/bin/ocamlopt.opt-I chemoelectric-pycaml-8614105 -cclib(quoted diff) +0
-2-lpycaml_stubs -ccopt -I/usr/include/python2.7 -ccopt -I/usr/include/x86_64-linux-gnu/python2.7 -cclib -lpython2.7 -cchemoelectric-pycaml-8614105/pycaml.ml -o chemoelectric-pycaml-8614105/pycaml.cmx
make[6]: *** No rule to make target 'chemoelectric-pycaml-8614105/pycaml.a', needed by 'all-opt-build-targets'. Stop.
make[6]: *** Waiting for unfinished jobs....
make[6]: Leaving directory '/home/mcgrof/coccinelle/bundles/pycaml' ../Makefile.bundles:33: recipe for target 'all.opt' failed make[5]: *** [all.opt] Error 2
make[5]: Leaving directory '/home/mcgrof/coccinelle/bundles/pycaml' Makefile:186: recipe for target '/home/mcgrof/coccinelle/bundles/pycaml/.opt' failed make[4]: *** [/home/mcgrof/coccinelle/bundles/pycaml/.opt] Error 2 make[4]: Leaving directory '/home/mcgrof/coccinelle' Makefile:180: recipe for target 'subdirs.opt' failed make[3]: *** [subdirs.opt] Error 1
make[3]: Leaving directory '/home/mcgrof/coccinelle' Makefile:163: recipe for target 'opt-compil' failed make[2]: *** [opt-compil] Error 2
make[2]: Leaving directory '/home/mcgrof/coccinelle' Makefile:143: recipe for target 'all-release' failed make[1]: *** [all-release] Error 2
make[1]: Leaving directory '/home/mcgrof/coccinelle' Makefile:121: recipe for target 'all' failed
make: *** [all] Error 2Signed-off-by Luis R. Rodriguez <[email protected]>--- bundles/pycaml/Makefile | 2 ++ 1 file changed, 2 insertions(+)bundles/pycaml/Makefile +0
-0@@ -17,6 +17,8 @@ export PYMAJOR=$(shell echo ${PYTHON_VERSION} | sed -e 's/\..*//') include ../Makefile.bundlesQuoted 8 lines
bundles/pycaml/Makefile export PYMAJOR=$(shell echo ${PYTHON_VERSION} | sed -e 's/\..*//') · line 24
Quoted 5 lines
bundles/pycaml/Makefile +2
-0+.NOTPARALLEL: .prepare + .prepare: $(MARKER) rm -f "$(SRC_DIR)/pycaml.mli" $(CC) -E -w -D PYMAJOR$(PYMAJOR) -xc pycaml.pp.ml >$(SRC_DIR)/pycaml.mlThanks for the patch. I'm not sure that it will address all possible issues,
Agreed.
Quoted 7 lines
Quoted 1 line
but if it is an improvement, then ok.
It makes parallel building work with a minimal change. The bigger hammer is to not trust any bundles and add +.NOTPARALLEL: all to the top level bundles/Makefile, that was my first approach, however it didn't seem like a good idea either as then you'd slow the bundle build generally.
I actually thing we also need .ONESHELL on all bundle Makefiles as well. This alone does not suffice to make it build correctly, however it seems this is correct. So the following patch seems more appropriate. Let me know.
Quoted 1 line
From
850cc1b16132d4984d1f4c61eea36472571db44bMon Sep 17 00:00:00 2001
From: "Luis R. Rodriguez" <[email protected]>
Date: Fri, 17 Jun 2016 17:08:40 +0000
Subject: [PATCH] bundles/pycaml/: use .ONESHELL and .NOTPARALLEL on bundles
Bundles need their preparation to run in one shell [0], for pycaml we also need to disable parallel builds on the preparation. The .ONESHELL can be placed anywhere in the file.
[0] https://www.gnu.org/software/make/manual/make.html#Execution
This fixes parallel compilation when using 32 threads on a 32-core system:
The error:
make[6]: Entering directory '/home/mcgrof/coccinelle/bundles/pycaml' /usr/bin/ocamlopt.opt -I chemoelectric-pycaml-8614105 -cclib -lpycaml_stubs -ccopt -I/usr/include/python2.7 -ccopt -I/usr/include/x86_64-linux-gnu/python2.7 -cclib -lpython2.7 -c chemoelectric-pycaml-8614105/pycaml.ml -o chemoelectric-pycaml-8614105/pycaml.cmx
make[6]: *** No rule to make target 'chemoelectric-pycaml-8614105/pycaml.a', needed by 'all-opt-build-targets'. Stop.
make[6]: *** Waiting for unfinished jobs....
make[6]: Leaving directory '/home/mcgrof/coccinelle/bundles/pycaml' ../Makefile.bundles:33: recipe for target 'all.opt' failed make[5]: *** [all.opt] Error 2
make[5]: Leaving directory '/home/mcgrof/coccinelle/bundles/pycaml' Makefile:186: recipe for target '/home/mcgrof/coccinelle/bundles/pycaml/.opt' failed make[4]: *** [/home/mcgrof/coccinelle/bundles/pycaml/.opt] Error 2 make[4]: Leaving directory '/home/mcgrof/coccinelle' Makefile:180: recipe for target 'subdirs.opt' failed make[3]: *** [subdirs.opt] Error 1
make[3]: Leaving directory '/home/mcgrof/coccinelle' Makefile:163: recipe for target 'opt-compil' failed make[2]: *** [opt-compil] Error 2
make[2]: Leaving directory '/home/mcgrof/coccinelle' Makefile:143: recipe for target 'all-release' failed make[1]: *** [all-release] Error 2
make[1]: Leaving directory '/home/mcgrof/coccinelle' Makefile:121: recipe for target 'all' failed
make: *** [all] Error 2
--- bundles/menhirLib/Makefile | 1 + bundles/parmap/Makefile | 1 + bundles/pcre/Makefile | 1 + bundles/pycaml/Makefile | 2 ++ 4 files changed, 5 insertions(+)
bundles/menhirLib/Makefile +1 -0
@@ -33,6 +33,7 @@ include ../Makefile.bundles depend: .depend @$(MAKE) all-build +.ONESHELL: .prepare: $(MARKER) echo "let libdir = \"$(shell pwd\)\"" > $(SRC_DIR)/installation.ml echo "let ocamlfind = false" >> $(SRC_DIR)/installation.ml
bundles/parmap/Makefile +1 -0
@@ -17,6 +17,7 @@ OCAMLCCFLAGS+=-ccopt -D_GNU_SOURCE include ../Makefile.bundles +.ONESHELL: .prepare: $(MARKER) cd $(SRC_DIR) && ./configure touch $@
bundles/pcre/Makefile +1 -0
@@ -24,6 +24,7 @@ EXTRALNKFLAGS=$(PCRE_LIBS:%=-ldopt %) include ../Makefile.bundles +.ONESHELL: .prepare: $(MARKER) if [ "$$OCAMLVERSION" "<" 4.02.0 ]; then \ cp $(SRC_DIR)/pcre_compat312.ml $(SRC_DIR)/pcre_compat.ml; \
bundles/pycaml/Makefile +2 -0
@@ -17,6 +17,8 @@ export PYMAJOR=$(shell echo ${PYTHON_VERSION} | sed -e 's/\..*//') include ../Makefile.bundles +.ONESHELL: +.NOTPARALLEL: .prepare .prepare: $(MARKER) rm -f "$(SRC_DIR)/pycaml.mli" $(CC) -E -w -D PYMAJOR$(PYMAJOR) -xc pycaml.pp.ml >$(SRC_DIR)/pycaml.ml
Signature · 1 line
2.8.1
PATCH 1/1 bundles/pycaml/: force pycaml preparation to be done serially
Quoted 8 lines
Quoted 5 lines
It makes parallel building work with a minimal change. The bigger hammer is to not trust any bundles and add +.NOTPARALLEL: all to the top level bundles/Makefile, that was my first approach, however it didn't seem like a good idea either as then you'd slow the bundle build generally.
I actually thing we also need .ONESHELL on all bundle Makefiles as well. This alone does not suffice to make it build correctly, however it seems this is correct. So the following patch seems more appropriate.
I do not agree on the suggested combination.
Quoted 1 line
Bundles need their preparation to run in one shell [0],
I do not think so. - Such a setting can make sense for efficiency reasons.
https://www.gnu.org/software/make/manual/html_node/One-Shell.html
Quoted 1 line
for pycaml we also need to disable parallel builds on the preparation.
I can agree to this view.
Quoted 6 lines
include ../Makefile.bundles
+.ONESHELL:
+.NOTPARALLEL: .prepare
.prepare: $(MARKER)
rm -f "$(SRC_DIR)/pycaml.mli"
I find this change mixture questionable.
1. Would such an adjustment be cleaner by separate commits?
2. You try to express that only such a preparation should be performed in a serial way. It seems that your approach does not completely fit to the documentation.
https://www.gnu.org/software/make/manual/html_node/Special-Targets.html :
"?
If .NOTPARALLEL is mentioned as a target, then this invocation of
make will be run serially, even if the ?-j? option is given. ? Any prerequisites on this target are ignored.
?"
I would appreciate if other parts of the affected make scripts can
still be executed in parallel. Do you want to switch off parallelisation for the whole script so far?
Regards,
Markus
PATCH 1/1 bundles/pycaml/: force pycaml preparation to be done serially
Quoted 1 line · SF Markus Elfring
Quoted 1 line
+.ONESHELL:
This ended causing some loop on some system, the only consistent fix seems to be to use .NOTPARALLEL on all bundles.
Luis
Quoted 2 lines
?, the only consistent fix seems to be to use
.NOTPARALLEL on all bundles.
How do you think about a bit more fine-tuning for this setting?
Would you really like to switch parallelisation off for the affected make scripts completely?
Regards,
Markus
Quoted 6 lines · SF Markus Elfring
Quoted 2 lines
?, the only consistent fix seems to be to use
.NOTPARALLEL on all bundles.How do you think about a bit more fine-tuning for this setting?
Would you really like to switch parallelisation off for the affected make scripts completely?
Building should work, period. Optimizations to building can be done, but since parallel building is brokena and something fixes it, that should be consdered first as it at least fixes general parallel building. The disabling of parallel building here is just for the prepare target.
Luis
Quoted 1 line
Building should work, period.
I would appreciate this also.
Quoted 1 line
The disabling of parallel building here is just for the prepare target.
I have got the understanding from the make documentation that the discussed setting does not work in the way you seem to expect here.
I suggest therefore to consider additional possibilities.
Regards,
Markus
Quoted 1 line · SF Markus Elfring
I suggest therefore to consider additional possibilities.
Patches welcomed.
Luis
Quoted 1 line
Patches welcomed.
Where do you want to switch parallelisation off exactly?
1. Whole make script
or
2. Treat only a specific preparation target specially.
Regards,
Markus
[Cocci] Further fine-tuning for parallel software builds?
In reply to Luis R. Rodriguezbundles/pycaml/Makefile export PYMAJOR=$(shell echo ${PYTHON_VERSION} | sed -e 's/\..*//') · line 23
Quoted 6 lines
include ../Makefile.bundles
+.NOTPARALLEL: .prepare
bundles/pycaml/Makefile +1
-0+ .prepare: $(MARKER) rm -f "$(SRC_DIR)/pycaml.mli"
Do you eventually need to consider further preparation for several processing stages?
Will any more dependencies matter between special software build targets?
Regards,
Markus