Skip to contentSkip to conversation

bundles/pycaml/: force pycaml preparation to be done serially

PATCH In review Luis R. Rodriguez 4 participants 12 messages Jun 17, 2016 – Jun 21, 2016

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 bundles/pycaml/: force pycaml preparation to be done serially

This fixes parallel compilation when using 32 threads on a 32-core system:

The error:

make[6]: Entering directory '/home/mcgrof/coccinelle/bundles/pycaml' /usr/bin/ocamlopt.opt -I chemoelectric-pycaml-8614105 -cclib -lpycaml_stubs -ccopt -I/usr/include/python2.7 -ccopt -I/usr/include/x86_64-linux-gnu/python2.7 -cclib -lpython2.7 -c chemoelectric-pycaml-8614105/pycaml.ml -o chemoelectric-pycaml-8614105/pycaml.cmx
make[6]: *** No rule to make target 'chemoelectric-pycaml-8614105/pycaml.a', needed by 'all-opt-build-targets'. Stop.
make[6]: *** Waiting for unfinished jobs....
make[6]: Leaving directory '/home/mcgrof/coccinelle/bundles/pycaml' ../Makefile.bundles:33: recipe for target 'all.opt' failed make[5]: *** [all.opt] Error 2
make[5]: Leaving directory '/home/mcgrof/coccinelle/bundles/pycaml' Makefile:186: recipe for target '/home/mcgrof/coccinelle/bundles/pycaml/.opt' failed make[4]: *** [/home/mcgrof/coccinelle/bundles/pycaml/.opt] Error 2 make[4]: Leaving directory '/home/mcgrof/coccinelle' Makefile:180: recipe for target 'subdirs.opt' failed make[3]: *** [subdirs.opt] Error 1
make[3]: Leaving directory '/home/mcgrof/coccinelle' Makefile:163: recipe for target 'opt-compil' failed make[2]: *** [opt-compil] Error 2
make[2]: Leaving directory '/home/mcgrof/coccinelle' Makefile:143: recipe for target 'all-release' failed make[1]: *** [all-release] Error 2
make[1]: Leaving directory '/home/mcgrof/coccinelle' Makefile:121: recipe for target 'all' failed
make: *** [all] Error 2

---
 bundles/pycaml/Makefile | 2 ++
 1 file changed, 2 insertions(+)
bundles/pycaml/Makefile +2 -0
index 5045c27..4a619f4 100644
@@ -17,6 +17,8 @@ export PYMAJOR=$(shell echo ${PYTHON_VERSION} | sed -e 's/\..*//')
 
 include ../Makefile.bundles
 
+.NOTPARALLEL: .prepare
+
 .prepare: $(MARKER)
 	rm -f "$(SRC_DIR)/pycaml.mli"
 	$(CC) -E -w -D PYMAJOR$(PYMAJOR) -xc pycaml.pp.ml >$(SRC_DIR)/pycaml.ml
Signature · 1 line
2.8.1
#1 lore raw

PATCH 1/1 bundles/pycaml/: force pycaml preparation to be done serially

Quoted 45 lines · Luis R. Rodriguez

This fixes parallel compilation when using 32 threads on a 32-core system:

The error:

make[6]: Entering directory '/home/mcgrof/coccinelle/bundles/pycaml' /usr/bin/ocamlopt.opt -I chemoelectric-pycaml-8614105 -cclib

(quoted diff) +0 -2
-lpycaml_stubs -ccopt -I/usr/include/python2.7 -ccopt
-I/usr/include/x86_64-linux-gnu/python2.7 -cclib -lpython2.7 -c

chemoelectric-pycaml-8614105/pycaml.ml -o chemoelectric-pycaml-8614105/pycaml.cmx
make[6]: *** No rule to make target 'chemoelectric-pycaml-8614105/pycaml.a', needed by 'all-opt-build-targets'. Stop.
make[6]: *** Waiting for unfinished jobs....
make[6]: Leaving directory '/home/mcgrof/coccinelle/bundles/pycaml' ../Makefile.bundles:33: recipe for target 'all.opt' failed make[5]: *** [all.opt] Error 2
make[5]: Leaving directory '/home/mcgrof/coccinelle/bundles/pycaml' Makefile:186: recipe for target '/home/mcgrof/coccinelle/bundles/pycaml/.opt' failed make[4]: *** [/home/mcgrof/coccinelle/bundles/pycaml/.opt] Error 2 make[4]: Leaving directory '/home/mcgrof/coccinelle' Makefile:180: recipe for target 'subdirs.opt' failed make[3]: *** [subdirs.opt] Error 1
make[3]: Leaving directory '/home/mcgrof/coccinelle' Makefile:163: recipe for target 'opt-compil' failed make[2]: *** [opt-compil] Error 2
make[2]: Leaving directory '/home/mcgrof/coccinelle' Makefile:143: recipe for target 'all-release' failed make[1]: *** [all-release] Error 2
make[1]: Leaving directory '/home/mcgrof/coccinelle' Makefile:121: recipe for target 'all' failed
make: *** [all] Error 2

---
 bundles/pycaml/Makefile | 2 ++
 1 file changed, 2 insertions(+)
bundles/pycaml/Makefile +0 -0
index 5045c27..4a619f4 100644

bundles/pycaml/Makefile export PYMAJOR=$(shell echo ${PYTHON_VERSION} | sed -e 's/\..*//') · line 24

Quoted 8 lines
bundles/pycaml/Makefile +2 -0
@@ -17,6 +17,8 @@ export PYMAJOR=$(shell echo ${PYTHON_VERSION} | sed -e 's/\..*//')
 
 include ../Makefile.bundles
 
+.NOTPARALLEL: .prepare
+
 .prepare: $(MARKER)
 	rm -f "$(SRC_DIR)/pycaml.mli"
 	$(CC) -E -w -D PYMAJOR$(PYMAJOR) -xc pycaml.pp.ml >$(SRC_DIR)/pycaml.ml

Thanks for the patch. I'm not sure that it will address all possible issues, but if it is an improvement, then ok.

julia

#2 lore raw

PATCH 1/1 bundles/pycaml/: force pycaml preparation to be done serially

Quoted 52 lines · Julia Lawall
Quoted 48 lines · Luis R. Rodriguez

This fixes parallel compilation when using 32 threads on a 32-core system:

The error:

make[6]: Entering directory '/home/mcgrof/coccinelle/bundles/pycaml' /usr/bin/ocamlopt.opt -I chemoelectric-pycaml-8614105 -cclib

(quoted diff) +0 -2
-lpycaml_stubs -ccopt -I/usr/include/python2.7 -ccopt
-I/usr/include/x86_64-linux-gnu/python2.7 -cclib -lpython2.7 -c

chemoelectric-pycaml-8614105/pycaml.ml -o chemoelectric-pycaml-8614105/pycaml.cmx
make[6]: *** No rule to make target 'chemoelectric-pycaml-8614105/pycaml.a', needed by 'all-opt-build-targets'. Stop.
make[6]: *** Waiting for unfinished jobs....
make[6]: Leaving directory '/home/mcgrof/coccinelle/bundles/pycaml' ../Makefile.bundles:33: recipe for target 'all.opt' failed make[5]: *** [all.opt] Error 2
make[5]: Leaving directory '/home/mcgrof/coccinelle/bundles/pycaml' Makefile:186: recipe for target '/home/mcgrof/coccinelle/bundles/pycaml/.opt' failed make[4]: *** [/home/mcgrof/coccinelle/bundles/pycaml/.opt] Error 2 make[4]: Leaving directory '/home/mcgrof/coccinelle' Makefile:180: recipe for target 'subdirs.opt' failed make[3]: *** [subdirs.opt] Error 1
make[3]: Leaving directory '/home/mcgrof/coccinelle' Makefile:163: recipe for target 'opt-compil' failed make[2]: *** [opt-compil] Error 2
make[2]: Leaving directory '/home/mcgrof/coccinelle' Makefile:143: recipe for target 'all-release' failed make[1]: *** [all-release] Error 2
make[1]: Leaving directory '/home/mcgrof/coccinelle' Makefile:121: recipe for target 'all' failed
make: *** [all] Error 2

---
 bundles/pycaml/Makefile | 2 ++
 1 file changed, 2 insertions(+)
bundles/pycaml/Makefile +0 -0
index 5045c27..4a619f4 100644
@@ -17,6 +17,8 @@ export PYMAJOR=$(shell echo ${PYTHON_VERSION} | sed -e 's/\..*//')
 
 include ../Makefile.bundles
 
Quoted 8 lines

bundles/pycaml/Makefile export PYMAJOR=$(shell echo ${PYTHON_VERSION} | sed -e 's/\..*//') · line 24

Quoted 5 lines
bundles/pycaml/Makefile +2 -0
+.NOTPARALLEL: .prepare
+
 .prepare: $(MARKER)
 	rm -f "$(SRC_DIR)/pycaml.mli"
 	$(CC) -E -w -D PYMAJOR$(PYMAJOR) -xc pycaml.pp.ml >$(SRC_DIR)/pycaml.ml

Thanks for the patch. I'm not sure that it will address all possible issues,

Agreed.

Quoted 1 line

but if it is an improvement, then ok.

It makes parallel building work with a minimal change. The bigger hammer is to not trust any bundles and add +.NOTPARALLEL: all to the top level bundles/Makefile, that was my first approach, however it didn't seem like a good idea either as then you'd slow the bundle build generally.

Luis

#3 lore raw

PATCH 1/1 bundles/pycaml/: force pycaml preparation to be done serially

Quoted 64 lines · Luis R. Rodriguez
Quoted 52 lines · Julia Lawall
Quoted 48 lines · Luis R. Rodriguez

This fixes parallel compilation when using 32 threads on a 32-core system:

The error:

make[6]: Entering directory '/home/mcgrof/coccinelle/bundles/pycaml' /usr/bin/ocamlopt.opt -I chemoelectric-pycaml-8614105 -cclib

(quoted diff) +0 -2
-lpycaml_stubs -ccopt -I/usr/include/python2.7 -ccopt
-I/usr/include/x86_64-linux-gnu/python2.7 -cclib -lpython2.7 -c

chemoelectric-pycaml-8614105/pycaml.ml -o chemoelectric-pycaml-8614105/pycaml.cmx
make[6]: *** No rule to make target 'chemoelectric-pycaml-8614105/pycaml.a', needed by 'all-opt-build-targets'. Stop.
make[6]: *** Waiting for unfinished jobs....
make[6]: Leaving directory '/home/mcgrof/coccinelle/bundles/pycaml' ../Makefile.bundles:33: recipe for target 'all.opt' failed make[5]: *** [all.opt] Error 2
make[5]: Leaving directory '/home/mcgrof/coccinelle/bundles/pycaml' Makefile:186: recipe for target '/home/mcgrof/coccinelle/bundles/pycaml/.opt' failed make[4]: *** [/home/mcgrof/coccinelle/bundles/pycaml/.opt] Error 2 make[4]: Leaving directory '/home/mcgrof/coccinelle' Makefile:180: recipe for target 'subdirs.opt' failed make[3]: *** [subdirs.opt] Error 1
make[3]: Leaving directory '/home/mcgrof/coccinelle' Makefile:163: recipe for target 'opt-compil' failed make[2]: *** [opt-compil] Error 2
make[2]: Leaving directory '/home/mcgrof/coccinelle' Makefile:143: recipe for target 'all-release' failed make[1]: *** [all-release] Error 2
make[1]: Leaving directory '/home/mcgrof/coccinelle' Makefile:121: recipe for target 'all' failed
make: *** [all] Error 2

---
 bundles/pycaml/Makefile | 2 ++
 1 file changed, 2 insertions(+)
bundles/pycaml/Makefile +0 -0
index 5045c27..4a619f4 100644
@@ -17,6 +17,8 @@ export PYMAJOR=$(shell echo ${PYTHON_VERSION} | sed -e 's/\..*//')
 
 include ../Makefile.bundles
 
Quoted 8 lines

bundles/pycaml/Makefile export PYMAJOR=$(shell echo ${PYTHON_VERSION} | sed -e 's/\..*//') · line 24

Quoted 5 lines
bundles/pycaml/Makefile +2 -0
+.NOTPARALLEL: .prepare
+
 .prepare: $(MARKER)
 	rm -f "$(SRC_DIR)/pycaml.mli"
 	$(CC) -E -w -D PYMAJOR$(PYMAJOR) -xc pycaml.pp.ml >$(SRC_DIR)/pycaml.ml

Thanks for the patch. I'm not sure that it will address all possible issues,

Agreed.

Quoted 7 lines
Quoted 1 line

but if it is an improvement, then ok.

It makes parallel building work with a minimal change. The bigger hammer is to not trust any bundles and add +.NOTPARALLEL: all to the top level bundles/Makefile, that was my first approach, however it didn't seem like a good idea either as then you'd slow the bundle build generally.

I actually thing we also need .ONESHELL on all bundle Makefiles as well. This alone does not suffice to make it build correctly, however it seems this is correct. So the following patch seems more appropriate. Let me know.

Quoted 1 line

From 850cc1b16132d4984d1f4c61eea36472571db44b Mon Sep 17 00:00:00 2001

From: "Luis R. Rodriguez" <[email protected]>
Date: Fri, 17 Jun 2016 17:08:40 +0000
Subject: [PATCH] bundles/pycaml/: use .ONESHELL and .NOTPARALLEL on bundles

Bundles need their preparation to run in one shell [0], for pycaml we also need to disable parallel builds on the preparation. The .ONESHELL can be placed anywhere in the file.

[0] https://www.gnu.org/software/make/manual/make.html#Execution

This fixes parallel compilation when using 32 threads on a 32-core system:

The error:

make[6]: Entering directory '/home/mcgrof/coccinelle/bundles/pycaml' /usr/bin/ocamlopt.opt -I chemoelectric-pycaml-8614105 -cclib -lpycaml_stubs -ccopt -I/usr/include/python2.7 -ccopt -I/usr/include/x86_64-linux-gnu/python2.7 -cclib -lpython2.7 -c chemoelectric-pycaml-8614105/pycaml.ml -o chemoelectric-pycaml-8614105/pycaml.cmx
make[6]: *** No rule to make target 'chemoelectric-pycaml-8614105/pycaml.a', needed by 'all-opt-build-targets'. Stop.
make[6]: *** Waiting for unfinished jobs....
make[6]: Leaving directory '/home/mcgrof/coccinelle/bundles/pycaml' ../Makefile.bundles:33: recipe for target 'all.opt' failed make[5]: *** [all.opt] Error 2
make[5]: Leaving directory '/home/mcgrof/coccinelle/bundles/pycaml' Makefile:186: recipe for target '/home/mcgrof/coccinelle/bundles/pycaml/.opt' failed make[4]: *** [/home/mcgrof/coccinelle/bundles/pycaml/.opt] Error 2 make[4]: Leaving directory '/home/mcgrof/coccinelle' Makefile:180: recipe for target 'subdirs.opt' failed make[3]: *** [subdirs.opt] Error 1
make[3]: Leaving directory '/home/mcgrof/coccinelle' Makefile:163: recipe for target 'opt-compil' failed make[2]: *** [opt-compil] Error 2
make[2]: Leaving directory '/home/mcgrof/coccinelle' Makefile:143: recipe for target 'all-release' failed make[1]: *** [all-release] Error 2
make[1]: Leaving directory '/home/mcgrof/coccinelle' Makefile:121: recipe for target 'all' failed
make: *** [all] Error 2

---
 bundles/menhirLib/Makefile | 1 +
 bundles/parmap/Makefile    | 1 +
 bundles/pcre/Makefile      | 1 +
 bundles/pycaml/Makefile    | 2 ++
 4 files changed, 5 insertions(+)
bundles/menhirLib/Makefile +1 -0
index 23ac0d6..59b8fe0 100644
@@ -33,6 +33,7 @@ include ../Makefile.bundles
 depend: .depend
 	@$(MAKE) all-build
 
+.ONESHELL:
 .prepare: $(MARKER)
 	echo "let libdir = \"$(shell pwd\)\"" > $(SRC_DIR)/installation.ml
 	echo "let ocamlfind = false" >> $(SRC_DIR)/installation.ml
bundles/parmap/Makefile +1 -0
index 67c65ec..4436ba9 100644
@@ -17,6 +17,7 @@ OCAMLCCFLAGS+=-ccopt -D_GNU_SOURCE
 
 include ../Makefile.bundles
 
+.ONESHELL:
 .prepare: $(MARKER)
 	cd $(SRC_DIR) && ./configure
 	touch $@
bundles/pcre/Makefile +1 -0
index 97aa12e..9cfbf88 100644
@@ -24,6 +24,7 @@ EXTRALNKFLAGS=$(PCRE_LIBS:%=-ldopt %)
 
 include ../Makefile.bundles
 
+.ONESHELL:
 .prepare: $(MARKER)
 	if [ "$$OCAMLVERSION" "<" 4.02.0 ]; then \
 	  cp $(SRC_DIR)/pcre_compat312.ml $(SRC_DIR)/pcre_compat.ml; \
bundles/pycaml/Makefile +2 -0
index 5045c27..bded209 100644
@@ -17,6 +17,8 @@ export PYMAJOR=$(shell echo ${PYTHON_VERSION} | sed -e 's/\..*//')
 
 include ../Makefile.bundles
 
+.ONESHELL:
+.NOTPARALLEL: .prepare
 .prepare: $(MARKER)
 	rm -f "$(SRC_DIR)/pycaml.mli"
 	$(CC) -E -w -D PYMAJOR$(PYMAJOR) -xc pycaml.pp.ml >$(SRC_DIR)/pycaml.ml
Signature · 1 line
2.8.1
#4 lore raw

PATCH 1/1 bundles/pycaml/: force pycaml preparation to be done serially

Quoted 8 lines
Quoted 5 lines

It makes parallel building work with a minimal change. The bigger hammer is to not trust any bundles and add +.NOTPARALLEL: all to the top level bundles/Makefile, that was my first approach, however it didn't seem like a good idea either as then you'd slow the bundle build generally.

I actually thing we also need .ONESHELL on all bundle Makefiles as well. This alone does not suffice to make it build correctly, however it seems this is correct. So the following patch seems more appropriate.

I do not agree on the suggested combination.

Quoted 1 line

Bundles need their preparation to run in one shell [0],

I do not think so. - Such a setting can make sense for efficiency reasons.
https://www.gnu.org/software/make/manual/html_node/One-Shell.html

Quoted 1 line

for pycaml we also need to disable parallel builds on the preparation.

I can agree to this view.

Quoted 6 lines

include ../Makefile.bundles

+.ONESHELL:
+.NOTPARALLEL: .prepare
.prepare: $(MARKER)
rm -f "$(SRC_DIR)/pycaml.mli"

I find this change mixture questionable.

1. Would such an adjustment be cleaner by separate commits?

2. You try to express that only such a preparation should be performed in a serial way. It seems that your approach does not completely fit to the documentation.

https://www.gnu.org/software/make/manual/html_node/Special-Targets.html :
"?
If .NOTPARALLEL is mentioned as a target, then this invocation of
make will be run serially, even if the ?-j? option is given. ? Any prerequisites on this target are ignored.
?"

I would appreciate if other parts of the affected make scripts can
still be executed in parallel. Do you want to switch off parallelisation for the whole script so far?

Regards,
Markus

#5 lore raw

PATCH 1/1 bundles/pycaml/: force pycaml preparation to be done serially

Quoted 1 line · SF Markus Elfring
Quoted 1 line

+.ONESHELL:

This ended causing some loop on some system, the only consistent fix seems to be to use .NOTPARALLEL on all bundles.

Luis

#6 lore raw
Quoted 2 lines

?, the only consistent fix seems to be to use
.NOTPARALLEL on all bundles.

How do you think about a bit more fine-tuning for this setting?

Would you really like to switch parallelisation off for the affected make scripts completely?

Regards,
Markus

#7 lore raw
Quoted 6 lines · SF Markus Elfring
Quoted 2 lines

?, the only consistent fix seems to be to use
.NOTPARALLEL on all bundles.

How do you think about a bit more fine-tuning for this setting?

Would you really like to switch parallelisation off for the affected make scripts completely?

Building should work, period. Optimizations to building can be done, but since parallel building is brokena and something fixes it, that should be consdered first as it at least fixes general parallel building. The disabling of parallel building here is just for the prepare target.

Luis

#8 lore raw
Quoted 1 line

Building should work, period.

I would appreciate this also.

Quoted 1 line

The disabling of parallel building here is just for the prepare target.

I have got the understanding from the make documentation that the discussed setting does not work in the way you seem to expect here.
I suggest therefore to consider additional possibilities.

Regards,
Markus

#9 lore raw
Quoted 1 line · SF Markus Elfring

I suggest therefore to consider additional possibilities.

Patches welcomed.

Luis

#10 lore raw
Quoted 1 line

Patches welcomed.

Where do you want to switch parallelisation off exactly?

1. Whole make script

or

2. Treat only a specific preparation target specially.

Regards,
Markus

#11 lore raw

[Cocci] Further fine-tuning for parallel software builds?

In reply to Luis R. Rodriguez

bundles/pycaml/Makefile export PYMAJOR=$(shell echo ${PYTHON_VERSION} | sed -e 's/\..*//') · line 23

Quoted 6 lines

include ../Makefile.bundles

+.NOTPARALLEL: .prepare

bundles/pycaml/Makefile +1 -0
+
 .prepare: $(MARKER)
 	rm -f "$(SRC_DIR)/pycaml.mli"

Do you eventually need to consider further preparation for several processing stages?
Will any more dependencies matter between special software build targets?

Regards,
Markus