Fix make -j race due to stubdom and tools make -C
cover Fix make -j race due to stubdom and tools make -C
Ian Jackson (2):
build system: make install-stubdom depend on install-tools again DO NOT APPLY git-checkout.sh: add sleep to demonstrate race
Makefile | 2 +- scripts/git-checkout.sh | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-)
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH 1/2 build system: make install-stubdom depend on install-tools again
In d290e325179ccee966cd679d0fed48be6f4cc1b7
"build system: don't let install-stubdom depend on install-tools"
the dependency of install-stubdom on install-tools was removed.
However, this was not correct. stubdom/Makefile contains this:
$(XEN_ROOT)/tools/qemu-xen-traditional-dir:
$(MAKE) DESTDIR= -C $(XEN_ROOT)/tools qemu-xen-traditional-dir-findAs I have written before:
With recursive make, it is necessary for the overall structure of the makefiles to sequence things so that each directory is entered exactly once, before its dependent directories are entered. (It is possible to violate this rule without creating races but it is tricky and inadvisable.)
Since d290e325179c, it can happen that the command for the qemu-xen-traditional-dir-find rule is run twice simultaneously - once as a result of $(MAKE) -C tools install, and once as a result of $(MAKE) -C stubdom install. If you get unlucky, this causes lossage. (This just happened to me in an osstest flight.)
In principle we could alternatively fix this by lifting the commands in the qemu-xen-traditional-dir-find target (and perhaps other things too) into the toplevel Makefile, as was done for mini-os.
But that seems overkill given how bad the stubdom build system is, and the fact that we think at some point this qemu-trad will go away entirely. Adding the tools dependency back to the stubdom build is by and large good enough.
(Someone who really wants to build stubdom without tools is welcome to do this separation if they really want to.)
--- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Makefile +1 -1
@@ -127,7 +127,7 @@ install-tools: install-tools-public-headers $(MAKE) -C tools install .PHONY: install-stubdom -install-stubdom: mini-os-dir install-tools-public-headers +install-stubdom: mini-os-dir install-tools $(MAKE) -C stubdom install ifeq (x86_64,$(XEN_TARGET_ARCH)) XEN_TARGET_ARCH=x86_32 $(MAKE) -C stubdom install-grub
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 1/2] build system: make install-stubdom depend on install-tools again
Quoted 32 lines · Ian Jackson
In
d290e325179ccee966cd679d0fed48be6f4cc1b7
"build system: don't let install-stubdom depend on install-tools"
the dependency of install-stubdom on install-tools was removed.However, this was not correct. stubdom/Makefile contains this:
$(XEN_ROOT)/tools/qemu-xen-traditional-dir: $(MAKE) DESTDIR= -C $(XEN_ROOT)/tools qemu-xen-traditional-dir-findAs I have written before:
With recursive make, it is necessary for the overall structure of the makefiles to sequence things so that each directory is entered exactly once, before its dependent directories are entered. (It is possible to violate this rule without creating races but it is tricky and inadvisable.)
Since
d290e325179c, it can happen that the command for the qemu-xen-traditional-dir-find rule is run twice simultaneously - once as a result of $(MAKE) -C tools install, and once as a result of $(MAKE) -C stubdom install. If you get unlucky, this causes lossage. (This just happened to me in an osstest flight.)In principle we could alternatively fix this by lifting the commands in the qemu-xen-traditional-dir-find target (and perhaps other things too) into the toplevel
Makefile, as was done for mini-os.But that seems overkill given how bad the stubdom build system is, and the fact that we think at some point this qemu-trad will go away entirely. Adding the tools dependency back to the stubdom build is by and large good enough.
Quoted 6 lines
(Someone who really wants to build stubdom without tools is welcome to do this separation if they really want to.)
CC Juergen Gross <[email protected]>CC Wei Liu <[email protected]>Signed-off-by Ian Jackson <[email protected]>
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 1/2] build system: make install-stubdom depend on install-tools again
Wei Liu writes ("Re: [PATCH 1/2] build system: make install-stubdom depend on install-tools again"):
Quoted 9 lines · Ian Jackson
Quoted 8 lines
In
d290e325179ccee966cd679d0fed48be6f4cc1b7
"build system: don't let install-stubdom depend on install-tools"
the dependency of install-stubdom on install-tools was removed.However, this was not correct. stubdom/Makefile contains this:
$(XEN_ROOT)/tools/qemu-xen-traditional-dir: $(MAKE) DESTDIR= -C $(XEN_ROOT)/tools qemu-xen-traditional-dir-find
...
Quoted 1 line
Acked-by Wei Liu <[email protected]>
Thanks. Since this is related to unblocking osstest (it is causing spurious build failures in various push gates), I have pushed it.
Unfortunately I think this patch is not suitable for backport so this bug will continue to trouble us in stable branches from 4.9 onwards.
Ian.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH 2/2 DO NOT APPLY git-checkout.sh: add sleep to demonstrate race
In reply to Ian JacksonThis allows easy reproduction of git-checkout races: each checkout waits until you kill its sleep. You can see the multiple identical checkout runes in ps etc. A good rune is:
watch 'ps -fHtpts/60'
(replacing pts/60 with the pty of your build job).
--- scripts/git-checkout.sh | 2 ++ 1 file changed, 2 insertions(+)
scripts/git-checkout.sh +2 -0
@@ -11,6 +11,8 @@ DIR=$3 set -e +sleep 1000 ||: + if test \! -d $DIR-remote; then rm -rf $DIR-remote $DIR-remote.tmp mkdir -p $DIR-remote.tmp; rmdir $DIR-remote.tmp
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel