Fix race while building external modules

When building an image with several external modules, a race is likely to occur. This race is when building scripts in the staged kernel tree (sysroot).

To prevent this, we promote the do_make_scripts() function to a task, which we protect by a lock. Tip on how to make this work was received by Richard in [1]. To avoid new problems, introduced by this promotion, we make this new task depend on do_populate_sysroot from the modules dependencies (i.e. virtual/kernel).

This patch has been extensively tested on the 2011-1 release branch, on a custom ARM-machine and distro with several external modules.

[1] http://lists.linuxtogo.org/pipermail/openembedded-core/2011-October/011163.html

The following changes since commit e31dd9b65f3b03f79cabab25eca157532de3bd9c:

fontconfig: fix fix-pkgconfig.patch (2011-10-18 18:13:47 +0100)

are available in the git repository at:
git://github.com/darander/oe-core ext-mod-race
https://github.com/darander/oe-core/tree/ext-mod-race

Anders Darander (1):
module.bbclass: add lock to prevent error bulding ext modules

 meta/classes/module.bbclass |    5 ++++-
 1 files changed, 4 insertions(+), 1 deletions(-)

1.7.7

When external modules are built, files in $STAGING_KERNEL_DIR/scripts/basic will/can get rebuilt.
This raises a potential race condition. Prevent this by adding a lock around the do_make_scripts() function. Further, make sure that the kernel has been installed to the sysroot, prior to executing this new task.

---
 meta/classes/module.bbclass |    5 ++++-
 1 files changed, 4 insertions(+), 1 deletions(-)

index 572df0d..53c16b7 100644
@@ -14,8 +14,11 @@ do_make_scripts() {
 	           -C ${STAGING_KERNEL_DIR} scripts
 }
 
+addtask make_scripts before do_compile
+do_make_scripts[lockfiles] = "${TMPDIR}/kernel-scripts.lock"
+do_make_scripts[deptask] = "do_populate_sysroot"
+
 module_do_compile() {
-	do_make_scripts
 	unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS
 	oe_runmake KERNEL_PATH=${STAGING_KERNEL_DIR}   \
 		   KERNEL_SRC=${STAGING_KERNEL_DIR}    \

1.7.7

Op 19 okt. 2011, om 13:15 heeft Anders Darander het volgende geschreven:

So what do external module recipe that aren't using do_compile from module.bbclass need to change to keep working?

I haven't tried any, do you know any widely available ones that I could check?

If they aren't using the do_compile from module.bbclass, are they inheriting module.bbclass anyway? If they aren't inheriting module.bbclass, then they shouldn't be affected.

If they inherit module.bbclass, than I guess that we should remove the call to do_make_scripts from their do_compile. _But_ this is untested, as I've got no such recipe (that I'm aware of). If my assumptions hold, we should probably rephrase the comment regarding do_make_scripts.

Cheers,
Anders

Anders Darander
ChargeStorm AB / eStorm AB

Merged into OE-Core

Thanks
Sau!