Skip to contentSkip to conversation

lib/oe/package_manager: Add utils function for multiprocess execution

PATCH In review Richard Purdie 2 participants 3 messages Aug 21, 2014 – Aug 23, 2014

lore
Share & tools
lore

Richard Purdie

author
#0 lore raw

PATCH 1/1 lib/oe/package_manager: Add utils function for multiprocess execution

Our usage of multitprocessing is problematic. In particular, there is a bug in python 2.7 multiprocessing where signals are not handled until command completion instead of immediately.

This factors the multiprocess code into a function which is enhanced with a workaround to ensure immediate signal handling and also better SIGINT handling which should happen in the parent, not the children to ensure clean exits. The workaround for the signals is being added to the core bb.utils function so it can benefit all users.

Signed-off-by Richard Purdie <[email protected]>
meta/lib/oe/package_manager.py +4 -28
index 8be3d41..f8fc3c2 100644
@@ -7,6 +7,7 @@ import multiprocessing
 import re
 import bb
 import tempfile
+import oe.utils
 
 
 # this can be used by all PM backends to create the index files in parallel

@@ -116,16 +117,7 @@ class RpmIndexer(Indexer):
             bb.note("There are no packages in %s" % self.deploy_dir)
             return
 
-        nproc = multiprocessing.cpu_count()
-        pool = bb.utils.multiprocessingpool(nproc)
-        results = list(pool.imap(create_index, index_cmds))
-        pool.close()
-        pool.join()
-
-        for result in results:
-            if result is not None:
-                return(result)
-
+        oe.utils.multiprocess_exec(index_cmds, create_index)
 
 class OpkgIndexer(Indexer):
     def write_index(self):

@@ -161,15 +153,7 @@ class OpkgIndexer(Indexer):
             bb.note("There are no packages in %s!" % self.deploy_dir)
             return
 
-        nproc = multiprocessing.cpu_count()
-        pool = bb.utils.multiprocessingpool(nproc)
-        results = list(pool.imap(create_index, index_cmds))
-        pool.close()
-        pool.join()
-
-        for result in results:
-            if result is not None:
-                return(result)
+        oe.utils.multiprocess_exec(index_cmds, create_index)
 
 
 class DpkgIndexer(Indexer):

@@ -210,15 +194,7 @@ class DpkgIndexer(Indexer):
             bb.note("There are no packages in %s" % self.deploy_dir)
             return
 
-        nproc = multiprocessing.cpu_count()
-        pool = bb.utils.multiprocessingpool(nproc)
-        results = list(pool.imap(create_index, index_cmds))
-        pool.close()
-        pool.join()
-
-        for result in results:
-            if result is not None:
-                return(result)
+        oe.utils.multiprocess_exec(index_cmds, create_index)
 
 
 class PkgsList(object):
meta/lib/oe/utils.py +29 -0
index 0a1d108..92e21a4 100644
@@ -151,3 +151,32 @@ def execute_pre_post_process(d, cmds):
         if cmd != '':
             bb.note("Executing %s ..." % cmd)
             bb.build.exec_func(cmd, d)
+
+def multiprocess_exec(commands, function):
+    import signal
+    import multiprocessing
+
+    if not commands:
+        return []
+
+    def init_worker():
+        signal.signal(signal.SIGINT, signal.SIG_IGN)
+
+    nproc = min(multiprocessing.cpu_count(), len(commands))
+    pool = bb.utils.multiprocessingpool(nproc, init_worker)
+    imap = pool.imap(function, commands)
+
+    try:
+        results = list(imap)
+        pool.close()
+        pool.join()
+        results = []
+        for result in results:
+            if result is not None:
+                results.append(result)
+        return results
+
+    except KeyboardInterrupt:
+        pool.terminate()
+        pool.join()
+        raise
#1 lore raw
Quoted 3 lines · Richard Purdie

Our usage of multitprocessing is problematic. In particular, there is a bug in python 2.7 multiprocessing where signals are not handled until command completion instead of immediately.

This looks good, but the subject is misleading, in my opinion:
"lib/oe/package_manager:
Add utils function for multiprocess execution". It should probably either mention both utils and package_manager, or be split into two commits.

Signature · 5 lines
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics
author
#2 lore raw
Quoted 8 lines · Christopher Larson

On Thu, Aug 21, 2014 at 1:46 PM, Richard Purdie <[email protected]> wrote:
Our usage of multitprocessing is problematic. In particular, there is a bug
in python 2.7 multiprocessing where signals are not handled until command
completion instead of immediately.

Quoted 4 lines

This looks good, but the subject is misleading, in my opinion:
"lib/oe/package_manager: Add utils function for multiprocess execution". It should probably either mention both utils and package_manager, or be split into two commits.

Good point, I tweaked before merging, thanks. It was the (bad) result of rearranging a few commits.

Cheers,

Richard