update.py: fetch repos parallelly
cover update.py: fetch repos parallelly
Hi Paul,
We have 124 layers, the "update.py -b <branch>" costs about 10 minutes to finish the update for one branch, and we need update several branches periodly, which is really a little slow. So I'd like to make it can run parally, here are some thoughts about it:
1) Make fetch can run parallelly, this is easy to do, and safe enough as the patches does
2) Make recipeparse can parse layers parallelly, we may need split update_layer.py into two parts:
- The one which only does recipeparse, this costs a lot of time, and can be parallell.
- The one which writes to database (can't be parallel, and doesn't have to)
I will work on it if no objections.
// Robert
The following changes since commit fa0b01082fb0596959841e95af8c516ae3f267a7:
admin: fix dependency display and search (2017-12-21 16:14:17 +1300)
are available in the git repository at:
git://git.pokylinux.org/poky-contrib rbt/layerindex_parallel
http://git.pokylinux.org/cgit.cgi//log/?h=rbt/layerindex_parallel
Robert Yang (1):
update.py: fetch repos parallelly
layerindex/update.py | 47 ++++++++++++++++++++++++++++++++++------------- settings.py | 2 ++ 2 files changed, 36 insertions(+), 13 deletions(-)
Signature · 1 line
2.7.4
RFC PATCH 1/1 update.py: fetch repos parallelly
This can save a lot of time, here is my testing data when PARALLEL_JOBS is 10, this is the fetch time only, I hacked it to stop when the fetch is done to get the data (124 layers):
$ update.py -b <branch> Before: 2m30 Now: 16s
--- layerindex/update.py | 47 ++++++++++++++++++++++++++++++++++------------- settings.py | 2 ++ 2 files changed, 36 insertions(+), 13 deletions(-)
layerindex/update.py +34 -13
@@ -20,6 +20,7 @@ from distutils.version import LooseVersion import utils import operator import re +import multiprocessing import warnings warnings.filterwarnings("ignore", category=DeprecationWarning)
@@ -127,6 +128,18 @@ def update_actual_branch(layerquery, fetchdir, branch, options, update_bitbake, for s in to_save: s.save() +def fetch_repo(vcs_url, repodir, urldir, fetchdir, layer_name): + logger.info("Fetching remote repository %s" % vcs_url) + try: + if not os.path.exists(repodir): + utils.runcmd("git clone %s %s" % (vcs_url, urldir), fetchdir, logger=logger, printerr=False) + else: + utils.runcmd("git fetch -p", repodir, logger=logger, printerr=False) + return (vcs_url, None) + except subprocess.CalledProcessError as e: + logger.error("Fetch of layer %s failed: %s" % (layer_name, e.output)) + return (vcs_url, e.output) + def main(): if LooseVersion(git.__version__) < '0.3.1': logger.error("Version of GitPython is too old, please install GitPython (python-git) 0.3.1 or later in order to use this script")
@@ -232,6 +245,9 @@ def main(): if not os.path.exists(fetchdir): os.makedirs(fetchdir) + + allrepos = {} + fetchedresult = [] fetchedrepos = [] failedrepos = {}
@@ -262,19 +278,24 @@ def main(): # Handle multiple layers in a single repo urldir = layer.get_fetch_dir() repodir = os.path.join(fetchdir, urldir) - if not (layer.vcs_url in fetchedrepos or layer.vcs_url in failedrepos): - logger.info("Fetching remote repository %s" % layer.vcs_url) - out = None - try: - if not os.path.exists(repodir): - out = utils.runcmd("git clone %s %s" % (layer.vcs_url, urldir), fetchdir, logger=logger, printerr=False) - else: - out = utils.runcmd("git fetch -p", repodir, logger=logger, printerr=False) - except subprocess.CalledProcessError as e: - logger.error("Fetch of layer %s failed: %s" % (layer.name, e.output)) - failedrepos[layer.vcs_url] = e.output - continue - fetchedrepos.append(layer.vcs_url) + if layer.vcs_url not in allrepos: + allrepos[layer.vcs_url] = (repodir, urldir, fetchdir, layer.name) + # Add bitbake + allrepos[settings.BITBAKE_REPO_URL] = (bitbakepath, "bitbake", fetchdir, "bitbake") + # Parallel fetching + pool = multiprocessing.Pool(int(settings.PARALLEL_JOBS)) + for url in allrepos: + fetchedresult.append(pool.apply_async(fetch_repo, \ + (url, allrepos[url][0], allrepos[url][1], allrepos[url][2], allrepos[url][3],))) + pool.close() + pool.join() + + for url in fetchedresult[:]: + # The format is (url, error), the error is None when succeed. + if url.get()[1]: + failedrepos[url.get()[0]] = url.get()[1] + else: + fetchedrepos.append(url.get()[0]) if not (fetchedrepos or update_bitbake): logger.error("No repositories could be fetched, exiting")
settings.py +2 -0
@@ -228,3 +228,5 @@ SUBMIT_EMAIL_SUBJECT = 'OE Layerindex layer submission' # RabbitMQ settings RABBIT_BROKER = 'amqp://' RABBIT_BACKEND = 'rpc://' + +PARALLEL_JOBS = "10"
Signature · 1 line
2.7.4
Hi Paul,
Quoted 14 lines · Robert Yang
Hi Paul,
We have 124 layers, the "
update.py-b <branch>" costs about 10 minutes to finish the update for one branch, and we need update several branches periodly, which is really a little slow. So I'd like to make it can run parally, here are some thoughts about it:
1) Make fetch can run parallelly, this is easy to do, and safe enough as the patches does
2) Make recipeparse can parse layers parallelly, we may need splitupdate_layer.pyinto two parts:
- The one which only does recipeparse, this costs a lot of time, and can be parallell.
- The one which writes to database (can't be parallel, and doesn't have to)
I found that there is an easy way to improve the performance, just use update_layer.py as a module rather than an program can save a lot of time, we can use multiprocessing.Process() to call the function to put it in a subprocess, and remove bb modules when switch branches since it may not be compatible, please see next email for details, it can reduce time from 9m20s to 1m43s for 124 layers when everything is up to date totally.
// Robert
Quoted 20 lines
I will work on it if no objections.
// Robert
The following changes since commit
fa0b01082fb0596959841e95af8c516ae3f267a7:admin: fix dependency display and search (2017-12-21 16:14:17 +1300)
are available in the git repository at:
git://git.pokylinux.org/poky-contribrbt/layerindex_parallel
http://git.pokylinux.org/cgit.cgi//log/?h=rbt/layerindex_parallelRobert Yang (1):
update.py: fetch repos parallellylayerindex/update.py | 47 ++++++++++++++++++++++++++++++++++------------- settings.py | 2 ++ 2 files changed, 36 insertions(+), 13 deletions(-)
Hi Robert,
Quoted 15 lines · NZDT Robert Yang
Quoted 4 lines · Robert Yang
We have 124 layers, the "
update.py-b <branch>" costs about 10 minutes to finish the update for one branch, and we need update several branches periodly, which is really a little slow. So I'd like to make it can run parally, here are some thoughts about it:Quoted 9 lines
1) Make fetch can run parallelly, this is easy to do, and safe enough as
the patches does
2) Make recipeparse can parse layers parallelly, we may need splitupdate_layer.pyinto two parts:
- The one which only does recipeparse, this costs a lot of time, and can be parallell.
- The one which writes to database (can't be parallel, and doesn't have to)
Quoted 6 lines
I found that there is an easy way to improve the performance, just use
update_layer.pyas a module rather than an program can save a lot of time, we can usemultiprocessing.Process()to call the function to put it in a subprocess, and remove bb modules when switch branches since it may not be compatible, please see next email for details, it can reduce time from 9m20s to 1m43s for 124 layers when everything is up to date totally.
OK, but:
1) Does this work across python 2 and 3?
2) Are you 100% sure that this clears out the memory of anything that happens to get loaded? One of the issues we used to see with the earlier single program structure was "corruption" of the python variable/module space; the current fully separate processes completely avoid that.
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Quoted 25 lines · Paul Eggleton
Hi Robert,
Quoted 15 lines · NZDT Robert Yang
Quoted 4 lines · Robert Yang
We have 124 layers, the "
update.py-b <branch>" costs about 10 minutes to finish the update for one branch, and we need update several branches periodly, which is really a little slow. So I'd like to make it can run parally, here are some thoughts about it:Quoted 9 lines
1) Make fetch can run parallelly, this is easy to do, and safe enough as
the patches does
2) Make recipeparse can parse layers parallelly, we may need splitupdate_layer.pyinto two parts:
- The one which only does recipeparse, this costs a lot of time, and can be parallell.
- The one which writes to database (can't be parallel, and doesn't have to)Quoted 6 lines
I found that there is an easy way to improve the performance, just use
update_layer.pyas a module rather than an program can save a lot of time, we can usemultiprocessing.Process()to call the function to put it in a subprocess, and remove bb modules when switch branches since it may not be compatible, please see next email for details, it can reduce time from 9m20s to 1m43s for 124 layers when everything is up to date totally.
Quoted 3 lines
OK, but:
1) Does this work across python 2 and 3?
Sorry, does it still needs support python2 ? My oldest bitbake only supported python3, so I didn't test it with python 2 since bitbake would raise the error:
$ /usr/bin/python ./update.py -b <branch>
File "/workspace1/lyang1/layers-fetched/bitbake/lib/bb/__init__.py", line 28,
in <module>
raise RuntimeError("Sorry, python 3.4.0 or later is required for this
version of bitbake")
Quoted 5 lines
2) Are you 100% sure that this clears out the memory of anything that happens to get loaded? One of the issues we used to see with the earlier single program structure was "corruption" of the python variable/module space; the current fully separate processes completely avoid that.
Yes, I'm sure, "del sys.modules[name]" can unload the modules when child process is terminated.
What I met before was tinfoil can't be unloaded, then there would be errors when switch from old branch (YP 2.2) to new one (YP 2.4), del sys.modules[name] can fix the problem.
// Robert
Quoted 4 lines
Cheers,
Paul