socketed based data connection
cover socketed based data connection
This patch set enables remote data access in the exec'd task process.
Some test data:
I've conducted a poky-image-minimal build on two different machines with and without the patch, the result are as follow:
2 core weybridge: worker count=4, make -j4
without patch
real: 161min
user: 237min
sys: 56min
with patch
real: 138min
user: 196min
sys: 45min
4 core nehalem: worker count=4, make -j4
without patch
real: 88min
user: 218min
sys: 37min
with patch
real: 83min
user: 160min
sys: 29minthe user+sys has an improvement of around 20%~30%, however the real runtime improvement is less significant. It's probably because of the thread workload unbalance on multicore, or in another word, some CPUs are not fully loaded during the build. e.g. when do_rootfs_rpm is run, there is probably no other tasks running.
Haven't done any test against single thread running, but it should have a much more considerable impact.
Known problems:
1. fetcher is not working at the moment, maybe bitbake needs an explicit InheritFromOS(), but that produces some inconsistency in PATH in my previous run
2. do_rootfs_rpm has some problem in the first run
Next step and possible future usage:
1. organize remote data to a separate class
2. although we have reverted to fork for task running, a remote data mechanism would work for a more separate build (e.g. distributed build)
3. the added code in bitbake-runtask is to run necessary initializations for a working environment, this is part of cooker. The current cooker seems too heavy and may be broken down into several pieces according to functionalitiies
Thanks,
Qing
Pull URL: git://git.pokylinux.org/poky-contrib.git
Branch: qhe/datasock
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=qhe/datasock
Thanks,
Qing He
---
Qing He (4):
python method fix
sstate.bbclass: workaround
data_smart: remote data
socket based worker data
bitbake/bin/bitbake-runtask | 80 +++++++++++++++++++++++++++++++++-------- bitbake/lib/bb/data_smart.py | 25 ++++++++++++- bitbake/lib/bb/methodpool.py | 6 +++ bitbake/lib/bb/parse/ast.py | 2 +- bitbake/lib/bb/runqueue.py | 68 ++++++++++++++++++++++++++++++++---- meta/classes/sstate.bbclass | 2 +- 6 files changed, 157 insertions(+), 26 deletions(-)
PATCH 1/4 python method fix
1. substitute '_' to '-' to avoid misrepresentation by smart data
2. export _parsed_fns
--- bitbake/lib/bb/methodpool.py | 6 ++++++ bitbake/lib/bb/parse/ast.py | 2 +- 2 files changed, 7 insertions(+), 1 deletions(-)
bitbake/lib/bb/methodpool.py +6 -0
@@ -82,3 +82,9 @@ def get_parsed_dict(): shortcut """ return _parsed_methods + +def get_parsed_fns(): + """ + used by environment restruct + """ + return _parsed_fns
bitbake/lib/bb/parse/ast.py +1 -1
@@ -138,7 +138,7 @@ class MethodNode: class PythonMethodNode(AstNode): def __init__(self, funcname, root, body, fn): - self.func_name = funcname + self.func_name = re.sub('_', '-', funcname) self.root = root self.body = body self.fn = fn
Signature · 1 line
1.7.0
PATCH 2/4 sstate.bbclass: workaround
In reply to Qing HeAvoid calculating task dependencies in task
This inhibit the generation of .siginfo
--- meta/classes/sstate.bbclass | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
meta/classes/sstate.bbclass +1 -1
@@ -287,7 +287,7 @@ def sstate_package(ss, d): bb.data.setVar('SSTATE_PKG', sstatepkg, d) bb.build.exec_func('sstate_create_package', d) - bb.siggen.dump_this_task(sstatepkg + ".siginfo", d) + #bb.siggen.dump_this_task(sstatepkg + ".siginfo", d) return
Signature · 1 line
1.7.0
PATCH 3/4 data_smart: remote data
In reply to Qing Head hoc version of remote data for data_smart
the data.dict["_remote_data"]["content"] is supposed to be the socket used for data connection
--- bitbake/lib/bb/data_smart.py | 25 ++++++++++++++++++++++++- 1 files changed, 24 insertions(+), 1 deletions(-)
bitbake/lib/bb/data_smart.py +24 -1
@@ -32,6 +32,13 @@ import copy, re, sys import bb from bb import utils from bb.COW import COWDictBase +import socket + +try: + import cPickle as pickle +except ImportError: + import pickle + bb.msg.note(1, bb.msg.domain.Cache, "Importing cPickle failed. Falling back to a very slow implementation.") __setvar_keyword__ = ["_append", "_prepend"]
@@ -189,6 +196,12 @@ class DataSmart: if var in dest: return dest[var] + if "_remote_data" in dest: + s = dest["_remote_data"]["content"] + s.send(var) + l = int(s.recv(4), 16) + return pickle.loads(s.recv(l, socket.MSG_WAITALL)) + if "_data" not in dest: break dest = dest["_data"]
@@ -351,8 +364,18 @@ class DataSmart: _keys(d["_data"], mykey) for key in d.keys(): - if key != "_data": + if key != "_data" and key != "_remote_data": mykey[key] = None + + if "_remote_data" in d: + s = d["_remote_data"]["content"] + s.send("_cmd_keys") + l = int(s.recv(4), 16) + rk = pickle.loads(s.recv(l, socket.MSG_WAITALL)) + for key in rk: + if key != "_data": + mykey[key] = None + keytab = {} _keys(self.dict, keytab) return keytab.keys()
Signature · 1 line
1.7.0
PATCH 4/4 socket based worker data
In reply to Qing HeThis patch tries to enable exec'd process to access bitbake data from a UNIX socket.
It's a PoC version.
--- bitbake/bin/bitbake-runtask | 80 ++++++++++++++++++++++++++++++++++-------- bitbake/lib/bb/runqueue.py | 68 ++++++++++++++++++++++++++++++++---- 2 files changed, 125 insertions(+), 23 deletions(-)
bitbake/bin/bitbake-runtask +64 -16
@@ -3,6 +3,7 @@ import os import sys import warnings +import re sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(sys.argv[0])), 'lib')) try:
@@ -71,8 +72,7 @@ bb.event.useStdout = False hashfile = sys.argv[1] buildfile = sys.argv[2] taskname = sys.argv[3] - -import bb.cooker +socketname = sys.argv[5] or None p = pickle.Unpickler(file(hashfile, "rb")) hashdata = p.load()
@@ -83,25 +83,68 @@ verbose = hashdata["verbose"] bb.utils.init_logger(bb.msg, verbose, debug, debug_domains) -cooker = bb.cooker.BBCooker(BBConfiguration(debug, debug_domains), None) -cooker.parseConfiguration() +if socketname: + import bb, bb.data, bb.parse, socket + import time + + fn = buildfile + + configuration = BBConfiguration(debug, debug_domains) + + try: + the_data = bb.data.init() + datasock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + datasock.connect(socketname) + bb.data.setVar("_remote_data", datasock, the_data) + except: + socketname = "" + +if socketname: + bb.codeparser.parser_cache_init(the_data) + bb.parse.init_parser(the_data, configuration.dump_signatures) + + datasock.send("_cmd_parsed_fns") + l = int(datasock.recv(4), 16) + parsed = pickle.loads(datasock.recv(l)) + for key in parsed.keys(): + funcname = re.sub('_', '-', key) + f = bb.data.getVar(funcname, the_data, False) + if f: + bb.methodpool.check_insert_method(key, f, fn) + + for var in bb.data.getVar('__BBHANDLERS', the_data) or []: + bb.event.register(var, bb.data.getVar(var, the_data)) -cooker.bb_cache = bb.cache.init(cooker) -cooker.status = bb.cache.CacheData() + if not bb.data.getVar("BUILDNAME", the_data): + bb.data.setVar("BUILDNAME", time.strftime('%Y%m%d%H%M'), the_data) + bb.data.setVar("BUILDSTART", time.strftime('%m/%d/%Y %H:%M:%S', time.gmtime()), the_data) -(fn, cls) = cooker.bb_cache.virtualfn2realfn(buildfile) -buildfile = cooker.matchFile(fn) -fn = cooker.bb_cache.realfn2virtual(buildfile, cls) -cooker.buildSetVars() + bb.data.setVar("BB_WORKERCONTEXT", "1", the_data) + bb.event.fire(bb.event.ConfigParsed(), the_data) -# Load data into the cache for fn and parse the loaded cache data -the_data = cooker.bb_cache.loadDataFull(fn, cooker.get_file_appends(fn), cooker.configuration.data) -cooker.bb_cache.setData(fn, buildfile, the_data) -cooker.bb_cache.handle_data(fn, cooker.status) +else: + import bb.cooker -#exportlist = bb.utils.preserved_envvars_export_list() -#bb.utils.filter_environment(exportlist) + cooker = bb.cooker.BBCooker(BBConfiguration(debug, debug_domains), None) + cooker.parseConfiguration() + + cooker.bb_cache = bb.cache.init(cooker) + cooker.status = bb.cache.CacheData() + + (fn, cls) = cooker.bb_cache.virtualfn2realfn(buildfile) + buildfile = cooker.matchFile(fn) + fn = cooker.bb_cache.realfn2virtual(buildfile, cls) + + cooker.buildSetVars() + + # Load data into the cache for fn and parse the loaded cache data + the_data = cooker.bb_cache.loadDataFull(fn, cooker.get_file_appends(fn), cooker.configuration.data) + cooker.bb_cache.setData(fn, buildfile, the_data) + cooker.bb_cache.handle_data(fn, cooker.status) + + #exportlist = bb.utils.preserved_envvars_export_list() + #bb.utils.filter_environment(exportlist) if taskname.endswith("_setscene"): the_data.setVarFlag(taskname, "quieterrors", "1")
@@ -116,5 +159,10 @@ for h in hashdata["deps"]: ret = 0 if sys.argv[4] != "True": ret = bb.build.exec_task(fn, taskname, the_data) + +if socketname: + datasock.send("_cmd_end") + datasock.close() + sys.exit(ret)
bitbake/lib/bb/runqueue.py +61 -7
129 lines not on this page · raw message
Signature · 1 line
1.7.0
Quoted 1 line · Qing
This patch set enables remote data access in the exec'd task process.
It's intended to be proof of concept to an alternative approach to work around the slowness introduced by exec bitbake-runtask. The patch is still WIP.
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=qhe/datasock
Thanks,
Qing
Quoted 76 lines
Some test data:
I've conducted a poky-image-minimal build on two different machines with and without the patch, the result are as follow:2 core weybridge: worker count=4, make -j4
without patchreal: 161min user: 237min sys: 56min with patch real: 138min user: 196min sys: 45min 4 core nehalem: worker count=4, make -j4 without patch real: 88min user: 218min sys: 37min with patch real: 83min user: 160min sys: 29minthe user+sys has an improvement of around 20%~30%, however the real runtime improvement is less significant. It's probably because of the thread workload unbalance on multicore, or in another word, some CPUs are not fully loaded during the build. e.g. when
do_rootfs_rpmis run, there is probably no other tasks running.Haven't done any test against single thread running, but it should have a much more considerable impact.
Known problems:
1. fetcher is not working at the moment, maybe bitbake needs an explicitInheritFromOS(), but that produces some inconsistency in PATH in my previous run
2.do_rootfs_rpmhas some problem in the first runNext step and possible future usage:
1. organize remote data to a separate class
2. although we have reverted to fork for task running, a remote data mechanism would work for a more separate build (e.g. distributed build)
3. the added code in bitbake-runtask is to run necessary initializations for a working environment, this is part of cooker. The current cooker seems too heavy and may be broken down into several pieces according to functionalitiiesThanks,
QingPull URL:
git://git.pokylinux.org/poky-contrib.git
Branch: qhe/datasock
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=qhe/datasockThanks,
Qing He
---Qing He (4):
python method fix
sstate.bbclass: workaround
data_smart: remote data
socket based worker databitbake/bin/bitbake-runtask | 80 +++++++++++++++++++++++++++++++++-------- bitbake/lib/bb/data_smart.py | 25 ++++++++++++- bitbake/lib/bb/methodpool.py | 6 +++ bitbake/lib/bb/parse/ast.py | 2 +- bitbake/lib/bb/runqueue.py | 68 ++++++++++++++++++++++++++++++++---- meta/classes/sstate.bbclass | 2 +- 6 files changed, 157 insertions(+), 26 deletions(-)
Quoted 11 lines
From: Qing He
Sent: Wednesday, December 22, 2010 1:25 AMQuoted 1 line · Qing
This patch set enables remote data access in the exec'd task process.
It's intended to be proof of concept to an alternative approach to work around the slowness introduced by exec bitbake-runtask. The patch is still WIP.
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=qhe/datasock
looks the change is not intrusive as I originally thought. :-)
Quoted 86 lines
Thanks,
QingQuoted 62 lines
Some test data:
I've conducted a poky-image-minimal build on two different machines with and without the patch, the result are as follow:2 core weybridge: worker count=4, make -j4
without patchreal: 161min user: 237min sys: 56min with patch real: 138min user: 196min sys: 45min 4 core nehalem: worker count=4, make -j4 without patch real: 88min user: 218min sys: 37min with patch real: 83min user: 160min sys: 29minthe user+sys has an improvement of around 20%~30%, however the real runtime improvement is less significant. It's probably because of the thread workload unbalance on multicore, or in another word, some CPUs are not fully loaded during the build. e.g. when
do_rootfs_rpmis run, there is probably no other tasks running.Haven't done any test against single thread running, but it should have a much more considerable impact.
Known problems:
1. fetcher is not working at the moment, maybe bitbake needs an explicitInheritFromOS(), but that produces some inconsistency in PATH in my previous run
2.do_rootfs_rpmhas some problem in the first runNext step and possible future usage:
1. organize remote data to a separate class
2. although we have reverted to fork for task running, a remote data mechanism would work for a more separate build (e.g. distributed build)
3. the added code in bitbake-runtask is to run necessary initializations for a working environment, this is part of cooker. The current cooker seems too heavy and may be broken down into several pieces according to functionalitiiesThanks,
QingPull URL:
git://git.pokylinux.org/poky-contrib.git
Branch: qhe/datasock
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=qhe/datasockThanks,
Qing He
---Quoted 7 lines
Qing He (4):
python method fix
sstate.bbclass: workaround
data_smart: remote data
socket based worker databitbake/bin/bitbake-runtask | 80+++++++++++++++++++++++++++++++++--------
Quoted 4 lines
bitbake/lib/bb/data_smart.py | 25 ++++++++++++- bitbake/lib/bb/methodpool.py | 6 +++ bitbake/lib/bb/parse/ast.py | 2 +- bitbake/lib/bb/runqueue.py | 68++++++++++++++++++++++++++++++++----
Quoted 3 lines
meta/classes/sstate.bbclass | 2 +- 6 files changed, 157 insertions(+), 26 deletions(-)_______________________________________________
poky mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/poky
Quoted 8 lines · Qing He
This patch set enables remote data access in the exec'd task process.
Some test data:
I've conducted a poky-image-minimal build on two different machines with and without the patch, the result are as follow:2 core weybridge: worker count=4, make -j4
without patch
Quoted 16 lines
real: 161min
user: 237min
sys: 56min
with patch
real: 138min
user: 196min
sys: 45min
4 core nehalem: worker count=4, make -j4
without patch
real: 88min
user: 218min
sys: 37min
with patch
real: 83min
user: 160min
sys: 29minFor comparison what numbers does master give? I'm assuming these numbers were with master using exec() so I'd be interested in the comparisons with fork().
Quoted 17 lines
the user+sys has an improvement of around 20%~30%, however the real runtime improvement is less significant. It's probably because of the thread workload unbalance on multicore, or in another word, some CPUs are not fully loaded during the build. e.g. when
do_rootfs_rpmis run, there is probably no other tasks running.Haven't done any test against single thread running, but it should have a much more considerable impact.
Known problems:
1. fetcher is not working at the moment, maybe bitbake needs an explicitInheritFromOS(), but that produces some inconsistency in PATH in my previous run
2.do_rootfs_rpmhas some problem in the first run
Quoted 4 lines
Next step and possible future usage:
1. organize remote data to a separate class
2. although we have reverted to fork for task running, a remote data mechanism would work for a more separate build (e.g. distributed build)
It could also be useful for UI interaction with the datastore.
Quoted 3 lines
3. the added code in bitbake-runtask is to run necessary initializations for a working environment, this is part of cooker. The current cooker seems too heavy and may be broken down into several pieces according to functionalitiies
I certainly agree cooker could use some serious refactoring in some form.
Cheers,
Richard