Skip to contentSkip to conversation

socketed based data connection

RFC In review Qing He 3 participants 8 messages Dec 21, 2010 – Dec 22, 2010

lore
Share & tools
lore
author
#1 lore raw

PATCH 1/4 python method fix

In reply to Qing He

1. substitute '_' to '-' to avoid misrepresentation by smart data
2. export _parsed_fns

---
 bitbake/lib/bb/methodpool.py |    6 ++++++
 bitbake/lib/bb/parse/ast.py  |    2 +-
 2 files changed, 7 insertions(+), 1 deletions(-)
bitbake/lib/bb/methodpool.py +6 -0
index 1485b13..299c54d 100644
@@ -82,3 +82,9 @@ def get_parsed_dict():
     shortcut
     """
     return _parsed_methods
+
+def get_parsed_fns():
+    """
+    used by environment restruct
+    """
+    return _parsed_fns
bitbake/lib/bb/parse/ast.py +1 -1
index 870ae65..dc56f21 100644
@@ -138,7 +138,7 @@ class MethodNode:
 
 class PythonMethodNode(AstNode):
     def __init__(self, funcname, root, body, fn):
-        self.func_name = funcname
+        self.func_name = re.sub('_', '-', funcname)
         self.root = root
         self.body = body
         self.fn = fn
Signature · 1 line
1.7.0
author
#2 lore raw

PATCH 2/4 sstate.bbclass: workaround

In reply to Qing He

Avoid calculating task dependencies in task
This inhibit the generation of .siginfo

---
 meta/classes/sstate.bbclass |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
meta/classes/sstate.bbclass +1 -1
index 3696a8c..cc5e672 100644
@@ -287,7 +287,7 @@ def sstate_package(ss, d):
     bb.data.setVar('SSTATE_PKG', sstatepkg, d)
     bb.build.exec_func('sstate_create_package', d)
     
-    bb.siggen.dump_this_task(sstatepkg + ".siginfo", d)
+    #bb.siggen.dump_this_task(sstatepkg + ".siginfo", d)
 
     return
 
Signature · 1 line
1.7.0
author
#3 lore raw

PATCH 3/4 data_smart: remote data

In reply to Qing He

ad hoc version of remote data for data_smart
the data.dict["_remote_data"]["content"] is supposed to be the socket used for data connection

---
 bitbake/lib/bb/data_smart.py |   25 ++++++++++++++++++++++++-
 1 files changed, 24 insertions(+), 1 deletions(-)
bitbake/lib/bb/data_smart.py +24 -1
index c8cd8f8..9b630eb 100644
@@ -32,6 +32,13 @@ import copy, re, sys
 import bb
 from bb   import utils
 from bb.COW  import COWDictBase
+import socket
+
+try:
+    import cPickle as pickle
+except ImportError:
+    import pickle
+    bb.msg.note(1, bb.msg.domain.Cache, "Importing cPickle failed. Falling back to a very slow implementation.")
 
 
 __setvar_keyword__ = ["_append", "_prepend"]
@@ -189,6 +196,12 @@ class DataSmart:
             if var in dest:
                 return dest[var]
 
+            if "_remote_data" in dest:
+                s = dest["_remote_data"]["content"]
+                s.send(var)
+                l = int(s.recv(4), 16)
+                return pickle.loads(s.recv(l, socket.MSG_WAITALL))
+
             if "_data" not in dest:
                 break
             dest = dest["_data"]
@@ -351,8 +364,18 @@ class DataSmart:
                 _keys(d["_data"], mykey)
 
             for key in d.keys():
-                if key != "_data":
+                if key != "_data" and key != "_remote_data":
                     mykey[key] = None
+
+            if "_remote_data" in d:
+                s = d["_remote_data"]["content"]
+                s.send("_cmd_keys")
+                l = int(s.recv(4), 16)
+                rk = pickle.loads(s.recv(l, socket.MSG_WAITALL))
+                for key in rk:
+                    if key != "_data":
+                        mykey[key] = None
+
         keytab = {}
         _keys(self.dict, keytab)
         return keytab.keys()
Signature · 1 line
1.7.0

Qing He

author
#0 lore raw

cover socketed based data connection

This patch set enables remote data access in the exec'd task process.

Some test data:
I've conducted a poky-image-minimal build on two different machines with and without the patch, the result are as follow:

2 core weybridge: worker count=4, make -j4
without patch

    real: 161min
    user: 237min
    sys:  56min
  with patch
    real: 138min
    user: 196min
    sys:  45min
4 core nehalem: worker count=4, make -j4
  without patch
    real: 88min
    user: 218min
    sys:  37min
  with patch
    real: 83min
    user: 160min
    sys:  29min

the user+sys has an improvement of around 20%~30%, however the real runtime improvement is less significant. It's probably because of the thread workload unbalance on multicore, or in another word, some CPUs are not fully loaded during the build. e.g. when do_rootfs_rpm is run, there is probably no other tasks running.

Haven't done any test against single thread running, but it should have a much more considerable impact.

Known problems:
1. fetcher is not working at the moment, maybe bitbake needs an explicit InheritFromOS(), but that produces some inconsistency in PATH in my previous run
2. do_rootfs_rpm has some problem in the first run

Next step and possible future usage:
1. organize remote data to a separate class
2. although we have reverted to fork for task running, a remote data mechanism would work for a more separate build (e.g. distributed build)
3. the added code in bitbake-runtask is to run necessary initializations for a working environment, this is part of cooker. The current cooker seems too heavy and may be broken down into several pieces according to functionalitiies

Thanks,
Qing

Pull URL: git://git.pokylinux.org/poky-contrib.git
Branch: qhe/datasock
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=qhe/datasock

Thanks,
Qing He
---

Qing He (4):
python method fix
sstate.bbclass: workaround
data_smart: remote data
socket based worker data

 bitbake/bin/bitbake-runtask  |   80 +++++++++++++++++++++++++++++++++--------
 bitbake/lib/bb/data_smart.py |   25 ++++++++++++-
 bitbake/lib/bb/methodpool.py |    6 +++
 bitbake/lib/bb/parse/ast.py  |    2 +-
 bitbake/lib/bb/runqueue.py   |   68 ++++++++++++++++++++++++++++++++----
 meta/classes/sstate.bbclass  |    2 +-
 6 files changed, 157 insertions(+), 26 deletions(-)
author
#4 lore raw

PATCH 4/4 socket based worker data

This patch tries to enable exec'd process to access bitbake data from a UNIX socket.

It's a PoC version.

---
 bitbake/bin/bitbake-runtask |   80 ++++++++++++++++++++++++++++++++++--------
 bitbake/lib/bb/runqueue.py  |   68 ++++++++++++++++++++++++++++++++----
 2 files changed, 125 insertions(+), 23 deletions(-)
bitbake/bin/bitbake-runtask +64 -16
index bee0f42..4699490 100755
@@ -3,6 +3,7 @@
 import os
 import sys
 import warnings
+import re
 sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(sys.argv[0])), 'lib'))
 
 try:
@@ -71,8 +72,7 @@ bb.event.useStdout = False
 hashfile = sys.argv[1]
 buildfile = sys.argv[2]
 taskname = sys.argv[3]
-
-import bb.cooker
+socketname = sys.argv[5] or None
 
 p = pickle.Unpickler(file(hashfile, "rb"))
 hashdata = p.load()
@@ -83,25 +83,68 @@ verbose = hashdata["verbose"]
 
 bb.utils.init_logger(bb.msg, verbose, debug, debug_domains)
 
-cooker = bb.cooker.BBCooker(BBConfiguration(debug, debug_domains), None)
-cooker.parseConfiguration()
+if socketname:
+    import bb, bb.data, bb.parse, socket
+    import time
+
+    fn = buildfile
+
+    configuration = BBConfiguration(debug, debug_domains)
+
+    try:
+        the_data = bb.data.init()
+        datasock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
+        datasock.connect(socketname)
+        bb.data.setVar("_remote_data", datasock, the_data)
+    except:
+        socketname = ""
+
+if socketname:
+    bb.codeparser.parser_cache_init(the_data)
+    bb.parse.init_parser(the_data, configuration.dump_signatures)
+
+    datasock.send("_cmd_parsed_fns")
+    l = int(datasock.recv(4), 16)
+    parsed = pickle.loads(datasock.recv(l))
+    for key in parsed.keys():
+        funcname = re.sub('_', '-', key)
+        f = bb.data.getVar(funcname, the_data, False)
+        if f:
+            bb.methodpool.check_insert_method(key, f, fn)
+
+    for var in bb.data.getVar('__BBHANDLERS', the_data) or []:
+        bb.event.register(var, bb.data.getVar(var, the_data))
 
-cooker.bb_cache = bb.cache.init(cooker)
-cooker.status = bb.cache.CacheData()
+    if not bb.data.getVar("BUILDNAME", the_data):
+        bb.data.setVar("BUILDNAME", time.strftime('%Y%m%d%H%M'), the_data)
+    bb.data.setVar("BUILDSTART", time.strftime('%m/%d/%Y %H:%M:%S', time.gmtime()), the_data)
 
-(fn, cls) = cooker.bb_cache.virtualfn2realfn(buildfile)
-buildfile = cooker.matchFile(fn)
-fn = cooker.bb_cache.realfn2virtual(buildfile, cls)
 
-cooker.buildSetVars()
+    bb.data.setVar("BB_WORKERCONTEXT", "1", the_data)
+    bb.event.fire(bb.event.ConfigParsed(), the_data)
 
-# Load data into the cache for fn and parse the loaded cache data
-the_data = cooker.bb_cache.loadDataFull(fn, cooker.get_file_appends(fn), cooker.configuration.data)
-cooker.bb_cache.setData(fn, buildfile, the_data)
-cooker.bb_cache.handle_data(fn, cooker.status)
+else:
+    import bb.cooker
 
-#exportlist = bb.utils.preserved_envvars_export_list()
-#bb.utils.filter_environment(exportlist)
+    cooker = bb.cooker.BBCooker(BBConfiguration(debug, debug_domains), None)
+    cooker.parseConfiguration()
+
+    cooker.bb_cache = bb.cache.init(cooker)
+    cooker.status = bb.cache.CacheData()
+
+    (fn, cls) = cooker.bb_cache.virtualfn2realfn(buildfile)
+    buildfile = cooker.matchFile(fn)
+    fn = cooker.bb_cache.realfn2virtual(buildfile, cls)
+
+    cooker.buildSetVars()
+
+    # Load data into the cache for fn and parse the loaded cache data
+    the_data = cooker.bb_cache.loadDataFull(fn, cooker.get_file_appends(fn), cooker.configuration.data)
+    cooker.bb_cache.setData(fn, buildfile, the_data)
+    cooker.bb_cache.handle_data(fn, cooker.status)
+
+    #exportlist = bb.utils.preserved_envvars_export_list()
+    #bb.utils.filter_environment(exportlist)
 
 if taskname.endswith("_setscene"):
     the_data.setVarFlag(taskname, "quieterrors", "1")
@@ -116,5 +159,10 @@ for h in hashdata["deps"]:
 ret = 0
 if sys.argv[4] != "True":
     ret = bb.build.exec_task(fn, taskname, the_data)
+
+if socketname:
+    datasock.send("_cmd_end")
+    datasock.close()
+
 sys.exit(ret)
 
bitbake/lib/bb/runqueue.py +61 -7

129 lines not on this page · raw message

Signature · 1 line
1.7.0
author
#5 lore raw
In reply to Qing He
Quoted 1 line · Qing

This patch set enables remote data access in the exec'd task process.

It's intended to be proof of concept to an alternative approach to work around the slowness introduced by exec bitbake-runtask. The patch is still WIP.

http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=qhe/datasock

Thanks,
Qing

Quoted 76 lines

Some test data:
I've conducted a poky-image-minimal build on two different machines with and without the patch, the result are as follow:

2 core weybridge: worker count=4, make -j4
without patch

    real: 161min
    user: 237min
    sys:  56min
  with patch
    real: 138min
    user: 196min
    sys:  45min
4 core nehalem: worker count=4, make -j4
  without patch
    real: 88min
    user: 218min
    sys:  37min
  with patch
    real: 83min
    user: 160min
    sys:  29min

the user+sys has an improvement of around 20%~30%, however the real runtime improvement is less significant. It's probably because of the thread workload unbalance on multicore, or in another word, some CPUs are not fully loaded during the build. e.g. when do_rootfs_rpm is run, there is probably no other tasks running.

Haven't done any test against single thread running, but it should have a much more considerable impact.

Known problems:
1. fetcher is not working at the moment, maybe bitbake needs an explicit InheritFromOS(), but that produces some inconsistency in PATH in my previous run
2. do_rootfs_rpm has some problem in the first run

Next step and possible future usage:
1. organize remote data to a separate class
2. although we have reverted to fork for task running, a remote data mechanism would work for a more separate build (e.g. distributed build)
3. the added code in bitbake-runtask is to run necessary initializations for a working environment, this is part of cooker. The current cooker seems too heavy and may be broken down into several pieces according to functionalitiies

Thanks,
Qing

Pull URL: git://git.pokylinux.org/poky-contrib.git
Branch: qhe/datasock
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=qhe/datasock

Thanks,
Qing He
---

Qing He (4):
python method fix
sstate.bbclass: workaround
data_smart: remote data
socket based worker data

 bitbake/bin/bitbake-runtask  |   80 +++++++++++++++++++++++++++++++++--------
 bitbake/lib/bb/data_smart.py |   25 ++++++++++++-
 bitbake/lib/bb/methodpool.py |    6 +++
 bitbake/lib/bb/parse/ast.py  |    2 +-
 bitbake/lib/bb/runqueue.py   |   68 ++++++++++++++++++++++++++++++++----
 meta/classes/sstate.bbclass  |    2 +-
 6 files changed, 157 insertions(+), 26 deletions(-)
#7 lore raw
In reply to Qing He
Quoted 8 lines · Qing He

This patch set enables remote data access in the exec'd task process.

Some test data:
I've conducted a poky-image-minimal build on two different machines with and without the patch, the result are as follow:

2 core weybridge: worker count=4, make -j4
without patch

Quoted 16 lines
    real: 161min
    user: 237min
    sys:  56min
  with patch
    real: 138min
    user: 196min
    sys:  45min
4 core nehalem: worker count=4, make -j4
  without patch
    real: 88min
    user: 218min
    sys:  37min
  with patch
    real: 83min
    user: 160min
    sys:  29min

For comparison what numbers does master give? I'm assuming these numbers were with master using exec() so I'd be interested in the comparisons with fork().

Quoted 17 lines

the user+sys has an improvement of around 20%~30%, however the real runtime improvement is less significant. It's probably because of the thread workload unbalance on multicore, or in another word, some CPUs are not fully loaded during the build. e.g. when do_rootfs_rpm is run, there is probably no other tasks running.

Haven't done any test against single thread running, but it should have a much more considerable impact.

Known problems:
1. fetcher is not working at the moment, maybe bitbake needs an explicit InheritFromOS(), but that produces some inconsistency in PATH in my previous run
2. do_rootfs_rpm has some problem in the first run

Quoted 4 lines

Next step and possible future usage:
1. organize remote data to a separate class
2. although we have reverted to fork for task running, a remote data mechanism would work for a more separate build (e.g. distributed build)

It could also be useful for UI interaction with the datastore.

Quoted 3 lines

3. the added code in bitbake-runtask is to run necessary initializations for a working environment, this is part of cooker. The current cooker seems too heavy and may be broken down into several pieces according to functionalitiies

I certainly agree cooker could use some serious refactoring in some form.

Cheers,

Richard

#6 lore raw
In reply to Qing He
Quoted 11 lines

From: Qing He
Sent: Wednesday, December 22, 2010 1:25 AM

Quoted 1 line · Qing

This patch set enables remote data access in the exec'd task process.

It's intended to be proof of concept to an alternative approach to work around the slowness introduced by exec bitbake-runtask. The patch is still WIP.

http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=qhe/datasock

looks the change is not intrusive as I originally thought. :-)

Quoted 86 lines

Thanks,
Qing

Quoted 62 lines

Some test data:
I've conducted a poky-image-minimal build on two different machines with and without the patch, the result are as follow:

2 core weybridge: worker count=4, make -j4
without patch

    real: 161min
    user: 237min
    sys:  56min
  with patch
    real: 138min
    user: 196min
    sys:  45min
4 core nehalem: worker count=4, make -j4
  without patch
    real: 88min
    user: 218min
    sys:  37min
  with patch
    real: 83min
    user: 160min
    sys:  29min

the user+sys has an improvement of around 20%~30%, however the real runtime improvement is less significant. It's probably because of the thread workload unbalance on multicore, or in another word, some CPUs are not fully loaded during the build. e.g. when do_rootfs_rpm is run, there is probably no other tasks running.

Haven't done any test against single thread running, but it should have a much more considerable impact.

Known problems:
1. fetcher is not working at the moment, maybe bitbake needs an explicit InheritFromOS(), but that produces some inconsistency in PATH in my previous run
2. do_rootfs_rpm has some problem in the first run

Next step and possible future usage:
1. organize remote data to a separate class
2. although we have reverted to fork for task running, a remote data mechanism would work for a more separate build (e.g. distributed build)
3. the added code in bitbake-runtask is to run necessary initializations for a working environment, this is part of cooker. The current cooker seems too heavy and may be broken down into several pieces according to functionalitiies

Thanks,
Qing

Pull URL: git://git.pokylinux.org/poky-contrib.git
Branch: qhe/datasock
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=qhe/datasock

Thanks,
Qing He
---

Quoted 7 lines

Qing He (4):
python method fix
sstate.bbclass: workaround
data_smart: remote data
socket based worker data

 bitbake/bin/bitbake-runtask  |   80

+++++++++++++++++++++++++++++++++--------

Quoted 4 lines
 bitbake/lib/bb/data_smart.py |   25 ++++++++++++-
 bitbake/lib/bb/methodpool.py |    6 +++
 bitbake/lib/bb/parse/ast.py  |    2 +-
 bitbake/lib/bb/runqueue.py   |   68

++++++++++++++++++++++++++++++++----

Quoted 3 lines
 meta/classes/sstate.bbclass  |    2 +-
 6 files changed, 157 insertions(+), 26 deletions(-)

_______________________________________________
poky mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/poky