Skip to content

RFC socketed based data connection

Qing He poky 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 RFC Latest4 patches 3 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/4python method fixbitbake/lib/bb/methodpool.py bitbake/lib/bb/parse/ast.py+7 -10 replies
2/4sstate.bbclass: workaroundmeta/classes/sstate.bbclass+1 -10 replies
3/4data_smart: remote databitbake/lib/bb/data_smart.py+24 -10 replies
4/4socket based worker databitbake/bin/bitbake-runtask bitbake/lib/bb/runqueue.py+125 -230 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [RFC 0/4] socketed based data connection

This patch set enables remote data access in the exec'd task process.

Some test data:
I've conducted a poky-image-minimal build on two different machines with and without the patch, the result are as follow:

2 core weybridge: worker count=4, make -j4
without patch

    real: 161min
    user: 237min
    sys:  56min
  with patch
    real: 138min
    user: 196min
    sys:  45min
4 core nehalem: worker count=4, make -j4
  without patch
    real: 88min
    user: 218min
    sys:  37min
  with patch
    real: 83min
    user: 160min
    sys:  29min

the user+sys has an improvement of around 20%~30%, however the real runtime improvement is less significant. It's probably because of the thread workload unbalance on multicore, or in another word, some CPUs are not fully loaded during the build. e.g. when do_rootfs_rpm is run, there is probably no other tasks running.

Haven't done any test against single thread running, but it should have a much more considerable impact.

Known problems:
1. fetcher is not working at the moment, maybe bitbake needs an explicit InheritFromOS(), but that produces some inconsistency in PATH in my previous run
2. do_rootfs_rpm has some problem in the first run

Next step and possible future usage:
1. organize remote data to a separate class
2. although we have reverted to fork for task running, a remote data mechanism would work for a more separate build (e.g. distributed build)
3. the added code in bitbake-runtask is to run necessary initializations for a working environment, this is part of cooker. The current cooker seems too heavy and may be broken down into several pieces according to functionalitiies

Thanks,
Qing

Pull URL: git://git.pokylinux.org/poky-contrib.git
Branch: qhe/datasock
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=qhe/datasock

Thanks,
Qing He
---

Qing He (4):
python method fix
sstate.bbclass: workaround
data_smart: remote data
socket based worker data

 bitbake/bin/bitbake-runtask  |   80 +++++++++++++++++++++++++++++++++--------
 bitbake/lib/bb/data_smart.py |   25 ++++++++++++-
 bitbake/lib/bb/methodpool.py |    6 +++
 bitbake/lib/bb/parse/ast.py  |    2 +-
 bitbake/lib/bb/runqueue.py   |   68 ++++++++++++++++++++++++++++++++----
 meta/classes/sstate.bbclass  |    2 +-
 6 files changed, 157 insertions(+), 26 deletions(-)