Poky environment tweaks + imagetest change
cover Poky environment tweaks + imagetest change
This patch set contains an RFC for some changes to the Poky environment handing. These are to allow us to minimise the nunmber of variables our standard tasks see and hence allow improvements to sstate to be made. I think the changes are very positive but they need further testing before they can go into master. In particular this removes variables like DISPLAY and DBUS socket addresses and so forth, only making them available to tasks that specifically require them. Some changes to interactive patch reject handling will probably be needed to complete this change.
The imagetest change I'd appreciate feed back from Scott Garman and Jiajun on.
Pull URL: git://git.pokylinux.org/poky-contrib.git
Branch: rpurdie/tweaks2
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=rpurdie/tweaks2
Cheers,
Richard
---
Richard Purdie (3):
imagetest-qemu: Allow the task to run after any rootfs is created and
also standalone
bitbake: Overhaul environment handling
devshell: Clean up devshell code so this task doesn't polute the
global namespace
bitbake/bin/bitbake-runtask | 3 ++ bitbake/lib/bb/cooker.py | 3 +- bitbake/lib/bb/data.py | 10 ++++++- bitbake/lib/bb/runqueue.py | 1 + bitbake/lib/bb/utils.py | 48 ++++++++++++++++++++++++----------- meta/classes/devshell.bbclass | 11 +++---- meta/classes/imagetest-qemu.bbclass | 19 ++++++++++--- 7 files changed, 67 insertions(+), 28 deletions(-)
PATCH 1/3 imagetest-qemu: Allow the task to run after any rootfs is created and also standalone
One bitbake invocation can cause multiple images to be generated. We should test each one and we cna do this by running after the rootfs task. Running the tests standlone is also still possible with a new separate standlone task.
--- meta/classes/imagetest-qemu.bbclass | 19 ++++++++++++++----- 1 files changed, 14 insertions(+), 5 deletions(-)
meta/classes/imagetest-qemu.bbclass +14 -5
@@ -1,8 +1,3 @@ -addtask qemuimagetest before do_build -# after do_rootfs -do_qemuimagetest[nostamp] = "1" -do_qemuimagetest[depends] += "qemu-native:do_populate_sysroot" - # Test related variables # By default, TEST_DIR is created under WORKDIR TEST_DIR ?= "${WORKDIR}/qemuimagetest"
@@ -12,6 +7,20 @@ TEST_TMP ?= "${TEST_DIR}/tmp" TEST_SCEN ?= "sanity" python do_qemuimagetest() { + qemuimagetest_main(d) +} +addtask qemuimagetest before do_build after do_rootfs +do_qemuimagetest[nostamp] = "1" +do_qemuimagetest[depends] += "qemu-native:do_populate_sysroot" + +python do_qemuimagetest_standalone() { + qemuimagetest_main(d) +} +addtask qemuimagetest_standalone +do_qemuimagetest_standalone[nostamp] = "1" +do_qemuimagetest_standalone[depends] += "qemu-native:do_populate_sysroot" + +python qemuimagetest_main(d) { import sys import re import os
Signature · 1 line
1.7.0.4
PATCH 2/3 bitbake: Overhaul environment handling
In reply to Richard PurdieCurrently, anything whitelisted in the environment makes it into the worker processes. This is undesireable and the worker environment should be as clean as possible. This patch adapts bitbake sosme variables are loaded into bitbake's datastore but not exported by default. Any variable can be exported by setting its export flag.
Currently, this code only finalises the environment in he worker as doing so in the server means variables are unavailable in the worker. If we switch back to fork() calls instead of exec() this code will need revisting.
--- bitbake/bin/bitbake-runtask | 3 ++ bitbake/lib/bb/cooker.py | 3 +- bitbake/lib/bb/data.py | 10 ++++++++- bitbake/lib/bb/runqueue.py | 1 + bitbake/lib/bb/utils.py | 48 +++++++++++++++++++++++++++++------------- 5 files changed, 48 insertions(+), 17 deletions(-)
bitbake/bin/bitbake-runtask +3 -0
@@ -100,6 +100,9 @@ the_data = cooker.bb_cache.loadDataFull(fn, cooker.get_file_appends(fn), cooker. cooker.bb_cache.setData(fn, buildfile, the_data) cooker.bb_cache.handle_data(fn, cooker.status) +exportlist = bb.utils.preserved_envvars_export_list() +bb.utils.filter_environment(exportlist) + if taskname.endswith("_setscene"): the_data.setVarFlag(taskname, "quieterrors", "1")
bitbake/lib/bb/cooker.py +2 -1
@@ -918,7 +918,8 @@ class BBCooker: def pre_serve(self): # Empty the environment. The environment will be populated as # necessary from the data store. - bb.utils.empty_environment() + #bb.utils.empty_environment() + return def post_serve(self): bb.event.fire(CookerExit(), self.configuration.event_data)
bitbake/lib/bb/data.py +9 -1
@@ -161,10 +161,12 @@ def expandKeys(alterdata, readdata = None): def inheritFromOS(d): """Inherit variables from the environment.""" + exportlist = bb.utils.preserved_envvars_export_list() for s in os.environ.keys(): try: setVar(s, os.environ[s], d) - setVarFlag(s, "export", True, d) + if s in exportlist: + setVarFlag(s, "export", True, d) except TypeError: pass
@@ -244,6 +246,12 @@ def export_vars(d): pass return ret +def export_envvars(v, d): + for s in os.environ.keys(): + if s not in v: + v[s] = os.environ[s] + return v + def emit_func(func, o=sys.__stdout__, d = init()): """Emits all items in the data store in a format such that it can be sourced by a shell."""
bitbake/lib/bb/runqueue.py +1 -0
@@ -1062,6 +1062,7 @@ class RunQueueExecute: the_data = self.cooker.bb_cache.loadDataFull(fn, self.cooker.get_file_appends(fn), self.cooker.configuration.data) env = bb.data.export_vars(the_data) + env = bb.data.export_envvars(env, the_data) taskdep = self.rqdata.dataCache.task_deps[fn] if 'fakeroot' in taskdep and taskname in taskdep['fakeroot']:
bitbake/lib/bb/utils.py +33 -15
75 lines not on this page · raw message
Signature · 1 line
1.7.0.4
PATCH 3/3 devshell: Clean up devshell code so this task doesn't polute the global namespace
In reply to Richard Purdie--- meta/classes/devshell.bbclass | 11 +++++------ 1 files changed, 5 insertions(+), 6 deletions(-)
meta/classes/devshell.bbclass +5 -6
@@ -1,15 +1,14 @@ -EXTRA_OEMAKE[export] = "1" - do_devshell[dirs] = "${S}" do_devshell[nostamp] = "1" -export DISPLAY -export DBUS_SESSION_BUS_ADDRESS -export XAUTHORITY ?= "${HOME}/.Xauthority" -export FAKEROOTENV +XAUTHORITY ?= "${HOME}/.Xauthority" devshell_do_devshell() { + export DISPLAY='${DISPLAY}' + export DBUS_SESSION_BUS_ADDRESS='${DBUS_SESSION_BUS_ADDRESS}' + export XAUTHORITY='${XAUTHORITY}' export TERMWINDOWTITLE="Bitbake Developer Shell" + export EXTRA_OEMAKE='${EXTRA_OEMAKE}' ${TERMCMD} if [ $? -ne 0 ]; then echo "Fatal: '${TERMCMD}' not found. Check TERMCMD variable."
Signature · 1 line
1.7.0.4
Re: [PATCH 1/3] imagetest-qemu: Allow the task to run after any rootfs is created and also standalone
In reply to Richard PurdieQuoted 27 lines
One bitbake invocation can cause multiple images to be generated. We
should test each one and we cna do this by running after the rootfs
task. Running the tests standlone is also still possible with a new separate standlone task.Signed-off-by Richard Purdie <[email protected]>--- meta/classes/imagetest-qemu.bbclass | 19 ++++++++++++++----- 1 files changed, 14 insertions(+), 5 deletions(-)(quoted diff) +0
-0b/meta/classes/imagetest-qemu.bbclass index 07bdf01..3c1586e 100644 --- a/meta/classes/imagetest-qemu.bbclass +++ b/meta/classes/imagetest-qemu.bbclass @@ -1,8 +1,3 @@ -addtask qemuimagetest before
do_build-# afterdo_rootfs-do_qemuimagetest[nostamp] = "1" -do_qemuimagetest[depends] += "qemu-native:do_populate_sysroot" -
# Test related variables
# By default,TEST_DIRis created under WORKDIRTEST_DIR?=
"${WORKDIR}/qemuimagetest"(quoted diff) +1
-0@@ -12,6 +7,20 @@ TEST_TMP ?= "${TEST_DIR}/tmp" TEST_SCEN ?= "sanity" python do_qemuimagetest() { + qemuimagetest_main(d) +} +addtask qemuimagetest before do_buildafter
do_rootfs+do_qemuimagetest[nostamp] = "1"(quoted diff) +3
-0+do_qemuimagetest[depends] += "qemu-native:do_populate_sysroot" + +python do_qemuimagetest_standalone() { + qemuimagetest_main(d) +} +addtask qemuimagetest_standalone +do_qemuimagetest_standalone[nostamp]
Quoted 5 lines
= "1" +do_qemuimagetest_standalone[depends] += "qemu-native:do_populate_sysroot" + +python
qemuimagetest_main(d){
import sys
import re
import os
Hi Richard,
Seems we should use keyword "def" qemuimagetest_main definition. After this modification, it can work on my side.
meta/classes/imagetest-qemu.bbclass +15 -7
@@ -1,8 +1,3 @@ -addtask qemuimagetest before do_build -# after do_rootfs -do_qemuimagetest[nostamp] = "1" -do_qemuimagetest[depends] += "qemu-native:do_populate_sysroot" - # Test related variables # By default, TEST_DIR is created under WORKDIR TEST_DIR ?= "${WORKDIR}/qemuimagetest"
@@ -11,7 +6,21 @@ TEST_RESULT ?= "${TEST_DIR}/result" TEST_TMP ?= "${TEST_DIR}/tmp" TEST_SCEN ?= "sanity" -python do_qemuimagetest() { +python do_qemuimagetest(){ + qemuimagetest_main(d) +} +addtask qemuimagetest before do_build after do_rootfs +do_qemuimagetest[nostamp] = "1" +do_qemuimagetest[depends] += "qemu-native:do_populate_sysroot" + +python do_qemuimagetest_standalone(){ + qemuimagetest_main(d) +} +addtask qemuimagetest_standalone +do_qemuimagetest_standalone[nostamp] = "1" +do_qemuimagetest_standalone[depends] += "qemu-native:do_populate_sysroot" + +def qemuimagetest_main(d): import sys import re import os
@@ -142,4 +151,3 @@ python do_qemuimagetest() { if ret != 0: raise bb.build.FuncFailed("Some testcases fail, pls. check test result and test log!!!") -}
Best Regards,
Jiajun
Re: [PATCH 1/3] imagetest-qemu: Allow the task to run after any rootfs is created and also standalone
In reply to Richard PurdieQuoted 5 lines · Richard Purdie
One bitbake invocation can cause multiple images to be generated. We should test each one and we cna do this by running after the rootfs task. Running the tests standlone is also still possible with a new separate standlone task.
Signed-off-by Richard Purdie<[email protected]>
(with Jiajun's modification)
Quoted 38 lines
--- meta/classes/imagetest-qemu.bbclass | 19 ++++++++++++++----- 1 files changed, 14 insertions(+), 5 deletions(-)meta/classes/imagetest-qemu.bbclass +14
-5@@ -1,8 +1,3 @@ -addtask qemuimagetest before do_build -# after do_rootfs -do_qemuimagetest[nostamp] = "1" -do_qemuimagetest[depends] += "qemu-native:do_populate_sysroot" - # Test related variables # By default, TEST_DIR is created under WORKDIR TEST_DIR ?= "${WORKDIR}/qemuimagetest"@@ -12,6 +7,20 @@ TEST_TMP ?= "${TEST_DIR}/tmp" TEST_SCEN ?= "sanity" python do_qemuimagetest() { + qemuimagetest_main(d) +} +addtask qemuimagetest before do_build after do_rootfs +do_qemuimagetest[nostamp] = "1" +do_qemuimagetest[depends] += "qemu-native:do_populate_sysroot" + +python do_qemuimagetest_standalone() { + qemuimagetest_main(d) +} +addtask qemuimagetest_standalone +do_qemuimagetest_standalone[nostamp] = "1" +do_qemuimagetest_standalone[depends] += "qemu-native:do_populate_sysroot" + +python qemuimagetest_main(d) { import sys import re import os
Signature · 2 lines
Scott Garman Embedded Linux Distro Engineer - Yocto Project
Re: [PATCH 1/3] imagetest-qemu: Allow the task to run after any rootfs is created and also standalone
Quoted 10 lines · Scott Garman
Quoted 5 lines · Richard Purdie
One bitbake invocation can cause multiple images to be generated. We should test each one and we cna do this by running after the rootfs task. Running the tests standlone is also still possible with a new separate standlone task.
Signed-off-by Richard Purdie<[email protected]>Acked-by Scott Garman <[email protected]>(with Jiajun's modification)
Thanks, I've merged this into master with the fix.
Cheers,
Richard