Skip to contentSkip to conversation

Poky environment tweaks + imagetest change

PATCH In review Richard Purdie 3 participants 7 messages Nov 29, 2010 – Dec 7, 2010

lore
Share & tools
lore
#0 lore raw

cover Poky environment tweaks + imagetest change

This patch set contains an RFC for some changes to the Poky environment handing. These are to allow us to minimise the nunmber of variables our standard tasks see and hence allow improvements to sstate to be made. I think the changes are very positive but they need further testing before they can go into master. In particular this removes variables like DISPLAY and DBUS socket addresses and so forth, only making them available to tasks that specifically require them. Some changes to interactive patch reject handling will probably be needed to complete this change.

The imagetest change I'd appreciate feed back from Scott Garman and Jiajun on.

Pull URL: git://git.pokylinux.org/poky-contrib.git
Branch: rpurdie/tweaks2
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=rpurdie/tweaks2

Cheers,

Richard
---

Richard Purdie (3):
imagetest-qemu: Allow the task to run after any rootfs is created and
also standalone
bitbake: Overhaul environment handling
devshell: Clean up devshell code so this task doesn't polute the
global namespace

 bitbake/bin/bitbake-runtask         |    3 ++
 bitbake/lib/bb/cooker.py            |    3 +-
 bitbake/lib/bb/data.py              |   10 ++++++-
 bitbake/lib/bb/runqueue.py          |    1 +
 bitbake/lib/bb/utils.py             |   48 ++++++++++++++++++++++++-----------
 meta/classes/devshell.bbclass       |   11 +++----
 meta/classes/imagetest-qemu.bbclass |   19 ++++++++++---
 7 files changed, 67 insertions(+), 28 deletions(-)
#1 lore raw

PATCH 1/3 imagetest-qemu: Allow the task to run after any rootfs is created and also standalone

One bitbake invocation can cause multiple images to be generated. We should test each one and we cna do this by running after the rootfs task. Running the tests standlone is also still possible with a new separate standlone task.

---
 meta/classes/imagetest-qemu.bbclass |   19 ++++++++++++++-----
 1 files changed, 14 insertions(+), 5 deletions(-)
meta/classes/imagetest-qemu.bbclass +14 -5
index 07bdf01..3c1586e 100644
@@ -1,8 +1,3 @@
-addtask qemuimagetest before do_build
-# after do_rootfs
-do_qemuimagetest[nostamp] = "1"
-do_qemuimagetest[depends] += "qemu-native:do_populate_sysroot"
-
 # Test related variables
 # By default, TEST_DIR is created under WORKDIR
 TEST_DIR ?= "${WORKDIR}/qemuimagetest"
@@ -12,6 +7,20 @@ TEST_TMP ?= "${TEST_DIR}/tmp"
 TEST_SCEN ?= "sanity"
 
 python do_qemuimagetest() {
+    qemuimagetest_main(d)
+}
+addtask qemuimagetest before do_build after do_rootfs
+do_qemuimagetest[nostamp] = "1"
+do_qemuimagetest[depends] += "qemu-native:do_populate_sysroot"
+
+python do_qemuimagetest_standalone() {
+    qemuimagetest_main(d)
+}
+addtask qemuimagetest_standalone
+do_qemuimagetest_standalone[nostamp] = "1"
+do_qemuimagetest_standalone[depends] += "qemu-native:do_populate_sysroot"
+
+python qemuimagetest_main(d) {
     import sys
     import re
     import os
Signature · 1 line
1.7.0.4
#2 lore raw

Re: [PATCH 1/3] imagetest-qemu: Allow the task to run after any rootfs is created and also standalone

Quoted 27 lines

One bitbake invocation can cause multiple images to be generated. We
should test each one and we cna do this by running after the rootfs
task. Running the tests standlone is also still possible with a new separate standlone task.

---
 meta/classes/imagetest-qemu.bbclass |   19 ++++++++++++++-----
 1 files changed, 14 insertions(+), 5 deletions(-)
(quoted diff) +0 -0

b/meta/classes/imagetest-qemu.bbclass index 07bdf01..3c1586e 100644 --- a/meta/classes/imagetest-qemu.bbclass +++ b/meta/classes/imagetest-qemu.bbclass @@ -1,8 +1,3 @@ -addtask qemuimagetest before do_build -# after do_rootfs -do_qemuimagetest[nostamp] = "1" -do_qemuimagetest[depends] += "qemu-native:do_populate_sysroot" -
# Test related variables
# By default, TEST_DIR is created under WORKDIR TEST_DIR ?=
"${WORKDIR}/qemuimagetest"

(quoted diff) +1 -0
@@ -12,6 +7,20 @@ TEST_TMP ?= "${TEST_DIR}/tmp"
 TEST_SCEN ?= "sanity"
 
 python do_qemuimagetest() {
+    qemuimagetest_main(d) +} +addtask qemuimagetest before do_build

after do_rootfs +do_qemuimagetest[nostamp] = "1"

(quoted diff) +3 -0
+do_qemuimagetest[depends] += "qemu-native:do_populate_sysroot" +
+python do_qemuimagetest_standalone() { +    qemuimagetest_main(d) +}
+addtask qemuimagetest_standalone +do_qemuimagetest_standalone[nostamp]
Quoted 5 lines

= "1" +do_qemuimagetest_standalone[depends] += "qemu-native:do_populate_sysroot" + +python qemuimagetest_main(d) {
import sys
import re
import os

Hi Richard,
Seems we should use keyword "def" qemuimagetest_main definition. After this modification, it can work on my side.

meta/classes/imagetest-qemu.bbclass +15 -7
index 07bdf01..8751eeb 100644
@@ -1,8 +1,3 @@
-addtask qemuimagetest before do_build
-# after do_rootfs
-do_qemuimagetest[nostamp] = "1"
-do_qemuimagetest[depends] += "qemu-native:do_populate_sysroot"
-
 # Test related variables
 # By default, TEST_DIR is created under WORKDIR
 TEST_DIR ?= "${WORKDIR}/qemuimagetest"
@@ -11,7 +6,21 @@ TEST_RESULT ?= "${TEST_DIR}/result"
 TEST_TMP ?= "${TEST_DIR}/tmp"
 TEST_SCEN ?= "sanity"
 
-python do_qemuimagetest() {
+python do_qemuimagetest(){
+    qemuimagetest_main(d)
+}
+addtask qemuimagetest before do_build after do_rootfs
+do_qemuimagetest[nostamp] = "1"
+do_qemuimagetest[depends] += "qemu-native:do_populate_sysroot"
+
+python do_qemuimagetest_standalone(){
+    qemuimagetest_main(d)
+}
+addtask qemuimagetest_standalone
+do_qemuimagetest_standalone[nostamp] = "1"
+do_qemuimagetest_standalone[depends] += "qemu-native:do_populate_sysroot"
+
+def qemuimagetest_main(d):
     import sys
     import re
     import os
@@ -142,4 +151,3 @@ python do_qemuimagetest() {
 
     if ret != 0:
         raise bb.build.FuncFailed("Some testcases fail, pls. check test result and test log!!!")
-}

Best Regards,
Jiajun

Acked-by on v1 1/3
#3 lore raw

Re: [PATCH 1/3] imagetest-qemu: Allow the task to run after any rootfs is created and also standalone

In reply to Richard Purdie
Quoted 5 lines · Richard Purdie

One bitbake invocation can cause multiple images to be generated. We should test each one and we cna do this by running after the rootfs task. Running the tests standlone is also still possible with a new separate standlone task.

(with Jiajun's modification)

Quoted 38 lines
---
  meta/classes/imagetest-qemu.bbclass |   19 ++++++++++++++-----
  1 files changed, 14 insertions(+), 5 deletions(-)
meta/classes/imagetest-qemu.bbclass +14 -5
index 07bdf01..3c1586e 100644
@@ -1,8 +1,3 @@
-addtask qemuimagetest before do_build
-# after do_rootfs
-do_qemuimagetest[nostamp] = "1"
-do_qemuimagetest[depends] += "qemu-native:do_populate_sysroot"
-
  # Test related variables
  # By default, TEST_DIR is created under WORKDIR
  TEST_DIR ?= "${WORKDIR}/qemuimagetest"
@@ -12,6 +7,20 @@ TEST_TMP ?= "${TEST_DIR}/tmp"
  TEST_SCEN ?= "sanity"
 
  python do_qemuimagetest() {
+    qemuimagetest_main(d)
+}
+addtask qemuimagetest before do_build after do_rootfs
+do_qemuimagetest[nostamp] = "1"
+do_qemuimagetest[depends] += "qemu-native:do_populate_sysroot"
+
+python do_qemuimagetest_standalone() {
+    qemuimagetest_main(d)
+}
+addtask qemuimagetest_standalone
+do_qemuimagetest_standalone[nostamp] = "1"
+do_qemuimagetest_standalone[depends] += "qemu-native:do_populate_sysroot"
+
+python qemuimagetest_main(d) {
      import sys
      import re
      import os
Signature · 2 lines
Scott Garman
Embedded Linux Distro Engineer - Yocto Project
#4 lore raw

Re: [PATCH 1/3] imagetest-qemu: Allow the task to run after any rootfs is created and also standalone

Quoted 10 lines · Scott Garman
Quoted 5 lines · Richard Purdie

One bitbake invocation can cause multiple images to be generated. We should test each one and we cna do this by running after the rootfs task. Running the tests standlone is also still possible with a new separate standlone task.

(with Jiajun's modification)

Thanks, I've merged this into master with the fix.

Cheers,

Richard

#5 lore raw

PATCH 2/3 bitbake: Overhaul environment handling

In reply to Richard Purdie

Currently, anything whitelisted in the environment makes it into the worker processes. This is undesireable and the worker environment should be as clean as possible. This patch adapts bitbake sosme variables are loaded into bitbake's datastore but not exported by default. Any variable can be exported by setting its export flag.

Currently, this code only finalises the environment in he worker as doing so in the server means variables are unavailable in the worker. If we switch back to fork() calls instead of exec() this code will need revisting.

---
 bitbake/bin/bitbake-runtask |    3 ++
 bitbake/lib/bb/cooker.py    |    3 +-
 bitbake/lib/bb/data.py      |   10 ++++++++-
 bitbake/lib/bb/runqueue.py  |    1 +
 bitbake/lib/bb/utils.py     |   48 +++++++++++++++++++++++++++++-------------
 5 files changed, 48 insertions(+), 17 deletions(-)
bitbake/bin/bitbake-runtask +3 -0
index 88101a5..9079f57 100755
@@ -100,6 +100,9 @@ the_data = cooker.bb_cache.loadDataFull(fn, cooker.get_file_appends(fn), cooker.
 cooker.bb_cache.setData(fn, buildfile, the_data)
 cooker.bb_cache.handle_data(fn, cooker.status)
 
+exportlist = bb.utils.preserved_envvars_export_list()
+bb.utils.filter_environment(exportlist)
+
 if taskname.endswith("_setscene"):
     the_data.setVarFlag(taskname, "quieterrors", "1")
 
bitbake/lib/bb/cooker.py +2 -1
index 054dac8..3cd358c 100644
@@ -918,7 +918,8 @@ class BBCooker:
     def pre_serve(self):
         # Empty the environment. The environment will be populated as
         # necessary from the data store.
-        bb.utils.empty_environment()
+        #bb.utils.empty_environment()
+        return
 
     def post_serve(self):
         bb.event.fire(CookerExit(), self.configuration.event_data)
bitbake/lib/bb/data.py +9 -1
index fee10cc..d4d43fd 100644
@@ -161,10 +161,12 @@ def expandKeys(alterdata, readdata = None):
 
 def inheritFromOS(d):
     """Inherit variables from the environment."""
+    exportlist = bb.utils.preserved_envvars_export_list()
     for s in os.environ.keys():
         try:
             setVar(s, os.environ[s], d)
-            setVarFlag(s, "export", True, d)
+            if s in exportlist:
+                setVarFlag(s, "export", True, d)
         except TypeError:
             pass
 
@@ -244,6 +246,12 @@ def export_vars(d):
             pass
     return ret
 
+def export_envvars(v, d):
+    for s in os.environ.keys():
+        if s not in v:
+            v[s] = os.environ[s]
+    return v
+
 def emit_func(func, o=sys.__stdout__, d = init()):
     """Emits all items in the data store in a format such that it can be sourced by a shell."""
 
bitbake/lib/bb/runqueue.py +1 -0
index 3300db7..e26aa4e 100644
@@ -1062,6 +1062,7 @@ class RunQueueExecute:
             the_data = self.cooker.bb_cache.loadDataFull(fn, self.cooker.get_file_appends(fn), self.cooker.configuration.data)
 
             env = bb.data.export_vars(the_data)
+            env = bb.data.export_envvars(env, the_data)
 
             taskdep = self.rqdata.dataCache.task_deps[fn]
             if 'fakeroot' in taskdep and taskname in taskdep['fakeroot']:
bitbake/lib/bb/utils.py +33 -15

75 lines not on this page · raw message

Signature · 1 line
1.7.0.4
#6 lore raw

PATCH 3/3 devshell: Clean up devshell code so this task doesn't polute the global namespace

In reply to Richard Purdie
---
 meta/classes/devshell.bbclass |   11 +++++------
 1 files changed, 5 insertions(+), 6 deletions(-)
meta/classes/devshell.bbclass +5 -6
index d08056b..4fb3e63 100644
@@ -1,15 +1,14 @@
-EXTRA_OEMAKE[export] = "1"
-
 do_devshell[dirs] = "${S}"
 do_devshell[nostamp] = "1"
 
-export DISPLAY
-export DBUS_SESSION_BUS_ADDRESS
-export XAUTHORITY ?= "${HOME}/.Xauthority"
-export FAKEROOTENV
+XAUTHORITY ?= "${HOME}/.Xauthority"
 
 devshell_do_devshell() {
+	export DISPLAY='${DISPLAY}'
+	export DBUS_SESSION_BUS_ADDRESS='${DBUS_SESSION_BUS_ADDRESS}'
+	export XAUTHORITY='${XAUTHORITY}'
 	export TERMWINDOWTITLE="Bitbake Developer Shell"
+	export EXTRA_OEMAKE='${EXTRA_OEMAKE}'
 	${TERMCMD}
 	if [ $? -ne 0 ]; then
 	    echo "Fatal: '${TERMCMD}' not found. Check TERMCMD variable."
Signature · 1 line
1.7.0.4