Enable wic in eSDK

From: Chang Rebecca Swee Fun <[email protected]>

Hi all,

Resend as v2 as I realized an issue with BUILDDIR being set was not correct. I made the correction on Patch 4 by setting BUILDDIR to sdkroot variable (eSDK base path) before we alter the variable for bitbake executable file path.

Regards,
Rebecca

-----

Hi all,

As the subject called out: this patch series enable wic in eSDK. The details of what I have done are documented within the commit message. Basically wic requires an OE build environment, but we are using a different environment setup script in eSDK. Hence, I have added some code for wic to explicitly export bitbake variables within eSDK. I have also make wic to use the shared code in scriptpath for oe lib and bitbake path addition to sys.path.

I have run the changes on wic oe-selftest and the tests are passing. What's next: I think it would better to have some test cases for wic within eSDK if this series are merged.

Thanks.

Regards,
Rebecca

The following changes since commit 364f8bcfcbd04e722490f363ad36a15fb7066ba7:

linux-firmware: Bump revision to 65b1c68c (2018-01-11 10:26:07 +0000)

are available in the Git repository at:

git://push.yoctoproject.org/poky-contrib rebeccas/wic-dev

Chang Rebecca Swee Fun (5):
scripts/wic: use scriptpath module to find bitbake path and oe lib
path
scripts/wic: append bitbake executable file path in eSDK environment scripts/wic: fix error of import wic module in eSDK environment scripts/wic: explicitly set BUILDDIR within eSDK classes/populate_sdk_ext: support wic in eSDK

 meta/classes/populate_sdk_ext.bbclass |  2 +-
 scripts/wic                           | 23 ++++++++++++++++++-----
 2 files changed, 19 insertions(+), 6 deletions(-)

2.15.0

From: Chang Rebecca Swee Fun <[email protected]>

Use the scriptpath module in order to standardize the adding of bitbake and meta/lib path to sys.path.

---
 scripts/wic | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

index 097084a6033..0d988757150 100755
@@ -43,13 +43,12 @@ from distutils import spawn
 scripts_path = os.path.abspath(os.path.dirname(__file__))
 lib_path = scripts_path + '/lib'
 sys.path.insert(0, lib_path)
-oe_lib_path = os.path.join(os.path.dirname(scripts_path), 'meta', 'lib')
-sys.path.insert(0, oe_lib_path)
+import scriptpath
+scriptpath.add_oe_lib_path()
 
 bitbake_exe = spawn.find_executable('bitbake')
 if bitbake_exe:
-    bitbake_path = os.path.join(os.path.dirname(bitbake_exe), '../lib')
-    sys.path.insert(0, bitbake_path)
+    bitbake_path = scriptpath.add_bitbake_lib_path()
     from bb import cookerdata
     from bb.main import bitbake_main, BitBakeConfigParameters
 else:

2.15.0

From: Chang Rebecca Swee Fun <[email protected]>

wic needs a set of tools to be available from sysroots. wic will find bitbake executable within the environment, and wic was unable to locate bitbake executable within eSDK because it wasn't setup with the OE build environment script. Hence, we need to add bitbake file path into the environment PATH for wic to be able to discover it and import bb modules.

---
 scripts/wic | 12 ++++++++++++
 1 file changed, 12 insertions(+)

index 0d988757150..293a216d71d 100755
@@ -46,6 +46,18 @@ sys.path.insert(0, lib_path)
 import scriptpath
 scriptpath.add_oe_lib_path()
 
+# Check whether wic is running within eSDK environment
+sdkroot = scripts_path
+if os.environ.get('SDKTARGETSYSROOT'):
+    while sdkroot != '' and sdkroot != os.sep:
+        if os.path.exists(os.path.join(sdkroot, '.devtoolbase')):
+            # .devtoolbase only exists within eSDK
+            # If found, initialize bitbake path for eSDK environment and append to PATH
+            sdkroot = os.path.join(os.path.dirname(scripts_path), 'bitbake', 'bin')
+            os.environ['PATH'] += ":" + sdkroot
+            break
+        sdkroot = os.path.dirname(sdkroot)
+
 bitbake_exe = spawn.find_executable('bitbake')
 if bitbake_exe:
     bitbake_path = scriptpath.add_bitbake_lib_path()

2.15.0

From: Chang Rebecca Swee Fun <[email protected]>

wic modules in scripts/lib/ are needed for wic to work, but path to the python module is not exported in eSDK environment and we were using an absolutized path of wic script within the sysroots.

We now changed to use real script path instead, where the wic modules are located. This will also resolved the tracebacks found when running wic from within the eSDK environment.

Traceback (most recent call last):
File "/tmp/deploy/sdk/poky_sdk/sysroots/x86_64-pokysdk-linux/usr/bin/wic", line 58, in <module>
from wic import WicError
ImportError: No module named 'wic'

---
 scripts/wic | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index 293a216d71d..d9bea228ad5 100755
@@ -40,7 +40,7 @@ from collections import namedtuple
 from distutils import spawn
 
 # External modules
-scripts_path = os.path.abspath(os.path.dirname(__file__))
+scripts_path = os.path.dirname(os.path.realpath(__file__))
 lib_path = scripts_path + '/lib'
 sys.path.insert(0, lib_path)
 import scriptpath

2.15.0

From: Chang Rebecca Swee Fun <[email protected]>

When we run wic within eSDK:

$ wic create mkefidisk -e core-image-minimal

ERROR: BUILDDIR not found, exiting. (Did you forget to source oe-init-build-env?)

In order to figure out variable values, one must have sourced the OE build environment setup script. However, when we are in within the eSDK environment which isn't initialised like the normal OE build environment, we can't use wic utility with eSDK.

Reference:
https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#wic-requirements

While wic ought to be fixed to be able to run without bitbake & native tools [YOCTO #11281], but this is a workaround to set BUILDDIR in the environment so that bitbake environment is setup for wic to build its required native tools.

---
 scripts/wic | 2 ++
 1 file changed, 2 insertions(+)

index d9bea228ad5..7392bc4e7f4 100755
@@ -51,6 +51,8 @@ sdkroot = scripts_path
 if os.environ.get('SDKTARGETSYSROOT'):
     while sdkroot != '' and sdkroot != os.sep:
         if os.path.exists(os.path.join(sdkroot, '.devtoolbase')):
+            # Set BUILDDIR for wic to work within eSDK
+            os.environ['BUILDDIR'] = sdkroot
             # .devtoolbase only exists within eSDK
             # If found, initialize bitbake path for eSDK environment and append to PATH
             sdkroot = os.path.join(os.path.dirname(scripts_path), 'bitbake', 'bin')

2.15.0

From: Chang Rebecca Swee Fun <[email protected]>

Make 'wic' image creation tool/command available in eSDK environment. This would allow eSDK users to manipulate images within eSDK environment.

[YOCTO #12177]

---
 meta/classes/populate_sdk_ext.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index eabc365300a..4f7100dd7e1 100644
@@ -533,7 +533,7 @@ def get_sdk_required_utilities(buildtools_fn, d):
 
 install_tools() {
 	install -d ${SDK_OUTPUT}/${SDKPATHNATIVE}${bindir_nativesdk}
-	scripts="devtool recipetool oe-find-native-sysroot runqemu*"
+	scripts="devtool recipetool oe-find-native-sysroot runqemu* wic"
 	for script in $scripts; do
 		for scriptfn in `find ${SDK_OUTPUT}/${SDKPATH}/${scriptrelpath} -maxdepth 1 -executable -name "$script"`; do
 			lnr ${scriptfn} ${SDK_OUTPUT}/${SDKPATHNATIVE}${bindir_nativesdk}/`basename $scriptfn`

2.15.0

Would it make sense to cover this new functionality by tests?

--
Regards,
Ed

Yes. The tests are in development. Stay tuned.