Skip to contentSkip to conversation

Extensible SDK: 3 fixes

PATCH In review Chen Qi 2 participants 5 messages Jun 10, 2015 – Jun 15, 2015

lore
Share & tools
lore

Chen Qi

author
#0 lore raw

v2 cover Extensible SDK: 3 fixes

Changes since V1:
*) Make use of bb.utils.edit_metadata.
*) Introduce a whitelist mechanism to allow exceptions.

The following changes since commit de6a26b95a7f7bd8f9dc47ab35d8b07ba671f4eb:

x264: use x86 over-ride instead of i586 (2015-06-08 17:32:46 +0100)

are available in the git repository at:

git://git.openembedded.org/openembedded-core-contrib ChenQi/ext-sdk-3-fixes
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=ChenQi/ext-sdk-3-fixes

Chen Qi (3):
populate_sdk_ext: install the latest buildtools-tarball
copy_buildsystem: make sure bitbake directory is copied
populate_sdk_ext: consider custom configuration in local.conf

 meta/classes/populate_sdk_ext.bbclass | 21 ++++++++++++++++++++-
 meta/lib/oe/copy_buildsystem.py       |  7 +++----
 2 files changed, 23 insertions(+), 5 deletions(-)
Signature · 1 line
1.9.1
author
#1 lore raw

PATCH v2 1/3 populate_sdk_ext: install the latest buildtools-tarball

If we do bitbake buildtools-tarball' and then after one day do bitbake core-image-minimal -c populate_sdk_ext', we would meet errors like below.

| install: cannot stat '/buildarea2/chenqi/poky/build-systemd/tmp/deploy/sdk/ poky-glibc-x86_64-buildtools-tarball-core2-64-buildtools-nativesdk-standalone -1.8+snapshot-20150429.sh': No such file or directory

The problem is that the output name for buildtools-tarball has ${DATE} in it. So if populate_sdk_ext task is executed but buildtools-tarball is not rebuilt, the above error appears.

Instead of hardcoding ${DISTRO_VERSION} which consists of ${DATE} in the install_tools() function, we should find the latest buildtools-tarball based on the modification time and install it.

[YOCTO #7674]

Signed-off-by Chen Qi <[email protected]>
---
 meta/classes/populate_sdk_ext.bbclass | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
meta/classes/populate_sdk_ext.bbclass +3 -1
index dc2c58e..2fc4c11 100644
@@ -168,7 +168,9 @@ install_tools() {
 	ln -sr ${SDK_OUTPUT}/${SDKPATH}/${scriptrelpath}/recipetool ${SDK_OUTPUT}/${SDKPATHNATIVE}${bindir_nativesdk}/recipetool
 	touch ${SDK_OUTPUT}/${SDKPATH}/.devtoolbase
 
-	install ${SDK_DEPLOY}/${DISTRO}-${TCLIBC}-${SDK_ARCH}-buildtools-tarball-${TUNE_PKGARCH}-buildtools-nativesdk-standalone-${DISTRO_VERSION}.sh ${SDK_OUTPUT}/${SDKPATH}
+	# find latest buildtools-tarball and install it
+	buildtools_path=`ls -t1 ${SDK_DEPLOY}/${DISTRO}-${TCLIBC}-${SDK_ARCH}-buildtools-tarball-${TUNE_PKGARCH}-buildtools-nativesdk-standalone-*.sh | head -n1`
+	install $buildtools_path ${SDK_OUTPUT}/${SDKPATH}
 
 	install ${SDK_DEPLOY}/${BUILD_ARCH}-nativesdk-libc.tar.bz2 ${SDK_OUTPUT}/${SDKPATH}
 }
Signature · 1 line
1.9.1
author
#2 lore raw

PATCH v2 2/3 copy_buildsystem: make sure bitbake directory is copied

In reply to Chen Qi

The previous code assumes that bitbake/ directory is under the core layer. This is the case for Yocto project. But users might clone oe-core and bitbake separately. So we use bb.__file__ to locate the bitbake directory to make sure it's copied into the extensible SDK.

Signed-off-by Chen Qi <[email protected]>
---
 meta/lib/oe/copy_buildsystem.py | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)
meta/lib/oe/copy_buildsystem.py +3 -4
index cf7fada..979578c 100644
@@ -28,11 +28,10 @@ class BuildSystem(object):
         layers.append(corebase)
 
         corebase_files = self.d.getVar('COREBASE_FILES', True).split()
-
-        # bitbake belongs in corebase so make sure it goes there
-        if "bitbake" not in corebase_files:
-            corebase_files.append("bitbake")
         corebase_files = [corebase + '/' +x for x in corebase_files]
+        # Make sure bitbake goes in
+        bitbake_dir = bb.__file__.rsplit('/', 3)[0]
+        corebase_files.append(bitbake_dir)
 
         for layer in layers:
             layerconf = os.path.join(layer, 'conf', 'layer.conf')
Signature · 1 line
1.9.1
author
#3 lore raw

PATCH v2 3/3 populate_sdk_ext: consider custom configuration in local.conf

In reply to Chen Qi

Copy the contents of local.conf under TOPDIR into the final generated local.conf. In this way, custom settings are also made into the final local.conf like IMAGE_INSTALL, DISTRO_FEATURES, VIRTUAL-RUNTIME_xxx, etc.

Before this change, installing extensible SDK would usually report failure when preparing the build system if the user has custom configuration for DISTRO_FEATURES in local.conf. Also, items in IMAGE_INSTALL_append in local.conf also don't get built correctly.

This patch solves the above problem by making use of the bb.utils.edit_metadata.

In addition, we check to avoid any setting that might lead to host path bleeding into SDK's configuration. Basically, variables with values starting with '/' are removed. A whitelist mechanism is introduced so that users could specify variables that should not be ignored. The name of the whitelist is SDK_LOCAL_CONF_WHITELIST.

[YOCTO #7616]

Signed-off-by Chen Qi <[email protected]>
---
 meta/classes/populate_sdk_ext.bbclass | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
meta/classes/populate_sdk_ext.bbclass +17 -0
index 2fc4c11..08130d4 100644
@@ -16,6 +16,7 @@ SDK_RDEPENDS_append_task-populate-sdk-ext = " ${SDK_TARGETS}"
 SDK_RELOCATE_AFTER_INSTALL_task-populate-sdk-ext = "0"
 
 SDK_META_CONF_WHITELIST ?= "MACHINE DISTRO PACKAGE_CLASSES"
+SDK_LOCAL_CONF_WHITELIST ?= ""
 
 SDK_TARGETS ?= "${PN}"
 OE_INIT_ENV_SCRIPT ?= "oe-init-build-env"

@@ -108,12 +109,28 @@ python copy_buildsystem () {
         f.write('    "\n')
 
     # Create local.conf
+    local_conf_whitelist = d.getVar('SDK_LOCAL_CONF_WHITELIST', True).split()
+    def handle_var(varname, origvalue, op, newlines):
+        if origvalue.strip().startswith('/') and not varname in local_conf_whitelist:
+            newlines.append('# Removed original setting of %s\n' % varname)
+            return None, op, 0, True
+        else:
+            return origvalue, op, 0, True
+    varlist = ['[^#=+ ]*']
+    builddir = d.getVar('TOPDIR', True)
+    with open(builddir + '/conf/local.conf', 'r') as f:
+        oldlines = f.readlines()
+    (updated, newlines) = bb.utils.edit_metadata(oldlines, varlist, handle_var)
+
     with open(baseoutpath + '/conf/local.conf', 'w') as f:
         f.write('# WARNING: this configuration has been automatically generated and in\n')
         f.write('# most cases should not be edited. If you need more flexibility than\n')
         f.write('# this configuration provides, it is strongly suggested that you set\n')
         f.write('# up a proper instance of the full build system and use that instead.\n\n')
 
+        for line in newlines:
+            f.write(line)
+
         f.write('INHERIT += "%s"\n\n' % 'uninative')
         f.write('CONF_VERSION = "%s"\n\n' % d.getVar('CONF_VERSION'))
Signature · 1 line
1.9.1
#4 lore raw

Re: [PATCH V2 3/3] populate_sdk_ext: consider custom configuration in local.conf

Quoted 61 lines · On Wednesday 10 June 2015 15:25:24 Chen Qi wrote:

Copy the contents of local.conf under TOPDIR into the final generated local.conf. In this way, custom settings are also made into the final local.conf like IMAGE_INSTALL, DISTRO_FEATURES, VIRTUAL-RUNTIME_xxx, etc.

Before this change, installing extensible SDK would usually report failure when preparing the build system if the user has custom configuration for DISTRO_FEATURES in local.conf. Also, items in IMAGE_INSTALL_append in local.conf also don't get built correctly.

This patch solves the above problem by making use of the bb.utils.edit_metadata.

In addition, we check to avoid any setting that might lead to host path bleeding into SDK's configuration. Basically, variables with values starting with '/' are removed. A whitelist mechanism is introduced so that users could specify variables that should not be ignored. The name of the whitelist is SDK_LOCAL_CONF_WHITELIST.

[YOCTO #7616]

Signed-off-by Chen Qi <[email protected]>
---
 meta/classes/populate_sdk_ext.bbclass | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
(quoted diff) +0 -0

b/meta/classes/populate_sdk_ext.bbclass index 2fc4c11..08130d4 100644

meta/classes/populate_sdk_ext.bbclass +0 -0
@@ -16,6 +16,7 @@ SDK_RDEPENDS_append_task-populate-sdk-ext = "

${SDK_TARGETS}" SDK_RELOCATE_AFTER_INSTALL_task-populate-sdk-ext = "0"

SDK_META_CONF_WHITELIST ?= "MACHINE DISTRO PACKAGE_CLASSES"

meta/classes/populate_sdk_ext.bbclass +2 -0
+SDK_LOCAL_CONF_WHITELIST ?= ""
 
 SDK_TARGETS ?= "${PN}"
 OE_INIT_ENV_SCRIPT ?= "oe-init-build-env"

@@ -108,12 +109,28 @@ python copy_buildsystem () {
         f.write('    "\n')
 
     # Create local.conf
+    local_conf_whitelist = d.getVar('SDK_LOCAL_CONF_WHITELIST',

True).split() + def handle_var(varname, origvalue, op, newlines):
+ if origvalue.strip().startswith('/') and not varname in
local_conf_whitelist: + newlines.append('# Removed original
setting of %s\n' % varname) + return None, op, 0, True

meta/classes/populate_sdk_ext.bbclass +7 -0
+        else:
+            return origvalue, op, 0, True
+    varlist = ['[^#=+ ]*']
+    builddir = d.getVar('TOPDIR', True)
+    with open(builddir + '/conf/local.conf', 'r') as f:
+        oldlines = f.readlines()
+    (updated, newlines) = bb.utils.edit_metadata(oldlines, varlist,

handle_var) +
with open(baseoutpath + '/conf/local.conf', 'w') as f:
f.write('# WARNING: this configuration has been automatically
generated and in\n') f.write('# most cases should not be edited. If you need more flexibility than\n') f.write('# this configuration provides, it is strongly suggested that you set\n') f.write('# up a proper instance of the full build system and use that instead.\n\n')

meta/classes/populate_sdk_ext.bbclass python copy_buildsystem () { · line 135

Quoted 5 lines
meta/classes/populate_sdk_ext.bbclass +3 -0
+        for line in newlines:
+            f.write(line)
+
         f.write('INHERIT += "%s"\n\n' % 'uninative')
         f.write('CONF_VERSION = "%s"\n\n' % d.getVar('CONF_VERSION'))

OK, this looks good - but do we really need two whitelists now (SDK_META_CONF_WHITELIST and SDK_LOCAL_CONF_WHITELIST)?

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre