Skip to contentSkip to conversation

populate_sdk_ext: two fixes

PATCH Superseded Chen Qi 3 participants 7 messages Apr 29, 2015

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Chen Qi

author
#0 lore raw

cover populate_sdk_ext: two fixes

The following changes since commit 4dd4b96b6d60246338bb30ede9f3ab1b2e757be9:

libxfont: Security Advisory - libxfont - CVE-2015-1804 (2015-04-28 07:56:01 +0100)

are available in the git repository at:

git://git.openembedded.org/openembedded-core-contrib ChenQi/populate_sdk_ext_2_fixes
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=ChenQi/populate_sdk_ext_2_fixes

Chen Qi (2):
populate_sdk_ext: consider custom configuration in local.conf
populate_sdk_ext: install the latest buildtools-tarball

 meta/classes/populate_sdk_ext.bbclass | 32 +++++++++++++++++++++++++++++++-
 1 file changed, 31 insertions(+), 1 deletion(-)
Signature · 1 line
1.9.1
author
#1 lore raw

PATCH 1/2 populate_sdk_ext: consider custom configuration in local.conf

Copy the contents of local.conf under TOPDIR into the final generated local.conf. In this way, custom settings are also made into the final local.conf like IMAGE_INSTALL, DISTRO_FEATURES, VIRTUAL-RUNTIME_xxx, etc.

Before this change, installing extensible SDK would usually report failure when preparing the build system if the user has custom configuration for DISTRO_FEATURES in local.conf. Also, items in IMAGE_INSTALL_append in local.conf also don't get built correctly.

This patch solves the above problem.

A blacklist mechanism is also introduced so that we can blacklist variables that should not be copied into the final local.conf file. Currently, the blacklist contains 'SSTATE_DIR' and 'DL_DIR'.

[YOCTO #7616]

Signed-off-by Chen Qi <[email protected]>
---
 meta/classes/populate_sdk_ext.bbclass | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)
meta/classes/populate_sdk_ext.bbclass +20 -0
index dc2c58e..7925ece 100644
@@ -16,6 +16,7 @@ SDK_RDEPENDS_append_task-populate-sdk-ext = " ${SDK_TARGETS}"
 SDK_RELOCATE_AFTER_INSTALL_task-populate-sdk-ext = "0"
 
 SDK_META_CONF_WHITELIST ?= "MACHINE DISTRO PACKAGE_CLASSES"
+SDK_META_CONF_BLACKLIST ?= "DL_DIR SSTATE_DIR"
 
 SDK_TARGETS ?= "${PN}"
 OE_INIT_ENV_SCRIPT ?= "oe-init-build-env"

@@ -114,6 +115,25 @@ python copy_buildsystem () {
         f.write('# this configuration provides, it is strongly suggested that you set\n')
         f.write('# up a proper instance of the full build system and use that instead.\n\n')
 
+        # Copy configurations from the current local.conf
+        builddir = d.getVar('TOPDIR', True)
+        with open(builddir + '/conf/local.conf', 'r') as lf:
+            varblacklist = d.getVar('SDK_META_CONF_BLACKLIST', True).split()
+            skip = False
+            for line in lf:
+                line = line.lstrip()
+                if line.startswith('#'):
+                    continue
+                for varname in varblacklist:
+                    if line.startswith(varname):
+                        skip = True
+                        break
+                if not skip:
+                    f.write(line)
+                skip = False
+        f.write('\n')
+
+        # Configurations in local.conf which are specific for extensible SDK
         f.write('INHERIT += "%s"\n\n' % 'uninative')
         f.write('CONF_VERSION = "%s"\n\n' % d.getVar('CONF_VERSION'))
Signature · 1 line
1.9.1
author
#5 lore raw

PATCH 2/2 populate_sdk_ext: install the latest buildtools-tarball

In reply to Chen Qi

If we do bitbake buildtools-tarball' and then after one day do bitbake core-image-minimal -c populate_sdk_ext', we would meet errors like below.

| install: cannot stat '/buildarea2/chenqi/poky/build-systemd/tmp/deploy/sdk/ poky-glibc-x86_64-buildtools-tarball-core2-64-buildtools-nativesdk-standalone -1.8+snapshot-20150429.sh': No such file or directory

The problem is that the output name for buildtools-tarball has ${DATE} in it. So if populate_sdk_ext task is executed but buildtools-tarball is not rebuilt, the above error appears.

Instead of hardcoding ${DISTRO_VERSION} which consists of ${DATE} in the install_tools() function, we should find the latest buildtools-tarball based on the modification time and install it.

[YOCTO #7674]

Signed-off-by Chen Qi <[email protected]>
---
 meta/classes/populate_sdk_ext.bbclass | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
meta/classes/populate_sdk_ext.bbclass +11 -1
index 7925ece..0e4ee31 100644
@@ -188,7 +188,17 @@ install_tools() {
 	ln -sr ${SDK_OUTPUT}/${SDKPATH}/${scriptrelpath}/recipetool ${SDK_OUTPUT}/${SDKPATHNATIVE}${bindir_nativesdk}/recipetool
 	touch ${SDK_OUTPUT}/${SDKPATH}/.devtoolbase
 
-	install ${SDK_DEPLOY}/${DISTRO}-${TCLIBC}-${SDK_ARCH}-buildtools-tarball-${TUNE_PKGARCH}-buildtools-nativesdk-standalone-${DISTRO_VERSION}.sh ${SDK_OUTPUT}/${SDKPATH}
+	# find latest buildtools-tarball and install it
+	maxseconds=0
+	buildtools_path=""
+	for f in `ls ${SDK_DEPLOY}/${DISTRO}-${TCLIBC}-${SDK_ARCH}-buildtools-tarball-${TUNE_PKGARCH}-buildtools-nativesdk-standalone-*.sh`; do
+		seconds=`stat -c "%Y" $f`
+		if [ $seconds -gt $maxseconds ]; then
+			maxseconds=$seconds
+			buildtools_path="$f"
+		fi
+	done
+	install $buildtools_path ${SDK_OUTPUT}/${SDKPATH}
 
 	install ${SDK_DEPLOY}/${BUILD_ARCH}-nativesdk-libc.tar.bz2 ${SDK_OUTPUT}/${SDKPATH}
 }
Signature · 1 line
1.9.1
#2 lore raw

Re: [PATCH 1/2] populate_sdk_ext: consider custom configuration in local.conf

In reply to Chen Qi

Hi Qi,

Quoted 29 lines · On Wednesday 29 April 2015 16:08:36 Chen Qi wrote:

Copy the contents of local.conf under TOPDIR into the final generated local.conf. In this way, custom settings are also made into the final local.conf like IMAGE_INSTALL, DISTRO_FEATURES, VIRTUAL-RUNTIME_xxx, etc.

Before this change, installing extensible SDK would usually report failure when preparing the build system if the user has custom configuration for DISTRO_FEATURES in local.conf. Also, items in IMAGE_INSTALL_append in local.conf also don't get built correctly.

This patch solves the above problem.

A blacklist mechanism is also introduced so that we can blacklist variables that should not be copied into the final local.conf file. Currently, the blacklist contains 'SSTATE_DIR' and 'DL_DIR'.

[YOCTO #7616]

Signed-off-by Chen Qi <[email protected]>
---
 meta/classes/populate_sdk_ext.bbclass | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)
(quoted diff) +0 -0

b/meta/classes/populate_sdk_ext.bbclass index dc2c58e..7925ece 100644

meta/classes/populate_sdk_ext.bbclass +0 -0
@@ -16,6 +16,7 @@ SDK_RDEPENDS_append_task-populate-sdk-ext = "

${SDK_TARGETS}" SDK_RELOCATE_AFTER_INSTALL_task-populate-sdk-ext = "0"

Quoted 2 lines

SDK_META_CONF_WHITELIST ?= "MACHINE DISTRO PACKAGE_CLASSES"
+SDK_META_CONF_BLACKLIST ?= "DL_DIR SSTATE_DIR"

I'm not totally opposed to this approach but we are going to have to get the default list right if we are going to do it - e.g. you've missed the fairly obvious TMPDIR. There are quite a large number of other variables the user might set that should be blacklisted - potentially any variable that points to a location where the build system would write files (DEPLOY_DIR, DEPLOY_DIR_*, WORKDIR, T, S, B, ...). We might go a step further and use the data dictionary to check if the value evaluates to somewhere outside of ${TMPDIR} (if we first make a copy of d and set TMPDIR to a dummy value); this way we would allow adjustments within TMPDIR but not outside of it.

My other concern (perhaps a pet issue of mine) is that people are far too reliant on putting things in local.conf that they really shouldn't. On the other hand building a potentially broken SDK or at least one that doesn't match up with the image is worse, but I'm still concerned about the way local.conf is often abused.

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
#6 lore raw

Re: [PATCH 2/2] populate_sdk_ext: install the latest buildtools-tarball

In reply to Chen Qi
Quoted 40 lines · On Wednesday 29 April 2015 16:08:37 Chen Qi wrote:

If we do bitbake buildtools-tarball' and then after one day do bitbake core-image-minimal -c populate_sdk_ext', we would meet errors like below.

| install: cannot stat
| '/buildarea2/chenqi/poky/build-systemd/tmp/deploy/sdk/

poky-glibc-x86_64-buildtools-tarball-core2-64-buildtools-nativesdk-standalon e -1.8+snapshot-20150429.sh': No such file or directory

The problem is that the output name for buildtools-tarball has ${DATE} in it. So if populate_sdk_ext task is executed but buildtools-tarball is not rebuilt, the above error appears.

Instead of hardcoding ${DISTRO_VERSION} which consists of ${DATE} in the install_tools() function, we should find the latest buildtools-tarball based on the modification time and install it.

[YOCTO #7674]

Signed-off-by Chen Qi <[email protected]>
---
 meta/classes/populate_sdk_ext.bbclass | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
(quoted diff) +0 -0

b/meta/classes/populate_sdk_ext.bbclass index 7925ece..0e4ee31 100644

meta/classes/populate_sdk_ext.bbclass +0 -0
@@ -188,7 +188,17 @@ install_tools() {
 	ln -sr ${SDK_OUTPUT}/${SDKPATH}/${scriptrelpath}/recipetool

${SDK_OUTPUT}/${SDKPATHNATIVE}${bindir_nativesdk}/recipetool touch ${SDK_OUTPUT}/${SDKPATH}/.devtoolbase

meta/classes/populate_sdk_ext.bbclass +0 -1
-	install

${SDK_DEPLOY}/${DISTRO}-${TCLIBC}-${SDK_ARCH}-buildtools-tarball-${TUNE_PKG ARCH}-buildtools-nativesdk-standalone-${DISTRO_VERSION}.sh
${SDK_OUTPUT}/${SDKPATH} + # find latest buildtools-tarball and install it

meta/classes/populate_sdk_ext.bbclass +3 -0
+	maxseconds=0
+	buildtools_path=""
+	for f in `ls
Quoted 2 lines

${SDK_DEPLOY}/${DISTRO}-${TCLIBC}-${SDK_ARCH}-buildtools-tarball-${TUNE_PKG
ARCH}-buildtools-nativesdk-standalone-*.sh; do + seconds=stat -c

"%Y" $f`

meta/classes/populate_sdk_ext.bbclass install_tools() { · line 201

Quoted 6 lines
meta/classes/populate_sdk_ext.bbclass +6 -0
+		if [ $seconds -gt $maxseconds ]; then
+			maxseconds=$seconds
+			buildtools_path="$f"
+		fi
+	done
+	install $buildtools_path ${SDK_OUTPUT}/${SDKPATH}

Surely it would be simpler to just ls -t ... | head -n1 ?

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
#3 lore raw

Re: [PATCH 1/2] populate_sdk_ext: consider custom configuration in local.conf

In reply to Paul Eggleton
Quoted 45 lines · Paul Eggleton

Hi Qi,

Quoted 29 lines · On Wednesday 29 April 2015 16:08:36 Chen Qi wrote:

Copy the contents of local.conf under TOPDIR into the final generated local.conf. In this way, custom settings are also made into the final local.conf like IMAGE_INSTALL, DISTRO_FEATURES, VIRTUAL-RUNTIME_xxx, etc.

Before this change, installing extensible SDK would usually report failure when preparing the build system if the user has custom configuration for DISTRO_FEATURES in local.conf. Also, items in IMAGE_INSTALL_append in local.conf also don't get built correctly.

This patch solves the above problem.

A blacklist mechanism is also introduced so that we can blacklist variables that should not be copied into the final local.conf file. Currently, the blacklist contains 'SSTATE_DIR' and 'DL_DIR'.

[YOCTO #7616]

Signed-off-by Chen Qi <[email protected]>
---
 meta/classes/populate_sdk_ext.bbclass | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)
(quoted diff) +0 -0

b/meta/classes/populate_sdk_ext.bbclass index dc2c58e..7925ece 100644

meta/classes/populate_sdk_ext.bbclass +0 -0
@@ -16,6 +16,7 @@ SDK_RDEPENDS_append_task-populate-sdk-ext = "

${SDK_TARGETS}" SDK_RELOCATE_AFTER_INSTALL_task-populate-sdk-ext = "0"

Quoted 2 lines

SDK_META_CONF_WHITELIST ?= "MACHINE DISTRO PACKAGE_CLASSES"
+SDK_META_CONF_BLACKLIST ?= "DL_DIR SSTATE_DIR"

I'm not totally opposed to this approach but we are going to have to get the default list right if we are going to do it - e.g. you've missed the fairly obvious TMPDIR. There are quite a large number of other variables the user might set that should be blacklisted - potentially any variable that points to a location where the build system would write files (DEPLOY_DIR, DEPLOY_DIR_*, WORKDIR, T, S, B, ...). We might go a step further and use the data dictionary to check if the value evaluates to somewhere outside of ${TMPDIR} (if we first make a copy of d and set TMPDIR to a dummy value); this way we would allow adjustments within TMPDIR but not outside of it.

Quoted 5 lines

My other concern (perhaps a pet issue of mine) is that people are far too reliant on putting things in local.conf that they really shouldn't. On the other hand building a potentially broken SDK or at least one that doesn't match up with the image is worse, but I'm still concerned about the way local.conf is often abused.

One thing we could do programatically is to build the bash config hash with and without local.conf. If they match, local.conf isn't influencing the build checksums. If they don't match, local.conf has something problematic in it and we could error and tell the user to fix it?

This would also cover site.conf, auto.conf and the other conf files we can detect.

Cheers,

Richard

#4 lore raw

Re: [PATCH 1/2] populate_sdk_ext: consider custom configuration in local.conf

Quoted 56 lines · On Wednesday 29 April 2015 09:59:37 Richard Purdie wrote:
Quoted 49 lines · Paul Eggleton
Quoted 33 lines · On Wednesday 29 April 2015 16:08:36 Chen Qi wrote:

Copy the contents of local.conf under TOPDIR into the final generated local.conf. In this way, custom settings are also made into the final local.conf like IMAGE_INSTALL, DISTRO_FEATURES, VIRTUAL-RUNTIME_xxx, etc.

Before this change, installing extensible SDK would usually report failure
when preparing the build system if the user has custom configuration for DISTRO_FEATURES in local.conf. Also, items in IMAGE_INSTALL_append in local.conf also don't get built correctly.

This patch solves the above problem.

A blacklist mechanism is also introduced so that we can blacklist variables
that should not be copied into the final local.conf file. Currently, the blacklist contains 'SSTATE_DIR' and 'DL_DIR'.

[YOCTO #7616]

Signed-off-by Chen Qi <[email protected]>

---

 meta/classes/populate_sdk_ext.bbclass | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)
(quoted diff) +0 -0

b/meta/classes/populate_sdk_ext.bbclass index dc2c58e..7925ece 100644

meta/classes/populate_sdk_ext.bbclass +0 -0
@@ -16,6 +16,7 @@ SDK_RDEPENDS_append_task-populate-sdk-ext = "

${SDK_TARGETS}" SDK_RELOCATE_AFTER_INSTALL_task-populate-sdk-ext = "0"

Quoted 3 lines

SDK_META_CONF_WHITELIST ?= "MACHINE DISTRO PACKAGE_CLASSES"

+SDK_META_CONF_BLACKLIST ?= "DL_DIR SSTATE_DIR"

I'm not totally opposed to this approach but we are going to have to get the default list right if we are going to do it - e.g. you've missed the fairly obvious TMPDIR. There are quite a large number of other variables the user might set that should be blacklisted - potentially any variable that points to a location where the build system would write files (DEPLOY_DIR, DEPLOY_DIR_*, WORKDIR, T, S, B, ...). We might go a step further and use the data dictionary to check if the value evaluates to somewhere outside of ${TMPDIR} (if we first make a copy of d and set TMPDIR to a dummy value); this way we would allow adjustments within TMPDIR but not outside of it.

Quoted 5 lines

My other concern (perhaps a pet issue of mine) is that people are far too reliant on putting things in local.conf that they really shouldn't. On the other hand building a potentially broken SDK or at least one that doesn't match up with the image is worse, but I'm still concerned about the way local.conf is often abused.

Quoted 7 lines

One thing we could do programatically is to build the bash config hash with and without local.conf. If they match, local.conf isn't influencing the build checksums. If they don't match, local.conf has something problematic in it and we could error and tell the user to fix it?

This would also cover site.conf, auto.conf and the other conf files we can detect.

Hmm, yes... this might be another check we could turn on when switching from development -> production (what I mentioned at OEDAM).

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre