Extensible SDK: 3 fixes

If we do bitbake buildtools-tarball' and then after one day do bitbake core-image-minimal -c populate_sdk_ext', we would meet errors like below.

| install: cannot stat '/buildarea2/chenqi/poky/build-systemd/tmp/deploy/sdk/

poky-glibc-x86_64-buildtools-tarball-core2-64-buildtools-nativesdk-standalone -1.8+snapshot-20150429.sh': No such file or directory

The problem is that the output name for buildtools-tarball has ${DATE} in it.
So if populate_sdk_ext task is executed but buildtools-tarball is not rebuilt,
the above error appears.

Instead of hardcoding ${DISTRO_VERSION} which consists of ${DATE} in the install_tools() function, we should find the latest buildtools-tarball based
on the modification time and install it.

[YOCTO #7674]

Signed-off-by Chen Qi <[email protected]>

Quoted 20 lines · Chen Qi

If we do bitbake buildtools-tarball' and then after one day do bitbake core-image-minimal -c populate_sdk_ext', we would meet errors like below.

| install: cannot stat '/buildarea2/chenqi/poky/build-systemd/tmp/deploy/sdk/

poky-glibc-x86_64-buildtools-tarball-core2-64-buildtools-nativesdk-standalone -1.8+snapshot-20150429.sh': No such file or directory

The problem is that the output name for buildtools-tarball has ${DATE} in it.
So if populate_sdk_ext task is executed but buildtools-tarball is not rebuilt,
the above error appears.

Instead of hardcoding ${DISTRO_VERSION} which consists of ${DATE} in the install_tools() function, we should find the latest buildtools-tarball based
on the modification time and install it.

Quoted 4 lines

[YOCTO #7674]

Signed-off-by Chen Qi <[email protected]>

Hmm, if buildtools-tarball is required, why not just pull it in via dependencies and build a current one, rather than assuming/forcing the user to have done so themselves?

Quoted 25 lines · Christopher Larson

On Tue, May 26, 2015 at 11:09 PM, Chen Qi <[email protected]> wrote:

If we do bitbake buildtools-tarball' and then after one day do bitbake

Quoted 22 lines

core-image-minimal -c populate_sdk_ext', we would meet errors like below.

| install: cannot stat '/buildarea2/chenqi/poky/build-systemd/tmp/deploy/sdk/

poky-glibc-x86_64-buildtools-tarball-core2-64-buildtools-nativesdk-standalone -1.8+snapshot-20150429.sh': No such file or directory

The problem is that the output name for buildtools-tarball has ${DATE} in it.
So if populate_sdk_ext task is executed but buildtools-tarball is not rebuilt,
the above error appears.

Instead of hardcoding ${DISTRO_VERSION} which consists of ${DATE} in the install_tools() function, we should find the latest buildtools-tarball based
on the modification time and install it.

[YOCTO #7674]

Quoted 8 lines

Quoted 3 lines

Signed-off-by Chen Qi <[email protected]>

Hmm, if buildtools-tarball is required, why not just pull it in via dependencies and build a current one, rather than assuming/forcing the user
to have done so themselves?

It is a dependency and will be built if it hasn't been built. But since

TOOLCHAIN_OUTPUTNAME ?= "${SDK_NAME}-buildtools-nativesdk-standalone-${DISTRO_VERSION}"

in buildtools-tarball.bb the filename will be different each time we try to copy it, unless we force buildtools-tarball to be rebuilt each time the sdk is constructed.

${DATE} being used in DISTRO_VERSION is misleading in my opinion because different dates don't necessarily mean different contents. So we could change DISTRO_VERSION to not use date, or buildtools-tarball.bb to not use DISTRO_VERSION.

Copy the contents of local.conf under TOPDIR into the final generated local.conf. In this way, custom settings are also made into the final local.conf like IMAGE_INSTALL, DISTRO_FEATURES, VIRTUAL-RUNTIME_xxx, etc.

Before this change, installing extensible SDK would usually report failure when preparing the build system if the user has custom configuration for DISTRO_FEATURES in local.conf. Also, items in IMAGE_INSTALL_append in local.conf also don't get built correctly.

This patch solves the above problem.

A blacklist mechanism is also introduced so that we can blacklist variables that should not be copied into the final local.conf file. Currently, the blacklist contains 'TMPDIR', 'SSTATE_DIR', 'DL_DIR', 'STAMPS_DIR', 'BASE_WORKDIR' and 'DEPLOY_DIR'. What these variables have in common is that they are set in bitbake.conf using '?=' or '??='.

In addition, we check to avoid any setting that might lead to host path bleeding into SDK's configuration.

[YOCTO #7616]

Signed-off-by Chen Qi <[email protected]>

---
 meta/classes/populate_sdk_ext.bbclass | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

(quoted diff) +0 -0

b/meta/classes/populate_sdk_ext.bbclass index 2fc4c11..49ba26b 100644

meta/classes/populate_sdk_ext.bbclass +0 -0

@@ -16,6 +16,7 @@ SDK_RDEPENDS_append_task-populate-sdk-ext = "

${SDK_TARGETS}" SDK_RELOCATE_AFTER_INSTALL_task-populate-sdk-ext = "0"

SDK_META_CONF_WHITELIST ?= "MACHINE DISTRO PACKAGE_CLASSES"
+SDK_META_CONF_BLACKLIST ?= "TMPDIR DL_DIR SSTATE_DIR STAMPS_DIR BASE_WORKDIR DEPLOY_DIR"

SDK_TARGETS ?= "${PN}"
OE_INIT_ENV_SCRIPT ?= "oe-init-build-env"

meta/classes/populate_sdk_ext.bbclass +0 -0

@@ -114,6 +115,28 @@ python copy_buildsystem () {
         f.write('# this configuration provides, it is strongly suggested

that you set\n') f.write('# up a proper instance of the full build system and use that instead.\n\n')

meta/classes/populate_sdk_ext.bbclass +4 -0

+        # Copy configurations from the current local.conf
+        builddir = d.getVar('TOPDIR', True)
+        with open(builddir + '/conf/local.conf', 'r') as lf:
+            varblacklist = d.getVar('SDK_META_CONF_BLACKLIST',

True).split() + skip = False

meta/classes/populate_sdk_ext.bbclass +17 -0

+            for line in lf:
+                line = line.lstrip()
+                if line.startswith('#'):
+                    continue
+                # avoid host path bleeding into SDK configuration
+                if line.find('"/') != -1:
+                    continue
+                for varname in varblacklist:
+                    if line.startswith(varname):
+                        skip = True
+                        break
+                if not skip:
+                    f.write(line)
+                skip = False
+        f.write('\n')
+
+        # Configurations in local.conf which are specific for extensible

SDK f.write('INHERIT += "%s"\n\n' % 'uninative')

         f.write('CONF_VERSION = "%s"\n\n' % d.getVar('CONF_VERSION'))

Hi Qi,

Quoted 68 lines · On Wednesday 27 May 2015 14:09:39 Chen Qi wrote:

Copy the contents of local.conf under TOPDIR into the final generated local.conf. In this way, custom settings are also made into the final local.conf like IMAGE_INSTALL, DISTRO_FEATURES, VIRTUAL-RUNTIME_xxx, etc.

Before this change, installing extensible SDK would usually report failure when preparing the build system if the user has custom configuration for DISTRO_FEATURES in local.conf. Also, items in IMAGE_INSTALL_append in local.conf also don't get built correctly.

This patch solves the above problem.

A blacklist mechanism is also introduced so that we can blacklist variables that should not be copied into the final local.conf file. Currently, the blacklist contains 'TMPDIR', 'SSTATE_DIR', 'DL_DIR', 'STAMPS_DIR', 'BASE_WORKDIR' and 'DEPLOY_DIR'. What these variables have in common is that they are set in bitbake.conf using '?=' or '??='.

In addition, we check to avoid any setting that might lead to host path bleeding into SDK's configuration.

[YOCTO #7616]

Signed-off-by Chen Qi <[email protected]>

---
  meta/classes/populate_sdk_ext.bbclass | 23 +++++++++++++++++++++++
  1 file changed, 23 insertions(+)

(quoted diff) +0 -0

b/meta/classes/populate_sdk_ext.bbclass index 2fc4c11..49ba26b 100644

meta/classes/populate_sdk_ext.bbclass +0 -0

@@ -16,6 +16,7 @@ SDK_RDEPENDS_append_task-populate-sdk-ext = "

${SDK_TARGETS}" SDK_RELOCATE_AFTER_INSTALL_task-populate-sdk-ext = "0"

SDK_META_CONF_WHITELIST ?= "MACHINE DISTRO PACKAGE_CLASSES"
+SDK_META_CONF_BLACKLIST ?= "TMPDIR DL_DIR SSTATE_DIR STAMPS_DIR BASE_WORKDIR DEPLOY_DIR"

  SDK_TARGETS ?= "${PN}"
  OE_INIT_ENV_SCRIPT ?= "oe-init-build-env"

meta/classes/populate_sdk_ext.bbclass +0 -0

@@ -114,6 +115,28 @@ python copy_buildsystem () {
          f.write('# this configuration provides, it is strongly suggested

that you set\n') f.write('# up a proper instance of the full build system and use that instead.\n\n')

meta/classes/populate_sdk_ext.bbclass +4 -0

+        # Copy configurations from the current local.conf
+        builddir = d.getVar('TOPDIR', True)
+        with open(builddir + '/conf/local.conf', 'r') as lf:
+            varblacklist = d.getVar('SDK_META_CONF_BLACKLIST',

True).split() + skip = False

meta/classes/populate_sdk_ext.bbclass +17 -0

+            for line in lf:
+                line = line.lstrip()
+                if line.startswith('#'):
+                    continue
+                # avoid host path bleeding into SDK configuration
+                if line.find('"/') != -1:
+                    continue
+                for varname in varblacklist:
+                    if line.startswith(varname):
+                        skip = True
+                        break
+                if not skip:
+                    f.write(line)
+                skip = False
+        f.write('\n')
+
+        # Configurations in local.conf which are specific for extensible

SDK f.write('INHERIT += "%s"\n\n' % 'uninative')

Quoted 1 line

          f.write('CONF_VERSION = "%s"\n\n' % d.getVar('CONF_VERSION'))

I've been thinking about this; since I've recently added edit_metadata() in bitbake's lib/bb/utils.py, it would make sense to use that rather than adding another piece of code that effectively parses bitbake configuration files (though I know this isn't adding much). I know this series has been going back and forth for a while now and the delays have largely been my fault, so I'd be happy to make this change and resend the series - it's up to you.

Cheers,
Paul

Quoted 89 lines · Paul Eggleton

Quoted 77 lines · On Wednesday 27 May 2015 14:09:39 Chen Qi wrote:

Copy the contents of local.conf under TOPDIR into the final generated local.conf. In this way, custom settings are also made into the final local.conf like IMAGE_INSTALL, DISTRO_FEATURES, VIRTUAL-RUNTIME_xxx, etc.

Before this change, installing extensible SDK would usually report failure
when preparing the build system if the user has custom configuration for DISTRO_FEATURES in local.conf. Also, items in IMAGE_INSTALL_append in local.conf also don't get built correctly.

This patch solves the above problem.

A blacklist mechanism is also introduced so that we can blacklist variables
that should not be copied into the final local.conf file. Currently, the blacklist contains 'TMPDIR', 'SSTATE_DIR', 'DL_DIR', 'STAMPS_DIR', 'BASE_WORKDIR' and 'DEPLOY_DIR'. What these variables have in common is that they are set in bitbake.conf using '?=' or '??='.

In addition, we check to avoid any setting that might lead to host path bleeding into SDK's configuration.

[YOCTO #7616]

Signed-off-by Chen Qi <[email protected]>

---

  meta/classes/populate_sdk_ext.bbclass | 23 +++++++++++++++++++++++
  1 file changed, 23 insertions(+)

(quoted diff) +0 -0

b/meta/classes/populate_sdk_ext.bbclass index 2fc4c11..49ba26b 100644

meta/classes/populate_sdk_ext.bbclass +0 -0

@@ -16,6 +16,7 @@ SDK_RDEPENDS_append_task-populate-sdk-ext = "

${SDK_TARGETS}" SDK_RELOCATE_AFTER_INSTALL_task-populate-sdk-ext = "0"

SDK_META_CONF_WHITELIST ?= "MACHINE DISTRO PACKAGE_CLASSES"

+SDK_META_CONF_BLACKLIST ?= "TMPDIR DL_DIR SSTATE_DIR STAMPS_DIR BASE_WORKDIR DEPLOY_DIR"

  SDK_TARGETS ?= "${PN}"
  OE_INIT_ENV_SCRIPT ?= "oe-init-build-env"

meta/classes/populate_sdk_ext.bbclass +0 -0

@@ -114,6 +115,28 @@ python copy_buildsystem () {
 
          f.write('# this configuration provides, it is strongly
          suggested
 

that you set\n') f.write('# up a proper instance of the full build system and use that instead.\n\n')

meta/classes/populate_sdk_ext.bbclass +4 -0

+        # Copy configurations from the current local.conf
+        builddir = d.getVar('TOPDIR', True)
+        with open(builddir + '/conf/local.conf', 'r') as lf:
+            varblacklist = d.getVar('SDK_META_CONF_BLACKLIST',

True).split() + skip = False

meta/classes/populate_sdk_ext.bbclass +17 -0

+            for line in lf:
+                line = line.lstrip()
+                if line.startswith('#'):
+                    continue
+                # avoid host path bleeding into SDK configuration
+                if line.find('"/') != -1:
+                    continue
+                for varname in varblacklist:
+                    if line.startswith(varname):
+                        skip = True
+                        break
+                if not skip:
+                    f.write(line)
+                skip = False
+        f.write('\n')
+
+        # Configurations in local.conf which are specific for extensible

SDK f.write('INHERIT += "%s"\n\n' % 'uninative')

Quoted 1 line

          f.write('CONF_VERSION = "%s"\n\n' % d.getVar('CONF_VERSION'))

I've been thinking about this; since I've recently added edit_metadata() in
bitbake's lib/bb/utils.py, it would make sense to use that rather than adding another piece of code that effectively parses bitbake configuration files (though I know this isn't adding much). I know this series has been going back and forth for a while now and the delays have largely been my fault, so I'd be happy to make this change and resend the series - it's up to you.

Quoted 2 lines

Cheers,
Paul

Hi Paul,

Thanks for your suggestion.
I've been looking into edit_metadata(). But it seems that it's used to modify *specific* variables.
In this case, we don't know which variables we are going to skip in local.conf. We just know that they start with '/'. So I don't see how to do it.