Skip to contentSkip to conversation

Extensible SDK fixes

PATCH New Paul Eggleton 1 participant 12 messages Sep 7, 2015

lore
Share & tools
lore
#0 lore raw

cover Extensible SDK fixes

This rolls up a set of previously posted patches from Qi and Brendan with some changes from me.

Changes since the previously posted versions:

* populate_sdk_ext: install the latest buildtools-tarball:
- Avoid poky-specific path usage
* populate_sdk_ext: consider custom configuration in local.conf:
- Allow blacklisting certain variables
- Prevent comments and blank lines from coming through
* toolchain-shar-extract.sh: ensure extensible SDK install path obeys restrictions
- Use grep instead of bash-specific =~ operator
* toolchain-shar-extract.sh: better default install path for extensible SDK:
- Fix default path overriding path specified on the command line with the -d option when installing the extensible SDK
* oe-publish-sdk: add script:
- Ensure errors in layer git repo setup exit immediately instead of continuing
- Change some debug logging to info for more coherent output
- Use -A option with "git add"
- Fix command usage info in comments
* devtool: add mechanism for updating extensible SDK:
- Allow default update server to be read from config file
- Squash in change to write SDK_TARGETS to config file

The following changes since commit 8402958cd2cb87b8283c8ee4e2d08e1a6717d67a:

pseudo_1.7.3.bb: New version of pseudo (2015-09-06 15:24:28 +0100)

are available in the git repository at:

git://git.openembedded.org/openembedded-core-contrib paule/extsdk4
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/extsdk4

Brendan Le Foll (3):
toolchain-shar-extract.sh: ensure extensible SDK install path obeys
restrictions
toolchain-shar-extract.sh: better default install path for extensible
SDK
toolchain-shar-extract.sh: explain why we cannot use sudo in
extensible SDK

Paul Eggleton (1):
classes/populate_sdk_ext: avoid poky-specific buildtools naming

[email protected] (7):
populate_sdk_ext: install the latest buildtools-tarball
copy_buildsystem: make sure bitbake directory is copied
populate_sdk_ext: consider custom configuration in local.conf
populate_sdk_ext: don't remove the native qemu dependencies Extensible SDK: allow for installation without preparing build system
oe-publish-sdk: add script
devtool: add mechanism for updating extensible SDK

 meta/classes/populate_sdk_base.bbclass |   2 +
 meta/classes/populate_sdk_ext.bbclass  |  59 +++++++---
 meta/files/toolchain-shar-extract.sh   |  41 +++++--
 meta/lib/oe/copy_buildsystem.py        |   7 +-
 scripts/lib/devtool/sdk.py             | 197 +++++++++++++++++++++++++++++++++
 scripts/oe-publish-sdk                 | 143 ++++++++++++++++++++++++
 6 files changed, 422 insertions(+), 27 deletions(-)
 create mode 100644 scripts/lib/devtool/sdk.py
 create mode 100755 scripts/oe-publish-sdk
Signature · 1 line
2.1.0
#1 lore raw

PATCH 1/11 toolchain-shar-extract.sh: ensure extensible SDK install path obeys restrictions

From: Brendan Le Foll <[email protected]>

There are some characters that cannot appear in the installation path, so we need to check for these

---
 meta/files/toolchain-shar-extract.sh | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)
meta/files/toolchain-shar-extract.sh +12 -3
index 3a50991..85719fa 100644
@@ -85,9 +85,18 @@ else
 	target_sdk_dir=$(readlink -m "$target_sdk_dir")
 fi
 
-if [ -n "$(echo $target_sdk_dir|grep ' ')" ]; then
-	echo "The target directory path ($target_sdk_dir) contains spaces. Abort!"
-	exit 1
+if [ "$SDK_EXTENSIBLE" = "1" ]; then
+	# We're going to be running the build system, additional restrictions apply
+	if echo "$target_sdk_dir" | grep -q '[+\ @]'; then
+		echo "The target directory path ($target_sdk_dir) contains illegal" \
+		     "characters such as spaces, @ or +. Abort!"
+		exit 1
+	fi
+else
+	if [ -n "$(echo $target_sdk_dir|grep ' ')" ]; then
+		echo "The target directory path ($target_sdk_dir) contains spaces. Abort!"
+		exit 1
+	fi
 fi
 
 if [ -e "$target_sdk_dir/environment-setup-@REAL_MULTIMACH_TARGET_SYS@" ]; then
Signature · 1 line
2.1.0
#2 lore raw

PATCH 2/11 toolchain-shar-extract.sh: better default install path for extensible SDK

In reply to Paul Eggleton

From: Brendan Le Foll <[email protected]>

Extensible SDK cannot be installed as root so by default offer to install it in user's home directory under distro/distro_version replacing the normal SDK version '+' char with a '_' as that's a restricted character for bitbake

---
 meta/classes/populate_sdk_base.bbclass |  2 ++
 meta/files/toolchain-shar-extract.sh   | 16 ++++++++++++----
 2 files changed, 14 insertions(+), 4 deletions(-)
meta/classes/populate_sdk_base.bbclass +2 -0
index a9e9bd7..b015bf0 100644
@@ -55,6 +55,7 @@ SDK_PRE_INSTALL_COMMAND ?= ""
 SDK_POST_INSTALL_COMMAND ?= ""
 SDK_RELOCATE_AFTER_INSTALL ?= "1"
 
+SDKEXTPATH ?= "~/${@d.getVar('DISTRO', True)}_sdk"
 SDK_TITLE ?= "${@d.getVar('DISTRO_NAME', True) or d.getVar('DISTRO', True)} SDK"
 
 SDK_TARGET_MANIFEST = "${SDK_DEPLOY}/${TOOLCHAIN_OUTPUTNAME}.target.manifest"
@@ -154,6 +155,7 @@ EOF
 	# substitute variables
 	sed -i -e 's#@SDK_ARCH@#${SDK_ARCH}#g' \
 		-e 's#@SDKPATH@#${SDKPATH}#g' \
+		-e 's#@SDKEXTPATH@#${SDKEXTPATH}#g' \
 		-e 's#@OLDEST_KERNEL@#${OLDEST_KERNEL}#g' \
 		-e 's#@REAL_MULTIMACH_TARGET_SYS@#${REAL_MULTIMACH_TARGET_SYS}#g' \
 		-e 's#@SDK_TITLE@#${SDK_TITLE}#g' \
meta/files/toolchain-shar-extract.sh +12 -4
index 85719fa..27a1607 100644
@@ -36,7 +36,6 @@ while getopts ":yd:DRS" OPT; do
 	case $OPT in
 	y)
 		answer="Y"
-		[ "$target_sdk_dir" = "" ] && target_sdk_dir=$DEFAULT_INSTALL_DIR
 		;;
 	d)
 		target_sdk_dir=$OPTARG
@@ -73,9 +72,18 @@ fi
 
 @SDK_PRE_INSTALL_COMMAND@
 
+# SDK_EXTENSIBLE is exposed from the SDK_PRE_INSTALL_COMMAND above
+if [ "$SDK_EXTENSIBLE" = "1" ]; then
+	DEFAULT_INSTALL_DIR="@SDKEXTPATH@"
+fi
+
 if [ "$target_sdk_dir" = "" ]; then
-	read -e -p "Enter target directory for SDK (default: $DEFAULT_INSTALL_DIR): " target_sdk_dir
-	[ "$target_sdk_dir" = "" ] && target_sdk_dir=$DEFAULT_INSTALL_DIR
+	if [ "$answer" = "Y" ]; then
+		target_sdk_dir="$DEFAULT_INSTALL_DIR"
+	else
+		read -e -p "Enter target directory for SDK (default: $DEFAULT_INSTALL_DIR): " target_sdk_dir
+		[ "$target_sdk_dir" = "" ] && target_sdk_dir=$DEFAULT_INSTALL_DIR
+	fi
 fi
 
 eval target_sdk_dir=$(echo "$target_sdk_dir"|sed 's/ /\\ /g')
@@ -155,7 +163,7 @@ echo "done"
 printf "Setting it up..."
 # fix environment paths
 for env_setup_script in `ls $target_sdk_dir/environment-setup-*`; do
-	$SUDO_EXEC sed -e "s:$DEFAULT_INSTALL_DIR:$target_sdk_dir:g" -i $env_setup_script
+	$SUDO_EXEC sed -e "s:@SDKPATH@:$target_sdk_dir:g" -i $env_setup_script
 done
 
 @SDK_POST_INSTALL_COMMAND@
Signature · 1 line
2.1.0
#3 lore raw

PATCH 3/11 toolchain-shar-extract.sh: explain why we cannot use sudo in extensible SDK

In reply to Paul Eggleton

From: Brendan Le Foll <[email protected]>

---
 meta/files/toolchain-shar-extract.sh | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
meta/files/toolchain-shar-extract.sh +2 -1
index 27a1607..3624940 100644
@@ -136,7 +136,8 @@ mkdir -p $target_sdk_dir >/dev/null 2>&1
 # if don't have the right to access dir, gain by sudo 
 if [ ! -x $target_sdk_dir -o ! -w $target_sdk_dir -o ! -r $target_sdk_dir ]; then 
 	if [ "$SDK_EXTENSIBLE" = "1" ]; then
-		echo "Unable to access \"$target_sdk_dir\"."
+		echo "Unable to access \"$target_sdk_dir\", will not attempt to use" \
+		     "sudo as as extensible SDK cannot be used as root."
 		exit 1
 	fi
 
Signature · 1 line
2.1.0
#4 lore raw

PATCH 4/11 classes/populate_sdk_ext: avoid poky-specific buildtools naming

In reply to Paul Eggleton

Only poky sets SDK_NAME to include ${IMAGE_BASENAME} (i.e. ${PN}), so we can't assume the buildtools filename will include it here. Change it to look for a file with "buildtools-nativesdk-standalone" in the name (the buildtools-tarball recipe itself sets TOOLCHAIN_OUTPUTNAME to include this.)

---
 meta/classes/populate_sdk_ext.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes/populate_sdk_ext.bbclass +1 -1
index a36bf16..8509f0c 100644
@@ -189,7 +189,7 @@ SDK_PRE_INSTALL_COMMAND_task-populate-sdk-ext = "${sdk_ext_preinst}"
 sdk_ext_postinst() {
 	printf "\nExtracting buildtools...\n"
 	cd $target_sdk_dir
-	printf "buildtools\ny" | ./*buildtools-tarball* > /dev/null
+	printf "buildtools\ny" | ./*buildtools-nativesdk-standalone* > /dev/null
 
 	# Make sure when the user sets up the environment, they also get
 	# the buildtools-tarball tools in their path.
Signature · 1 line
2.1.0
#5 lore raw

PATCH 5/11 populate_sdk_ext: install the latest buildtools-tarball

In reply to Paul Eggleton

From: "[email protected]" <[email protected]>

If we do bitbake buildtools-tarball' and then after one day do bitbake core-image-minimal -c populate_sdk_ext', we would meet errors like below.

| install: cannot stat '/buildarea2/chenqi/poky/build-systemd/tmp/deploy/sdk/ poky-glibc-x86_64-buildtools-tarball-core2-64-buildtools-nativesdk-standalone -1.8+snapshot-20150429.sh': No such file or directory

The problem is that the output name for buildtools-tarball has ${DATE} in it. So if populate_sdk_ext task is executed but buildtools-tarball is not rebuilt, the above error appears.

Instead of hardcoding ${DISTRO_VERSION} which consists of ${DATE} in the install_tools() function, we should find the latest buildtools-tarball based on the modification time and install it.

[YOCTO #7674]

---
 meta/classes/populate_sdk_ext.bbclass | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
meta/classes/populate_sdk_ext.bbclass +9 -1
index 8509f0c..5dd6051 100644
@@ -163,13 +163,21 @@ python copy_buildsystem () {
         pass
 }
 
+def extsdk_get_buildtools_filename(d):
+    # This is somewhat of a hack
+    localdata = bb.data.createCopy(d)
+    localdata.setVar('PN', 'buildtools-tarball')
+    return localdata.expand('${SDK_NAME}-buildtools-nativesdk-standalone-*.sh')
+
 install_tools() {
 	install -d ${SDK_OUTPUT}/${SDKPATHNATIVE}${bindir_nativesdk}
 	lnr ${SDK_OUTPUT}/${SDKPATH}/${scriptrelpath}/devtool ${SDK_OUTPUT}/${SDKPATHNATIVE}${bindir_nativesdk}/devtool
 	lnr ${SDK_OUTPUT}/${SDKPATH}/${scriptrelpath}/recipetool ${SDK_OUTPUT}/${SDKPATHNATIVE}${bindir_nativesdk}/recipetool
 	touch ${SDK_OUTPUT}/${SDKPATH}/.devtoolbase
 
-	install ${SDK_DEPLOY}/${DISTRO}-${TCLIBC}-${SDK_ARCH}-buildtools-tarball-${TUNE_PKGARCH}-buildtools-nativesdk-standalone-${DISTRO_VERSION}.sh ${SDK_OUTPUT}/${SDKPATH}
+	# find latest buildtools-tarball and install it
+	buildtools_path=`ls -t1 ${SDK_DEPLOY}/${@extsdk_get_buildtools_filename(d)} | head -n1`
+	install $buildtools_path ${SDK_OUTPUT}/${SDKPATH}
 
 	install ${SDK_DEPLOY}/${BUILD_ARCH}-nativesdk-libc.tar.bz2 ${SDK_OUTPUT}/${SDKPATH}
 }
Signature · 1 line
2.1.0
#6 lore raw

PATCH 6/11 copy_buildsystem: make sure bitbake directory is copied

In reply to Paul Eggleton

From: "[email protected]" <[email protected]>

The previous code assumes that bitbake/ directory is under the core layer. This is the case for Yocto project. But users might clone oe-core and bitbake separately. So we use bb.__file__ to locate the bitbake directory to make sure it's copied into the extensible SDK.

---
 meta/lib/oe/copy_buildsystem.py | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)
meta/lib/oe/copy_buildsystem.py +3 -4
index cf7fada..979578c 100644
@@ -28,11 +28,10 @@ class BuildSystem(object):
         layers.append(corebase)
 
         corebase_files = self.d.getVar('COREBASE_FILES', True).split()
-
-        # bitbake belongs in corebase so make sure it goes there
-        if "bitbake" not in corebase_files:
-            corebase_files.append("bitbake")
         corebase_files = [corebase + '/' +x for x in corebase_files]
+        # Make sure bitbake goes in
+        bitbake_dir = bb.__file__.rsplit('/', 3)[0]
+        corebase_files.append(bitbake_dir)
 
         for layer in layers:
             layerconf = os.path.join(layer, 'conf', 'layer.conf')
Signature · 1 line
2.1.0
#7 lore raw

PATCH 7/11 populate_sdk_ext: consider custom configuration in local.conf

In reply to Paul Eggleton

From: "[email protected]" <[email protected]>

Copy the contents of local.conf under TOPDIR into the final generated local.conf. In this way, custom settings are also made into the final local.conf like IMAGE_INSTALL, DISTRO_FEATURES, VIRTUAL-RUNTIME_xxx, etc. Comments and blank lines are filtered out.

Before this change, installing extensible SDK would usually report failure when preparing the build system if the user has custom configuration for DISTRO_FEATURES in local.conf. Also, items in IMAGE_INSTALL_append in local.conf also don't get built correctly.

This patch solves the above problem by making use of bb.utils.edit_metadata.

In addition, we check to avoid any setting that might lead to host paths bleeding into the SDK's configuration. Basically, variables with values starting with '/' are removed. A whitelist mechanism is introduced so that users could specify variables that should not be ignored. The name of the whitelist is SDK_LOCAL_CONF_WHITELIST.

The SDK_META_CONF_WHITELIST is removed as it's of no use after this change.

SDK_LOCAL_CONF_BLACKLIST can be used to prevent copying specific variable settings to the extensible SDK's local.conf; the default is to exclude PRSERV_HOST (since this is likely to be internal). Similarly, SDK_INHERIT_BLACKLIST to forbit local.conf in SDK to inherit certain classes such as 'buildhistory' or 'icecc' that would not normally make sense in an SDK environment.

[YOCTO #7616]

---
 meta/classes/populate_sdk_ext.bbclass | 26 +++++++++++++++++++++++---
 1 file changed, 23 insertions(+), 3 deletions(-)
meta/classes/populate_sdk_ext.bbclass +23 -3
index 5dd6051..fa36a91 100644
@@ -15,7 +15,9 @@ SDK_RDEPENDS_append_task-populate-sdk-ext = " ${SDK_TARGETS}"
 
 SDK_RELOCATE_AFTER_INSTALL_task-populate-sdk-ext = "0"
 
-SDK_META_CONF_WHITELIST ?= "MACHINE DISTRO PACKAGE_CLASSES"
+SDK_LOCAL_CONF_WHITELIST ?= ""
+SDK_LOCAL_CONF_BLACKLIST ?= "CONF_VERSION BB_NUMBER_THREADS PARALLEL_MAKE PRSERV_HOST"
+SDK_INHERIT_BLACKLIST ?= "buildhistory icecc"
 
 SDK_TARGETS ?= "${PN}"
 OE_INIT_ENV_SCRIPT ?= "oe-init-build-env"
@@ -109,15 +111,35 @@ python copy_buildsystem () {
         f.write('    "\n')
 
     # Create local.conf
+    local_conf_whitelist = (d.getVar('SDK_LOCAL_CONF_WHITELIST', True) or '').split()
+    local_conf_blacklist = (d.getVar('SDK_LOCAL_CONF_BLACKLIST', True) or '').split()
+    def handle_var(varname, origvalue, op, newlines):
+        if varname in local_conf_blacklist or (origvalue.strip().startswith('/') and not varname in local_conf_whitelist):
+            newlines.append('# Removed original setting of %s\n' % varname)
+            return None, op, 0, True
+        else:
+            return origvalue, op, 0, True
+    varlist = ['[^#=+ ]*']
+    builddir = d.getVar('TOPDIR', True)
+    with open(builddir + '/conf/local.conf', 'r') as f:
+        oldlines = f.readlines()
+    (updated, newlines) = bb.utils.edit_metadata(oldlines, varlist, handle_var)
+
     with open(baseoutpath + '/conf/local.conf', 'w') as f:
         f.write('# WARNING: this configuration has been automatically generated and in\n')
         f.write('# most cases should not be edited. If you need more flexibility than\n')
         f.write('# this configuration provides, it is strongly suggested that you set\n')
         f.write('# up a proper instance of the full build system and use that instead.\n\n')
+        for line in newlines:
+            if line.strip() and not line.startswith('#'):
+                f.write(line)
 
         f.write('INHERIT += "%s"\n\n' % 'uninative')
         f.write('CONF_VERSION = "%s"\n\n' % d.getVar('CONF_VERSION', False))
 
+        # Some classes are not suitable for SDK, remove them from INHERIT
+        f.write('INHERIT_remove = "%s"\n' % d.getVar('SDK_INHERIT_BLACKLIST'))
+
         # This is a bit of a hack, but we really don't want these dependencies
         # (we're including them in the SDK as nativesdk- versions instead)
         f.write('POKYQEMUDEPS_forcevariable = ""\n\n')
@@ -134,8 +156,6 @@ python copy_buildsystem () {
         # Ensure locked sstate cache objects are re-used without error
         f.write('SIGGEN_LOCKEDSIGS_CHECK_LEVEL = "warn"\n\n')
 
-        for varname in d.getVar('SDK_META_CONF_WHITELIST', True).split():
-            f.write('%s = "%s"\n' % (varname, d.getVar(varname, True)))
         f.write('require conf/locked-sigs.inc\n')
         f.write('require conf/work-config.inc\n')
 
Signature · 1 line
2.1.0
#8 lore raw

PATCH 8/11 populate_sdk_ext: don't remove the native qemu dependencies

In reply to Paul Eggleton

From: "[email protected]" <[email protected]>

These dependencies were deliberately removed because it was assumed that they were provided by nativesdk packages. On the one hand, nativesdk packages in extensible SDK don't have these packages; on the other hand, even if we add these nativesdk packages, they are still not useful because we we need runqemu to run correctly.

So we don't remove these native qemu dependencies.

---
 meta/classes/populate_sdk_ext.bbclass | 5 -----
 1 file changed, 5 deletions(-)
meta/classes/populate_sdk_ext.bbclass +0 -5
index fa36a91..fc2d96f 100644
@@ -140,11 +140,6 @@ python copy_buildsystem () {
         # Some classes are not suitable for SDK, remove them from INHERIT
         f.write('INHERIT_remove = "%s"\n' % d.getVar('SDK_INHERIT_BLACKLIST'))
 
-        # This is a bit of a hack, but we really don't want these dependencies
-        # (we're including them in the SDK as nativesdk- versions instead)
-        f.write('POKYQEMUDEPS_forcevariable = ""\n\n')
-        f.write('EXTRA_IMAGEDEPENDS_remove = "qemu-native qemu-helper-native"\n\n')
-
         # Bypass the default connectivity check if any
         f.write('CONNECTIVITY_CHECK_URIS = ""\n\n')
 
Signature · 1 line
2.1.0
#9 lore raw

PATCH 9/11 Extensible SDK: allow for installation without preparing build system

In reply to Paul Eggleton

From: "[email protected]" <[email protected]>

When publishing SDK, what we want is basically its metadata and sstate cache objects. We don't want the SDK to be prepared with running bitbake as it takes time which reproduces meaningless output for the published SDK.

So this patch adds an option to allow for SDK to be extracted without preparing the build system.

---
 meta/classes/populate_sdk_ext.bbclass | 12 +++++++-----
 meta/files/toolchain-shar-extract.sh  |  7 ++++++-
 2 files changed, 13 insertions(+), 6 deletions(-)
meta/classes/populate_sdk_ext.bbclass +7 -5
index fc2d96f..0b012eb 100644
@@ -228,11 +228,13 @@ sdk_ext_postinst() {
 	# For now this is where uninative.bbclass expects the tarball
 	mv *-nativesdk-libc.tar.* $target_sdk_dir/`dirname ${oe_init_build_env_path}`
 
-	printf "Preparing build system...\n"
-	# dash which is /bin/sh on Ubuntu will not preserve the
-	# current working directory when first ran, nor will it set $1 when
-	# sourcing a script. That is why this has to look so ugly.
-	sh -c ". buildtools/environment-setup* > preparing_build_system.log && cd $target_sdk_dir/`dirname ${oe_init_build_env_path}` && set $target_sdk_dir && . $target_sdk_dir/${oe_init_build_env_path} $target_sdk_dir >> preparing_build_system.log && bitbake ${SDK_TARGETS} >> preparing_build_system.log" || { echo "SDK preparation failed: see `pwd`/preparing_build_system.log" ; exit 1 ; }
+	if [ "$prepare_buildsystem" != "no" ]; then
+	    printf "Preparing build system...\n"
+	    # dash which is /bin/sh on Ubuntu will not preserve the
+	    # current working directory when first ran, nor will it set $1 when
+	    # sourcing a script. That is why this has to look so ugly.
+	    sh -c ". buildtools/environment-setup* > preparing_build_system.log && cd $target_sdk_dir/`dirname ${oe_init_build_env_path}` && set $target_sdk_dir && . $target_sdk_dir/${oe_init_build_env_path} $target_sdk_dir >> preparing_build_system.log && bitbake ${SDK_TARGETS} >> preparing_build_system.log" || { echo "SDK preparation failed: see `pwd`/preparing_build_system.log" ; exit 1 ; }
+	fi
 	echo done
 }
 
meta/files/toolchain-shar-extract.sh +6 -1
index 3624940..cd0a547 100644
@@ -32,7 +32,7 @@ answer=""
 relocate=1
 savescripts=0
 verbose=0
-while getopts ":yd:DRS" OPT; do
+while getopts ":yd:nDRS" OPT; do
 	case $OPT in
 	y)
 		answer="Y"
@@ -40,6 +40,9 @@ while getopts ":yd:DRS" OPT; do
 	d)
 		target_sdk_dir=$OPTARG
 		;;
+	n)
+		prepare_buildsystem="no"
+		;;
 	D)
 		verbose=1
 		;;
@@ -54,6 +57,8 @@ while getopts ":yd:DRS" OPT; do
 		echo "Usage: $(basename $0) [-y] [-d <dir>]"
 		echo "  -y         Automatic yes to all prompts"
 		echo "  -d <dir>   Install the SDK to <dir>"
+		echo "======== Extensible SDK only options ============"
+		echo "  -n         Do not prepare the build system"
 		echo "======== Advanced DEBUGGING ONLY OPTIONS ========"
 		echo "  -S         Save relocation scripts"
 		echo "  -R         Do not relocate executables"
Signature · 1 line
2.1.0
#10 lore raw

PATCH 10/11 oe-publish-sdk: add script

In reply to Paul Eggleton

From: "[email protected]" <[email protected]>

Add a script to publish an extensible SDK that has previously been built to a specified destination. This published SDK is intended to be accessed by the devtool sdk-update command from an installed copy of the extensible SDK.

e.g.
oe-publish-sdk <ext-sdk> <destination>

---
 scripts/oe-publish-sdk | 143 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 143 insertions(+)
 create mode 100755 scripts/oe-publish-sdk
scripts/oe-publish-sdk new file +143 -0
new file mode 100755
index 0000000..1737c9f
@@ -0,0 +1,143 @@
+#!/usr/bin/env python
+
+# OpenEmbedded SDK publishing tool
+
+# oe-publish-sdk publish <ext-sdk> <destination>
+# <ext-sdk>: extensible SDK to publish (path to the installer shell script)
+# <destination>: local or remote location which servers as an SDK update server
+# e.g.
+# oe-publish-sdk /path/to/sdk-ext.sh /mnt/poky/sdk-ext
+# oe-publish-sdk /path/to/sdk-ext.sh user@host:/opt/poky/sdk-ext
+#
+
+import sys
+import os
+import argparse
+import glob
+import re
+import subprocess
+import logging
+import shutil
+import errno
+
+scripts_path = os.path.dirname(os.path.realpath(__file__))
+lib_path = scripts_path + '/lib'
+sys.path = sys.path + [lib_path]
+import scriptutils
+logger = scriptutils.logger_create('sdktool')
+
+def mkdir(d):
+    try:
+        os.makedirs(d)
+    except OSError as e:
+        if e.errno != errno.EEXIST:
+            raise e
+
+def publish(args):
+    logger.debug("In publish function")
+    target_sdk = args.sdk
+    destination = args.dest
+    logger.debug("target_sdk = %s, update_server = %s" % (target_sdk, destination))
+    sdk_basename = os.path.basename(target_sdk)
+
+    # Ensure the SDK exists
+    if not os.path.exists(target_sdk):
+        logger.error("%s doesn't exist" % target_sdk)
+        return -1
+
+    if ':' in destination:
+        is_remote = True
+        host, destdir = destination.split(':')
+        dest_sdk = os.path.join(destdir, sdk_basename)
+    else:
+        is_remote = False
+        dest_sdk = os.path.join(destination, sdk_basename)
+
+    # Making sure the directory exists
+    logger.debug("Making sure the destination directory exists")
+    if not is_remote:
+        mkdir(destination)
+    else:
+        cmd = "ssh %s 'mkdir -p %s'" % (host, destdir)
+        ret = subprocess.call(cmd, shell=True)
+        if ret != 0:
+            logger.error("Making directory %s on %s failed" % (destdir, host))
+            return ret
+
+    # Copying the SDK to the destination
+    logger.info("Copying the SDK to destination")
+    if not is_remote:
+        if os.path.exists(dest_sdk):
+            os.remove(dest_sdk)
+        if (os.stat(target_sdk).st_dev == os.stat(destination).st_dev):
+            os.link(target_sdk, dest_sdk)
+        else:
+            shutil.copy(target_sdk, dest_sdk)
+    else:
+        cmd = "scp %s %s" % (target_sdk, destination)
+        ret = subprocess.call(cmd, shell=True)
+        if ret != 0:
+            logger.error("scp %s %s failed" % (target_sdk, destination))
+            return ret
+
+    # Unpack the SDK
+    logger.info("Unpacking SDK")
+    if not is_remote:
+        cmd = "sh %s -n -y -d %s" % (dest_sdk, destination)
+        ret = subprocess.call(cmd, shell=True)
+        if ret == 0:
+            logger.info('Successfully unpacked %s to %s' % (dest_sdk, destination))
+        else:
+            logger.error('Failed to unpack %s to %s' % (dest_sdk, destination))
+            return ret
+    else:
+        cmd = "ssh %s 'sh %s -n -y -d %s'" % (host, dest_sdk, destdir)
+        ret = subprocess.call(cmd, shell=True)
+        if ret == 0:
+            logger.info('Successfully unpacked %s to %s' % (dest_sdk, destdir))
+        else:
+            logger.error('Failed to unpack %s to %s' % (dest_sdk, destdir))
+            return ret
+
+    # Setting up the git repo
+    if not is_remote:
+        cmd = 'set -e; cd %s/layers; if [ ! -e .git ]; then git init .; mv .git/hooks/post-update.sample .git/hooks/post-update; fi; git add -A .; git commit -q -m "init repo" || true;' % destination
+    else:
+        cmd = "ssh %s 'set -e; cd %s/layers; if [ ! -e .git ]; then git init .; mv .git/hooks/post-update.sample .git/hooks/post-update; fi; git add -A .; git commit -q -m \"init repo\" || true;'" % (host, destdir)
+    ret = subprocess.call(cmd, shell=True)
+    if ret == 0:
+        logger.info('SDK published successfully')
+    else:
+        logger.error('Failed to set up layer git repo')
+    return ret
+
+
+def main():
+    parser = argparse.ArgumentParser(description="OpenEmbedded development tool",
+                                     epilog="Use %(prog)s <subcommand> --help to get help on a specific command")
+    parser.add_argument('-d', '--debug', help='Enable debug output', action='store_true')
+    parser.add_argument('-q', '--quiet', help='Print only errors', action='store_true')
+
+    parser.add_argument('sdk', help='Extensible SDK to publish')
+    parser.add_argument('dest', help='Destination to publish SDK to')
+
+    parser.set_defaults(func=publish)
+
+    args = parser.parse_args()
+
+    if args.debug:
+        logger.setLevel(logging.DEBUG)
+    elif args.quiet:
+        logger.setLevel(logging.ERROR)
+
+    ret = args.func(args)
+    return ret
+
+if __name__ == "__main__":
+    try:
+        ret = main()
+    except Exception:
+        ret = 1
+        import traceback
+        traceback.print_exc(5)
+    sys.exit(ret)
Signature · 1 line
2.1.0
#11 lore raw

PATCH 11/11 devtool: add mechanism for updating extensible SDK

In reply to Paul Eggleton

From: "[email protected]" <[email protected]>

Enable updating the installed extensible SDK from a local or remote server, avoiding the need to install it again from scratch when updating. (This assumes that the updated SDK has been built and then published somewhere using the oe-publish-sdk script beforehand.)

This plugin is only enabled when devtool is used within the extensible SDK since it doesn't make sense to use it next to a normal install of the build system.

E.g.
devtool sdk-update /mnt/sdk-repo/
devtool sdk-update http://mysdkhost/sdk

---
 meta/classes/populate_sdk_ext.bbclass |   6 ++
 scripts/lib/devtool/sdk.py            | 197 ++++++++++++++++++++++++++++++++++
 2 files changed, 203 insertions(+)
 create mode 100644 scripts/lib/devtool/sdk.py
meta/classes/populate_sdk_ext.bbclass +6 -0
index 0b012eb..4ef8838 100644
@@ -18,6 +18,7 @@ SDK_RELOCATE_AFTER_INSTALL_task-populate-sdk-ext = "0"
 SDK_LOCAL_CONF_WHITELIST ?= ""
 SDK_LOCAL_CONF_BLACKLIST ?= "CONF_VERSION BB_NUMBER_THREADS PARALLEL_MAKE PRSERV_HOST"
 SDK_INHERIT_BLACKLIST ?= "buildhistory icecc"
+SDK_UPDATE_URL ?= ""
 
 SDK_TARGETS ?= "${PN}"
 OE_INIT_ENV_SCRIPT ?= "oe-init-build-env"
@@ -85,6 +86,11 @@ python copy_buildsystem () {
     config.set('General', 'bitbake_subdir', conf_bbpath)
     config.set('General', 'init_path', conf_initpath)
     config.set('General', 'core_meta_subdir', core_meta_subdir)
+    config.add_section('SDK')
+    config.set('SDK', 'sdk_targets', d.getVar('SDK_TARGETS', True))
+    updateurl = d.getVar('SDK_UPDATE_URL', True)
+    if updateurl:
+        config.set('SDK', 'updateserver', updateurl)
     bb.utils.mkdirhier(os.path.join(baseoutpath, 'conf'))
     with open(os.path.join(baseoutpath, 'conf', 'devtool.conf'), 'w') as f:
         config.write(f)
scripts/lib/devtool/sdk.py new file +197 -0

197 lines not on this page · raw message

Signature · 1 line
2.1.0