Fixes for eSDK and testsdkext

PATCH Superseded Robert Yang 4 participants 33 messages Nov 17, 2016 – Dec 14, 2016 1 message not in the archive

lore

Share & tools

lore

Robert Yang

author

2016-11-17 06:19 UTC #0 lore raw

cover Fixes for eSDK and testsdkext

The following changes since commit a675b2c89e477af088faee9b3be96eae19a85f0b:

sanity.bbclass: fix logging of an error (2016-11-15 15:18:50 +0000)

are available in the git repository at:

git://git.openembedded.org/openembedded-core-contrib rbt/eSDK
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=rbt/eSDK

Robert Yang (8):
populate_sdk_ext.bbclass: install multilib targets as populate_sdk
does
oeqa/sdkext/devtool.py: remove workspace/sources before running test
cases
oe-publish-sdk: make cmd easier to read
oe-publish-sdk: add pyshtables.py to .gitignore oeqa/sdkext/devtool.py: don't reset when the test is failed oeqa/oetest.py: add hasLockedSig() oeqa/sdkext/devtool.py: skip test_extend_autotools_recipe_creation
when no libxml2
oe/copy_buildsystem.py: add SDK_LAYERS_EXCLUDE_PATTERN

 meta/classes/populate_sdk_ext.bbclass | 61 ++++++++++++++++++++++-------------
 meta/lib/oe/copy_buildsystem.py       | 18 +++++++++++
 meta/lib/oeqa/oetest.py               | 13 ++++++++
 meta/lib/oeqa/sdkext/devtool.py       | 13 ++++----
 scripts/oe-publish-sdk                | 19 +++++++++--
 5 files changed, 93 insertions(+), 31 deletions(-)

Signature · 1 line

2.10.2

Robert Yang

author

2016-11-17 06:19 UTC #1 lore raw

PATCH 1/8 populate_sdk_ext.bbclass: install multilib targets as populate_sdk does

Fixed:
MACHINE = "qemux86-64"
require conf/multilib.conf
MULTILIBS = "multilib:lib32"
DEFAULTTUNE_virtclass-multilib-lib32 = "x86"

$ bitbake core-image-minimal -cpopulate_sdk_ext
[snip]
Testing /buildarea/lyang1/test_po/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/testsdkext//tc/environment-setup-x86-pokymllib32-linux
test_cvs (oeqa.sdk.buildcvs.BuildCvsTest) ... FAIL
[snip]

It was failed because no lib32 toolchains.

The fixes include:
* Set SDK_TARGETS correctly
* Return multilib depends in get_ext_sdk_depends()
* Write information to all environment-setup-* scripts.

[YOCTO #10647]

Signed-off-by Robert Yang <[email protected]>

---
 meta/classes/populate_sdk_ext.bbclass | 61 ++++++++++++++++++++++-------------
 1 file changed, 38 insertions(+), 23 deletions(-)

meta/classes/populate_sdk_ext.bbclass +38 ~~-23~~

index 26b5ca6..ce9c40a 100644
@@ -39,7 +39,7 @@ SDK_LOCAL_CONF_BLACKLIST ?= "CONF_VERSION \
 SDK_INHERIT_BLACKLIST ?= "buildhistory icecc"
 SDK_UPDATE_URL ?= ""
 
-SDK_TARGETS ?= "${PN}"
+SDK_TARGETS ?= "${@multilib_pkg_extend(d, d.getVar('BPN', True))}"
 
 def get_sdk_install_targets(d, images_only=False):
     sdk_install_targets = ''

@@ -562,38 +562,52 @@ SDK_PRE_INSTALL_COMMAND_task-populate-sdk-ext = "${sdk_ext_preinst}"
 sdk_ext_postinst() {
 	printf "\nExtracting buildtools...\n"
 	cd $target_sdk_dir
-	env_setup_script="$target_sdk_dir/environment-setup-${REAL_MULTIMACH_TARGET_SYS}"
-	printf "buildtools\ny" | ./${SDK_BUILDTOOLS_INSTALLER} > buildtools.log || { printf 'ERROR: buildtools installation failed:\n' ; cat buildtools.log ; echo "printf 'ERROR: this SDK was not fully installed and needs reinstalling\n'" >> $env_setup_script ; exit 1 ; }
+	env_setup_scripts="`ls $target_sdk_dir/environment-setup-*`"
+	./${SDK_BUILDTOOLS_INSTALLER} -d buildtools -y > buildtools.log
+	if [ $? -ne 0 ]; then
+		echo 'ERROR: buildtools installation failed:'
+		cat buildtools.log
+		for e in $env_setup_scripts; do
+			echo "echo 'ERROR: this SDK was not fully installed and needs reinstalling'" >> $e
+		done
+		exit 1
+	fi
 
 	# Delete the buildtools tar file since it won't be used again
 	rm -f ./${SDK_BUILDTOOLS_INSTALLER}
 	# We don't need the log either since it succeeded
 	rm -f buildtools.log
 
-	# Make sure when the user sets up the environment, they also get
-	# the buildtools-tarball tools in their path.
-	echo ". $target_sdk_dir/buildtools/environment-setup*" >> $env_setup_script
-
-	# Allow bitbake environment setup to be ran as part of this sdk.
-	echo "export OE_SKIP_SDK_CHECK=1" >> $env_setup_script
+	for e in $env_setup_scripts; do
+		# Make sure when the user sets up the environment, they also get
+		# the buildtools-tarball tools in their path.
+		echo ". $target_sdk_dir/buildtools/environment-setup*" >> $e
 
-	# A bit of another hack, but we need this in the path only for devtool
-	# so put it at the end of $PATH.
-	echo "export PATH=$target_sdk_dir/sysroots/${SDK_SYS}${bindir_nativesdk}:\$PATH" >> $env_setup_script
+		# Allow bitbake environment setup to be ran as part of this sdk.
+		echo "export OE_SKIP_SDK_CHECK=1" >> $e
 
-	echo "printf 'SDK environment now set up; additionally you may now run devtool to perform development tasks.\nRun devtool --help for further details.\n'" >> $env_setup_script
-
-	# Warn if trying to use external bitbake and the ext SDK together
-	echo "(which bitbake > /dev/null 2>&1 && echo 'WARNING: attempting to use the extensible SDK in an environment set up to run bitbake - this may lead to unexpected results. Please source this script in a new shell session instead.') || true" >> $env_setup_script
+		# A bit of another hack, but we need this in the path only for devtool
+		# so put it at the end of $PATH.
+		echo "export PATH=$target_sdk_dir/sysroots/${SDK_SYS}${bindir_nativesdk}:\$PATH" >> $e
+		echo "printf 'SDK environment now set up; additionally you may now run devtool to perform development tasks.\nRun devtool --help for further details.\n'" >> $e
+		# Warn if trying to use external bitbake and the ext SDK together
+		echo "(which bitbake > /dev/null 2>&1 && echo 'WARNING: attempting to use the extensible SDK in an environment set up to run bitbake - this may lead to unexpected results. Please source this script in a new shell session instead.') || true" >> $e
+	done
 
 	if [ "$prepare_buildsystem" != "no" ]; then
-		printf "Preparing build system...\n"
+		echo "Preparing build system..."
 		# dash which is /bin/sh on Ubuntu will not preserve the
 		# current working directory when first ran, nor will it set $1 when
 		# sourcing a script. That is why this has to look so ugly.
 		LOGFILE="$target_sdk_dir/preparing_build_system.log"
-		sh -c ". buildtools/environment-setup* > $LOGFILE && cd $target_sdk_dir/`dirname ${oe_init_build_env_path}` && set $target_sdk_dir && . $target_sdk_dir/${oe_init_build_env_path} $target_sdk_dir >> $LOGFILE && python $target_sdk_dir/ext-sdk-prepare.py $LOGFILE '${SDK_INSTALL_TARGETS}'" || { echo "printf 'ERROR: this SDK was not fully installed and needs reinstalling\n'" >> $env_setup_script ; exit 1 ; }
-		rm $target_sdk_dir/ext-sdk-prepare.py
+		sh -c ". buildtools/environment-setup* > $LOGFILE && cd $target_sdk_dir/`dirname ${oe_init_build_env_path}` && set $target_sdk_dir && . $target_sdk_dir/${oe_init_build_env_path} $target_sdk_dir >> $LOGFILE && python $target_sdk_dir/ext-sdk-prepare.py $LOGFILE '${SDK_INSTALL_TARGETS}'"
+		if [ $? -ne 0 ]; then
+			for e in $env_setup_scripts; do
+				echo "echo 'ERROR: this SDK was not fully installed and needs reinstalling'" >> $e
+			done
+			exit 1
+		fi
+		rm -f $target_sdk_dir/ext-sdk-prepare.py
 	fi
 	echo done
 }

@@ -624,10 +638,11 @@ fakeroot python do_populate_sdk_ext() {
 def get_ext_sdk_depends(d):
     # Note: the deps varflag is a list not a string, so we need to specify expand=False
     deps = d.getVarFlag('do_image_complete', 'deps', False)
-    pn = d.getVar('PN', True)
-    deplist = ['%s:%s' % (pn, dep) for dep in deps]
-    for task in ['do_image_complete', 'do_rootfs', 'do_build']:
-        deplist.extend((d.getVarFlag(task, 'depends', True) or '').split())
+    deplist = []
+    for pn in multilib_pkg_extend(d, d.getVar('BPN', True)).split():
+        deplist += ['%s:%s' % (pn, dep) for dep in deps]
+        for task in ['do_image_complete', 'do_rootfs', 'do_build']:
+            deplist.extend((d.getVarFlag(task, 'depends', True) or '').split())
     return ' '.join(deplist)
 
 python do_sdk_depends() {

Signature · 1 line

2.10.2

Paul Eggleton

2016-12-13 04:56 UTC #2 lore raw

Re: [PATCH 1/8] populate_sdk_ext.bbclass: install multilib targets as populate_sdk does

Hi Robert,

There are a bunch of changes in here that don't relate to the multilib fix - details below.

Quoted 43 lines · On Wed, 16 Nov 2016 22:19:30 Robert Yang wrote:

Fixed:
MACHINE = "qemux86-64"
require conf/multilib.conf
MULTILIBS = "multilib:lib32"
DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
$ bitbake core-image-minimal -cpopulate_sdk_ext
[snip]
Testing
/buildarea/lyang1/test_po/tmp/work/qemux86_64-poky-linux/core-image-minimal
/1.0-r0/testsdkext//tc/environment-setup-x86-pokymllib32-linux test_cvs
(oeqa.sdk.buildcvs.BuildCvsTest) ... FAIL
[snip]
It was failed because no lib32 toolchains.
The fixes include:
* Set SDK_TARGETS correctly
* Return multilib depends in get_ext_sdk_depends()
* Write information to all environment-setup-* scripts.
[YOCTO #10647]
Signed-off-by Robert Yang <[email protected]>
---
 meta/classes/populate_sdk_ext.bbclass | 61
++++++++++++++++++++++------------- 1 file changed, 38 insertions(+), 23 deletions(-)
(quoted diff) +0 -0
b/meta/classes/populate_sdk_ext.bbclass index 26b5ca6..ce9c40a 100644
meta/classes/populate_sdk_ext.bbclass +1 -1
@@ -39,7 +39,7 @@ SDK_LOCAL_CONF_BLACKLIST ?= "CONF_VERSION \
 SDK_INHERIT_BLACKLIST ?= "buildhistory icecc"
 SDK_UPDATE_URL ?= ""
 
-SDK_TARGETS ?= "${PN}"
+SDK_TARGETS ?= "${@multilib_pkg_extend(d, d.getVar('BPN', True))}"
 
 def get_sdk_install_targets(d, images_only=False):
     sdk_install_targets = ''

meta/classes/populate_sdk_ext.bbclass SDK_PRE_INSTALL_COMMAND_task-populate-sdk-ext = · line 565

Quoted 4 lines

meta/classes/populate_sdk_ext.bbclass +0 -1

@@ -562,38 +562,52 @@ SDK_PRE_INSTALL_COMMAND_task-populate-sdk-ext =
 "${sdk_ext_preinst}" sdk_ext_postinst() {
 	printf "\nExtracting buildtools...\n"
 	cd $target_sdk_dir
-	env_setup_script="$target_sdk_dir/environment-

setup-${REAL_MULTIMACH_TARGE

Quoted 13 lines

T_SYS}" - printf "buildtools\ny" | ./${SDK_BUILDTOOLS_INSTALLER} >
buildtools.log || { printf 'ERROR: buildtools installation failed:\n' ; cat buildtools.log ; echo "printf 'ERROR: this SDK was not fully installed and needs reinstalling\n'" >> $env_setup_script ; exit 1 ; }
meta/classes/populate_sdk_ext.bbclass +7 -0
+	env_setup_scripts="`ls $target_sdk_dir/environment-setup-*`"
+	./${SDK_BUILDTOOLS_INSTALLER} -d buildtools -y > buildtools.log
+	if [ $? -ne 0 ]; then
+		echo 'ERROR: buildtools installation failed:'
+		cat buildtools.log
+		for e in $env_setup_scripts; do
+			echo "echo 'ERROR: this SDK was not fully installed and needs
reinstalling'" >> $e + done

meta/classes/populate_sdk_ext.bbclass

Quoted 2 lines

meta/classes/populate_sdk_ext.bbclass +2 -0
+		exit 1
+	fi

This change isn't entirely related to the multilib changes.

Quoted 19 lines

meta/classes/populate_sdk_ext.bbclass +0 -3

 	# Delete the buildtools tar file since it won't be used again
 	rm -f ./${SDK_BUILDTOOLS_INSTALLER}
 	# We don't need the log either since it succeeded
 	rm -f buildtools.log
 
-	# Make sure when the user sets up the environment, they also get
-	# the buildtools-tarball tools in their path.
-	echo ". $target_sdk_dir/buildtools/environment-setup*" >>

$env_setup_script -

meta/classes/populate_sdk_ext.bbclass +4 -5

-	# Allow bitbake environment setup to be ran as part of this sdk.
-	echo "export OE_SKIP_SDK_CHECK=1" >> $env_setup_script
+	for e in $env_setup_scripts; do
+		# Make sure when the user sets up the environment, they also get
+		# the buildtools-tarball tools in their path.
+		echo ". $target_sdk_dir/buildtools/environment-setup*" >> $e
 
-	# A bit of another hack, but we need this in the path only for devtool
-	# so put it at the end of $PATH.
-	echo "export

Quoted 2 lines

PATH=$target_sdk_dir/sysroots/${SDK_SYS}${bindir_nativesdk}:\$PATH" >>
$env_setup_script + # Allow bitbake environment setup to be ran as part

Quoted 10 lines

this sdk. + echo "export OE_SKIP_SDK_CHECK=1" >> $e
meta/classes/populate_sdk_ext.bbclass +0 -1
-	echo "printf 'SDK environment now set up; additionally you may now run
devtool to perform development tasks.\nRun devtool --help for further details.\n'" >> $env_setup_script -
meta/classes/populate_sdk_ext.bbclass +0 -2
-	# Warn if trying to use external bitbake and the ext SDK together
-	echo "(which bitbake > /dev/null 2>&1 && echo 'WARNING: attempting to use
the extensible SDK in an environment set up to run bitbake - this may lead to unexpected results. Please source this script in a new shell session
instead.') || true" >> $env_setup_script + # A bit of another hack, but

Quoted 1 line

need this in the path only for devtool + # so put it at the end of

$PATH.

meta/classes/populate_sdk_ext.bbclass

Quoted 3 lines

meta/classes/populate_sdk_ext.bbclass +1 -0
+		echo "export
PATH=$target_sdk_dir/sysroots/${SDK_SYS}${bindir_nativesdk}:\$PATH" >> $e
meta/classes/populate_sdk_ext.bbclass +1 -0
+		echo "printf 'SDK environment now set up; additionally you may now 

run

Quoted 2 lines

devtool to perform development tasks.\nRun devtool --help for further
details.\n'" >> $e + # Warn if trying to use external bitbake and the

ext

meta/classes/populate_sdk_ext.bbclass SDK_PRE_INSTALL_COMMAND_task-populate-sdk-ext = "${sdk_ext_preinst}" · line 598

Quoted 8 lines

SDK together + echo "(which bitbake > /dev/null 2>&1 && echo 'WARNING:
attempting to use the extensible SDK in an environment set up to run bitbake - this may lead to unexpected results. Please source this script in
a new shell session instead.') || true" >> $e + done
meta/classes/populate_sdk_ext.bbclass +1 -1
 	if [ "$prepare_buildsystem" != "no" ]; then
-		printf "Preparing build system...\n"
+		echo "Preparing build system..."

Why did you change this?

Quoted 19 lines

meta/classes/populate_sdk_ext.bbclass +0 -1
 		# dash which is /bin/sh on Ubuntu will not preserve the
 		# current working directory when first ran, nor will it set $1 when
 		# sourcing a script. That is why this has to look so ugly.
 		LOGFILE="$target_sdk_dir/preparing_build_system.log"
-		sh -c ". buildtools/environment-setup* > $LOGFILE && cd
$target_sdk_dir/dirname ${oe_init_build_env_path} && set $target_sdk_dir && . $target_sdk_dir/${oe_init_build_env_path} $target_sdk_dir >> $LOGFILE && python $target_sdk_dir/ext-sdk-prepare.py $LOGFILE '${SDK_INSTALL_TARGETS}'" || { echo "printf 'ERROR: this SDK was not fully installed and needs reinstalling\n'" >> $env_setup_script ; exit 1 ; }
meta/classes/populate_sdk_ext.bbclass +1 -1
-		rm $target_sdk_dir/ext-sdk-prepare.py
+		sh -c ". buildtools/environment-setup* > $LOGFILE && cd
$target_sdk_dir/dirname ${oe_init_build_env_path} && set $target_sdk_dir && . $target_sdk_dir/${oe_init_build_env_path} $target_sdk_dir >> $LOGFILE && python $target_sdk_dir/ext-sdk-prepare.py $LOGFILE
'${SDK_INSTALL_TARGETS}'" + if [ $? -ne 0 ]; then
meta/classes/populate_sdk_ext.bbclass +2 -0
+			for e in $env_setup_scripts; do
+				echo "echo 'ERROR: this SDK was not fully installed and needs
reinstalling'" >> $e + done

meta/classes/populate_sdk_ext.bbclass SDK_PRE_INSTALL_COMMAND_task-populate-sdk-ext = "${sdk_ext_preinst}" · line 610

Quoted 3 lines

meta/classes/populate_sdk_ext.bbclass +3 -0
+			exit 1
+		fi
+		rm -f $target_sdk_dir/ext-sdk-prepare.py

That last line is also unrelated.

Cheers,
Paul

Signature · 2 lines

Paul Eggleton
Intel Open Source Technology Centre

Robert Yang

author

2016-12-13 06:03 UTC #3 lore raw

Re: [PATCH 1/8] populate_sdk_ext.bbclass: install multilib targets as populate_sdk does

Quoted 4 lines · Paul Eggleton

Hi Robert,
There are a bunch of changes in here that don't relate to the multilib fix - details below.

Thanks, I will split them into two commits.

Quoted 115 lines · On Wed, 16 Nov 2016 22:19:30 Robert Yang wrote:

Quoted 43 lines
Fixed:
MACHINE = "qemux86-64"
require conf/multilib.conf
MULTILIBS = "multilib:lib32"
DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
$ bitbake core-image-minimal -cpopulate_sdk_ext
[snip]
Testing
/buildarea/lyang1/test_po/tmp/work/qemux86_64-poky-linux/core-image-minimal
/1.0-r0/testsdkext//tc/environment-setup-x86-pokymllib32-linux test_cvs
(oeqa.sdk.buildcvs.BuildCvsTest) ... FAIL
[snip]
It was failed because no lib32 toolchains.
The fixes include:
* Set SDK_TARGETS correctly
* Return multilib depends in get_ext_sdk_depends()
* Write information to all environment-setup-* scripts.
[YOCTO #10647]
Signed-off-by Robert Yang <[email protected]>
---
 meta/classes/populate_sdk_ext.bbclass | 61
++++++++++++++++++++++------------- 1 file changed, 38 insertions(+), 23 deletions(-)
(quoted diff) +0 -0
b/meta/classes/populate_sdk_ext.bbclass index 26b5ca6..ce9c40a 100644
meta/classes/populate_sdk_ext.bbclass +1 -1
@@ -39,7 +39,7 @@ SDK_LOCAL_CONF_BLACKLIST ?= "CONF_VERSION \
 SDK_INHERIT_BLACKLIST ?= "buildhistory icecc"
 SDK_UPDATE_URL ?= ""
 
-SDK_TARGETS ?= "${PN}"
+SDK_TARGETS ?= "${@multilib_pkg_extend(d, d.getVar('BPN', True))}"
 
 def get_sdk_install_targets(d, images_only=False):
     sdk_install_targets = ''
meta/classes/populate_sdk_ext.bbclass SDK_PRE_INSTALL_COMMAND_task-populate-sdk-ext = · line 565
Quoted 4 lines
meta/classes/populate_sdk_ext.bbclass +0 -1
@@ -562,38 +562,52 @@ SDK_PRE_INSTALL_COMMAND_task-populate-sdk-ext =
 "${sdk_ext_preinst}" sdk_ext_postinst() {
 	printf "\nExtracting buildtools...\n"
 	cd $target_sdk_dir
-	env_setup_script="$target_sdk_dir/environment-
setup-${REAL_MULTIMACH_TARGE
Quoted 13 lines
T_SYS}" - printf "buildtools\ny" | ./${SDK_BUILDTOOLS_INSTALLER} >
buildtools.log || { printf 'ERROR: buildtools installation failed:\n' ; cat buildtools.log ; echo "printf 'ERROR: this SDK was not fully installed and needs reinstalling\n'" >> $env_setup_script ; exit 1 ; }
meta/classes/populate_sdk_ext.bbclass +7 -0
+	env_setup_scripts="`ls $target_sdk_dir/environment-setup-*`"
+	./${SDK_BUILDTOOLS_INSTALLER} -d buildtools -y > buildtools.log
+	if [ $? -ne 0 ]; then
+		echo 'ERROR: buildtools installation failed:'
+		cat buildtools.log
+		for e in $env_setup_scripts; do
+			echo "echo 'ERROR: this SDK was not fully installed and needs
reinstalling'" >> $e + done
meta/classes/populate_sdk_ext.bbclass
Quoted 2 lines
meta/classes/populate_sdk_ext.bbclass +2 -0
+		exit 1
+	fi
This change isn't entirely related to the multilib changes.
Quoted 19 lines
meta/classes/populate_sdk_ext.bbclass +0 -3
 	# Delete the buildtools tar file since it won't be used again
 	rm -f ./${SDK_BUILDTOOLS_INSTALLER}
 	# We don't need the log either since it succeeded
 	rm -f buildtools.log
 
-	# Make sure when the user sets up the environment, they also get
-	# the buildtools-tarball tools in their path.
-	echo ". $target_sdk_dir/buildtools/environment-setup*" >>
$env_setup_script -
meta/classes/populate_sdk_ext.bbclass +4 -5
-	# Allow bitbake environment setup to be ran as part of this sdk.
-	echo "export OE_SKIP_SDK_CHECK=1" >> $env_setup_script
+	for e in $env_setup_scripts; do
+		# Make sure when the user sets up the environment, they also get
+		# the buildtools-tarball tools in their path.
+		echo ". $target_sdk_dir/buildtools/environment-setup*" >> $e
 
-	# A bit of another hack, but we need this in the path only for devtool
-	# so put it at the end of $PATH.
-	echo "export
Quoted 2 lines
PATH=$target_sdk_dir/sysroots/${SDK_SYS}${bindir_nativesdk}:\$PATH" >>
$env_setup_script + # Allow bitbake environment setup to be ran as part
of
Quoted 10 lines
this sdk. + echo "export OE_SKIP_SDK_CHECK=1" >> $e
meta/classes/populate_sdk_ext.bbclass +0 -1
-	echo "printf 'SDK environment now set up; additionally you may now run
devtool to perform development tasks.\nRun devtool --help for further details.\n'" >> $env_setup_script -
meta/classes/populate_sdk_ext.bbclass +0 -2
-	# Warn if trying to use external bitbake and the ext SDK together
-	echo "(which bitbake > /dev/null 2>&1 && echo 'WARNING: attempting to use
the extensible SDK in an environment set up to run bitbake - this may lead to unexpected results. Please source this script in a new shell session
instead.') || true" >> $env_setup_script + # A bit of another hack, but
we
Quoted 1 line
need this in the path only for devtool + # so put it at the end of
$PATH.
meta/classes/populate_sdk_ext.bbclass
Quoted 3 lines
meta/classes/populate_sdk_ext.bbclass +1 -0
+		echo "export
PATH=$target_sdk_dir/sysroots/${SDK_SYS}${bindir_nativesdk}:\$PATH" >> $e
meta/classes/populate_sdk_ext.bbclass +1 -0
+		echo "printf 'SDK environment now set up; additionally you may now
run
Quoted 2 lines
devtool to perform development tasks.\nRun devtool --help for further
details.\n'" >> $e + # Warn if trying to use external bitbake and the
ext
Quoted 5 lines
SDK together + echo "(which bitbake > /dev/null 2>&1 && echo 'WARNING:
attempting to use the extensible SDK in an environment set up to run bitbake - this may lead to unexpected results. Please source this script in
a new shell session instead.') || true" >> $e + done

Quoted 5 lines

meta/classes/populate_sdk_ext.bbclass SDK_PRE_INSTALL_COMMAND_task-populate-sdk-ext = "${sdk_ext_preinst}" · line 598
Quoted 3 lines
meta/classes/populate_sdk_ext.bbclass +1 -1
 	if [ "$prepare_buildsystem" != "no" ]; then
-		printf "Preparing build system...\n"
+		echo "Preparing build system..."
Why did you change this?

We don't need complex format here, so so echo is simpler ?

// Robert

Quoted 29 lines

Quoted 20 lines
meta/classes/populate_sdk_ext.bbclass +0 -1
 		# dash which is /bin/sh on Ubuntu will not preserve the
 		# current working directory when first ran, nor will it set $1 when
 		# sourcing a script. That is why this has to look so ugly.
 		LOGFILE="$target_sdk_dir/preparing_build_system.log"
-		sh -c ". buildtools/environment-setup* > $LOGFILE && cd
$target_sdk_dir/dirname ${oe_init_build_env_path} && set $target_sdk_dir && . $target_sdk_dir/${oe_init_build_env_path} $target_sdk_dir >> $LOGFILE && python $target_sdk_dir/ext-sdk-prepare.py $LOGFILE '${SDK_INSTALL_TARGETS}'" || { echo "printf 'ERROR: this SDK was not fully installed and needs reinstalling\n'" >> $env_setup_script ; exit 1 ; }
meta/classes/populate_sdk_ext.bbclass +1 -1
-		rm $target_sdk_dir/ext-sdk-prepare.py
+		sh -c ". buildtools/environment-setup* > $LOGFILE && cd
$target_sdk_dir/dirname ${oe_init_build_env_path} && set $target_sdk_dir && . $target_sdk_dir/${oe_init_build_env_path} $target_sdk_dir >> $LOGFILE && python $target_sdk_dir/ext-sdk-prepare.py $LOGFILE
'${SDK_INSTALL_TARGETS}'" + if [ $? -ne 0 ]; then
meta/classes/populate_sdk_ext.bbclass +2 -0
+			for e in $env_setup_scripts; do
+				echo "echo 'ERROR: this SDK was not fully installed and needs
reinstalling'" >> $e + done
meta/classes/populate_sdk_ext.bbclass SDK_PRE_INSTALL_COMMAND_task-populate-sdk-ext = "${sdk_ext_preinst}" · line 610
Quoted 3 lines
meta/classes/populate_sdk_ext.bbclass +3 -0
+			exit 1
+		fi
+		rm -f $target_sdk_dir/ext-sdk-prepare.py
That last line is also unrelated.
Cheers,
Paul

Robert Yang

author

2016-12-14 02:25 UTC #4 lore raw

Re: [PATCH 1/8] populate_sdk_ext.bbclass: install multilib targets as populate_sdk does

In reply to Paul Eggleton

Hi Paul,

After I looked into the code again, these changes are related, so that I can't split them into 2, please see my comments inline.

Quoted 63 lines · Paul Eggleton

Hi Robert,
There are a bunch of changes in here that don't relate to the multilib fix - details below.
Quoted 43 lines · On Wed, 16 Nov 2016 22:19:30 Robert Yang wrote:
Fixed:
MACHINE = "qemux86-64"
require conf/multilib.conf
MULTILIBS = "multilib:lib32"
DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
$ bitbake core-image-minimal -cpopulate_sdk_ext
[snip]
Testing
/buildarea/lyang1/test_po/tmp/work/qemux86_64-poky-linux/core-image-minimal
/1.0-r0/testsdkext//tc/environment-setup-x86-pokymllib32-linux test_cvs
(oeqa.sdk.buildcvs.BuildCvsTest) ... FAIL
[snip]
It was failed because no lib32 toolchains.
The fixes include:
* Set SDK_TARGETS correctly
* Return multilib depends in get_ext_sdk_depends()
* Write information to all environment-setup-* scripts.
[YOCTO #10647]
Signed-off-by Robert Yang <[email protected]>
---
 meta/classes/populate_sdk_ext.bbclass | 61
++++++++++++++++++++++------------- 1 file changed, 38 insertions(+), 23 deletions(-)
(quoted diff) +0 -0
b/meta/classes/populate_sdk_ext.bbclass index 26b5ca6..ce9c40a 100644
meta/classes/populate_sdk_ext.bbclass +1 -1
@@ -39,7 +39,7 @@ SDK_LOCAL_CONF_BLACKLIST ?= "CONF_VERSION \
 SDK_INHERIT_BLACKLIST ?= "buildhistory icecc"
 SDK_UPDATE_URL ?= ""
 
-SDK_TARGETS ?= "${PN}"
+SDK_TARGETS ?= "${@multilib_pkg_extend(d, d.getVar('BPN', True))}"
 
 def get_sdk_install_targets(d, images_only=False):
     sdk_install_targets = ''
meta/classes/populate_sdk_ext.bbclass SDK_PRE_INSTALL_COMMAND_task-populate-sdk-ext = · line 565
Quoted 4 lines
meta/classes/populate_sdk_ext.bbclass +0 -1
@@ -562,38 +562,52 @@ SDK_PRE_INSTALL_COMMAND_task-populate-sdk-ext =
 "${sdk_ext_preinst}" sdk_ext_postinst() {
 	printf "\nExtracting buildtools...\n"
 	cd $target_sdk_dir
-	env_setup_script="$target_sdk_dir/environment-
setup-${REAL_MULTIMACH_TARGE
Quoted 9 lines
T_SYS}" - printf "buildtools\ny" | ./${SDK_BUILDTOOLS_INSTALLER} >
buildtools.log || { printf 'ERROR: buildtools installation failed:\n' ; cat buildtools.log ; echo "printf 'ERROR: this SDK was not fully installed and needs reinstalling\n'" >> $env_setup_script ; exit 1 ; }
meta/classes/populate_sdk_ext.bbclass +5 -0
+	env_setup_scripts="`ls $target_sdk_dir/environment-setup-*`"
+	./${SDK_BUILDTOOLS_INSTALLER} -d buildtools -y > buildtools.log
+	if [ $? -ne 0 ]; then
+		echo 'ERROR: buildtools installation failed:'
+		cat buildtools.log

Quoted 8 lines

meta/classes/populate_sdk_ext.bbclass
Quoted 6 lines
meta/classes/populate_sdk_ext.bbclass +2 -0
+		for e in $env_setup_scripts; do
+			echo "echo 'ERROR: this SDK was not fully installed and needs
reinstalling'" >> $e + done
meta/classes/populate_sdk_ext.bbclass +2 -0
+		exit 1
+	fi
This change isn't entirely related to the multilib changes.

The old code only considers one env_setup_script, but there are multiple ones now, so they are related.

Quoted 48 lines

Quoted 20 lines
meta/classes/populate_sdk_ext.bbclass +0 -3
 	# Delete the buildtools tar file since it won't be used again
 	rm -f ./${SDK_BUILDTOOLS_INSTALLER}
 	# We don't need the log either since it succeeded
 	rm -f buildtools.log
 
-	# Make sure when the user sets up the environment, they also get
-	# the buildtools-tarball tools in their path.
-	echo ". $target_sdk_dir/buildtools/environment-setup*" >>
$env_setup_script -
meta/classes/populate_sdk_ext.bbclass +4 -5
-	# Allow bitbake environment setup to be ran as part of this sdk.
-	echo "export OE_SKIP_SDK_CHECK=1" >> $env_setup_script
+	for e in $env_setup_scripts; do
+		# Make sure when the user sets up the environment, they also get
+		# the buildtools-tarball tools in their path.
+		echo ". $target_sdk_dir/buildtools/environment-setup*" >> $e
 
-	# A bit of another hack, but we need this in the path only for devtool
-	# so put it at the end of $PATH.
-	echo "export
Quoted 2 lines
PATH=$target_sdk_dir/sysroots/${SDK_SYS}${bindir_nativesdk}:\$PATH" >>
$env_setup_script + # Allow bitbake environment setup to be ran as part
of
Quoted 10 lines
this sdk. + echo "export OE_SKIP_SDK_CHECK=1" >> $e
meta/classes/populate_sdk_ext.bbclass +0 -1
-	echo "printf 'SDK environment now set up; additionally you may now run
devtool to perform development tasks.\nRun devtool --help for further details.\n'" >> $env_setup_script -
meta/classes/populate_sdk_ext.bbclass +0 -2
-	# Warn if trying to use external bitbake and the ext SDK together
-	echo "(which bitbake > /dev/null 2>&1 && echo 'WARNING: attempting to use
the extensible SDK in an environment set up to run bitbake - this may lead to unexpected results. Please source this script in a new shell session
instead.') || true" >> $env_setup_script + # A bit of another hack, but
we
Quoted 1 line
need this in the path only for devtool + # so put it at the end of
$PATH.
meta/classes/populate_sdk_ext.bbclass
Quoted 3 lines
meta/classes/populate_sdk_ext.bbclass +1 -0
+		echo "export
PATH=$target_sdk_dir/sysroots/${SDK_SYS}${bindir_nativesdk}:\$PATH" >> $e
meta/classes/populate_sdk_ext.bbclass +1 -0
+		echo "printf 'SDK environment now set up; additionally you may now
run
Quoted 2 lines
devtool to perform development tasks.\nRun devtool --help for further
details.\n'" >> $e + # Warn if trying to use external bitbake and the
ext
Quoted 5 lines
SDK together + echo "(which bitbake > /dev/null 2>&1 && echo 'WARNING:
attempting to use the extensible SDK in an environment set up to run bitbake - this may lead to unexpected results. Please source this script in
a new shell session instead.') || true" >> $e + done

Quoted 5 lines

meta/classes/populate_sdk_ext.bbclass SDK_PRE_INSTALL_COMMAND_task-populate-sdk-ext = "${sdk_ext_preinst}" · line 598
Quoted 3 lines
meta/classes/populate_sdk_ext.bbclass +1 -1
 	if [ "$prepare_buildsystem" != "no" ]; then
-		printf "Preparing build system...\n"
+		echo "Preparing build system..."
Why did you change this?

Will drop this in V2.

Quoted 17 lines

Quoted 17 lines
meta/classes/populate_sdk_ext.bbclass +0 -1
 		# dash which is /bin/sh on Ubuntu will not preserve the
 		# current working directory when first ran, nor will it set $1 when
 		# sourcing a script. That is why this has to look so ugly.
 		LOGFILE="$target_sdk_dir/preparing_build_system.log"
-		sh -c ". buildtools/environment-setup* > $LOGFILE && cd
$target_sdk_dir/dirname ${oe_init_build_env_path} && set $target_sdk_dir && . $target_sdk_dir/${oe_init_build_env_path} $target_sdk_dir >> $LOGFILE && python $target_sdk_dir/ext-sdk-prepare.py $LOGFILE '${SDK_INSTALL_TARGETS}'" || { echo "printf 'ERROR: this SDK was not fully installed and needs reinstalling\n'" >> $env_setup_script ; exit 1 ; }
meta/classes/populate_sdk_ext.bbclass +1 -1
-		rm $target_sdk_dir/ext-sdk-prepare.py
+		sh -c ". buildtools/environment-setup* > $LOGFILE && cd
$target_sdk_dir/dirname ${oe_init_build_env_path} && set $target_sdk_dir && . $target_sdk_dir/${oe_init_build_env_path} $target_sdk_dir >> $LOGFILE && python $target_sdk_dir/ext-sdk-prepare.py $LOGFILE
'${SDK_INSTALL_TARGETS}'" + if [ $? -ne 0 ]; then

Quoted 8 lines

meta/classes/populate_sdk_ext.bbclass SDK_PRE_INSTALL_COMMAND_task-populate-sdk-ext = "${sdk_ext_preinst}" · line 610
Quoted 6 lines
meta/classes/populate_sdk_ext.bbclass +2 -0
+			for e in $env_setup_scripts; do
+				echo "echo 'ERROR: this SDK was not fully installed and needs
reinstalling'" >> $e + done
meta/classes/populate_sdk_ext.bbclass +3 -0
+			exit 1
+		fi
+		rm -f $target_sdk_dir/ext-sdk-prepare.py
That last line is also unrelated.

The old code only considers one env_setup_script, but there are multiple ones now, so they are related.

// Robert

Quoted 4 lines

Cheers,
Paul

Paul Eggleton

2016-12-14 02:34 UTC #5 lore raw

Re: [PATCH 1/8] populate_sdk_ext.bbclass: install multilib targets as populate_sdk does

Quoted 2 lines · On Wed, 14 Dec 2016 10:25:36 Robert Yang wrote:

After I looked into the code again, these changes are related, so that I can't split them into 2, please see my comments inline.

I've replied inline.

Quoted 80 lines · Paul Eggleton

Quoted 70 lines
There are a bunch of changes in here that don't relate to the multilib fix
- details below.
Quoted 48 lines · On Wed, 16 Nov 2016 22:19:30 Robert Yang wrote:
Fixed:
MACHINE = "qemux86-64"
require conf/multilib.conf
MULTILIBS = "multilib:lib32"
DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
$ bitbake core-image-minimal -cpopulate_sdk_ext
[snip]
Testing
/buildarea/lyang1/test_po/tmp/work/qemux86_64-poky-linux/core-image-minim
al
/1.0-r0/testsdkext//tc/environment-setup-x86-pokymllib32-linux test_cvs
(oeqa.sdk.buildcvs.BuildCvsTest) ... FAIL
[snip]
It was failed because no lib32 toolchains.
The fixes include:
* Set SDK_TARGETS correctly
* Return multilib depends in get_ext_sdk_depends()
* Write information to all environment-setup-* scripts.
[YOCTO #10647]
Signed-off-by Robert Yang <[email protected]>
---
 meta/classes/populate_sdk_ext.bbclass | 61
++++++++++++++++++++++------------- 1 file changed, 38 insertions(+), 23 deletions(-)
(quoted diff) +0 -0
b/meta/classes/populate_sdk_ext.bbclass index 26b5ca6..ce9c40a 100644
meta/classes/populate_sdk_ext.bbclass +1 -1
@@ -39,7 +39,7 @@ SDK_LOCAL_CONF_BLACKLIST ?= "CONF_VERSION \
 
 SDK_INHERIT_BLACKLIST ?= "buildhistory icecc"
 SDK_UPDATE_URL ?= ""
 
-SDK_TARGETS ?= "${PN}"
+SDK_TARGETS ?= "${@multilib_pkg_extend(d, d.getVar('BPN', True))}"
 
 def get_sdk_install_targets(d, images_only=False):
sdk_install_targets = ''
meta/classes/populate_sdk_ext.bbclass SDK_PRE_INSTALL_COMMAND_task-populate-sdk-ext = · line 567
Quoted 6 lines
meta/classes/populate_sdk_ext.bbclass +0 -1
@@ -562,38 +562,52 @@ SDK_PRE_INSTALL_COMMAND_task-populate-sdk-ext =
 "${sdk_ext_preinst}" sdk_ext_postinst() {
 
 	printf "\nExtracting buildtools...\n"
 	cd $target_sdk_dir
 
-	env_setup_script="$target_sdk_dir/environment-
setup-${REAL_MULTIMACH_TARGE
Quoted 9 lines
T_SYS}"
meta/classes/populate_sdk_ext.bbclass +0 -1
-	printf "buildtools\ny" | ./${SDK_BUILDTOOLS_INSTALLER} >
buildtools.log || { printf 'ERROR: buildtools installation failed:\n' ; cat buildtools.log ; echo "printf 'ERROR: this SDK was not fully installed and needs reinstalling\n'" >> $env_setup_script ; exit 1 ; }
meta/classes/populate_sdk_ext.bbclass +4 -0
+	env_setup_scripts="`ls $target_sdk_dir/environment-setup-*`"
+	./${SDK_BUILDTOOLS_INSTALLER} -d buildtools -y > buildtools.log
+	if [ $? -ne 0 ]; then
+		echo 'ERROR: buildtools installation failed:'
Quoted 8 lines
meta/classes/populate_sdk_ext.bbclass
Quoted 6 lines
meta/classes/populate_sdk_ext.bbclass +3 -0
+		cat buildtools.log
+		for e in $env_setup_scripts; do
+			echo "echo 'ERROR: this SDK was not fully installed and needs
reinstalling'" >> $e + done
meta/classes/populate_sdk_ext.bbclass +2 -0
+		exit 1
+	fi
This change isn't entirely related to the multilib changes.

Quoted 2 lines

The old code only considers one env_setup_script, but there are multiple ones now, so they are related.

Yes, I understand that. What I meant was, you've unrolled the single line *and* added handling for multiple env setup scripts - both are good changes but I'd like to see them in separate patches so that it's clear what's being done.

Quoted 32 lines

Quoted 23 lines
Quoted 23 lines
meta/classes/populate_sdk_ext.bbclass +0 -1
 		# dash which is /bin/sh on Ubuntu will not preserve the
 		# current working directory when first ran, nor will it set $1
 		when
 		# sourcing a script. That is why this has to look so ugly.
 		LOGFILE="$target_sdk_dir/preparing_build_system.log"
 
-		sh -c ". buildtools/environment-setup* > $LOGFILE && cd
$target_sdk_dir/dirname ${oe_init_build_env_path} && set $target_sdk_dir
&& . $target_sdk_dir/${oe_init_build_env_path} $target_sdk_dir >> $LOGFILE
&& python $target_sdk_dir/ext-sdk-prepare.py $LOGFILE '${SDK_INSTALL_TARGETS}'" || { echo "printf 'ERROR: this SDK was not fully
installed and needs reinstalling\n'" >> $env_setup_script ; exit 1 ; }
meta/classes/populate_sdk_ext.bbclass +1 -1
-		rm $target_sdk_dir/ext-sdk-prepare.py
+		sh -c ". buildtools/environment-setup* > $LOGFILE && cd
$target_sdk_dir/dirname ${oe_init_build_env_path} && set $target_sdk_dir
&& . $target_sdk_dir/${oe_init_build_env_path} $target_sdk_dir >> $LOGFILE
&& python $target_sdk_dir/ext-sdk-prepare.py $LOGFILE
'${SDK_INSTALL_TARGETS}'" + if [ $? -ne 0 ]; then
Quoted 8 lines
meta/classes/populate_sdk_ext.bbclass SDK_PRE_INSTALL_COMMAND_task-populate-sdk-ext = "${sdk_ext_preinst}" · line 610
Quoted 6 lines
meta/classes/populate_sdk_ext.bbclass +2 -0
+			for e in $env_setup_scripts; do
+				echo "echo 'ERROR: this SDK was not fully installed and
needs reinstalling'" >> $e + done
meta/classes/populate_sdk_ext.bbclass +3 -0
+			exit 1
+		fi
+		rm -f $target_sdk_dir/ext-sdk-prepare.py
That last line is also unrelated.

Quoted 2 lines

The old code only considers one env_setup_script, but there are multiple ones now, so they are related.

Specifically in the last line you are doing rm -f. That is a change from rm previously. Again, there's nothing wrong with that, but it's not related to the multilib handling.

Thanks,
Paul

Signature · 2 lines

Paul Eggleton
Intel Open Source Technology Centre

Robert Yang

author

2016-11-17 06:19 UTC #6 lore raw

PATCH 2/8 oeqa/sdkext/devtool.py: remove workspace/sources before running test cases

In reply to Robert Yang

Fixed:
MACHINE = "qemux86-64"
require conf/multilib.conf
MULTILIBS = "multilib:lib32"
DEFAULTTUNE_virtclass-multilib-lib32 = "x86"

$ bitbake core-image-minimal -cpopulate_sdk_ext
[snip]
ERROR: Source tree path /path/to/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/testsdkext/tc/workspace/sources/v4l2loopback-driver already exists and is not empty\n'
[snip]

This is because the test case will run twice (environment-setup-core2-64-poky-linux and environment-setup-x86-pokymllib32-linux), it would fail in the second run, 'devtool reset' can not remove sources, so remove it before running test cases.

[YOCTO #10647]

Signed-off-by Robert Yang <[email protected]>

---
 meta/lib/oeqa/sdkext/devtool.py | 3 +++
 1 file changed, 3 insertions(+)

meta/lib/oeqa/sdkext/devtool.py +3 -0

index 65f41f6..f101eb6 100644
@@ -15,6 +15,9 @@ class DevtoolTest(oeSDKExtTest):
         self.myapp_cmake_dst = os.path.join(self.tc.sdktestdir, "myapp_cmake")
         shutil.copytree(self.myapp_cmake_src, self.myapp_cmake_dst)
 
+        # Clean sources dir to make "git clone" can run again
+        shutil.rmtree(os.path.join(self.tc.sdktestdir, "tc/workspace/sources"), True)
+
     def _test_devtool_build(self, directory):
         self._run('devtool add myapp %s' % directory)
         try:

Signature · 1 line

2.10.2

Paul Eggleton

2016-12-13 04:45 UTC #7 lore raw

Re: [PATCH 2/8] oeqa/sdkext/devtool.py: remove workspace/sources before running test cases

Quoted 36 lines · On Wed, 16 Nov 2016 22:19:31 Robert Yang wrote:

Fixed:
MACHINE = "qemux86-64"
require conf/multilib.conf
MULTILIBS = "multilib:lib32"
DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
$ bitbake core-image-minimal -cpopulate_sdk_ext
[snip]
ERROR: Source tree path
/path/to/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/testsdkex
t/tc/workspace/sources/v4l2loopback-driver already exists and is not
empty\n' [snip]
This is because the test case will run twice (environment-setup-core2-64-poky-linux and environment-setup-x86-pokymllib32-linux), it would fail in the second run, 'devtool reset' can not remove sources, so remove it before running test cases.
[YOCTO #10647]
Signed-off-by Robert Yang <[email protected]>
---
 meta/lib/oeqa/sdkext/devtool.py | 3 +++
 1 file changed, 3 insertions(+)
(quoted diff) +0 -0
b/meta/lib/oeqa/sdkext/devtool.py index 65f41f6..f101eb6 100644
meta/lib/oeqa/sdkext/devtool.py +2 -0
@@ -15,6 +15,9 @@ class DevtoolTest(oeSDKExtTest):
         self.myapp_cmake_dst = os.path.join(self.tc.sdktestdir,
 "myapp_cmake") shutil.copytree(self.myapp_cmake_src, self.myapp_cmake_dst)
 
+        # Clean sources dir to make "git clone" can run again
+        shutil.rmtree(os.path.join(self.tc.sdktestdir,

Quoted 4 lines

"tc/workspace/sources"), True) +
def _test_devtool_build(self, directory):
self._run('devtool add myapp %s' % directory) try:

It seems to me that's what's missing here is a proper teardown process like we have for oe-selftest, so that tests clean up after themselves whether they succeed or fail. I'm unsure as to whether that is part of the plan for the new QA refactoring though.

In the absence of that however I guess we don't have much choice but to do something like this.

Cheers,
Paul

Signature · 2 lines

Paul Eggleton
Intel Open Source Technology Centre

Robert Yang

author

2016-12-13 05:56 UTC #8 lore raw

Re: [PATCH 2/8] oeqa/sdkext/devtool.py: remove workspace/sources before running test cases

Hi Paul,

Thanks for reply, please see my comments inline.

Quoted 35 lines · Paul Eggleton

Quoted 34 lines · On Wed, 16 Nov 2016 22:19:31 Robert Yang wrote:
Fixed:
MACHINE = "qemux86-64"
require conf/multilib.conf
MULTILIBS = "multilib:lib32"
DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
$ bitbake core-image-minimal -cpopulate_sdk_ext
[snip]
ERROR: Source tree path
/path/to/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/testsdkex
t/tc/workspace/sources/v4l2loopback-driver already exists and is not
empty\n' [snip]
This is because the test case will run twice (environment-setup-core2-64-poky-linux and environment-setup-x86-pokymllib32-linux), it would fail in the second run, 'devtool reset' can not remove sources, so remove it before running test cases.
[YOCTO #10647]
Signed-off-by Robert Yang <[email protected]>
---
 meta/lib/oeqa/sdkext/devtool.py | 3 +++
 1 file changed, 3 insertions(+)
(quoted diff) +0 -0
b/meta/lib/oeqa/sdkext/devtool.py index 65f41f6..f101eb6 100644
meta/lib/oeqa/sdkext/devtool.py +0 -0
@@ -15,6 +15,9 @@ class DevtoolTest(oeSDKExtTest):
         self.myapp_cmake_dst = os.path.join(self.tc.sdktestdir,
 "myapp_cmake") shutil.copytree(self.myapp_cmake_src, self.myapp_cmake_dst)
 

Quoted 11 lines

Quoted 6 lines
meta/lib/oeqa/sdkext/devtool.py +2 -0
+        # Clean sources dir to make "git clone" can run again
+        shutil.rmtree(os.path.join(self.tc.sdktestdir,
"tc/workspace/sources"), True) +
def _test_devtool_build(self, directory):
self._run('devtool add myapp %s' % directory) try:
It seems to me that's what's missing here is a proper teardown process like we have for oe-selftest, so that tests clean up after themselves whether they succeed or fail. I'm unsure as to whether that is part of the plan for the new QA refactoring though.

There is already a 'devtool reset' which can do the cleanup, but it can't remove sources as I said in the commit log.

// Robert

Quoted 7 lines

In the absence of that however I guess we don't have much choice but to do something like this.
Cheers,
Paul

Paul Eggleton

2016-12-13 06:29 UTC #9 lore raw

Re: [PATCH 2/8] oeqa/sdkext/devtool.py: remove workspace/sources before running test cases

Quoted 52 lines · On Tue, 13 Dec 2016 13:56:05 Robert Yang wrote:

Hi Paul,
Thanks for reply, please see my comments inline.
Quoted 47 lines · Paul Eggleton
Quoted 45 lines · On Wed, 16 Nov 2016 22:19:31 Robert Yang wrote:
Fixed:
MACHINE = "qemux86-64"
require conf/multilib.conf
MULTILIBS = "multilib:lib32"
DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
$ bitbake core-image-minimal -cpopulate_sdk_ext
[snip]
ERROR: Source tree path
/path/to/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/testsdk
ex
t/tc/workspace/sources/v4l2loopback-driver already exists and is not
empty\n' [snip]
This is because the test case will run twice (environment-setup-core2-64-poky-linux and environment-setup-x86-pokymllib32-linux), it would fail in the second run, 'devtool reset' can not remove sources, so remove it before running test cases.
[YOCTO #10647]
Signed-off-by Robert Yang <[email protected]>
---
 meta/lib/oeqa/sdkext/devtool.py | 3 +++
 1 file changed, 3 insertions(+)
(quoted diff) +0 -0
b/meta/lib/oeqa/sdkext/devtool.py index 65f41f6..f101eb6 100644
meta/lib/oeqa/sdkext/devtool.py +2 -0
@@ -15,6 +15,9 @@ class DevtoolTest(oeSDKExtTest):
         self.myapp_cmake_dst = os.path.join(self.tc.sdktestdir,
 
 "myapp_cmake") shutil.copytree(self.myapp_cmake_src,
 self.myapp_cmake_dst)
 
+        # Clean sources dir to make "git clone" can run again
+        shutil.rmtree(os.path.join(self.tc.sdktestdir,
"tc/workspace/sources"), True) +
def _test_devtool_build(self, directory):
self._run('devtool add myapp %s' % directory)

Quoted 8 lines

Quoted 5 lines
Quoted 1 line
try:
It seems to me that's what's missing here is a proper teardown process like we have for oe-selftest, so that tests clean up after themselves whether they succeed or fail. I'm unsure as to whether that is part of the plan for the new QA refactoring though.
There is already a 'devtool reset' which can do the cleanup, but it can't remove sources as I said in the commit log.

devtool reset not deleting the source tree is intentional - I don't want people accidentally losing changes to their source. In any case though it's not about what we do here but where we do it.

Cheers,
Paul

Signature · 2 lines

Paul Eggleton
Intel Open Source Technology Centre

Robert Yang

author

2016-12-13 06:47 UTC #10 lore raw

Re: [PATCH 2/8] oeqa/sdkext/devtool.py: remove workspace/sources before running test cases

Quoted 59 lines · Paul Eggleton

Quoted 58 lines · On Tue, 13 Dec 2016 13:56:05 Robert Yang wrote:
Hi Paul,
Thanks for reply, please see my comments inline.
Quoted 52 lines · Paul Eggleton
Quoted 43 lines · On Wed, 16 Nov 2016 22:19:31 Robert Yang wrote:
Fixed:
MACHINE = "qemux86-64"
require conf/multilib.conf
MULTILIBS = "multilib:lib32"
DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
$ bitbake core-image-minimal -cpopulate_sdk_ext
[snip]
ERROR: Source tree path
/path/to/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/testsdk
ex
t/tc/workspace/sources/v4l2loopback-driver already exists and is not
empty\n' [snip]
This is because the test case will run twice (environment-setup-core2-64-poky-linux and environment-setup-x86-pokymllib32-linux), it would fail in the second run, 'devtool reset' can not remove sources, so remove it before running test cases.
[YOCTO #10647]
Signed-off-by Robert Yang <[email protected]>
---
 meta/lib/oeqa/sdkext/devtool.py | 3 +++
 1 file changed, 3 insertions(+)
(quoted diff) +0 -0
b/meta/lib/oeqa/sdkext/devtool.py index 65f41f6..f101eb6 100644
meta/lib/oeqa/sdkext/devtool.py +2 -0
@@ -15,6 +15,9 @@ class DevtoolTest(oeSDKExtTest):
         self.myapp_cmake_dst = os.path.join(self.tc.sdktestdir,
 
 "myapp_cmake") shutil.copytree(self.myapp_cmake_src,
 self.myapp_cmake_dst)
 
+        # Clean sources dir to make "git clone" can run again
+        shutil.rmtree(os.path.join(self.tc.sdktestdir,
"tc/workspace/sources"), True) +
Quoted 4 lines
def _test_devtool_build(self, directory):
self._run('devtool add myapp %s' % directory)
try:
It seems to me that's what's missing here is a proper teardown process like we have for oe-selftest, so that tests clean up after themselves whether they succeed or fail. I'm unsure as to whether that is part of the plan for the new QA refactoring though.

Quoted 4 lines

Quoted 2 lines
There is already a 'devtool reset' which can do the cleanup, but it can't remove sources as I said in the commit log.
devtool reset not deleting the source tree is intentional - I don't want

Add an option like "devtool reset --force" ? When I met the error, the first I did was check "devtool reset", but there is no way to remove resources, so I have to remove it here to allow multilib test cases runs. Or maybe fix "devtool add" to check the git repo correclty ? For example, when there is already a repo, just update it rather than clone it again.

The multilib + testsdkext would fail without this fix.

// Robert

Quoted 9 lines

people accidentally losing changes to their source. In any case though it's not about what we do here but where we do it.
Cheers,
Paul

Paul Eggleton

2016-12-13 08:21 UTC #11 lore raw

Re: [PATCH 2/8] oeqa/sdkext/devtool.py: remove workspace/sources before running test cases

Quoted 8 lines · On Tue, 13 Dec 2016 14:47:10 Robert Yang wrote:

Quoted 7 lines · Paul Eggleton
Quoted 6 lines · On Tue, 13 Dec 2016 13:56:05 Robert Yang wrote:
Quoted 4 lines · Paul Eggleton
It seems to me that's what's missing here is a proper teardown process like we have for oe-selftest, so that tests clean up after themselves whether they succeed or fail. I'm unsure as to whether that is part of the plan for the new QA refactoring though.

Quoted 8 lines

Quoted 4 lines
Quoted 2 lines
There is already a 'devtool reset' which can do the cleanup, but it can't remove sources as I said in the commit log.
devtool reset not deleting the source tree is intentional - I don't want
Add an option like "devtool reset --force" ? When I met the error, the first I did was check "devtool reset", but there is no way to remove resources, so I have to remove it here to allow multilib test cases runs.

I've considered this and decided against it - it just seems to me that at least one person will use it every time until one day when they wipe out their source with their changes in it. I'd really rather not be enabling that. Besides, in everyday use repeated runs of devtool add / modify on the same recipe should be at a minimum, if that's not the case then we have other issues to look at.

You say there's no way to remove these, but there is - just delete the files explicitly.

Quoted 2 lines

Or maybe fix "devtool add" to check the git repo correclty ? For example, when there is already a repo, just update it rather than clone it again.

Right, that has been suggested - and it's all fine if the source tree is unmodified and just a straight update to get from where it is currently to match the upstream. However, what if it's not fast-forward, or a different repo entirely? What if there are local modifications?

Quoted 1 line

The multilib + testsdkext would fail without this fix.

Yes, that's why I ultimately said we haven't much choice but to apply this patch. We must acknowledge however that the problem exists because the tests aren't cleaning up after themselves, which should be their responsibility.

Cheers,
Paul

Signature · 2 lines

Paul Eggleton
Intel Open Source Technology Centre

Robert Yang

author

2016-12-13 08:31 UTC #12 lore raw

Re: [PATCH 2/8] oeqa/sdkext/devtool.py: remove workspace/sources before running test cases

Quoted 36 lines · Paul Eggleton

Quoted 8 lines · On Tue, 13 Dec 2016 14:47:10 Robert Yang wrote:
Quoted 7 lines · Paul Eggleton
Quoted 6 lines · On Tue, 13 Dec 2016 13:56:05 Robert Yang wrote:
Quoted 4 lines · Paul Eggleton
It seems to me that's what's missing here is a proper teardown process like we have for oe-selftest, so that tests clean up after themselves whether they succeed or fail. I'm unsure as to whether that is part of the plan for the new QA refactoring though.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
There is already a 'devtool reset' which can do the cleanup, but it can't remove sources as I said in the commit log.
devtool reset not deleting the source tree is intentional - I don't want
Add an option like "devtool reset --force" ? When I met the error, the first I did was check "devtool reset", but there is no way to remove resources, so I have to remove it here to allow multilib test cases runs.
I've considered this and decided against it - it just seems to me that at least one person will use it every time until one day when they wipe out their source with their changes in it. I'd really rather not be enabling that. Besides, in everyday use repeated runs of devtool add / modify on the same recipe should be at a minimum, if that's not the case then we have other issues to look at.
You say there's no way to remove these, but there is - just delete the files explicitly.
Quoted 2 lines
Or maybe fix "devtool add" to check the git repo correclty ? For example, when there is already a repo, just update it rather than clone it again.
Right, that has been suggested - and it's all fine if the source tree is unmodified and just a straight update to get from where it is currently to match the upstream. However, what if it's not fast-forward, or a different repo entirely? What if there are local modifications?

Quoted 4 lines

Quoted 1 line
The multilib + testsdkext would fail without this fix.
Yes, that's why I ultimately said we haven't much choice but to apply this patch. We must acknowledge however that the problem exists because the tests

Got it, thanks.

// Robert

Quoted 5 lines

aren't cleaning up after themselves, which should be their responsibility.
Cheers,
Paul

Lopez, Mariano

2016-12-13 13:58 UTC #13 lore raw

Re: [PATCH 2/8] oeqa/sdkext/devtool.py: remove workspace/sources before running test cases

In reply to Paul Eggleton

Quoted 35 lines · Paul Eggleton

Quoted 34 lines · On Wed, 16 Nov 2016 22:19:31 Robert Yang wrote:
Fixed:
MACHINE = "qemux86-64"
require conf/multilib.conf
MULTILIBS = "multilib:lib32"
DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
$ bitbake core-image-minimal -cpopulate_sdk_ext
[snip]
ERROR: Source tree path
/path/to/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/testsdkex
t/tc/workspace/sources/v4l2loopback-driver already exists and is not
empty\n' [snip]
This is because the test case will run twice (environment-setup-core2-64-poky-linux and environment-setup-x86-pokymllib32-linux), it would fail in the second run, 'devtool reset' can not remove sources, so remove it before running test cases.
[YOCTO #10647]
Signed-off-by Robert Yang <[email protected]>
---
  meta/lib/oeqa/sdkext/devtool.py | 3 +++
  1 file changed, 3 insertions(+)
(quoted diff) +0 -0
b/meta/lib/oeqa/sdkext/devtool.py index 65f41f6..f101eb6 100644
meta/lib/oeqa/sdkext/devtool.py +0 -0
@@ -15,6 +15,9 @@ class DevtoolTest(oeSDKExtTest):
          self.myapp_cmake_dst = os.path.join(self.tc.sdktestdir,
 "myapp_cmake") shutil.copytree(self.myapp_cmake_src, self.myapp_cmake_dst)
 

Quoted 10 lines

Quoted 6 lines
meta/lib/oeqa/sdkext/devtool.py +2 -0
+        # Clean sources dir to make "git clone" can run again
+        shutil.rmtree(os.path.join(self.tc.sdktestdir,
"tc/workspace/sources"), True) +
def _test_devtool_build(self, directory):
self._run('devtool add myapp %s' % directory) try:
It seems to me that's what's missing here is a proper teardown process like we have for oe-selftest, so that tests clean up after themselves whether they succeed or fail. I'm unsure as to whether that is part of the plan for the new QA refactoring though.

To clean directories before/after the test it is not in the plans of the QA refactoring, they way Robert did the clean up is appropriated, in the setUpClass method, this way it will run before every class test and only one time.

Mariano

Quoted 7 lines

In the absence of that however I guess we don't have much choice but to do something like this.
Cheers,
Paul

Robert Yang

author

2016-11-17 06:19 UTC #14 lore raw

PATCH 3/8 oe-publish-sdk: make cmd easier to read

In reply to Robert Yang

The command was too long to read and maintain.

Signed-off-by Robert Yang <[email protected]>

---
 scripts/oe-publish-sdk | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)

scripts/oe-publish-sdk +17 -2

index 4fe8974..d95c623 100755
@@ -113,10 +113,25 @@ def publish(args):
             return ret
 
     # Setting up the git repo
+    cmd_common = "if [ ! -e .git ]; then"
+    cmd_common += "    git init .;"
+    cmd_common += "    mv .git/hooks/post-update.sample .git/hooks/post-update;"
+    cmd_common += "    echo '*.pyc\n*.pyo' > .gitignore;"
+    cmd_common += "fi;"
+    cmd_common += "git add -A .;"
+    cmd_common += "git config user.email '[email protected]' && git config user.name 'OE' && git commit -q -m 'init repo' || true;"
     if not is_remote:
-        cmd = 'set -e; mkdir -p %s/layers; cd %s/layers; if [ ! -e .git ]; then git init .; mv .git/hooks/post-update.sample .git/hooks/post-update; echo "*.pyc\n*.pyo" > .gitignore; fi; git add -A .; git config user.email "[email protected]" && git config user.name "OE" && git commit -q -m "init repo" || true; git update-server-info' % (destination, destination)
+        cmd = "set -e;"
+        cmd += "mkdir -p %s/layers;" % destination
+        cmd += "cd %s/layers;" % destination
+        cmd += cmd_common
+        cmd += "git update-server-info"
     else:
-        cmd = "ssh %s 'set -e; mkdir -p %s/layers; cd %s/layers; if [ ! -e .git ]; then git init .; mv .git/hooks/post-update.sample .git/hooks/post-update; echo '*.pyc\n*.pyo' > .gitignore; fi; git add -A .; git config user.email '[email protected]' && git config user.name 'OE' && git commit -q -m \"init repo\" || true; git update-server-info'" % (host, destdir, destdir)
+        cmd = "ssh %s 'set -e;" % host
+        cmd += "mkdir -p %s/layers;" % destdir
+        cmd += "cd %s/layers;" % destdir
+        cmd += cmd_common
+        cmd += "git update-server-info'"
     ret = subprocess.call(cmd, shell=True)
     if ret == 0:
         logger.info('SDK published successfully')

Signature · 1 line

2.10.2

Robert Yang

author

2016-11-17 06:19 UTC #15 lore raw

PATCH 4/8 oe-publish-sdk: add pyshtables.py to .gitignore

In reply to Robert Yang

Fixed:
MACHINE = "qemux86-64"
require conf/multilib.conf
MULTILIBS = "multilib:lib32"
DEFAULTTUNE_virtclass-multilib-lib32 = "x86"

$ bitbake core-image-minimal -cpopulate_sdk_ext
[snip]
ERROR: Failed to update metadata as there have been changes made to it. Aborting.\nERROR: Changed files:\nb' M poky/bitbake/lib/bb/pysh/pyshtables.py\\n'\n"
[snip]

This is because the test case will run twice (environment-setup-core2-64-poky-linux and environment-setup-x86-pokymllib32-linux), it would fail in the second run since pyshtables.py is regenerated in the first run. This file is generated automatically, publish it doesn't make any sense, so add it to .gitignore.

[YOCTO #10647]

Signed-off-by Robert Yang <[email protected]>

---
 scripts/oe-publish-sdk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

scripts/oe-publish-sdk +1 -1

index d95c623..e2b1b95 100755
@@ -116,7 +116,7 @@ def publish(args):
     cmd_common = "if [ ! -e .git ]; then"
     cmd_common += "    git init .;"
     cmd_common += "    mv .git/hooks/post-update.sample .git/hooks/post-update;"
-    cmd_common += "    echo '*.pyc\n*.pyo' > .gitignore;"
+    cmd_common += "    echo '*.pyc\n*.pyo\npyshtables.py' > .gitignore;"
     cmd_common += "fi;"
     cmd_common += "git add -A .;"
     cmd_common += "git config user.email '[email protected]' && git config user.name 'OE' && git commit -q -m 'init repo' || true;"

Signature · 1 line

2.10.2

Paul Eggleton

2016-12-13 04:59 UTC #16 lore raw

Re: [PATCH 4/8] oe-publish-sdk: add pyshtables.py to .gitignore

Quoted 12 lines · On Wed, 16 Nov 2016 22:19:33 Robert Yang wrote:

Fixed:
MACHINE = "qemux86-64"
require conf/multilib.conf
MULTILIBS = "multilib:lib32"
DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
$ bitbake core-image-minimal -cpopulate_sdk_ext
[snip]
ERROR: Failed to update metadata as there have been changes made to it.
Aborting.\nERROR: Changed files:\nb' M
poky/bitbake/lib/bb/pysh/pyshtables.py\\n'\n" [snip]

Quoted 8 lines

This is because the test case will run twice (environment-setup-core2-64-poky-linux and environment-setup-x86-pokymllib32-linux), it would fail in the second run since pyshtables.py is regenerated in the first run. This file is generated automatically, publish it doesn't make any sense, so add it to .gitignore.
[YOCTO #10647]

The actual fix looks OK but I don't think 10647 is really the right bug number. Coincidentally today someone else reported this issue, so this would probably be a better one:

https://bugzilla.yoctoproject.org/show_bug.cgi?id=10796

Quoted 16 lines

Signed-off-by Robert Yang <[email protected]>
---
 scripts/oe-publish-sdk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/oe-publish-sdk +0 -0
index d95c623..e2b1b95 100755
@@ -116,7 +116,7 @@ def publish(args):
     cmd_common = "if [ ! -e .git ]; then"
     cmd_common += "    git init .;"
     cmd_common += "    mv .git/hooks/post-update.sample
.git/hooks/post-update;" - cmd_common += " echo '*.pyc\n*.pyo' >
.gitignore;"
+ cmd_common += " echo '*.pyc\n*.pyo\npyshtables.py' > .gitignore;"

Quoted 4 lines

     cmd_common += "fi;"
     cmd_common += "git add -A .;"
     cmd_common += "git config user.email '[email protected]' && git config user.name

'OE' && git commit -q -m 'init repo' || true;"

Cheers,
Paul

Signature · 2 lines

Paul Eggleton
Intel Open Source Technology Centre

Robert Yang

author

2016-12-13 06:03 UTC #17 lore raw

Re: [PATCH 4/8] oe-publish-sdk: add pyshtables.py to .gitignore

Quoted 20 lines · Paul Eggleton

Quoted 19 lines · On Wed, 16 Nov 2016 22:19:33 Robert Yang wrote:
Fixed:
MACHINE = "qemux86-64"
require conf/multilib.conf
MULTILIBS = "multilib:lib32"
DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
$ bitbake core-image-minimal -cpopulate_sdk_ext
[snip]
ERROR: Failed to update metadata as there have been changes made to it.
Aborting.\nERROR: Changed files:\nb' M
poky/bitbake/lib/bb/pysh/pyshtables.py\\n'\n" [snip]
This is because the test case will run twice (environment-setup-core2-64-poky-linux and environment-setup-x86-pokymllib32-linux), it would fail in the second run since pyshtables.py is regenerated in the first run. This file is generated automatically, publish it doesn't make any sense, so add it to .gitignore.

Quoted 7 lines

Quoted 1 line
[YOCTO #10647]
The actual fix looks OK but I don't think 10647 is really the right bug number. Coincidentally today someone else reported this issue, so this would probably be a better one:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=10796

Thanks, I will update it.

// Robert

Quoted 25 lines

Quoted 17 lines
Signed-off-by Robert Yang <[email protected]>
---
 scripts/oe-publish-sdk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/oe-publish-sdk +0 -0
index d95c623..e2b1b95 100755
@@ -116,7 +116,7 @@ def publish(args):
     cmd_common = "if [ ! -e .git ]; then"
     cmd_common += "    git init .;"
     cmd_common += "    mv .git/hooks/post-update.sample
.git/hooks/post-update;" - cmd_common += " echo '*.pyc\n*.pyo' >
.gitignore;"
+ cmd_common += " echo '*.pyc\n*.pyo\npyshtables.py' > .gitignore;"
Quoted 4 lines
     cmd_common += "fi;"
     cmd_common += "git add -A .;"
     cmd_common += "git config user.email '[email protected]' && git config user.name
'OE' && git commit -q -m 'init repo' || true;"
Cheers,
Paul

Robert Yang

author

2016-11-17 06:19 UTC #18 lore raw

PATCH 5/8 oeqa/sdkext/devtool.py: don't reset when the test is failed

In reply to Robert Yang

The contents are helpful to debug when the error happens.

Signed-off-by Robert Yang <[email protected]>

---
 meta/lib/oeqa/sdkext/devtool.py | 5 -----
 1 file changed, 5 deletions(-)

meta/lib/oeqa/sdkext/devtool.py +0 -5

index f101eb6..c91090e 100644
@@ -24,7 +24,6 @@ class DevtoolTest(oeSDKExtTest):
             self._run('devtool build myapp')
         except Exception as e:
             print(e.output)
-            self._run('devtool reset myapp')
             raise e
         self._run('devtool reset myapp')

@@ -34,7 +33,6 @@ class DevtoolTest(oeSDKExtTest):
             self._run('devtool package myapp')
         except Exception as e:
             print(e.output)
-            self._run('devtool reset myapp')
             raise e
         self._run('devtool reset myapp')

@@ -73,7 +71,6 @@ class DevtoolTest(oeSDKExtTest):
             self._run('devtool build %s' % recipe)
         except Exception as e:
             print(e.output)
-            self._run('devtool reset %s' % recipe)
             raise e
         self._run('devtool reset %s' % recipe)

@@ -87,7 +84,6 @@ class DevtoolTest(oeSDKExtTest):
             self._run('devtool build %s' % recipe)
         except Exception as e:
             print(e.output)
-            self._run('devtool reset %s' % recipe)
             raise e
         self._run('devtool reset %s' % recipe)

@@ -100,7 +96,6 @@ class DevtoolTest(oeSDKExtTest):
             self._run('devtool build %s ' % package_nodejs)
         except Exception as e:
             print(e.output)
-            self._run('devtool reset %s' % package_nodejs)
             raise e
         self._run('devtool reset %s '% package_nodejs)

Signature · 1 line

2.10.2

Paul Eggleton

2016-12-13 04:48 UTC #19 lore raw

Re: [PATCH 5/8] oeqa/sdkext/devtool.py: don't reset when the test is failed

Quoted 1 line · On Wed, 16 Nov 2016 22:19:34 Robert Yang wrote:

The contents are helpful to debug when the error happens.

In this case removing this is OK because we're operating on an eSDK that the tests install. However, this wouldn't be appropriate for the devtool oe- selftest tests since they are operating on the user's build system itself - not that you probably have any intention of changing those, I just wanted to note that.

Cheers,
Paul

Signature · 2 lines

Paul Eggleton
Intel Open Source Technology Centre

Robert Yang

author

2016-12-13 06:01 UTC #20 lore raw

Re: [PATCH 5/8] oeqa/sdkext/devtool.py: don't reset when the test is failed

Quoted 8 lines · Paul Eggleton

Quoted 1 line · On Wed, 16 Nov 2016 22:19:34 Robert Yang wrote:
The contents are helpful to debug when the error happens.
In this case removing this is OK because we're operating on an eSDK that the tests install. However, this wouldn't be appropriate for the devtool oe- selftest tests since they are operating on the user's build system itself - not that you probably have any intention of changing those, I just wanted to note that.

Sorry, I don't quite understand, does selftest uses oeqa/sdkext/devtool.py, please ? Even if it uses devtool.py, I still think that we need keep the error contents for debugging when the error happens. It's very hard to debug when there is no error log or no workspace.

// Robert

Quoted 4 lines

Cheers,
Paul

Paul Eggleton

2016-12-13 06:33 UTC #21 lore raw

Re: [PATCH 5/8] oeqa/sdkext/devtool.py: don't reset when the test is failed

Quoted 4 lines · On Tue, 13 Dec 2016 14:01:08 Robert Yang wrote:

Quoted 3 lines · Paul Eggleton
Quoted 1 line · On Wed, 16 Nov 2016 22:19:34 Robert Yang wrote:
The contents are helpful to debug when the error happens.

Quoted 8 lines

Quoted 5 lines
In this case removing this is OK because we're operating on an eSDK that the tests install. However, this wouldn't be appropriate for the devtool oe- selftest tests since they are operating on the user's build system itself - not that you probably have any intention of changing those, I just wanted to note that.
Sorry, I don't quite understand, does selftest uses oeqa/sdkext/devtool.py, please ?

No, this is for bitbake -c testsdkext.

Quoted 3 lines

Even if it uses devtool.py, I still think that we need keep the error contents for debugging when the error happens. It's very hard to debug when there is no error log or no workspace.

I mean the equivalent oe-selftest tests in oeqa/selftest/devtool.py.

Actually, I've realised something that applies here as well - leaving the files around would be OK if we stopped on the first failure, but we don't - the next test proceeds after it. How will you be able to rely on what's in the workspace if several other tests have run afterwards - not to mention ensure those tests aren't disrupted by the leftover files?

Cheers,
Paul

Signature · 2 lines

Paul Eggleton
Intel Open Source Technology Centre

Robert Yang

author

2016-12-13 06:39 UTC #22 lore raw

Re: [PATCH 5/8] oeqa/sdkext/devtool.py: don't reset when the test is failed

Quoted 20 lines · Paul Eggleton

Quoted 4 lines · On Tue, 13 Dec 2016 14:01:08 Robert Yang wrote:
Quoted 3 lines · Paul Eggleton
Quoted 1 line · On Wed, 16 Nov 2016 22:19:34 Robert Yang wrote:
The contents are helpful to debug when the error happens.
Quoted 8 lines
Quoted 5 lines
In this case removing this is OK because we're operating on an eSDK that the tests install. However, this wouldn't be appropriate for the devtool oe- selftest tests since they are operating on the user's build system itself - not that you probably have any intention of changing those, I just wanted to note that.
Sorry, I don't quite understand, does selftest uses oeqa/sdkext/devtool.py, please ?
No, this is for bitbake -c testsdkext.
Quoted 3 lines
Even if it uses devtool.py, I still think that we need keep the error contents for debugging when the error happens. It's very hard to debug when there is no error log or no workspace.

Quoted 7 lines

I mean the equivalent oe-selftest tests in oeqa/selftest/devtool.py.
Actually, I've realised something that applies here as well - leaving the files around would be OK if we stopped on the first failure, but we don't - the next test proceeds after it. How will you be able to rely on what's in the workspace if several other tests have run afterwards - not to mention ensure those tests aren't disrupted by the leftover files?

That's a problem, but we really need consider debugging, otherwise it's painful when test are failed but nothing left. How about run test cases in different workspaces ?

// Robert

Quoted 4 lines

Cheers,
Paul

Paul Eggleton

2016-12-13 08:07 UTC #23 lore raw

Re: [PATCH 5/8] oeqa/sdkext/devtool.py: don't reset when the test is failed

Quoted 31 lines · On Tue, 13 Dec 2016 14:39:46 Robert Yang wrote:

Quoted 21 lines · Paul Eggleton
Quoted 10 lines · On Tue, 13 Dec 2016 14:01:08 Robert Yang wrote:
Quoted 8 lines · Paul Eggleton
Quoted 1 line · On Wed, 16 Nov 2016 22:19:34 Robert Yang wrote:
The contents are helpful to debug when the error happens.
In this case removing this is OK because we're operating on an eSDK that the tests install. However, this wouldn't be appropriate for the devtool oe- selftest tests since they are operating on the user's build system itself - not that you probably have any intention of changing those, I just wanted to note that.
Quoted 3 lines
Sorry, I don't quite understand, does selftest uses oeqa/sdkext/devtool.py,
please ?
No, this is for bitbake -c testsdkext.
Quoted 3 lines
Even if it uses devtool.py, I still think that we need keep the error contents for debugging when the error happens. It's very hard to debug when there is no error log or no workspace.
Quoted 8 lines
I mean the equivalent oe-selftest tests in oeqa/selftest/devtool.py.
Actually, I've realised something that applies here as well - leaving the files around would be OK if we stopped on the first failure, but we don't -
the next test proceeds after it. How will you be able to rely on what's in the workspace if several other tests have run afterwards - not to mention ensure those tests aren't disrupted by the leftover files?

Quoted 3 lines

That's a problem, but we really need consider debugging, otherwise it's painful when test are failed but nothing left. How about run test cases in different workspaces ?

Currently when I need to debug a test, I tend to either run the test commands manually, or comment out the cleanup. If we really want to improve debugging then we ought to do it properly - have proper teardown infrastructure and then provide a mode that disables it.

Cheers,
Paul

Signature · 2 lines

Paul Eggleton
Intel Open Source Technology Centre

Robert Yang

author

2016-12-13 08:09 UTC #24 lore raw

Re: [PATCH 5/8] oeqa/sdkext/devtool.py: don't reset when the test is failed

Quoted 32 lines · Paul Eggleton

Quoted 31 lines · On Tue, 13 Dec 2016 14:39:46 Robert Yang wrote:
Quoted 29 lines · Paul Eggleton
Quoted 10 lines · On Tue, 13 Dec 2016 14:01:08 Robert Yang wrote:
Quoted 8 lines · Paul Eggleton
Quoted 1 line · On Wed, 16 Nov 2016 22:19:34 Robert Yang wrote:
The contents are helpful to debug when the error happens.
In this case removing this is OK because we're operating on an eSDK that the tests install. However, this wouldn't be appropriate for the devtool oe- selftest tests since they are operating on the user's build system itself - not that you probably have any intention of changing those, I just wanted to note that.
Quoted 3 lines
Sorry, I don't quite understand, does selftest uses oeqa/sdkext/devtool.py,
please ?
No, this is for bitbake -c testsdkext.
Quoted 3 lines
Even if it uses devtool.py, I still think that we need keep the error contents for debugging when the error happens. It's very hard to debug when there is no error log or no workspace.
I mean the equivalent oe-selftest tests in oeqa/selftest/devtool.py.
Actually, I've realised something that applies here as well - leaving the files around would be OK if we stopped on the first failure, but we don't -
the next test proceeds after it. How will you be able to rely on what's in the workspace if several other tests have run afterwards - not to mention ensure those tests aren't disrupted by the leftover files?

Quoted 8 lines

Quoted 3 lines
That's a problem, but we really need consider debugging, otherwise it's painful when test are failed but nothing left. How about run test cases in different workspaces ?
Currently when I need to debug a test, I tend to either run the test commands manually, or comment out the cleanup. If we really want to improve debugging then we ought to do it properly - have proper teardown infrastructure and then provide a mode that disables it.

Thanks, I will drop the patch atm.

// Robert

Quoted 4 lines

Cheers,
Paul

Robert Yang

author

2016-11-17 06:19 UTC #25 lore raw

PATCH 6/8 oeqa/oetest.py: add hasLockedSig()

In reply to Robert Yang

It checks whether there is a "recipe:do_populate_sysroot:" in locked-sigs.inc, which will help to determine whether the testcase will run or not.

Signed-off-by Robert Yang <[email protected]>

---
 meta/lib/oeqa/oetest.py | 13 +++++++++++++
 1 file changed, 13 insertions(+)

meta/lib/oeqa/oetest.py +13 -0

index 95d3bf7..d12381d 100644
@@ -171,6 +171,12 @@ class oeSDKExtTest(oeSDKTest):
         return subprocess.check_output(". %s > /dev/null;"\
             " %s;" % (self.tc.sdkenv, cmd), stderr=subprocess.STDOUT, shell=True, env=env).decode("utf-8")
 
+    @classmethod
+    def hasLockedSig(self, recipe):
+        if re.search(" " + recipe + ":do_populate_sysroot:", oeTest.tc.locked_sigs):
+            return True
+        return False
+
 def getmodule(pos=2):
     # stack returns a list of tuples containg frame information
     # First element of the list the is current frame, caller is 1

@@ -708,6 +714,13 @@ class SDKExtTestContext(SDKTestContext):
         self.sdkextfilesdir = os.path.join(os.path.dirname(os.path.abspath(
             oeqa.sdkext.__file__)), "files")
 
+        self.locked_sig_file = os.path.join(self.sdktestdir, "tc/conf/locked-sigs.inc")
+        if os.path.exists(self.locked_sig_file):
+            with open(self.locked_sig_file) as f:
+                self.locked_sigs = f.read()
+        else:
+            bb.fatal("%s not found. Did you build the ext sdk image?\n%s" % e)
+
     def _get_test_namespace(self):
         if self.cm:
             return "sdk"

Signature · 1 line

2.10.2

Robert Yang

author

2016-11-17 06:19 UTC #26 lore raw

PATCH 7/8 oeqa/sdkext/devtool.py: skip test_extend_autotools_recipe_creation when no libxml2

In reply to Robert Yang

The librdfa requires libxml2 to build, otherwise, it would fail:

| configure: error: Package requirements (libxml-2.0 >= 2.6.26) were not met:
|
| No package 'libxml-2.0' found

Signed-off-by Robert Yang <[email protected]>

---
 meta/lib/oeqa/sdkext/devtool.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

meta/lib/oeqa/sdkext/devtool.py +4 -1

index c91090e..9ff69f9 100644
@@ -1,7 +1,7 @@
 import shutil
 import subprocess
 import urllib.request
-from oeqa.oetest import oeSDKExtTest
+from oeqa.oetest import oeSDKExtTest, skipModule
 from oeqa.utils.decorators import *
 
 class DevtoolTest(oeSDKExtTest):

@@ -64,6 +64,9 @@ class DevtoolTest(oeSDKExtTest):
     @testcase(1482)
     @skipUnlessPassed('test_devtool_location')
     def test_extend_autotools_recipe_creation(self):
+        # librdfa requires libxml2
+        if not oeSDKExtTest.hasLockedSig("libxml2"):
+            skipModule("No libxml2 package in the eSDK")
         req = 'https://github.com/rdfa/librdfa'
         recipe = "bbexample"
         self._run('devtool add %s %s' % (recipe, req) )

Signature · 1 line

2.10.2

Robert Yang

author

2016-11-17 06:19 UTC #27 lore raw

PATCH 8/8 oe/copy_buildsystem.py: add SDK_LAYERS_EXCLUDE_PATTERN

In reply to Robert Yang

It is helpful we want to exclude a lot of layers.
It uses python re, and supports multiple patterns (separated by space).

Signed-off-by Robert Yang <[email protected]>

---
 meta/lib/oe/copy_buildsystem.py | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

meta/lib/oe/copy_buildsystem.py +18 -0

index 29ac6d4..e5d00c7 100644
@@ -1,5 +1,12 @@
 # This class should provide easy access to the different aspects of the
 # buildsystem such as layers, bitbake location, etc.
+#
+# SDK_LAYERS_EXCLUDE: Layers which will be excluded from SDK layers.
+# SDK_LAYERS_EXCLUDE_PATTERN: The simiar to SDK_LAYERS_EXCLUDE, this supports
+#                             python regular expression, use space as separator,
+#                              e.g.: ".*-downloads closed-.*"
+#
+
 import stat
 import shutil

@@ -23,8 +30,10 @@ class BuildSystem(object):
         self.context = context
         self.layerdirs = [os.path.abspath(pth) for pth in d.getVar('BBLAYERS', True).split()]
         self.layers_exclude = (d.getVar('SDK_LAYERS_EXCLUDE', True) or "").split()
+        self.layers_exclude_pattern = d.getVar('SDK_LAYERS_EXCLUDE_PATTERN', True)
 
     def copy_bitbake_and_layers(self, destdir, workspace_name=None):
+        import re
         # Copy in all metadata layers + bitbake (as repositories)
         layers_copied = []
         bb.utils.mkdirhier(destdir)

@@ -36,8 +45,17 @@ class BuildSystem(object):
         # Exclude layers
         for layer_exclude in self.layers_exclude:
             if layer_exclude in layers:
+                bb.note('Excluded %s from sdk layers since it is in SDK_LAYERS_EXCLUDE' % layer_exclude)
                 layers.remove(layer_exclude)
 
+        if self.layers_exclude_pattern:
+            layers_cp = layers[:]
+            for pattern in self.layers_exclude_pattern.split():
+                for layer in layers_cp:
+                    if re.match(pattern, layer):
+                        bb.note('Excluded %s from sdk layers since matched SDK_LAYERS_EXCLUDE_PATTERN' % layer)
+                        layers.remove(layer)
+
         workspace_newname = workspace_name
         if workspace_newname:
             layernames = [os.path.basename(layer) for layer in layers]

Signature · 1 line

2.10.2

A message that is not in the archive (sent off-list or not yet received): [email protected]

Burton, Ross

2016-11-17 09:02 UTC #29 lore raw

Re: ✗ patchtest: failure for Fixes for eSDK and testsdkext

Quoted 6 lines · Patchwork

* Patch [7/8] oeqa/sdkext/devtool.py: skip
test_extend_autotools_recipe_creation when no libxml2
Issue Commit shortlog is too long [test_shortlog_length]
Suggested fix Edit shortlog so that it is 80 characters or less
(currently 82 characters)

I'm not sure we've ever been super-strict on this, so I'm wondering if warning over 90 is sensible so that we allow an extra two characters when function names are long but still detect when someone has put an essay in the first line (which does happen if the submitter isn't aware of the git convention).

Ross

Paul Eggleton

2016-11-17 17:12 UTC #30 lore raw

Re: ✗ patchtest: failure for Fixes for eSDK and testsdkext

Quoted 9 lines · Ross

On 17 November 2016 at 06:55, Patchwork <
Quoted 6 lines · [email protected]> wrote:
* Patch [7/8] oeqa/sdkext/devtool.py: skip
test_extend_autotools_recipe_creation when no libxml2
Issue Commit shortlog is too long [test_shortlog_length]
Suggested fix Edit shortlog so that it is 80 characters or less

Quoted 7 lines

Quoted 1 line
(currently 82 characters)
I'm not sure we've ever been super-strict on this, so I'm wondering if warning over 90 is sensible so that we allow an extra two characters when function names are long but still detect when someone has put an essay in the first line (which does happen if the submitter isn't aware of the git convention).

Yep I'd support that.

Cheers,
Paul

Signature · 2 lines

Paul Eggleton
Intel Open Source Technology Centre

Robert Yang

author

2016-11-29 07:38 UTC #31 lore raw

Re: ✗ patchtest: failure for Fixes for eSDK and testsdkext

In reply to Burton, Ross

Quoted 11 lines · Ross

On 17 November 2016 at 06:55, Patchwork <[email protected] <mailto:[email protected]>> wrote:
* Patch [7/8] oeqa/sdkext/devtool.py: skip
test_extend_autotools_recipe_creation when no libxml2
Issue Commit shortlog is too long [test_shortlog_length]
Suggested fix Edit shortlog so that it is 80 characters or less
(currently 82 characters)

Quoted 4 lines

I'm not sure we've ever been super-strict on this, so I'm wondering if warning over 90 is sensible so that we allow an extra two characters when function names are long but still detect when someone has put an essay in the first line (which does happen if the submitter isn't aware of the git convention).

Ping, do I have to make the subject line shorter and resend the patches, please ?

// Robert

Quoted 4 lines

Ross

Paul Eggleton

2016-11-29 09:10 UTC #32 lore raw

Re: ✗ patchtest: failure for Fixes for eSDK and testsdkext

Quoted 12 lines · On Tue, 29 Nov 2016 15:38:50 Robert Yang wrote:

Quoted 11 lines · Ross
On 17 November 2016 at 06:55, Patchwork <[email protected]> <mailto:[email protected]>> wrote:
* Patch [7/8] oeqa/sdkext/devtool.py: skip
test_extend_autotools_recipe_creation when no libxml2
Issue Commit shortlog is too long [test_shortlog_length]
Suggested fix Edit shortlog so that it is 80 characters or less
(currently 82 characters)

Quoted 8 lines

Quoted 5 lines
I'm not sure we've ever been super-strict on this, so I'm wondering if warning over 90 is sensible so that we allow an extra two characters when function names are long but still detect when someone has put an essay in the first line (which does happen if the submitter isn't aware of the git convention).
Ping, do I have to make the subject line shorter and resend the patches, please ?

No, no need to do that.

Cheers,
Paul

Signature · 2 lines

Paul Eggleton
Intel Open Source Technology Centre

Robert Yang

author

2016-12-13 02:58 UTC #33 lore raw

In reply to Robert Yang

Ping, any comments on this thread, please ?

// Robert

Quoted 29 lines · Robert Yang

The following changes since commit a675b2c89e477af088faee9b3be96eae19a85f0b:
sanity.bbclass: fix logging of an error (2016-11-15 15:18:50 +0000)
are available in the git repository at:
git://git.openembedded.org/openembedded-core-contrib rbt/eSDK
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=rbt/eSDK
Robert Yang (8):
populate_sdk_ext.bbclass: install multilib targets as populate_sdk
does
oeqa/sdkext/devtool.py: remove workspace/sources before running test
cases
oe-publish-sdk: make cmd easier to read
oe-publish-sdk: add pyshtables.py to .gitignore oeqa/sdkext/devtool.py: don't reset when the test is failed oeqa/oetest.py: add hasLockedSig() oeqa/sdkext/devtool.py: skip test_extend_autotools_recipe_creation
when no libxml2
oe/copy_buildsystem.py: add SDK_LAYERS_EXCLUDE_PATTERN
 meta/classes/populate_sdk_ext.bbclass | 61 ++++++++++++++++++++++-------------
 meta/lib/oe/copy_buildsystem.py       | 18 +++++++++++
 meta/lib/oeqa/oetest.py               | 13 ++++++++
 meta/lib/oeqa/sdkext/devtool.py       | 13 ++++----
 scripts/oe-publish-sdk                | 19 +++++++++--
 5 files changed, 93 insertions(+), 31 deletions(-)