populate_sdk_ext.bbclass: set METADATA_REVISION with an DISTRO override
PATCH 1/3 populate_sdk_ext.bbclass: set METADATA_REVISION with an DISTRO override
* otherwise it ends '<unknown>' inside esdk, because of parsing order:
# $METADATA_REVISION [3 operations]
# set /OE/build/test-D/conf/local.conf:43
# "f2da54ef432eac89b0f18eaad68e602b6990b5de"
# immediate /OE/build/test-D/layers/poky/meta/classes/metadata_scm.bbclass:9
# "${@oe.buildcfg.detect_revision(d)}"
# set /OE/build/test-D/layers/poky/meta/classes/metadata_scm.bbclass:10
# [vardepvalue] "${METADATA_REVISION}"
# pre-expansion value: # "<unknown>" METADATA_REVISION="<unknown>"
* This causes base-files.do_install and following tasks to have different signatures between esdk and the build directory where this esdk was created:
bitbake-diffsigs {test-D,poky/build-uninative-disabled}/tmp/stamps/qemux86_64-poky-linux/base-files/*do_install*sigdata*
NOTE: Starting bitbake server...
basehash changed from 5b6981cf58bfd57d416b0e31611b73a26baae635dd1ac31c08d46f95064c3ffc to dbdce042da4d7813d632b6d1cc87a16f728ad20e55fecbc392830e6acf72babd Variable METADATA_REVISION value changed from '<unknown>' to 'f2da54ef432eac89b0f18eaad68e602b6990b5de'
and an warning from "python3 /OE/build/test-D/ext-sdk-prepare.py" when eSDK is being prepared for use:
WARNING: The base-files:do_install sig is computed to be 83b9c9a6ef1145baac5a1e0d08814b9156af239c58fc42df95c25a9cd8a7f201,
but the sig is locked to 3dc22233059075978e5503691e98e79e7cc60db94259dfcd886bca2291c0add7 in SIGGEN_LOCKEDSIGS_t-qemux86-64
--- meta/classes-recipe/populate_sdk_ext.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes-recipe/populate_sdk_ext.bbclass +1 -1
@@ -376,7 +376,7 @@ python copy_buildsystem () { f.write('BUILDCFG_HEADER = ""\n\n') # Write METADATA_REVISION - f.write('METADATA_REVISION = "%s"\n\n' % d.getVar('METADATA_REVISION')) + f.write('METADATA_REVISION:%s = "%s"\n\n' % (d.getVar('DISTRO'), d.getVar('METADATA_REVISION'))) f.write('# Provide a flag to indicate we are in the EXT_SDK Context\n') f.write('WITHIN_EXT_SDK = "1"\n\n')
Signature · 1 line
2.40.0
PATCH 2/3 populate_sdk_ext.bbclass: redirect stderr to stdout so that both end in LOGFILE
* this in the end doesn't help much, I was debugging warning (about base-files.do_install signature being different than expected) from:
python3 $target_sdk_dir/ext-sdk-prepare.py $LOGFILE '${SDK_INSTALL_TARGETS}'this shows the warning on console, but it doesn't end in $LOGFILE, because it writes only contents of cooker log into the $LOGFILE with:
with open(logfile, 'a') as logf:
logf.write('Preparing SDK for %s...\n' % ', '.join(sdk_targets))
ret = run_command_interruptible('BB_SETSCENE_ENFORCE=1 bitbake --quiet %s' % ' '.join(sdk_targets)) if not ret:
ret = run_command_interruptible('bitbake --quiet build-sysroots')
lastlog = get_last_consolelog()
if lastlog:
with open(lastlog, 'r') as f:
for line in f:
logf.write(line)
if ret:
print('ERROR: SDK preparation failed: error log written to %s' % logfile) return ret
maybe we could remove whole support for $LOGFILE parameter and just redirect the output like other commands on this line
--- meta/classes-recipe/populate_sdk_ext.bbclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/classes-recipe/populate_sdk_ext.bbclass +1 -1
@@ -739,7 +739,7 @@ sdk_ext_postinst() { # current working directory when first ran, nor will it set $1 when # sourcing a script. That is why this has to look so ugly. LOGFILE="$target_sdk_dir/preparing_build_system.log" - sh -c ". buildtools/environment-setup* > $LOGFILE && cd $target_sdk_dir/`dirname ${oe_init_build_env_path}` && set $target_sdk_dir && . $target_sdk_dir/${oe_init_build_env_path} $target_sdk_dir >> $LOGFILE && python3 $target_sdk_dir/ext-sdk-prepare.py $LOGFILE '${SDK_INSTALL_TARGETS}'" || { echo "printf 'ERROR: this SDK was not fully installed and needs reinstalling\n'" >> $env_setup_script ; exit 1 ; } + sh -c ". buildtools/environment-setup* > $LOGFILE 2>&1 && cd $target_sdk_dir/`dirname ${oe_init_build_env_path}` && set $target_sdk_dir && . $target_sdk_dir/${oe_init_build_env_path} $target_sdk_dir >> $LOGFILE 2>&1 && python3 $target_sdk_dir/ext-sdk-prepare.py $LOGFILE '${SDK_INSTALL_TARGETS}'" || { echo "printf 'ERROR: this SDK was not fully installed and needs reinstalling\n'" >> $env_setup_script ; exit 1 ; } fi if [ -e $target_sdk_dir/ext-sdk-prepare.py ]; then rm $target_sdk_dir/ext-sdk-prepare.py
Signature · 1 line
2.40.0
PATCH 3/3 oeqa: print stderr in the AssertionError as well
In reply to Martin Jansa--- meta/lib/oeqa/utils/commands.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/lib/oeqa/utils/commands.py +1 -1
@@ -209,7 +209,7 @@ def runCmd(command, ignore_status=False, timeout=None, assert_error=True, sync=T exc_output = "\n... (last %d lines of output)\n" % limit_exc_output + \ '\n'.join(split[-limit_exc_output:]) if assert_error: - raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, exc_output)) + raise AssertionError("Command '%s' returned non-zero exit status %d:\nstderr: %s\nstdout:\n%s" % (command, result.status, result.error, exc_output)) else: raise CommandError(result.status, command, exc_output)
Signature · 1 line
2.40.0
Re: [OE-core] [PATCH 3/3] oeqa: print stderr in the AssertionError as well
Hello Martin,
Quoted 10 lines · On Tue, 11 Apr 2023 11:03:48 +0200 "Martin Jansa" <[email protected]>…
Signed-off-by Martin Jansa <[email protected]>--- meta/lib/oeqa/utils/commands.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)meta/lib/oeqa/utils/commands.py +0
-0
meta/lib/oeqa/utils/commands.py def runCmd(command, ignore_status=False, timeout=None, assert_error=True, sync=T · line 212
Quoted 5 lines
meta/lib/oeqa/utils/commands.py +1
-1@@ -209,7 +209,7 @@ def runCmd(command, ignore_status=False, timeout=None, assert_error=True, sync=T exc_output = "\n... (last %d lines of output)\n" % limit_exc_output + \ '\n'.join(split[-limit_exc_output:]) if assert_error: - raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, exc_output)) + raise AssertionError("Command '%s' returned non-zero exit status %d:\nstderr: %s\nstdout:\n%s" % (command, result.status, result.error, exc_output))
Testing with this series produces errors on the selftest builds:
---------------------------------8<---------------------------------
AssertionError: Command 'echo foobar >&2; false' returned non-zero exit status 1:
stderr:
stdout:
foobar
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/runcmd.py", line 61, in test_result_assertion
self.assertRaisesRegexp(AssertionError, "Command 'echo .* false' returned non-zero exit status 1:\nfoobar",
File "/usr/lib/python3.9/unittest/case.py", line 1329, in deprecated_func
return original_func(*args, **kwargs)
File "/usr/lib/python3.9/unittest/case.py", line 1276, in assertRaisesRegex
return context.handle('assertRaisesRegex', args, kwargs)
File "/usr/lib/python3.9/unittest/case.py", line 201, in handle
callable_obj(*args, **kwargs)
File "/usr/lib/python3.9/unittest/case.py", line 239, in __exit__
self._raiseFailure('"{}" does not match "{}"'.format( File "/usr/lib/python3.9/unittest/case.py", line 163, in _raiseFailure
raise self.test_case.failureException(msg)
AssertionError: "Command 'echo .* false' returned non-zero exit status 1:
foobar" does not match "Command 'echo foobar >&2; false' returned non-zero exit status 1:
stderr:
stdout:
foobar"
---------------------------------8<---------------------------------
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5002/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5044/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5053/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5083/steps/15/logs/stdio
Signature · 3 lines
Luca Ceresoli, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Re: [OE-core] [PATCH 3/3] oeqa: print stderr in the AssertionError as well
Hello Luca,
I'm sorry, I've updated the runcmd.RunCmdTests.test_result_assertion test case to pass with this change just to realize that:
"stderr": subprocess.STDOUT
in self.defaultopts does work and even the stderr output is ending in stdout, so this change isn't very useful (as I don't see any runcmd users passing different options than defaultopts).
Please drop 3/3 other 2 changes are independent on it.
I have one more change in this area which is the real fix for esdk tests with uninative disabled, but that has some weird side-effect which is triggering:
NOTE: Running task 3807 of 3809 (/OE/build/poky/meta/recipes-core/images/core-image-minimal.bb:do_populate_lic_deploy)
ERROR: When reparsing /OE/build/poky/meta/recipes-core/images/core-image-minimal.bb:do_populate_sdk_ext, the basehash value changed from 8249821d6529e111a2f7f1101632fe140639df54a700e2c604a99ea150541a6d to 86086c51c409799a0d4d2a4fcac9e7e245cb36d86f236dcf7093e428a7c37ac9. The metadata is not deterministic and this needs to be fixed.
ERROR: The following commands may help:
ERROR: $ bitbake core-image-minimal -cdo_populate_sdk_ext -Snone
ERROR: Then:
ERROR: $ bitbake core-image-minimal -cdo_populate_sdk_ext -Sprintdiff
which I haven't resolved yet, so it might be a while before I send this 4th change.
Regards,
Quoted 80 lines · Luca Ceresoli
Hello Martin,
Review
Quoted 6 lines · On Tue, 11 Apr 2023 11:03:48 +0200 "Martin Jansa" <[email protected]>…
Signed-off-by Martin Jansa <[email protected]>--- meta/lib/oeqa/utils/commands.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)(quoted diff) +0
-0
b/meta/lib/oeqa/utils/commands.pymeta/lib/oeqa/utils/commands.py def runCmd(command, ignore_status=False,
Quoted 4 lines
index
c1f533802e..7b2f6ec124100644meta/lib/oeqa/utils/commands.py +0
-0@@ -209,7 +209,7 @@ def runCmd(command, ignore_status=False,timeout=None, assert_error=True, sync=T
meta/lib/oeqa/utils/commands.py def runCmd(command, ignore_status=False, · line 209
Quoted 1 line
meta/lib/oeqa/utils/commands.py +0
-0exc_output = "\n... (last %d lines of output)\n" %
limit_exc_output+ \meta/lib/oeqa/utils/commands.py def runCmd(command, ignore_status=False, timeout=None, assert_error=True, sync=T
Quoted 3 lines
meta/lib/oeqa/utils/commands.py +0
-1'\n'.join(split[-limit_exc_output:]) if assert_error: - raise AssertionError("Command '%s' returned non-zero exitstatus %d:\n%s" % (command, result.status,
exc_output))meta/lib/oeqa/utils/commands.py def runCmd(command, ignore_status=False, · line 212
Quoted 1 line
meta/lib/oeqa/utils/commands.py +1
-0+ raise AssertionError("Command '%s' returned non-zero exitstatus %d:\nstderr: %s\nstdout:\n%s" % (command, result.status, result.error,
exc_output))Testing with this series produces errors on the selftest builds:
---------------------------------8<---------------------------------
AssertionError: Command 'echo foobar >&2; false' returned non-zero exit status 1:
stderr:
stdout:
foobarDuring handling of the above exception, another exception occurred:
Traceback (most recent call last):
File
"/home/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/runcmd.py", line 61, intest_result_assertion
self.assertRaisesRegexp(AssertionError, "Command 'echo .* false'
returned non-zero exit status 1:\nfoobar",
File "/usr/lib/python3.9/unittest/case.py", line 1329, indeprecated_func
returnoriginal_func(*args, **kwargs)
File "/usr/lib/python3.9/unittest/case.py", line 1276, in
assertRaisesRegex
returncontext.handle('assertRaisesRegex', args, kwargs)
File "/usr/lib/python3.9/unittest/case.py", line 201, in handle
callable_obj(*args, **kwargs)
File "/usr/lib/python3.9/unittest/case.py", line 239, in__exit__self._raiseFailure('"{}" does not match "{}"'.format(File "
/usr/lib/python3.9/unittest/case.py", line 163, in_raiseFailure
raiseself.test_case.failureException(msg)
AssertionError: "Command 'echo .* false' returned non-zero exit status 1:
foobar" does not match "Command 'echo foobar >&2; false' returned non-zero exit status 1:
stderr:
stdout:
foobar"---------------------------------8<---------------------------------
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5002/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5044/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5053/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5083/steps/15/logs/stdio
--
Luca Ceresoli, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com