rootfs-postcommands: Set vardeps for write_image_test_data
PATCH v2 1/1 rootfs-postcommands: Set vardeps for write_image_test_data
From: Peter Hoyes <[email protected]>
The testdata.json file generated as part of the rootfs postprocess commands currently contains almost all Bitbake variables and is used by OEQA test cases to inspect the build environment. However only a small number of variables are actually used and the testdata.json is not automatically updated when the variables are updated.
Introduce the TESTIMAGE_EXPORT_VARS variable to explicitly define the variables to be collected for testdata.json and populate with all variables used by runtime tests in OE-core. Use this variable to set the vardeps of write_image_test_data.
Modify export2json to take an explicit list of keys to export, instead of exporting everything in the datastore.
Add a demonstrative OE selftest to rootfspostcommandstests.
--- .../rootfs-postcommands.bbclass | 28 ++++++++++++++++++- meta/lib/oe/data.py | 19 ++----------- .../selftest/cases/rootfspostcommandstests.py | 19 +++++++++++++ 3 files changed, 48 insertions(+), 18 deletions(-)
meta/classes-recipe/rootfs-postcommands.bbclass +27 -1
@@ -453,6 +453,30 @@ rootfs_sysroot_relativelinks () { sysroot-relativelinks.py ${SDK_OUTPUT}/${SDKTARGETSYSROOT} } +TESTIMAGE_EXPORT_VARS ?= "\ + ARCH \ + DEPLOY_DIR \ + DEPLOY_DIR_DEB \ + DISTRO_FEATURES \ + DL_DIR \ + IMAGE \ + IMAGE_FEATURES \ + libdir \ + MACHINE \ + MULTILIB_VARIANTS \ + PACKAGE_FEED_GPG_NAME \ + PTEST_EXPECT_FAILURE \ + QEMU_USE_KVM \ + SDK_DEPLOY \ + SDKPATH \ + T \ + TEST_LOG_DIR \ + TOOLCHAINEXT_OUTPUTNAME \ + TUNE_PKGARCH \ + VIRTUAL-RUNTIME_init_manager \ + WORKDIR \ +" + # Generated test data json file python write_image_test_data() { from oe.data import export2json
@@ -462,7 +486,8 @@ python write_image_test_data() { testdata_name = os.path.join(deploy_dir, "%s.testdata.json" % d.getVar('IMAGE_NAME')) searchString = "%s/"%(d.getVar("TOPDIR")).replace("//","/") - export2json(d, testdata_name, searchString=searchString, replaceString="") + exportkeys = d.getVar('TESTIMAGE_EXPORT_VARS').split() + export2json(d, exportkeys, testdata_name, searchString=searchString, replaceString="") if os.path.exists(testdata_name) and link_name: testdata_link = os.path.join(deploy_dir, "%s.testdata.json" % link_name)
@@ -471,6 +496,7 @@ python write_image_test_data() { os.remove(testdata_link) os.symlink(os.path.basename(testdata_name), testdata_link) } +write_image_test_data[vardeps] += "${TESTIMAGE_EXPORT_VARS}" write_image_test_data[vardepsexclude] += "TOPDIR" # Check for unsatisfied recommendations (RRECOMMENDS)
meta/lib/oe/data.py +2 -17
@@ -23,25 +23,10 @@ def typed_value(key, d): except (TypeError, ValueError) as exc: bb.msg.fatal("Data", "%s: %s" % (key, str(exc))) -def export2json(d, json_file, expand=True, searchString="",replaceString=""): +def export2json(d, keys, json_file, expand=True, searchString="",replaceString=""): data2export = {} - keys2export = [] - for key in d.keys(): - if key.startswith("_"): - continue - elif key.startswith("BB"): - continue - elif key.startswith("B_pn"): - continue - elif key.startswith("do_"): - continue - elif d.getVarFlag(key, "func"): - continue - - keys2export.append(key) - - for key in keys2export: + for key in keys: try: data2export[key] = d.getVar(key, expand).replace(searchString,replaceString) except bb.data_smart.ExpansionError:
meta/lib/oeqa/selftest/cases/rootfspostcommandstests.py +19 -0
@@ -2,6 +2,7 @@ # # SPDX-License-Identifier: MIT +import json import os import oe import unittest
@@ -95,3 +96,21 @@ class ShadowUtilsTidyFiles(OESelftestTestCase): unsorted.append(file) if (unsorted): raise Exception("The following files were not sorted by ID as expected: %s" % unsorted) + + +class TestDataTests(OESelftestTestCase): + def test_vardeps(self): + """ + Test that variables changes are reflected in testdata.json + """ + test_image = "core-image-minimal" + self.write_config('TESTIMAGE_EXPORT_VARS:append = " TEST_VARIABLE"\nTEST_VARIABLE = "VALUE1"') + bitbake(test_image) + self.append_config('TEST_VARIABLE = "VALUE2"') + bitbake(test_image) + + vars = get_bb_vars(('DEPLOY_DIR_IMAGE', 'IMAGE_LINK_NAME'), test_image) + testdata_json = "%s/%s.testdata.json" % (vars['DEPLOY_DIR_IMAGE'], vars['IMAGE_LINK_NAME']) + with open(testdata_json, 'r') as tf: + testdata_vars = json.load(tf) + self.assertEqual(testdata_vars['TEST_VARIABLE'], 'VALUE2')
Signature · 1 line
2.34.1
Hello Peter,
Quoted 13 lines · On Thu, 22 Jun 2023 17:34:13 +0100 "Peter Hoyes" <[email protected]> wrote:
From: Peter Hoyes <[email protected]>
The testdata.json file generated as part of the rootfs postprocess commands currently contains almost all Bitbake variables and is used by OEQA test cases to inspect the build environment. However only a small number of variables are actually used and the testdata.json is not automatically updated when the variables are updated.
Introduce the
TESTIMAGE_EXPORT_VARSvariable to explicitly define the variables to be collected for testdata.json and populate with all variables used by runtime tests in OE-core. Use this variable to set the vardeps ofwrite_image_test_data.
Quoted 6 lines
Modify export2json to take an explicit list of keys to export, instead of exporting everything in the datastore.
Add a demonstrative OE selftest to rootfspostcommandstests.
Signed-off-by Peter Hoyes <[email protected]>
This patch is triggering some failures on the autobuilders:
Exception: TypeError: export2json() missing 1 required positional argument: 'json_file'
Some logs:
https://autobuilder.yoctoproject.org/typhoon/#/builders/20/builds/7744/steps/11/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/20/builds/7744/steps/16/logs/stdio
Luca
Signature · 3 lines
Luca Ceresoli, Bootlin Embedded Linux and Kernel engineering https://bootlin.com