Skip to contentSkip to conversation

oeqa: loader.py: show warning when skipping selected module and abort if all are skipped

PATCH New Martin Jansa 1 participant 2 messages Mar 13, 2023

lore
Share & tools
lore
#0 lore raw

PATCH 1/2 oeqa: loader.py: show warning when skipping selected module and abort if all are skipped

* skipped modules were triggering an ERROR before:

poky $ oe-selftest -v -r eSDK.oeSDKExtSelfTest.test_install_libraries_headers imagefeatures.ImageFeatures.test_image_gen_debugfs -K -B /OE/build/poky/build-eSDK

  2023-03-13 15:07:53,430 - oe-selftest - ERROR - Not found eSDK.oeSDKExtSelfTest.test_install_libraries_headers in loaded test cases

* but didn't show the reason why it wasn't loaded and more importantly -r was ignored when all selected modules were silently skipped

* add a warning when skipping some module and abort if some modules were selected, but all ended being skipped:

poky $ oe-selftest -v -r eSDK.oeSDKExtSelfTest.test_install_libraries_headers -K -B /OE/build/poky/build-eSDK

  2023-03-13 15:11:51,028 - oe-selftest - WARNING - module 'eSDK.oeSDKExtSelfTest.test_install_libraries_headers' was skipped from selected modules, because it doesn't match with module name assumptions: package and module names do not contain upper case characters, whereas class names do
  2023-03-13 15:11:51,028 - oe-selftest - ERROR - All selected modules were skipped, this would trigger selftest with all tests and -r ignored.

* I was hit by this in oe-selftest -r eSDK.oeSDKExtSelfTest.test_install_libraries_headers which is skipped due to upper case characters in module name and selftest started to run all tests (archiver.Archiver.test_archiver_allows_to_filter_on_recipe_name as first from 529)

poky $ oe-selftest -v -r eSDK.oeSDKExtSelfTest.test_install_libraries_headers -K -B /OE/build/poky/build-eSDK

  2023-03-13 14:00:52,955 - oe-selftest - DEBUG - Selected tests with -r: ['eSDK.oeSDKExtSelfTest.test_install_libraries_headers']
  2023-03-13 14:00:55,531 - oe-selftest - INFO - Changing cwd to /OE/build/poky/build
  ..
  2023-03-13 14:00:58,128 - oe-selftest - INFO - test_archiver_allows_to_filter_on_recipe_name (archiver.Archiver.test_archiver_allows_to_filter_on_recipe_name)

I'll rename eSDK to esdk in next commit to avoid this.

* also fix small typo in context I've noticed when debugging this

---
 meta/lib/oeqa/core/context.py | 2 +-
 meta/lib/oeqa/core/loader.py  | 9 +++++++--
 2 files changed, 8 insertions(+), 3 deletions(-)
meta/lib/oeqa/core/context.py +1 -1
index 2abe353d27..9313271f58 100644
@@ -81,7 +81,7 @@ class OETestContext(object):
     def runTests(self, processes=None, skips=[]):
         self.runner = self.runnerClass(self, descriptions=False, verbosity=2)
 
-        # Dinamically skip those tests specified though arguments
+        # Dynamically skip those tests specified though arguments
         self.skipTests(skips)
 
         self._run_start_time = time.time()
meta/lib/oeqa/core/loader.py +7 -2
index 11978213b8..f25b5970e9 100644
@@ -37,7 +37,7 @@ def _find_duplicated_modules(suite, directory):
         if path:
             raise ImportError("Duplicated %s module found in %s" % (module, path))
 
-def _built_modules_dict(modules):
+def _built_modules_dict(modules, logger):
     modules_dict = {}
 
     if modules == None:
@@ -48,6 +48,9 @@ def _built_modules_dict(modules):
         # characters, whereas class names do
         m = re.match(r'^([0-9a-z_.]+)(?:\.(\w[^.]*)(?:\.([^.]+))?)?$', module, flags=re.ASCII)
         if not m:
+            logger.warn("module '%s' was skipped from selected modules, "\
+                "because it doesn't match with module name assumptions: "\
+                "package and module names do not contain upper case characters, whereas class names do" % module)
             continue
 
         module_name, class_name, test_name = m.groups()
@@ -58,6 +61,8 @@ def _built_modules_dict(modules):
             modules_dict[module_name][class_name] = []
         if test_name and test_name not in modules_dict[module_name][class_name]:
             modules_dict[module_name][class_name].append(test_name)
+    if modules and not modules_dict:
+        raise OEQATestNotFound("All selected modules were skipped, this would trigger selftest with all tests and -r ignored.")
 
     return modules_dict
 
@@ -71,7 +76,7 @@ class OETestLoader(unittest.TestLoader):
             *args, **kwargs):
         self.tc = tc
 
-        self.modules = _built_modules_dict(modules)
+        self.modules = _built_modules_dict(modules, tc.logger)
 
         self.tests = tests
         self.modules_required = modules_required
Signature · 1 line
2.39.2
#1 lore raw

PATCH 2/2 selftest: eSDK rename to esdk

* I was hit by oe-selftest -r eSDK.oeSDKExtSelfTest.test_install_libraries_headers running all tests except only this selected one:

poky $ oe-selftest -v -r eSDK.oeSDKExtSelfTest.test_install_libraries_headers -K -B /OE/build/poky/build-eSDK

  2023-03-13 14:00:52,955 - oe-selftest - DEBUG - Selected tests with -r: ['eSDK.oeSDKExtSelfTest.test_install_libraries_headers']
  2023-03-13 14:00:55,531 - oe-selftest - INFO - Changing cwd to /OE/build/poky/build
  ..
  2023-03-13 14:00:58,128 - oe-selftest - INFO - test_archiver_allows_to_filter_on_recipe_name (archiver.Archiver.test_archiver_allows_to_filter_on_recipe_name)

this is caused by _built_modules_dict(modules) function which filters out eSDK.oeSDKExtSelfTest.test_install_libraries_headers based on the regexp and then it runs all loaded tests, because modules are empty

the initial regexp and comment from 2017:
https://git.openembedded.org/openembedded-core/commit/?id=80db3d999ae26d298d9d5418a32b11a4f27af9d5

# Assumption: package and module names do not contain upper case # characters, whereas class names do
m = re.match(r'^([^A-Z]+)(?:\.([A-Z][^.]*)(?:\.([^.]+))?)?$', module)

might still be valid, but it was loosened in 2018 to accept upper case in module:

https://git.openembedded.org/openembedded-core/commit/?id=1ecf48fd286a77078451b67879a44f9c9dc7a894
Some test cases (eSDK.oeSDK*, runtime_test/*) does not match with current regex, fix it accept all.

Then skipping the not matching modules was added later in 2018:

     https://git.openembedded.org/openembedded-core/commit/?id=f2042bf3638ed4edfb167e7f7d4be6da60997ead

and regexp was updated again in 2020 not to accept upper case in modules:

https://git.openembedded.org/openembedded-core/commit/?id=ad81ea90a815389e45ff302a85151724c71f71c3
oeqa/core/loader: refine regex to find module
test case in format <module name>.<class name>.<test case name> this is clear when test cases is only 3 item deep.

but confused when it is 4 item deep, eg,
oelib.types.TestList.test_list_nosep

I'm afraid that changing this regexp again to accept eSDK will break someone's favorite test case, renaming eSDK looks much safer option

There is only 1 such case in poky:

  $ oe-selftest --list-modules | grep INFO.- | sed 's/^.*INFO - //g' | grep -v '^[a-z_\.]*$'
  Listing all available test modules:
  eSDK

Most modules are just a-z (52x), then oelib.<foo> (6x) and 7 modules with underscore '_'.

---
 meta/lib/oeqa/selftest/cases/{eSDK.py => esdk.py} | 0
 1 file changed, 0 insertions(+), 0 deletions(-)
 rename meta/lib/oeqa/selftest/cases/{eSDK.py => esdk.py} (100%)
meta/lib/oeqa/selftest/cases/esdk.py renamed from meta/lib/oeqa/selftest/cases/eSDK.py +0 -0
similarity index 100%
rename from meta/lib/oeqa/selftest/cases/eSDK.py
rename to meta/lib/oeqa/selftest/cases/esdk.py
Signature · 1 line
2.39.2