Skip to contentSkip to conversation

oeqa/runtime/parselogs improvements

PATCH Superseded [email protected] 2 participants 10 messages Sep 21, 2023

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover oeqa/runtime/parselogs improvements

From: Ross Burton <[email protected]>

This is part one of my parselogs series, which clean up the code and refactor slightly in preparation for a more exciting part two which I shall send shortly.

Ross Burton (8):
oeqa/runtime/parselogs: remove unused imports oeqa/runtime/parselogs: don't bother to show target hardware
information
oeqa/runtime/parselogs: remove obsolete LSB testing support oeqa/runtime/parselogs: inline single-caller functions oeqa/runtime/parselogs: improve find call oeqa/runtime/parselogs: don't pass around members oeqa/runtime/parselogs: move some variables out of global scope oeqa/runtime/parselogs: select the correct machine-specific ignores
early

 meta/lib/oeqa/runtime/cases/parselogs.py | 109 +++++++----------------
 1 file changed, 33 insertions(+), 76 deletions(-)
Signature · 1 line
2.34.1
#1 lore raw

PATCH 1/8 oeqa/runtime/parselogs: remove unused imports

From: Ross Burton <[email protected]>

---
 meta/lib/oeqa/runtime/cases/parselogs.py | 2 --
 1 file changed, 2 deletions(-)
meta/lib/oeqa/runtime/cases/parselogs.py +0 -2
index e67d3750dad..8fd5a86e141 100644
@@ -10,8 +10,6 @@ from subprocess import check_output
 from shutil import rmtree
 from oeqa.runtime.case import OERuntimeTestCase
 from oeqa.core.decorator.depends import OETestDepends
-from oeqa.core.decorator.data import skipIfDataVar
-from oeqa.runtime.decorator.package import OEHasPackage
 
 #in the future these lists could be moved outside of module
 errors = ["error", "cannot", "can\'t", "failed"]
Signature · 1 line
2.34.1
#2 lore raw

PATCH 2/8 oeqa/runtime/parselogs: don't bother to show target hardware information

In reply to [email protected]

From: Ross Burton <[email protected]>

This information is pretty useless as we know what the target is from the build configuration (be it a qemu machine or real hardware).

---
 meta/lib/oeqa/runtime/cases/parselogs.py | 29 ------------------------
 1 file changed, 29 deletions(-)
meta/lib/oeqa/runtime/cases/parselogs.py +0 -29
index 8fd5a86e141..e62ab2c44ec 100644
@@ -237,34 +237,6 @@ class ParseLogsTest(OERuntimeTestCase):
     def getWorkdir(self):
         return self.td.get('WORKDIR', '')
 
-    # Get some information on the CPU of the machine to display at the
-    # beginning of the output. This info might be useful in some cases.
-    def getHardwareInfo(self):
-        hwi = ""
-        cmd = ('cat /proc/cpuinfo | grep "model name" | head -n1 | '
-               " awk 'BEGIN{FS=\":\"}{print $2}'")
-        _, cpu_name = self.target.run(cmd)
-
-        cmd = ('cat /proc/cpuinfo | grep "cpu cores" | head -n1 | '
-               "awk {'print $4'}")
-        _, cpu_physical_cores = self.target.run(cmd)
-
-        cmd = 'cat /proc/cpuinfo | grep "processor" | wc -l'
-        _, cpu_logical_cores = self.target.run(cmd)
-
-        _, cpu_arch = self.target.run('uname -m')
-
-        hwi += 'Machine information: \n'
-        hwi += '*******************************\n'
-        hwi += 'Machine name: ' + self.getMachine() + '\n'
-        hwi += 'CPU: ' + str(cpu_name) + '\n'
-        hwi += 'Arch: ' + str(cpu_arch)+ '\n'
-        hwi += 'Physical cores: ' + str(cpu_physical_cores) + '\n'
-        hwi += 'Logical cores: ' + str(cpu_logical_cores) + '\n'
-        hwi += '*******************************\n'
-
-        return hwi
-
     # Go through the log locations provided and if it's a folder
     # create a list with all the .log files in it, if it's a file
     # just add it to that list.
@@ -376,7 +348,6 @@ class ParseLogsTest(OERuntimeTestCase):
         self.write_dmesg()
         log_list = self.get_local_log_list(self.log_locations)
         result = self.parse_logs(self.errors, self.ignore_errors, log_list)
-        print(self.getHardwareInfo())
         errcount = 0
         for log in result:
             self.msg += 'Log: ' + log + '\n'
Signature · 1 line
2.34.1
#3 lore raw

PATCH 3/8 oeqa/runtime/parselogs: remove obsolete LSB testing support

In reply to [email protected]

From: Ross Burton <[email protected]>

The LSB compliance tests were removed in 2019[1], so this is obsolete.

[1] oe-core fb064356

---
 meta/lib/oeqa/runtime/cases/parselogs.py | 8 --------
 1 file changed, 8 deletions(-)
meta/lib/oeqa/runtime/cases/parselogs.py +0 -8
index e62ab2c44ec..e670df3cedd 100644
@@ -69,9 +69,6 @@ common_errors = [
     "Failed to read LoaderEntryOneShot variable, ignoring: Operation not supported",
     ]
 
-video_related = [
-]
-
 x86_common = [
     '[drm:psb_do_init] *ERROR* Debug is',
     'wrong ELF class',
@@ -225,11 +222,6 @@ class ParseLogsTest(OERuntimeTestCase):
         cls.ignore_errors = ignore_errors
         cls.log_locations = log_locations
         cls.msg = ''
-        is_lsb, _ = cls.tc.target.run("which LSB_Test.sh")
-        if is_lsb == 0:
-            for machine in cls.ignore_errors:
-                cls.ignore_errors[machine] = cls.ignore_errors[machine] \
-                                             + video_related
 
     def getMachine(self):
         return self.td.get('MACHINE', '')
Signature · 1 line
2.34.1
#4 lore raw

PATCH 4/8 oeqa/runtime/parselogs: inline single-caller functions

In reply to [email protected]

From: Ross Burton <[email protected]>

There's no need to have one-liner functions to get the MACHINE or WORKDIR when they're only called once.

---
 meta/lib/oeqa/runtime/cases/parselogs.py | 10 ++--------
 1 file changed, 2 insertions(+), 8 deletions(-)
meta/lib/oeqa/runtime/cases/parselogs.py +2 -8
index e670df3cedd..6e5dc753060 100644
@@ -223,12 +223,6 @@ class ParseLogsTest(OERuntimeTestCase):
         cls.log_locations = log_locations
         cls.msg = ''
 
-    def getMachine(self):
-        return self.td.get('MACHINE', '')
-
-    def getWorkdir(self):
-        return self.td.get('WORKDIR', '')
-
     # Go through the log locations provided and if it's a folder
     # create a list with all the .log files in it, if it's a file
     # just add it to that list.
@@ -251,7 +245,7 @@ class ParseLogsTest(OERuntimeTestCase):
 
     # Copy the log files to be parsed locally
     def transfer_logs(self, log_list):
-        workdir = self.getWorkdir()
+        workdir = self.td.get('WORKDIR')
         self.target_logs = workdir + '/' + 'target_logs'
         target_logs = self.target_logs
         if os.path.exists(target_logs):
@@ -278,7 +272,7 @@ class ParseLogsTest(OERuntimeTestCase):
         grepcmd += '" ' + str(log) + " | grep -Eiv \'"
 
         try:
-            errorlist = ignore_errors[self.getMachine()]
+            errorlist = ignore_errors[self.td.get('MACHINE')]
         except KeyError:
             self.msg += 'No ignore list found for this machine, using default\n'
             errorlist = ignore_errors['default']
Signature · 1 line
2.34.1
#5 lore raw

PATCH 5/8 oeqa/runtime/parselogs: improve find call

In reply to [email protected]

From: Ross Burton <[email protected]>

getLogList() uses remote find invocations to find the logs. Instead of relying on shell expansion of wildcards and redundant use of -maxdepth (pointless as the shell expansion means the find is passed the files to return), invoke find idiomatically by telling it what directory to search for and escape the glob so find processes it.

Also remove many pointless str() calls.

---
 meta/lib/oeqa/runtime/cases/parselogs.py | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
meta/lib/oeqa/runtime/cases/parselogs.py +5 -5
index 6e5dc753060..93782b844b9 100644
@@ -229,18 +229,18 @@ class ParseLogsTest(OERuntimeTestCase):
     def getLogList(self, log_locations):
         logs = []
         for location in log_locations:
-            status, _ = self.target.run('test -f ' + str(location))
+            status, _ = self.target.run('test -f %s' % location)
             if status == 0:
-                logs.append(str(location))
+                logs.append(location)
             else:
-                status, _ = self.target.run('test -d ' + str(location))
+                status, _ = self.target.run('test -d %s' % location)
                 if status == 0:
-                    cmd = 'find ' + str(location) + '/*.log -maxdepth 1 -type f'
+                    cmd = 'find %s -name \\*.log -maxdepth 1 -type f' % location
                     status, output = self.target.run(cmd)
                     if status == 0:
                         output = output.splitlines()
                         for logfile in output:
-                            logs.append(os.path.join(location, str(logfile)))
+                            logs.append(os.path.join(location, logfile))
         return logs
 
     # Copy the log files to be parsed locally
Signature · 1 line
2.34.1
#6 lore raw

PATCH 6/8 oeqa/runtime/parselogs: don't pass around members

In reply to [email protected]

From: Ross Burton <[email protected]>

There's no point in passing around member fields, just access them directly.

---
 meta/lib/oeqa/runtime/cases/parselogs.py | 24 ++++++++++++++----------
 1 file changed, 14 insertions(+), 10 deletions(-)
meta/lib/oeqa/runtime/cases/parselogs.py +14 -10
index 93782b844b9..0262f574d1a 100644
@@ -263,19 +263,19 @@ class ParseLogsTest(OERuntimeTestCase):
         return logs
 
     # Build the grep command to be used with filters and exclusions
-    def build_grepcmd(self, errors, ignore_errors, log):
+    def build_grepcmd(self, log):
         grepcmd = 'grep '
         grepcmd += '-Ei "'
-        for error in errors:
+        for error in self.errors:
             grepcmd += r'\<' + error + r'\>' + '|'
         grepcmd = grepcmd[:-1]
         grepcmd += '" ' + str(log) + " | grep -Eiv \'"
 
         try:
-            errorlist = ignore_errors[self.td.get('MACHINE')]
+            errorlist = self.ignore_errors[self.td.get('MACHINE')]
         except KeyError:
             self.msg += 'No ignore list found for this machine, using default\n'
-            errorlist = ignore_errors['default']
+            errorlist = self.ignore_errors['default']
 
         for ignore_error in errorlist:
             ignore_error = ignore_error.replace('(', r'\(')
@@ -292,17 +292,21 @@ class ParseLogsTest(OERuntimeTestCase):
 
         return grepcmd
 
-    # Grep only the errors so that their context could be collected.
-    # Default context is 10 lines before and after the error itself
-    def parse_logs(self, errors, ignore_errors, logs,
-                   lines_before = 10, lines_after = 10):
+    def parse_logs(self, logs, lines_before=10, lines_after=10):
+        """
+        Search the log files @logs looking for error lines (marked by
+        @self.errors), ignoring anything listed in @self.ignore_errors.
+
+        Returns a dictionary of log filenames to a dictionary of error lines to
+        the error context (controlled by @lines_before and @lines_after).
+        """
         results = {}
         rez = []
         grep_output = ''
 
         for log in logs:
             result = None
-            thegrep = self.build_grepcmd(errors, ignore_errors, log)
+            thegrep = self.build_grepcmd(log)
 
             try:
                 result = check_output(thegrep, shell=True).decode('utf-8')
@@ -333,7 +337,7 @@ class ParseLogsTest(OERuntimeTestCase):
     def test_parselogs(self):
         self.write_dmesg()
         log_list = self.get_local_log_list(self.log_locations)
-        result = self.parse_logs(self.errors, self.ignore_errors, log_list)
+        result = self.parse_logs(log_list)
         errcount = 0
         for log in result:
             self.msg += 'Log: ' + log + '\n'
Signature · 1 line
2.34.1
#7 lore raw

PATCH 7/8 oeqa/runtime/parselogs: move some variables out of global scope

In reply to [email protected]

From: Ross Burton <[email protected]>

errors and log_locations can be trivially set in the class directly, instead of being defined in the module and then copied into the class.

---
 meta/lib/oeqa/runtime/cases/parselogs.py | 19 +++++++++----------
 1 file changed, 9 insertions(+), 10 deletions(-)
meta/lib/oeqa/runtime/cases/parselogs.py +9 -10
index 0262f574d1a..3f205661ea9 100644
@@ -11,9 +11,6 @@ from shutil import rmtree
 from oeqa.runtime.case import OERuntimeTestCase
 from oeqa.core.decorator.depends import OETestDepends
 
-#in the future these lists could be moved outside of module
-errors = ["error", "cannot", "can\'t", "failed"]
-
 common_errors = [
     "(WW) warning, (EE) error, (NI) not implemented, (??) unknown.",
     "dma timeout",
@@ -201,17 +198,19 @@ ignore_errors = {
         ] + common_errors,
 }
 
-log_locations = ["/var/log/","/var/log/dmesg", "/tmp/dmesg_output.log"]
-
 class ParseLogsTest(OERuntimeTestCase):
 
+    # Which log files should be collected
+    log_locations = ["/var/log/", "/var/log/dmesg", "/tmp/dmesg_output.log"]
+
+    # The keywords that identify error messages in the log files
+    errors = ["error", "cannot", "can't", "failed"]
+
     @classmethod
     def setUpClass(cls):
-        cls.errors = errors
-
         # When systemd is enabled we need to notice errors on
         # circular dependencies in units.
-        if 'systemd' in cls.td.get('DISTRO_FEATURES', ''):
+        if 'systemd' in cls.td.get('DISTRO_FEATURES'):
             cls.errors.extend([
                 'Found ordering cycle on',
                 'Breaking ordering cycle by deleting job',
@@ -220,8 +219,6 @@ class ParseLogsTest(OERuntimeTestCase):
                 ])
 
         cls.ignore_errors = ignore_errors
-        cls.log_locations = log_locations
-        cls.msg = ''
 
     # Go through the log locations provided and if it's a folder
     # create a list with all the .log files in it, if it's a file
@@ -338,7 +335,9 @@ class ParseLogsTest(OERuntimeTestCase):
         self.write_dmesg()
         log_list = self.get_local_log_list(self.log_locations)
         result = self.parse_logs(log_list)
+
         errcount = 0
+        self.msg = ""
         for log in result:
             self.msg += 'Log: ' + log + '\n'
             self.msg += '-----------------------\n'
Signature · 1 line
2.34.1
#8 lore raw

PATCH 8/8 oeqa/runtime/parselogs: select the correct machine-specific ignores early

In reply to [email protected]

From: Ross Burton <[email protected]>

This has no impact to the execution, but makes the following changes neater.

---
 meta/lib/oeqa/runtime/cases/parselogs.py | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)
meta/lib/oeqa/runtime/cases/parselogs.py +6 -7
index 3f205661ea9..c889be8f6ea 100644
@@ -218,7 +218,11 @@ class ParseLogsTest(OERuntimeTestCase):
                 'Ordering cycle found, skipping',
                 ])
 
-        cls.ignore_errors = ignore_errors
+        try:
+            cls.ignore_errors = ignore_errors[cls.td.get('MACHINE')]
+        except KeyError:
+            cls.logger.info('No ignore list found for this machine, using default')
+            cls.ignore_errors = cls.ignore_errors['default']
 
     # Go through the log locations provided and if it's a folder
     # create a list with all the .log files in it, if it's a file
@@ -268,13 +272,8 @@ class ParseLogsTest(OERuntimeTestCase):
         grepcmd = grepcmd[:-1]
         grepcmd += '" ' + str(log) + " | grep -Eiv \'"
 
-        try:
-            errorlist = self.ignore_errors[self.td.get('MACHINE')]
-        except KeyError:
-            self.msg += 'No ignore list found for this machine, using default\n'
-            errorlist = self.ignore_errors['default']
 
-        for ignore_error in errorlist:
+        for ignore_error in self.ignore_errors:
             ignore_error = ignore_error.replace('(', r'\(')
             ignore_error = ignore_error.replace(')', r'\)')
             ignore_error = ignore_error.replace("'", '.')
Signature · 1 line
2.34.1
#9 lore raw
In reply to [email protected]
Quoted 18 lines · Ross Burton

From: Ross Burton <[email protected]>

This is part one of my parselogs series, which clean up the code and refactor slightly in preparation for a more exciting part two which I shall send shortly.

Ross Burton (8):
oeqa/runtime/parselogs: remove unused imports oeqa/runtime/parselogs: don't bother to show target hardware
information
oeqa/runtime/parselogs: remove obsolete LSB testing support oeqa/runtime/parselogs: inline single-caller functions oeqa/runtime/parselogs: improve find call oeqa/runtime/parselogs: don't pass around members oeqa/runtime/parselogs: move some variables out of global scope oeqa/runtime/parselogs: select the correct machine-specific ignores
early

Quoted 2 lines
 meta/lib/oeqa/runtime/cases/parselogs.py | 109 +++++++----------------
 1 file changed, 33 insertions(+), 76 deletions(-)

This might be too exciting already!

https://autobuilder.yoctoproject.org/typhoon/#/builders/47/builds/7818
https://autobuilder.yoctoproject.org/typhoon/#/builders/110/builds/6710

I've noted the helpful error messages, presumably from setupClass :(

Cheers,

Richard