oeqa/runtime/parselogs improvements
cover oeqa/runtime/parselogs improvements
From: Ross Burton <[email protected]>
This is part one of my parselogs series, which clean up the code and refactor slightly in preparation for a more exciting part two which I shall send shortly.
Ross Burton (8):
oeqa/runtime/parselogs: remove unused imports oeqa/runtime/parselogs: don't bother to show target hardware
information
oeqa/runtime/parselogs: remove obsolete LSB testing support oeqa/runtime/parselogs: inline single-caller functions oeqa/runtime/parselogs: improve find call oeqa/runtime/parselogs: don't pass around members oeqa/runtime/parselogs: move some variables out of global scope oeqa/runtime/parselogs: select the correct machine-specific ignores
early
meta/lib/oeqa/runtime/cases/parselogs.py | 109 +++++++---------------- 1 file changed, 33 insertions(+), 76 deletions(-)
Signature · 1 line
2.34.1
PATCH 1/8 oeqa/runtime/parselogs: remove unused imports
From: Ross Burton <[email protected]>
--- meta/lib/oeqa/runtime/cases/parselogs.py | 2 -- 1 file changed, 2 deletions(-)
meta/lib/oeqa/runtime/cases/parselogs.py +0 -2
@@ -10,8 +10,6 @@ from subprocess import check_output from shutil import rmtree from oeqa.runtime.case import OERuntimeTestCase from oeqa.core.decorator.depends import OETestDepends -from oeqa.core.decorator.data import skipIfDataVar -from oeqa.runtime.decorator.package import OEHasPackage #in the future these lists could be moved outside of module errors = ["error", "cannot", "can\'t", "failed"]
Signature · 1 line
2.34.1
PATCH 2/8 oeqa/runtime/parselogs: don't bother to show target hardware information
In reply to [email protected]From: Ross Burton <[email protected]>
This information is pretty useless as we know what the target is from the build configuration (be it a qemu machine or real hardware).
--- meta/lib/oeqa/runtime/cases/parselogs.py | 29 ------------------------ 1 file changed, 29 deletions(-)
meta/lib/oeqa/runtime/cases/parselogs.py +0 -29
@@ -237,34 +237,6 @@ class ParseLogsTest(OERuntimeTestCase): def getWorkdir(self): return self.td.get('WORKDIR', '') - # Get some information on the CPU of the machine to display at the - # beginning of the output. This info might be useful in some cases. - def getHardwareInfo(self): - hwi = "" - cmd = ('cat /proc/cpuinfo | grep "model name" | head -n1 | ' - " awk 'BEGIN{FS=\":\"}{print $2}'") - _, cpu_name = self.target.run(cmd) - - cmd = ('cat /proc/cpuinfo | grep "cpu cores" | head -n1 | ' - "awk {'print $4'}") - _, cpu_physical_cores = self.target.run(cmd) - - cmd = 'cat /proc/cpuinfo | grep "processor" | wc -l' - _, cpu_logical_cores = self.target.run(cmd) - - _, cpu_arch = self.target.run('uname -m') - - hwi += 'Machine information: \n' - hwi += '*******************************\n' - hwi += 'Machine name: ' + self.getMachine() + '\n' - hwi += 'CPU: ' + str(cpu_name) + '\n' - hwi += 'Arch: ' + str(cpu_arch)+ '\n' - hwi += 'Physical cores: ' + str(cpu_physical_cores) + '\n' - hwi += 'Logical cores: ' + str(cpu_logical_cores) + '\n' - hwi += '*******************************\n' - - return hwi - # Go through the log locations provided and if it's a folder # create a list with all the .log files in it, if it's a file # just add it to that list.
@@ -376,7 +348,6 @@ class ParseLogsTest(OERuntimeTestCase): self.write_dmesg() log_list = self.get_local_log_list(self.log_locations) result = self.parse_logs(self.errors, self.ignore_errors, log_list) - print(self.getHardwareInfo()) errcount = 0 for log in result: self.msg += 'Log: ' + log + '\n'
Signature · 1 line
2.34.1
PATCH 3/8 oeqa/runtime/parselogs: remove obsolete LSB testing support
In reply to [email protected]From: Ross Burton <[email protected]>
The LSB compliance tests were removed in 2019[1], so this is obsolete.
[1] oe-core fb064356
--- meta/lib/oeqa/runtime/cases/parselogs.py | 8 -------- 1 file changed, 8 deletions(-)
meta/lib/oeqa/runtime/cases/parselogs.py +0 -8
@@ -69,9 +69,6 @@ common_errors = [ "Failed to read LoaderEntryOneShot variable, ignoring: Operation not supported", ] -video_related = [ -] - x86_common = [ '[drm:psb_do_init] *ERROR* Debug is', 'wrong ELF class',
@@ -225,11 +222,6 @@ class ParseLogsTest(OERuntimeTestCase): cls.ignore_errors = ignore_errors cls.log_locations = log_locations cls.msg = '' - is_lsb, _ = cls.tc.target.run("which LSB_Test.sh") - if is_lsb == 0: - for machine in cls.ignore_errors: - cls.ignore_errors[machine] = cls.ignore_errors[machine] \ - + video_related def getMachine(self): return self.td.get('MACHINE', '')
Signature · 1 line
2.34.1
PATCH 4/8 oeqa/runtime/parselogs: inline single-caller functions
In reply to [email protected]From: Ross Burton <[email protected]>
There's no need to have one-liner functions to get the MACHINE or WORKDIR when they're only called once.
--- meta/lib/oeqa/runtime/cases/parselogs.py | 10 ++-------- 1 file changed, 2 insertions(+), 8 deletions(-)
meta/lib/oeqa/runtime/cases/parselogs.py +2 -8
@@ -223,12 +223,6 @@ class ParseLogsTest(OERuntimeTestCase): cls.log_locations = log_locations cls.msg = '' - def getMachine(self): - return self.td.get('MACHINE', '') - - def getWorkdir(self): - return self.td.get('WORKDIR', '') - # Go through the log locations provided and if it's a folder # create a list with all the .log files in it, if it's a file # just add it to that list.
@@ -251,7 +245,7 @@ class ParseLogsTest(OERuntimeTestCase): # Copy the log files to be parsed locally def transfer_logs(self, log_list): - workdir = self.getWorkdir() + workdir = self.td.get('WORKDIR') self.target_logs = workdir + '/' + 'target_logs' target_logs = self.target_logs if os.path.exists(target_logs):
@@ -278,7 +272,7 @@ class ParseLogsTest(OERuntimeTestCase): grepcmd += '" ' + str(log) + " | grep -Eiv \'" try: - errorlist = ignore_errors[self.getMachine()] + errorlist = ignore_errors[self.td.get('MACHINE')] except KeyError: self.msg += 'No ignore list found for this machine, using default\n' errorlist = ignore_errors['default']
Signature · 1 line
2.34.1
PATCH 5/8 oeqa/runtime/parselogs: improve find call
In reply to [email protected]From: Ross Burton <[email protected]>
getLogList() uses remote find invocations to find the logs. Instead of relying on shell expansion of wildcards and redundant use of -maxdepth (pointless as the shell expansion means the find is passed the files to return), invoke find idiomatically by telling it what directory to search for and escape the glob so find processes it.
Also remove many pointless str() calls.
--- meta/lib/oeqa/runtime/cases/parselogs.py | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
meta/lib/oeqa/runtime/cases/parselogs.py +5 -5
@@ -229,18 +229,18 @@ class ParseLogsTest(OERuntimeTestCase): def getLogList(self, log_locations): logs = [] for location in log_locations: - status, _ = self.target.run('test -f ' + str(location)) + status, _ = self.target.run('test -f %s' % location) if status == 0: - logs.append(str(location)) + logs.append(location) else: - status, _ = self.target.run('test -d ' + str(location)) + status, _ = self.target.run('test -d %s' % location) if status == 0: - cmd = 'find ' + str(location) + '/*.log -maxdepth 1 -type f' + cmd = 'find %s -name \\*.log -maxdepth 1 -type f' % location status, output = self.target.run(cmd) if status == 0: output = output.splitlines() for logfile in output: - logs.append(os.path.join(location, str(logfile))) + logs.append(os.path.join(location, logfile)) return logs # Copy the log files to be parsed locally
Signature · 1 line
2.34.1
PATCH 6/8 oeqa/runtime/parselogs: don't pass around members
In reply to [email protected]From: Ross Burton <[email protected]>
There's no point in passing around member fields, just access them directly.
--- meta/lib/oeqa/runtime/cases/parselogs.py | 24 ++++++++++++++---------- 1 file changed, 14 insertions(+), 10 deletions(-)
meta/lib/oeqa/runtime/cases/parselogs.py +14 -10
@@ -263,19 +263,19 @@ class ParseLogsTest(OERuntimeTestCase): return logs # Build the grep command to be used with filters and exclusions - def build_grepcmd(self, errors, ignore_errors, log): + def build_grepcmd(self, log): grepcmd = 'grep ' grepcmd += '-Ei "' - for error in errors: + for error in self.errors: grepcmd += r'\<' + error + r'\>' + '|' grepcmd = grepcmd[:-1] grepcmd += '" ' + str(log) + " | grep -Eiv \'" try: - errorlist = ignore_errors[self.td.get('MACHINE')] + errorlist = self.ignore_errors[self.td.get('MACHINE')] except KeyError: self.msg += 'No ignore list found for this machine, using default\n' - errorlist = ignore_errors['default'] + errorlist = self.ignore_errors['default'] for ignore_error in errorlist: ignore_error = ignore_error.replace('(', r'\(')
@@ -292,17 +292,21 @@ class ParseLogsTest(OERuntimeTestCase): return grepcmd - # Grep only the errors so that their context could be collected. - # Default context is 10 lines before and after the error itself - def parse_logs(self, errors, ignore_errors, logs, - lines_before = 10, lines_after = 10): + def parse_logs(self, logs, lines_before=10, lines_after=10): + """ + Search the log files @logs looking for error lines (marked by + @self.errors), ignoring anything listed in @self.ignore_errors. + + Returns a dictionary of log filenames to a dictionary of error lines to + the error context (controlled by @lines_before and @lines_after). + """ results = {} rez = [] grep_output = '' for log in logs: result = None - thegrep = self.build_grepcmd(errors, ignore_errors, log) + thegrep = self.build_grepcmd(log) try: result = check_output(thegrep, shell=True).decode('utf-8')
@@ -333,7 +337,7 @@ class ParseLogsTest(OERuntimeTestCase): def test_parselogs(self): self.write_dmesg() log_list = self.get_local_log_list(self.log_locations) - result = self.parse_logs(self.errors, self.ignore_errors, log_list) + result = self.parse_logs(log_list) errcount = 0 for log in result: self.msg += 'Log: ' + log + '\n'
Signature · 1 line
2.34.1
PATCH 7/8 oeqa/runtime/parselogs: move some variables out of global scope
In reply to [email protected]From: Ross Burton <[email protected]>
errors and log_locations can be trivially set in the class directly, instead of being defined in the module and then copied into the class.
--- meta/lib/oeqa/runtime/cases/parselogs.py | 19 +++++++++---------- 1 file changed, 9 insertions(+), 10 deletions(-)
meta/lib/oeqa/runtime/cases/parselogs.py +9 -10
@@ -11,9 +11,6 @@ from shutil import rmtree from oeqa.runtime.case import OERuntimeTestCase from oeqa.core.decorator.depends import OETestDepends -#in the future these lists could be moved outside of module -errors = ["error", "cannot", "can\'t", "failed"] - common_errors = [ "(WW) warning, (EE) error, (NI) not implemented, (??) unknown.", "dma timeout",
@@ -201,17 +198,19 @@ ignore_errors = { ] + common_errors, } -log_locations = ["/var/log/","/var/log/dmesg", "/tmp/dmesg_output.log"] - class ParseLogsTest(OERuntimeTestCase): + # Which log files should be collected + log_locations = ["/var/log/", "/var/log/dmesg", "/tmp/dmesg_output.log"] + + # The keywords that identify error messages in the log files + errors = ["error", "cannot", "can't", "failed"] + @classmethod def setUpClass(cls): - cls.errors = errors - # When systemd is enabled we need to notice errors on # circular dependencies in units. - if 'systemd' in cls.td.get('DISTRO_FEATURES', ''): + if 'systemd' in cls.td.get('DISTRO_FEATURES'): cls.errors.extend([ 'Found ordering cycle on', 'Breaking ordering cycle by deleting job',
@@ -220,8 +219,6 @@ class ParseLogsTest(OERuntimeTestCase): ]) cls.ignore_errors = ignore_errors - cls.log_locations = log_locations - cls.msg = '' # Go through the log locations provided and if it's a folder # create a list with all the .log files in it, if it's a file
@@ -338,7 +335,9 @@ class ParseLogsTest(OERuntimeTestCase): self.write_dmesg() log_list = self.get_local_log_list(self.log_locations) result = self.parse_logs(log_list) + errcount = 0 + self.msg = "" for log in result: self.msg += 'Log: ' + log + '\n' self.msg += '-----------------------\n'
Signature · 1 line
2.34.1
PATCH 8/8 oeqa/runtime/parselogs: select the correct machine-specific ignores early
In reply to [email protected]From: Ross Burton <[email protected]>
This has no impact to the execution, but makes the following changes neater.
--- meta/lib/oeqa/runtime/cases/parselogs.py | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-)
meta/lib/oeqa/runtime/cases/parselogs.py +6 -7
@@ -218,7 +218,11 @@ class ParseLogsTest(OERuntimeTestCase): 'Ordering cycle found, skipping', ]) - cls.ignore_errors = ignore_errors + try: + cls.ignore_errors = ignore_errors[cls.td.get('MACHINE')] + except KeyError: + cls.logger.info('No ignore list found for this machine, using default') + cls.ignore_errors = cls.ignore_errors['default'] # Go through the log locations provided and if it's a folder # create a list with all the .log files in it, if it's a file
@@ -268,13 +272,8 @@ class ParseLogsTest(OERuntimeTestCase): grepcmd = grepcmd[:-1] grepcmd += '" ' + str(log) + " | grep -Eiv \'" - try: - errorlist = self.ignore_errors[self.td.get('MACHINE')] - except KeyError: - self.msg += 'No ignore list found for this machine, using default\n' - errorlist = self.ignore_errors['default'] - for ignore_error in errorlist: + for ignore_error in self.ignore_errors: ignore_error = ignore_error.replace('(', r'\(') ignore_error = ignore_error.replace(')', r'\)') ignore_error = ignore_error.replace("'", '.')
Signature · 1 line
2.34.1
Quoted 18 lines · Ross Burton
From: Ross Burton <[email protected]>
This is part one of my parselogs series, which clean up the code and refactor slightly in preparation for a more exciting part two which I shall send shortly.
Ross Burton (8):
oeqa/runtime/parselogs: remove unused imports oeqa/runtime/parselogs: don't bother to show target hardware
information
oeqa/runtime/parselogs: remove obsolete LSB testing support oeqa/runtime/parselogs: inline single-caller functions oeqa/runtime/parselogs: improve find call oeqa/runtime/parselogs: don't pass around members oeqa/runtime/parselogs: move some variables out of global scope oeqa/runtime/parselogs: select the correct machine-specific ignores
early
Quoted 2 lines
meta/lib/oeqa/runtime/cases/parselogs.py | 109 +++++++---------------- 1 file changed, 33 insertions(+), 76 deletions(-)
This might be too exciting already!
https://autobuilder.yoctoproject.org/typhoon/#/builders/47/builds/7818
https://autobuilder.yoctoproject.org/typhoon/#/builders/110/builds/6710
I've noted the helpful error messages, presumably from setupClass :(
Cheers,
Richard