oeqa: target: ssh: log SSH errors during tests
cover oeqa: target: ssh: log SSH errors during tests
Most tests running SSH commands ask for no error to be raised when the returned status is not 0, and handle the returned status themselves. But most of the tests do not check if the non-zero status is caused by a fail of the command run on the target or by a fail of SSH itself.
This can lead to confusing test results, such as the one described in bugzilla issue #15981.
This series introduce a check for SSH failures by default, only bypassed in some specific cases.
---
Mathieu Dubois-Briand (4):
oeqa: target: ssh: Fail on SSH error even when errors are ignored
oeqa: runtime: Ignore SSH errors during setup and tear down
oeqa: postactions: Ignore SSH errors
oeqa: runtime: ssh: Manage any SSH failure locally
meta/lib/oeqa/core/target/ssh.py | 12 ++++++++---- meta/lib/oeqa/runtime/cases/logrotate.py | 7 ++++--- meta/lib/oeqa/runtime/cases/ssh.py | 2 +- meta/lib/oeqa/runtime/cases/weston.py | 2 +- meta/lib/oeqa/utils/postactions.py | 4 ++-- 5 files changed, 16 insertions(+), 11 deletions(-)
---
base-commit: 9dfd4f44d4f40e7926dc88cb564baa2345c2a24f
change-id: 20251007-mathieu-ssh-fails-0c01b1865511
Best regards,
Signature · 1 line
Mathieu Dubois-Briand <[email protected]>
PATCH 1/4 oeqa: target: ssh: Fail on SSH error even when errors are ignored
Most tests running SSH commands ask for no error to be raised when the returned status is not 0. As run() will return this status, they may later use its value to do a similar check on their own, or completely ignore it. But most of the tests do not check if the non-zero status is caused by a fail of the command run on the target or by a fail of SSH itself.
This can lead to confusion when the error does not come from the command executed on the target but from SSH itself: test might wrongfully be marked as PASSED or might fail with incoherent errors.
As SSH errors are always reported with exit code 255, we can easily filter these.
Modify OESSHTarget.run() behaviour so an AssertionError is raised on SSH failures, even when ignore_status parameter is True. Still allow to explicitly ignore this error for the rare cases where this can be needed.
--- meta/lib/oeqa/core/target/ssh.py | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-)
meta/lib/oeqa/core/target/ssh.py +8 -4
@@ -55,7 +55,7 @@ class OESSHTarget(OETarget): def stop(self, **kwargs): pass - def _run(self, command, timeout=None, ignore_status=True, raw=False): + def _run(self, command, timeout=None, ignore_status=True, raw=False, ignore_ssh_fails=False): """ Runs command in target using SSHProcess. """
@@ -66,13 +66,17 @@ class OESSHTarget(OETarget): self.logger.debug("[Command returned '%d' after %.2f seconds]" "" % (status, time.time() - starttime)) - if status and not ignore_status: + if status == 255 and not ignore_ssh_fails: + raise AssertionError("ssh exited with status '255' for command " + "'%s': this is likely an SSH failure\n%s" + % (command, output)) + elif status and not ignore_status: raise AssertionError("Command '%s' returned non-zero exit " "status %d:\n%s" % (command, status, output)) return (status, output) - def run(self, command, timeout=None, ignore_status=True, raw=False): + def run(self, command, timeout=None, ignore_status=True, raw=False, ignore_ssh_fails=False): """ Runs command in target.
@@ -91,7 +95,7 @@ class OESSHTarget(OETarget): else: processTimeout = self.timeout - status, output = self._run(sshCmd, processTimeout, ignore_status, raw) + status, output = self._run(sshCmd, processTimeout, ignore_status, raw, ignore_ssh_fails) if len(output) > (64 * 1024): self.logger.debug('Command: %s\nStatus: %d Output length: %s\n' % (command, status, len(output))) else:
Signature · 1 line
2.47.3
PATCH 2/4 oeqa: runtime: Ignore SSH errors during setup and tear down
In reply to Mathieu Dubois-BriandTests using SSH will fail when no SSH server is present on the target. These tests are disabled in these cases, by being marked with a dependency on ssh.SSHTest.test_ssh, which in turns has a dependency on having either dropbear or openssh-sshd in the image.
But setUpClass() and tearDownClass() functions are always executed, even on tests failing the dependency checks, leading to unexpected failed tests.
Ignoring SSH errors in setup and tear down allows to avoid these test errors.
--- meta/lib/oeqa/runtime/cases/logrotate.py | 7 ++++--- meta/lib/oeqa/runtime/cases/weston.py | 2 +- 2 files changed, 5 insertions(+), 4 deletions(-)
meta/lib/oeqa/runtime/cases/logrotate.py +4 -3
@@ -15,12 +15,13 @@ class LogrotateTest(OERuntimeTestCase): @classmethod def setUpClass(cls): - cls.tc.target.run('cp /etc/logrotate.d/wtmp $HOME/wtmp.oeqabak') + cls.tc.target.run('cp /etc/logrotate.d/wtmp $HOME/wtmp.oeqabak', + ignore_ssh_fails=True) @classmethod def tearDownClass(cls): - cls.tc.target.run('mv -f $HOME/wtmp.oeqabak /etc/logrotate.d/wtmp && rm -rf /var/log//logrotate_dir') - cls.tc.target.run('rm -rf /var/log/logrotate_testfile && rm -rf /etc/logrotate.d/logrotate_testfile') + cls.tc.target.run('mv -f $HOME/wtmp.oeqabak /etc/logrotate.d/wtmp && rm -rf /var/log//logrotate_dir', ignore_ssh_fails=True) + cls.tc.target.run('rm -rf /var/log/logrotate_testfile && rm -rf /etc/logrotate.d/logrotate_testfile', ignore_ssh_fails=True) @OETestDepends(['ssh.SSHTest.test_ssh']) @OEHasPackage(['logrotate'])
meta/lib/oeqa/runtime/cases/weston.py +1 -1
@@ -16,7 +16,7 @@ class WestonTest(OERuntimeTestCase): @classmethod def tearDownClass(cls): - cls.tc.target.run('rm %s' % cls.weston_log_file) + cls.tc.target.run('rm %s' % cls.weston_log_file, ignore_ssh_fails=True) @OETestDepends(['ssh.SSHTest.test_ssh']) @OEHasPackage(['weston'])
Signature · 1 line
2.47.3
Re: [OE-core] [PATCH 2/4] oeqa: runtime: Ignore SSH errors during setup and tear down
Quoted 9 lines · Mathieu Dubois-Briand via lists.openembedded.org
Tests using SSH will fail when no SSH server is present on the target. These tests are disabled in these cases, by being marked with a dependency on ssh.SSHTest.test_ssh, which in turns has a dependency on having either dropbear or openssh-sshd in the image.
But
setUpClass()andtearDownClass()functions are always executed, even on tests failing the dependency checks, leading to unexpected failed tests.
Well that’s just silly. I wonder if OEHasPackage() works on the class scope too and does actually avoid doing the setup/teardown?
However, this does demonstrate that doing non-trivial setup/teardown is a bad idea in test cases, and copying files to the target is definitely non-trivial.
I suggest merging all of the logrotate tests and setup/teardown into a single test that uses finally: to cleanup, and just removing the teardown from the weston test.
Cheers,
Ross
Re: [OE-core] [PATCH 2/4] oeqa: runtime: Ignore SSH errors during setup and tear down
Quoted 11 lines · Ross Burton via lists.openembedded.org
Quoted 9 lines · Mathieu Dubois-Briand via lists.openembedded.org
Tests using SSH will fail when no SSH server is present on the target. These tests are disabled in these cases, by being marked with a dependency on ssh.SSHTest.test_ssh, which in turns has a dependency on having either dropbear or openssh-sshd in the image.
But
setUpClass()andtearDownClass()functions are always executed, even on tests failing the dependency checks, leading to unexpected failed tests.
Quoted 6 lines
Well that’s just silly. I wonder if
OEHasPackage()works on the class scope too and does actually avoid doing the setup/teardown?However, this does demonstrate that doing non-trivial setup/teardown is a bad idea in test cases, and copying files to the target is definitely non-trivial.
I suggest merging all of the logrotate tests and setup/teardown into a single test that uses finally: to cleanup, and just removing the teardown from the weston test.
I am going to send some patches to clean this.
On the logrotate one, I choose to keep two separate tests, but we can highly simplify this cleanup part and integrate that in the tests. On weston, let's remove the log file before starting weston, so we can still be sure the log is clean.
Signature · 3 lines
Mathieu Dubois-Briand, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 3/4 oeqa: postactions: Ignore SSH errors
In reply to Mathieu Dubois-BriandPostactions are not part of the tests but allow to retrieve useful data from the target. They try to do this using SSH, but this can fail when no SSH server is present on the target. Ignore these fails.
--- meta/lib/oeqa/utils/postactions.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
meta/lib/oeqa/utils/postactions.py +2 -2
@@ -22,7 +22,7 @@ from oeqa.utils import get_artefact_dir def get_target_disk_usage(d, tc, artifacts_list, outputdir): output_file = os.path.join(outputdir, "target_disk_usage.txt") try: - (status, output) = tc.target.run('df -h') + (status, output) = tc.target.run('df -h', ignore_ssh_fails=True) with open(output_file, 'w') as f: f.write(output) f.write("\n")
@@ -49,7 +49,7 @@ def get_artifacts_list(target, raw_list): for raw_path in raw_list.split(): cmd = f"for p in {raw_path}; do if [ -e $p ]; then echo $p; fi; done" try: - status, output = target.run(cmd) + status, output = target.run(cmd, ignore_ssh_fails=True) if status != 0 or not output: raise Exception() result += output.split()
Signature · 1 line
2.47.3
PATCH 4/4 oeqa: runtime: ssh: Manage any SSH failure locally
In reply to Mathieu Dubois-BriandThis is the SSH test, it makes sense to ignore SSH failures in the SSH helper and manage them in the test body.
--- meta/lib/oeqa/runtime/cases/ssh.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/lib/oeqa/runtime/cases/ssh.py +1 -1
@@ -17,7 +17,7 @@ class SSHTest(OERuntimeTestCase): @OEHasPackage(['dropbear', 'openssh-sshd']) def test_ssh(self): for i in range(5): - status, output = self.target.run("uname -a", timeout=30) + status, output = self.target.run("uname -a", timeout=30, ignore_ssh_fails=True) if status == 0: break elif status == 255 or status == -signal.SIGTERM:
Signature · 1 line
2.47.3