Skip to contentSkip to conversation

oeqa: target: ssh: log SSH errors during tests

PATCH In review Mathieu Dubois-Briand 2 participants 7 messages Oct 7, 2025 – Oct 13, 2025

lore
Share & tools
lore
#0 lore raw

cover oeqa: target: ssh: log SSH errors during tests

Most tests running SSH commands ask for no error to be raised when the returned status is not 0, and handle the returned status themselves. But most of the tests do not check if the non-zero status is caused by a fail of the command run on the target or by a fail of SSH itself.

This can lead to confusing test results, such as the one described in bugzilla issue #15981.

This series introduce a check for SSH failures by default, only bypassed in some specific cases.

---
Mathieu Dubois-Briand (4):
oeqa: target: ssh: Fail on SSH error even when errors are ignored
oeqa: runtime: Ignore SSH errors during setup and tear down
oeqa: postactions: Ignore SSH errors
oeqa: runtime: ssh: Manage any SSH failure locally

 meta/lib/oeqa/core/target/ssh.py         | 12 ++++++++----
 meta/lib/oeqa/runtime/cases/logrotate.py |  7 ++++---
 meta/lib/oeqa/runtime/cases/ssh.py       |  2 +-
 meta/lib/oeqa/runtime/cases/weston.py    |  2 +-
 meta/lib/oeqa/utils/postactions.py       |  4 ++--
 5 files changed, 16 insertions(+), 11 deletions(-)

---
base-commit: 9dfd4f44d4f40e7926dc88cb564baa2345c2a24f
change-id: 20251007-mathieu-ssh-fails-0c01b1865511

Best regards,

Signature · 1 line
Mathieu Dubois-Briand <[email protected]>
#1 lore raw

PATCH 1/4 oeqa: target: ssh: Fail on SSH error even when errors are ignored

Most tests running SSH commands ask for no error to be raised when the returned status is not 0. As run() will return this status, they may later use its value to do a similar check on their own, or completely ignore it. But most of the tests do not check if the non-zero status is caused by a fail of the command run on the target or by a fail of SSH itself.

This can lead to confusion when the error does not come from the command executed on the target but from SSH itself: test might wrongfully be marked as PASSED or might fail with incoherent errors.

As SSH errors are always reported with exit code 255, we can easily filter these.

Modify OESSHTarget.run() behaviour so an AssertionError is raised on SSH failures, even when ignore_status parameter is True. Still allow to explicitly ignore this error for the rare cases where this can be needed.

---
 meta/lib/oeqa/core/target/ssh.py | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)
meta/lib/oeqa/core/target/ssh.py +8 -4
index 8b5c450a058fd172f36af6cbd9569983aa3d2d51..0ac3ae438895c97e89599b2dfb2797dabbf383dd 100644
@@ -55,7 +55,7 @@ class OESSHTarget(OETarget):
     def stop(self, **kwargs):
         pass
 
-    def _run(self, command, timeout=None, ignore_status=True, raw=False):
+    def _run(self, command, timeout=None, ignore_status=True, raw=False, ignore_ssh_fails=False):
         """
             Runs command in target using SSHProcess.
         """
@@ -66,13 +66,17 @@ class OESSHTarget(OETarget):
         self.logger.debug("[Command returned '%d' after %.2f seconds]"
                  "" % (status, time.time() - starttime))
 
-        if status and not ignore_status:
+        if status == 255 and not ignore_ssh_fails:
+            raise AssertionError("ssh exited with status '255' for command "
+                                 "'%s': this is likely an SSH failure\n%s"
+                                 % (command, output))
+        elif status and not ignore_status:
             raise AssertionError("Command '%s' returned non-zero exit "
                                  "status %d:\n%s" % (command, status, output))
 
         return (status, output)
 
-    def run(self, command, timeout=None, ignore_status=True, raw=False):
+    def run(self, command, timeout=None, ignore_status=True, raw=False, ignore_ssh_fails=False):
         """
             Runs command in target.
 
@@ -91,7 +95,7 @@ class OESSHTarget(OETarget):
         else:
             processTimeout = self.timeout
 
-        status, output = self._run(sshCmd, processTimeout, ignore_status, raw)
+        status, output = self._run(sshCmd, processTimeout, ignore_status, raw, ignore_ssh_fails)
         if len(output) > (64 * 1024):
             self.logger.debug('Command: %s\nStatus: %d Output length:  %s\n' % (command, status, len(output)))
         else:
Signature · 1 line
2.47.3
#2 lore raw

PATCH 2/4 oeqa: runtime: Ignore SSH errors during setup and tear down

In reply to Mathieu Dubois-Briand

Tests using SSH will fail when no SSH server is present on the target. These tests are disabled in these cases, by being marked with a dependency on ssh.SSHTest.test_ssh, which in turns has a dependency on having either dropbear or openssh-sshd in the image.

But setUpClass() and tearDownClass() functions are always executed, even on tests failing the dependency checks, leading to unexpected failed tests.

Ignoring SSH errors in setup and tear down allows to avoid these test errors.

---
 meta/lib/oeqa/runtime/cases/logrotate.py | 7 ++++---
 meta/lib/oeqa/runtime/cases/weston.py    | 2 +-
 2 files changed, 5 insertions(+), 4 deletions(-)
meta/lib/oeqa/runtime/cases/logrotate.py +4 -3
index 6ad980cb6adc0cc1660989228442633db3a9827d..7cb43d98c56d73865838ed8418a6a88732840a28 100644
@@ -15,12 +15,13 @@ class LogrotateTest(OERuntimeTestCase):
 
     @classmethod
     def setUpClass(cls):
-        cls.tc.target.run('cp /etc/logrotate.d/wtmp $HOME/wtmp.oeqabak')
+        cls.tc.target.run('cp /etc/logrotate.d/wtmp $HOME/wtmp.oeqabak',
+                          ignore_ssh_fails=True)
 
     @classmethod
     def tearDownClass(cls):
-        cls.tc.target.run('mv -f $HOME/wtmp.oeqabak /etc/logrotate.d/wtmp && rm -rf /var/log//logrotate_dir')
-        cls.tc.target.run('rm -rf /var/log/logrotate_testfile && rm -rf /etc/logrotate.d/logrotate_testfile')
+        cls.tc.target.run('mv -f $HOME/wtmp.oeqabak /etc/logrotate.d/wtmp && rm -rf /var/log//logrotate_dir', ignore_ssh_fails=True)
+        cls.tc.target.run('rm -rf /var/log/logrotate_testfile && rm -rf /etc/logrotate.d/logrotate_testfile', ignore_ssh_fails=True)
 
     @OETestDepends(['ssh.SSHTest.test_ssh'])
     @OEHasPackage(['logrotate'])
meta/lib/oeqa/runtime/cases/weston.py +1 -1
index ee4d336482e5e7b4e61625599d38ab2fbf09ace0..e2cecffe83536c9bfeaae62892bad832f75cb541 100644
@@ -16,7 +16,7 @@ class WestonTest(OERuntimeTestCase):
 
     @classmethod
     def tearDownClass(cls):
-        cls.tc.target.run('rm %s' % cls.weston_log_file)
+        cls.tc.target.run('rm %s' % cls.weston_log_file, ignore_ssh_fails=True)
 
     @OETestDepends(['ssh.SSHTest.test_ssh'])
     @OEHasPackage(['weston'])
Signature · 1 line
2.47.3
#3 lore raw

Re: [OE-core] [PATCH 2/4] oeqa: runtime: Ignore SSH errors during setup and tear down

Quoted 9 lines · Mathieu Dubois-Briand via lists.openembedded.org

Tests using SSH will fail when no SSH server is present on the target. These tests are disabled in these cases, by being marked with a dependency on ssh.SSHTest.test_ssh, which in turns has a dependency on having either dropbear or openssh-sshd in the image.

But setUpClass() and tearDownClass() functions are always executed, even on tests failing the dependency checks, leading to unexpected failed tests.

Well that’s just silly. I wonder if OEHasPackage() works on the class scope too and does actually avoid doing the setup/teardown?

However, this does demonstrate that doing non-trivial setup/teardown is a bad idea in test cases, and copying files to the target is definitely non-trivial.

I suggest merging all of the logrotate tests and setup/teardown into a single test that uses finally: to cleanup, and just removing the teardown from the weston test.

Cheers,
Ross

#4 lore raw

Re: [OE-core] [PATCH 2/4] oeqa: runtime: Ignore SSH errors during setup and tear down

Quoted 11 lines · Ross Burton via lists.openembedded.org
Quoted 9 lines · Mathieu Dubois-Briand via lists.openembedded.org

Tests using SSH will fail when no SSH server is present on the target. These tests are disabled in these cases, by being marked with a dependency on ssh.SSHTest.test_ssh, which in turns has a dependency on having either dropbear or openssh-sshd in the image.

But setUpClass() and tearDownClass() functions are always executed, even on tests failing the dependency checks, leading to unexpected failed tests.

Quoted 6 lines

Well that’s just silly. I wonder if OEHasPackage() works on the class scope too and does actually avoid doing the setup/teardown?

However, this does demonstrate that doing non-trivial setup/teardown is a bad idea in test cases, and copying files to the target is definitely non-trivial.

I suggest merging all of the logrotate tests and setup/teardown into a single test that uses finally: to cleanup, and just removing the teardown from the weston test.

I am going to send some patches to clean this.

On the logrotate one, I choose to keep two separate tests, but we can highly simplify this cleanup part and integrate that in the tests. On weston, let's remove the log file before starting weston, so we can still be sure the log is clean.

Signature · 3 lines
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#5 lore raw

PATCH 3/4 oeqa: postactions: Ignore SSH errors

In reply to Mathieu Dubois-Briand

Postactions are not part of the tests but allow to retrieve useful data from the target. They try to do this using SSH, but this can fail when no SSH server is present on the target. Ignore these fails.

---
 meta/lib/oeqa/utils/postactions.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
meta/lib/oeqa/utils/postactions.py +2 -2
index c69481db6cbf29b5e3f267170151d85e73baa0d0..7b967edaad0f0a418347f18a7fa85073672ceeff 100644
@@ -22,7 +22,7 @@ from oeqa.utils import get_artefact_dir
 def get_target_disk_usage(d, tc, artifacts_list, outputdir):
     output_file = os.path.join(outputdir, "target_disk_usage.txt")
     try:
-        (status, output) = tc.target.run('df -h')
+        (status, output) = tc.target.run('df -h', ignore_ssh_fails=True)
         with open(output_file, 'w') as f:
             f.write(output)
             f.write("\n")
@@ -49,7 +49,7 @@ def get_artifacts_list(target, raw_list):
     for raw_path in raw_list.split():
         cmd = f"for p in {raw_path}; do if [ -e $p ]; then echo $p; fi; done"
         try:
-            status, output = target.run(cmd)
+            status, output = target.run(cmd, ignore_ssh_fails=True)
             if status != 0 or not output:
                 raise Exception()
             result += output.split()
Signature · 1 line
2.47.3
#6 lore raw

PATCH 4/4 oeqa: runtime: ssh: Manage any SSH failure locally

In reply to Mathieu Dubois-Briand

This is the SSH test, it makes sense to ignore SSH failures in the SSH helper and manage them in the test body.

---
 meta/lib/oeqa/runtime/cases/ssh.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/lib/oeqa/runtime/cases/ssh.py +1 -1
index b632a29a0178bf1c0015b84bca8e3fc3fd18fa74..3e9503277e39b281ffa7dc5f08f975ba2e96cee5 100644
@@ -17,7 +17,7 @@ class SSHTest(OERuntimeTestCase):
     @OEHasPackage(['dropbear', 'openssh-sshd'])
     def test_ssh(self):
         for i in range(5):
-          status, output = self.target.run("uname -a", timeout=30)
+          status, output = self.target.run("uname -a", timeout=30, ignore_ssh_fails=True)
           if status == 0:
               break
           elif status == 255 or status == -signal.SIGTERM:
Signature · 1 line
2.47.3