Skip to contentSkip to conversation

oeqa/runtime/ssh: Rework ssh timeout

PATCH Richard Purdie 1 participant 1 message Oct 10, 2024 1 message not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw
Quoted 34 lines · Richard Purdie via lists.openembedded.org

After the changes to improve this test, we keep seeing image testing ssh failures, particularly on mips.

It looks like part of the problem is that on a loaded system, 5s is too short for mips to reliably establish an ssh connection. I've seen logs where it keeps timing out and fails to work, then the debug code successfully uses ssh later after everything else fails.

Change the timings/retries to give slow platforms enough time to respond.

---
 meta/lib/oeqa/runtime/cases/ssh.py | 4 ++--

 1 file changed, 2 insertions(+), 2 deletions(-)
meta/lib/oeqa/runtime/cases/ssh.py +0 -0
index b86428002f6..f5b76edff40 100644
@@ -16,7 +16,7 @@ class SSHTest(OERuntimeTestCase):
      @OETestDepends(['ping.PingTest.test_ping'])
      @OEHasPackage(['dropbear', 'openssh-sshd'])

     def test_ssh(self):
-        for i in range(20):
+        for i in range(5):
           status, output = self.target.run("uname -a", timeout=5) if status == 0:
               break

meta/lib/oeqa/runtime/cases/ssh.py +1 -1
@@ -29,7 +29,7 @@ class SSHTest(OERuntimeTestCase):
                # give it time for the port to open.
                # We sometimes see -15 (SIGTERM) on slow emulation machines too, likely
                # from boot/init not being 100% complete, retry for these too.
-              time.sleep(5)
+              time.sleep(30)
Quoted 3 lines

               continue
           else:
               self.fail("uname failed with \"%s\" (exit code %s)" % (output, status))

The ssh test changes and resulting failures is making me quite grumpy. We took code which was working reliably, changed to make it much nicer and in the process made the autobuilder builds unstable.

I've mentioned it several times and whilst people have looked at and said "don't know" or "it shouldn't do that", nobody is actually picking up the issue and resolving it.

I could have reverted it but I agree the new code is nicer. I'd have been asked to "prove" it was the commit, or to revert the revert at some point and then we'd be back to the same instability again.

I have looked at a log of one of the failing builds:

https://valkyrie.yocto.io/pub/shared-failure-data/log.do_testimage.2960382
from
https://valkyrie.yoctoproject.org/#/builders/32/builds/198

and concluded it is simply just slower than the 5s being allowed. Hopefully this patch fixes the issue. If it does not, I would appreciate some help in resolving these failures.

Cheers,

Richard