Skip to contentSkip to conversation

tests/acceptance: Fix race conditions in s390x tests & skip fedora on gitlab-CI

PATCH In review Thomas Huth 5 participants 9 messages Jan 8, 2021 – Jan 14, 2021

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 tests/acceptance: Fix race conditions in s390x tests & skip fedora on gitlab-CI

There was a race condition in the first test where there was already the "crw" output in the dmesg, but the "0.0.4711" entry has not been created in the /sys fs yet. Fix it by waiting until it is there.

The second test has even more problems on gitlab-CI. Even after adding some more synchronization points (that wait for some messages in the "dmesg" output to make sure that the modules got loaded correctly), there are still occasionally some hangs in this test when it is running in the gitlab-CI. So far I was unable to reproduce these hangs locally on my computer, so this issue might take a while to debug. Thus disable the 2nd test in the gitlab-CI until the problems are better understood and fixed.

---
 tests/acceptance/machine_s390_ccw_virtio.py | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
tests/acceptance/machine_s390_ccw_virtio.py +12 -2
index eccf26b262..4028c99afc 100644
@@ -12,6 +12,7 @@
 import os
 import tempfile
 
+from avocado import skipIf
 from avocado_qemu import Test
 from avocado_qemu import exec_command_and_wait_for_pattern
 from avocado_qemu import wait_for_console_pattern
@@ -133,8 +134,10 @@ class S390CCWVirtioMachine(Test):
         self.vm.command('device_add', driver='virtio-net-ccw',
                         devno='fe.0.4711', id='net_4711')
         self.wait_for_crw_reports()
-        exec_command_and_wait_for_pattern(self, 'ls /sys/bus/ccw/devices/',
-                                          '0.0.4711')
+        exec_command_and_wait_for_pattern(self, 'for i in 1 2 3 4 5 6 7 ; do '
+                    'if [ -e /sys/bus/ccw/devices/*4711 ]; then break; fi ;'
+                    'sleep 1 ; done ; ls /sys/bus/ccw/devices/',
+                    '0.0.4711')
         # and detach it again
         self.clear_guest_dmesg()
         self.vm.command('device_del', id='net_4711')
@@ -155,6 +158,7 @@ class S390CCWVirtioMachine(Test):
                                           'MemTotal:         115640 kB')
 
 
+    @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
     def test_s390x_fedora(self):
 
         """
@@ -199,6 +203,9 @@ class S390CCWVirtioMachine(Test):
 
         # Some tests to see whether the CLI options have been considered:
         self.log.info("Test whether QEMU CLI options have been considered")
+        exec_command_and_wait_for_pattern(self,
+                        'while ! (dmesg | grep enP7p0s0) ; do sleep 1 ; done',
+                        'virtio_net virtio0 enP7p0s0: renamed')
         exec_command_and_wait_for_pattern(self, 'lspci',
                              '0007:00:00.0 Class 0200: Device 1af4:1000')
         exec_command_and_wait_for_pattern(self,
@@ -222,6 +229,9 @@ class S390CCWVirtioMachine(Test):
         # can simply read the written "magic bytes" back from the PPM file to
         # check whether the framebuffer is working as expected.
         self.log.info("Test screendump of virtio-gpu device")
+        exec_command_and_wait_for_pattern(self,
+                        'while ! (dmesg | grep gpudrmfb) ; do sleep 1 ; done',
+                        'virtio_gpudrmfb frame buffer device')
         exec_command_and_wait_for_pattern(self,
             'echo -e "\e[?25l" > /dev/tty0', ':/#')
         exec_command_and_wait_for_pattern(self, 'for ((i=0;i<250;i++)); do '
Signature · 1 line
2.27.0
Reviewed-by Tested-by on v1 1/1
#1 lore raw
Quoted 13 lines · Thomas Huth

There was a race condition in the first test where there was already the "crw" output in the dmesg, but the "0.0.4711" entry has not been created in the /sys fs yet. Fix it by waiting until it is there.

The second test has even more problems on gitlab-CI. Even after adding some more synchronization points (that wait for some messages in the "dmesg" output to make sure that the modules got loaded correctly), there are still occasionally some hangs in this test when it is running in the gitlab-CI. So far I was unable to reproduce these hangs locally on my computer, so this issue might take a while to debug. Thus disable the 2nd test in the gitlab-CI until the problems are better understood and fixed.

Quoted 5 lines
---
 tests/acceptance/machine_s390_ccw_virtio.py | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

JOB ID : 6b2b3c1f6f6b0c4c2e9fd694b475bd12c193adbd
JOB LOG : /home/linux1/src/qemu.dev/build/tests/results/job-2021-01-08T16.24-6b2b3c1/job.log
(1/2) tests/acceptance/machine_s390_ccw_virtio.py:S390CCWVirtioMachine.test_s390x_devices:
PASS (8.78 s)
(2/2) tests/acceptance/machine_s390_ccw_virtio.py:S390CCWVirtioMachine.test_s390x_fedora:
PASS (23.86 s)
RESULTS : PASS 2 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0
| CANCEL 0
JOB TIME : 33.02 s

#2 lore raw
In reply to Thomas Huth
Quoted 30 lines · On Fri, 8 Jan 2021 19:56:45 +0100 Thomas Huth <[email protected]> wrote:

There was a race condition in the first test where there was already the "crw" output in the dmesg, but the "0.0.4711" entry has not been created in the /sys fs yet. Fix it by waiting until it is there.

The second test has even more problems on gitlab-CI. Even after adding some more synchronization points (that wait for some messages in the "dmesg" output to make sure that the modules got loaded correctly), there are still occasionally some hangs in this test when it is running in the gitlab-CI. So far I was unable to reproduce these hangs locally on my computer, so this issue might take a while to debug. Thus disable the 2nd test in the gitlab-CI until the problems are better understood and fixed.

---
 tests/acceptance/machine_s390_ccw_virtio.py | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
tests/acceptance/machine_s390_ccw_virtio.py +1 -0
index eccf26b262..4028c99afc 100644
@@ -12,6 +12,7 @@
 import os
 import tempfile
 
+from avocado import skipIf
 from avocado_qemu import Test
 from avocado_qemu import exec_command_and_wait_for_pattern
 from avocado_qemu import wait_for_console_pattern

tests/acceptance/machine_s390_ccw_virtio.py class S390CCWVirtioMachine(Test): · line 140

Quoted 9 lines
tests/acceptance/machine_s390_ccw_virtio.py +4 -2
@@ -133,8 +134,10 @@ class S390CCWVirtioMachine(Test):
         self.vm.command('device_add', driver='virtio-net-ccw',
                         devno='fe.0.4711', id='net_4711')
         self.wait_for_crw_reports()
-        exec_command_and_wait_for_pattern(self, 'ls /sys/bus/ccw/devices/',
-                                          '0.0.4711')
+        exec_command_and_wait_for_pattern(self, 'for i in 1 2 3 4 5 6 7 ; do '
+                    'if [ -e /sys/bus/ccw/devices/*4711 ]; then break; fi ;'
+                    'sleep 1 ; done ; ls /sys/bus/ccw/devices/',
+                    '0.0.4711')

I'm wondering whether we should introduce a generic helper function for "execute command repeatedly, if the expected result did not yet show up", or "wait for a file/directory to exist". It's probably not uncommon for a desired outcome to arrive asynchronously, and having a function for waiting/retrying could be handy.

Quoted 3 lines
tests/acceptance/machine_s390_ccw_virtio.py +0 -0
         # and detach it again
         self.clear_guest_dmesg()
         self.vm.command('device_del', id='net_4711')
#3 lore raw
Quoted 40 lines · Cornelia Huck
Quoted 37 lines · On Fri, 8 Jan 2021 19:56:45 +0100 Thomas Huth <[email protected]> wrote:

There was a race condition in the first test where there was already the "crw" output in the dmesg, but the "0.0.4711" entry has not been created in the /sys fs yet. Fix it by waiting until it is there.

The second test has even more problems on gitlab-CI. Even after adding some more synchronization points (that wait for some messages in the "dmesg" output to make sure that the modules got loaded correctly), there are still occasionally some hangs in this test when it is running in the gitlab-CI. So far I was unable to reproduce these hangs locally on my computer, so this issue might take a while to debug. Thus disable the 2nd test in the gitlab-CI until the problems are better understood and fixed.

---
 tests/acceptance/machine_s390_ccw_virtio.py | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
tests/acceptance/machine_s390_ccw_virtio.py +3 -2
index eccf26b262..4028c99afc 100644
@@ -12,6 +12,7 @@
 import os
 import tempfile
 
+from avocado import skipIf
 from avocado_qemu import Test
 from avocado_qemu import exec_command_and_wait_for_pattern
 from avocado_qemu import wait_for_console_pattern
@@ -133,8 +134,10 @@ class S390CCWVirtioMachine(Test):
         self.vm.command('device_add', driver='virtio-net-ccw',
                         devno='fe.0.4711', id='net_4711')
         self.wait_for_crw_reports()
-        exec_command_and_wait_for_pattern(self, 'ls /sys/bus/ccw/devices/',
-                                          '0.0.4711')
+        exec_command_and_wait_for_pattern(self, 'for i in 1 2 3 4 5 6 7 ; do '
+                    'if [ -e /sys/bus/ccw/devices/*4711 ]; then break; fi ;'
Quoted 8 lines

tests/acceptance/machine_s390_ccw_virtio.py class S390CCWVirtioMachine(Test): · line 140

Quoted 2 lines
tests/acceptance/machine_s390_ccw_virtio.py +2 -0
+                    'sleep 1 ; done ; ls /sys/bus/ccw/devices/',
+                    '0.0.4711')

I'm wondering whether we should introduce a generic helper function for "execute command repeatedly, if the expected result did not yet show up", or "wait for a file/directory to exist". It's probably not uncommon for a desired outcome to arrive asynchronously, and having a function for waiting/retrying could be handy.

We don't really want to encourage fragile shell scripts in the guest so something that makes it easy to encode these loops in python. Currently the _console_interaction helper fails the test if failure_message is seen so I guess we need a slightly more liberal interaction which accepts a command can fail so we can write something like:

while True:

      if exec_command_and_check(self, "stat -t /sys/bus/ccw/devices/0.0.4711",
                                "/sys/bus/ccw/devices/0.0.4711"):
          break

?

Quoted 4 lines
Quoted 4 lines
tests/acceptance/machine_s390_ccw_virtio.py +0 -0
         # and detach it again
         self.clear_guest_dmesg()
         self.vm.command('device_del', id='net_4711')
Signature · 1 line
Alex Bennée
#4 lore raw
Quoted 57 lines · On Tue, 12 Jan 2021 11:32:44 +0000 Alex Bennée <[email protected]> wrote:
Quoted 40 lines · Cornelia Huck
Quoted 37 lines · On Fri, 8 Jan 2021 19:56:45 +0100 Thomas Huth <[email protected]> wrote:

There was a race condition in the first test where there was already the "crw" output in the dmesg, but the "0.0.4711" entry has not been created in the /sys fs yet. Fix it by waiting until it is there.

The second test has even more problems on gitlab-CI. Even after adding some more synchronization points (that wait for some messages in the "dmesg" output to make sure that the modules got loaded correctly), there are still occasionally some hangs in this test when it is running in the gitlab-CI. So far I was unable to reproduce these hangs locally on my computer, so this issue might take a while to debug. Thus disable the 2nd test in the gitlab-CI until the problems are better understood and fixed.

---
 tests/acceptance/machine_s390_ccw_virtio.py | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
tests/acceptance/machine_s390_ccw_virtio.py +3 -2
index eccf26b262..4028c99afc 100644
@@ -12,6 +12,7 @@
 import os
 import tempfile
 
+from avocado import skipIf
 from avocado_qemu import Test
 from avocado_qemu import exec_command_and_wait_for_pattern
 from avocado_qemu import wait_for_console_pattern
@@ -133,8 +134,10 @@ class S390CCWVirtioMachine(Test):
         self.vm.command('device_add', driver='virtio-net-ccw',
                         devno='fe.0.4711', id='net_4711')
         self.wait_for_crw_reports()
-        exec_command_and_wait_for_pattern(self, 'ls /sys/bus/ccw/devices/',
-                                          '0.0.4711')
+        exec_command_and_wait_for_pattern(self, 'for i in 1 2 3 4 5 6 7 ; do '
+                    'if [ -e /sys/bus/ccw/devices/*4711 ]; then break; fi ;'
Quoted 8 lines

tests/acceptance/machine_s390_ccw_virtio.py class S390CCWVirtioMachine(Test): · line 140

Quoted 2 lines
tests/acceptance/machine_s390_ccw_virtio.py +2 -0
+                    'sleep 1 ; done ; ls /sys/bus/ccw/devices/',
+                    '0.0.4711')  

I'm wondering whether we should introduce a generic helper function for "execute command repeatedly, if the expected result did not yet show up", or "wait for a file/directory to exist". It's probably not uncommon for a desired outcome to arrive asynchronously, and having a function for waiting/retrying could be handy.

We don't really want to encourage fragile shell scripts in the guest so something that makes it easy to encode these loops in python. Currently the _console_interaction helper fails the test if failure_message is seen so I guess we need a slightly more liberal interaction which accepts a command can fail so we can write something like:

Quoted 6 lines

while True:

      if exec_command_and_check(self, "stat -t /sys/bus/ccw/devices/0.0.4711",
                                "/sys/bus/ccw/devices/0.0.4711"):
          break

?

Yes, something like that. The caller can decide whether they want to limit retries.

Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
tests/acceptance/machine_s390_ccw_virtio.py +0 -0
         # and detach it again
         self.clear_guest_dmesg()
         self.vm.command('device_del', id='net_4711')  
#5 lore raw
Quoted 65 lines · Cornelia Huck
Quoted 62 lines · On Tue, 12 Jan 2021 11:32:44 +0000 Alex Bennée <[email protected]> wrote:
Quoted 40 lines · Cornelia Huck
Quoted 37 lines · On Fri, 8 Jan 2021 19:56:45 +0100 Thomas Huth <[email protected]> wrote:

There was a race condition in the first test where there was already the "crw" output in the dmesg, but the "0.0.4711" entry has not been created in the /sys fs yet. Fix it by waiting until it is there.

The second test has even more problems on gitlab-CI. Even after adding some more synchronization points (that wait for some messages in the "dmesg" output to make sure that the modules got loaded correctly), there are still occasionally some hangs in this test when it is running in the gitlab-CI. So far I was unable to reproduce these hangs locally on my computer, so this issue might take a while to debug. Thus disable the 2nd test in the gitlab-CI until the problems are better understood and fixed.

---
  tests/acceptance/machine_s390_ccw_virtio.py | 14 ++++++++++++--
  1 file changed, 12 insertions(+), 2 deletions(-)
tests/acceptance/machine_s390_ccw_virtio.py +3 -2
index eccf26b262..4028c99afc 100644
@@ -12,6 +12,7 @@
  import os
  import tempfile
 
+from avocado import skipIf
  from avocado_qemu import Test
  from avocado_qemu import exec_command_and_wait_for_pattern
  from avocado_qemu import wait_for_console_pattern
@@ -133,8 +134,10 @@ class S390CCWVirtioMachine(Test):
          self.vm.command('device_add', driver='virtio-net-ccw',
                          devno='fe.0.4711', id='net_4711')
          self.wait_for_crw_reports()
-        exec_command_and_wait_for_pattern(self, 'ls /sys/bus/ccw/devices/',
-                                          '0.0.4711')
+        exec_command_and_wait_for_pattern(self, 'for i in 1 2 3 4 5 6 7 ; do '
+                    'if [ -e /sys/bus/ccw/devices/*4711 ]; then break; fi ;'
Quoted 8 lines

tests/acceptance/machine_s390_ccw_virtio.py class S390CCWVirtioMachine(Test): · line 140

Quoted 2 lines
tests/acceptance/machine_s390_ccw_virtio.py +2 -0
+                    'sleep 1 ; done ; ls /sys/bus/ccw/devices/',
+                    '0.0.4711')

I'm wondering whether we should introduce a generic helper function for "execute command repeatedly, if the expected result did not yet show up", or "wait for a file/directory to exist". It's probably not uncommon for a desired outcome to arrive asynchronously, and having a function for waiting/retrying could be handy.

We don't really want to encourage fragile shell scripts in the guest so something that makes it easy to encode these loops in python. Currently the _console_interaction helper fails the test if failure_message is seen so I guess we need a slightly more liberal interaction which accepts a command can fail so we can write something like:

   while True:
       if exec_command_and_check(self, "stat -t /sys/bus/ccw/devices/0.0.4711",
                                 "/sys/bus/ccw/devices/0.0.4711"):
           break
Quoted 4 lines
Quoted 1 line

?

Yes, something like that. The caller can decide whether they want to limit retries.

Fine for me, but I think we should use a timeout, not an amount of retries.

I already put my patch into my pull-request yesterday (so that people are not running into failures with their gitlab-CI), so if someone wants to have a go at creating such a function in python, feel free to do so by refactoring that code again.

Thomas

#6 lore raw

+Willian

Quoted 78 lines · Thomas Huth
Quoted 77 lines · Cornelia Huck
Quoted 74 lines · On Tue, 12 Jan 2021 11:32:44 +0000 Alex Bennée <[email protected]> wrote:
Quoted 51 lines · Cornelia Huck

On Fri,  8 Jan 2021 19:56:45 +0100
Thomas Huth <[email protected]> wrote:

Quoted 48 lines

There was a race condition in the first test where there was already the
"crw" output in the dmesg, but the "0.0.4711" entry has not been created
in the /sys fs yet. Fix it by waiting until it is there.

The second test has even more problems on gitlab-CI. Even after adding some
more synchronization points (that wait for some messages in the "dmesg"
output to make sure that the modules got loaded correctly), there are still
occasionally some hangs in this test when it is running in the gitlab-CI.
So far I was unable to reproduce these hangs locally on my computer, so
this issue might take a while to debug. Thus disable the 2nd test in the
gitlab-CI until the problems are better understood and fixed.

---
  tests/acceptance/machine_s390_ccw_virtio.py | 14 ++++++++++++--

  1 file changed, 12 insertions(+), 2 deletions(-)
tests/acceptance/machine_s390_ccw_virtio.py +0 -0
index eccf26b262..4028c99afc 100644
@@ -12,6 +12,7 @@

  import os
  import tempfile
  +from avocado import skipIf
  from avocado_qemu import Test
  from avocado_qemu import exec_command_and_wait_for_pattern from avocado_qemu import wait_for_console_pattern

tests/acceptance/machine_s390_ccw_virtio.py +1 -2
@@ -133,8 +134,10 @@ class S390CCWVirtioMachine(Test):
           self.vm.command('device_add', driver='virtio-net-ccw',
                           devno='fe.0.4711', id='net_4711')
           self.wait_for_crw_reports()
-        exec_command_and_wait_for_pattern(self, 'ls
 /sys/bus/ccw/devices/',
-                                          '0.0.4711')
+        exec_command_and_wait_for_pattern(self, 'for i in 1 2 3 4

5 6 7 ; do '
+                    'if [ -e /sys/bus/ccw/devices/*4711 ]; then break; fi ;'

Quoted 8 lines
Quoted 2 lines

+                    'sleep 1 ; done ; ls /sys/bus/ccw/devices/', +                    '0.0.4711')

I'm wondering whether we should introduce a generic helper function for "execute command repeatedly, if the expected result did not yet show up", or "wait for a file/directory to exist". It's probably not uncommon for a desired outcome to arrive asynchronously, and having a function for waiting/retrying could be handy.

We don't really want to encourage fragile shell scripts in the guest so something that makes it easy to encode these loops in python. Currently the _console_interaction helper fails the test if failure_message is seen so I guess we need a slightly more liberal interaction which accepts a command can fail so we can write something like:

   while True:
       if exec_command_and_check(self, "stat -t /sys/bus/ccw/devices/0.0.4711", "/sys/bus/ccw/devices/0.0.4711"):
           break

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

?

Yes, something like that. The caller can decide whether they want to limit retries.

Fine for me, but I think we should use a timeout, not an amount of retries.

We explained this feature request to the Avocado project at the KVM forum 2018 in Lyon. There was an entry filled on their Trello dashboard. Then the project switched to GitLab and I lost track of it.

Cleber, if you remember, can you point us at the new ticket please?

Thanks,

Phil.

Quoted 7 lines

I already put my patch into my pull-request yesterday (so that people are not running into failures with their gitlab-CI), so if someone wants to have a go at creating such a function in python, feel free to do so by refactoring that code again.

 Thomas

#7 lore raw
Quoted 1 line · Philippe Mathieu-Daudé

...

Quoted 4 lines

We explained this feature request to the Avocado project at the KVM forum 2018 in Lyon. There was an entry filled on their Trello dashboard. Then the project switched to GitLab and I lost track of it.

We moved our Avocado Trello board to the GitHub issues, but the Avocado-QEMU Trello board stays live [1], with little or no action in the last months as we are mostly using GitHub issued to track work any on Avocado.

I don't recall or I was not involved in a discussion about a feature like the one described in this thread in the KVM Forum of 2019 (Lyon).

Quoted 2 lines

Cleber, if you remember, can you point us at the new ticket please?

I went through the Trello board and could not find a card related to this feature. Anyway, just like the avocado_qemu package implements a wait_for_console_pattern, it is possible to implement the needed feature there.

Quoted 13 lines

Thanks,

Phil.

Quoted 7 lines

I already put my patch into my pull-request yesterday (so that people are not running into failures with their gitlab-CI), so if someone wants to have a go at creating such a function in python, feel free to do so by refactoring that code again.

Thomas

#8 lore raw

Avocado-QEMU Trello board missing in my previous mail.

Quoted 10 lines · Willian Rampazzo
Quoted 1 line · Philippe Mathieu-Daudé

...

Quoted 4 lines

We explained this feature request to the Avocado project at the KVM forum 2018 in Lyon. There was an entry filled on their Trello dashboard. Then the project switched to GitLab and I lost track of it.

Quoted 4 lines

We moved our Avocado Trello board to the GitHub issues, but the Avocado-QEMU Trello board stays live [1], with little or no action in the last months as we are mostly using GitHub issued to track work any on Avocado.

[1] https://trello.com/b/6Qi1pxVn/avocado-qemu

Quoted 25 lines

I don't recall or I was not involved in a discussion about a feature like the one described in this thread in the KVM Forum of 2019 (Lyon).

Quoted 2 lines

Cleber, if you remember, can you point us at the new ticket please?

I went through the Trello board and could not find a card related to this feature. Anyway, just like the avocado_qemu package implements a wait_for_console_pattern, it is possible to implement the needed feature there.

Quoted 13 lines

Thanks,

Phil.

Quoted 7 lines

I already put my patch into my pull-request yesterday (so that people are not running into failures with their gitlab-CI), so if someone wants to have a go at creating such a function in python, feel free to do so by refactoring that code again.

Thomas