Skip to contentSkip to conversation

tests/functional: Make socat wait longer in migration exec test

PATCH In review Fabiano Rosas 4 participants 4 messages Apr 22 – Apr 23

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 tests/functional: Make socat wait longer in migration exec test

The migration_with_exec test is failing sporadically for all architectures due to a race when the destination socat process takes too long to start listening while the source process is already issuing connect().

The race is inherent because the exec: migration spawns the to-be-exec'ed command asynchronously and returns from the migrate-incoming command. The localhost-only testcase is not representative of the majority of migrations. In a real scenario between two different hosts that race wouldn't happen.

Fix the testcase by configuring the source socat command to wait indefinitely while trying to connect.

---
 tests/functional/migration.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/functional/migration.py +1 -1
index 144f091ba8..3b7674af3b 100644
@@ -85,5 +85,5 @@ def migration_with_exec(self):
         with Ports() as ports:
             free_port = self._get_free_port(ports)
             dst_uri = 'exec:socat TCP-LISTEN:%u -' % free_port
-            src_uri = 'exec:socat - TCP:localhost:%u' % free_port
+            src_uri = 'exec:socat - TCP:localhost:%u,forever' % free_port
             self.migrate(dst_uri, src_uri)
Signature · 1 line
2.51.0
Reviewed-by on v1 1/1
#1 lore raw
Quoted 24 lines · Fabiano Rosas

The migration_with_exec test is failing sporadically for all architectures due to a race when the destination socat process takes too long to start listening while the source process is already issuing connect().

The race is inherent because the exec: migration spawns the to-be-exec'ed command asynchronously and returns from the migrate-incoming command. The localhost-only testcase is not representative of the majority of migrations. In a real scenario between two different hosts that race wouldn't happen.

Fix the testcase by configuring the source socat command to wait indefinitely while trying to connect.

---
  tests/functional/migration.py | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
tests/functional/migration.py +0 -0
index 144f091ba8..3b7674af3b 100644

tests/functional/migration.py def migration_with_exec(self): · line 88

Quoted 5 lines
tests/functional/migration.py +1 -1
@@ -85,5 +85,5 @@ def migration_with_exec(self):
          with Ports() as ports:
              free_port = self._get_free_port(ports)
              dst_uri = 'exec:socat TCP-LISTEN:%u -' % free_port
-            src_uri = 'exec:socat - TCP:localhost:%u' % free_port
+            src_uri = 'exec:socat - TCP:localhost:%u,forever' % free_port

Maybe use "retry=90" instead? OTOH, we have the high level timeout from the meson runner, so we should not hang here forever by accident anyway.

Thus:

Reviewed-by
#2 lore raw
Quoted 20 lines · Thomas Huth
Quoted 19 lines · Fabiano Rosas

The migration_with_exec test is failing sporadically for all architectures due to a race when the destination socat process takes too long to start listening while the source process is already issuing connect().

The race is inherent because the exec: migration spawns the to-be-exec'ed command asynchronously and returns from the migrate-incoming command. The localhost-only testcase is not representative of the majority of migrations. In a real scenario between two different hosts that race wouldn't happen.

Fix the testcase by configuring the source socat command to wait indefinitely while trying to connect.

---
  tests/functional/migration.py | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 13 lines

tests/functional/migration.py def migration_with_exec(self): · line 88

Quoted 10 lines
tests/functional/migration.py +1 -1
index 144f091ba8..3b7674af3b 100644
@@ -85,5 +85,5 @@ def migration_with_exec(self):
          with Ports() as ports:
              free_port = self._get_free_port(ports)
              dst_uri = 'exec:socat TCP-LISTEN:%u -' % free_port
-            src_uri = 'exec:socat - TCP:localhost:%u' % free_port
+            src_uri = 'exec:socat - TCP:localhost:%u,forever' % free_port

Maybe use "retry=90" instead? OTOH, we have the high level timeout from the meson runner, so we should not hang here forever by accident anyway.

Yes, IMHO it'll be good to stick with one timeout mechanism rather than adding more magical timeouts.

Quoted 3 lines

Thus:

Signature · 1 line
Peter Xu
#3 lore raw
Quoted 35 lines · Peter Xu
Quoted 20 lines · Thomas Huth
Quoted 19 lines · Fabiano Rosas

The migration_with_exec test is failing sporadically for all architectures due to a race when the destination socat process takes too long to start listening while the source process is already issuing connect().

The race is inherent because the exec: migration spawns the to-be-exec'ed command asynchronously and returns from the migrate-incoming command. The localhost-only testcase is not representative of the majority of migrations. In a real scenario between two different hosts that race wouldn't happen.

Fix the testcase by configuring the source socat command to wait indefinitely while trying to connect.

---
   tests/functional/migration.py | 2 +-
   1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 13 lines

tests/functional/migration.py def migration_with_exec(self): · line 88

Quoted 10 lines
tests/functional/migration.py +1 -1
index 144f091ba8..3b7674af3b 100644
@@ -85,5 +85,5 @@ def migration_with_exec(self):
           with Ports() as ports:
               free_port = self._get_free_port(ports)
               dst_uri = 'exec:socat TCP-LISTEN:%u -' % free_port
-            src_uri = 'exec:socat - TCP:localhost:%u' % free_port
+            src_uri = 'exec:socat - TCP:localhost:%u,forever' % free_port

Maybe use "retry=90" instead? OTOH, we have the high level timeout from the meson runner, so we should not hang here forever by accident anyway.

Quoted 8 lines

Yes, IMHO it'll be good to stick with one timeout mechanism rather than adding more magical timeouts.

Quoted 3 lines

Thus: