net: fix non-deterministic failures of the 'netdev-socket' qtest
Subsystems (2)
cover net: fix non-deterministic failures of the 'netdev-socket' qtest
We've previously bumped up the timeouts in the netdev-socket qtest to supposedly fix non-deterministic failures, however, the failures are still hitting CI.
A simple 'listen()' and 'connect()' pairing across 2 QEMU processes should be very quick to execute, even under high system load, so it was never likely that the test was failing due to timeouts being reached.
The actual root cause was a race condition in the test design. It was spawning a QEMU with a 'server' netdev, and then spawning one with the 'client' netdev. There was insufficient synchronization, however, so it was possible for the 2nd QEMU process to attempt to 'connect()' before the 'listen()' call was made by the 1st QEMU.
In the test scenarios that did not use the 'reconnect' flag, this would result in the client QEMU never getting into the expected state. The test code would thus loop on 'info network' until hitting the maximum wait time.
This series reverts the increased timeouts, and fixes synchronization in the test scenarios. It also improves reporting of errors in the socket netdev backend so that 'info network' reports what actually went wrong rather than a useless generic 'connection error' string. This will help us diagnose any future CI problems, should they occurr.
Daniel P. Berrangé (6):
Revert "netdev: set timeout depending on loadavg" Revert "osdep: add getloadavg"
Revert "tests/qtest/netdev-socket: Raise connection timeout to 120
seconds"
net: add explicit info about connecting/listening state
net: handle QIOTask completion to report useful error message
qtest: ensure netdev-socket tests have non-overlapping names
include/qemu/osdep.h | 10 --------- meson.build | 1 - net/stream.c | 18 +++++++++++----- tests/qtest/netdev-socket.c | 42 +++++++------------------------------ 4 files changed, 21 insertions(+), 50 deletions(-)
Signature · 1 line
2.43.0
PATCH 1/6 Revert "netdev: set timeout depending on loadavg"
This reverts commit cadfc7293977ecadc2d6c48d7cffc553ed2f85f1.
The test was not timing out because of slow execution. It was timing out due to a race condition leading to the client QEMU attempting (and fatally failing) to connect before the server QEMU was listening.
--- tests/qtest/netdev-socket.c | 28 +--------------------------- 1 file changed, 1 insertion(+), 27 deletions(-)
tests/qtest/netdev-socket.c +1 -27
@@ -18,32 +18,6 @@#define CONNECTION_TIMEOUT 120 -static double connection_timeout (void ) -{ -double load; -int ret =getloadavg (&load,1 ); - -/* -* If we can't get load data, or load is low because we just started -* running, assume load of 1 (we are alone in this system). -*/ -if (ret <1 || load <1.0 ) { - load =1.0 ; - } -/* -* No one wants to wait more than 10 minutes for this test. Higher load? -* Too bad. -*/ -if (load >10.0 ) { -fprintf (stderr,"Warning: load %f higher than 10 - test might timeout\n" , - load); - load =10.0 ; - } - -/* if load is high increase timeout as we might not get a chance to run */ -return load *CONNECTION_TIMEOUT ; -} -#define EXPECT_STATE (q, e, t) \ do { \ char *resp = NULL; \
@@ -57,7 +31,7 @@ do { \if (g_str_equal (resp, e)) { \break ; \ } \ - }while (g_test_timer_elapsed () <connection_timeout ()); \ + }while (g_test_timer_elapsed () <CONNECTION_TIMEOUT ); \ g_assert_cmpstr(resp, ==, e); \g_free (resp); \ }while (0 )
Signature · 1 line
2.43.0
PATCH 2/6 Revert "osdep: add getloadavg"
In reply to Daniel P. BerrangéThis reverts commit dc864d3a3777424187280e50c9bfb84dced54f12.
This functionality is not required after the previous revert
--- include/qemu/osdep.h | 10 ---------- meson.build | 1 - 2 files changed, 11 deletions(-)
include/qemu/osdep.h +0 -10
@@ -779,16 +779,6 @@ static inline int platform_does_not_support_system(const char *command) }#endif /* !HAVE_SYSTEM_FUNCTION */ -/** -* If the load average was unobtainable, -1 is returned -*/ -#ifndef HAVE_GETLOADAVG_FUNCTION -static inline int getloadavg (double loadavg[],int nelem) -{ -return -1 ; -} -#endif /* !HAVE_GETLOADAVG_FUNCTION */ -#ifdef __cplusplus }#endif
meson.build +0 -1
@@ -2296,7 +2296,6 @@ config_host_data.set('HAVE_GLIB_WITH_SLICE_ALLOCATOR', glib_has_gslice) config_host_data.set('HAVE_OPENPTY', cc.has_function('openpty', dependencies: util)) config_host_data.set('HAVE_STRCHRNUL', cc.has_function('strchrnul')) config_host_data.set('HAVE_SYSTEM_FUNCTION', cc.has_function('system', prefix: '#include <stdlib.h>')) -config_host_data.set('HAVE_GETLOADAVG_FUNCTION', cc.has_function('getloadavg', prefix: '#include <stdlib.h>')) if rbd.found() config_host_data.set('HAVE_RBD_NAMESPACE_EXISTS', cc.has_function('rbd_namespace_exists',
Signature · 1 line
2.43.0
PATCH 3/6 Revert "tests/qtest/netdev-socket: Raise connection timeout to 120 seconds"
In reply to Daniel P. BerrangéThis reverts commit 0daaf2761f6d268ffaa2d01d450e202e127452b1.
The test was not timing out because of slow execution. It was timing out due to a race condition leading to the client QEMU attempting (and fatally failing) to connect before the server QEMU was listening.
--- tests/qtest/netdev-socket.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tests/qtest/netdev-socket.c +1 -1
@@ -16,7 +16,7 @@#include "qapi/qobject-input-visitor.h" #include "qapi/qapi-visit-sockets.h" -#define CONNECTION_TIMEOUT 120 +#define CONNECTION_TIMEOUT 60#define EXPECT_STATE (q, e, t) \ do { \
Signature · 1 line
2.43.0
PATCH 4/6 net: add explicit info about connecting/listening state
In reply to Daniel P. BerrangéWhen running 'info network', if the stream backend is still in the process of connecting, or waiting for an incoming connection, no information is displayed.
There is also no way to distinguish whether the server is still in the process of setting up the listener socket, or whether it is ready to accept incoming client connections.
This leads to a race condition in the netdev-socket qtest which launches a server process followed by a client process. Under high load conditions it is possible for the client to attempt to connect before the server is accepting clients. For the scenarios which do not set the 'reconnect' option, this opens up a race which can lead to the test scenario failing to reach the expected state.
Now that 'info network' can distinguish between initialization phase and the listening phase, the netdev-socket qtest will correctly synchronize, such that the client QEMU is not spawned until the server is ready.
This should solve the non-deterministic failures seen with the netdev-socket qtest.
--- net/stream.c | 5 ++++- tests/qtest/netdev-socket.c | 10 +++++----- 2 files changed, 9 insertions(+), 6 deletions(-)
net/stream.c +4 -1
@@ -173,7 +173,7 @@ static gboolean net_stream_send(QIOChannel *ioc,net_socket_rs_init (&s->rs, net_stream_rs_finalize, false); s->nc.link_down = true; -qemu_set_info_str (&s->nc,"%s" ,"" ); +qemu_set_info_str (&s->nc,"listening" );qapi_event_send_netdev_stream_disconnected (s->nc.name);net_stream_arm_reconnect (s);
@@ -292,6 +292,7 @@ static void net_stream_server_listening(QIOTask *task, gpointer opaque) s->nc.link_down = true; s->listener =qio_net_listener_new (); +qemu_set_info_str (&s->nc,"listening" );net_socket_rs_init (&s->rs, net_stream_rs_finalize, false);qio_net_listener_set_client_func (s->listener, net_stream_listen, s,NULL );qio_net_listener_add (s->listener, listen_sioc);
@@ -309,6 +310,7 @@ static int net_stream_server_init(NetClientState *peer, nc =qemu_new_net_client (&net_stream_info, peer, model, name); s =DO_UPCAST (NetStreamState, nc, nc); +qemu_set_info_str (&s->nc,"initializing" ); s->listen_ioc =QIO_CHANNEL (listen_sioc); qio_channel_socket_listen_async(listen_sioc, addr, 0,
@@ -400,6 +402,7 @@ static int net_stream_client_init(NetClientState *peer, nc =qemu_new_net_client (&net_stream_info, peer, model, name); s =DO_UPCAST (NetStreamState, nc, nc); +qemu_set_info_str (&s->nc,"connecting" ); s->ioc =QIO_CHANNEL (sioc); s->nc.link_down = true;
tests/qtest/netdev-socket.c +5 -5
@@ -127,7 +127,7 @@ static void test_stream_inet_ipv4(void)"addr.ipv4=on,addr.ipv6=off," "addr.host=127.0.0.1,addr.port=%d", port); -EXPECT_STATE (qts0,"st0: index=0,type=stream,\r\n" ,0 ); +EXPECT_STATE (qts0,"st0: index=0,type=stream,listening\r\n" ,0 ); qts1 =qtest_initf ("-nodefaults -M none " "-netdev stream,server=false,id=st0,addr.type=inet,"
@@ -200,7 +200,7 @@ static void test_stream_unix_reconnect(void)"-netdev stream,id=st0,server=true,addr.type=unix," "addr.path=%s", path); -EXPECT_STATE (qts0,"st0: index=0,type=stream,\r\n" ,0 ); +EXPECT_STATE (qts0,"st0: index=0,type=stream,listening\r\n" ,0 ); qts1 =qtest_initf ("-nodefaults -M none " "-netdev stream,server=false,id=st0,addr.type=unix,"
@@ -250,7 +250,7 @@ static void test_stream_inet_ipv6(void)"addr.ipv4=off,addr.ipv6=on," "addr.host=::1,addr.port=%d", port); -EXPECT_STATE (qts0,"st0: index=0,type=stream,\r\n" ,0 ); +EXPECT_STATE (qts0,"st0: index=0,type=stream,listening\r\n" ,0 ); qts1 =qtest_initf ("-nodefaults -M none " "-netdev stream,server=false,id=st0,addr.type=inet,"
@@ -282,7 +282,7 @@ static void test_stream_unix(void)"addr.type=unix,addr.path=%s," , path); -EXPECT_STATE (qts0,"st0: index=0,type=stream,\r\n" ,0 ); +EXPECT_STATE (qts0,"st0: index=0,type=stream,listening\r\n" ,0 ); qts1 =qtest_initf ("-nodefaults -M none " "-netdev stream,id=st0,server=false,"
@@ -314,7 +314,7 @@ static void test_stream_unix_abstract(void)"addr.abstract=on" , path); -EXPECT_STATE (qts0,"st0: index=0,type=stream,\r\n" ,0 ); +EXPECT_STATE (qts0,"st0: index=0,type=stream,listening\r\n" ,0 ); qts1 =qtest_initf ("-nodefaults -M none " "-netdev stream,id=st0,server=false,"
Signature · 1 line
2.43.0
PATCH 5/6 net: handle QIOTask completion to report useful error message
In reply to Daniel P. BerrangéThe network stream backend uses the async QIO socket APIs for listening and connecting sockets. It does not check the task object completion status, however, instead just looking at whether the socket FD is -1 or not.
By checking the task completion, we can set a useful error message for users instead of the non-actionable "connection error" string.
eg so users will see:
(qemu) info network
net: index=0,type=stream,error: Failed to connect to '/foo.unix': No such file or directory
--- net/stream.c | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-)
net/stream.c +10 -5
@@ -165,6 +165,7 @@ static gboolean net_stream_send(QIOChannel *ioc, s->ioc_write_tag =0 ; }if (s->listener) { +qemu_set_info_str (&s->nc,"listening" );qio_net_listener_set_client_func (s->listener, net_stream_listen, s,NULL ); }
@@ -173,7 +174,6 @@ static gboolean net_stream_send(QIOChannel *ioc,net_socket_rs_init (&s->rs, net_stream_rs_finalize, false); s->nc.link_down = true; -qemu_set_info_str (&s->nc,"listening" );qapi_event_send_netdev_stream_disconnected (s->nc.name);net_stream_arm_reconnect (s);
@@ -272,9 +272,11 @@ static void net_stream_server_listening(QIOTask *task, gpointer opaque)QIOChannelSocket *listen_sioc =QIO_CHANNEL_SOCKET (s->listen_ioc);SocketAddress *addr;int ret; +Error *err =NULL ; -if (listen_sioc->fd <0 ) { -qemu_set_info_str (&s->nc,"connection error" ); +if (qio_task_propagate_error (task, &err)) { +qemu_set_info_str (&s->nc,"error: %s" ,error_get_pretty (err)); +error_free (err);return ; }
@@ -327,9 +329,11 @@ static void net_stream_client_connected(QIOTask *task, gpointer opaque)SocketAddress *addr;gchar *uri;int ret; +Error *err =NULL ; -if (sioc->fd <0 ) { -qemu_set_info_str (&s->nc,"connection error" ); +if (qio_task_propagate_error (task, &err)) { +qemu_set_info_str (&s->nc,"error: %s" ,error_get_pretty (err)); +error_free (err); goto error; }
@@ -384,6 +388,7 @@ static gboolean net_stream_reconnect(gpointer data)static void net_stream_arm_reconnect (NetStreamState *s) {if (s->reconnect && s->timer_tag ==0 ) { +qemu_set_info_str (&s->nc,"connecting" ); s->timer_tag =g_timeout_add_seconds (s->reconnect, net_stream_reconnect, s); }
Signature · 1 line
2.43.0
PATCH 6/6 qtest: ensure netdev-socket tests have non-overlapping names
In reply to Daniel P. BerrangéWhen naming glib tests if the name of one test is a substring of the name of another test, it is not possible to use the '-p /the/name' option to run a single test.
--- tests/qtest/netdev-socket.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tests/qtest/netdev-socket.c +1 -1
@@ -526,7 +526,7 @@ int main(int argc, char **argv)#ifndef _WIN32qtest_add_func ("/netdev/dgram/unix" , test_dgram_unix);#endif -qtest_add_func ("/netdev/stream/unix" , test_stream_unix); +qtest_add_func ("/netdev/stream/unix/oneshot" , test_stream_unix);qtest_add_func ("/netdev/stream/unix/reconnect" , test_stream_unix_reconnect);#ifdef CONFIG_LINUX
Signature · 1 line
2.43.0
Re: [PATCH 6/6] qtest: ensure netdev-socket tests have non-overlapping names
Quoted 8 lines · Daniel P. Berrangé
When naming glib tests if the name of one test is a substring of the name of another test, it is not possible to use the '-p /the/name' option to run a single test.
Signed-off-by Daniel P. Berrangé <[email protected]>--- tests/qtest/netdev-socket.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 36 lines · Daniel P. Berrangé
We've previously bumped up the timeouts in the netdev-socket qtest to supposedly fix non-deterministic failures, however, the failures are still hitting CI.
A simple '
listen()' and 'connect()' pairing across 2 QEMU processes should be very quick to execute, even under high system load, so it was never likely that the test was failing due to timeouts being reached.The actual root cause was a race condition in the test design. It was spawning a QEMU with a 'server' netdev, and then spawning one with the 'client' netdev. There was insufficient synchronization, however, so it was possible for the 2nd QEMU process to attempt to '
connect()' before the 'listen()' call was made by the 1st QEMU.In the test scenarios that did not use the 'reconnect' flag, this would result in the client QEMU never getting into the expected state. The test code would thus loop on 'info network' until hitting the maximum wait time.
This series reverts the increased timeouts, and fixes synchronization in the test scenarios. It also improves reporting of errors in the socket netdev backend so that 'info network' reports what actually went wrong rather than a useless generic 'connection error' string. This will help us diagnose any future CI problems, should they occurr.
Daniel P. Berrangé (6):
Revert "netdev: set timeout depending on loadavg" Revert "osdep: add getloadavg"
Revert "tests/qtest/netdev-socket: Raise connection timeout to 120
seconds"
net: add explicit info about connecting/listening state
net: handle QIOTask completion to report useful error message
qtest: ensure netdev-socket tests have non-overlapping names
Quoted 5 lines
include/qemu/osdep.h | 10 --------- meson.build | 1 - net/stream.c | 18 +++++++++++----- tests/qtest/netdev-socket.c | 42 +++++++------------------------------ 4 files changed, 21 insertions(+), 50 deletions(-)
Quoted 38 lines · Stefan Hajnoczi
Quoted 36 lines · Daniel P. Berrangé
We've previously bumped up the timeouts in the netdev-socket qtest to supposedly fix non-deterministic failures, however, the failures are still hitting CI.
A simple '
listen()' and 'connect()' pairing across 2 QEMU processes should be very quick to execute, even under high system load, so it was never likely that the test was failing due to timeouts being reached.The actual root cause was a race condition in the test design. It was spawning a QEMU with a 'server' netdev, and then spawning one with the 'client' netdev. There was insufficient synchronization, however, so it was possible for the 2nd QEMU process to attempt to '
connect()' before the 'listen()' call was made by the 1st QEMU.In the test scenarios that did not use the 'reconnect' flag, this would result in the client QEMU never getting into the expected state. The test code would thus loop on 'info network' until hitting the maximum wait time.
This series reverts the increased timeouts, and fixes synchronization in the test scenarios. It also improves reporting of errors in the socket netdev backend so that 'info network' reports what actually went wrong rather than a useless generic 'connection error' string. This will help us diagnose any future CI problems, should they occurr.
Daniel P. Berrangé (6):
Revert "netdev: set timeout depending on loadavg" Revert "osdep: add getloadavg"
Revert "tests/qtest/netdev-socket: Raise connection timeout to 120
seconds"
net: add explicit info about connecting/listening state
net: handle QIOTask completion to report useful error message
qtest: ensure netdev-socket tests have non-overlapping names
Quoted 8 lines
Quoted 5 lines
include/qemu/osdep.h | 10 --------- meson.build | 1 - net/stream.c | 18 +++++++++++----- tests/qtest/netdev-socket.c | 42 +++++++------------------------------ 4 files changed, 21 insertions(+), 50 deletions(-)Reviewed-by Stefan Hajnoczi <[email protected]>
Queued.
Thanks
Quoted 46 lines · Jason Wang
Quoted 44 lines · Stefan Hajnoczi
Quoted 41 lines · Daniel P. Berrangé
We've previously bumped up the timeouts in the netdev-socket qtest to supposedly fix non-deterministic failures, however, the failures are still hitting CI.
A simple '
listen()' and 'connect()' pairing across 2 QEMU processes should be very quick to execute, even under high system load, so it was never likely that the test was failing due to timeouts being reached.The actual root cause was a race condition in the test design. It was spawning a QEMU with a 'server' netdev, and then spawning one with the 'client' netdev. There was insufficient synchronization, however, so it was possible for the 2nd QEMU process to attempt to '
connect()' before the 'listen()' call was made by the 1st QEMU.In the test scenarios that did not use the 'reconnect' flag, this would result in the client QEMU never getting into the expected state. The test code would thus loop on 'info network' until hitting the maximum wait time.
This series reverts the increased timeouts, and fixes synchronization in the test scenarios. It also improves reporting of errors in the socket netdev backend so that 'info network' reports what actually went wrong rather than a useless generic 'connection error' string. This will help us diagnose any future CI problems, should they occurr.
Daniel P. Berrangé (6):
Revert "netdev: set timeout depending on loadavg" Revert "osdep: add getloadavg"
Revert "tests/qtest/netdev-socket: Raise connection timeout to 120
seconds"
net: add explicit info about connecting/listening state
net: handle QIOTask completion to report useful error message
qtest: ensure netdev-socket tests have non-overlapping namesinclude/qemu/osdep.h | 10 --------- meson.build | 1 - net/stream.c | 18 +++++++++++----- tests/qtest/netdev-socket.c | 42 +++++++------------------------------ 4 files changed, 21 insertions(+), 50 deletions(-)
Quoted 4 lines
Quoted 2 lines
Reviewed-by Stefan Hajnoczi <[email protected]>Queued.
These are already upstream, via thuth's testing queue.
thanks
-- PMM
Quoted 51 lines · Peter Maydell
Quoted 49 lines · Jason Wang
Quoted 46 lines · Stefan Hajnoczi
Quoted 36 lines · Daniel P. Berrangé
We've previously bumped up the timeouts in the netdev-socket qtest to supposedly fix non-deterministic failures, however, the failures are still hitting CI.
A simple '
listen()' and 'connect()' pairing across 2 QEMU processes should be very quick to execute, even under high system load, so it was never likely that the test was failing due to timeouts being reached.The actual root cause was a race condition in the test design. It was spawning a QEMU with a 'server' netdev, and then spawning one with the 'client' netdev. There was insufficient synchronization, however, so it was possible for the 2nd QEMU process to attempt to '
connect()' before the 'listen()' call was made by the 1st QEMU.In the test scenarios that did not use the 'reconnect' flag, this would result in the client QEMU never getting into the expected state. The test code would thus loop on 'info network' until hitting the maximum wait time.
This series reverts the increased timeouts, and fixes synchronization in the test scenarios. It also improves reporting of errors in the socket netdev backend so that 'info network' reports what actually went wrong rather than a useless generic 'connection error' string. This will help us diagnose any future CI problems, should they occurr.
Daniel P. Berrangé (6):
Revert "netdev: set timeout depending on loadavg" Revert "osdep: add getloadavg"
Revert "tests/qtest/netdev-socket: Raise connection timeout to 120
seconds"
net: add explicit info about connecting/listening state
net: handle QIOTask completion to report useful error message
qtest: ensure netdev-socket tests have non-overlapping namesQuoted 5 lines
include/qemu/osdep.h | 10 --------- meson.build | 1 - net/stream.c | 18 +++++++++++----- tests/qtest/netdev-socket.c | 42 +++++++------------------------------ 4 files changed, 21 insertions(+), 50 deletions(-)Reviewed-by Stefan Hajnoczi <[email protected]>
Quoted 7 lines
Quoted 1 line
Queued.
These are already upstream, via thuth's testing queue.
thanks
-- PMM
Great, so I dropped this.
Thanks
Hi Jason,
As the net/ maintainer, could you take a look at this series. This failure has been causing pain for CI for quite a while.
If you're happy with it, I can include it in a pending pull request of other misc patches I have.
Quoted 35 lines · Daniel P. Berrangé
We've previously bumped up the timeouts in the netdev-socket qtest to supposedly fix non-deterministic failures, however, the failures are still hitting CI.
A simple '
listen()' and 'connect()' pairing across 2 QEMU processes should be very quick to execute, even under high system load, so it was never likely that the test was failing due to timeouts being reached.The actual root cause was a race condition in the test design. It was spawning a QEMU with a 'server' netdev, and then spawning one with the 'client' netdev. There was insufficient synchronization, however, so it was possible for the 2nd QEMU process to attempt to '
connect()' before the 'listen()' call was made by the 1st QEMU.In the test scenarios that did not use the 'reconnect' flag, this would result in the client QEMU never getting into the expected state. The test code would thus loop on 'info network' until hitting the maximum wait time.
This series reverts the increased timeouts, and fixes synchronization in the test scenarios. It also improves reporting of errors in the socket netdev backend so that 'info network' reports what actually went wrong rather than a useless generic 'connection error' string. This will help us diagnose any future CI problems, should they occurr.
Daniel P. Berrangé (6):
Revert "netdev: set timeout depending on loadavg" Revert "osdep: add getloadavg"
Revert "tests/qtest/netdev-socket: Raise connection timeout to 120
seconds"
net: add explicit info about connecting/listening state
net: handle QIOTask completion to report useful error message
qtest: ensure netdev-socket tests have non-overlapping names
Quoted 9 lines
include/qemu/osdep.h | 10 --------- meson.build | 1 - net/stream.c | 18 +++++++++++----- tests/qtest/netdev-socket.c | 42 +++++++------------------------------ 4 files changed, 21 insertions(+), 50 deletions(-)Signature · 1 line
2.43.0
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|