selftests: mptcp: increase test coverage
Subsystems (3)
Florian Westphal
v3 cover selftests: mptcp: increase test coverage
This is resend of my selftest changes. All commits may be sqaushed to "mptcp: add basic kselftest for mptcp".
1. make tc delays random. This is to make sure mptcp-level retransmits don't trigger more frequent when delay is large.
Note that the existing selftests always test low delay in the loopback test case and in the ns1 <-> ns2 case, because there is no artificial delay in both cases.
2. add accept/getpeername checks to make sure the sockaddr returned from kernel is the expected endpoint.
3. add ipv6 capability to the test.
This was tested by changing the test script to use TCP <-> TCP and setting both capture=1 ipv6=true in the script.
The tcpdump shows ipv6 is used.
4. use a random combination of tso/gso/gro. Presence (or lack) of any of them cause slight changes in behaviour, we should make sure include as many combinations as possible.
Because test duration is already large (~3m 45s in my vm), don't add more test cases but instead make it use random combination.
5. allow parallel execution of the test case. This allows you to run e.g. 10 parallel instances instead of waiting for sequential completion.
6. Fix a possible deadlock in the test program where both ends block on write(), waiting for other side to drain the receive buffer.
7. track duration of each test. This is useful to spot bugs in mptcp:
if MPTCP <-> MPTCP test takes 31 seconds but TCP fallback took 3, something is wrong.
With current export branch, I saw no absurd divergences, so mptcp seems fine right now.
Average test run time is about 2 minutes after these changes.
The changes are also available via git-pull:
The following changes since commit 85c93de446d3fd90e47605ba8ff161bef7558f09:
mptcp: Implement basic path manager (2019-10-23 01:31:04 +0000)
are available in the Git repository at:
git://git.breakpoint.cc/fw/mptcp-next.git mptcp_selftest_03
for you to fetch changes up to fc53dcc1b003887a25d3efb7238a351dab37ba8f:
selftests: mptcp: track duration of each test (2019-10-23 15:08:17 +0200)
----------------------------------------------------------------
Florian Westphal (7):
selftests: mptcp: make tc delays random
selftests: mptcp: add accept/getpeer checks
selftests: mptcp: extend mptcp_connect tool for ipv6 family
selftests: mptcp: random ethtool tweaking
selftests: allow parallel execution
selftests: fix spurious deadlock
selftests: mptcp: track duration of each test
tools/testing/selftests/net/mptcp/mptcp_connect.c | 176 +++++++++++++--- tools/testing/selftests/net/mptcp/mptcp_connect.sh | 224 +++++++++++++++------ 2 files changed, 314 insertions(+), 86 deletions(-)
Florian Westphal
PATCH v3 1/7 selftests: mptcp: make tc delays random
test with less predictable setups:
tc qdisc delay is now random, same for reordering and loss. Motivation is to cover more scenarios without a increase in test-time.
--- .../selftests/net/mptcp/mptcp_connect.sh | 26 +++++++++++++++++-- 1 file changed, 24 insertions(+), 2 deletions(-)
tools/testing/selftests/net/mptcp/mptcp_connect.sh +24 -2
@@ -302,8 +302,30 @@ for sender in 1 2 3 4;do do_ping ns4 ns$sender 10.0.3.1 done -tc -net ns2 qdisc add dev ns2eth3 root netem loss random 1 -tc -net ns3 qdisc add dev ns3eth4 root netem delay 10ms reorder 25% 50% gap 5 +loss=$((RANDOM%101)) +if [ $loss -eq 100 ] ;then + loss=1% + tc -net ns2 qdisc add dev ns2eth3 root netem loss random $loss +elif [ $loss -ge 10 ]; then + loss=0.$loss% + tc -net ns2 qdisc add dev ns2eth3 root netem loss random $loss +elif [ $loss -ge 1 ]; then + loss=0.0$loss% + tc -net ns2 qdisc add dev ns2eth3 root netem loss random $loss +fi + +delay=$((RANDOM%1000)) +reorder1=$((RANDOM%10)) +reorder1=$((100 - reorder1)) +reorder2=$((RANDOM%100)) + +if [ $reorder1 -lt 100 ] && [ $reorder2 -gt 0 ]; then + tc -net ns3 qdisc add dev ns3eth4 root netem delay ${delay}ms reorder ${reorder1}% ${reorder2}% + echo "INFO: Using loss of $loss, delay $delay ms, reorder: $reorder1, $reorder2 on ns3eth4" +elif [ $delay -gt 0 ]; then + tc -net ns3 qdisc add dev ns3eth4 root netem delay ${delay}ms + echo "INFO: Using loss of $loss, delay $delay ms on ns3eth4" +fi for sender in 1 2 3 4;do run_tests ns1 ns$sender 10.0.1.1
Signature · 1 line
2.21.0
Mat Martineau
[MPTCP] Re: [PATCH v3 1/7] selftests: mptcp: make tc delays random
Quoted 33 lines · Florian Westphal
test with less predictable setups:
tc qdisc delay is now random, same for reordering and loss. Motivation is to cover more scenarios without a increase in test-time.Signed-off-by Florian Westphal <fw(a)strlen.de>---
.../selftests/net/mptcp/mptcp_connect.sh| 26 +++++++++++++++++--1 file changed, 24 insertions(+), 2 deletions(-)tools/testing/selftests/net/mptcp/mptcp_connect.sh +0
-0@@ -302,8 +302,30 @@ for sender in 1 2 3 4;do do_ping ns4 ns$sender 10.0.3.1done
tools/testing/selftests/net/mptcp/mptcp_connect.sh +14
-2-tc -net ns2 qdisc add dev ns2eth3 root netem loss random 1 -tc -net ns3 qdisc add dev ns3eth4 root netem delay 10ms reorder 25% 50% gap 5 +loss=$((RANDOM%101)) +if [ $loss -eq 100 ] ;then + loss=1% + tc -net ns2 qdisc add dev ns2eth3 root netem loss random $loss +elif [ $loss -ge 10 ]; then + loss=0.$loss% + tc -net ns2 qdisc add dev ns2eth3 root netem loss random $loss +elif [ $loss -ge 1 ]; then + loss=0.0$loss% + tc -net ns2 qdisc add dev ns2eth3 root netem loss random $loss +fi + +delay=$((RANDOM%1000)) +reorder1=$((RANDOM%10))
tools/testing/selftests/net/mptcp/mptcp_connect.sh for sender in 1 2 3 4;do · line 328
Quoted 10 lines
tools/testing/selftests/net/mptcp/mptcp_connect.sh +10
-0+reorder1=$((100 - reorder1)) +reorder2=$((RANDOM%100)) + +if [ $reorder1 -lt 100 ] && [ $reorder2 -gt 0 ]; then + tc -net ns3 qdisc add dev ns3eth4 root netem delay ${delay}ms reorder ${reorder1}% ${reorder2}% + echo "INFO: Using loss of $loss, delay $delay ms, reorder: $reorder1, $reorder2 on ns3eth4" +elif [ $delay -gt 0 ]; then + tc -net ns3 qdisc add dev ns3eth4 root netem delay ${delay}ms + echo "INFO: Using loss of $loss, delay $delay ms on ns3eth4" +fi
I think it's ok to merge this patch, but randomly configuring with/without reordering does seem like a significant difference in test cases (and maybe the gso/tso/gro changes in patch 4 reach that threshold too).
There are situations where execution time is less of a concern than coverage. What do you think about randomizing the netem and ethtool settings by default, but having a command line option to run a larger set of test cases?
--
Mat Martineau
Intel
Florian Westphal
[MPTCP] Re: [PATCH v3 1/7] selftests: mptcp: make tc delays random
Quoted 17 lines · Mat Martineau
tools/testing/selftests/net/mptcp/mptcp_connect.sh for sender in 1 2 3 4;do · line 328
Quoted 12 lines
tools/testing/selftests/net/mptcp/mptcp_connect.sh +12
-0+delay=$((RANDOM%1000)) +reorder1=$((RANDOM%10)) +reorder1=$((100 - reorder1)) +reorder2=$((RANDOM%100)) + +if [ $reorder1 -lt 100 ] && [ $reorder2 -gt 0 ]; then + tc -net ns3 qdisc add dev ns3eth4 root netem delay ${delay}ms reorder ${reorder1}% ${reorder2}% + echo "INFO: Using loss of $loss, delay $delay ms, reorder: $reorder1, $reorder2 on ns3eth4" +elif [ $delay -gt 0 ]; then + tc -net ns3 qdisc add dev ns3eth4 root netem delay ${delay}ms + echo "INFO: Using loss of $loss, delay $delay ms on ns3eth4" +fiI think it's ok to merge this patch, but randomly configuring with/without reordering does seem like a significant difference in test cases (and maybe the gso/tso/gro changes in patch 4 reach that threshold too).
Quoted 4 lines
There are situations where execution time is less of a concern than coverage. What do you think about randomizing the netem and ethtool settings by default, but having a command line option to run a larger set of test cases?
Thats a good idea, I'll think about what kind of settings would be useful to have.
We can also reduce test time by not testing the fallback case anymore except for the loopback, i.e. ns1 <-> ns1 with tcp -> mptcp, mptcp -> tcp, and mptcp -> mptcp, but for everything else just do the mptcp <-> mptcp tests.
What do you think?
Mat Martineau
[MPTCP] Re: [PATCH v3 1/7] selftests: mptcp: make tc delays random
Quoted 26 lines · Florian Westphal
Quoted 17 lines · Mat Martineau
tools/testing/selftests/net/mptcp/mptcp_connect.sh for sender in 1 2 3 4;do · line 328
Quoted 12 lines
tools/testing/selftests/net/mptcp/mptcp_connect.sh +12
-0+delay=$((RANDOM%1000)) +reorder1=$((RANDOM%10)) +reorder1=$((100 - reorder1)) +reorder2=$((RANDOM%100)) + +if [ $reorder1 -lt 100 ] && [ $reorder2 -gt 0 ]; then + tc -net ns3 qdisc add dev ns3eth4 root netem delay ${delay}ms reorder ${reorder1}% ${reorder2}% + echo "INFO: Using loss of $loss, delay $delay ms, reorder: $reorder1, $reorder2 on ns3eth4" +elif [ $delay -gt 0 ]; then + tc -net ns3 qdisc add dev ns3eth4 root netem delay ${delay}ms + echo "INFO: Using loss of $loss, delay $delay ms on ns3eth4" +fiI think it's ok to merge this patch, but randomly configuring with/without reordering does seem like a significant difference in test cases (and maybe the gso/tso/gro changes in patch 4 reach that threshold too).
Quoted 4 lines
There are situations where execution time is less of a concern than coverage. What do you think about randomizing the netem and ethtool settings by default, but having a command line option to run a larger set of test cases?
Thats a good idea, I'll think about what kind of settings would be useful to have.
Quoted 6 lines
We can also reduce test time by not testing the fallback case anymore except for the loopback, i.e. ns1 <-> ns1 with tcp -> mptcp, mptcp -> tcp, and mptcp -> mptcp, but for everything else just do the mptcp <-> mptcp tests.
What do you think?
That would be a good way to save time. I don't think the combinations other than ns1 <-> ns1 do anything meaningfully different when everything from SYNACK onward is regular TCP.
--
Mat Martineau
Intel
Florian Westphal
PATCH v3 2/7 selftests: mptcp: add accept/getpeer checks
In reply to Florian WestphalCheck that the result coming from accept matches that of getpeername. For initiator side, check getpeername matches address passed to connect().
At this time, kernel returns the address of the first subflow in the list. Right now, we do not yet implement fate-sharing, i.e. if the original subflow goes away, the result of getpeername would change.
There are different ways to fix that. One would be to copy the quadruple from the first subflow socket to the mptcp socket.
If we do that, this test should catch a possible bug (such as accidental reversal of saddr/daddr).
--- .../selftests/net/mptcp/mptcp_connect.c | 107 ++++++++++++++++++ 1 file changed, 107 insertions(+)
tools/testing/selftests/net/mptcp/mptcp_connect.c +107 -0
@@ -54,6 +54,21 @@ static const char *getxinfo_strerr(int err)return gai_strerror (err); } +static void xgetnameinfo (const struct sockaddr *addr,socklen_t addrlen, +char *host,socklen_t hostlen, +char *serv,socklen_t servlen) +{ +int flags =NI_NUMERICHOST |NI_NUMERICSERV ; +int err =getnameinfo (addr, addrlen, host, hostlen, serv, servlen, flags); + +if (err) { +const char *errstr =getxinfo_strerr (err); + +fprintf (stderr,"Fatal: getnameinfo: %s\n" , errstr); +exit (1 ); + } +} +static void xgetaddrinfo (const char *node,const char *service,const struct addrinfo *hints,struct addrinfo **res)
@@ -327,6 +342,94 @@ static int copyfd_io(int infd, int peerfd, int outfd)return 0 ; } +static void check_sockaddr (int pf,struct sockaddr_storage *ss, +socklen_t salen) +{ +char addr[INET6_ADDRSTRLEN ]; +char serv[INET6_ADDRSTRLEN ]; +struct sockaddr_in6 *sin6; +struct sockaddr_in *sin; +int wanted_size =0 ; + +switch (pf) { +case AF_INET : + wanted_size =sizeof (*sin); + sin = (void *)ss; +if (!sin->sin_port) +fprintf (stderr,"accept: something wrong: ip connection from port 0" ); +break ; +case AF_INET6 : + wanted_size =sizeof (*sin6); + sin6 = (void *)ss; +if (!sin6->sin6_port) +fprintf (stderr,"accept: something wrong: ipv6 connection from port 0" ); +break ; +default : +fprintf (stderr,"accept: Unknown pf %d, salen %u\n" , pf, salen); +return ; + } + +if (salen != wanted_size) +fprintf (stderr,"accept: size mismatch, got %d expected %d\n" , + (int )salen, wanted_size); + +if (ss->ss_family != pf) +fprintf (stderr,"accept: pf mismatch, expect %d, ss_family is %d\n" , + (int )ss->ss_family, pf); +} + +static void check_getpeername (int fd,struct sockaddr_storage *ss,socklen_t salen) +{ +struct sockaddr_storage peerss; +socklen_t peersalen =sizeof (peerss); + +if (getpeername (fd, (struct sockaddr *)&peerss, &peersalen) <0 ) { +perror ("getpeername" ); +return ; + } + +if (peersalen != salen) { +fprintf (stderr,"%s: %d vs %d\n" , __func__, peersalen, salen); +return ; + } + +if (memcmp (ss, &peerss, peersalen)) { +char a[INET6_ADDRSTRLEN ]; +char b[INET6_ADDRSTRLEN ]; +char c[INET6_ADDRSTRLEN ]; +char d[INET6_ADDRSTRLEN ]; + +xgetnameinfo ((struct sockaddr *)ss, salen, + a,sizeof (a), b,sizeof (b)); + +xgetnameinfo ((struct sockaddr *)&peerss, peersalen, + c,sizeof (c), d,sizeof (d)); + +fprintf (stderr,"%s: memcmp failure: accept %s vs peername %s, %s vs %s salen %d vs %d\n" , + __func__, a, c, b, d, peersalen, salen); + } +} + +static void check_getpeername_connect (int fd) +{ +struct sockaddr_storage ss; +socklen_t salen =sizeof (ss); +char a[INET6_ADDRSTRLEN ]; +char b[INET6_ADDRSTRLEN ]; + +if (getpeername (fd, (struct sockaddr *)&ss, &salen) <0 ) { +perror ("getpeername" ); +return ; + } + +xgetnameinfo ((struct sockaddr *)&ss, salen, + a,sizeof (a), b,sizeof (b)); + +if (strcmp (cfg_host, a) ||strcmp (cfg_port, b)) +fprintf (stderr,"%s: %s vs %s, %s vs %s\n" , __func__, + cfg_host, a, cfg_port, b); +} +int main_loop_s (int listensock) {struct sockaddr_storage ss;
@@ -350,6 +453,9 @@ int main_loop_s(int listensock) salen =sizeof (ss); remotesock =accept (listensock, (struct sockaddr *)&ss, &salen);if (remotesock >=0 ) { +check_sockaddr (pf, &ss, salen); +check_getpeername (remotesock, &ss, salen); +copyfd_io (0 , remotesock,1 );return 0 ; }
@@ -384,6 +490,7 @@ int main_loop(void)if (fd <0 )return 2 ; +check_getpeername_connect (fd);return copyfd_io (0 , fd,1 ); }
Signature · 1 line
2.21.0
Mat Martineau
[MPTCP] Re: [PATCH v3 2/7] selftests: mptcp: add accept/getpeer checks
Quoted 45 lines · Florian Westphal
Check that the result coming from accept matches that of getpeername. For initiator side, check getpeername matches address passed to
connect().At this time, kernel returns the address of the first subflow in the list. Right now, we do not yet implement fate-sharing, i.e. if the original subflow goes away, the result of getpeername would change.
There are different ways to fix that. One would be to copy the quadruple from the first subflow socket to the mptcp socket.
If we do that, this test should catch a possible bug (such as accidental reversal of saddr/daddr).
Signed-off-by Florian Westphal <fw(a)strlen.de>---
.../selftests/net/mptcp/mptcp_connect.c| 107 ++++++++++++++++++1 file changed, 107 insertions(+)tools/testing/selftests/net/mptcp/mptcp_connect.c +15
-0@@ -54,6 +54,21 @@ static const char *getxinfo_strerr(int err)return gai_strerror (err); } +static void xgetnameinfo (const struct sockaddr *addr,socklen_t addrlen, +char *host,socklen_t hostlen, +char *serv,socklen_t servlen) +{ +int flags =NI_NUMERICHOST |NI_NUMERICSERV ; +int err =getnameinfo (addr, addrlen, host, hostlen, serv, servlen, flags); + +if (err) { +const char *errstr =getxinfo_strerr (err); + +fprintf (stderr,"Fatal: getnameinfo: %s\n" , errstr); +exit (1 ); + } +} +static void xgetaddrinfo (const char *node,const char *service,const struct addrinfo *hints,struct addrinfo **res)
tools/testing/selftests/net/mptcp/mptcp_connect.c static int copyfd_io(int infd, int peerfd, int outfd) · line 349
Quoted 8 lines
tools/testing/selftests/net/mptcp/mptcp_connect.c +5
-0@@ -327,6 +342,94 @@ static int copyfd_io(int infd, int peerfd, int outfd)return 0 ; } +static void check_sockaddr (int pf,struct sockaddr_storage *ss, +socklen_t salen) +{ +char addr[INET6_ADDRSTRLEN ]; +char serv[INET6_ADDRSTRLEN ];
addr and serv are not used and cause a compile warning.
The rest of the patch looks good to me.
--
Mat Martineau
Intel
Florian Westphal
[MPTCP] Re: [PATCH v3 2/7] selftests: mptcp: add accept/getpeer checks
Quoted 7 lines · Mat Martineau
tools/testing/selftests/net/mptcp/mptcp_connect.c static int copyfd_io(int infd, int peerfd, int outfd) · line 349
Quoted 5 lines
tools/testing/selftests/net/mptcp/mptcp_connect.c +5
-0+static void check_sockaddr (int pf,struct sockaddr_storage *ss, +socklen_t salen) +{ +char addr[INET6_ADDRSTRLEN ]; +char serv[INET6_ADDRSTRLEN ];addr and serv are not used and cause a compile warning.
Note to self: update your VM image :-/
Thanks, I'll fix this up for next iteration.
Florian Westphal
PATCH v3 3/7 selftests: mptcp: extend mptcp_connect tool for ipv6 family
In reply to Florian WestphalAt this time socket() will fail when requesting an ipv6 mptcp socket. This is ok for now, as the test script won't request ipv6 tests yet.
Once someone starts to implement mptcp v6 support, just set ipv6=true in the script and the selftest will attempt to connect via ipv6.
Tested with a tcp <-> tcp connection.
--- .../selftests/net/mptcp/mptcp_connect.c | 16 ++-- .../selftests/net/mptcp/mptcp_connect.sh | 77 ++++++++++++++++--- 2 files changed, 77 insertions(+), 16 deletions(-)
tools/testing/selftests/net/mptcp/mptcp_connect.c +11 -5
@@ -37,11 +37,11 @@ static const char *cfg_host;static const char *cfg_port ="12000" ;static int cfg_sock_proto =IPPROTO_MPTCP ;static bool tcpulp_audit; - +static int pf =AF_INET ;static void die_usage (void ) { -fprintf (stderr,"Usage: mptcp_connect [-u] [-s MPTCP|TCP] [-p port] " +fprintf (stderr,"Usage: mptcp_connect [-6] [-u] [-s MPTCP|TCP] [-p port] " "[ -l ] [ -t timeout ] connect_address\n" );exit (1 ); }
@@ -94,12 +94,13 @@ static int sock_listen_mptcp(const char * const listenaddr, .ai_flags = AI_PASSIVE | AI_NUMERICHOST }; - hints.ai_family =AF_INET ; + hints.ai_family = pf;struct addrinfo *a, *addr;int one =1 ;xgetaddrinfo (listenaddr, port, &hints, &addr); + hints.ai_family = pf;for (a = addr; a; a = a->ai_next) { sock =socket (a->ai_family, a->ai_socktype, cfg_sock_proto);
@@ -182,7 +183,7 @@ static int sock_connect_mptcp(const char * const remoteaddr,struct addrinfo *a, *addr;int sock =-1 ; - hints.ai_family =AF_INET ; + hints.ai_family = pf;xgetaddrinfo (remoteaddr, port, &hints, &addr);for (a = addr; a; a = a->ai_next) {
@@ -512,7 +513,7 @@ static void parse_opts(int argc, char **argv) {int c; -while ((c =getopt (argc, argv,"lp:s:hut:" )) !=-1 ) { +while ((c =getopt (argc, argv,"6lp:s:hut:" )) !=-1 ) {switch (c) {case 'l' : listen_mode = true;
@@ -528,6 +529,8 @@ static void parse_opts(int argc, char **argv)break ;case 'u' : tcpulp_audit = true; +case '6' : + pf =AF_INET6 ;break ;case 't' : poll_timeout =atoi (optarg) *1000 ;
@@ -540,6 +543,9 @@ static void parse_opts(int argc, char **argv)if (optind +1 != argc)die_usage (); cfg_host = argv[optind]; + +if (strchr (cfg_host,':' )) + pf =AF_INET6 ; }int main (int argc,char *argv[])
tools/testing/selftests/net/mptcp/mptcp_connect.sh +66 -11
175 lines not on this page · raw message
Signature · 1 line
2.21.0
Florian Westphal
PATCH v3 4/7 selftests: mptcp: random ethtool tweaking
In reply to Florian WestphalInstead of unconditionally disabling TSO in ns3, turn off any of gso/tso/gro in ns3 and/or ns4.
This gets us various combinations of GRO/GSO/TSO without a large impact on the average test run time.
The old approach is already not deterministic, and cannot possibly be made so. For protocol tests mptcp packetdrill is a better fit.
--- .../selftests/net/mptcp/mptcp_connect.sh | 25 ++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-)
tools/testing/selftests/net/mptcp/mptcp_connect.sh +24 -1
@@ -91,7 +91,6 @@ $ipv6 && ip -net ns3 addr add dead:beef:3::2/64 dev ns3eth4 ip -net ns3 link set ns3eth4 up ip -net ns3 route add default via 10.0.2.1 $ipv6 && ip -net ns3 route add default via dead:beef:2::1 -ip netns exec ns3 ethtool -K ns3eth2 tso off 2>/dev/null ip netns exec ns3 sysctl -q net.ipv4.ip_forward=1 $ipv6 && ip netns exec ns3 sysctl -q net.ipv6.conf.all.forwarding=1
@@ -101,6 +100,30 @@ ip -net ns4 link set ns4eth3 up ip -net ns4 route add default via 10.0.3.2 $ipv6 && ip -net ns4 route add default via dead:beef:3::2 +set_ethtool_flags() { + ns=$1 + dev=$2 + + r=$RANDOM + + pick1=$((r & 1)) + pick2=$((r & 2)) + pick3=$((r & 4)) + + flags="" + [ $pick1 -ne 0 ] && flags="tso off" + [ $pick2 -ne 0 ] && flags="$flags gso off" + [ $pick3 -ne 0 ] && flags="$flags gro off" + + [ -z "$flags" ] && return + + ip netns exec $ns ethtool -K $dev $flags 2>/dev/null + [ $? -eq 0 ] && echo "INFO: set $ns dev $dev: ethtool -K $flags" +} + +set_ethtool_flags ns3 ns3eth2 +set_ethtool_flags ns4 ns4eth3 + print_file_err() { ls -l "$1" 1>&2
Signature · 1 line
2.21.0
Florian Westphal
PATCH v3 5/7 selftests: allow parallel execution
In reply to Florian WestphalThis allows to run multiple instances of the test script in parallel on the same host.
Should it make easier to spot e.g. resource leaks or race conditions.
--- .../selftests/net/mptcp/mptcp_connect.sh | 165 +++++++++--------- 1 file changed, 86 insertions(+), 79 deletions(-)
tools/testing/selftests/net/mptcp/mptcp_connect.sh +86 -79
@@ -11,6 +11,13 @@ capture=0 timeout=30 ipv6=false +sec=$(date +%s) +rndh=$(printf %x $sec)-$(mktemp -u XXXXXX) +ns1="ns1-$rndh" +ns2="ns2-$rndh" +ns3="ns3-$rndh" +ns4="ns4-$rndh" + TEST_COUNT=0 cleanup()
@@ -19,8 +26,8 @@ cleanup() rm -f "$sin" "$sout" rm -f "$capout" - for i in 1 2 3 4; do - ip netns del ns$i + for netns in "$ns1" "$ns2" "$ns3" "$ns4";do + ip netns del $netns done }
@@ -43,62 +50,62 @@ cout=$(mktemp) capout=$(mktemp) trap cleanup EXIT -for i in 1 2 3 4;do - ip netns add ns$i || exit $ksft_skip - ip -net ns$i link set lo up +for i in "$ns1" "$ns2" "$ns3" "$ns4";do + ip netns add $i || exit $ksft_skip + ip -net $i link set lo up done -# ns1 ns2 ns3 ns4 +# "$ns1" ns2 ns3 ns4 # ns1eth2 ns2eth1 ns2eth3 ns3eth2 ns3eth4 ns4eth3 # - drop 1% -> reorder 25% # <- TSO off - -ip link add ns1eth2 netns ns1 type veth peer name ns2eth1 netns ns2 -ip link add ns2eth3 netns ns2 type veth peer name ns3eth2 netns ns3 -ip link add ns3eth4 netns ns3 type veth peer name ns4eth3 netns ns4 +ip link add ns1eth2 netns "$ns1" type veth peer name ns2eth1 netns "$ns2" +ip link add ns2eth3 netns "$ns2" type veth peer name ns3eth2 netns "$ns3" +ip link add ns3eth4 netns "$ns3" type veth peer name ns4eth3 netns "$ns4" -ip -net ns1 addr add 10.0.1.1/24 dev ns1eth2 +ip -net "$ns1" addr add 10.0.1.1/24 dev ns1eth2 if $ipv6 ; then - ip -net ns1 addr add dead:beef:1::1/64 dev ns1eth2 + ip -net "$ns1" addr add dead:beef:1::1/64 dev ns1eth2 if [ $? -ne 0 ] ;then echo "SKIP: Can't add ipv6 address, skip ipv6 tests" 1>&2 ipv6=false fi fi -ip -net ns1 link set ns1eth2 up -ip -net ns1 route add default via 10.0.1.2 -$ipv6 && ip -net ns1 route add default via dead:beef:1::2 - -ip -net ns2 addr add 10.0.1.2/24 dev ns2eth1 -$ipv6 && ip -net ns2 addr add dead:beef:1::2/64 dev ns2eth1 -ip -net ns2 link set ns2eth1 up - -ip -net ns2 addr add 10.0.2.1/24 dev ns2eth3 -$ipv6 && ip -net ns2 addr add dead:beef:2::1/64 dev ns2eth3 -ip -net ns2 link set ns2eth3 up -ip -net ns2 route add default via 10.0.2.2 -$ipv6 && ip -net ns2 route add default via dead:beef:2::2 -ip netns exec ns2 sysctl -q net.ipv4.ip_forward=1 -$ipv6 && ip netns exec ns2 sysctl -q net.ipv6.conf.all.forwarding=1 - -ip -net ns3 addr add 10.0.2.2/24 dev ns3eth2 -$ipv6 && ip -net ns3 addr add dead:beef:2::2/64 dev ns3eth2 -ip -net ns3 link set ns3eth2 up - -ip -net ns3 addr add 10.0.3.2/24 dev ns3eth4 -$ipv6 && ip -net ns3 addr add dead:beef:3::2/64 dev ns3eth4 -ip -net ns3 link set ns3eth4 up -ip -net ns3 route add default via 10.0.2.1 -$ipv6 && ip -net ns3 route add default via dead:beef:2::1 -ip netns exec ns3 sysctl -q net.ipv4.ip_forward=1 -$ipv6 && ip netns exec ns3 sysctl -q net.ipv6.conf.all.forwarding=1 - -ip -net ns4 addr add 10.0.3.1/24 dev ns4eth3 -$ipv6 && ip -net ns4 addr add dead:beef:3::1/64 dev ns4eth3 -ip -net ns4 link set ns4eth3 up -ip -net ns4 route add default via 10.0.3.2 -$ipv6 && ip -net ns4 route add default via dead:beef:3::2 +ip -net "$ns1" link set ns1eth2 up +ip -net "$ns1" route add default via 10.0.1.2 +$ipv6 && ip -net "$ns1" route add default via dead:beef:1::2 + +ip -net "$ns2" addr add 10.0.1.2/24 dev ns2eth1 +$ipv6 && ip -net "$ns2" addr add dead:beef:1::2/64 dev ns2eth1 +ip -net "$ns2" link set ns2eth1 up + +ip -net "$ns2" addr add 10.0.2.1/24 dev ns2eth3 +$ipv6 && ip -net "$ns2" addr add dead:beef:2::1/64 dev ns2eth3 +ip -net "$ns2" link set ns2eth3 up +ip -net "$ns2" route add default via 10.0.2.2 +$ipv6 && ip -net "$ns2" route add default via dead:beef:2::2 +ip netns exec "$ns2" sysctl -q net.ipv4.ip_forward=1 +$ipv6 && ip netns exec "$ns2" sysctl -q net.ipv6.conf.all.forwarding=1 + +ip -net "$ns3" addr add 10.0.2.2/24 dev ns3eth2 +$ipv6 && ip -net "$ns3" addr add dead:beef:2::2/64 dev ns3eth2 +ip -net "$ns3" link set ns3eth2 up + +ip -net "$ns3" addr add 10.0.3.2/24 dev ns3eth4 +$ipv6 && ip -net "$ns3" addr add dead:beef:3::2/64 dev ns3eth4 +ip -net "$ns3" link set ns3eth4 up +ip -net "$ns3" route add default via 10.0.2.1 +$ipv6 && ip -net "$ns3" route add default via dead:beef:2::1 +ip netns exec "$ns3" sysctl -q net.ipv4.ip_forward=1 +$ipv6 && ip netns exec "$ns3" sysctl -q net.ipv6.conf.all.forwarding=1 + +ip -net "$ns4" addr add 10.0.3.1/24 dev ns4eth3 +$ipv6 && ip -net "$ns4" addr add dead:beef:3::1/64 dev ns4eth3 +ip -net "$ns4" link set ns4eth3 up +ip -net "$ns4" route add default via 10.0.3.2 +$ipv6 && ip -net "$ns4" route add default via dead:beef:3::2 set_ethtool_flags() { ns=$1
@@ -121,8 +128,8 @@ set_ethtool_flags() { [ $? -eq 0 ] && echo "INFO: set $ns dev $dev: ethtool -K $flags" } -set_ethtool_flags ns3 ns3eth2 -set_ethtool_flags ns4 ns4eth3 +set_ethtool_flags "$ns3" ns3eth2 +set_ethtool_flags "$ns4" ns4eth3 print_file_err() {
@@ -151,7 +158,7 @@ check_transfer() check_mptcp_disabled() { - disabled_ns="ns_disabled" + disabled_ns="ns_disabled-$sech-$(mktemp -u XXXXXX)" ip netns add ${disabled_ns} || exit $ksft_skip # net.mptcp.enabled should be enabled by default
@@ -179,7 +186,7 @@ check_mptcp_disabled() check_mptcp_ulp_setsockopt() { - local t="ns_ulp" retval= + local t="ns_ulp-$sech-$(mktemp -u XXXXXX)" retval= ip netns add ${t} || exit $ksft_skip if ! ip netns exec ${t} ./mptcp_connect -u -p 10000 -s TCP 127.0.0.1 2>&1; then
@@ -344,40 +351,40 @@ check_mptcp_ulp_setsockopt # Allow DAD to finish $ipv6 && sleep 2 -for sender in 1 2 3 4;do - do_ping ns1 ns$sender 10.0.1.1 +for sender in "$ns1" "$ns2" "$ns3" "$ns4";do + do_ping "$ns1" $sender 10.0.1.1 if $ipv6;then - do_ping ns1 ns$sender dead:beef:1::1 + do_ping "$ns1" $sender dead:beef:1::1 if [ $? -ne 0 ]; then echo "SKIP: IPv6 tests" 2>&1 ipv6=false fi fi - do_ping ns2 ns$sender 10.0.1.2 - $ipv6 && do_ping ns2 ns$sender dead:beef:1::2 - do_ping ns2 ns$sender 10.0.2.1 - $ipv6 && do_ping ns2 ns$sender dead:beef:2::1 + do_ping "$ns2" $sender 10.0.1.2 + $ipv6 && do_ping "$ns2" $sender dead:beef:1::2 + do_ping "$ns2" $sender 10.0.2.1 + $ipv6 && do_ping "$ns2" $sender dead:beef:2::1 - do_ping ns3 ns$sender 10.0.2.2 - $ipv6 && do_ping ns3 ns$sender dead:beef:2::2 - do_ping ns3 ns$sender 10.0.3.2 - $ipv6 && do_ping ns3 ns$sender dead:beef:3::2 + do_ping "$ns3" $sender 10.0.2.2 + $ipv6 && do_ping "$ns3" $sender dead:beef:2::2 + do_ping "$ns3" $sender 10.0.3.2 + $ipv6 && do_ping "$ns3" $sender dead:beef:3::2 - do_ping ns4 ns$sender 10.0.3.1 - $ipv6 && do_ping ns4 ns$sender dead:beef:3::1 + do_ping ns4 $sender 10.0.3.1 + $ipv6 && do_ping "$ns4" $sender dead:beef:3::1 done loss=$((RANDOM%101)) if [ $loss -eq 100 ] ;then loss=1% - tc -net ns2 qdisc add dev ns2eth3 root netem loss random $loss + tc -net "$ns2" qdisc add dev ns2eth3 root netem loss random $loss elif [ $loss -ge 10 ]; then loss=0.$loss% - tc -net ns2 qdisc add dev ns2eth3 root netem loss random $loss + tc -net "$ns2" qdisc add dev ns2eth3 root netem loss random $loss elif [ $loss -ge 1 ]; then loss=0.0$loss% - tc -net ns2 qdisc add dev ns2eth3 root netem loss random $loss + tc -net "$ns2" qdisc add dev ns2eth3 root netem loss random $loss fi delay=$((RANDOM%1000))
@@ -386,35 +393,35 @@ reorder1=$((100 - reorder1)) reorder2=$((RANDOM%100)) if [ $reorder1 -lt 100 ] && [ $reorder2 -gt 0 ]; then - tc -net ns3 qdisc add dev ns3eth4 root netem delay ${delay}ms reorder ${reorder1}% ${reorder2}% + tc -net "$ns3" qdisc add dev ns3eth4 root netem delay ${delay}ms reorder ${reorder1}% ${reorder2}% echo "INFO: Using loss of $loss, delay $delay ms, reorder: $reorder1, $reorder2 on ns3eth4" elif [ $delay -gt 0 ]; then - tc -net ns3 qdisc add dev ns3eth4 root netem delay ${delay}ms + tc -net "$ns3" qdisc add dev ns3eth4 root netem delay ${delay}ms echo "INFO: Using loss of $loss, delay $delay ms on ns3eth4" fi -for sender in 1 2 3 4;do - run_tests ns1 ns$sender 10.0.1.1 0.0.0.0 +for sender in $ns1 $ns2 $ns3 $ns4;do + run_tests "$ns1" $sender 10.0.1.1 0.0.0.0 if $ipv6;then - run_tests ns1 ns$sender dead:beef:1::1 :: + run_tests "$ns1" $sender dead:beef:1::1 :: if [ $? -ne 0 ] ;then echo "SKIP: IPv6 tests" 2>&1 ipv6=false fi fi - run_tests ns2 ns$sender 10.0.1.2 0.0.0.0 - $ipv6 && run_tests ns2 ns$sender dead:beef:1::2 :: - run_tests ns2 ns$sender 10.0.2.1 0.0.0.0 - $ipv6 && run_tests ns2 ns$sender dead:beef:2::1 :: + run_tests "$ns2" $sender 10.0.1.2 0.0.0.0 + $ipv6 && run_tests "$ns2" $sender dead:beef:1::2 :: + run_tests "$ns2" $sender 10.0.2.1 0.0.0.0 + $ipv6 && run_tests "$ns2" $sender dead:beef:2::1 :: - run_tests ns3 ns$sender 10.0.2.2 0.0.0.0 - $ipv6 && run_tests ns3 ns$sender dead:beef:2::2 :: - run_tests ns3 ns$sender 10.0.3.2 0.0.0.0 - $ipv6 && run_tests ns3 ns$sender dead:beef:3::2 :: + run_tests "$ns3" $sender 10.0.2.2 0.0.0.0 + $ipv6 && run_tests "$ns3" $sender dead:beef:2::2 :: + run_tests "$ns3" $sender 10.0.3.2 0.0.0.0 + $ipv6 && run_tests "$ns3" $sender dead:beef:3::2 :: - run_tests ns4 ns$sender 10.0.3.1 0.0.0.0 - $ipv6 && run_tests ns4 ns$sender dead:beef:3::1 :: + run_tests "$ns4" $sender 10.0.3.1 0.0.0.0 + $ipv6 && run_tests "$ns4" $sender dead:beef:3::1 :: done exit $ret
Signature · 1 line
2.21.0
Florian Westphal
PATCH v3 6/7 selftests: fix spurious deadlock
In reply to Florian Westphaldo_rnd_write() may call write on the mptcp socket repeatedly. As the peerfd is blocking, this prevents the process from draining the receive buffer.
Its unlikely but possible that both sides are blocked in write() on the tcp socket, with both ends announcing a zero window, waiting for read() to drain the socket receive buffer.
An alternative fix would be to make the peerfds nonblocking. This change instead makes it so we call write at most once after poll revents has the POLLOUT bit set.
--- .../selftests/net/mptcp/mptcp_connect.c | 53 ++++++++++--------- 1 file changed, 28 insertions(+), 25 deletions(-)
tools/testing/selftests/net/mptcp/mptcp_connect.c +28 -25
@@ -207,27 +207,18 @@ static int sock_connect_mptcp(const char * const remoteaddr,static size_t do_rnd_write (const int fd,char *buf,const size_t len) { -size_t offset =0 ; - -while (offset < len) { -unsigned int do_w; -size_t written; -ssize_t bw; +unsigned int do_w; +ssize_t bw; - do_w =rand () &0xffff ; -if (do_w ==0 || do_w > (len - offset)) - do_w = len - offset; + do_w =rand () &0xffff ; +if (do_w ==0 || do_w > len) + do_w = len; - bw =write (fd, buf + offset, do_w); -if (bw <0 ) { -perror ("write" ); -return 0 ; - } + bw =write (fd, buf, do_w); +if (bw <0 ) +perror ("write" ); - written = (size_t )bw; - offset += written; - } -return offset; +return bw; }static size_t do_write (const int fd,char *buf,const size_t len)
@@ -271,9 +262,11 @@ static int copyfd_io(int infd, int peerfd, int outfd) .fd = peerfd, .events = POLLIN | POLLOUT, }; +unsigned int woff =0 , wlen =0 ; +char wbuf[8192 ];for (;;) { -char buf[8192 ]; +char rbuf[8192 ];ssize_t len;if (fds.events ==0 )
@@ -293,7 +286,7 @@ static int copyfd_io(int infd, int peerfd, int outfd) }if (fds.revents &POLLIN ) { - len =do_rnd_read (peerfd, buf,sizeof (buf)); + len =do_rnd_read (peerfd, rbuf,sizeof (rbuf));if (len ==0 ) {/* no more data to receive: * peer has closed its write side
@@ -310,15 +303,25 @@ static int copyfd_io(int infd, int peerfd, int outfd)return 3 ; } -do_write (outfd, buf, len); +do_write (outfd, rbuf, len); }if (fds.revents &POLLOUT ) { - len =do_rnd_read (infd, buf,sizeof (buf)); -if (len >0 ) { -if (!do_rnd_write (peerfd, buf, len)) +if (wlen ==0 ) { + woff =0 ; + wlen =read (infd, wbuf,sizeof (wbuf)); + } + +if (wlen >0 ) { +ssize_t bw; + + bw =do_rnd_write (peerfd, wbuf + woff, wlen); +if (bw <0 )return 111 ; - }else if (len ==0 ) { + + woff += bw; + wlen -= bw; + }else if (wlen ==0 ) {/* We have no more data to send. */ fds.events &= ~POLLOUT ;
Signature · 1 line
2.21.0
Florian Westphal
PATCH v3 7/7 selftests: mptcp: track duration of each test
In reply to Florian WestphalThis allows to detect discrepancies, e.g.
MPTCP -> TCP 1s
TCP -> MPTCP 0s
MPTCP -> MPTCP 41s
The expecation of course is that test duration is pretty similar for tests between the same network namespaces, regardless of wheter MPTCP or TCP fallback is used.
--- tools/testing/selftests/net/mptcp/mptcp_connect.sh | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
tools/testing/selftests/net/mptcp/mptcp_connect.sh +7 -2
@@ -256,6 +256,7 @@ do_transfer() sleep 1 + start=$(date +%s) ip netns exec ${connector_ns} ./mptcp_connect -t $timeout -p $port -s ${cl_proto} $connect_addr < "$cin" > "$cout" & cpid=$!
@@ -264,13 +265,17 @@ do_transfer() wait $spid rets=$? + stop=$(date +%s) + if [ $capture -eq 1 ]; then sleep 1 kill $cappid fi + duration=$((stop-start)) + duration=$(printf "(duration %03ss)" $duration) if [ ${rets} -ne 0 ] || [ ${retc} -ne 0 ]; then - echo "[ FAIL ] client exit code $retc, server $rets" 1>&2 + echo "$duration [ FAIL ] client exit code $retc, server $rets" 1>&2 echo "\nnetns ${listener_ns} socket stat for $port:" 1>&2 ip netns exec ${listener_ns} ss -nita 1>&2 -o "sport = :$port" echo "\nnetns ${connector_ns} socket stat for $port:" 1>&2
@@ -286,7 +291,7 @@ do_transfer() rets=$? if [ $retc -eq 0 ] && [ $rets -eq 0 ];then - echo "[ OK ]" + echo "$duration [ OK ]" cat "$capout" return 0 fi
Signature · 1 line
2.21.0
Mat Martineau
Quoted 30 lines · Florian Westphal
This is resend of my selftest changes. All commits may be sqaushed to "mptcp: add basic kselftest for mptcp".
1. make tc delays random. This is to make sure mptcp-level retransmits don't trigger more frequent when delay is large.
Note that the existing selftests always test low delay in the loopback test case and in the ns1 <-> ns2 case, because there is no artificial delay in both cases.2. add accept/getpeername checks to make sure the sockaddr returned from kernel is the expected endpoint.
3. add ipv6 capability to the test.
This was tested by changing the test script to use TCP <-> TCP and setting both capture=1 ipv6=true in the script.
The tcpdump shows ipv6 is used.4. use a random combination of tso/gso/gro. Presence (or lack) of any of them cause slight changes in behaviour, we should make sure include as many combinations as possible.
Because test duration is already large (~3m 45s in my vm), don't add more test cases but instead make it use random combination.5. allow parallel execution of the test case. This allows you to run e.g. 10 parallel instances instead of waiting for sequential completion.
6. Fix a possible deadlock in the test program where both ends block on
write(), waiting for other side to drain the receive buffer.
Quoted 8 lines
7. track duration of each test. This is useful to spot bugs in mptcp:
if MPTCP <-> MPTCP test takes 31 seconds but TCP fallback took 3, something is wrong.
With current export branch, I saw no absurd divergences, so mptcp seems fine right now.Average test run time is about 2 minutes after these changes.
Other than the extra variables in patch 2 (see the reply to that message), I think the series is ok to merge.
--
Mat Martineau
Intel