Skip to contentSkip to conversation

Speedup TCP remote hosts connections

PATCH New Vladislav Grishenko 3 participants 10 messages Sep 27, 2020 – Oct 5, 2020

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Speedup TCP remote hosts connections

For non-blocking TCP/Unix connection, OpenVPN checks was it established in loop and if not - sleeps or handles management for next one second. Since the first check is made right after the connection attempt, it will likely be always unsuccessful, resulting in redundant wait for one or more seconds:

    00:00:00.667607 fcntl(5, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
    00:00:00.667713 connect(5, {...}, 16) = -1 EINPROGRESS (Operation now in progress)
    00:00:00.667832 poll([{fd=5, events=POLLOUT}], 1, 0) = 0 (Timeout)
    00:00:00.667954 nanosleep({tv_sec=1, tv_nsec=0}, 0x7fff52450270) = 0
    00:00:01.668608 poll([{fd=5, events=POLLOUT}], 1, 0) = 1 ([{fd=5, revents=POLLOUT}])

With this patch openvpn_connect() will wait for connection establishment for the first second (if possible) and will return success immediately: this speedups connection and provides real connection time that can be used for detection of the fastest remote server in subsequent patches:

    00:00:00.019133 fcntl(5, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
    00:00:00.019265 connect(5, {...}, 16) = -1 EINPROGRESS (Operation now in progress)
    00:00:00.019388 poll([{fd=5, events=POLLOUT}], 1, 1000) = 1 ([{fd=5, revents=POLLOUT}])

If connection still can't established - this should be treated as either too slow/far or non-responding server, so imprecise connection checks every next one second in loop will be performed as usual.

---
 src/openvpn/socket.c | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)
src/openvpn/socket.c +19 -1
index 76bdbfc5..66b1e4e2 100644
@@ -1464,13 +1464,26 @@ openvpn_connect(socket_descriptor_t sd,
 #endif
         )
     {
+        /* Non-blocking connection is in the progress and can be likely
+         * established within the first second with no need for imprecise
+         * sleeping in loop.
+         */
+        bool connect_wait = (connect_timeout > 0);
         while (true)
         {
 #if POLL
+            int timeout = 0;
             struct pollfd fds[1];
+
             fds[0].fd = sd;
             fds[0].events = POLLOUT;
-            status = poll(fds, 1, 0);
+            if (connect_wait)
+            {
+                connect_wait = false;
+                timeout = 1000;
+            }
+
+            status = poll(fds, 1, timeout);
 #else
             fd_set writes;
             struct timeval tv;
@@ -1479,6 +1492,11 @@ openvpn_connect(socket_descriptor_t sd,
             openvpn_fd_set(sd, &writes);
             tv.tv_sec = 0;
             tv.tv_usec = 0;
+            if (connect_wait)
+            {
+                connect_wait = false;
+                tv.tv_sec = 1;
+            }
 
             status = select(sd + 1, NULL, &writes, NULL, &tv);
 #endif
Signature · 1 line
2.17.1
#1 lore raw

PATCH 1/1 Speedup TCP remote hosts connections

For non-blocking TCP/Unix connection, OpenVPN checks was it established in loop and if not - sleeps or handles management for next one second. Since the first check is made right after the connection attempt, it will likely be always unsuccessful, resulting in redundant wait for one or more seconds:

    00:00:00.667607 fcntl(5, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
    00:00:00.667713 connect(5, {...}, 16) = -1 EINPROGRESS (Operation now in progress)
    00:00:00.667832 poll([{fd=5, events=POLLOUT}], 1, 0) = 0 (Timeout)
    00:00:00.667954 nanosleep({tv_sec=1, tv_nsec=0}, 0x7fff52450270) = 0
    00:00:01.668608 poll([{fd=5, events=POLLOUT}], 1, 0) = 1 ([{fd=5, revents=POLLOUT}])

With this patch openvpn_connect() will wait for connection establishment for the first second (if possible) and will return success immediately: this speedups connection and provides real connection time that can be used for detection of the fastest remote server in subsequent patches:

    00:00:00.019133 fcntl(5, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
    00:00:00.019265 connect(5, {...}, 16) = -1 EINPROGRESS (Operation now in progress)
    00:00:00.019388 poll([{fd=5, events=POLLOUT}], 1, 1000) = 1 ([{fd=5, revents=POLLOUT}])

If connection still can't established - this should be treated as either too slow/far or non-responding server, so imprecise connection checks every next one second in loop will be performed as usual.

v2: cosmetics, decrease connection_timeout to avoid wait more than it in total

---
 src/openvpn/socket.c | 22 +++++++++++++++++++++-
 1 file changed, 21 insertions(+), 1 deletion(-)
src/openvpn/socket.c +21 -1
index 76bdbfc5..049216ff 100644
@@ -1464,13 +1464,27 @@ openvpn_connect(socket_descriptor_t sd,
 #endif
         )
     {
+        /* Non-blocking connection is in the progress and can be likely
+         * established within the first second with no need for imprecise
+         * sleeping in loop.
+         */
+        bool connect_wait = (connect_timeout > 0);
         while (true)
         {
 #if POLL
+            int timeout = 0;
             struct pollfd fds[1];
+
             fds[0].fd = sd;
             fds[0].events = POLLOUT;
-            status = poll(fds, 1, 0);
+            if (connect_wait)
+            {
+                connect_wait = false;
+                connect_timeout--;
+                timeout = 1000;
+            }
+
+            status = poll(fds, 1, timeout);
 #else
             fd_set writes;
             struct timeval tv;
@@ -1479,6 +1493,12 @@ openvpn_connect(socket_descriptor_t sd,
             openvpn_fd_set(sd, &writes);
             tv.tv_sec = 0;
             tv.tv_usec = 0;
+            if (connect_wait)
+            {
+                connect_wait = false;
+                connect_timeout--;
+                tv.tv_sec = 1;
+            }
 
             status = select(sd + 1, NULL, &writes, NULL, &tv);
 #endif
Signature · 1 line
2.17.1
#2 lore raw

Am 28.09.20 um 01:32 schrieb Vladislav Grishenko:

Quoted 4 lines

For non-blocking TCP/Unix connection, OpenVPN checks was it established in loop and if not - sleeps or handles management for next one second. Since the first check is made right after the connection attempt, it will likely be always unsuccessful, resulting in redundant wait for one or more seconds:

Okay, I looked at the code and you are right. That code is a mess. It basically calls select/pool with a timeout of 0, then actively does select/pool in management for 1s and does this in a loop. That this totally wrong indeed.

I think we should be more radical here. Always do a 1s poll/select timeout and do a 0s management sleep/poll. That will also make the code a bit simpler.

Arne

#3 lore raw

Hi Arne,

Quoted 6 lines

Okay, I looked at the code and you are right. That code is a mess. It basically calls select/pool with a timeout of 0, then actively does select/pool in management for 1s and does this in a loop. That this totally wrong indeed.

I think we should be more radical here. Always do a 1s poll/select timeout and do a 0s management sleep/poll. That will also make the code a bit simpler.

Othe thing, management_sleep(0) results in infinite wait, so I;ve changed api in subsequent patch.
Now, management_sleep() can handle only non-negative value and managemet_event_loop_n_seconds() can take negative values to have infinite wait, if necessary. Since there were no negative or zero parameter for management_sleep() users, no side effect behavior change is expected.
Seems, a bit simpler can't be achieved :)

--
Best Regards, Vladislav Grishenko

#4 lore raw

PATCH v3 1/1 Speedup TCP remote hosts connections

In reply to Vladislav Grishenko

For non-blocking TCP/Unix connection, OpenVPN checks was it established in loop and if not - sleeps or handles management for next one second. Since the first check is made right after the connection attempt, it will likely be always unsuccessful, causing redundant wait for one or more seconds:

    00:00:00.667607 fcntl(5, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
    00:00:00.667713 connect(5, {...}, 16) = -1 EINPROGRESS (Operation now in progress)
    00:00:00.667832 poll([{fd=5, events=POLLOUT}], 1, 0) = 0 (Timeout)
    00:00:00.667954 nanosleep({tv_sec=1, tv_nsec=0}, 0x7fff52450270) = 0
    00:00:01.668608 poll([{fd=5, events=POLLOUT}], 1, 0) = 1 ([{fd=5, revents=POLLOUT}])

After this patch openvpn_connect() will perform blocking wait for connection establishment (if possible) and just check for management events once in one second (if management enabled) w/o sleep. This speedups TCP/Unix connection establishment and provides almost real connection time that can be used for detection of the fastest remote server in subsequent patches:

    00:00:00.790510 fcntl(5, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
    00:00:00.790616 connect(5, {...}, 16) = -1 EINPROGRESS (Operation now in progress)
    00:00:00.790877 poll([{fd=5, events=POLLOUT}], 1, 1000) = 0 (Timeout)
    00:00:01.792880 poll([{fd=5, events=POLLOUT}], 1, 1000) = 1 ([{fd=5, revents=POLLOUT}])

Or, with management interface enabled:

    00:00:00.906421 fcntl(5, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
    00:00:00.906527 connect(6, {...}, 16) = -1 EINPROGRESS (Operation now in progress)
    00:00:00.906779 poll([{fd=6, events=POLLOUT}], 1, 1000) = 0 (Timeout)
    00:00:01.910418 poll([{fd=3, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
    00:00:01.911365 poll([{fd=6, events=POLLOUT}], 1, 1000) = 0 ([{fd=6, revents=POLLOUT}])

v2: cosmetics, decrease connection_timeout to avoid wait more than it
v3: teach management_sleep() to handle zero timeout and reject negative
use 1s timeout for connection and 0s timeout for management events

---
 src/openvpn/manage.c | 30 +++++++++++++++++++++++-------
 src/openvpn/socket.c |  6 +++---
 2 files changed, 26 insertions(+), 10 deletions(-)
src/openvpn/manage.c +23 -7
index 898cb3b3..ac142177 100644
@@ -3310,12 +3310,17 @@ man_block(struct management *man, volatile int *signal_received, const time_t ex
 
     if (man_standalone_ok(man))
     {
+        /* expire time can be already overdue, for this case init zero
+         * timeout to avoid waiting first time and exit loop early with
+         * either obtained event or timeout.
+         */
+        tv.tv_usec = 0;
+        tv.tv_sec = 0;
+
         while (true)
         {
             event_reset(man->connection.es);
             management_socket_set(man, man->connection.es, NULL, NULL);
-            tv.tv_usec = 0;
-            tv.tv_sec = 1;
             if (man_check_for_signals(signal_received))
             {
                 status = -1;
@@ -3343,6 +3348,10 @@ man_block(struct management *man, volatile int *signal_received, const time_t ex
                 }
                 break;
             }
+
+            /* wait one second more */
+            tv.tv_sec = 1;
+            tv.tv_usec = 0;
         }
     }
     return status;
@@ -3444,7 +3453,7 @@ management_event_loop_n_seconds(struct management *man, int sec)
 
         /* set expire time */
         update_time();
-        if (sec)
+        if (sec >= 0)
         {
             expire = now + sec;
         }
@@ -3474,7 +3483,7 @@ management_event_loop_n_seconds(struct management *man, int sec)
         /* revert state */
         man->persist.standalone_disabled = standalone_disabled_save;
     }
-    else
+    else if (sec > 0)
     {
         sleep(sec);
     }
@@ -4117,11 +4126,15 @@ log_history_ref(const struct log_history *h, const int index)
 void
 management_sleep(const int n)
 {
-    if (management)
+    if (n < 0)
+    {
+        return;
+    }
+    else if (management)
     {
         management_event_loop_n_seconds(management, n);
     }
-    else
+    else if (n > 0)
     {
         sleep(n);
     }
@@ -4132,7 +4145,10 @@ management_sleep(const int n)
 void
 management_sleep(const int n)
 {
-    sleep(n);
+    if (n > 0)
+    {
+        sleep(n);
+    }
 }
 
 #endif /* ENABLE_MANAGEMENT */
src/openvpn/socket.c +3 -3
index 76bdbfc5..155780e3 100644
@@ -1470,14 +1470,14 @@ openvpn_connect(socket_descriptor_t sd,
             struct pollfd fds[1];
             fds[0].fd = sd;
             fds[0].events = POLLOUT;
-            status = poll(fds, 1, 0);
+            status = poll(fds, 1, (connect_timeout > 0) ? 1000 : 0);
 #else
             fd_set writes;
             struct timeval tv;
 
             FD_ZERO(&writes);
             openvpn_fd_set(sd, &writes);
-            tv.tv_sec = 0;
+            tv.tv_sec = (connect_timeout > 0) ? 1 : 0;
             tv.tv_usec = 0;
 
             status = select(sd + 1, NULL, &writes, NULL, &tv);
@@ -1507,7 +1507,7 @@ openvpn_connect(socket_descriptor_t sd,
 #endif
                     break;
                 }
-                management_sleep(1);
+                management_sleep(0);
                 continue;
             }
 
Signature · 1 line
2.17.1
Acked-by on v3 1/1
#5 lore raw

Am 02.10.20 um 00:53 schrieb Vladislav Grishenko:

Quoted 30 lines

For non-blocking TCP/Unix connection, OpenVPN checks was it established in loop and if not - sleeps or handles management for next one second. Since the first check is made right after the connection attempt, it will likely be always unsuccessful, causing redundant wait for one or more seconds:

    00:00:00.667607 fcntl(5, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
    00:00:00.667713 connect(5, {...}, 16) = -1 EINPROGRESS (Operation now in progress)
    00:00:00.667832 poll([{fd=5, events=POLLOUT}], 1, 0) = 0 (Timeout)
    00:00:00.667954 nanosleep({tv_sec=1, tv_nsec=0}, 0x7fff52450270) = 0
    00:00:01.668608 poll([{fd=5, events=POLLOUT}], 1, 0) = 1 ([{fd=5, revents=POLLOUT}])

After this patch openvpn_connect() will perform blocking wait for connection establishment (if possible) and just check for management events once in one second (if management enabled) w/o sleep. This speedups TCP/Unix connection establishment and provides almost real connection time that can be used for detection of the fastest remote server in subsequent patches:

    00:00:00.790510 fcntl(5, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
    00:00:00.790616 connect(5, {...}, 16) = -1 EINPROGRESS (Operation now in progress)
    00:00:00.790877 poll([{fd=5, events=POLLOUT}], 1, 1000) = 0 (Timeout)
    00:00:01.792880 poll([{fd=5, events=POLLOUT}], 1, 1000) = 1 ([{fd=5, revents=POLLOUT}])

Or, with management interface enabled:

    00:00:00.906421 fcntl(5, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
    00:00:00.906527 connect(6, {...}, 16) = -1 EINPROGRESS (Operation now in progress)
    00:00:00.906779 poll([{fd=6, events=POLLOUT}], 1, 1000) = 0 (Timeout)
    00:00:01.910418 poll([{fd=3, events=POLLIN|POLLPRI}], 1, 0) = 0 (Timeout)
    00:00:01.911365 poll([{fd=6, events=POLLOUT}], 1, 1000) = 0 ([{fd=6, revents=POLLOUT}])
Quoted 3 lines

v2: cosmetics, decrease connection_timeout to avoid wait more than it
v3: teach management_sleep() to handle zero timeout and reject negative
use 1s timeout for connection and 0s timeout for management events

I like this version much more. It doesn't add extra complexity.

#6 lore raw
In reply to Vladislav Grishenko

Your patch has been applied to the master branch.

I have run a number of TCP client connect tests, to a very close server (speedup very noticeable!), to distant server (still some difference), and to a non-reachable IP address (because I was worried that this would block out management, but it doesn't - it just shifts the "delay 1 s" to openvpn_connect(), and makes it a "max 1s" instead of a hard sleep).

I have not looked into manage.c as closely, but from a quick glance, this also looks reasonable.

Since we need a RC3 anyway (due to the redirect-gateway bug), and Arne's motto for 2.5 is "MAKE THINGS FAST!", I think this still fits into 2.5.

commit b68aa00603332357e6c711e91c5b4ba04d78294b (master) commit d08b66f39cf80733a51e1607386dc4993faef09f (release/2.5)

Author: Vladislav Grishenko
Date:   Fri Oct 2 03:53:19 2020 +0500
     Speedup TCP remote hosts connections

     Signed-off-by: Vladislav Grishenko <[email protected]...>
     Acked-by: Arne Schwabe <[email protected]...>
     Message-Id: <[email protected]...>
     URL: https://www.mail-archive.com/[email protected]/msg21139.html
     Signed-off-by: Gert Doering <[email protected]...>

--
kind regards,

Gert Doering

#7 lore raw

Hi Gert,

Thanks for that.
Perhaps same approach can be applied to server's tcp listening, would require testing of more management cases.

--
Best Regards, Vladislav Grishenko

Quoted 10 lines

-----Original Message-----
From: Gert Doering <[email protected]...>
Sent: Sunday, October 4, 2020 5:19 PM
To: Vladislav Grishenko <[email protected]...>
Cc: [email protected]
Subject: [PATCH applied] Re: Speedup TCP remote hosts connections

Your patch has been applied to the master branch.

I have run a number of TCP client connect tests, to a very close server

(speedup

Quoted 1 line

very noticeable!), to distant server (still some difference), and to a

non-

Quoted 2 lines

reachable IP address (because I was worried that this would block out management, but it doesn't - it just shifts the "delay 1 s" to

openvpn_connect(),

Quoted 3 lines

and makes it a "max 1s" instead of a hard sleep).

I have not looked into manage.c as closely, but from a quick glance, this

also

Quoted 24 lines

looks reasonable.

Since we need a RC3 anyway (due to the redirect-gateway bug), and Arne's motto for 2.5 is "MAKE THINGS FAST!", I think this still fits into 2.5.

commit b68aa00603332357e6c711e91c5b4ba04d78294b (master) commit d08b66f39cf80733a51e1607386dc4993faef09f (release/2.5)

Author: Vladislav Grishenko
Date:   Fri Oct 2 03:53:19 2020 +0500
     Speedup TCP remote hosts connections

     Signed-off-by: Vladislav Grishenko <[email protected]...>
     Acked-by: Arne Schwabe <[email protected]...>
     Message-Id: <[email protected]...>
     URL: https://www.mail-archive.com/openvpn-
[email protected]/msg21139.html
     Signed-off-by: Gert Doering <[email protected]...>

--
kind regards,

Gert Doering

#8 lore raw

Hi,

Quoted 2 lines · Vladislav Grishenko

Perhaps same approach can be applied to server's tcp listening, would require testing of more management cases.

There's two code paths here

--tcp-server

and

--mode server + --tcp

I think the "mode server" code path is already very quick (= if I test with your fast TCP client, against a "git master" server, I get 0.16s connection setup time). This is what people use on "more than one client" servers, so it's already fine.

The "--tcp-server" code path is "point to point". Not sure we currently test this at all (I have UDP p2p instance, but no TCP yet... seems I need to add one). This might indeed be slow, I had a look at the accept() path in socket.c recently and was wondering "how can this work at all?" - but that's "socket.c" vs. "mtcp.c", I think.

Since this is somewhat of a niche case, I'd put that into the "2.6" bin :-)
- the other one, TCP on the client, is much more heavily used, so 2.5 for that was appropriate.

gert

Signature · 6 lines
"If was one thing all people took for granted, was conviction that if you 
 feed honest figures into a computer, honest figures come out. Never doubted 
 it myself till I met a computer with a sense of humor."
                             Robert A. Heinlein, The Moon is a Harsh Mistress

Gert Doering - Munich, Germany                             [email protected]...
#9 lore raw

Hi Gert,

Quoted 1 line

"--tcp-server"

Yep, mean it, even poll doesn't used there. Have no any prio about it tho, just related thoughts.

--
Best Regards, Vladislav Grishenko

Quoted 16 lines

-----Original Message-----
From: Gert Doering <[email protected]...>
Sent: Monday, October 5, 2020 10:28 PM
To: Vladislav Grishenko <[email protected]...>
Cc: 'Gert Doering' <[email protected]...>; openvpn- [email protected]
Subject: Re: [PATCH applied] Re: Speedup TCP remote hosts connections

Hi,

Quoted 2 lines · Vladislav Grishenko

Perhaps same approach can be applied to server's tcp listening, would require testing of more management cases.

There's two code paths here

Quoted 7 lines

--tcp-server

and

--mode server + --tcp

I think the "mode server" code path is already very quick (= if I test

with your fast

Quoted 1 line

TCP client, against a "git master" server, I get 0.16s connection setup

time). This

Quoted 1 line

is what people use on "more than one client" servers, so it's already

fine.

Quoted 2 lines

The "--tcp-server" code path is "point to point". Not sure we currently

test this

Quoted 1 line

at all (I have UDP p2p instance, but no TCP yet... seems I need to add

one). This

Quoted 2 lines

might indeed be slow, I had a look at the
accept() path in socket.c recently and was wondering "how can this work at

all?"

Quoted 4 lines

- but that's "socket.c" vs. "mtcp.c", I think.

Since this is somewhat of a niche case, I'd put that into the "2.6" bin

:-)

Quoted 7 lines

- the other one, TCP on the client, is much more heavily used, so 2.5 for that was appropriate.

gert
--
"If was one thing all people took for granted, was conviction that if you
feed honest figures into a computer, honest figures come out. Never

doubted

Quoted 2 lines

it myself till I met a computer with a sense of humor."
Robert A. Heinlein, The Moon is a Harsh

Mistress

Quoted 2 lines

Gert Doering - Munich, Germany

[email protected]...