Skip to contentSkip to conversation

Various patches to improve DCO behaviour

PATCH In review Arne Schwabe 2 participants 21 messages Dec 24, 2022 – Dec 25, 2022

lore
Share & tools
lore
#0 lore raw

cover Various patches to improve DCO behaviour

This is a list of various patches that improve certain corner cases of DCO behaviour or improve. The remaining issues seem to be in the ovpn-dco module and need to fixed there first. Remaining issues in ovpn-dco:

- if there are a lot of event, the recvfrom the netlink socket will give a ENOMEM. Using Control-S and Control-Q to temporarily pause the OpenVPN daemon is a reliable way for me to trigger it.

- OpenVPN tcp sessions that are gone, especially when doing TCP reset are not notified to the user space

- With UDP and bombarding server already when starting up gives me a hard freeze

- Unload ovpn-dco sometimes does not work. Getting 'In use by xy ' failures.

- Latest FreeBSD dco module seems to be very broken. Reverting latest commit fixes it.

Arne Schwabe (9):
Rename TM_UNTRUSTED to TM_INITIAL
Always start session in TM_INITIAL rather than TM_ACTIVE or TM_INITIAL

These two could be smashed into one commit but two makes reviewing easier. They clean up the state machine a bit and by doing so fix an obscore corner case that would not work in p2p dco.

Move dco_installed back to link_socket from link_socket.info.actual

With the intention of moving back to using normal send/recv instead of using netlink, the special handling of figuring out which peer address uses wht mechansim is no longer necessary. Instead of hunting the bug down, just always use sendto when wanting to send something.

Ensure we do not promote a TA_TIMEOUT to a TA_WRITE/TA_READ event with
dco
Ensures that we do not go to a socket write/read on a DCO installed TCP socket.

Also drop incoming dco packet content when dropping the packet otherwise we would not clear the received packet and the dco code would refuse to put a new one there.

Do not set nl socket buffer size
This does not change anything and is more cosmetic.

Bail out when trying to install a TCP socket with residual data to DCO

This is something we could probably fix if we spent a lot more time and only ever read as little as possible (i.e. read 2 bytes to know next packet length, and then read only the length of the next packet). But instead of adding a lot of extra code, I rather want to wait for the plans of ovpn-dco and tcp and how the API evolves.

Improve logging when seeing a message for an unkown peer Ignore OVPN_DEL_PEER_REASON_USERSPACE to avoid race conditions

This commit fixes a race condition that lead to OpenVPN trying to add a peer that already existed.

 src/openvpn/dco.c        |  23 +-------
 src/openvpn/dco_linux.c  |   3 --
 src/openvpn/forward.c    |  27 ++--------
 src/openvpn/forward.h    |  30 +++++++++++
 src/openvpn/init.c       |   2 +-
 src/openvpn/mtcp.c       |  20 +++++--
 src/openvpn/mudp.c       |   2 +-
 src/openvpn/multi.c      |  45 +++++++++++++---
 src/openvpn/socket.c     |   8 +--
 src/openvpn/socket.h     |  11 ++--
 src/openvpn/ssl.c        | 113 +++++++++++++++------------------------
 src/openvpn/ssl.h        |   2 +-
 src/openvpn/ssl_common.h |   2 +-
 13 files changed, 144 insertions(+), 144 deletions(-)
Signature · 1 line
2.37.1 (Apple Git-137.1)
#1 lore raw

PATCH 1/9 Rename TM_UNTRUSTED to TM_INITIAL

---
 src/openvpn/ssl.c        | 16 ++++++++--------
 src/openvpn/ssl.h        |  2 +-
 src/openvpn/ssl_common.h |  2 +-
 3 files changed, 10 insertions(+), 10 deletions(-)
src/openvpn/ssl.c +8 -8
index 9e5480528..a5fb4fd22 100644
@@ -890,8 +890,8 @@ session_index_name(int index)
         case TM_ACTIVE:
             return "TM_ACTIVE";
 
-        case TM_UNTRUSTED:
-            return "TM_UNTRUSTED";
+        case TM_INITIAL:
+            return "TM_INITIAL";
 
         case TM_LAME_DUCK:
             return "TM_LAME_DUCK";
@@ -1330,7 +1330,7 @@ tls_multi_init_finalize(struct tls_multi *multi, int tls_mtu)
 
     if (!multi->opt.single_session)
     {
-        tls_session_init(multi, &multi->session[TM_UNTRUSTED]);
+        tls_session_init(multi, &multi->session[TM_INITIAL]);
     }
 }
 
@@ -3250,7 +3250,7 @@ tls_multi_process(struct tls_multi *multi,
     if (multi->multi_state >= CAS_CONNECT_DONE)
     {
         /* Only generate keys for the TM_ACTIVE session. We defer generating
-         * keys for TM_UNTRUSTED until we actually trust it.
+         * keys for TM_INITIAL until we actually trust it.
          * For TM_LAME_DUCK it makes no sense to generate new keys. */
         struct tls_session *session = &multi->session[TM_ACTIVE];
         struct key_state *ks = &session->key[KS_PRIMARY];
@@ -3299,9 +3299,9 @@ tls_multi_process(struct tls_multi *multi,
      * verification failed.  A semi-trusted session can forward data on the
      * TLS control channel but not on the tunnel channel.
      */
-    if (TLS_AUTHENTICATED(multi, &multi->session[TM_UNTRUSTED].key[KS_PRIMARY]))
+    if (TLS_AUTHENTICATED(multi, &multi->session[TM_INITIAL].key[KS_PRIMARY]))
     {
-        move_session(multi, TM_ACTIVE, TM_UNTRUSTED, true);
+        move_session(multi, TM_ACTIVE, TM_INITIAL, true);
         msg(D_TLS_DEBUG_LOW, "TLS: tls_multi_process: untrusted session promoted to %strusted",
             tas == TLS_AUTHENTICATION_SUCCEEDED ? "" : "semi-");
 
@@ -3720,7 +3720,7 @@ tls_pre_decrypt(struct tls_multi *multi,
             print_link_socket_actual(from, &gc));
 
         new_link = true;
-        i = TM_UNTRUSTED;
+        i = TM_INITIAL;
         session->untrusted_addr = *from;
     }
     else
@@ -3731,7 +3731,7 @@ tls_pre_decrypt(struct tls_multi *multi,
         /*
          * Packet must belong to an existing session.
          */
-        if (i != TM_ACTIVE && i != TM_UNTRUSTED)
+        if (i != TM_ACTIVE && i != TM_INITIAL)
         {
             msg(D_TLS_ERRORS,
                 "TLS Error: Unroutable control packet received from %s (si=%d op=%s)",
src/openvpn/ssl.h +1 -1
index 55c672d44..bd27e57a0 100644
@@ -159,7 +159,7 @@ struct tls_multi *tls_multi_init(struct tls_options *tls_options);
  * @ingroup control_processor
  *
  * This function initializes the \c TM_ACTIVE \c tls_session, and in
- * server mode also the \c TM_UNTRUSTED \c tls_session, associated with
+ * server mode also the \c TM_INITIAL \c tls_session, associated with
  * this \c tls_multi structure.  It also configures the control channel's
  * \c frame structure based on the data channel's \c frame given in
  * argument \a frame.
src/openvpn/ssl_common.h +1 -1
index 978a9fca0..7d9c2460b 100644
@@ -512,7 +512,7 @@ struct tls_session
  *
  *  @{ */
 #define TM_ACTIVE    0          /**< Active \c tls_session. */
-#define TM_UNTRUSTED 1          /**< As yet un-trusted \c tls_session
+#define TM_INITIAL   1          /**< As yet un-trusted \c tls_session
                                  *   being negotiated. */
 #define TM_LAME_DUCK 2          /**< Old \c tls_session. */
 #define TM_SIZE      3          /**< Size of the \c tls_multi.session
Signature · 1 line
2.37.1 (Apple Git-137.1)
Acked-by on v1 1/9
#2 lore raw

[Openvpn-devel] [PATCH applied] Re: Rename TM_UNTRUSTED to TM_INITIAL

This is trivial renaming, and makes sense together with 2/9.

I will also squash these two, because this alone does not compile... (one TM_UNTRUSTED in ssl.c survived The Great Rename).

Your patch will been applied to the master & release/2.6 branch, together with 2/9, if 2/9 passes stare-at-code and review.

commit eecf9ee30f27b34533fde963b256176829cfffcf (to be squashed)

Author: Arne Schwabe
Date:   Sat Dec 24 20:42:45 2022 +0100
     Rename TM_UNTRUSTED to TM_INITIAL

     Signed-off-by: Arne Schwabe <[email protected]...>
     Acked-by: Gert Doering <[email protected]...>
     Message-Id: <[email protected]...>
     URL: https://www.mail-archive.com/[email protected]/msg25798.html
     Signed-off-by: Gert Doering <[email protected]...>

--
kind regards,

Gert Doering

#3 lore raw

PATCH 2/9 Always start session in TM_INITIAL rather than TM_ACTIVE or TM_INITIAL

In reply to Arne Schwabe

Currently we start new session in TM_ACTIVE or TM_INITIAL depending if we already have an active session in TM_ACTIVE or not.

With this change, all session will be started in TM_INITIAL both initiated by a peer but also session by ourselves. This simplifies state transitions and eliminates the wacky state transition that when we have a failed reneogitiation (and move TM_ACTIVE to TM_LAME_DUCK) that a new session of a peer starts in TM_ACTIVE rather than TM_INITIAL

---
 src/openvpn/mudp.c |  2 +-
 src/openvpn/ssl.c  | 99 ++++++++++++++++------------------------------
 2 files changed, 36 insertions(+), 65 deletions(-)
src/openvpn/mudp.c +1 -1
index 458152335..c27c6da5b 100644
@@ -257,7 +257,7 @@ multi_get_create_instance_udp(struct multi_context *m, bool *floated)
                             && session_id_defined((&state.peer_session_id)))
                         {
                             mi->context.c2.tls_multi->n_sessions++;
-                            struct tls_session *session = &mi->context.c2.tls_multi->session[TM_ACTIVE];
+                            struct tls_session *session = &mi->context.c2.tls_multi->session[TM_INITIAL];
                             session_skip_to_pre_start(session, &state, &m->top.c2.from);
                         }
                     }
src/openvpn/ssl.c +35 -64

160 lines not on this page · raw message

Signature · 1 line
2.37.1 (Apple Git-137.1)
Acked-by on v1 2/9
#4 lore raw

[Openvpn-devel] [PATCH applied] Re: Always start session in TM_INITIAL rather than TM_ACTIVE or TM_INITIAL

"Simplifying state machine much good" :-)

I have stared at the code a bit ("seems to make sense") and fed this to the full test rig - extensive client side tests on Linux and FreeBSD, full set of server side tests on Linux (DCO and no DCO). Didn't test FreeBSD/DCO as this is really just TLS handshakes, and all the weirdness in the past related to TLS handshake have hit both platforms the same way.

I did have an extra eye on the p2p TLS tests that tended to fail if the timing was just right - repaired keepalive already fixed those, so I tried without keepalive, with the usual timing (reneg-sec 300, reconnect after 400s).

Dec 24 22:42:13 ubuntu2004 tun-udp-p2p-tls-sha256[1805147]: TLS: move_session: dest=TM_LAME_DUCK src=TM_ACTIVE reinit_src=1

.. but this still confuses the --tls-server...

Dec 24 22:43:14 ubuntu2004 tun-udp-p2p-tls-sha256[1805147]: TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
Dec 24 22:43:14 ubuntu2004 tun-udp-p2p-tls-sha256[1805147]: TLS Error: TLS handshake failed
Dec 24 22:43:20 ubuntu2004 tun-udp-p2p-tls-sha256[1805147]: TLS Error: Received control packet from unexpected IP addr: [AF_INET6]::ffff:194.97.140.5:49828

.. and it will then fail to establish connections. So this is no worse than without this patch (and no better). The server will eventually recover (after 3600s), but --keepalive will fix it as well.

Let's see if the "send UDP directly" patch will fix that one.

Your patch has been applied to the master and release/2.6 branch, squashed together with 1/9.

commit 7dcde87b7a4323ffb173576d4559e14fcfe4e627 (master) commit 9828c7045a27e7dc5e6f430798323a1abd003fbf (release/2.6)

Author: Arne Schwabe
Date:   Sat Dec 24 20:42:46 2022 +0100
     Always start session in TM_INITIAL rather than TM_ACTIVE or TM_INITIAL

     Signed-off-by: Arne Schwabe <[email protected]...>
     Acked-by: Gert Doering <[email protected]...>
     Message-Id: <[email protected]...>
     URL: https://www.mail-archive.com/[email protected]/msg25795.html
     Signed-off-by: Gert Doering <[email protected]...>

--
kind regards,

Gert Doering

#5 lore raw

PATCH 3/9 Move dco_installed back to link_socket from link_socket.info.actual

In reply to Arne Schwabe

this change was done in order to be able to differentiate when needing to use dco and when to use normal socket sendto. Since we want to eventually completely use the userspace sockets for sending/receiving, we just switch to always use UDP sendto even if the socket is already installed in the kernel.

---
 src/openvpn/dco.c     | 23 ++---------------------
 src/openvpn/forward.c | 13 +++++++------
 src/openvpn/init.c    |  2 +-
 src/openvpn/mtcp.c    |  6 +++---
 src/openvpn/socket.c  |  8 ++++----
 src/openvpn/socket.h  | 11 +++++------
 6 files changed, 22 insertions(+), 41 deletions(-)
src/openvpn/dco.c +2 -21
index 993265188..2f4d0f779 100644
@@ -456,22 +456,6 @@ dco_check_pull_options(int msglevel, const struct options *o)
     return true;
 }
 
-static void
-addr_set_dco_installed(struct context *c)
-{
-    /* We ensure that all addresses we currently hold have the dco_installed
-     * bit set */
-    for (int i = 0; i < KEY_SCAN_SIZE; ++i)
-    {
-        struct key_state *ks = get_key_scan(c->c2.tls_multi, i);
-        if (ks)
-        {
-            ks->remote_addr.dco_installed = true;
-        }
-    }
-    get_link_socket_info(c)->lsa->actual.dco_installed = true;
-}
-
 int
 dco_p2p_add_new_peer(struct context *c)
 {
@@ -484,8 +468,6 @@ dco_p2p_add_new_peer(struct context *c)
 
     ASSERT(ls->info.connection_established);
 
-    addr_set_dco_installed(c);
-
     struct sockaddr *remoteaddr = &ls->info.lsa->actual.dest.addr.sa;
     struct tls_multi *multi = c->c2.tls_multi;
 #ifdef TARGET_FREEBSD
@@ -505,7 +487,7 @@ dco_p2p_add_new_peer(struct context *c)
     }
 
     c->c2.tls_multi->dco_peer_id = multi->peer_id;
-    c->c2.link_socket->info.lsa->actual.dco_installed = true;
+    c->c2.link_socket->dco_installed = true;
 
     return 0;
 }
@@ -595,7 +577,6 @@ dco_multi_add_new_peer(struct multi_context *m, struct multi_instance *mi)
         ASSERT(c->c2.link_socket_info->connection_established);
         remoteaddr = &c->c2.link_socket_info->lsa->actual.dest.addr.sa;
     }
-    addr_set_dco_installed(c);
 
     /* In server mode we need to fetch the remote addresses from the push config */
     struct in_addr vpn_ip4 = { 0 };
@@ -633,7 +614,7 @@ dco_multi_add_new_peer(struct multi_context *m, struct multi_instance *mi)
         {
             msg(D_DCO|M_ERRNO, "error closing TCP socket after DCO handover");
         }
-        c->c2.link_socket->info.lsa->actual.dco_installed = true;
+        c->c2.link_socket->dco_installed = true;
         c->c2.link_socket->sd = SOCKET_UNDEFINED;
     }
 
src/openvpn/forward.c +7 -6
index c04511ee1..64c8ee6a0 100644
@@ -1674,9 +1674,10 @@ process_ip_header(struct context *c, unsigned int flags, struct buffer *buf)
     }
 }
 
-/* Linux DCO implementations pass the socket to the kernel and
- * disallow usage of it from userland, so (control) packets sent and
- * received by OpenVPN need to go through the DCO interface.
+/*
+ * Linux DCO implementations pass the socket to the kernel and
+ * disallow usage of it from userland for TCP, so (control) packets
+ * sent and received by OpenVPN need to go through the DCO interface.
  *
  * Windows DCO needs control packets to be sent via the normal
  * standard Overlapped I/O.
@@ -1688,10 +1689,10 @@ process_ip_header(struct context *c, unsigned int flags, struct buffer *buf)
  * in the future...) in a small inline function.
  */
 static inline bool
-should_use_dco_socket(struct link_socket_actual *actual)
+should_use_dco_socket(struct link_socket *ls)
 {
 #if defined(TARGET_LINUX)
-    return actual->dco_installed;
+    return ls->dco_installed && proto_is_tcp(ls->info.proto);
 #else
     return false;
 #endif
@@ -1770,7 +1771,7 @@ process_outgoing_link(struct context *c)
                 socks_preprocess_outgoing_link(c, &to_addr, &size_delta);
 
                 /* Send packet */
-                if (should_use_dco_socket(c->c2.to_link_addr))
+                if (should_use_dco_socket(c->c2.link_socket))
                 {
                     size = dco_do_write(&c->c1.tuntap->dco,
                                         c->c2.tls_multi->dco_peer_id,
src/openvpn/init.c +1 -1
index 409a8be2a..3380ed9e6 100644
@@ -3696,7 +3696,7 @@ do_close_link_socket(struct context *c)
      * closed in do_close_tun(). Set it to UNDEFINED so
      * we won't use WinSock API to close it. */
     if (tuntap_is_dco_win(c->c1.tuntap) && c->c2.link_socket
-        && c->c2.link_socket->info.lsa->actual.dco_installed)
+        && c->c2.link_socket->dco_installed)
     {
         c->c2.link_socket->sd = SOCKET_UNDEFINED;
     }
src/openvpn/mtcp.c +3 -3
index 07da15a6d..ac06ddc64 100644
@@ -402,7 +402,7 @@ multi_tcp_wait_lite(struct multi_context *m, struct multi_instance *mi, const in
 
     tv_clear(&c->c2.timeval); /* ZERO-TIMEOUT */
 
-    if (mi && mi->context.c2.link_socket->info.lsa->actual.dco_installed)
+    if (mi && mi->context.c2.link_socket->dco_installed)
     {
         /* If we got a socket that has been handed over to the kernel
          * we must not call the normal socket function to figure out
@@ -537,7 +537,7 @@ multi_tcp_dispatch(struct multi_context *m, struct multi_instance *mi, const int
 
         case TA_INITIAL:
             ASSERT(mi);
-            if (!mi->context.c2.link_socket->info.lsa->actual.dco_installed)
+            if (!mi->context.c2.link_socket->dco_installed)
             {
                 multi_tcp_set_global_rw_flags(m, mi);
             }
@@ -590,7 +590,7 @@ multi_tcp_post(struct multi_context *m, struct multi_instance *mi, const int act
             }
             else
             {
-                if (!c->c2.link_socket->info.lsa->actual.dco_installed)
+                if (!c->c2.link_socket->dco_installed)
                 {
                     multi_tcp_set_global_rw_flags(m, mi);
                 }
src/openvpn/socket.c +4 -4
index 82787f9f2..c7ec0e06d 100644
@@ -2147,7 +2147,7 @@ create_socket_dco_win(struct context *c, struct link_socket *sock,
                       get_server_poll_remaining_time(sock->server_poll_timeout),
                       signal_received);
 
-    sock->info.lsa->actual.dco_installed = true;
+    sock->dco_installed = true;
 
     if (*signal_received)
     {
@@ -3480,7 +3480,7 @@ link_socket_write_udp_posix_sendmsg(struct link_socket *sock,
 static int
 socket_get_last_error(const struct link_socket *sock)
 {
-    if (sock->info.lsa->actual.dco_installed)
+    if (sock->dco_installed)
     {
         return GetLastError();
     }
@@ -3521,7 +3521,7 @@ socket_recv_queue(struct link_socket *sock, int maxsize)
         ASSERT(ResetEvent(sock->reads.overlapped.hEvent));
         sock->reads.flags = 0;
 
-        if (sock->info.lsa->actual.dco_installed)
+        if (sock->dco_installed)
         {
             status = ReadFile((HANDLE)sock->sd, wsabuf[0].buf, wsabuf[0].len,
                               &sock->reads.size, &sock->reads.overlapped);
@@ -3626,7 +3626,7 @@ socket_send_queue(struct link_socket *sock, struct buffer *buf, const struct lin
         ASSERT(ResetEvent(sock->writes.overlapped.hEvent));
         sock->writes.flags = 0;
 
-        if (sock->info.lsa->actual.dco_installed)
+        if (sock->dco_installed)
         {
             status = WriteFile((HANDLE)sock->sd, wsabuf[0].buf, wsabuf[0].len,
                                &sock->writes.size, &sock->writes.overlapped);
src/openvpn/socket.h +5 -6
index 929ef8187..05c31b104 100644
@@ -88,7 +88,6 @@ struct link_socket_actual
     /*int dummy;*/ /* add offset to force a bug if dest not explicitly dereferenced */
 
     struct openvpn_sockaddr dest;
-    bool dco_installed;
 #if ENABLE_IP_PKTINFO
     union {
 #if defined(HAVE_IN_PKTINFO) && defined(HAVE_IPI_SPEC_DST)
@@ -169,6 +168,7 @@ struct link_socket
 
     socket_descriptor_t sd;
     socket_descriptor_t ctrl_sd; /* only used for UDP over Socks */
+    bool dco_installed;
 
 #ifdef _WIN32
     struct overlapped_io reads;
@@ -1036,7 +1036,7 @@ link_socket_read_udp_win32(struct link_socket *sock,
                            struct link_socket_actual *from)
 {
     sockethandle_t sh = { .s = sock->sd };
-    if (sock->info.lsa->actual.dco_installed)
+    if (sock->dco_installed)
     {
         *from = sock->info.lsa->actual;
         sh.is_handle = true;
@@ -1058,8 +1058,7 @@ link_socket_read(struct link_socket *sock,
                  struct buffer *buf,
                  struct link_socket_actual *from)
 {
-    if (proto_is_udp(sock->info.proto)
-        || sock->info.lsa->actual.dco_installed)
+    if (proto_is_udp(sock->info.proto) || sock->dco_installed)
     /* unified UDPv4 and UDPv6, for DCO the kernel
      * will strip the length header */
     {
@@ -1102,7 +1101,7 @@ link_socket_write_win32(struct link_socket *sock,
 {
     int err = 0;
     int status = 0;
-    sockethandle_t sh = { .s = sock->sd, .is_handle = sock->info.lsa->actual.dco_installed };
+    sockethandle_t sh = { .s = sock->sd, .is_handle = sock->dco_installed };
     if (overlapped_io_active(&sock->writes))
     {
         status = sockethandle_finalize(sh, &sock->writes, NULL, NULL);
@@ -1176,7 +1175,7 @@ link_socket_write(struct link_socket *sock,
                   struct buffer *buf,
                   struct link_socket_actual *to)
 {
-    if (proto_is_udp(sock->info.proto) || to->dco_installed)
+    if (proto_is_udp(sock->info.proto) || sock->dco_installed)
     {
         /* unified UDPv4 and UDPv6 and DCO (kernel adds size header) */
         return link_socket_write_udp(sock, buf, to);
Signature · 1 line
2.37.1 (Apple Git-137.1)
Acked-by on v1 3/9
#6 lore raw

[Openvpn-devel] [PATCH applied] Re: Move dco_installed back to link_socket from link_socket.info.actual

Tested on Linux/non-DCO (client and server) and DCO (client and server), especially focusing on p2p --tls-server, which was the original reason for moving dco_installed around. Tests still succeed, good :-)

Test without --keepalive on the --tls-server (11 11t) and that still gets confused, though it looks a bit different...

Dec 24 23:54:30 ubuntu2004 tun-udp-p2p-tls-sha256[1826875]: UDPv6 READ [14] from [AF_INET6]::ffff:194.97.140.5:51628: P_CONTROL_HARD_RESET_CLIENT_V2 kid=0 [ ] pid=0 DATA len=0
Dec 24 23:54:30 ubuntu2004 tun-udp-p2p-tls-sha256[1826875]: TLS Error: Received control packet from unexpected IP addr: [AF_INET6]::ffff:194.97.140.5:51628
Dec 24 23:54:30 ubuntu2004 tun-udp-p2p-tls-sha256[1826875]: UDPv6 WRITE [26] to [AF_INET6]::ffff:194.97.140.5:52318: P_CONTROL_HARD_RESET_SERVER_V2 kid=0 [ 0 ] pid=0 DATA len=0

..

Dec 24 23:55:38 ubuntu2004 tun-udp-p2p-tls-sha256[1826875]: UDPv6 READ [14] from [AF_INET6]2001:608:0:814::fb00:14:21675: P_CONTROL_HARD_RESET_CLIENT_V2 kid=0 [ ] pid=0 DATA len=0
Dec 24 23:55:38 ubuntu2004 tun-udp-p2p-tls-sha256[1826875]: TLS Error: Received control packet from unexpected IP addr: [AF_INET6]2001:608:0:814::fb00:14:21675
Dec 24 23:55:42 ubuntu2004 tun-udp-p2p-tls-sha256[1826875]: UDPv6 READ [14] from [AF_INET6]2001:608:0:814::fb00:14:21675: P_CONTROL_HARD_RESET_CLIENT_V2 kid=0 [ ] pid=0 DATA len=0
Dec 24 23:55:42 ubuntu2004 tun-udp-p2p-tls-sha256[1826875]: TLS Error: Received control packet from unexpected IP addr: [AF_INET6]2001:608:0:814::fb00:14:21675
Dec 24 23:55:42 ubuntu2004 tun-udp-p2p-tls-sha256[1826875]: UDPv6 WRITE [26] to [AF_INET6]::ffff:194.97.140.5:52318: P_CONTROL_HARD_RESET_SERVER_V2 kid=0 [ 0 ] pid=0 DATA len=0

.. and it still sticks to "send packets to the old address" - but going back to the test run before this patch, it's the same problem, just without "dco_do_write()" being involved. So, not better, but no worse either - and since this change brought other issues Arne was experiencing, this new approach seems to be a good thing.

The socket.h change breaks my bandaid patch, of which I'll send a v2 right away (because I still find that useful as an extra safety belt).

Your patch has been applied to the master branch.

commit 1413b38d0eacafb6c03c701236fe546f44f39a8d (master) commit b87f69cdaad54a73bf5029b5511d33db06014602 (release/2.6)

Author: Arne Schwabe
Date:   Sat Dec 24 20:42:47 2022 +0100
     Move dco_installed back to link_socket from link_socket.info.actual

     Signed-off-by: Arne Schwabe <[email protected]...>
     Acked-by: Gert Doering <[email protected]...>
     Message-Id: <[email protected]...>
     URL: https://www.mail-archive.com/[email protected]/msg25792.html
     Signed-off-by: Gert Doering <[email protected]...>

--
kind regards,

Gert Doering

#7 lore raw

PATCH 4/9 Ensure we do not promote a TA_TIMEOUT to a TA_WRITE/TA_READ event with dco

In reply to Arne Schwabe

with dco sometimes we end up promoting a timeout event to write event or read event. For the residual read, this problem is probably not solvable without changing the kernel DCO API

---
 src/openvpn/forward.c | 24 ------------------------
 src/openvpn/forward.h | 30 ++++++++++++++++++++++++++++++
 src/openvpn/mtcp.c    | 14 ++++++++++++--
 3 files changed, 42 insertions(+), 26 deletions(-)
src/openvpn/forward.c +0 -24
index 64c8ee6a0..17a14f0bd 100644
@@ -1674,30 +1674,6 @@ process_ip_header(struct context *c, unsigned int flags, struct buffer *buf)
     }
 }
 
-/*
- * Linux DCO implementations pass the socket to the kernel and
- * disallow usage of it from userland for TCP, so (control) packets
- * sent and received by OpenVPN need to go through the DCO interface.
- *
- * Windows DCO needs control packets to be sent via the normal
- * standard Overlapped I/O.
- *
- * FreeBSD DCO allows control packets to pass through the socket in both
- * directions.
- *
- * Hide that complexity (...especially if more platforms show up
- * in the future...) in a small inline function.
- */
-static inline bool
-should_use_dco_socket(struct link_socket *ls)
-{
-#if defined(TARGET_LINUX)
-    return ls->dco_installed && proto_is_tcp(ls->info.proto);
-#else
-    return false;
-#endif
-}
-
 /*
  * Input: c->c2.to_link
  */
src/openvpn/forward.h +30 -0
index bd2d96010..e50f235da 100644
@@ -424,4 +424,34 @@ connection_established(struct context *c)
     }
 }
 
+
+/**
+ * @param ls the link_socket the decision should be made for
+ * @return if we should use the dco kernel api or normal socket APIs for
+ *         write/send
+ *
+ *
+ * Linux DCO implementations pass the socket to the kernel and
+ * disallow usage of it from userland for TCP, so (control) packets
+ * sent and received by OpenVPN need to go through the DCO interface.
+ *
+ * Windows DCO needs control packets to be sent via the normal
+ * standard Overlapped I/O.
+ *
+ * FreeBSD DCO allows control packets to pass through the socket in both
+ * directions.
+ *
+ * Hide that complexity (...especially if more platforms show up
+ * in the future...) in a small inline function.
+ */
+static inline bool
+should_use_dco_socket(struct link_socket *ls)
+{
+#if defined(TARGET_LINUX)
+    return ls->dco_installed && proto_is_tcp(ls->info.proto);
+#else
+    return false;
+#endif
+}
+
 #endif /* FORWARD_H */
src/openvpn/mtcp.c +12 -2
index ac06ddc64..263f4d994 100644
@@ -407,7 +407,7 @@ multi_tcp_wait_lite(struct multi_context *m, struct multi_instance *mi, const in
         /* If we got a socket that has been handed over to the kernel
          * we must not call the normal socket function to figure out
          * if it is readable or writable */
-        /* Assert that we only have the DCO exptected flags */
+        /* Assert that we only have the DCO expected flags */
         ASSERT(action & (TA_SOCKET_READ | TA_SOCKET_WRITE));
 
         /* We are always ready! */
@@ -567,7 +567,7 @@ multi_tcp_post(struct multi_context *m, struct multi_instance *mi, const int act
     {
         flags |= MTP_TUN_OUT;
     }
-    if (LINK_OUT(c))
+    if (LINK_OUT(c) && !should_use_dco_socket(c->c2.link_socket))
     {
         flags |= MTP_LINK_OUT;
     }
@@ -586,6 +586,16 @@ multi_tcp_post(struct multi_context *m, struct multi_instance *mi, const int act
         case MTP_NONE:
             if (mi && socket_read_residual(c->c2.link_socket))
             {
+                if (should_use_dco_socket(c->c2.link_socket))
+                {
+                    struct gc_arena gc = gc_new();
+                    msg(M_INFO, "ovpn-dco installed socket with residual read "
+                        "len=%d, mi=%s. This connection will probably"
+                        " break.",  BLEN(&c->c2.link_socket->stream_buf.residual),
+                        multi_instance_string(mi, false, &gc));
+                    gc_free(&gc);
+                    break;
+                }
                 newaction = TA_SOCKET_READ_RESIDUAL;
             }
             else
Signature · 1 line
2.37.1 (Apple Git-137.1)
#8 lore raw

Re: [Openvpn-devel] [PATCH 4/9] Ensure we do not promote a TA_TIMEOUT to a TA_WRITE/TA_READ event with dco

Hi,

Quoted 5 lines · Arne Schwabe

with dco sometimes we end up promoting a timeout event to write event or read event. For the residual read, this problem is probably not solvable without changing the kernel DCO API

This one breaks TCP on a DCO-enabled server for me, though I can't see any obvious reason why. Basic p2mp tcp server, nothing fancy.

With "3/7" (1413b38d0eacafb6, as in master) it works fine.

With "4/7" (just this patch, on top of master) the "TCP gremlin test" looks good - that's what I started last night, and no crashes/BUGs - but an actual TCP client fails...

Client log:

2022-12-25 09:41:32 VERIFY OK: depth=0, C=US, ST=California, L=Pleasanton, O=OpenVPN community project, CN=server, [email protected]...
2022-12-25 09:41:32 WARNING: 'link-mtu' is used inconsistently, local='link-mtu 1547', remote='link-mtu 1544'
2022-12-25 09:41:32 Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, peer certificate: 2048 bit RSA, signature: RSA-SHA1
2022-12-25 09:41:32 [server] Peer Connection Initiated with [AF_INET]195.30.8.84:51194
2022-12-25 09:41:33 SENT CONTROL [server]: 'PUSH_REQUEST' (status=1)
2022-12-25 09:41:39 SENT CONTROL [server]: 'PUSH_REQUEST' (status=1)
2022-12-25 09:41:43 TLS Error: local/remote TLS keys are out of sync: [AF_INET]195.30.8.84:51194 (received key id: 0, known key ids:  [key#0 state=S_ACTIVE auth=KS_AUTH_TRUE id=0 sid=87065d72 c3889e05] [key#1 state=S_UNDEF auth=KS_AUTH_FALSE id=0 sid=00000000 00000000] [key#2 state=S_UNDEF auth=KS_AUTH_FALSE id=0 sid=00000000 00000000])
2022-12-25 09:41:44 SENT CONTROL [server]: 'PUSH_REQUEST' (status=1)
2022-12-25 09:41:50 SENT CONTROL [server]: 'PUSH_REQUEST' (status=1)
2022-12-25 09:41:53 TLS Error: local/remote TLS keys are out of sync: [AF_INET]195.30.8.84:51194 (received key id: 0, known key ids:  [key#0 state=S_ACTIVE auth=KS_AUTH_TRUE id=0 sid=87065d72 c3889e05] [key#1 state=S_UNDEF auth=KS_AUTH_FALSE id=0 sid=00000000 00000000] [key#2 state=S_UNDEF auth=KS_AUTH_FALSE id=0 sid=00000000 00000000])
2022-12-25 09:41:55 SENT CONTROL [server]: 'PUSH_REQUEST' (status=1)

server log

2022-12-25 09:41:32 us=809562 TCP connection established with [AF_INET6]::ffff:194.97.140.5:28517
2022-12-25 09:41:32 us=809600 TCPv6_SERVER link local: (not bound)
2022-12-25 09:41:32 us=809615 TCPv6_SERVER link remote: [AF_INET6]::ffff:194.97.140.5:28517
2022-12-25 09:41:32 us=809775 194.97.140.5:28517 TLS: Initial packet from [AF_INET6]::ffff:194.97.140.5:28517, sid=130a42ff 79d3e7e6
2022-12-25 09:41:32 us=822866 194.97.140.5:28517 VERIFY OK: depth=1, C=US, ST=California, L=Pleasanton, O=OpenVPN community project, CN=OpenVPN community project CA, [email protected]...
2022-12-25 09:41:32 us=823211 194.97.140.5:28517 VERIFY OK: depth=0, C=DE, ST=Bavaria, L=Munich, O=OpenVPN community project, OU=DCO F14 client, CN=freebsd-14-amd64, [email protected]...
...
2022-12-25 09:41:32 us=824045 194.97.140.5:28517 TLS: move_session: dest=TM_ACTIVE src=TM_INITIAL reinit_src=1
2022-12-25 09:41:32 us=824186 194.97.140.5:28517 TLS: tls_multi_process: initial untrusted session promoted to trusted
2022-12-25 09:41:32 us=863377 194.97.140.5:28517 Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, peer certificate: 2048 bit RSA, signature: RSA-SHA256
2022-12-25 09:41:32 us=863449 194.97.140.5:28517 [freebsd-14-amd64] Peer Connection Initiated with [AF_INET6]::ffff:194.97.140.5:28517
...
2022-12-25 09:41:32 us=863895 freebsd-14-amd64/194.97.140.5:28517 Data Channel: using negotiated cipher 'AES-256-GCM'
2022-12-25 09:41:32 us=863935 freebsd-14-amd64/194.97.140.5:28517 Data Channel MTU parms [ mss_fix:1366 max_frag:0 tun_mtu:1500 tun_max_mtu:1600 headroom:136 payload:1768 tailroom:562 ET:0 ]
2022-12-25 09:41:32 us=864176 freebsd-14-amd64/194.97.140.5:28517 SENT CONTROL [freebsd-14-amd64]: 'PUSH_REPLY,route 10.220.0.0 255.255.0.0,route-ipv6 fd00:abcd:220::/48,tun-ipv6,route-gateway 10.220.1.1,topology subnet,ping 10,ping-restart 30,compress stub-v2,ifconfig-ipv6 fd00:abcd:220:1::1002/64 fd00:abcd:220:1::1,ifconfig 10.220.1.4 255.255.255.0,peer-id 0,cipher AES-256-GCM,key-derivation tls-ekm' (status=1)

and then nothing more, until the client gives up and reconnects.

With --verb 6, I see this dance...

2022-12-25 09:45:51 us=194027 freebsd-14-amd64/194.97.140.5:47015 Data Channel: using negotiated cipher 'AES-256-GCM'
2022-12-25 09:45:51 us=194058 freebsd-14-amd64/194.97.140.5:47015 Data Channel MTU parms [ mss_fix:1366 max_frag:0 tun_mtu:1500 tun_max_mtu:1600 headroom:136 payload:1768 tailroom:562 ET:0 ]
2022-12-25 09:45:51 us=194134 freebsd-14-amd64/194.97.140.5:47015 dco_install_key: peer_id=0 keyid=0, currently 0 keys installed
2022-12-25 09:45:51 us=194167 freebsd-14-amd64/194.97.140.5:47015 dco_new_key: slot 0, key-id 0, peer-id 0, cipher AES-256-GCM
2022-12-25 09:45:51 us=194317 freebsd-14-amd64/194.97.140.5:47015 SENT CONTROL [freebsd-14-amd64]: 'PUSH_REPLY,route 10.220.0.0 255.255.0.0,route-ipv6 fd00:abcd:220::/48,tun-ipv6,route-gateway 10.220.1.1,topology subnet,ping 10,ping-restart 30,compress stub-v2,ifconfig-ipv6 fd00:abcd:220:1::1002/64 fd00:abcd:220:1::1,ifconfig 10.220.1.4 255.255.255.0,peer-id 0,cipher AES-256-GCM,key-derivation tls-ekm' (status=1)
2022-12-25 09:45:51 us=194363 freebsd-14-amd64/194.97.140.5:47015 dco_update_keys: peer_id=0
2022-12-25 09:45:52 us=265607 freebsd-14-amd64/194.97.140.5:47015 dco_update_keys: peer_id=0
2022-12-25 09:45:52 us=455630 dco_do_read
2022-12-25 09:45:52 us=455674 ovpn-dco: received OVPN_PACKET_ATTR_PACKET, ifindex: 31370 peer-id: 0, len 49
2022-12-25 09:45:53 us=336677 freebsd-14-amd64/194.97.140.5:47015 dco_update_keys: peer_id=0
2022-12-25 09:45:54 us=408023 freebsd-14-amd64/194.97.140.5:47015 dco_update_keys: peer_id=0
2022-12-25 09:45:54 us=740566 dco_do_read
2022-12-25 09:45:54 us=740653 ovpn-dco: received OVPN_PACKET_ATTR_PACKET, ifindex: 31370 peer-id: 0, len 49
2022-12-25 09:45:55 us=478633 freebsd-14-amd64/194.97.140.5:47015 dco_update_keys: peer_id=0
2022-12-25 09:45:56 us=550011 freebsd-14-amd64/194.97.140.5:47015 dco_update_keys: peer_id=0
2022-12-25 09:45:57 us=620239 freebsd-14-amd64/194.97.140.5:47015 dco_update_keys: peer_id=0
2022-12-25 09:45:58 us=85776 dco_do_read
2022-12-25 09:45:58 us=85865 ovpn-dco: received OVPN_PACKET_ATTR_PACKET, ifindex: 31370 peer-id: 0, len 49
2022-12-25 09:45:58 us=86702 dco_do_read
2022-12-25 09:45:58 us=86754 ovpn-dco: received OVPN_PACKET_ATTR_PACKET, ifindex: 31370 peer-id: 0, len 49

... but no incoming PUSH_REQUEST...

This is fully reproducible, no special timing or anything required, just "TCP p2mp server with DCO, tcp client".

With DCO disabled (same server config, --disable-dco) initialization progresses normally, and everything works..

gert

Signature · 6 lines
"If was one thing all people took for granted, was conviction that if you 
 feed honest figures into a computer, honest figures come out. Never doubted 
 it myself till I met a computer with a sense of humor."
                             Robert A. Heinlein, The Moon is a Harsh Mistress

Gert Doering - Munich, Germany                             [email protected]...
#9 lore raw

PATCH 5/9 Also drop incoming dco packet content when dropping the packet

In reply to Arne Schwabe

If we get a message from a mismatched packet we need to clear the incoming message buffer to ensure we can receive another packet.

---
 src/openvpn/forward.c | 2 ++
 src/openvpn/multi.c   | 2 ++
 2 files changed, 4 insertions(+)
src/openvpn/forward.c +2 -0
index 17a14f0bd..61caf1146 100644
@@ -1194,6 +1194,8 @@ process_incoming_dco(struct context *c)
         msg(D_DCO_DEBUG, "%s: received message for mismatching peer-id %d, "
             "expected %d", __func__, dco->dco_message_peer_id,
             c->c2.tls_multi->dco_peer_id);
+        /* ensure we also drop a message if there is one in the buffer */
+        buf_init(&dco->dco_packet_in, 0);
         return;
     }
 
src/openvpn/multi.c +2 -0
index fcb308151..9a20112e2 100644
@@ -3276,6 +3276,8 @@ multi_process_incoming_dco(struct multi_context *m)
     else
     {
         msg(D_DCO, "Received packet for peer-id unknown to OpenVPN: %d", peer_id);
+        /* Also clear the buffer if this was incoming packet for a dropped peer */
+        buf_init(&dco->dco_packet_in, 0);
     }
 
     dco->dco_message_type = 0;
Signature · 1 line
2.37.1 (Apple Git-137.1)
Acked-by on v1 5/9
#10 lore raw

[Openvpn-devel] [PATCH applied] Re: Also drop incoming dco packet content when dropping the packet

I'm not sure if my gremlins have ever seen this particular problem, but it seems yours have been more active :-)

Stare-at-code confirms that "buf_init()" is the way we clear dco_packet_in in the existing code paths, "if a packet has been consumed", and doing so in these two places ("we do not want any packet, go away") sounds like the right thing to do (otherwise dco_linux::ovpn_handle_msg() will get very upset trying to read the next incoming message).

Tested on the Linux/DCO test rig, client+server. FreeBSD never sends data packets via the DCO interface, and Windows does not even have this structure element.

Your patch has been applied to the master and release/2.6 branch.

commit 7433618cb4bea017ae7c360da42093f49cf014b4 (master) commit 474fe4df7d04de0eaa4cfe48b06497008275451a (release/2.6)

Author: Arne Schwabe
Date:   Sat Dec 24 20:42:49 2022 +0100
     Also drop incoming dco packet content when dropping the packet

     Signed-off-by: Arne Schwabe <[email protected]...>
     Acked-by: Gert Doering <[email protected]...>
     Message-Id: <[email protected]...>
     URL: https://www.mail-archive.com/[email protected]/msg25797.html
     Signed-off-by: Gert Doering <[email protected]...>

--
kind regards,

Gert Doering

#11 lore raw

PATCH 6/9 Do not set nl socket buffer size

In reply to Arne Schwabe

libnl increases the sizes we pass to 8192 anyway. Currently when we have a lot of events queued we might run into a NLE_NOMEM message and that terminates the server. So rather let the kernel decide the buffer sizes.

---
 src/openvpn/dco_linux.c | 3 ---
 1 file changed, 3 deletions(-)
src/openvpn/dco_linux.c +0 -3
index 222537fc1..d4bca555d 100644
@@ -348,9 +348,6 @@ ovpn_dco_init_netlink(dco_context_t *dco)
         msg(M_ERR, "Cannot create netlink socket");
     }
 
-    /* TODO: Why are we setting this buffer size? */
-    nl_socket_set_buffer_size(dco->nl_sock, 8192, 8192);
-
     int ret = genl_connect(dco->nl_sock);
     if (ret)
     {
Signature · 1 line
2.37.1 (Apple Git-137.1)
Acked-by on v1 6/9
#12 lore raw

[Openvpn-devel] [PATCH applied] Re: Do not set nl socket buffer size

"Trivial enough", removing code that we're not sure why we have it in the first place. Tested on Linux/DCO, client and server, and all is well behaved (for normal operations, did not test gremlins - but my gremlins never hit sitnl queue limits either).

Your patch has been applied to the master and release/2.6 branch.

commit b2ca179ce2678115c3a61fd3f843c64c1d059fbc (master) commit fa2a330f72d29a5ce630327ca288ed520aa2d697 (release/2.6)

Author: Arne Schwabe
Date:   Sat Dec 24 20:42:50 2022 +0100
     Do not set nl socket buffer size

     Signed-off-by: Arne Schwabe <[email protected]...>
     Acked-by: Gert Doering <[email protected]...>
     Message-Id: <[email protected]...>
     URL: https://www.mail-archive.com/[email protected]/msg25789.html
     Signed-off-by: Gert Doering <[email protected]...>

--
kind regards,

Gert Doering

#13 lore raw

PATCH 7/9 Bail out when trying to install a TCP socket with residual data to DCO

In reply to Arne Schwabe

Instead of getting the server in a very weird state, we bail out here. This is only a bandaid solution but better than the alternatives.

---
 src/openvpn/mtcp.c  |  2 +-
 src/openvpn/multi.c | 10 +++++++++-
 2 files changed, 10 insertions(+), 2 deletions(-)
src/openvpn/mtcp.c +1 -1
index 263f4d994..3837ccbab 100644
@@ -591,7 +591,7 @@ multi_tcp_post(struct multi_context *m, struct multi_instance *mi, const int act
                     struct gc_arena gc = gc_new();
                     msg(M_INFO, "ovpn-dco installed socket with residual read "
                         "len=%d, mi=%s. This connection will probably"
-                        " break.",  BLEN(&c->c2.link_socket->stream_buf.residual),
+                        " break.",  BLEN(&c->c2.link_socket->stream_buf.buf),
                         multi_instance_string(mi, false, &gc));
                     gc_free(&gc);
                     break;
src/openvpn/multi.c +9 -1
index 9a20112e2..d29b7efe3 100644
@@ -2316,9 +2316,17 @@ multi_client_setup_dco_initial(struct multi_context *m,
 {
     if (!dco_enabled(&mi->context.options))
     {
-        /* DCO not enabled, nothing to do, return sucess */
+        /* DCO not enabled, nothing to do, return success */
         return true;
     }
+
+    if (socket_read_residual(mi->context.c2.link_socket))
+    {
+        msg(M_INFO, "TCP socket with half read packet. Cannot install to "
+            "DCO: %s",  multi_instance_string(mi, false, gc));
+        return false;
+    }
+
     int ret = dco_multi_add_new_peer(m, mi);
     if (ret < 0)
     {
Signature · 1 line
2.37.1 (Apple Git-137.1)
#14 lore raw

Re: [Openvpn-devel] [PATCH 7/9] Bail out when trying to install a TCP socket with residual data to DCO

Hi,

Quoted 4 lines · Arne Schwabe

Instead of getting the server in a very weird state, we bail out here. This is only a bandaid solution but better than the alternatives.

Having a clear failure indication in the log and towards the client is good, so definitely Feature-ACK.

I did also observe this when running the gremlin TCP test last night, with 4/9 applied

Dec 25 02:49:44 ubuntu2004 tun-tcp-p2mp-username-cn[1828024]: ovpn-dco installed socket with residual read len=0, mi=gremlin84091/194.97.140.21:10133. This connection will probably break.
Dec 25 02:49:46 ubuntu2004 tun-tcp-p2mp-username-cn[1828024]: ovpn-dco installed socket with residual read len=0, mi=gremlin84091/194.97.140.21:10133. This connection will probably break.
Dec 25 02:49:47 ubuntu2004 tun-tcp-p2mp-username-cn[1828024]: ovpn-dco installed socket with residual read len=0, mi=gremlin84091/194.97.140.21:10133. This connection will probably break.
Dec 25 02:49:48 ubuntu2004 tun-tcp-p2mp-username-cn[1828024]: ovpn-dco installed socket with residual read len=0, mi=gremlin84091/194.97.140.21:10133. This connection will probably break.

... with a single instance (gremlin84091) having 2905 lines of log - so this is certainly not a state we want the server to be in (other instances only show up once, or 4 or 22 times... whatever OpenVPN does here).

This does not apply as it needs 4/9, and *that* one breaks inbound TCP for me, so postponing this patch until we understand why 4/ doesn't work.

gert

Signature · 6 lines
"If was one thing all people took for granted, was conviction that if you 
 feed honest figures into a computer, honest figures come out. Never doubted 
 it myself till I met a computer with a sense of humor."
                             Robert A. Heinlein, The Moon is a Harsh Mistress

Gert Doering - Munich, Germany                             [email protected]...
#15 lore raw

PATCH 8/9 Improve logging when seeing a message for an unkown peer

In reply to Arne Schwabe
---
 src/openvpn/multi.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
src/openvpn/multi.c +11 -1
index d29b7efe3..6c6385c6e 100644
@@ -3283,7 +3283,17 @@ multi_process_incoming_dco(struct multi_context *m)
     }
     else
     {
-        msg(D_DCO, "Received packet for peer-id unknown to OpenVPN: %d", peer_id);
+        int msglevel = D_DCO;
+        if (dco->dco_message_type == OVPN_CMD_DEL_PEER
+            && dco->dco_del_peer_reason == OVPN_DEL_PEER_REASON_USERSPACE)
+        {
+            /* we get notified after we kill the peer ourselves and probably
+             * have already forgotten about it. This is expected */
+            msglevel = D_DCO_DEBUG;
+        }
+        msg(msglevel, "Received packet for peer-id unknown to OpenVPN: %d, "
+            "type %d, reason %d", peer_id, dco->dco_message_type,
+            dco->dco_del_peer_reason);
         /* Also clear the buffer if this was incoming packet for a dropped peer */
         buf_init(&dco->dco_packet_in, 0);
     }
Signature · 1 line
2.37.1 (Apple Git-137.1)
Acked-by on v1 8/9
#16 lore raw

[Openvpn-devel] [PATCH applied] Re: Improve logging when seeing a message for an unkown peer

This looks like a good change - only log the "this is not really relevant" messages if we're interested (verb 6), and also add more details.

Run on Linux-DCO client/server testbed, and I see quite a number of them...

Dec 25 19:22:01 ubuntu2004 tun-udp-p2mp-topology-subnet[1965839]: Received packet for peer-id unknown to OpenVPN: -1, type 0, reason 2
Dec 25 19:22:01 ubuntu2004 tun-udp-p2mp[1965830]: Received packet for peer-id unknown to OpenVPN: -1, type 0, reason 0
Dec 25 19:22:01 ubuntu2004 tun-tcp-p2mp[1965819]: Received packet for peer-id unknown to OpenVPN: -1, type 0, reason 1
Dec 25 19:22:01 ubuntu2004 tun-udp-p2mp-topology-subnet[1965839]: Received packet for peer-id unknown to OpenVPN: -1, type 0, reason 2
Dec 25 19:22:01 ubuntu2004 tun-udp-p2mp[1965830]: Received packet for peer-id unknown to OpenVPN: -1, type 0, reason 0
Dec 25 19:22:01 ubuntu2004 tun-tcp-p2mp[1965819]: Received packet for peer-id unknown to OpenVPN: -1, type 0, reason 1

.. though I can't claim to understand what is happening here...

Why is the kernel sending packets for "peer-id -1"? Why are these being sent to all OpenVPN instances at the same moment? Even if we're ignoring these messages, they leave me with an uneasy feeling... (type = 0 is "CMD_UNSPEC", reason 0/1/2 would be TEARDOWN, USERSPACE, EXPIRED - shouldn't these be DEL_PEER messages, so type=3?)

Your patch has been applied to the master and release/2.6 branch.

commit 8516b4b3665aa94d3114194f33fbb9c34119ab71 (master) commit 5d47df8d69c73629eccc1e443b3dfedbb9e8fd2d (release/2.6)

Author: Arne Schwabe
Date:   Sat Dec 24 20:42:52 2022 +0100
     Improve logging when seeing a message for an unkown peer

     Signed-off-by: Arne Schwabe <[email protected]...>
     Acked-by: Gert Doering <[email protected]...>
     Message-Id: <[email protected]...>
     URL: https://www.mail-archive.com/[email protected]/msg25793.html
     Signed-off-by: Gert Doering <[email protected]...>

--
kind regards,

Gert Doering

#17 lore raw

PATCH 9/9 Ignore OVPN_DEL_PEER_REASON_USERSPACE to avoid race conditions

In reply to Arne Schwabe
---
 src/openvpn/multi.c | 21 +++++++++++++++++----
 1 file changed, 17 insertions(+), 4 deletions(-)
src/openvpn/multi.c +17 -4
index 6c6385c6e..858d602ca 100644
@@ -473,6 +473,10 @@ multi_instance_string(const struct multi_instance *mi, bool null, struct gc_aren
             buf_printf(&out, "%s/", cn);
         }
         buf_printf(&out, "%s", mroute_addr_print(&mi->real, gc));
+        if (mi->context.c2.tls_multi)
+        {
+            buf_printf(&out, " peer-id=%d", mi->context.c2.tls_multi->peer_id);
+        }
         return BSTR(&out);
     }
     else if (null)
@@ -3243,10 +3247,19 @@ process_incoming_del_peer(struct multi_context *m, struct multi_instance *mi,
             break;
 
         case OVPN_DEL_PEER_REASON_USERSPACE:
-            /* This very likely ourselves but might be another process, so
-             * still process it */
-            reason = "ovpn-dco: userspace request";
-            break;
+            /* We assume that is ourselves. UUnfortunately, sometimes these
+             * events happen with enough delay that they can have an order of
+             *
+             * dco_del_peer x
+             * [new client connecting]
+             * dco_new_peer x
+             * event from dco_del_peer arrives.
+             *
+             * if we do not ignore this we get desynced with the kernel
+             * since we assume the peer-id is free again. The other way would
+             * be to send a dco_del_peer again
+             */
+            return;
     }
 
     /* When kernel already deleted the peer, the socket is no longer
Signature · 1 line
2.37.1 (Apple Git-137.1)
#18 lore raw

Re: [Openvpn-devel] [PATCH 9/9] Ignore OVPN_DEL_PEER_REASON_USERSPACE to avoid race conditions

Hi,

src/openvpn/multi.c multi_instance_string(const struct multi_instance *mi, bool null, struct gc_aren · line 479

Quoted 5 lines · Arne Schwabe
         buf_printf(&out, "%s", mroute_addr_print(&mi->real, gc));
src/openvpn/multi.c +4 -0
+        if (mi->context.c2.tls_multi)
+        {
+            buf_printf(&out, " peer-id=%d", mi->context.c2.tls_multi->peer_id);
+        }

Do I read this right, this will add the peer-id=%d to every single log line in p2mp mode? In DCO mode, this might be a relevant information (... for things related to DCO), but in non-DCO mode, it's mostly "uninteresting", no?

I'm all for complete log information, but making the lines too long isn't that helpful either...

(The other part of that patch will be looked at, when 4/9 has survived)

gert

Signature · 6 lines
"If was one thing all people took for granted, was conviction that if you 
 feed honest figures into a computer, honest figures come out. Never doubted 
 it myself till I met a computer with a sense of humor."
                             Robert A. Heinlein, The Moon is a Harsh Mistress

Gert Doering - Munich, Germany                             [email protected]...
#19 lore raw

Re: [Openvpn-devel] [PATCH 9/9] Ignore OVPN_DEL_PEER_REASON_USERSPACE to avoid race conditions

Am 25.12.22 um 00:13 schrieb Gert Doering:

Quoted 9 lines

Hi,

src/openvpn/multi.c multi_instance_string(const struct multi_instance *mi, bool null, struct gc_aren · line 479

Quoted 5 lines · Arne Schwabe
          buf_printf(&out, "%s", mroute_addr_print(&mi->real, gc));
src/openvpn/multi.c +4 -0
+        if (mi->context.c2.tls_multi)
+        {
+            buf_printf(&out, " peer-id=%d", mi->context.c2.tls_multi->peer_id);
+        }
Quoted 7 lines

Do I read this right, this will add the peer-id=%d to every single log line in p2mp mode? In DCO mode, this might be a relevant information (... for things related to DCO), but in non-DCO mode, it's mostly "uninteresting", no?

I'm all for complete log information, but making the lines too long isn't that helpful either...

Yeah. This accidentially slipped into the patch set. I am sorry for including it. It was however really helpful for doing the debugging.

Arne

#20 lore raw

Re: [Openvpn-devel] [PATCH 9/9] Ignore OVPN_DEL_PEER_REASON_USERSPACE to avoid race conditions

Hi,

Quoted 6 lines · Arne Schwabe
Quoted 4 lines

src/openvpn/multi.c multi_instance_string(const struct multi_instance *mi, bool null, struct gc_aren · line 479

Quoted 4 lines
src/openvpn/multi.c +4 -0
+        if (mi->context.c2.tls_multi)
+        {
+            buf_printf(&out, " peer-id=%d", mi->context.c2.tls_multi->peer_id);
+        }

Yeah. This accidentially slipped into the patch set. I am sorry for including it. It was however really helpful for doing the debugging.

Thinking more about it, I wonder if we can have it conditionalized on "DCO active" and "verb >=6" (D_DCO_DEBUG). This would make it useful for, well, DCO debugging, but not get in the way otherwise.

Do we have enough information available at this place?

gert

Signature · 6 lines
"If was one thing all people took for granted, was conviction that if you 
 feed honest figures into a computer, honest figures come out. Never doubted 
 it myself till I met a computer with a sense of humor."
                             Robert A. Heinlein, The Moon is a Harsh Mistress

Gert Doering - Munich, Germany                             [email protected]...