Skip to contentSkip to conversation

Ensure we do not promote a TA_TIMEOUT to a TA_READ event with dco

PATCH Superseded Arne Schwabe 2 participants 8 messages Dec 27, 2022 – Jan 10, 2023

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/4 Ensure we do not promote a TA_TIMEOUT to a TA_READ event with dco

with dco sometimes we end up promoting a timeout event to read event. For the residual read, this problem is probably not solvable without changing the kernel DCO API (ie. passing our residual on new_peer to let the kernel handle assembling the next packet.)

---
 src/openvpn/forward.c | 24 ------------------------
 src/openvpn/forward.h | 30 ++++++++++++++++++++++++++++++
 src/openvpn/mtcp.c    | 12 +++++++++++-
 3 files changed, 41 insertions(+), 25 deletions(-)
src/openvpn/forward.c +0 -24
index af4ed05da..61caf1146 100644
@@ -1676,30 +1676,6 @@ process_ip_header(struct context *c, unsigned int flags, struct buffer *buf)
     }
 }
 
-/*
- * Linux DCO implementations pass the socket to the kernel and
- * disallow usage of it from userland for TCP, so (control) packets
- * sent and received by OpenVPN need to go through the DCO interface.
- *
- * Windows DCO needs control packets to be sent via the normal
- * standard Overlapped I/O.
- *
- * FreeBSD DCO allows control packets to pass through the socket in both
- * directions.
- *
- * Hide that complexity (...especially if more platforms show up
- * in the future...) in a small inline function.
- */
-static inline bool
-should_use_dco_socket(struct link_socket *ls)
-{
-#if defined(TARGET_LINUX)
-    return ls->dco_installed && proto_is_tcp(ls->info.proto);
-#else
-    return false;
-#endif
-}
-
 /*
  * Input: c->c2.to_link
  */
src/openvpn/forward.h +30 -0
index bd2d96010..5cddb5995 100644
@@ -424,4 +424,34 @@ connection_established(struct context *c)
     }
 }
 
+
+/**
+ * @param ls the link_socket the decision should be made for
+ * @return if we should use the dco kernel api or normal socket APIs for
+ *         write/send
+ *
+ *
+ * Linux DCO implementations pass the socket to the kernel and
+ * disallow usage of it from userland for TCP, so (control) packets
+ * sent and received by OpenVPN need to go through the DCO interface.
+ *
+ * Windows DCO needs control packets to be sent via the normal
+ * standard Overlapped I/O.
+ *
+ * FreeBSD DCO allows control packets to pass through the socket in both
+ * directions.
+ *
+ * Hide that complexity (...especially if more platforms show up
+ * in the future...) in a small inline function.
+ */
+static inline bool
+should_use_dco_socket(const struct link_socket *ls)
+{
+#if defined(TARGET_LINUX)
+    return ls->dco_installed && proto_is_tcp(ls->info.proto);
+#else
+    return false;
+#endif
+}
+
 #endif /* FORWARD_H */
src/openvpn/mtcp.c +11 -1
index ac06ddc64..519630544 100644
@@ -407,7 +407,7 @@ multi_tcp_wait_lite(struct multi_context *m, struct multi_instance *mi, const in
         /* If we got a socket that has been handed over to the kernel
          * we must not call the normal socket function to figure out
          * if it is readable or writable */
-        /* Assert that we only have the DCO exptected flags */
+        /* Assert that we only have the DCO expected flags */
         ASSERT(action & (TA_SOCKET_READ | TA_SOCKET_WRITE));
 
         /* We are always ready! */
@@ -586,6 +586,16 @@ multi_tcp_post(struct multi_context *m, struct multi_instance *mi, const int act
         case MTP_NONE:
             if (mi && socket_read_residual(c->c2.link_socket))
             {
+                if (should_use_dco_socket(c->c2.link_socket))
+                {
+                    struct gc_arena gc = gc_new();
+                    msg(M_INFO, "ovpn-dco installed socket with residual read "
+                        "len=%d, mi=%s. This connection will probably"
+                        " break.",  BLEN(&c->c2.link_socket->stream_buf.residual),
+                        multi_instance_string(mi, false, &gc));
+                    gc_free(&gc);
+                    break;
+                }
                 newaction = TA_SOCKET_READ_RESIDUAL;
             }
             else
Signature · 1 line
2.37.1 (Apple Git-137.1)
#1 lore raw

PATCH v2 2/4 Bail out when trying to install a TCP socket with residual data to DCO

Instead of getting the server in a very weird state, we bail out here. This is only a bandaid solution but better than the alternatives.

---
 src/openvpn/mtcp.c  |  2 +-
 src/openvpn/multi.c | 10 +++++++++-
 2 files changed, 10 insertions(+), 2 deletions(-)
src/openvpn/mtcp.c +1 -1
index 519630544..c07b5d592 100644
@@ -591,7 +591,7 @@ multi_tcp_post(struct multi_context *m, struct multi_instance *mi, const int act
                     struct gc_arena gc = gc_new();
                     msg(M_INFO, "ovpn-dco installed socket with residual read "
                         "len=%d, mi=%s. This connection will probably"
-                        " break.",  BLEN(&c->c2.link_socket->stream_buf.residual),
+                        " break.",  BLEN(&c->c2.link_socket->stream_buf.buf),
                         multi_instance_string(mi, false, &gc));
                     gc_free(&gc);
                     break;
src/openvpn/multi.c +9 -1
index 92e63dd26..6c6385c6e 100644
@@ -2316,9 +2316,17 @@ multi_client_setup_dco_initial(struct multi_context *m,
 {
     if (!dco_enabled(&mi->context.options))
     {
-        /* DCO not enabled, nothing to do, return sucess */
+        /* DCO not enabled, nothing to do, return success */
         return true;
     }
+
+    if (socket_read_residual(mi->context.c2.link_socket))
+    {
+        msg(M_INFO, "TCP socket with half read packet. Cannot install to "
+            "DCO: %s",  multi_instance_string(mi, false, gc));
+        return false;
+    }
+
     int ret = dco_multi_add_new_peer(m, mi);
     if (ret < 0)
     {
Signature · 1 line
2.37.1 (Apple Git-137.1)
#2 lore raw

PATCH v2 3/4 Ignore OVPN_DEL_PEER_REASON_USERSPACE to avoid race conditions

In reply to Arne Schwabe
---
 src/openvpn/multi.c | 17 +++++++++++++----
 1 file changed, 13 insertions(+), 4 deletions(-)
src/openvpn/multi.c +13 -4
index 6c6385c6e..50d88f19a 100644
@@ -3243,10 +3243,19 @@ process_incoming_del_peer(struct multi_context *m, struct multi_instance *mi,
             break;
 
         case OVPN_DEL_PEER_REASON_USERSPACE:
-            /* This very likely ourselves but might be another process, so
-             * still process it */
-            reason = "ovpn-dco: userspace request";
-            break;
+            /* We assume that is ourselves. UUnfortunately, sometimes these
+             * events happen with enough delay that they can have an order of
+             *
+             * dco_del_peer x
+             * [new client connecting]
+             * dco_new_peer x
+             * event from dco_del_peer arrives.
+             *
+             * if we do not ignore this we get desynced with the kernel
+             * since we assume the peer-id is free again. The other way would
+             * be to send a dco_del_peer again
+             */
+            return;
     }
 
     /* When kernel already deleted the peer, the socket is no longer
Signature · 1 line
2.37.1 (Apple Git-137.1)
Acked-by on v2 3/4
#3 lore raw

[Openvpn-devel] [PATCH applied] Re: Ignore OVPN_DEL_PEER_REASON_USERSPACE to avoid race conditions

I have not actually seen this particular race (or if I have seen it it was drowned in lots of other events) - but yeah, to fully ignore this seems better than "kill peer <x> just because we ended a peer with the very same peer-id <x> a few seconds ago".

To achieve the "some other program might have sent this" original idea, one would need to add a sending PID ("we ignore everything with our own PID"), or some sort of ACK mechanism ("we do not re-assign the peer-id <x> until we have confirmation from the kernel"). This would certainly be nice, but for now, ignore it is.

Tested on the ubuntu2004 DCO system (client/server), with no gremlins.

Your patch has been applied to the master branch.

commit 6ad66b0c2950c0d7674a5867085fef8115f61d11 (master) commit eec054a95820ffa7e548f47078ec0101d6949907 (release/2.6)

Author: Arne Schwabe
Date:   Tue Dec 27 03:24:03 2022 +0100
     Ignore OVPN_DEL_PEER_REASON_USERSPACE to avoid race conditions

     Signed-off-by: Arne Schwabe <[email protected]...>
     Acked-by: Gert Doering <[email protected]...>
     Message-Id: <[email protected]...>
     URL: https://www.mail-archive.com/[email protected]/msg25820.html
     Signed-off-by: Gert Doering <[email protected]...>

--
kind regards,

Gert Doering

#4 lore raw

PATCH v2 4/4 Log peer-id if loglevel is D_DCO_DEBUG and dco is enabled

In reply to Arne Schwabe

This enables logging the peer id in p2mp mode if dco is enabled and the log level is high enough

---
 src/openvpn/multi.c | 6 ++++++
 1 file changed, 6 insertions(+)
src/openvpn/multi.c +6 -0
index 50d88f19a..f8366be28 100644
@@ -473,6 +473,12 @@ multi_instance_string(const struct multi_instance *mi, bool null, struct gc_aren
             buf_printf(&out, "%s/", cn);
         }
         buf_printf(&out, "%s", mroute_addr_print(&mi->real, gc));
+        if (mi->context.c2.tls_multi
+            && mi->context.options.verbosity >= D_DCO_DEBUG
+            && dco_enabled(&mi->context.options))
+        {
+            buf_printf(&out, " peer-id=%d", mi->context.c2.tls_multi->peer_id);
+        }
         return BSTR(&out);
     }
     else if (null)
Signature · 1 line
2.37.1 (Apple Git-137.1)
#5 lore raw

Re: [Openvpn-devel] [PATCH v2 4/4] Log peer-id if loglevel is D_DCO_DEBUG and dco is enabled

Hi,

Quoted 2 lines · Arne Schwabe

This enables logging the peer id in p2mp mode if dco is enabled and the log level is high enough

Ho hum, this does not work for me...

TLS/TCP server, DCO enabled, --verb 6...

2022-12-27 11:05:41 us=111855 something/194.97.140.21:16006 dco_do_write: peer-id 0, len=390
2022-12-27 11:05:41 us=111931 something/194.97.140.21:16006 dco_update_keys: peer_id=0
2022-12-27 11:05:42 us=173611 dco_do_read
2022-12-27 11:05:42 us=173764 ovpn-dco: received OVPN_PACKET_ATTR_PACKET, ifindex: 32018 peer-id: 0, len 22
2022-12-27 11:05:42 us=173825 something/194.97.140.21:16006 TCPv6_SERVER READ [22] from [AF_INET6]::ffff:194.97.140.21:16006: P_ACK_V1 kid=0 [ 5 ] DATA len=0
2022-12-27 11:05:42 us=267059 something/194.97.140.21:16006 dco_update_keys: peer_id=0
2022-12-27 11:05:43 us=423410 something/194.97.140.21:16006 dco_update_keys: peer_id=0
2022-12-27 11:05:58 us=584121 something/194.97.140.21:16006 dco_update_keys: peer_id=0

... there *should* be a peer-id somewhere in that line...

Playing around with the patch a bit, the offending piece seems to be "mi->context.options.verbosity >= D_DCO_DEBUG" - which is unsurprising, as D_DCO_DEBUG is not "6" but "LOGLEV(6, 69, M_DEBUG)", which translates to

  #define LOGLEV(log_level, mute_level, other) ((log_level) | ENCODE_MUTE_LEVEL(mute_level) | other)

(numerically, D_DCO_DEBUG = 1157628038)

The test for dco_enabled() works.

gert

Signature · 6 lines
"If was one thing all people took for granted, was conviction that if you 
 feed honest figures into a computer, honest figures come out. Never doubted 
 it myself till I met a computer with a sense of humor."
                             Robert A. Heinlein, The Moon is a Harsh Mistress

Gert Doering - Munich, Germany                             [email protected]...
#6 lore raw

Re: [Openvpn-devel] [PATCH v2 4/4] Log peer-id if loglevel is D_DCO_DEBUG and dco is enabled

Hi,

Quoted 8 lines · Gert Doering

Playing around with the patch a bit, the offending piece seems to be "mi->context.options.verbosity >= D_DCO_DEBUG" - which is unsurprising, as D_DCO_DEBUG is not "6" but "LOGLEV(6, 69, M_DEBUG)", which translates to

  #define LOGLEV(log_level, mute_level, other) ((log_level) | ENCODE_MUTE_LEVEL(mute_level) | other)

(numerically, D_DCO_DEBUG = 1157628038)

... looking at error.h, there is a "check_debug_level(flags)" thing, so changing the condition to

+ if (mi->context.c2.tls_multi

+            // && mi->context.options.verbosity >= D_DCO_DEBUG
+            && check_debug_level(D_DCO_DEBUG)
+            && dco_enabled(&mi->context.options))

+ {
+ buf_printf(&out, " peer-id=%d", mi->context.c2.tls_multi->peer_id);
+ }

works - "if --verb 6 or higher, *and* DCO, then have peer-id=%d as part if the prefix".

Can you resend a v3?

gert

Signature · 6 lines
"If was one thing all people took for granted, was conviction that if you 
 feed honest figures into a computer, honest figures come out. Never doubted 
 it myself till I met a computer with a sense of humor."
                             Robert A. Heinlein, The Moon is a Harsh Mistress

Gert Doering - Munich, Germany                             [email protected]...
#7 lore raw
In reply to Arne Schwabe

Hi,

Quoted 6 lines · Arne Schwabe

with dco sometimes we end up promoting a timeout event to read event. For the residual read, this problem is probably not solvable without changing the kernel DCO API (ie. passing our residual on new_peer to let the kernel handle assembling the next packet.)

So after a long and extensive round of testing (this one as it is, and 1/4+2/4 against the gremlins) I can say that it does not break anything, and fixes the "too many TCP connects will cause a fd=-1 recv() and subsequent crash".

After some more discussion on IRC neither Arne and I seem to really fully understand the residual stuff, so not merging this yet, more study is needed...

gert

Signature · 6 lines
"If was one thing all people took for granted, was conviction that if you 
 feed honest figures into a computer, honest figures come out. Never doubted 
 it myself till I met a computer with a sense of humor."
                             Robert A. Heinlein, The Moon is a Harsh Mistress

Gert Doering - Munich, Germany                             [email protected]...