Ensure we do not promote a TA_TIMEOUT to a TA_READ event with dco
PATCH v2 1/4 Ensure we do not promote a TA_TIMEOUT to a TA_READ event with dco
with dco sometimes we end up promoting a timeout event to read event. For the residual read, this problem is probably not solvable without changing the kernel DCO API (ie. passing our residual on new_peer to let the kernel handle assembling the next packet.)
--- src/openvpn/forward.c | 24 ------------------------ src/openvpn/forward.h | 30 ++++++++++++++++++++++++++++++ src/openvpn/mtcp.c | 12 +++++++++++- 3 files changed, 41 insertions(+), 25 deletions(-)
src/openvpn/forward.c +0 -24
@@ -1676,30 +1676,6 @@ process_ip_header(struct context *c, unsigned int flags, struct buffer *buf) } } -/* -* Linux DCO implementations pass the socket to the kernel and -* disallow usage of it from userland for TCP, so (control) packets -* sent and received by OpenVPN need to go through the DCO interface. -* -* Windows DCO needs control packets to be sent via the normal -* standard Overlapped I/O. -* -* FreeBSD DCO allows control packets to pass through the socket in both -* directions. -* -* Hide that complexity (...especially if more platforms show up -* in the future...) in a small inline function. -*/ -static inline bool -should_use_dco_socket (struct link_socket *ls) -{ -#if defined(TARGET_LINUX ) -return ls->dco_installed &&proto_is_tcp (ls->info.proto); -#else -return false; -#endif -} -/* * Input: c->c2.to_link */
src/openvpn/forward.h +30 -0
@@ -424,4 +424,34 @@ connection_established(struct context *c) } } + +/** +* @param ls the link_socket the decision should be made for +* @return if we should use the dco kernel api or normal socket APIs for +* write/send +* +* +* Linux DCO implementations pass the socket to the kernel and +* disallow usage of it from userland for TCP, so (control) packets +* sent and received by OpenVPN need to go through the DCO interface. +* +* Windows DCO needs control packets to be sent via the normal +* standard Overlapped I/O. +* +* FreeBSD DCO allows control packets to pass through the socket in both +* directions. +* +* Hide that complexity (...especially if more platforms show up +* in the future...) in a small inline function. +*/ +static inline bool +should_use_dco_socket (const struct link_socket *ls) +{ +#if defined(TARGET_LINUX ) +return ls->dco_installed &&proto_is_tcp (ls->info.proto); +#else +return false; +#endif +} +#endif /* FORWARD_H */
src/openvpn/mtcp.c +11 -1
@@ -407,7 +407,7 @@ multi_tcp_wait_lite(struct multi_context *m, struct multi_instance *mi, const in/* If we got a socket that has been handed over to the kernel * we must not call the normal socket function to figure out * if it is readable or writable */ -/* Assert that we only have the DCO exptected flags */ +/* Assert that we only have the DCO expected flags */ ASSERT (action & (TA_SOCKET_READ |TA_SOCKET_WRITE ));/* We are always ready! */
@@ -586,6 +586,16 @@ multi_tcp_post(struct multi_context *m, struct multi_instance *mi, const int actcase MTP_NONE :if (mi &&socket_read_residual (c->c2.link_socket)) { +if (should_use_dco_socket (c->c2.link_socket)) + { +struct gc_arena gc =gc_new (); +msg (M_INFO ,"ovpn-dco installed socket with residual read " +"len=%d, mi=%s. This connection will probably" +" break." ,BLEN (&c->c2.link_socket->stream_buf.residual), +multi_instance_string (mi, false, &gc)); +gc_free (&gc); +break ; + } newaction =TA_SOCKET_READ_RESIDUAL ; }else
Signature · 1 line
2.37.1 (Apple Git-137.1)
PATCH v2 2/4 Bail out when trying to install a TCP socket with residual data to DCO
Instead of getting the server in a very weird state, we bail out here. This is only a bandaid solution but better than the alternatives.
--- src/openvpn/mtcp.c | 2 +- src/openvpn/multi.c | 10 +++++++++- 2 files changed, 10 insertions(+), 2 deletions(-)
src/openvpn/mtcp.c +1 -1
@@ -591,7 +591,7 @@ multi_tcp_post(struct multi_context *m, struct multi_instance *mi, const int actstruct gc_arena gc =gc_new ();msg (M_INFO ,"ovpn-dco installed socket with residual read " "len=%d, mi=%s. This connection will probably" -" break." ,BLEN (&c->c2.link_socket->stream_buf.residual), +" break." ,BLEN (&c->c2.link_socket->stream_buf.buf),multi_instance_string (mi, false, &gc));gc_free (&gc);break ;
src/openvpn/multi.c +9 -1
@@ -2316,9 +2316,17 @@ multi_client_setup_dco_initial(struct multi_context *m, {if (!dco_enabled (&mi->context.options)) { -/* DCO not enabled, nothing to do, return sucess */ +/* DCO not enabled, nothing to do, return success */ return true; } + +if (socket_read_residual (mi->context.c2.link_socket)) + { +msg (M_INFO ,"TCP socket with half read packet. Cannot install to " +"DCO: %s" ,multi_instance_string (mi, false, gc)); +return false; + } +int ret =dco_multi_add_new_peer (m, mi);if (ret <0 ) {
Signature · 1 line
2.37.1 (Apple Git-137.1)
PATCH v2 3/4 Ignore OVPN_DEL_PEER_REASON_USERSPACE to avoid race conditions
In reply to Arne Schwabe--- src/openvpn/multi.c | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-)
src/openvpn/multi.c +13 -4
@@ -3243,10 +3243,19 @@ process_incoming_del_peer(struct multi_context *m, struct multi_instance *mi,break ;case OVPN_DEL_PEER_REASON_USERSPACE : -/* This very likely ourselves but might be another process, so -* still process it */ - reason ="ovpn-dco: userspace request" ; -break ; +/* We assume that is ourselves. UUnfortunately, sometimes these +* events happen with enough delay that they can have an order of +* +* dco_del_peer x +* [new client connecting] +* dco_new_peer x +* event from dco_del_peer arrives. +* +* if we do not ignore this we get desynced with the kernel +* since we assume the peer-id is free again. The other way would +* be to send a dco_del_peer again +*/ +return ; }/* When kernel already deleted the peer, the socket is no longer
Signature · 1 line
2.37.1 (Apple Git-137.1)
PATCH v2 4/4 Log peer-id if loglevel is D_DCO_DEBUG and dco is enabled
In reply to Arne SchwabeThis enables logging the peer id in p2mp mode if dco is enabled and the log level is high enough
--- src/openvpn/multi.c | 6 ++++++ 1 file changed, 6 insertions(+)
src/openvpn/multi.c +6 -0
@@ -473,6 +473,12 @@ multi_instance_string(const struct multi_instance *mi, bool null, struct gc_arenbuf_printf (&out,"%s/" , cn); }buf_printf (&out,"%s" ,mroute_addr_print (&mi->real, gc)); +if (mi->context.c2.tls_multi + && mi->context.options.verbosity >=D_DCO_DEBUG + &&dco_enabled (&mi->context.options)) + { +buf_printf (&out," peer-id=%d" , mi->context.c2.tls_multi->peer_id); + }return BSTR (&out); }else if (null)
Signature · 1 line
2.37.1 (Apple Git-137.1)
Re: [Openvpn-devel] [PATCH v2 4/4] Log peer-id if loglevel is D_DCO_DEBUG and dco is enabled
Hi,
Quoted 2 lines · Arne Schwabe
This enables logging the peer id in p2mp mode if dco is enabled and the log level is high enough
Ho hum, this does not work for me...
TLS/TCP server, DCO enabled, --verb 6...
2022-12-27 11:05:41 us=111855 something/194.97.140.21:16006 dco_do_write: peer-id 0, len=390 2022-12-27 11:05:41 us=111931 something/194.97.140.21:16006 dco_update_keys: peer_id=0 2022-12-27 11:05:42 us=173611 dco_do_read 2022-12-27 11:05:42 us=173764 ovpn-dco: received OVPN_PACKET_ATTR_PACKET, ifindex: 32018 peer-id: 0, len 22 2022-12-27 11:05:42 us=173825 something/194.97.140.21:16006 TCPv6_SERVER READ [22] from [AF_INET6]::ffff:194.97.140.21:16006: P_ACK_V1 kid=0 [ 5 ] DATA len=0 2022-12-27 11:05:42 us=267059 something/194.97.140.21:16006 dco_update_keys: peer_id=0 2022-12-27 11:05:43 us=423410 something/194.97.140.21:16006 dco_update_keys: peer_id=0 2022-12-27 11:05:58 us=584121 something/194.97.140.21:16006 dco_update_keys: peer_id=0
... there *should* be a peer-id somewhere in that line...
Playing around with the patch a bit, the offending piece seems to be "mi->context.options.verbosity >= D_DCO_DEBUG" - which is unsurprising, as D_DCO_DEBUG is not "6" but "LOGLEV(6, 69, M_DEBUG)", which translates to
#define LOGLEV(log_level, mute_level, other) ((log_level) | ENCODE_MUTE_LEVEL(mute_level) | other)
(numerically, D_DCO_DEBUG = 1157628038)
The test for dco_enabled() works.
gert
Signature · 6 lines
"If was one thing all people took for granted, was conviction that if you
feed honest figures into a computer, honest figures come out. Never doubted
it myself till I met a computer with a sense of humor."
Robert A. Heinlein, The Moon is a Harsh Mistress
Gert Doering - Munich, Germany [email protected]...Hi,
Quoted 6 lines · Arne Schwabe
with dco sometimes we end up promoting a timeout event to read event. For the residual read, this problem is probably not solvable without changing the kernel DCO API (ie. passing our residual on
new_peerto let the kernel handle assembling the next packet.)Signed-off-by Arne Schwabe <[email protected]...>
So after a long and extensive round of testing (this one as it is, and 1/4+2/4 against the gremlins) I can say that it does not break anything, and fixes the "too many TCP connects will cause a fd=-1 recv() and subsequent crash".
After some more discussion on IRC neither Arne and I seem to really fully understand the residual stuff, so not merging this yet, more study is needed...
gert
Signature · 6 lines
"If was one thing all people took for granted, was conviction that if you
feed honest figures into a computer, honest figures come out. Never doubted
it myself till I met a computer with a sense of humor."
Robert A. Heinlein, The Moon is a Harsh Mistress
Gert Doering - Munich, Germany [email protected]...[Openvpn-devel] [PATCH applied] Re: Ignore OVPN_DEL_PEER_REASON_USERSPACE to avoid race conditions
In reply to Arne SchwabeI have not actually seen this particular race (or if I have seen it it was drowned in lots of other events) - but yeah, to fully ignore this seems better than "kill peer <x> just because we ended a peer with the very same peer-id <x> a few seconds ago".
To achieve the "some other program might have sent this" original idea, one would need to add a sending PID ("we ignore everything with our own PID"), or some sort of ACK mechanism ("we do not re-assign the peer-id <x> until we have confirmation from the kernel"). This would certainly be nice, but for now, ignore it is.
Tested on the ubuntu2004 DCO system (client/server), with no gremlins.
Your patch has been applied to the master branch.
commit 6ad66b0c2950c0d7674a5867085fef8115f61d11 (master) commit eec054a95820ffa7e548f47078ec0101d6949907 (release/2.6)
Author: Arne Schwabe Date: Tue Dec 27 03:24:03 2022 +0100
Ignore OVPN_DEL_PEER_REASON_USERSPACE to avoid race conditions
Signed-off-by: Arne Schwabe <[email protected]...>
Acked-by: Gert Doering <[email protected]...>
Message-Id: <[email protected]...>
URL: https://www.mail-archive.com/[email protected]/msg25820.html
Signed-off-by: Gert Doering <[email protected]...>--
kind regards,
Gert Doering
Re: [Openvpn-devel] [PATCH v2 4/4] Log peer-id if loglevel is D_DCO_DEBUG and dco is enabled
In reply to Gert DoeringHi,
Quoted 8 lines · Gert Doering
Playing around with the patch a bit, the offending piece seems to be "
mi->context.options.verbosity>=D_DCO_DEBUG" - which is unsurprising, asD_DCO_DEBUGis not "6" but "LOGLEV(6, 69, M_DEBUG)", which translates to#define LOGLEV(log_level, mute_level, other) ((log_level) | ENCODE_MUTE_LEVEL(mute_level) | other)(numerically,
D_DCO_DEBUG= 1157628038)
... looking at error.h, there is a "check_debug_level(flags)" thing, so changing the condition to
+ if (mi->context.c2.tls_multi
+ // && mi->context.options.verbosity >= D_DCO_DEBUG + && check_debug_level(D_DCO_DEBUG) + && dco_enabled(&mi->context.options))
+ {
+ buf_printf(&out, " peer-id=%d", mi->context.c2.tls_multi->peer_id);
+ }
works - "if --verb 6 or higher, *and* DCO, then have peer-id=%d as part if the prefix".
Can you resend a v3?
gert
Signature · 6 lines
"If was one thing all people took for granted, was conviction that if you
feed honest figures into a computer, honest figures come out. Never doubted
it myself till I met a computer with a sense of humor."
Robert A. Heinlein, The Moon is a Harsh Mistress
Gert Doering - Munich, Germany [email protected]...