Skip to contentSkip to conversation

Another take on preserving SIGTERM

PATCH In review Selva Nair 2 participants 5 messages Jun 7, 2016

lore
Share & tools
lore

Selva Nair

author
#0 lore raw

cover Another take on preserving SIGTERM

Hi,

Here is an alternative to my previous patch addressing the same point:

"Save exit-notify timer during a restart"
(http://article.gmane.org/gmane.network.openvpn.devel/11807)

This one (patch in next email) may be a more palatable approach.

Selva

Selva Nair (1):
Ignore SIGUSR1/SIGHUP during exit notification

 src/openvpn/sig.c |   21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)
Signature · 1 line
1.7.10.4
author
#1 lore raw

PATCH 1/1 Ignore SIGUSR1/SIGHUP during exit notification

This allows exit notification to complete and finally trigger SIGTERM. The current practice of allowing a restart in this state clears the exit notification timer data and thus loses the SIGTERM.

Trac #687

---
 src/openvpn/sig.c |   25 ++++++++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)
src/openvpn/sig.c +24 -1
index f903fc0..718b786 100644
@@ -376,12 +376,35 @@ process_sigterm (struct context *c)
   return ret;
 }
 
+/**
+ * If a restart signal is received during exit-notification, reset the
+ * signal and return true.
+ */
+static bool
+ignore_restart_signals (struct context *c)
+{
+  bool ret = false;
+#ifdef ENABLE_OCC
+  if ( (c->sig->signal_received == SIGUSR1 || c->sig->signal_received == SIGHUP) &&
+        event_timeout_defined(&c->c2.explicit_exit_notification_interval) )
+    {
+       msg (M_INFO, "Ignoring %s received during exit notification",
+            signal_name(c->sig->signal_received, true));
+       signal_reset (c->sig);
+       ret = true;
+    }
+#endif
+  return ret;
+}
+
 bool
 process_signal (struct context *c)
 {
   bool ret = true;
 
-  if (c->sig->signal_received == SIGTERM || c->sig->signal_received == SIGINT)
+  if (ignore_restart_signals (c))
+    ret = false;
+  else if (c->sig->signal_received == SIGTERM || c->sig->signal_received == SIGINT)
     {
       ret = process_sigterm (c);
     }
Signature · 1 line
1.7.10.4
#2 lore raw

Re: [Openvpn-devel] [PATCH 1/1] Ignore SIGUSR1/SIGHUP during exit notification

Hi,

Quoted 3 lines · Selva Nair

This allows exit notification to complete and finally trigger SIGTERM. The current practice of allowing a restart in this state clears the exit notification timer data and thus loses the SIGTERM.

This is along the lines I was thinking as well, so I like it better than the first try (which was sort of fiddling with the symptoms, but not with the underlying signal handling problem).

What if SIGTERM+SIGUSR1 are received in very quick succession, so that we haven't even entered "explicit_exit_notification_interval" yet?

I'm not sure if I understand the full chain of events that happen on a (soft or hard) SIGTERM, what state ->signal_received has when the exit notification timer is active, and under which circumstances get_signal() is called to collect signal info into the context... this is magic stuff...

My original idea was "the place that sets signal_received will not do that for SIGUSR1 if a SIGTERM has already been received" (so, de-coupled from the exit notification timer), but it seems that this is not easy to do. So maybe your patch is the pragmatic way to solve this.

Arne?

gert

Signature · 4 lines
USENET is *not* the non-clickable part of WWW!
                                                           //www.muc.de/~gert/
Gert Doering - Munich, Germany                             [email protected]...
fax: +49-89-35655025                        [email protected]...
author
#3 lore raw

Re: [Openvpn-devel] [PATCH 1/1] Ignore SIGUSR1/SIGHUP during exit notification

Quoted 7 lines · Gert Doering

Hi,

Quoted 3 lines · Selva Nair

This allows exit notification to complete and finally trigger SIGTERM. The current practice of allowing a restart in this state clears the exit notification timer data and thus loses the SIGTERM.

Quoted 7 lines

This is along the lines I was thinking as well, so I like it better than the first try (which was sort of fiddling with the symptoms, but not with the underlying signal handling problem).

What if SIGTERM+SIGUSR1 are received in very quick succession, so that we haven't even entered "explicit_exit_notification_interval" yet?

This patch only addresses the issue of restart if exit notify is ongoing. The second problem of signals getting overwritten before getting processed has to be addressed independently. I am not yet sure how best to do this as there are many places and many ways this can happen.

Quoted 7 lines

I'm not sure if I understand the full chain of events that happen on a (soft or hard) SIGTERM, what state ->signal_received has when the exit notification timer is active, and under which circumstances get_signal() is called to collect signal info into the context... this is magic stuff...

In most cases, the signal gets noticed through the EVENT_LOOP_CHECK_SIG (same as P2P_CHECK_SIG) which breaks the loop or through 'if (IS_SIG) return'.
When exit notification is ongoing, signal_received is zero: SIGTERM received --> processed --> removed and exit-notify started. This is needed to make sure we don't exit until the notification is completed. The SIGTERM gets re-instated (signal_received = SIGTERM) when the notification completes and then we'll exit the next time the signal is checked.

This patch handles only the period when exit-notify is ongoing so a SIGTERM is impending.

Quoted 5 lines

My original idea was "the place that sets signal_received will not do that for SIGUSR1 if a SIGTERM has already been received" (so, de-coupled from the exit notification timer), but it seems that this is not easy to do. So maybe your patch is the pragmatic way to solve this.

We still have to do something like that but as you say its hard as the signal is set in different ways -- through throw_signal, register_signal, directly over-writing signal_received etc.. And no solution is going to be perfect as checking and setting the signal needs two operations during which a hard signal may be delivered. But that is something we can live with, I suppose -- I do not mind having to press ctrl-c twice to get noticed. But we should make sure soft signals (those internally generated or received through the management interface) are never lost or over-written by lower priority signals.

Selva

#4 lore raw

[Openvpn-devel] [PATCH applied] Re: Ignore SIGUSR1/SIGHUP during exit notification

In reply to Selva Nair

ACK.

Stared at the code, and managed to reproduce the race by pressing ctrl-c at exactly the right spot (shortly before the "pull-filter reject" would trigger a SIGUSR1) and saw my ctrl-c being ignored - with the patch, things work as expected:

Tue Jun 7 23:00:08 2016 us=714409 Ignoring SIGUSR1 received during exit notification

As we have discussed, this is not "the perfect solution" yet, but I think it solves the "large" race condition (during the explicit-exit-notify time window) without making code elsewhere even more complicated. We should continue working on this, though, and eventually get rid of this workaround again :-)

As this is bugfix it goes into release/2.3.

Your patch has been applied to the master and release/2.3 branch.

commit 63b3e000c9141f4ca03a374354da26334257bc18 (master) commit 859c9f412455665635964519f3e44a0b4151a787 (release/2.3)

Author: Selva Nair
Date:   Tue Jun 7 00:44:20 2016 -0400
     Ignore SIGUSR1/SIGHUP during exit notification

     Signed-off-by: Selva Nair <[email protected]...>
     Acked-by: Gert Doering <[email protected]...>
     Message-Id: <[email protected]...>
     URL: http://article.gmane.org/gmane.network.openvpn.devel/11814
     Signed-off-by: Gert Doering <[email protected]...>

--
kind regards,

Gert Doering