systemd: Enable systemd's auto-restart feature for server profiles
PATCH 1/1 systemd: Enable systemd's auto-restart feature for server profiles
Systemd supervises services it has started and can act upon unexpected scenarios. This change will restart OpenVPN after 5 seconds if the OpenVPN process exits unexpectedly.
The on-failure mode is the recommended mode by upstream systemd.
This change have been tested on a test server for some month, and it works indeed as intended when provoking the OpenVPN process to stop.
--- distro/systemd/[email protected]... | 2 ++ 1 file changed, 2 insertions(+)
distro/systemd/[email protected]... +2 -0
@@ -18,6 +18,8 @@ DeviceAllow=/dev/net/tun rw ProtectSystem=true ProtectHome=true KillMode=process +RestartSec=5s +Restart=on-failure [Install] WantedBy=multi-user.target
Signature · 1 line
2.13.5
Hi,
distro/systemd/[email protected]... DeviceAllow=/dev/net/tun rw · line 22
Quoted 6 lines · David Sommerseth
distro/systemd/[email protected]... +2
-0@@ -18,6 +18,8 @@ DeviceAllow=/dev/net/tun rw ProtectSystem=true ProtectHome=true KillMode=process +RestartSec=5s +Restart=on-failure
Is there a way to get exponential backoff on restart?
Restarting is good, but if there is something faulty that leads to "the process always dies right away", this can lead to very quickly filling disks with not-so-useful logging...
(Otherwise, yes, restarting is good :-) )
gert
Signature · 4 lines
USENET is *not* the non-clickable part of WWW!
//www.muc.de/~gert/
Gert Doering - Munich, Germany [email protected]...
fax: +49-89-35655025 [email protected]...Quoted 12 lines · Gert Doering
Hi,
distro/systemd/[email protected]... DeviceAllow=/dev/net/tun rw · line 22
Quoted 6 lines · David Sommerseth
distro/systemd/[email protected]... +2
-0@@ -18,6 +18,8 @@ DeviceAllow=/dev/net/tun rw ProtectSystem=true ProtectHome=true KillMode=process +RestartSec=5s +Restart=on-failureIs there a way to get exponential backoff on restart?
Quoted 6 lines
Restarting is good, but if there is something faulty that leads to "the process always dies right away", this can lead to very quickly filling disks with not-so-useful logging...
(Otherwise, yes, restarting is good :-) )
Hi,
From systemd.unit man-page[1]:
StartLimitIntervalSec=, StartLimitBurst=
Configure unit start rate limiting. By default, units which are started more than 5 times within 10 seconds are not permitted to start any more times until the 10 second interval ends.
I verified this behavior on CentOS 7 using another daemon (monit) by setting "Restart=on-failure" for it, breaking its config file and forcibly killing it. Note that RestartSec is the default, i.e. 100ms:
---
Sep 07 09:55:37 centos-7 systemd[1]: monit.service: control process
exited, code=exited status=1
Sep 07 09:55:37 centos-7 systemd[1]: Unit monit.service entered failed
state.
Sep 07 09:55:37 centos-7 systemd[1]: monit.service holdoff time over,
scheduling restart.
Sep 07 09:55:37 centos-7 systemd[1]: Stopping Pro-active monitoring
utility for unix systems...
Sep 07 09:55:37 centos-7 systemd[1]: Starting Pro-active monitoring
utility for unix systems...
Sep 07 09:55:37 centos-7 systemd[1]: monit.service start request
repeated too quickly, refusing to start.
Sep 07 09:55:37 centos-7 systemd[1]: Failed to start Pro-active
monitoring utility for unix systems.
Sep 07 09:55:37 centos-7 systemd[1]: Unit monit.service entered failed
state.
---
As you can see, systemd quickly realizes that monit will not come back up and stops trying.
However, when I added "RestartSec=5s" the StartLimit* thresholds were never triggered. This meant that systemd never ceased trying to restart the monit service.
David: any particular reason why you added RestartSec? Why not just let it be the default (100ms)?
Signature · 7 lines
Samuli Seppänen Community Manager OpenVPN Technologies, Inc irc freenode net: mattock [1] <https://www.freedesktop.org/software/systemd/man/systemd.unit.html>
Quoted 65 lines · Samuli Seppänen
Quoted 12 lines · Gert Doering
Hi,
distro/systemd/[email protected]... DeviceAllow=/dev/net/tun rw · line 22
Quoted 6 lines · David Sommerseth
distro/systemd/[email protected]... +2
-0@@ -18,6 +18,8 @@ DeviceAllow=/dev/net/tun rw ProtectSystem=true ProtectHome=true KillMode=process +RestartSec=5s +Restart=on-failureIs there a way to get exponential backoff on restart?
Quoted 6 lines
Restarting is good, but if there is something faulty that leads to "the process always dies right away", this can lead to very quickly filling disks with not-so-useful logging...
(Otherwise, yes, restarting is good :-) )
Hi,
From systemd.unit man-page[1]:
StartLimitIntervalSec=, StartLimitBurst=
Configure unit start rate limiting. By default, units which are started more than 5 times within 10 seconds are not permitted to start any more times until the 10 second interval ends.
I verified this behavior on CentOS 7 using another daemon (monit) by setting "Restart=on-failure" for it, breaking its config file and forcibly killing it. Note that RestartSec is the default, i.e. 100ms:
---
Sep 07 09:55:37 centos-7 systemd[1]: monit.service: control processexited, code=exited status=1
Sep 07 09:55:37 centos-7 systemd[1]: Unit monit.service entered failedstate.
Sep 07 09:55:37 centos-7 systemd[1]: monit.service holdoff time over,scheduling restart.
Sep 07 09:55:37 centos-7 systemd[1]: Stopping Pro-active monitoringutility for unix systems...
Sep 07 09:55:37 centos-7 systemd[1]: Starting Pro-active monitoringutility for unix systems...
Sep 07 09:55:37 centos-7 systemd[1]: monit.service start requestrepeated too quickly, refusing to start.
Sep 07 09:55:37 centos-7 systemd[1]: Failed to start Pro-activemonitoring utility for unix systems.
Sep 07 09:55:37 centos-7 systemd[1]: Unit monit.service entered failedstate.
---
As you can see, systemd quickly realizes that monit will not come back up and stops trying.
Quoted 7 lines
However, when I added "RestartSec=5s" the StartLimit* thresholds were never triggered. This meant that systemd never ceased trying to restart the monit service.
David: any particular reason why you added RestartSec? Why not just let it be the default (100ms)?
Actually the man-page quote above is a bit ambiguous. But later on the man-page explain the logic pretty clearly:
"Note that units which are configured for Restart= and which reach the start limit are not attempted to be restarted anymore; however, they may still be restarted manually at a later point, from which point on, the restart logic is again activated."
Signature · 5 lines
Samuli Seppänen Community Manager OpenVPN Technologies, Inc irc freenode net: mattock
Hi,
Quoted 4 lines · Samuli Seppänen
"Note that units which are configured for Restart= and which reach the start limit are not attempted to be restarted anymore; however, they may still be restarted manually at a later point, from which point on, the restart logic is again activated."
Which is not what I hoped for... "turn it off and leave it so" is non helpful (it might be a transient error preventing the startup).
Meh.
gert
Signature · 4 lines
USENET is *not* the non-clickable part of WWW!
//www.muc.de/~gert/
Gert Doering - Munich, Germany [email protected]...
fax: +49-89-35655025 [email protected]...Quoted 11 lines · Gert Doering
Hi,
Quoted 4 lines · Samuli Seppänen
"Note that units which are configured for Restart= and which reach the start limit are not attempted to be restarted anymore; however, they may still be restarted manually at a later point, from which point on, the restart logic is again activated."
Which is not what I hoped for... "turn it off and leave it so" is non helpful (it might be a transient error preventing the startup).
Good point. Systemd seems to be able to adjust its restart behavior depending on exit code of the main service process (i.e. OpenVPN) using "RestartForceExitStatus" and "RsstartPreventExitStatus"[1]. Perhaps these could be helpful in our case...
Signature · 7 lines
Samuli Seppänen Community Manager OpenVPN Technologies, Inc irc freenode net: mattock [1] <https://www.freedesktop.org/software/systemd/man/systemd.service.html>
Quoted 9 lines · Samuli Seppänen
Quoted 8 lines · Gert Doering
Hi,
Quoted 4 lines · Samuli Seppänen
"Note that units which are configured for Restart= and which reach the start limit are not attempted to be restarted anymore; however, they may still be restarted manually at a later point, from which point on, the restart logic is again activated."
Quoted 8 lines
Quoted 3 lines
Which is not what I hoped for... "turn it off and leave it so" is non helpful (it might be a transient error preventing the startup).
Good point. Systemd seems to be able to adjust its restart behavior depending on exit code of the main service process (i.e. OpenVPN) using "RestartForceExitStatus" and "RsstartPreventExitStatus"[1]. Perhaps these could be helpful in our case...
Okay, lets try to align what OpenVPN does in various scenarios and compare it how we want restarts to happen.
First, have a look at the man page:
<https://www.freedesktop.org/software/systemd/man/systemd.service.html#Restart=>
Then realise that OpenVPN (AFAIK) only operates with two exit codes:
#define OPENVPN_EXIT_STATUS_GOOD 0 #define OPENVPN_EXIT_STATUS_ERROR 1 #define OPENVPN_EXIT_STATUS_USAGE 1 #define OPENVPN_EXIT_STATUS_CANNOT_OPEN_DEBUG_FILE 1
(from error.h)
So the RestartForceExitStatus/RestartPreventExitStatus is not going to be helpful if all graceful errors results in 1, which is the most common way OpenVPN stops - through the M_FATAL. So that leaves us with SIGSEGV, SIGABRT and similar unclean exit signals.
To avoid restarts on faulty configurations or if we can define scenarios where we do not want OpenVPN to be restarted automatically, we need to introduce more exit codes. This way we can implicitly tell systemd if it should restart OpenVPN or not.
Signature · 4 lines
kind regards, David Sommerseth OpenVPN Technologies, Inc
Hi,
Quoted 9 lines · David Sommerseth
So the RestartForceExitStatus/RestartPreventExitStatus is not going to be helpful if all graceful errors results in 1, which is the most common way OpenVPN stops - through the
M_FATAL. So that leaves us with SIGSEGV, SIGABRT and similar unclean exit signals.To avoid restarts on faulty configurations or if we can define scenarios where we do not want OpenVPN to be restarted automatically, we need to introduce more exit codes. This way we can implicitly tell systemd if it should restart OpenVPN or not.
Indeed. Just to agree publically here :-) - it seems we'll need to revisit exit codes as well, for "restarting this won't do any good" (fatal config errors, for example) vs. "we have crashed, and please do restart!!" (out of memory as something obvious we can't easily catch, most other error situations wrt network *should* not cause a M_FATAL, even if some still do).
Where do we keep this as "we need to look into it, but it's complicated" reminder? trac?
gert
Signature · 4 lines
USENET is *not* the non-clickable part of WWW!
//www.muc.de/~gert/
Gert Doering - Munich, Germany [email protected]...
fax: +49-89-35655025 [email protected]...Quoted 20 lines · Gert Doering
Hi,
Quoted 9 lines · David Sommerseth
So the RestartForceExitStatus/RestartPreventExitStatus is not going to be helpful if all graceful errors results in 1, which is the most common way OpenVPN stops - through the
M_FATAL. So that leaves us with SIGSEGV, SIGABRT and similar unclean exit signals.To avoid restarts on faulty configurations or if we can define scenarios where we do not want OpenVPN to be restarted automatically, we need to introduce more exit codes. This way we can implicitly tell systemd if it should restart OpenVPN or not.
Indeed. Just to agree publically here :-) - it seems we'll need to revisit exit codes as well, for "restarting this won't do any good" (fatal config errors, for example) vs. "we have crashed, and please do restart!!" (out of memory as something obvious we can't easily catch, most other error situations wrt network *should* not cause a
M_FATAL, even if some still do).
Quoted 2 lines
Where do we keep this as "we need to look into it, but it's complicated" reminder? trac?
Yeah, a trac ticket is probably the best.
Signature · 4 lines
kind regards, David Sommerseth OpenVPN Technologies, Inc
Quoted 10 lines · Gert Doering
Hi,
Quoted 4 lines · Samuli Seppänen
"Note that units which are configured for Restart= and which reach the start limit are not attempted to be restarted anymore; however, they may still be restarted manually at a later point, from which point on, the restart logic is again activated."
Which is not what I hoped for... "turn it off and leave it so" is non helpful (it might be a transient error preventing the startup).
I'm confused. What is it you want?
* try restarting in an endless loop?
* try restarting X times and then stop trying?
Signature · 4 lines
kind regards, David Sommerseth OpenVPN Technologies, Inc
Hi,
Quoted 7 lines · David Sommerseth
Quoted 2 lines
Which is not what I hoped for... "turn it off and leave it so" is non helpful (it might be a transient error preventing the startup).
I'm confused. What is it you want?
* try restarting in an endless loop?
* try restarting X times and then stop trying?
"exponential decay"
Restart a few times quickly, and then slow down, like "re-try after 1s, 3s, 5s, 10s, 15s, 30s, 60s, ... and then stick to every-5-minutes" or so (like OpenVPN's own connection re-try logic :-) ).
That way you get quick restart if a temporary failure happens, and avoid filling logs with useless crap if the error persists - like "bind to an IP address that is not present in the system, because the ppp0 interface where it would show up is down", or something like this.
But if that cannot be done, that's how it is - and in that case, "stop trying" is not what I want.
gert
Signature · 4 lines
USENET is *not* the non-clickable part of WWW!
//www.muc.de/~gert/
Gert Doering - Munich, Germany [email protected]...
fax: +49-89-35655025 [email protected]...Quoted 8 lines · Gert Doering
Hi,
Quoted 5 lines · David Sommerseth
Quoted 2 lines
Which is not what I hoped for... "turn it off and leave it so" is non helpful (it might be a transient error preventing the startup).
I'm confused. What is it you want?
Quoted 8 lines
Quoted 2 lines
* try restarting in an endless loop?
* try restarting X times and then stop trying?"exponential decay"
Restart a few times quickly, and then slow down, like "re-try after 1s, 3s, 5s, 10s, 15s, 30s, 60s, ... and then stick to every-5-minutes" or so (like OpenVPN's own connection re-try logic :-) ).
Yes, that is a nice idea. But to my knowledge, that is not something systemd supports today. So we can't have that easily. We could look into a ExcePreStart=, which tracks how long time it has gone since the last restart and adds the "exponential decay" sleep. But that means more code to maintain.
Quoted 7 lines
That way you get quick restart if a temporary failure happens, and avoid filling logs with useless crap if the error persists - like "bind to an IP address that is not present in the system, because the ppp0 interface where it would show up is down", or something like this.
But if that cannot be done, that's how it is - and in that case, "stop trying" is not what I want.
Which means the current patch is what is possible to achieve today. :)
Signature · 4 lines
kind regards, David Sommerseth OpenVPN Technologies, Inc
Quoted 66 lines · Samuli Seppänen
Quoted 65 lines · Samuli Seppänen
Quoted 12 lines · Gert Doering
Hi,
distro/systemd/[email protected]... DeviceAllow=/dev/net/tun rw · line 22
Quoted 6 lines · David Sommerseth
distro/systemd/[email protected]... +2
-0@@ -18,6 +18,8 @@ DeviceAllow=/dev/net/tun rw ProtectSystem=true ProtectHome=true KillMode=process +RestartSec=5s +Restart=on-failureIs there a way to get exponential backoff on restart?
Quoted 6 lines
Restarting is good, but if there is something faulty that leads to "the process always dies right away", this can lead to very quickly filling disks with not-so-useful logging...
(Otherwise, yes, restarting is good :-) )
Hi,
From systemd.unit man-page[1]:
StartLimitIntervalSec=, StartLimitBurst=
Configure unit start rate limiting. By default, units which are started more than 5 times within 10 seconds are not permitted to start any more times until the 10 second interval ends.
I verified this behavior on CentOS 7 using another daemon (monit) by setting "Restart=on-failure" for it, breaking its config file and forcibly killing it. Note that RestartSec is the default, i.e. 100ms:
---
Sep 07 09:55:37 centos-7 systemd[1]: monit.service: control processexited, code=exited status=1
Sep 07 09:55:37 centos-7 systemd[1]: Unit monit.service entered failedstate.
Sep 07 09:55:37 centos-7 systemd[1]: monit.service holdoff time over,scheduling restart.
Sep 07 09:55:37 centos-7 systemd[1]: Stopping Pro-active monitoringutility for unix systems...
Sep 07 09:55:37 centos-7 systemd[1]: Starting Pro-active monitoringutility for unix systems...
Sep 07 09:55:37 centos-7 systemd[1]: monit.service start requestrepeated too quickly, refusing to start.
Sep 07 09:55:37 centos-7 systemd[1]: Failed to start Pro-activemonitoring utility for unix systems.
Sep 07 09:55:37 centos-7 systemd[1]: Unit monit.service entered failedstate.
---
As you can see, systemd quickly realizes that monit will not come back up and stops trying.
Quoted 6 lines
Quoted 6 lines
However, when I added "RestartSec=5s" the StartLimit* thresholds were never triggered. This meant that systemd never ceased trying to restart the monit service.
David: any particular reason why you added RestartSec? Why not just let it be the default (100ms)?
Partly to avoid respawning too fast. We don't know what kind of additional plug-ins or management interface tools have been integrated and how they react if OpenVPN goes into a tight restart-loop. And partly to _escape_ the stopping of restarts.
I'm not sure I'm easily buying into the "faulty configuration" argument.
Because these restart scenarios mostly happens when a sys-admin is
_not_ around. If you're playing with configurations, do a restart to test the new config, you are around to handle a failed situation. The only area where this can fail is when we break options and OpenVPN get updated automatically. But in both these scenarios, a restart delay of 5 seconds won't cause too much stress on the system.
I also opted for not much longer delay, as if a restart happens successfully, most users won't notice that too much. If it is 30 seconds or 1 minute, that is much more noticeable. But I'm open for adjusting this too.
Signature · 4 lines
kind regards, David Sommerseth OpenVPN Technologies, Inc
Quoted 74 lines · David Sommerseth
Quoted 66 lines · Samuli Seppänen
Quoted 65 lines · Samuli Seppänen
Quoted 12 lines · Gert Doering
Hi,
distro/systemd/[email protected]... DeviceAllow=/dev/net/tun rw · line 22
Quoted 6 lines · David Sommerseth
distro/systemd/[email protected]... +2
-0@@ -18,6 +18,8 @@ DeviceAllow=/dev/net/tun rw ProtectSystem=true ProtectHome=true KillMode=process +RestartSec=5s +Restart=on-failureIs there a way to get exponential backoff on restart?
Quoted 6 lines
Restarting is good, but if there is something faulty that leads to "the process always dies right away", this can lead to very quickly filling disks with not-so-useful logging...
(Otherwise, yes, restarting is good :-) )
Hi,
From systemd.unit man-page[1]:
StartLimitIntervalSec=, StartLimitBurst=
Configure unit start rate limiting. By default, units which are started more than 5 times within 10 seconds are not permitted to start any more times until the 10 second interval ends.
I verified this behavior on CentOS 7 using another daemon (monit) by setting "Restart=on-failure" for it, breaking its config file and forcibly killing it. Note that RestartSec is the default, i.e. 100ms:
---
Sep 07 09:55:37 centos-7 systemd[1]: monit.service: control processexited, code=exited status=1
Sep 07 09:55:37 centos-7 systemd[1]: Unit monit.service entered failedstate.
Sep 07 09:55:37 centos-7 systemd[1]: monit.service holdoff time over,scheduling restart.
Sep 07 09:55:37 centos-7 systemd[1]: Stopping Pro-active monitoringutility for unix systems...
Sep 07 09:55:37 centos-7 systemd[1]: Starting Pro-active monitoringutility for unix systems...
Sep 07 09:55:37 centos-7 systemd[1]: monit.service start requestrepeated too quickly, refusing to start.
Sep 07 09:55:37 centos-7 systemd[1]: Failed to start Pro-activemonitoring utility for unix systems.
Sep 07 09:55:37 centos-7 systemd[1]: Unit monit.service entered failedstate.
---
As you can see, systemd quickly realizes that monit will not come back up and stops trying.
Quoted 6 lines
Quoted 6 lines
However, when I added "RestartSec=5s" the StartLimit* thresholds were never triggered. This meant that systemd never ceased trying to restart the monit service.
David: any particular reason why you added RestartSec? Why not just let it be the default (100ms)?
Quoted 6 lines
Partly to avoid respawning too fast. We don't know what kind of additional plug-ins or management interface tools have been integrated and how they react if OpenVPN goes into a tight restart-loop. And partly to _escape_ the stopping of restarts.
I'm not sure I'm easily buying into the "faulty configuration" argument.
There is no "faulty configuration argument". I broke monit's configuration file intentionally for the sole purpose of testing systemd's restart functionality.
Quoted 11 lines
Because these restart scenarios mostly happens when a sys-admin is
_not_ around. If you're playing with configurations, do a restart to test the new config, you are around to handle a failed situation. The only area where this can fail is when we break options and OpenVPN get updated automatically. But in both these scenarios, a restart delay of 5 seconds won't cause too much stress on the system.I also opted for not much longer delay, as if a restart happens successfully, most users won't notice that too much. If it is 30 seconds or 1 minute, that is much more noticeable. But I'm open for adjusting this too.
Quoted 4 lines · Gert Doering
Restarting is good, but if there is something faulty that leads to "the process always dies right away", this can lead to very quickly filling disks with not-so-useful logging...
Oh, I overlooked this one. Just one comment in regards to the "filling disks". That naturally depends on how logging is configured on the system, and that varies a lot. But my experience based on defaults in the environments I use:
* systemd journal have some reasonably sane defaults to avoid this to happen; IIRC it defaults to rotate the journal when reaching 10-15% of available disk space *or* 4GB of log data.
* RHEL (and clones) usually have rsyslog installed too, which the journal forwards log data too. And it most commonly it also have logrotate installed too (at least on the server variant) which runs on a regular basis. But it also depends on how big the partition where /var/log resides is.
So the risks for such restarts to cause full disks should be fairly minimal. And for those who have enabled remote logging, that is often to pay more attention to log events, so then such scenarios would probably be detected even quicker.
Signature · 4 lines
kind regards, David Sommerseth OpenVPN Technologies, Inc
Hi,
Quoted 3 lines · David Sommerseth
Systemd supervises services it has started and can act upon unexpected scenarios. This change will restart OpenVPN after 5 seconds if the OpenVPN process exits unexpectedly.
Define "unexpectedly"
(my2c: something needs to be fixed)
Quoted 1 line
The on-failure mode is the recommended mode by upstream systemd.
IE:
This is the way "upstream systemd" recommend to use systemd *if* "some failed process" is to be restarted.
(my2c: A decision better left to the sys-admin not openvpn developer)
Quoted 2 lines
This change have been tested on a test server for some month, and it works indeed as intended when provoking the OpenVPN process to stop.
Define "provoking"
(my2c: not unexpectedly at all, nothing to fix here)
Either way,
it is only a few lines of a systemd unit.service file to be deleted/customised by the relevant sys-admin.
Regards,
Quoted 10 lines · fragmentux
Hi,
Quoted 4 lines · David Sommerseth
Systemd supervises services it has started and can act upon unexpected scenarios. This change will restart OpenVPN after 5 seconds if the OpenVPN
process exits unexpectedly.Define "unexpectedly"
(my2c: something needs to be fixed)
Yes, that _could_ be, but is not for sure. It is hard to define "unexpectedly". Or to flip it around, if we could define "unexpectedly" properly, there is nothing unexpected happening.
But _some_ scenarios
- The out-of-memory killer picks an openvpn process
Fix: Add more memory, reduce the number of memory hungry processes, usually nothing OpenVPN can do anything with
- The server process segfaults dies (segfault)
This happens most commonly due to poor/badly written plug-ins
Fix: Get an updated/fixed plug-in, not something OpenVPN can fully control.
- A fatal error happened
Which can have many reasons (grep M_FATAL in the source), most of them it is safe to restart the OpenVPN process. It could be caused by the crypto layer not getting enough random data, it could be a malicious attempt of an attack, triggering some of the ASSERT() checks in the code. M_FATAL in general is more of the error type "It is not safe to continue running this process" but it doesn't mean it isn't safe to restart the process. Again, situations outside OpenVPN influences its ability to run.
And there's a plethora of other scenarios too.
So to define "unexpectedly" is not far away from trying to define "life".
Quoted 7 lines
Quoted 1 line
The on-failure mode is the recommended mode by upstream systemd.
IE:
This is the way "upstream systemd" recommend to use systemd *if* "some failed process" is to be restarted.(my2c: A decision better left to the sys-admin not openvpn developer)
Why is it bad to have this behaviour being the default? The sys-admin still have the ability to override this.
And even though I have the role of developer here in the OpenVPN community, outside of my day-work role I do also have the role as a sys-admin. From my point of view, this isn't a "this sounds like a cool feature". It is actually something I have enabled on systems manually for some time, so if my smaller scale environment finds this useful - why wouldn't others find it useful? And also having had sys-admin roles in former work life, this is something we deployed on many services in far larger environments.
Quoted 5 lines
Quoted 2 lines
This change have been tested on a test server for some month, and it works indeed as intended when provoking the OpenVPN process to stop.
Define "provoking"
(my2c: not unexpectedly at all, nothing to fix here)
Like adding a plug-in which triggers a segfault on the first connection attempt. Or sending a SIGKILL to the process. This was needed to test and verify that the restart mechanism did work. Otherwise I would not have had confidence in believing this was done correctly. And even though that my triggering isn't unexpected, it is unexpected by the OpenVPN process.
But sure, if you have a way to trigger OpenVPN to unexpectedly failing, we'd be delighted to hear about it .... as that is most likely something we should fix in OpenVPN ;-)
In case you haven't noticed, OpenVPN is quite stable and seldom fails by itself. But when it does, it is for most end-users depending on it a nice feature that things do try to recover on its own.
These events do also not happen in complete silence. They are logged. And a responsible sys-admin will check logs at regular intervals, some even have automated alerts happening. But most sys-admins will be pleased that they don't have to go into panic rescue mode instantly because a VPN server went down and they have 50 users on the phone complaining they can't reach the intranet.
Quoted 3 lines
Either way,
it is only a few lines of a systemd unit.service file to be deleted/customised by the relevant sys-admin.
systemctl edit [email protected]...
then add Restart=no
Which have the advantage of disabling this feature on selected configurations, not all configs by default. If you want the default, I believe you just do: systemctl edit openvpn-server@
Signature · 4 lines
kind regards, David Sommerseth OpenVPN Technologies, Inc
Hi,
all your comment are totally valid from a sys-admin point of view but from an openvpn POV, the only responsibility is to provide a secure VPN.
Use all of systemd's functions to maximize openvpn's process *security* But *forcing* restart as an almost unconditional default is nonsense. How would you do this for non-systemd systems ?
I disagree with making this change to the default
[email protected]... unit file.
If you really want to include them then how about:
Either:
[email protected]... (responsible for start/stop etc actions) [email protected]... (speaks for itself)
Or rather
include extra .service files in ./contrib. as samples or such.
regards,
Quoted 102 lines · David Sommerseth
Quoted 10 lines · fragmentux
Hi,
Quoted 4 lines · David Sommerseth
Systemd supervises services it has started and can act upon unexpected scenarios. This change will restart OpenVPN after 5 seconds if the OpenVPN
process exits unexpectedly.Define "unexpectedly"
(my2c: something needs to be fixed)Yes, that _could_ be, but is not for sure. It is hard to define "unexpectedly". Or to flip it around, if we could define "unexpectedly" properly, there is nothing unexpected happening.
But _some_ scenarios
- The out-of-memory killer picks an openvpn process
Fix: Add more memory, reduce the number of memory hungry processes, usually nothing OpenVPN can do anything with- The server process segfaults dies (segfault)
This happens most commonly due to poor/badly written plug-ins
Fix: Get an updated/fixed plug-in, not something OpenVPN can fully control.- A fatal error happened
Which can have many reasons (grepM_FATALin the source), most of them it is safe to restart the OpenVPN process. It could be caused by the crypto layer not getting enough random data, it could be a malicious attempt of an attack, triggering some of theASSERT()checks in the code.M_FATALin general is more of the error type "It is not safe to continue running this process" but it doesn't mean it isn't safe to restart the process. Again, situations outside OpenVPN influences its ability to run.And there's a plethora of other scenarios too.
So to define "unexpectedly" is not far away from trying to define "life".
Quoted 7 lines
Quoted 1 line
The on-failure mode is the recommended mode by upstream systemd.
IE:
This is the way "upstream systemd" recommend to use systemd *if* "some failed process" is to be restarted.(my2c: A decision better left to the sys-admin not openvpn developer)
Why is it bad to have this behaviour being the default? The sys-admin still have the ability to override this.
And even though I have the role of developer here in the OpenVPN community, outside of my day-work role I do also have the role as a sys-admin. From my point of view, this isn't a "this sounds like a cool feature". It is actually something I have enabled on systems manually for some time, so if my smaller scale environment finds this useful - why wouldn't others find it useful? And also having had sys-admin roles in former work life, this is something we deployed on many services in far larger environments.
Quoted 5 lines
Quoted 2 lines
This change have been tested on a test server for some month, and it works indeed as intended when provoking the OpenVPN process to stop.
Define "provoking"
(my2c: not unexpectedly at all, nothing to fix here)Like adding a plug-in which triggers a segfault on the first connection attempt. Or sending a SIGKILL to the process. This was needed to test and verify that the restart mechanism did work. Otherwise I would not have had confidence in believing this was done correctly. And even though that my triggering isn't unexpected, it is unexpected by the OpenVPN process.
But sure, if you have a way to trigger OpenVPN to unexpectedly failing, we'd be delighted to hear about it .... as that is most likely something we should fix in OpenVPN ;-)
In case you haven't noticed, OpenVPN is quite stable and seldom fails by itself. But when it does, it is for most end-users depending on it a nice feature that things do try to recover on its own.
These events do also not happen in complete silence. They are logged. And a responsible sys-admin will check logs at regular intervals, some even have automated alerts happening. But most sys-admins will be pleased that they don't have to go into panic rescue mode instantly because a VPN server went down and they have 50 users on the phone complaining they can't reach the intranet.
Quoted 3 lines
Either way,
it is only a few lines of a systemd unit.service file to be deleted/customised by the relevant sys-admin.systemctl edit [email protected]...
then add Restart=noWhich have the advantage of disabling this feature on selected configurations, not all configs by default. If you want the default, I believe you just do: systemctl edit openvpn-server@
On 07/09/17 23:02, fragmentux wrote:
i,
Quoted 6 lines
all your comment are totally valid from a sys-admin point of view but from an openvpn POV, the only responsibility is to provide a secure VPN.
Use all of systemd's functions to maximize openvpn's process *security* But *forcing* restart as an almost unconditional default is nonsense.
We are in the position to promote sane and good defaults. This behaviour is considered sane and good by many sys-admins. So when these two view-points intersects, I see no harm of us actually promoting this change.
Quoted 1 line
How would you do this for non-systemd systems ?
Isn't that obvious? systemd unit files are for systemd. Non-systemd systems doesn't have systemd unit files, thus there is very little we can do about them.
Quoted 2 lines
I disagree with making this change to the default
[email protected]... unit file.
Your opposition have been noted.
Quoted 5 lines
If you really want to include them then how about:
Either:
[email protected]... (responsible for start/stop etc actions) [email protected]... (speaks for itself)
NAK. This is not how the design around systemd unit files is intended to be used. Plus: it already exists a Debian bug ticket where there are comments about us adding 2 more unit files. If adding even more, I can already sense the heat increasing on that ticket.
Quoted 2 lines
Or rather
include extra .service files in ./contrib. as samples or such.
NAK. I rather have a document simply describing how to change the defaults using 'systemctl edit'. Which is exactly how systemd is designed to be used. But we should have a baseline of recommended defaults, and sys-admins can choose to opt-out of these defaults through standard mechanisms, not by adding complexity through more unit files to scan through.
Just image a system which actively uses both openvpn-server@ and [email protected]... Unless we also split up /etc/openvpn/server ... it will be even more confusing when investigating a server in 2 years why something is misbehaving. "Did this config run through this or that unit file?". openvpn-server@ is clear and specific, it handles server configurations. Period.
If you want a specific configuration or all openvpn-server@ OpenVPN configurations to behave differently from the recommended defaults, then you do that through 'systemctl edit', where it is very visible if this specific configuration have some additional tweaks not - through 'systemctl status'. This way sys-admins won't have remember or research which 'sub-unit file' of openvpn-server@ to achieve a specific behaviour.
Signature · 4 lines
kind regards, David Sommerseth OpenVPN Technologies, Inc
Hi,
Quoted 6 lines · David Sommerseth
On 07/09/17 23:02, fragmentux wrote:
i,Quoted 4 lines
all your comment are totally valid from a sys-admin point of view but from an openvpn POV, the only responsibility is to provide a secure VPN.
Quoted 7 lines
Quoted 2 lines
Use all of systemd's functions to maximize openvpn's process *security* But *forcing* restart as an almost unconditional default is nonsense.
We are in the position to promote sane and good defaults. This behaviour is considered sane and good by many sys-admins. So when these two view-points intersects, I see no harm of us actually promoting this change.
Do you want to discuss the "sanity" of said sys-admins ?
I do not subscribe to "popular POV" .: I disagree with your sys-admins. I believe the restart decision should remain in the hands of the owner.
Quoted 5 lines
Quoted 1 line
How would you do this for non-systemd systems ?
Isn't that obvious? systemd unit files are for systemd. Non-systemd systems doesn't have systemd unit files, thus there is very little we can do about them.
Consistent behaviour is clearly *not* your primary concern for OpenVPN. ^Take Note^
Quoted 4 lines
Quoted 2 lines
I disagree with making this change to the default
[email protected]... unit file.Your opposition have been noted.
Yeah right ..
Quoted 8 lines
Quoted 5 lines
If you really want to include them then how about:
Either:
[email protected]... (responsible for start/stop etc actions) [email protected]... (speaks for itself)NAK. This is not how the design around systemd unit files is intended to be used.
"design around systemd" .. Not OpenVPN
So what does Mr. https://github.com/poettering recommend ?
I prefer my "broken" servers to Fail outright
regardless of what Pot. or you think.
Quoted 3 lines
Plus: it already exists a Debian bug ticket where there are comments about us adding 2 more unit files. If adding even more, I can already sense the heat increasing on that ticket.
Thus .. see Next
Quoted 4 lines
Quoted 2 lines
Or rather
include extra .service files in ./contrib. as samples or such.NAK. I rather have a document
Another badly (if at all) maintained OpenVPN document ..
and lets face facts here .. this is *systemd* not "$(I deleted what I wanted to say .. maybe you can guess)" or some such.
Quoted 7 lines
simply describing how to change the
defaults using 'systemctl edit'. Which is exactly how systemd is designed to be used. But we should have a baseline of recommended defaults, and sys-admins can choose to opt-out of these defaults through standard mechanisms, not by adding complexity through more unit files to scan through.
Auto Restart on Failure for OpenVPN is *not* a recommended default other than by you and I stand against it.
Quoted 1 line
Just image a system
Must I .... ? To what end .... ?
Quoted 6 lines
which actively uses both openvpn-server@ and [email protected]... Unless we also split up
/etc/openvpn/server... it will be even more confusing when investigating a server in 2 years why something is misbehaving. "Did this config run through this or that unit file?". openvpn-server@ is clear and specific, it handles server configurations. Period.
Thus it should *not* be the decision of an openvpn developer to decide if "somebody else's" server should behave like yours ..
IE: Forced Restart .. By default.
*NAK*
Quoted 4 lines
If you want a specific configuration or all openvpn-server@ OpenVPN configurations to behave differently from the recommended defaults, then you do that through 'systemctl edit', where it is very visible if this specific configuration have some additional tweaks not - through
I know enough about systemd thanks.
I reject the idea that *all* servers under the control of /systemd/ should _inherit_ an almost forced restart due to *your* preference.
Quoted 2 lines
'systemctl status'. This way sys-admins won't have remember or research which 'sub-unit file' of openvpn-server@ to achieve a specific behaviour.
"sys-admins won't have to remember" .. word fail me.
People have a job to do ..
My final note:
1. As soon as I read this patch I was against it ..
2. I understand that I am not the most educated of Linux users
3. This "patch" has nothing to to with the function of OpenVPN therefore it should remain that way.
regards,
WRT systemd and 3rd party 'servers'
I am curious to know of any other "Server service" who decide .. to *rely* on ..
a *forced* 'systemd (almost) unconditional restart by default'
Regards,
Quoted 5 lines · fragmentux
WRT systemd and 3rd party 'servers'
I am curious to know of any other "Server service" who decide .. to *rely* on ..
a *forced* 'systemd (almost) unconditional restart by default'
As a basic list:
/usr/lib/systemd/system $ grep -r Restart *.service [email protected]...:Restart=always
[email protected]...:RestartSec=0
console-getty.service:Restart=always
console-getty.service:RestartSec=0
[email protected]...:Restart=always [email protected]...:RestartSec=0 dbus-org.freedesktop.login1.service:Restart=always dbus-org.freedesktop.login1.service:RestartSec=0 debug-shell.service:Restart=always
debug-shell.service:RestartSec=0
[email protected]...:Restart=always
[email protected]...:RestartSec=0
mariadb.service:# Restart crashed server only, on-failure would also restart, for example, when mariadb.service:Restart=on-abort
mariadb.service:RestartSec=5s
NetworkManager.service:Restart=on-failure
rsyslog.service:Restart=on-failure
rsyslog.service:Restart=on-failure
[email protected]...:Restart=always
sshd.service:Restart=on-failure
sshd.service:RestartSec=42s
sssd-autofs.service:Restart=on-failure
sssd-nss.service:Restart=on-failure
sssd-pac.service:Restart=on-failure
sssd-pam.service:Restart=on-failure
sssd-ssh.service:Restart=on-failure
sssd-sudo.service:Restart=on-failure
systemd-journald.service:Restart=always
systemd-journald.service:RestartSec=0
systemd-logind.service:Restart=always
systemd-logind.service:RestartSec=0 [email protected]...:RestartForceExitStatus=133 systemd-udevd.service:Restart=always
systemd-udevd.service:RestartSec=0
Signature · 4 lines
Steven Haigh ? [email protected]... ? http://www.crc.id.au ? +61 (3) 9001 6090 ? 0412 935 897
Hi,
Quoted 10 lines · Steven Haigh
Quoted 5 lines · fragmentux
WRT systemd and 3rd party 'servers'
I am curious to know of any other "Server service" who decide .. to *rely* on ..
a *forced* 'systemd (almost) unconditional restart by default'As a basic list:
/usr/lib/systemd/system$ grep -r Restart *.service
This is the wrong place to grep ..
Quoted 26 lines
[email protected]...:Restart=always
[email protected]...:RestartSec=0
console-getty.service:Restart=always
console-getty.service:RestartSec=0
[email protected]...:Restart=always [email protected]...:RestartSec=0 dbus-org.freedesktop.login1.service:Restart=always dbus-org.freedesktop.login1.service:RestartSec=0 debug-shell.service:Restart=always
debug-shell.service:RestartSec=0
[email protected]...:Restart=always
[email protected]...:RestartSec=0
mariadb.service:# Restart crashed server only, on-failure would also restart, for example, when mariadb.service:Restart=on-abort
mariadb.service:RestartSec=5s
NetworkManager.service:Restart=on-failure
rsyslog.service:Restart=on-failure
rsyslog.service:Restart=on-failure
[email protected]...:Restart=always
sshd.service:Restart=on-failure
sshd.service:RestartSec=42s
sssd-autofs.service:Restart=on-failure
sssd-nss.service:Restart=on-failure
sssd-pac.service:Restart=on-failure
sssd-pam.service:Restart=on-failure
sssd-ssh.service:Restart=on-failure
Quoted 8 lines
sssd-sudo.service:Restart=on-failure systemd-journald.service:Restart=always systemd-journald.service:RestartSec=0 systemd-logind.service:Restart=always systemd-logind.service:RestartSec=0 [email protected]...:RestartForceExitStatus=133 systemd-udevd.service:Restart=always systemd-udevd.service:RestartSec=0
I said Server as in 'Server' not localhost.
(other than sshd ..
but that is a whole `nother ball game, let us not get into that)
cheers :)
Regards,
Quoted 10 lines · fragmentux
Hi,
Quoted 7 lines · Steven Haigh
Quoted 5 lines · fragmentux
WRT systemd and 3rd party 'servers'
I am curious to know of any other "Server service" who decide .. to *rely* on ..
a *forced* 'systemd (almost) unconditional restart by default'
Quoted 6 lines
Quoted 3 lines
As a basic list:
/usr/lib/systemd/system$ grep -r Restart *.serviceThis is the wrong place to grep ..
Oops .. maybe not "the wrong place" ..
but I stand by my objection.
regards,
Hi,
Quoted 5 lines · fragmentux
WRT systemd and 3rd party 'servers'
I am curious to know of any other "Server service" who decide .. to *rely* on ..
a *forced* 'systemd (almost) unconditional restart by default'
Some software provides their own wrapper scripts to ensure restarts (like squid, for example) because not all systems have facilities to do that. So being able to use systemd to ensure the daemon stays up relieves the developers from cooking their own scripts.
mgetty+sendfax' faxrunqd, for example, is started from /etc/inittab on systems that do not have systemd - for exactly the same reason: if something happens, faxrunqd gets automatically restarted.
gert
Signature · 4 lines
USENET is *not* the non-clickable part of WWW!
//www.muc.de/~gert/
Gert Doering - Munich, Germany [email protected]...
fax: +49-89-35655025 [email protected]...Hi,
Quoted 2 lines · fragmentux
all your comment are totally valid from a sys-admin point of view but from an openvpn POV, the only responsibility is to provide a secure VPN.
Uh, no. Part of our responsibility is also to provide a reasonably good experience to sysadmins installing and running openvpn - and if we can provide a systemd unit file that does "the right thing" for "most of the time", users will be happier.
(I do not always agree with David on what the best approach to "good experience" is, so you'll see discussions - but we *do* agree on the general principle "it should work and not annoy users or sysadmins")
Another example: why do we bother to have warnings about option combinations that will not work right in our code, instead of "just exit(1) with no further explanation"? Because "secure VPN" alone isn't good enought, it needs to be reasonably robust against misuse, and *tell* users what is wrong so they can fix their errors.
gert
Signature · 4 lines
USENET is *not* the non-clickable part of WWW!
//www.muc.de/~gert/
Gert Doering - Munich, Germany [email protected]...
fax: +49-89-35655025 [email protected]...-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Applying lazy-ack policy to this patch. Despite one reservation to this change, the overall discussion was in favour of this enhancement.
Your patch has been applied to the following branches
commit a4686e99b047081f0ef6f7945450183088464aa5 (master) commit 1c112c38d46207905bff97969cf787baada59711 (release/2.4)
Author: David Sommerseth Date: Thu Sep 7 01:52:02 2017 +0200
systemd: Enable systemd's auto-restart feature for server profiles
Signed-off-by: David Sommerseth <[email protected]...>
Acked-by: David Sommerseth <[email protected]...>
Message-Id: <[email protected]...>
URL: https://www.mail-archive.com/[email protected]/msg15370.html
Signed-off-by: David Sommerseth <[email protected]...>- --
kind regards,
David Sommerseth
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
iQIcBAEBCgAGBQJZxFCQAAoJEIbPlEyWcf3yjWcQAL1RFXTYuZLw+eJ1eoNIuZx2 N21HiIh887SNvexzgKthJNVGj5AeDQo4DXRdd5UxxGK6Czh2FZUuZSExv4lNsugI t56Y3UmWXB6J3PcvmkSyTIPuQVPXNW0qCL3VDV88Bx7Is3JvZsm/kLVLuzXsDwFS vQyHpJ6y3nAavPMigTvP6kYNXN8oz6L7oSeQxWfj5p21cLYFNablSILxFcuCXJtN iyC0tZ+gr7WhqE7g/cY1GG7I8mlfEFj6DrtQCb08T6XGbnPNI4qmL/PujAHlpATO qyEfjTpMVPJVz4ylPKGRe1B/GWa4USOh1M6PgdVIDJYAZAbAcncJMNyHf+gseUMt RGgCz8ri9eR+uOcM4ZDiZNVywj/BOsMDAKjZ/tGWI9DfKxkQu5WjmaCslkdhSM0e 2lk/SA3PCCcyFF7WQ/bynYqUi0aVM+2F/5YSH8xJXFxyd6VSJseoiyhGLxXKl1lo M9GjujoiEjimNUJNQQxDBWfnRaNGSVGlCLZtCgsj2VkkZ9fGE8FhLjPLJMNL+c8I Iba6E3HsDsI5vMI9ZFRGQ0MHdr9gEbRee+IoMnQZx9lSHuW583tQ6IcIOBbatgGH OsG6wCPU9vjFOLh6ABRlgGCStfK+CHzYq9GRaCw7jb3GjBNrckuJA7SYBqGSvVS+ mKGFSB6n+Y7fhlZ/sdXL
=orEs
-----END PGP SIGNATURE-----