nfs-utils: systemd units bug fixes and comments.
Subsystems (2)
cover nfs-utils: systemd units bug fixes and comments.
Bug Fixes:
The /proc/net/rpc/use-gss-proxy file can not be used as a start up trigger for rpc.svcsgssd since it always exists, with or without gss-proxy installed.
Adding the Wants= to the nfs server unit cause a systemd ordering cycle which caused reboots to take forever.
Comment One:
Even though nfs-client does conditionally start the rpc.gssd service when /etc/krb5.keytab exists (which good), but that's all it does. Meaning 'systemctl status' does not show that rpc.gssd is running and 'systemctl restart' does not restart rpc.gssd and 'systemctl stop' does not stop the daemon.
It just seems odd to me that we will be using one target unit to enable a daemon then another service unit (rpc-gssd) to control it.
I thinking we should have one service unit, when enabled, control both the rpc.gssd and the rpc.svcgssd daemons. The start up trigger for both daemons will be the existence of /etc/krb5.keytab and rpc.svcgssd will only be started if the nfs server is enabled (if that is possible).
Comment Two:
How about renaming the nfs-utils unit to nfs-services since a 'systemctl restart' of the unit start all the server and client daemons (even when the server is not enabled, which is probably a bug).
Since a 'systemctl restart nfs-utils' starts all the daemons shouldn't 'systemctl stop nfs-utils' bring them all down as well?
Another oddity, going a 'systemctl restart nfs-utils' causes v3 mounts to go stale... Meaning going a ls on a v3 mount point after the restart errors out with ESTALE... Not sure why...
Comment Three:
I'm not seeing how the nfs-utils_env.sh file, called by each unit, is all that useful. The main reason is you can not tell which unit its being called from so how do know what should be done? I guess I'm just missing the concept on how and what it should be used for.
Steve Dickson (2):
rpc-svcgssd.service: removed a the start up triggers
systemd: Removed the "ordering cycle" from nfs-server.service
systemd/nfs-server.service | 2 -- systemd/rpc-svcgssd.service | 1 - 2 files changed, 3 deletions(-)
Signature · 1 line
1.8.5.3
PATCH 1/2 rpc-svcgssd.service: removed a the start up triggers
The file /proc/net/rpc/use-gss-proxy exists whether gss-proxy is or is not installed. So is existence can not be used as start up trigger.
--- systemd/rpc-svcgssd.service | 1 - 1 file changed, 1 deletion(-)
systemd/rpc-svcgssd.service +0 -1
@@ -7,7 +7,6 @@ PartOf=nfs-utils.service After=gssproxy.service ConditionPathExists=|!@localstatedir@/run/gssproxy.pid -ConditionPathExists=|!/proc/net/rpc/use-gss-proxy ConditionPathExists=/etc/krb5.keytab [Service]
Signature · 1 line
1.8.5.3
Re: [PATCH 1/2] rpc-svcgssd.service: removed a the start up triggers
Quoted 14 lines · On Tue, 18 Feb 2014 03:48:48 -0500 Steve Dickson <[email protected]> wrote:
The file
/proc/net/rpc/use-gss-proxyexists whether gss-proxy is or is not installed. So is existence can not be used as start up trigger.Signed-off-by Steve Dickson <[email protected]>--- systemd/rpc-svcgssd.service | 1 - 1 file changed, 1 deletion(-)systemd/rpc-svcgssd.service +0
-0
systemd/rpc-svcgssd.service PartOf=nfs-utils.service · line 12
Quoted 7 lines
systemd/rpc-svcgssd.service +0
-1@@ -7,7 +7,6 @@ PartOf=nfs-utils.service After=gssproxy.service ConditionPathExists=|!@localstatedir@/run/gssproxy.pid -ConditionPathExists=|!/proc/net/rpc/use-gss-proxy ConditionPathExists=/etc/krb5.keytab [Service]
This is testing for non-existence, not existence.
We need rpc-svcgssd unless gssproxy can perform that task for us, in which case we don't want rpc-svcgssd.
gssproxy can only perform the task for us if:
1/ it is running (hence the test on the pid file)
2/ the kernel knows how to talk to it (hence the test on use-gss-proxy)
With you version, if gssproxy were running on a kernel which did not include support for nfsd talking to gssproxy, then rpc-svcgssd would fail to start, which isn't what we want.
This may be an unusual configuration but unless it is an impossible configuration I would rather the code remained as it was. (This should work on *all* distros, not just sane distros).
Thanks,
NeilBrown
Re: [PATCH 1/2] rpc-svcgssd.service: removed a the start up triggers
Quoted 26 lines · NeilBrown
Quoted 14 lines · On Tue, 18 Feb 2014 03:48:48 -0500 Steve Dickson <[email protected]> wrote:
The file
/proc/net/rpc/use-gss-proxyexists whether gss-proxy is or is not installed. So is existence can not be used as start up trigger.Signed-off-by Steve Dickson <[email protected]>--- systemd/rpc-svcgssd.service | 1 - 1 file changed, 1 deletion(-)systemd/rpc-svcgssd.service +0
-0systemd/rpc-svcgssd.service PartOf=nfs-utils.service · line 12
Quoted 7 lines
systemd/rpc-svcgssd.service +0
-1@@ -7,7 +7,6 @@ PartOf=nfs-utils.service After=gssproxy.service ConditionPathExists=|!@localstatedir@/run/gssproxy.pid -ConditionPathExists=|!/proc/net/rpc/use-gss-proxy ConditionPathExists=/etc/krb5.keytab [Service]This is testing for non-existence, not existence.
Quoted 5 lines
We need rpc-svcgssd unless gssproxy can perform that task for us, in which case we don't want rpc-svcgssd.
gssproxy can only perform the task for us if:
1/ it is running (hence the test on the pid file)
could be stale
Quoted 1 line
2/ the kernel knows how to talk to it (hence the test on use-gss-proxy)
use-gss-proxy is always available whether gss-proxy is being used or not as it is the interface used by gss-proxy to tell the kernel it is in use.
Quoted 3 lines
With you version, if gssproxy were running on a kernel which did not include support for nfsd talking to gssproxy, then rpc-svcgssd would fail to start, which isn't what we want.
True the right way to handle this would probably be to
make /proc/net/rpc/use-gss-proxy readable and start rpc.svcgssd if the value is 0 ?
Quoted 3 lines
This may be an unusual configuration but unless it is an impossible configuration I would rather the code remained as it was. (This should work on *all* distros, not just sane distros).
I agree with the sentiment.
Simo.
Signature · 1 line
Simo Sorce * Red Hat, Inc * New York
Re: [PATCH 1/2] rpc-svcgssd.service: removed a the start up triggers
Quoted 27 lines · On Wed, 19 Feb 2014 11:03:36 -0500 Simo Sorce <[email protected]> wrote:
Quoted 26 lines · NeilBrown
Quoted 14 lines · On Tue, 18 Feb 2014 03:48:48 -0500 Steve Dickson <[email protected]> wrote:
The file
/proc/net/rpc/use-gss-proxyexists whether gss-proxy is or is not installed. So is existence can not be used as start up trigger.Signed-off-by Steve Dickson <[email protected]>--- systemd/rpc-svcgssd.service | 1 - 1 file changed, 1 deletion(-)systemd/rpc-svcgssd.service +0
-0systemd/rpc-svcgssd.service PartOf=nfs-utils.service · line 12
Quoted 7 lines
systemd/rpc-svcgssd.service +0
-1@@ -7,7 +7,6 @@ PartOf=nfs-utils.service After=gssproxy.service ConditionPathExists=|!@localstatedir@/run/gssproxy.pid -ConditionPathExists=|!/proc/net/rpc/use-gss-proxy ConditionPathExists=/etc/krb5.keytab [Service]This is testing for non-existence, not existence.
Quoted 7 lines
Quoted 5 lines
We need rpc-svcgssd unless gssproxy can perform that task for us, in which case we don't want rpc-svcgssd.
gssproxy can only perform the task for us if:
1/ it is running (hence the test on the pid file)could be stale
Agreed. But it is the best we have. Systemd doesn't support
ConditionUnitActive
or similar.
Quoted 6 lines
Quoted 2 lines
2/ the kernel knows how to talk to it (hence the test on use-gss-proxy)
use-gss-proxy is always available whether gss-proxy is being used or not as it is the interface used by gss-proxy to tell the kernel it is in use.
It is not available in linux 3.9 or earlier.
Quoted 8 lines
Quoted 4 lines
With you version, if gssproxy were running on a kernel which did not include support for nfsd talking to gssproxy, then rpc-svcgssd would fail to start, which isn't what we want.
True the right way to handle this would probably be to
make/proc/net/rpc/use-gss-proxyreadable and start rpc.svcgssd if the value is 0 ?
Pity systemd doesn't have
ConditionPathIsFileContaining
Even udev has that!
Quoted 6 lines
Quoted 4 lines
This may be an unusual configuration but unless it is an impossible configuration I would rather the code remained as it was. (This should work on *all* distros, not just sane distros).
I agree with the sentiment.
Thanks,
NeilBrown
Quoted 3 lines
Simo.
PATCH 2/2 systemd: Removed the "ordering cycle" from nfs-server.service
In reply to Steve DicksonCommit 88ff0417 introduce 'Wants=' statements that conditionally started other services. Those services need to be removed from the 'After=' statements otherwise a systemd ordering cycle is created.
--- systemd/nfs-server.service | 2 -- 1 file changed, 2 deletions(-)
systemd/nfs-server.service +0 -2
@@ -7,8 +7,6 @@ Wants=rpc-statd.service nfs-idmapd.service rpc-gssd.service rpc-svcgssd.service Wants=rpc-statd-notify.service After= network.target proc-fs-nfsd.mount rpcbind.target nfs-mountd.service -After= nfs-idmapd.service rpc-statd.service -After= rpc-gssd.service rpc-svcgssd.service Before= rpc-statd-notify.service [Service]
Signature · 1 line
1.8.5.3
Re: [PATCH 2/2] systemd: Removed the "ordering cycle" from nfs-server.service
Quoted 15 lines · On Tue, 18 Feb 2014 03:48:49 -0500 Steve Dickson <[email protected]> wrote:
Commit
88ff0417introduce 'Wants=' statements that conditionally started other services. Those services need to be removed from the 'After=' statements otherwise a systemd ordering cycle is created.Signed-off-by Steve Dickson <[email protected]>--- systemd/nfs-server.service | 2 -- 1 file changed, 2 deletions(-)systemd/nfs-server.service +0
-0
systemd/nfs-server.service Wants=rpc-statd.service nfs-idmapd.service rpc-gssd.service rpc-svcgssd.service · line 12
Quoted 8 lines
systemd/nfs-server.service +0
-2@@ -7,8 +7,6 @@ Wants=rpc-statd.service nfs-idmapd.service rpc-gssd.service rpc-svcgssd.service Wants=rpc-statd-notify.service After= network.target proc-fs-nfsd.mount rpcbind.target nfs-mountd.service -After= nfs-idmapd.service rpc-statd.service -After= rpc-gssd.service rpc-svcgssd.service Before= rpc-statd-notify.service [Service]
"Wants" does not imply ordering. If A Wants B, then A and B can be started at the same time.
But we really want these various services to be running before nfsd is started. So the "After" are important. Please leave them.
The lack of ordering is explicit in the documentation for Requires:
If a unit foo.service requires a unit bar.service as
configured with Requires= and no ordering is configured with After= or Before=, then both units will be started simultaneously and without any delay between them if foo.service is activated.
and implicit in the documentation for Wants:
Wants=
A weaker version of Requires=.
Thanks,
NeilBrown
Quoted 6 lines · Steve Dickson
Bug Fixes:
The
/proc/net/rpc/use-gss-proxyfile can not be used as a start up trigger for rpc.svcsgssd since it always exists, with or without gss-proxy installed.
Quoted 7 lines
Adding the Wants= to the nfs server unit cause a systemd ordering cycle which caused reboots to take forever.
Comment One:
Even though nfs-client does conditionally start the rpc.gssd service when
/etc/krb5.keytabexists (which good),
No, that's bad. rpc.gssd has to run in cases where there is no /etc/krb5.ktab. See
http://www.spinics.net/lists/linux-nfs/msg41585.html
The existence of /etc/krb5.conf is a better choice.
Quoted 43 lines
but that's
all it does. Meaning 'systemctl status' does not show that rpc.gssd is running and 'systemctl restart' does not restart rpc.gssd and 'systemctl stop' does not stop the daemon.It just seems odd to me that we will be using one target unit to enable a daemon then another service unit (rpc-gssd) to control it.
I thinking we should have one service unit, when enabled, control both the rpc.gssd and the rpc.svcgssd daemons. The start up trigger for both daemons will be the existence of
/etc/krb5.keytaband rpc.svcgssd will only be started if the nfs server is enabled (if that is possible).Comment Two:
How about renaming the nfs-utils unit to nfs-services since a 'systemctl restart' of the unit start all the server and client daemons (even when the server is not enabled, which is probably a bug).
Since a 'systemctl restart nfs-utils' starts all the daemons shouldn't 'systemctl stop nfs-utils' bring them all down as well?
Another oddity, going a 'systemctl restart nfs-utils' causes v3 mounts to go stale... Meaning going a ls on a v3 mount point after the restart errors out with ESTALE... Not sure why...
Comment Three:
I'm not seeing how the
nfs-utils_env.shfile, called by each unit, is all that useful. The main reason is you can not tell which unit its being called from so how do know what should be done? I guess I'm just missing the concept on how and what it should be used for.Steve Dickson (2):
rpc-svcgssd.service: removed a the start up triggers
systemd: Removed the "ordering cycle" from nfs-server.servicesystemd/nfs-server.service | 2 --
systemd/rpc-svcgssd.service | 1 -2 files changed, 3 deletions(-)
Signature · 2 lines
Chuck Lever chuck[dot]lever[at]oracle[dot]com
Quoted 11 lines · Chuck Lever
Quoted 9 lines · Steve Dickson
Quoted 9 lines
Bug Fixes:
The
/proc/net/rpc/use-gss-proxyfile can not be used as a start up trigger for rpc.svcsgssd since it always exists, with or without gss-proxy installed.Adding the Wants= to the nfs server unit cause a systemd ordering cycle which caused reboots to take forever.
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Comment One:
Even though nfs-client does conditionally start the rpc.gssd service when
/etc/krb5.keytabexists (which good),No, that's bad. rpc.gssd has to run in cases where there is no
/etc/krb5.ktab. See
At this point its a pipe dream for rpc.gssd to run with no keytab. It logs a ton of errors messages on every upcall which means on every mount these days.
We either have to tone down the error messages or check for the existence of the keytab before processing the upcall. I think the latter would better...
Quoted 3 lines
The existence of
/etc/krb5.confis a better choice.
Fine... If there is no objection... I'm ok with it..
steved.
Hi Steve-
Quoted 14 lines · Steve Dickson
Quoted 11 lines · Chuck Lever
Quoted 9 lines · Steve Dickson
Quoted 9 lines
Bug Fixes:
The
/proc/net/rpc/use-gss-proxyfile can not be used as a start up trigger for rpc.svcsgssd since it always exists, with or without gss-proxy installed.Adding the Wants= to the nfs server unit cause a systemd ordering cycle which caused reboots to take forever.
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Comment One:
Even though nfs-client does conditionally start the rpc.gssd service when
/etc/krb5.keytabexists (which good),No, that's bad. rpc.gssd has to run in cases where there is no
/etc/krb5.ktab. SeeAt this point its a pipe dream for rpc.gssd to run with no keytab.
This is not a pipe dream. I’m talking about the common use case where a user kinit’s as root then uses the “-n” option on gssd so that root’s credential is used as the client’s machine credential, instead of using the keytab to establish a GSS context.
With the exception of kernels 3.9 - 3.12, this has always worked, does require gssd to be running, and does not need to have a keytab on the client to operate correctly. When 3.9 broke this feature, people (including NeilB!) complained loudly.
Quoted 5 lines
It logs a ton of errors messages on every upcall which means on every mount these days.
We either have to tone down the error messages or check for the existence of the keytab before processing the upcall. I think the latter would better…
gssd default verbosity is a legacy of the days when Kerberized NFS was new and we wanted verbose logging to monitor gssd activity. It seems like a harmless step forward to eliminate them or hide some or all of them behind a gssd command line option.
The bug here is gssd log diarrhea.
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
Quoted 2 lines · Chuck Lever
Quoted 1 line
At this point its a pipe dream for rpc.gssd to run with no keytab.
This is not a pipe dream.
I meant its a pipe dream in the current state due to, your nicely termed, "gssd log diarrhea" ;-)
steved.
Quoted 12 lines
I’m talking about the common use case where a user kinit’s as root then uses the “-n” option on gssd so that root’s credential is used as the client’s machine credential, instead of using the keytab to establish a GSS context.
With the exception of kernels 3.9 - 3.12, this has always worked, does require gssd to be running, and does not need to have a keytab on the client to operate correctly. When 3.9 broke this feature, people (including NeilB!) complained loudly.
Quoted 5 lines
Quoted 5 lines
It logs a ton of errors messages on every upcall which means on every mount these days.
We either have to tone down the error messages or check for the existence of the keytab before processing the upcall. I think the latter would better…gssd default verbosity is a legacy of the days when Kerberized NFS was new and we wanted verbose logging to monitor gssd activity. It seems like a harmless step forward to eliminate them or hide some or all of them behind a gssd command line option.
The bug here is gssd log diarrhea.
Quoted 9 lines · On Tue, 18 Feb 2014 03:48:47 -0500 Steve Dickson <[email protected]> wrote:
Bug Fixes:
The
/proc/net/rpc/use-gss-proxyfile can not be used as a start up trigger for rpc.svcsgssd since it always exists, with or without gss-proxy installed.Adding the Wants= to the nfs server unit cause a systemd ordering cycle which caused reboots to take forever.
Quoted 7 lines
Comment One:
Even though nfs-client does conditionally start the rpc.gssd service when
/etc/krb5.keytabexists (which good), but that's all it does. Meaning 'systemctl status' does not show that rpc.gssd is running and 'systemctl restart' does not restart rpc.gssd and 'systemctl stop' does not stop the daemon.
Well it doesn't *only* start rpc.gssd, it also starts e.g. sm-notify. But maybe you meant that. It only starts things, it doesn't stop them. This how systemd works. Certainly there is no way I can see for
systemctl status foo
to report on anything but a single service: foo. It might be nice if it could report on a collection of services, but that doesn't seem to be supported.
We could arrange that when you 'stop' nfs-client it would also stop rpc-gssd (unless the server was running and wanted it) by setting
StopWhenUnneeded=yes
That isn't the default in systemd so I'm cautious of setting it without a good reason. However I don't object if it turns out that it works.
Restart is a bit tricky - again because both the client and the server want some services.
You can enable Restart propagation by using PartOf=. However that also enables Stop propagation and we don't want that. i.e. we don't want
systemctl stop nfs-client
to unilaterally stop rpc.gssd - in particular not if nfs-server is running.
In short: I think that the inter-relationships in nfs-utils are too complex to allow the simplicity that you are seeking. Or systemd isn't sophisticated enough to represent them.
Quoted 10 lines
It just seems odd to me that we will be using one target unit to enable a daemon then another service unit (rpc-gssd) to control it.
I thinking we should have one service unit, when enabled, control both the rpc.gssd and the rpc.svcgssd daemons. The start up trigger for both daemons will be the existence of
/etc/krb5.keytaband rpc.svcgssd will only be started if the nfs server is enabled (if that is possible).
I originally thought that would not practical as the requirements for server and client are different. However I now see (thanks Bruce) that both the client and the server need both rpc.gssd and rpc.svcgssd. So we can have an
nfs-secure
(or similar) which Wants both rpc-gssd and rpc-svcgssd. Then both server and client Want it.
However I don't think it should be "enabled". It should automatically start whenever nfs-server or nfs-client is started, providing the relevant files exist.
(It can be 'masked' if someone has installed krb5 but really doesn't want the gss daemons running).
So you would still use a different service to enable it than to restart it.
Also "systemctl status nfs-secure" would not be able to show you the status of both daemons.
Quoted 7 lines
Comment Two:
How about renaming the nfs-utils unit to nfs-services since a 'systemctl restart' of the unit start all the server and client daemons (even when the server is not enabled, which is probably a bug).
I don't object to a renaming. The name "nfs-utils" was simply the first thing I thought of.
systemctl restart nfs-utils
should *not* start any service that is not already started. I think I tested that. However if I'm wrong we could argue that "try-restart" should be used if you don't want to start services that aren't already running.... though I'm not sure that would work. Must test....
systemctl stop nfs-utils
should stop services. It uses "PartOf" which is supposed to apply to 'restart' and 'stop' equally.
Quoted 2 lines
Since a 'systemctl restart nfs-utils' starts all the daemons shouldn't 'systemctl stop nfs-utils' bring them all down as well?
Doesn't it?
Quoted 5 lines
Another oddity, going a 'systemctl restart nfs-utils' causes v3 mounts to go stale... Meaning going a ls on a v3 mount point after the restart errors out with ESTALE... Not sure why...
That is weird and certainly needs looking in to. I could be due to
ExecStopPost=/usr/sbin/exportfs -au
I wonder if we really need that.
I hope to get some time tomorrow to experiment with all this and I will definitely look into this issue (unless you explain it first :-)
Quoted 7 lines
Comment Three:
I'm not seeing how the
nfs-utils_env.shfile, called by each unit, is all that useful. The main reason is you can not tell which unit its being called from so how do know what should be done? I guess I'm just missing the concept on how and what it should be used for.
I've got a patch which changes this so it is only called once in a separate service. However the principle remains unchanged.
The purpose of the script is to read /etc/sysconfig/nfs (or similar) and write out /run/sysconfig/nfs-utils containing *all* the environment variables used in *any* nfs-utils unit file. It allows a translation from what is expected in the sysconfig file to what the daemons expect as their arguments.
I see this as a transitional thing. Ultimately the various daemons should be able to inspect their environment, and expect exactly the things that might be in the config file. So systemd just reads /etc/sysconfig/nfs and then runs the daemons. But that goal is a little ways off yet.
One thing we have in our sysconfig file is "NFS3_SERVER_SUPPORT". If that is set to "no", then as well as constructing appropriate arguments for rpc.mountd and rpc.nfs we don't run rpc.statd (unless a v3 filesystem is mounted). Handling the first part with nfs-utils_env.sh is trivial. Handling the second bit is something I haven't figured out yet.
Thanks for your comments.
NeilBrown
Quoted 9 lines
Steve Dickson (2):
rpc-svcgssd.service: removed a the start up triggers
systemd: Removed the "ordering cycle" from nfs-server.servicesystemd/nfs-server.service | 2 -- systemd/rpc-svcgssd.service | 1 - 2 files changed, 3 deletions(-)