Skip to contentSkip to conversation

nfs-utils: systemd units bug fixes and comments.

PATCH In review Steve Dickson 4 participants 12 messages Feb 18, 2014 – Feb 19, 2014

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover nfs-utils: systemd units bug fixes and comments.

Bug Fixes:

The /proc/net/rpc/use-gss-proxy file can not be used as a start up trigger for rpc.svcsgssd since it always exists, with or without gss-proxy installed.

Adding the Wants= to the nfs server unit cause a systemd ordering cycle which caused reboots to take forever.

Comment One:

Even though nfs-client does conditionally start the rpc.gssd service when /etc/krb5.keytab exists (which good), but that's all it does. Meaning 'systemctl status' does not show that rpc.gssd is running and 'systemctl restart' does not restart rpc.gssd and 'systemctl stop' does not stop the daemon.

It just seems odd to me that we will be using one target unit to enable a daemon then another service unit (rpc-gssd) to control it.

I thinking we should have one service unit, when enabled, control both the rpc.gssd and the rpc.svcgssd daemons. The start up trigger for both daemons will be the existence of /etc/krb5.keytab and rpc.svcgssd will only be started if the nfs server is enabled (if that is possible).

Comment Two:

How about renaming the nfs-utils unit to nfs-services since a 'systemctl restart' of the unit start all the server and client daemons (even when the server is not enabled, which is probably a bug).

Since a 'systemctl restart nfs-utils' starts all the daemons shouldn't 'systemctl stop nfs-utils' bring them all down as well?

Another oddity, going a 'systemctl restart nfs-utils' causes v3 mounts to go stale... Meaning going a ls on a v3 mount point after the restart errors out with ESTALE... Not sure why...

Comment Three:

I'm not seeing how the nfs-utils_env.sh file, called by each unit, is all that useful. The main reason is you can not tell which unit its being called from so how do know what should be done? I guess I'm just missing the concept on how and what it should be used for.

Steve Dickson (2):
rpc-svcgssd.service: removed a the start up triggers
systemd: Removed the "ordering cycle" from nfs-server.service

 systemd/nfs-server.service  | 2 --
 systemd/rpc-svcgssd.service | 1 -
 2 files changed, 3 deletions(-)
Signature · 1 line
1.8.5.3
#1 lore raw

PATCH 1/2 rpc-svcgssd.service: removed a the start up triggers

The file /proc/net/rpc/use-gss-proxy exists whether gss-proxy is or is not installed. So is existence can not be used as start up trigger.

---
 systemd/rpc-svcgssd.service | 1 -
 1 file changed, 1 deletion(-)
systemd/rpc-svcgssd.service +0 -1
index 6bd4588..8e10b98 100644
@@ -7,7 +7,6 @@ PartOf=nfs-utils.service
 
 After=gssproxy.service
 ConditionPathExists=|!@localstatedir@/run/gssproxy.pid
-ConditionPathExists=|!/proc/net/rpc/use-gss-proxy
 ConditionPathExists=/etc/krb5.keytab
 
 [Service]
Signature · 1 line
1.8.5.3
#2 lore raw

Re: [PATCH 1/2] rpc-svcgssd.service: removed a the start up triggers

Quoted 14 lines · On Tue, 18 Feb 2014 03:48:48 -0500 Steve Dickson <[email protected]> wrote:

The file /proc/net/rpc/use-gss-proxy exists whether gss-proxy is or is not installed. So is existence can not be used as start up trigger.

---
 systemd/rpc-svcgssd.service | 1 -
 1 file changed, 1 deletion(-)
systemd/rpc-svcgssd.service +0 -0
index 6bd4588..8e10b98 100644

systemd/rpc-svcgssd.service PartOf=nfs-utils.service · line 12

Quoted 7 lines
systemd/rpc-svcgssd.service +0 -1
@@ -7,7 +7,6 @@ PartOf=nfs-utils.service
 
 After=gssproxy.service
 ConditionPathExists=|!@localstatedir@/run/gssproxy.pid
-ConditionPathExists=|!/proc/net/rpc/use-gss-proxy
 ConditionPathExists=/etc/krb5.keytab
 
 [Service]

This is testing for non-existence, not existence.

We need rpc-svcgssd unless gssproxy can perform that task for us, in which case we don't want rpc-svcgssd.

gssproxy can only perform the task for us if:
1/ it is running (hence the test on the pid file)
2/ the kernel knows how to talk to it (hence the test on use-gss-proxy)

With you version, if gssproxy were running on a kernel which did not include support for nfsd talking to gssproxy, then rpc-svcgssd would fail to start, which isn't what we want.

This may be an unusual configuration but unless it is an impossible configuration I would rather the code remained as it was. (This should work on *all* distros, not just sane distros).

Thanks,
NeilBrown

#3 lore raw

Re: [PATCH 1/2] rpc-svcgssd.service: removed a the start up triggers

Quoted 26 lines · NeilBrown
Quoted 14 lines · On Tue, 18 Feb 2014 03:48:48 -0500 Steve Dickson <[email protected]> wrote:

The file /proc/net/rpc/use-gss-proxy exists whether gss-proxy is or is not installed. So is existence can not be used as start up trigger.

---
 systemd/rpc-svcgssd.service | 1 -
 1 file changed, 1 deletion(-)
systemd/rpc-svcgssd.service +0 -0
index 6bd4588..8e10b98 100644

systemd/rpc-svcgssd.service PartOf=nfs-utils.service · line 12

Quoted 7 lines
systemd/rpc-svcgssd.service +0 -1
@@ -7,7 +7,6 @@ PartOf=nfs-utils.service
 
 After=gssproxy.service
 ConditionPathExists=|!@localstatedir@/run/gssproxy.pid
-ConditionPathExists=|!/proc/net/rpc/use-gss-proxy
 ConditionPathExists=/etc/krb5.keytab
 
 [Service]

This is testing for non-existence, not existence.

Quoted 5 lines

We need rpc-svcgssd unless gssproxy can perform that task for us, in which case we don't want rpc-svcgssd.

gssproxy can only perform the task for us if:
1/ it is running (hence the test on the pid file)

could be stale

Quoted 1 line

2/ the kernel knows how to talk to it (hence the test on use-gss-proxy)

use-gss-proxy is always available whether gss-proxy is being used or not as it is the interface used by gss-proxy to tell the kernel it is in use.

Quoted 3 lines

With you version, if gssproxy were running on a kernel which did not include support for nfsd talking to gssproxy, then rpc-svcgssd would fail to start, which isn't what we want.

True the right way to handle this would probably be to
make /proc/net/rpc/use-gss-proxy readable and start rpc.svcgssd if the value is 0 ?

Quoted 3 lines

This may be an unusual configuration but unless it is an impossible configuration I would rather the code remained as it was. (This should work on *all* distros, not just sane distros).

I agree with the sentiment.

Simo.

Signature · 1 line
Simo Sorce * Red Hat, Inc * New York
#4 lore raw

Re: [PATCH 1/2] rpc-svcgssd.service: removed a the start up triggers

Quoted 27 lines · On Wed, 19 Feb 2014 11:03:36 -0500 Simo Sorce <[email protected]> wrote:
Quoted 26 lines · NeilBrown
Quoted 14 lines · On Tue, 18 Feb 2014 03:48:48 -0500 Steve Dickson <[email protected]> wrote:

The file /proc/net/rpc/use-gss-proxy exists whether gss-proxy is or is not installed. So is existence can not be used as start up trigger.

---
 systemd/rpc-svcgssd.service | 1 -
 1 file changed, 1 deletion(-)
systemd/rpc-svcgssd.service +0 -0
index 6bd4588..8e10b98 100644

systemd/rpc-svcgssd.service PartOf=nfs-utils.service · line 12

Quoted 7 lines
systemd/rpc-svcgssd.service +0 -1
@@ -7,7 +7,6 @@ PartOf=nfs-utils.service
 
 After=gssproxy.service
 ConditionPathExists=|!@localstatedir@/run/gssproxy.pid
-ConditionPathExists=|!/proc/net/rpc/use-gss-proxy
 ConditionPathExists=/etc/krb5.keytab
 
 [Service]

This is testing for non-existence, not existence.

Quoted 7 lines
Quoted 5 lines

We need rpc-svcgssd unless gssproxy can perform that task for us, in which case we don't want rpc-svcgssd.

gssproxy can only perform the task for us if:
1/ it is running (hence the test on the pid file)

could be stale

Agreed. But it is the best we have. Systemd doesn't support
ConditionUnitActive
or similar.

Quoted 6 lines
Quoted 2 lines

2/ the kernel knows how to talk to it (hence the test on use-gss-proxy)

use-gss-proxy is always available whether gss-proxy is being used or not as it is the interface used by gss-proxy to tell the kernel it is in use.

It is not available in linux 3.9 or earlier.

Quoted 8 lines
Quoted 4 lines

With you version, if gssproxy were running on a kernel which did not include support for nfsd talking to gssproxy, then rpc-svcgssd would fail to start, which isn't what we want.

True the right way to handle this would probably be to
make /proc/net/rpc/use-gss-proxy readable and start rpc.svcgssd if the value is 0 ?

Pity systemd doesn't have

ConditionPathIsFileContaining

Even udev has that!

Quoted 6 lines
Quoted 4 lines

This may be an unusual configuration but unless it is an impossible configuration I would rather the code remained as it was. (This should work on *all* distros, not just sane distros).

I agree with the sentiment.

Thanks,

NeilBrown

Quoted 3 lines

Simo.

#5 lore raw

PATCH 2/2 systemd: Removed the "ordering cycle" from nfs-server.service

In reply to Steve Dickson

Commit 88ff0417 introduce 'Wants=' statements that conditionally started other services. Those services need to be removed from the 'After=' statements otherwise a systemd ordering cycle is created.

---
 systemd/nfs-server.service | 2 --
 1 file changed, 2 deletions(-)
systemd/nfs-server.service +0 -2
index 26df656..43be4aa 100644
@@ -7,8 +7,6 @@ Wants=rpc-statd.service nfs-idmapd.service rpc-gssd.service rpc-svcgssd.service
 Wants=rpc-statd-notify.service
 
 After= network.target proc-fs-nfsd.mount rpcbind.target nfs-mountd.service
-After= nfs-idmapd.service rpc-statd.service
-After= rpc-gssd.service rpc-svcgssd.service
 Before= rpc-statd-notify.service
 
 [Service]
Signature · 1 line
1.8.5.3
#6 lore raw

Re: [PATCH 2/2] systemd: Removed the "ordering cycle" from nfs-server.service

Quoted 15 lines · On Tue, 18 Feb 2014 03:48:49 -0500 Steve Dickson <[email protected]> wrote:

Commit 88ff0417 introduce 'Wants=' statements that conditionally started other services. Those services need to be removed from the 'After=' statements otherwise a systemd ordering cycle is created.

---
 systemd/nfs-server.service | 2 --
 1 file changed, 2 deletions(-)
systemd/nfs-server.service +0 -0
index 26df656..43be4aa 100644

systemd/nfs-server.service Wants=rpc-statd.service nfs-idmapd.service rpc-gssd.service rpc-svcgssd.service · line 12

Quoted 8 lines
systemd/nfs-server.service +0 -2
@@ -7,8 +7,6 @@ Wants=rpc-statd.service nfs-idmapd.service rpc-gssd.service rpc-svcgssd.service
 Wants=rpc-statd-notify.service
 
 After= network.target proc-fs-nfsd.mount rpcbind.target nfs-mountd.service
-After= nfs-idmapd.service rpc-statd.service
-After= rpc-gssd.service rpc-svcgssd.service
 Before= rpc-statd-notify.service
 
 [Service]

"Wants" does not imply ordering. If A Wants B, then A and B can be started at the same time.
But we really want these various services to be running before nfsd is started. So the "After" are important. Please leave them.

The lack of ordering is explicit in the documentation for Requires:

If a unit foo.service requires a unit bar.service as
configured with Requires= and no ordering is configured with After= or Before=, then both units will be started simultaneously and without any delay between them if foo.service is activated.

and implicit in the documentation for Wants:

Wants=
A weaker version of Requires=.

Thanks,
NeilBrown

#7 lore raw
In reply to Steve Dickson
Quoted 6 lines · Steve Dickson

Bug Fixes:

The /proc/net/rpc/use-gss-proxy file can not be used as a start up trigger for rpc.svcsgssd since it always exists, with or without gss-proxy installed.

Quoted 7 lines

Adding the Wants= to the nfs server unit cause a systemd ordering cycle which caused reboots to take forever.

Comment One:

Even though nfs-client does conditionally start the rpc.gssd service when /etc/krb5.keytab exists (which good),

No, that's bad. rpc.gssd has to run in cases where there is no /etc/krb5.ktab. See

http://www.spinics.net/lists/linux-nfs/msg41585.html

The existence of /etc/krb5.conf is a better choice.

Quoted 43 lines

but that's
all it does. Meaning 'systemctl status' does not show that rpc.gssd is running and 'systemctl restart' does not restart rpc.gssd and 'systemctl stop' does not stop the daemon.

It just seems odd to me that we will be using one target unit to enable a daemon then another service unit (rpc-gssd) to control it.

I thinking we should have one service unit, when enabled, control both the rpc.gssd and the rpc.svcgssd daemons. The start up trigger for both daemons will be the existence of /etc/krb5.keytab and rpc.svcgssd will only be started if the nfs server is enabled (if that is possible).

Comment Two:

How about renaming the nfs-utils unit to nfs-services since a 'systemctl restart' of the unit start all the server and client daemons (even when the server is not enabled, which is probably a bug).

Since a 'systemctl restart nfs-utils' starts all the daemons shouldn't 'systemctl stop nfs-utils' bring them all down as well?

Another oddity, going a 'systemctl restart nfs-utils' causes v3 mounts to go stale... Meaning going a ls on a v3 mount point after the restart errors out with ESTALE... Not sure why...

Comment Three:

I'm not seeing how the nfs-utils_env.sh file, called by each unit, is all that useful. The main reason is you can not tell which unit its being called from so how do know what should be done? I guess I'm just missing the concept on how and what it should be used for.

Steve Dickson (2):
rpc-svcgssd.service: removed a the start up triggers
systemd: Removed the "ordering cycle" from nfs-server.service

systemd/nfs-server.service | 2 --
systemd/rpc-svcgssd.service | 1 -

2 files changed, 3 deletions(-)
Signature · 2 lines
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
#8 lore raw
Quoted 11 lines · Chuck Lever
Quoted 9 lines · Steve Dickson
Quoted 9 lines

Bug Fixes:

The /proc/net/rpc/use-gss-proxy file can not be used as a start up trigger for rpc.svcsgssd since it always exists, with or without gss-proxy installed.

Adding the Wants= to the nfs server unit cause a systemd ordering cycle which caused reboots to take forever.

Quoted 7 lines
Quoted 4 lines
Quoted 4 lines

Comment One:

Even though nfs-client does conditionally start the rpc.gssd service when /etc/krb5.keytab exists (which good),

No, that's bad. rpc.gssd has to run in cases where there is no /etc/krb5.ktab. See

http://www.spinics.net/lists/linux-nfs/msg41585.html

At this point its a pipe dream for rpc.gssd to run with no keytab. It logs a ton of errors messages on every upcall which means on every mount these days.

We either have to tone down the error messages or check for the existence of the keytab before processing the upcall. I think the latter would better...

Quoted 3 lines

The existence of /etc/krb5.conf is a better choice.

Fine... If there is no objection... I'm ok with it..

steved.

#9 lore raw

Hi Steve-

Quoted 14 lines · Steve Dickson
Quoted 11 lines · Chuck Lever
Quoted 9 lines · Steve Dickson
Quoted 9 lines

Bug Fixes:

The /proc/net/rpc/use-gss-proxy file can not be used as a start up trigger for rpc.svcsgssd since it always exists, with or without gss-proxy installed.

Adding the Wants= to the nfs server unit cause a systemd ordering cycle which caused reboots to take forever.

Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines

Comment One:

Even though nfs-client does conditionally start the rpc.gssd service when /etc/krb5.keytab exists (which good),

No, that's bad. rpc.gssd has to run in cases where there is no /etc/krb5.ktab. See

http://www.spinics.net/lists/linux-nfs/msg41585.html

At this point its a pipe dream for rpc.gssd to run with no keytab.

This is not a pipe dream. I’m talking about the common use case where a user kinit’s as root then uses the “-n” option on gssd so that root’s credential is used as the client’s machine credential, instead of using the keytab to establish a GSS context.

With the exception of kernels 3.9 - 3.12, this has always worked, does require gssd to be running, and does not need to have a keytab on the client to operate correctly. When 3.9 broke this feature, people (including NeilB!) complained loudly.

Quoted 5 lines

It logs a ton of errors messages on every upcall which means on every mount these days.
We either have to tone down the error messages or check for the existence of the keytab before processing the upcall. I think the latter would better…

gssd default verbosity is a legacy of the days when Kerberized NFS was new and we wanted verbose logging to monitor gssd activity. It seems like a harmless step forward to eliminate them or hide some or all of them behind a gssd command line option.

The bug here is gssd log diarrhea.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

#10 lore raw
Quoted 2 lines · Chuck Lever
Quoted 1 line

At this point its a pipe dream for rpc.gssd to run with no keytab.

This is not a pipe dream.

I meant its a pipe dream in the current state due to, your nicely termed, "gssd log diarrhea" ;-)

steved.

Quoted 12 lines

I’m talking about the common use case where a user kinit’s as root then uses the “-n” option on gssd so that root’s credential is used as the client’s machine credential, instead of using the keytab to establish a GSS context.

With the exception of kernels 3.9 - 3.12, this has always worked, does require gssd to be running, and does not need to have a keytab on the client to operate correctly. When 3.9 broke this feature, people (including NeilB!) complained loudly.

Quoted 5 lines
Quoted 5 lines

It logs a ton of errors messages on every upcall which means on every mount these days.
We either have to tone down the error messages or check for the existence of the keytab before processing the upcall. I think the latter would better…

gssd default verbosity is a legacy of the days when Kerberized NFS was new and we wanted verbose logging to monitor gssd activity. It seems like a harmless step forward to eliminate them or hide some or all of them behind a gssd command line option.

The bug here is gssd log diarrhea.

#11 lore raw
In reply to Steve Dickson
Quoted 9 lines · On Tue, 18 Feb 2014 03:48:47 -0500 Steve Dickson <[email protected]> wrote:

Bug Fixes:

The /proc/net/rpc/use-gss-proxy file can not be used as a start up trigger for rpc.svcsgssd since it always exists, with or without gss-proxy installed.

Adding the Wants= to the nfs server unit cause a systemd ordering cycle which caused reboots to take forever.

Quoted 7 lines

Comment One:

Even though nfs-client does conditionally start the rpc.gssd service when /etc/krb5.keytab exists (which good), but that's all it does. Meaning 'systemctl status' does not show that rpc.gssd is running and 'systemctl restart' does not restart rpc.gssd and 'systemctl stop' does not stop the daemon.

Well it doesn't *only* start rpc.gssd, it also starts e.g. sm-notify. But maybe you meant that. It only starts things, it doesn't stop them. This how systemd works. Certainly there is no way I can see for
systemctl status foo
to report on anything but a single service: foo. It might be nice if it could report on a collection of services, but that doesn't seem to be supported.

We could arrange that when you 'stop' nfs-client it would also stop rpc-gssd (unless the server was running and wanted it) by setting
StopWhenUnneeded=yes

That isn't the default in systemd so I'm cautious of setting it without a good reason. However I don't object if it turns out that it works.

Restart is a bit tricky - again because both the client and the server want some services.
You can enable Restart propagation by using PartOf=. However that also enables Stop propagation and we don't want that. i.e. we don't want
systemctl stop nfs-client
to unilaterally stop rpc.gssd - in particular not if nfs-server is running.

In short: I think that the inter-relationships in nfs-utils are too complex to allow the simplicity that you are seeking. Or systemd isn't sophisticated enough to represent them.

Quoted 10 lines

It just seems odd to me that we will be using one target unit to enable a daemon then another service unit (rpc-gssd) to control it.

I thinking we should have one service unit, when enabled, control both the rpc.gssd and the rpc.svcgssd daemons. The start up trigger for both daemons will be the existence of /etc/krb5.keytab and rpc.svcgssd will only be started if the nfs server is enabled (if that is possible).

I originally thought that would not practical as the requirements for server and client are different. However I now see (thanks Bruce) that both the client and the server need both rpc.gssd and rpc.svcgssd. So we can have an
nfs-secure
(or similar) which Wants both rpc-gssd and rpc-svcgssd. Then both server and client Want it.

However I don't think it should be "enabled". It should automatically start whenever nfs-server or nfs-client is started, providing the relevant files exist.
(It can be 'masked' if someone has installed krb5 but really doesn't want the gss daemons running).

So you would still use a different service to enable it than to restart it.

Also "systemctl status nfs-secure" would not be able to show you the status of both daemons.

Quoted 7 lines

Comment Two:

How about renaming the nfs-utils unit to nfs-services since a 'systemctl restart' of the unit start all the server and client daemons (even when the server is not enabled, which is probably a bug).

I don't object to a renaming. The name "nfs-utils" was simply the first thing I thought of.

systemctl restart nfs-utils

should *not* start any service that is not already started. I think I tested that. However if I'm wrong we could argue that "try-restart" should be used if you don't want to start services that aren't already running.... though I'm not sure that would work. Must test....

systemctl stop nfs-utils

should stop services. It uses "PartOf" which is supposed to apply to 'restart' and 'stop' equally.

Quoted 2 lines

Since a 'systemctl restart nfs-utils' starts all the daemons shouldn't 'systemctl stop nfs-utils' bring them all down as well?

Doesn't it?

Quoted 5 lines

Another oddity, going a 'systemctl restart nfs-utils' causes v3 mounts to go stale... Meaning going a ls on a v3 mount point after the restart errors out with ESTALE... Not sure why...

That is weird and certainly needs looking in to. I could be due to

ExecStopPost=/usr/sbin/exportfs -au

I wonder if we really need that.

I hope to get some time tomorrow to experiment with all this and I will definitely look into this issue (unless you explain it first :-)

Quoted 7 lines

Comment Three:

I'm not seeing how the nfs-utils_env.sh file, called by each unit, is all that useful. The main reason is you can not tell which unit its being called from so how do know what should be done? I guess I'm just missing the concept on how and what it should be used for.

I've got a patch which changes this so it is only called once in a separate service. However the principle remains unchanged.

The purpose of the script is to read /etc/sysconfig/nfs (or similar) and write out /run/sysconfig/nfs-utils containing *all* the environment variables used in *any* nfs-utils unit file. It allows a translation from what is expected in the sysconfig file to what the daemons expect as their arguments.

I see this as a transitional thing. Ultimately the various daemons should be able to inspect their environment, and expect exactly the things that might be in the config file. So systemd just reads /etc/sysconfig/nfs and then runs the daemons. But that goal is a little ways off yet.

One thing we have in our sysconfig file is "NFS3_SERVER_SUPPORT". If that is set to "no", then as well as constructing appropriate arguments for rpc.mountd and rpc.nfs we don't run rpc.statd (unless a v3 filesystem is mounted). Handling the first part with nfs-utils_env.sh is trivial. Handling the second bit is something I haven't figured out yet.

Thanks for your comments.

NeilBrown

Quoted 9 lines

Steve Dickson (2):
rpc-svcgssd.service: removed a the start up triggers
systemd: Removed the "ordering cycle" from nfs-server.service

 systemd/nfs-server.service  | 2 --
 systemd/rpc-svcgssd.service | 1 -
 2 files changed, 3 deletions(-)