Boot time improvement with systemd and nfs-utils
Subsystems (2)
rpc-statd-notify is causing a 10 second hang on my system during boot due to an unwanted dependency on network-online.target. This dependency isn't needed anyway, because rpc-statd-notify (sm-notify) will wait for the network to come online if it isn't already (up to 15 minutes, so no risk of timeout that would be avoided by systemd)
=============================================
From c90bd7e701c2558606907f08bf27ae9be3f8e0bf Mon Sep 17 00:00:00 2001
From: Hackintosh 5 <[email protected]>
Date: Sat, 2 Jan 2021 14:28:30 +0000
Subject: [PATCH] systemd: network-online.target is not needed for
rpc-statd-notify.service
Commit 09e5c6c2 changed the After line for rpc-statd-notify to change network.target to network-online.target, which is incorrect, because sm-notify has a default timeout of 15 minutes, which is longer than the timeout for network-online.target. In other words, the dependency on network-online.target is useless and delays system boot by ~10 seconds.
--- systemd/rpc-statd-notify.service | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
(quoted diff) +0 -0
b/systemd/rpc-statd-notify.service
index aad4c0d2..8a40e862 100644
systemd/rpc-statd-notify.service +2 -2
@@ -1,8 +1,8 @@ [Unit] Description=Notify NFS peers of a restart DefaultDependencies=no -Wants=network-online.target -After=local-fs.target network-online.target nss-lookup.target +Wants=network.target +After=local-fs.target network.target nss-lookup.target # if we run an nfs server, it needs to be running before we # tell clients that it has restarted.
Signature · 1 line
2.29.2
Hello, thanks for your report.
The dependency you are removing addresses a bug -- if the network is not configured when rpc.statd is started, the rpc.statd process continues to use incorrect local address information even after the network is up.
Quoted 38 lines · Hackintosh Five
rpc-statd-notify is causing a 10 second hang on my system during boot due to an unwanted dependency on network-online.target. This dependency isn't needed anyway, because rpc-statd-notify (sm-notify) will wait for the network to come online if it isn't already (up to 15 minutes, so no risk of timeout that would be avoided by systemd)
=============================================
Fromc90bd7e701c2558606907f08bf27ae9be3f8e0bfMon Sep 17 00:00:00 2001
From: Hackintosh 5 <[email protected]>
Date: Sat, 2 Jan 2021 14:28:30 +0000
Subject: [PATCH] systemd: network-online.target is not needed for rpc-statd-notify.serviceCommit
09e5c6c2changed the After line for rpc-statd-notify to change network.target to network-online.target, which is incorrect, because sm-notify has a default timeout of 15 minutes, which is longer than the timeout for network-online.target. In other words, the dependency on network-online.target is useless and delays system boot by ~10 seconds.
---
systemd/rpc-statd-notify.service | 4 ++--1 file changed, 2 insertions(+), 2 deletions(-)systemd/rpc-statd-notify.service +0
-0@@ -1,8 +1,8 @@[Unit]
Description=Notify NFS peers of a restart
DefaultDependencies=nosystemd/rpc-statd-notify.service +2
-2-Wants=network-online.target -After=local-fs.target network-online.target nss-lookup.target +Wants=network.target +After=local-fs.target network.target nss-lookup.target
Quoted 4 lines
# if we run an nfs server, it needs to be running before we # tell clients that it has restarted.Signature · 1 line
2.29.2
--
Chuck Lever
Hi, thanks for the fast reply
I have never even used nfs and I'm not a systemd expert, so I'm not at all sure this interpretation is correct, but here goes. I only removed the dependency from rpc.statd.notify, not rpc.statd. I didn't remove the After=nfs-server line, and for nfs-server to be up, network-online must be up first (there's an After requirement in the nfs-server unit). So if the nfs-server is enabled, the rpc-statd-notify will order itself after the server is up, which depends on the network. That means that, if there is a server, the server must be up before it sends notifications, so it will have the right hostname. This only improves boot speed on nfs clients, where nfs-client.target pulls in rpc-statd-notify.service.
Quoted 54 lines · Chuck Lever
Hello, thanks for your report.
The dependency you are removing addresses a bug -- if the network is not configured when rpc.statd is started, the rpc.statd process continues to use incorrect local address information even after the network is up.
Quoted 38 lines · Hackintosh Five
rpc-statd-notify is causing a 10 second hang on my system during boot due to an unwanted dependency on network-online.target. This dependency isn't needed anyway, because rpc-statd-notify (sm-notify) will wait for the network to come online if it isn't already (up to 15 minutes, so no risk of timeout that would be avoided by systemd)
=============================================
Fromc90bd7e701c2558606907f08bf27ae9be3f8e0bfMon Sep 17 00:00:00 2001
From: Hackintosh 5 <[email protected]>
Date: Sat, 2 Jan 2021 14:28:30 +0000
Subject: [PATCH] systemd: network-online.target is not needed for rpc-statd-notify.serviceCommit
09e5c6c2changed the After line for rpc-statd-notify to change network.target to network-online.target, which is incorrect, because sm-notify has a default timeout of 15 minutes, which is longer than the timeout for network-online.target. In other words, the dependency on network-online.target is useless and delays system boot by ~10 seconds.
---
systemd/rpc-statd-notify.service | 4 ++--1 file changed, 2 insertions(+), 2 deletions(-)systemd/rpc-statd-notify.service +0
-0@@ -1,8 +1,8 @@[Unit]
Description=Notify NFS peers of a restart
DefaultDependencies=nosystemd/rpc-statd-notify.service +2
-2-Wants=network-online.target -After=local-fs.target network-online.target nss-lookup.target +Wants=network.target +After=local-fs.target network.target nss-lookup.targetQuoted 4 lines
# if we run an nfs server, it needs to be running before we # tell clients that it has restarted. -- 2.29.2--
Chuck Lever
Hi-
Quoted 7 lines · Hackintosh Five
Hi, thanks for the fast reply
I have never even used nfs and I'm not a systemd expert, so I'm not at all sure this interpretation is correct, but here goes. I only removed the dependency from rpc.statd.notify, not rpc.statd.
Same problem exists for sm-notify.
Quoted 6 lines
I didn't remove
theAfter=nfs-serverline, and for nfs-server to be up, network-online must be up first (there's an After requirement in the nfs-server unit). So if the nfs-server is enabled, the rpc-statd-notify will order itself after the server is up, which depends on the network.
IIRC sm-notify runs on clients too. That's why the dependency is on the network and not on nfs-server.
Quoted 6 lines
That means that, if there is a server, the
server must be up before it sends notifications, so it will have the right hostname. This only improves boot speed on nfs clients, where nfs-client.target pulls in rpc-statd-notify.service.
Quoted 55 lines
Quoted 54 lines · Chuck Lever
Hello, thanks for your report.
The dependency you are removing addresses a bug -- if the network is not configured when rpc.statd is started, the rpc.statd process continues to use incorrect local address information even after the network is up.
Quoted 38 lines · Hackintosh Five
rpc-statd-notify is causing a 10 second hang on my system during boot due to an unwanted dependency on network-online.target. This dependency isn't needed anyway, because rpc-statd-notify (sm-notify) will wait for the network to come online if it isn't already (up to 15 minutes, so no risk of timeout that would be avoided by systemd)
=============================================
Fromc90bd7e701c2558606907f08bf27ae9be3f8e0bfMon Sep 17 00:00:00 2001
From: Hackintosh 5 <[email protected]>
Date: Sat, 2 Jan 2021 14:28:30 +0000
Subject: [PATCH] systemd: network-online.target is not needed for rpc-statd-notify.serviceCommit
09e5c6c2changed the After line for rpc-statd-notify to change network.target to network-online.target, which is incorrect, because sm-notify has a default timeout of 15 minutes, which is longer than the timeout for network-online.target. In other words, the dependency on network-online.target is useless and delays system boot by ~10 seconds.
---
systemd/rpc-statd-notify.service | 4 ++--1 file changed, 2 insertions(+), 2 deletions(-)systemd/rpc-statd-notify.service +0
-0@@ -1,8 +1,8 @@[Unit]
Description=Notify NFS peers of a restart
DefaultDependencies=nosystemd/rpc-statd-notify.service +2
-2-Wants=network-online.target -After=local-fs.target network-online.target nss-lookup.target +Wants=network.target +After=local-fs.target network.target nss-lookup.targetQuoted 4 lines
# if we run an nfs server, it needs to be running before we # tell clients that it has restarted. -- 2.29.2--
Chuck Lever
--
Chuck Lever
A message that is not in the archive (sent off-list or not yet received): CAL5u83Fxd2rGuYuaghcC4irUtscmXr5-p36Qqf4+FwtctZJFaQ@mail.gmail.com
A message that is not in the archive (sent off-list or not yet received): [email protected]
I see. Does rpc-statd-notify HAVE to start before nfs-client? If not, perhaps a one-off timer unit with no delay could be made so that the startup of rpc-statd-notify doesn't block the boot process, while still running after network-online?
Quoted 107 lines · Chuck Lever
The problem is not in sm-notify itself, it's in the C library functions. The system's DNS resolver configuration is set during network startup. When a process first attempts a DNS query, it retrieves the system DNS configuration as it is at that moment, and keeps that configuration until the process exits. If sm-notify starts before the system's DNS resolver is configured, then it simply doesn't work because it can't perform DNS queries correctly.
On Jan 4, 2021, at 8:13 AM, Hackintosh Five <[email protected]> wrote:
Yep, sm-notify is Wanted by nfs-utils.target, and hence clients. But I can't see anywhere in the source of lm-notify where being offline would make a difference (beyond triggering a retry). If such a function does exist and I missed it, perhaps it could be moved to the end of the program (so the hostname is calculated only when the network is available, or whatever)
Quoted 87 lines · On Mon, 4 Jan 2021, 12:54 Chuck Lever, <[email protected]> wrote:
Hi-
Quoted 7 lines · Hackintosh Five
Hi, thanks for the fast reply
I have never even used nfs and I'm not a systemd expert, so I'm not at all sure this interpretation is correct, but here goes. I only removed the dependency from rpc.statd.notify, not rpc.statd.
Same problem exists for sm-notify.
Quoted 6 lines
I didn't remove
theAfter=nfs-serverline, and for nfs-server to be up, network-online must be up first (there's an After requirement in the nfs-server unit). So if the nfs-server is enabled, the rpc-statd-notify will order itself after the server is up, which depends on the network.IIRC sm-notify runs on clients too. That's why the dependency is on the network and not on nfs-server.
Quoted 6 lines
That means that, if there is a server, the
server must be up before it sends notifications, so it will have the right hostname. This only improves boot speed on nfs clients, where nfs-client.target pulls in rpc-statd-notify.service.Quoted 55 lines
Quoted 54 lines · Chuck Lever
Hello, thanks for your report.
The dependency you are removing addresses a bug -- if the network is not configured when rpc.statd is started, the rpc.statd process continues to use incorrect local address information even after the network is up.
Quoted 38 lines · Hackintosh Five
rpc-statd-notify is causing a 10 second hang on my system during boot due to an unwanted dependency on network-online.target. This dependency isn't needed anyway, because rpc-statd-notify (sm-notify) will wait for the network to come online if it isn't already (up to 15 minutes, so no risk of timeout that would be avoided by systemd)
=============================================
Fromc90bd7e701c2558606907f08bf27ae9be3f8e0bfMon Sep 17 00:00:00 2001
From: Hackintosh 5 <[email protected]>
Date: Sat, 2 Jan 2021 14:28:30 +0000
Subject: [PATCH] systemd: network-online.target is not needed for rpc-statd-notify.serviceCommit
09e5c6c2changed the After line for rpc-statd-notify to change network.target to network-online.target, which is incorrect, because sm-notify has a default timeout of 15 minutes, which is longer than the timeout for network-online.target. In other words, the dependency on network-online.target is useless and delays system boot by ~10 seconds.
---
systemd/rpc-statd-notify.service | 4 ++--1 file changed, 2 insertions(+), 2 deletions(-)systemd/rpc-statd-notify.service +0
-0@@ -1,8 +1,8 @@[Unit]
Description=Notify NFS peers of a restart
DefaultDependencies=nosystemd/rpc-statd-notify.service +2
-2-Wants=network-online.target -After=local-fs.target network-online.target nss-lookup.target +Wants=network.target +After=local-fs.target network.target nss-lookup.targetQuoted 4 lines
# if we run an nfs server, it needs to be running before we # tell clients that it has restarted. -- 2.29.2--
Chuck LeverQuoted 5 lines
--
Chuck Lever--
Chuck Lever
I'm not aware of a similar reason why the notify step would need to block the boot process. Maybe Type=forking is wrong, but I thought sm-notify was a forking/daemonizing type of utility.
Quoted 10 lines · Hackintosh Five
I see. Does rpc-statd-notify HAVE to start before nfs-client? If not, perhaps a one-off timer unit with no delay could be made so that the startup of rpc-statd-notify doesn't block the boot process, while still running after network-online?
Quoted 2 lines · Chuck Lever
The problem is not in sm-notify itself, it's in the C library functions. The system's DNS resolver configuration is set during network startup. When a process first attempts a DNS query, it retrieves the system DNS configuration as it is at that moment, and keeps that configuration until the process exits. If sm-notify starts before the system's DNS resolver is configured, then it simply doesn't work because it can't perform DNS queries correctly.
--
Chuck Lever
The issue isn't with the forking type, that's certainly correct (since it does indeed fork). The problem is that systemd is putting a dependency between nfs-client (required by multi-user.target) and rpc-statd-notify (which requires network-online), resulting in gdm waiting for network-online. The only workaround I was able to make work was to create a new timer unit which simply launches sm-notify after 1 second. nfs-client can start the timer unit, which then *asynchronously* starts sm-notify, meaning that sm-notify gets to keep its dependency on network-online. Patch with that method will be sent in a moment (if git send-email decides to work)
Quoted 23 lines · Chuck Lever
I'm not aware of a similar reason why the notify step would need to block the boot process. Maybe Type=forking is wrong, but I thought sm-notify was a forking/daemonizing type of utility.
Quoted 10 lines · Hackintosh Five
I see. Does rpc-statd-notify HAVE to start before nfs-client? If not, perhaps a one-off timer unit with no delay could be made so that the startup of rpc-statd-notify doesn't block the boot process, while still running after network-online?
Quoted 2 lines · Chuck Lever
The problem is not in sm-notify itself, it's in the C library functions. The system's DNS resolver configuration is set during network startup. When a process first attempts a DNS query, it retrieves the system DNS configuration as it is at that moment, and keeps that configuration until the process exits. If sm-notify starts before the system's DNS resolver is configured, then it simply doesn't work because it can't perform DNS queries correctly.
--
Chuck Lever
PATCH v2 1/1 systemd: rpc-statd-notify.service can run in the background
In reply to Chuck LeverThis allows rpc-statd-notify to run in the background when it is only in use by a client. This is done by a timer unit with a one second timeout, which is Wanted by nfs-client.target. The result is that there is no longer a dependency on network-online.target by multi-user.target, so everyone gets faster boot times yay.
--- systemd/nfs-client.target | 2 +- systemd/rpc-statd-notify.timer | 9 +++++++++ 2 files changed, 10 insertions(+), 1 deletion(-) create mode 100644 systemd/rpc-statd-notify.timer
systemd/nfs-client.target +1 -1
@@ -5,7 +5,7 @@ Wants=remote-fs-pre.target # Note: we don't "Wants=rpc-statd.service" as "mount.nfs" will arrange to # start that on demand if needed. -Wants=rpc-statd-notify.service +Wants=rpc-statd-notify.timer # GSS services dependencies and ordering Wants=auth-rpcgss-module.service
systemd/rpc-statd-notify.timer new file +9 -0
@@ -0,0 +1,9 @@ +[Unit] +Description=Notify NFS peers of a restart +RefuseManualStart=true +RefuseManualStop=true + +[Timer] +OnActiveSec=1 +Unit=rpc-statd-notify.service +RemainAfterElapse=false
Signature · 1 line
2.29.2
Tom Talpey
reviewerRe: [PATCH v2] systemd: rpc-statd-notify.service can run in the background
Quoted 5 lines · On 1/4/2021 1:55 PM, Hackintosh 5 wrote:
This allows rpc-statd-notify to run in the background when it is only in use by a client. This is done by a timer unit with a one second timeout, which is Wanted by nfs-client.target. The result is that there is no longer a dependency on network-online.target by multi-user.target, so everyone gets faster boot times yay.
I'm concerned that this change may allow the nfs client to start before the sm-notify has a chance to send its "I'm back" message to the server, and for the server to process it. This will lead to lock failures.
Also, I'm unclear how an apparently arbitrary 1-second delay is fixing this. Is this really a systemd thing? If so, changing the NFS behavior is the wrong approach.
Tom.
Quoted 35 lines
--- systemd/nfs-client.target | 2 +- systemd/rpc-statd-notify.timer | 9 +++++++++ 2 files changed, 10 insertions(+), 1 deletion(-)create mode 100644 systemd/rpc-statd-notify.timer
systemd/nfs-client.target +1
-1@@ -5,7 +5,7 @@ Wants=remote-fs-pre.target # Note: we don't "Wants=rpc-statd.service" as "mount.nfs" will arrange to # start that on demand if needed. -Wants=rpc-statd-notify.service +Wants=rpc-statd-notify.timer # GSS services dependencies and ordering Wants=auth-rpcgss-module.servicesystemd/rpc-statd-notify.timer new file +9
-0@@ -0,0 +1,9 @@ +[Unit] +Description=Notify NFS peers of a restart +RefuseManualStart=true +RefuseManualStop=true + +[Timer] +OnActiveSec=1 +Unit=rpc-statd-notify.service +RemainAfterElapse=false