Skip to contentSkip to conversation

Boot time improvement with systemd and nfs-utils

PATCH In review Hackintosh Five 3 participants 9 messages Jan 4, 2021 – Jan 5, 2021 2 messages not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

rpc-statd-notify is causing a 10 second hang on my system during boot due to an unwanted dependency on network-online.target. This dependency isn't needed anyway, because rpc-statd-notify (sm-notify) will wait for the network to come online if it isn't already (up to 15 minutes, so no risk of timeout that would be avoided by systemd)
=============================================
From c90bd7e701c2558606907f08bf27ae9be3f8e0bf Mon Sep 17 00:00:00 2001
From: Hackintosh 5 <[email protected]>
Date: Sat, 2 Jan 2021 14:28:30 +0000
Subject: [PATCH] systemd: network-online.target is not needed for
rpc-statd-notify.service

Commit 09e5c6c2 changed the After line for rpc-statd-notify to change network.target to network-online.target, which is incorrect, because sm-notify has a default timeout of 15 minutes, which is longer than the timeout for network-online.target. In other words, the dependency on network-online.target is useless and delays system boot by ~10 seconds.

---
 systemd/rpc-statd-notify.service | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
(quoted diff) +0 -0

b/systemd/rpc-statd-notify.service
index aad4c0d2..8a40e862 100644

systemd/rpc-statd-notify.service +2 -2
@@ -1,8 +1,8 @@
 [Unit]
 Description=Notify NFS peers of a restart
 DefaultDependencies=no
-Wants=network-online.target
-After=local-fs.target network-online.target nss-lookup.target
+Wants=network.target
+After=local-fs.target network.target nss-lookup.target
 
 # if we run an nfs server, it needs to be running before we
 # tell clients that it has restarted.
Signature · 1 line
2.29.2
#1 lore raw

Hello, thanks for your report.

The dependency you are removing addresses a bug -- if the network is not configured when rpc.statd is started, the rpc.statd process continues to use incorrect local address information even after the network is up.

Quoted 38 lines · Hackintosh Five

rpc-statd-notify is causing a 10 second hang on my system during boot due to an unwanted dependency on network-online.target. This dependency isn't needed anyway, because rpc-statd-notify (sm-notify) will wait for the network to come online if it isn't already (up to 15 minutes, so no risk of timeout that would be avoided by systemd)
=============================================
From c90bd7e701c2558606907f08bf27ae9be3f8e0bf Mon Sep 17 00:00:00 2001
From: Hackintosh 5 <[email protected]>
Date: Sat, 2 Jan 2021 14:28:30 +0000
Subject: [PATCH] systemd: network-online.target is not needed for rpc-statd-notify.service

Commit 09e5c6c2 changed the After line for rpc-statd-notify to change network.target to network-online.target, which is incorrect, because sm-notify has a default timeout of 15 minutes, which is longer than the timeout for network-online.target. In other words, the dependency on network-online.target is useless and delays system boot by ~10 seconds.
---
systemd/rpc-statd-notify.service | 4 ++--

1 file changed, 2 insertions(+), 2 deletions(-)
systemd/rpc-statd-notify.service +0 -0
index aad4c0d2..8a40e862 100644
@@ -1,8 +1,8 @@

[Unit]
Description=Notify NFS peers of a restart
DefaultDependencies=no

systemd/rpc-statd-notify.service +2 -2
-Wants=network-online.target
-After=local-fs.target network-online.target nss-lookup.target
+Wants=network.target
+After=local-fs.target network.target nss-lookup.target
Quoted 4 lines
# if we run an nfs server, it needs to be running before we
# tell clients that it has restarted.
Signature · 1 line
2.29.2

--
Chuck Lever

#2 lore raw

Hi, thanks for the fast reply

I have never even used nfs and I'm not a systemd expert, so I'm not at all sure this interpretation is correct, but here goes. I only removed the dependency from rpc.statd.notify, not rpc.statd. I didn't remove the After=nfs-server line, and for nfs-server to be up, network-online must be up first (there's an After requirement in the nfs-server unit). So if the nfs-server is enabled, the rpc-statd-notify will order itself after the server is up, which depends on the network. That means that, if there is a server, the server must be up before it sends notifications, so it will have the right hostname. This only improves boot speed on nfs clients, where nfs-client.target pulls in rpc-statd-notify.service.

Quoted 54 lines · Chuck Lever

Hello, thanks for your report.

The dependency you are removing addresses a bug -- if the network is not configured when rpc.statd is started, the rpc.statd process continues to use incorrect local address information even after the network is up.

Quoted 38 lines · Hackintosh Five

rpc-statd-notify is causing a 10 second hang on my system during boot due to an unwanted dependency on network-online.target. This dependency isn't needed anyway, because rpc-statd-notify (sm-notify) will wait for the network to come online if it isn't already (up to 15 minutes, so no risk of timeout that would be avoided by systemd)
=============================================
From c90bd7e701c2558606907f08bf27ae9be3f8e0bf Mon Sep 17 00:00:00 2001
From: Hackintosh 5 <[email protected]>
Date: Sat, 2 Jan 2021 14:28:30 +0000
Subject: [PATCH] systemd: network-online.target is not needed for rpc-statd-notify.service

Commit 09e5c6c2 changed the After line for rpc-statd-notify to change network.target to network-online.target, which is incorrect, because sm-notify has a default timeout of 15 minutes, which is longer than the timeout for network-online.target. In other words, the dependency on network-online.target is useless and delays system boot by ~10 seconds.
---
systemd/rpc-statd-notify.service | 4 ++--

1 file changed, 2 insertions(+), 2 deletions(-)
systemd/rpc-statd-notify.service +0 -0
index aad4c0d2..8a40e862 100644
@@ -1,8 +1,8 @@

[Unit]
Description=Notify NFS peers of a restart
DefaultDependencies=no

systemd/rpc-statd-notify.service +2 -2
-Wants=network-online.target
-After=local-fs.target network-online.target nss-lookup.target
+Wants=network.target
+After=local-fs.target network.target nss-lookup.target
Quoted 4 lines
# if we run an nfs server, it needs to be running before we
# tell clients that it has restarted.
--
2.29.2

--
Chuck Lever

#3 lore raw

Hi-

Quoted 7 lines · Hackintosh Five

Hi, thanks for the fast reply

I have never even used nfs and I'm not a systemd expert, so I'm not at all sure this interpretation is correct, but here goes. I only removed the dependency from rpc.statd.notify, not rpc.statd.

Same problem exists for sm-notify.

Quoted 6 lines

I didn't remove
the After=nfs-server line, and for nfs-server to be up, network-online must be up first (there's an After requirement in the nfs-server unit). So if the nfs-server is enabled, the rpc-statd-notify will order itself after the server is up, which depends on the network.

IIRC sm-notify runs on clients too. That's why the dependency is on the network and not on nfs-server.

Quoted 6 lines

That means that, if there is a server, the
server must be up before it sends notifications, so it will have the right hostname. This only improves boot speed on nfs clients, where nfs-client.target pulls in rpc-statd-notify.service.

Quoted 55 lines
Quoted 54 lines · Chuck Lever

Hello, thanks for your report.

The dependency you are removing addresses a bug -- if the network is not configured when rpc.statd is started, the rpc.statd process continues to use incorrect local address information even after the network is up.

Quoted 38 lines · Hackintosh Five

rpc-statd-notify is causing a 10 second hang on my system during boot due to an unwanted dependency on network-online.target. This dependency isn't needed anyway, because rpc-statd-notify (sm-notify) will wait for the network to come online if it isn't already (up to 15 minutes, so no risk of timeout that would be avoided by systemd)
=============================================
From c90bd7e701c2558606907f08bf27ae9be3f8e0bf Mon Sep 17 00:00:00 2001
From: Hackintosh 5 <[email protected]>
Date: Sat, 2 Jan 2021 14:28:30 +0000
Subject: [PATCH] systemd: network-online.target is not needed for rpc-statd-notify.service

Commit 09e5c6c2 changed the After line for rpc-statd-notify to change network.target to network-online.target, which is incorrect, because sm-notify has a default timeout of 15 minutes, which is longer than the timeout for network-online.target. In other words, the dependency on network-online.target is useless and delays system boot by ~10 seconds.
---
systemd/rpc-statd-notify.service | 4 ++--

1 file changed, 2 insertions(+), 2 deletions(-)
systemd/rpc-statd-notify.service +0 -0
index aad4c0d2..8a40e862 100644
@@ -1,8 +1,8 @@

[Unit]
Description=Notify NFS peers of a restart
DefaultDependencies=no

systemd/rpc-statd-notify.service +2 -2
-Wants=network-online.target
-After=local-fs.target network-online.target nss-lookup.target
+Wants=network.target
+After=local-fs.target network.target nss-lookup.target
Quoted 4 lines
# if we run an nfs server, it needs to be running before we
# tell clients that it has restarted.
--
2.29.2

--
Chuck Lever

--
Chuck Lever

A message that is not in the archive (sent off-list or not yet received): CAL5u83Fxd2rGuYuaghcC4irUtscmXr5-p36Qqf4+FwtctZJFaQ@mail.gmail.com

A message that is not in the archive (sent off-list or not yet received): [email protected]

#6 lore raw

I see. Does rpc-statd-notify HAVE to start before nfs-client? If not, perhaps a one-off timer unit with no delay could be made so that the startup of rpc-statd-notify doesn't block the boot process, while still running after network-online?

Quoted 107 lines · Chuck Lever

The problem is not in sm-notify itself, it's in the C library functions. The system's DNS resolver configuration is set during network startup. When a process first attempts a DNS query, it retrieves the system DNS configuration as it is at that moment, and keeps that configuration until the process exits. If sm-notify starts before the system's DNS resolver is configured, then it simply doesn't work because it can't perform DNS queries correctly.

On Jan 4, 2021, at 8:13 AM, Hackintosh Five <[email protected]> wrote:

Yep, sm-notify is Wanted by nfs-utils.target, and hence clients. But I can't see anywhere in the source of lm-notify where being offline would make a difference (beyond triggering a retry). If such a function does exist and I missed it, perhaps it could be moved to the end of the program (so the hostname is calculated only when the network is available, or whatever)

Quoted 87 lines · On Mon, 4 Jan 2021, 12:54 Chuck Lever, <[email protected]> wrote:

Hi-

Quoted 7 lines · Hackintosh Five

Hi, thanks for the fast reply

I have never even used nfs and I'm not a systemd expert, so I'm not at all sure this interpretation is correct, but here goes. I only removed the dependency from rpc.statd.notify, not rpc.statd.

Same problem exists for sm-notify.

Quoted 6 lines

I didn't remove
the After=nfs-server line, and for nfs-server to be up, network-online must be up first (there's an After requirement in the nfs-server unit). So if the nfs-server is enabled, the rpc-statd-notify will order itself after the server is up, which depends on the network.

IIRC sm-notify runs on clients too. That's why the dependency is on the network and not on nfs-server.

Quoted 6 lines

That means that, if there is a server, the
server must be up before it sends notifications, so it will have the right hostname. This only improves boot speed on nfs clients, where nfs-client.target pulls in rpc-statd-notify.service.

Quoted 55 lines
Quoted 54 lines · Chuck Lever

Hello, thanks for your report.

The dependency you are removing addresses a bug -- if the network is not configured when rpc.statd is started, the rpc.statd process continues to use incorrect local address information even after the network is up.

Quoted 38 lines · Hackintosh Five

rpc-statd-notify is causing a 10 second hang on my system during boot due to an unwanted dependency on network-online.target. This dependency isn't needed anyway, because rpc-statd-notify (sm-notify) will wait for the network to come online if it isn't already (up to 15 minutes, so no risk of timeout that would be avoided by systemd)
=============================================
From c90bd7e701c2558606907f08bf27ae9be3f8e0bf Mon Sep 17 00:00:00 2001
From: Hackintosh 5 <[email protected]>
Date: Sat, 2 Jan 2021 14:28:30 +0000
Subject: [PATCH] systemd: network-online.target is not needed for rpc-statd-notify.service

Commit 09e5c6c2 changed the After line for rpc-statd-notify to change network.target to network-online.target, which is incorrect, because sm-notify has a default timeout of 15 minutes, which is longer than the timeout for network-online.target. In other words, the dependency on network-online.target is useless and delays system boot by ~10 seconds.
---
systemd/rpc-statd-notify.service | 4 ++--

1 file changed, 2 insertions(+), 2 deletions(-)
systemd/rpc-statd-notify.service +0 -0
index aad4c0d2..8a40e862 100644
@@ -1,8 +1,8 @@

[Unit]
Description=Notify NFS peers of a restart
DefaultDependencies=no

systemd/rpc-statd-notify.service +2 -2
-Wants=network-online.target
-After=local-fs.target network-online.target nss-lookup.target
+Wants=network.target
+After=local-fs.target network.target nss-lookup.target
Quoted 4 lines
# if we run an nfs server, it needs to be running before we
# tell clients that it has restarted.
--
2.29.2

--
Chuck Lever

Quoted 5 lines

--
Chuck Lever

--
Chuck Lever

#7 lore raw

I'm not aware of a similar reason why the notify step would need to block the boot process. Maybe Type=forking is wrong, but I thought sm-notify was a forking/daemonizing type of utility.

Quoted 10 lines · Hackintosh Five

I see. Does rpc-statd-notify HAVE to start before nfs-client? If not, perhaps a one-off timer unit with no delay could be made so that the startup of rpc-statd-notify doesn't block the boot process, while still running after network-online?

Quoted 2 lines · Chuck Lever

The problem is not in sm-notify itself, it's in the C library functions. The system's DNS resolver configuration is set during network startup. When a process first attempts a DNS query, it retrieves the system DNS configuration as it is at that moment, and keeps that configuration until the process exits. If sm-notify starts before the system's DNS resolver is configured, then it simply doesn't work because it can't perform DNS queries correctly.

--
Chuck Lever

#8 lore raw

The issue isn't with the forking type, that's certainly correct (since it does indeed fork). The problem is that systemd is putting a dependency between nfs-client (required by multi-user.target) and rpc-statd-notify (which requires network-online), resulting in gdm waiting for network-online. The only workaround I was able to make work was to create a new timer unit which simply launches sm-notify after 1 second. nfs-client can start the timer unit, which then *asynchronously* starts sm-notify, meaning that sm-notify gets to keep its dependency on network-online. Patch with that method will be sent in a moment (if git send-email decides to work)

Quoted 23 lines · Chuck Lever

I'm not aware of a similar reason why the notify step would need to block the boot process. Maybe Type=forking is wrong, but I thought sm-notify was a forking/daemonizing type of utility.

Quoted 10 lines · Hackintosh Five

I see. Does rpc-statd-notify HAVE to start before nfs-client? If not, perhaps a one-off timer unit with no delay could be made so that the startup of rpc-statd-notify doesn't block the boot process, while still running after network-online?

Quoted 2 lines · Chuck Lever

The problem is not in sm-notify itself, it's in the C library functions. The system's DNS resolver configuration is set during network startup. When a process first attempts a DNS query, it retrieves the system DNS configuration as it is at that moment, and keeps that configuration until the process exits. If sm-notify starts before the system's DNS resolver is configured, then it simply doesn't work because it can't perform DNS queries correctly.

--
Chuck Lever

#9 lore raw

PATCH v2 1/1 systemd: rpc-statd-notify.service can run in the background

In reply to Chuck Lever

This allows rpc-statd-notify to run in the background when it is only in use by a client. This is done by a timer unit with a one second timeout, which is Wanted by nfs-client.target. The result is that there is no longer a dependency on network-online.target by multi-user.target, so everyone gets faster boot times yay.

---
 systemd/nfs-client.target      | 2 +-
 systemd/rpc-statd-notify.timer | 9 +++++++++
 2 files changed, 10 insertions(+), 1 deletion(-)
 create mode 100644 systemd/rpc-statd-notify.timer
systemd/nfs-client.target +1 -1
index 8a8300a1..b7cce746 100644
@@ -5,7 +5,7 @@ Wants=remote-fs-pre.target
 
 # Note: we don't "Wants=rpc-statd.service" as "mount.nfs" will arrange to
 # start that on demand if needed.
-Wants=rpc-statd-notify.service
+Wants=rpc-statd-notify.timer
 
 # GSS services dependencies and ordering
 Wants=auth-rpcgss-module.service
systemd/rpc-statd-notify.timer new file +9 -0
new file mode 100644
index 00000000..bac68817
@@ -0,0 +1,9 @@
+[Unit]
+Description=Notify NFS peers of a restart
+RefuseManualStart=true
+RefuseManualStop=true
+
+[Timer]
+OnActiveSec=1
+Unit=rpc-statd-notify.service
+RemainAfterElapse=false
Signature · 1 line
2.29.2
reviewer
#10 lore raw

Re: [PATCH v2] systemd: rpc-statd-notify.service can run in the background

Quoted 5 lines · On 1/4/2021 1:55 PM, Hackintosh 5 wrote:

This allows rpc-statd-notify to run in the background when it is only in use by a client. This is done by a timer unit with a one second timeout, which is Wanted by nfs-client.target. The result is that there is no longer a dependency on network-online.target by multi-user.target, so everyone gets faster boot times yay.

I'm concerned that this change may allow the nfs client to start before the sm-notify has a chance to send its "I'm back" message to the server, and for the server to process it. This will lead to lock failures.

Also, I'm unclear how an apparently arbitrary 1-second delay is fixing this. Is this really a systemd thing? If so, changing the NFS behavior is the wrong approach.

Tom.

Quoted 35 lines
---
  systemd/nfs-client.target      | 2 +-
  systemd/rpc-statd-notify.timer | 9 +++++++++
  2 files changed, 10 insertions(+), 1 deletion(-)

create mode 100644 systemd/rpc-statd-notify.timer

systemd/nfs-client.target +1 -1
index 8a8300a1..b7cce746 100644
@@ -5,7 +5,7 @@ Wants=remote-fs-pre.target
 
  # Note: we don't "Wants=rpc-statd.service" as "mount.nfs" will arrange to
  # start that on demand if needed.
-Wants=rpc-statd-notify.service
+Wants=rpc-statd-notify.timer
 
  # GSS services dependencies and ordering
  Wants=auth-rpcgss-module.service
systemd/rpc-statd-notify.timer new file +9 -0
new file mode 100644
index 00000000..bac68817
@@ -0,0 +1,9 @@
+[Unit]
+Description=Notify NFS peers of a restart
+RefuseManualStart=true
+RefuseManualStop=true
+
+[Timer]
+OnActiveSec=1
+Unit=rpc-statd-notify.service
+RemainAfterElapse=false