Skip to contentSkip to conversation

Make sm-notify faster if there are no servers to notify

PATCH Phil Endecott 5 participants 29 messages Oct 29, 2008 – Dec 11, 2008 8 messages not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Dear Experts,

sm-notify was taking a long time while my laptop booted. This was odd because I use NFS only rarely - via autofs - on that machine, and sm-notify actually has no-one to notify most of the time. So I have patched it as follows. Is this a legitimate thing to do?

utils/statd/sm-notify.c +4 -0
diff -ur nfs-utils-1.1.3.orig/utils/statd/sm-notify.c nfs-utils-1.1.3/utils/statd/sm-notify.c
@@ -169,6 +169,10 @@
 	backup_hosts(_SM_DIR_PATH, _SM_BAK_PATH);
 	get_hosts(_SM_BAK_PATH);
 
+	if (!hosts) {
+		return 0;
+	}
+
 	/* Get and update the NSM state. This will call sync() */
 	nsm_state = nsm_get_state(opt_update_state);
 	set_kernel_nsm_state(nsm_state);

Regards, Phil.
(Please Cc: me in any replies.)

A message that is not in the archive (sent off-list or not yet received): 1225239200402-YnoLgZYwwYuCbKHnblo0pmrPP3OPMK55cpQHUIT47Ck@public.gmane.org

#2 lore raw
Quoted 6 lines · Phil Endecott

Dear Experts,

sm-notify was taking a long time while my laptop booted. This was odd because I use NFS only rarely - via autofs - on that machine, and sm-notify actually has no-one to notify most of the time. So I have patched it as follows. Is this a legitimate thing to do?

It looks like your patch was committed to nfs-utils a couple weeks ago:
see c8d18e26d2a53d9036a32c2dafebccaf4ce1634d from

git://linux-nfs.org/nfs-utils

--b.

Quoted 26 lines

diff -ur nfs-utils-1.1.3.orig/utils/statd/sm-notify.c nfs-utils-1.1.3/utils/statd/sm-notify.c

utils/statd/sm-notify.c +4 -0
@@ -169,6 +169,10 @@
 	backup_hosts(_SM_DIR_PATH, _SM_BAK_PATH);
 	get_hosts(_SM_BAK_PATH);
 
+	if (!hosts) {
+		return 0;
+	}
+
 	/* Get and update the NSM state. This will call sync() */
 	nsm_state = nsm_get_state(opt_update_state);
 	set_kernel_nsm_state(nsm_state);

Regards, Phil.
(Please Cc: me in any replies.)

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#3 lore raw
Quoted 11 lines · J. Bruce Fields
Quoted 3 lines · Phil Endecott

Dear Experts,

sm-notify was taking a long time while my laptop booted. This was odd because I use NFS only rarely - via autofs - on that machine, and sm-notify actually has no-one to notify most of the time. So I have patched it as follows. Is this a legitimate thing to do?

It looks like your patch was committed to nfs-utils a couple weeks ago:
see c8d18e26d2a53d9036a32c2dafebccaf4ce1634d from

git://linux-nfs.org/nfs-utils

--b.

How curious. I guess someone saw my Debian bug report. No mention of it on this list as far as I can see though.

I presume from this that it is considered a safe thing to do.

Phil.

A message that is not in the archive (sent off-list or not yet received): 1225301403729-YnoLgZYwwYuCbKHnblo0pmrPP3OPMK55cpQHUIT47Ck@public.gmane.org

#5 lore raw
Quoted 12 lines · Phil Endecott
Quoted 11 lines · J. Bruce Fields
Quoted 6 lines · Phil Endecott

Dear Experts,

sm-notify was taking a long time while my laptop booted. This was odd because I use NFS only rarely - via autofs - on that machine, and sm-notify actually has no-one to notify most of the time. So I have patched it as follows. Is this a legitimate thing to do?

It looks like your patch was committed to nfs-utils a couple weeks ago:
see c8d18e26d2a53d9036a32c2dafebccaf4ce1634d from

Quoted 8 lines
Quoted 3 lines

git://linux-nfs.org/nfs-utils

--b.

How curious. I guess someone saw my Debian bug report. No mention of it on this list as far as I can see though.

I presume from this that it is considered a safe thing to do.

It looks right to me. Hopefully somebody actually has tested this on a client that holds locks when it reboots?

I remember this was one of the things Arjan mentioned having to disable in his "5-second boot" talk at the Linux Plumbers Conference, so you're not the only one to have noticed the problem....

--b.

#6 lore raw
Quoted 12 lines · J. Bruce Fields
Quoted 11 lines · Phil Endecott
Quoted 10 lines · J. Bruce Fields
Quoted 3 lines · Phil Endecott

Dear Experts,

sm-notify was taking a long time while my laptop booted. This was odd because I use NFS only rarely - via autofs - on that machine, and sm-notify actually has no-one to notify most of the time. So I have patched it as follows. Is this a legitimate thing to do?

It looks like your patch was committed to nfs-utils a couple weeks ago:
see c8d18e26d2a53d9036a32c2dafebccaf4ce1634d from

git://linux-nfs.org/nfs-utils

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

--b.

How curious. I guess someone saw my Debian bug report. No mention of it on this list as far as I can see though.

I presume from this that it is considered a safe thing to do.

It looks right to me. Hopefully somebody actually has tested this on a client that holds locks when it reboots?

Not me.

Quoted 3 lines

I remember this was one of the things Arjan mentioned having to disable in his "5-second boot" talk at the Linux Plumbers Conference, so you're not the only one to have noticed the problem....

Yes; I noticed the huge pause due to the sync() and applied this fix. It was only later that I looked at Arjan's slides (I had to wait until someone had converted then from PowerPoint to PDF...) and saw that he had the same thing in his bootchart.

Phil.

A message that is not in the archive (sent off-list or not yet received): 1225302305994-YnoLgZYwwYuCbKHnblo0pmrPP3OPMK55cpQHUIT47Ck@public.gmane.org

#8 lore raw
Quoted 22 lines · Phil Endecott
Quoted 21 lines · J. Bruce Fields
Quoted 20 lines · Phil Endecott
Quoted 9 lines · J. Bruce Fields
Quoted 7 lines · Phil Endecott

Dear Experts,

sm-notify was taking a long time while my laptop booted. This was odd because I use NFS only rarely - via autofs - on that machine, and sm-notify actually has no-one to notify most of the time. So I have patched it as follows. Is this a legitimate thing to do?

Quoted 6 lines

It looks like your patch was committed to nfs-utils a couple weeks ago:
see c8d18e26d2a53d9036a32c2dafebccaf4ce1634d from

git://linux-nfs.org/nfs-utils

--b.

How curious. I guess someone saw my Debian bug report. No mention of it on this list as far as I can see though.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

I presume from this that it is considered a safe thing to do.

It looks right to me. Hopefully somebody actually has tested this on a client that holds locks when it reboots?

Not me.

Ugh.

We really need someone doing regular lock recovery tests with both the latest kernel and latest nfs-utils.

Ideally we'd also have tests that could be easily run by anyone. Though there may be too many site-specific details involved in writing scripts that interact with (and reboot) multiple machines.

Quoted 8 lines
Quoted 3 lines

I remember this was one of the things Arjan mentioned having to disable in his "5-second boot" talk at the Linux Plumbers Conference, so you're not the only one to have noticed the problem....

Yes; I noticed the huge pause due to the sync() and applied this fix. It was only later that I looked at Arjan's slides (I had to wait until someone had converted then from PowerPoint to PDF...) and saw that he had the same thing in his bootchart.

Oh, so the time was all spent in the sync() in nsm_get_state()?

Anyway, I think the nsm state updating shouldn't matter if you don't even have any peers to notify.

--b.

#9 lore raw
Quoted 43 lines · J. Bruce Fields
Quoted 22 lines · Phil Endecott
Quoted 21 lines · J. Bruce Fields
Quoted 20 lines · Phil Endecott
Quoted 9 lines · J. Bruce Fields
Quoted 7 lines · Phil Endecott

Dear Experts,

sm-notify was taking a long time while my laptop booted. This was odd because I use NFS only rarely - via autofs - on that machine, and sm-notify actually has no-one to notify most of the time. So I have patched it as follows. Is this a legitimate thing to do?

Quoted 6 lines

It looks like your patch was committed to nfs-utils a couple weeks ago:
see c8d18e26d2a53d9036a32c2dafebccaf4ce1634d from

git://linux-nfs.org/nfs-utils

--b.

How curious. I guess someone saw my Debian bug report. No mention of it on this list as far as I can see though.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

I presume from this that it is considered a safe thing to do.

It looks right to me. Hopefully somebody actually has tested this on a client that holds locks when it reboots?

Not me.

Ugh.

We really need someone doing regular lock recovery tests with both the latest kernel and latest nfs-utils.

Ideally we'd also have tests that could be easily run by anyone. Though there may be too many site-specific details involved in writing scripts that interact with (and reboot) multiple machines.

Quoted 4 lines
Quoted 3 lines

I remember this was one of the things Arjan mentioned having to disable in his "5-second boot" talk at the Linux Plumbers Conference, so you're not the only one to have noticed the problem....

Quoted 6 lines
Quoted 4 lines

Yes; I noticed the huge pause due to the sync() and applied this fix. It was only later that I looked at Arjan's slides (I had to wait until someone had converted then from PowerPoint to PDF...) and saw that he had the same thing in his bootchart.

Oh, so the time was all spent in the sync() in nsm_get_state()?

I assume sync() is required because this logic performs a rename as well as a simple write?

Quoted 2 lines

Anyway, I think the nsm state updating shouldn't matter if you don't even have any peers to notify.

It probably does matter.

When a system is initially installed, it likely does not have a state file in /var/lib/nfs. This may be harmless if it's not present; rpc.statd probably does the right thing in this case.

However, the rest of the logic in nsm_get_state() is needed to bump the system's state value properly after every reboot. It may be inconsequential if there were no mounts or no NFS clients during the last reboot, but this is subtle. I wouldn't bet on it.

Quoted 7 lines

--b.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

#10 lore raw
Quoted 2 lines · Chuck Lever

I assume sync() is required because this logic performs a rename as well as a simple write?

I think an fsync() on the containing directory (together with an fsync() of the file itself) would do the job if you wanted to avoid the globaly sync(). I don't think ext3 is capable of doing anything finer-grained than a whole-filesystem sync, though, so this doesn't help many people in practice right now.

In any case, the rename adds an extra level of safety by ensuring the nsm state is updated atomically, so we shouldn't get rid of it.

Quoted 8 lines
Quoted 2 lines

Anyway, I think the nsm state updating shouldn't matter if you don't even have any peers to notify.

It probably does matter.

When a system is initially installed, it likely does not have a state file in /var/lib/nfs. This may be harmless if it's not present; rpc.statd probably does the right thing in this case.

The "right thing" in that case would be, I guess, to create a state file with "0" in it. It doesn't do that. So this patch *does* break stuff. Oops!

So should we revert it and do something else, or patch statd to create a new state file if necessary?

Quoted 4 lines

However, the rest of the logic in nsm_get_state() is needed to bump the system's state value properly after every reboot. It may be inconsequential if there were no mounts or no NFS clients during the last reboot, but this is subtle. I wouldn't bet on it.

If the state is only every communicated to hosts by notifications, then if we're not notifying, the update of the state can't matter.

(So is that premise correct?)

--b.

#11 lore raw
Quoted 22 lines · bfields
Quoted 2 lines · Chuck Lever

I assume sync() is required because this logic performs a rename as well as a simple write?

I think an fsync() on the containing directory (together with an fsync() of the file itself) would do the job if you wanted to avoid the globaly sync(). I don't think ext3 is capable of doing anything finer-grained than a whole-filesystem sync, though, so this doesn't help many people in practice right now.

In any case, the rename adds an extra level of safety by ensuring the nsm state is updated atomically, so we shouldn't get rid of it.

Quoted 8 lines
Quoted 2 lines

Anyway, I think the nsm state updating shouldn't matter if you don't even have any peers to notify.

It probably does matter.

When a system is initially installed, it likely does not have a state file in /var/lib/nfs. This may be harmless if it's not present; rpc.statd probably does the right thing in this case.

Quoted 6 lines

The "right thing" in that case would be, I guess, to create a state file with "0" in it. It doesn't do that. So this patch *does* break stuff. Oops!

So should we revert it and do something else, or patch statd to create a new state file if necessary?

It looks like this still needs to be fixed? I think it would be good enough just to teach rpc.nfsd to create the file if it doesn't exist.

--b.

Quoted 12 lines
Quoted 5 lines

However, the rest of the logic in nsm_get_state() is needed to bump the system's state value properly after every reboot. It may be inconsequential if there were no mounts or no NFS clients during the last reboot, but this is subtle. I wouldn't bet on it.

If the state is only every communicated to hosts by notifications, then if we're not notifying, the update of the state can't matter.

(So is that premise correct?)

--b.

#12 lore raw
Quoted 27 lines · J. Bruce Fields
Quoted 26 lines · bfields
Quoted 2 lines · Chuck Lever

I assume sync() is required because this logic performs a rename as well as a simple write?

I think an fsync() on the containing directory (together with an fsync() of the file itself) would do the job if you wanted to avoid the globaly sync(). I don't think ext3 is capable of doing anything finer-grained than a whole-filesystem sync, though, so this doesn't help many people in practice right now.

In any case, the rename adds an extra level of safety by ensuring the nsm state is updated atomically, so we shouldn't get rid of it.

Quoted 8 lines
Quoted 2 lines

Anyway, I think the nsm state updating shouldn't matter if you don't even have any peers to notify.

It probably does matter.

When a system is initially installed, it likely does not have a state file in /var/lib/nfs. This may be harmless if it's not present; rpc.statd probably does the right thing in this case.

The "right thing" in that case would be, I guess, to create a state file with "0" in it. It doesn't do that. So this patch *does* break stuff. Oops!

Quoted 5 lines
Quoted 2 lines

So should we revert it and do something else, or patch statd to create a new state file if necessary?

It looks like this still needs to be fixed? I think it would be good enough just to teach rpc.nfsd to create the file if it doesn't exist.

Meh. I'd rather manage the state file in one place, rather than have multiple user space entities fiddle with it.

I think we should find out exactly what breaks when sm-notify quits early. Steve hasn't found a problem with the patch already in nfs-utils, but the corner cases here are really narrow.

Without a lot of testing (which we currently don't have the resources for), I don't feel 100% positive about sm-notify quitting early. My preferred solution would involve working around the sync(2) call instead (ie fixing sm-notify so we don't need it, or somehow doing it in the background so it doesn't hold up the boot-up process). I think we will end up waiting until this actually bites someone, but chances are it will be a long wait.

Quoted 14 lines

--b.

Quoted 12 lines
Quoted 5 lines

However, the rest of the logic in nsm_get_state() is needed to bump the system's state value properly after every reboot. It may be inconsequential if there were no mounts or no NFS clients during the last reboot, but this is subtle. I wouldn't bet on it.

If the state is only every communicated to hosts by notifications, then if we're not notifying, the update of the state can't matter.

(So is that premise correct?)

--b.

Signature · 2 lines
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
#13 lore raw
Quoted 37 lines · Chuck Lever
Quoted 36 lines · J. Bruce Fields
Quoted 35 lines · bfields
Quoted 3 lines · Chuck Lever

I assume sync() is required because this logic performs a rename as well
as a simple write?

I think an fsync() on the containing directory (together with an fsync()
of the file itself) would do the job if you wanted to avoid the globaly
sync(). I don't think ext3 is capable of doing anything finer- grained
than a whole-filesystem sync, though, so this doesn't help many people
in practice right now.

In any case, the rename adds an extra level of safety by ensuring the nsm state is updated atomically, so we shouldn't get rid of it.

Quoted 10 lines
Quoted 3 lines

Anyway, I think the nsm state updating shouldn't matter if you don't
even have any peers to notify.

It probably does matter.

When a system is initially installed, it likely does not have a state
file in /var/lib/nfs. This may be harmless if it's not present; rpc.statd probably does the right thing in this case.

The "right thing" in that case would be, I guess, to create a state file
with "0" in it. It doesn't do that. So this patch *does* break stuff.
Oops!

Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

So should we revert it and do something else, or patch statd to create
a new state file if necessary?

It looks like this still needs to be fixed? I think it would be good enough just to teach rpc.nfsd to create the file if it doesn't exist.

			^^^^^^^^

(Err, sorry, note I meant rpc.statd, not rpc.nfsd.)

--b.

Quoted 14 lines

Meh. I'd rather manage the state file in one place, rather than have multiple user space entities fiddle with it.

I think we should find out exactly what breaks when sm-notify quits early. Steve hasn't found a problem with the patch already in nfs- utils, but the corner cases here are really narrow.

Without a lot of testing (which we currently don't have the resources for), I don't feel 100% positive about sm-notify quitting early. My preferred solution would involve working around the sync(2) call instead (ie fixing sm-notify so we don't need it, or somehow doing it in the background so it doesn't hold up the boot-up process). I think we will end up waiting until this actually bites someone, but chances are it will be a long wait.

#14 lore raw
Quoted 42 lines · J. Bruce Fields
Quoted 41 lines · Chuck Lever
Quoted 40 lines · J. Bruce Fields
Quoted 38 lines · bfields
Quoted 3 lines · Chuck Lever

I assume sync() is required because this logic performs a rename as well
as a simple write?

I think an fsync() on the containing directory (together with an fsync()
of the file itself) would do the job if you wanted to avoid the globaly
sync(). I don't think ext3 is capable of doing anything finer- grained
than a whole-filesystem sync, though, so this doesn't help many people
in practice right now.

In any case, the rename adds an extra level of safety by ensuring the nsm state is updated atomically, so we shouldn't get rid of it.

Quoted 10 lines
Quoted 3 lines

Anyway, I think the nsm state updating shouldn't matter if you don't
even have any peers to notify.

It probably does matter.

When a system is initially installed, it likely does not have a state
file in /var/lib/nfs. This may be harmless if it's not present; rpc.statd probably does the right thing in this case.

The "right thing" in that case would be, I guess, to create a state file
with "0" in it. It doesn't do that. So this patch *does* break stuff.
Oops!

So should we revert it and do something else, or patch statd to create
a new state file if necessary?

Quoted 10 lines
Quoted 2 lines
Quoted 2 lines

It looks like this still needs to be fixed? I think it would be good enough just to teach rpc.nfsd to create the file if it doesn't exist.

			^^^^^^^^

(Err, sorry, note I meant rpc.statd, not rpc.nfsd.)

--b.

Quoted 2 lines

Meh. I'd rather manage the state file in one place, rather than have multiple user space entities fiddle with it.

Acknowledged. My comments below still stand, I think. More testing! Yay!

Quoted 11 lines
Quoted 11 lines

I think we should find out exactly what breaks when sm-notify quits early. Steve hasn't found a problem with the patch already in nfs- utils, but the corner cases here are really narrow.

Without a lot of testing (which we currently don't have the resources for), I don't feel 100% positive about sm-notify quitting early. My preferred solution would involve working around the sync(2) call instead (ie fixing sm-notify so we don't need it, or somehow doing it in the background so it doesn't hold up the boot-up process). I think we will end up waiting until this actually bites someone, but chances are it will be a long wait.

In any event, I will keep this in mind (along with the idea of allowing only client or server peers to be notified of a reboot) as I audit this code for IPv6 support.

Signature · 2 lines
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
#15 lore raw
In reply to Chuck Lever
Quoted 1 line · Chuck Lever

My preferred solution would involve working around the sync(2) call instead (ie fixing sm-notify so we don't need it, or somehow doing it in the background so it doesn't hold up the boot-up process).

Doing it in the background doesn't really help, as it's the huge blob of disk writes that it generates that's the problem as much as the elapsed time. Boot time is more likely to be I/O bound than CPU bound.

BTW, I have written an article about my efforts to speed up booting which will should appear at debian-administration.org later today.

Cheers, Phil.

#16 lore raw
In reply to J. Bruce Fields
Quoted 21 lines · J. Bruce Fields
Quoted 2 lines · Chuck Lever

I assume sync() is required because this logic performs a rename as well as a simple write?

I think an fsync() on the containing directory (together with an fsync() of the file itself) would do the job if you wanted to avoid the globaly sync(). I don't think ext3 is capable of doing anything finer-grained than a whole-filesystem sync, though, so this doesn't help many people in practice right now.

In any case, the rename adds an extra level of safety by ensuring the nsm state is updated atomically, so we shouldn't get rid of it.

Quoted 7 lines
Quoted 2 lines

Anyway, I think the nsm state updating shouldn't matter if you don't even have any peers to notify.

It probably does matter.

When a system is initially installed, it likely does not have a state file in /var/lib/nfs. This may be harmless if it's not present; rpc.statd probably does the right thing in this case.

Quoted 6 lines

The "right thing" in that case would be, I guess, to create a state file with "0" in it. It doesn't do that. So this patch *does* break stuff. Oops!

So should we revert it and do something else, or patch statd to create a new state file if necessary?

I have to agree with Chuck, that managing the state file from one place is desirable... Although does it make sense to move that management into a init script? Maybe insuring the state is different before any daemons are started? Would we still need the sync()?

Quoted 8 lines
Quoted 5 lines

However, the rest of the logic in nsm_get_state() is needed to bump the system's state value properly after every reboot. It may be inconsequential if there were no mounts or no NFS clients during the last reboot, but this is subtle. I wouldn't bet on it.

If the state is only every communicated to hosts by notifications, then if we're not notifying, the update of the state can't matter.

I had the same notion... If there are no clients to notify, why would any clients care if our state changed?? With that said.... I do have a sinking feeling that this patch will come back to bite us... It was just too easy of a fix... :-)

steved.

A message that is not in the archive (sent off-list or not yet received): [email protected]

#18 lore raw
Quoted 39 lines · Steve Dickson
Quoted 21 lines · J. Bruce Fields
Quoted 2 lines · Chuck Lever

I assume sync() is required because this logic performs a rename as well as a simple write?

I think an fsync() on the containing directory (together with an fsync() of the file itself) would do the job if you wanted to avoid the globaly sync(). I don't think ext3 is capable of doing anything finer-grained than a whole-filesystem sync, though, so this doesn't help many people in practice right now.

In any case, the rename adds an extra level of safety by ensuring the nsm state is updated atomically, so we shouldn't get rid of it.

Quoted 7 lines
Quoted 2 lines

Anyway, I think the nsm state updating shouldn't matter if you don't even have any peers to notify.

It probably does matter.

When a system is initially installed, it likely does not have a state file in /var/lib/nfs. This may be harmless if it's not present; rpc.statd probably does the right thing in this case.

Quoted 6 lines

The "right thing" in that case would be, I guess, to create a state file with "0" in it. It doesn't do that. So this patch *does* break stuff. Oops!

So should we revert it and do something else, or patch statd to create a new state file if necessary?

I have to agree with Chuck, that managing the state file from one place is desirable... Although does it make sense to move that management into a init script? Maybe insuring the state is different before any daemons are started? Would we still need the sync()?

Quoted 6 lines
Quoted 5 lines

However, the rest of the logic in nsm_get_state() is needed to bump the system's state value properly after every reboot. It may be inconsequential if there were no mounts or no NFS clients during the last reboot, but this is subtle. I wouldn't bet on it.

Quoted 6 lines
Quoted 2 lines

If the state is only every communicated to hosts by notifications, then if we're not notifying, the update of the state can't matter.

I had the same notion... If there are no clients to notify, why would any clients care if our state changed?? With that said.... I do have a sinking feeling that this patch will come back to bite us... It was just too easy of a fix... :-)

Any progress on this? I don't think we can release in the current state, since as far as I can tell that means on a new system, unless the install scripts create /var/lib/nfs/state, neither sm-notify nor statd ever writes to /proc/sys/fs/nfs/nsm_local_state, and (without testing) it looks to me like that means lockd defaults to a state of 0, which is nonsense?

One possible compromise that I think would be correct might be to skip the sync in the case where there are no peers to notify, instead of exiting early in that case. As long as a sync happens at some point before we grant any locks (which statd can guarantee), I think that will work....

--b.

#19 lore raw
Quoted 7 lines · On Thursday December 4, [email protected] wrote:

Any progress on this? I don't think we can release in the current state, since as far as I can tell that means on a new system, unless the install scripts create /var/lib/nfs/state, neither sm-notify nor statd ever writes to /proc/sys/fs/nfs/nsm_local_state, and (without testing) it looks to me like that means lockd defaults to a state of 0, which is nonsense?

Why is '0' nonsense?
The only real requirement on 'state' is that it changes when the host reboots while some peer is monitoring it. Even if it got reset to zero every time the sm and sm.bak became empty it would still work just fine. If we reboot and find that both sm and sm.bak are empty there is really no point in changing 'state'.

I think it would still be valuable to replace the 'sync' with two 'fsync's, one of the file, one on the directory.
This may not be a win on ext3 today (I'm not 100% certain about that) but there are other filesystems and more seem to be coming.

NeilBrown

A message that is not in the archive (sent off-list or not yet received): 18744.41310.635618.148281-wvvUuzkyo1EYVZTmpyfIwg@public.gmane.org

#21 lore raw
Quoted 10 lines · Neil Brown
Quoted 7 lines · On Thursday December 4, [email protected] wrote:

Any progress on this? I don't think we can release in the current state, since as far as I can tell that means on a new system, unless the install scripts create /var/lib/nfs/state, neither sm-notify nor statd ever writes to /proc/sys/fs/nfs/nsm_local_state, and (without testing) it looks to me like that means lockd defaults to a state of 0, which is nonsense?

Why is '0' nonsense?

Even numbers are supposed to mean that a host is down, odd that it's up. In practice noone ever uses that fact--they only ever advertise odd numbers. So we might get away with using an even number. But then again a peer would be in its rights to check the parity and complain, and some may well do that.

Quoted 4 lines

The only real requirement on 'state' is that it changes when the host reboots while some peer is monitoring it. Even if it got reset to zero every time the sm and sm.bak became empty it would still work just fine.

Hm, I don't know; it might be that an inconveniently timed network partition combined with nsm states that repeat themselves could prevent a client from knowing about some reboot that it should have known about.

It's safer to keep a counter and ensure that the state never repeats (well, anyway, not until it overflows after a few billion reboots).

Quoted 2 lines

If we reboot and find that both sm and sm.bak are empty there is really no point in changing 'state'.

I think I've convinced myself of that, yes. For the purposes of locking, a reboot which didn't require notifying any client may as well have not been a reboot, since no state was lost.

But we should at least make sure that the state is properly initialized and nondecreasing.

Quoted 2 lines

I think it would still be valuable to replace the 'sync' with two 'fsync's, one of the file, one on the directory.

Sure, may as well.--b.

Quoted 2 lines

This may not be a win on ext3 today (I'm not 100% certain about that) but there are other filesystems and more seem to be coming.

#22 lore raw
Quoted 6 lines · J. Bruce Fields
Quoted 3 lines

I think it would still be valuable to replace the 'sync' with two 'fsync's, one of the file, one on the directory.

Sure, may as well.--b.

Something similar to this:

utils/statd/sm-notify.c renamed from utils/statd/sm-notify.c.orig +3 -7
diff -up nfs-utils/utils/statd/sm-notify.c.orig nfs-utils/utils/statd/sm-notify.c
@@ -211,12 +211,6 @@ usage:		fprintf(stderr,
 	backup_hosts(_SM_DIR_PATH, _SM_BAK_PATH);
 	get_hosts(_SM_BAK_PATH);
 
-	/* If there are not hosts to notify, just exit */
-	if (!hosts) {
-		nsm_log(LOG_DEBUG, "No hosts to notify; exiting");
-		return 0;
-	}
-
 	/* Get and update the NSM state. This will call sync() */
 	nsm_state = nsm_get_state(opt_update_state);
 	set_kernel_nsm_state(nsm_state);
@@ -694,6 +688,7 @@ nsm_get_state(int update)
 {
 	char		newfile[PATH_MAX];
 	int		fd, state;
+	DIR 	*fp;
 
 	if ((fd = open(_SM_STATE_PATH, O_RDONLY)) < 0) {
 		if (!opt_quiet) {
@@ -730,13 +725,16 @@ nsm_get_state(int update)
 				"Failed to write state to %s", newfile);
 			exit(1);
 		}
+		fsync(fd);
 		close(fd);
+		fp = opendir(_SM_STATE_PATH);
 		if (rename(newfile, _SM_STATE_PATH) < 0) {
 			nsm_log(LOG_ERR,
 				"Cannot create %s: %m", _SM_STATE_PATH);
 			exit(1);
 		}
-		sync();
 

If so, I'll make it happen...

steved.

A message that is not in the archive (sent off-list or not yet received): [email protected]

#24 lore raw
Quoted 13 lines · Steve Dickson
Quoted 6 lines · J. Bruce Fields
Quoted 3 lines

I think it would still be valuable to replace the 'sync' with two 'fsync's, one of the file, one on the directory.

Sure, may as well.--b.

Something similar to this:

diff -up nfs-utils/utils/statd/sm-notify.c.orig nfs-utils/utils/statd/sm-notify.c

utils/statd/sm-notify.c usage: fprintf(stderr, · line 218

Quoted 8 lines
utils/statd/sm-notify.c renamed from utils/statd/sm-notify.c.orig +0 -5
@@ -211,12 +211,6 @@ usage:		fprintf(stderr,
 	backup_hosts(_SM_DIR_PATH, _SM_BAK_PATH);
 	get_hosts(_SM_BAK_PATH);
 
-	/* If there are not hosts to notify, just exit */
-	if (!hosts) {
-		nsm_log(LOG_DEBUG, "No hosts to notify; exiting");
-		return 0;
-	}

This was still a huge boot-time win in the common case, so now that we've committed to it I'd rather not regress. Let's just skip the sync()s/fsncy()s in the !hosts case--that looks to me like the simplest correct solution for now.

Quoted 13 lines
utils/statd/sm-notify.c renamed from utils/statd/sm-notify.c.orig +1 -1
-
 	/* Get and update the NSM state. This will call sync() */
 	nsm_state = nsm_get_state(opt_update_state);
 	set_kernel_nsm_state(nsm_state);
@@ -694,6 +688,7 @@ nsm_get_state(int update)
 {
 	char		newfile[PATH_MAX];
 	int		fd, state;
+	DIR 	*fp;
 
 	if ((fd = open(_SM_STATE_PATH, O_RDONLY)) < 0) {
 		if (!opt_quiet) {

utils/statd/sm-notify.c nsm_get_state(int update) · line 730

Quoted 6 lines
utils/statd/sm-notify.c renamed from utils/statd/sm-notify.c.orig +2 -0
@@ -730,13 +725,16 @@ nsm_get_state(int update)
 				"Failed to write state to %s", newfile);
 			exit(1);
 		}
+		fsync(fd);
 		close(fd);
+		fp = opendir(_SM_STATE_PATH);

Also, I think you meant to:

		fsync(fp);
		close(fp);

?

--b.

Quoted 10 lines
utils/statd/sm-notify.c renamed from utils/statd/sm-notify.c.orig +0 -1
 		if (rename(newfile, _SM_STATE_PATH) < 0) {
 			nsm_log(LOG_ERR,
 				"Cannot create %s: %m", _SM_STATE_PATH);
 			exit(1);
 		}
-		sync();
 

If so, I'll make it happen...

steved.

#25 lore raw
Quoted 17 lines · bfields
Quoted 16 lines · Steve Dickson
Quoted 6 lines · J. Bruce Fields
Quoted 3 lines

I think it would still be valuable to replace the 'sync' with two 'fsync's, one of the file, one on the directory.

Sure, may as well.--b.

Something similar to this:

diff -up nfs-utils/utils/statd/sm-notify.c.orig nfs-utils/utils/statd/sm-notify.c

utils/statd/sm-notify.c renamed from utils/statd/sm-notify.c.orig +0 -0
@@ -211,12 +211,6 @@ usage:		fprintf(stderr,
 	backup_hosts(_SM_DIR_PATH, _SM_BAK_PATH);
 	get_hosts(_SM_BAK_PATH);
 
Quoted 10 lines

utils/statd/sm-notify.c usage: fprintf(stderr, · line 218

Quoted 5 lines
utils/statd/sm-notify.c renamed from utils/statd/sm-notify.c.orig +0 -5
-	/* If there are not hosts to notify, just exit */
-	if (!hosts) {
-		nsm_log(LOG_DEBUG, "No hosts to notify; exiting");
-		return 0;
-	}

This was still a huge boot-time win in the common case, so now that we've committed to it I'd rather not regress. Let's just skip the sync()s/fsncy()s in the !hosts case--that looks to me like the simplest correct solution for now.

My argument for correctness: if we don't sync in that case, then on reboot the rename that updates the state will either have happened or (if a crash comes too soon) not.

It is OK for that update to not happen as long as we're assured it happens before the first lock request is made or replied to, or the first monitor request completes, as, in the absence of any notifies, those are the only points at which the new state will be exposed to the outside world.

The first lock request will also require an upcall to statd. So we're OK as long as any monitor requests (from either the local kernel or remote peers) do a sync.

And statd should be doing a sync before responding to any monitor request. As long as the SM_DIR is on the same filesystem as the state file, that would do the job.... But now that I look, I see statd is using an open with O_SYNC to ensure the new statd record hits stable storage. Which we can't count on being enough.

How about adding an explicit fsync() of the state file (and parent directory) to statd's first succesful creation of a statd record, together with a comment explaining this? So around about line 194 in utils/statd/monitor.c:sm_mon_1_svc()?

In fact, we could delete the sync entirely and do the same before the first notification, and then we wouldn't have to wait for the sync in the case host records are present either.... (statd would, but perhaps we could still get other work done in the mean time).

(Am I missing something?

--b.

#26 lore raw
Quoted 56 lines · J. Bruce Fields
Quoted 19 lines · bfields
Quoted 18 lines · Steve Dickson
Quoted 6 lines · J. Bruce Fields
Quoted 3 lines

I think it would still be valuable to replace the 'sync' with two 'fsync's, one of the file, one on the directory.

Sure, may as well.--b.

Something similar to this:

diff -up nfs-utils/utils/statd/sm-notify.c.orig nfs-utils/utils/statd/sm-notify.c
--- nfs-utils/utils/statd/sm-notify.c.orig 2008-11-17
15:06:13.000000000 -0500
+++ nfs-utils/utils/statd/sm-notify.c 2008-12-05
08:21:52.000000000 -0500

utils/statd/sm-notify.c renamed from utils/statd/sm-notify.c.orig +0 -0
@@ -211,12 +211,6 @@ usage:		fprintf(stderr,
 	backup_hosts(_SM_DIR_PATH, _SM_BAK_PATH);
 	get_hosts(_SM_BAK_PATH);
 
Quoted 10 lines

utils/statd/sm-notify.c usage: fprintf(stderr, · line 218

Quoted 5 lines
utils/statd/sm-notify.c renamed from utils/statd/sm-notify.c.orig +0 -5
-	/* If there are not hosts to notify, just exit */
-	if (!hosts) {
-		nsm_log(LOG_DEBUG, "No hosts to notify; exiting");
-		return 0;
-	}

This was still a huge boot-time win in the common case, so now that we've committed to it I'd rather not regress. Let's just skip the sync()s/fsncy()s in the !hosts case--that looks to me like the simplest correct solution for now.

My argument for correctness: if we don't sync in that case, then on reboot the rename that updates the state will either have happened or (if a crash comes too soon) not.

It is OK for that update to not happen as long as we're assured it happens before the first lock request is made or replied to, or the first monitor request completes, as, in the absence of any notifies, those are the only points at which the new state will be exposed to the outside world.

The first lock request will also require an upcall to statd. So we're OK as long as any monitor requests (from either the local kernel or remote peers) do a sync.

And statd should be doing a sync before responding to any monitor request. As long as the SM_DIR is on the same filesystem as the state file, that would do the job.... But now that I look, I see statd is using an open with O_SYNC to ensure the new statd record hits stable storage. Which we can't count on being enough.

How about adding an explicit fsync() of the state file (and parent directory) to statd's first succesful creation of a statd record, together with a comment explaining this? So around about line 194 in utils/statd/monitor.c:sm_mon_1_svc()?

Quoted 6 lines

In fact, we could delete the sync entirely and do the same before the first notification, and then we wouldn't have to wait for the sync in the case host records are present either.... (statd would, but perhaps we could still get other work done in the mean time).

(Am I missing something?)

This all might work, but I think we're adding a lot of complexity as a workaround. Someone should fix the real problem, which is the implementation of sync().

That would probably have other benefits too.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

#27 lore raw
Quoted 12 lines · Chuck Lever
Quoted 11 lines · J. Bruce Fields

How about adding an explicit fsync() of the state file (and parent directory) to statd's first succesful creation of a statd record, together with a comment explaining this? So around about line 194 in utils/statd/monitor.c:sm_mon_1_svc()?

In fact, we could delete the sync entirely and do the same before the first notification, and then we wouldn't have to wait for the sync in the case host records are present either.... (statd would, but perhaps
we could still get other work done in the mean time).

Quoted 4 lines
Quoted 1 line

(Am I missing something?)

This all might work, but I think we're adding a lot of complexity as a workaround.

I think you mean the justification is too subtle--the code itself (just a couple syncs or fsyncs) is pretty simple.

Quoted 2 lines

Someone should fix the real problem, which is the
implementation of sync().

I think you mean of fsync(). My understanding of past discussions on the issue is that it's not really fixable on ext3, at least. So default setups will have this problem for a while.

--b.

Quoted 1 line

That would probably have other benefits too.

#28 lore raw

Hi Bruce-

Quoted 12 lines · J. Bruce Fields
Quoted 11 lines · Chuck Lever
Quoted 10 lines · J. Bruce Fields

How about adding an explicit fsync() of the state file (and parent directory) to statd's first succesful creation of a statd record, together with a comment explaining this? So around about line 194 in utils/statd/monitor.c:sm_mon_1_svc()?

In fact, we could delete the sync entirely and do the same before the first notification, and then we wouldn't have to wait for the sync in the case host records are present either.... (statd would, but perhaps we could still get other work done in the mean time).

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

(Am I missing something?)

This all might work, but I think we're adding a lot of complexity as a workaround.

I think you mean the justification is too subtle--the code itself (just a couple syncs or fsyncs) is pretty simple.

Let me state it another way. Your suggested redesign makes assumptions about the order in which these operations are performed. Can the code provide any guarantee that this ordering is always true? This may have been feasible when rpc.statd did notification too, but these are now two separate programs. Can you think of a way of implementing this where the ordering dependencies are coded instead of commented?

It would be significantly less fragile (ie immune to ordering problems and long-term changes to system start-up and nfs-utils code) and simpler to understand if sm-notify does whatever syncing it needs, and rpc.statd does whatever syncing it needs.

All I'm saying is this scheme may be easy to break by accident, and any future problem here is probably not going to show up until someone really needs this to work right.

Quoted 6 lines
Quoted 2 lines

Someone should fix the real problem, which is the
implementation of sync().

I think you mean of fsync(). My understanding of past discussions on the issue is that it's not really fixable on ext3, at least. So default setups will have this problem for a while.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

#29 lore raw
Quoted 28 lines · Chuck Lever

Hi Bruce-

Quoted 17 lines · J. Bruce Fields
Quoted 16 lines · Chuck Lever
Quoted 14 lines · J. Bruce Fields

How about adding an explicit fsync() of the state file (and parent directory) to statd's first succesful creation of a statd record, together with a comment explaining this? So around about line 194 in
utils/statd/monitor.c:sm_mon_1_svc()?

In fact, we could delete the sync entirely and do the same before the
first notification, and then we wouldn't have to wait for the sync in
the case host records are present either.... (statd would, but perhaps we could still get other work done in the mean time).

(Am I missing something?)

Quoted 7 lines
Quoted 3 lines

This all might work, but I think we're adding a lot of complexity as a
workaround.

I think you mean the justification is too subtle--the code itself (just
a couple syncs or fsyncs) is pretty simple.

Quoted 6 lines

Let me state it another way. Your suggested redesign makes assumptions about the order in which these operations are performed. Can the code provide any guarantee that this ordering is always true? This may have been feasible when rpc.statd did notification too, but these are now two separate programs. Can you think of a way of implementing this where the ordering dependencies are coded instead of commented?

The existing code requires that we update the on-disk nsm state number before doing notification or accepting statd calls. That's the only ordering I'd require.

Maybe I should try to write a patch.

--b.

Quoted 20 lines

It would be significantly less fragile (ie immune to ordering problems and long-term changes to system start-up and nfs-utils code) and simpler to understand if sm-notify does whatever syncing it needs, and rpc.statd does whatever syncing it needs.

All I'm saying is this scheme may be easy to break by accident, and any future problem here is probably not going to show up until someone really needs this to work right.

Quoted 6 lines
Quoted 2 lines

Someone should fix the real problem, which is the
implementation of sync().

I think you mean of fsync(). My understanding of past discussions on the issue is that it's not really fixable on ext3, at least. So default setups will have this problem for a while.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

#30 lore raw
In reply to J. Bruce Fields
Quoted 30 lines · J. Bruce Fields
Quoted 15 lines · bfields
Quoted 14 lines · Steve Dickson
Quoted 4 lines · J. Bruce Fields
Quoted 2 lines

I think it would still be valuable to replace the 'sync' with two 'fsync's, one of the file, one on the directory.

Sure, may as well.--b.

Something similar to this:

diff -up nfs-utils/utils/statd/sm-notify.c.orig nfs-utils/utils/statd/sm-notify.c

utils/statd/sm-notify.c renamed from utils/statd/sm-notify.c.orig +0 -0
@@ -211,12 +211,6 @@ usage:		fprintf(stderr,
 	backup_hosts(_SM_DIR_PATH, _SM_BAK_PATH);
 	get_hosts(_SM_BAK_PATH);
 
Quoted 9 lines

utils/statd/sm-notify.c usage: fprintf(stderr, · line 218

Quoted 5 lines
utils/statd/sm-notify.c renamed from utils/statd/sm-notify.c.orig +0 -5
-	/* If there are not hosts to notify, just exit */
-	if (!hosts) {
-		nsm_log(LOG_DEBUG, "No hosts to notify; exiting");
-		return 0;
-	}

This was still a huge boot-time win in the common case, so now that we've committed to it I'd rather not regress. Let's just skip the sync()s/fsncy()s in the !hosts case--that looks to me like the simplest correct solution for now.

My argument for correctness: if we don't sync in that case, then on reboot the rename that updates the state will either have happened or (if a crash comes too soon) not.

Quoted 5 lines

It is OK for that update to not happen as long as we're assured it happens before the first lock request is made or replied to, or the first monitor request completes, as, in the absence of any notifies, those are the only points at which the new state will be exposed to the outside world.

Doesn't the sync() have to happen before the file is first read by stated. Meaning before statd:main() calls load_state_number()?

Quoted 11 lines

The first lock request will also require an upcall to statd. So we're OK as long as any monitor requests (from either the local kernel or remote peers) do a sync.

And statd should be doing a sync before responding to any monitor request. As long as the SM_DIR is on the same filesystem as the state file, that would do the job.... But now that I look, I see statd is using an open with O_SYNC to ensure the new statd record hits stable storage. Which we can't count on being enough.

Quoted 4 lines

How about adding an explicit fsync() of the state file (and parent directory) to statd's first succesful creation of a statd record, together with a comment explaining this? So around about line 194 in utils/statd/monitor.c:sm_mon_1_svc()?

If we do the sync()/fsync() here we will also have to update MY_STATE since that's what is the number used in the RPCs. But also I think doing the sync this late be a bit waste since there is real good chance the rename has already been sync-ed out by previous sync() during boot up... or am I missing something...

steved.

A message that is not in the archive (sent off-list or not yet received): [email protected]

#32 lore raw
Quoted 31 lines · Steve Dickson
Quoted 30 lines · J. Bruce Fields
Quoted 15 lines · bfields
Quoted 14 lines · Steve Dickson
Quoted 4 lines · J. Bruce Fields
Quoted 2 lines

I think it would still be valuable to replace the 'sync' with two 'fsync's, one of the file, one on the directory.

Sure, may as well.--b.

Something similar to this:

diff -up nfs-utils/utils/statd/sm-notify.c.orig nfs-utils/utils/statd/sm-notify.c

utils/statd/sm-notify.c renamed from utils/statd/sm-notify.c.orig +0 -0
@@ -211,12 +211,6 @@ usage:		fprintf(stderr,
 	backup_hosts(_SM_DIR_PATH, _SM_BAK_PATH);
 	get_hosts(_SM_BAK_PATH);
 
Quoted 9 lines

utils/statd/sm-notify.c usage: fprintf(stderr, · line 218

Quoted 5 lines
utils/statd/sm-notify.c renamed from utils/statd/sm-notify.c.orig +0 -5
-	/* If there are not hosts to notify, just exit */
-	if (!hosts) {
-		nsm_log(LOG_DEBUG, "No hosts to notify; exiting");
-		return 0;
-	}

This was still a huge boot-time win in the common case, so now that we've committed to it I'd rather not regress. Let's just skip the sync()s/fsncy()s in the !hosts case--that looks to me like the simplest correct solution for now.

My argument for correctness: if we don't sync in that case, then on reboot the rename that updates the state will either have happened or (if a crash comes too soon) not.

Quoted 7 lines
Quoted 5 lines

It is OK for that update to not happen as long as we're assured it happens before the first lock request is made or replied to, or the first monitor request completes, as, in the absence of any notifies, those are the only points at which the new state will be exposed to the outside world.

Doesn't the sync() have to happen before the file is first read by stated. Meaning before statd:main() calls load_state_number()?

I can't see why. sync() of course can't have any effect on a subsequent read of the file.

Quoted 10 lines
Quoted 10 lines

The first lock request will also require an upcall to statd. So we're OK as long as any monitor requests (from either the local kernel or remote peers) do a sync.

And statd should be doing a sync before responding to any monitor request. As long as the SM_DIR is on the same filesystem as the state file, that would do the job.... But now that I look, I see statd is using an open with O_SYNC to ensure the new statd record hits stable storage. Which we can't count on being enough.

Quoted 6 lines
Quoted 4 lines

How about adding an explicit fsync() of the state file (and parent directory) to statd's first succesful creation of a statd record, together with a comment explaining this? So around about line 194 in utils/statd/monitor.c:sm_mon_1_svc()?

If we do the sync()/fsync() here we will also have to update MY_STATE since that's what is the number used in the RPCs.

I don't believe that's true.

Quoted 4 lines

But also I think
doing the sync this late be a bit waste since there is real good chance the rename has already been sync-ed out by previous sync() during boot up... or am I missing something...

The whole boot gets held up waiting for this one sync to complete; those later sync's mainly only delay bringing up nfs.

--b.

#33 lore raw
In reply to J. Bruce Fields
Quoted 15 lines · J. Bruce Fields
Quoted 14 lines · Steve Dickson
Quoted 4 lines · J. Bruce Fields
Quoted 2 lines

I think it would still be valuable to replace the 'sync' with two 'fsync's, one of the file, one on the directory.

Sure, may as well.--b.

Something similar to this:

diff -up nfs-utils/utils/statd/sm-notify.c.orig nfs-utils/utils/statd/sm-notify.c

utils/statd/sm-notify.c renamed from utils/statd/sm-notify.c.orig +0 -0
@@ -211,12 +211,6 @@ usage:		fprintf(stderr,
 	backup_hosts(_SM_DIR_PATH, _SM_BAK_PATH);
 	get_hosts(_SM_BAK_PATH);
 
Quoted 8 lines

utils/statd/sm-notify.c usage: fprintf(stderr, · line 218

Quoted 5 lines
utils/statd/sm-notify.c renamed from utils/statd/sm-notify.c.orig +0 -5
-	/* If there are not hosts to notify, just exit */
-	if (!hosts) {
-		nsm_log(LOG_DEBUG, "No hosts to notify; exiting");
-		return 0;
-	}

This was still a huge boot-time win in the common case, so now that we've committed to it I'd rather not regress.

I thought the idea was the state had to be updated even when there are no hosts...

Quoted 10 lines

utils/statd/sm-notify.c nsm_get_state(int update) · line 730

Quoted 7 lines
utils/statd/sm-notify.c renamed from utils/statd/sm-notify.c.orig +2 -0
@@ -730,13 +725,16 @@ nsm_get_state(int update)
 				"Failed to write state to %s", newfile);
 			exit(1);
 		}
+		fsync(fd);
 		close(fd);
+		fp = opendir(_SM_STATE_PATH);

Also, I think you meant to:

Quoted 2 lines
		fsync(fp);
		close(fp);

No... I don't think so... the fd is an open() of the new file and the fp is a DIR stream... But there was a cut/paste error... The fsync() of the DIR stream was missing... here is is again...

@@ -694,6 +688,7 @@ nsm_get_state(int update)
{
char newfile[PATH_MAX];
int fd, state;
+ DIR *fp;

if ((fd = open(_SM_STATE_PATH, O_RDONLY)) < 0) {
if (!opt_quiet) {
@@ -730,13 +725,18 @@ nsm_get_state(int update)
"Failed to write state to %s", newfile);
exit(1);
}
+ fsync(fd);
close(fd);
+ fp = opendir(_SM_STATE_PATH);
if (rename(newfile, _SM_STATE_PATH) < 0) {
nsm_log(LOG_ERR,
"Cannot create %s: %m", _SM_STATE_PATH);
exit(1);
}
- sync();
+ if (fp != NULL) {
+ fsync(dirfd(fp));
+ closedir(fp);
+ }
}

return state;

A message that is not in the archive (sent off-list or not yet received): [email protected]

#35 lore raw
Quoted 24 lines · Steve Dickson
Quoted 15 lines · J. Bruce Fields
Quoted 14 lines · Steve Dickson
Quoted 4 lines · J. Bruce Fields
Quoted 2 lines

I think it would still be valuable to replace the 'sync' with two 'fsync's, one of the file, one on the directory.

Sure, may as well.--b.

Something similar to this:

diff -up nfs-utils/utils/statd/sm-notify.c.orig nfs-utils/utils/statd/sm-notify.c

utils/statd/sm-notify.c renamed from utils/statd/sm-notify.c.orig +0 -0
@@ -211,12 +211,6 @@ usage:		fprintf(stderr,
 	backup_hosts(_SM_DIR_PATH, _SM_BAK_PATH);
 	get_hosts(_SM_BAK_PATH);
 
Quoted 8 lines

utils/statd/sm-notify.c usage: fprintf(stderr, · line 218

Quoted 5 lines
utils/statd/sm-notify.c renamed from utils/statd/sm-notify.c.orig +0 -5
-	/* If there are not hosts to notify, just exit */
-	if (!hosts) {
-		nsm_log(LOG_DEBUG, "No hosts to notify; exiting");
-		return 0;
-	}

This was still a huge boot-time win in the common case, so now that we've committed to it I'd rather not regress.

Quoted 2 lines

I thought the idea was the state had to be updated even when there are no hosts...

I would prefer to increment the state file even in the absence of hosts (it might be useful e.g. to a client that missed a previous grace period due to a network problem to know that the current boot instance is later than the one it needed to reclaim in). But I don't believe it's necessary.

The more immediate problem is that the state file no longer gets created
at all on a new system. (Unless the package install scripts do it. I
haven't checked.)

So, we could either make sure the state file gets created some other way, or delay the syncs as described before.

Quoted 10 lines
Quoted 10 lines

utils/statd/sm-notify.c nsm_get_state(int update) · line 730

Quoted 7 lines
utils/statd/sm-notify.c renamed from utils/statd/sm-notify.c.orig +2 -0
@@ -730,13 +725,16 @@ nsm_get_state(int update)
 				"Failed to write state to %s", newfile);
 			exit(1);
 		}
+		fsync(fd);
 		close(fd);
+		fp = opendir(_SM_STATE_PATH);

Also, I think you meant to:

Quoted 5 lines
Quoted 2 lines
		fsync(fp);
		close(fp);

No... I don't think so... the fd is an open() of the new file and the fp is a DIR stream... But there was a cut/paste error... The fsync() of the DIR stream was missing... here is is again...

OK.

Quoted 11 lines
utils/statd/sm-notify.c renamed from utils/statd/sm-notify.c.orig +1 -0
@@ -694,6 +688,7 @@ nsm_get_state(int update)
 {
 	char		newfile[PATH_MAX];
 	int		fd, state;
+	DIR 	*fp;
 
 	if ((fd = open(_SM_STATE_PATH, O_RDONLY)) < 0) {
 		if (!opt_quiet) {
@@ -730,13 +725,18 @@ nsm_get_state(int update)
 				"Failed to write state to %s", newfile);
 			exit(1);

utils/statd/sm-notify.c nsm_get_state(int update) · line 739

Quoted 10 lines
utils/statd/sm-notify.c renamed from utils/statd/sm-notify.c.orig +2 -1
 		}
+		fsync(fd);
 		close(fd);
+		fp = opendir(_SM_STATE_PATH);
 		if (rename(newfile, _SM_STATE_PATH) < 0) {
 			nsm_log(LOG_ERR,
 				"Cannot create %s: %m", _SM_STATE_PATH);
 			exit(1);
 		}
-		sync();

May as well move that opendir down here where it's used, though.

Review nsm_get_state(int update) · line 742

Quoted 7 lines
(quoted diff) +4 -0
+		if (fp != NULL) {
+			fsync(dirfd(fp));
+			closedir(fp);
+		}
 	}
 
 	return state;

--b.

#36 lore raw
In reply to J. Bruce Fields
Quoted 14 lines · J. Bruce Fields
Quoted 13 lines · Phil Endecott
Quoted 12 lines · J. Bruce Fields
Quoted 6 lines · Phil Endecott

Dear Experts,

sm-notify was taking a long time while my laptop booted. This was odd because I use NFS only rarely - via autofs - on that machine, and sm-notify actually has no-one to notify most of the time. So I have patched it as follows. Is this a legitimate thing to do?

It looks like your patch was committed to nfs-utils a couple weeks ago:
see c8d18e26d2a53d9036a32c2dafebccaf4ce1634d from

git://linux-nfs.org/nfs-utils

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

--b.

How curious. I guess someone saw my Debian bug report. No mention of it on this list as far as I can see though.

I presume from this that it is considered a safe thing to do.

It looks right to me. Hopefully somebody actually has tested this on a client that holds locks when it reboots?

I did... things worked as expected....

steved.