server's statd and lockd will not sync after its nfslock restart
Subsystems (2)
Hi,
When testing the NLM at the latest kernel(2.6.32), i find a bug. When a client hold locks, after server restart its nfslock service, server's statd will not synchronize with lockd.
If server restart nfslock twice or more, client's lock will be lost.
Test process:
Step1: client open nfs file.
Step2: client using fcntl to get lock.
Step3: server restart it's nfslock service.
After step3, server's lockd records client holding locks, but statd's /var/lib/nfs/statd/sm/ directory is empty. It means statd and lockd are not sync. If server restart it's nfslock again, client's locks will be lost.
The Primary Reason:
At step3, when client's reclaimed lock request is sent to server,
client's host(the host struct) is reused but not be re-monitored at server's lockd. After that, statd and lockd are not sync.
Question:
In my opinion, if lockd is allowed reuseing the client's host, it should send a SM_MON to statd when reuse. If not allowed, the client's host should be destroyed immediately.
What should lockd to do? Reuse ? Destroy ? Or some other action?
thanks,
Mi Jinlong
Quoted 7 lines · Mi Jinlong
Hi,
When testing the NLM at the latest kernel(2.6.32), i find a bug. When a client hold locks, after server restart its nfslock service, server's statd will not synchronize with lockd.
If server restart nfslock twice or more, client's lock will be lost.
Quoted 5 lines
Test process:
Step1: client open nfs file.
Step2: client using fcntl to get lock.
Step3: server restart it's nfslock service.
I don't know what you mean; what did you actually do in step 3?
--b.
Quoted 24 lines
After step3, server's lockd records client holding locks, but statd's
/var/lib/nfs/statd/sm/directory is empty. It means statd and lockd are not sync. If server restart it's nfslock again, client's locks will be lost.The Primary Reason:
At step3, when client's reclaimed lock request is sent to server,
client's host(the host struct) is reused but not be re-monitored at server's lockd. After that, statd and lockd are not sync.Question:
In my opinion, if lockd is allowed reuseing the client's host, it should send a
SM_MONto statd when reuse. If not allowed, the client's host should be destroyed immediately.What should lockd to do? Reuse ? Destroy ? Or some other action?
thanks,
Mi Jinlong
Quoted 7 lines · Mi Jinlong
Hi,
When testing the NLM at the latest kernel(2.6.32), i find a bug. When a client hold locks, after server restart its nfslock service, server's statd will not synchronize with lockd.
If server restart nfslock twice or more, client's lock will be lost.
Quoted 5 lines
Test process:
Step1: client open nfs file.
Step2: client using fcntl to get lock.
Step3: server restart it's nfslock service.
I'll assume here that you mean the equivalent of "service nfslock restart". This restarts statd and possibly runs sm-notify, but it has no effect on lockd.
Again, this test seems artificial to me. Is there a real world use case where someone would deliberately restart statd while an NFS server is serving files? I pose this question because I've worked on statd only for a year or so, and I am quite likely ignorant of all the ways it can be deployed.
Quoted 9 lines
After step3, server's lockd records client holding locks, but statd's
/var/lib/nfs/statd/sm/directory is empty. It means statd and lockd are not sync. If server restart it's nfslock again, client's locks will be lost.The Primary Reason:
At step3, when client's reclaimed lock request is sent to server,
client's host(the host struct) is reused but not be re-monitored at server's lockd. After that, statd and lockd are not sync.
The kernel squashes SM_MON upcalls for hosts that it already believes are monitored. This is a scalability feature.
Quoted 7 lines
Question:
In my opinion, if lockd is allowed reuseing the client's host, it should send a
SM_MONto statd when reuse. If not allowed, the client's host should be destroyed immediately.What should lockd to do? Reuse ? Destroy ? Or some other action?
I don't immediately see why lockd should change it's behavior. Perhaps statd/sm-notify were incorrect to delete the monitor list when you restarted the nfslock service?
Can you show exactly how statd's state (ie it's on-disk monitor list in /var/lib/nfs/statd/sm) changed across the restart? Did sm-notify run when you restarted statd? If so, why didn't the sm-notify pid file stop it?
Signature · 2 lines
Chuck Lever chuck[dot]lever[at]oracle[dot]com
J. Bruce Fields :
Quoted 8 lines · Mi Jinlong
Quoted 7 lines
Hi,
When testing the NLM at the latest kernel(2.6.32), i find a bug. When a client hold locks, after server restart its nfslock service, server's statd will not synchronize with lockd.
If server restart nfslock twice or more, client's lock will be lost.
Quoted 7 lines
Quoted 5 lines
Test process:
Step1: client open nfs file.
Step2: client using fcntl to get lock.
Step3: server restart it's nfslock service.I don't know what you mean; what did you actually do in step 3?
I use command "service nfslock restart" at server.
I mean, after server restart nfslock service, lockd and statd are not synchronizing. Because, after nfslock restart, server go into grace_period, and client's reclaimed lock request will be processed by lockd. At this, client's host(the host struct) will be reused which is created before nfslock restart, but, the lockd don't send a SM_MON to statd.
After locks are reclaimed, server's lockd records client hold locks, but statd don't monitor client.
thanks,
Mi Jinlong
Quoted 28 lines
--b.
Quoted 23 lines
After step3, server's lockd records client holding locks, but statd's
/var/lib/nfs/statd/sm/directory is empty. It means statd and lockd are not sync. If server restart it's nfslock again, client's locks will be lost.The Primary Reason:
At step3, when client's reclaimed lock request is sent to server,
client's host(the host struct) is reused but not be re-monitored at server's lockd. After that, statd and lockd are not sync.Question:
In my opinion, if lockd is allowed reuseing the client's host, it should send a
SM_MONto statd when reuse. If not allowed, the client's host should be destroyed immediately.What should lockd to do? Reuse ? Destroy ? Or some other action?
thanks,
Mi Jinlong
Signature · 2 lines
Regards Mi Jinlong
Chuck Lever:
Quoted 10 lines · Mi Jinlong
Quoted 9 lines
Hi,
When testing the NLM at the latest kernel(2.6.32), i find a bug. When a client hold locks, after server restart its nfslock service, server's statd will not synchronize with lockd.
If server restart nfslock twice or more, client's lock will be lost.Test process:
Quoted 7 lines
Quoted 3 lines
Step1: client open nfs file.
Step2: client using fcntl to get lock.
Step3: server restart it's nfslock service.I'll assume here that you mean the equivalent of "service nfslock restart". This restarts statd and possibly runs sm-notify, but it has no effect on lockd.
Yes, i used "service nfslock restart".
It has effect on lockd too, when service stop, lockd will get a KILL signal. Lockd will release all client's locks, and go into grace_period and wait client reclaime it's lock.
Quoted 6 lines
Again, this test seems artificial to me. Is there a real world use case where someone would deliberately restart statd while an NFS server is serving files? I pose this question because I've worked on statd only for a year or so, and I am quite likely ignorant of all the ways it can be deployed.
^/^, but maybe someone will restart nfslock when an NFS server is serving files. It is inevitable.
Quoted 6 lines
Quoted 6 lines
After step3, server's lockd records client holding locks, but statd's
/var/lib/nfs/statd/sm/directory is empty. It means statd and lockd are not sync. If server restart it's nfslock again, client's locks will be lost.
Quoted 8 lines
Quoted 5 lines
The Primary Reason:
At step3, when client's reclaimed lock request is sent to server,
client's host(the host struct) is reused but not be re-monitored at server's lockd. After that, statd and lockd are not sync.The kernel squashes
SM_MONupcalls for hosts that it already believes are monitored. This is a scalability feature.
When statd start, it will move files from /var/lib/nfs/statd/sm/ to /var/lib/nfs/statd/sm.bak/. If lockd don't send a SM_MON to statd, statd will not monitor those client which be monitored before statd restart. I don't make sure, is it right?
Quoted 8 lines
Quoted 8 lines
Question:
In my opinion, if lockd is allowed reuseing the client's host, it should send a
SM_MONto statd when reuse. If not allowed, the client's host should
be destroyed immediately.
Quoted 5 lines
Quoted 1 line
What should lockd to do? Reuse ? Destroy ? Or some other action?
I don't immediately see why lockd should change it's behavior. Perhaps statd/sm-notify were incorrect to delete the monitor list when you restarted the nfslock service?
Sorry, maybe i did not express clearly.
I mean, lockd reuse the host struct which was created before statd restart.
It seems have deleted the monitor list when nfslock restart.
Quoted 6 lines
Can you show exactly how statd's state (ie it's on-disk monitor list in
/var/lib/nfs/statd/sm) changed across the restart? Did sm-notify run when you restarted statd? If so, why didn't the sm-notify pid file stop it?
The statd and lockd's state at server when nfslock restart:
lockd statd |
|
host(monitored = 1) /sm/client | client get locks success at first
(locks) |
|
host(monitored = 1) /sm/client | nfslock stop (lockd release client's locks)
(no locks) |
|
host(monitored = 1) /sm/ | nfslock start (client reclaim locks)
(locks) | (but statd don't monitor it) note: host(monitored=1) means: client's host struct is created, and is marked be monitored.
(locks), (no locks)means: host strcut holds locks, or not.
/sm/client means: there have a file under /var/lib/nfs/statd/sm directory
/sm/ means: /var/lib/nfs/statd/sm is empty!
thanks,
Mi Jinlong
Quoted 14 lines · On Wed, 16 Dec 2009 18:27:09 +0800 Mi Jinlong <[email protected]> wrote:
The statd and lockd's state at server when nfslock restart:
lockd statd | | host(monitored = 1) /sm/client | client get locks success at first (locks) | | host(monitored = 1) /sm/client | nfslock stop (lockd release client's locks) (no locks) | | host(monitored = 1) /sm/ | nfslock start (client reclaim locks) (locks) | (but statd don't monitor it)
Quoted 6 lines
note: host(monitored=1) means: client's host struct is created, and is marked be monitored.
(locks), (no locks)means: host strcut holds locks, or not.
/sm/client means: there have a file under/var/lib/nfs/statd/smdirectory
/sm/ means:/var/lib/nfs/statd/smis empty!
Perhaps we ought to clear the cached list of monitored hosts (i.e. set them all to monitored = 0) when lockd gets a SIGKILL.
Signature · 1 line
Jeff Layton <[email protected]>
Quoted 49 lines · Mi Jinlong
Chuck Lever:
Quoted 10 lines · Mi Jinlong
Quoted 9 lines
Hi,
When testing the NLM at the latest kernel(2.6.32), i find a bug. When a client hold locks, after server restart its nfslock service, server's statd will not synchronize with lockd.
If server restart nfslock twice or more, client's lock will be lost.Test process:
Quoted 7 lines
Quoted 3 lines
Step1: client open nfs file.
Step2: client using fcntl to get lock.
Step3: server restart it's nfslock service.I'll assume here that you mean the equivalent of "service nfslock restart". This restarts statd and possibly runs sm-notify, but it has no effect on lockd.
Yes, i used "service nfslock restart".
It has effect on lockd too, when service stop, lockd will get a
KILL signal.
Lockd will release all client's locks, and go intograce_periodand
wait
client reclaime it's lock.Quoted 6 lines
Again, this test seems artificial to me. Is there a real world use case where someone would deliberately restart statd while an NFS server is serving files? I pose this question because I've worked on statd only for a year or so, and I am quite likely ignorant of all the ways it can be deployed.
^/^, but maybe someone will restart nfslock when an NFS server is
serving files.
It is inevitable.Quoted 11 lines
Quoted 10 lines
After step3, server's lockd records client holding locks, but statd's
/var/lib/nfs/statd/sm/directory is empty. It means statd and lockd are not sync. If server restart it's nfslock again, client's locks will be lost.The Primary Reason:
At step3, when client's reclaimed lock request is sent to server, client's host(the host struct) is reused but not be re-monitored at server's lockd. After that, statd and lockd are not sync.
Quoted 5 lines
Quoted 2 lines
The kernel squashes
SM_MONupcalls for hosts that it already believes are monitored. This is a scalability feature.When statd start, it will move files from
/var/lib/nfs/statd/sm/to/var/lib/nfs/statd/sm.bak/.
Well, it's really sm-notify that does this. sm-notify is run by rpc.statd when it starts up.
However, sm-notify should only retire the monitor list the first time it is run after a reboot. Simply restarting statd should not change the on-disk monitor list in the slightest. If it does, there's some kind of problem with the way sm-notify's pid file is managed, or perhaps with the nfslock script.
Quoted 17 lines
If lockd don't send a
SM_MONto statd,
statd will not monitor those client which be monitored before statd
restart.Quoted 7 lines
Quoted 7 lines
Question:
In my opinion, if lockd is allowed reuseing the client's host, it should send a
SM_MONto statd when reuse. If not allowed, the client's host should
be destroyed immediately.Quoted 5 lines
Quoted 1 line
What should lockd to do? Reuse ? Destroy ? Or some other action?
I don't immediately see why lockd should change it's behavior. Perhaps statd/sm-notify were incorrect to delete the monitor list when you restarted the nfslock service?
Quoted 5 lines
Sorry, maybe i did not express clearly.
I mean, lockd reuse the host struct which was created before statd
restart.It seems have deleted the monitor list when nfslock restart.
lockd does not touch any user space files; the on-disk monitor list is managed by statd and sm-notify. A remote peer rebooting does not clear the "monitored" flag for that peer in the local kernel's lockd, so it won't send another SM_MON request.
Now, it may be the case that "service nfslock start" uses a command line option that forces a fresh sm-notify run, and that is what is wiping the on-disk monitor list. That would be the bug in this case -- sm-notify can and should be allowed to make its own determination of whether the monitor list gets retired. Notification should not normally be forced by command line options in the nfslock script.
Quoted 33 lines
Quoted 4 lines
Can you show exactly how statd's state (ie it's on-disk monitor list in
/var/lib/nfs/statd/sm) changed across the restart? Did sm-notify run when you restarted statd? If so, why didn't the sm-notify pid file stop it?The statd and lockd's state at server when nfslock restart:
lockd statd | | host(monitored = 1) /sm/client | client get locks success at first (locks) | | host(monitored = 1) /sm/client | nfslock stop (lockd release client's locks) (no locks) | | host(monitored = 1) /sm/ | nfslock start(client reclaim locks)
(locks) | (but
statd don't monitor it)note: host(monitored=1) means: client's host struct is created,
and is marked be monitored.
(locks), (no locks)means: host strcut holds locks, or not.
/sm/client means: there have a file under/var/lib/
nfs/statd/sm directory
/sm/ means:/var/lib/nfs/statd/smis empty!thanks,
Mi Jinlong
Signature · 2 lines
Chuck Lever chuck[dot]lever[at]oracle[dot]com
Jeff Layton :
Quoted 23 lines
Quoted 13 lines · On Wed, 16 Dec 2009 18:27:09 +0800 Mi Jinlong <[email protected]> wrote:
The statd and lockd's state at server when nfslock restart:
lockd statd | | host(monitored = 1) /sm/client | client get locks success at first (locks) | | host(monitored = 1) /sm/client | nfslock stop (lockd release client's locks) (no locks) | | host(monitored = 1) /sm/ | nfslock start (client reclaim locks) (locks) | (but statd don't monitor it)Quoted 6 lines
note: host(monitored=1) means: client's host struct is created, and is marked be monitored.
(locks), (no locks)means: host strcut holds locks, or not.
/sm/client means: there have a file under/var/lib/nfs/statd/smdirectory
/sm/ means:/var/lib/nfs/statd/smis empty!
Quoted 2 lines
Perhaps we ought to clear the cached list of monitored hosts (i.e. set them all to monitored = 0) when lockd gets a SIGKILL.
Yes, if lockd can reuse the host struct, clear the cached list of monitored hosts is a good way when lockd gets a SIGKILL.
thanks,
Mi Jinlong
Chuck Lever :
Quoted 4 lines · Mi Jinlong
Quoted 3 lines
Chuck Lever:
Quoted 2 lines · Mi Jinlong
Quoted 1 line
Hi,
...snip...
Quoted 13 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
The Primary Reason:
At step3, when client's reclaimed lock request is sent to server, client's host(the host struct) is reused but not be re-monitored at server's lockd. After that, statd and lockd are not sync.
Quoted 5 lines
Quoted 2 lines
The kernel squashes
SM_MONupcalls for hosts that it already believes are monitored. This is a scalability feature.When statd start, it will move files from
/var/lib/nfs/statd/sm/to/var/lib/nfs/statd/sm.bak/.
Quoted 8 lines
Well, it's really sm-notify that does this. sm-notify is run by rpc.statd when it starts up.
However, sm-notify should only retire the monitor list the first time it is run after a reboot. Simply restarting statd should not change the on-disk monitor list in the slightest. If it does, there's some kind of problem with the way sm-notify's pid file is managed, or perhaps with the nfslock script.
When starting, statd will call run_sm_notify() function to run sm-notify. Using command "service nfslock restart" will case statd stop and start, so sm-notify will be run. If sm-notify run, the on-disk monitor list will be changed.
Quoted 23 lines
Quoted 23 lines
If lockd don't send a
SM_MONto statd,
statd will not monitor those client which be monitored before statd
restart.Quoted 8 lines
Quoted 8 lines
Question:
In my opinion, if lockd is allowed reuseing the client's host, it should
send aSM_MONto statd when reuse. If not allowed, the client's host should
be destroyed immediately.Quoted 5 lines
Quoted 1 line
What should lockd to do? Reuse ? Destroy ? Or some other action?
I don't immediately see why lockd should change it's behavior. Perhaps statd/sm-notify were incorrect to delete the monitor list when you restarted the nfslock service?
Sorry, maybe i did not express clearly.
I mean, lockd reuse the host struct which was created before statd
restart.
Quoted 6 lines
Quoted 1 line
It seems have deleted the monitor list when nfslock restart.
lockd does not touch any user space files; the on-disk monitor list is managed by statd and sm-notify. A remote peer rebooting does not clear the "monitored" flag for that peer in the local kernel's lockd, so it won't send another
SM_MONrequest.
Yes, that's right.
But, this case refers to server's lockd, not the remote peer. I thank, when local system's nfslock restart, local kernel's lockd clear all other client's host strcut's "monitored" flag.
Quoted 7 lines
Now, it may be the case that "service nfslock start" uses a command line option that forces a fresh sm-notify run, and that is what is wiping the on-disk monitor list. That would be the bug in this case -- sm-notify can and should be allowed to make its own determination of whether the monitor list gets retired. Notification should not normally be forced by command line options in the nfslock script.
A fresh sm-notify run is cause by statd start.
I find it through codes by followed.
utils/statd/statd.c
...
478 if (! (run_mode & MODE_NO_NOTIFY))
479 switch (pid = fork()) {
480 case 0:
481 run_sm_notify(out_port);
482 break;
483 case -1:
484 break;
485 default:
486 waitpid(pid, NULL, 0);
487 }
....
I thank, when statd restart and call sm-notify, the on-disk monitor list will be deleted, so lockd should clear all other client's host strcut's "monitored" flag. After that, a reused host struct will be re-monitored, a on-disk monitor will be re-created. Like that, lockd and statd will sync .
thanks,
Mi Jinlong
Quoted 99 lines · Mi Jinlong
Chuck Lever :
Quoted 4 lines · Mi Jinlong
Quoted 3 lines
Chuck Lever:
Quoted 2 lines · Mi Jinlong
Quoted 1 line
Hi,
...snip...
Quoted 13 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
The Primary Reason:
At step3, when client's reclaimed lock request is sent to server, client's host(the host struct) is reused but not be re-monitored at server's lockd. After that, statd and lockd are not sync.
Quoted 5 lines
Quoted 2 lines
The kernel squashes
SM_MONupcalls for hosts that it already believes are monitored. This is a scalability feature.When statd start, it will move files from
/var/lib/nfs/statd/sm/to/var/lib/nfs/statd/sm.bak/.Quoted 8 lines
Well, it's really sm-notify that does this. sm-notify is run by rpc.statd when it starts up.
However, sm-notify should only retire the monitor list the first time it is run after a reboot. Simply restarting statd should not change the on-disk monitor list in the slightest. If it does, there's some kind of problem with the way sm-notify's pid file is managed, or perhaps with the nfslock script.
When starting, statd will call
run_sm_notify()function to run sm-
notify.
Using command "service nfslock restart" will case statd stop and
start,
so sm-notify will be run. If sm-notify run, the on-disk monitor list will be changed.Quoted 23 lines
Quoted 23 lines
If lockd don't send a
SM_MONto statd,
statd will not monitor those client which be monitored before statd restart.Quoted 8 lines
Quoted 8 lines
Question:
In my opinion, if lockd is allowed reuseing the client's host, it should
send aSM_MONto statd when reuse. If not allowed, the client's host should
be destroyed immediately.Quoted 5 lines
Quoted 1 line
What should lockd to do? Reuse ? Destroy ? Or some other action?
I don't immediately see why lockd should change it's behavior. Perhaps statd/sm-notify were incorrect to delete the monitor list when you restarted the nfslock service?
Sorry, maybe i did not express clearly.
I mean, lockd reuse the host struct which was created before statd restart.Quoted 6 lines
Quoted 1 line
It seems have deleted the monitor list when nfslock restart.
lockd does not touch any user space files; the on-disk monitor list is managed by statd and sm-notify. A remote peer rebooting does not clear the "monitored" flag for that peer in the local kernel's lockd, so it won't send another
SM_MONrequest.Yes, that's right.
But, this case refers to server's lockd, not the remote peer. I thank, when local system's nfslock restart, local kernel's lockd clear all other client's host strcut's "monitored" flag.
Quoted 7 lines
Now, it may be the case that "service nfslock start" uses a command line option that forces a fresh sm-notify run, and that is what is wiping the on-disk monitor list. That would be the bug in this case -- sm-notify can and should be allowed to make its own determination of whether the monitor list gets retired. Notification should not normally be forced by command line options in the nfslock script.
A fresh sm-notify run is cause by statd start.
I find it through codes by followed.
utils/statd/statd.c
...
478 if (! (run_mode&MODE_NO_NOTIFY))
479 switch (pid =fork()) {
480 case 0:
481run_sm_notify(out_port);
482 break;
483 case -1:
484 break;
485 default:
486waitpid(pid, NULL, 0);
487 }
....
Quoted 4 lines
I thank, when statd restart and call sm-notify, the on-disk monitor list will be deleted, so lockd should clear all other client's host strcut's "monitored" flag. After that, a reused host struct will be re-monitored, a on-disk monitor will be re-created. Like that, lockd and statd will sync .
run_sm_notify() simply forks and execs the sm-notify program. This program checks for the existence of a pid file. If the pid file exists, then sm-notify exits. If it does not, then sm-notify retires the records in /var/lib/nfs/statd/sm and posts reboot notifications.
Jeff Layton pointed out to me yesterday that Red Hat's nfslock script unconditionally deletes sm-notify's pid file every time "service nfslock start" is done, which effectively defeats sm-notify's reboot detection.
sm-notify was written by a developer at SuSE. SuSE Linux uses a tmpfs for /var/run, but Red Hat uses permanent storage for this directory. Thus on SuSE, the pid file gets deleted automatically by a reboot, but on Red Hat, the pid file must be deleted "by hand" or reboot notification never occurs.
So the root cause of this problem is that the current mechanism sm-notify uses to detect a reboot is not portable across distributions.
My new-statd prototype used a semaphor instead of a pid file to detect reboots. A semaphor is shared (visible to other processes) and will continue to exist until it is deleted or the system reboots. It is a resource that is not destroyed automatically when the sm-notify process exits. If creating the semaphor fails, sm-notify exits. If creating it succeeds, it runs.
Would anyone strongly object to using a semaphor instead of a pid file here? Is support for semaphors always built into kernels? Would there be any problems with the small size of the semaphor name space? Is there another similar facility that might be better?
Signature · 2 lines
Chuck Lever chuck[dot]lever[at]oracle[dot]com
Quoted 2 lines · Chuck Lever
run_sm_notify()simply forks and execs the sm-notify program. This =20 program checks for the existence of a pid file. If the pid file exis=
ts,=20
Quoted 1 line
then sm-notify exits. If it does not, then sm-notify retires the rec=
ords=20
Quoted 3 lines
in
/var/lib/nfs/statd/smand posts reboot notifications.Jeff Layton pointed out to me yesterday that Red Hat's nfslock script=
=20
Quoted 1 line
unconditionally deletes sm-notify's pid file every time "service nfsl=
ock=20
Quoted 1 line
start" is done, which effectively defeats sm-notify's reboot detectio=
n.
Quoted 2 lines
sm-notify was written by a developer at SuSE. SuSE Linux uses a tmpf=
s =20
Quoted 1 line
for
/var/run, but Red Hat uses permanent storage for this directory. =
=20
Quoted 1 line
Thus on SuSE, the pid file gets deleted automatically by a reboot, bu=
t =20
Quoted 7 lines
on Red Hat, the pid file must be deleted "by hand" or reboot =20 notification never occurs.
So the root cause of this problem is that the current mechanism sm-=20 notify uses to detect a reboot is not portable across distributions.
My new-statd prototype used a semaphor instead of a pid file to detec=
t =20
Quoted 1 line
reboots. A semaphor is shared (visible to other processes) and will =
=20
Quoted 1 line
continue to exist until it is deleted or the system reboots. It is a=
=20
Quoted 1 line
resource that is not destroyed automatically when the sm-notify proce=
ss=20
Quoted 1 line
exits. If creating the semaphor fails, sm-notify exits. If creating=
it=20
Quoted 3 lines
succeeds, it runs.
Would anyone strongly object to using a semaphor instead of a pid fil=
e =20
Quoted 1 line
here? Is support for semaphors always built into kernels? Would the=
re=20
Quoted 1 line
be any problems with the small size of the semaphor name space? Is t=
here=20
Quoted 1 line
another similar facility that might be better?
I don't know much about those (except that I think there's an e at the end); looks like sem_overview(7) is the place to start?
It says:
" Prior to kernel 2.6, Linux only supported unnamed,
thread-shared sema=E2=80=90 phores. On a system with Linux 2.6 and =
a
glibc that provides the NPTL threading implementation, a complete implementation of POSIX semaphores is provided."
So would it mean dropping support for 2.4?
--b.
Quoted 147 lines · Chuck Lever
Quoted 112 lines · Mi Jinlong
Chuck Lever :
Quoted 4 lines · Mi Jinlong
Quoted 3 lines
Chuck Lever:
Quoted 2 lines · Mi Jinlong
Quoted 1 line
Hi,
...snip...
Quoted 18 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
The Primary Reason:
At step3, when client's reclaimed lock request is sent to server, client's host(the host struct) is reused but not be re-monitored at
server's lockd. After that, statd and lockd are not sync.Quoted 6 lines
Quoted 3 lines
The kernel squashes
SM_MONupcalls for hosts that it already believes
are monitored. This is a scalability feature.When statd start, it will move files from
/var/lib/nfs/statd/sm/to/var/lib/nfs/statd/sm.bak/.Well, it's really sm-notify that does this. sm-notify is run by rpc.statd when it starts up.
Quoted 7 lines
However, sm-notify should only retire the monitor list the first time it
is run after a reboot. Simply restarting statd should not change the on-disk monitor list in the slightest. If it does, there's some kind of
problem with the way sm-notify's pid file is managed, or perhaps with the nfslock script.When starting, statd will call
run_sm_notify()function to run sm-
notify.
Using command "service nfslock restart" will case statd stop and
start,
so sm-notify will be run. If sm-notify run, the on-disk monitor list will be changed.Quoted 25 lines
Quoted 25 lines
If lockd don't send a
SM_MONto statd,
statd will not monitor those client which be monitored before statd restart.Quoted 9 lines
Quoted 9 lines
Question:
In my opinion, if lockd is allowed reuseing the client's host, it should
send aSM_MONto statd when reuse. If not allowed, the client's host
should
be destroyed immediately.Quoted 6 lines
Quoted 1 line
What should lockd to do? Reuse ? Destroy ? Or some other action?
I don't immediately see why lockd should change it's behavior. Perhaps
statd/sm-notify were incorrect to delete the monitor list when you restarted the nfslock service?Sorry, maybe i did not express clearly.
I mean, lockd reuse the host struct which was created before statd restart.Quoted 8 lines
Quoted 1 line
It seems have deleted the monitor list when nfslock restart.
lockd does not touch any user space files; the on-disk monitor list is
managed by statd and sm-notify. A remote peer rebooting does not clear
the "monitored" flag for that peer in the local kernel's lockd, so it won't send anotherSM_MONrequest.Yes, that's right.
But, this case refers to server's lockd, not the remote peer. I thank, when local system's nfslock restart, local kernel's lockd clear all other client's host strcut's "monitored" flag.
Quoted 12 lines
Now, it may be the case that "service nfslock start" uses a command line
option that forces a fresh sm-notify run, and that is what is wiping the
on-disk monitor list. That would be the bug in this case -- sm- notify
can and should be allowed to make its own determination of whether the
monitor list gets retired. Notification should not normally be forced
by command line options in the nfslock script.A fresh sm-notify run is cause by statd start.
I find it through codes by followed.
utils/statd/statd.c
...
478 if (! (run_mode&MODE_NO_NOTIFY))
479 switch (pid =fork()) {
480 case 0:
481run_sm_notify(out_port);
482 break;
483 case -1:
484 break;
485 default:
486waitpid(pid, NULL, 0);
487 }
....Quoted 7 lines
I thank, when statd restart and call sm-notify, the on-disk monitor list will
be deleted, so lockd should clear all other client's host strcut's "monitored" flag.
After that, a reused host struct will be re-monitored, a on-disk monitor
will be re-created. Like that, lockd and statd will sync .
run_sm_notify()simply forks and execs the sm-notify program. This program checks for the existence of a pid file. If the pid file exists, then sm-notify exits. If it does not, then sm-notify retires the records in/var/lib/nfs/statd/smand posts reboot notifications.Jeff Layton pointed out to me yesterday that Red Hat's nfslock script unconditionally deletes sm-notify's pid file every time "service nfslock start" is done, which effectively defeats sm-notify's reboot detection.
sm-notify was written by a developer at SuSE. SuSE Linux uses a tmpfs for
/var/run, but Red Hat uses permanent storage for this directory. Thus on SuSE, the pid file gets deleted automatically by a reboot, but on Red Hat, the pid file must be deleted "by hand" or reboot notification never occurs.So the root cause of this problem is that the current mechanism sm- notify uses to detect a reboot is not portable across distributions.
My new-statd prototype used a semaphor instead of a pid file to detect reboots. A semaphor is shared (visible to other processes) and will continue to exist until it is deleted or the system reboots. It is a resource that is not destroyed automatically when the sm-notify process exits. If creating the semaphor fails, sm-notify exits. If creating it succeeds, it runs.
Quoted 5 lines
Would anyone strongly object to using a semaphor instead of a pid file here? Is support for semaphors always built into kernels? Would there be any problems with the small size of the semaphor name space? Is there another similar facility that might be better?
One alternative might be to just record the kernel's random boot_id in the pid file. That gets regenerated on each boot, so should be unique.
Trond
Quoted 148 lines · Trond Myklebust
Quoted 147 lines · Chuck Lever
Quoted 112 lines · Mi Jinlong
Chuck Lever :
Quoted 4 lines · Mi Jinlong
Quoted 3 lines
Chuck Lever:
Quoted 2 lines · Mi Jinlong
Quoted 1 line
Hi,
...snip...
Quoted 18 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
The Primary Reason:
At step3, when client's reclaimed lock request is sent to server, client's host(the host struct) is reused but not be re-monitored at
server's lockd. After that, statd and lockd are not sync.Quoted 6 lines
Quoted 3 lines
The kernel squashes
SM_MONupcalls for hosts that it already believes
are monitored. This is a scalability feature.When statd start, it will move files from
/var/lib/nfs/statd/sm/to/var/lib/nfs/statd/sm.bak/.Well, it's really sm-notify that does this. sm-notify is run by rpc.statd when it starts up.
Quoted 7 lines
However, sm-notify should only retire the monitor list the first time it
is run after a reboot. Simply restarting statd should not change the on-disk monitor list in the slightest. If it does, there's some kind of
problem with the way sm-notify's pid file is managed, or perhaps with the nfslock script.When starting, statd will call
run_sm_notify()function to run sm- notify.
Using command "service nfslock restart" will case statd stop and start,
so sm-notify will be run. If sm-notify run, the on-disk monitor list will be changed.Quoted 25 lines
Quoted 25 lines
If lockd don't send a
SM_MONto statd,
statd will not monitor those client which be monitored before statd restart.Quoted 9 lines
Quoted 9 lines
Question:
In my opinion, if lockd is allowed reuseing the client's host, it should
send aSM_MONto statd when reuse. If not allowed, the client's host
should
be destroyed immediately.Quoted 6 lines
Quoted 1 line
What should lockd to do? Reuse ? Destroy ? Or some other action?
I don't immediately see why lockd should change it's behavior. Perhaps
statd/sm-notify were incorrect to delete the monitor list when you restarted the nfslock service?Sorry, maybe i did not express clearly.
I mean, lockd reuse the host struct which was created before statd restart.Quoted 8 lines
Quoted 1 line
It seems have deleted the monitor list when nfslock restart.
lockd does not touch any user space files; the on-disk monitor list is
managed by statd and sm-notify. A remote peer rebooting does not clear
the "monitored" flag for that peer in the local kernel's lockd, so it won't send anotherSM_MONrequest.Yes, that's right.
But, this case refers to server's lockd, not the remote peer. I thank, when local system's nfslock restart, local kernel's lockd clear all other client's host strcut's "monitored" flag.
Quoted 12 lines
Now, it may be the case that "service nfslock start" uses a command line
option that forces a fresh sm-notify run, and that is what is wiping the
on-disk monitor list. That would be the bug in this case -- sm- notify
can and should be allowed to make its own determination of whether the
monitor list gets retired. Notification should not normally be forced
by command line options in the nfslock script.A fresh sm-notify run is cause by statd start.
I find it through codes by followed.
utils/statd/statd.c
...
478 if (! (run_mode&MODE_NO_NOTIFY))
479 switch (pid =fork()) {
480 case 0:
481run_sm_notify(out_port);
482 break;
483 case -1:
484 break;
485 default:
486waitpid(pid, NULL, 0);
487 }
....Quoted 7 lines
I thank, when statd restart and call sm-notify, the on-disk monitor list will
be deleted, so lockd should clear all other client's host strcut's "monitored" flag.
After that, a reused host struct will be re-monitored, a on-disk monitor
will be re-created. Like that, lockd and statd will sync .
run_sm_notify()simply forks and execs the sm-notify program. This program checks for the existence of a pid file. If the pid file exists, then sm-notify exits. If it does not, then sm-notify retires the records in/var/lib/nfs/statd/smand posts reboot notifications.Jeff Layton pointed out to me yesterday that Red Hat's nfslock script unconditionally deletes sm-notify's pid file every time "service nfslock start" is done, which effectively defeats sm-notify's reboot detection.
sm-notify was written by a developer at SuSE. SuSE Linux uses a tmpfs for
/var/run, but Red Hat uses permanent storage for this directory. Thus on SuSE, the pid file gets deleted automatically by a reboot, but on Red Hat, the pid file must be deleted "by hand" or reboot notification never occurs.So the root cause of this problem is that the current mechanism sm- notify uses to detect a reboot is not portable across distributions.
My new-statd prototype used a semaphor instead of a pid file to detect reboots. A semaphor is shared (visible to other processes) and will continue to exist until it is deleted or the system reboots. It is a resource that is not destroyed automatically when the sm-notify process exits. If creating the semaphor fails, sm-notify exits. If creating it succeeds, it runs.
Quoted 7 lines
Quoted 4 lines
Would anyone strongly object to using a semaphor instead of a pid file here? Is support for semaphors always built into kernels? Would there be any problems with the small size of the semaphor name space? Is there another similar facility that might be better?
One alternative might be to just record the kernel's random
boot_idin the pid file. That gets regenerated on each boot, so should be unique.
Where do you get it in user space? Is it available on earlier kernels? ("should be unique" -- I hope it doesn't have the same problem we had with XID replay on diskless systems).
Fwiw, I tried using the boot time stamp at one point, but unfortunately that's adjusted by the ntp offset, so it can take different values over time. It was difficult to compare it to a time stamp recorded in a file.
Signature · 2 lines
Chuck Lever chuck[dot]lever[at]oracle[dot]com
Quoted 9 lines · J. Bruce Fields
Quoted 8 lines · Chuck Lever
run_sm_notify()simply forks and execs the sm-notify program. This program checks for the existence of a pid file. If the pid file =20 exists,
then sm-notify exits. If it does not, then sm-notify retires the =20 records
in/var/lib/nfs/statd/smand posts reboot notifications.Jeff Layton pointed out to me yesterday that Red Hat's nfslock scrip=
t
Quoted 13 lines
Quoted 13 lines
unconditionally deletes sm-notify's pid file every time "service =20 nfslock
start" is done, which effectively defeats sm-notify's reboot =20 detection.sm-notify was written by a developer at SuSE. SuSE Linux uses a =20 tmpfs
for/var/run, but Red Hat uses permanent storage for this directory. Thus on SuSE, the pid file gets deleted automatically by a reboot, =20 but
on Red Hat, the pid file must be deleted "by hand" or reboot notification never occurs.
Quoted 7 lines
Quoted 7 lines
So the root cause of this problem is that the current mechanism sm- notify uses to detect a reboot is not portable across distributions.
My new-statd prototype used a semaphor instead of a pid file to =20 detect
reboots. A semaphor is shared (visible to other processes) and will continue to exist until it is deleted or the system reboots. It is =
a
Quoted 11 lines
Quoted 11 lines
resource that is not destroyed automatically when the sm-notify =20 process
exits. If creating the semaphor fails, sm-notify exits. If =20 creating it
succeeds, it runs.Would anyone strongly object to using a semaphor instead of a pid =20 file
here? Is support for semaphors always built into kernels? Would =20 there
be any problems with the small size of the semaphor name space? Is =
=20
Quoted 4 lines
Quoted 2 lines
there
another similar facility that might be better?I don't know much about those (except that I think there's an e at th=
e
Quoted 6 lines
end); looks like
sem_overview(7)is the place to start?It says:
" Prior to kernel 2.6, Linux only supported unnamed,
thread-shared sema=E2=80=90 phores. On a system with Linux 2.6 an=
d a
Quoted 4 lines
glibc that provides the NPTL threading implementation, a complete implementation of POSIX semaphores is provided."
So would it mean dropping support for 2.4?
No, it would mean using them only on systems that supported shared =20 semaphores.
--=20
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
Quoted 7 lines · Chuck Lever
Quoted 2 lines · Trond Myklebust
One alternative might be to just record the kernel's random
boot_idin the pid file. That gets regenerated on each boot, so should be unique.Where do you get it in user space? Is it available on earlier kernels? ("should be unique" -- I hope it doesn't have the same problem we had with XID replay on diskless systems).
You can access it from userland as the 'kernel.random.boot_id' sysctl.
It is available on 2.4 kernels and newer.
It is based on the kernel random number generator, so should be reasonably unique.
Quoted 4 lines
Fwiw, I tried using the boot time stamp at one point, but unfortunately that's adjusted by the ntp offset, so it can take different values over time. It was difficult to compare it to a time stamp recorded in a file.
Agreed. You can't rely on time stamps.
Trond
Quoted 10 lines · On Thu, 17 Dec 2009 11:18:53 -0500 Chuck Lever <[email protected]> wrote:
Jeff Layton pointed out to me yesterday that Red Hat's nfslock script unconditionally deletes sm-notify's pid file every time "service nfslock start" is done, which effectively defeats sm-notify's reboot detection.
sm-notify was written by a developer at SuSE. SuSE Linux uses a tmpfs for
/var/run, but Red Hat uses permanent storage for this directory. Thus on SuSE, the pid file gets deleted automatically by a reboot, but on Red Hat, the pid file must be deleted "by hand" or reboot notification never occurs.
Just to make sure the facts are straight:
SuSE does not use tmpfs for /var/run (much as I personally think that would be a very sensible approach for both /var/run and /var/locks). It appears that Debian can use tmpfs for these, but doesn't by default.
Both SuSE and Debian have boot time scripts that clean up /var/run and other directories. They remove all non-directories other than /var/run/utmp.
If Redhat doesn't clean up /var/run at boot time, then I would think that is very odd. The files in there represent something that is running. At boot, nothing is running, so it should all be cleaned up. Are you sure Redhat doesn't clean out /var/run???
I just had a look at master.kernel.org (the only fedora machine I can think of that I have access to) and in /etc/rc.d/rc.sysinit I find
find /var/lock /var/run ! -type d -exec rm -f {} \;So I'm thinking that if you just remove
# Make sure locks are recovered
rm -f /var/run/sm-notify.pid
from /etc/init.d/nfslock, then it will do the right thing.
NeilBrown
Quoted 30 lines · Neil Brown
Quoted 10 lines · On Thu, 17 Dec 2009 11:18:53 -0500 Chuck Lever <[email protected]> wrote:
Jeff Layton pointed out to me yesterday that Red Hat's nfslock script unconditionally deletes sm-notify's pid file every time "service nfslock start" is done, which effectively defeats sm-notify's reboot detection.
sm-notify was written by a developer at SuSE. SuSE Linux uses a tmpfs for
/var/run, but Red Hat uses permanent storage for this directory. Thus on SuSE, the pid file gets deleted automatically by a reboot, but on Red Hat, the pid file must be deleted "by hand" or reboot notification never occurs.Just to make sure the facts are straight:
SuSE does not use tmpfs for/var/run(much as I personally think that would be a very sensible approach for both/var/runand/var/locks). It appears that Debian can use tmpfs for these, but doesn't by default.Both SuSE and Debian have boot time scripts that clean up
/var/runand other directories. They remove all non-directories other than/var/run/utmp.If Redhat doesn't clean up
/var/runat boot time, then I would think that is very odd. The files in there represent something that is running. At boot, nothing is running, so it should all be cleaned up. Are you sure Redhat doesn't clean out/var/run???I just had a look at master.kernel.org (the only fedora machine I can think of that I have access to) and in
/etc/rc.d/rc.sysinitI find
Quoted 8 lines
find /var/lock /var/run ! -type d -exec rm -f {} \;So I'm thinking that if you just remove
# Make sure locks are recovered
rm -f/var/run/sm-notify.pidfrom
/etc/init.d/nfslock, then it will do the right thing.
Makes sense. Steve, can you look into this for supported releases (like F12 and RHEL5)? Or, perhaps you can clarify why that "rm" is required.
Meanwhile, I'm going to prototype a mechanism that tries to use the kernel's boot_id, if present.
Signature · 2 lines
Chuck Lever chuck[dot]lever[at]oracle[dot]com
Quoted 41 lines · Chuck Lever
Quoted 38 lines · Neil Brown
Quoted 10 lines · On Thu, 17 Dec 2009 11:18:53 -0500 Chuck Lever <[email protected]> wrote:
Jeff Layton pointed out to me yesterday that Red Hat's nfslock script unconditionally deletes sm-notify's pid file every time "service nfslock start" is done, which effectively defeats sm-notify's reboot detection.
sm-notify was written by a developer at SuSE. SuSE Linux uses a tmpfs for
/var/run, but Red Hat uses permanent storage for this directory. Thus on SuSE, the pid file gets deleted automatically by a reboot, but on Red Hat, the pid file must be deleted "by hand" or reboot notification never occurs.Just to make sure the facts are straight:
SuSE does not use tmpfs for/var/run(much as I personally think that would be a very sensible approach for both/var/runand/var/locks). It appears that Debian can use tmpfs for these, but doesn't by default.Both SuSE and Debian have boot time scripts that clean up
/var/runand other
directories. They remove all non-directories other than/var/run/utmp.If Redhat doesn't clean up
/var/runat boot time, then I would think that is
very odd. The files in there represent something that is running. At boot,
nothing is running, so it should all be cleaned up. Are you sure Redhat doesn't clean out/var/run???I just had a look at master.kernel.org (the only fedora machine I can think
of that I have access to) and in/etc/rc.d/rc.sysinitI findfind /var/lock /var/run ! -type d -exec rm -f {} \;So I'm thinking that if you just remove
Quoted 7 lines
Quoted 4 lines
# Make sure locks are recovered
rm -f/var/run/sm-notify.pidfrom
/etc/init.d/nfslock, then it will do the right thing.Makes sense. Steve, can you look into this for supported releases (like F12 and RHEL5)? Or, perhaps you can clarify why that "rm" is required.
I know at the time I added that code the pid file was not being removed and by explicitly removing it caused sm-notify to *always* run which, at the time, seem like the right thing to do.. The change was made in early January of 08, so let me take to see if things have changed...
steved.