Skip to contentSkip to conversation

After nfs restart, locks can't be recovered which record by lockd before

RFC Mi Jinlong 3 participants 13 messages Jan 13, 2010 – Jan 19, 2010 2 messages not in the archive

Subsystems (2)
lore
Share & tools
lore

Mi Jinlong

author
#0 lore raw

Hi,

When testing the nfs's lock at NFSv3, I get a problem. So I want someone help me, thanks!

Test Process:
Step1, ClientA get an exclusive lock success.
Step2, Using command "service nfs restart" to restart server's nfs se=
rvice.
Step3. ClientB get lock should fail, but success.

I think after step2 (nfs service restart), clientA's lock should be rec= overed.
But like above, clientA's lock doesn=E2=80=99t be recovered.

When tracing the kernel, I find nfsd will cause lockd stop when it stop= =2E
When lockd stop, all locks will be release which is record before at lo= ckd.

When nfsd start, the lockd will start also, but the statd don't know wh= at happened
at kernel, so after that, locks will be lost.

Is it right when nfs stop, the lockd will stop too?=20 If it's right, should locks be recovered after lockd start?

thanks,
Mi Jinlong

#1 lore raw
Quoted 8 lines · On Wed, 13 Jan 2010 17:51:25 +0800 Mi Jinlong <[email protected]> wrote:

Hi,
=20
When testing the nfs's lock at NFSv3, I get a problem.
So I want someone help me, thanks!
=20
Test Process:
Step1, ClientA get an exclusive lock success.
Step2, Using command "service nfs restart" to restart server's nfs =

service.

"service nfs restart" on an RH-derived distro is the same as running "/etc/init.d/nfs restart". Init scripts vary between distros (and even between releases on the same distro). Since you're asking this in a more generic forum, you should probably be specific about what's actually being restarted (and in what order). Understanding that may also help you answer your own question here.

Quoted 3 lines

Step3. ClientB get lock should fail, but success.
=20
I think after step2 (nfs service restart), clientA's lock should be r=

ecovered.

Quoted 3 lines

But like above, clientA's lock doesn=E2=80=99t be recovered. =20
When tracing the kernel, I find nfsd will cause lockd stop when it st=

op.

Quoted 1 line

When lockd stop, all locks will be release which is record before at =

lockd.

Quoted 2 lines

=20
When nfsd start, the lockd will start also, but the statd don't know =

what happened

Quoted 5 lines

at kernel, so after that, locks will be lost.
=20
Is it right when nfs stop, the lockd will stop too?=20 If it's right, should locks be recovered after lockd start? =20

Assuming you're using a RH-derived distro like Fedora or RHEL, then no. statd is controlled by a separate init script (nfslock) and when you run "service nfs restart" you're not restarting it. NSM notifications are not sent and clients generally won't reclaim their locks.

IOW, "you're doing it wrong". If you want locks to be reclaimed then you probably need to restart the nfslock service too.

--=20
Jeff Layton <[email protected]>

#3 lore raw
In reply to a message not in the archive
Quoted 11 lines · Jeff Layton
Quoted 8 lines · On Wed, 13 Jan 2010 17:51:25 +0800 Mi Jinlong<[email protected]> wrote:

Hi,

When testing the nfs's lock at NFSv3, I get a problem. So I want someone help me, thanks!

Test Process:
Step1, ClientA get an exclusive lock success.
Step2, Using command "service nfs restart" to restart server's nf=

s service.

Quoted 3 lines

"service nfs restart" on an RH-derived distro is the same as running "/etc/init.d/nfs restart". Init scripts vary between distros (and eve=

n

Quoted 8 lines

between releases on the same distro). Since you're asking this in a more generic forum, you should probably be specific about what's actually being restarted (and in what order). Understanding that may also help you answer your own question here.

Quoted 3 lines

Step3. ClientB get lock should fail, but success.

I think after step2 (nfs service restart), clientA's lock should be =

recovered.

Quoted 3 lines
Quoted 3 lines

But like above, clientA's lock doesn=E2=80=99t be recovered.

When tracing the kernel, I find nfsd will cause lockd stop when it s=

top.

Quoted 1 line
Quoted 1 line

When lockd stop, all locks will be release which is record before at=

lockd.

Quoted 2 lines
Quoted 2 lines

When nfsd start, the lockd will start also, but the statd don't know=

what happened

Quoted 7 lines
Quoted 5 lines

at kernel, so after that, locks will be lost.

Is it right when nfs stop, the lockd will stop too? If it's right, should locks be recovered after lockd start?

Assuming you're using a RH-derived distro like Fedora or RHEL, then n=

o.

Quoted 6 lines

statd is controlled by a separate init script (nfslock) and when you run "service nfs restart" you're not restarting it. NSM notifications are not sent and clients generally won't reclaim their locks.

IOW, "you're doing it wrong". If you want locks to be reclaimed then you probably need to restart the nfslock service too.

Mi Jinlong is exercising another case we know doesn't work right, but w= e=20
don't expect admins will ever perform this kind of "down-up" on a norma= l=20
production server. In other words, we expect it to work this way, and=20 it's been good enough, so far.

As Jeff points out, the "nfs" and the "nfslock" services are separate.=20 This is because "nfslock" is required for both client and server side=20 NFS, but "nfs" is required only on the server. This split also dictate= s=20
the way sm-notify works, since it has to behave differently on NFS=20 clients and servers.

Two other points:

+ lockd would not restart itself in this case if there happened to b=
e=20
NFS mounts on that system

+ lockd doesn't currently poke statd when it restarts to tell it to=20
send reboot notifications, but it probably should

We know that lockd will start up when someone mounts the first NFS=20 share, or when the NFS server is started. If lockd sent statd an=20 SM_SIMU_CRASH (or something like it) every time it cold started, statd=20 could send reboot notifications at the right time on both servers and=20 clients without extra logic in the init scripts, and we wouldn't need=20 that kludge in sm-notify to know when a machine has rebooted.

--=20
chuck[dot]lever[at]oracle[dot]com

author
#11 lore raw
In reply to a message not in the archive

Hi Jeff,

Jeff Layton =E5=86=99=E9=81=93:

Quoted 11 lines

On Wed, 13 Jan 2010 17:51:25 +0800
Mi Jinlong <[email protected]> wrote:
=20

Quoted 8 lines

Hi,

When testing the nfs's lock at NFSv3, I get a problem. So I want someone help me, thanks!

Test Process:
Step1, ClientA get an exclusive lock success.
Step2, Using command "service nfs restart" to restart server's nfs=

service.

Quoted 3 lines

=20
"service nfs restart" on an RH-derived distro is the same as running "/etc/init.d/nfs restart". Init scripts vary between distros (and eve=

n

Quoted 4 lines

between releases on the same distro). Since you're asking this in a more generic forum, you should probably be specific about what's actually being restarted (and in what order). Understanding that may also help you answer your own question here.

I test it on RHEL5(kernel 2.6.18, nfs-utils 1.0.9) and Fedora 12(kern=
el 2.6.33-rc3,
nfs-utils 1.2.0).

Quoted 4 lines

=20

Quoted 3 lines

Step3. ClientB get lock should fail, but success.

I think after step2 (nfs service restart), clientA's lock should be =

recovered.

Quoted 3 lines
Quoted 3 lines

But like above, clientA's lock doesn=E2=80=99t be recovered.

When tracing the kernel, I find nfsd will cause lockd stop when it s=

top.

Quoted 1 line
Quoted 1 line

When lockd stop, all locks will be release which is record before at=

lockd.

Quoted 2 lines
Quoted 2 lines

When nfsd start, the lockd will start also, but the statd don't know=

what happened

Quoted 7 lines
Quoted 5 lines

at kernel, so after that, locks will be lost.

Is it right when nfs stop, the lockd will stop too?=20 If it's right, should locks be recovered after lockd start?

=20
Assuming you're using a RH-derived distro like Fedora or RHEL, then n=

o.

Quoted 3 lines

statd is controlled by a separate init script (nfslock) and when you run "service nfs restart" you're not restarting it. NSM notifications are not sent and clients generally won't reclaim their locks.

Yes, the statd don't restart when nfs restart. But, nfs restart will cause lockd restart. If lockd restart, locks which record before will=
be lost.

Quoted 3 lines

IOW, "you're doing it wrong". If you want locks to be reclaimed then you probably need to restart the nfslock service too.
=20

Don't you think it's more reasonable that locks should be recovered=20 when nfs restart cause lockd restart?

--=20
Regards
Mi Jinlong

author
#4 lore raw
In reply to Chuck Lever

Hi Chuck,

Chuck Lever =E5=86=99=E9=81=93:

Quoted 5 lines · Jeff Layton
Quoted 4 lines

On Wed, 13 Jan 2010 17:51:25 +0800
Mi Jinlong<[email protected]> wrote:

Assuming you're using a RH-derived distro like Fedora or RHEL, then =

no.

Quoted 2 lines
Quoted 2 lines

statd is controlled by a separate init script (nfslock) and when you run "service nfs restart" you're not restarting it. NSM notification=

s

Quoted 6 lines
Quoted 4 lines

are not sent and clients generally won't reclaim their locks.

IOW, "you're doing it wrong". If you want locks to be reclaimed then you probably need to restart the nfslock service too.

=20
Mi Jinlong is exercising another case we know doesn't work right, but=

we

Quoted 1 line

don't expect admins will ever perform this kind of "down-up" on a nor=

mal

Quoted 1 line

production server. In other words, we expect it to work this way, an=

d

Quoted 3 lines

it's been good enough, so far.
=20
As Jeff points out, the "nfs" and the "nfslock" services are separate=

=2E

Quoted 2 lines

This is because "nfslock" is required for both client and server side NFS, but "nfs" is required only on the server. This split also dicta=

tes

Quoted 6 lines

the way sm-notify works, since it has to behave differently on NFS clients and servers.
=20
Two other points:
=20
+ lockd would not restart itself in this case if there happened to =

be

Quoted 1 line

NFS mounts on that system

When testing, i find nfs restart will cause lockd restart. I find some codes which cause the lock stop when nfs stop.

At kernel 2.6.18, fs/lockd/svc.c
...
354 if (nlmsvc_users) {
355 if (--nlmsvc_users)
356 goto out;
357 } else
358 printk(KERN_WARNING "lockd_down: no users! pid=3D%=
d\n", nlmsvc_pid);
...
366=20
367 kill_proc(nlmsvc_pid, SIGKILL, 1);
...
=20
At kernel 2.6.18, fs/lockd/svc.c
...
344 if (nlmsvc_users) {
345 if (--nlmsvc_users)
346 goto out;
347 } else {
348 printk(KERN_ERR "lockd_down: no users! task=3D%p\n=
",
349 nlmsvc_task);
350 BUG();
351 }
....
357 kthread_stop(nlmsvc_task);
358 svc_exit_thread(nlmsvc_rqst);
...

As above, when nlmsvc_users <=3D 1, the lockd will be killed.

Quoted 3 lines

=20
+ lockd doesn't currently poke statd when it restarts to tell it to
send reboot notifications, but it probably should

Yes, I agree with you. But now, when some reason cause lockd restart =
but
statd not restart, locks which hold before will lost.

Maybe, the kernel should fix this.

Quoted 4 lines

=20
We know that lockd will start up when someone mounts the first NFS share, or when the NFS server is started. If lockd sent statd an SM_SIMU_CRASH (or something like it) every time it cold started, stat=

d

Quoted 3 lines

could send reboot notifications at the right time on both servers and clients without extra logic in the init scripts, and we wouldn't need that kludge in sm-notify to know when a machine has rebooted.

What's the meaning of cold start?? System reboot? Or statd reboot?

I want to know when using cammond "service nfslock restart" restart t=
he=20
nfslock service(means restart statd and lockd), will the statd call s=
m-notify=20
to notify other client? Or don't?

At RHEL5 and Fedora, the sm-notify will be call and send SM-NOTIFY ev=
ery time=20
when nfslock restart.

thanks,
Mi Jinlong

#12 lore raw
In reply to Mi Jinlong
Quoted 7 lines · On Thu, 14 Jan 2010 17:41:13 +0800 Mi Jinlong <[email protected]> wrote:
Quoted 3 lines

IOW, "you're doing it wrong". If you want locks to be reclaimed then you probably need to restart the nfslock service too.

Don't you think it's more reasonable that locks should be recovered when nfs restart cause lockd restart?

Sure, that's the idea. That said, I'm leery of trying to make this too automatic. Extra complexity in this sort of thing often means that we get emergent behaviors from the system that we don't expect. I'm not sure that risk is worth it to guard against admins restarting lockd like this without forcing notifications to be sent.

Signature · 1 line
Jeff Layton <[email protected]>
#5 lore raw
In reply to Mi Jinlong
Quoted 10 lines · Mi Jinlong

Hi Chuck,

Chuck Lever =E5=86=99=E9=81=93:

Quoted 7 lines · Jeff Layton
Quoted 6 lines

On Wed, 13 Jan 2010 17:51:25 +0800
Mi Jinlong<[email protected]> wrote:

Assuming you're using a RH-derived distro like Fedora or RHEL, =20 then no.
statd is controlled by a separate init script (nfslock) and when yo=

u

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

run "service nfs restart" you're not restarting it. NSM =20 notifications
are not sent and clients generally won't reclaim their locks.

IOW, "you're doing it wrong". If you want locks to be reclaimed the=

n

Quoted 10 lines
Quoted 10 lines
Quoted 1 line

you probably need to restart the nfslock service too.

Mi Jinlong is exercising another case we know doesn't work right, =20 but we
don't expect admins will ever perform this kind of "down-up" on a =20 normal
production server. In other words, we expect it to work this way, =20 and
it's been good enough, so far.

Quoted 3 lines
Quoted 3 lines

As Jeff points out, the "nfs" and the "nfslock" services are =20 separate.
This is because "nfslock" is required for both client and server sid=

e

Quoted 7 lines
Quoted 7 lines

NFS, but "nfs" is required only on the server. This split also =20 dictates
the way sm-notify works, since it has to behave differently on NFS clients and servers.
Two other points:

+ lockd would not restart itself in this case if there happened to =

=20

Quoted 6 lines
Quoted 2 lines

be
NFS mounts on that system

When testing, i find nfs restart will cause lockd restart. I find some codes which cause the lock stop when nfs stop.

Quoted 7 lines

At kernel 2.6.18, fs/lockd/svc.c
...
354 if (nlmsvc_users) {
355 if (--nlmsvc_users)
356 goto out;
357 } else
358 printk(KERN_WARNING "lockd_down: no users! pid=3D=

%d=20

Quoted 6 lines

\n", nlmsvc_pid);
...
366
367 kill_proc(nlmsvc_pid, SIGKILL, 1);
...

Quoted 7 lines

At kernel 2.6.18, fs/lockd/svc.c
...
344 if (nlmsvc_users) {
345 if (--nlmsvc_users)
346 goto out;
347 } else {
348 printk(KERN_ERR "lockd_down: no users! task=3D%p\=

n",

Quoted 11 lines

349 nlmsvc_task);
350 BUG();
351 }
....
357 kthread_stop(nlmsvc_task);
358 svc_exit_thread(nlmsvc_rqst);
...

As above, when nlmsvc_users <=3D 1, the lockd will be killed.

Quoted 1 line
Quoted 8 lines
Quoted 2 lines

+ lockd doesn't currently poke statd when it restarts to tell it to
send reboot notifications, but it probably should

Yes, I agree with you. But now, when some reason cause lockd =20
restart but
statd not restart, locks which hold before will lost.

Maybe, the kernel should fix this.

What did you have in mind?

Quoted 5 lines
Quoted 5 lines

We know that lockd will start up when someone mounts the first NFS share, or when the NFS server is started. If lockd sent statd an SM_SIMU_CRASH (or something like it) every time it cold started, =20 statd
could send reboot notifications at the right time on both servers an=

d

Quoted 1 line
Quoted 1 line

clients without extra logic in the init scripts, and we wouldn't nee=

d

Quoted 3 lines
Quoted 1 line

that kludge in sm-notify to know when a machine has rebooted.

What's the meaning of cold start?? System reboot? Or statd reboot?

cold start meaning that lockd is shutdown and rmmod'd, then started up = =20
and re-loaded.

This can also happen on a client if all NFS mounts go away. =20 lockd_down is invoked, and lockd.ko is removed. On the next NFS =20 mount, lockd is loaded again.

Quoted 1 line

I want to know when using cammond "service nfslock restart" restart =

=20

Quoted 2 lines

the
nfslock service(means restart statd and lockd), will the statd call =

=20

Quoted 2 lines

sm-notify
to notify other client? Or don't?

Currently "service nfslock restart" always causes a notification to be = =20
sent. Since "service nfslock restart" causes lockd to drop its locks =20 (I assume that's what that "killproc lockd" does) I guess we need to =20 force reboot notifications here. (I still argue that removing the =20 pidfile in the "start" case is not correct).

It appears that both the nfs and nfslock start up scripts do something = =20
to lockd (as well as the case when the number of NFS mounts goes to =20 zero). However, only the nfslock script forces sm-notify to send =20 notifications.

I suppose a naive fix for your server restart issue might be to add an = =20
"sm-notify -f" to the "restart" case in /etc/init.d/nfs. This would =20 cause reboot notifications to be sent if the monitor list was not =20 empty during a server restart.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

author
#14 lore raw
In reply to a message not in the archive

Hi Jeff,

Jeff Layton :

Quoted 10 lines
Quoted 6 lines · On Thu, 14 Jan 2010 17:41:13 +0800 Mi Jinlong <[email protected]> wrote:
Quoted 3 lines

IOW, "you're doing it wrong". If you want locks to be reclaimed then you probably need to restart the nfslock service too.

Don't you think it's more reasonable that locks should be recovered when nfs restart cause lockd restart?

Quoted 6 lines

Sure, that's the idea. That said, I'm leery of trying to make this too automatic. Extra complexity in this sort of thing often means that we get emergent behaviors from the system that we don't expect. I'm not sure that risk is worth it to guard against admins restarting lockd like this without forcing notifications to be sent.

IMO, when lockd restart without statd restart, lockd should recovered those locks. If the NFSv3's designers think it's not needed when the design it, we would respect theirs idea at first. But, I think this is indeed a problem!

Signature · 2 lines
Regards
Mi Jinlong
author
#6 lore raw
In reply to Chuck Lever

Hi Chuck,

Chuck Lever =E5=86=99=E9=81=93:

Quoted 9 lines · Mi Jinlong
Quoted 8 lines

Hi Chuck,

Chuck Lever =E5=86=99=E9=81=93:

Quoted 5 lines · Jeff Layton
Quoted 4 lines

On Wed, 13 Jan 2010 17:51:25 +0800
Mi Jinlong<[email protected]> wrote:

Assuming you're using a RH-derived distro like Fedora or RHEL, the=

n no.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

statd is controlled by a separate init script (nfslock) and when y=

ou

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

run "service nfs restart" you're not restarting it. NSM notificati=

ons

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

are not sent and clients generally won't reclaim their locks.

IOW, "you're doing it wrong". If you want locks to be reclaimed th=

en

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

you probably need to restart the nfslock service too.

Mi Jinlong is exercising another case we know doesn't work right, b=

ut we

Quoted 1 line
Quoted 1 line
Quoted 1 line

don't expect admins will ever perform this kind of "down-up" on a n=

ormal

Quoted 1 line
Quoted 1 line
Quoted 1 line

production server. In other words, we expect it to work this way, =

and

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

it's been good enough, so far.

As Jeff points out, the "nfs" and the "nfslock" services are separa=

te.

Quoted 1 line
Quoted 1 line
Quoted 1 line

This is because "nfslock" is required for both client and server si=

de

Quoted 1 line
Quoted 1 line
Quoted 1 line

NFS, but "nfs" is required only on the server. This split also dic=

tates

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

the way sm-notify works, since it has to behave differently on NFS clients and servers.
Two other points:

+ lockd would not restart itself in this case if there happened to=

be

Quoted 18 lines
Quoted 18 lines
Quoted 1 line

NFS mounts on that system

When testing, i find nfs restart will cause lockd restart. I find some codes which cause the lock stop when nfs stop.

At kernel 2.6.18, fs/lockd/svc.c
...
354 if (nlmsvc_users) {
355 if (--nlmsvc_users)
356 goto out;
357 } else
358 printk(KERN_WARNING "lockd_down: no users!
pid=3D%d\n", nlmsvc_pid);
...
366
367 kill_proc(nlmsvc_pid, SIGKILL, 1);
...

Quoted 7 lines
Quoted 7 lines

At kernel 2.6.18, fs/lockd/svc.c
...
344 if (nlmsvc_users) {
345 if (--nlmsvc_users)
346 goto out;
347 } else {
348 printk(KERN_ERR "lockd_down: no users! task=3D%p=

\n",

Quoted 8 lines
Quoted 8 lines

349 nlmsvc_task);
350 BUG();
351 }
....
357 kthread_stop(nlmsvc_task);
358 svc_exit_thread(nlmsvc_rqst);
...

Quoted 4 lines
Quoted 4 lines

As above, when nlmsvc_users <=3D 1, the lockd will be killed.

Quoted 2 lines

+ lockd doesn't currently poke statd when it restarts to tell it t=

o

Quoted 3 lines
Quoted 3 lines
Quoted 1 line

send reboot notifications, but it probably should

Yes, I agree with you. But now, when some reason cause lockd restar=

t but

Quoted 5 lines
Quoted 3 lines

statd not restart, locks which hold before will lost.

Maybe, the kernel should fix this.

=20
What did you have in mind?

I think when lockd restart, statd should restart too and sent sm-noti=
fy to other client.
But now, in kernel and nfs-uitls, it don't implemented.
As the communication style between lockd and statd, this is indeed no=
t easy to implement it.

So, I think it's should more easy to implement it through the mechani=
sm that exposes
the kernel's nlm_host cache via /sys you show me before.

Quoted 4 lines

=20

Quoted 3 lines
Quoted 3 lines

We know that lockd will start up when someone mounts the first NFS share, or when the NFS server is started. If lockd sent statd an SM_SIMU_CRASH (or something like it) every time it cold started, st=

atd

Quoted 1 line
Quoted 1 line
Quoted 1 line

could send reboot notifications at the right time on both servers a=

nd

Quoted 1 line
Quoted 1 line
Quoted 1 line

clients without extra logic in the init scripts, and we wouldn't ne=

ed

Quoted 5 lines
Quoted 3 lines
Quoted 1 line

that kludge in sm-notify to know when a machine has rebooted.

What's the meaning of cold start?? System reboot? Or statd reboot?

=20
cold start meaning that lockd is shutdown and rmmod'd, then started u=

p

Quoted 3 lines

and re-loaded.
=20
This can also happen on a client if all NFS mounts go away. lockd_do=

wn

Quoted 2 lines

is invoked, and lockd.ko is removed. On the next NFS mount, lockd is loaded again.

Thanks.

Quoted 2 lines

=20

Quoted 1 line

I want to know when using cammond "service nfslock restart" restart=

the

Quoted 5 lines
Quoted 3 lines

nfslock service(means restart statd and lockd), will the statd call
sm-notify
to notify other client? Or don't?

=20
Currently "service nfslock restart" always causes a notification to b=

e

Quoted 1 line

sent. Since "service nfslock restart" causes lockd to drop its locks=

(I

Quoted 1 line

assume that's what that "killproc lockd" does) I guess we need to for=

ce

Quoted 1 line

reboot notifications here. (I still argue that removing the pidfile =

in

Quoted 3 lines

the "start" case is not correct).
=20
It appears that both the nfs and nfslock start up scripts do somethin=

g

Quoted 3 lines

to lockd (as well as the case when the number of NFS mounts goes to zero). However, only the nfslock script forces sm-notify to send notifications.

But, at RHLE5 and Fedora, when using cammond "service nfslock restart=
" restart
the nfslock service, the lockd isn't shutdown and rmmod'd.

Is it a bug?

Quoted 2 lines

=20
I suppose a naive fix for your server restart issue might be to add a=

n

Quoted 2 lines

"sm-notify -f" to the "restart" case in /etc/init.d/nfs. This would cause reboot notifications to be sent if the monitor list was not emp=

ty

Quoted 1 line

during a server restart.

Hehe. ^/^

thinks,
Mi Jinlong

#7 lore raw
Quoted 14 lines · Mi Jinlong

Hi Chuck,

Chuck Lever =E5=86=99=E9=81=93:

Quoted 11 lines · Mi Jinlong
Quoted 10 lines

Hi Chuck,

Chuck Lever =E5=86=99=E9=81=93:

Quoted 7 lines · Jeff Layton
Quoted 6 lines

On Wed, 13 Jan 2010 17:51:25 +0800
Mi Jinlong<[email protected]> wrote:

Assuming you're using a RH-derived distro like Fedora or RHEL, =20 then no.
statd is controlled by a separate init script (nfslock) and when =

=20

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 8 lines

you
run "service nfs restart" you're not restarting it. NSM =20 notifications
are not sent and clients generally won't reclaim their locks.

IOW, "you're doing it wrong". If you want locks to be reclaimed =20 then
you probably need to restart the nfslock service too.

Mi Jinlong is exercising another case we know doesn't work right, =

=20

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

but we
don't expect admins will ever perform this kind of "down-up" on a =

=20

Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
Quoted 15 lines

normal
production server. In other words, we expect it to work this =20 way, and
it's been good enough, so far.

As Jeff points out, the "nfs" and the "nfslock" services are =20 separate.
This is because "nfslock" is required for both client and server =20 side
NFS, but "nfs" is required only on the server. This split also =20 dictates
the way sm-notify works, since it has to behave differently on NFS clients and servers.
Two other points:

Quoted 3 lines

+ lockd would not restart itself in this case if there happened =20 to be
NFS mounts on that system

When testing, i find nfs restart will cause lockd restart. I find some codes which cause the lock stop when nfs stop.

At kernel 2.6.18, fs/lockd/svc.c
...
354 if (nlmsvc_users) {
355 if (--nlmsvc_users)
356 goto out;
357 } else
358 printk(KERN_WARNING "lockd_down: no users!
pid=3D%d\n", nlmsvc_pid);
...
366
367 kill_proc(nlmsvc_pid, SIGKILL, 1);
...

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

At kernel 2.6.18, fs/lockd/svc.c
...
344 if (nlmsvc_users) {
345 if (--nlmsvc_users)
346 goto out;
347 } else {
348 printk(KERN_ERR "lockd_down: no users! task=3D%=

p=20

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

\n",
349 nlmsvc_task);
350 BUG();
351 }
....
357 kthread_stop(nlmsvc_task);
358 svc_exit_thread(nlmsvc_rqst);
...

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

As above, when nlmsvc_users <=3D 1, the lockd will be killed.

Quoted 2 lines

+ lockd doesn't currently poke statd when it restarts to tell it t=

o

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 1 line

send reboot notifications, but it probably should

Yes, I agree with you. But now, when some reason cause lockd =20 restart but
statd not restart, locks which hold before will lost.

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

Maybe, the kernel should fix this.

What did you have in mind?

I think when lockd restart, statd should restart too and sent sm-=20
notify to other client.

Sending notifications is likely the correct thing to do if lockd is =20 restarted while there are active locks. A statd restart isn't =20 necessarily required to send reboot notifications, however. You can =20 do it with "sm-notify -f".

The problem with "sm-notify -f" is that it deletes the on-disk monitor = =20
list while statd is still running. This means the on-disk monitor =20 list and statd's in-memory monitor list will be out of sync. I seem =20 to recall that sm-notify is run by itself by cluster scripts, and that = =20
could be a real problem.

As implemented on RH, "service nfslock restart" will restart statd and = =20
force an sm-notify anyway, so no real harm done, but that's pretty =20 heavyweight (and requires that admins do "service nfs stop; service =20 nfslock restart; service nfs start" or something like that if they =20 want to get proper lock recovery).

A simple restart of statd (outside of the nfslock script) probably =20 won't be adequate, though. It will respect the sm-notify pidfile, and = =20
not send notifications when started up. I don't see a flag on statd =20 to force it to send notifications on restart (-N only sends =20 notifications; it doesn't also start the statd daemon).

In a perfect world, when lockd restarts, it would send up an =20 SM_SIMU_CRASH, and statd would do the right thing: if there are =20 monitored peers, it would send reboot notifications, and adjust it's =20 monitor list accordingly; if there were no monitored peers, it would =20 do nothing. Thus no statd restart would be needed.

Quoted 4 lines

But now, in kernel and nfs-uitls, it don't implemented.
As the communication style between lockd and statd, this is indeed =20
not easy to implement it.

Quoted 11 lines

So, I think it's should more easy to implement it through the =20
mechanism that exposes
the kernel's nlm_host cache via /sys you show me before.

Quoted 7 lines
Quoted 5 lines

I want to know when using cammond "service nfslock restart" =20 restart the
nfslock service(means restart statd and lockd), will the statd call sm-notify
to notify other client? Or don't?

Currently "service nfslock restart" always causes a notification to =

=20

Quoted 15 lines
Quoted 9 lines

be
sent. Since "service nfslock restart" causes lockd to drop its =20 locks (I
assume that's what that "killproc lockd" does) I guess we need to =20 force
reboot notifications here. (I still argue that removing the =20 pidfile in
the "start" case is not correct).

Quoted 5 lines

It appears that both the nfs and nfslock start up scripts do =20 something
to lockd (as well as the case when the number of NFS mounts goes to zero). However, only the nfslock script forces sm-notify to send notifications.

Quoted 5 lines

But, at RHLE5 and Fedora, when using cammond "service nfslock =20
restart" restart
the nfslock service, the lockd isn't shutdown and rmmod'd.

Is it a bug?

=46or the "no more NFS mounts case" and the server shutdown case, the =20 NFS client or server, both being in the kernel, call lockd_down enough = =20
times to make the user count go to zero. lockd.ko can be removed at =20 that point. I seem to recall there being some kind of automatic =20 mechanism for module removal after a period of zero module refcount. =20 In other words, lockd.ko is removed as a side effect, afaict.

The nfslock script doesn't stop either the kernel client or server =20 code, so it doesn't really cause a lockd_down call. But, nfslock does = =20
do a "killproc lockd". My assumption is that causes all locks to be =20 dropped. So it's not a cold restart of lockd, but we still =20 potentially lose a lot of lock state here.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

author
#8 lore raw

Hi Chuck,

Chuck Lever =E5=86=99=E9=81=93:

Quoted 1 line · Mi Jinlong

=2E..snip...

Quoted 11 lines
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Maybe, the kernel should fix this.

What did you have in mind?

I think when lockd restart, statd should restart too and sent
sm-notify to other client.

=20
Sending notifications is likely the correct thing to do if lockd is restarted while there are active locks. A statd restart isn't necessarily required to send reboot notifications, however. You can =

do

Quoted 3 lines

it with "sm-notify -f".
=20
The problem with "sm-notify -f" is that it deletes the on-disk monito=

r

Quoted 1 line

list while statd is still running. This means the on-disk monitor li=

st

Quoted 5 lines

and statd's in-memory monitor list will be out of sync. I seem to recall that sm-notify is run by itself by cluster scripts, and that could be a real problem.
=20
As implemented on RH, "service nfslock restart" will restart statd an=

d

Quoted 3 lines

force an sm-notify anyway, so no real harm done, but that's pretty heavyweight (and requires that admins do "service nfs stop; service nfslock restart; service nfs start" or something like that if they wa=

nt

Quoted 3 lines

to get proper lock recovery).
=20
A simple restart of statd (outside of the nfslock script) probably wo=

n't

Quoted 3 lines

be adequate, though. It will respect the sm-notify pidfile, and not send notifications when started up. I don't see a flag on statd to force it to send notifications on restart (-N only sends notification=

s;

Quoted 6 lines

it doesn't also start the statd daemon).
=20
In a perfect world, when lockd restarts, it would send up an SM_SIMU_CRASH, and statd would do the right thing: if there are monitored peers, it would send reboot notifications, and adjust it's monitor list accordingly; if there were no monitored peers, it would=

do

Quoted 1 line

nothing. Thus no statd restart would be needed.

Did this part have implemented at kernel?
I don't find the codes about SM_SIMU_CRASH.

IMO, if SM_SIMU_CRASH can work correctly, after lockd restart, the lockd and statd would work correctly (SYN) too. And the problem I asked above will not be happened.

Quoted 10 lines

=20

Quoted 7 lines

But now, in kernel and nfs-uitls, it don't implemented. As the communication style between lockd and statd, this is indeed
not easy to implement it.

So, I think it's should more easy to implement it through the
mechanism that exposes
the kernel's nlm_host cache via /sys you show me before.

=20

Quoted 1 line
Quoted 1 line
Quoted 1 line

I want to know when using cammond "service nfslock restart" restar=

t the

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

nfslock service(means restart statd and lockd), will the statd cal=

l

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

sm-notify
to notify other client? Or don't?

Currently "service nfslock restart" always causes a notification to=

be

Quoted 1 line
Quoted 1 line
Quoted 1 line

sent. Since "service nfslock restart" causes lockd to drop its loc=

ks (I

Quoted 1 line
Quoted 1 line
Quoted 1 line

assume that's what that "killproc lockd" does) I guess we need to f=

orce

Quoted 1 line
Quoted 1 line
Quoted 1 line

reboot notifications here. (I still argue that removing the pidfil=

e in

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

the "start" case is not correct).

It appears that both the nfs and nfslock start up scripts do someth=

ing

Quoted 11 lines
Quoted 9 lines
Quoted 3 lines

to lockd (as well as the case when the number of NFS mounts goes to zero). However, only the nfslock script forces sm-notify to send notifications.

But, at RHLE5 and Fedora, when using cammond "service nfslock
restart" restart
the nfslock service, the lockd isn't shutdown and rmmod'd.

Is it a bug?

=20
For the "no more NFS mounts case" and the server shutdown case, the N=

=46S

Quoted 1 line

client or server, both being in the kernel, call lockd_down enough ti=

mes

Quoted 6 lines

to make the user count go to zero. lockd.ko can be removed at that point. I seem to recall there being some kind of automatic mechanism for module removal after a period of zero module refcount. In other words, lockd.ko is removed as a side effect, afaict.
=20
The nfslock script doesn't stop either the kernel client or server co=

de,

Quoted 3 lines

so it doesn't really cause a lockd_down call. But, nfslock does do a "killproc lockd". My assumption is that causes all locks to be dropped. So it's not a cold restart of lockd, but we still potential=

ly

Quoted 1 line

lose a lot of lock state here.

Thanks for your explain!

Thanks,
Mi Jinlong

#9 lore raw
Quoted 52 lines · Mi Jinlong

Hi Chuck,

Chuck Lever =E5=86=99=E9=81=93:

Quoted 1 line · Mi Jinlong

...snip...

Quoted 38 lines
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Maybe, the kernel should fix this.

What did you have in mind?

I think when lockd restart, statd should restart too and sent sm-notify to other client.

Sending notifications is likely the correct thing to do if lockd is restarted while there are active locks. A statd restart isn't necessarily required to send reboot notifications, however. You =20 can do
it with "sm-notify -f".

The problem with "sm-notify -f" is that it deletes the on-disk =20 monitor
list while statd is still running. This means the on-disk monitor =20 list
and statd's in-memory monitor list will be out of sync. I seem to recall that sm-notify is run by itself by cluster scripts, and that could be a real problem.

As implemented on RH, "service nfslock restart" will restart statd =20 and
force an sm-notify anyway, so no real harm done, but that's pretty heavyweight (and requires that admins do "service nfs stop; service nfslock restart; service nfs start" or something like that if they =20 want
to get proper lock recovery).

A simple restart of statd (outside of the nfslock script) probably =20 won't
be adequate, though. It will respect the sm-notify pidfile, and not send notifications when started up. I don't see a flag on statd to force it to send notifications on restart (-N only sends =20 notifications;
it doesn't also start the statd daemon).

Quoted 6 lines

In a perfect world, when lockd restarts, it would send up an SM_SIMU_CRASH, and statd would do the right thing: if there are monitored peers, it would send reboot notifications, and adjust it's monitor list accordingly; if there were no monitored peers, it =20 would do
nothing. Thus no statd restart would be needed.

Quoted 2 lines

Did this part have implemented at kernel?
I don't find the codes about SM_SIMU_CRASH.

There isn't such code in the current kernel. I'm simply suggesting a =20 possible solution.

I'm coding up a patch that does this so we can experiment with it. =20 I'm a little worried that the current statd code won't do the right =20 thing, or even worse, it would crash. That would make it difficult to = =20
introduce such a patch without triggering regressions.

--=20
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

author
#10 lore raw

Chuck Lever =E5=86=99=E9=81=93:

Quoted 21 lines

=20
On Jan 18, 2010, at 5:51 AM, Mi Jinlong wrote:
=20

Quoted 18 lines

Hi Chuck,

Chuck Lever =E5=86=99=E9=81=93:

Quoted 1 line · Mi Jinlong

...snip...

Quoted 11 lines
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Maybe, the kernel should fix this.

What did you have in mind?

I think when lockd restart, statd should restart too and sent sm-notify to other client.

Sending notifications is likely the correct thing to do if lockd is restarted while there are active locks. A statd restart isn't necessarily required to send reboot notifications, however. You ca=

n do

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

it with "sm-notify -f".

The problem with "sm-notify -f" is that it deletes the on-disk moni=

tor

Quoted 1 line
Quoted 1 line
Quoted 1 line

list while statd is still running. This means the on-disk monitor =

list

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

and statd's in-memory monitor list will be out of sync. I seem to recall that sm-notify is run by itself by cluster scripts, and that could be a real problem.

As implemented on RH, "service nfslock restart" will restart statd =

and

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

force an sm-notify anyway, so no real harm done, but that's pretty heavyweight (and requires that admins do "service nfs stop; service nfslock restart; service nfs start" or something like that if they =

want

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

to get proper lock recovery).

A simple restart of statd (outside of the nfslock script) probably =

won't

Quoted 1 line
Quoted 1 line
Quoted 1 line

be adequate, though. It will respect the sm-notify pidfile, and no=

t

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

send notifications when started up. I don't see a flag on statd to force it to send notifications on restart (-N only sends notificati=

ons;

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

it doesn't also start the statd daemon).

In a perfect world, when lockd restarts, it would send up an SM_SIMU_CRASH, and statd would do the right thing: if there are monitored peers, it would send reboot notifications, and adjust it'=

s

Quoted 1 line
Quoted 1 line
Quoted 1 line

monitor list accordingly; if there were no monitored peers, it wou=

ld do

Quoted 9 lines
Quoted 4 lines
Quoted 1 line

nothing. Thus no statd restart would be needed.

Did this part have implemented at kernel?
I don't find the codes about SM_SIMU_CRASH.

=20
There isn't such code in the current kernel. I'm simply suggesting a possible solution.
=20
I'm coding up a patch that does this so we can experiment with it. I=

'm

Quoted 1 line

a little worried that the current statd code won't do the right thing=

,

Quoted 3 lines

or even worse, it would crash. That would make it difficult to introduce such a patch without triggering regressions. =20

That's why I don't find the codes! Thanks!
The statd code does have some rough, but, we can modify it when we ne=
ed.

Do you have implemented those codes? If have, please give me a copy,=20 or a git commit.

thanks,
Mi Jinlong