Found issue with BKL not held during call to nfs_asynch_unlink_release()
Subsystems (2)
We have found a problem with the BKL not being held during a call to nfs_asynch_unlink_release(), the call stack (this is an rpciod process) is:
nfs_async_unlink_release (called via tk_ops->rpc_release()) rpc_release_task__rpc_executerun_workqueueworker_thread
kthreadkernel_thread
I have been talking to Chuck Lever about this because I had noted one of his patches for removing the BKL added a spin lock to protect the nfs_deletes list used in fs/nfs/unlink.c. This patch happens to fix the specific problem we are seeing, but discussion with Chuck suggests there may be a larger problem.
Chuck pointed out that tk_ops is a recent addition, and that perhaps the think to do would be to add a lock_kernel() around the call to tk_ops->rpc_release().
When I had been investigating removing the BKL, I had noted that it seemed inconsistent whether the BKL was held for calls to rpc_execute(), and in some cases, the BKL was held ONLY for the call to rpc_execute(). My investigations found that rpciod does not hold the BKL when calling __rpc_execute(), and I also think some of the direct I/O code in fs/nfs/direct.c does not hold the BKL, or possible the asynch code (I didn't keep good notes - I had made a cursory examination, then noted for sure there were code paths where rpc_execute was called without the BKL, and naively assumed it was not needed, and didn't pursue exactly which calls did and did not hold the BKL - it seems that assumption may be wrong).
At first, I thought that using Chuck's patch to unlink.c would be a good fix since it would not introduce new lock_kernel() calls, however, if there may be other holes, it might be simpler to put the proper lock_kernel() calls in, and then cleanly remove all uses of the BKL when Chuck's patch set is adopted.
Frank Filz
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 11 lines · Frank Filz
We have found a problem with the BKL not being held during a call to
nfs_asynch_unlink_release(), the call stack (this is an rpciod process) is:
nfs_async_unlink_release(called viatk_ops->rpc_release())rpc_release_task__rpc_executerun_workqueueworker_thread
kthreadkernel_thread
Until we fix the unlink call (and possibly others), then that looks like a bug.
Quoted 9 lines
I have been talking to Chuck Lever about this because I had noted one of his patches for removing the BKL added a spin lock to protect the
nfs_deleteslist used infs/nfs/unlink.c. This patch happens to fix the specific problem we are seeing, but discussion with Chuck suggests there may be a larger problem.Chuck pointed out that
tk_opsis a recent addition, and that perhaps the think to do would be to add alock_kernel()around the call totk_ops->rpc_release().
Agreed.
Quoted 12 lines
When I had been investigating removing the BKL, I had noted that it seemed inconsistent whether the BKL was held for calls to
rpc_execute(), and in some cases, the BKL was held ONLY for the call torpc_execute(). My investigations found that rpciod does not hold the BKL when calling__rpc_execute(), and I also think some of the direct I/O code infs/nfs/direct.cdoes not hold the BKL, or possible the asynch code (I didn't keep good notes - I had made a cursory examination, then noted for sure there were code paths whererpc_executewas called without the BKL, and naively assumed it was not needed, and didn't pursue exactly which calls did and did not hold the BKL - it seems that assumption may be wrong).
Quoted 5 lines
At first, I thought that using Chuck's patch to
unlink.cwould be a good fix since it would not introduce newlock_kernel()calls, however, if there may be other holes, it might be simpler to put the properlock_kernel()calls in, and then cleanly remove all uses of the BKL when Chuck's patch set is adopted.
No. The RPC code itself should no longer require the BKL in order to function correctly. The callbacks into the NFS or lockd layers are the only routines that might care, and they only care for the duration of the callback itself (holding the BKL across the entire RPC call is unnecessary).
Cheers,
Trond
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 46 lines · Trond Myklebust
Quoted 11 lines · Frank Filz
We have found a problem with the BKL not being held during a call to
nfs_asynch_unlink_release(), the call stack (this is an rpciod process) is:
nfs_async_unlink_release(called viatk_ops->rpc_release())rpc_release_task__rpc_executerun_workqueueworker_thread
kthreadkernel_threadUntil we fix the unlink call (and possibly others), then that looks like a bug.
Quoted 9 lines
I have been talking to Chuck Lever about this because I had noted one of his patches for removing the BKL added a spin lock to protect the
nfs_deleteslist used infs/nfs/unlink.c. This patch happens to fix the specific problem we are seeing, but discussion with Chuck suggests there may be a larger problem.Chuck pointed out that
tk_opsis a recent addition, and that perhaps the think to do would be to add alock_kernel()around the call totk_ops->rpc_release().Agreed.
Quoted 12 lines
When I had been investigating removing the BKL, I had noted that it seemed inconsistent whether the BKL was held for calls to
rpc_execute(), and in some cases, the BKL was held ONLY for the call torpc_execute(). My investigations found that rpciod does not hold the BKL when calling__rpc_execute(), and I also think some of the direct I/O code infs/nfs/direct.cdoes not hold the BKL, or possible the asynch code (I didn't keep good notes - I had made a cursory examination, then noted for sure there were code paths whererpc_executewas called without the BKL, and naively assumed it was not needed, and didn't pursue exactly which calls did and did not hold the BKL - it seems that assumption may be wrong).Quoted 5 lines
At first, I thought that using Chuck's patch to
unlink.cwould be a good fix since it would not introduce newlock_kernel()calls, however, if there may be other holes, it might be simpler to put the properlock_kernel()calls in, and then cleanly remove all uses of the BKL when Chuck's patch set is adopted.
Quoted 5 lines
No. The RPC code itself should no longer require the BKL in order to function correctly. The callbacks into the NFS or lockd layers are thehe only routines that might care, and they only care for the duration of the callback itself (holding the BKL across the entire RPC call is unnecessary).
Ok. Should the BKL be acquired for all the tk_ops calls? I see these four:
net/sunrpc/clnt.c rpc_call_async 535 tk_ops->rpc_release(data);
net/sunrpc/sched.c rpc_prepare_task 568 task->tk_ops->rpc_call_prepare (task, task->tk_calldata);
net/sunrpc/sched.c rpc_exit_task 578 task->tk_ops->rpc_call_done(task, task->tk_calldata);
net/sunrpc/sched.c rpc_release_task 892 tk_ops->rpc_release(calldata);
On the surface, it looks like it may only be necessary for rpc_release.
Frank
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 4 lines · Frank Filz
Ok. Should the BKL be acquired for all the
tk_opscalls? I see these four:
net/sunrpc/clnt.crpc_call_async535tk_ops->rpc_release(data);
Yup. This needs a BKL wrapper.
Quoted 2 lines
net/sunrpc/sched.crpc_prepare_task568task->tk_ops->rpc_call_prepare(task,task->tk_calldata);
Nope. Should be covered by the BKL wrapper of task->tk_action() in __rpc_execute().
Quoted 2 lines
net/sunrpc/sched.crpc_exit_task578task->tk_ops->rpc_call_done(task, task->tk_calldata);
Nope. Should be covered by the BKL wrapper of task->tk_action() in __rpc_execute().
Quoted 1 line
net/sunrpc/sched.crpc_release_task892tk_ops->rpc_release(calldata);
Yup. This needs a BKL wrapper.
Note that this would allow us to immediately get rid of all the BKL wrappers around rpc_call_*() and friends.
Cheers,
Trond
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 17 lines · Trond Myklebust
Quoted 4 lines · Frank Filz
Ok. Should the BKL be acquired for all the
tk_opscalls? I see these four:
net/sunrpc/clnt.crpc_call_async535tk_ops->rpc_release(data);Yup. This needs a BKL wrapper.
Quoted 2 lines
net/sunrpc/sched.crpc_prepare_task568task->tk_ops->rpc_call_prepare(task,task->tk_calldata);Nope. Should be covered by the BKL wrapper of
task->tk_action()in__rpc_execute().Quoted 2 lines
net/sunrpc/sched.crpc_exit_task578task->tk_ops->rpc_call_done(task, task->tk_calldata);
Quoted 6 lines
Nope. Should be covered by the BKL wrapper of
task->tk_action()in__rpc_execute().Quoted 1 line
net/sunrpc/sched.crpc_release_task892tk_ops->rpc_release(calldata);Yup. This needs a BKL wrapper.
One more, in rpc_run_task()...
Quoted 2 lines
Note that this would allow us to immediately get rid of all the BKL wrappers around rpc_call_*() and friends.
Do you mean rpc_call_{a,}sync() or do you mean the tk_ops calls?
Signature · 11 lines
"We who cut mere stones must always be envisioning cathedrals" -- Quarry worker's creed ------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 18 lines · Chuck Lever
Quoted 17 lines · Trond Myklebust
Quoted 4 lines · Frank Filz
Ok. Should the BKL be acquired for all the
tk_opscalls? I see these four:
net/sunrpc/clnt.crpc_call_async535tk_ops->rpc_release(data);Yup. This needs a BKL wrapper.
Quoted 2 lines
net/sunrpc/sched.crpc_prepare_task568task->tk_ops->rpc_call_prepare(task,task->tk_calldata);Nope. Should be covered by the BKL wrapper of
task->tk_action()in__rpc_execute().Quoted 2 lines
net/sunrpc/sched.crpc_exit_task578task->tk_ops->rpc_call_done(task, task->tk_calldata);
Quoted 8 lines
Quoted 6 lines
Nope. Should be covered by the BKL wrapper of
task->tk_action()in__rpc_execute().Quoted 1 line
net/sunrpc/sched.crpc_release_task892tk_ops->rpc_release(calldata);Yup. This needs a BKL wrapper.
One more, in
rpc_run_task()...
Oops, missed that one. I'll code up and test a patch to add these.
Quoted 2 lines
Quoted 2 lines
Note that this would allow us to immediately get rid of all the BKL wrappers around rpc_call_*() and friends.
I'll also code up a separate patch or patch set to remove these.
Quoted 1 line
Do you mean rpc_call_{a,}sync() or do you mean the
tk_opscalls?
I assume rpc_call_async() and rpc_call_sync() along with rpc_execute().
Frank
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 1 line · Chuck Lever
One more, in
rpc_run_task()...
Yep.
Quoted 4 lines
Quoted 2 lines
Note that this would allow us to immediately get rid of all the BKL wrappers around rpc_call_*() and friends.
Do you mean rpc_call_{a,}sync() or do you mean the
tk_opscalls?
All direct calls to rpc_call_(a|)sync, rpc_run_task, and rpc_execute that are being wrapped. See for instance nfs_execute_write, nfs_execute_read, nfs_direct_read_schedule, nfs_direct_commit_schedule, nfs_direct_write_schedule.
Also see nfs_symlink_filler, nfs3_proc_readdir, nfs_proc_readdir.
Cheers,
Trond
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 9 lines · Trond Myklebust
Quoted 1 line · Chuck Lever
One more, in
rpc_run_task()...Yep.
Quoted 3 lines
Quoted 2 lines
Note that this would allow us to immediately get rid of all the BKL wrappers around rpc_call_*() and friends.
Quoted 8 lines
Quoted 1 line
Do you mean rpc_call_{a,}sync() or do you mean the
tk_opscalls?All direct calls to rpc_call_(a|)sync,
rpc_run_task, andrpc_executethat are being wrapped. See for instancenfs_execute_write,nfs_execute_read,nfs_direct_read_schedule,nfs_direct_commit_schedule,nfs_direct_write_schedule.Also see
nfs_symlink_filler,nfs3_proc_readdir,nfs_proc_readdir.
I looked over all of those. A few had a bit more than just a call to rpc inside the BKL, but I'm pretty sure they are all safe (references to the inode and such, which should be locked, or references to stack variables). I won't get a chance to test this until next week, but looks promising. This will definitely remove a bunch of lock_kernel() calls at the expense of one new one.
Frank
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs