autofs4 hang in 2.6.37-rc1
A 2.6.37-rc1 (f6614b7bb405a) setup hangs in autofs4:
INFO: task automount:10399 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
automount D 0000000000000000 0 10399 1 0x00000000
ffff88012a059da8 0000000000000082 0000000000000000 0000000000000000 ffff88012a058010 ffff88012a059fd8 0000000000011800 ffff8801290f2c40 ffff8801290f2fb0 ffff8801290f2fa8 0000000000011800 0000000000011800
Call Trace: [<ffffffff813a3c3c>] __mutex_lock_common+0x126/0x18b [<ffffffff813a3cb5>] __mutex_lock_slowpath+0x14/0x16 [<ffffffff813a3dbc>] mutex_lock+0x31/0x4b [<ffffffff811b17f7>] autofs4_root_ioctl+0x28/0x53 [<ffffffff810f5e5c>] do_vfs_ioctl+0x557/0x5bb [<ffffffff810eca65>] ? sys_newlstat+0x2d/0x38 [<ffffffff810f5f02>] sys_ioctl+0x42/0x65 [<ffffffff81002b42>] system_call_fastpath+0x16/0x1b
This is a simple /home automount. More info on request.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 18 lines · Avi Kivity
A 2.6.37-rc1 (
f6614b7bb405a) setup hangs in autofs4:INFO: task automount:10399 blocked for more than 120 seconds."echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
automount D 0000000000000000 0 10399 10x00000000
ffff88012a059da80000000000000082 0000000000000000 0000000000000000ffff88012a058010ffff88012a059fd80000000000011800ffff8801290f2c40ffff8801290f2fb0ffff8801290f2fa80000000000011800 0000000000011800Call Trace: [<ffffffff813a3c3c>] __mutex_lock_common+0x126/0x18b [<ffffffff813a3cb5>] __mutex_lock_slowpath+0x14/0x16 [<ffffffff813a3dbc>] mutex_lock+0x31/0x4b [<ffffffff811b17f7>] autofs4_root_ioctl+0x28/0x53 [<ffffffff810f5e5c>] do_vfs_ioctl+0x557/0x5bb [<ffffffff810eca65>] ? sys_newlstat+0x2d/0x38 [<ffffffff810f5f02>] sys_ioctl+0x42/0x65 [<ffffffff81002b42>] system_call_fastpath+0x16/0x1b
Quoted 2 lines
This is a simple /home automount. More info on request.
Likely culprit:
automount S ffff88012a28a680 0 399 1 0x00000000
ffff88012a07bd08 0000000000000082 0000000000000000 0000000000000000 ffff88012a07a010 ffff88012a07bfd8 0000000000011800 ffff88012693c260 ffff88012693c5d0 ffff88012693c5c8 0000000000011800 0000000000011800
Call Trace: [<ffffffff81056197>] ? prepare_to_wait+0x67/0x74 [<ffffffff811b23eb>] autofs4_wait+0x5a4/0x6d5 [<ffffffff81055f25>] ? autoremove_wake_function+0x0/0x34 [<ffffffff811b2ba5>] autofs4_do_expire_multi+0x5b/0xa3 [<ffffffff811b2c39>] autofs4_expire_multi+0x4c/0x54 [<ffffffff811b1750>] autofs4_root_ioctl_unlocked+0x23e/0x252 [<ffffffff811b1808>] autofs4_root_ioctl+0x39/0x53 [<ffffffff810f5e5c>] do_vfs_ioctl+0x557/0x5bb [<ffffffff810ca644>] ? remove_vma+0x6e/0x76 [<ffffffff810cb6a2>] ? do_munmap+0x31c/0x33e [<ffffffff810f5f02>] sys_ioctl+0x42/0x65 [<ffffffff81002b42>] system_call_fastpath+0x16/0x1b
Shouldn't we drop autofs4_ioctl_mutex while we wait?
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 19 lines · On Sunday 14 November 2010 14:51:04 Avi Kivity wrote:
automount S
ffff88012a28a6800 399 10x00000000
ffff88012a07bd080000000000000082 0000000000000000 0000000000000000ffff88012a07a010ffff88012a07bfd80000000000011800ffff88012693c260ffff88012693c5d0ffff88012693c5c80000000000011800 0000000000011800Call Trace: [<ffffffff81056197>] ? prepare_to_wait+0x67/0x74 [<ffffffff811b23eb>] autofs4_wait+0x5a4/0x6d5 [<ffffffff81055f25>] ? autoremove_wake_function+0x0/0x34 [<ffffffff811b2ba5>] autofs4_do_expire_multi+0x5b/0xa3 [<ffffffff811b2c39>] autofs4_expire_multi+0x4c/0x54 [<ffffffff811b1750>] autofs4_root_ioctl_unlocked+0x23e/0x252 [<ffffffff811b1808>] autofs4_root_ioctl+0x39/0x53 [<ffffffff810f5e5c>] do_vfs_ioctl+0x557/0x5bb [<ffffffff810ca644>] ? remove_vma+0x6e/0x76 [<ffffffff810cb6a2>] ? do_munmap+0x31c/0x33e [<ffffffff810f5f02>] sys_ioctl+0x42/0x65 [<ffffffff81002b42>] system_call_fastpath+0x16/0x1b
Quoted 1 line
Shouldn't we drop
autofs4_ioctl_mutexwhile we wait?
If the ioctl can sleep for multiple seconds, the mutex should indeed be dropped, and that would be safe because we used to do the same with the BKL.
The question is why this would sleep for more than 120 seconds.
Arnd
Quoted 20 lines · Arnd Bergmann
Quoted 19 lines · On Sunday 14 November 2010 14:51:04 Avi Kivity wrote:
automount S
ffff88012a28a6800 399 10x00000000
ffff88012a07bd080000000000000082 0000000000000000 0000000000000000ffff88012a07a010ffff88012a07bfd80000000000011800ffff88012693c260ffff88012693c5d0ffff88012693c5c80000000000011800 0000000000011800Call Trace: [<ffffffff81056197>] ? prepare_to_wait+0x67/0x74 [<ffffffff811b23eb>] autofs4_wait+0x5a4/0x6d5 [<ffffffff81055f25>] ? autoremove_wake_function+0x0/0x34 [<ffffffff811b2ba5>] autofs4_do_expire_multi+0x5b/0xa3 [<ffffffff811b2c39>] autofs4_expire_multi+0x4c/0x54 [<ffffffff811b1750>] autofs4_root_ioctl_unlocked+0x23e/0x252 [<ffffffff811b1808>] autofs4_root_ioctl+0x39/0x53 [<ffffffff810f5e5c>] do_vfs_ioctl+0x557/0x5bb [<ffffffff810ca644>] ? remove_vma+0x6e/0x76 [<ffffffff810cb6a2>] ? do_munmap+0x31c/0x33e [<ffffffff810f5f02>] sys_ioctl+0x42/0x65 [<ffffffff81002b42>] system_call_fastpath+0x16/0x1b
Quoted 8 lines
Quoted 1 line
Shouldn't we drop
autofs4_ioctl_mutexwhile we wait?If the ioctl can sleep for multiple seconds, the mutex should indeed be dropped, and that would be safe because we used to do the same with the BKL.
The question is why this would sleep for more than 120 seconds.
Let's fix first and ask questions later.
Signature · 1 line
error compiling committee.c: too many arguments to function
Ian Kent
maintainerQuoted 20 lines · Arnd Bergmann
Quoted 19 lines · On Sunday 14 November 2010 14:51:04 Avi Kivity wrote:
automount S
ffff88012a28a6800 399 10x00000000
ffff88012a07bd080000000000000082 0000000000000000 0000000000000000ffff88012a07a010ffff88012a07bfd80000000000011800ffff88012693c260ffff88012693c5d0ffff88012693c5c80000000000011800 0000000000011800Call Trace: [<ffffffff81056197>] ? prepare_to_wait+0x67/0x74 [<ffffffff811b23eb>] autofs4_wait+0x5a4/0x6d5 [<ffffffff81055f25>] ? autoremove_wake_function+0x0/0x34 [<ffffffff811b2ba5>] autofs4_do_expire_multi+0x5b/0xa3 [<ffffffff811b2c39>] autofs4_expire_multi+0x4c/0x54 [<ffffffff811b1750>] autofs4_root_ioctl_unlocked+0x23e/0x252 [<ffffffff811b1808>] autofs4_root_ioctl+0x39/0x53 [<ffffffff810f5e5c>] do_vfs_ioctl+0x557/0x5bb [<ffffffff810ca644>] ? remove_vma+0x6e/0x76 [<ffffffff810cb6a2>] ? do_munmap+0x31c/0x33e [<ffffffff810f5f02>] sys_ioctl+0x42/0x65 [<ffffffff81002b42>] system_call_fastpath+0x16/0x1b
Quoted 7 lines
Quoted 1 line
Shouldn't we drop
autofs4_ioctl_mutexwhile we wait?If the ioctl can sleep for multiple seconds, the mutex should indeed be dropped, and that would be safe because we used to do the same with the BKL.
The question is why this would sleep for more than 120 seconds.
umount against a server that isn't responding can easily take more than 2 minutes.
Ian
Ian Kent
maintainerQuoted 23 lines · Avi Kivity
Quoted 22 lines · Arnd Bergmann
Quoted 20 lines · On Sunday 14 November 2010 14:51:04 Avi Kivity wrote:
automount S
ffff88012a28a6800 399 10x00000000
ffff88012a07bd080000000000000082 0000000000000000 0000000000000000ffff88012a07a010ffff88012a07bfd80000000000011800ffff88012693c260ffff88012693c5d0ffff88012693c5c80000000000011800 0000000000011800Call Trace: [<ffffffff81056197>] ? prepare_to_wait+0x67/0x74 [<ffffffff811b23eb>] autofs4_wait+0x5a4/0x6d5 [<ffffffff81055f25>] ? autoremove_wake_function+0x0/0x34 [<ffffffff811b2ba5>] autofs4_do_expire_multi+0x5b/0xa3 [<ffffffff811b2c39>] autofs4_expire_multi+0x4c/0x54 [<ffffffff811b1750>] autofs4_root_ioctl_unlocked+0x23e/0x252 [<ffffffff811b1808>] autofs4_root_ioctl+0x39/0x53 [<ffffffff810f5e5c>] do_vfs_ioctl+0x557/0x5bb [<ffffffff810ca644>] ? remove_vma+0x6e/0x76 [<ffffffff810cb6a2>] ? do_munmap+0x31c/0x33e [<ffffffff810f5f02>] sys_ioctl+0x42/0x65 [<ffffffff81002b42>] system_call_fastpath+0x16/0x1bShouldn't we drop
autofs4_ioctl_mutexwhile we wait?
Quoted 8 lines
Quoted 6 lines
If the ioctl can sleep for multiple seconds, the mutex should indeed be dropped, and that would be safe because we used to do the same with the BKL.
The question is why this would sleep for more than 120 seconds.
Let's fix first and ask questions later.
You can't hold an exclusive mutex during an autofs expire because the daemon will start by calling the ioctl to check for a dentry to expire then call back to the daemon to perform the umount and wait for a status return (also an ioctl).
From memory the expire is the only ioctl that is sensitive to this deadlock.
So, either the mutex must be released while waiting for the status return or get rid of the autofs4_ioctl_mutex altogether.
Ian
Quoted 4 lines · On Monday 15 November 2010 02:45:33 Ian Kent wrote:
You can't hold an exclusive mutex during an autofs expire because the daemon will start by calling the ioctl to check for a dentry to expire then call back to the daemon to perform the umount and wait for a status return (also an ioctl).
Ok, I see. So it's my fault for not realizing that there are long blocking ioctls. I was under the assumption that all of these ioctl commands were simple non-blocking commands.
Quoted 5 lines
Quoted 1 line
From memory the expire is the only ioctl that is sensitive to this
deadlock.
So, either the mutex must be released while waiting for the status return or get rid of the
autofs4_ioctl_mutexaltogether.
Right. As I said with the original patch, I don't think the mutex is really needed, but using it seemed to be the safer alternative. It was in the sense that it guaranteed the breakage to be obvious rather than silent...
Ian, if you can prove that the lock is not needed, I think we shold just remove it.
Arnd
Quoted 9 lines · Ian Kent
Quoted 6 lines
If the ioctl can sleep for multiple seconds, the mutex should indeed be dropped, and that would be safe because we used to do the same with the BKL.
The question is why this would sleep for more than 120 seconds.
umount against a server that isn't responding can easily take more than 2 minutes.
Well, in my setup, the server should be responding.
Signature · 1 line
error compiling committee.c: too many arguments to function
Ian Kent
maintainerQuoted 10 lines · Arnd Bergmann
Quoted 4 lines · On Monday 15 November 2010 02:45:33 Ian Kent wrote:
You can't hold an exclusive mutex during an autofs expire because the daemon will start by calling the ioctl to check for a dentry to expire then call back to the daemon to perform the umount and wait for a status return (also an ioctl).
Ok, I see. So it's my fault for not realizing that there are long blocking ioctls. I was under the assumption that all of these ioctl commands were simple non-blocking commands.
This isn't anyone's fault (except maybe mine) because I'm the one most likely to realize it was a problem and didn't notice it. I've even been caught by this deadlock (when holding a singular lock) before when I tried to use .. ummm .. netlink (I think, not even sure what it's called any more) instead of an ioctl interface for the new autofs control interface.
Quoted 7 lines
Quoted 6 lines
Quoted 2 lines
From memory the expire is the only ioctl that is sensitive to this
deadlock.
So, either the mutex must be released while waiting for the status return or get rid of the
autofs4_ioctl_mutexaltogether.
Quoted 7 lines
Right. As I said with the original patch, I don't think the mutex is really needed, but using it seemed to be the safer alternative. It was in the sense that it guaranteed the breakage to be obvious rather than silent...
Ian, if you can prove that the lock is not needed, I think we shold just remove it.
I don't think I can prove it but I will have a long look at the code. I don't think it is needed and I expect I'll recommend it be removed.
Oh and btw ... please excuse this off-topic question.
In your recent commit 6e9624b8caec290d28b4c6d9ec75749df6372b87 regarding BKL removal you implied that blkdev_{get,put} shouldn't need the BLK. I'm working on a btrfs problem and one of the issues is a deadlock caused by the out of order acquisition of the BLK and the bdev->bd_mutex between these two functions. Clearly this isn't a problem from 2.6.36 but do you think it would be safe just to apply the hunks for blkdev_{get,put} from your commit to fix my problem for older an older kernel, say 2.6.35?
Ian
Quoted 5 lines · Ian Kent
Quoted 2 lines
Ian, if you can prove that the lock is not needed, I think we shold just remove it.
I don't think I can prove it but I will have a long look at the code. I don't think it is needed and I expect I'll recommend it be removed.
I've been running with the lock removed for a while with no ill effect. Of course it doesn't prove anything but at least it's a workaround for me.
Signature · 1 line
error compiling committee.c: too many arguments to function
Ian Kent
maintainerQuoted 9 lines · Avi Kivity
Quoted 5 lines · Ian Kent
Quoted 2 lines
Ian, if you can prove that the lock is not needed, I think we shold just remove it.
I don't think I can prove it but I will have a long look at the code. I don't think it is needed and I expect I'll recommend it be removed.
I've been running with the lock removed for a while with no ill effect. Of course it doesn't prove anything but at least it's a workaround for me.
Yeah, I tried pretty hard over quite a long time, with the expectation that the BKL would be removed, to try and make the code independent of it. At one point patched the kernel to use the unlocked ioctl entry point during some development testing and found only one fix that was needed, although a lot has changed since then too.
Ian
Ian Kent
maintainerQuoted 8 lines · Ian Kent
Quoted 7 lines · Avi Kivity
Quoted 5 lines · Ian Kent
Quoted 2 lines
Ian, if you can prove that the lock is not needed, I think we shold just remove it.
I don't think I can prove it but I will have a long look at the code. I don't think it is needed and I expect I'll recommend it be removed.
Quoted 8 lines
Quoted 2 lines
I've been running with the lock removed for a while with no ill effect. Of course it doesn't prove anything but at least it's a workaround for me.
Yeah, I tried pretty hard over quite a long time, with the expectation that the BKL would be removed, to try and make the code independent of it. At one point patched the kernel to use the unlocked ioctl entry point during some development testing and found only one fix that was needed, although a lot has changed since then too.
Hahaha, although as you say, I won't really know if there are races until I get people really hammering autofs. But, since that's were this is at maybe that's reason enough to remove it so we can get people to start applying pressure to the code so we find and fix any problems.
Quoted 3 lines
Ian
Ian Kent
maintainerQuoted 17 lines · Arnd Bergmann
Quoted 4 lines · On Monday 15 November 2010 02:45:33 Ian Kent wrote:
You can't hold an exclusive mutex during an autofs expire because the daemon will start by calling the ioctl to check for a dentry to expire then call back to the daemon to perform the umount and wait for a status return (also an ioctl).
Ok, I see. So it's my fault for not realizing that there are long blocking ioctls. I was under the assumption that all of these ioctl commands were simple non-blocking commands.
Quoted 5 lines
Quoted 1 line
From memory the expire is the only ioctl that is sensitive to this
deadlock.
So, either the mutex must be released while waiting for the status return or get rid of the
autofs4_ioctl_mutexaltogether.
Quoted 7 lines
Right. As I said with the original patch, I don't think the mutex is really needed, but using it seemed to be the safer alternative. It was in the sense that it guaranteed the breakage to be obvious rather than silent...
Ian, if you can prove that the lock is not needed, I think we shold just remove it.
I've looked through the old ioctl interface code and that looks fine.
But the important thing to notice is that the new ioctl interface (in fs/autofs4/dev-ioctl.c) used the unlocked_ioctl method since it was merged in 2.6.28 and that calls back into the core ioctl code for its major functionality. So the core function of the ioctl interface has been used without the BKL for quite a while now and has been heavily exercised in subsequent testing since the new ioctl interface has been in place.
I can't see any reason for keeping the autofs4_ioctl_mutex.
Ian
Quoted 11 lines · Avi Kivity
Quoted 8 lines · Ian Kent
Quoted 5 lines
If the ioctl can sleep for multiple seconds, the mutex should indeed be dropped, and that would be safe because we used to do the same with the BKL.
The question is why this would sleep for more than 120 seconds.
umount against a server that isn't responding can easily take more than 2 minutes.
Well, in my setup, the server should be responding.
Same as in mine. Server's up, but automount is locking up solid. trying to access the share, or links pointing to anything on it also lock up and can't be killed.
[ 5520.863130] INFO: task automount:1491 blocked for more than 120 seconds. [ 5520.863139] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 5520.863145] automount D 0000000000000001 0 1491 1 0x00000000 [ 5520.863157] ffff880137f5fdc8 0000000000000086 ffff880137f5fd48 0000000000000086 [ 5520.863172] ffff8800379efd78 ffff880137f5e000 ffff880137f5e000 ffff88013f1c06a0 [ 5520.863184] ffff880137f5e010 ffff88013f1c0960 ffff880137f5ffd8 ffff880137f5ffd8 [ 5520.863197] Call Trace: [ 5520.863214] [<ffffffff81038541>] ? get_parent_ip+0x11/0x50 [ 5520.863232] [<ffffffff813914ea>] __mutex_lock_slowpath+0x11a/0x1e0 [ 5520.863242] [<ffffffff813962fd>] ? sub_preempt_count+0x9d/0xd0 [ 5520.863251] [<ffffffff8139106e>] mutex_lock+0x1e/0x40 [ 5520.863264] [<ffffffffa03390a8>] autofs4_root_ioctl+0x38/0x70 [autofs4] [ 5520.863274] [<ffffffff8112d4e7>] do_vfs_ioctl+0x97/0x530 [ 5520.863283] [<ffffffff8104af2c>] ? sys_wait4+0xac/0xf0 [ 5520.863291] [<ffffffff8112d9ca>] sys_ioctl+0x4a/0x80 [ 5520.863302] [<ffffffff81002feb>] system_call_fastpath+0x16/0x1b
Got about 8+ of those in my dmesg right now. Another machine running .36 right now, is playing music off the same share, and this machine doesn't lock up at all when running .36 as well. Tried 2.6.37-rc1 and now -rc3.
Signature · 2 lines
Thomas Fjellstrom [email protected]
Quoted 9 lines · Ian Kent
But the important thing to notice is that the new ioctl interface (in
fs/autofs4/dev-ioctl.c) used theunlocked_ioctlmethod since it was merged in 2.6.28 and that calls back into the core ioctl code for its major functionality. So the core function of the ioctl interface has been used without the BKL for quite a while now and has been heavily exercised in subsequent testing since the new ioctl interface has been in place.I can't see any reason for keeping the
autofs4_ioctl_mutex.
Ok. Are you submitting a patch to remove it? Your reasoning absolutely makes sense and we need to fix this regression.
Arnd