5.0.5 non-expiring mounts
Subsystems (2)
I've pulled down the 5.0.5 tree and all patches.
I've set the logging level to debug and can see messages of expiring mounts in /var/log/messages...but when I check /etc/mtab or /proc/mounts or df, i can still see them there. Any ideas if this is a known issue or if I can give more info than below?
Linux sc-sim-208-144 2.6.29.4 #8 SMP Tue May 25 14:00:33 PDT 2010 x86_64 x86_64 x86_64 GNU/Linux
# automount -V
Linux automount version 5.0.5
Directories:
config dir: /etc/sysconfig
maps dir: /etc
modules dir: /usr/lib/autofsCompile options:
DISABLE_MOUNT_LOCKING ENABLE_IGNORE_BUSY_MOUNTS LIBXML2_WORKAROUND
Ian Kent
maintainerQuoted 6 lines · Philip Ong Jr.
I've pulled down the 5.0.5 tree and all patches.
I've set the logging level to debug and can see messages of expiring mounts in
/var/log/messages...butwhen I check/etc/mtabor/proc/mountsor df, i can still see them there. Any ideas if this is a known issue or if I can give more info than below?
You will need to post a full debug log somewhere.
Make sure that syslog is capturing daemon.* to the syslog so that you get all the debug log messages.
Quoted 20 lines
Linux sc-sim-208-144 2.6.29.4 #8 SMP Tue May 25 14:00:33 PDT 2010
x86_64x86_64x86_64GNU/Linux# automount -VLinux automount version 5.0.5
Directories:
config dir: /etc/sysconfig maps dir: /etc modules dir: /usr/lib/autofsCompile options:
DISABLE_MOUNT_LOCKINGENABLE_IGNORE_BUSY_MOUNTSLIBXML2_WORKAROUND_______________________________________________
autofs mailing list
[email protected]
http://linux.kernel.org/mailman/listinfo/autofs
Quoted 4 lines · Philip Ong Jr.
I've set the logging level to debug and can see messages of expiring mounts in
/var/log/messages...butwhen I check/etc/mtabor/proc/mountsor df, i can still see them there. Any ideas if this is a known issue or if I can give more info than below?
Out of curiosity... are these direct or indirect mounts? We're seeing an issue where our huge maps served from LDAP are seeing indirects not umounting while logging expires just fine, but the direct maps are working great. I believe a co-worker has replicated this exact same behavior with local files based maps too though, so pretty sure it's not really related to LDAP at all, but we've found using LDAP helped clear out a lot of old issues we had a few years ago related to the hashing bits.
I've even done network sniffing while doing a kill -USR1 and the automount daemon opens new sockets for each indirect (our homedirs) path it wants to expire, but there never is any actual network traffic sent across the wire. Even better, the sockets stay stuck in ESTABLISHED for a long time. Some more USR1/expires and/or HUPs can help clear is, or I just run a script that does an fuser then umount on unused homedirs and that flushes the ports out. When the ports pile up to >300 or so, you start getting NFS failures with syslog showing "couldn't read superblock."
Oh, and when I say huge maps.. I mean it. We're >10k direct mounts and I don't know how many indirect mounts, but it's even more then that.
Signature · 3 lines
Mike Marion-Unix SysAdmin/Staff IT Engineer-http://www.qualcomm.com Antonio: "It's no fair. Some men drink deep from the fountain of life, while Antonio takes one sip and it goes down the wrong pipe!"
Ian Kent
maintainerQuoted 24 lines · Mike Marion
Quoted 4 lines · Philip Ong Jr.
I've set the logging level to debug and can see messages of expiring mounts in
/var/log/messages...butwhen I check/etc/mtabor/proc/mountsor df, i can still see them there. Any ideas if this is a known issue or if I can give more info than below?Out of curiosity... are these direct or indirect mounts? We're seeing an issue where our huge maps served from LDAP are seeing indirects not umounting while logging expires just fine, but the direct maps are working great. I believe a co-worker has replicated this exact same behavior with local files based maps too though, so pretty sure it's not really related to LDAP at all, but we've found using LDAP helped clear out a lot of old issues we had a few years ago related to the hashing bits.
I've even done network sniffing while doing a kill -USR1 and the automount daemon opens new sockets for each indirect (our homedirs) path it wants to expire, but there never is any actual network traffic sent across the wire. Even better, the sockets stay stuck in ESTABLISHED for a long time. Some more USR1/expires and/or HUPs can help clear is, or I just run a script that does an fuser then umount on unused homedirs and that flushes the ports out. When the ports pile up to >300 or so, you start getting NFS failures with syslog showing "couldn't read superblock."
Quoted 2 lines
Oh, and when I say huge maps.. I mean it. We're >10k direct mounts and I don't know how many indirect mounts, but it's even more then that.
What kernel and source?
Ian
Ian Kent
maintainerQuoted 25 lines · Ian Kent
Quoted 24 lines · Mike Marion
Quoted 4 lines · Philip Ong Jr.
I've set the logging level to debug and can see messages of expiring mounts in
/var/log/messages...butwhen I check/etc/mtabor/proc/mountsor df, i can still see them there. Any ideas if this is a known issue or if I can give more info than below?Out of curiosity... are these direct or indirect mounts? We're seeing an issue where our huge maps served from LDAP are seeing indirects not umounting while logging expires just fine, but the direct maps are working great. I believe a co-worker has replicated this exact same behavior with local files based maps too though, so pretty sure it's not really related to LDAP at all, but we've found using LDAP helped clear out a lot of old issues we had a few years ago related to the hashing bits.
I've even done network sniffing while doing a kill -USR1 and the automount daemon opens new sockets for each indirect (our homedirs) path it wants to expire, but there never is any actual network traffic sent across the wire. Even better, the sockets stay stuck in ESTABLISHED for a long time. Some more USR1/expires and/or HUPs can help clear is, or I just run a script that does an fuser then umount on unused homedirs and that flushes the ports out. When the ports pile up to >300 or so, you start getting NFS failures with syslog showing "couldn't read superblock."
Quoted 4 lines
Quoted 2 lines
Oh, and when I say huge maps.. I mean it. We're >10k direct mounts and I don't know how many indirect mounts, but it's even more then that.
What kernel and source?
That is kernel revision and autofs revision?
Quoted 6 lines
Ian
_______________________________________________
autofs mailing list
[email protected]
http://linux.kernel.org/mailman/listinfo/autofs
Quoted 1 line · Ian Kent
That is kernel revision and autofs revision?
2.6.16.60-0.59.1 (Sles10 sp3 with an updated, but not bleeding edge, patch). autofs 5.0.5 with most of the patches up to a couple months ago. It's hard to get exacts because it's a PTF from Novell (we really pushed them to upgrade to 5.0.5) but it should be pretty much equal to the patch they just released for sle 11 sp1 that they're recommending as they default going forward.
Signature · 4 lines
Mike Marion-Unix SysAdmin/Staff IT Engineer-http://www.qualcomm.com Vir: "Londo, are you deliberatly trying to drive me insane?" Londo: "The Universe is already mad. Anything else would be redundant." ==> Babylon 5
Ian Kent
maintainerQuoted 9 lines · Mike Marion
Quoted 1 line · Ian Kent
That is kernel revision and autofs revision?
2.6.16.60-0.59.1 (Sles10 sp3 with an updated, but not bleeding edge, patch). autofs 5.0.5 with most of the patches up to a couple months ago. It's hard to get exacts because it's a PTF from Novell (we really pushed them to upgrade to 5.0.5) but it should be pretty much equal to the patch they just released for sle 11 sp1 that they're recommending as they default going forward.
That make it hard, as you know.
But I wouldn't mind spending a bit of time on it, if you can also.
Let's assume that it's a user space problem for now.
Could you post a debug log somewhere and the current sles patched source somewhere. If you could spend a little time trying to compare the patches in the sles package to the ones in the Rawhide autofs repo that would be a big help too.
Ian
Quoted 11 lines · Ian Kent
Quoted 9 lines · Mike Marion
Quoted 1 line · Ian Kent
That is kernel revision and autofs revision?
2.6.16.60-0.59.1 (Sles10 sp3 with an updated, but not bleeding edge, patch). autofs 5.0.5 with most of the patches up to a couple months ago. It's hard to get exacts because it's a PTF from Novell (we really pushed them to upgrade to 5.0.5) but it should be pretty much equal to the patch they just released for sle 11 sp1 that they're recommending as they default going forward.
Quoted 5 lines
That make it hard, as you know.
But I wouldn't mind spending a bit of time on it, if you can also.
Let's assume that it's a user space problem for now.
Here are the call traces for all automount processes on the kernel side:
PID: 4017 TASK: ffff8104202a9080 CPU: 4 COMMAND: "automount"
#0 [ffff81040426fc58] schedule at ffffffff802efc13 #1 [ffff81040426fd30] schedule_timeout at ffffffff802f0748 #2 [ffff81040426fd80] futex_wait_abstime at ffffffff8014b703 #3 [ffff81040426fe60] do_futex at ffffffff8014bb33 #4 [ffff81040426ff20] sys_futex at ffffffff8014c2ed #5 [ffff81040426ff80] system_call at ffffffff8010ae16
PID: 4018 TASK: ffff81041cefd810 CPU: 2 COMMAND: "automount"
#0 [ffff810404211d68] schedule at ffffffff802efc13 #1 [ffff810404211e40] schedule_timeout at ffffffff802f07b4 #2 [ffff810404211e90] futex_wait_abstime at ffffffff8014b703 #3 [ffff810404211f70] futex_wait_restart at ffffffff8014b7d2 #4 [ffff810404211f80] system_call at ffffffff8010ae16
PID: 4019 TASK: ffff810420ab8080 CPU: 5 COMMAND: "automount"
#0 [ffff810404213c58] schedule at ffffffff802efc13 #1 [ffff810404213d30] schedule_timeout at ffffffff802f0748 #2 [ffff810404213d80] futex_wait_abstime at ffffffff8014b703 #3 [ffff810404213e60] do_futex at ffffffff8014bb33 #4 [ffff810404213f20] sys_futex at ffffffff8014c2ed #5 [ffff810404213f80] system_call at ffffffff8010ae16
PID: 4022 TASK: ffff81042055b7d0 CPU: 5 COMMAND: "automount"
#0 [ffff810404c71c58] schedule at ffffffff802efc13 #1 [ffff810404c71d30] schedule_timeout at ffffffff802f0748 #2 [ffff810404c71d80] futex_wait_abstime at ffffffff8014b703 #3 [ffff810404c71e60] do_futex at ffffffff8014bb33 #4 [ffff810404c71f20] sys_futex at ffffffff8014c2ed #5 [ffff810404c71f80] system_call at ffffffff8010ae16
PID: 4025 TASK: ffff81041cefd080 CPU: 2 COMMAND: "automount"
#0 [ffff8104001d7da8] schedule at ffffffff802efc13 #1 [ffff8104001d7e80] schedule_timeout at ffffffff802f07b4 #2 [ffff8104001d7ed0] do_sys_poll at ffffffff80199526 #3 [ffff8104001d7f60] sys_poll at ffffffff80199640 #4 [ffff8104001d7f80] system_call at ffffffff8010ae16
PID: 4026 TASK: ffff81041e1bd080 CPU: 7 COMMAND: "automount"
#0 [ffff8104001f7c58] schedule at ffffffff802efc13 #1 [ffff8104001f7d30] schedule_timeout at ffffffff802f0748 #2 [ffff8104001f7d80] futex_wait_abstime at ffffffff8014b703 #3 [ffff8104001f7e60] do_futex at ffffffff8014bb33 #4 [ffff8104001f7f20] sys_futex at ffffffff8014c2ed #5 [ffff8104001f7f80] system_call at ffffffff8010ae16
PID: 6851 TASK: ffff81013c9187d0 CPU: 6 COMMAND: "automount"
#0 [ffff810304f97c58] schedule at ffffffff802efc13 #1 [ffff810304f97d30] schedule_timeout at ffffffff802f0748 #2 [ffff810304f97d80] futex_wait_abstime at ffffffff8014b703 #3 [ffff810304f97e60] do_futex at ffffffff8014bb33 #4 [ffff810304f97f20] sys_futex at ffffffff8014c2ed #5 [ffff810304f97f80] system_call at ffffffff8010ae16
PID: 7060 TASK: ffff810188ef5080 CPU: 4 COMMAND: "automount"
#0 [ffff810319ebdc58] schedule at ffffffff802efc13 #1 [ffff810319ebdd30] schedule_timeout at ffffffff802f0748 #2 [ffff810319ebdd80] futex_wait_abstime at ffffffff8014b703 #3 [ffff810319ebde60] do_futex at ffffffff8014bb33 #4 [ffff810319ebdf20] sys_futex at ffffffff8014c2ed #5 [ffff810319ebdf80] system_call at ffffffff8010ae16
PID: 20702 TASK: ffff810230be5080 CPU: 7 COMMAND: "automount"
#0 [ffff81026c80dc18] schedule at ffffffff802efc13
#1 [ffff81026c80dcf0] autofs4_wait at ffffffff8831e5b2 [autofs4] #2 [ffff81026c80deb0] autofs4_expire_multi at ffffffff8831ed06 [autofs4] #3 [ffff81026c80dee0] do_ioctl at ffffffff80198acd
#4 [ffff81026c80df00] vfs_ioctl at ffffffff80198d2b
#5 [ffff81026c80df40] sys_ioctl at ffffffff80198da5
#6 [ffff81026c80df80] system_call at ffffffff8010ae16
They look regular to me, do you see something suspect?
And here are the call traces from the user land daemon:
Thread 9 (Thread 4017):
#0 0x00002b56e465d6a8 in __lll_mutex_lock_wait () from /lib64/libpthread.so.0
#1 0x00002b56e46599fb in _L_mutex_lock_92 () from /lib64/libpthread.so.0
#2 0x00002b56e4659455 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00005555555746cd in master_mutex_lock () at master.c:49
#4 0x000055555555d260 in do_hup_signal (master=0x55555568d010, age=1296063258)
at automount.c:1276
#5 0x0000555555560bd3 in statemachine (arg=<value optimized out>)
at automount.c:1354#6 main (arg=<value optimized out>) at automount.c:2142
Thread 8 (Thread 20702):
#0 0x00002b56e4dd62a7 in brk () from /lib64/libc.so.6
#1 0x0000555555577dfe in expire (logopt=2, cmd=<value optimized out>, fd=21,
ioctlfd=21, path=0x55555569ca20 "/usr2", arg=0x41c27ef4)
at dev-ioctl-lib.c:657
#2 0x0000555555577ebe in ioctl_expire (logopt=21, ioctlfd=-1,
path=0x55555569ca20 "/usr2", when=0) at dev-ioctl-lib.c:701
#3 0x0000555555561e4e in expire_proc_indirect (arg=<value optimized out>)
at indirect.c:545
#4 0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0
#5 0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6
#6 0x0000000000000000 in ?? ()Thread 7 (Thread 7060):
#0 0x00002b56e465ac77 in pthread_rwlock_wrlock () from /lib64/libpthread.so.0 #1 0x00005555555752ea in master_source_writelock (entry=<value optimized out>)
at master.c:527
#2 0x0000555555575f8f in master_add_map_source (entry=0x5555556a10b0,
type=0x0, format=0x0, age=1296059657, argc=1, argv=<value optimized out>)
at master.c:191
#3 0x0000555555579ee3 in master_parse_entry (buffer=<value optimized out>,
default_timeout=86400, logging=<value optimized out>, age=1296059657)
at master_parse.y:823
#4 0x00002aaaaaab83fe in lookup_read_master (master=<value optimized out>,
age=1296059657, context=<value optimized out>) at lookup_ldap.c:1625
#5 0x0000555555569052 in do_read_master (master=0x55555568d010,
type=<value optimized out>, age=1296059657) at lookup.c:96
#6 0x000055555556aa3c in lookup_nss_read_master (master=0x55555568d010,
age=1296059657) at lookup.c:229
#7 0x0000555555575c28 in master_read_master (master=0x55555568d010,
age=1296059657, readall=1) at master.c:831
#8 0x000055555555d844 in do_read_master (arg=<value optimized out>)
at automount.c:1259#9 0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0 #10 0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6 #11 0x0000000000000000 in ?? ()
Thread 6 (Thread 6851):
#0 0x00002b56e465aa3d in pthread_rwlock_rdlock () from /lib64/libpthread.so.0 #1 0x000055555556deb6 in cache_readlock (mc=0x55555568e5b8) at cache.c:60 #2 0x000055555556baff in do_readmap (arg=<value optimized out>) at state.c:479 #3 0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0 #4 0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6 #5 0x0000000000000000 in ?? ()
Thread 5 (Thread 4026):
#0 0x00002b56e465d6a8 in __lll_mutex_lock_wait () from /lib64/libpthread.so.0
#1 0x00002b56e46599fb in _L_mutex_lock_92 () from /lib64/libpthread.so.0
#2 0x00002b56e4659455 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00005555555746cd in master_mutex_lock () at master.c:49
#4 0x0000555555560ff9 in handle_packet_missing_indirect (ap=0x55555569c940,
pkt=0x41823ec0) at indirect.c:808
#5 0x000055555555fa32 in handle_packet (ap=<value optimized out>)
at automount.c:1026
#6 handle_mounts (ap=<value optimized out>) at automount.c:1551
#7 0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0
#8 0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6
#9 0x0000000000000000 in ?? ()Thread 4 (Thread 4025):
#0 0x00002b56e4dd4b26 in fts_read () from /lib64/libc.so.6 #1 0x00007fff37314e80 in ?? () #2 0x0000000000000000 in ?? ()
Thread 3 (Thread 4022):
#0 0x00002b56e465ac77 in pthread_rwlock_wrlock () from /lib64/libpthread.so.0 #1 0x00005555555752ea in master_source_writelock (entry=<value optimized out>)
at master.c:527
#2 0x00005555555631af in handle_packet_missing_direct (ap=0x5555556a11c0,
pkt=0x40821ec0) at direct.c:1296
#3 0x000055555555fa1d in handle_packet (ap=<value optimized out>)
at automount.c:1029
#4 handle_mounts (ap=<value optimized out>) at automount.c:1551
#5 0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0
#6 0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6
#7 0x0000000000000000 in ?? ()Thread 2 (Thread 4019):
#0 0x00002b56e465b326 in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib64/libpthread.so.0
#1 0x000055555556c8b3 in st_queue_handler (arg=<value optimized out>)
at state.c:1104 #2 0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0 #3 0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6 #4 0x0000000000000000 in ?? ()
Thread 1 (Thread 4018):
#0 0x00002b56e465b517 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
from /lib64/libpthread.so.0
#1 0x0000555555573762 in alarm_handler (arg=<value optimized out>)
at alarm.c:206 #2 0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0 #3 0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6 #4 0x0000000000000000 in ?? ()
Quoted 2 lines
Could you post a debug log somewhere and the current sles patched source somewhere.
Unfortunately I don't have the full debug logs yet, so I'll leave this to Mike.
Quoted 3 lines
If you could spend a little time trying to compare the patches in the sles package to the ones in the Rawhide autofs repo that would be a big help too.
The package includes all upstream patches released so far (ie, up to autofs-5.0.5-fix-submount-shutdown-wait.patch), minus the following ones:
autofs-5.0.4 - use intr option as hosts mount default (configuration change) autofs-5.0.5 - add dump maps option (recent feature) autofs-5.0.5 - add external bind method (recent feature) autofs-5.0.5 - fix ext4 fsck at mount (no ext4 on SLES 10)
Thanks,
Leonardo
Ian Kent
maintainerQuoted 12 lines · Leonardo Chiquitto
Quoted 11 lines · Ian Kent
Quoted 9 lines · Mike Marion
Quoted 1 line · Ian Kent
That is kernel revision and autofs revision?
2.6.16.60-0.59.1 (Sles10 sp3 with an updated, but not bleeding edge, patch). autofs 5.0.5 with most of the patches up to a couple months ago. It's hard to get exacts because it's a PTF from Novell (we really pushed them to upgrade to 5.0.5) but it should be pretty much equal to the patch they just released for sle 11 sp1 that they're recommending as they default going forward.
Quoted 7 lines
Quoted 5 lines
That make it hard, as you know.
But I wouldn't mind spending a bit of time on it, if you can also.
Let's assume that it's a user space problem for now.
Here are the call traces for all automount processes on the kernel side:
I think it's a user space problem.
snip ...
Quoted 99 lines
And here are the call traces from the user land daemon:
Thread 9 (Thread 4017):
#0 0x00002b56e465d6a8 in __lll_mutex_lock_wait () from /lib64/libpthread.so.0 #1 0x00002b56e46599fb in _L_mutex_lock_92 () from /lib64/libpthread.so.0 #2 0x00002b56e4659455 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00005555555746cd in master_mutex_lock () at master.c:49 #4 0x000055555555d260 in do_hup_signal (master=0x55555568d010, age=1296063258) at automount.c:1276 #5 0x0000555555560bd3 in statemachine (arg=<value optimized out>) at automount.c:1354#6 main (arg=<value optimized out>) at automount.c:2142
Thread 8 (Thread 20702):
#00x00002b56e4dd62a7in brk () from/lib64/libc.so.6
#10x0000555555577dfein expire (logopt=2, cmd=<value optimized out>, fd=21,ioctlfd=21, path=0x55555569ca20 "/usr2", arg=0x41c27ef4) at dev-ioctl-lib.c:657 #2 0x0000555555577ebe in ioctl_expire (logopt=21, ioctlfd=-1, path=0x55555569ca20 "/usr2", when=0) at dev-ioctl-lib.c:701 #3 0x0000555555561e4e in expire_proc_indirect (arg=<value optimized out>) at indirect.c:545 #4 0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0 #5 0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6 #6 0x0000000000000000 in ?? ()Thread 7 (Thread 7060):
#00x00002b56e465ac77inpthread_rwlock_wrlock() from/lib64/libpthread.so.0#10x00005555555752eainmaster_source_writelock(entry=<value optimized out>)at master.c:527 #2 0x0000555555575f8f in master_add_map_source (entry=0x5555556a10b0, type=0x0, format=0x0, age=1296059657, argc=1, argv=<value optimized out>) at master.c:191 #3 0x0000555555579ee3 in master_parse_entry (buffer=<value optimized out>, default_timeout=86400, logging=<value optimized out>, age=1296059657) at master_parse.y:823 #4 0x00002aaaaaab83fe in lookup_read_master (master=<value optimized out>, age=1296059657, context=<value optimized out>) at lookup_ldap.c:1625 #5 0x0000555555569052 in do_read_master (master=0x55555568d010, type=<value optimized out>, age=1296059657) at lookup.c:96 #6 0x000055555556aa3c in lookup_nss_read_master (master=0x55555568d010, age=1296059657) at lookup.c:229 #7 0x0000555555575c28 in master_read_master (master=0x55555568d010, age=1296059657, readall=1) at master.c:831 #8 0x000055555555d844 in do_read_master (arg=<value optimized out>) at automount.c:1259#9
0x00002b56e4657193instart_thread() from/lib64/libpthread.so.0#100x00002b56e4ddcdfdin sysctl () from/lib64/libc.so.6#110x0000000000000000in ?? ()Thread 6 (Thread 6851):
#0 0x00002b56e465aa3d in pthread_rwlock_rdlock () from /lib64/libpthread.so.0 #1 0x000055555556deb6 in cache_readlock (mc=0x55555568e5b8) at cache.c:60 #2 0x000055555556baff in do_readmap (arg=<value optimized out>) at state.c:479 #3 0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0 #4 0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6 #5 0x0000000000000000 in ?? ()Thread 5 (Thread 4026):
#0 0x00002b56e465d6a8 in __lll_mutex_lock_wait () from /lib64/libpthread.so.0 #1 0x00002b56e46599fb in _L_mutex_lock_92 () from /lib64/libpthread.so.0 #2 0x00002b56e4659455 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00005555555746cd in master_mutex_lock () at master.c:49 #4 0x0000555555560ff9 in handle_packet_missing_indirect (ap=0x55555569c940, pkt=0x41823ec0) at indirect.c:808 #5 0x000055555555fa32 in handle_packet (ap=<value optimized out>) at automount.c:1026 #6 handle_mounts (ap=<value optimized out>) at automount.c:1551 #7 0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0 #8 0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6 #9 0x0000000000000000 in ?? ()Thread 4 (Thread 4025):
#0 0x00002b56e4dd4b26 in fts_read () from /lib64/libc.so.6 #1 0x00007fff37314e80 in ?? () #2 0x0000000000000000 in ?? ()Thread 3 (Thread 4022):
#00x00002b56e465ac77inpthread_rwlock_wrlock() from/lib64/libpthread.so.0#10x00005555555752eainmaster_source_writelock(entry=<value optimized out>)at master.c:527 #2 0x00005555555631af in handle_packet_missing_direct (ap=0x5555556a11c0, pkt=0x40821ec0) at direct.c:1296 #3 0x000055555555fa1d in handle_packet (ap=<value optimized out>) at automount.c:1029 #4 handle_mounts (ap=<value optimized out>) at automount.c:1551 #5 0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0 #6 0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6 #7 0x0000000000000000 in ?? ()Thread 2 (Thread 4019):
#00x00002b56e465b326in pthread_cond_wait@@GLIBC_2.3.2 ()
from/lib64/libpthread.so.0
#10x000055555556c8b3inst_queue_handler(arg=<value optimized out>)at state.c:1104 #2 0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0 #3 0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6 #4 0x0000000000000000 in ?? ()
Quoted 9 lines
Thread 1 (Thread 4018):
#00x00002b56e465b517in pthread_cond_timedwait@@GLIBC_2.3.2 ()
from/lib64/libpthread.so.0
#10x0000555555573762inalarm_handler(arg=<value optimized out>)at alarm.c:206 #2 0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0 #3 0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6 #4 0x0000000000000000 in ?? ()
It looks like a lock ordering bug to me and I think it's the same hang that Steve Thompson reported.
This will be hard to test I think, hopefully Mike or Steve will be able to use a couple of machines for testing, even so it may take a while before we know.
Could you prepare a test package for Mike using this patch please. It should apply OK but of course you'll need to discard or fix the CHANGELOG hunk.
I'll prepare a test package for Steve soon as I get time.
autofs-5.0.5 - fix out of order locking in readmap
From: Ian Kent <[email protected]>
When re-reading the master map and a mount lookup for a particular map entry arrives at the same time a map source is being added to the map entry and a map entry re-read is also initiated at the same time an out of order mutex locking bug which results in a hang.
---
CHANGELOG | 1 + lib/master.c | 8 ++------ 2 files changed, 3 insertions(+), 6 deletions(-)
CHANGELOG +1 -0
@@ -55,6 +55,7 @@ - add external bind method. - fix add simple bind auth. - add option to dump configured automount maps. +- fix out of order locking in readmap. 03/09/2009 autofs-5.0.5 -----------------------
lib/master.c +2 -6
@@ -611,8 +611,6 @@ struct master_mapent *master_find_mapent(struct master *master, const char *path {struct list_head *head, *p; -master_mutex_lock (); - head = &master->mounts;list_for_each (p, head) {struct master_mapent *entry;
@@ -625,8 +623,6 @@ struct master_mapent *master_find_mapent(struct master *master, const char *path } } -master_mutex_unlock (); -return NULL ; }
@@ -703,9 +699,7 @@ struct master_mapent *master_new_mapent(struct master *master, const char *path,void master_add_mapent (struct master *master,struct master_mapent *entry) { -master_mutex_lock ();list_add_tail (&entry->list, &master->mounts); -master_mutex_unlock ();return ; }
@@ -813,6 +807,7 @@ int master_read_master(struct master *master, time_t age, int readall)* We need to clear and re-populate the null map entry cache * before alowing anyone else to use it. */ +master_mutex_lock ();if (master->nc) {cache_writelock (master->nc); nc = master->nc;
@@ -831,6 +826,7 @@ int master_read_master(struct master *master, time_t age, int readall)master_init_scan ();lookup_nss_read_master (master, age);cache_unlock (nc); +master_mutex_unlock ();if (!master->read_fail)master_mount_mounts (master, age, readall);
Ian Kent
maintainerQuoted 5 lines · Ian Kent
Quoted 4 lines · Leonardo Chiquitto
Quoted 3 lines · Ian Kent
Quoted 2 lines · Mike Marion
On Mon, Feb 14, 2011 at 07:37:01PM -0800, Ian Kent wrote:
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
That is kernel revision and autofs revision?
2.6.16.60-0.59.1 (Sles10 sp3 with an updated, but not bleeding edge, patch). autofs 5.0.5 with most of the patches up to a couple months ago. It's hard to get exacts because it's a PTF from Novell (we really pushed them to upgrade to 5.0.5) but it should be pretty much equal to the patch they just released for sle 11 sp1 that they're recommending as they default going forward.
Now I'm confused?
I thought that Mike had mention he had seen hangs, similar to Steve, and this backtrace was an example of that. But the mail thread doesn't read like that and oddly enough I seem to have identified a locking problem from looking at the code based on the backtrace.
A backtrace generally doesn't do us any good when were trying to find an expire problem, the debug log is where we have to start on these.
Quoted 214 lines
Quoted 8 lines
Quoted 6 lines
That make it hard, as you know.
But I wouldn't mind spending a bit of time on it, if you can also.
Let's assume that it's a user space problem for now.
Here are the call traces for all automount processes on the kernel side:
I think it's a user space problem.
snip ...
Quoted 99 lines
And here are the call traces from the user land daemon:
Thread 9 (Thread 4017):
#0 0x00002b56e465d6a8 in __lll_mutex_lock_wait () from /lib64/libpthread.so.0 #1 0x00002b56e46599fb in _L_mutex_lock_92 () from /lib64/libpthread.so.0 #2 0x00002b56e4659455 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00005555555746cd in master_mutex_lock () at master.c:49 #4 0x000055555555d260 in do_hup_signal (master=0x55555568d010, age=1296063258) at automount.c:1276 #5 0x0000555555560bd3 in statemachine (arg=<value optimized out>) at automount.c:1354#6 main (arg=<value optimized out>) at automount.c:2142
Thread 8 (Thread 20702):
#00x00002b56e4dd62a7in brk () from/lib64/libc.so.6
#10x0000555555577dfein expire (logopt=2, cmd=<value optimized out>, fd=21,ioctlfd=21, path=0x55555569ca20 "/usr2", arg=0x41c27ef4) at dev-ioctl-lib.c:657 #2 0x0000555555577ebe in ioctl_expire (logopt=21, ioctlfd=-1, path=0x55555569ca20 "/usr2", when=0) at dev-ioctl-lib.c:701 #3 0x0000555555561e4e in expire_proc_indirect (arg=<value optimized out>) at indirect.c:545 #4 0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0 #5 0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6 #6 0x0000000000000000 in ?? ()Thread 7 (Thread 7060):
#00x00002b56e465ac77inpthread_rwlock_wrlock() from/lib64/libpthread.so.0#10x00005555555752eainmaster_source_writelock(entry=<value optimized out>)at master.c:527 #2 0x0000555555575f8f in master_add_map_source (entry=0x5555556a10b0, type=0x0, format=0x0, age=1296059657, argc=1, argv=<value optimized out>) at master.c:191 #3 0x0000555555579ee3 in master_parse_entry (buffer=<value optimized out>, default_timeout=86400, logging=<value optimized out>, age=1296059657) at master_parse.y:823 #4 0x00002aaaaaab83fe in lookup_read_master (master=<value optimized out>, age=1296059657, context=<value optimized out>) at lookup_ldap.c:1625 #5 0x0000555555569052 in do_read_master (master=0x55555568d010, type=<value optimized out>, age=1296059657) at lookup.c:96 #6 0x000055555556aa3c in lookup_nss_read_master (master=0x55555568d010, age=1296059657) at lookup.c:229 #7 0x0000555555575c28 in master_read_master (master=0x55555568d010, age=1296059657, readall=1) at master.c:831 #8 0x000055555555d844 in do_read_master (arg=<value optimized out>) at automount.c:1259#9
0x00002b56e4657193instart_thread() from/lib64/libpthread.so.0#100x00002b56e4ddcdfdin sysctl () from/lib64/libc.so.6#110x0000000000000000in ?? ()Thread 6 (Thread 6851):
#0 0x00002b56e465aa3d in pthread_rwlock_rdlock () from /lib64/libpthread.so.0 #1 0x000055555556deb6 in cache_readlock (mc=0x55555568e5b8) at cache.c:60 #2 0x000055555556baff in do_readmap (arg=<value optimized out>) at state.c:479 #3 0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0 #4 0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6 #5 0x0000000000000000 in ?? ()Thread 5 (Thread 4026):
#0 0x00002b56e465d6a8 in __lll_mutex_lock_wait () from /lib64/libpthread.so.0 #1 0x00002b56e46599fb in _L_mutex_lock_92 () from /lib64/libpthread.so.0 #2 0x00002b56e4659455 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00005555555746cd in master_mutex_lock () at master.c:49 #4 0x0000555555560ff9 in handle_packet_missing_indirect (ap=0x55555569c940, pkt=0x41823ec0) at indirect.c:808 #5 0x000055555555fa32 in handle_packet (ap=<value optimized out>) at automount.c:1026 #6 handle_mounts (ap=<value optimized out>) at automount.c:1551 #7 0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0 #8 0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6 #9 0x0000000000000000 in ?? ()Thread 4 (Thread 4025):
#0 0x00002b56e4dd4b26 in fts_read () from /lib64/libc.so.6 #1 0x00007fff37314e80 in ?? () #2 0x0000000000000000 in ?? ()Thread 3 (Thread 4022):
#00x00002b56e465ac77inpthread_rwlock_wrlock() from/lib64/libpthread.so.0#10x00005555555752eainmaster_source_writelock(entry=<value optimized out>)at master.c:527 #2 0x00005555555631af in handle_packet_missing_direct (ap=0x5555556a11c0, pkt=0x40821ec0) at direct.c:1296 #3 0x000055555555fa1d in handle_packet (ap=<value optimized out>) at automount.c:1029 #4 handle_mounts (ap=<value optimized out>) at automount.c:1551 #5 0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0 #6 0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6 #7 0x0000000000000000 in ?? ()Thread 2 (Thread 4019):
#00x00002b56e465b326in pthread_cond_wait@@GLIBC_2.3.2 ()
from/lib64/libpthread.so.0
#10x000055555556c8b3inst_queue_handler(arg=<value optimized out>)at state.c:1104 #2 0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0 #3 0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6 #4 0x0000000000000000 in ?? ()Quoted 9 lines
Thread 1 (Thread 4018):
#00x00002b56e465b517in pthread_cond_timedwait@@GLIBC_2.3.2 ()
from/lib64/libpthread.so.0
#10x0000555555573762inalarm_handler(arg=<value optimized out>)at alarm.c:206 #2 0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0 #3 0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6 #4 0x0000000000000000 in ?? ()It looks like a lock ordering bug to me and I think it's the same hang that Steve Thompson reported.
This will be hard to test I think, hopefully Mike or Steve will be able to use a couple of machines for testing, even so it may take a while before we know.
Could you prepare a test package for Mike using this patch please. It should apply OK but of course you'll need to discard or fix the CHANGELOG hunk.
I'll prepare a test package for Steve soon as I get time.
autofs-5.0.5 - fix out of order locking in readmap
From: Ian Kent <[email protected]>
When re-reading the master map and a mount lookup for a particular map entry arrives at the same time a map source is being added to the map entry and a map entry re-read is also initiated at the same time an out of order mutex locking bug which results in a hang.
---CHANGELOG | 1 + lib/master.c | 8 ++------ 2 files changed, 3 insertions(+), 6 deletions(-)CHANGELOG +1
-0@@ -55,6 +55,7 @@ - add external bind method. - fix add simple bind auth. - add option to dump configured automount maps. +- fix out of order locking in readmap. 03/09/2009 autofs-5.0.5 -----------------------lib/master.c +2
-6@@ -611,8 +611,6 @@ struct master_mapent *master_find_mapent(struct master *master, const char *path {struct list_head *head, *p; -master_mutex_lock (); - head = &master->mounts;list_for_each (p, head) {struct master_mapent *entry;@@ -625,8 +623,6 @@ struct master_mapent *master_find_mapent(struct master *master, const char *path } } -master_mutex_unlock (); -return NULL ; }@@ -703,9 +699,7 @@ struct master_mapent *master_new_mapent(struct master *master, const char *path,void master_add_mapent (struct master *master,struct master_mapent *entry) { -master_mutex_lock ();list_add_tail (&entry->list, &master->mounts); -master_mutex_unlock ();return ; }@@ -813,6 +807,7 @@ int master_read_master(struct master *master, time_t age, int readall)* We need to clear and re-populate the null map entry cache * before alowing anyone else to use it. */ +master_mutex_lock ();if (master->nc) {cache_writelock (master->nc); nc = master->nc;@@ -831,6 +826,7 @@ int master_read_master(struct master *master, time_t age, int readall)master_init_scan ();lookup_nss_read_master (master, age);cache_unlock (nc); +master_mutex_unlock ();if (!master->read_fail)master_mount_mounts (master, age, readall);
Quoted 137 lines · Ian Kent
Quoted 12 lines · Leonardo Chiquitto
Quoted 11 lines · Ian Kent
Quoted 9 lines · Mike Marion
Quoted 1 line · Ian Kent
That is kernel revision and autofs revision?
2.6.16.60-0.59.1 (Sles10 sp3 with an updated, but not bleeding edge, patch). autofs 5.0.5 with most of the patches up to a couple months ago. It's hard to get exacts because it's a PTF from Novell (we really pushed them to upgrade to 5.0.5) but it should be pretty much equal to the patch they just released for sle 11 sp1 that they're recommending as they default going forward.
Quoted 7 lines
Quoted 5 lines
That make it hard, as you know.
But I wouldn't mind spending a bit of time on it, if you can also.
Let's assume that it's a user space problem for now.
Here are the call traces for all automount processes on the kernel side:
I think it's a user space problem.
snip ...
Quoted 99 lines
And here are the call traces from the user land daemon:
Thread 9 (Thread 4017):
#0 0x00002b56e465d6a8 in__lll_mutex_lock_wait() from/lib64/libpthread.so.0#1 0x00002b56e46599fb in_L_mutex_lock_92() from/lib64/libpthread.so.0#2 0x00002b56e4659455 inpthread_mutex_lock() from/lib64/libpthread.so.0#3 0x00005555555746cd inmaster_mutex_lock() at master.c:49
#4 0x000055555555d260 indo_hup_signal(master=0x55555568d010, age=1296063258) at automount.c:1276
#5 0x0000555555560bd3 in statemachine (arg=<value optimized out>) at automount.c:1354
#6 main (arg=<value optimized out>) at automount.c:2142Thread 8 (Thread 20702):
#0 0x00002b56e4dd62a7 in brk () from/lib64/libc.so.6
#1 0x0000555555577dfe in expire (logopt=2, cmd=<value optimized out>, fd=21, ioctlfd=21, path=0x55555569ca20 "/usr2", arg=0x41c27ef4)
at dev-ioctl-lib.c:657
#2 0x0000555555577ebe inioctl_expire(logopt=21, ioctlfd=-1, path=0x55555569ca20 "/usr2", when=0) at dev-ioctl-lib.c:701 #3 0x0000555555561e4e inexpire_proc_indirect(arg=<value optimized out>) at indirect.c:545
#4 0x00002b56e4657193 instart_thread() from/lib64/libpthread.so.0#5 0x00002b56e4ddcdfd in sysctl () from/lib64/libc.so.6
#6 0x0000000000000000 in ?? ()Thread 7 (Thread 7060):
#0 0x00002b56e465ac77 inpthread_rwlock_wrlock() from/lib64/libpthread.so.0#1 0x00005555555752ea inmaster_source_writelock(entry=<value optimized out>) at master.c:527
#2 0x0000555555575f8f inmaster_add_map_source(entry=0x5555556a10b0, type=0x0, format=0x0, age=1296059657, argc=1, argv=<value optimized out>) at master.c:191
#3 0x0000555555579ee3 inmaster_parse_entry(buffer=<value optimized out>, default_timeout=86400, logging=<value optimized out>, age=1296059657) at master_parse.y:823
#4 0x00002aaaaaab83fe inlookup_read_master(master=<value optimized out>, age=1296059657, context=<value optimized out>) at lookup_ldap.c:1625 #5 0x0000555555569052 indo_read_master(master=0x55555568d010, type=<value optimized out>, age=1296059657) at lookup.c:96
#6 0x000055555556aa3c inlookup_nss_read_master(master=0x55555568d010, age=1296059657) at lookup.c:229
#7 0x0000555555575c28 inmaster_read_master(master=0x55555568d010, age=1296059657, readall=1) at master.c:831
#8 0x000055555555d844 indo_read_master(arg=<value optimized out>) at automount.c:1259
#9 0x00002b56e4657193 instart_thread() from/lib64/libpthread.so.0#100x00002b56e4ddcdfdin sysctl () from/lib64/libc.so.6
#110x0000000000000000in ?? ()Thread 6 (Thread 6851):
#0 0x00002b56e465aa3d inpthread_rwlock_rdlock() from/lib64/libpthread.so.0#1 0x000055555556deb6 incache_readlock(mc=0x55555568e5b8) at cache.c:60 #2 0x000055555556baff indo_readmap(arg=<value optimized out>) at state.c:479 #3 0x00002b56e4657193 instart_thread() from/lib64/libpthread.so.0#4 0x00002b56e4ddcdfd in sysctl () from/lib64/libc.so.6
#5 0x0000000000000000 in ?? ()Thread 5 (Thread 4026):
#0 0x00002b56e465d6a8 in__lll_mutex_lock_wait() from/lib64/libpthread.so.0#1 0x00002b56e46599fb in_L_mutex_lock_92() from/lib64/libpthread.so.0#2 0x00002b56e4659455 inpthread_mutex_lock() from/lib64/libpthread.so.0#3 0x00005555555746cd inmaster_mutex_lock() at master.c:49
#4 0x0000555555560ff9 inhandle_packet_missing_indirect(ap=0x55555569c940, pkt=0x41823ec0) at indirect.c:808
#5 0x000055555555fa32 inhandle_packet(ap=<value optimized out>) at automount.c:1026
#6 handle_mounts (ap=<value optimized out>) at automount.c:1551 #7 0x00002b56e4657193 instart_thread() from/lib64/libpthread.so.0#8 0x00002b56e4ddcdfd in sysctl () from/lib64/libc.so.6
#9 0x0000000000000000 in ?? ()Thread 4 (Thread 4025):
#0 0x00002b56e4dd4b26 infts_read() from/lib64/libc.so.6#1 0x00007fff37314e80 in ?? ()
#2 0x0000000000000000 in ?? ()Thread 3 (Thread 4022):
#0 0x00002b56e465ac77 inpthread_rwlock_wrlock() from/lib64/libpthread.so.0#1 0x00005555555752ea inmaster_source_writelock(entry=<value optimized out>) at master.c:527
#2 0x00005555555631af inhandle_packet_missing_direct(ap=0x5555556a11c0, pkt=0x40821ec0) at direct.c:1296
#3 0x000055555555fa1d inhandle_packet(ap=<value optimized out>) at automount.c:1029
#4 handle_mounts (ap=<value optimized out>) at automount.c:1551 #5 0x00002b56e4657193 instart_thread() from/lib64/libpthread.so.0#6 0x00002b56e4ddcdfd in sysctl () from/lib64/libc.so.6
#7 0x0000000000000000 in ?? ()Thread 2 (Thread 4019):
#0 0x00002b56e465b326 in pthread_cond_wait@@GLIBC_2.3.2 () from/lib64/libpthread.so.0
#1 0x000055555556c8b3 inst_queue_handler(arg=<value optimized out>) at state.c:1104
#2 0x00002b56e4657193 instart_thread() from/lib64/libpthread.so.0#3 0x00002b56e4ddcdfd in sysctl () from/lib64/libc.so.6#4 0x0000000000000000 in ?? ()Quoted 9 lines
Thread 1 (Thread 4018):
#0 0x00002b56e465b517 in pthread_cond_timedwait@@GLIBC_2.3.2 () from/lib64/libpthread.so.0
#1 0x0000555555573762 inalarm_handler(arg=<value optimized out>) at alarm.c:206
#2 0x00002b56e4657193 instart_thread() from/lib64/libpthread.so.0#3 0x00002b56e4ddcdfd in sysctl () from/lib64/libc.so.6#4 0x0000000000000000 in ?? ()It looks like a lock ordering bug to me and I think it's the same hang that Steve Thompson reported.
Quoted 7 lines
This will be hard to test I think, hopefully Mike or Steve will be able to use a couple of machines for testing, even so it may take a while before we know.
Could you prepare a test package for Mike using this patch please. It should apply OK but of course you'll need to discard or fix the CHANGELOG hunk.
Ian, thanks for the patch. I built the test package and forwarded it to Mike.
Quoted 1 line
I'll prepare a test package for Steve soon as I get time.
I took the liberty of building a test package for CentOS and RHEL too. I used autofs-5.0.1-0.rc2.143.el5_5.6.src.rpm as base and added your patch. The packages are now available at:
http://download.opensuse.org/repositories/home:/leonardocf:/autofs/
Unfortunately I failed to persuade the Build Service to keep the package release number, so Steve might need to pass --oldpackage to rpm.
Thanks,
Leonardo
Quoted 14 lines · Ian Kent
Quoted 5 lines · Ian Kent
Quoted 4 lines · Leonardo Chiquitto
Quoted 3 lines · Ian Kent
Quoted 2 lines · Mike Marion
On Mon, Feb 14, 2011 at 07:37:01PM -0800, Ian Kent wrote:
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
That is kernel revision and autofs revision?
2.6.16.60-0.59.1 (Sles10 sp3 with an updated, but not bleeding edge, patch). autofs 5.0.5 with most of the patches up to a couple months ago. It's hard to get exacts because it's a PTF from Novell (we really pushed them to upgrade to 5.0.5) but it should be pretty much equal to the patch they just released for sle 11 sp1 that they're recommending as they default going forward.
Quoted 4 lines
Now I'm confused?
I thought that Mike had mention he had seen hangs, similar to Steve, and this backtrace was an example of that.
This is right. The backtrace I posted was sent to me as an example of a hang.
Quoted 3 lines
But the mail thread doesn't read
like that and oddly enough I seem to have identified a locking problem from looking at the code based on the backtrace.
I'm sorry that we ended up hijacking the "non-expiring mounts" thread.
Leonardo
Quoted 215 lines · Ian Kent
Quoted 12 lines · Leonardo Chiquitto
Quoted 11 lines · Ian Kent
Quoted 9 lines · Mike Marion
Quoted 1 line · Ian Kent
That is kernel revision and autofs revision?
2.6.16.60-0.59.1 (Sles10 sp3 with an updated, but not bleeding edge, patch). autofs 5.0.5 with most of the patches up to a couple months ago. It's hard to get exacts because it's a PTF from Novell (we really pushed them to upgrade to 5.0.5) but it should be pretty much equal to the patch they just released for sle 11 sp1 that they're recommending as they default going forward.
Quoted 7 lines
Quoted 5 lines
That make it hard, as you know.
But I wouldn't mind spending a bit of time on it, if you can also.
Let's assume that it's a user space problem for now.
Here are the call traces for all automount processes on the kernel side:
I think it's a user space problem.
snip ...
Quoted 99 lines
And here are the call traces from the user land daemon:
Thread 9 (Thread 4017):
#0 0x00002b56e465d6a8 in__lll_mutex_lock_wait() from/lib64/libpthread.so.0#1 0x00002b56e46599fb in_L_mutex_lock_92() from/lib64/libpthread.so.0#2 0x00002b56e4659455 inpthread_mutex_lock() from/lib64/libpthread.so.0#3 0x00005555555746cd inmaster_mutex_lock() at master.c:49
#4 0x000055555555d260 indo_hup_signal(master=0x55555568d010, age=1296063258) at automount.c:1276
#5 0x0000555555560bd3 in statemachine (arg=<value optimized out>) at automount.c:1354
#6 main (arg=<value optimized out>) at automount.c:2142Thread 8 (Thread 20702):
#0 0x00002b56e4dd62a7 in brk () from/lib64/libc.so.6
#1 0x0000555555577dfe in expire (logopt=2, cmd=<value optimized out>, fd=21, ioctlfd=21, path=0x55555569ca20 "/usr2", arg=0x41c27ef4)
at dev-ioctl-lib.c:657
#2 0x0000555555577ebe inioctl_expire(logopt=21, ioctlfd=-1, path=0x55555569ca20 "/usr2", when=0) at dev-ioctl-lib.c:701 #3 0x0000555555561e4e inexpire_proc_indirect(arg=<value optimized out>) at indirect.c:545
#4 0x00002b56e4657193 instart_thread() from/lib64/libpthread.so.0#5 0x00002b56e4ddcdfd in sysctl () from/lib64/libc.so.6
#6 0x0000000000000000 in ?? ()Thread 7 (Thread 7060):
#0 0x00002b56e465ac77 inpthread_rwlock_wrlock() from/lib64/libpthread.so.0#1 0x00005555555752ea inmaster_source_writelock(entry=<value optimized out>) at master.c:527
#2 0x0000555555575f8f inmaster_add_map_source(entry=0x5555556a10b0, type=0x0, format=0x0, age=1296059657, argc=1, argv=<value optimized out>) at master.c:191
#3 0x0000555555579ee3 inmaster_parse_entry(buffer=<value optimized out>, default_timeout=86400, logging=<value optimized out>, age=1296059657) at master_parse.y:823
#4 0x00002aaaaaab83fe inlookup_read_master(master=<value optimized out>, age=1296059657, context=<value optimized out>) at lookup_ldap.c:1625 #5 0x0000555555569052 indo_read_master(master=0x55555568d010, type=<value optimized out>, age=1296059657) at lookup.c:96
#6 0x000055555556aa3c inlookup_nss_read_master(master=0x55555568d010, age=1296059657) at lookup.c:229
#7 0x0000555555575c28 inmaster_read_master(master=0x55555568d010, age=1296059657, readall=1) at master.c:831
#8 0x000055555555d844 indo_read_master(arg=<value optimized out>) at automount.c:1259
#9 0x00002b56e4657193 instart_thread() from/lib64/libpthread.so.0#100x00002b56e4ddcdfdin sysctl () from/lib64/libc.so.6
#110x0000000000000000in ?? ()Thread 6 (Thread 6851):
#0 0x00002b56e465aa3d inpthread_rwlock_rdlock() from/lib64/libpthread.so.0#1 0x000055555556deb6 incache_readlock(mc=0x55555568e5b8) at cache.c:60 #2 0x000055555556baff indo_readmap(arg=<value optimized out>) at state.c:479 #3 0x00002b56e4657193 instart_thread() from/lib64/libpthread.so.0#4 0x00002b56e4ddcdfd in sysctl () from/lib64/libc.so.6
#5 0x0000000000000000 in ?? ()Thread 5 (Thread 4026):
#0 0x00002b56e465d6a8 in__lll_mutex_lock_wait() from/lib64/libpthread.so.0#1 0x00002b56e46599fb in_L_mutex_lock_92() from/lib64/libpthread.so.0#2 0x00002b56e4659455 inpthread_mutex_lock() from/lib64/libpthread.so.0#3 0x00005555555746cd inmaster_mutex_lock() at master.c:49
#4 0x0000555555560ff9 inhandle_packet_missing_indirect(ap=0x55555569c940, pkt=0x41823ec0) at indirect.c:808
#5 0x000055555555fa32 inhandle_packet(ap=<value optimized out>) at automount.c:1026
#6 handle_mounts (ap=<value optimized out>) at automount.c:1551 #7 0x00002b56e4657193 instart_thread() from/lib64/libpthread.so.0#8 0x00002b56e4ddcdfd in sysctl () from/lib64/libc.so.6
#9 0x0000000000000000 in ?? ()Thread 4 (Thread 4025):
#0 0x00002b56e4dd4b26 infts_read() from/lib64/libc.so.6#1 0x00007fff37314e80 in ?? ()
#2 0x0000000000000000 in ?? ()Thread 3 (Thread 4022):
#0 0x00002b56e465ac77 inpthread_rwlock_wrlock() from/lib64/libpthread.so.0#1 0x00005555555752ea inmaster_source_writelock(entry=<value optimized out>) at master.c:527
#2 0x00005555555631af inhandle_packet_missing_direct(ap=0x5555556a11c0, pkt=0x40821ec0) at direct.c:1296
#3 0x000055555555fa1d inhandle_packet(ap=<value optimized out>) at automount.c:1029
#4 handle_mounts (ap=<value optimized out>) at automount.c:1551 #5 0x00002b56e4657193 instart_thread() from/lib64/libpthread.so.0#6 0x00002b56e4ddcdfd in sysctl () from/lib64/libc.so.6
#7 0x0000000000000000 in ?? ()Thread 2 (Thread 4019):
#0 0x00002b56e465b326 in pthread_cond_wait@@GLIBC_2.3.2 () from/lib64/libpthread.so.0
#1 0x000055555556c8b3 inst_queue_handler(arg=<value optimized out>) at state.c:1104
#2 0x00002b56e4657193 instart_thread() from/lib64/libpthread.so.0#3 0x00002b56e4ddcdfd in sysctl () from/lib64/libc.so.6#4 0x0000000000000000 in ?? ()Quoted 9 lines
Thread 1 (Thread 4018):
#0 0x00002b56e465b517 in pthread_cond_timedwait@@GLIBC_2.3.2 () from/lib64/libpthread.so.0
#1 0x0000555555573762 inalarm_handler(arg=<value optimized out>) at alarm.c:206
#2 0x00002b56e4657193 instart_thread() from/lib64/libpthread.so.0#3 0x00002b56e4ddcdfd in sysctl () from/lib64/libc.so.6#4 0x0000000000000000 in ?? ()It looks like a lock ordering bug to me and I think it's the same hang that Steve Thompson reported.
This will be hard to test I think, hopefully Mike or Steve will be able to use a couple of machines for testing, even so it may take a while before we know.
Could you prepare a test package for Mike using this patch please. It should apply OK but of course you'll need to discard or fix the CHANGELOG hunk.
I'll prepare a test package for Steve soon as I get time.
autofs-5.0.5 - fix out of order locking in readmap
From: Ian Kent <[email protected]>
When re-reading the master map and a mount lookup for a particular map entry arrives at the same time a map source is being added to the map entry and a map entry re-read is also initiated at the same time an out of order mutex locking bug which results in a hang.
---CHANGELOG | 1 +
lib/master.c | 8 ++------2 files changed, 3 insertions(+), 6 deletions(-)CHANGELOG +0
-0@@ -55,6 +55,7 @@- add external bind method.
- fix add simple bind auth.
- add option to dump configured automount maps.
+- fix out of order locking in readmap.03/09/2009 autofs-5.0.5
-----------------------lib/master.c +1
-6@@ -611,8 +611,6 @@ struct master_mapent *master_find_mapent(struct master *master, const char *path { struct list_head *head, *p; - master_mutex_lock(); - head = &master->mounts; list_for_each(p, head) { struct master_mapent *entry;@@ -625,8 +623,6 @@ struct master_mapent *master_find_mapent(struct master *master, const char *path } } - master_mutex_unlock(); - return NULL; }@@ -703,9 +699,7 @@ struct master_mapent *master_new_mapent(struct master *master, const char *path, void master_add_mapent(struct master *master, struct master_mapent *entry) { - master_mutex_lock(); list_add_tail(&entry->list, &master->mounts); - master_mutex_unlock(); return; }@@ -813,6 +807,7 @@ int master_read_master(struct master *master, time_t age, int readall)* We need to clear and re-populate the null map entry cache * before alowing anyone else to use it. */ + master_mutex_lock(); if (master->nc) { cache_writelock(master->nc); nc = master->nc;
lib/master.c int master_read_master(struct master *master, time_t age, int readall) · line 832
Quoted 7 lines
lib/master.c +1
-0@@ -831,6 +826,7 @@ int master_read_master(struct master *master, time_t age, int readall) master_init_scan(); lookup_nss_read_master(master, age); cache_unlock(nc); + master_mutex_unlock(); if (!master->read_fail) master_mount_mounts(master, age, readall);
Hello Ian,
I just got a reply confirming that the hangs are no longer reproducible with this
patch applied. Please, do you think you can consider it for upstream inclusion?
Thanks,
Leonardo
Ian Kent
maintainerQuoted 221 lines · Leonardo Chiquitto
Quoted 220 lines · Ian Kent
Quoted 12 lines · Leonardo Chiquitto
Quoted 11 lines · Ian Kent
Quoted 9 lines · Mike Marion
Quoted 1 line · Ian Kent
That is kernel revision and autofs revision?
2.6.16.60-0.59.1 (Sles10 sp3 with an updated, but not bleeding edge, patch). autofs 5.0.5 with most of the patches up to a couple months ago. It's hard to get exacts because it's a PTF from Novell (we really pushed them to upgrade to 5.0.5) but it should be pretty much equal to the patch they just released for sle 11 sp1 that they're recommending as they default going forward.
Quoted 7 lines
Quoted 5 lines
That make it hard, as you know.
But I wouldn't mind spending a bit of time on it, if you can also.
Let's assume that it's a user space problem for now.
Here are the call traces for all automount processes on the kernel side:
I think it's a user space problem.
snip ...
Quoted 99 lines
And here are the call traces from the user land daemon:
Thread 9 (Thread 4017):
#0 0x00002b56e465d6a8 in __lll_mutex_lock_wait () from /lib64/libpthread.so.0 #1 0x00002b56e46599fb in _L_mutex_lock_92 () from /lib64/libpthread.so.0 #2 0x00002b56e4659455 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00005555555746cd in master_mutex_lock () at master.c:49 #4 0x000055555555d260 in do_hup_signal (master=0x55555568d010, age=1296063258) at automount.c:1276 #5 0x0000555555560bd3 in statemachine (arg=<value optimized out>) at automount.c:1354#6 main (arg=<value optimized out>) at automount.c:2142
Thread 8 (Thread 20702):
#00x00002b56e4dd62a7in brk () from/lib64/libc.so.6
#10x0000555555577dfein expire (logopt=2, cmd=<value optimized out>, fd=21,ioctlfd=21, path=0x55555569ca20 "/usr2", arg=0x41c27ef4) at dev-ioctl-lib.c:657 #2 0x0000555555577ebe in ioctl_expire (logopt=21, ioctlfd=-1, path=0x55555569ca20 "/usr2", when=0) at dev-ioctl-lib.c:701 #3 0x0000555555561e4e in expire_proc_indirect (arg=<value optimized out>) at indirect.c:545 #4 0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0 #5 0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6 #6 0x0000000000000000 in ?? ()Thread 7 (Thread 7060):
#00x00002b56e465ac77inpthread_rwlock_wrlock() from/lib64/libpthread.so.0#10x00005555555752eainmaster_source_writelock(entry=<value optimized out>)at master.c:527 #2 0x0000555555575f8f in master_add_map_source (entry=0x5555556a10b0, type=0x0, format=0x0, age=1296059657, argc=1, argv=<value optimized out>) at master.c:191 #3 0x0000555555579ee3 in master_parse_entry (buffer=<value optimized out>, default_timeout=86400, logging=<value optimized out>, age=1296059657) at master_parse.y:823 #4 0x00002aaaaaab83fe in lookup_read_master (master=<value optimized out>, age=1296059657, context=<value optimized out>) at lookup_ldap.c:1625 #5 0x0000555555569052 in do_read_master (master=0x55555568d010, type=<value optimized out>, age=1296059657) at lookup.c:96 #6 0x000055555556aa3c in lookup_nss_read_master (master=0x55555568d010, age=1296059657) at lookup.c:229 #7 0x0000555555575c28 in master_read_master (master=0x55555568d010, age=1296059657, readall=1) at master.c:831 #8 0x000055555555d844 in do_read_master (arg=<value optimized out>) at automount.c:1259#9
0x00002b56e4657193instart_thread() from/lib64/libpthread.so.0#100x00002b56e4ddcdfdin sysctl () from/lib64/libc.so.6#110x0000000000000000in ?? ()Thread 6 (Thread 6851):
#0 0x00002b56e465aa3d in pthread_rwlock_rdlock () from /lib64/libpthread.so.0 #1 0x000055555556deb6 in cache_readlock (mc=0x55555568e5b8) at cache.c:60 #2 0x000055555556baff in do_readmap (arg=<value optimized out>) at state.c:479 #3 0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0 #4 0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6 #5 0x0000000000000000 in ?? ()Thread 5 (Thread 4026):
#0 0x00002b56e465d6a8 in __lll_mutex_lock_wait () from /lib64/libpthread.so.0 #1 0x00002b56e46599fb in _L_mutex_lock_92 () from /lib64/libpthread.so.0 #2 0x00002b56e4659455 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00005555555746cd in master_mutex_lock () at master.c:49 #4 0x0000555555560ff9 in handle_packet_missing_indirect (ap=0x55555569c940, pkt=0x41823ec0) at indirect.c:808 #5 0x000055555555fa32 in handle_packet (ap=<value optimized out>) at automount.c:1026 #6 handle_mounts (ap=<value optimized out>) at automount.c:1551 #7 0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0 #8 0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6 #9 0x0000000000000000 in ?? ()Thread 4 (Thread 4025):
#0 0x00002b56e4dd4b26 in fts_read () from /lib64/libc.so.6 #1 0x00007fff37314e80 in ?? () #2 0x0000000000000000 in ?? ()Thread 3 (Thread 4022):
#00x00002b56e465ac77inpthread_rwlock_wrlock() from/lib64/libpthread.so.0#10x00005555555752eainmaster_source_writelock(entry=<value optimized out>)at master.c:527 #2 0x00005555555631af in handle_packet_missing_direct (ap=0x5555556a11c0, pkt=0x40821ec0) at direct.c:1296 #3 0x000055555555fa1d in handle_packet (ap=<value optimized out>) at automount.c:1029 #4 handle_mounts (ap=<value optimized out>) at automount.c:1551 #5 0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0 #6 0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6 #7 0x0000000000000000 in ?? ()Thread 2 (Thread 4019):
#00x00002b56e465b326in pthread_cond_wait@@GLIBC_2.3.2 ()
from/lib64/libpthread.so.0
#10x000055555556c8b3inst_queue_handler(arg=<value optimized out>)at state.c:1104 #2 0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0 #3 0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6 #4 0x0000000000000000 in ?? ()Quoted 9 lines
Thread 1 (Thread 4018):
#00x00002b56e465b517in pthread_cond_timedwait@@GLIBC_2.3.2 ()
from/lib64/libpthread.so.0
#10x0000555555573762inalarm_handler(arg=<value optimized out>)at alarm.c:206 #2 0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0 #3 0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6 #4 0x0000000000000000 in ?? ()It looks like a lock ordering bug to me and I think it's the same hang that Steve Thompson reported.
This will be hard to test I think, hopefully Mike or Steve will be able to use a couple of machines for testing, even so it may take a while before we know.
Could you prepare a test package for Mike using this patch please. It should apply OK but of course you'll need to discard or fix the CHANGELOG hunk.
I'll prepare a test package for Steve soon as I get time.
autofs-5.0.5 - fix out of order locking in readmap
From: Ian Kent <[email protected]>
When re-reading the master map and a mount lookup for a particular map entry arrives at the same time a map source is being added to the map entry and a map entry re-read is also initiated at the same time an out of order mutex locking bug which results in a hang.
---CHANGELOG | 1 + lib/master.c | 8 ++------ 2 files changed, 3 insertions(+), 6 deletions(-)CHANGELOG +1
-0@@ -55,6 +55,7 @@ - add external bind method. - fix add simple bind auth. - add option to dump configured automount maps. +- fix out of order locking in readmap. 03/09/2009 autofs-5.0.5 -----------------------lib/master.c +2
-6@@ -611,8 +611,6 @@ struct master_mapent *master_find_mapent(struct master *master, const char *path {struct list_head *head, *p; -master_mutex_lock (); - head = &master->mounts;list_for_each (p, head) {struct master_mapent *entry;@@ -625,8 +623,6 @@ struct master_mapent *master_find_mapent(struct master *master, const char *path } } -master_mutex_unlock (); -return NULL ; }@@ -703,9 +699,7 @@ struct master_mapent *master_new_mapent(struct master *master, const char *path,void master_add_mapent (struct master *master,struct master_mapent *entry) { -master_mutex_lock ();list_add_tail (&entry->list, &master->mounts); -master_mutex_unlock ();return ; }@@ -813,6 +807,7 @@ int master_read_master(struct master *master, time_t age, int readall)* We need to clear and re-populate the null map entry cache * before alowing anyone else to use it. */ +master_mutex_lock ();if (master->nc) {cache_writelock (master->nc); nc = master->nc;@@ -831,6 +826,7 @@ int master_read_master(struct master *master, time_t age, int readall)master_init_scan ();lookup_nss_read_master (master, age);cache_unlock (nc); +master_mutex_unlock ();
Quoted 8 lines
lib/master.c int master_read_master(struct master *master, time_t age, int readall) · line 832
Quoted 2 lines
lib/master.c +0
-0if (!master->read_fail)master_mount_mounts (master, age, readall);Hello Ian,
I just got a reply confirming that the hangs are no longer reproducible with this
patch applied. Please, do you think you can consider it for upstream inclusion?
Oh wow, that's just plain luck I think.
I'll put the patch back on my commit queue but can't say yet when the next time I will push updates. At least, as long as it's in the queue it will be included in the next release, which is long overdue.
Quoted 3 lines
Thanks,
Leonardo
Quoted 6 lines · Philip Ong Jr.
I've pulled down the 5.0.5 tree and all patches.
I've set the logging level to debug and can see messages of expiring mounts in
/var/log/messages...butwhen I check/etc/mtabor/proc/mountsor df, i can still see them there. Any ideas if this is a known issue or if I can give more info than below?
Hello,
I finally had time to return to this issue. To avoid the confusion caused by old kernels, I reproduced the problem on openSUSE Factory (kernel 2.6.38 and autofs 5.0.5 with all kernel.org patches applied). Here's the configuration used:
# cat /etc/auto.master /data /etc/auto.data
# cat /etc/auto.data isos -fstype=nfs,ro,rsize=8192,wsize=8192,intr,nolock,nosuid libre:/isos
# cat /etc/sysconfig/autofs | grep -v '^#' AUTOFS_OPTIONS="" LOCAL_OPTIONS="" APPEND_OPTIONS="yes" USE_MISC_DEVICE="yes" DEFAULT_MASTER_MAP_NAME="auto.master" DEFAULT_TIMEOUT=600 DEFAULT_BROWSE_MODE="yes" DEFAULT_LOGGING="debug" DEFAULT_MAP_OBJECT_CLASS="nisMap" DEFAULT_ENTRY_OBJECT_CLASS="nisObject" DEFAULT_MAP_ATTRIBUTE="nisMapName" DEFAULT_ENTRY_ATTRIBUTE="cn" DEFAULT_VALUE_ATTRIBUTE="nisMapEntry" DEFAULT_AUTH_CONF_FILE="etc/autofs_ldap_auth.conf" MAP_HASH_TABLE_SIZE=1024
I'm attaching the automount debug logs showing the following sequence:
- automount startup
- mount of an NFS volume (/data/isos)
- failed attempt to trigger the expiration of the mounted volume (sending SIGUSR1)
- successful attempt to trigger the expiration of the mounted volume (sending SIGUSR1 again)
I also confirmed that the problem no longer happens if I revert the following commit:
commit 08aafab4c1d0ab6227c80f8cd1086ae78556a370
Author: Ian Kent <[email protected]> Date: Thu Sep 9 11:10:47 2010 +0800
autofs-5.0.5 - fix direct map not updating on reread
Philip, do you think you could try to revert it in your setup/package just to confirm this "works"?
Thanks,
Leonardo
== STARTUP
Mar 24 18:41:33 n47 automount[4681]: Starting automounter version 5.0.5, master map auto.master Mar 24 18:41:33 n47 automount[4681]: using kernel protocol version 5.02 Mar 24 18:41:33 n47 automount[4681]: lookup_nss_read_master: reading master files auto.master Mar 24 18:41:33 n47 automount[4681]: parse_init: parse(sun): init gathered global options: (null) Mar 24 18:41:33 n47 automount[4681]: lookup_read_master: lookup(file): read entry /data Mar 24 18:41:33 n47 automount[4681]: master_do_mount: mounting /data Mar 24 18:41:33 n47 automount[4681]: automount_path_to_fifo: fifo name /var/run/autofs.fifo-data Mar 24 18:41:33 n47 automount[4681]: lookup_nss_read_map: reading map file /etc/auto.data Mar 24 18:41:33 n47 automount[4681]: parse_init: parse(sun): init gathered global options: (null) Mar 24 18:41:33 n47 automount[4681]: mounted indirect on /data with timeout 600, freq 150 seconds Mar 24 18:41:33 n47 automount[4681]: st_ready: st_ready(): state = 0 path /data Mar 24 18:41:33 n47 automount[4681]: ghosting enabled
== COMMAND: ls /data/isos
Mar 24 18:41:54 n47 automount[4681]: handle_packet: type = 3
Mar 24 18:41:54 n47 automount[4681]: handle_packet_missing_indirect: token 3, name isos, request pid 4691
Mar 24 18:41:54 n47 automount[4681]: attempting to mount entry /data/isos
Mar 24 18:41:54 n47 automount[4681]: lookup_mount: lookup(file): looking up isos
Mar 24 18:41:54 n47 automount[4681]: lookup_mount: lookup(file): isos -> -fstype=nfs,ro,rsize=8192,wsize=8192,intr,nolock,nosuid libre:/isos
Mar 24 18:41:54 n47 automount[4681]: parse_mount: parse(sun): expanded entry: -fstype=nfs,ro,rsize=8192,wsize=8192,intr,nolock,nosuid libre:/isos
Mar 24 18:41:54 n47 automount[4681]: parse_mount: parse(sun): gathered options: fstype=nfs,ro,rsize=8192,wsize=8192,intr,nolock,nosuid
Mar 24 18:41:54 n47 automount[4681]: parse_mount: parse(sun): dequote("libre:/isos") -> libre:/isos
Mar 24 18:41:54 n47 automount[4681]: parse_mount: parse(sun): core of entry: options=fstype=nfs,ro,rsize=8192,wsize=8192,intr,nolock,nosuid, loc=libre:/isos
Mar 24 18:41:54 n47 automount[4681]: sun_mount: parse(sun): mounting root /data, mountpoint isos, what libre:/isos, fstype nfs, options ro,rsize=8192,wsize=8192,intr,nolock,nosuid
Mar 24 18:41:54 n47 automount[4681]: mount_mount: mount(nfs): root=/data name=isos what=libre:/isos, fstype=nfs, options=ro,rsize=8192,wsize=8192,intr,nolock,nosuid
Mar 24 18:41:54 n47 automount[4681]: mount_mount: mount(nfs): nfs options="ro,rsize=8192,wsize=8192,intr,nolock,nosuid", nosymlink=0, ro=1
Mar 24 18:41:54 n47 automount[4681]: mount_mount: mount(nfs): calling mkdir_path /data/isos
Mar 24 18:41:54 n47 automount[4681]: mount_mount: mount(nfs): calling mount -t nfs -s -o ro,rsize=8192,wsize=8192,intr,nolock,nosuid libre:/isos /data/isos
Mar 24 18:41:54 n47 automount[4681]: mount_mount: mount(nfs): mounted libre:/isos on /data/isos
Mar 24 18:41:54 n47 automount[4681]: dev_ioctl_send_ready: token = 3
Mar 24 18:41:54 n47 automount[4681]: mounted /data/isos
Mar 24 18:41:54 n47 automount[4681]: st_readmap: state 1 path /data
Mar 24 18:41:54 n47 automount[4681]: re-reading map for /data
Mar 24 18:41:54 n47 automount[4681]: lookup_nss_read_map: reading map file /etc/auto.data
Mar 24 18:41:54 n47 automount[4681]: parse_init: parse(sun): init gathered global options: (null)
Mar 24 18:41:54 n47 automount[4681]: st_ready: st_ready(): state = 4 path /data== COMMAND: kill -USR1 $(pidof automount)
Mar 24 18:42:07 n47 automount[4681]: do_notify_state: signal 10 Mar 24 18:42:07 n47 automount[4681]: master_notify_state_change: sig 10 switching /data from 1 to 3
== COMMAND: kill -USR1 $(pidof automount)
Mar 24 18:42:45 n47 automount[4681]: do_notify_state: signal 10 Mar 24 18:42:45 n47 automount[4681]: master_notify_state_change: sig 10 switching /data from 1 to 3 Mar 24 18:42:45 n47 automount[4681]: st_prune: state 1 path /data Mar 24 18:42:45 n47 automount[4681]: expire_proc: exp_proc = 140224412317440 path /data Mar 24 18:42:45 n47 automount[4681]: expire_proc_indirect: expire /data/isos Mar 24 18:42:45 n47 automount[4681]: handle_packet: type = 4 Mar 24 18:42:45 n47 automount[4681]: handle_packet_expire_indirect: token 4, name isos Mar 24 18:42:45 n47 automount[4681]: expiring path /data/isos Mar 24 18:42:45 n47 automount[4681]: umount_multi: path /data/isos incl 1 Mar 24 18:42:45 n47 automount[4681]: umount_subtree_mounts: unmounting dir = /data/isos Mar 24 18:42:45 n47 automount[4681]: expired /data/isos Mar 24 18:42:45 n47 automount[4681]: dev_ioctl_send_ready: token = 4 Mar 24 18:42:45 n47 automount[4681]: expire_cleanup: got thid 140224412317440 path /data stat 0 Mar 24 18:42:45 n47 automount[4681]: expire_cleanup: sigchld: exp 140224412317440 finished, switching from 3 to 1 Mar 24 18:42:45 n47 automount[4681]: st_ready: st_ready(): state = 3 path /data
_______________________________________________
autofs mailing list
[email protected]
http://linux.kernel.org/mailman/listinfo/autofs
Ian Kent
maintainerQuoted 41 lines · Leonardo Chiquitto
I finally had time to return to this issue. To avoid the confusion caused by old kernels, I reproduced the problem on openSUSE Factory (kernel 2.6.38 and autofs 5.0.5 with all kernel.org patches applied). Here's the configuration used:
# cat /etc/auto.master /data /etc/auto.data# cat /etc/auto.data isos -fstype=nfs,ro,rsize=8192,wsize=8192,intr,nolock,nosuid libre:/isos# cat /etc/sysconfig/autofs | grep -v '^#' AUTOFS_OPTIONS="" LOCAL_OPTIONS="" APPEND_OPTIONS="yes" USE_MISC_DEVICE="yes" DEFAULT_MASTER_MAP_NAME="auto.master" DEFAULT_TIMEOUT=600 DEFAULT_BROWSE_MODE="yes" DEFAULT_LOGGING="debug" DEFAULT_MAP_OBJECT_CLASS="nisMap" DEFAULT_ENTRY_OBJECT_CLASS="nisObject" DEFAULT_MAP_ATTRIBUTE="nisMapName" DEFAULT_ENTRY_ATTRIBUTE="cn" DEFAULT_VALUE_ATTRIBUTE="nisMapEntry" DEFAULT_AUTH_CONF_FILE="etc/autofs_ldap_auth.conf" MAP_HASH_TABLE_SIZE=1024I'm attaching the automount debug logs showing the following sequence:
- automount startup
- mount of an NFS volume (/data/isos)
- failed attempt to trigger the expiration of the mounted volume (sending SIGUSR1)
- successful attempt to trigger the expiration of the mounted volume (sending SIGUSR1 again)I also confirmed that the problem no longer happens if I revert the following commit:
Quoted 8 lines
commit
08aafab4c1d0ab6227c80f8cd1086ae78556a370Author: Ian Kent <[email protected]> Date: Thu Sep 9 11:10:47 2010 +0800autofs-5.0.5 - fix direct map not updating on reread
Philip, do you think you could try to revert it in your setup/package just to confirm this "works"?
The problem was actually introduced by autofs-5.0.5-remove-state-machine-timed-wait.patch.
There had been a long standing bug in the state queue handling which I thought was a pthreads problem. When I added the above patch everything appeared to work OK.
But the change you mentioned above exposed the bug and the log you provided allowed me to work out what was broken and fix it.
But the stale map processing has also been broken somewhere along the way and I'm also working on fixing that.
Once that's done there's not much more needed for 5.0.6, at last.
Ian