Skip to contentSkip to conversation

5.0.5 non-expiring mounts

DISCUSSION Philip Ong Jr. 4 participants 16 messages Feb 8, 2011 – Mar 28, 2011

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

I've pulled down the 5.0.5 tree and all patches.

I've set the logging level to debug and can see messages of expiring mounts in /var/log/messages...but when I check /etc/mtab or /proc/mounts or df, i can still see them there. Any ideas if this is a known issue or if I can give more info than below?

Linux sc-sim-208-144 2.6.29.4 #8 SMP Tue May 25 14:00:33 PDT 2010 x86_64 x86_64 x86_64 GNU/Linux

# automount -V

Linux automount version 5.0.5

Directories:

        config dir:     /etc/sysconfig
        maps dir:       /etc
        modules dir:    /usr/lib/autofs

Compile options:
DISABLE_MOUNT_LOCKING ENABLE_IGNORE_BUSY_MOUNTS LIBXML2_WORKAROUND

maintainer
#1 lore raw
Quoted 6 lines · Philip Ong Jr.

I've pulled down the 5.0.5 tree and all patches.

I've set the logging level to debug and can see messages of expiring mounts in /var/log/messages...but when I check /etc/mtab or /proc/mounts or df, i can still see them there. Any ideas if this is a known issue or if I can give more info than below?

You will need to post a full debug log somewhere.
Make sure that syslog is capturing daemon.* to the syslog so that you get all the debug log messages.

Quoted 20 lines

Linux sc-sim-208-144 2.6.29.4 #8 SMP Tue May 25 14:00:33 PDT 2010 x86_64 x86_64 x86_64 GNU/Linux

# automount -V

Linux automount version 5.0.5

Directories:

         config dir:     /etc/sysconfig
         maps dir:       /etc
         modules dir:    /usr/lib/autofs

Compile options:
DISABLE_MOUNT_LOCKING ENABLE_IGNORE_BUSY_MOUNTS LIBXML2_WORKAROUND

_______________________________________________
autofs mailing list
[email protected]
http://linux.kernel.org/mailman/listinfo/autofs

#2 lore raw
In reply to Philip Ong Jr.
Quoted 4 lines · Philip Ong Jr.

I've set the logging level to debug and can see messages of expiring mounts in /var/log/messages...but when I check /etc/mtab or /proc/mounts or df, i can still see them there. Any ideas if this is a known issue or if I can give more info than below?

Out of curiosity... are these direct or indirect mounts? We're seeing an issue where our huge maps served from LDAP are seeing indirects not umounting while logging expires just fine, but the direct maps are working great. I believe a co-worker has replicated this exact same behavior with local files based maps too though, so pretty sure it's not really related to LDAP at all, but we've found using LDAP helped clear out a lot of old issues we had a few years ago related to the hashing bits.

I've even done network sniffing while doing a kill -USR1 and the automount daemon opens new sockets for each indirect (our homedirs) path it wants to expire, but there never is any actual network traffic sent across the wire. Even better, the sockets stay stuck in ESTABLISHED for a long time. Some more USR1/expires and/or HUPs can help clear is, or I just run a script that does an fuser then umount on unused homedirs and that flushes the ports out. When the ports pile up to >300 or so, you start getting NFS failures with syslog showing "couldn't read superblock."

Oh, and when I say huge maps.. I mean it. We're >10k direct mounts and I don't know how many indirect mounts, but it's even more then that.

Signature · 3 lines
Mike Marion-Unix SysAdmin/Staff IT Engineer-http://www.qualcomm.com
Antonio: "It's no fair.  Some men drink deep from the fountain of life, while
Antonio takes one sip and it goes down the wrong pipe!"
maintainer
#3 lore raw
Quoted 24 lines · Mike Marion
Quoted 4 lines · Philip Ong Jr.

I've set the logging level to debug and can see messages of expiring mounts in /var/log/messages...but when I check /etc/mtab or /proc/mounts or df, i can still see them there. Any ideas if this is a known issue or if I can give more info than below?

Out of curiosity... are these direct or indirect mounts? We're seeing an issue where our huge maps served from LDAP are seeing indirects not umounting while logging expires just fine, but the direct maps are working great. I believe a co-worker has replicated this exact same behavior with local files based maps too though, so pretty sure it's not really related to LDAP at all, but we've found using LDAP helped clear out a lot of old issues we had a few years ago related to the hashing bits.

I've even done network sniffing while doing a kill -USR1 and the automount daemon opens new sockets for each indirect (our homedirs) path it wants to expire, but there never is any actual network traffic sent across the wire. Even better, the sockets stay stuck in ESTABLISHED for a long time. Some more USR1/expires and/or HUPs can help clear is, or I just run a script that does an fuser then umount on unused homedirs and that flushes the ports out. When the ports pile up to >300 or so, you start getting NFS failures with syslog showing "couldn't read superblock."

Quoted 2 lines

Oh, and when I say huge maps.. I mean it. We're >10k direct mounts and I don't know how many indirect mounts, but it's even more then that.

What kernel and source?
Ian

maintainer
#4 lore raw
Quoted 25 lines · Ian Kent
Quoted 24 lines · Mike Marion
Quoted 4 lines · Philip Ong Jr.

I've set the logging level to debug and can see messages of expiring mounts in /var/log/messages...but when I check /etc/mtab or /proc/mounts or df, i can still see them there. Any ideas if this is a known issue or if I can give more info than below?

Out of curiosity... are these direct or indirect mounts? We're seeing an issue where our huge maps served from LDAP are seeing indirects not umounting while logging expires just fine, but the direct maps are working great. I believe a co-worker has replicated this exact same behavior with local files based maps too though, so pretty sure it's not really related to LDAP at all, but we've found using LDAP helped clear out a lot of old issues we had a few years ago related to the hashing bits.

I've even done network sniffing while doing a kill -USR1 and the automount daemon opens new sockets for each indirect (our homedirs) path it wants to expire, but there never is any actual network traffic sent across the wire. Even better, the sockets stay stuck in ESTABLISHED for a long time. Some more USR1/expires and/or HUPs can help clear is, or I just run a script that does an fuser then umount on unused homedirs and that flushes the ports out. When the ports pile up to >300 or so, you start getting NFS failures with syslog showing "couldn't read superblock."

Quoted 4 lines
Quoted 2 lines

Oh, and when I say huge maps.. I mean it. We're >10k direct mounts and I don't know how many indirect mounts, but it's even more then that.

What kernel and source?

That is kernel revision and autofs revision?

Quoted 6 lines

Ian

_______________________________________________
autofs mailing list
[email protected]
http://linux.kernel.org/mailman/listinfo/autofs

#5 lore raw
Quoted 1 line · Ian Kent

That is kernel revision and autofs revision?

2.6.16.60-0.59.1 (Sles10 sp3 with an updated, but not bleeding edge, patch). autofs 5.0.5 with most of the patches up to a couple months ago. It's hard to get exacts because it's a PTF from Novell (we really pushed them to upgrade to 5.0.5) but it should be pretty much equal to the patch they just released for sle 11 sp1 that they're recommending as they default going forward.

Signature · 4 lines
Mike Marion-Unix SysAdmin/Staff IT Engineer-http://www.qualcomm.com
Vir: "Londo, are you deliberatly trying to drive me insane?"
Londo: "The Universe is already mad.  Anything else would be redundant."
==> Babylon 5
maintainer
#6 lore raw
Quoted 9 lines · Mike Marion
Quoted 1 line · Ian Kent

That is kernel revision and autofs revision?

2.6.16.60-0.59.1 (Sles10 sp3 with an updated, but not bleeding edge, patch). autofs 5.0.5 with most of the patches up to a couple months ago. It's hard to get exacts because it's a PTF from Novell (we really pushed them to upgrade to 5.0.5) but it should be pretty much equal to the patch they just released for sle 11 sp1 that they're recommending as they default going forward.

That make it hard, as you know.

But I wouldn't mind spending a bit of time on it, if you can also.

Let's assume that it's a user space problem for now.
Could you post a debug log somewhere and the current sles patched source somewhere. If you could spend a little time trying to compare the patches in the sles package to the ones in the Rawhide autofs repo that would be a big help too.

Ian

#7 lore raw
Quoted 11 lines · Ian Kent
Quoted 9 lines · Mike Marion
Quoted 1 line · Ian Kent

That is kernel revision and autofs revision?

2.6.16.60-0.59.1 (Sles10 sp3 with an updated, but not bleeding edge, patch). autofs 5.0.5 with most of the patches up to a couple months ago.  It's hard to get exacts because it's a PTF from Novell (we really pushed them to upgrade to 5.0.5) but it should be pretty much equal to the patch they just released for sle 11 sp1 that they're recommending as they default going forward.

Quoted 5 lines

That make it hard, as you know.

But I wouldn't mind spending a bit of time on it, if you can also.

Let's assume that it's a user space problem for now.

Here are the call traces for all automount processes on the kernel side:

PID: 4017 TASK: ffff8104202a9080 CPU: 4 COMMAND: "automount"
#0 [ffff81040426fc58] schedule at ffffffff802efc13 #1 [ffff81040426fd30] schedule_timeout at ffffffff802f0748 #2 [ffff81040426fd80] futex_wait_abstime at ffffffff8014b703 #3 [ffff81040426fe60] do_futex at ffffffff8014bb33 #4 [ffff81040426ff20] sys_futex at ffffffff8014c2ed #5 [ffff81040426ff80] system_call at ffffffff8010ae16

PID: 4018 TASK: ffff81041cefd810 CPU: 2 COMMAND: "automount"
#0 [ffff810404211d68] schedule at ffffffff802efc13 #1 [ffff810404211e40] schedule_timeout at ffffffff802f07b4 #2 [ffff810404211e90] futex_wait_abstime at ffffffff8014b703 #3 [ffff810404211f70] futex_wait_restart at ffffffff8014b7d2 #4 [ffff810404211f80] system_call at ffffffff8010ae16

PID: 4019 TASK: ffff810420ab8080 CPU: 5 COMMAND: "automount"
#0 [ffff810404213c58] schedule at ffffffff802efc13 #1 [ffff810404213d30] schedule_timeout at ffffffff802f0748 #2 [ffff810404213d80] futex_wait_abstime at ffffffff8014b703 #3 [ffff810404213e60] do_futex at ffffffff8014bb33 #4 [ffff810404213f20] sys_futex at ffffffff8014c2ed #5 [ffff810404213f80] system_call at ffffffff8010ae16

PID: 4022 TASK: ffff81042055b7d0 CPU: 5 COMMAND: "automount"
#0 [ffff810404c71c58] schedule at ffffffff802efc13 #1 [ffff810404c71d30] schedule_timeout at ffffffff802f0748 #2 [ffff810404c71d80] futex_wait_abstime at ffffffff8014b703 #3 [ffff810404c71e60] do_futex at ffffffff8014bb33 #4 [ffff810404c71f20] sys_futex at ffffffff8014c2ed #5 [ffff810404c71f80] system_call at ffffffff8010ae16

PID: 4025 TASK: ffff81041cefd080 CPU: 2 COMMAND: "automount"
#0 [ffff8104001d7da8] schedule at ffffffff802efc13 #1 [ffff8104001d7e80] schedule_timeout at ffffffff802f07b4 #2 [ffff8104001d7ed0] do_sys_poll at ffffffff80199526 #3 [ffff8104001d7f60] sys_poll at ffffffff80199640 #4 [ffff8104001d7f80] system_call at ffffffff8010ae16

PID: 4026 TASK: ffff81041e1bd080 CPU: 7 COMMAND: "automount"
#0 [ffff8104001f7c58] schedule at ffffffff802efc13 #1 [ffff8104001f7d30] schedule_timeout at ffffffff802f0748 #2 [ffff8104001f7d80] futex_wait_abstime at ffffffff8014b703 #3 [ffff8104001f7e60] do_futex at ffffffff8014bb33 #4 [ffff8104001f7f20] sys_futex at ffffffff8014c2ed #5 [ffff8104001f7f80] system_call at ffffffff8010ae16

PID: 6851 TASK: ffff81013c9187d0 CPU: 6 COMMAND: "automount"
#0 [ffff810304f97c58] schedule at ffffffff802efc13 #1 [ffff810304f97d30] schedule_timeout at ffffffff802f0748 #2 [ffff810304f97d80] futex_wait_abstime at ffffffff8014b703 #3 [ffff810304f97e60] do_futex at ffffffff8014bb33 #4 [ffff810304f97f20] sys_futex at ffffffff8014c2ed #5 [ffff810304f97f80] system_call at ffffffff8010ae16

PID: 7060 TASK: ffff810188ef5080 CPU: 4 COMMAND: "automount"
#0 [ffff810319ebdc58] schedule at ffffffff802efc13 #1 [ffff810319ebdd30] schedule_timeout at ffffffff802f0748 #2 [ffff810319ebdd80] futex_wait_abstime at ffffffff8014b703 #3 [ffff810319ebde60] do_futex at ffffffff8014bb33 #4 [ffff810319ebdf20] sys_futex at ffffffff8014c2ed #5 [ffff810319ebdf80] system_call at ffffffff8010ae16

PID: 20702 TASK: ffff810230be5080 CPU: 7 COMMAND: "automount"
#0 [ffff81026c80dc18] schedule at ffffffff802efc13
#1 [ffff81026c80dcf0] autofs4_wait at ffffffff8831e5b2 [autofs4] #2 [ffff81026c80deb0] autofs4_expire_multi at ffffffff8831ed06 [autofs4] #3 [ffff81026c80dee0] do_ioctl at ffffffff80198acd
#4 [ffff81026c80df00] vfs_ioctl at ffffffff80198d2b
#5 [ffff81026c80df40] sys_ioctl at ffffffff80198da5
#6 [ffff81026c80df80] system_call at ffffffff8010ae16

They look regular to me, do you see something suspect?

And here are the call traces from the user land daemon:

Thread 9 (Thread 4017):

#0  0x00002b56e465d6a8 in __lll_mutex_lock_wait () from /lib64/libpthread.so.0
#1  0x00002b56e46599fb in _L_mutex_lock_92 () from /lib64/libpthread.so.0
#2  0x00002b56e4659455 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00005555555746cd in master_mutex_lock () at master.c:49
#4  0x000055555555d260 in do_hup_signal (master=0x55555568d010, age=1296063258)
    at automount.c:1276
#5  0x0000555555560bd3 in statemachine (arg=<value optimized out>)
    at automount.c:1354

#6 main (arg=<value optimized out>) at automount.c:2142

Thread 8 (Thread 20702):
#0 0x00002b56e4dd62a7 in brk () from /lib64/libc.so.6
#1 0x0000555555577dfe in expire (logopt=2, cmd=<value optimized out>, fd=21,

    ioctlfd=21, path=0x55555569ca20 "/usr2", arg=0x41c27ef4)
    at dev-ioctl-lib.c:657
#2  0x0000555555577ebe in ioctl_expire (logopt=21, ioctlfd=-1,
    path=0x55555569ca20 "/usr2", when=0) at dev-ioctl-lib.c:701
#3  0x0000555555561e4e in expire_proc_indirect (arg=<value optimized out>)
    at indirect.c:545
#4  0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0
#5  0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 7 (Thread 7060):
#0 0x00002b56e465ac77 in pthread_rwlock_wrlock () from /lib64/libpthread.so.0 #1 0x00005555555752ea in master_source_writelock (entry=<value optimized out>)

    at master.c:527
#2  0x0000555555575f8f in master_add_map_source (entry=0x5555556a10b0,
    type=0x0, format=0x0, age=1296059657, argc=1, argv=<value optimized out>)
    at master.c:191
#3  0x0000555555579ee3 in master_parse_entry (buffer=<value optimized out>,
    default_timeout=86400, logging=<value optimized out>, age=1296059657)
    at master_parse.y:823
#4  0x00002aaaaaab83fe in lookup_read_master (master=<value optimized out>,
    age=1296059657, context=<value optimized out>) at lookup_ldap.c:1625
#5  0x0000555555569052 in do_read_master (master=0x55555568d010,
    type=<value optimized out>, age=1296059657) at lookup.c:96
#6  0x000055555556aa3c in lookup_nss_read_master (master=0x55555568d010,
    age=1296059657) at lookup.c:229
#7  0x0000555555575c28 in master_read_master (master=0x55555568d010,
    age=1296059657, readall=1) at master.c:831
#8  0x000055555555d844 in do_read_master (arg=<value optimized out>)
    at automount.c:1259

#9 0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0 #10 0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6 #11 0x0000000000000000 in ?? ()

Thread 6 (Thread 6851):

#0  0x00002b56e465aa3d in pthread_rwlock_rdlock () from /lib64/libpthread.so.0
#1  0x000055555556deb6 in cache_readlock (mc=0x55555568e5b8) at cache.c:60
#2  0x000055555556baff in do_readmap (arg=<value optimized out>) at state.c:479
#3  0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0
#4  0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6
#5  0x0000000000000000 in ?? ()

Thread 5 (Thread 4026):

#0  0x00002b56e465d6a8 in __lll_mutex_lock_wait () from /lib64/libpthread.so.0
#1  0x00002b56e46599fb in _L_mutex_lock_92 () from /lib64/libpthread.so.0
#2  0x00002b56e4659455 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00005555555746cd in master_mutex_lock () at master.c:49
#4  0x0000555555560ff9 in handle_packet_missing_indirect (ap=0x55555569c940,
    pkt=0x41823ec0) at indirect.c:808
#5  0x000055555555fa32 in handle_packet (ap=<value optimized out>)
    at automount.c:1026
#6  handle_mounts (ap=<value optimized out>) at automount.c:1551
#7  0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0
#8  0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6
#9  0x0000000000000000 in ?? ()

Thread 4 (Thread 4025):

#0  0x00002b56e4dd4b26 in fts_read () from /lib64/libc.so.6
#1  0x00007fff37314e80 in ?? ()
#2  0x0000000000000000 in ?? ()

Thread 3 (Thread 4022):
#0 0x00002b56e465ac77 in pthread_rwlock_wrlock () from /lib64/libpthread.so.0 #1 0x00005555555752ea in master_source_writelock (entry=<value optimized out>)

    at master.c:527
#2  0x00005555555631af in handle_packet_missing_direct (ap=0x5555556a11c0,
    pkt=0x40821ec0) at direct.c:1296
#3  0x000055555555fa1d in handle_packet (ap=<value optimized out>)
    at automount.c:1029
#4  handle_mounts (ap=<value optimized out>) at automount.c:1551
#5  0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0
#6  0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6
#7  0x0000000000000000 in ?? ()

Thread 2 (Thread 4019):
#0 0x00002b56e465b326 in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib64/libpthread.so.0
#1 0x000055555556c8b3 in st_queue_handler (arg=<value optimized out>)

    at state.c:1104
#2  0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0
#3  0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6
#4  0x0000000000000000 in ?? ()

Thread 1 (Thread 4018):
#0 0x00002b56e465b517 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
from /lib64/libpthread.so.0
#1 0x0000555555573762 in alarm_handler (arg=<value optimized out>)

    at alarm.c:206
#2  0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0
#3  0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6
#4  0x0000000000000000 in ?? ()
Quoted 2 lines

Could you post a debug log somewhere and the current sles patched source somewhere.

Unfortunately I don't have the full debug logs yet, so I'll leave this to Mike.

Quoted 3 lines

If you could spend a little time trying to compare the patches in the sles package to the ones in the Rawhide autofs repo that would be a big help too.

The package includes all upstream patches released so far (ie, up to autofs-5.0.5-fix-submount-shutdown-wait.patch), minus the following ones:

autofs-5.0.4 - use intr option as hosts mount default (configuration change) autofs-5.0.5 - add dump maps option (recent feature) autofs-5.0.5 - add external bind method (recent feature) autofs-5.0.5 - fix ext4 fsck at mount (no ext4 on SLES 10)

Thanks,
Leonardo

maintainer
#8 lore raw
Quoted 12 lines · Leonardo Chiquitto
Quoted 11 lines · Ian Kent
Quoted 9 lines · Mike Marion
Quoted 1 line · Ian Kent

That is kernel revision and autofs revision?

2.6.16.60-0.59.1 (Sles10 sp3 with an updated, but not bleeding edge, patch). autofs 5.0.5 with most of the patches up to a couple months ago. It's hard to get exacts because it's a PTF from Novell (we really pushed them to upgrade to 5.0.5) but it should be pretty much equal to the patch they just released for sle 11 sp1 that they're recommending as they default going forward.

Quoted 7 lines
Quoted 5 lines

That make it hard, as you know.

But I wouldn't mind spending a bit of time on it, if you can also.

Let's assume that it's a user space problem for now.

Here are the call traces for all automount processes on the kernel side:

I think it's a user space problem.

snip ...

Quoted 99 lines

And here are the call traces from the user land daemon:

Thread 9 (Thread 4017):

#0  0x00002b56e465d6a8 in __lll_mutex_lock_wait () from /lib64/libpthread.so.0
#1  0x00002b56e46599fb in _L_mutex_lock_92 () from /lib64/libpthread.so.0
#2  0x00002b56e4659455 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00005555555746cd in master_mutex_lock () at master.c:49
#4  0x000055555555d260 in do_hup_signal (master=0x55555568d010, age=1296063258)
    at automount.c:1276
#5  0x0000555555560bd3 in statemachine (arg=<value optimized out>)
    at automount.c:1354

#6 main (arg=<value optimized out>) at automount.c:2142

Thread 8 (Thread 20702):
#0 0x00002b56e4dd62a7 in brk () from /lib64/libc.so.6
#1 0x0000555555577dfe in expire (logopt=2, cmd=<value optimized out>, fd=21,

    ioctlfd=21, path=0x55555569ca20 "/usr2", arg=0x41c27ef4)
    at dev-ioctl-lib.c:657
#2  0x0000555555577ebe in ioctl_expire (logopt=21, ioctlfd=-1,
    path=0x55555569ca20 "/usr2", when=0) at dev-ioctl-lib.c:701
#3  0x0000555555561e4e in expire_proc_indirect (arg=<value optimized out>)
    at indirect.c:545
#4  0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0
#5  0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 7 (Thread 7060):
#0 0x00002b56e465ac77 in pthread_rwlock_wrlock () from /lib64/libpthread.so.0 #1 0x00005555555752ea in master_source_writelock (entry=<value optimized out>)

    at master.c:527
#2  0x0000555555575f8f in master_add_map_source (entry=0x5555556a10b0,
    type=0x0, format=0x0, age=1296059657, argc=1, argv=<value optimized out>)
    at master.c:191
#3  0x0000555555579ee3 in master_parse_entry (buffer=<value optimized out>,
    default_timeout=86400, logging=<value optimized out>, age=1296059657)
    at master_parse.y:823
#4  0x00002aaaaaab83fe in lookup_read_master (master=<value optimized out>,
    age=1296059657, context=<value optimized out>) at lookup_ldap.c:1625
#5  0x0000555555569052 in do_read_master (master=0x55555568d010,
    type=<value optimized out>, age=1296059657) at lookup.c:96
#6  0x000055555556aa3c in lookup_nss_read_master (master=0x55555568d010,
    age=1296059657) at lookup.c:229
#7  0x0000555555575c28 in master_read_master (master=0x55555568d010,
    age=1296059657, readall=1) at master.c:831
#8  0x000055555555d844 in do_read_master (arg=<value optimized out>)
    at automount.c:1259

#9 0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0 #10 0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6 #11 0x0000000000000000 in ?? ()

Thread 6 (Thread 6851):

#0  0x00002b56e465aa3d in pthread_rwlock_rdlock () from /lib64/libpthread.so.0
#1  0x000055555556deb6 in cache_readlock (mc=0x55555568e5b8) at cache.c:60
#2  0x000055555556baff in do_readmap (arg=<value optimized out>) at state.c:479
#3  0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0
#4  0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6
#5  0x0000000000000000 in ?? ()

Thread 5 (Thread 4026):

#0  0x00002b56e465d6a8 in __lll_mutex_lock_wait () from /lib64/libpthread.so.0
#1  0x00002b56e46599fb in _L_mutex_lock_92 () from /lib64/libpthread.so.0
#2  0x00002b56e4659455 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00005555555746cd in master_mutex_lock () at master.c:49
#4  0x0000555555560ff9 in handle_packet_missing_indirect (ap=0x55555569c940,
    pkt=0x41823ec0) at indirect.c:808
#5  0x000055555555fa32 in handle_packet (ap=<value optimized out>)
    at automount.c:1026
#6  handle_mounts (ap=<value optimized out>) at automount.c:1551
#7  0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0
#8  0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6
#9  0x0000000000000000 in ?? ()

Thread 4 (Thread 4025):

#0  0x00002b56e4dd4b26 in fts_read () from /lib64/libc.so.6
#1  0x00007fff37314e80 in ?? ()
#2  0x0000000000000000 in ?? ()

Thread 3 (Thread 4022):
#0 0x00002b56e465ac77 in pthread_rwlock_wrlock () from /lib64/libpthread.so.0 #1 0x00005555555752ea in master_source_writelock (entry=<value optimized out>)

    at master.c:527
#2  0x00005555555631af in handle_packet_missing_direct (ap=0x5555556a11c0,
    pkt=0x40821ec0) at direct.c:1296
#3  0x000055555555fa1d in handle_packet (ap=<value optimized out>)
    at automount.c:1029
#4  handle_mounts (ap=<value optimized out>) at automount.c:1551
#5  0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0
#6  0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6
#7  0x0000000000000000 in ?? ()

Thread 2 (Thread 4019):
#0 0x00002b56e465b326 in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib64/libpthread.so.0
#1 0x000055555556c8b3 in st_queue_handler (arg=<value optimized out>)

    at state.c:1104
#2  0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0
#3  0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6
#4  0x0000000000000000 in ?? ()
Quoted 9 lines

Thread 1 (Thread 4018):
#0 0x00002b56e465b517 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
from /lib64/libpthread.so.0
#1 0x0000555555573762 in alarm_handler (arg=<value optimized out>)

    at alarm.c:206
#2  0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0
#3  0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6
#4  0x0000000000000000 in ?? ()

It looks like a lock ordering bug to me and I think it's the same hang that Steve Thompson reported.

This will be hard to test I think, hopefully Mike or Steve will be able to use a couple of machines for testing, even so it may take a while before we know.

Could you prepare a test package for Mike using this patch please. It should apply OK but of course you'll need to discard or fix the CHANGELOG hunk.

I'll prepare a test package for Steve soon as I get time.

autofs-5.0.5 - fix out of order locking in readmap

From: Ian Kent <[email protected]>

When re-reading the master map and a mount lookup for a particular map entry arrives at the same time a map source is being added to the map entry and a map entry re-read is also initiated at the same time an out of order mutex locking bug which results in a hang.
---

 CHANGELOG    |    1 +
 lib/master.c |    8 ++------
 2 files changed, 3 insertions(+), 6 deletions(-)
CHANGELOG +1 -0
index 11054da..631094f 100644
@@ -55,6 +55,7 @@
 - add external bind method.
 - fix add simple bind auth.
 - add option to dump configured automount maps.
+- fix out of order locking in readmap.
 
 03/09/2009 autofs-5.0.5
 -----------------------
lib/master.c +2 -6
index 95bd3fb..daedf66 100644
@@ -611,8 +611,6 @@ struct master_mapent *master_find_mapent(struct master *master, const char *path
 {
 	struct list_head *head, *p;
 
-	master_mutex_lock();
-
 	head = &master->mounts;
 	list_for_each(p, head) {
 		struct master_mapent *entry;
@@ -625,8 +623,6 @@ struct master_mapent *master_find_mapent(struct master *master, const char *path
 		}
 	}
 
-	master_mutex_unlock();
-
 	return NULL;
 }
 
@@ -703,9 +699,7 @@ struct master_mapent *master_new_mapent(struct master *master, const char *path,
 
 void master_add_mapent(struct master *master, struct master_mapent *entry)
 {
-	master_mutex_lock();
 	list_add_tail(&entry->list, &master->mounts);
-	master_mutex_unlock();
 	return;
 }
 
@@ -813,6 +807,7 @@ int master_read_master(struct master *master, time_t age, int readall)
 	 * We need to clear and re-populate the null map entry cache
 	 * before alowing anyone else to use it.
 	 */
+	master_mutex_lock();
 	if (master->nc) {
 		cache_writelock(master->nc);
 		nc = master->nc;
@@ -831,6 +826,7 @@ int master_read_master(struct master *master, time_t age, int readall)
 	master_init_scan();
 	lookup_nss_read_master(master, age);
 	cache_unlock(nc);
+	master_mutex_unlock();
 
 	if (!master->read_fail)
 		master_mount_mounts(master, age, readall);
maintainer
#9 lore raw
Quoted 5 lines · Ian Kent
Quoted 4 lines · Leonardo Chiquitto
Quoted 3 lines · Ian Kent
Quoted 2 lines · Mike Marion

On Mon, Feb 14, 2011 at 07:37:01PM -0800, Ian Kent wrote:

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 1 line

That is kernel revision and autofs revision?

2.6.16.60-0.59.1 (Sles10 sp3 with an updated, but not bleeding edge, patch). autofs 5.0.5 with most of the patches up to a couple months ago. It's hard to get exacts because it's a PTF from Novell (we really pushed them to upgrade to 5.0.5) but it should be pretty much equal to the patch they just released for sle 11 sp1 that they're recommending as they default going forward.

Now I'm confused?

I thought that Mike had mention he had seen hangs, similar to Steve, and this backtrace was an example of that. But the mail thread doesn't read like that and oddly enough I seem to have identified a locking problem from looking at the code based on the backtrace.

A backtrace generally doesn't do us any good when were trying to find an expire problem, the debug log is where we have to start on these.

Quoted 214 lines
Quoted 8 lines
Quoted 6 lines

That make it hard, as you know.

But I wouldn't mind spending a bit of time on it, if you can also.

Let's assume that it's a user space problem for now.

Here are the call traces for all automount processes on the kernel side:

I think it's a user space problem.

snip ...

Quoted 99 lines

And here are the call traces from the user land daemon:

Thread 9 (Thread 4017):

#0  0x00002b56e465d6a8 in __lll_mutex_lock_wait () from /lib64/libpthread.so.0
#1  0x00002b56e46599fb in _L_mutex_lock_92 () from /lib64/libpthread.so.0
#2  0x00002b56e4659455 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00005555555746cd in master_mutex_lock () at master.c:49
#4  0x000055555555d260 in do_hup_signal (master=0x55555568d010, age=1296063258)
    at automount.c:1276
#5  0x0000555555560bd3 in statemachine (arg=<value optimized out>)
    at automount.c:1354

#6 main (arg=<value optimized out>) at automount.c:2142

Thread 8 (Thread 20702):
#0 0x00002b56e4dd62a7 in brk () from /lib64/libc.so.6
#1 0x0000555555577dfe in expire (logopt=2, cmd=<value optimized out>, fd=21,

    ioctlfd=21, path=0x55555569ca20 "/usr2", arg=0x41c27ef4)
    at dev-ioctl-lib.c:657
#2  0x0000555555577ebe in ioctl_expire (logopt=21, ioctlfd=-1,
    path=0x55555569ca20 "/usr2", when=0) at dev-ioctl-lib.c:701
#3  0x0000555555561e4e in expire_proc_indirect (arg=<value optimized out>)
    at indirect.c:545
#4  0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0
#5  0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 7 (Thread 7060):
#0 0x00002b56e465ac77 in pthread_rwlock_wrlock () from /lib64/libpthread.so.0 #1 0x00005555555752ea in master_source_writelock (entry=<value optimized out>)

    at master.c:527
#2  0x0000555555575f8f in master_add_map_source (entry=0x5555556a10b0,
    type=0x0, format=0x0, age=1296059657, argc=1, argv=<value optimized out>)
    at master.c:191
#3  0x0000555555579ee3 in master_parse_entry (buffer=<value optimized out>,
    default_timeout=86400, logging=<value optimized out>, age=1296059657)
    at master_parse.y:823
#4  0x00002aaaaaab83fe in lookup_read_master (master=<value optimized out>,
    age=1296059657, context=<value optimized out>) at lookup_ldap.c:1625
#5  0x0000555555569052 in do_read_master (master=0x55555568d010,
    type=<value optimized out>, age=1296059657) at lookup.c:96
#6  0x000055555556aa3c in lookup_nss_read_master (master=0x55555568d010,
    age=1296059657) at lookup.c:229
#7  0x0000555555575c28 in master_read_master (master=0x55555568d010,
    age=1296059657, readall=1) at master.c:831
#8  0x000055555555d844 in do_read_master (arg=<value optimized out>)
    at automount.c:1259

#9 0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0 #10 0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6 #11 0x0000000000000000 in ?? ()

Thread 6 (Thread 6851):

#0  0x00002b56e465aa3d in pthread_rwlock_rdlock () from /lib64/libpthread.so.0
#1  0x000055555556deb6 in cache_readlock (mc=0x55555568e5b8) at cache.c:60
#2  0x000055555556baff in do_readmap (arg=<value optimized out>) at state.c:479
#3  0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0
#4  0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6
#5  0x0000000000000000 in ?? ()

Thread 5 (Thread 4026):

#0  0x00002b56e465d6a8 in __lll_mutex_lock_wait () from /lib64/libpthread.so.0
#1  0x00002b56e46599fb in _L_mutex_lock_92 () from /lib64/libpthread.so.0
#2  0x00002b56e4659455 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00005555555746cd in master_mutex_lock () at master.c:49
#4  0x0000555555560ff9 in handle_packet_missing_indirect (ap=0x55555569c940,
    pkt=0x41823ec0) at indirect.c:808
#5  0x000055555555fa32 in handle_packet (ap=<value optimized out>)
    at automount.c:1026
#6  handle_mounts (ap=<value optimized out>) at automount.c:1551
#7  0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0
#8  0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6
#9  0x0000000000000000 in ?? ()

Thread 4 (Thread 4025):

#0  0x00002b56e4dd4b26 in fts_read () from /lib64/libc.so.6
#1  0x00007fff37314e80 in ?? ()
#2  0x0000000000000000 in ?? ()

Thread 3 (Thread 4022):
#0 0x00002b56e465ac77 in pthread_rwlock_wrlock () from /lib64/libpthread.so.0 #1 0x00005555555752ea in master_source_writelock (entry=<value optimized out>)

    at master.c:527
#2  0x00005555555631af in handle_packet_missing_direct (ap=0x5555556a11c0,
    pkt=0x40821ec0) at direct.c:1296
#3  0x000055555555fa1d in handle_packet (ap=<value optimized out>)
    at automount.c:1029
#4  handle_mounts (ap=<value optimized out>) at automount.c:1551
#5  0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0
#6  0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6
#7  0x0000000000000000 in ?? ()

Thread 2 (Thread 4019):
#0 0x00002b56e465b326 in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib64/libpthread.so.0
#1 0x000055555556c8b3 in st_queue_handler (arg=<value optimized out>)

    at state.c:1104
#2  0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0
#3  0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6
#4  0x0000000000000000 in ?? ()
Quoted 9 lines

Thread 1 (Thread 4018):
#0 0x00002b56e465b517 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
from /lib64/libpthread.so.0
#1 0x0000555555573762 in alarm_handler (arg=<value optimized out>)

    at alarm.c:206
#2  0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0
#3  0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6
#4  0x0000000000000000 in ?? ()

It looks like a lock ordering bug to me and I think it's the same hang that Steve Thompson reported.

This will be hard to test I think, hopefully Mike or Steve will be able to use a couple of machines for testing, even so it may take a while before we know.

Could you prepare a test package for Mike using this patch please. It should apply OK but of course you'll need to discard or fix the CHANGELOG hunk.

I'll prepare a test package for Steve soon as I get time.

autofs-5.0.5 - fix out of order locking in readmap

From: Ian Kent <[email protected]>

When re-reading the master map and a mount lookup for a particular map entry arrives at the same time a map source is being added to the map entry and a map entry re-read is also initiated at the same time an out of order mutex locking bug which results in a hang.
---

 CHANGELOG    |    1 +
 lib/master.c |    8 ++------
 2 files changed, 3 insertions(+), 6 deletions(-)
CHANGELOG +1 -0
index 11054da..631094f 100644
@@ -55,6 +55,7 @@
 - add external bind method.
 - fix add simple bind auth.
 - add option to dump configured automount maps.
+- fix out of order locking in readmap.
 
 03/09/2009 autofs-5.0.5
 -----------------------
lib/master.c +2 -6
index 95bd3fb..daedf66 100644
@@ -611,8 +611,6 @@ struct master_mapent *master_find_mapent(struct master *master, const char *path
 {
 	struct list_head *head, *p;
 
-	master_mutex_lock();
-
 	head = &master->mounts;
 	list_for_each(p, head) {
 		struct master_mapent *entry;
@@ -625,8 +623,6 @@ struct master_mapent *master_find_mapent(struct master *master, const char *path
 		}
 	}
 
-	master_mutex_unlock();
-
 	return NULL;
 }
 
@@ -703,9 +699,7 @@ struct master_mapent *master_new_mapent(struct master *master, const char *path,
 
 void master_add_mapent(struct master *master, struct master_mapent *entry)
 {
-	master_mutex_lock();
 	list_add_tail(&entry->list, &master->mounts);
-	master_mutex_unlock();
 	return;
 }
 
@@ -813,6 +807,7 @@ int master_read_master(struct master *master, time_t age, int readall)
 	 * We need to clear and re-populate the null map entry cache
 	 * before alowing anyone else to use it.
 	 */
+	master_mutex_lock();
 	if (master->nc) {
 		cache_writelock(master->nc);
 		nc = master->nc;
@@ -831,6 +826,7 @@ int master_read_master(struct master *master, time_t age, int readall)
 	master_init_scan();
 	lookup_nss_read_master(master, age);
 	cache_unlock(nc);
+	master_mutex_unlock();
 
 	if (!master->read_fail)
 		master_mount_mounts(master, age, readall);
#11 lore raw
In reply to Ian Kent
Quoted 137 lines · Ian Kent
Quoted 12 lines · Leonardo Chiquitto
Quoted 11 lines · Ian Kent
Quoted 9 lines · Mike Marion
Quoted 1 line · Ian Kent

That is kernel revision and autofs revision?

2.6.16.60-0.59.1 (Sles10 sp3 with an updated, but not bleeding edge, patch). autofs 5.0.5 with most of the patches up to a couple months ago.  It's hard to get exacts because it's a PTF from Novell (we really pushed them to upgrade to 5.0.5) but it should be pretty much equal to the patch they just released for sle 11 sp1 that they're recommending as they default going forward.

Quoted 7 lines
Quoted 5 lines

That make it hard, as you know.

But I wouldn't mind spending a bit of time on it, if you can also.

Let's assume that it's a user space problem for now.

Here are the call traces for all automount processes on the kernel side:

I think it's a user space problem.

snip ...

Quoted 99 lines

And here are the call traces from the user land daemon:

Thread 9 (Thread 4017):
#0  0x00002b56e465d6a8 in __lll_mutex_lock_wait () from /lib64/libpthread.so.0 #1  0x00002b56e46599fb in _L_mutex_lock_92 () from /lib64/libpthread.so.0 #2  0x00002b56e4659455 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3  0x00005555555746cd in master_mutex_lock () at master.c:49
#4  0x000055555555d260 in do_hup_signal (master=0x55555568d010, age=1296063258) at automount.c:1276
#5  0x0000555555560bd3 in statemachine (arg=<value optimized out>) at automount.c:1354
#6  main (arg=<value optimized out>) at automount.c:2142

Thread 8 (Thread 20702):
#0  0x00002b56e4dd62a7 in brk () from /lib64/libc.so.6
#1  0x0000555555577dfe in expire (logopt=2, cmd=<value optimized out>, fd=21, ioctlfd=21, path=0x55555569ca20 "/usr2", arg=0x41c27ef4)
    at dev-ioctl-lib.c:657
#2  0x0000555555577ebe in ioctl_expire (logopt=21, ioctlfd=-1, path=0x55555569ca20 "/usr2", when=0) at dev-ioctl-lib.c:701 #3  0x0000555555561e4e in expire_proc_indirect (arg=<value optimized out>) at indirect.c:545
#4  0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0 #5  0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 7 (Thread 7060):
#0  0x00002b56e465ac77 in pthread_rwlock_wrlock () from /lib64/libpthread.so.0 #1  0x00005555555752ea in master_source_writelock (entry=<value optimized out>) at master.c:527
#2  0x0000555555575f8f in master_add_map_source (entry=0x5555556a10b0, type=0x0, format=0x0, age=1296059657, argc=1, argv=<value optimized out>) at master.c:191
#3  0x0000555555579ee3 in master_parse_entry (buffer=<value optimized out>, default_timeout=86400, logging=<value optimized out>, age=1296059657) at master_parse.y:823
#4  0x00002aaaaaab83fe in lookup_read_master (master=<value optimized out>, age=1296059657, context=<value optimized out>) at lookup_ldap.c:1625 #5  0x0000555555569052 in do_read_master (master=0x55555568d010, type=<value optimized out>, age=1296059657) at lookup.c:96
#6  0x000055555556aa3c in lookup_nss_read_master (master=0x55555568d010, age=1296059657) at lookup.c:229
#7  0x0000555555575c28 in master_read_master (master=0x55555568d010, age=1296059657, readall=1) at master.c:831
#8  0x000055555555d844 in do_read_master (arg=<value optimized out>) at automount.c:1259
#9  0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0 #10 0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6
#11 0x0000000000000000 in ?? ()

Thread 6 (Thread 6851):
#0  0x00002b56e465aa3d in pthread_rwlock_rdlock () from /lib64/libpthread.so.0 #1  0x000055555556deb6 in cache_readlock (mc=0x55555568e5b8) at cache.c:60 #2  0x000055555556baff in do_readmap (arg=<value optimized out>) at state.c:479 #3  0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0 #4  0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6
#5  0x0000000000000000 in ?? ()

Thread 5 (Thread 4026):
#0  0x00002b56e465d6a8 in __lll_mutex_lock_wait () from /lib64/libpthread.so.0 #1  0x00002b56e46599fb in _L_mutex_lock_92 () from /lib64/libpthread.so.0 #2  0x00002b56e4659455 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3  0x00005555555746cd in master_mutex_lock () at master.c:49
#4  0x0000555555560ff9 in handle_packet_missing_indirect (ap=0x55555569c940, pkt=0x41823ec0) at indirect.c:808
#5  0x000055555555fa32 in handle_packet (ap=<value optimized out>) at automount.c:1026
#6  handle_mounts (ap=<value optimized out>) at automount.c:1551 #7  0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0 #8  0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6
#9  0x0000000000000000 in ?? ()

Thread 4 (Thread 4025):
#0  0x00002b56e4dd4b26 in fts_read () from /lib64/libc.so.6 #1  0x00007fff37314e80 in ?? ()
#2  0x0000000000000000 in ?? ()

Thread 3 (Thread 4022):
#0  0x00002b56e465ac77 in pthread_rwlock_wrlock () from /lib64/libpthread.so.0 #1  0x00005555555752ea in master_source_writelock (entry=<value optimized out>) at master.c:527
#2  0x00005555555631af in handle_packet_missing_direct (ap=0x5555556a11c0, pkt=0x40821ec0) at direct.c:1296
#3  0x000055555555fa1d in handle_packet (ap=<value optimized out>) at automount.c:1029
#4  handle_mounts (ap=<value optimized out>) at automount.c:1551 #5  0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0 #6  0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6
#7  0x0000000000000000 in ?? ()

Thread 2 (Thread 4019):
#0  0x00002b56e465b326 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000055555556c8b3 in st_queue_handler (arg=<value optimized out>) at state.c:1104
#2  0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0 #3  0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6 #4  0x0000000000000000 in ?? ()

Quoted 9 lines

Thread 1 (Thread 4018):
#0  0x00002b56e465b517 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000555555573762 in alarm_handler (arg=<value optimized out>) at alarm.c:206
#2  0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0 #3  0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6 #4  0x0000000000000000 in ?? ()

It looks like a lock ordering bug to me and I think it's the same hang that Steve Thompson reported.

Quoted 7 lines

This will be hard to test I think, hopefully Mike or Steve will be able to use a couple of machines for testing, even so it may take a while before we know.

Could you prepare a test package for Mike using this patch please. It should apply OK but of course you'll need to discard or fix the CHANGELOG hunk.

Ian, thanks for the patch. I built the test package and forwarded it to Mike.

Quoted 1 line

I'll prepare a test package for Steve soon as I get time.

I took the liberty of building a test package for CentOS and RHEL too. I used autofs-5.0.1-0.rc2.143.el5_5.6.src.rpm as base and added your patch. The packages are now available at:

http://download.opensuse.org/repositories/home:/leonardocf:/autofs/

Unfortunately I failed to persuade the Build Service to keep the package release number, so Steve might need to pass --oldpackage to rpm.

Thanks,
Leonardo

#10 lore raw
In reply to Ian Kent
Quoted 14 lines · Ian Kent
Quoted 5 lines · Ian Kent
Quoted 4 lines · Leonardo Chiquitto
Quoted 3 lines · Ian Kent
Quoted 2 lines · Mike Marion

On Mon, Feb 14, 2011 at 07:37:01PM -0800, Ian Kent wrote:

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 1 line

That is kernel revision and autofs revision?

2.6.16.60-0.59.1 (Sles10 sp3 with an updated, but not bleeding edge, patch). autofs 5.0.5 with most of the patches up to a couple months ago.  It's hard to get exacts because it's a PTF from Novell (we really pushed them to upgrade to 5.0.5) but it should be pretty much equal to the patch they just released for sle 11 sp1 that they're recommending as they default going forward.

Quoted 4 lines

Now I'm confused?

I thought that Mike had mention he had seen hangs, similar to Steve, and this backtrace was an example of that.

This is right. The backtrace I posted was sent to me as an example of a hang.

Quoted 3 lines

But the mail thread doesn't read
like that and oddly enough I seem to have identified a locking problem from looking at the code based on the backtrace.

I'm sorry that we ended up hijacking the "non-expiring mounts" thread.

Leonardo

#12 lore raw
In reply to Ian Kent
Quoted 215 lines · Ian Kent
Quoted 12 lines · Leonardo Chiquitto
Quoted 11 lines · Ian Kent
Quoted 9 lines · Mike Marion
Quoted 1 line · Ian Kent

That is kernel revision and autofs revision?

2.6.16.60-0.59.1 (Sles10 sp3 with an updated, but not bleeding edge, patch). autofs 5.0.5 with most of the patches up to a couple months ago.  It's hard to get exacts because it's a PTF from Novell (we really pushed them to upgrade to 5.0.5) but it should be pretty much equal to the patch they just released for sle 11 sp1 that they're recommending as they default going forward.

Quoted 7 lines
Quoted 5 lines

That make it hard, as you know.

But I wouldn't mind spending a bit of time on it, if you can also.

Let's assume that it's a user space problem for now.

Here are the call traces for all automount processes on the kernel side:

I think it's a user space problem.

snip ...

Quoted 99 lines

And here are the call traces from the user land daemon:

Thread 9 (Thread 4017):
#0  0x00002b56e465d6a8 in __lll_mutex_lock_wait () from /lib64/libpthread.so.0 #1  0x00002b56e46599fb in _L_mutex_lock_92 () from /lib64/libpthread.so.0 #2  0x00002b56e4659455 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3  0x00005555555746cd in master_mutex_lock () at master.c:49
#4  0x000055555555d260 in do_hup_signal (master=0x55555568d010, age=1296063258) at automount.c:1276
#5  0x0000555555560bd3 in statemachine (arg=<value optimized out>) at automount.c:1354
#6  main (arg=<value optimized out>) at automount.c:2142

Thread 8 (Thread 20702):
#0  0x00002b56e4dd62a7 in brk () from /lib64/libc.so.6
#1  0x0000555555577dfe in expire (logopt=2, cmd=<value optimized out>, fd=21, ioctlfd=21, path=0x55555569ca20 "/usr2", arg=0x41c27ef4)
    at dev-ioctl-lib.c:657
#2  0x0000555555577ebe in ioctl_expire (logopt=21, ioctlfd=-1, path=0x55555569ca20 "/usr2", when=0) at dev-ioctl-lib.c:701 #3  0x0000555555561e4e in expire_proc_indirect (arg=<value optimized out>) at indirect.c:545
#4  0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0 #5  0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 7 (Thread 7060):
#0  0x00002b56e465ac77 in pthread_rwlock_wrlock () from /lib64/libpthread.so.0 #1  0x00005555555752ea in master_source_writelock (entry=<value optimized out>) at master.c:527
#2  0x0000555555575f8f in master_add_map_source (entry=0x5555556a10b0, type=0x0, format=0x0, age=1296059657, argc=1, argv=<value optimized out>) at master.c:191
#3  0x0000555555579ee3 in master_parse_entry (buffer=<value optimized out>, default_timeout=86400, logging=<value optimized out>, age=1296059657) at master_parse.y:823
#4  0x00002aaaaaab83fe in lookup_read_master (master=<value optimized out>, age=1296059657, context=<value optimized out>) at lookup_ldap.c:1625 #5  0x0000555555569052 in do_read_master (master=0x55555568d010, type=<value optimized out>, age=1296059657) at lookup.c:96
#6  0x000055555556aa3c in lookup_nss_read_master (master=0x55555568d010, age=1296059657) at lookup.c:229
#7  0x0000555555575c28 in master_read_master (master=0x55555568d010, age=1296059657, readall=1) at master.c:831
#8  0x000055555555d844 in do_read_master (arg=<value optimized out>) at automount.c:1259
#9  0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0 #10 0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6
#11 0x0000000000000000 in ?? ()

Thread 6 (Thread 6851):
#0  0x00002b56e465aa3d in pthread_rwlock_rdlock () from /lib64/libpthread.so.0 #1  0x000055555556deb6 in cache_readlock (mc=0x55555568e5b8) at cache.c:60 #2  0x000055555556baff in do_readmap (arg=<value optimized out>) at state.c:479 #3  0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0 #4  0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6
#5  0x0000000000000000 in ?? ()

Thread 5 (Thread 4026):
#0  0x00002b56e465d6a8 in __lll_mutex_lock_wait () from /lib64/libpthread.so.0 #1  0x00002b56e46599fb in _L_mutex_lock_92 () from /lib64/libpthread.so.0 #2  0x00002b56e4659455 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3  0x00005555555746cd in master_mutex_lock () at master.c:49
#4  0x0000555555560ff9 in handle_packet_missing_indirect (ap=0x55555569c940, pkt=0x41823ec0) at indirect.c:808
#5  0x000055555555fa32 in handle_packet (ap=<value optimized out>) at automount.c:1026
#6  handle_mounts (ap=<value optimized out>) at automount.c:1551 #7  0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0 #8  0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6
#9  0x0000000000000000 in ?? ()

Thread 4 (Thread 4025):
#0  0x00002b56e4dd4b26 in fts_read () from /lib64/libc.so.6 #1  0x00007fff37314e80 in ?? ()
#2  0x0000000000000000 in ?? ()

Thread 3 (Thread 4022):
#0  0x00002b56e465ac77 in pthread_rwlock_wrlock () from /lib64/libpthread.so.0 #1  0x00005555555752ea in master_source_writelock (entry=<value optimized out>) at master.c:527
#2  0x00005555555631af in handle_packet_missing_direct (ap=0x5555556a11c0, pkt=0x40821ec0) at direct.c:1296
#3  0x000055555555fa1d in handle_packet (ap=<value optimized out>) at automount.c:1029
#4  handle_mounts (ap=<value optimized out>) at automount.c:1551 #5  0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0 #6  0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6
#7  0x0000000000000000 in ?? ()

Thread 2 (Thread 4019):
#0  0x00002b56e465b326 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000055555556c8b3 in st_queue_handler (arg=<value optimized out>) at state.c:1104
#2  0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0 #3  0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6 #4  0x0000000000000000 in ?? ()

Quoted 9 lines

Thread 1 (Thread 4018):
#0  0x00002b56e465b517 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000555555573762 in alarm_handler (arg=<value optimized out>) at alarm.c:206
#2  0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0 #3  0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6 #4  0x0000000000000000 in ?? ()

It looks like a lock ordering bug to me and I think it's the same hang that Steve Thompson reported.

This will be hard to test I think, hopefully Mike or Steve will be able to use a couple of machines for testing, even so it may take a while before we know.

Could you prepare a test package for Mike using this patch please. It should apply OK but of course you'll need to discard or fix the CHANGELOG hunk.

I'll prepare a test package for Steve soon as I get time.

autofs-5.0.5 - fix out of order locking in readmap

From: Ian Kent <[email protected]>

When re-reading the master map and a mount lookup for a particular map entry arrives at the same time a map source is being added to the map entry and a map entry re-read is also initiated at the same time an out of order mutex locking bug which results in a hang.
---

 CHANGELOG    |    1 +
 lib/master.c |    8 ++------

 2 files changed, 3 insertions(+), 6 deletions(-)
CHANGELOG +0 -0
index 11054da..631094f 100644
@@ -55,6 +55,7 @@

 - add external bind method.
 - fix add simple bind auth.
 - add option to dump configured automount maps.
+- fix out of order locking in readmap.

 03/09/2009 autofs-5.0.5
 -----------------------

lib/master.c +1 -6
index 95bd3fb..daedf66 100644
@@ -611,8 +611,6 @@ struct master_mapent *master_find_mapent(struct master *master, const char *path
  {
        struct list_head *head, *p;
 
-       master_mutex_lock();
-
        head = &master->mounts;
        list_for_each(p, head) {
                struct master_mapent *entry;
@@ -625,8 +623,6 @@ struct master_mapent *master_find_mapent(struct master *master, const char *path
                }
        }
 
-       master_mutex_unlock();
-
        return NULL;
  }
 
@@ -703,9 +699,7 @@ struct master_mapent *master_new_mapent(struct master *master, const char *path,
 
  void master_add_mapent(struct master *master, struct master_mapent *entry)
  {
-       master_mutex_lock();
        list_add_tail(&entry->list, &master->mounts);
-       master_mutex_unlock();
        return;
  }
 
@@ -813,6 +807,7 @@ int master_read_master(struct master *master, time_t age, int readall)
         * We need to clear and re-populate the null map entry cache
         * before alowing anyone else to use it.
         */
+       master_mutex_lock();
        if (master->nc) {
                cache_writelock(master->nc);
                nc = master->nc;

lib/master.c int master_read_master(struct master *master, time_t age, int readall) · line 832

Quoted 7 lines
lib/master.c +1 -0
@@ -831,6 +826,7 @@ int master_read_master(struct master *master, time_t age, int readall)
        master_init_scan();
        lookup_nss_read_master(master, age);
        cache_unlock(nc);
+       master_mutex_unlock();
 
        if (!master->read_fail)
                master_mount_mounts(master, age, readall);

Hello Ian,

I just got a reply confirming that the hangs are no longer reproducible with this
patch applied. Please, do you think you can consider it for upstream inclusion?

Thanks,
Leonardo

maintainer
#13 lore raw
Quoted 221 lines · Leonardo Chiquitto
Quoted 220 lines · Ian Kent
Quoted 12 lines · Leonardo Chiquitto
Quoted 11 lines · Ian Kent
Quoted 9 lines · Mike Marion
Quoted 1 line · Ian Kent

That is kernel revision and autofs revision?

2.6.16.60-0.59.1 (Sles10 sp3 with an updated, but not bleeding edge, patch). autofs 5.0.5 with most of the patches up to a couple months ago. It's hard to get exacts because it's a PTF from Novell (we really pushed them to upgrade to 5.0.5) but it should be pretty much equal to the patch they just released for sle 11 sp1 that they're recommending as they default going forward.

Quoted 7 lines
Quoted 5 lines

That make it hard, as you know.

But I wouldn't mind spending a bit of time on it, if you can also.

Let's assume that it's a user space problem for now.

Here are the call traces for all automount processes on the kernel side:

I think it's a user space problem.

snip ...

Quoted 99 lines

And here are the call traces from the user land daemon:

Thread 9 (Thread 4017):

#0  0x00002b56e465d6a8 in __lll_mutex_lock_wait () from /lib64/libpthread.so.0
#1  0x00002b56e46599fb in _L_mutex_lock_92 () from /lib64/libpthread.so.0
#2  0x00002b56e4659455 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00005555555746cd in master_mutex_lock () at master.c:49
#4  0x000055555555d260 in do_hup_signal (master=0x55555568d010, age=1296063258)
    at automount.c:1276
#5  0x0000555555560bd3 in statemachine (arg=<value optimized out>)
    at automount.c:1354

#6 main (arg=<value optimized out>) at automount.c:2142

Thread 8 (Thread 20702):
#0 0x00002b56e4dd62a7 in brk () from /lib64/libc.so.6
#1 0x0000555555577dfe in expire (logopt=2, cmd=<value optimized out>, fd=21,

    ioctlfd=21, path=0x55555569ca20 "/usr2", arg=0x41c27ef4)
    at dev-ioctl-lib.c:657
#2  0x0000555555577ebe in ioctl_expire (logopt=21, ioctlfd=-1,
    path=0x55555569ca20 "/usr2", when=0) at dev-ioctl-lib.c:701
#3  0x0000555555561e4e in expire_proc_indirect (arg=<value optimized out>)
    at indirect.c:545
#4  0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0
#5  0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 7 (Thread 7060):
#0 0x00002b56e465ac77 in pthread_rwlock_wrlock () from /lib64/libpthread.so.0 #1 0x00005555555752ea in master_source_writelock (entry=<value optimized out>)

    at master.c:527
#2  0x0000555555575f8f in master_add_map_source (entry=0x5555556a10b0,
    type=0x0, format=0x0, age=1296059657, argc=1, argv=<value optimized out>)
    at master.c:191
#3  0x0000555555579ee3 in master_parse_entry (buffer=<value optimized out>,
    default_timeout=86400, logging=<value optimized out>, age=1296059657)
    at master_parse.y:823
#4  0x00002aaaaaab83fe in lookup_read_master (master=<value optimized out>,
    age=1296059657, context=<value optimized out>) at lookup_ldap.c:1625
#5  0x0000555555569052 in do_read_master (master=0x55555568d010,
    type=<value optimized out>, age=1296059657) at lookup.c:96
#6  0x000055555556aa3c in lookup_nss_read_master (master=0x55555568d010,
    age=1296059657) at lookup.c:229
#7  0x0000555555575c28 in master_read_master (master=0x55555568d010,
    age=1296059657, readall=1) at master.c:831
#8  0x000055555555d844 in do_read_master (arg=<value optimized out>)
    at automount.c:1259

#9 0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0 #10 0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6 #11 0x0000000000000000 in ?? ()

Thread 6 (Thread 6851):

#0  0x00002b56e465aa3d in pthread_rwlock_rdlock () from /lib64/libpthread.so.0
#1  0x000055555556deb6 in cache_readlock (mc=0x55555568e5b8) at cache.c:60
#2  0x000055555556baff in do_readmap (arg=<value optimized out>) at state.c:479
#3  0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0
#4  0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6
#5  0x0000000000000000 in ?? ()

Thread 5 (Thread 4026):

#0  0x00002b56e465d6a8 in __lll_mutex_lock_wait () from /lib64/libpthread.so.0
#1  0x00002b56e46599fb in _L_mutex_lock_92 () from /lib64/libpthread.so.0
#2  0x00002b56e4659455 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00005555555746cd in master_mutex_lock () at master.c:49
#4  0x0000555555560ff9 in handle_packet_missing_indirect (ap=0x55555569c940,
    pkt=0x41823ec0) at indirect.c:808
#5  0x000055555555fa32 in handle_packet (ap=<value optimized out>)
    at automount.c:1026
#6  handle_mounts (ap=<value optimized out>) at automount.c:1551
#7  0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0
#8  0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6
#9  0x0000000000000000 in ?? ()

Thread 4 (Thread 4025):

#0  0x00002b56e4dd4b26 in fts_read () from /lib64/libc.so.6
#1  0x00007fff37314e80 in ?? ()
#2  0x0000000000000000 in ?? ()

Thread 3 (Thread 4022):
#0 0x00002b56e465ac77 in pthread_rwlock_wrlock () from /lib64/libpthread.so.0 #1 0x00005555555752ea in master_source_writelock (entry=<value optimized out>)

    at master.c:527
#2  0x00005555555631af in handle_packet_missing_direct (ap=0x5555556a11c0,
    pkt=0x40821ec0) at direct.c:1296
#3  0x000055555555fa1d in handle_packet (ap=<value optimized out>)
    at automount.c:1029
#4  handle_mounts (ap=<value optimized out>) at automount.c:1551
#5  0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0
#6  0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6
#7  0x0000000000000000 in ?? ()

Thread 2 (Thread 4019):
#0 0x00002b56e465b326 in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib64/libpthread.so.0
#1 0x000055555556c8b3 in st_queue_handler (arg=<value optimized out>)

    at state.c:1104
#2  0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0
#3  0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6
#4  0x0000000000000000 in ?? ()
Quoted 9 lines

Thread 1 (Thread 4018):
#0 0x00002b56e465b517 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
from /lib64/libpthread.so.0
#1 0x0000555555573762 in alarm_handler (arg=<value optimized out>)

    at alarm.c:206
#2  0x00002b56e4657193 in start_thread () from /lib64/libpthread.so.0
#3  0x00002b56e4ddcdfd in sysctl () from /lib64/libc.so.6
#4  0x0000000000000000 in ?? ()

It looks like a lock ordering bug to me and I think it's the same hang that Steve Thompson reported.

This will be hard to test I think, hopefully Mike or Steve will be able to use a couple of machines for testing, even so it may take a while before we know.

Could you prepare a test package for Mike using this patch please. It should apply OK but of course you'll need to discard or fix the CHANGELOG hunk.

I'll prepare a test package for Steve soon as I get time.

autofs-5.0.5 - fix out of order locking in readmap

From: Ian Kent <[email protected]>

When re-reading the master map and a mount lookup for a particular map entry arrives at the same time a map source is being added to the map entry and a map entry re-read is also initiated at the same time an out of order mutex locking bug which results in a hang.
---

 CHANGELOG    |    1 +
 lib/master.c |    8 ++------
 2 files changed, 3 insertions(+), 6 deletions(-)
CHANGELOG +1 -0
index 11054da..631094f 100644
@@ -55,6 +55,7 @@
 - add external bind method.
 - fix add simple bind auth.
 - add option to dump configured automount maps.
+- fix out of order locking in readmap.
 
 03/09/2009 autofs-5.0.5
 -----------------------
lib/master.c +2 -6
index 95bd3fb..daedf66 100644
@@ -611,8 +611,6 @@ struct master_mapent *master_find_mapent(struct master *master, const char *path
 {
       struct list_head *head, *p;
 
-       master_mutex_lock();
-
       head = &master->mounts;
       list_for_each(p, head) {
               struct master_mapent *entry;
@@ -625,8 +623,6 @@ struct master_mapent *master_find_mapent(struct master *master, const char *path
               }
       }
 
-       master_mutex_unlock();
-
       return NULL;
 }
 
@@ -703,9 +699,7 @@ struct master_mapent *master_new_mapent(struct master *master, const char *path,
 
 void master_add_mapent(struct master *master, struct master_mapent *entry)
 {
-       master_mutex_lock();
       list_add_tail(&entry->list, &master->mounts);
-       master_mutex_unlock();
       return;
 }
 
@@ -813,6 +807,7 @@ int master_read_master(struct master *master, time_t age, int readall)
        * We need to clear and re-populate the null map entry cache
        * before alowing anyone else to use it.
        */
+       master_mutex_lock();
       if (master->nc) {
               cache_writelock(master->nc);
               nc = master->nc;
@@ -831,6 +826,7 @@ int master_read_master(struct master *master, time_t age, int readall)
       master_init_scan();
       lookup_nss_read_master(master, age);
       cache_unlock(nc);
+       master_mutex_unlock();
 
Quoted 8 lines

lib/master.c int master_read_master(struct master *master, time_t age, int readall) · line 832

Quoted 2 lines
lib/master.c +0 -0
       if (!master->read_fail)
               master_mount_mounts(master, age, readall);

Hello Ian,

I just got a reply confirming that the hangs are no longer reproducible with this
patch applied. Please, do you think you can consider it for upstream inclusion?

Oh wow, that's just plain luck I think.

I'll put the patch back on my commit queue but can't say yet when the next time I will push updates. At least, as long as it's in the queue it will be included in the next release, which is long overdue.

Quoted 3 lines

Thanks,
Leonardo

#14 lore raw
In reply to Philip Ong Jr.
Quoted 6 lines · Philip Ong Jr.

I've pulled down the 5.0.5 tree and all patches.

I've set the logging level to debug and can see messages of expiring mounts in /var/log/messages...but when I check /etc/mtab or /proc/mounts or df, i can still see them there. Any ideas if this is a known issue or if I can give more info than below?

Hello,

I finally had time to return to this issue. To avoid the confusion caused by old kernels, I reproduced the problem on openSUSE Factory (kernel 2.6.38 and autofs 5.0.5 with all kernel.org patches applied). Here's the configuration used:

# cat /etc/auto.master
/data	/etc/auto.data
# cat /etc/auto.data
isos	-fstype=nfs,ro,rsize=8192,wsize=8192,intr,nolock,nosuid libre:/isos
# cat /etc/sysconfig/autofs | grep -v '^#'
AUTOFS_OPTIONS=""
LOCAL_OPTIONS=""
APPEND_OPTIONS="yes"
USE_MISC_DEVICE="yes"
DEFAULT_MASTER_MAP_NAME="auto.master"
DEFAULT_TIMEOUT=600
DEFAULT_BROWSE_MODE="yes"
DEFAULT_LOGGING="debug"
DEFAULT_MAP_OBJECT_CLASS="nisMap"
DEFAULT_ENTRY_OBJECT_CLASS="nisObject"
DEFAULT_MAP_ATTRIBUTE="nisMapName"
DEFAULT_ENTRY_ATTRIBUTE="cn"
DEFAULT_VALUE_ATTRIBUTE="nisMapEntry"
DEFAULT_AUTH_CONF_FILE="etc/autofs_ldap_auth.conf"
MAP_HASH_TABLE_SIZE=1024

I'm attaching the automount debug logs showing the following sequence:

- automount startup
- mount of an NFS volume (/data/isos)
- failed attempt to trigger the expiration of the mounted volume (sending SIGUSR1)
- successful attempt to trigger the expiration of the mounted volume (sending SIGUSR1 again)

I also confirmed that the problem no longer happens if I revert the following commit:

commit 08aafab4c1d0ab6227c80f8cd1086ae78556a370

Author: Ian Kent <[email protected]>
Date:   Thu Sep 9 11:10:47 2010 +0800

autofs-5.0.5 - fix direct map not updating on reread

Philip, do you think you could try to revert it in your setup/package just to confirm this "works"?

Thanks,
Leonardo

== STARTUP

Mar 24 18:41:33 n47 automount[4681]: Starting automounter version 5.0.5, master map auto.master
Mar 24 18:41:33 n47 automount[4681]: using kernel protocol version 5.02
Mar 24 18:41:33 n47 automount[4681]: lookup_nss_read_master: reading master files auto.master
Mar 24 18:41:33 n47 automount[4681]: parse_init: parse(sun): init gathered global options: (null)
Mar 24 18:41:33 n47 automount[4681]: lookup_read_master: lookup(file): read entry /data
Mar 24 18:41:33 n47 automount[4681]: master_do_mount: mounting /data
Mar 24 18:41:33 n47 automount[4681]: automount_path_to_fifo: fifo name /var/run/autofs.fifo-data
Mar 24 18:41:33 n47 automount[4681]: lookup_nss_read_map: reading map file /etc/auto.data
Mar 24 18:41:33 n47 automount[4681]: parse_init: parse(sun): init gathered global options: (null)
Mar 24 18:41:33 n47 automount[4681]: mounted indirect on /data with timeout 600, freq 150 seconds
Mar 24 18:41:33 n47 automount[4681]: st_ready: st_ready(): state = 0 path /data
Mar 24 18:41:33 n47 automount[4681]: ghosting enabled

== COMMAND: ls /data/isos

Mar 24 18:41:54 n47 automount[4681]: handle_packet: type = 3
Mar 24 18:41:54 n47 automount[4681]: handle_packet_missing_indirect: token 3, name isos, request pid 4691
Mar 24 18:41:54 n47 automount[4681]: attempting to mount entry /data/isos
Mar 24 18:41:54 n47 automount[4681]: lookup_mount: lookup(file): looking up isos
Mar 24 18:41:54 n47 automount[4681]: lookup_mount: lookup(file): isos -> -fstype=nfs,ro,rsize=8192,wsize=8192,intr,nolock,nosuid libre:/isos
Mar 24 18:41:54 n47 automount[4681]: parse_mount: parse(sun): expanded entry: -fstype=nfs,ro,rsize=8192,wsize=8192,intr,nolock,nosuid libre:/isos
Mar 24 18:41:54 n47 automount[4681]: parse_mount: parse(sun): gathered options: fstype=nfs,ro,rsize=8192,wsize=8192,intr,nolock,nosuid
Mar 24 18:41:54 n47 automount[4681]: parse_mount: parse(sun): dequote("libre:/isos") -> libre:/isos
Mar 24 18:41:54 n47 automount[4681]: parse_mount: parse(sun): core of entry: options=fstype=nfs,ro,rsize=8192,wsize=8192,intr,nolock,nosuid, loc=libre:/isos
Mar 24 18:41:54 n47 automount[4681]: sun_mount: parse(sun): mounting root /data, mountpoint isos, what libre:/isos, fstype nfs, options ro,rsize=8192,wsize=8192,intr,nolock,nosuid
Mar 24 18:41:54 n47 automount[4681]: mount_mount: mount(nfs): root=/data name=isos what=libre:/isos, fstype=nfs, options=ro,rsize=8192,wsize=8192,intr,nolock,nosuid
Mar 24 18:41:54 n47 automount[4681]: mount_mount: mount(nfs): nfs options="ro,rsize=8192,wsize=8192,intr,nolock,nosuid", nosymlink=0, ro=1
Mar 24 18:41:54 n47 automount[4681]: mount_mount: mount(nfs): calling mkdir_path /data/isos
Mar 24 18:41:54 n47 automount[4681]: mount_mount: mount(nfs): calling mount -t nfs -s -o ro,rsize=8192,wsize=8192,intr,nolock,nosuid libre:/isos /data/isos
Mar 24 18:41:54 n47 automount[4681]: mount_mount: mount(nfs): mounted libre:/isos on /data/isos
Mar 24 18:41:54 n47 automount[4681]: dev_ioctl_send_ready: token = 3
Mar 24 18:41:54 n47 automount[4681]: mounted /data/isos
Mar 24 18:41:54 n47 automount[4681]: st_readmap: state 1 path /data
Mar 24 18:41:54 n47 automount[4681]: re-reading map for /data
Mar 24 18:41:54 n47 automount[4681]: lookup_nss_read_map: reading map file /etc/auto.data
Mar 24 18:41:54 n47 automount[4681]: parse_init: parse(sun): init gathered global options: (null)
Mar 24 18:41:54 n47 automount[4681]: st_ready: st_ready(): state = 4 path /data

== COMMAND: kill -USR1 $(pidof automount)

Mar 24 18:42:07 n47 automount[4681]: do_notify_state: signal 10
Mar 24 18:42:07 n47 automount[4681]: master_notify_state_change: sig 10 switching /data from 1 to 3

== COMMAND: kill -USR1 $(pidof automount)

Mar 24 18:42:45 n47 automount[4681]: do_notify_state: signal 10
Mar 24 18:42:45 n47 automount[4681]: master_notify_state_change: sig 10 switching /data from 1 to 3
Mar 24 18:42:45 n47 automount[4681]: st_prune: state 1 path /data
Mar 24 18:42:45 n47 automount[4681]: expire_proc: exp_proc = 140224412317440 path /data
Mar 24 18:42:45 n47 automount[4681]: expire_proc_indirect: expire /data/isos
Mar 24 18:42:45 n47 automount[4681]: handle_packet: type = 4
Mar 24 18:42:45 n47 automount[4681]: handle_packet_expire_indirect: token 4, name isos
Mar 24 18:42:45 n47 automount[4681]: expiring path /data/isos
Mar 24 18:42:45 n47 automount[4681]: umount_multi: path /data/isos incl 1
Mar 24 18:42:45 n47 automount[4681]: umount_subtree_mounts: unmounting dir = /data/isos
Mar 24 18:42:45 n47 automount[4681]: expired /data/isos
Mar 24 18:42:45 n47 automount[4681]: dev_ioctl_send_ready: token = 4
Mar 24 18:42:45 n47 automount[4681]: expire_cleanup: got thid 140224412317440 path /data stat 0
Mar 24 18:42:45 n47 automount[4681]: expire_cleanup: sigchld: exp 140224412317440 finished, switching from 3 to 1
Mar 24 18:42:45 n47 automount[4681]: st_ready: st_ready(): state = 3 path /data

_______________________________________________
autofs mailing list
[email protected]
http://linux.kernel.org/mailman/listinfo/autofs

maintainer
#15 lore raw
Quoted 41 lines · Leonardo Chiquitto

I finally had time to return to this issue. To avoid the confusion caused by old kernels, I reproduced the problem on openSUSE Factory (kernel 2.6.38 and autofs 5.0.5 with all kernel.org patches applied). Here's the configuration used:

# cat /etc/auto.master
/data	/etc/auto.data
# cat /etc/auto.data
isos	-fstype=nfs,ro,rsize=8192,wsize=8192,intr,nolock,nosuid libre:/isos
# cat /etc/sysconfig/autofs | grep -v '^#'
AUTOFS_OPTIONS=""
LOCAL_OPTIONS=""
APPEND_OPTIONS="yes"
USE_MISC_DEVICE="yes"
DEFAULT_MASTER_MAP_NAME="auto.master"
DEFAULT_TIMEOUT=600
DEFAULT_BROWSE_MODE="yes"
DEFAULT_LOGGING="debug"
DEFAULT_MAP_OBJECT_CLASS="nisMap"
DEFAULT_ENTRY_OBJECT_CLASS="nisObject"
DEFAULT_MAP_ATTRIBUTE="nisMapName"
DEFAULT_ENTRY_ATTRIBUTE="cn"
DEFAULT_VALUE_ATTRIBUTE="nisMapEntry"
DEFAULT_AUTH_CONF_FILE="etc/autofs_ldap_auth.conf"
MAP_HASH_TABLE_SIZE=1024

I'm attaching the automount debug logs showing the following sequence:

- automount startup
- mount of an NFS volume (/data/isos)
- failed attempt to trigger the expiration of the mounted volume (sending SIGUSR1)
- successful attempt to trigger the expiration of the mounted volume (sending SIGUSR1 again)

I also confirmed that the problem no longer happens if I revert the following commit:

Quoted 8 lines

commit 08aafab4c1d0ab6227c80f8cd1086ae78556a370

Author: Ian Kent <[email protected]>
Date:   Thu Sep 9 11:10:47 2010 +0800

autofs-5.0.5 - fix direct map not updating on reread

Philip, do you think you could try to revert it in your setup/package just to confirm this "works"?

The problem was actually introduced by autofs-5.0.5-remove-state-machine-timed-wait.patch.

There had been a long standing bug in the state queue handling which I thought was a pthreads problem. When I added the above patch everything appeared to work OK.

But the change you mentioned above exposed the bug and the log you provided allowed me to work out what was broken and fix it.

But the stale map processing has also been broken somewhere along the way and I'm also working on fixing that.

Once that's done there's not much more needed for 5.0.6, at last.

Ian