Seeing some 5.0.1 stop expiring mounts
autofs-5.0.1 with patches:
autofs-5.0.1-bad-cast.patch autofs-5.0.1-check-mtab-updated.patch autofs-5.0.1-check-user-info-return.patch autofs-5.0.1-cmd-global-options-fix.patch autofs-5.0.1-cmd-global-options.patch autofs-5.0.1-code-cleanups.patch autofs-5.0.1-conf-append-global.patch autofs-5.0.1-configure-cleanups.patch autofs-5.0.1-correct-hesiod-check.patch autofs-5.0.1-disable-exports-check.patch autofs-5.0.1-drop-default-prefix-from-config.patch autofs-5.0.1-export-check-network-fix-2.patch autofs-5.0.1-file-map-allow-white-space-only-line.patch autofs-5.0.1-fix-browse-dir-create.patch autofs-5.0.1-hosts-simple-fail.patch autofs-5.0.1-localfs-label-check.patch autofs-5.0.1-map-update-source-only.patch autofs-5.0.1-network_match-fix.patch autofs-5.0.1-null-domain-fix.patch autofs-5.0.1-random-selection.patch autofs-5.0.1-remove-macro-automount-8.patch autofs-5.0.1-remove-redundant-ident-macros.patch autofs-5.0.1-update-kernel-patches.patch
kernel 2.6.16.21-0.9 on sles9-sp3 (took sles10 kernel src.rpm and built new kernel on top of sles9).
All maps in LDAP. Startup is fine... 6200+ entries in /proc/mounts (Yes we have that many). Some hosts get into a state where they aren't expiring mounts anymore:
$ mount -t nfs | wc
476 2856 70188Differences I see in ps are that there's another thread hanging around (wedged?):
ps axsm output:
0 2840 0000000000010201 - -
- - ? 384:58 automoun 0 - 0000000000000000
fffffffe7ffbfeff 0000000000000000 0000000180000000 Tsl -
0:00 -
0 - 0000000000000000 fffffffe7ffbfeff 0000000000000000
0000000180000000 Ssl - 0:00 -
0 - 0000000000000000 fffffffe7ffbfeff 0000000000000000
0000000180000000 Ssl - 0:16 -
0 - 0000000000000000 fffffffe7ffbfeff 0000000000000000
0000000180000000 Ssl - 0:04 -
0 - 0000000000000000 fffffffe7ffbfeff 0000000000000000
0000000180000000 Ssl - 0:10 -
0 - 0000000000000000 fffffffe7ffbfeff 0000000000000000
0000000180000000 Ssl - 0:09 -
0 - 0000000000000000 fffffffe7ffbfeff 0000000000000000
0000000180000000 Ssl - 0:00 -
0 - 0000000000000000 fffffffe7ffbfeff 0000000000000000
0000000180000000 Ssl - 0:00 -
0 - 0000000000000000 fffffffe7ffbfeff 00000000000000000000000180000000 Ssl - 0:02 -
ps -eLF output:
root 2840 1 2840 0 9 15576 7208 2 Jul05 ?
00:00:00 automount
root 2840 1 2841 0 9 15576 7208 2 Jul05 ?
00:00:00 automount
root 2840 1 2842 0 9 15576 7208 2 Jul05 ?
00:00:16 automount
root 2840 1 2845 0 9 15576 7208 1 Jul05 ?
00:00:04 automount
root 2840 1 2848 0 9 15576 7208 0 Jul05 ?
00:00:10 automount
root 2840 1 2849 0 9 15576 7208 3 Jul05 ?
00:00:09 automount
root 2840 1 32417 0 9 15576 7208 2 Jul13 ?
00:00:00 automount
root 2840 1 2230 0 9 15576 7208 3 Jul14 ?
00:00:00 automount
root 2840 1 2235 0 9 15576 7208 1 Jul14 ?
00:00:02 automount
Hosts that are ok spawn a new thread periodically, the wedged hosts don't. Kill -USR1/HUP don't seem to have any effect, even stracing process while sending that signal shows that it never seems to see it (no SIGUSR1 info in strace).
Strace shows host stuck doing nothing but futex and time calls like this:
[pid 2840] futex(0x555555685ac4, FUTEX_WAIT, 2949131, {0, 993887000}) = -1 ETIMEDOUT (Connection timed out)
[pid 2840] futex(0x555555685a80, FUTEX_WAKE, 1) = 0
[pid 2840] clock_gettime(CLOCK_REALTIME, {1184701903, 6010000}) = 0
[pid 2840] futex(0x555555685ac4, FUTEX_WAIT, 2949133, {0, 993990000}) =
-1 ETIMEDOUT (Connection timed out)
[pid 2840] futex(0x555555685a80, FUTEX_WAKE, 1) = 0
[pid 2840] clock_gettime(CLOCK_REALTIME, {1184701904, 5912000}) = 0
[pid 2840] futex(0x555555685ac4, FUTEX_WAIT, 2949135, {0, 994088000}) =
-1 ETIMEDOUT (Connection timed out)
[pid 2840] futex(0x555555685a80, FUTEX_WAKE, 1) = 0
[pid 2840] clock_gettime(CLOCK_REALTIME, {1184701905, 5758000}) = 0
[pid 2840] futex(0x555555685ac4, FUTEX_WAIT, 2949137, {0, 994242000}) =
-1 ETIMEDOUT (Connection timed out)
[pid 2840] futex(0x555555685a80, FUTEX_WAKE, 1) = 0
[pid 2840] clock_gettime(CLOCK_REALTIME, {1184701906, 5657000}) = 0
[pid 2840] futex(0x555555685ac4, FUTEX_WAIT, 2949139, {0, 994343000}) =
-1 ETIMEDOUT (Connection timed out)
[pid 2840] futex(0x555555685a80, FUTEX_WAKE, 1) = 0
[pid 2840] clock_gettime(CLOCK_REALTIME, {1184701907, 5625000}) = 0
[pid 2840] futex(0x555555685ac4, FUTEX_WAIT, 2949141, {0, 994375000}) =
-1 ETIMEDOUT (Connection timed out)
[pid 2840] futex(0x555555685a80, FUTEX_WAKE, 1) = 0
[pid 2840] clock_gettime(CLOCK_REALTIME, {1184701908, 5506000}) = 0
[pid 2840] futex(0x555555685ac4, FUTEX_WAIT, 2949143, {0, 994494000}) =-1 ETIMEDOUT (Connection timed out)
[pid 2840] futex(0x555555685a80, FUTEX_WAKE, 1) = 0
[pid 2840] clock_gettime(CLOCK_REALTIME, {1184701909, 5422000}) = 0
Wondering if anyone else has seen and/or has any idea what might be the cause.
Signature · 4 lines
Mike Marion-Unix SysAdmin/Staff IT Engineer-http://www.qualcomm.com "I've never used their tech support, but the word is that it sucks." I believe it's composed entirely of monkeys that couldn't get the Shakespeare gig. :-) ==> /. users talking about @home tech support
Quoted 2 lines · Mike Marion
Wondering if anyone else has seen and/or has any idea what might be the cause.
Hmm.. might've found the cause at least.. common thread appears to be OOM situation hit on hosts in this state. Yes, our users do this a LOT.. it's a big pain, we're trying to find a balance between enforcing limits without killing batch jobs incorrectly.
Luckily autofs5 seems to be the first autofs IME that you can kill -9 then start a new one and have it take over the mounts properly.
Signature · 2 lines
Mike Marion-Unix SysAdmin/Staff IT Engineer-http://www.qualcomm.com My pid is Inigo Montoya. You kill -9 my parent process. Prepare to vi.
Ian Kent
maintainerQuoted 5 lines · Mike Marion
Quoted 2 lines · Mike Marion
Wondering if anyone else has seen and/or has any idea what might be the cause.
Quoted 7 lines
Hmm.. might've found the cause at least.. common thread appears to be OOM situation hit on hosts in this state. Yes, our users do this a LOT.. it's a big pain, we're trying to find a balance between enforcing limits without killing batch jobs incorrectly.
Luckily autofs5 seems to be the first autofs IME that you can kill -9 then start a new one and have it take over the mounts properly.
Yep, it's seems to work well so far.
If you configure with --enable-ignore-busy (and for good measure --disable-mount-locking) then you should be able to achieve the same result with "service autofs restart", no kill needed at all.
Ian
Ian Kent
maintainerQuoted 17 lines · Mike Marion
autofs-5.0.1 with patches:
autofs-5.0.1-bad-cast.patch autofs-5.0.1-check-mtab-updated.patch autofs-5.0.1-check-user-info-return.patch autofs-5.0.1-cmd-global-options-fix.patch autofs-5.0.1-cmd-global-options.patch autofs-5.0.1-code-cleanups.patch autofs-5.0.1-conf-append-global.patch autofs-5.0.1-configure-cleanups.patch autofs-5.0.1-correct-hesiod-check.patch autofs-5.0.1-disable-exports-check.patch autofs-5.0.1-drop-default-prefix-from-config.patch autofs-5.0.1-export-check-network-fix-2.patch autofs-5.0.1-file-map-allow-white-space-only-line.patch autofs-5.0.1-fix-browse-dir-create.patch autofs-5.0.1-hosts-simple-fail.patch
Quoted 8 lines
autofs-5.0.1-localfs-label-check.patch
autofs-5.0.1-map-update-source-only.patch
autofs-5.0.1-network_match-fix.patch
autofs-5.0.1-null-domain-fix.patch
autofs-5.0.1-random-selection.patch
autofs-5.0.1-remove-macro-automount-8.patch
autofs-5.0.1-remove-redundant-ident-macros.patch
autofs-5.0.1-update-kernel-patches.patch
Might be worth considering going to 5.0.2, especially since you have a busy site, as a nasty deadlock in the alarm handler has been fixed.
There are patches for 5.0.1 on kernel.org as well.
Ian
Quoted 2 lines · Ian Kent
Might be worth considering going to 5.0.2, especially since you have a busy site, as a nasty deadlock in the alarm handler has been fixed.
Unfortunately, we can't. I've since found out (through trial and error) that the patch:
Quoted 1 line
Quoted 1 line
autofs-5.0.1-map-update-source-only.patch
is completely broken for us, and it appears to be part of the 5.0.2 codebase now.
Our main setup for auotofs5 clients now is pure ldap.. auto_master is in ldap, nsswitch.conf has
automount: ldap
everything is in ldap. We remove all /etc/auto.* files too.
With the above patch applied to 5.0.1 (or using 5.0.2) as soon as the daemon gets a HUP signal, it flushes out all the auto.projects (our main map) entries from /proc/mounts and they're gone forever.
When first started, and until the daemon gets a HUP, it works fine. Our/proc/mounts has 6200+ entries (we have a crapload of paths) and
they'll mount great. Entries look as expected, e.g.:
auto.projects /prj/qct/gv autofs rw,fd=6,pgrp=2571,timeout=600,minproto=5,maxproto=5,direct 0 0
then if you mount it, it adds in:
ronald:/vol/eng_ice_0014/qct_gv /prj/qct/gv nfs rw,v3,rsize=32768,wsize=32768,acregmin=1,acregmax=5,acdirmin=1,acdirmax=5,hard,lock,proto=tcp,addr=ronald 0 0
After the HUP, the thing flushes, then logs a ton of rm_dir errors.. like so:
Start of daemon:
automount[2571]: mounted direct mount on /prj/qct/gv with timeout 600, freq 150 seconds
Flush after HUP:
automount[2572]: umounted direct mount /prj/qct/gv
After all umounts.. these errors show for every path:
automount[2549]: rmdir_path: lstat of /prj/qct/gv failed.
I did a test with 5.0.1 with all patchs sans the autofs-5.0.1-map-update-source-only.patch and it's fine... I can HUP left and right, do kill -USR1 to flush, etc. Works right. But rebuild again with that patch and first HUP breaks all our auto.projects paths. Weird thing is the /net and /usr2 (indirect home dirs) stay working. Those entries look like:
$ egrep 'auto.home|/net' /proc/mounts | grep -v auto.projects -hosts /net autofs rw,fd=9,pgrp=13948,timeout=600,minproto=5,maxproto=5,indirect 0 0 auto.home /usr2 autofs rw,fd=14,pgrp=13948,timeout=600,minproto=5,maxproto=5,indirect 0 0
I can try to get more info out of the ldap setup if needed, but unfortunately the admin that really knows the ldap side is out until next week. I can poke around some though, and use his docs.
Signature · 3 lines
Mike Marion-Unix SysAdmin/Staff IT Engineer-http://www.qualcomm.com The needs of the many outweigh the needs of the few, or the one. That's from Trek. Pretty cool, huh? -- Dennis Miller
Ian Kent
maintainerQuoted 8 lines · Mike Marion
Quoted 2 lines · Ian Kent
Might be worth considering going to 5.0.2, especially since you have a busy site, as a nasty deadlock in the alarm handler has been fixed.
Unfortunately, we can't. I've since found out (through trial and error) that the patch:
Quoted 1 line
Quoted 1 line
autofs-5.0.1-map-update-source-only.patch
Oops!
Quoted 38 lines
is completely broken for us, and it appears to be part of the 5.0.2 codebase now.
Our main setup for auotofs5 clients now is pure ldap..
auto_masteris in ldap, nsswitch.conf has
automount: ldap
everything is in ldap. We remove all/etc/auto.*files too.With the above patch applied to 5.0.1 (or using 5.0.2) as soon as the daemon gets a HUP signal, it flushes out all the auto.projects (our main map) entries from
/proc/mountsand they're gone forever.When first started, and until the daemon gets a HUP, it works fine. Our
/proc/mountshas 6200+ entries (we have a crapload of paths) and
they'll mount great. Entries look as expected, e.g.:
auto.projects /prj/qct/gv autofs rw,fd=6,pgrp=2571,timeout=600,minproto=5,maxproto=5,direct 0 0
then if you mount it, it adds in:
ronald:/vol/eng_ice_0014/qct_gv /prj/qct/gv nfs rw,v3,rsize=32768,wsize=32768,acregmin=1,acregmax=5,acdirmin=1,acdirmax=5,hard,lock,proto=tcp,addr=ronald 0 0After the HUP, the thing flushes, then logs a ton of
rm_direrrors.. like so:Start of daemon:
automount[2571]: mounted direct mount on /prj/qct/gv with timeout 600, freq 150 secondsFlush after HUP:
automount[2572]: umounted direct mount /prj/qct/gvAfter all umounts.. these errors show for every path:
automount[2549]:rmdir_path: lstat of /prj/qct/gv failed.I did a test with 5.0.1 with all patchs sans the autofs-5.0.1-map-update-source-only.patch and it's fine... I can HUP left and right, do kill -USR1 to flush, etc. Works right. But rebuild again with that patch and first HUP breaks all our auto.projects paths. Weird thing is the /net and /usr2 (indirect home dirs) stay working. Those entries look like:
Quoted 3 lines
$ egrep 'auto.home|/net' /proc/mounts | grep -v auto.projects -hosts /net autofs rw,fd=9,pgrp=13948,timeout=600,minproto=5,maxproto=5,indirect 0 0 auto.home /usr2 autofs rw,fd=14,pgrp=13948,timeout=600,minproto=5,maxproto=5,indirect 0 0
So it sounds like the direct mounts are unhappy with this or possibly the LDAP module.
I'll try to duplicate the problem.
Ian
Ian Kent
maintainerQuoted 11 lines · Mike Marion
Quoted 2 lines · Ian Kent
Might be worth considering going to 5.0.2, especially since you have a busy site, as a nasty deadlock in the alarm handler has been fixed.
Unfortunately, we can't. I've since found out (through trial and error) that the patch:
Quoted 1 line
Quoted 1 line
autofs-5.0.1-map-update-source-only.patch
is completely broken for us, and it appears to be part of the 5.0.2 codebase now.
Indeed.
I think I've identified the problem but I'll need to look more closely at it.
In the meantime it would be great if you could try this patch, in addition to the patch above, and check if there are any other problems with it.
The patch is against the current git repo but applies against a fairly up to date 5.0.1 tree (it looks correct) with a small offset.
Ian
---
daemon/lookup.c +2 -0
@@ -325,6 +325,7 @@ static int read_file_source_instance(struct autofs_point *ap, struct map_source instance->recurse = map->recurse; instance->depth = map->depth; } + instance->stale = map->stale;return do_read_map (ap, instance, age); }
@@ -346,6 +347,7 @@ static int read_source_instance(struct autofs_point *ap, struct map_source *map, instance->recurse = map->recurse; instance->depth = map->depth; } + instance->stale = map->stale;return do_read_map (ap, instance, age); }
Ian Kent
maintainerQuoted 5 lines · Mike Marion
Quoted 2 lines · Ian Kent
Might be worth considering going to 5.0.2, especially since you have a busy site, as a nasty deadlock in the alarm handler has been fixed.
snip ...
Quoted 3 lines
With the above patch applied to 5.0.1 (or using 5.0.2) as soon as the daemon gets a HUP signal, it flushes out all the auto.projects (our main map) entries from
/proc/mountsand they're gone forever.
fwiw, the map-update-source-only patch is quite an important change from the point of view of the version 5 design and the correct operation of the map update logic. The reason it was needed is that I changed the way instances of maps are sorted during initial development but didn't properly account for it.
I really appreciate any time you can spare to help with this.
Thanks
Ian
Quoted 1 line · Ian Kent
I really appreciate any time you can spare to help with this.
No problem. I'm finishing up some stuff needed today but should have time to try out the other patch and see if it still has that expire problem tomorrow.
Signature · 4 lines
Mike Marion-Unix SysAdmin/Staff IT Engineer-http://www.qualcomm.com Girl: "She dumped ya, didn't she?" Drew: "It was mutual. We both hated me, and she was the only one that had nerve enough to leave." ==> Drew Cary Show
Ian Kent
maintainerQuoted 7 lines · Mike Marion
Quoted 1 line · Ian Kent
I really appreciate any time you can spare to help with this.
No problem. I'm finishing up some stuff needed today but should have time to try out the other patch and see if it still has that expire problem tomorrow.
Don't mean to be a pain but have you had time to check this?
Ian
Quoted 1 line · Ian Kent
Don't mean to be a pain but have you had time to check this?
Not a pain.. too much got in the way this week. I was able to test that patch both against 5.0.1 with the autofs-5.0.1-map-update-source-only.patch
and against 5.0.2 and it's good.
Just to show, here's 5.0.2 without those 2 lines patched in2
vbr2-001:/usr/src/packages/SOURCES # ps auxw | grep automount
root 6812 0.7 0.0 28552 7180 ? Ssl 13:59 0:00
automount
root 6935 0.0 0.0 2888 704 pts/1 S+ 14:00 0:00 grep
automount
vbr2-001:/usr/src/packages/SOURCES # wc /proc/mounts; mount | wc ; kill -HUP 6812; while [ 1 ]; do wc /proc/mounts ; mount | wc; sleep 5; done 6314 37884 730725 /proc/mounts
12 72 441
6314 37884 730725 /proc/mounts
12 72 441
6107 36642 706276 /proc/mounts
12 72 441
5964 35784 689905 /proc/mounts
12 72 441
5833 34998 674567 /proc/mounts
12 72 441
5695 34170 658506 /proc/mounts
12 72 441
5477 32862 633119 /proc/mounts
12 72 441and on down until all the auto.projects entries are gone... which takes awhile.
The syslog shows them all being unmounted, unlike 5.0.1 it only shows the "rmdir lmstat of <path> failed" messages for things that were actually nfs mounted vs the entire auto.projects tree though.
Signature · 7 lines
Mike Marion-Unix SysAdmin/Staff IT Engineer-http://www.qualcomm.com Lisa: "As you know, we've been swimming. And we've developed a taste for it. We agree that getting our own pool is the way to go. Now before you respond, you should know that your refusal will result in months and months of..." Bart, Lisa: "CanwehaveapoolDad? CanwehaveapoolDad? CanwehaveapoolDad? CanwehaveapoolDad? CanwehaveapoolDad?" Homer: "I understand. Let us celebrate our agreement with the adding of chocolate to milk." -- Simpsons
Ian Kent
maintainerQuoted 8 lines · Mike Marion
Quoted 1 line · Ian Kent
Don't mean to be a pain but have you had time to check this?
Not a pain.. too much got in the way this week. I was able to test that patch both against 5.0.1 with the autofs-5.0.1-map-update-source-only.patch
and against 5.0.2 and it's good.
Cool .. I'll commit the patch to the repo and deal with any other issues as we need this anyway.
Thanks
Quoted 33 lines
Just to show, here's 5.0.2 without those 2 lines patched in2
vbr2-001:/usr/src/packages/SOURCES # ps auxw | grep automount
root 6812 0.7 0.0 28552 7180 ? Ssl 13:59 0:00
automount
root 6935 0.0 0.0 2888 704 pts/1 S+ 14:00 0:00 grep
automount
vbr2-001:/usr/src/packages/SOURCES # wc/proc/mounts; mount | wc ; kill -HUP 6812; while [ 1 ]; do wc/proc/mounts; mount | wc; sleep 5; done 6314 37884 730725/proc/mounts12 72 441 6314 37884 730725 /proc/mounts 12 72 441 6107 36642 706276 /proc/mounts 12 72 441 5964 35784 689905 /proc/mounts 12 72 441 5833 34998 674567 /proc/mounts 12 72 441 5695 34170 658506 /proc/mounts 12 72 441 5477 32862 633119 /proc/mounts 12 72 441and on down until all the auto.projects entries are gone... which takes awhile.
The syslog shows them all being unmounted, unlike 5.0.1 it only shows the "rmdir lmstat of <path> failed" messages for things that were actually nfs mounted vs the entire auto.projects tree though.