Skip to contentSkip to conversation

Seeing some 5.0.1 stop expiring mounts

DISCUSSION Mike Marion 2 participants 12 messages Jul 17, 2007 – Jul 28, 2007 Kernel automounter

lore
Share & tools
lore
#0 lore raw

autofs-5.0.1 with patches:

autofs-5.0.1-bad-cast.patch autofs-5.0.1-check-mtab-updated.patch autofs-5.0.1-check-user-info-return.patch autofs-5.0.1-cmd-global-options-fix.patch autofs-5.0.1-cmd-global-options.patch autofs-5.0.1-code-cleanups.patch autofs-5.0.1-conf-append-global.patch autofs-5.0.1-configure-cleanups.patch autofs-5.0.1-correct-hesiod-check.patch autofs-5.0.1-disable-exports-check.patch autofs-5.0.1-drop-default-prefix-from-config.patch autofs-5.0.1-export-check-network-fix-2.patch autofs-5.0.1-file-map-allow-white-space-only-line.patch autofs-5.0.1-fix-browse-dir-create.patch autofs-5.0.1-hosts-simple-fail.patch autofs-5.0.1-localfs-label-check.patch autofs-5.0.1-map-update-source-only.patch autofs-5.0.1-network_match-fix.patch autofs-5.0.1-null-domain-fix.patch autofs-5.0.1-random-selection.patch autofs-5.0.1-remove-macro-automount-8.patch autofs-5.0.1-remove-redundant-ident-macros.patch autofs-5.0.1-update-kernel-patches.patch

kernel 2.6.16.21-0.9 on sles9-sp3 (took sles10 kernel src.rpm and built new kernel on top of sles9).

All maps in LDAP. Startup is fine... 6200+ entries in /proc/mounts (Yes we have that many). Some hosts get into a state where they aren't expiring mounts anymore:

$ mount -t nfs | wc
    476    2856   70188

Differences I see in ps are that there's another thread hanging around (wedged?):
ps axsm output:

    0  2840 0000000000010201                -                -
- -    ?        384:58 automoun    0     - 0000000000000000
  fffffffe7ffbfeff 0000000000000000 0000000180000000 Tsl  -
0:00 -
    0     - 0000000000000000 fffffffe7ffbfeff 0000000000000000
0000000180000000 Ssl  -          0:00 -
    0     - 0000000000000000 fffffffe7ffbfeff 0000000000000000
0000000180000000 Ssl  -          0:16 -
    0     - 0000000000000000 fffffffe7ffbfeff 0000000000000000
0000000180000000 Ssl  -          0:04 -
    0     - 0000000000000000 fffffffe7ffbfeff 0000000000000000
0000000180000000 Ssl  -          0:10 -
    0     - 0000000000000000 fffffffe7ffbfeff 0000000000000000
0000000180000000 Ssl  -          0:09 -
    0     - 0000000000000000 fffffffe7ffbfeff 0000000000000000
0000000180000000 Ssl  -          0:00 -
    0     - 0000000000000000 fffffffe7ffbfeff 0000000000000000
0000000180000000 Ssl  -          0:00 -
    0     - 0000000000000000 fffffffe7ffbfeff 0000000000000000

0000000180000000 Ssl - 0:02 -

ps -eLF output:
root 2840 1 2840 0 9 15576 7208 2 Jul05 ?
00:00:00 automount
root 2840 1 2841 0 9 15576 7208 2 Jul05 ?
00:00:00 automount
root 2840 1 2842 0 9 15576 7208 2 Jul05 ?
00:00:16 automount
root 2840 1 2845 0 9 15576 7208 1 Jul05 ?
00:00:04 automount
root 2840 1 2848 0 9 15576 7208 0 Jul05 ?
00:00:10 automount
root 2840 1 2849 0 9 15576 7208 3 Jul05 ?
00:00:09 automount
root 2840 1 32417 0 9 15576 7208 2 Jul13 ?
00:00:00 automount
root 2840 1 2230 0 9 15576 7208 3 Jul14 ?
00:00:00 automount
root 2840 1 2235 0 9 15576 7208 1 Jul14 ?
00:00:02 automount

Hosts that are ok spawn a new thread periodically, the wedged hosts don't. Kill -USR1/HUP don't seem to have any effect, even stracing process while sending that signal shows that it never seems to see it (no SIGUSR1 info in strace).

Strace shows host stuck doing nothing but futex and time calls like this:
[pid 2840] futex(0x555555685ac4, FUTEX_WAIT, 2949131, {0, 993887000}) = -1 ETIMEDOUT (Connection timed out)

[pid  2840] futex(0x555555685a80, FUTEX_WAKE, 1) = 0
[pid  2840] clock_gettime(CLOCK_REALTIME, {1184701903, 6010000}) = 0
[pid  2840] futex(0x555555685ac4, FUTEX_WAIT, 2949133, {0, 993990000}) =
-1 ETIMEDOUT (Connection timed out)
[pid  2840] futex(0x555555685a80, FUTEX_WAKE, 1) = 0
[pid  2840] clock_gettime(CLOCK_REALTIME, {1184701904, 5912000}) = 0
[pid  2840] futex(0x555555685ac4, FUTEX_WAIT, 2949135, {0, 994088000}) =
-1 ETIMEDOUT (Connection timed out)
[pid  2840] futex(0x555555685a80, FUTEX_WAKE, 1) = 0
[pid  2840] clock_gettime(CLOCK_REALTIME, {1184701905, 5758000}) = 0
[pid  2840] futex(0x555555685ac4, FUTEX_WAIT, 2949137, {0, 994242000}) =
-1 ETIMEDOUT (Connection timed out)
[pid  2840] futex(0x555555685a80, FUTEX_WAKE, 1) = 0
[pid  2840] clock_gettime(CLOCK_REALTIME, {1184701906, 5657000}) = 0
[pid  2840] futex(0x555555685ac4, FUTEX_WAIT, 2949139, {0, 994343000}) =
-1 ETIMEDOUT (Connection timed out)
[pid  2840] futex(0x555555685a80, FUTEX_WAKE, 1) = 0
[pid  2840] clock_gettime(CLOCK_REALTIME, {1184701907, 5625000}) = 0
[pid  2840] futex(0x555555685ac4, FUTEX_WAIT, 2949141, {0, 994375000}) =
-1 ETIMEDOUT (Connection timed out)
[pid  2840] futex(0x555555685a80, FUTEX_WAKE, 1) = 0
[pid  2840] clock_gettime(CLOCK_REALTIME, {1184701908, 5506000}) = 0
[pid  2840] futex(0x555555685ac4, FUTEX_WAIT, 2949143, {0, 994494000}) =

-1 ETIMEDOUT (Connection timed out)
[pid 2840] futex(0x555555685a80, FUTEX_WAKE, 1) = 0
[pid 2840] clock_gettime(CLOCK_REALTIME, {1184701909, 5422000}) = 0

Wondering if anyone else has seen and/or has any idea what might be the cause.

Signature · 4 lines
Mike Marion-Unix SysAdmin/Staff IT Engineer-http://www.qualcomm.com
"I've never used their tech support, but the word is that it sucks."
I believe it's composed entirely of monkeys that couldn't get the Shakespeare
gig. :-) ==> /. users talking about @home tech support
#1 lore raw
Quoted 2 lines · Mike Marion

Wondering if anyone else has seen and/or has any idea what might be the cause.

Hmm.. might've found the cause at least.. common thread appears to be OOM situation hit on hosts in this state. Yes, our users do this a LOT.. it's a big pain, we're trying to find a balance between enforcing limits without killing batch jobs incorrectly.

Luckily autofs5 seems to be the first autofs IME that you can kill -9 then start a new one and have it take over the mounts properly.

Signature · 2 lines
Mike Marion-Unix SysAdmin/Staff IT Engineer-http://www.qualcomm.com
My pid is Inigo Montoya.  You kill -9 my parent process. Prepare to vi.
maintainer
#2 lore raw
Quoted 5 lines · Mike Marion
Quoted 2 lines · Mike Marion

Wondering if anyone else has seen and/or has any idea what might be the cause.

Quoted 7 lines

Hmm.. might've found the cause at least.. common thread appears to be OOM situation hit on hosts in this state. Yes, our users do this a LOT.. it's a big pain, we're trying to find a balance between enforcing limits without killing batch jobs incorrectly.

Luckily autofs5 seems to be the first autofs IME that you can kill -9 then start a new one and have it take over the mounts properly.

Yep, it's seems to work well so far.

If you configure with --enable-ignore-busy (and for good measure --disable-mount-locking) then you should be able to achieve the same result with "service autofs restart", no kill needed at all.

Ian

maintainer
#3 lore raw
In reply to Mike Marion
Quoted 17 lines · Mike Marion

autofs-5.0.1 with patches:

autofs-5.0.1-bad-cast.patch autofs-5.0.1-check-mtab-updated.patch autofs-5.0.1-check-user-info-return.patch autofs-5.0.1-cmd-global-options-fix.patch autofs-5.0.1-cmd-global-options.patch autofs-5.0.1-code-cleanups.patch autofs-5.0.1-conf-append-global.patch autofs-5.0.1-configure-cleanups.patch autofs-5.0.1-correct-hesiod-check.patch autofs-5.0.1-disable-exports-check.patch autofs-5.0.1-drop-default-prefix-from-config.patch autofs-5.0.1-export-check-network-fix-2.patch autofs-5.0.1-file-map-allow-white-space-only-line.patch autofs-5.0.1-fix-browse-dir-create.patch autofs-5.0.1-hosts-simple-fail.patch

Quoted 8 lines

autofs-5.0.1-localfs-label-check.patch
autofs-5.0.1-map-update-source-only.patch
autofs-5.0.1-network_match-fix.patch
autofs-5.0.1-null-domain-fix.patch
autofs-5.0.1-random-selection.patch
autofs-5.0.1-remove-macro-automount-8.patch
autofs-5.0.1-remove-redundant-ident-macros.patch
autofs-5.0.1-update-kernel-patches.patch

Might be worth considering going to 5.0.2, especially since you have a busy site, as a nasty deadlock in the alarm handler has been fixed.

There are patches for 5.0.1 on kernel.org as well.

Ian

#4 lore raw
Quoted 2 lines · Ian Kent

Might be worth considering going to 5.0.2, especially since you have a busy site, as a nasty deadlock in the alarm handler has been fixed.

Unfortunately, we can't. I've since found out (through trial and error) that the patch:

Quoted 1 line
Quoted 1 line

autofs-5.0.1-map-update-source-only.patch

is completely broken for us, and it appears to be part of the 5.0.2 codebase now.

Our main setup for auotofs5 clients now is pure ldap.. auto_master is in ldap, nsswitch.conf has
automount: ldap
everything is in ldap. We remove all /etc/auto.* files too.

With the above patch applied to 5.0.1 (or using 5.0.2) as soon as the daemon gets a HUP signal, it flushes out all the auto.projects (our main map) entries from /proc/mounts and they're gone forever.

When first started, and until the daemon gets a HUP, it works fine. Our
/proc/mounts has 6200+ entries (we have a crapload of paths) and
they'll mount great. Entries look as expected, e.g.:
auto.projects /prj/qct/gv autofs rw,fd=6,pgrp=2571,timeout=600,minproto=5,maxproto=5,direct 0 0
then if you mount it, it adds in:
ronald:/vol/eng_ice_0014/qct_gv /prj/qct/gv nfs rw,v3,rsize=32768,wsize=32768,acregmin=1,acregmax=5,acdirmin=1,acdirmax=5,hard,lock,proto=tcp,addr=ronald 0 0

After the HUP, the thing flushes, then logs a ton of rm_dir errors.. like so:

Start of daemon:
automount[2571]: mounted direct mount on /prj/qct/gv with timeout 600, freq 150 seconds

Flush after HUP:
automount[2572]: umounted direct mount /prj/qct/gv

After all umounts.. these errors show for every path:
automount[2549]: rmdir_path: lstat of /prj/qct/gv failed.

I did a test with 5.0.1 with all patchs sans the autofs-5.0.1-map-update-source-only.patch and it's fine... I can HUP left and right, do kill -USR1 to flush, etc. Works right. But rebuild again with that patch and first HUP breaks all our auto.projects paths. Weird thing is the /net and /usr2 (indirect home dirs) stay working. Those entries look like:

$ egrep 'auto.home|/net' /proc/mounts | grep -v auto.projects
-hosts /net autofs rw,fd=9,pgrp=13948,timeout=600,minproto=5,maxproto=5,indirect 0 0
auto.home /usr2 autofs rw,fd=14,pgrp=13948,timeout=600,minproto=5,maxproto=5,indirect 0 0

I can try to get more info out of the ldap setup if needed, but unfortunately the admin that really knows the ldap side is out until next week. I can poke around some though, and use his docs.

Signature · 3 lines
Mike Marion-Unix SysAdmin/Staff IT Engineer-http://www.qualcomm.com
The needs of the many outweigh the needs of the few, or the one.  That's from
Trek.  Pretty cool, huh? -- Dennis Miller
maintainer
#5 lore raw
Quoted 8 lines · Mike Marion
Quoted 2 lines · Ian Kent

Might be worth considering going to 5.0.2, especially since you have a busy site, as a nasty deadlock in the alarm handler has been fixed.

Unfortunately, we can't. I've since found out (through trial and error) that the patch:

Quoted 1 line
Quoted 1 line

autofs-5.0.1-map-update-source-only.patch

Oops!

Quoted 38 lines

is completely broken for us, and it appears to be part of the 5.0.2 codebase now.

Our main setup for auotofs5 clients now is pure ldap.. auto_master is in ldap, nsswitch.conf has
automount: ldap
everything is in ldap. We remove all /etc/auto.* files too.

With the above patch applied to 5.0.1 (or using 5.0.2) as soon as the daemon gets a HUP signal, it flushes out all the auto.projects (our main map) entries from /proc/mounts and they're gone forever.

When first started, and until the daemon gets a HUP, it works fine. Our
/proc/mounts has 6200+ entries (we have a crapload of paths) and
they'll mount great. Entries look as expected, e.g.:
auto.projects /prj/qct/gv autofs rw,fd=6,pgrp=2571,timeout=600,minproto=5,maxproto=5,direct 0 0
then if you mount it, it adds in:
ronald:/vol/eng_ice_0014/qct_gv /prj/qct/gv nfs rw,v3,rsize=32768,wsize=32768,acregmin=1,acregmax=5,acdirmin=1,acdirmax=5,hard,lock,proto=tcp,addr=ronald 0 0

After the HUP, the thing flushes, then logs a ton of rm_dir errors.. like so:

Start of daemon:
automount[2571]: mounted direct mount on /prj/qct/gv with timeout 600, freq 150 seconds

Flush after HUP:
automount[2572]: umounted direct mount /prj/qct/gv

After all umounts.. these errors show for every path:
automount[2549]: rmdir_path: lstat of /prj/qct/gv failed.

I did a test with 5.0.1 with all patchs sans the autofs-5.0.1-map-update-source-only.patch and it's fine... I can HUP left and right, do kill -USR1 to flush, etc. Works right. But rebuild again with that patch and first HUP breaks all our auto.projects paths. Weird thing is the /net and /usr2 (indirect home dirs) stay working. Those entries look like:

Quoted 3 lines
$ egrep 'auto.home|/net' /proc/mounts | grep -v auto.projects
-hosts /net autofs rw,fd=9,pgrp=13948,timeout=600,minproto=5,maxproto=5,indirect 0 0
auto.home /usr2 autofs rw,fd=14,pgrp=13948,timeout=600,minproto=5,maxproto=5,indirect 0 0

So it sounds like the direct mounts are unhappy with this or possibly the LDAP module.

I'll try to duplicate the problem.

Ian

maintainer
#6 lore raw
In reply to Mike Marion
Quoted 11 lines · Mike Marion
Quoted 2 lines · Ian Kent

Might be worth considering going to 5.0.2, especially since you have a busy site, as a nasty deadlock in the alarm handler has been fixed.

Unfortunately, we can't. I've since found out (through trial and error) that the patch:

Quoted 1 line
Quoted 1 line

autofs-5.0.1-map-update-source-only.patch

is completely broken for us, and it appears to be part of the 5.0.2 codebase now.

Indeed.

I think I've identified the problem but I'll need to look more closely at it.

In the meantime it would be great if you could try this patch, in addition to the patch above, and check if there are any other problems with it.

The patch is against the current git repo but applies against a fairly up to date 5.0.1 tree (it looks correct) with a small offset.

Ian

---

daemon/lookup.c +2 -0
index 70b9e02..4f2b318 100644
@@ -325,6 +325,7 @@ static int read_file_source_instance(struct autofs_point *ap, struct map_source
 		instance->recurse = map->recurse;
 		instance->depth = map->depth;
 	}
+	instance->stale = map->stale;
 
 	return do_read_map(ap, instance, age);
 }
@@ -346,6 +347,7 @@ static int read_source_instance(struct autofs_point *ap, struct map_source *map,
 		instance->recurse = map->recurse;
 		instance->depth = map->depth;
 	}
+	instance->stale = map->stale;
 
 	return do_read_map(ap, instance, age);
 }
maintainer
#7 lore raw
In reply to Mike Marion
Quoted 5 lines · Mike Marion
Quoted 2 lines · Ian Kent

Might be worth considering going to 5.0.2, especially since you have a busy site, as a nasty deadlock in the alarm handler has been fixed.

snip ...

Quoted 3 lines

With the above patch applied to 5.0.1 (or using 5.0.2) as soon as the daemon gets a HUP signal, it flushes out all the auto.projects (our main map) entries from /proc/mounts and they're gone forever.

fwiw, the map-update-source-only patch is quite an important change from the point of view of the version 5 design and the correct operation of the map update logic. The reason it was needed is that I changed the way instances of maps are sorted during initial development but didn't properly account for it.

I really appreciate any time you can spare to help with this.

Thanks
Ian

#8 lore raw
Quoted 1 line · Ian Kent

I really appreciate any time you can spare to help with this.

No problem. I'm finishing up some stuff needed today but should have time to try out the other patch and see if it still has that expire problem tomorrow.

Signature · 4 lines
Mike Marion-Unix SysAdmin/Staff IT Engineer-http://www.qualcomm.com
Girl: "She dumped ya, didn't she?"
Drew: "It was mutual.  We both hated me, and she was the only one that had 
nerve enough to leave."  ==> Drew Cary Show
maintainer
#9 lore raw
Quoted 7 lines · Mike Marion
Quoted 1 line · Ian Kent

I really appreciate any time you can spare to help with this.

No problem. I'm finishing up some stuff needed today but should have time to try out the other patch and see if it still has that expire problem tomorrow.

Don't mean to be a pain but have you had time to check this?

Ian

#10 lore raw
Quoted 1 line · Ian Kent

Don't mean to be a pain but have you had time to check this?

Not a pain.. too much got in the way this week. I was able to test that patch both against 5.0.1 with the autofs-5.0.1-map-update-source-only.patch
and against 5.0.2 and it's good.

Just to show, here's 5.0.2 without those 2 lines patched in2

vbr2-001:/usr/src/packages/SOURCES # ps auxw | grep automount
root 6812 0.7 0.0 28552 7180 ? Ssl 13:59 0:00
automount
root 6935 0.0 0.0 2888 704 pts/1 S+ 14:00 0:00 grep
automount
vbr2-001:/usr/src/packages/SOURCES # wc /proc/mounts; mount | wc ; kill -HUP 6812; while [ 1 ]; do wc /proc/mounts ; mount | wc; sleep 5; done 6314 37884 730725 /proc/mounts

     12      72     441
6314 37884 730725 /proc/mounts
     12      72     441
6107 36642 706276 /proc/mounts
     12      72     441
5964 35784 689905 /proc/mounts
     12      72     441
5833 34998 674567 /proc/mounts
     12      72     441
5695 34170 658506 /proc/mounts
     12      72     441
5477 32862 633119 /proc/mounts
     12      72     441

and on down until all the auto.projects entries are gone... which takes awhile.

The syslog shows them all being unmounted, unlike 5.0.1 it only shows the "rmdir lmstat of <path> failed" messages for things that were actually nfs mounted vs the entire auto.projects tree though.

Signature · 7 lines
Mike Marion-Unix SysAdmin/Staff IT Engineer-http://www.qualcomm.com
Lisa: "As you know, we've been swimming. And we've developed a taste for it. We
agree that getting our own pool is the way to go. Now before you respond, you 
should know that your refusal will result in months and months of..."
Bart, Lisa: "CanwehaveapoolDad? CanwehaveapoolDad? CanwehaveapoolDad? 
CanwehaveapoolDad? CanwehaveapoolDad?"
Homer: "I understand. Let us celebrate our agreement with the adding of chocolate to milk." -- Simpsons
maintainer
#11 lore raw
Quoted 8 lines · Mike Marion
Quoted 1 line · Ian Kent

Don't mean to be a pain but have you had time to check this?

Not a pain.. too much got in the way this week. I was able to test that patch both against 5.0.1 with the autofs-5.0.1-map-update-source-only.patch
and against 5.0.2 and it's good.

Cool .. I'll commit the patch to the repo and deal with any other issues as we need this anyway.

Thanks

Quoted 33 lines

Just to show, here's 5.0.2 without those 2 lines patched in2

vbr2-001:/usr/src/packages/SOURCES # ps auxw | grep automount
root 6812 0.7 0.0 28552 7180 ? Ssl 13:59 0:00
automount
root 6935 0.0 0.0 2888 704 pts/1 S+ 14:00 0:00 grep
automount
vbr2-001:/usr/src/packages/SOURCES # wc /proc/mounts; mount | wc ; kill -HUP 6812; while [ 1 ]; do wc /proc/mounts ; mount | wc; sleep 5; done 6314 37884 730725 /proc/mounts

     12      72     441
6314 37884 730725 /proc/mounts
     12      72     441
6107 36642 706276 /proc/mounts
     12      72     441
5964 35784 689905 /proc/mounts
     12      72     441
5833 34998 674567 /proc/mounts
     12      72     441
5695 34170 658506 /proc/mounts
     12      72     441
5477 32862 633119 /proc/mounts
     12      72     441

and on down until all the auto.projects entries are gone... which takes awhile.

The syslog shows them all being unmounted, unlike 5.0.1 it only shows the "rmdir lmstat of <path> failed" messages for things that were actually nfs mounted vs the entire auto.projects tree though.