Long timeout on broken DNS server
Subsystems (2)
Today I upgraded one of my three DNS servers, the first listed in /etc/resolv.conf, and until I got it back on the air, autofs mounting was glacial. To investigate, on my own workstation I prepended an IP address not assigned to any machine, and reproduced the problem. The symptom is that automount takes 40 to 45 seconds, while other actions involving DNS or hostnames take around 5 seconds, as expected:
"dig math.ucla.edu. soa" and "mount -t nfs tupelo:/h1 /mnt" were tested. The command to test automounting was "ls /net/$HOST/$DIR/.", in this case, "ls /net/serval/h1/." but I use a different host each time to make sure DNS resolution happens. We use nscd, but the result is the same whether or not it is running.
I've appended a debug log, but there is no smoking gun; in fact there is no sign of any misbehavior except for a long delay. Does anyone have any idea why, when autofs resolves a hostname, the DNS timeout is so much longer than the default of 5 seconds?
The distro on the test machine is OpenSuSE 10.3. The autofs version is 5.0.2, although 4.1.4 is affected similarly. Kernel is 2.6.22.17 (SMP).
=-- auto.master --- (comments omitted in all conf files)
/net /etc/auto.net <== giving trouble
/home yp:auto.home
=-- auto.net ---
* -rsize=8192,wsize=8192,retry=1,soft,fstype=autofs,-DSERVER=& file:/etc/auto.net.generic
=-- auto.net.generic ---
* ${SERVER}:/&
(The effect is that when you refer to /net/$HOST/$DIR it starts a sub-daemon for $HOST which then mounts the exported filesystem(s).)
=-- nsswitch.conf --
passwd: compat nis shadow: compat nis group: compat nis
hosts: files dns <== servers tested are not in /etc/hosts networks: files dns
netgroup: nis aliases: files nis
services: files protocols: files rpc: files ethers: files netmasks: files publickey: files
bootparams: files automount: files
=-- resolv.conf ---
nameserver 128.97.4.35 <== dead machine prepended for test
nameserver 128.97.4.254 nameserver 128.97.12.3 search math.ucla.edu pic.ucla.edu
=-- /etc/sysconfig/autofs --- (comments deleted)
AUTOFS_OPTIONS=""
NISMASTERMAP="auto.master" <== map exists but not actually used
UNDERSCORETODOT="yes" <== we have no underscores
LOCAL_OPTIONS=""
APPEND_OPTIONS="yes"
DEFAULT_MASTER_MAP_NAME="auto.master"
DEFAULT_TIMEOUT=600
DEFAULT_BROWSE_MODE="yes"
DEFAULT_LOGGING="none" <== changed to "debug"
DEFAULT_MAP_OBJECT_CLASS="nisMap" <== pro forma; we have no LDAP
DEFAULT_ENTRY_OBJECT_CLASS="nisObject" DEFAULT_MAP_ATTRIBUTE="nisMapName" DEFAULT_ENTRY_ATTRIBUTE="cn" DEFAULT_VALUE_ATTRIBUTE="nisMapEntry" DEFAULT_AUTH_CONF_FILE="etc/autofs_ldap_auth.conf"
=-- Debug output (annotated) ---
Mar 19 21:14:06 simba automount[4468]: handle_packet: type = 3
Mar 19 21:14:06 simba automount[4468]: handle_packet_missing_indirect:
token 41600, name serval, request pid 4529
Mar 19 21:14:06 simba automount[4468]: attempting to mount entry /net/serval
Mar 19 21:14:06 simba automount[4468]: lookup_mount: lookup(file):
looking up serval
Mar 19 21:14:06 simba automount[4468]: lookup_mount: lookup(file):
serval -> -rsize=8192,wsize=8192,retry=1,soft,fstype=autofs,-DSERVER=&
file:/etc/auto.net.generic
Mar 19 21:14:06 simba automount[4468]: parse_mount: parse(sun):
expanded entry: -rsize=8192,wsize=8192,retry=1,soft,fstype=autofs,
-DSERVER=serval file:/etc/auto.net.generic
Mar 19 21:14:06 simba automount[4468]: parse_mount: parse(sun):
gathered options: rsize=8192,wsize=8192,retry=1,soft,fstype=autofs,
-DSERVER=serval
Mar 19 21:14:06 simba automount[4468]: parse_mount: parse(sun):
dequote("file:/etc/auto.net.generic") -> file:/etc/auto.net.generic
Mar 19 21:14:06 simba automount[4468]: parse_mount: parse(sun):
core of entry: options=rsize=8192,wsize=8192,retry=1,soft,
fstype=autofs,-DSERVER=serval, loc=file:/etc/auto.net.generic
Mar 19 21:14:06 simba automount[4468]: sun_mount: parse(sun):
mounting root /net, mountpoint serval, what file:/etc/auto.net.generic,
fstype autofs, options rsize=8192,wsize=8192,retry=1,soft,-DSERVER=serval
Mar 19 21:14:06 simba automount[4468]: do_mount:
file:/etc/auto.net.generic /net/serval type autofs options
rsize=8192,wsize=8192,retry=1,soft,-DSERVER=serval using module autofs
Mar 19 21:14:06 simba automount[4468]: mount_mount: mount(autofs):
fullpath=/net/serval what=file:/etc/auto.net.generic
options=rsize=8192,wsize=8192,retry=1,soft,-DSERVER=serval
Mar 19 21:14:06 simba automount[4468]: lookup_nss_read_map: reading map
file /etc/auto.net.generic
Mar 19 21:14:06 simba automount[4468]: parse_init: parse(sun):
init gathered global options: rsize=8192,wsize=8192,retry=1,soft
Mar 19 21:14:06 simba automount[4468]: mounted indirect mount on
/net/serval with timeout 600, freq 150 seconds
Mar 19 21:14:06 simba automount[4468]: ghosting enabled
Mar 19 21:14:06 simba automount[4468]: send_ready: token = 41600
Mar 19 21:14:06 simba automount[4468]: mounted /net/serval
Mar 19 21:14:06 simba automount[4468]: handle_packet: type = 3
Mar 19 21:14:06 simba automount[4468]: handle_packet_missing_indirect:
token 41601, name m1, request pid 4529
Mar 19 21:14:06 simba automount[4468]: attempting to mount
entry /net/serval/m1
Mar 19 21:14:06 simba automount[4468]: lookup_mount: lookup(file):
looking up m1
Mar 19 21:14:06 simba automount[4468]: lookup_mount: lookup(file):
m1 -> ${SERVER}:/&
Mar 19 21:14:06 simba automount[4468]: parse_mount: parse(sun):
expanded entry: serval:/m1
Mar 19 21:14:06 simba automount[4468]: parse_mount: parse(sun):
gathered options: rsize=8192,wsize=8192,retry=1,soft
Mar 19 21:14:06 simba automount[4468]: parse_mount: parse(sun):
dequote("serval:/m1") -> serval:/m1
Mar 19 21:14:06 simba automount[4468]: parse_mount: parse(sun):
core of entry: options=rsize=8192,wsize=8192,retry=1,soft, loc=serval:/m1
Mar 19 21:14:06 simba automount[4468]: sun_mount: parse(sun):
mounting root /net/serval, mountpoint m1, what serval:/m1, fstype nfs,
options rsize=8192,wsize=8192,retry=1,soft
Mar 19 21:14:06 simba automount[4468]: mount_mount: mount(nfs):
root=/net/serval name=m1 what=serval:/m1, fstype=nfs,
options=rsize=8192,wsize=8192,retry=1,soft
Mar 19 21:14:06 simba automount[4468]: mount_mount: mount(nfs): nfs
options="rsize=8192,wsize=8192,retry=1,soft", nosymlink=0, ro=0=== While waiting, mounts from previous testing expired.
Mar 19 21:14:14 simba automount[4468]: st_expire: state 1 path /net
Mar 19 21:14:14 simba automount[4468]: expire_proc: exp_proc = 3046157200
path /net
Mar 19 21:14:14 simba automount[4468]: expire_proc_indirect:
expire /net/bustamove/m1
Mar 19 21:14:14 simba automount[4468]: st_expire: state 1 path /net/bustamove
Mar 19 21:14:14 simba automount[4468]: expire_proc: exp_proc = 3045104528
path /net/bustamove
Mar 19 21:14:14 simba automount[4468]: expire_proc_indirect:
expire /net/bustamove/m1
Mar 19 21:14:15 simba automount[4468]: expire_proc_indirect: 1 remaining
in /net/bustamove
Mar 19 21:14:15 simba automount[4468]: mount still busy /net/bustamove
Mar 19 21:14:15 simba automount[4468]: expire_cleanup: got thid 3045104528
path /net/bustamove stat 2
Mar 19 21:14:15 simba automount[4468]: expire_cleanup: sigchld:
exp 3045104528 finished, switching from 2 to 1
Mar 19 21:14:15 simba automount[4468]: st_ready: st_ready(): state = 2
path /net/bustamove=== and 2 more, omitted, then it noticed it was working on Serval:
Mar 19 21:14:17 simba automount[4468]: mount still busy /net/serval
Mar 19 21:14:17 simba automount[4468]: expire_cleanup: got thid 3045104528
path /net/serval stat 0
Mar 19 21:14:17 simba automount[4468]: expire_cleanup: sigchld:
exp 3045104528 finished, switching from 2 to 1
Mar 19 21:14:17 simba automount[4468]: st_ready: st_ready(): state = 2
path /net/serval
Mar 19 21:14:17 simba automount[4468]: st_expire: state 1 path /net/simba=== omitted stuff for (not) expiring Simba and Julia
Mar 19 21:14:22 simba automount[4468]: expire_proc_indirect:
4 submounts remaining in /net
Mar 19 21:14:22 simba automount[4468]: expire_proc_indirect:
4 remaining in /net
Mar 19 21:14:22 simba automount[4468]: mount still busy /net
Mar 19 21:14:22 simba automount[4468]: expire_cleanup: got thid 3046157200
path /net stat 5
Mar 19 21:14:22 simba automount[4468]: expire_cleanup: sigchld:
exp 3046157200 finished, switching from 2 to 1
Mar 19 21:14:22 simba automount[4468]: st_ready: st_ready():
state = 2 path /net=== It sits there for 20 seconds more...
Mar 19 21:14:41 simba automount[4468]: mount_mount: mount(nfs):
calling mkdir_path /net/serval/m1
Mar 19 21:14:41 simba automount[4468]: mount_mount: mount(nfs): calling
mount -t nfs -s -o rsize=8192,wsize=8192,retry=1,soft
serval:/m1 /net/serval/m1=== Looks like a normal 5 second timeout in /sbin/mount
Mar 19 21:14:46 simba automount[4468]: mount(nfs):
mounted serval:/m1 on /net/serval/m1
Mar 19 21:14:46 simba automount[4468]: send_ready: token = 41601
Mar 19 21:14:46 simba automount[4468]: mounted /net/serval/m1=== and the content was delivered.
James F. Carter Voice 310 825 2897 FAX 310 206 6673
UCLA-Mathnet; 6115 MSA; 405 Hilgard Ave.; Los Angeles, CA, USA 90095-1555
Email: [email protected] http://www.math.ucla.edu/~jimc (q.v. for PGP key)
Ian Kent
maintainerQuoted 11 lines · Jim Carter
Today I upgraded one of my three DNS servers, the first listed in
/etc/resolv.conf, and until I got it back on the air, autofs mounting was glacial. To investigate, on my own workstation I prepended an IP address not assigned to any machine, and reproduced the problem. The symptom is that automount takes 40 to 45 seconds, while other actions involving DNS or hostnames take around 5 seconds, as expected:
"dig math.ucla.edu. soa" and "mount -t nfs tupelo:/h1 /mnt" were tested. The command to test automounting was "ls /net/$HOST/$DIR/.", in this case, "ls /net/serval/h1/." but I use a different host each time to make sure DNS resolution happens. We use nscd, but the result is the same whether or not it is running.
Not sure about the 5 seconds, I thought the timeout was longer, but I think autofs is doing DNS lookus for the same name more often than it needs to.
I'll have a look and see if I can put something together for you to test.
Ian
Quoted 8 lines · Ian Kent
Quoted 3 lines · Jim Carter
Today I upgraded one of my three DNS servers, the first listed in
/etc/resolv.conf, and until I got it back on the air, autofs mounting was glacial...Not sure about the 5 seconds, I thought the timeout was longer, but I think autofs is doing DNS lookus for the same name more often than it needs to.
The current man page for resolv.conf(5) says the default timeout is 5 secs, and the apps behave as if that were true. There's an option to tweak the timeout, and it's been suggested to shorten the time, since on a local lookup if you don't get an answer in 1 sec you're never going to get it -- but the timeout applies to global lookups too, which can take longer legitimately, if several stages of forwarders and recursion are involved.
Quoted 2 lines
I'll have a look and see if I can put something together for you to test.
Thanks for looking into this -- I'll test the package on one of the machines running 5.0.2 presently.
James F. Carter Voice 310 825 2897 FAX 310 206 6673
UCLA-Mathnet; 6115 MSA; 520 Portola Plaza; Los Angeles, CA, USA 90095-1555
Email: [email protected] http://www.math.ucla.edu/~jimc (q.v. for PGP key)
Ian Kent
maintainerQuoted 17 lines · Jim Carter
Quoted 8 lines · Ian Kent
Quoted 3 lines · Jim Carter
Today I upgraded one of my three DNS servers, the first listed in
/etc/resolv.conf, and until I got it back on the air, autofs mounting was glacial...Not sure about the 5 seconds, I thought the timeout was longer, but I think autofs is doing DNS lookus for the same name more often than it needs to.
The current man page for
resolv.conf(5)says the default timeout is 5 secs, and the apps behave as if that were true. There's an option to tweak the timeout, and it's been suggested to shorten the time, since on a local lookup if you don't get an answer in 1 sec you're never going to get it -- but the timeout applies to global lookups too, which can take longer legitimately, if several stages of forwarders and recursion are involved.
Quoted 5 lines
Quoted 2 lines
I'll have a look and see if I can put something together for you to test.
Thanks for looking into this -- I'll test the package on one of the machines running 5.0.2 presently.
Try this patch and let me know what happens.
Not sure it will actually make much difference. I've made it against the autofs in opensuse, revision 30.2.
---
include/rpc_subs.h renamed from include/rpc_subs.h.dns-name-lookup +3 -1
@@ -46,6 +46,8 @@struct conn_info {const char *host; +const char *addr; +size_t addr_len;unsigned short port;unsigned long program;unsigned long version;
@@ -61,7 +63,7 @@ int rpc_udp_getclient(struct conn_info *void rpc_destroy_udp_client (struct conn_info *);int rpc_tcp_getclient (struct conn_info *,unsigned int ,unsigned int );void rpc_destroy_tcp_client (struct conn_info *); -int rpc_portmap_getclient (struct conn_info *,const char *,const char *,unsigned int ); +int rpc_portmap_getclient (struct conn_info *,const char *,const char *,size_t ,const char *,unsigned int );unsigned short rpc_portmap_getport (struct conn_info *,struct pmap *);int rpc_ping_proto (struct conn_info *);int rpc_ping (const char *,long ,long ,unsigned int );
include/replicated.h renamed from include/replicated.h.dns-name-lookup +1 -0
@@ -52,6 +52,7 @@struct host {char *name;char *addr; +size_t addr_len;char *path;unsigned int version;unsigned int proximity;
modules/replicated.c renamed from modules/replicated.c.dns-name-lookup +19 -6
@@ -227,7 +227,9 @@ static unsigned int get_proximity(constreturn PROXIMITY_OTHER ; } -static struct host *new_host (const char *name,const char *addr,unsigned int proximity,unsigned int weight) +static struct host *new_host (const char *name, +const char *addr,size_t addr_len, +unsigned int proximity,unsigned int weight) {struct host *new;char *tmp1, *tmp2;
@@ -239,11 +241,12 @@ static struct host *new_host(const charif (!tmp1)return NULL ; - tmp2 =strdup (addr); + tmp2 =malloc (addr_len);if (!tmp2) {free (tmp1);return NULL ; } +memcpy (tmp2, addr, addr_len); new =malloc (sizeof (struct host ));if (!new) {
@@ -255,6 +258,7 @@ static struct host *new_host(const charmemset (new,0 ,sizeof (struct host )); new->name = tmp1; + new->addr_len = addr_len; new->addr = tmp2; new->proximity = proximity; new->weight = weight;
@@ -439,7 +443,8 @@ static unsigned int get_nfs_info(struct v3_ver:if (!have_port_opt) { status =rpc_portmap_getclient (pm_info, - host->name, proto,RPC_CLOSE_DEFAULT ); + host->name, host->addr, host->addr_len, + proto,RPC_CLOSE_DEFAULT );if (!status) goto done_ver; }
@@ -551,6 +556,8 @@ static int get_vers_and_cost(struct host timeout =RPC_TIMEOUT *8 ; rpc_info.host = host->name; + rpc_info.addr = host->addr; + rpc_info.addr_len = host->addr_len; rpc_info.program =NFS_PROGRAM ; rpc_info.timeout.tv_sec = timeout; rpc_info.close_option =RPC_CLOSE_DEFAULT ;
@@ -600,6 +607,8 @@ static int get_supported_ver_and_cost(st timeout =RPC_TIMEOUT *8 ; rpc_info.host = host->name; + rpc_info.addr = host->addr; + rpc_info.addr_len = host->addr_len; rpc_info.program =NFS_PROGRAM ; rpc_info.timeout.tv_sec = timeout; rpc_info.close_option =RPC_CLOSE_DEFAULT ;
@@ -627,7 +636,8 @@ static int get_supported_ver_and_cost(streturn 0 ; }else {int ret =rpc_portmap_getclient (&pm_info, - host->name, proto,RPC_CLOSE_DEFAULT ); + host->name, host->addr, host->addr_len, + proto,RPC_CLOSE_DEFAULT );if (!ret)return 0 ;
@@ -840,7 +850,7 @@ static int add_host_addrs(struct host **if (prx ==PROXIMITY_ERROR )return 0 ; -if (!(new =new_host (host, thost, prx, weight))) +if (!(new =new_host (host, thost,sizeof (saddr.sin_addr), prx, weight)))return 0 ;if (!add_host (list, new))
@@ -865,11 +875,14 @@ static int add_host_addrs(struct host ** }for (haddr = phe->h_addr_list; *haddr; haddr++) { +struct in_addr tt; + prx =get_proximity (*haddr, phe->h_length);if (prx ==PROXIMITY_ERROR )return 0 ; -if (!(new =new_host (host, *haddr, prx, weight))) +memcpy (&tt, *haddr,sizeof (struct in_addr )); +if (!(new =new_host (host, *haddr, phe->h_length, prx, weight)))return 0 ;if (!add_host (list, new)) {
lib/rpc_subs.c renamed from lib/rpc_subs.c.dns-name-lookup +20 -2
@@ -89,6 +89,11 @@ static CLIENT *create_udp_client(structmemset (&raddr,0 ,sizeof (raddr)); raddr.sin_family =AF_INET ; +if (info->addr) { +memcpy (&raddr.sin_addr.s_addr, info->addr, info->addr_len); + goto got_addr; + } +if (inet_aton (info->host, &raddr.sin_addr)) goto got_addr;
@@ -298,6 +303,11 @@ static CLIENT *create_tcp_client(structmemset (&addr,0 ,sizeof (addr)); addr.sin_family =AF_INET ; +if (info->addr) { +memcpy (&addr.sin_addr.s_addr, info->addr, info->addr_len); + goto got_addr; + } +if (inet_aton (info->host, &addr.sin_addr)) goto got_addr;
@@ -410,8 +420,8 @@ void rpc_destroy_tcp_client(struct conn_ }int rpc_portmap_getclient (struct conn_info *info, -const char *host,const char *proto, -unsigned int option) +const char *host,const char *addr,size_t addr_len, +const char *proto,unsigned int option) {struct protoent *pe_proto;CLIENT *client;
@@ -421,6 +431,8 @@ int rpc_portmap_getclient(struct conn_inreturn 0 ; info->host = host; + info->addr = addr; + info->addr_len = addr_len; info->program =PMAPPROG ; info->port =PMAPPORT ; info->version =PMAPVERS ;
@@ -465,6 +477,8 @@ unsigned short rpc_portmap_getport(struc client = info->client; else { pmap_info.host = info->host; + pmap_info.addr = info->addr; + pmap_info.addr_len = info->addr_len; pmap_info.port =PMAPPORT ; pmap_info.program =PMAPPROG ; pmap_info.version =PMAPVERS ;
@@ -592,6 +606,8 @@ static unsigned int __rpc_ping(const chastruct pmap parms; info.host = host; + info.addr =NULL ; + info.addr_len =0 ; info.program =NFS_PROGRAM ; info.version = version; info.send_sz =0 ;
@@ -1339,6 +1355,8 @@ exports rpc_get_exports(const char *hostint status; info.host = host; + info.addr =NULL ; + info.addr_len =0 ; info.program =MOUNTPROG ; info.version =MOUNTVERS ; info.send_sz =0 ;
Quoted 6 lines · Ian Kent
Quoted 5 lines · Jim Carter
Quoted 4 lines · Ian Kent
Quoted 3 lines · Jim Carter
Today I upgraded one of my three DNS servers, the first listed in
/etc/resolv.conf, and until I got it back on the air, autofs mounting was glacial...
--snip--
Quoted 4 lines
Try this patch and let me know what happens.
Not sure it will actually make much difference. I've made it against the autofs in opensuse, revision 30.2.
Patch and build were clean. I installed it on a machine running autofs-5.0.2-30.2 and it worked. To test, I prepended to /etc/resolv.conf a nameserver (IP address) which was known to not exist, and then automounted something. This took 10.093 secs elapsed, corresponding to one timeout (default length 5 secs) when autofs resolves the hostname, and a second timeout when mount resolves the name. This is the best that can be done.
Thanks very much for looking into this! When the lead nameserver dies the user experience will be much improved, even if not perfect.
I think the real solution is for us to have a forwarding nameserver on every Linux box. /etc/resolv.conf would point only to it. BIND learns which forwarder gives the quickest response, so when that one is lost the situation is self-healing, in contrast to resolv.conf which has no memory.
The only problem is with Windows: it only has an equivalent of resolv.conf. But it would be pointing to the domain controller(s), which forward to the Linux DNS servers. I hope the Windows DNS server can re-order the forwarders like BIND can. In our "experiment" it wasn't possible to tell one way or the other due to not properly using the available redundancy.
James F. Carter Voice 310 825 2897 FAX 310 206 6673
UCLA-Mathnet; 6115 MSA; 405 Hilgard Ave.; Los Angeles, CA, USA 90095-1555
Email: [email protected] http://www.math.ucla.edu/~jimc (q.v. for PGP key)
Ian Kent
maintainerQuoted 13 lines · Jim Carter
Quoted 6 lines · Ian Kent
Quoted 5 lines · Jim Carter
Quoted 4 lines · Ian Kent
Quoted 3 lines · Jim Carter
Today I upgraded one of my three DNS servers, the first listed in
/etc/resolv.conf, and until I got it back on the air, autofs mounting was glacial...--snip--
Quoted 4 lines
Try this patch and let me know what happens.
Not sure it will actually make much difference. I've made it against the autofs in opensuse, revision 30.2.
Quoted 7 lines
Patch and build were clean. I installed it on a machine running autofs-5.0.2-30.2 and it worked. To test, I prepended to
/etc/resolv.confa nameserver (IP address) which was known to not exist, and then automounted something. This took 10.093 secs elapsed, corresponding to one timeout (default length 5 secs) when autofs resolves the hostname, and a second timeout when mount resolves the name. This is the best that can be done.
OK, I can't think of any possible side effects from this so I'll merge it.
Thanks
Ian