fast mounting problems
Subsystems (2)
Relating to Autofs 4.1
This relates to a bug I've filed with RHAT
( https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=128966 )
In our testing enviroment we use autofs a lot and have been experiancing problems of being unable to mount directories every now and again
To this end I build a test script to exercise the autofs daemon which nicely exposes the problem (again covered in the bugzilla report)
What appears to be happening is that autofs is spawning forked copies of itself to help facilitate loading FS's
however, initially when debugging it seemed like the first batch of NFS directories were mounted before the second forked process times out trying to rpc_ping the server because the initial process never bothered to close the socket.
Now Jeff Moyer at RedHat spotted that and gave me a patch for rpc_stubs.c which alleviates the problem slightly, however, while the parent and then the first child process play nice, any further forked processes also run into this rpc_ping port being held open and consequently fail to mount and you'd see something like this in the system messages log
Aug 18 18:59:12 localhost automount[8746]: attempting to mount entry/test/947 Aug 18 18:59:12 localhost automount[9050]: >> nfs bindresvport: Address already in use Aug 18 18:59:12 localhost automount[9050]: mount(nfs): nfs: mount failure 192.168.1.100:/compile/nfs/947 on /test/947
soooooo,.... ideas on how to make the other children processes play nice?
Phil
=--=
[email protected]
maintainerQuoted 24 lines · Bryce as root
Relating to Autofs 4.1
This relates to a bug I've filed with RHAT
( https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=128966 )In our testing enviroment we use autofs a lot and have been experiancing problems of being unable to mount directories every now and again
To this end I build a test script to exercise the autofs daemon which nicely exposes the problem (again covered in the bugzilla report)
What appears to be happening is that autofs is spawning forked copies of itself to help facilitate loading FS's
however, initially when debugging it seemed like the first batch of NFS directories were mounted before the second forked process times out trying torpc_pingthe server because the initial process never bothered to close the socket.Now Jeff Moyer at RedHat spotted that and gave me a patch for
rpc_stubs.cwhich alleviates the problem slightly, however, while the parent and then the first child process play nice, any further forked processes also run into thisrpc_pingport being held open and consequently fail to mount and you'd see something like this in the system messages log
Quoted 5 lines
Aug 18 18:59:12 localhost automount[8746]: attempting to mount entry/test/947 Aug 18 18:59:12 localhost automount[9050]: >> nfs bindresvport: Address already in use Aug 18 18:59:12 localhost automount[9050]: mount(nfs): nfs: mount failure 192.168.1.100:/compile/nfs/947 on /test/947soooooo,.... ideas on how to make the other children processes play nice?
I've read the bugzilla report.
Thanks for you huge effort in profiling this problem Phil.
After investigating it looks like Jeffs' original patch was not quite right. Can you try the attached patch and let me know how it goes please. It assumes that Jeffs' original sock leak fix patch has been applied.
Ian
lib/rpc_subs.c renamed from lib/rpc_subs.c.auto_close +25 -5
@@ -32,9 +32,27 @@#define PMAP_TOUT_UDP 2#define PMAP_TOUT_TCP 3 +/* +* TCP RPC private data struct - from sunrpc/clnt_tcp.c. +* Hopefully it won't be changing any time soon. +*/ + +#define MCALL_MSG_SIZE 24 + +struct ct_data { +int ct_sock; +bool_t ct_closeit; +struct timeval ct_wait; +bool_t ct_waitset; +struct sockaddr_in ct_addr; +struct rpc_err ct_error; +char ct_mcall[MCALL_MSG_SIZE ]; +u_int ct_mpos; +XDR ct_xdrs; +}; +struct conn_info {const char *host; -int fd;unsigned short port;unsigned long program;unsigned long version;
@@ -49,6 +67,7 @@*/ static CLIENT *create_udp_client (struct conn_info *info) { +int fd =RPC_ANYSOCK ;CLIENT *client;struct sockaddr_in addr;struct hostent *hp;
@@ -62,14 +81,13 @@if (!hp)return NULL ; - info->fd =RPC_ANYSOCK ; addr.sin_family =AF_INET ; addr.sin_port =htons (info->port);memcpy (&addr.sin_addr.s_addr, hp->h_addr, hp->h_length); client =clntudp_bufcreate (&addr, info->program, info->version, - info->timeout, &info->fd, + info->timeout, &fd, info->send_sz, info->recv_sz);return client;
@@ -149,6 +167,7 @@ {int fd;CLIENT *client; +struct ct_data *ct;struct sockaddr_in addr;struct hostent *hp;int ret;
@@ -180,7 +199,9 @@if (!client) goto out_close; - info->fd = fd; + ct = (struct ct_data *) client->cl_private; + ct->ct_closeit =1 ; +return client; out_close:
@@ -226,7 +247,6 @@ pmap_info.timeout);clnt_destroy (client); -close (pmap_info.fd);if (stat !=RPC_SUCCESS )return 0 ;
_______________________________________________
autofs mailing list
[email protected]
http://linux.kernel.org/mailman/listinfo/autofs
==> Regarding Re: [autofs] fast mounting problems; [email protected] adds:
Quoted 24 lines · Bryce as root
Quoted 24 lines
Relating to Autofs 4.1
This relates to a bug I've filed with RHAT (
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=128966 )In our testing enviroment we use autofs a lot and have been experiancing problems of being unable to mount directories every now and again
To this end I build a test script to exercise the autofs daemon which nicely exposes the problem (again covered in the bugzilla report)
What appears to be happening is that autofs is spawning forked copies of itself to help facilitate loading FS's however, initially when debugging it seemed like the first batch of NFS directories were mounted before the second forked process times out trying to
rpc_pingthe server because the initial process never bothered to close the socket.Now Jeff Moyer at RedHat spotted that and gave me a patch for
rpc_stubs.cwhich alleviates the problem slightly, however, while the parent and then the first child process play nice, any further forked processes also run into thisrpc_pingport being held open and consequently fail to mount and you'd see something like this in the system messages log
Quoted 8 lines
Quoted 8 lines
Aug 18 18:59:12 localhost automount[8746]: attempting to mountentry/test/947 Aug 18 18:59:12 localhost automount[9050]: >> nfs
bindresvport: Address already in use Aug 18 18:59:12 localhost automount[9050]: mount(nfs): nfs: mount failure 192.168.1.100:/compile/nfs/947 on /test/947soooooo,.... ideas on how to make the other children processes play nice?
raven> I've read the bugzilla report.
raven> Thanks for you huge effort in profiling this problem Phil.
raven> After investigating it looks like Jeffs' original patch was not raven> quite right. Can you try the attached patch and let me know how it raven> goes please. It assumes that Jeffs' original sock leak fix patch has raven> been applied.
Hi, Ian,
OK, so I've finally gotten 'round to looking at your patch. You can avoid including internal rpc data structures by using the clnt_control call with CLSET_FD_CLOSE. Aside from that, I like your approach better. ;) So, how about this patch instead of the other two?
-Jeff
lib/rpc_subs.c renamed from lib/rpc_subs.c~ +2 -1
@@ -48,7 +48,7 @@ struct conn_info {*/ static CLIENT *create_udp_client (struct conn_info *info) { -int fd =-1 ; +int fd =RPC_ANYSOCK ;CLIENT *client;struct sockaddr_in addr;struct hostent *hp;
@@ -169,6 +169,7 @@ static CLIENT* create_tcp_client(structif (!client) goto out_close; +clnt_control (client,CLSET_FD_CLOSE ,NULL );return client; out_close:
[email protected]
maintainerQuoted 67 lines · Jeff Moyer
==> Regarding Re: [autofs] fast mounting problems; [email protected] adds:
Quoted 24 lines · Bryce as root
Quoted 24 lines
Relating to Autofs 4.1
This relates to a bug I've filed with RHAT (
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=128966 )In our testing enviroment we use autofs a lot and have been experiancing problems of being unable to mount directories every now and again
To this end I build a test script to exercise the autofs daemon which nicely exposes the problem (again covered in the bugzilla report)
What appears to be happening is that autofs is spawning forked copies of itself to help facilitate loading FS's however, initially when debugging it seemed like the first batch of NFS directories were mounted before the second forked process times out trying to
rpc_pingthe server because the initial process never bothered to close the socket.Now Jeff Moyer at RedHat spotted that and gave me a patch for
rpc_stubs.cwhich alleviates the problem slightly, however, while the parent and then the first child process play nice, any further forked processes also run into thisrpc_pingport being held open and consequently fail to mount and you'd see something like this in the system messages logQuoted 8 lines
Quoted 8 lines
Aug 18 18:59:12 localhost automount[8746]: attempting to mountentry/test/947 Aug 18 18:59:12 localhost automount[9050]: >> nfs
bindresvport: Address already in use Aug 18 18:59:12 localhost automount[9050]: mount(nfs): nfs: mount failure 192.168.1.100:/compile/nfs/947 on /test/947soooooo,.... ideas on how to make the other children processes play nice?
raven> I've read the bugzilla report.
raven> Thanks for you huge effort in profiling this problem Phil.
raven> After investigating it looks like Jeffs' original patch was not raven> quite right. Can you try the attached patch and let me know how it raven> goes please. It assumes that Jeffs' original sock leak fix patch has raven> been applied.
Hi, Ian,
OK, so I've finally gotten 'round to looking at your patch. You can avoid including internal rpc data structures by using the
clnt_controlcall withCLSET_FD_CLOSE. Aside from that, I like your approach better. ;) So, how about this patch instead of the other two?(quoted diff) +0
-1-Jefflib/rpc_subs.c renamed from lib/rpc_subs.c~ +1
-1@@ -48,7 +48,7 @@ struct conn_info {*/ static CLIENT *create_udp_client (struct conn_info *info) { -int fd =-1 ; +int fd =RPC_ANYSOCK ;CLIENT *client;struct sockaddr_in addr;struct hostent *hp;
lib/rpc_subs.c static CLIENT* create_tcp_client(struct · line 175
Quoted 7 lines
lib/rpc_subs.c renamed from lib/rpc_subs.c~ +1
-0@@ -169,6 +169,7 @@ static CLIENT* create_tcp_client(structif (!client) goto out_close; +clnt_control (client,CLSET_FD_CLOSE ,NULL );return client; out_close:
OK. I guess my if around the clnt_control is redundant as this particular call should always succeed if client is non-null. Cool.
Ian