Skip to contentSkip to conversation

fast mounting problems

DISCUSSION Bryce as root 3 participants 4 messages Aug 31, 2004 – Sep 14, 2004

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Relating to Autofs 4.1

This relates to a bug I've filed with RHAT
( https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=128966 )

In our testing enviroment we use autofs a lot and have been experiancing problems of being unable to mount directories every now and again

To this end I build a test script to exercise the autofs daemon which nicely exposes the problem (again covered in the bugzilla report)

What appears to be happening is that autofs is spawning forked copies of itself to help facilitate loading FS's
however, initially when debugging it seemed like the first batch of NFS directories were mounted before the second forked process times out trying to rpc_ping the server because the initial process never bothered to close the socket.

Now Jeff Moyer at RedHat spotted that and gave me a patch for rpc_stubs.c which alleviates the problem slightly, however, while the parent and then the first child process play nice, any further forked processes also run into this rpc_ping port being held open and consequently fail to mount and you'd see something like this in the system messages log

Aug 18 18:59:12 localhost automount[8746]: attempting to mount entry/test/947
Aug 18 18:59:12 localhost automount[9050]: >> nfs bindresvport: Address already in use
Aug 18 18:59:12 localhost automount[9050]: mount(nfs): nfs: mount failure 192.168.1.100:/compile/nfs/947 on /test/947

soooooo,.... ideas on how to make the other children processes play nice?

Phil
=--=

#1 lore raw
Quoted 24 lines · Bryce as root

Relating to Autofs 4.1

This relates to a bug I've filed with RHAT
( https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=128966 )

In our testing enviroment we use autofs a lot and have been experiancing problems of being unable to mount directories every now and again

To this end I build a test script to exercise the autofs daemon which nicely exposes the problem (again covered in the bugzilla report)

What appears to be happening is that autofs is spawning forked copies of itself to help facilitate loading FS's
however, initially when debugging it seemed like the first batch of NFS directories were mounted before the second forked process times out trying to rpc_ping the server because the initial process never bothered to close the socket.

Now Jeff Moyer at RedHat spotted that and gave me a patch for rpc_stubs.c which alleviates the problem slightly, however, while the parent and then the first child process play nice, any further forked processes also run into this rpc_ping port being held open and consequently fail to mount and you'd see something like this in the system messages log

Quoted 5 lines
Aug 18 18:59:12 localhost automount[8746]: attempting to mount entry/test/947
Aug 18 18:59:12 localhost automount[9050]: >> nfs bindresvport: Address already in use
Aug 18 18:59:12 localhost automount[9050]: mount(nfs): nfs: mount failure 192.168.1.100:/compile/nfs/947 on /test/947

soooooo,.... ideas on how to make the other children processes play nice?

I've read the bugzilla report.

Thanks for you huge effort in profiling this problem Phil.

After investigating it looks like Jeffs' original patch was not quite right. Can you try the attached patch and let me know how it goes please. It assumes that Jeffs' original sock leak fix patch has been applied.

Ian

lib/rpc_subs.c renamed from lib/rpc_subs.c.auto_close +25 -5
@@ -32,9 +32,27 @@
 #define PMAP_TOUT_UDP	2
 #define PMAP_TOUT_TCP	3
 
+/*
+ * TCP RPC private data struct - from sunrpc/clnt_tcp.c.
+ * Hopefully it won't be changing any time soon.
+ */
+
+#define MCALL_MSG_SIZE 24
+
+struct ct_data {
+	int ct_sock;
+	bool_t ct_closeit;
+	struct timeval ct_wait;
+	bool_t ct_waitset;
+	struct sockaddr_in ct_addr;
+	struct rpc_err ct_error;
+	char ct_mcall[MCALL_MSG_SIZE];
+	u_int ct_mpos;
+	XDR ct_xdrs;
+};
+
 struct conn_info {
 	const char *host;
-	int fd;
 	unsigned short port;
 	unsigned long program;
 	unsigned long version;
@@ -49,6 +67,7 @@
  */
 static CLIENT* create_udp_client(struct conn_info *info)
 {
+	int fd = RPC_ANYSOCK;
 	CLIENT *client;
 	struct sockaddr_in addr;
 	struct hostent *hp;
@@ -62,14 +81,13 @@
 	if (!hp)
 		return NULL;
 
-	info->fd = RPC_ANYSOCK;
 	addr.sin_family = AF_INET;
 	addr.sin_port = htons(info->port);
 	memcpy(&addr.sin_addr.s_addr, hp->h_addr, hp->h_length);
 
 	client = clntudp_bufcreate(&addr,
 				   info->program, info->version,
-				   info->timeout, &info->fd,
+				   info->timeout, &fd,
 				   info->send_sz, info->recv_sz);
 
 	return client;
@@ -149,6 +167,7 @@
 {
 	int fd;
 	CLIENT *client;
+	struct ct_data *ct;
 	struct sockaddr_in addr;
 	struct hostent *hp;
 	int ret;
@@ -180,7 +199,9 @@
 	if (!client)
 		goto out_close;
 
-	info->fd = fd;
+	ct = (struct ct_data *) client->cl_private;
+	ct->ct_closeit = 1;
+
 	return client;
 
 out_close:
@@ -226,7 +247,6 @@
 			 pmap_info.timeout);
 
 	clnt_destroy(client);
-	close(pmap_info.fd);
 
 	if (stat != RPC_SUCCESS)
 		return 0;

_______________________________________________
autofs mailing list
[email protected]
http://linux.kernel.org/mailman/listinfo/autofs

#2 lore raw

==> Regarding Re: [autofs] fast mounting problems; [email protected] adds:

Quoted 24 lines · Bryce as root
Quoted 24 lines

Relating to Autofs 4.1

This relates to a bug I've filed with RHAT (
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=128966 )

In our testing enviroment we use autofs a lot and have been experiancing problems of being unable to mount directories every now and again

To this end I build a test script to exercise the autofs daemon which nicely exposes the problem (again covered in the bugzilla report)

What appears to be happening is that autofs is spawning forked copies of itself to help facilitate loading FS's however, initially when debugging it seemed like the first batch of NFS directories were mounted before the second forked process times out trying to rpc_ping the server because the initial process never bothered to close the socket.

Now Jeff Moyer at RedHat spotted that and gave me a patch for rpc_stubs.c which alleviates the problem slightly, however, while the parent and then the first child process play nice, any further forked processes also run into this rpc_ping port being held open and consequently fail to mount and you'd see something like this in the system messages log

Quoted 8 lines
Quoted 8 lines
Aug 18 18:59:12 localhost automount[8746]: attempting to mount

entry/test/947 Aug 18 18:59:12 localhost automount[9050]: >> nfs
bindresvport: Address already in use Aug 18 18:59:12 localhost automount[9050]: mount(nfs): nfs: mount failure 192.168.1.100:/compile/nfs/947 on /test/947

soooooo,.... ideas on how to make the other children processes play nice?

raven> I've read the bugzilla report.

raven> Thanks for you huge effort in profiling this problem Phil.

raven> After investigating it looks like Jeffs' original patch was not raven> quite right. Can you try the attached patch and let me know how it raven> goes please. It assumes that Jeffs' original sock leak fix patch has raven> been applied.

Hi, Ian,

OK, so I've finally gotten 'round to looking at your patch. You can avoid including internal rpc data structures by using the clnt_control call with CLSET_FD_CLOSE. Aside from that, I like your approach better. ;) So, how about this patch instead of the other two?

-Jeff

lib/rpc_subs.c renamed from lib/rpc_subs.c~ +2 -1
@@ -48,7 +48,7 @@ struct conn_info {
  */
 static CLIENT* create_udp_client(struct conn_info *info)
 {
-	int fd = -1;
+	int fd = RPC_ANYSOCK;
 	CLIENT *client;
 	struct sockaddr_in addr;
 	struct hostent *hp;
@@ -169,6 +169,7 @@ static CLIENT* create_tcp_client(struct 
 	if (!client)
 		goto out_close;
 
+	clnt_control(client, CLSET_FD_CLOSE, NULL);
 	return client;
 
 out_close:
#3 lore raw
Quoted 67 lines · Jeff Moyer

==> Regarding Re: [autofs] fast mounting problems; [email protected] adds:

Quoted 24 lines · Bryce as root
Quoted 24 lines

Relating to Autofs 4.1

This relates to a bug I've filed with RHAT (
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=128966 )

In our testing enviroment we use autofs a lot and have been experiancing problems of being unable to mount directories every now and again

To this end I build a test script to exercise the autofs daemon which nicely exposes the problem (again covered in the bugzilla report)

What appears to be happening is that autofs is spawning forked copies of itself to help facilitate loading FS's however, initially when debugging it seemed like the first batch of NFS directories were mounted before the second forked process times out trying to rpc_ping the server because the initial process never bothered to close the socket.

Now Jeff Moyer at RedHat spotted that and gave me a patch for rpc_stubs.c which alleviates the problem slightly, however, while the parent and then the first child process play nice, any further forked processes also run into this rpc_ping port being held open and consequently fail to mount and you'd see something like this in the system messages log

Quoted 8 lines
Quoted 8 lines
Aug 18 18:59:12 localhost automount[8746]: attempting to mount

entry/test/947 Aug 18 18:59:12 localhost automount[9050]: >> nfs
bindresvport: Address already in use Aug 18 18:59:12 localhost automount[9050]: mount(nfs): nfs: mount failure 192.168.1.100:/compile/nfs/947 on /test/947

soooooo,.... ideas on how to make the other children processes play nice?

raven> I've read the bugzilla report.

raven> Thanks for you huge effort in profiling this problem Phil.

raven> After investigating it looks like Jeffs' original patch was not raven> quite right. Can you try the attached patch and let me know how it raven> goes please. It assumes that Jeffs' original sock leak fix patch has raven> been applied.

Hi, Ian,

OK, so I've finally gotten 'round to looking at your patch. You can avoid including internal rpc data structures by using the clnt_control call with CLSET_FD_CLOSE. Aside from that, I like your approach better. ;) So, how about this patch instead of the other two?

(quoted diff) +0 -1
-Jeff
lib/rpc_subs.c renamed from lib/rpc_subs.c~ +1 -1
@@ -48,7 +48,7 @@ struct conn_info {
  */
 static CLIENT* create_udp_client(struct conn_info *info)
 {
-	int fd = -1;
+	int fd = RPC_ANYSOCK;
 	CLIENT *client;
 	struct sockaddr_in addr;
 	struct hostent *hp;

lib/rpc_subs.c static CLIENT* create_tcp_client(struct · line 175

Quoted 7 lines
lib/rpc_subs.c renamed from lib/rpc_subs.c~ +1 -0
@@ -169,6 +169,7 @@ static CLIENT* create_tcp_client(struct 
 	if (!client)
 		goto out_close;
 
+	clnt_control(client, CLSET_FD_CLOSE, NULL);
 	return client;
 
 out_close:

OK. I guess my if around the clnt_control is redundant as this particular call should always succeed if client is non-null. Cool.

Ian