Skip to contentSkip to conversation

synchronous AF_LOCAL connect

PATCH J. Bruce Fields 4 participants 16 messages Feb 18, 2013 – Feb 21, 2013

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

The rpc code expects all connects to be done asynchronously by a workqueue. But that doesn't seem necessary in the AF_LOCAL case. The only user (rpcbind) actually wants the connect done in the context of a process with the right namespace. (And that will be true of gss proxy too, which also wants to use AF_LOCAL.)

But maybe I'm missing something.

Also, I haven't really tried to understand this code--I just assumed I could basically call xs_local_setup_socket from ->setup instead of the workqueue, and that seems to work based on a very superficial test. At a minimum I guess the PF_FSTRANS fiddling shouldn't be there.

--b.

commit 93713cdf3ba42789b4e960ad457d2d802746b3ff

Author: J. Bruce Fields <[email protected]>
Date:   Fri Feb 15 17:54:26 2013 -0500

rpc: simplify AF_LOCAL connect

It doesn't appear that anyone actually needs to connect asynchronously.

Also, using a workqueue for the connect means we lose the namespace information from the original process. This is a problem since there's no way to explicitly pass in a filesystem namespace for resolution of an AF_LOCAL address.

net/sunrpc/xprtsock.c +53 -44
index bbc0915..72765f1 100644
@@ -1866,57 +1866,14 @@ static int xs_local_finish_connecting(struct rpc_xprt *xprt,
  * @xprt: RPC transport to connect
  * @transport: socket transport to connect
  * @create_sock: function to create a socket of the correct type
- *
- * Invoked by a work queue tasklet.
  */
 static void xs_local_setup_socket(struct work_struct *work)
 {
 	struct sock_xprt *transport =
 		container_of(work, struct sock_xprt, connect_worker.work);
 	struct rpc_xprt *xprt = &transport->xprt;
-	struct socket *sock;
-	int status = -EIO;
-
-	current->flags |= PF_FSTRANS;
-
-	clear_bit(XPRT_CONNECTION_ABORT, &xprt->state);
-	status = __sock_create(xprt->xprt_net, AF_LOCAL,
-					SOCK_STREAM, 0, &sock, 1);
-	if (status < 0) {
-		dprintk("RPC:       can't create AF_LOCAL "
-			"transport socket (%d).\n", -status);
-		goto out;
-	}
-	xs_reclassify_socketu(sock);
-
-	dprintk("RPC:       worker connecting xprt %p via AF_LOCAL to %s\n",
-			xprt, xprt->address_strings[RPC_DISPLAY_ADDR]);
 
-	status = xs_local_finish_connecting(xprt, sock);
-	switch (status) {
-	case 0:
-		dprintk("RPC:       xprt %p connected to %s\n",
-				xprt, xprt->address_strings[RPC_DISPLAY_ADDR]);
-		xprt_set_connected(xprt);
-		break;
-	case -ENOENT:
-		dprintk("RPC:       xprt %p: socket %s does not exist\n",
-				xprt, xprt->address_strings[RPC_DISPLAY_ADDR]);
-		break;
-	case -ECONNREFUSED:
-		dprintk("RPC:       xprt %p: connection refused for %s\n",
-				xprt, xprt->address_strings[RPC_DISPLAY_ADDR]);
-		break;
-	default:
-		printk(KERN_ERR "%s: unhandled error (%d) connecting to %s\n",
-				__func__, -status,
-				xprt->address_strings[RPC_DISPLAY_ADDR]);
-	}
-
-out:
-	xprt_clear_connecting(xprt);
-	xprt_wake_pending_tasks(xprt, status);
-	current->flags &= ~PF_FSTRANS;
+	xprt_wake_pending_tasks(xprt, -EIO);
 }
 
 #ifdef CONFIG_SUNRPC_SWAP
@@ -2588,6 +2545,55 @@ static const struct rpc_timeout xs_local_default_timeout = {
 	.to_retries = 2,
 };
 
+
+static int xs_local_connect(struct sock_xprt *transport)
+{
+	struct rpc_xprt *xprt = &transport->xprt;
+	struct socket *sock;
+	int status = -EIO;
+
+	current->flags |= PF_FSTRANS;
+
+	clear_bit(XPRT_CONNECTION_ABORT, &xprt->state);
+	status = __sock_create(xprt->xprt_net, AF_LOCAL,
+					SOCK_STREAM, 0, &sock, 1);
+	if (status < 0) {
+		dprintk("RPC:       can't create AF_LOCAL "
+			"transport socket (%d).\n", -status);
+		goto out;
+	}
+	xs_reclassify_socketu(sock);
+
+	dprintk("RPC:       worker connecting xprt %p via AF_LOCAL to %s\n",
+			xprt, xprt->address_strings[RPC_DISPLAY_ADDR]);
+
+	status = xs_local_finish_connecting(xprt, sock);
+	switch (status) {
+	case 0:
+		dprintk("RPC:       xprt %p connected to %s\n",
+				xprt, xprt->address_strings[RPC_DISPLAY_ADDR]);
+		xprt_set_connected(xprt);
+		break;
+	case -ENOENT:
+		dprintk("RPC:       xprt %p: socket %s does not exist\n",
+				xprt, xprt->address_strings[RPC_DISPLAY_ADDR]);
+		break;
+	case -ECONNREFUSED:
+		dprintk("RPC:       xprt %p: connection refused for %s\n",
+				xprt, xprt->address_strings[RPC_DISPLAY_ADDR]);
+		break;
+	default:
+		printk(KERN_ERR "%s: unhandled error (%d) connecting to %s\n",
+				__func__, -status,
+				xprt->address_strings[RPC_DISPLAY_ADDR]);
+	}
+out:
+	xprt_clear_connecting(xprt);
+	xprt_wake_pending_tasks(xprt, status);
+	current->flags &= ~PF_FSTRANS;
+	return status;
+}
+
 /**
  * xs_setup_local - Set up transport to use an AF_LOCAL socket
  * @args: rpc transport creation arguments
@@ -2630,6 +2636,9 @@ static struct rpc_xprt *xs_setup_local(struct xprt_create *args)
 		INIT_DELAYED_WORK(&transport->connect_worker,
 					xs_local_setup_socket);
 		xs_format_peer_addresses(xprt, "local", RPCBIND_NETID_LOCAL);
+		ret = ERR_PTR(xs_local_connect(transport));
+		if (ret)
+			goto out_err;
 		break;
 	default:
 		ret = ERR_PTR(-EAFNOSUPPORT);
#1 lore raw
Quoted 6 lines · bfields

The rpc code expects all connects to be done asynchronously by a workqueue. But that doesn't seem necessary in the AF_LOCAL case. The only user (rpcbind) actually wants the connect done in the context of a process with the right namespace. (And that will be true of gss proxy too, which also wants to use AF_LOCAL.)

Quoted 6 lines

But maybe I'm missing something.

Also, I haven't really tried to understand this code--I just assumed I could basically call xs_local_setup_socket from ->setup instead of the workqueue, and that seems to work based on a very superficial test. At a minimum I guess the PF_FSTRANS fiddling shouldn't be there.

Here it is with that and the other extraneous xprt stuff gone.

See any problem with doing this?

(This is basically my attempt to implement Trond's solution to the AF_LOCAL-connect-in-a-namespace problem:

	http://mid.gmane.org/<4FA345DA4F4AE44899BD2B03EEEC2FA9092E0A40@SACEXCMBX04-PRD.hq.netapp.com>

)

--b.

commit 49413f389f7b38b4cbeb2d1c7f25ebcbf00f25f6

Author: J. Bruce Fields <[email protected]>
Date:   Fri Feb 15 17:54:26 2013 -0500

rpc: simplify AF_LOCAL connect

It doesn't appear that anyone actually needs to connect asynchronously.

Also, using a workqueue for the connect means we lose the namespace information from the original process. This is a problem since there's no way to explicitly pass in a filesystem namespace for resolution of an AF_LOCAL address.

net/sunrpc/xprtsock.c +46 -44
index bbc0915..b7d60e4 100644
@@ -1866,57 +1866,14 @@ static int xs_local_finish_connecting(struct rpc_xprt *xprt,
  * @xprt: RPC transport to connect
  * @transport: socket transport to connect
  * @create_sock: function to create a socket of the correct type
- *
- * Invoked by a work queue tasklet.
  */
 static void xs_local_setup_socket(struct work_struct *work)
 {
 	struct sock_xprt *transport =
 		container_of(work, struct sock_xprt, connect_worker.work);
 	struct rpc_xprt *xprt = &transport->xprt;
-	struct socket *sock;
-	int status = -EIO;
-
-	current->flags |= PF_FSTRANS;
-
-	clear_bit(XPRT_CONNECTION_ABORT, &xprt->state);
-	status = __sock_create(xprt->xprt_net, AF_LOCAL,
-					SOCK_STREAM, 0, &sock, 1);
-	if (status < 0) {
-		dprintk("RPC:       can't create AF_LOCAL "
-			"transport socket (%d).\n", -status);
-		goto out;
-	}
-	xs_reclassify_socketu(sock);
 
-	dprintk("RPC:       worker connecting xprt %p via AF_LOCAL to %s\n",
-			xprt, xprt->address_strings[RPC_DISPLAY_ADDR]);
-
-	status = xs_local_finish_connecting(xprt, sock);
-	switch (status) {
-	case 0:
-		dprintk("RPC:       xprt %p connected to %s\n",
-				xprt, xprt->address_strings[RPC_DISPLAY_ADDR]);
-		xprt_set_connected(xprt);
-		break;
-	case -ENOENT:
-		dprintk("RPC:       xprt %p: socket %s does not exist\n",
-				xprt, xprt->address_strings[RPC_DISPLAY_ADDR]);
-		break;
-	case -ECONNREFUSED:
-		dprintk("RPC:       xprt %p: connection refused for %s\n",
-				xprt, xprt->address_strings[RPC_DISPLAY_ADDR]);
-		break;
-	default:
-		printk(KERN_ERR "%s: unhandled error (%d) connecting to %s\n",
-				__func__, -status,
-				xprt->address_strings[RPC_DISPLAY_ADDR]);
-	}
-
-out:
-	xprt_clear_connecting(xprt);
-	xprt_wake_pending_tasks(xprt, status);
-	current->flags &= ~PF_FSTRANS;
+	xprt_wake_pending_tasks(xprt, -EIO);
 }
 
 #ifdef CONFIG_SUNRPC_SWAP
@@ -2588,6 +2545,48 @@ static const struct rpc_timeout xs_local_default_timeout = {
 	.to_retries = 2,
 };
 
+
+static int xs_local_connect(struct sock_xprt *transport)
+{
+	struct rpc_xprt *xprt = &transport->xprt;
+	struct socket *sock;
+	int status = -EIO;
+
+	status = __sock_create(xprt->xprt_net, AF_LOCAL,
+					SOCK_STREAM, 0, &sock, 1);
+	if (status < 0) {
+		dprintk("RPC:       can't create AF_LOCAL "
+			"transport socket (%d).\n", -status);
+		return status;
+	}
+	xs_reclassify_socketu(sock);
+
+	dprintk("RPC:       worker connecting xprt %p via AF_LOCAL to %s\n",
+			xprt, xprt->address_strings[RPC_DISPLAY_ADDR]);
+
+	status = xs_local_finish_connecting(xprt, sock);
+	switch (status) {
+	case 0:
+		dprintk("RPC:       xprt %p connected to %s\n",
+				xprt, xprt->address_strings[RPC_DISPLAY_ADDR]);
+		xprt_set_connected(xprt);
+		break;
+	case -ENOENT:
+		dprintk("RPC:       xprt %p: socket %s does not exist\n",
+				xprt, xprt->address_strings[RPC_DISPLAY_ADDR]);
+		break;
+	case -ECONNREFUSED:
+		dprintk("RPC:       xprt %p: connection refused for %s\n",
+				xprt, xprt->address_strings[RPC_DISPLAY_ADDR]);
+		break;
+	default:
+		printk(KERN_ERR "%s: unhandled error (%d) connecting to %s\n",
+				__func__, -status,
+				xprt->address_strings[RPC_DISPLAY_ADDR]);
+	}
+	return status;
+}
+
 /**
  * xs_setup_local - Set up transport to use an AF_LOCAL socket
  * @args: rpc transport creation arguments
@@ -2630,6 +2629,9 @@ static struct rpc_xprt *xs_setup_local(struct xprt_create *args)
 		INIT_DELAYED_WORK(&transport->connect_worker,
 					xs_local_setup_socket);
 		xs_format_peer_addresses(xprt, "local", RPCBIND_NETID_LOCAL);
+		ret = ERR_PTR(xs_local_connect(transport));
+		if (ret)
+			goto out_err;
 		break;
 	default:
 		ret = ERR_PTR(-EAFNOSUPPORT);
#2 lore raw
Quoted 9 lines · J. Bruce Fields
Quoted 8 lines · bfields

The rpc code expects all connects to be done asynchronously by a workqueue. But that doesn't seem necessary in the AF_LOCAL case. The only user (rpcbind) actually wants the connect done in the context of a process with the right namespace. (And that will be true of gss proxy too, which also wants to use AF_LOCAL.)

But maybe I'm missing something.

Quoted 8 lines
Quoted 4 lines

Also, I haven't really tried to understand this code--I just assumed I could basically call xs_local_setup_socket from ->setup instead of the workqueue, and that seems to work based on a very superficial test. At a minimum I guess the PF_FSTRANS fiddling shouldn't be there.

Here it is with that and the other extraneous xprt stuff gone.

See any problem with doing this?

Nothing is screaming at me. As long as an AF_LOCAL connect operation doesn't ever sleep, this should be safe, I think.

How did you test it?

Quoted 150 lines

(This is basically my attempt to implement Trond's solution to the AF_LOCAL-connect-in-a-namespace problem:

	http://mid.gmane.org/<4FA345DA4F4AE44899BD2B03EEEC2FA9092E0A40@SACEXCMBX04-PRD.hq.netapp.com>

)

--b.

commit 49413f389f7b38b4cbeb2d1c7f25ebcbf00f25f6

Author: J. Bruce Fields <[email protected]>
Date:   Fri Feb 15 17:54:26 2013 -0500

rpc: simplify AF_LOCAL connect

It doesn't appear that anyone actually needs to connect asynchronously.

Also, using a workqueue for the connect means we lose the namespace information from the original process. This is a problem since there's no way to explicitly pass in a filesystem namespace for resolution of an AF_LOCAL address.

net/sunrpc/xprtsock.c +46 -45
index bbc0915..b7d60e4 100644
@@ -1866,57 +1866,14 @@ static int xs_local_finish_connecting(struct rpc_xprt *xprt,
 * @xprt: RPC transport to connect
 * @transport: socket transport to connect
 * @create_sock: function to create a socket of the correct type
- *
- * Invoked by a work queue tasklet.
 */
 static void xs_local_setup_socket(struct work_struct *work)
 {
 	struct sock_xprt *transport =
 		container_of(work, struct sock_xprt, connect_worker.work);
 	struct rpc_xprt *xprt = &transport->xprt;
-	struct socket *sock;
-	int status = -EIO;
-
-	current->flags |= PF_FSTRANS;
-
-	clear_bit(XPRT_CONNECTION_ABORT, &xprt->state);
-	status = __sock_create(xprt->xprt_net, AF_LOCAL,
-					SOCK_STREAM, 0, &sock, 1);
-	if (status < 0) {
-		dprintk("RPC:       can't create AF_LOCAL "
-			"transport socket (%d).\n", -status);
-		goto out;
-	}
-	xs_reclassify_socketu(sock);
 
-	dprintk("RPC:       worker connecting xprt %p via AF_LOCAL to %s\n",
-			xprt, xprt->address_strings[RPC_DISPLAY_ADDR]);
-
-	status = xs_local_finish_connecting(xprt, sock);
-	switch (status) {
-	case 0:
-		dprintk("RPC:       xprt %p connected to %s\n",
-				xprt, xprt->address_strings[RPC_DISPLAY_ADDR]);
-		xprt_set_connected(xprt);
-		break;
-	case -ENOENT:
-		dprintk("RPC:       xprt %p: socket %s does not exist\n",
-				xprt, xprt->address_strings[RPC_DISPLAY_ADDR]);
-		break;
-	case -ECONNREFUSED:
-		dprintk("RPC:       xprt %p: connection refused for %s\n",
-				xprt, xprt->address_strings[RPC_DISPLAY_ADDR]);
-		break;
-	default:
-		printk(KERN_ERR "%s: unhandled error (%d) connecting to %s\n",
-				__func__, -status,
-				xprt->address_strings[RPC_DISPLAY_ADDR]);
-	}
-
-out:
-	xprt_clear_connecting(xprt);
-	xprt_wake_pending_tasks(xprt, status);
-	current->flags &= ~PF_FSTRANS;
+	xprt_wake_pending_tasks(xprt, -EIO);
 }
 
 #ifdef CONFIG_SUNRPC_SWAP
@@ -2588,6 +2545,48 @@ static const struct rpc_timeout xs_local_default_timeout = {
 	.to_retries = 2,
 };
 
+
+static int xs_local_connect(struct sock_xprt *transport)
+{
+	struct rpc_xprt *xprt = &transport->xprt;
+	struct socket *sock;
+	int status = -EIO;
+
+	status = __sock_create(xprt->xprt_net, AF_LOCAL,
+					SOCK_STREAM, 0, &sock, 1);
+	if (status < 0) {
+		dprintk("RPC:       can't create AF_LOCAL "
+			"transport socket (%d).\n", -status);
+		return status;
+	}
+	xs_reclassify_socketu(sock);
+
+	dprintk("RPC:       worker connecting xprt %p via AF_LOCAL to %s\n",
+			xprt, xprt->address_strings[RPC_DISPLAY_ADDR]);
+
+	status = xs_local_finish_connecting(xprt, sock);
+	switch (status) {
+	case 0:
+		dprintk("RPC:       xprt %p connected to %s\n",
+				xprt, xprt->address_strings[RPC_DISPLAY_ADDR]);
+		xprt_set_connected(xprt);
+		break;
+	case -ENOENT:
+		dprintk("RPC:       xprt %p: socket %s does not exist\n",
+				xprt, xprt->address_strings[RPC_DISPLAY_ADDR]);
+		break;
+	case -ECONNREFUSED:
+		dprintk("RPC:       xprt %p: connection refused for %s\n",
+				xprt, xprt->address_strings[RPC_DISPLAY_ADDR]);
+		break;
+	default:
+		printk(KERN_ERR "%s: unhandled error (%d) connecting to %s\n",
+				__func__, -status,
+				xprt->address_strings[RPC_DISPLAY_ADDR]);
+	}
+	return status;
+}
+
 /**
 * xs_setup_local - Set up transport to use an AF_LOCAL socket
 * @args: rpc transport creation arguments
@@ -2630,6 +2629,9 @@ static struct rpc_xprt *xs_setup_local(struct xprt_create *args)
 		INIT_DELAYED_WORK(&transport->connect_worker,
 					xs_local_setup_socket);
 		xs_format_peer_addresses(xprt, "local", RPCBIND_NETID_LOCAL);
+		ret = ERR_PTR(xs_local_connect(transport));
+		if (ret)
+			goto out_err;
 		break;
 	default:
 		ret = ERR_PTR(-EAFNOSUPPORT);
--

To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 2 lines
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
#3 lore raw
Quoted 19 lines · Chuck Lever
Quoted 15 lines · J. Bruce Fields
Quoted 13 lines · bfields

The rpc code expects all connects to be done asynchronously by a workqueue. But that doesn't seem necessary in the AF_LOCAL case. The only user (rpcbind) actually wants the connect done in the context of a process with the right namespace. (And that will be true of gss proxy too, which also wants to use AF_LOCAL.)

But maybe I'm missing something.

Also, I haven't really tried to understand this code--I just assumed I could basically call xs_local_setup_socket from ->setup instead of the workqueue, and that seems to work based on a very superficial test. At a minimum I guess the PF_FSTRANS fiddling shouldn't be there.

Quoted 6 lines
Quoted 3 lines

Here it is with that and the other extraneous xprt stuff gone.

See any problem with doing this?

Nothing is screaming at me. As long as an AF_LOCAL connect operation doesn't ever sleep, this should be safe, I think.

I'm sure it must sleep. Why would that make any difference?

Quoted 1 line

How did you test it?

I'm just doing my usual set of connectathon runs, and assuming mounts would fail if the server's rpcbind registration failed.

--b.

Quoted 160 lines
Quoted 152 lines

(This is basically my attempt to implement Trond's solution to the AF_LOCAL-connect-in-a-namespace problem:

	http://mid.gmane.org/<4FA345DA4F4AE44899BD2B03EEEC2FA9092E0A40@SACEXCMBX04-PRD.hq.netapp.com>

)

--b.

commit 49413f389f7b38b4cbeb2d1c7f25ebcbf00f25f6

Author: J. Bruce Fields <[email protected]>
Date:   Fri Feb 15 17:54:26 2013 -0500

rpc: simplify AF_LOCAL connect

It doesn't appear that anyone actually needs to connect asynchronously.

Also, using a workqueue for the connect means we lose the namespace information from the original process. This is a problem since there's no way to explicitly pass in a filesystem namespace for resolution of an AF_LOCAL address.

net/sunrpc/xprtsock.c +46 -45
index bbc0915..b7d60e4 100644
@@ -1866,57 +1866,14 @@ static int xs_local_finish_connecting(struct rpc_xprt *xprt,
 * @xprt: RPC transport to connect
 * @transport: socket transport to connect
 * @create_sock: function to create a socket of the correct type
- *
- * Invoked by a work queue tasklet.
 */
 static void xs_local_setup_socket(struct work_struct *work)
 {
 	struct sock_xprt *transport =
 		container_of(work, struct sock_xprt, connect_worker.work);
 	struct rpc_xprt *xprt = &transport->xprt;
-	struct socket *sock;
-	int status = -EIO;
-
-	current->flags |= PF_FSTRANS;
-
-	clear_bit(XPRT_CONNECTION_ABORT, &xprt->state);
-	status = __sock_create(xprt->xprt_net, AF_LOCAL,
-					SOCK_STREAM, 0, &sock, 1);
-	if (status < 0) {
-		dprintk("RPC:       can't create AF_LOCAL "
-			"transport socket (%d).\n", -status);
-		goto out;
-	}
-	xs_reclassify_socketu(sock);
 
-	dprintk("RPC:       worker connecting xprt %p via AF_LOCAL to %s\n",
-			xprt, xprt->address_strings[RPC_DISPLAY_ADDR]);
-
-	status = xs_local_finish_connecting(xprt, sock);
-	switch (status) {
-	case 0:
-		dprintk("RPC:       xprt %p connected to %s\n",
-				xprt, xprt->address_strings[RPC_DISPLAY_ADDR]);
-		xprt_set_connected(xprt);
-		break;
-	case -ENOENT:
-		dprintk("RPC:       xprt %p: socket %s does not exist\n",
-				xprt, xprt->address_strings[RPC_DISPLAY_ADDR]);
-		break;
-	case -ECONNREFUSED:
-		dprintk("RPC:       xprt %p: connection refused for %s\n",
-				xprt, xprt->address_strings[RPC_DISPLAY_ADDR]);
-		break;
-	default:
-		printk(KERN_ERR "%s: unhandled error (%d) connecting to %s\n",
-				__func__, -status,
-				xprt->address_strings[RPC_DISPLAY_ADDR]);
-	}
-
-out:
-	xprt_clear_connecting(xprt);
-	xprt_wake_pending_tasks(xprt, status);
-	current->flags &= ~PF_FSTRANS;
+	xprt_wake_pending_tasks(xprt, -EIO);
 }
 
 #ifdef CONFIG_SUNRPC_SWAP
@@ -2588,6 +2545,48 @@ static const struct rpc_timeout xs_local_default_timeout = {
 	.to_retries = 2,
 };
 
+
+static int xs_local_connect(struct sock_xprt *transport)
+{
+	struct rpc_xprt *xprt = &transport->xprt;
+	struct socket *sock;
+	int status = -EIO;
+
+	status = __sock_create(xprt->xprt_net, AF_LOCAL,
+					SOCK_STREAM, 0, &sock, 1);
+	if (status < 0) {
+		dprintk("RPC:       can't create AF_LOCAL "
+			"transport socket (%d).\n", -status);
+		return status;
+	}
+	xs_reclassify_socketu(sock);
+
+	dprintk("RPC:       worker connecting xprt %p via AF_LOCAL to %s\n",
+			xprt, xprt->address_strings[RPC_DISPLAY_ADDR]);
+
+	status = xs_local_finish_connecting(xprt, sock);
+	switch (status) {
+	case 0:
+		dprintk("RPC:       xprt %p connected to %s\n",
+				xprt, xprt->address_strings[RPC_DISPLAY_ADDR]);
+		xprt_set_connected(xprt);
+		break;
+	case -ENOENT:
+		dprintk("RPC:       xprt %p: socket %s does not exist\n",
+				xprt, xprt->address_strings[RPC_DISPLAY_ADDR]);
+		break;
+	case -ECONNREFUSED:
+		dprintk("RPC:       xprt %p: connection refused for %s\n",
+				xprt, xprt->address_strings[RPC_DISPLAY_ADDR]);
+		break;
+	default:
+		printk(KERN_ERR "%s: unhandled error (%d) connecting to %s\n",
+				__func__, -status,
+				xprt->address_strings[RPC_DISPLAY_ADDR]);
+	}
+	return status;
+}
+
 /**
 * xs_setup_local - Set up transport to use an AF_LOCAL socket
 * @args: rpc transport creation arguments
@@ -2630,6 +2629,9 @@ static struct rpc_xprt *xs_setup_local(struct xprt_create *args)
 		INIT_DELAYED_WORK(&transport->connect_worker,
 					xs_local_setup_socket);
 		xs_format_peer_addresses(xprt, "local", RPCBIND_NETID_LOCAL);
+		ret = ERR_PTR(xs_local_connect(transport));
+		if (ret)
+			goto out_err;
 		break;
 	default:
 		ret = ERR_PTR(-EAFNOSUPPORT);
--

To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 2 lines
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
#4 lore raw
Quoted 20 lines · J. Bruce Fields
Quoted 19 lines · Chuck Lever
Quoted 15 lines · J. Bruce Fields
Quoted 13 lines · bfields

The rpc code expects all connects to be done asynchronously by a workqueue. But that doesn't seem necessary in the AF_LOCAL case. The only user (rpcbind) actually wants the connect done in the context of a process with the right namespace. (And that will be true of gss proxy too, which also wants to use AF_LOCAL.)

But maybe I'm missing something.

Also, I haven't really tried to understand this code--I just assumed I could basically call xs_local_setup_socket from ->setup instead of the workqueue, and that seems to work based on a very superficial test. At a minimum I guess the PF_FSTRANS fiddling shouldn't be there.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

Here it is with that and the other extraneous xprt stuff gone.

See any problem with doing this?

Nothing is screaming at me. As long as an AF_LOCAL connect operation doesn't ever sleep, this should be safe, I think.

I'm sure it must sleep. Why would that make any difference?

As I understand it, sometimes an ASYNC RPC task is driving the connect, and such a task must never sleep when calling outside of rpciod. rpciod must be allowed to put that task on a wait queue and go do other work if the connect operation doesn't succeed immediately, otherwise all ASYNC RPC operations hang (or worse, an oops occurs).

Quoted 4 lines
Quoted 1 line

How did you test it?

I'm just doing my usual set of connectathon runs, and assuming mounts would fail if the server's rpcbind registration failed.

Have you tried killing rpcbind first to see how the error cases are handled?

Does rpcbind get the registration's "owner" field correct when namespaces are involved?

Signature · 2 lines
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
#5 lore raw
Quoted 28 lines · Chuck Lever
Quoted 24 lines · J. Bruce Fields
Quoted 23 lines · Chuck Lever
Quoted 18 lines · J. Bruce Fields
Quoted 6 lines · bfields

The rpc code expects all connects to be done asynchronously by a workqueue. But that doesn't seem necessary in the AF_LOCAL case. The only user (rpcbind) actually wants the connect done in the context of a process with the right namespace. (And that will be true of gss proxy too, which also wants to use AF_LOCAL.)

Quoted 7 lines

But maybe I'm missing something.

Also, I haven't really tried to understand this code--I just assumed I could basically call xs_local_setup_socket from ->setup instead of the workqueue, and that seems to work based on a very superficial test. At a minimum I guess the PF_FSTRANS fiddling shouldn't be there.

Here it is with that and the other extraneous xprt stuff gone.

See any problem with doing this?

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

Nothing is screaming at me. As long as an AF_LOCAL connect operation doesn't ever sleep, this should be safe, I think.

I'm sure it must sleep. Why would that make any difference?

As I understand it, sometimes an ASYNC RPC task is driving the connect, and such a task must never sleep when calling outside of rpciod.

AF_LOCAL is currently only used to register rpc services. I can't see any case when it's called asynchronously.

(And the same will be true of the gss-proxy calls, which also plan to use AF_LOCAL.)

Quoted 10 lines

rpciod must be allowed to put that task on a wait queue and go do other work if the connect operation doesn't succeed immediately, otherwise all ASYNC RPC operations hang (or worse, an oops occurs).

Quoted 4 lines
Quoted 1 line

How did you test it?

I'm just doing my usual set of connectathon runs, and assuming mounts would fail if the server's rpcbind registration failed.

Have you tried killing rpcbind first to see how the error cases are handled?

No, thanks for the suggestion, I'll check.

Quoted 2 lines

Does rpcbind get the registration's "owner" field correct when namespaces are involved?

Looking at rpcb_clnt.c.... I only ever see r_owner set to "" or "0". I can't see why that would need to change in a container.

--b.

#6 lore raw
Quoted 38 lines · J. Bruce Fields
Quoted 28 lines · Chuck Lever
Quoted 24 lines · J. Bruce Fields
Quoted 23 lines · Chuck Lever
Quoted 18 lines · J. Bruce Fields
Quoted 6 lines · bfields

The rpc code expects all connects to be done asynchronously by a workqueue. But that doesn't seem necessary in the AF_LOCAL case. The only user (rpcbind) actually wants the connect done in the context of a process with the right namespace. (And that will be true of gss proxy too, which also wants to use AF_LOCAL.)

Quoted 7 lines

But maybe I'm missing something.

Also, I haven't really tried to understand this code--I just assumed I could basically call xs_local_setup_socket from ->setup instead of the workqueue, and that seems to work based on a very superficial test. At a minimum I guess the PF_FSTRANS fiddling shouldn't be there.

Here it is with that and the other extraneous xprt stuff gone.

See any problem with doing this?

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

Nothing is screaming at me. As long as an AF_LOCAL connect operation doesn't ever sleep, this should be safe, I think.

I'm sure it must sleep. Why would that make any difference?

As I understand it, sometimes an ASYNC RPC task is driving the connect, and such a task must never sleep when calling outside of rpciod.

Quoted 5 lines

AF_LOCAL is currently only used to register rpc services. I can't see any case when it's called asynchronously.

(And the same will be true of the gss-proxy calls, which also plan to use AF_LOCAL.)

Yes, but AF_LOCAL is supposed to be a generic transport for RPC. This is not a feature we just made up, it's actually a well-defined API that exists on other platforms (it's even specified in RFCs). Right now I would hesitate to restrict the use of AF_LOCAL upcalls to only synchronous contexts, because eventually we may want to use the transport in asynchronous contexts.

If we were to go with using a synchronous connect, however, I think there should be some kind of safety check to make sure the xs connect function is not being invoked from an asynchronous context. This is a restriction that does not exist for other transports supported by the kernel RPC client, so it should be underscored in the code.

Alternately, perhaps your new code could continue to invoke the kernel connect operation in a non-blocking mode, and simply have the local transport setup function fail if the non-blocking connect operation does not return with success. That would be an interesting test to see if an AF_LOCAL connect operation actually does try to sleep in common code paths.

Quoted 12 lines
Quoted 11 lines

rpciod must be allowed to put that task on a wait queue and go do other work if the connect operation doesn't succeed immediately, otherwise all ASYNC RPC operations hang (or worse, an oops occurs).

Quoted 4 lines
Quoted 1 line

How did you test it?

I'm just doing my usual set of connectathon runs, and assuming mounts would fail if the server's rpcbind registration failed.

Have you tried killing rpcbind first to see how the error cases are handled?

Quoted 7 lines

No, thanks for the suggestion, I'll check.

Quoted 2 lines

Does rpcbind get the registration's "owner" field correct when namespaces are involved?

Looking at rpcb_clnt.c.... I only ever see r_owner set to "" or "0". I can't see why that would need to change in a container.

rpcbind scrapes the remote end of the socket to see who's calling. IIRC the r_owner field is ignored if it's not root who is making the registration request.

This is the raison d'être to register RPC services via AF_LOCAL transports. Thus it should be explicitly tested that rpcbind gets registration correct after changes are made to AF_LOCAL support in the kernel's RPC client.

Signature · 2 lines
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
#7 lore raw
Quoted 1 line · Chuck Lever

Yes, but AF_LOCAL is supposed to be a generic transport for RPC. This is not a feature we just made up, it's actually a well-defined API that exists on other platforms (it's even specified in RFCs). Right now I would hesitate to restrict the use of AF_LOCAL upcalls to only synchronous contexts, because eventually we may want to use the transport in asynchronous contexts.

The whole problem is that it is a piss-poorly defined feature in an asynchronous kernel context.

Sockets carry around a well defined net namespace context that allow them to resolve IP addresses. However they carry none of the file namespace context information that is required to make sense of AF_LOCAL "addresses".

IOW we have 3 options:

1. Drop AF_LOCAL support altogether
2. Add file namespace context to the RPC or socket layers
3. Drop asynchronous support, so that we have a reliable userspace-defined context.

1) involves a user space api change, which will bring down the iron fist of the Finn.
2) involves cooperation from the VFS and socket folks which doesn't seem to be happening.

so that leaves (3), which is perfectly doable since we do _not_ want to expose the rpc layer to anything outside the kernel. It's not intended as a generic libtirpc...

Quoted 1 line

If we were to go with using a synchronous connect, however, I think there should be some kind of safety check to make sure the xs connect function is not being invoked from an asynchronous context. This is a restriction that does not exist for other transports supported by the kernel RPC client, so it should be underscored in the code.

void xs_connect_local(struct rpc_task *task)

{
	if (RPC_IS_ASYNC(task))
		rpc_exit(task, -ENOTCONN);

.....
}

...done.

Signature · 6 lines
Trond Myklebust
Linux NFS client maintainer

NetApp
[email protected]
www.netapp.com
#8 lore raw
Quoted 30 lines · Trond
Quoted 1 line · Chuck Lever

Yes, but AF_LOCAL is supposed to be a generic transport for RPC. This is not a feature we just made up, it's actually a well-defined API that exists on other platforms (it's even specified in RFCs). Right now I would hesitate to restrict the use of AF_LOCAL upcalls to only synchronous contexts, because eventually we may want to use the transport in asynchronous contexts.

The whole problem is that it is a piss-poorly defined feature in an asynchronous kernel context.

Sockets carry around a well defined net namespace context that allow them to resolve IP addresses. However they carry none of the file namespace context information that is required to make sense of AF_LOCAL "addresses".

IOW we have 3 options:

1. Drop AF_LOCAL support altogether
2. Add file namespace context to the RPC or socket layers
3. Drop asynchronous support, so that we have a reliable userspace-defined context.

1) involves a user space api change, which will bring down the iron fist of the Finn.
2) involves cooperation from the VFS and socket folks which doesn't seem to be happening.

so that leaves (3), which is perfectly doable since we do _not_ want to expose the rpc layer to anything outside the kernel. It's not intended as a generic libtirpc...

Quoted 1 line

If we were to go with using a synchronous connect, however, I think there should be some kind of safety check to make sure the xs connect function is not being invoked from an asynchronous context. This is a restriction that does not exist for other transports supported by the kernel RPC client, so it should be underscored in the code.

Quoted 9 lines

void xs_connect_local(struct rpc_task *task)

{
	if (RPC_IS_ASYNC(task))
		rpc_exit(task, -ENOTCONN);

.....
}

...done.

This seems the most reasonable approach to me too, and makes the code simpler for now.

Simo.

Signature · 1 line
Simo Sorce * Red Hat, Inc * New York
#14 lore raw
In reply to Myklebust, Trond
Quoted 11 lines · Trond
Quoted 1 line · Chuck Lever

Yes, but AF_LOCAL is supposed to be a generic transport for RPC. This is not a feature we just made up, it's actually a well-defined API that exists on other platforms (it's even specified in RFCs). Right now I would hesitate to restrict the use of AF_LOCAL upcalls to only synchronous contexts, because eventually we may want to use the transport in asynchronous contexts.

The whole problem is that it is a piss-poorly defined feature in an asynchronous kernel context.

Sockets carry around a well defined net namespace context that allow them to resolve IP addresses. However they carry none of the file namespace context information that is required to make sense of AF_LOCAL "addresses".

I recognize this problem, but I fail to see how it is connected to asynchronicity in general. The issue seems to be specifically how rpciod implements asynchronicity.

Quoted 9 lines

IOW we have 3 options:

1. Drop AF_LOCAL support altogether
2. Add file namespace context to the RPC or socket layers
3. Drop asynchronous support, so that we have a reliable userspace-defined context.

1) involves a user space api change, which will bring down the iron fist of the Finn.

The problem with 1) is that rpcbind uses a special feature of AF_LOCAL to protect registrations from being removed by a malicious or ignorant registrant. That's why I added AF_LOCAL. Somehow we would have to replace that feature.

Quoted 2 lines

2) involves cooperation from the VFS and socket folks which doesn't seem to be happening.

Yep, I'm aware of that.

Quoted 3 lines

so that leaves (3), which is perfectly doable since we do _not_ want to expose the rpc layer to anything outside the kernel. It's not intended as a generic libtirpc...

I hoped for a better alternative, but I see that folks do not have the patience for that. ;-)

Quoted 11 lines
Quoted 2 lines

If we were to go with using a synchronous connect, however, I think there should be some kind of safety check to make sure the xs connect function is not being invoked from an asynchronous context. This is a restriction that does not exist for other transports supported by the kernel RPC client, so it should be underscored in the code.

void xs_connect_local(struct rpc_task *task)

{
	if (RPC_IS_ASYNC(task))
		rpc_exit(task, -ENOTCONN);

.....
}

...done.

That's what I had in mind. I might even add a WARN_ON_ONCE().

Signature · 2 lines
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
#15 lore raw
Quoted 9 lines · Chuck Lever
Quoted 7 lines · Trond
Quoted 1 line · Chuck Lever

Yes, but AF_LOCAL is supposed to be a generic transport for RPC. This is not a feature we just made up, it's actually a well-defined API that exists on other platforms (it's even specified in RFCs). Right now I would hesitate to restrict the use of AF_LOCAL upcalls to only synchronous contexts, because eventually we may want to use the transport in asynchronous contexts.

The whole problem is that it is a piss-poorly defined feature in an asynchronous kernel context.

Quoted 6 lines
Quoted 4 lines

Sockets carry around a well defined net namespace context that allow them to resolve IP addresses. However they carry none of the file namespace context information that is required to make sense of AF_LOCAL "addresses".

I recognize this problem, but I fail to see how it is connected to asynchronicity in general. The issue seems to be specifically how rpciod implements asynchronicity.

The problem is connected to asynchronicity in general because there is no userspace context to fall back on. Asynchronous contexts are pretty much by definition required to drop their connection to the original context in order to be useful.

As for adding more userspace context information to rpciod, I refuse point blank to even start that discussion until someone has a valid use case.

Quoted 37 lines
Quoted 9 lines

IOW we have 3 options:

1. Drop AF_LOCAL support altogether
2. Add file namespace context to the RPC or socket layers
3. Drop asynchronous support, so that we have a reliable userspace-defined context.

1) involves a user space api change, which will bring down the iron fist of the Finn.

The problem with 1) is that rpcbind uses a special feature of AF_LOCAL to protect registrations from being removed by a malicious or ignorant registrant. That's why I added AF_LOCAL. Somehow we would have to replace that feature.

Quoted 2 lines

2) involves cooperation from the VFS and socket folks which doesn't seem to be happening.

Yep, I'm aware of that.

Quoted 3 lines

so that leaves (3), which is perfectly doable since we do _not_ want to expose the rpc layer to anything outside the kernel. It's not intended as a generic libtirpc...

I hoped for a better alternative, but I see that folks do not have the patience for that. ;-)

Quoted 11 lines
Quoted 2 lines

If we were to go with using a synchronous connect, however, I think there should be some kind of safety check to make sure the xs connect function is not being invoked from an asynchronous context. This is a restriction that does not exist for other transports supported by the kernel RPC client, so it should be underscored in the code.

void xs_connect_local(struct rpc_task *task)

{
	if (RPC_IS_ASYNC(task))
		rpc_exit(task, -ENOTCONN);

.....
}

...done.

That's what I had in mind. I might even add a WARN_ON_ONCE().

Signature · 6 lines
Trond Myklebust
Linux NFS client maintainer

NetApp
[email protected]
www.netapp.com
#9 lore raw
In reply to Simo Sorce
Quoted 38 lines · Simo Sorce
Quoted 37 lines · Trond
Quoted 1 line · Chuck Lever

Yes, but AF_LOCAL is supposed to be a generic transport for RPC. This is not a feature we just made up, it's actually a well-defined API that exists on other platforms (it's even specified in RFCs). Right now I would hesitate to restrict the use of AF_LOCAL upcalls to only synchronous contexts, because eventually we may want to use the transport in asynchronous contexts.

The whole problem is that it is a piss-poorly defined feature in an asynchronous kernel context.

Sockets carry around a well defined net namespace context that allow them to resolve IP addresses. However they carry none of the file namespace context information that is required to make sense of AF_LOCAL "addresses".

IOW we have 3 options:

1. Drop AF_LOCAL support altogether
2. Add file namespace context to the RPC or socket layers
3. Drop asynchronous support, so that we have a reliable userspace-defined context.

1) involves a user space api change, which will bring down the iron fist of the Finn.
2) involves cooperation from the VFS and socket folks which doesn't seem to be happening.

so that leaves (3), which is perfectly doable since we do _not_ want to expose the rpc layer to anything outside the kernel. It's not intended as a generic libtirpc...

Quoted 1 line

If we were to go with using a synchronous connect, however, I think there should be some kind of safety check to make sure the xs connect function is not being invoked from an asynchronous context. This is a restriction that does not exist for other transports supported by the kernel RPC client, so it should be underscored in the code.

void xs_connect_local(struct rpc_task *task)

{
	if (RPC_IS_ASYNC(task))
		rpc_exit(task, -ENOTCONN);

.....
}

Quoted 5 lines
Quoted 2 lines

...done.

This seems the most reasonable approach to me too, and makes the code simpler for now.

OK, I've added that check and fixed some other bugs (thanks to Chuck for some help in IRC).

I think that gets rpcbind working in containers fine.

gss-proxy has one more problem: it needs to do upcalls from nfsd threads which won't have the right filesystem namespace.

I get a write from gss-proxy when it starts and can do an initial connect then using its context. But if we disconnect after that I'm stuck.

Does it cause any problems if I just set the idle_timeout to 0 for AF_LOCAL?

--b.

#10 lore raw
Quoted 8 lines · J. Bruce Fields

OK, I've added that check and fixed some other bugs (thanks to Chuck for some help in IRC).

I think that gets rpcbind working in containers fine.

gss-proxy has one more problem: it needs to do upcalls from nfsd threads which won't have the right filesystem namespace.

Quoted 6 lines

I get a write from gss-proxy when it starts and can do an initial connect then using its context. But if we disconnect after that I'm stuck.

Does it cause any problems if I just set the idle_timeout to 0 for AF_LOCAL?

That gives me the following three patches. They work for me.

Would it make more sense to make the idle timeout configurable? I couldn't see why disconnecting idle AF_LOCAL rpcbind connections would be particularly important anyway.

--b.

commit 6656841afa0602f7aae3e42648eb44bfe79f7389

Author: J. Bruce Fields <[email protected]>
Date:   Wed Feb 20 17:52:19 2013 -0500

SUNRPC: make AF_LOCAL connect synchronous

It doesn't appear that anyone actually needs to connect asynchronously.

Also, using a workqueue for the connect means we lose the namespace information from the original process. This is a problem since there's no way to explicitly pass in a filesystem namespace for resolution of an AF_LOCAL address.

net/sunrpc/xprtsock.c +27 -8
index bbc0915..b1df874 100644
@@ -1866,13 +1866,9 @@ static int xs_local_finish_connecting(struct rpc_xprt *xprt,
  * @xprt: RPC transport to connect
  * @transport: socket transport to connect
  * @create_sock: function to create a socket of the correct type
- *
- * Invoked by a work queue tasklet.
  */
-static void xs_local_setup_socket(struct work_struct *work)
+static void xs_local_setup_socket(struct sock_xprt *transport)
 {
-	struct sock_xprt *transport =
-		container_of(work, struct sock_xprt, connect_worker.work);
 	struct rpc_xprt *xprt = &transport->xprt;
 	struct socket *sock;
 	int status = -EIO;
@@ -1919,6 +1915,31 @@ out:
 	current->flags &= ~PF_FSTRANS;
 }
 
+static void xs_local_connect(struct rpc_task *task)
+{
+	struct rpc_xprt *xprt = task->tk_xprt;
+	struct sock_xprt *transport = container_of(xprt, struct sock_xprt, xprt);
+	unsigned long timeout;
+
+	 if (RPC_IS_ASYNC(task))
+	 	rpc_exit(task, -ENOTCONN);
+
+	if (transport->sock != NULL && !RPC_IS_SOFTCONN(task)) {
+		dprintk("RPC:       xs_connect delayed xprt %p for %lu "
+				"seconds\n",
+				xprt, xprt->reestablish_timeout / HZ);
+		timeout = xprt->reestablish_timeout;
+		xprt->reestablish_timeout <<= 1;
+		if (xprt->reestablish_timeout < XS_TCP_INIT_REEST_TO)
+			xprt->reestablish_timeout = XS_TCP_INIT_REEST_TO;
+		if (xprt->reestablish_timeout > XS_TCP_MAX_REEST_TO)
+			xprt->reestablish_timeout = XS_TCP_MAX_REEST_TO;
+		rpc_delay(task, timeout);
+	} else
+		dprintk("RPC:       xs_connect scheduled xprt %p\n", xprt);
+	xs_local_setup_socket(transport);
+}
+
 #ifdef CONFIG_SUNRPC_SWAP
 static void xs_set_memalloc(struct rpc_xprt *xprt)
 {
@@ -2454,7 +2475,7 @@ static struct rpc_xprt_ops xs_local_ops = {
 	.alloc_slot		= xprt_alloc_slot,
 	.rpcbind		= xs_local_rpcbind,
 	.set_port		= xs_local_set_port,
-	.connect		= xs_connect,
+	.connect		= xs_local_connect,
 	.buf_alloc		= rpc_malloc,
 	.buf_free		= rpc_free,
 	.send_request		= xs_local_send_request,
@@ -2627,8 +2648,6 @@ static struct rpc_xprt *xs_setup_local(struct xprt_create *args)
 			goto out_err;
 		}
 		xprt_set_bound(xprt);
-		INIT_DELAYED_WORK(&transport->connect_worker,
-					xs_local_setup_socket);
 		xs_format_peer_addresses(xprt, "local", RPCBIND_NETID_LOCAL);
 		break;
 	default:

commit 3d622fe729b9b4382785c3ef2ef61e484df1b3ec

Author: J. Bruce Fields <[email protected]>
Date:   Thu Feb 21 10:14:22 2013 -0500

SUNRPC: attempt AF_LOCAL connect on setup

In the gss-proxy case, setup time is when I know I'll have the right namespace for the connect.

In other cases, it might be useful to get any connection errors earlier--though actually in practice it doesn't make any difference for rpcbind.

net/sunrpc/xprtsock.c +5 -1
index b1df874..f2cf652 100644
@@ -1867,7 +1867,7 @@ static int xs_local_finish_connecting(struct rpc_xprt *xprt,
  * @transport: socket transport to connect
  * @create_sock: function to create a socket of the correct type
  */
-static void xs_local_setup_socket(struct sock_xprt *transport)
+static int xs_local_setup_socket(struct sock_xprt *transport)
 {
 	struct rpc_xprt *xprt = &transport->xprt;
 	struct socket *sock;
@@ -1913,6 +1913,7 @@ out:
 	xprt_clear_connecting(xprt);
 	xprt_wake_pending_tasks(xprt, status);
 	current->flags &= ~PF_FSTRANS;
+	return status;
 }
 
 static void xs_local_connect(struct rpc_task *task)
@@ -2649,6 +2650,9 @@ static struct rpc_xprt *xs_setup_local(struct xprt_create *args)
 		}
 		xprt_set_bound(xprt);
 		xs_format_peer_addresses(xprt, "local", RPCBIND_NETID_LOCAL);
+		ret = ERR_PTR(xs_local_setup_socket(transport));
+		if (ret)
+			goto out_err;
 		break;
 	default:
 		ret = ERR_PTR(-EAFNOSUPPORT);

commit 1a67db92015506ca07e6fc7a24583917adcbb43d

Author: J. Bruce Fields <[email protected]>
Date:   Wed Feb 20 18:08:52 2013 -0500

SUNRPC: no idle timeout for AF_LOCAL sockets

In the gss-proxy case I don't want to have to reconnect at random--I want to connect only on gss-proxy startup when I can steal gss-proxy's context to do the connect in the right namespace.

And surely an AF_LOCAL socket isn't a ton of state to keep around--how about we just turn off the idle timeout for AF_LOCAL sockets.

net/sunrpc/xprtsock.c +1 -1
index f2cf652..a32227e 100644
@@ -2635,7 +2635,7 @@ static struct rpc_xprt *xs_setup_local(struct xprt_create *args)
 
 	xprt->bind_timeout = XS_BIND_TO;
 	xprt->reestablish_timeout = XS_TCP_INIT_REEST_TO;
-	xprt->idle_timeout = XS_IDLE_DISC_TO;
+	xprt->idle_timeout = 0;
 
 	xprt->ops = &xs_local_ops;
 	xprt->timeout = &xs_local_default_timeout;
#11 lore raw
Quoted 13 lines · J. Bruce Fields
Quoted 12 lines · J. Bruce Fields

OK, I've added that check and fixed some other bugs (thanks to Chuck for some help in IRC).

I think that gets rpcbind working in containers fine.

gss-proxy has one more problem: it needs to do upcalls from nfsd threads which won't have the right filesystem namespace.

I get a write from gss-proxy when it starts and can do an initial connect then using its context. But if we disconnect after that I'm stuck.

Quoted 8 lines
Quoted 2 lines

Does it cause any problems if I just set the idle_timeout to 0 for AF_LOCAL?

That gives me the following three patches. They work for me.

Would it make more sense to make the idle timeout configurable? I couldn't see why disconnecting idle AF_LOCAL rpcbind connections would be particularly important anyway.

I was expecting you to add a new flag to the rpc_clnt (like "disconnect on retry") that would disable the transport idle timeout.

Quoted 166 lines

--b.

commit 6656841afa0602f7aae3e42648eb44bfe79f7389

Author: J. Bruce Fields <[email protected]>
Date:   Wed Feb 20 17:52:19 2013 -0500

SUNRPC: make AF_LOCAL connect synchronous

It doesn't appear that anyone actually needs to connect asynchronously.

Also, using a workqueue for the connect means we lose the namespace information from the original process. This is a problem since there's no way to explicitly pass in a filesystem namespace for resolution of an AF_LOCAL address.

net/sunrpc/xprtsock.c +27 -8
index bbc0915..b1df874 100644
@@ -1866,13 +1866,9 @@ static int xs_local_finish_connecting(struct rpc_xprt *xprt,
 * @xprt: RPC transport to connect
 * @transport: socket transport to connect
 * @create_sock: function to create a socket of the correct type
- *
- * Invoked by a work queue tasklet.
 */
-static void xs_local_setup_socket(struct work_struct *work)
+static void xs_local_setup_socket(struct sock_xprt *transport)
 {
-	struct sock_xprt *transport =
-		container_of(work, struct sock_xprt, connect_worker.work);
 	struct rpc_xprt *xprt = &transport->xprt;
 	struct socket *sock;
 	int status = -EIO;
@@ -1919,6 +1915,31 @@ out:
 	current->flags &= ~PF_FSTRANS;
 }
 
+static void xs_local_connect(struct rpc_task *task)
+{
+	struct rpc_xprt *xprt = task->tk_xprt;
+	struct sock_xprt *transport = container_of(xprt, struct sock_xprt, xprt);
+	unsigned long timeout;
+
+	 if (RPC_IS_ASYNC(task))
+	 	rpc_exit(task, -ENOTCONN);
+
+	if (transport->sock != NULL && !RPC_IS_SOFTCONN(task)) {
+		dprintk("RPC:       xs_connect delayed xprt %p for %lu "
+				"seconds\n",
+				xprt, xprt->reestablish_timeout / HZ);
+		timeout = xprt->reestablish_timeout;
+		xprt->reestablish_timeout <<= 1;
+		if (xprt->reestablish_timeout < XS_TCP_INIT_REEST_TO)
+			xprt->reestablish_timeout = XS_TCP_INIT_REEST_TO;
+		if (xprt->reestablish_timeout > XS_TCP_MAX_REEST_TO)
+			xprt->reestablish_timeout = XS_TCP_MAX_REEST_TO;
+		rpc_delay(task, timeout);
+	} else
+		dprintk("RPC:       xs_connect scheduled xprt %p\n", xprt);
+	xs_local_setup_socket(transport);
+}
+
 #ifdef CONFIG_SUNRPC_SWAP
 static void xs_set_memalloc(struct rpc_xprt *xprt)
 {
@@ -2454,7 +2475,7 @@ static struct rpc_xprt_ops xs_local_ops = {
 	.alloc_slot		= xprt_alloc_slot,
 	.rpcbind		= xs_local_rpcbind,
 	.set_port		= xs_local_set_port,
-	.connect		= xs_connect,
+	.connect		= xs_local_connect,
 	.buf_alloc		= rpc_malloc,
 	.buf_free		= rpc_free,
 	.send_request		= xs_local_send_request,
@@ -2627,8 +2648,6 @@ static struct rpc_xprt *xs_setup_local(struct xprt_create *args)
 			goto out_err;
 		}
 		xprt_set_bound(xprt);
-		INIT_DELAYED_WORK(&transport->connect_worker,
-					xs_local_setup_socket);
 		xs_format_peer_addresses(xprt, "local", RPCBIND_NETID_LOCAL);
 		break;
 	default:

commit 3d622fe729b9b4382785c3ef2ef61e484df1b3ec

Author: J. Bruce Fields <[email protected]>
Date:   Thu Feb 21 10:14:22 2013 -0500

SUNRPC: attempt AF_LOCAL connect on setup

In the gss-proxy case, setup time is when I know I'll have the right namespace for the connect.

In other cases, it might be useful to get any connection errors earlier--though actually in practice it doesn't make any difference for rpcbind.

net/sunrpc/xprtsock.c +5 -1
index b1df874..f2cf652 100644
@@ -1867,7 +1867,7 @@ static int xs_local_finish_connecting(struct rpc_xprt *xprt,
 * @transport: socket transport to connect
 * @create_sock: function to create a socket of the correct type
 */
-static void xs_local_setup_socket(struct sock_xprt *transport)
+static int xs_local_setup_socket(struct sock_xprt *transport)
 {
 	struct rpc_xprt *xprt = &transport->xprt;
 	struct socket *sock;
@@ -1913,6 +1913,7 @@ out:
 	xprt_clear_connecting(xprt);
 	xprt_wake_pending_tasks(xprt, status);
 	current->flags &= ~PF_FSTRANS;
+	return status;
 }
 
 static void xs_local_connect(struct rpc_task *task)
@@ -2649,6 +2650,9 @@ static struct rpc_xprt *xs_setup_local(struct xprt_create *args)
 		}
 		xprt_set_bound(xprt);
 		xs_format_peer_addresses(xprt, "local", RPCBIND_NETID_LOCAL);
+		ret = ERR_PTR(xs_local_setup_socket(transport));
+		if (ret)
+			goto out_err;
 		break;
 	default:
 		ret = ERR_PTR(-EAFNOSUPPORT);

commit 1a67db92015506ca07e6fc7a24583917adcbb43d

Author: J. Bruce Fields <[email protected]>
Date:   Wed Feb 20 18:08:52 2013 -0500

SUNRPC: no idle timeout for AF_LOCAL sockets

In the gss-proxy case I don't want to have to reconnect at random--I want to connect only on gss-proxy startup when I can steal gss-proxy's context to do the connect in the right namespace.

And surely an AF_LOCAL socket isn't a ton of state to keep around--how about we just turn off the idle timeout for AF_LOCAL sockets.

net/sunrpc/xprtsock.c +1 -2
index f2cf652..a32227e 100644
@@ -2635,7 +2635,7 @@ static struct rpc_xprt *xs_setup_local(struct xprt_create *args)
 
 	xprt->bind_timeout = XS_BIND_TO;
 	xprt->reestablish_timeout = XS_TCP_INIT_REEST_TO;
-	xprt->idle_timeout = XS_IDLE_DISC_TO;
+	xprt->idle_timeout = 0;
 
 	xprt->ops = &xs_local_ops;
 	xprt->timeout = &xs_local_default_timeout;
--

To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 2 lines
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
#12 lore raw
Quoted 19 lines · Chuck Lever
Quoted 16 lines · J. Bruce Fields
Quoted 14 lines · J. Bruce Fields

OK, I've added that check and fixed some other bugs (thanks to Chuck for some help in IRC).

I think that gets rpcbind working in containers fine.

gss-proxy has one more problem: it needs to do upcalls from nfsd threads which won't have the right filesystem namespace.

I get a write from gss-proxy when it starts and can do an initial connect then using its context. But if we disconnect after that I'm stuck.

Does it cause any problems if I just set the idle_timeout to 0 for AF_LOCAL?

Quoted 7 lines
Quoted 5 lines

That gives me the following three patches. They work for me.

Would it make more sense to make the idle timeout configurable? I couldn't see why disconnecting idle AF_LOCAL rpcbind connections would be particularly important anyway.

I was expecting you to add a new flag to the rpc_clnt (like "disconnect on retry") that would disable the transport idle timeout.

That would be OK with me too. What's best?

--b.

Quoted 176 lines
Quoted 168 lines

--b.

commit 6656841afa0602f7aae3e42648eb44bfe79f7389

Author: J. Bruce Fields <[email protected]>
Date:   Wed Feb 20 17:52:19 2013 -0500

SUNRPC: make AF_LOCAL connect synchronous

It doesn't appear that anyone actually needs to connect asynchronously.

Also, using a workqueue for the connect means we lose the namespace information from the original process. This is a problem since there's no way to explicitly pass in a filesystem namespace for resolution of an AF_LOCAL address.

net/sunrpc/xprtsock.c +27 -8
index bbc0915..b1df874 100644
@@ -1866,13 +1866,9 @@ static int xs_local_finish_connecting(struct rpc_xprt *xprt,
 * @xprt: RPC transport to connect
 * @transport: socket transport to connect
 * @create_sock: function to create a socket of the correct type
- *
- * Invoked by a work queue tasklet.
 */
-static void xs_local_setup_socket(struct work_struct *work)
+static void xs_local_setup_socket(struct sock_xprt *transport)
 {
-	struct sock_xprt *transport =
-		container_of(work, struct sock_xprt, connect_worker.work);
 	struct rpc_xprt *xprt = &transport->xprt;
 	struct socket *sock;
 	int status = -EIO;
@@ -1919,6 +1915,31 @@ out:
 	current->flags &= ~PF_FSTRANS;
 }
 
+static void xs_local_connect(struct rpc_task *task)
+{
+	struct rpc_xprt *xprt = task->tk_xprt;
+	struct sock_xprt *transport = container_of(xprt, struct sock_xprt, xprt);
+	unsigned long timeout;
+
+	 if (RPC_IS_ASYNC(task))
+	 	rpc_exit(task, -ENOTCONN);
+
+	if (transport->sock != NULL && !RPC_IS_SOFTCONN(task)) {
+		dprintk("RPC:       xs_connect delayed xprt %p for %lu "
+				"seconds\n",
+				xprt, xprt->reestablish_timeout / HZ);
+		timeout = xprt->reestablish_timeout;
+		xprt->reestablish_timeout <<= 1;
+		if (xprt->reestablish_timeout < XS_TCP_INIT_REEST_TO)
+			xprt->reestablish_timeout = XS_TCP_INIT_REEST_TO;
+		if (xprt->reestablish_timeout > XS_TCP_MAX_REEST_TO)
+			xprt->reestablish_timeout = XS_TCP_MAX_REEST_TO;
+		rpc_delay(task, timeout);
+	} else
+		dprintk("RPC:       xs_connect scheduled xprt %p\n", xprt);
+	xs_local_setup_socket(transport);
+}
+
 #ifdef CONFIG_SUNRPC_SWAP
 static void xs_set_memalloc(struct rpc_xprt *xprt)
 {
@@ -2454,7 +2475,7 @@ static struct rpc_xprt_ops xs_local_ops = {
 	.alloc_slot		= xprt_alloc_slot,
 	.rpcbind		= xs_local_rpcbind,
 	.set_port		= xs_local_set_port,
-	.connect		= xs_connect,
+	.connect		= xs_local_connect,
 	.buf_alloc		= rpc_malloc,
 	.buf_free		= rpc_free,
 	.send_request		= xs_local_send_request,
@@ -2627,8 +2648,6 @@ static struct rpc_xprt *xs_setup_local(struct xprt_create *args)
 			goto out_err;
 		}
 		xprt_set_bound(xprt);
-		INIT_DELAYED_WORK(&transport->connect_worker,
-					xs_local_setup_socket);
 		xs_format_peer_addresses(xprt, "local", RPCBIND_NETID_LOCAL);
 		break;
 	default:

commit 3d622fe729b9b4382785c3ef2ef61e484df1b3ec

Author: J. Bruce Fields <[email protected]>
Date:   Thu Feb 21 10:14:22 2013 -0500

SUNRPC: attempt AF_LOCAL connect on setup

In the gss-proxy case, setup time is when I know I'll have the right namespace for the connect.

In other cases, it might be useful to get any connection errors earlier--though actually in practice it doesn't make any difference for rpcbind.

net/sunrpc/xprtsock.c +5 -1
index b1df874..f2cf652 100644
@@ -1867,7 +1867,7 @@ static int xs_local_finish_connecting(struct rpc_xprt *xprt,
 * @transport: socket transport to connect
 * @create_sock: function to create a socket of the correct type
 */
-static void xs_local_setup_socket(struct sock_xprt *transport)
+static int xs_local_setup_socket(struct sock_xprt *transport)
 {
 	struct rpc_xprt *xprt = &transport->xprt;
 	struct socket *sock;
@@ -1913,6 +1913,7 @@ out:
 	xprt_clear_connecting(xprt);
 	xprt_wake_pending_tasks(xprt, status);
 	current->flags &= ~PF_FSTRANS;
+	return status;
 }
 
 static void xs_local_connect(struct rpc_task *task)
@@ -2649,6 +2650,9 @@ static struct rpc_xprt *xs_setup_local(struct xprt_create *args)
 		}
 		xprt_set_bound(xprt);
 		xs_format_peer_addresses(xprt, "local", RPCBIND_NETID_LOCAL);
+		ret = ERR_PTR(xs_local_setup_socket(transport));
+		if (ret)
+			goto out_err;
 		break;
 	default:
 		ret = ERR_PTR(-EAFNOSUPPORT);

commit 1a67db92015506ca07e6fc7a24583917adcbb43d

Author: J. Bruce Fields <[email protected]>
Date:   Wed Feb 20 18:08:52 2013 -0500

SUNRPC: no idle timeout for AF_LOCAL sockets

In the gss-proxy case I don't want to have to reconnect at random--I want to connect only on gss-proxy startup when I can steal gss-proxy's context to do the connect in the right namespace.

And surely an AF_LOCAL socket isn't a ton of state to keep around--how about we just turn off the idle timeout for AF_LOCAL sockets.

net/sunrpc/xprtsock.c +1 -2
index f2cf652..a32227e 100644
@@ -2635,7 +2635,7 @@ static struct rpc_xprt *xs_setup_local(struct xprt_create *args)
 
 	xprt->bind_timeout = XS_BIND_TO;
 	xprt->reestablish_timeout = XS_TCP_INIT_REEST_TO;
-	xprt->idle_timeout = XS_IDLE_DISC_TO;
+	xprt->idle_timeout = 0;
 
 	xprt->ops = &xs_local_ops;
 	xprt->timeout = &xs_local_default_timeout;
--

To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 2 lines
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
#13 lore raw
In reply to J. Bruce Fields
Quoted 13 lines · J. Bruce Fields
Quoted 12 lines · J. Bruce Fields

OK, I've added that check and fixed some other bugs (thanks to Chuck for some help in IRC).

I think that gets rpcbind working in containers fine.

gss-proxy has one more problem: it needs to do upcalls from nfsd threads which won't have the right filesystem namespace.

I get a write from gss-proxy when it starts and can do an initial connect then using its context. But if we disconnect after that I'm stuck.

Quoted 8 lines
Quoted 2 lines

Does it cause any problems if I just set the idle_timeout to 0 for AF_LOCAL?

That gives me the following three patches. They work for me.

Would it make more sense to make the idle timeout configurable? I couldn't see why disconnecting idle AF_LOCAL rpcbind connections would be particularly important anyway.

Also posted a new version of the gss-proxy series including these patches.

--b.