Skip to contentSkip to conversation

NFSv4 callback find client fix Version 3

PATCH In review Andy Adamson 2 participants 19 messages Dec 13, 2010 – Dec 15, 2010

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover NFSv4 callback find client fix Version 3

Version 3 of these patches
Applies to Tronds nfs-for-next branch

The back channel server svc_process_common RPC layer pg_authenticate call [nfs_callback_authenticate] is shared by both the NFSv4.0 and the NFSv4.1 callback threads. It authenticates the incoming request by finding (and referencing) an nfs_client struct based on the incoming request address and the NFS version (4). This is akin to the NFS server which authenticates requests by matching the server address to the exports file client list.

Since there is no minorversion in the search, it may find the wrong nfs_client struct. For the nfsv4.0 callback service thread, this means it could find an NFSv4.1 nfs_client. For the NFSv4.1 callback service thread, it could find an NFSv4.0 instead of v4.1, or find an NFSv4.1 nfs_client with the wrong session.

The nfs_client is dereferenced at the end of pg_authenticate. Another nfs_find_client call is done in the NFS layouer per operation dispatcher routines for NFSv4.0 and in the cb_sequence operation dispatcher routine for NFSv4.1 after decoding.

This means the callback server could start processing a callback, passing the pg_authenticate test, and have the nfs_client struct freed between the pg_authenticate call and the dispatcher operation call. Or, it could have found the wrong nfs_client in the pg_authenticate call.

The current code has this behavior: If the nfs_client is not found in pg_authenticate, the request is simply dropped (SVC_DROP). If an nfs_client is not found in the dispatcher routines NFS4ERR_BADSESSION is returned for v4.1 requests and NFS4ERR_BADHANDLE for v4.0 requests.

The fix is to implement the v4.0 SETCLIENTID callback_ident and use it to find the one and only client for v4.0 callbacks, and to associate the sessionid with the sessions based callback service (v4.1) and use it to identify the one and only client. This can be done in the NFS layer, in CB_COMPOUND header processing for v4.0 and in CB_SEQUENCE processing in v4.1.
In both cases, a reference to the found client is held across CB_COMPOUND processing.

The pg_authenticate method does not have the callback_ident for CB_NULL or CB_COMPOUND v4.0 processing, so just the server address, nfsversion and minorversion is used for the client search

For sessions based callback service, the sessionID can't be set until the return of the CREATE_SESSION call, yet the CB_NULL ping from a server can be initiated by the server at the receipt of the CREATE_SESSION call before the response is sent. So for CB_NULL, the sessionid is (usually) not set, and the sessionid is not used for CB_NULL pg_authenticate client searches.

0001-NFS-dont-free-shared-socket-on-backchannel-put-xprt.patch 0002-NFS-implement-v4.0-callback_ident.patch 0003-NFS-associate-sessionid-with-callback-connection.patch 0004-NFS-reference-nfs_client-across-cb_compound-processi.patch 0005-NFS-RPC_AUTH_GSS-unsupported-on-v4.1-back-channel.patch 0006-NFS-add-session-back-channel-draining.patch 0007-NFS-only-set-bc_xprt-upon-success.patch

-->Andy

#1 lore raw

PATCH 1/7 NFS dont free shared socket on backchannel put xprt

Fixes this bug:
fedora-64 kernel: Invoking bc_svc_procass()
fedora-64 kernel: nfs_callback_authenticate SVC_DROP
fedora-64 kernel: BUG: unable to handle kernel NULL pointer dereference at 0000000000000018 IP: [<ffffffffa0156140>] svc_sock_free+0x32/0x56 [sunrpc]

---
 fs/nfs/callback.c               |    3 +++
 include/linux/sunrpc/svc_xprt.h |    1 +
 net/sunrpc/svc_xprt.c           |    3 ++-
 3 files changed, 6 insertions(+), 1 deletions(-)
fs/nfs/callback.c +3 -0
index 93a8b3b..023a9eb 100644
@@ -193,6 +193,9 @@ nfs41_callback_up(struct svc_serv *serv, struct rpc_xprt *xprt)
 	serv->bc_xprt = bc_xprt;
 	xprt->bc_serv = serv;
 
+	/* socket is shared with the fore channel */
+	set_bit(XPT_SHARE_SOCK, &bc_xprt->xpt_flags);
+
 	INIT_LIST_HEAD(&serv->sv_cb_list);
 	spin_lock_init(&serv->sv_cb_lock);
 	init_waitqueue_head(&serv->sv_cb_waitq);
include/linux/sunrpc/svc_xprt.h +1 -0
index aea0d43..600c669 100644
@@ -62,6 +62,7 @@ struct svc_xprt {
 #define	XPT_DETACHED	10		/* detached from tempsocks list */
 #define XPT_LISTENER	11		/* listening endpoint */
 #define XPT_CACHE_AUTH	12		/* cache auth info */
+#define XPT_SHARE_SOCK	13		/* fore and back channel share socket */
 
 	struct svc_pool		*xpt_pool;	/* current pool iff queued */
 	struct svc_serv		*xpt_server;	/* service for transport */
net/sunrpc/svc_xprt.c +2 -1
index ea2ff78..8c4d9ad 100644
@@ -128,7 +128,8 @@ static void svc_xprt_free(struct kref *kref)
 	if (test_bit(XPT_CACHE_AUTH, &xprt->xpt_flags))
 		svcauth_unix_info_release(xprt);
 	put_net(xprt->xpt_net);
-	xprt->xpt_ops->xpo_free(xprt);
+	if (!test_bit(XPT_SHARE_SOCK, &xprt->xpt_flags))
+		xprt->xpt_ops->xpo_free(xprt);
 	module_put(owner);
 }
 
Signature · 1 line
1.6.6
#2 lore raw

PATCH 2/7 NFS implement v4.0 callback_ident

Provide a unique callback identifier per SETCLIENTID call used to identify the v4.0 callback service associated with the clientid.

Do not worry about wrap around. Zero value means unset.

---
 fs/nfs/nfs4proc.c         |    5 +++++
 fs/nfs/nfs4state.c        |    1 +
 include/linux/nfs_fs_sb.h |    1 +
 include/linux/nfs_xdr.h   |    1 +
 4 files changed, 8 insertions(+), 0 deletions(-)
fs/nfs/nfs4proc.c +5 -0
index 4435e5e..ab5a2c4 100644
@@ -3470,6 +3470,8 @@ do_state_recovery:
 	return -EAGAIN;
 }
 
+static u32 current_cb_ident = 1;
+
 int nfs4_proc_setclientid(struct nfs_client *clp, u32 program,
 		unsigned short port, struct rpc_cred *cred,
 		struct nfs4_setclientid_res *res)
@@ -3478,6 +3480,7 @@ int nfs4_proc_setclientid(struct nfs_client *clp, u32 program,
 	struct nfs4_setclientid setclientid = {
 		.sc_verifier = &sc_verifier,
 		.sc_prog = program,
+		.sc_cb_ident = current_cb_ident++,
 	};
 	struct rpc_message msg = {
 		.rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_SETCLIENTID],
@@ -3522,6 +3525,8 @@ int nfs4_proc_setclientid(struct nfs_client *clp, u32 program,
 			if (++clp->cl_id_uniquifier == 0)
 				break;
 	}
+	if (status == NFS_OK)
+		res->cb_ident = setclientid.sc_cb_ident;
 	return status;
 }
 
fs/nfs/nfs4state.c +1 -0
index f575a31..fe61422 100644
@@ -79,6 +79,7 @@ int nfs4_init_clientid(struct nfs_client *clp, struct rpc_cred *cred)
 	if (status != 0)
 		goto out;
 	clp->cl_clientid = clid.clientid;
+	clp->cl_cb_ident = clid.cb_ident;
 	nfs4_schedule_state_renewal(clp);
 out:
 	return status;
include/linux/nfs_fs_sb.h +1 -0
index 452d964..1eaa054 100644
@@ -71,6 +71,7 @@ struct nfs_client {
 	 */
 	char			cl_ipaddr[48];
 	unsigned char		cl_id_uniquifier;
+	u32			cl_cb_ident;	/* v4.0 callback identifier */
 	const struct nfs4_minor_version_ops *cl_mvops;
 #endif /* CONFIG_NFS_V4 */
 
include/linux/nfs_xdr.h +1 -0
index 80f0719..24e77a6 100644
@@ -868,6 +868,7 @@ struct nfs4_setclientid {
 struct nfs4_setclientid_res {
 	u64				clientid;
 	nfs4_verifier			confirm;
+	u32				cb_ident;
 };
 
 struct nfs4_statfs_arg {
Signature · 1 line
1.6.6
#3 lore raw

PATCH 3/7 NFS associate sessionid with callback connection

The sessions based callback service is started prior to the CREATE_SESSION call so that it can handle CB_NULL requests which can be sent before the CREATE_SESSION call returns and the session ID is known.

Set the callback sessionid after a sucessful CREATE_SESSION.

---
 fs/nfs/callback.c               |   31 +++++++++++++++++++++++++++++++
 fs/nfs/callback.h               |    1 +
 fs/nfs/nfs4state.c              |    6 ++++++
 include/linux/sunrpc/svc_xprt.h |    1 +
 net/sunrpc/xprt.c               |    2 ++
 5 files changed, 41 insertions(+), 0 deletions(-)
fs/nfs/callback.c +31 -0
index 023a9eb..558d858 100644
@@ -137,6 +137,33 @@ out_err:
 
 #if defined(CONFIG_NFS_V4_1)
 /*
+ *  * CB_SEQUENCE operations will fail until the callback sessionid is set.
+ *   */
+int nfs4_set_callback_sessionid(struct nfs_client *clp)
+{
+	struct svc_serv *serv = clp->cl_rpcclient->cl_xprt->bc_serv;
+	struct nfs4_sessionid *bc_sid;
+
+	if (!serv->bc_xprt)
+		return -EINVAL;
+
+	/* on success freed in xprt_free */
+	bc_sid = kmalloc(sizeof(struct nfs4_sessionid), GFP_KERNEL);
+	if (!bc_sid)
+		return -ENOMEM;
+	memcpy(bc_sid->data, &clp->cl_session->sess_id.data,
+		NFS4_MAX_SESSIONID_LEN);
+	spin_lock(&serv->sv_cb_lock);
+	serv->bc_xprt->xpt_bc_sid = bc_sid;
+	spin_unlock(&serv->sv_cb_lock);
+	dprintk("%s set xpt_bc_sid=%u:%u:%u:%u for bc_xprt %p\n", __func__,
+		((u32 *)bc_sid->data)[0], ((u32 *)bc_sid->data)[1],
+		((u32 *)bc_sid->data)[2], ((u32 *)bc_sid->data)[3],
+		serv->bc_xprt);
+	return 0;
+}
+
+/*
  * The callback service for NFSv4.1 callbacks
  */
 static int
@@ -236,6 +263,10 @@ static inline void nfs_callback_bc_serv(u32 minorversion, struct rpc_xprt *xprt,
 		struct nfs_callback_data *cb_info)
 {
 }
+int nfs4_set_callback_sessionid(struct nfs_client *clp)
+{
+	return 0;
+}
 #endif /* CONFIG_NFS_V4_1 */
 
 /*
fs/nfs/callback.h +1 -0
index 85a7cfd..58d61a8 100644
@@ -137,6 +137,7 @@ extern int nfs_callback_up(u32 minorversion, struct rpc_xprt *xprt);
 extern void nfs_callback_down(int minorversion);
 extern int nfs4_validate_delegation_stateid(struct nfs_delegation *delegation,
 					    const nfs4_stateid *stateid);
+extern int nfs4_set_callback_sessionid(struct nfs_client *clp);
 #endif /* CONFIG_NFS_V4 */
 /*
  * nfs41: Callbacks are expected to not cause substantial latency,
fs/nfs/nfs4state.c +6 -0
index fe61422..11290de 100644
@@ -193,6 +193,12 @@ int nfs41_init_clientid(struct nfs_client *clp, struct rpc_cred *cred)
 	status = nfs4_proc_create_session(clp);
 	if (status != 0)
 		goto out;
+	status = nfs4_set_callback_sessionid(clp);
+	if (status != 0) {
+		printk(KERN_WARNING "Sessionid not set. No callback service\n");
+		nfs_callback_down(1);
+		status = 0;
+	}
 	nfs41_setup_state_renewal(clp);
 	nfs_mark_client_ready(clp, NFS_CS_READY);
 out:
include/linux/sunrpc/svc_xprt.h +1 -0
index 600c669..aa15126 100644
@@ -79,6 +79,7 @@ struct svc_xprt {
 	size_t			xpt_remotelen;	/* length of address */
 	struct rpc_wait_queue	xpt_bc_pending;	/* backchannel wait queue */
 	struct list_head	xpt_users;	/* callbacks on free */
+	void			*xpt_bc_sid;	/* back channel session ID */
 
 	struct net		*xpt_net;
 };
net/sunrpc/xprt.c +2 -0
index 4c8f18a..18783cb 100644
@@ -985,6 +985,8 @@ void xprt_free(struct rpc_xprt *xprt)
 {
 	put_net(xprt->xprt_net);
 	kfree(xprt->slot);
+	if (xprt->bc_xprt)
+		kfree(xprt->bc_xprt->xpt_bc_sid);
 	kfree(xprt);
 }
 EXPORT_SYMBOL_GPL(xprt_free);
Signature · 1 line
1.6.6
#4 lore raw

PATCH 4/7 NFS reference nfs_client across cb_compound processing

In the NFS layer, V4.0 clients are found using the callback_ident field in the CB_COMPOUND header. V4.1 clients are found using the sessionID in the CB_SEQUENCE operation which is also compared against the sessionID assigned at back channel thread creation.

Each of these methods finds the one an only nfs_client associated with the incoming callback request - so nfs_find_client_next is not needed.

Pass the referenced nfs_client to each CB_COMPOUND operation being proceesed via the new cb_process_state structure.

In the RPC layer, the v4.0 callback identifier is not known for the pg_authenticate call, so a zero value (unset) is used to find the nfs_client. The sessionid for the sessions based callback service has (usually) not been set for the pg_authenticate on a CB_NULL call, so the sessionid is not used to find the client in pg_authenticate for CB_NULL calls. It is used for pg_authenticate CB_COMPOUND calls.

Use the new cb_process_state struct to move the NFS4ERR_RETRY_UNCACHED_REP processing from process_op into nfs4_callback_sequence where it belongs.

---
 fs/nfs/callback.c              |   21 ++++-
 fs/nfs/callback.h              |   24 +++++-
 fs/nfs/callback_proc.c         |  167 ++++++++++++++++------------------------
 fs/nfs/callback_xdr.c          |   40 ++++++----
 fs/nfs/client.c                |   97 ++++++++++++++++-------
 fs/nfs/internal.h              |    7 +-
 include/linux/sunrpc/bc_xprt.h |   13 +++
 7 files changed, 214 insertions(+), 155 deletions(-)
fs/nfs/callback.c +18 -3
index 558d858..7c21824 100644
@@ -16,9 +16,7 @@
 #include <linux/freezer.h>
 #include <linux/kthread.h>
 #include <linux/sunrpc/svcauth_gss.h>
-#if defined(CONFIG_NFS_V4_1)
 #include <linux/sunrpc/bc_xprt.h>
-#endif
 
 #include <net/inet_sock.h>
 
@@ -379,6 +377,23 @@ static int check_gss_callback_principal(struct nfs_client *clp,
 	return SVC_OK;
 }
 
+/* pg_authenticate method helper */
+static struct nfs_client *nfs_cb_find_client(struct svc_rqst *rqstp)
+{
+	struct nfs4_sessionid *sessionid = bc_xprt_sid(rqstp);
+	int is_cb_compound = rqstp->rq_proc == CB_COMPOUND ? 1 : 0;
+
+	dprintk("--> %s rq_proc %d\n", __func__, rqstp->rq_proc);
+	if (svc_is_backchannel(rqstp))
+		/* Sessionid (usually) set after CB_NULL ping */
+		return nfs4_find_client_sessionid(svc_addr(rqstp), sessionid,
+						  is_cb_compound);
+	else
+		/* No callback identifier in pg_authenticate */
+		return nfs4_find_client_ident(svc_addr(rqstp), 0);
+}
+
+/* pg_authenticate method for nfsv4 callback threads. */
 static int nfs_callback_authenticate(struct svc_rqst *rqstp)
 {
 	struct nfs_client *clp;
@@ -386,7 +401,7 @@ static int nfs_callback_authenticate(struct svc_rqst *rqstp)
 	int ret = SVC_OK;
 
 	/* Don't talk to strangers */
-	clp = nfs_find_client(svc_addr(rqstp), 4);
+	clp = nfs_cb_find_client(rqstp);
 	if (clp == NULL)
 		return SVC_DROP;
 
fs/nfs/callback.h +19 -5
index 58d61a8..d700d47 100644
@@ -34,10 +34,17 @@ enum nfs4_callback_opnum {
 	OP_CB_ILLEGAL = 10044,
 };
 
+struct cb_process_state {
+	__be32			drc_status;
+	struct nfs_client	*clp;
+	struct nfs4_sessionid	*svc_sid; /* v4.1 callback service sessionid */
+};
+
 struct cb_compound_hdr_arg {
 	unsigned int taglen;
 	const char *tag;
 	unsigned int minorversion;
+	unsigned int cb_ident; /* v4.0 callback identifier */
 	unsigned nops;
 };
 
@@ -104,7 +111,8 @@ struct cb_sequenceres {
 };
 
 extern unsigned nfs4_callback_sequence(struct cb_sequenceargs *args,
-				       struct cb_sequenceres *res);
+				       struct cb_sequenceres *res,
+				       struct cb_process_state *cps);
 
 extern int nfs41_validate_delegation_stateid(struct nfs_delegation *delegation,
 					     const nfs4_stateid *stateid);
@@ -118,19 +126,25 @@ struct cb_recallanyargs {
 	uint32_t	craa_type_mask;
 };
 
-extern unsigned nfs4_callback_recallany(struct cb_recallanyargs *args, void *dummy);
+extern unsigned nfs4_callback_recallany(struct cb_recallanyargs *args,
+					void *dummy,
+					struct cb_process_state *cps);
 
 struct cb_recallslotargs {
 	struct sockaddr	*crsa_addr;
 	uint32_t	crsa_target_max_slots;
 };
 extern unsigned nfs4_callback_recallslot(struct cb_recallslotargs *args,
-					  void *dummy);
+					 void *dummy,
+					 struct cb_process_state *cps);
 
 #endif /* CONFIG_NFS_V4_1 */
 
-extern __be32 nfs4_callback_getattr(struct cb_getattrargs *args, struct cb_getattrres *res);
-extern __be32 nfs4_callback_recall(struct cb_recallargs *args, void *dummy);
+extern __be32 nfs4_callback_getattr(struct cb_getattrargs *args,
+				    struct cb_getattrres *res,
+				    struct cb_process_state *cps);
+extern __be32 nfs4_callback_recall(struct cb_recallargs *args, void *dummy,
+				   struct cb_process_state *cps);
 
 #ifdef CONFIG_NFS_V4
 extern int nfs_callback_up(u32 minorversion, struct rpc_xprt *xprt);
fs/nfs/callback_proc.c +65 -102

288 lines not on this page · raw message

fs/nfs/callback_xdr.c +25 -15
index 05af212..85cbb8f 100644
@@ -10,8 +10,10 @@
 #include <linux/nfs4.h>
 #include <linux/nfs_fs.h>
 #include <linux/slab.h>
+#include <linux/sunrpc/bc_xprt.h>
 #include "nfs4_fs.h"
 #include "callback.h"
+#include "internal.h"
 
 #define CB_OP_TAGLEN_MAXSZ	(512)
 #define CB_OP_HDR_RES_MAXSZ	(2 + CB_OP_TAGLEN_MAXSZ)
@@ -33,7 +35,8 @@
 /* Internal error code */
 #define NFS4ERR_RESOURCE_HDR	11050
 
-typedef __be32 (*callback_process_op_t)(void *, void *);
+typedef __be32 (*callback_process_op_t)(void *, void *,
+					struct cb_process_state *);
 typedef __be32 (*callback_decode_arg_t)(struct svc_rqst *, struct xdr_stream *, void *);
 typedef __be32 (*callback_encode_res_t)(struct svc_rqst *, struct xdr_stream *, void *);
 
@@ -160,7 +163,7 @@ static __be32 decode_compound_hdr_arg(struct xdr_stream *xdr, struct cb_compound
 	hdr->minorversion = ntohl(*p++);
 	/* Check minor version is zero or one. */
 	if (hdr->minorversion <= 1) {
-		p++;	/* skip callback_ident */
+		hdr->cb_ident = ntohl(*p++); /* ignored by v4.1 */
 	} else {
 		printk(KERN_WARNING "%s: NFSv4 server callback with "
 			"illegal minor version %u!\n",
@@ -621,7 +624,8 @@ preprocess_nfs4_op(unsigned int op_nr, struct callback_op **op)
 static __be32 process_op(uint32_t minorversion, int nop,
 		struct svc_rqst *rqstp,
 		struct xdr_stream *xdr_in, void *argp,
-		struct xdr_stream *xdr_out, void *resp, int* drc_status)
+		struct xdr_stream *xdr_out, void *resp,
+		struct cb_process_state *cps)
 {
 	struct callback_op *op = &callback_ops[0];
 	unsigned int op_nr;
@@ -644,8 +648,8 @@ static __be32 process_op(uint32_t minorversion, int nop,
 	if (status)
 		goto encode_hdr;
 
-	if (*drc_status) {
-		status = *drc_status;
+	if (cps->drc_status) {
+		status = cps->drc_status;
 		goto encode_hdr;
 	}
 
@@ -653,16 +657,10 @@ static __be32 process_op(uint32_t minorversion, int nop,
 	if (maxlen > 0 && maxlen < PAGE_SIZE) {
 		status = op->decode_args(rqstp, xdr_in, argp);
 		if (likely(status == 0))
-			status = op->process_op(argp, resp);
+			status = op->process_op(argp, resp, cps);
 	} else
 		status = htonl(NFS4ERR_RESOURCE);
 
-	/* Only set by OP_CB_SEQUENCE processing */
-	if (status == htonl(NFS4ERR_RETRY_UNCACHED_REP)) {
-		*drc_status = status;
-		status = 0;
-	}
-
 encode_hdr:
 	res = encode_op_hdr(xdr_out, op_nr, status);
 	if (unlikely(res))
@@ -681,8 +679,11 @@ static __be32 nfs4_callback_compound(struct svc_rqst *rqstp, void *argp, void *r
 	struct cb_compound_hdr_arg hdr_arg = { 0 };
 	struct cb_compound_hdr_res hdr_res = { NULL };
 	struct xdr_stream xdr_in, xdr_out;
-	__be32 *p;
-	__be32 status, drc_status = 0;
+	__be32 *p, status;
+	struct cb_process_state cps = {
+		.drc_status = 0,
+		.clp = NULL,
+	};
 	unsigned int nops = 0;
 
 	dprintk("%s: start\n", __func__);
@@ -696,6 +697,14 @@ static __be32 nfs4_callback_compound(struct svc_rqst *rqstp, void *argp, void *r
 	if (status == __constant_htonl(NFS4ERR_RESOURCE))
 		return rpc_garbage_args;
 
+	if (hdr_arg.minorversion == 0) {
+		cps.clp = nfs4_find_client_ident(svc_addr(rqstp),
+						 hdr_arg.cb_ident);
+		if (!cps.clp)
+			return rpc_drop_reply;
+	} else
+		cps.svc_sid = bc_xprt_sid(rqstp);
+
 	hdr_res.taglen = hdr_arg.taglen;
 	hdr_res.tag = hdr_arg.tag;
 	if (encode_compound_hdr_res(&xdr_out, &hdr_res) != 0)
@@ -703,7 +712,7 @@ static __be32 nfs4_callback_compound(struct svc_rqst *rqstp, void *argp, void *r
 
 	while (status == 0 && nops != hdr_arg.nops) {
 		status = process_op(hdr_arg.minorversion, nops, rqstp,
-				    &xdr_in, argp, &xdr_out, resp, &drc_status);
+				    &xdr_in, argp, &xdr_out, resp, &cps);
 		nops++;
 	}
 
@@ -716,6 +725,7 @@ static __be32 nfs4_callback_compound(struct svc_rqst *rqstp, void *argp, void *r
 
 	*hdr_res.status = status;
 	*hdr_res.nops = htonl(nops);
+	nfs_put_client(cps.clp);
 	dprintk("%s: done, status = %u\n", __func__, ntohl(status));
 	return rpc_success;
 }
fs/nfs/client.c +69 -28

133 lines not on this page · raw message

fs/nfs/internal.h +5 -2
index e6356b7..f65602c 100644
@@ -129,8 +129,11 @@ extern void nfs_umount(const struct nfs_mount_request *info);
 extern struct rpc_program nfs_program;
 
 extern void nfs_put_client(struct nfs_client *);
-extern struct nfs_client *nfs_find_client(const struct sockaddr *, u32);
-extern struct nfs_client *nfs_find_client_next(struct nfs_client *);
+extern struct nfs_client *nfs4_find_client_ident(const struct sockaddr *,
+						    u32);
+extern struct nfs_client *
+nfs4_find_client_sessionid(const struct sockaddr *, struct nfs4_sessionid *,
+			   int);
 extern struct nfs_server *nfs_create_server(
 					const struct nfs_parsed_mount_data *,
 					struct nfs_fh *);
include/linux/sunrpc/bc_xprt.h +13 -0
index 7c91260..2c60e09 100644
@@ -47,6 +47,14 @@ static inline int svc_is_backchannel(const struct svc_rqst *rqstp)
 		return 1;
 	return 0;
 }
+static inline struct nfs4_sessionid *bc_xprt_sid(struct svc_rqst *rqstp)
+{
+	if (svc_is_backchannel(rqstp))
+		return (struct nfs4_sessionid *)
+					rqstp->rq_server->bc_xprt->xpt_bc_sid;
+	return NULL;
+}
+
 #else /* CONFIG_NFS_V4_1 */
 static inline int xprt_setup_backchannel(struct rpc_xprt *xprt,
 					 unsigned int min_reqs)
@@ -59,6 +67,11 @@ static inline int svc_is_backchannel(const struct svc_rqst *rqstp)
 	return 0;
 }
 
+static inline struct nfs4_sessionid *bc_xprt_sid(struct svc_rqst *rqstp)
+{
+	return NULL;
+}
+
 static inline void xprt_free_bc_request(struct rpc_rqst *req)
 {
 }
Signature · 1 line
1.6.6
#5 lore raw

PATCH 5/7 NFS RPC_AUTH_GSS unsupported on v4.1 back channel

---
 fs/nfs/callback.c |    3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)
fs/nfs/callback.c +3 -0
index 7c21824..dc2ead1 100644
@@ -360,6 +360,9 @@ static int check_gss_callback_principal(struct nfs_client *clp,
 	struct rpc_clnt *r = clp->cl_rpcclient;
 	char *p = svc_gss_principal(rqstp);
 
+	/* No RPC_AUTH_GSS on NFSv4.1 back channel yet */
+	if (clp->cl_minorversion != 0)
+		return SVC_DENIED;
 	/*
 	 * It might just be a normal user principal, in which case
 	 * userspace won't bother to tell us the name at all.
Signature · 1 line
1.6.6
#6 lore raw

PATCH 6/7 NFS add session back channel draining

Currently session draining only drains the fore channel. The back channel processing must also be drained.

Use the back channel highest_slot_used to indicate that a callback is being processed by the callback thread. Move the session complete to be per channel.

When the session is draininig, wait for any current back channel processing to complete and stop all new back channel processing by returning NFS4ERR_DELAY to the back channel client.

Drain the back channel, then the fore channel.

---
 fs/nfs/callback.h         |    2 +-
 fs/nfs/callback_proc.c    |    6 ++++++
 fs/nfs/callback_xdr.c     |   24 ++++++++++++++++++++++++
 fs/nfs/nfs4proc.c         |   26 +++++++++++++++++++-------
 fs/nfs/nfs4state.c        |   29 ++++++++++++++++++++++-------
 include/linux/nfs_fs_sb.h |    2 +-
 6 files changed, 73 insertions(+), 16 deletions(-)
fs/nfs/callback.h +1 -1
index d700d47..2d927d4 100644
@@ -138,6 +138,7 @@ extern unsigned nfs4_callback_recallslot(struct cb_recallslotargs *args,
 					 void *dummy,
 					 struct cb_process_state *cps);
 
+extern void nfs4_check_drain_bc_complete(struct nfs4_session *ses);
 #endif /* CONFIG_NFS_V4_1 */
 
 extern __be32 nfs4_callback_getattr(struct cb_getattrargs *args,
@@ -145,7 +146,6 @@ extern __be32 nfs4_callback_getattr(struct cb_getattrargs *args,
 				    struct cb_process_state *cps);
 extern __be32 nfs4_callback_recall(struct cb_recallargs *args, void *dummy,
 				   struct cb_process_state *cps);
-
 #ifdef CONFIG_NFS_V4
 extern int nfs_callback_up(u32 minorversion, struct rpc_xprt *xprt);
 extern void nfs_callback_down(int minorversion);
fs/nfs/callback_proc.c +6 -0
index b70e46d..a6af836 100644
@@ -253,6 +253,12 @@ __be32 nfs4_callback_sequence(struct cb_sequenceargs *args,
 	if (clp == NULL)
 		goto out;
 
+	/* state manager is resetting the session */
+	if (test_bit(NFS4_SESSION_DRAINING, &clp->cl_session->session_state)) {
+		status = NFS4ERR_DELAY;
+		goto out;
+	}
+
 	status = validate_seqid(&clp->cl_session->bc_slot_table, args);
 	if (status)
 		goto out;
fs/nfs/callback_xdr.c +24 -0
index 85cbb8f..09ca7c4 100644
@@ -596,6 +596,26 @@ preprocess_nfs41_op(int nop, unsigned int op_nr, struct callback_op **op)
 	return htonl(NFS_OK);
 }
 
+static void nfs4_callback_free_slot(struct nfs4_session *session)
+{
+	struct nfs4_slot_table *tbl = &session->bc_slot_table;
+
+	spin_lock(&tbl->slot_tbl_lock);
+	/*
+	 * Let the state manager know callback processing done.
+	 * A single slot, so highest used slotid is either 0 or -1
+	 */
+	tbl->highest_used_slotid--;
+	nfs4_check_drain_bc_complete(session);
+	spin_unlock(&tbl->slot_tbl_lock);
+}
+
+static void nfs4_cb_free_slot(struct nfs_client *clp)
+{
+	if (clp && clp->cl_session)
+		nfs4_callback_free_slot(clp->cl_session);
+}
+
 #else /* CONFIG_NFS_V4_1 */
 
 static __be32
@@ -604,6 +624,9 @@ preprocess_nfs41_op(int nop, unsigned int op_nr, struct callback_op **op)
 	return htonl(NFS4ERR_MINOR_VERS_MISMATCH);
 }
 
+static void nfs4_cb_free_slot(struct nfs_client *clp)
+{
+}
 #endif /* CONFIG_NFS_V4_1 */
 
 static __be32
@@ -725,6 +748,7 @@ static __be32 nfs4_callback_compound(struct svc_rqst *rqstp, void *argp, void *r
 
 	*hdr_res.status = status;
 	*hdr_res.nops = htonl(nops);
+	nfs4_cb_free_slot(cps.clp);
 	nfs_put_client(cps.clp);
 	dprintk("%s: done, status = %u\n", __func__, ntohl(status));
 	return rpc_success;
fs/nfs/nfs4proc.c +19 -7
index ab5a2c4..ecb2a5e 100644
@@ -355,9 +355,9 @@ nfs4_free_slot(struct nfs4_slot_table *tbl, struct nfs4_slot *free_slot)
 }
 
 /*
- * Signal state manager thread if session is drained
+ * Signal state manager thread if session fore channel is drained
  */
-static void nfs41_check_drain_session_complete(struct nfs4_session *ses)
+static void nfs4_check_drain_fc_complete(struct nfs4_session *ses)
 {
 	struct rpc_task *task;
 
@@ -371,8 +371,20 @@ static void nfs41_check_drain_session_complete(struct nfs4_session *ses)
 	if (ses->fc_slot_table.highest_used_slotid != -1)
 		return;
 
-	dprintk("%s COMPLETE: Session Drained\n", __func__);
-	complete(&ses->complete);
+	dprintk("%s COMPLETE: Session Fore Channel Drained\n", __func__);
+	complete(&ses->fc_slot_table.complete);
+}
+
+/*
+ * Signal state manager thread if session back channel is drained
+ */
+void nfs4_check_drain_bc_complete(struct nfs4_session *ses)
+{
+	if (!test_bit(NFS4_SESSION_DRAINING, &ses->session_state) ||
+	    ses->bc_slot_table.highest_used_slotid != -1)
+		return;
+	dprintk("%s COMPLETE: Session Back Channel Drained\n", __func__);
+	complete(&ses->bc_slot_table.complete);
 }
 
 static void nfs41_sequence_free_slot(struct nfs4_sequence_res *res)
@@ -389,7 +401,7 @@ static void nfs41_sequence_free_slot(struct nfs4_sequence_res *res)
 
 	spin_lock(&tbl->slot_tbl_lock);
 	nfs4_free_slot(tbl, res->sr_slot);
-	nfs41_check_drain_session_complete(res->sr_session);
+	nfs4_check_drain_fc_complete(res->sr_session);
 	spin_unlock(&tbl->slot_tbl_lock);
 	res->sr_slot = NULL;
 }
@@ -4781,17 +4793,17 @@ struct nfs4_session *nfs4_alloc_session(struct nfs_client *clp)
 	if (!session)
 		return NULL;
 
-	init_completion(&session->complete);
-
 	tbl = &session->fc_slot_table;
 	tbl->highest_used_slotid = -1;
 	spin_lock_init(&tbl->slot_tbl_lock);
 	rpc_init_priority_wait_queue(&tbl->slot_tbl_waitq, "ForeChannel Slot table");
+	init_completion(&tbl->complete);
 
 	tbl = &session->bc_slot_table;
 	tbl->highest_used_slotid = -1;
 	spin_lock_init(&tbl->slot_tbl_lock);
 	rpc_init_wait_queue(&tbl->slot_tbl_waitq, "BackChannel Slot table");
+	init_completion(&tbl->complete);
 
 	session->session_state = 1<<NFS4_SESSION_INITING;
 
fs/nfs/nfs4state.c +22 -7
index 11290de..fb23a32 100644
@@ -143,6 +143,11 @@ static int nfs41_setup_state_renewal(struct nfs_client *clp)
 	return status;
 }
 
+/*
+ * Back channel returns NFS4ERR_DELAY for new requests when
+ * NFS4_SESSION_DRAINING is set so there is no work to be done when draining
+ * is ended.
+ */
 static void nfs4_end_drain_session(struct nfs_client *clp)
 {
 	struct nfs4_session *ses = clp->cl_session;
@@ -166,22 +171,32 @@ static void nfs4_end_drain_session(struct nfs_client *clp)
 	}
 }
 
-static int nfs4_begin_drain_session(struct nfs_client *clp)
+static int nfs4_wait_on_slot_tbl(struct nfs4_slot_table *tbl)
 {
-	struct nfs4_session *ses = clp->cl_session;
-	struct nfs4_slot_table *tbl = &ses->fc_slot_table;
-
 	spin_lock(&tbl->slot_tbl_lock);
-	set_bit(NFS4_SESSION_DRAINING, &ses->session_state);
 	if (tbl->highest_used_slotid != -1) {
-		INIT_COMPLETION(ses->complete);
+		INIT_COMPLETION(tbl->complete);
 		spin_unlock(&tbl->slot_tbl_lock);
-		return wait_for_completion_interruptible(&ses->complete);
+		return wait_for_completion_interruptible(&tbl->complete);
 	}
 	spin_unlock(&tbl->slot_tbl_lock);
 	return 0;
 }
 
+static int nfs4_begin_drain_session(struct nfs_client *clp)
+{
+	struct nfs4_session *ses = clp->cl_session;
+	int ret = 0;
+
+	set_bit(NFS4_SESSION_DRAINING, &ses->session_state);
+	/* back channel */
+	ret = nfs4_wait_on_slot_tbl(&ses->bc_slot_table);
+	if (ret)
+		return ret;
+	/* fore channel */
+	return nfs4_wait_on_slot_tbl(&ses->fc_slot_table);
+}
+
 int nfs41_init_clientid(struct nfs_client *clp, struct rpc_cred *cred)
 {
 	int status;
include/linux/nfs_fs_sb.h +1 -1
index 1eaa054..e93ada0 100644
@@ -197,6 +197,7 @@ struct nfs4_slot_table {
 						 * op for dynamic resizing */
 	int		target_max_slots;	/* Set by CB_RECALL_SLOT as
 						 * the new max_slots */
+	struct completion complete;
 };
 
 static inline int slot_idx(struct nfs4_slot_table *tbl, struct nfs4_slot *sp)
@@ -213,7 +214,6 @@ struct nfs4_session {
 	unsigned long			session_state;
 	u32				hash_alg;
 	u32				ssv_len;
-	struct completion		complete;
 
 	/* The fore and back channel */
 	struct nfs4_channel_attrs	fc_attrs;
Signature · 1 line
1.6.6
#7 lore raw

PATCH 7/7 NFS only set bc_xprt upon success

Avoid a double svc_sock_destroy.
nfs41_callback_up destroys the bc_xprt upon failure. nfs_callback_up calls svc_destroy on failure which also calls svc_sock_destroy on the bc_xprt.

---
 fs/nfs/callback.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)
fs/nfs/callback.c +2 -1
index dc2ead1..8129511 100644
@@ -215,7 +215,6 @@ nfs41_callback_up(struct svc_serv *serv, struct rpc_xprt *xprt)
 	 * Save the svc_serv in the transport so that it can
 	 * be referenced when the session backchannel is initialized
 	 */
-	serv->bc_xprt = bc_xprt;
 	xprt->bc_serv = serv;
 
 	/* socket is shared with the fore channel */
@@ -227,6 +226,8 @@ nfs41_callback_up(struct svc_serv *serv, struct rpc_xprt *xprt)
 	rqstp = svc_prepare_thread(serv, &serv->sv_pools[0]);
 	if (IS_ERR(rqstp))
 		svc_sock_destroy(bc_xprt);
+	else
+		serv->bc_xprt = bc_xprt;
 out:
 	dprintk("--> %s return %p\n", __func__, rqstp);
 	return rqstp;
Signature · 1 line
1.6.6
#8 lore raw

Re: [PATCH_V3 3/7] NFS associate sessionid with callback connection

In reply to Andy Adamson
Quoted 110 lines · Andy Adamson

The sessions based callback service is started prior to the CREATE_SESSION call so that it can handle CB_NULL requests which can be sent before the CREATE_SESSION call returns and the session ID is known.

Set the callback sessionid after a sucessful CREATE_SESSION.

---
 fs/nfs/callback.c               |   31 +++++++++++++++++++++++++++++++
 fs/nfs/callback.h               |    1 +
 fs/nfs/nfs4state.c              |    6 ++++++
 include/linux/sunrpc/svc_xprt.h |    1 +
 net/sunrpc/xprt.c               |    2 ++
 5 files changed, 41 insertions(+), 0 deletions(-)
fs/nfs/callback.c +31 -0
index 023a9eb..558d858 100644
@@ -137,6 +137,33 @@ out_err:
 
 #if defined(CONFIG_NFS_V4_1)
 /*
+ *  * CB_SEQUENCE operations will fail until the callback sessionid is set.
+ *   */
+int nfs4_set_callback_sessionid(struct nfs_client *clp)
+{
+	struct svc_serv *serv = clp->cl_rpcclient->cl_xprt->bc_serv;
+	struct nfs4_sessionid *bc_sid;
+
+	if (!serv->bc_xprt)
+		return -EINVAL;
+
+	/* on success freed in xprt_free */
+	bc_sid = kmalloc(sizeof(struct nfs4_sessionid), GFP_KERNEL);
+	if (!bc_sid)
+		return -ENOMEM;
+	memcpy(bc_sid->data, &clp->cl_session->sess_id.data,
+		NFS4_MAX_SESSIONID_LEN);
+	spin_lock(&serv->sv_cb_lock);
+	serv->bc_xprt->xpt_bc_sid = bc_sid;
+	spin_unlock(&serv->sv_cb_lock);
+	dprintk("%s set xpt_bc_sid=%u:%u:%u:%u for bc_xprt %p\n", __func__,
+		((u32 *)bc_sid->data)[0], ((u32 *)bc_sid->data)[1],
+		((u32 *)bc_sid->data)[2], ((u32 *)bc_sid->data)[3],
+		serv->bc_xprt);
+	return 0;
+}
+
+/*
  * The callback service for NFSv4.1 callbacks
  */
 static int
@@ -236,6 +263,10 @@ static inline void nfs_callback_bc_serv(u32 minorversion, struct rpc_xprt *xprt,
 		struct nfs_callback_data *cb_info)
 {
 }
+int nfs4_set_callback_sessionid(struct nfs_client *clp)
+{
+	return 0;
+}
 #endif /* CONFIG_NFS_V4_1 */
 
 /*
fs/nfs/callback.h +1 -0
index 85a7cfd..58d61a8 100644
@@ -137,6 +137,7 @@ extern int nfs_callback_up(u32 minorversion, struct rpc_xprt *xprt);
 extern void nfs_callback_down(int minorversion);
 extern int nfs4_validate_delegation_stateid(struct nfs_delegation *delegation,
 					    const nfs4_stateid *stateid);
+extern int nfs4_set_callback_sessionid(struct nfs_client *clp);
 #endif /* CONFIG_NFS_V4 */
 /*
  * nfs41: Callbacks are expected to not cause substantial latency,
fs/nfs/nfs4state.c +6 -0
index fe61422..11290de 100644
@@ -193,6 +193,12 @@ int nfs41_init_clientid(struct nfs_client *clp, struct rpc_cred *cred)
 	status = nfs4_proc_create_session(clp);
 	if (status != 0)
 		goto out;
+	status = nfs4_set_callback_sessionid(clp);
+	if (status != 0) {
+		printk(KERN_WARNING "Sessionid not set. No callback service\n");
+		nfs_callback_down(1);
+		status = 0;
+	}
 	nfs41_setup_state_renewal(clp);
 	nfs_mark_client_ready(clp, NFS_CS_READY);
 out:
include/linux/sunrpc/svc_xprt.h +1 -0
index 600c669..aa15126 100644
@@ -79,6 +79,7 @@ struct svc_xprt {
 	size_t			xpt_remotelen;	/* length of address */
 	struct rpc_wait_queue	xpt_bc_pending;	/* backchannel wait queue */
 	struct list_head	xpt_users;	/* callbacks on free */
+	void			*xpt_bc_sid;	/* back channel session ID */
 
 	struct net		*xpt_net;
 };
net/sunrpc/xprt.c +0 -0
index 4c8f18a..18783cb 100644

net/sunrpc/xprt.c void xprt_free(struct rpc_xprt *xprt) · line 989

Quoted 5 lines
net/sunrpc/xprt.c +2 -0
@@ -985,6 +985,8 @@ void xprt_free(struct rpc_xprt *xprt)
 {
 	put_net(xprt->xprt_net);
 	kfree(xprt->slot);
+	if (xprt->bc_xprt)
+		kfree(xprt->bc_xprt->xpt_bc_sid);

If bc_xprt points to an svc_xprt, shouldn't you be waiting till you free that svc_xprt before freeing anything it points to?

The idea of leaving around that bc_xprt with its xpt_bc_sid field now pointing at freed memory makes me nervous....

--b.

Quoted 10 lines
net/sunrpc/xprt.c +0 -0
 	kfree(xprt);
 }
 EXPORT_SYMBOL_GPL(xprt_free);
Signature · 6 lines
1.6.6

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#9 lore raw

Re: [PATCH_V3 3/7] NFS associate sessionid with callback connection

Quoted 125 lines · J. Bruce Fields
Quoted 115 lines · Andy Adamson

The sessions based callback service is started prior to the CREATE_SESSION call so that it can handle CB_NULL requests which can be sent before the CREATE_SESSION call returns and the session ID is known.

Set the callback sessionid after a sucessful CREATE_SESSION.

---
fs/nfs/callback.c | 31 +++++++++++++++++++++++++++++
++

fs/nfs/callback.h               |    1 +
fs/nfs/nfs4state.c              |    6 ++++++
include/linux/sunrpc/svc_xprt.h |    1 +
net/sunrpc/xprt.c               |    2 ++
5 files changed, 41 insertions(+), 0 deletions(-)
fs/nfs/callback.c +1 -0
index 023a9eb..558d858 100644
@@ -137,6 +137,33 @@ out_err:
 
 #if defined(CONFIG_NFS_V4_1)
 /*
+ *  * CB_SEQUENCE operations will fail until the callback  

sessionid is set.
+ * */

fs/nfs/callback.c +25 -0
+int nfs4_set_callback_sessionid(struct nfs_client *clp)
+{
+	struct svc_serv *serv = clp->cl_rpcclient->cl_xprt->bc_serv;
+	struct nfs4_sessionid *bc_sid;
+
+	if (!serv->bc_xprt)
+		return -EINVAL;
+
+	/* on success freed in xprt_free */
+	bc_sid = kmalloc(sizeof(struct nfs4_sessionid), GFP_KERNEL);
+	if (!bc_sid)
+		return -ENOMEM;
+	memcpy(bc_sid->data, &clp->cl_session->sess_id.data,
+		NFS4_MAX_SESSIONID_LEN);
+	spin_lock(&serv->sv_cb_lock);
+	serv->bc_xprt->xpt_bc_sid = bc_sid;
+	spin_unlock(&serv->sv_cb_lock);
+	dprintk("%s set xpt_bc_sid=%u:%u:%u:%u for bc_xprt %p\n", __func__,
+		((u32 *)bc_sid->data)[0], ((u32 *)bc_sid->data)[1],
+		((u32 *)bc_sid->data)[2], ((u32 *)bc_sid->data)[3],
+		serv->bc_xprt);
+	return 0;
+}
+
+/*
 * The callback service for NFSv4.1 callbacks
 */

static int

fs/nfs/callback.c +4 -0
@@ -236,6 +263,10 @@ static inline void nfs_callback_bc_serv(u32  
 minorversion, struct rpc_xprt *xprt,
 		struct nfs_callback_data *cb_info)
 {
 }
+int nfs4_set_callback_sessionid(struct nfs_client *clp)
+{
+	return 0;
+}
 #endif /* CONFIG_NFS_V4_1 */
 

/*

fs/nfs/callback.h +1 -0
index 85a7cfd..58d61a8 100644
@@ -137,6 +137,7 @@ extern int nfs_callback_up(u32 minorversion,  
 struct rpc_xprt *xprt);
 extern void nfs_callback_down(int minorversion);
 extern int nfs4_validate_delegation_stateid(struct nfs_delegation *delegation,
 					    const nfs4_stateid *stateid);
+extern int nfs4_set_callback_sessionid(struct nfs_client *clp);
 #endif /* CONFIG_NFS_V4 */
 /*

* nfs41: Callbacks are expected to not cause substantial latency,

fs/nfs/nfs4state.c +6 -0
index fe61422..11290de 100644
@@ -193,6 +193,12 @@ int nfs41_init_clientid(struct nfs_client  
 *clp, struct rpc_cred *cred)
 	status = nfs4_proc_create_session(clp);
 	if (status != 0)
 		goto out;
+	status = nfs4_set_callback_sessionid(clp);
+	if (status != 0) {
+		printk(KERN_WARNING "Sessionid not set. No callback service\n");
+		nfs_callback_down(1);
+		status = 0;
+	}
 	nfs41_setup_state_renewal(clp);
 	nfs_mark_client_ready(clp, NFS_CS_READY);

out:

include/linux/sunrpc/svc_xprt.h +1 -0
index 600c669..aa15126 100644
@@ -79,6 +79,7 @@ struct svc_xprt {
 	size_t			xpt_remotelen;	/* length of address */
 	struct rpc_wait_queue	xpt_bc_pending;	/* backchannel wait queue */
 	struct list_head	xpt_users;	/* callbacks on free */
+	void			*xpt_bc_sid;	/* back channel session ID */
 
 	struct net		*xpt_net;
 };
net/sunrpc/xprt.c +0 -0
index 4c8f18a..18783cb 100644

net/sunrpc/xprt.c void xprt_free(struct rpc_xprt *xprt) · line 989

Quoted 5 lines
net/sunrpc/xprt.c +2 -0
@@ -985,6 +985,8 @@ void xprt_free(struct rpc_xprt *xprt)
 {
 	put_net(xprt->xprt_net);
 	kfree(xprt->slot);
+	if (xprt->bc_xprt)
+		kfree(xprt->bc_xprt->xpt_bc_sid);

If bc_xprt points to an svc_xprt, shouldn't you be waiting till you free that svc_xprt before freeing anything it points to?

Quoted 2 lines

The idea of leaving around that bc_xprt with its xpt_bc_sid field now pointing at freed memory makes me nervous....

You are correct - the xpt_bc_sid should be freed in svc_sock_destroy. I'll resend.

Thanks for the review!

-->Andy

Quoted 13 lines

--b.

Quoted 10 lines
net/sunrpc/xprt.c +0 -0
 	kfree(xprt);
 }
 EXPORT_SYMBOL_GPL(xprt_free);
Signature · 6 lines
1.6.6

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#10 lore raw

Re: [PATCH_V3 1/7] NFS dont free shared socket on backchannel put xprt

In reply to Andy Adamson
Quoted 43 lines · Andy Adamson

Fixes this bug:
fedora-64 kernel: Invoking bc_svc_procass()
fedora-64 kernel: nfs_callback_authenticate SVC_DROP
fedora-64 kernel: BUG: unable to handle kernel NULL pointer dereference at 0000000000000018 IP: [<ffffffffa0156140>] svc_sock_free+0x32/0x56 [sunrpc]

---
 fs/nfs/callback.c               |    3 +++
 include/linux/sunrpc/svc_xprt.h |    1 +
 net/sunrpc/svc_xprt.c           |    3 ++-
 3 files changed, 6 insertions(+), 1 deletions(-)
fs/nfs/callback.c +3 -0
index 93a8b3b..023a9eb 100644
@@ -193,6 +193,9 @@ nfs41_callback_up(struct svc_serv *serv, struct rpc_xprt *xprt)
 	serv->bc_xprt = bc_xprt;
 	xprt->bc_serv = serv;
 
+	/* socket is shared with the fore channel */
+	set_bit(XPT_SHARE_SOCK, &bc_xprt->xpt_flags);
+
 	INIT_LIST_HEAD(&serv->sv_cb_list);
 	spin_lock_init(&serv->sv_cb_lock);
 	init_waitqueue_head(&serv->sv_cb_waitq);
include/linux/sunrpc/svc_xprt.h +1 -0
index aea0d43..600c669 100644
@@ -62,6 +62,7 @@ struct svc_xprt {
 #define	XPT_DETACHED	10		/* detached from tempsocks list */
 #define XPT_LISTENER	11		/* listening endpoint */
 #define XPT_CACHE_AUTH	12		/* cache auth info */
+#define XPT_SHARE_SOCK	13		/* fore and back channel share socket */
 
 	struct svc_pool		*xpt_pool;	/* current pool iff queued */
 	struct svc_serv		*xpt_server;	/* service for transport */
net/sunrpc/svc_xprt.c +0 -0
index ea2ff78..8c4d9ad 100644

net/sunrpc/svc_xprt.c static void svc_xprt_free(struct kref *kref) · line 132

Quoted 6 lines
net/sunrpc/svc_xprt.c +2 -1
@@ -128,7 +128,8 @@ static void svc_xprt_free(struct kref *kref)
 	if (test_bit(XPT_CACHE_AUTH, &xprt->xpt_flags))
 		svcauth_unix_info_release(xprt);
 	put_net(xprt->xpt_net);
-	xprt->xpt_ops->xpo_free(xprt);
+	if (!test_bit(XPT_SHARE_SOCK, &xprt->xpt_flags))
+		xprt->xpt_ops->xpo_free(xprt);

So when does the svc_xprt get freed if not here? And why is it OK to do the put_net() and module_put() but not the xpo_free()?

Something feels wrong here....

--b.

Quoted 10 lines
net/sunrpc/svc_xprt.c +0 -0
 	module_put(owner);
 }
 
Signature · 6 lines
1.6.6

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#11 lore raw

Re: [PATCH_V3 1/7] NFS dont free shared socket on backchannel put xprt

Quoted 38 lines · J. Bruce Fields
Quoted 37 lines · Andy Adamson

Fixes this bug:
fedora-64 kernel: Invoking bc_svc_procass()
fedora-64 kernel: nfs_callback_authenticate SVC_DROP
fedora-64 kernel: BUG: unable to handle kernel NULL pointer dereference at 0000000000000018 IP: [<ffffffffa0156140>] svc_sock_free+0x32/0x56 [sunrpc]

---

fs/nfs/callback.c               |    3 +++
include/linux/sunrpc/svc_xprt.h |    1 +
net/sunrpc/svc_xprt.c           |    3 ++-
3 files changed, 6 insertions(+), 1 deletions(-)
fs/nfs/callback.c +3 -0
index 93a8b3b..023a9eb 100644
@@ -193,6 +193,9 @@ nfs41_callback_up(struct svc_serv *serv, struct  
 rpc_xprt *xprt)
 	serv->bc_xprt = bc_xprt;
 	xprt->bc_serv = serv;
 
+	/* socket is shared with the fore channel */
+	set_bit(XPT_SHARE_SOCK, &bc_xprt->xpt_flags);
+
 	INIT_LIST_HEAD(&serv->sv_cb_list);
 	spin_lock_init(&serv->sv_cb_lock);
	init_waitqueue_head(&serv->sv_cb_waitq);
include/linux/sunrpc/svc_xprt.h +1 -0
index aea0d43..600c669 100644
@@ -62,6 +62,7 @@ struct svc_xprt {
 #define	XPT_DETACHED	10		/* detached from tempsocks list */
 #define XPT_LISTENER	11		/* listening endpoint */
 #define XPT_CACHE_AUTH	12		/* cache auth info */
+#define XPT_SHARE_SOCK	13		/* fore and back channel share socket */
 
Quoted 15 lines

net/sunrpc/svc_xprt.c static void svc_xprt_free(struct kref *kref) · line 132

Quoted 13 lines
include/linux/sunrpc/svc_xprt.h +0 -0
 	struct svc_pool		*xpt_pool;	/* current pool iff queued */
 	struct svc_serv		*xpt_server;	/* service for transport */
net/sunrpc/svc_xprt.c +2 -1
index ea2ff78..8c4d9ad 100644
@@ -128,7 +128,8 @@ static void svc_xprt_free(struct kref *kref)
 	if (test_bit(XPT_CACHE_AUTH, &xprt->xpt_flags))
 		svcauth_unix_info_release(xprt);
 	put_net(xprt->xpt_net);
-	xprt->xpt_ops->xpo_free(xprt);
+	if (!test_bit(XPT_SHARE_SOCK, &xprt->xpt_flags))
+		xprt->xpt_ops->xpo_free(xprt);

So when does the svc_xprt get freed if not here?

svc_sock_destroy frees the bc_xprt, called by svc_destroy on the serv->bc_xprt.

Quoted 2 lines

And why is it OK to do
the put_net() and module_put() but not the xpo_free()?

svc_sock_create() called only by nfs41_callback_up creates the bc_xprt. It calls svc_xprt_init which calls get_net(), so put_net is needed.

Hmmm. It looks like the module_put is also not needed - the matching module_get is in svc_create_xprt() which is not called for the NFSv4.1 callback service. (but is called for the NFSv4.0 callback service ... sheese).

I'll move the module_put so it is also not called when the XPT_SHARE_SOCK flag is set.

Quoted 2 lines

Something feels wrong here....

Yeah - how about naming the server back channel xprt (bc_xprt) the same as the client back channel xprt (bc_xprt)!!! How confusing is THAT! :)

Thanks for the review.

-->Andy

Quoted 17 lines

--b.

Quoted 10 lines
net/sunrpc/svc_xprt.c +0 -0
 	module_put(owner);
 }
 
Signature · 6 lines
1.6.6

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#12 lore raw

Re: [PATCH_V3 1/7] NFS dont free shared socket on backchannel put xprt

Quoted 60 lines · Andy Adamson
Quoted 41 lines · J. Bruce Fields
Quoted 40 lines · Andy Adamson

Fixes this bug:
fedora-64 kernel: Invoking bc_svc_procass() fedora-64 kernel: nfs_callback_authenticate SVC_DROP fedora-64 kernel: BUG: unable to handle kernel NULL pointer dereference at 0000000000000018 IP: [<ffffffffa0156140>] svc_sock_free+0x32/0x56 [sunrpc]

---

fs/nfs/callback.c               |    3 +++
include/linux/sunrpc/svc_xprt.h |    1 +
net/sunrpc/svc_xprt.c           |    3 ++-
3 files changed, 6 insertions(+), 1 deletions(-)
fs/nfs/callback.c +3 -0
index 93a8b3b..023a9eb 100644
@@ -193,6 +193,9 @@ nfs41_callback_up(struct svc_serv *serv,
 struct rpc_xprt *xprt)
 	serv->bc_xprt = bc_xprt;
 	xprt->bc_serv = serv;
 
+	/* socket is shared with the fore channel */
+	set_bit(XPT_SHARE_SOCK, &bc_xprt->xpt_flags);
+
 	INIT_LIST_HEAD(&serv->sv_cb_list);
 	spin_lock_init(&serv->sv_cb_lock);
	init_waitqueue_head(&serv->sv_cb_waitq);
include/linux/sunrpc/svc_xprt.h +1 -0
index aea0d43..600c669 100644
@@ -62,6 +62,7 @@ struct svc_xprt {
 #define	XPT_DETACHED	10		/* detached from tempsocks list */
 #define XPT_LISTENER	11		/* listening endpoint */
 #define XPT_CACHE_AUTH	12		/* cache auth info */
+#define XPT_SHARE_SOCK	13		/* fore and back channel share socket */
 
Quoted 15 lines

net/sunrpc/svc_xprt.c static void svc_xprt_free(struct kref *kref) · line 132

Quoted 13 lines
include/linux/sunrpc/svc_xprt.h +0 -0
 	struct svc_pool		*xpt_pool;	/* current pool iff queued */
 	struct svc_serv		*xpt_server;	/* service for transport */
net/sunrpc/svc_xprt.c +2 -1
index ea2ff78..8c4d9ad 100644
@@ -128,7 +128,8 @@ static void svc_xprt_free(struct kref *kref)
 	if (test_bit(XPT_CACHE_AUTH, &xprt->xpt_flags))
 		svcauth_unix_info_release(xprt);
 	put_net(xprt->xpt_net);
-	xprt->xpt_ops->xpo_free(xprt);
+	if (!test_bit(XPT_SHARE_SOCK, &xprt->xpt_flags))
+		xprt->xpt_ops->xpo_free(xprt);

So when does the svc_xprt get freed if not here?

Quoted 2 lines

svc_sock_destroy frees the bc_xprt, called by svc_destroy on the serv->bc_xprt.

Can you remove the

	#if defined(CONFIG_NFS_V4_1)
        	svc_sock_destroy(serv->bc_xprt);
	#endif /* CONFIG_NFS_V4_1 */

from svc_destroy instead?

Quoted 7 lines
Quoted 2 lines

And why is it OK to do
the put_net() and module_put() but not the xpo_free()?

svc_sock_create() called only by nfs41_callback_up creates the bc_xprt. It calls svc_xprt_init which calls get_net(), so put_net is needed.

Quoted 7 lines

Hmmm. It looks like the module_put is also not needed - the matching module_get is in svc_create_xprt() which is not called for the NFSv4.1 callback service. (but is called for the NFSv4.0 callback service ... sheese).

I'll move the module_put so it is also not called when the XPT_SHARE_SOCK flag is set.

The xpt_net reference counting doesn't look right either.

I think we shouldn't need XPT_SHARE_SOCK if we just get the reference counting on the svc_xprt right instead.

Quoted 7 lines
Quoted 3 lines

Something feels wrong here....

Yeah - how about naming the server back channel xprt (bc_xprt) the same as the client back channel xprt (bc_xprt)!!! How confusing is THAT! :)

Well, and it's also the name of a field in svc_serv. If nothing else, would you mind renaming that to sv_bc_xprt?

--b.

Quoted 25 lines

Thanks for the review.

-->Andy

Quoted 19 lines

--b.

Quoted 11 lines
net/sunrpc/svc_xprt.c +0 -0
 	module_put(owner);
 }
 
Signature · 7 lines
1.6.6

--
To unsubscribe from this list: send the line "unsubscribe linux-
nfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#13 lore raw

Re: [PATCH_V3 1/7] NFS dont free shared socket on backchannel put xprt

Quoted 65 lines · J. Bruce Fields
Quoted 61 lines · Andy Adamson
Quoted 42 lines · J. Bruce Fields
Quoted 41 lines · Andy Adamson

Fixes this bug:
fedora-64 kernel: Invoking bc_svc_procass() fedora-64 kernel: nfs_callback_authenticate SVC_DROP fedora-64 kernel: BUG: unable to handle kernel NULL pointer dereference at 0000000000000018 IP: [<ffffffffa0156140>] svc_sock_free+0x32/0x56 [sunrpc]

---

fs/nfs/callback.c               |    3 +++
include/linux/sunrpc/svc_xprt.h |    1 +
net/sunrpc/svc_xprt.c           |    3 ++-
3 files changed, 6 insertions(+), 1 deletions(-)
fs/nfs/callback.c +3 -0
index 93a8b3b..023a9eb 100644
@@ -193,6 +193,9 @@ nfs41_callback_up(struct svc_serv *serv,
 struct rpc_xprt *xprt)
 	serv->bc_xprt = bc_xprt;
 	xprt->bc_serv = serv;
 
+	/* socket is shared with the fore channel */
+	set_bit(XPT_SHARE_SOCK, &bc_xprt->xpt_flags);
+
 	INIT_LIST_HEAD(&serv->sv_cb_list);
 	spin_lock_init(&serv->sv_cb_lock);
	init_waitqueue_head(&serv->sv_cb_waitq);
include/linux/sunrpc/svc_xprt.h +1 -0
index aea0d43..600c669 100644
@@ -62,6 +62,7 @@ struct svc_xprt {
 #define	XPT_DETACHED	10		/* detached from tempsocks list */
 #define XPT_LISTENER	11		/* listening endpoint */
 #define XPT_CACHE_AUTH	12		/* cache auth info */
+#define XPT_SHARE_SOCK	13		/* fore and back channel share socket  
 */
 
Quoted 15 lines

net/sunrpc/svc_xprt.c static void svc_xprt_free(struct kref *kref) · line 132

Quoted 13 lines
include/linux/sunrpc/svc_xprt.h +0 -0
 	struct svc_pool		*xpt_pool;	/* current pool iff queued */

struct svc_serv *xpt_server; /* service for transport */

net/sunrpc/svc_xprt.c +2 -1
index ea2ff78..8c4d9ad 100644
@@ -128,7 +128,8 @@ static void svc_xprt_free(struct kref *kref)
 	if (test_bit(XPT_CACHE_AUTH, &xprt->xpt_flags))
 		svcauth_unix_info_release(xprt);
 	put_net(xprt->xpt_net);
-	xprt->xpt_ops->xpo_free(xprt);
+	if (!test_bit(XPT_SHARE_SOCK, &xprt->xpt_flags))
+		xprt->xpt_ops->xpo_free(xprt);

So when does the svc_xprt get freed if not here?

Quoted 2 lines

svc_sock_destroy frees the bc_xprt, called by svc_destroy on the serv->bc_xprt.

Quoted 7 lines

Can you remove the

	#if defined(CONFIG_NFS_V4_1)
       	svc_sock_destroy(serv->bc_xprt);
	#endif /* CONFIG_NFS_V4_1 */

from svc_destroy instead?

Instead of what? Where do you want svc_sock_destroy to be called? From svc_xprt_free?

Quoted 13 lines
Quoted 13 lines
Quoted 3 lines

And why is it OK to do
the put_net() and module_put() but not the xpo_free()?

svc_sock_create() called only by nfs41_callback_up creates the bc_xprt. It calls svc_xprt_init which calls get_net(), so put_net is needed.

Hmmm. It looks like the module_put is also not needed - the matching module_get is in svc_create_xprt() which is not called for the NFSv4.1 callback service. (but is called for the NFSv4.0 callback service ... sheese).

Quoted 4 lines
Quoted 2 lines

I'll move the module_put so it is also not called when the XPT_SHARE_SOCK flag is set.

The xpt_net reference counting doesn't look right either.

I'm not familiar with the xpt_net reference counting.

Quoted 3 lines

I think we shouldn't need XPT_SHARE_SOCK if we just get the reference counting on the svc_xprt right instead.

At the end of the day svc_xprt_free will be called on this svc_xprt. At that time, xpo_free can not be called because xpo_create was not used to create it. svc_sock_destroy needs to be called to free the storage kzalloc'ed in svc_sock_create.

I don't get what you are suggesting.....

Quoted 11 lines
Quoted 8 lines
Quoted 4 lines

Something feels wrong here....

Yeah - how about naming the server back channel xprt (bc_xprt) the same as the client back channel xprt (bc_xprt)!!! How confusing is THAT! :)

Well, and it's also the name of a field in svc_serv. If nothing else, would you mind renaming that to sv_bc_xprt?

I can clean up the naming after we figure out the above.

-->Andy

Quoted 28 lines

--b.

Quoted 25 lines

Thanks for the review.

-->Andy

Quoted 19 lines

--b.

Quoted 11 lines
net/sunrpc/svc_xprt.c +0 -0
 	module_put(owner);
 }
 
Signature · 7 lines
1.6.6

--
To unsubscribe from this list: send the line "unsubscribe linux-
nfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#14 lore raw

Re: [PATCH_V3 1/7] NFS dont free shared socket on backchannel put xprt

Quoted 70 lines · Andy Adamson
Quoted 67 lines · J. Bruce Fields
Quoted 61 lines · Andy Adamson
Quoted 42 lines · J. Bruce Fields
Quoted 41 lines · Andy Adamson

Fixes this bug:
fedora-64 kernel: Invoking bc_svc_procass() fedora-64 kernel: nfs_callback_authenticate SVC_DROP fedora-64 kernel: BUG: unable to handle kernel NULL pointer dereference at 0000000000000018 IP: [<ffffffffa0156140>] svc_sock_free+0x32/0x56 [sunrpc]

---

fs/nfs/callback.c               |    3 +++
include/linux/sunrpc/svc_xprt.h |    1 +
net/sunrpc/svc_xprt.c           |    3 ++-
3 files changed, 6 insertions(+), 1 deletions(-)
fs/nfs/callback.c +3 -0
index 93a8b3b..023a9eb 100644
@@ -193,6 +193,9 @@ nfs41_callback_up(struct svc_serv *serv,
 struct rpc_xprt *xprt)
 	serv->bc_xprt = bc_xprt;
 	xprt->bc_serv = serv;
 
+	/* socket is shared with the fore channel */
+	set_bit(XPT_SHARE_SOCK, &bc_xprt->xpt_flags);
+
 	INIT_LIST_HEAD(&serv->sv_cb_list);
 	spin_lock_init(&serv->sv_cb_lock);
	init_waitqueue_head(&serv->sv_cb_waitq);
include/linux/sunrpc/svc_xprt.h +1 -0
index aea0d43..600c669 100644
@@ -62,6 +62,7 @@ struct svc_xprt {
 #define	XPT_DETACHED	10		/* detached from tempsocks list */
 #define XPT_LISTENER	11		/* listening endpoint */
 #define XPT_CACHE_AUTH	12		/* cache auth info */
+#define XPT_SHARE_SOCK	13		/* fore and back channel share

socket */

Quoted 15 lines

net/sunrpc/svc_xprt.c static void svc_xprt_free(struct kref *kref) · line 132

Quoted 13 lines

struct svc_pool *xpt_pool; /* current pool iff queued */
struct svc_serv *xpt_server; /* service for transport */

net/sunrpc/svc_xprt.c +2 -1
index ea2ff78..8c4d9ad 100644
@@ -128,7 +128,8 @@ static void svc_xprt_free(struct kref *kref)
 	if (test_bit(XPT_CACHE_AUTH, &xprt->xpt_flags))
 		svcauth_unix_info_release(xprt);
 	put_net(xprt->xpt_net);
-	xprt->xpt_ops->xpo_free(xprt);
+	if (!test_bit(XPT_SHARE_SOCK, &xprt->xpt_flags))
+		xprt->xpt_ops->xpo_free(xprt);

So when does the svc_xprt get freed if not here?

Quoted 2 lines

svc_sock_destroy frees the bc_xprt, called by svc_destroy on the serv->bc_xprt.

Can you remove the

Quoted 7 lines
Quoted 5 lines
	#if defined(CONFIG_NFS_V4_1)
      	svc_sock_destroy(serv->bc_xprt);
	#endif /* CONFIG_NFS_V4_1 */

from svc_destroy instead?

Instead of what?

Instead of the patch above.

--b.

Quoted 82 lines

Where do you want svc_sock_destroy to be called?

From svc_xprt_free?

Quoted 13 lines
Quoted 13 lines
Quoted 3 lines

And why is it OK to do
the put_net() and module_put() but not the xpo_free()?

svc_sock_create() called only by nfs41_callback_up creates the bc_xprt. It calls svc_xprt_init which calls get_net(), so put_net is needed.

Hmmm. It looks like the module_put is also not needed - the matching module_get is in svc_create_xprt() which is not called for the NFSv4.1 callback service. (but is called for the NFSv4.0 callback service ... sheese).

Quoted 4 lines
Quoted 2 lines

I'll move the module_put so it is also not called when the XPT_SHARE_SOCK flag is set.

The xpt_net reference counting doesn't look right either.

I'm not familiar with the xpt_net reference counting.

Quoted 4 lines

I think we shouldn't need XPT_SHARE_SOCK if we just get the reference counting
on the svc_xprt right instead.

At the end of the day svc_xprt_free will be called on this svc_xprt. At that time, xpo_free can not be called because xpo_create was not used to create it. svc_sock_destroy needs to be called to free the storage kzalloc'ed in svc_sock_create.

I don't get what you are suggesting.....

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

Something feels wrong here....

Quoted 7 lines
Quoted 3 lines

Yeah - how about naming the server back channel xprt (bc_xprt) the same as the client back channel xprt (bc_xprt)!!! How confusing is THAT! :)

Well, and it's also the name of a field in svc_serv. If nothing else, would
you mind renaming that to sv_bc_xprt?

I can clean up the naming after we figure out the above.

-->Andy

Quoted 28 lines

--b.

Quoted 25 lines

Thanks for the review.

-->Andy

Quoted 19 lines

--b.

Quoted 11 lines
net/sunrpc/svc_xprt.c +0 -0
 	module_put(owner);
 }
 
Signature · 7 lines
1.6.6

--
To unsubscribe from this list: send the line "unsubscribe linux-
nfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#15 lore raw

Re: [PATCH_V3 1/7] NFS dont free shared socket on backchannel put xprt

Quoted 75 lines · J. Bruce Fields
Quoted 74 lines · Andy Adamson
Quoted 71 lines · J. Bruce Fields
Quoted 61 lines · Andy Adamson
Quoted 42 lines · J. Bruce Fields
Quoted 41 lines · Andy Adamson

Fixes this bug:
fedora-64 kernel: Invoking bc_svc_procass() fedora-64 kernel: nfs_callback_authenticate SVC_DROP fedora-64 kernel: BUG: unable to handle kernel NULL pointer dereference at 0000000000000018 IP: [<ffffffffa0156140>] svc_sock_free+0x32/0x56 [sunrpc]

---

fs/nfs/callback.c               |    3 +++
include/linux/sunrpc/svc_xprt.h |    1 +
net/sunrpc/svc_xprt.c           |    3 ++-
3 files changed, 6 insertions(+), 1 deletions(-)
fs/nfs/callback.c +3 -0
index 93a8b3b..023a9eb 100644
@@ -193,6 +193,9 @@ nfs41_callback_up(struct svc_serv *serv,
 struct rpc_xprt *xprt)
 	serv->bc_xprt = bc_xprt;
 	xprt->bc_serv = serv;
 
+	/* socket is shared with the fore channel */
+	set_bit(XPT_SHARE_SOCK, &bc_xprt->xpt_flags);
+
 	INIT_LIST_HEAD(&serv->sv_cb_list);
 	spin_lock_init(&serv->sv_cb_lock);
	init_waitqueue_head(&serv->sv_cb_waitq);
include/linux/sunrpc/svc_xprt.h +1 -0
index aea0d43..600c669 100644
@@ -62,6 +62,7 @@ struct svc_xprt {
 #define	XPT_DETACHED	10		/* detached from tempsocks list */
 #define XPT_LISTENER	11		/* listening endpoint */
 #define XPT_CACHE_AUTH	12		/* cache auth info */
+#define XPT_SHARE_SOCK	13		/* fore and back channel share

socket */

Quoted 15 lines

net/sunrpc/svc_xprt.c static void svc_xprt_free(struct kref *kref) · line 132

Quoted 13 lines

struct svc_pool *xpt_pool; /* current pool iff queued */
struct svc_serv *xpt_server; /* service for transport */

net/sunrpc/svc_xprt.c +2 -1
index ea2ff78..8c4d9ad 100644
@@ -128,7 +128,8 @@ static void svc_xprt_free(struct kref *kref)
 	if (test_bit(XPT_CACHE_AUTH, &xprt->xpt_flags))
 		svcauth_unix_info_release(xprt);
 	put_net(xprt->xpt_net);
-	xprt->xpt_ops->xpo_free(xprt);
+	if (!test_bit(XPT_SHARE_SOCK, &xprt->xpt_flags))
+		xprt->xpt_ops->xpo_free(xprt);

So when does the svc_xprt get freed if not here?

Quoted 2 lines

svc_sock_destroy frees the bc_xprt, called by svc_destroy on the serv->bc_xprt.

Can you remove the

	#if defined(CONFIG_NFS_V4_1)
     	svc_sock_destroy(serv->bc_xprt);
	#endif /* CONFIG_NFS_V4_1 */
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

from svc_destroy instead?

Instead of what?

Instead of the patch above.

Sorry, I just don't understand how that will solve the sock_free BUG above.

-->Andy

Quoted 85 lines

--b.

Quoted 82 lines

Where do you want svc_sock_destroy to be called?

From svc_xprt_free?

Quoted 13 lines
Quoted 13 lines
Quoted 3 lines

And why is it OK to do
the put_net() and module_put() but not the xpo_free()?

svc_sock_create() called only by nfs41_callback_up creates the bc_xprt. It calls svc_xprt_init which calls get_net(), so put_net is needed.

Hmmm. It looks like the module_put is also not needed - the matching module_get is in svc_create_xprt() which is not called for the NFSv4.1 callback service. (but is called for the NFSv4.0 callback service ... sheese).

Quoted 4 lines
Quoted 2 lines

I'll move the module_put so it is also not called when the XPT_SHARE_SOCK flag is set.

The xpt_net reference counting doesn't look right either.

I'm not familiar with the xpt_net reference counting.

Quoted 4 lines

I think we shouldn't need XPT_SHARE_SOCK if we just get the reference counting
on the svc_xprt right instead.

At the end of the day svc_xprt_free will be called on this svc_xprt. At that time, xpo_free can not be called because xpo_create was not used to create it. svc_sock_destroy needs to be called to free the storage kzalloc'ed in svc_sock_create.

I don't get what you are suggesting.....

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

Something feels wrong here....

Quoted 7 lines
Quoted 3 lines

Yeah - how about naming the server back channel xprt (bc_xprt) the same as the client back channel xprt (bc_xprt)!!! How confusing is THAT! :)

Well, and it's also the name of a field in svc_serv. If nothing else, would
you mind renaming that to sv_bc_xprt?

I can clean up the naming after we figure out the above.

-->Andy

Quoted 28 lines

--b.

Quoted 25 lines

Thanks for the review.

-->Andy

Quoted 19 lines

--b.

Quoted 11 lines
net/sunrpc/svc_xprt.c +0 -0
 	module_put(owner);
 }
 
Signature · 7 lines
1.6.6

--
To unsubscribe from this list: send the line "unsubscribe linux-
nfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#16 lore raw

Re: [PATCH_V3 1/7] NFS dont free shared socket on backchannel put xprt

Quoted 78 lines · Andy Adamson
Quoted 75 lines · J. Bruce Fields
Quoted 74 lines · Andy Adamson
Quoted 71 lines · J. Bruce Fields
Quoted 61 lines · Andy Adamson
Quoted 42 lines · J. Bruce Fields
Quoted 41 lines · Andy Adamson

Fixes this bug:
fedora-64 kernel: Invoking bc_svc_procass() fedora-64 kernel: nfs_callback_authenticate SVC_DROP fedora-64 kernel: BUG: unable to handle kernel NULL pointer dereference at 0000000000000018 IP: [<ffffffffa0156140>] svc_sock_free+0x32/0x56 [sunrpc]

---

fs/nfs/callback.c               |    3 +++
include/linux/sunrpc/svc_xprt.h |    1 +
net/sunrpc/svc_xprt.c           |    3 ++-
3 files changed, 6 insertions(+), 1 deletions(-)
fs/nfs/callback.c +3 -0
index 93a8b3b..023a9eb 100644
@@ -193,6 +193,9 @@ nfs41_callback_up(struct svc_serv *serv,
 struct rpc_xprt *xprt)
 	serv->bc_xprt = bc_xprt;
 	xprt->bc_serv = serv;
 
+	/* socket is shared with the fore channel */
+	set_bit(XPT_SHARE_SOCK, &bc_xprt->xpt_flags);
+
 	INIT_LIST_HEAD(&serv->sv_cb_list);
 	spin_lock_init(&serv->sv_cb_lock);
	init_waitqueue_head(&serv->sv_cb_waitq);
include/linux/sunrpc/svc_xprt.h +1 -0
index aea0d43..600c669 100644
@@ -62,6 +62,7 @@ struct svc_xprt {
 #define	XPT_DETACHED	10		/* detached from tempsocks list */
 #define XPT_LISTENER	11		/* listening endpoint */
 #define XPT_CACHE_AUTH	12		/* cache auth info */
+#define XPT_SHARE_SOCK	13		/* fore and back channel share

socket */

Quoted 15 lines

net/sunrpc/svc_xprt.c static void svc_xprt_free(struct kref *kref) · line 132

Quoted 13 lines

struct svc_pool *xpt_pool; /* current pool iff queued */
struct svc_serv *xpt_server; /* service for transport */

net/sunrpc/svc_xprt.c +2 -1
index ea2ff78..8c4d9ad 100644
@@ -128,7 +128,8 @@ static void svc_xprt_free(struct kref *kref)
 	if (test_bit(XPT_CACHE_AUTH, &xprt->xpt_flags))
 		svcauth_unix_info_release(xprt);
 	put_net(xprt->xpt_net);
-	xprt->xpt_ops->xpo_free(xprt);
+	if (!test_bit(XPT_SHARE_SOCK, &xprt->xpt_flags))
+		xprt->xpt_ops->xpo_free(xprt);

So when does the svc_xprt get freed if not here?

Quoted 2 lines

svc_sock_destroy frees the bc_xprt, called by svc_destroy on the serv->bc_xprt.

Can you remove the

	#if defined(CONFIG_NFS_V4_1)
    	svc_sock_destroy(serv->bc_xprt);
	#endif /* CONFIG_NFS_V4_1 */
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

from svc_destroy instead?

Instead of what?

Instead of the patch above.

Sorry, I just don't understand how that will solve the sock_free BUG above.

It may not, sorry, I'd need to look at it more closely. Maybe you could explain in more detail how the bug happens and why? (Which pointer is it that's null, and why?)

--b.

#17 lore raw

Re: [PATCH_V3 1/7] NFS dont free shared socket on backchannel put xprt

Quoted 84 lines · J. Bruce Fields
Quoted 83 lines · Andy Adamson
Quoted 80 lines · J. Bruce Fields
Quoted 78 lines · Andy Adamson
Quoted 66 lines · J. Bruce Fields
Quoted 62 lines · Andy Adamson
Quoted 42 lines · J. Bruce Fields
Quoted 41 lines · Andy Adamson

Fixes this bug:
fedora-64 kernel: Invoking bc_svc_procass() fedora-64 kernel: nfs_callback_authenticate SVC_DROP fedora-64 kernel: BUG: unable to handle kernel NULL pointer dereference at 0000000000000018 IP: [<ffffffffa0156140>] svc_sock_free+0x32/0x56 [sunrpc]

---

fs/nfs/callback.c               |    3 +++
include/linux/sunrpc/svc_xprt.h |    1 +
net/sunrpc/svc_xprt.c           |    3 ++-
3 files changed, 6 insertions(+), 1 deletions(-)
fs/nfs/callback.c +3 -0
index 93a8b3b..023a9eb 100644
@@ -193,6 +193,9 @@ nfs41_callback_up(struct svc_serv *serv,
 struct rpc_xprt *xprt)
 	serv->bc_xprt = bc_xprt;
 	xprt->bc_serv = serv;
 
+	/* socket is shared with the fore channel */
+	set_bit(XPT_SHARE_SOCK, &bc_xprt->xpt_flags);
+
 	INIT_LIST_HEAD(&serv->sv_cb_list);
 	spin_lock_init(&serv->sv_cb_lock);
	init_waitqueue_head(&serv->sv_cb_waitq);
include/linux/sunrpc/svc_xprt.h +1 -0
index aea0d43..600c669 100644
@@ -62,6 +62,7 @@ struct svc_xprt {
 #define	XPT_DETACHED	10		/* detached from tempsocks list */
 #define XPT_LISTENER	11		/* listening endpoint */
 #define XPT_CACHE_AUTH	12		/* cache auth info */
+#define XPT_SHARE_SOCK	13		/* fore and back channel share

socket */

Quoted 16 lines
Quoted 7 lines

struct svc_pool *xpt_pool; /* current pool iff queued */
struct svc_serv *xpt_server; /* service for transport */

net/sunrpc/svc_xprt.c +0 -0
index ea2ff78..8c4d9ad 100644

net/sunrpc/svc_xprt.c static void svc_xprt_free(struct kref · line 133

Quoted 7 lines
net/sunrpc/svc_xprt.c +2 -1
@@ -128,7 +128,8 @@ static void svc_xprt_free(struct kref  
 *kref)
 	if (test_bit(XPT_CACHE_AUTH, &xprt->xpt_flags))
 		svcauth_unix_info_release(xprt);
 	put_net(xprt->xpt_net);
-	xprt->xpt_ops->xpo_free(xprt);
+	if (!test_bit(XPT_SHARE_SOCK, &xprt->xpt_flags))
+		xprt->xpt_ops->xpo_free(xprt);

So when does the svc_xprt get freed if not here?

Quoted 2 lines

svc_sock_destroy frees the bc_xprt, called by svc_destroy on the serv->bc_xprt.

Quoted 7 lines

Can you remove the

	#if defined(CONFIG_NFS_V4_1)
   	svc_sock_destroy(serv->bc_xprt);
	#endif /* CONFIG_NFS_V4_1 */

from svc_destroy instead?

Instead of what?

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Instead of the patch above.

Sorry, I just don't understand how that will solve the sock_free BUG above.

It may not, sorry, I'd need to look at it more closely. Maybe you could explain in more detail how the bug happens and why? (Which pointer is it that's null, and why?)

I force an SVC_DROP in nfs_callback_authenticate. svc_process_comon then calls svc_drop -> svc_xprt_release -> svc_xprt_put -> svc_xprt_free -> bc_xprt-> xpt_ops->xpt_free -> svc_sock_free where the svc_sock->sk_sock pointer is NULL - set that way at creation.

After more investigation, the bc_xprt.xpt_ref is not incremented across svc processing, so if there is an error such as svc_drop, svc_xprt_put call ends up trying to free the bc_xprt.

What we want is to look at the single bc_xprt as the "pool" of svc_xprts for the back channel. It should have a lifetime equal to that of the svc_serv. If we take a reference on the bc_xprt across processing (e.g. "recv" takes a reference, and upon no error, drop the reference after "send" then we will keep the bc_xprt around.

I'm thinking of doing the following:
- svc_xprt_get to bc_svc_process, and an svc_xprt_put at the end of bc_send.
- look into creating an svc_xprt_ops for the back channel and perhaps adding a module_get on the xpt_class->xcl_owner in nfs41_callback_up so that svc_xprt_put works correctly.
- calling svc_xprt_put instead of svc_sock_destroy in svc_destroy.

Does this sound like a good plan?

I note that svc_destroy is not even called because for some reason, nfs_callback_down gets an nfs_client struct with the cl_mvop->minor_version set to 0, so svc_exit_thread is not even called. I'll figure this out as well.

-->Andy

Quoted 2 lines

--b.

#18 lore raw

Re: [PATCH_V3 1/7] NFS dont free shared socket on backchannel put xprt

Quoted 106 lines · Andy Adamson
Quoted 86 lines · J. Bruce Fields
Quoted 85 lines · Andy Adamson
Quoted 81 lines · J. Bruce Fields
Quoted 71 lines · Andy Adamson
Quoted 68 lines · J. Bruce Fields
Quoted 62 lines · Andy Adamson
Quoted 42 lines · J. Bruce Fields
Quoted 41 lines · Andy Adamson

Fixes this bug:
fedora-64 kernel: Invoking bc_svc_procass() fedora-64 kernel: nfs_callback_authenticate SVC_DROP fedora-64 kernel: BUG: unable to handle kernel NULL pointer dereference at 0000000000000018 IP: [<ffffffffa0156140>] svc_sock_free+0x32/0x56 [sunrpc]

---

fs/nfs/callback.c               |    3 +++
include/linux/sunrpc/svc_xprt.h |    1 +
net/sunrpc/svc_xprt.c           |    3 ++-
3 files changed, 6 insertions(+), 1 deletions(-)
fs/nfs/callback.c +3 -0
index 93a8b3b..023a9eb 100644
@@ -193,6 +193,9 @@ nfs41_callback_up(struct svc_serv *serv,
 struct rpc_xprt *xprt)
 	serv->bc_xprt = bc_xprt;
 	xprt->bc_serv = serv;
 
+	/* socket is shared with the fore channel */
+	set_bit(XPT_SHARE_SOCK, &bc_xprt->xpt_flags);
+
 	INIT_LIST_HEAD(&serv->sv_cb_list);
 	spin_lock_init(&serv->sv_cb_lock);
	init_waitqueue_head(&serv->sv_cb_waitq);
include/linux/sunrpc/svc_xprt.h +1 -0
index aea0d43..600c669 100644
@@ -62,6 +62,7 @@ struct svc_xprt {
 #define	XPT_DETACHED	10		/* detached from tempsocks list */
 #define XPT_LISTENER	11		/* listening endpoint */
 #define XPT_CACHE_AUTH	12		/* cache auth info */
+#define XPT_SHARE_SOCK	13		/* fore and back channel share

socket */

Quoted 16 lines
Quoted 7 lines

struct svc_pool *xpt_pool; /* current pool iff queued */
struct svc_serv *xpt_server; /* service for transport */

net/sunrpc/svc_xprt.c +0 -0
index ea2ff78..8c4d9ad 100644

net/sunrpc/svc_xprt.c static void svc_xprt_free(struct · line 133

Quoted 7 lines
net/sunrpc/svc_xprt.c +2 -1
@@ -128,7 +128,8 @@ static void svc_xprt_free(struct
 kref *kref)
 	if (test_bit(XPT_CACHE_AUTH, &xprt->xpt_flags))
 		svcauth_unix_info_release(xprt);
 	put_net(xprt->xpt_net);
-	xprt->xpt_ops->xpo_free(xprt);
+	if (!test_bit(XPT_SHARE_SOCK, &xprt->xpt_flags))
+		xprt->xpt_ops->xpo_free(xprt);

So when does the svc_xprt get freed if not here?

Quoted 2 lines

svc_sock_destroy frees the bc_xprt, called by svc_destroy on the serv->bc_xprt.

Can you remove the

Quoted 7 lines
Quoted 5 lines
	#if defined(CONFIG_NFS_V4_1)
  	svc_sock_destroy(serv->bc_xprt);
	#endif /* CONFIG_NFS_V4_1 */

from svc_destroy instead?

Instead of what?

Instead of the patch above.

Quoted 7 lines
Quoted 2 lines

Sorry, I just don't understand how that will solve the sock_free BUG above.

It may not, sorry, I'd need to look at it more closely. Maybe you could
explain in more detail how the bug happens and why? (Which pointer is it that's null, and why?)

I force an SVC_DROP in nfs_callback_authenticate. svc_process_comon then calls svc_drop -> svc_xprt_release -> svc_xprt_put -> svc_xprt_free -> bc_xprt-> xpt_ops->xpt_free -> svc_sock_free where the svc_sock->sk_sock pointer is NULL - set that way at creation.

After more investigation, the bc_xprt.xpt_ref is not incremented across svc processing, so if there is an error such as svc_drop, svc_xprt_put call ends up trying to free the bc_xprt.

Quoted 5 lines

What we want is to look at the single bc_xprt as the "pool" of svc_xprts for the back channel. It should have a lifetime equal to that of the svc_serv. If we take a reference on the bc_xprt across processing (e.g. "recv" takes a reference, and upon no error, drop the reference after "send" then we will keep the bc_xprt around.

That's starting to make more sense to me now, thanks!

Quoted 9 lines

I'm thinking of doing the following:
- svc_xprt_get to bc_svc_process, and an svc_xprt_put at the end of bc_send.
- look into creating an svc_xprt_ops for the back channel and perhaps adding a module_get on the xpt_class->xcl_owner in nfs41_callback_up so that svc_xprt_put works correctly.
- calling svc_xprt_put instead of svc_sock_destroy in svc_destroy.

Does this sound like a good plan?

I haven't thought it through, but yes, something like that sounds better.

Quoted 4 lines

I note that svc_destroy is not even called because for some reason, nfs_callback_down gets an nfs_client struct with the cl_mvop-

Quoted 1 line

minor_version set to 0, so svc_exit_thread is not even called. I'll

figure this out as well.

Hm, OK, thanks for looking into this more closely!

--b.