Skip to contentSkip to conversation

Fixes for NFS/RDMA device removal

PATCH In review [email protected] 3 participants 7 messages Aug 12, 2024 – Aug 21, 2024

Subsystems (4)
lore
Share & tools
lore

[email protected]

author maintainer
#0 lore raw

cover Fixes for NFS/RDMA device removal

From: Chuck Lever <[email protected]>

Fix a handful of nits for the NFS/RDMA device removal code that went into v6.11-rc.

Chuck Lever (3):
rpcrdma: Device kref is over-incremented on error from xa_alloc
rpcrdma: Use XA_FLAGS_ALLOC instead of XA_FLAGS_ALLOC1
rpcrdma: Trace connection registration and unregistration

 include/trace/events/rpcrdma.h  | 36 +++++++++++++++++++++++++++++++++
 net/sunrpc/xprtrdma/ib_client.c |  6 ++++--
 2 files changed, 40 insertions(+), 2 deletions(-)
Signature · 1 line
2.45.1
author maintainer
#1 lore raw

PATCH 1/3 rpcrdma: Device kref is over-incremented on error from xa_alloc

From: Chuck Lever <[email protected]>

If the device's reference count is too high, the device completion callback never fires.

---
 net/sunrpc/xprtrdma/ib_client.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
net/sunrpc/xprtrdma/ib_client.c +1 -1
index a938c19c3490..4d1e9fa89573 100644
@@ -62,9 +62,9 @@ int rpcrdma_rn_register(struct ib_device *device,
 	if (!rd || test_bit(RPCRDMA_RD_F_REMOVING, &rd->rd_flags))
 		return -ENETUNREACH;
 
-	kref_get(&rd->rd_kref);
 	if (xa_alloc(&rd->rd_xa, &rn->rn_index, rn, xa_limit_32b, GFP_KERNEL) < 0)
 		return -ENOMEM;
+	kref_get(&rd->rd_kref);
 	rn->rn_done = done;
 	return 0;
 }
Signature · 1 line
2.45.1
author maintainer
#2 lore raw

PATCH 2/3 rpcrdma: Use XA_FLAGS_ALLOC instead of XA_FLAGS_ALLOC1

In reply to [email protected]

From: Chuck Lever <[email protected]>

Nit: The built-in xa_limit_32b range starts at 0, but XA_FLAGS_ALLOC1 configures the xarray's allocator to start at 1. Adopt the more conventional XA_FLAGS_ALLOC because there's no mechanical reason to skip 0.

---
 net/sunrpc/xprtrdma/ib_client.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
net/sunrpc/xprtrdma/ib_client.c +1 -1
index 4d1e9fa89573..7913d7bad23d 100644
@@ -111,7 +111,7 @@ static int rpcrdma_add_one(struct ib_device *device)
 		return -ENOMEM;
 
 	kref_init(&rd->rd_kref);
-	xa_init_flags(&rd->rd_xa, XA_FLAGS_ALLOC1);
+	xa_init_flags(&rd->rd_xa, XA_FLAGS_ALLOC);
 	rd->rd_device = device;
 	init_completion(&rd->rd_done);
 	ib_set_client_data(device, &rpcrdma_ib_client, rd);
Signature · 1 line
2.45.1
author maintainer
#3 lore raw

PATCH 3/3 rpcrdma: Trace connection registration and unregistration

In reply to [email protected]

From: Chuck Lever <[email protected]>

These new trace points record xarray indices and the time of endpoint registration and unregistration, to co-ordinate with device removal events.

---
 include/trace/events/rpcrdma.h  | 36 +++++++++++++++++++++++++++++++++
 net/sunrpc/xprtrdma/ib_client.c |  2 ++
 2 files changed, 38 insertions(+)
include/trace/events/rpcrdma.h +36 -0
index 9141398591e0..e6a72646c507 100644
@@ -2300,6 +2300,42 @@ DEFINE_CLIENT_DEVICE_EVENT(rpcrdma_client_remove_one);
 DEFINE_CLIENT_DEVICE_EVENT(rpcrdma_client_wait_on);
 DEFINE_CLIENT_DEVICE_EVENT(rpcrdma_client_remove_one_done);
 
+DECLARE_EVENT_CLASS(rpcrdma_client_register_class,
+	TP_PROTO(
+		const struct ib_device *device,
+		const struct rpcrdma_notification *rn
+	),
+
+	TP_ARGS(device, rn),
+
+	TP_STRUCT__entry(
+		__string(name, device->name)
+		__field(void *, callback)
+		__field(u32, index)
+	),
+
+	TP_fast_assign(
+		__assign_str(name);
+		__entry->callback = rn->rn_done;
+		__entry->index = rn->rn_index;
+	),
+
+	TP_printk("device=%s index=%u done callback=%pS\n",
+		__get_str(name), __entry->index, __entry->callback
+	)
+);
+
+#define DEFINE_CLIENT_REGISTER_EVENT(name)				\
+	DEFINE_EVENT(rpcrdma_client_register_class, name,		\
+	TP_PROTO(							\
+		const struct ib_device *device,				\
+		const struct rpcrdma_notification *rn			\
+	),								\
+	TP_ARGS(device, rn))
+
+DEFINE_CLIENT_REGISTER_EVENT(rpcrdma_client_register);
+DEFINE_CLIENT_REGISTER_EVENT(rpcrdma_client_unregister);
+
 #endif /* _TRACE_RPCRDMA_H */
 
 #include <trace/define_trace.h>
net/sunrpc/xprtrdma/ib_client.c +2 -0
index 7913d7bad23d..8507cd4d8921 100644
@@ -66,6 +66,7 @@ int rpcrdma_rn_register(struct ib_device *device,
 		return -ENOMEM;
 	kref_get(&rd->rd_kref);
 	rn->rn_done = done;
+	trace_rpcrdma_client_register(device, rn);
 	return 0;
 }
 
@@ -91,6 +92,7 @@ void rpcrdma_rn_unregister(struct ib_device *device,
 	if (!rd)
 		return;
 
+	trace_rpcrdma_client_unregister(device, rn);
 	xa_erase(&rd->rd_xa, rn->rn_index);
 	kref_put(&rd->rd_kref, rpcrdma_rn_release);
 }
Signature · 1 line
2.45.1
maintainer
#4 lore raw
In reply to [email protected]

Hi Chuck,

Quoted 5 lines · On Mon, Aug 12, 2024 at 11:48 AM <[email protected]> wrote:

From: Chuck Lever <[email protected]>

Fix a handful of nits for the NFS/RDMA device removal code that went into v6.11-rc.

Are you intending these for 6.11-rc or 6.12?

Anna

Quoted 13 lines

Chuck Lever (3):
rpcrdma: Device kref is over-incremented on error from xa_alloc
rpcrdma: Use XA_FLAGS_ALLOC instead of XA_FLAGS_ALLOC1
rpcrdma: Trace connection registration and unregistration

 include/trace/events/rpcrdma.h  | 36 +++++++++++++++++++++++++++++++++
 net/sunrpc/xprtrdma/ib_client.c |  6 ++++--
 2 files changed, 40 insertions(+), 2 deletions(-)

--
2.45.1

#5 lore raw
Quoted 14 lines · Anna Schumaker

Hi Chuck,

Quoted 5 lines · On Mon, Aug 12, 2024 at 11:48 AM <[email protected]> wrote:

From: Chuck Lever <[email protected]>

Fix a handful of nits for the NFS/RDMA device removal code that went into v6.11-rc.

Are you intending these for 6.11-rc or 6.12?

IIRC the patches they fix went into 6.11-rc1, so I'd like to see these go into a 6.11-rc release.

Quoted 15 lines

Anna

Quoted 13 lines

Chuck Lever (3):
rpcrdma: Device kref is over-incremented on error from xa_alloc
rpcrdma: Use XA_FLAGS_ALLOC instead of XA_FLAGS_ALLOC1
rpcrdma: Trace connection registration and unregistration

include/trace/events/rpcrdma.h | 36 +++++++++++++++++++++++++++++++++ net/sunrpc/xprtrdma/ib_client.c | 6 ++++--

2 files changed, 40 insertions(+), 2 deletions(-)

--
2.45.1

--
Chuck Lever

maintainer
#6 lore raw
Quoted 11 lines · Chuck Lever III
Quoted 11 lines · Anna Schumaker

Hi Chuck,

Quoted 3 lines · On Mon, Aug 12, 2024 at 11:48 AM <[email protected]> wrote:

From: Chuck Lever <[email protected]>

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Fix a handful of nits for the NFS/RDMA device removal code that went into v6.11-rc.

Are you intending these for 6.11-rc or 6.12?

IIRC the patches they fix went into 6.11-rc1, so I'd like to see these go into a 6.11-rc release.

Sounds good, thanks!

Anna

Quoted 22 lines
Quoted 17 lines

Anna

Quoted 13 lines

Chuck Lever (3):
rpcrdma: Device kref is over-incremented on error from xa_alloc
rpcrdma: Use XA_FLAGS_ALLOC instead of XA_FLAGS_ALLOC1
rpcrdma: Trace connection registration and unregistration

include/trace/events/rpcrdma.h | 36 +++++++++++++++++++++++++++++++++ net/sunrpc/xprtrdma/ib_client.c | 6 ++++--

2 files changed, 40 insertions(+), 2 deletions(-)

--
2.45.1

--
Chuck Lever