Skip to contentSkip to conversation

xprtrdma: Make sure Send CQ is allocated on an existing compvec

PATCH Superseded Nicolas Morey-Chaisemartin 3 participants 4 messages Jan 24, 2019 – Jan 25, 2019

Subsystems (3)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 xprtrdma: Make sure Send CQ is allocated on an existing compvec

Make sure the device has at least 2 completion vectors before allocating to compvec#1

Fixes a4699f5647f3 (xprtrdma: Put Send CQ in IB_POLL_WORKQUEUE mode)

---
Fixes since v1:
- Use num_comp_vector instead online_cpus

 net/sunrpc/xprtrdma/verbs.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
net/sunrpc/xprtrdma/verbs.c +1 -1
index b725911c0f3f..db913bcef984 100644
@@ -546,7 +546,7 @@ rpcrdma_ep_create(struct rpcrdma_ep *ep, struct rpcrdma_ia *ia,
 
 	sendcq = ib_alloc_cq(ia->ri_device, NULL,
 			     ep->rep_attr.cap.max_send_wr + 1,
-			     1, IB_POLL_WORKQUEUE);
+			     ia->ri_device->num_comp_vectors > 1 ? 1 : 0, IB_POLL_WORKQUEUE);
 	if (IS_ERR(sendcq)) {
 		rc = PTR_ERR(sendcq);
 		dprintk("RPC:       %s: failed to create send CQ: %i\n",
Signature · 1 line
2.18.0
reviewer
#1 lore raw
Quoted 1 line · Nicolas Morey-Chaisemartin

Make sure the device has at least 2 completion vectors before allocating to compvec#1

I agree this fixes the bug, but wouldn't it be better to attempt to steer the affinity to chosen cores, rather than unconditionally using static values? At least make some determination that the device, and the memory that was just allocated, were pointed to slightly optimal NUMA node to handle its completions.

Does the verbs API not support this? I thought that was discussed a while back.

Tom.

Quoted 23 lines
Fixes a4699f5647f3 (xprtrdma: Put Send CQ in IB_POLL_WORKQUEUE mode)

---
Fixes since v1:
- Use num_comp_vector instead online_cpus

  net/sunrpc/xprtrdma/verbs.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
net/sunrpc/xprtrdma/verbs.c +1 -1
index b725911c0f3f..db913bcef984 100644
@@ -546,7 +546,7 @@ rpcrdma_ep_create(struct rpcrdma_ep *ep, struct rpcrdma_ia *ia,
 
  	sendcq = ib_alloc_cq(ia->ri_device, NULL,
  			     ep->rep_attr.cap.max_send_wr + 1,
-			     1, IB_POLL_WORKQUEUE);
+			     ia->ri_device->num_comp_vectors > 1 ? 1 : 0, IB_POLL_WORKQUEUE);
  	if (IS_ERR(sendcq)) {
  		rc = PTR_ERR(sendcq);
  		dprintk("RPC:       %s: failed to create send CQ: %i\n",
#2 lore raw
Quoted 10 lines · Tom Talpey
Quoted 1 line · Nicolas Morey-Chaisemartin

Make sure the device has at least 2 completion vectors before allocating to compvec#1

I agree this fixes the bug, but wouldn't it be better to attempt to steer the affinity to chosen cores, rather than unconditionally using static values?

The goal for the moment is to create a patch that can be backported to stable kernels with minimal fuss.

Quoted 6 lines

At least make some determination that the
device, and the memory that was just allocated, were pointed to slightly optimal NUMA node to handle its completions.

Does the verbs API not support this? I thought that was discussed a while back.

It is true that spreading CQs across interrupt vectors is desirable, but for various reasons, it has been difficult to do it in a way that doesn't have bad behavior in some common cases.

Sagi has constructed some infrastructure that works well for ULPs that can create many transport instances, but we don't yet have that ability for NFS.

I'm open to suggestions for a subsequent patch after Nicolas' fix is merged.

Quoted 13 lines

Tom.

Quoted 11 lines
Fixes a4699f5647f3 (xprtrdma: Put Send CQ in IB_POLL_WORKQUEUE mode)

---
Fixes since v1:
- Use num_comp_vector instead online_cpus

 net/sunrpc/xprtrdma/verbs.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
net/sunrpc/xprtrdma/verbs.c +0 -0
index b725911c0f3f..db913bcef984 100644
Quoted 8 lines

net/sunrpc/xprtrdma/verbs.c rpcrdma_ep_create(struct rpcrdma_ep *ep, struct rpcrdma_ia *ia, · line 551

Quoted 8 lines
net/sunrpc/xprtrdma/verbs.c +1 -1
@@ -546,7 +546,7 @@ rpcrdma_ep_create(struct rpcrdma_ep *ep, struct rpcrdma_ia *ia,
   	sendcq = ib_alloc_cq(ia->ri_device, NULL,
 			     ep->rep_attr.cap.max_send_wr + 1,
-			     1, IB_POLL_WORKQUEUE);
+			     ia->ri_device->num_comp_vectors > 1 ? 1 : 0, IB_POLL_WORKQUEUE);
 	if (IS_ERR(sendcq)) {
 		rc = PTR_ERR(sendcq);
 		dprintk("RPC:       %s: failed to create send CQ: %i\n",

--
Chuck Lever

Reviewed-by on v2 1/1
#3 lore raw
In reply to Nicolas Morey-Chaisemartin
Quoted 20 lines · Nicolas Morey-Chaisemartin

Make sure the device has at least 2 completion vectors before allocating to compvec#1

Fixes a4699f5647f3 (xprtrdma: Put Send CQ in IB_POLL_WORKQUEUE mode)

---
Fixes since v1:
- Use num_comp_vector instead online_cpus

net/sunrpc/xprtrdma/verbs.c | 2 +-

1 file changed, 1 insertion(+), 1 deletion(-)
net/sunrpc/xprtrdma/verbs.c +0 -0
index b725911c0f3f..db913bcef984 100644

net/sunrpc/xprtrdma/verbs.c rpcrdma_ep_create(struct rpcrdma_ep *ep, struct rpcrdma_ia *ia, · line 549

Quoted 5 lines
net/sunrpc/xprtrdma/verbs.c +1 -1
@@ -546,7 +546,7 @@ rpcrdma_ep_create(struct rpcrdma_ep *ep, struct rpcrdma_ia *ia,
 
 	sendcq = ib_alloc_cq(ia->ri_device, NULL,
 			     ep->rep_attr.cap.max_send_wr + 1,
-			     1, IB_POLL_WORKQUEUE);
+			     ia->ri_device->num_comp_vectors > 1 ? 1 : 0, IB_POLL_WORKQUEUE);

Nit: You should split this line. Otherwise:

And I'll look into smarter compvec allocation again.

Quoted 6 lines
net/sunrpc/xprtrdma/verbs.c +0 -0
 	if (IS_ERR(sendcq)) {
 		rc = PTR_ERR(sendcq);
 		dprintk("RPC:       %s: failed to create send CQ: %i\n",
Signature · 1 line
2.18.0

--
Chuck Lever