Skip to contentSkip to conversation

RDMA/rxe: Fix qp reference counting for atomic ops

PATCH Applied Bob Pearson 4 participants 12 messages Jun 4, 2021 – Jun 16, 2021

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 RDMA/rxe: Fix qp reference counting for atomic ops

Currently the rdma_rxe driver attempts to protect atomic responder resources by taking a reference to the qp which is only freed when the resource is recycled for a new read or atomic operation. This means that in normal circumstances there is almost always an extra qp reference once an atomic operation has been executed which prevents cleaning up the qp and associated pd and cqs when the qp is destroyed.

This patch removes the call to rxe_add_ref() in send_atomic_ack() and the call to rxe_drop_ref() in free_rd_atomic_resource(). If the qp is destroyed while a peer is retrying an atomic op it will cause the operation to fail which is acceptable.

Fixes 86af61764151 ("IB/rxe: remove unnecessary skb_clone")
---
 drivers/infiniband/sw/rxe/rxe_qp.c   | 1 -
 drivers/infiniband/sw/rxe/rxe_resp.c | 2 --
 2 files changed, 3 deletions(-)
drivers/infiniband/sw/rxe/rxe_qp.c +0 -1
index 34ae957a315c..b6d83d82e4f9 100644
@@ -136,7 +136,6 @@ static void free_rd_atomic_resources(struct rxe_qp *qp)
 void free_rd_atomic_resource(struct rxe_qp *qp, struct resp_res *res)
 {
 	if (res->type == RXE_ATOMIC_MASK) {
-		rxe_drop_ref(qp);
 		kfree_skb(res->atomic.skb);
 	} else if (res->type == RXE_READ_MASK) {
 		if (res->read.mr)
drivers/infiniband/sw/rxe/rxe_resp.c +0 -2
index 2b220659bddb..39dc39be586e 100644
@@ -966,8 +966,6 @@ static int send_atomic_ack(struct rxe_qp *qp, struct rxe_pkt_info *pkt,
 		goto out;
 	}
 
-	rxe_add_ref(qp);
-
 	res = &qp->resp.resources[qp->resp.res_head];
 	free_rd_atomic_resource(qp, res);
 	rxe_advance_resp_resource(qp);
Signature · 1 line
2.30.2
#1 lore raw
Quoted 38 lines · Bob Pearson

Currently the rdma_rxe driver attempts to protect atomic responder resources by taking a reference to the qp which is only freed when the resource is recycled for a new read or atomic operation. This means that in normal circumstances there is almost always an extra qp reference once an atomic operation has been executed which prevents cleaning up the qp and associated pd and cqs when the qp is destroyed.

This patch removes the call to rxe_add_ref() in send_atomic_ack() and the call to rxe_drop_ref() in free_rd_atomic_resource(). If the qp is destroyed while a peer is retrying an atomic op it will cause the operation to fail which is acceptable.

Fixes 86af61764151 ("IB/rxe: remove unnecessary skb_clone")
---
 drivers/infiniband/sw/rxe/rxe_qp.c   | 1 -
 drivers/infiniband/sw/rxe/rxe_resp.c | 2 --
 2 files changed, 3 deletions(-)
drivers/infiniband/sw/rxe/rxe_qp.c +0 -1
index 34ae957a315c..b6d83d82e4f9 100644
@@ -136,7 +136,6 @@ static void free_rd_atomic_resources(struct rxe_qp *qp)
 void free_rd_atomic_resource(struct rxe_qp *qp, struct resp_res *res)
 {
 	if (res->type == RXE_ATOMIC_MASK) {
-		rxe_drop_ref(qp);
 		kfree_skb(res->atomic.skb);
 	} else if (res->type == RXE_READ_MASK) {
 		if (res->read.mr)
drivers/infiniband/sw/rxe/rxe_resp.c +0 -0
index 2b220659bddb..39dc39be586e 100644

drivers/infiniband/sw/rxe/rxe_resp.c static int send_atomic_ack(struct rxe_qp *qp, struct rxe_pkt_info *pkt, · line 971

Quoted 8 lines
drivers/infiniband/sw/rxe/rxe_resp.c +0 -2
@@ -966,8 +966,6 @@ static int send_atomic_ack(struct rxe_qp *qp, struct rxe_pkt_info *pkt,
 		goto out;
 	}
 
-	rxe_add_ref(qp);
-
 	res = &qp->resp.resources[qp->resp.res_head];
 	free_rd_atomic_resource(qp, res);
 	rxe_advance_resp_resource(qp);

There is another way to fix this which is to add cleanup_rd_atomic_resources to rxe_qp_destroy() but it has the exact same behavior because it removes the responder resource causing a retried operation to fail. You can't add it to the qp cleanup routines because you never get there until the ref count goes to zero. I think the above is the cleanest solution.

Bob

maintainer
#2 lore raw
In reply to Bob Pearson
Quoted 10 lines · Bob Pearson

Currently the rdma_rxe driver attempts to protect atomic responder resources by taking a reference to the qp which is only freed when the resource is recycled for a new read or atomic operation. This means that in normal circumstances there is almost always an extra qp reference once an atomic operation has been executed which prevents cleaning up the qp and associated pd and cqs when the qp is destroyed.

This patch removes the call to rxe_add_ref() in send_atomic_ack() and the call to rxe_drop_ref() in free_rd_atomic_resource(). If the qp is

Not sure if it is a good way to fix this problem by removing the call to rxe_add_ref.
Because taking a reference to the qp is to protect atomic responder resources.

Removing rxe_add_ref is to decrease the protection of the atomic responder resources.

Zhu Yanjun

Quoted 39 lines

destroyed while a peer is retrying an atomic op it will cause the operation to fail which is acceptable.

Fixes 86af61764151 ("IB/rxe: remove unnecessary skb_clone")
---
 drivers/infiniband/sw/rxe/rxe_qp.c   | 1 -
 drivers/infiniband/sw/rxe/rxe_resp.c | 2 --
 2 files changed, 3 deletions(-)
drivers/infiniband/sw/rxe/rxe_qp.c +0 -1
index 34ae957a315c..b6d83d82e4f9 100644
@@ -136,7 +136,6 @@ static void free_rd_atomic_resources(struct rxe_qp *qp)
 void free_rd_atomic_resource(struct rxe_qp *qp, struct resp_res *res)
 {
        if (res->type == RXE_ATOMIC_MASK) {
-               rxe_drop_ref(qp);
                kfree_skb(res->atomic.skb);
        } else if (res->type == RXE_READ_MASK) {
                if (res->read.mr)
drivers/infiniband/sw/rxe/rxe_resp.c +0 -3
index 2b220659bddb..39dc39be586e 100644
@@ -966,8 +966,6 @@ static int send_atomic_ack(struct rxe_qp *qp, struct rxe_pkt_info *pkt,
                goto out;
        }
 
-       rxe_add_ref(qp);
-
        res = &qp->resp.resources[qp->resp.res_head];
        free_rd_atomic_resource(qp, res);
        rxe_advance_resp_resource(qp);
--

2.30.2

#3 lore raw
Quoted 12 lines · Zhu Yanjun
Quoted 10 lines · Bob Pearson

Currently the rdma_rxe driver attempts to protect atomic responder resources by taking a reference to the qp which is only freed when the resource is recycled for a new read or atomic operation. This means that in normal circumstances there is almost always an extra qp reference once an atomic operation has been executed which prevents cleaning up the qp and associated pd and cqs when the qp is destroyed.

This patch removes the call to rxe_add_ref() in send_atomic_ack() and the call to rxe_drop_ref() in free_rd_atomic_resource(). If the qp is

Quoted 6 lines

Not sure if it is a good way to fix this problem by removing the call to rxe_add_ref.
Because taking a reference to the qp is to protect atomic responder resources.

Removing rxe_add_ref is to decrease the protection of the atomic responder resources.

All those rxe_add_ref/rxe_drop_ref in RXE are horrid. It will be good to delete them all.

Thanks

maintainer
#4 lore raw
Quoted 14 lines · Leon Romanovsky
Quoted 12 lines · Zhu Yanjun
Quoted 10 lines · Bob Pearson

Currently the rdma_rxe driver attempts to protect atomic responder resources by taking a reference to the qp which is only freed when the resource is recycled for a new read or atomic operation. This means that in normal circumstances there is almost always an extra qp reference once an atomic operation has been executed which prevents cleaning up the qp and associated pd and cqs when the qp is destroyed.

This patch removes the call to rxe_add_ref() in send_atomic_ack() and the call to rxe_drop_ref() in free_rd_atomic_resource(). If the qp is

Quoted 9 lines
Quoted 6 lines

Not sure if it is a good way to fix this problem by removing the call to rxe_add_ref.
Because taking a reference to the qp is to protect atomic responder resources.

Removing rxe_add_ref is to decrease the protection of the atomic responder resources.

All those rxe_add_ref/rxe_drop_ref in RXE are horrid. It will be good to delete them all.

I made tests with this commit. After this commit is applied, this problem disappeared.

Zhu Yanjun

Quoted 1 line

Thanks

#5 lore raw
Quoted 16 lines · Zhu Yanjun
Quoted 15 lines · Leon Romanovsky
Quoted 14 lines · Zhu Yanjun
Quoted 9 lines · Bob Pearson

Currently the rdma_rxe driver attempts to protect atomic responder resources by taking a reference to the qp which is only freed when the resource is recycled for a new read or atomic operation. This means that in normal circumstances there is almost always an extra qp reference once an atomic operation has been executed which prevents cleaning up the qp and associated pd and cqs when the qp is destroyed.

This patch removes the call to rxe_add_ref() in send_atomic_ack() and the call to rxe_drop_ref() in free_rd_atomic_resource(). If the qp is

Not sure if it is a good way to fix this problem by removing the call to rxe_add_ref.
Because taking a reference to the qp is to protect atomic responder resources.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

Removing rxe_add_ref is to decrease the protection of the atomic responder resources.

All those rxe_add_ref/rxe_drop_ref in RXE are horrid. It will be good to delete them all.

I made tests with this commit. After this commit is applied, this problem disappeared.

You were testing MW when you saw this bug. Does that mean that now MW is working for you?

Quoted 4 lines

Zhu Yanjun

Quoted 1 line

Thanks

maintainer
#6 lore raw
Quoted 19 lines · Robert B
Quoted 16 lines · Zhu Yanjun
Quoted 15 lines · Leon Romanovsky
Quoted 14 lines · Zhu Yanjun
Quoted 9 lines · Bob Pearson

Currently the rdma_rxe driver attempts to protect atomic responder resources by taking a reference to the qp which is only freed when the resource is recycled for a new read or atomic operation. This means that in normal circumstances there is almost always an extra qp reference once an atomic operation has been executed which prevents cleaning up the qp and associated pd and cqs when the qp is destroyed.

This patch removes the call to rxe_add_ref() in send_atomic_ack() and the call to rxe_drop_ref() in free_rd_atomic_resource(). If the qp is

Not sure if it is a good way to fix this problem by removing the call to rxe_add_ref.
Because taking a reference to the qp is to protect atomic responder resources.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

Removing rxe_add_ref is to decrease the protection of the atomic responder resources.

All those rxe_add_ref/rxe_drop_ref in RXE are horrid. It will be good to delete them all.

I made tests with this commit. After this commit is applied, this problem disappeared.

You were testing MW when you saw this bug. Does that mean that now MW is working for you?

Your MW patches are huge. After these patches are applied, I found 2 problems in my test environment.
So IMO, can you send the test cases about MW to rdma-core? So we can verify these MW patches with them.

In previous mails, you mentioned these MW test cases.

Thanks a lot.
Zhu Yanjun

Quoted 4 lines
Quoted 4 lines

Zhu Yanjun

Quoted 1 line

Thanks

#7 lore raw
Quoted 21 lines · Zhu Yanjun
Quoted 20 lines · Robert B
Quoted 18 lines · Zhu Yanjun
Quoted 17 lines · Leon Romanovsky
Quoted 16 lines · Zhu Yanjun
Quoted 9 lines · Bob Pearson

Currently the rdma_rxe driver attempts to protect atomic responder resources by taking a reference to the qp which is only freed when the resource is recycled for a new read or atomic operation. This means that in normal circumstances there is almost always an extra qp reference once an atomic operation has been executed which prevents cleaning up the qp and associated pd and cqs when the qp is destroyed.

This patch removes the call to rxe_add_ref() in send_atomic_ack() and the call to rxe_drop_ref() in free_rd_atomic_resource(). If the qp is

Not sure if it is a good way to fix this problem by removing the call to rxe_add_ref.
Because taking a reference to the qp is to protect atomic responder resources.

Removing rxe_add_ref is to decrease the protection of the atomic responder resources.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

All those rxe_add_ref/rxe_drop_ref in RXE are horrid. It will be good to delete them all.

I made tests with this commit. After this commit is applied, this problem disappeared.

You were testing MW when you saw this bug. Does that mean that now MW is working for you?

Your MW patches are huge. After these patches are applied, I found 2 problems in my test environment.

The trace you showed looked like the pyverbs tests all passed and then there were leaked QP/PD/CQ. I also saw those. After fixing the QP reference count bug (not in MW) I did not see any errors from the pyverbs tests of MW. Or any other errors for that matter. What was the other problem? Was that the memory barrier one (also not in MW)?

Mostly I want to know if you currently see any errors in the kernel related to MW. The test case bug (in test_qpex.py) is a separate issue that is not a rxe bug at all.

Bob

Quoted 11 lines

So IMO, can you send the test cases about MW to rdma-core? So we can verify these MW patches with them.

In previous mails, you mentioned these MW test cases.

Thanks a lot.
Zhu Yanjun

Quoted 3 lines
Quoted 3 lines

Zhu Yanjun

Quoted 1 line

Thanks

maintainer
#8 lore raw
Quoted 33 lines · Robert B
Quoted 21 lines · Zhu Yanjun
Quoted 20 lines · Robert B
Quoted 18 lines · Zhu Yanjun
Quoted 17 lines · Leon Romanovsky
Quoted 16 lines · Zhu Yanjun
Quoted 9 lines · Bob Pearson

Currently the rdma_rxe driver attempts to protect atomic responder resources by taking a reference to the qp which is only freed when the resource is recycled for a new read or atomic operation. This means that in normal circumstances there is almost always an extra qp reference once an atomic operation has been executed which prevents cleaning up the qp and associated pd and cqs when the qp is destroyed.

This patch removes the call to rxe_add_ref() in send_atomic_ack() and the call to rxe_drop_ref() in free_rd_atomic_resource(). If the qp is

Not sure if it is a good way to fix this problem by removing the call to rxe_add_ref.
Because taking a reference to the qp is to protect atomic responder resources.

Removing rxe_add_ref is to decrease the protection of the atomic responder resources.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

All those rxe_add_ref/rxe_drop_ref in RXE are horrid. It will be good to delete them all.

I made tests with this commit. After this commit is applied, this problem disappeared.

You were testing MW when you saw this bug. Does that mean that now MW is working for you?

Your MW patches are huge. After these patches are applied, I found 2 problems in my test environment.

Quoted 8 lines

The trace you showed looked like the pyverbs tests all passed and then there were leaked QP/PD/CQ. I also saw those. After fixing the QP reference count bug (not in MW) I did not see any errors from the pyverbs tests of MW. Or any other errors for that matter. What was the other problem? Was that the memory barrier one (also not in MW)?

Mostly I want to know if you currently see any errors in the kernel related to MW. The test case bug (in test_qpex.py) is a separate issue

The current test cases in rdma-core just confirm a regression in RXE.

Zhu Yanjun

Quoted 15 lines

that is not a rxe bug at all.

Bob

Quoted 11 lines

So IMO, can you send the test cases about MW to rdma-core? So we can verify these MW patches with them.

In previous mails, you mentioned these MW test cases.

Thanks a lot.
Zhu Yanjun

Quoted 3 lines
Quoted 3 lines

Zhu Yanjun

Quoted 1 line

Thanks

#9 lore raw
Quoted 37 lines · Zhu Yanjun
Quoted 36 lines · Robert B
Quoted 20 lines · Zhu Yanjun
Quoted 19 lines · Robert B
Quoted 18 lines · Zhu Yanjun
Quoted 17 lines · Leon Romanovsky
Quoted 16 lines · Zhu Yanjun
Quoted 9 lines · Bob Pearson

Currently the rdma_rxe driver attempts to protect atomic responder resources by taking a reference to the qp which is only freed when the resource is recycled for a new read or atomic operation. This means that in normal circumstances there is almost always an extra qp reference once an atomic operation has been executed which prevents cleaning up the qp and associated pd and cqs when the qp is destroyed.

This patch removes the call to rxe_add_ref() in send_atomic_ack() and the call to rxe_drop_ref() in free_rd_atomic_resource(). If the qp is

Not sure if it is a good way to fix this problem by removing the call to rxe_add_ref.
Because taking a reference to the qp is to protect atomic responder resources.

Removing rxe_add_ref is to decrease the protection of the atomic responder resources.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

All those rxe_add_ref/rxe_drop_ref in RXE are horrid. It will be good to delete them all.

I made tests with this commit. After this commit is applied, this problem disappeared.

You were testing MW when you saw this bug. Does that mean that now MW is working for you?

Your MW patches are huge. After these patches are applied, I found 2 problems in my test environment.

The trace you showed looked like the pyverbs tests all passed and then there were leaked QP/PD/CQ. I also saw those. After fixing the QP reference count bug (not in MW) I did not see any errors from the pyverbs tests of MW. Or any other errors for that matter. What was the other problem? Was that the memory barrier one (also not in MW)?

Quoted 5 lines
Quoted 2 lines

Mostly I want to know if you currently see any errors in the kernel related to MW. The test case bug (in test_qpex.py) is a separate issue

The current test cases in rdma-core just confirm a regression in RXE.

Zhu Yanjun

Which test cases are you referring to. Currently all test cases either pass or are skipped because they are not supported with one single exception. That test in test_qpex.py is *not* a regression. It used to skip until I added support for the extended MW bind operation to the user code today. It now fails because the test is actually wrong. It didn't set the access flags for the MR to support bind MW so the driver fails the WR with a bind MW error which is the correct behavior. The traditional QP WR API (ibv_post_send) exercises the same exact functionality of the driver and they all set the MR access correctly and pass. There are *no* actual errors being reported by the rdma-core tests.

Bob

Quoted 16 lines
Quoted 16 lines

that is not a rxe bug at all.

Bob

Quoted 11 lines

So IMO, can you send the test cases about MW to rdma-core? So we can verify these MW patches with them.

In previous mails, you mentioned these MW test cases.

Thanks a lot.
Zhu Yanjun

Quoted 3 lines
Quoted 3 lines

Zhu Yanjun

Quoted 1 line

Thanks

#10 lore raw
In reply to Zhu Yanjun
Quoted 8 lines · Zhu Yanjun
Quoted 7 lines · Bob Pearson

Currently the rdma_rxe driver attempts to protect atomic responder resources by taking a reference to the qp which is only freed when the resource is recycled for a new read or atomic operation. This means that in normal circumstances there is almost always an extra qp reference once an atomic operation has been executed which prevents cleaning up the qp and associated pd and cqs when the qp is destroyed.

Quoted 5 lines
Quoted 2 lines

This patch removes the call to rxe_add_ref() in send_atomic_ack() and the call to rxe_drop_ref() in free_rd_atomic_resource(). If the qp is

Not sure if it is a good way to fix this problem by removing the call to rxe_add_ref.
Because taking a reference to the qp is to protect atomic responder resources.

There is no good way to 'protect' responder resources. They are there to recover from a lost response packet which can occur multiple times. The peer can retry the atomic request an unpredictable number of times. There are no acks for response packets. So the QP just has to leave it in place until the max number of read/atomic requests has been received and then it re-uses the responder resource. So to 'protect' the responder resource means you have to leave the QP around potentially forever which is the root cause of the bug. In fact you should treat a retry of a read/atomic the same as a new request and when the QP is destroyed it stops responding to new requests.

There is a reference to the QP taken by a received packet which lasts until the packet is freed to prevent kernel seg faults if the QP is destroyed while a request is in process of being responded to.

Quoted 45 lines

Removing rxe_add_ref is to decrease the protection of the atomic responder resources.

Zhu Yanjun

Quoted 39 lines

destroyed while a peer is retrying an atomic op it will cause the operation to fail which is acceptable.

Fixes 86af61764151 ("IB/rxe: remove unnecessary skb_clone")
---
  drivers/infiniband/sw/rxe/rxe_qp.c   | 1 -
  drivers/infiniband/sw/rxe/rxe_resp.c | 2 --
  2 files changed, 3 deletions(-)
drivers/infiniband/sw/rxe/rxe_qp.c +0 -1
index 34ae957a315c..b6d83d82e4f9 100644
@@ -136,7 +136,6 @@ static void free_rd_atomic_resources(struct rxe_qp *qp)
  void free_rd_atomic_resource(struct rxe_qp *qp, struct resp_res *res)
  {
         if (res->type == RXE_ATOMIC_MASK) {
-               rxe_drop_ref(qp);
                 kfree_skb(res->atomic.skb);
         } else if (res->type == RXE_READ_MASK) {
                 if (res->read.mr)
drivers/infiniband/sw/rxe/rxe_resp.c +0 -3
index 2b220659bddb..39dc39be586e 100644
@@ -966,8 +966,6 @@ static int send_atomic_ack(struct rxe_qp *qp, struct rxe_pkt_info *pkt,
                 goto out;
         }
 
-       rxe_add_ref(qp);
-
         res = &qp->resp.resources[qp->resp.res_head];
         free_rd_atomic_resource(qp, res);
         rxe_advance_resp_resource(qp);
--

2.30.2

maintainer Applied
#11 lore raw
In reply to Bob Pearson
Quoted 12 lines · Bob Pearson

Currently the rdma_rxe driver attempts to protect atomic responder resources by taking a reference to the qp which is only freed when the resource is recycled for a new read or atomic operation. This means that in normal circumstances there is almost always an extra qp reference once an atomic operation has been executed which prevents cleaning up the qp and associated pd and cqs when the qp is destroyed.

This patch removes the call to rxe_add_ref() in send_atomic_ack() and the call to rxe_drop_ref() in free_rd_atomic_resource(). If the qp is destroyed while a peer is retrying an atomic op it will cause the operation to fail which is acceptable.

Quoted 7 lines
Fixes 86af61764151 ("IB/rxe: remove unnecessary skb_clone")
---
 drivers/infiniband/sw/rxe/rxe_qp.c   | 1 -
 drivers/infiniband/sw/rxe/rxe_resp.c | 2 --
 2 files changed, 3 deletions(-)

Applied to for-next, thanks

Jason