Skip to contentSkip to conversation

tests: Fix test_mr_rereg_pd

RFC PATCH In review Bob Pearson 2 participants 3 messages May 25, 2023 – May 29, 2023 InfiniBand

lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 tests: Fix test_mr_rereg_pd

This patch adds a util method drain_cq which drains out the cq associated with a client or server. This is then added to the method restate_qps in tests/test_mr.py. This allows correct operation when recovering test state from an error which may have also left stray completions in the cqs before resetting the qps for use.

Fixes 4bc72d894481 ("tests: Add rereg MR tests")
---
 tests/test_mr.py |  2 ++
 tests/utils.py   | 14 ++++++++++++++
 2 files changed, 16 insertions(+)
tests/test_mr.py +2 -0
index 534df46a..73dfbff2 100644
@@ -109,6 +109,8 @@ class MRTest(RDMATestCase):
         self.server.qp.to_rts(self.server_qp_attr)
         self.client.qp.modify(QPAttr(qp_state=e.IBV_QPS_RESET), e.IBV_QP_STATE)
         self.client.qp.to_rts(self.client_qp_attr)
+        u.drain_cq(self.client.cq)
+        u.drain_cq(self.server.cq)
 
     def test_mr_rereg_access(self):
         self.create_players(MRRes)
tests/utils.py +14 -0
index a1dfa7d8..f6966b1a 100644
@@ -672,6 +672,20 @@ def poll_cq_ex(cqex, count=1, data=None, sgid=None):
     finally:
         cqex.end_poll()
 
+def drain_cq(cq):
+    """
+    Drain completions from a CQ.
+    :param cq: CQ to drain
+    :return: None
+    """
+    channel = cq.comp_channel
+    while 1:
+        if channel:
+            channel.get_cq_event(cq)
+            cq.req_notify()
+        nc, tmp_wcs = cq.poll(1)
+        if nc == 0:
+            break
 
 def validate(received_str, is_server, msg_size):
     """
Signature · 1 line
2.39.2
#1 lore raw

I see that the traffic() function used in this test posts just one receive/send WR each iteration. Meaning that once the first CQE with error occurs, there are no more CQEs left to drain.

If that is the case, I'm not sure why you still see stray completions in the CQ.

Quoted 35 lines · Bob Pearson

External email: Use caution opening links or attachments

This patch adds a util method drain_cq which drains out the cq associated with a client or server. This is then added to the method restate_qps in tests/test_mr.py. This allows correct operation when recovering test state from an error which may have also left stray completions in the cqs before resetting the qps for use.

Fixes 4bc72d894481 ("tests: Add rereg MR tests")
---
  tests/test_mr.py |  2 ++
  tests/utils.py   | 14 ++++++++++++++
  2 files changed, 16 insertions(+)
tests/test_mr.py +2 -0
index 534df46a..73dfbff2 100644
@@ -109,6 +109,8 @@ class MRTest(RDMATestCase):
          self.server.qp.to_rts(self.server_qp_attr)
          self.client.qp.modify(QPAttr(qp_state=e.IBV_QPS_RESET), e.IBV_QP_STATE)
          self.client.qp.to_rts(self.client_qp_attr)
+        u.drain_cq(self.client.cq)
+        u.drain_cq(self.server.cq)
 
      def test_mr_rereg_access(self):
          self.create_players(MRRes)
tests/utils.py +0 -0
index a1dfa7d8..f6966b1a 100644

tests/utils.py def poll_cq_ex(cqex, count=1, data=None, sgid=None): · line 673

Quoted 2 lines
tests/utils.py +0 -0
@@ -672,6 +672,20 @@ def poll_cq_ex(cqex, count=1, data=None, sgid=None):
      finally:
          cqex.end_poll()

Keep two blank lines before and after module-level function definition (PEP-8 convention)

tests/utils.py def poll_cq_ex(cqex, count=1, data=None, sgid=None): · line 686

Quoted 12 lines
tests/utils.py +12 -0
+def drain_cq(cq):
+    """
+    Drain completions from a CQ.
+    :param cq: CQ to drain
+    :return: None
+    """
+    channel = cq.comp_channel
+    while 1:
+        if channel:
+            channel.get_cq_event(cq)
+            cq.req_notify()
+        nc, tmp_wcs = cq.poll(1)

tmp_wcs is unused. You can do this instead:

nc, _ = cq.poll(1)

Quoted 8 lines
tests/utils.py +2 -1
+        if nc == 0:
+            break
 
  def validate(received_str, is_server, msg_size):
      """
--

2.39.2

#2 lore raw
Quoted 2 lines · Edward Srouji

I see that the traffic() function used in this test posts just one receive/send WR each iteration. Meaning that once the first CQE with error occurs, there are no more CQEs left to drain.

I believe it is the receive WQEs that are getting flushed.

Quoted 55 lines

If that is the case, I'm not sure why you still see stray completions in the CQ.

Quoted 35 lines · Bob Pearson

External email: Use caution opening links or attachments

This patch adds a util method drain_cq which drains out the cq associated with a client or server. This is then added to the method restate_qps in tests/test_mr.py. This allows correct operation when recovering test state from an error which may have also left stray completions in the cqs before resetting the qps for use.

Fixes 4bc72d894481 ("tests: Add rereg MR tests")

---
  tests/test_mr.py |  2 ++
  tests/utils.py   | 14 ++++++++++++++

  2 files changed, 16 insertions(+)
tests/test_mr.py +2 -0
index 534df46a..73dfbff2 100644
@@ -109,6 +109,8 @@ class MRTest(RDMATestCase):
           self.server.qp.to_rts(self.server_qp_attr)
           self.client.qp.modify(QPAttr(qp_state=e.IBV_QPS_RESET), e.IBV_QP_STATE)
           self.client.qp.to_rts(self.client_qp_attr)
+        u.drain_cq(self.client.cq)
+        u.drain_cq(self.server.cq)
 

      def test_mr_rereg_access(self):
          self.create_players(MRRes)

tests/utils.py +0 -0
index a1dfa7d8..f6966b1a 100644
@@ -672,6 +672,20 @@ def poll_cq_ex(cqex, count=1, data=None, sgid=None):
Quoted 2 lines

      finally:
          cqex.end_poll()

Keep two blank lines before and after module-level function definition (PEP-8 convention)

Quoted 4 lines

+def drain_cq(cq):
+    """
+    Drain completions from a CQ.
+    :param cq: CQ to drain

Quoted 8 lines

+    :return: None
+    """
+    channel = cq.comp_channel
+    while 1:
+        if channel:
+            channel.get_cq_event(cq)
+            cq.req_notify()
+        nc, tmp_wcs = cq.poll(1)

Quoted 12 lines

tmp_wcs is unused. You can do this instead:

nc, _ = cq.poll(1)

Quoted 8 lines

+        if nc == 0:
+            break

  def validate(received_str, is_server, msg_size):
      """

Signature · 1 line
2.39.2

Thanks, I can fix those.

Bob