Skip to contentSkip to conversation

RDMA/core: test_qpex.py attempts invalid MW bind operation

Bug Report Pearson, Robert B 3 participants 12 messages Jun 7, 2021 – Jun 8, 2021 InfiniBand

lore
Share & tools
lore
#0 lore raw

sorry/this time without the HTML.

======================================================================
ERROR: test_qp_ex_rc_bind_mw (tests.test_qpex.QpExTestCase)
Verify bind memory window operation using the new post_send API.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/rpearson/src/rdma-core/tests/test_qpex.py", line 292, in test_qp_ex_rc_bind_mw u.poll_cq(server.cq)
  File "/home/rpearson/src/rdma-core/tests/utils.py", line 538, in poll_cq
    raise PyverbsRDMAError('Completion status is {s}'.
pyverbs.pyverbs_error.PyverbsRDMAError: Completion status is Memory window bind error. Errno: 6, No such device or address

This test attempts to bind a type 2 MW to an MR that does not have bind mw access set and expects the test to succeed.

Bob Pearson

#1 lore raw
Quoted 15 lines · Robert B

sorry/this time without the HTML.

======================================================================
ERROR: test_qp_ex_rc_bind_mw (tests.test_qpex.QpExTestCase) Verify bind memory window operation using the new post_send API.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/rpearson/src/rdma-core/tests/test_qpex.py", line 292, in test_qp_ex_rc_bind_mw
    u.poll_cq(server.cq)
  File "/home/rpearson/src/rdma-core/tests/utils.py", line 538, in poll_cq raise PyverbsRDMAError('Completion status is {s}'. pyverbs.pyverbs_error.PyverbsRDMAError: Completion status is Memory window bind error. Errno: 6, No such device or address

Quoted 2 lines

This test attempts to bind a type 2 MW to an MR that does not have bind mw access set and expects the test to succeed.

Does the test break after your MW series? Or will it break not-merged code yet?

Generally speaking, we expect that developers run rdma-core tests and fixed/extend them prior to the submission.

Thanks

Quoted 3 lines

Bob Pearson

#2 lore raw
Quoted 18 lines · Leon Romanovsky
Quoted 15 lines · Robert B

sorry/this time without the HTML.

======================================================================
ERROR: test_qp_ex_rc_bind_mw (tests.test_qpex.QpExTestCase) Verify bind memory window operation using the new post_send API.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/rpearson/src/rdma-core/tests/test_qpex.py", line 292, in
test_qp_ex_rc_bind_mw
    u.poll_cq(server.cq)
  File "/home/rpearson/src/rdma-core/tests/utils.py", line 538, in poll_cq raise PyverbsRDMAError('Completion status is {s}'.
pyverbs.pyverbs_error.PyverbsRDMAError: Completion status is Memory window bind error. Errno: 6, No such device or address

Quoted 2 lines

This test attempts to bind a type 2 MW to an MR that does not have bind mw access set and expects the test to succeed.

Quoted 9 lines

Does the test break after your MW series? Or will it break not-merged code yet?

Generally speaking, we expect that developers run rdma-core tests and fixed/extend them prior to the submission.

Thanks

Quoted 1 line

Bob Pearson

Nope. I don't have real RNICs at home to test. But (see my note to Zhu) the non extended APIs do set the access flags correctly and the extended test case does not. The wr_bind_mw() function can't fix this for the test case. It has to set the access flags when it creates the MR and it didn't. It is possible that mlx5 doesn't check the bind access flag but that seems unlikely.

Bob

#3 lore raw
Quoted 28 lines · Robert B
Quoted 26 lines · Leon Romanovsky
Quoted 15 lines · Robert B

sorry/this time without the HTML.

======================================================================
ERROR: test_qp_ex_rc_bind_mw (tests.test_qpex.QpExTestCase) Verify bind memory window operation using the new post_send API.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/rpearson/src/rdma-core/tests/test_qpex.py", line 292, in
test_qp_ex_rc_bind_mw
    u.poll_cq(server.cq)
  File "/home/rpearson/src/rdma-core/tests/utils.py", line 538, in poll_cq raise PyverbsRDMAError('Completion status is {s}'.
pyverbs.pyverbs_error.PyverbsRDMAError: Completion status is Memory window bind error. Errno: 6, No such device or address

Quoted 2 lines

This test attempts to bind a type 2 MW to an MR that does not have bind mw access set and expects the test to succeed.

Does the test break after your MW series? Or will it break not-merged code yet?

Generally speaking, we expect that developers run rdma-core tests and fixed/extend them prior to the submission.

Thanks

Quoted 8 lines
Quoted 1 line
Quoted 1 line

Bob Pearson

Nope. I don't have real RNICs at home to test. But (see my note to Zhu) the non extended APIs do set the access flags correctly and the extended test case does not. The wr_bind_mw() function can't fix this for the test case. It has to set the access flags when it creates the MR and it didn't. It is possible that mlx5 doesn't check the bind access flag but that seems unlikely.

mlx5 devices support MW 1 & 2 and kernel checks that only these types can be accepted from the user space. This is why mlx5 doesn't need to check access flags again.

   903 static int ib_uverbs_alloc_mw(struct uverbs_attr_bundle *attrs)
   904 {

....

   927         if (cmd.mw_type != IB_MW_TYPE_1 && cmd.mw_type != IB_MW_TYPE_2) {
   928                 ret = -EINVAL;
   929                 goto err_put;
   930         }

Thanks

Quoted 3 lines

Bob

#4 lore raw
Quoted 18 lines · Leon Romanovsky
Quoted 17 lines · Robert B
Quoted 16 lines · Leon Romanovsky
Quoted 15 lines · Robert B

sorry/this time without the HTML.

======================================================================
ERROR: test_qp_ex_rc_bind_mw (tests.test_qpex.QpExTestCase) Verify bind memory window operation using the new post_send API.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/rpearson/src/rdma-core/tests/test_qpex.py", line 292, in
test_qp_ex_rc_bind_mw
    u.poll_cq(server.cq)
  File "/home/rpearson/src/rdma-core/tests/utils.py", line 538, in poll_cq raise PyverbsRDMAError('Completion status is {s}'.
pyverbs.pyverbs_error.PyverbsRDMAError: Completion status is Memory window bind error. Errno: 6, No such device or address

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

This test attempts to bind a type 2 MW to an MR that does not have bind mw access set and expects the test to succeed.

You're right, looks like a test bug. I'll send a fix upstream.

Can you please confirm that this solves your issue:

tests/test_qpex.py +0 -0
index 4b58260f..c2d67ee8 100644
@@ -149,7 +149,7 @@ class QpExRCBindMw(RCResources):

         create_qp_ex(self, e.IBV_QPT_RC, e.IBV_QP_EX_WITH_BIND_MW)

     def create_mr(self):
-        self.mr = u.create_custom_mr(self, e.IBV_ACCESS_REMOTE_WRITE)
+        self.mr = u.create_custom_mr(self, e.IBV_ACCESS_REMOTE_WRITE | e.IBV_ACCESS_MW_BIND)

Quoted 24 lines
Quoted 8 lines
Quoted 8 lines

Does the test break after your MW series? Or will it break not-merged code yet?

Generally speaking, we expect that developers run rdma-core tests and fixed/extend them prior to the submission.

Thanks

Quoted 7 lines
Quoted 1 line
Quoted 1 line

Bob Pearson

Nope. I don't have real RNICs at home to test. But (see my note to Zhu) the non extended APIs do set the access flags correctly and the extended test case does not. The wr_bind_mw() function can't fix this for the test case. It has to set the access flags when it creates the MR and it didn't. It is possible that mlx5 doesn't check the bind access flag but that seems unlikely.

mlx5 devices support MW 1 & 2 and kernel checks that only these types can be accepted from the user space. This is why mlx5 doesn't need to check access flags again.

903 static int ib_uverbs_alloc_mw(struct uverbs_attr_bundle *attrs) 904 {

....

Quoted 7 lines
    927         if (cmd.mw_type != IB_MW_TYPE_1 && cmd.mw_type != IB_MW_TYPE_2) {
    928                 ret = -EINVAL;
    929                 goto err_put;
    930         }

Thanks

I see that mlx5 checks the access flags in userspace only if MW_DEBUG is turned on (in set_bind_wr()).

I guess that's for the sake of performance, as it's part of the data path.

Quoted 2 lines
Quoted 2 lines

Bob

#5 lore raw
Quoted 22 lines · Edward Srouji
Quoted 20 lines · Leon Romanovsky
Quoted 19 lines · Robert B
Quoted 18 lines · Leon Romanovsky
Quoted 17 lines · Robert B

sorry/this time without the HTML.

======================================================================

ERROR: test_qp_ex_rc_bind_mw (tests.test_qpex.QpExTestCase) Verify bind memory window operation using the new post_send API.
----------------------------------------------------------------------

Traceback (most recent call last):
    File "/home/rpearson/src/rdma-core/tests/test_qpex.py", line 292, in test_qp_ex_rc_bind_mw
      u.poll_cq(server.cq)
    File "/home/rpearson/src/rdma-core/tests/utils.py", line 538, in poll_cq raise PyverbsRDMAError('Completion status is {s}'. pyverbs.pyverbs_error.PyverbsRDMAError: Completion status is Memory window bind error. Errno: 6, No such device or address

Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

This test attempts to bind a type 2 MW to an MR that does not have bind mw access set and expects the test to succeed.

You're right, looks like a test bug. I'll send a fix upstream.

Can you please confirm that this solves your issue:

Well I get further. I am hitting a seg fault in python at

        client.qp.wr_rdma_write(new_key, server.mr.buf)

in test_qp_ex_rc_bind_mw.

I'm trying to track it down. I'm not very familiar with python and don't know how to run the test under gdb.

Thanks for the fix.

Bob

Quoted 51 lines
tests/test_qpex.py +0 -0
index 4b58260f..c2d67ee8 100644
@@ -149,7 +149,7 @@ class QpExRCBindMw(RCResources):
          create_qp_ex(self, e.IBV_QPT_RC, e.IBV_QP_EX_WITH_BIND_MW)
 

     def create_mr(self):
-        self.mr = u.create_custom_mr(self, e.IBV_ACCESS_REMOTE_WRITE) +        self.mr = u.create_custom_mr(self, e.IBV_ACCESS_REMOTE_WRITE | e.IBV_ACCESS_MW_BIND)

Quoted 24 lines
Quoted 8 lines
Quoted 8 lines

Does the test break after your MW series? Or will it break not-merged code yet?

Generally speaking, we expect that developers run rdma-core tests and fixed/extend them prior to the submission.

Thanks

Quoted 7 lines
Quoted 1 line
Quoted 1 line

Bob Pearson

Nope. I don't have real RNICs at home to test. But (see my note to Zhu) the non extended APIs do set the access flags correctly and the extended test case does not. The wr_bind_mw() function can't fix this for the test case. It has to set the access flags when it creates the MR and it didn't. It is possible that mlx5 doesn't check the bind access flag but that seems unlikely.

mlx5 devices support MW 1 & 2 and kernel checks that only these types can be accepted from the user space. This is why mlx5 doesn't need to check access flags again.

    903 static int ib_uverbs_alloc_mw(struct uverbs_attr_bundle *attrs) 904 {

....

Quoted 7 lines

    927         if (cmd.mw_type != IB_MW_TYPE_1 && cmd.mw_type != IB_MW_TYPE_2) { 928                 ret = -EINVAL;
    929                 goto err_put;
    930         }

Thanks

I see that mlx5 checks the access flags in userspace only if MW_DEBUG is turned on (in set_bind_wr()).

I guess that's for the sake of performance, as it's part of the data path.

Quoted 2 lines
Quoted 2 lines

Bob

#6 lore raw
Quoted 38 lines · Robert B

External email: Use caution opening links or attachments

Quoted 27 lines · Edward Srouji
Quoted 25 lines · Leon Romanovsky
Quoted 24 lines · Robert B
Quoted 23 lines · Leon Romanovsky
Quoted 22 lines · Robert B

sorry/this time without the HTML.

======================================================================

ERROR: test_qp_ex_rc_bind_mw (tests.test_qpex.QpExTestCase) Verify bind memory window operation using the new post_send API.
----------------------------------------------------------------------

Traceback (most recent call last):
    File "/home/rpearson/src/rdma-core/tests/test_qpex.py", line 292, in
test_qp_ex_rc_bind_mw
      u.poll_cq(server.cq)
    File "/home/rpearson/src/rdma-core/tests/utils.py", line 538, in poll_cq
      raise PyverbsRDMAError('Completion status is {s}'. pyverbs.pyverbs_error.PyverbsRDMAError: Completion status is Memory window
bind error. Errno: 6, No such device or address

Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

This test attempts to bind a type 2 MW to an MR that does not have bind mw
access set and expects the test to succeed.

You're right, looks like a test bug. I'll send a fix upstream.

Can you please confirm that this solves your issue:

Quoted 9 lines

Well I get further. I am hitting a seg fault in python at

        client.qp.wr_rdma_write(new_key, server.mr.buf)

in test_qp_ex_rc_bind_mw.

I'm trying to track it down. I'm not very familiar with python and don't know how to run the test under gdb.

I don't see the issue on mlx devices / rxe.

You can use gdb for python using: "gdb python"

Then : "run <program>" e.g.: "run tests/run_tests.py -k test_qp_ex_rc_bind_mw" (if you ran the command from the rdma-core root dir).

You can set breakpoints for C functions as you regularly do. If you want to add a breakpoint to the python test, I suggest you do "import pdb; pdb.set_trace()" at the line you want to set the bp at.

Good luck.

Quoted 62 lines

Thanks for the fix.

Bob

Quoted 58 lines
tests/test_qpex.py +0 -0
index 4b58260f..c2d67ee8 100644
@@ -149,7 +149,7 @@ class QpExRCBindMw(RCResources):
          create_qp_ex(self, e.IBV_QPT_RC, e.IBV_QP_EX_WITH_BIND_MW)
 

     def create_mr(self):
-        self.mr = u.create_custom_mr(self, e.IBV_ACCESS_REMOTE_WRITE) +        self.mr = u.create_custom_mr(self, e.IBV_ACCESS_REMOTE_WRITE | e.IBV_ACCESS_MW_BIND)

Quoted 28 lines
Quoted 9 lines
Quoted 9 lines

Does the test break after your MW series? Or will it break not-merged code yet?

Generally speaking, we expect that developers run rdma-core tests and fixed/extend them prior to the submission.

Thanks

Quoted 1 line

Bob Pearson

Quoted 10 lines

Nope. I don't have real RNICs at home to test. But (see my note to Zhu) the
non extended APIs do set the access flags correctly and the extended test
case does not. The wr_bind_mw() function can't fix this for the test case.
It has to set the access flags when it creates the MR and it didn't. It is
possible that mlx5 doesn't check the bind access flag but that seems unlikely.

mlx5 devices support MW 1 & 2 and kernel checks that only these types can be accepted from the user space. This is why mlx5 doesn't need to check access flags again.

    903 static int ib_uverbs_alloc_mw(struct uverbs_attr_bundle *attrs) 904 {

....

Quoted 8 lines

    927         if (cmd.mw_type != IB_MW_TYPE_1 && cmd.mw_type != IB_MW_TYPE_2) {
    928                 ret = -EINVAL;
    929                 goto err_put;
    930         }

Thanks

I see that mlx5 checks the access flags in userspace only if MW_DEBUG is turned on (in set_bind_wr()).

I guess that's for the sake of performance, as it's part of the data path.

Quoted 2 lines
Quoted 2 lines

Bob

#7 lore raw
In reply to Pearson, Robert B
Quoted 34 lines · Robert B
Quoted 22 lines · Edward Srouji
Quoted 20 lines · Leon Romanovsky
Quoted 19 lines · Robert B
Quoted 18 lines · Leon Romanovsky
Quoted 17 lines · Robert B

sorry/this time without the HTML.

======================================================================

ERROR: test_qp_ex_rc_bind_mw (tests.test_qpex.QpExTestCase) Verify bind memory window operation using the new post_send API.
----------------------------------------------------------------------

Traceback (most recent call last):
    File "/home/rpearson/src/rdma-core/tests/test_qpex.py", line 292, in test_qp_ex_rc_bind_mw
      u.poll_cq(server.cq)
    File "/home/rpearson/src/rdma-core/tests/utils.py", line 538, in poll_cq raise PyverbsRDMAError('Completion status is {s}'. pyverbs.pyverbs_error.PyverbsRDMAError: Completion status is Memory window bind error. Errno: 6, No such device or address

Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

This test attempts to bind a type 2 MW to an MR that does not have bind mw access set and expects the test to succeed.

You're right, looks like a test bug. I'll send a fix upstream.

Can you please confirm that this solves your issue:

Well I get further. I am hitting a seg fault in python at

        client.qp.wr_rdma_write(new_key, server.mr.buf)

Quoted 7 lines

in test_qp_ex_rc_bind_mw.

I'm trying to track it down. I'm not very familiar with python and don't know how to run the test under gdb.

Thanks for the fix.

Bob

OK got it. In the setup for the test you write

    class QpExRCBindMw(RCResources):
        def create_qps(self):
            create_qp_ex(self, e.IBV_QPT_RC, e.IBV_QP_EX_WITH_BIND_MW)

        def create_mr(self):
            self.mr = u.create_custom_mr(self, e.IBV_ACCESS_REMOTE_WRITE | e.IBV_ACCESS_MW_BIND)

which asks for qp_ex->wr_bind_mw() to be set but later in the test you write

    client.qp.wr_rdma_write(new_key, server.mr.buf)

which calls qp_ex->wr_rdma_write() which is not set causing the seg fault. I think you should have written

            create_qp_ex(self, e.IBV_QPT_RC, e.IBV_QP_EX_WITH_BIND_MW | e.IBV_QP_EX_WITH_RDMA_WRITE)

since you need both extended QP operations.

Bob

Quoted 52 lines
Quoted 52 lines
tests/test_qpex.py +0 -0
index 4b58260f..c2d67ee8 100644
@@ -149,7 +149,7 @@ class QpExRCBindMw(RCResources):
          create_qp_ex(self, e.IBV_QPT_RC, e.IBV_QP_EX_WITH_BIND_MW)
 

     def create_mr(self):
-        self.mr = u.create_custom_mr(self, e.IBV_ACCESS_REMOTE_WRITE) +        self.mr = u.create_custom_mr(self, e.IBV_ACCESS_REMOTE_WRITE | e.IBV_ACCESS_MW_BIND)

Quoted 24 lines
Quoted 8 lines
Quoted 8 lines

Does the test break after your MW series? Or will it break not-merged code yet?

Generally speaking, we expect that developers run rdma-core tests and fixed/extend them prior to the submission.

Thanks

Quoted 7 lines
Quoted 1 line
Quoted 1 line

Bob Pearson

Nope. I don't have real RNICs at home to test. But (see my note to Zhu) the non extended APIs do set the access flags correctly and the extended test case does not. The wr_bind_mw() function can't fix this for the test case. It has to set the access flags when it creates the MR and it didn't. It is possible that mlx5 doesn't check the bind access flag but that seems unlikely.

mlx5 devices support MW 1 & 2 and kernel checks that only these types can be accepted from the user space. This is why mlx5 doesn't need to check access flags again.

    903 static int ib_uverbs_alloc_mw(struct uverbs_attr_bundle *attrs) 904 {

....

Quoted 7 lines

    927         if (cmd.mw_type != IB_MW_TYPE_1 && cmd.mw_type != IB_MW_TYPE_2) { 928                 ret = -EINVAL;
    929                 goto err_put;
    930         }

Thanks

I see that mlx5 checks the access flags in userspace only if MW_DEBUG is turned on (in set_bind_wr()).

I guess that's for the sake of performance, as it's part of the data path.

Quoted 2 lines
Quoted 2 lines

Bob

#8 lore raw
Quoted 58 lines · Robert B
Quoted 34 lines · Robert B
Quoted 22 lines · Edward Srouji
Quoted 20 lines · Leon Romanovsky
Quoted 19 lines · Robert B
Quoted 18 lines · Leon Romanovsky
Quoted 17 lines · Robert B

sorry/this time without the HTML.

======================================================================

ERROR: test_qp_ex_rc_bind_mw (tests.test_qpex.QpExTestCase) Verify bind memory window operation using the new post_send API.
----------------------------------------------------------------------

Traceback (most recent call last):
    File "/home/rpearson/src/rdma-core/tests/test_qpex.py", line 292, in test_qp_ex_rc_bind_mw
      u.poll_cq(server.cq)
    File "/home/rpearson/src/rdma-core/tests/utils.py", line 538, in poll_cq raise PyverbsRDMAError('Completion status is {s}'. pyverbs.pyverbs_error.PyverbsRDMAError: Completion status is Memory window bind error. Errno: 6, No such device or address

Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

This test attempts to bind a type 2 MW to an MR that does not have bind mw access set and expects the test to succeed.

You're right, looks like a test bug. I'll send a fix upstream.

Can you please confirm that this solves your issue:

Well I get further. I am hitting a seg fault in python at

        client.qp.wr_rdma_write(new_key, server.mr.buf)

Quoted 7 lines

in test_qp_ex_rc_bind_mw.

I'm trying to track it down. I'm not very familiar with python and don't know how to run the test under gdb.

Thanks for the fix.

Bob

OK got it. In the setup for the test you write

    class QpExRCBindMw(RCResources):
        def create_qps(self):
            create_qp_ex(self, e.IBV_QPT_RC, e.IBV_QP_EX_WITH_BIND_MW)

        def create_mr(self):
            self.mr = u.create_custom_mr(self, e.IBV_ACCESS_REMOTE_WRITE | e.IBV_ACCESS_MW_BIND)

which asks for qp_ex->wr_bind_mw() to be set but later in the test you write

    client.qp.wr_rdma_write(new_key, server.mr.buf)

Quoted 7 lines

which calls qp_ex->wr_rdma_write() which is not set causing the seg fault. I think you should have written

            create_qp_ex(self, e.IBV_QPT_RC, e.IBV_QP_EX_WITH_BIND_MW | e.IBV_QP_EX_WITH_RDMA_WRITE)

since you need both extended QP operations.

Bob

With this patch the test is now running correctly

tests/test_qpex.py +0 -0
index 20288d45..0316bfcb 100644
@@ -146,10 +146,12 @@ class QpExRCAtomicFetchAdd(RCResources):
 

 class QpExRCBindMw(RCResources):
     def create_qps(self):
-        create_qp_ex(self, e.IBV_QPT_RC, e.IBV_QP_EX_WITH_BIND_MW) +        create_qp_ex(self, e.IBV_QPT_RC, e.IBV_QP_EX_WITH_BIND_MW | +                       e.IBV_QP_EX_WITH_RDMA_WRITE)

     def create_mr(self):
-        self.mr = u.create_custom_mr(self, e.IBV_ACCESS_REMOTE_WRITE) +        self.mr = u.create_custom_mr(self, e.IBV_ACCESS_REMOTE_WRITE | +                       e.IBV_ACCESS_MW_BIND)

 class QpExTestCase(RDMATestCase):

Quoted 53 lines
Quoted 53 lines
Quoted 53 lines
tests/test_qpex.py +0 -0
index 4b58260f..c2d67ee8 100644
@@ -149,7 +149,7 @@ class QpExRCBindMw(RCResources):
          create_qp_ex(self, e.IBV_QPT_RC, e.IBV_QP_EX_WITH_BIND_MW)
 

     def create_mr(self):
-        self.mr = u.create_custom_mr(self, e.IBV_ACCESS_REMOTE_WRITE) +        self.mr = u.create_custom_mr(self, e.IBV_ACCESS_REMOTE_WRITE | e.IBV_ACCESS_MW_BIND)

Quoted 24 lines
Quoted 8 lines
Quoted 8 lines

Does the test break after your MW series? Or will it break not-merged code yet?

Generally speaking, we expect that developers run rdma-core tests and fixed/extend them prior to the submission.

Thanks

Quoted 7 lines
Quoted 1 line
Quoted 1 line

Bob Pearson

Nope. I don't have real RNICs at home to test. But (see my note to Zhu) the non extended APIs do set the access flags correctly and the extended test case does not. The wr_bind_mw() function can't fix this for the test case. It has to set the access flags when it creates the MR and it didn't. It is possible that mlx5 doesn't check the bind access flag but that seems unlikely.

mlx5 devices support MW 1 & 2 and kernel checks that only these types can be accepted from the user space. This is why mlx5 doesn't need to check access flags again.

    903 static int ib_uverbs_alloc_mw(struct uverbs_attr_bundle *attrs) 904 {

....

Quoted 7 lines

    927         if (cmd.mw_type != IB_MW_TYPE_1 && cmd.mw_type != IB_MW_TYPE_2) { 928                 ret = -EINVAL;
    929                 goto err_put;
    930         }

Thanks

I see that mlx5 checks the access flags in userspace only if MW_DEBUG is turned on (in set_bind_wr()).

I guess that's for the sake of performance, as it's part of the data path.

Quoted 2 lines
Quoted 2 lines

Bob

#9 lore raw
Quoted 95 lines · Robert B

External email: Use caution opening links or attachments

Quoted 70 lines · Robert B
Quoted 40 lines · Robert B
Quoted 27 lines · Edward Srouji
Quoted 25 lines · Leon Romanovsky
Quoted 24 lines · Robert B
Quoted 23 lines · Leon Romanovsky
Quoted 22 lines · Robert B

sorry/this time without the HTML.

======================================================================

ERROR: test_qp_ex_rc_bind_mw (tests.test_qpex.QpExTestCase) Verify bind memory window operation using the new post_send API.
----------------------------------------------------------------------

Traceback (most recent call last):
    File "/home/rpearson/src/rdma-core/tests/test_qpex.py", line 292, in
test_qp_ex_rc_bind_mw
      u.poll_cq(server.cq)
    File "/home/rpearson/src/rdma-core/tests/utils.py", line 538, in poll_cq
      raise PyverbsRDMAError('Completion status is {s}'. pyverbs.pyverbs_error.PyverbsRDMAError: Completion status is Memory window
bind error. Errno: 6, No such device or address

Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

This test attempts to bind a type 2 MW to an MR that does not have bind mw
access set and expects the test to succeed.

You're right, looks like a test bug. I'll send a fix upstream.

Can you please confirm that this solves your issue:

Well I get further. I am hitting a seg fault in python at

        client.qp.wr_rdma_write(new_key, server.mr.buf)

Quoted 8 lines

in test_qp_ex_rc_bind_mw.

I'm trying to track it down. I'm not very familiar with python and don't know how to run the test under gdb.

Thanks for the fix.

Bob

OK got it. In the setup for the test you write

    class QpExRCBindMw(RCResources):
        def create_qps(self):
            create_qp_ex(self, e.IBV_QPT_RC, e.IBV_QP_EX_WITH_BIND_MW)

        def create_mr(self):
            self.mr = u.create_custom_mr(self,
e.IBV_ACCESS_REMOTE_WRITE |
                            e.IBV_ACCESS_MW_BIND)

which asks for qp_ex->wr_bind_mw() to be set but later in the test you write

    client.qp.wr_rdma_write(new_key, server.mr.buf)

which calls qp_ex->wr_rdma_write() which is not set causing the seg fault. I think you should have written

Quoted 6 lines

            create_qp_ex(self, e.IBV_QPT_RC, e.IBV_QP_EX_WITH_BIND_MW | e.IBV_QP_EX_WITH_RDMA_WRITE)

since you need both extended QP operations.

Bob

With this patch the test is now running correctly

tests/test_qpex.py +0 -0
index 20288d45..0316bfcb 100644
@@ -146,10 +146,12 @@ class QpExRCAtomicFetchAdd(RCResources):
 

 class QpExRCBindMw(RCResources):
     def create_qps(self):
-        create_qp_ex(self, e.IBV_QPT_RC, e.IBV_QP_EX_WITH_BIND_MW) +        create_qp_ex(self, e.IBV_QPT_RC, e.IBV_QP_EX_WITH_BIND_MW | +                       e.IBV_QP_EX_WITH_RDMA_WRITE)

Quoted 6 lines

     def create_mr(self):
-        self.mr = u.create_custom_mr(self, e.IBV_ACCESS_REMOTE_WRITE) +        self.mr = u.create_custom_mr(self, e.IBV_ACCESS_REMOTE_WRITE | +                       e.IBV_ACCESS_MW_BIND)

I've sent a fix patch for upstream (you can see at github).

Quoted 65 lines

 class QpExTestCase(RDMATestCase):

Quoted 63 lines
Quoted 63 lines
Quoted 63 lines
tests/test_qpex.py +0 -0
index 4b58260f..c2d67ee8 100644
@@ -149,7 +149,7 @@ class QpExRCBindMw(RCResources):
          create_qp_ex(self, e.IBV_QPT_RC, e.IBV_QP_EX_WITH_BIND_MW)
 

     def create_mr(self):
-        self.mr = u.create_custom_mr(self, e.IBV_ACCESS_REMOTE_WRITE) +        self.mr = u.create_custom_mr(self, e.IBV_ACCESS_REMOTE_WRITE | e.IBV_ACCESS_MW_BIND)

Quoted 31 lines
Quoted 11 lines
Quoted 11 lines

Does the test break after your MW series? Or will it break not-merged
code yet?

Generally speaking, we expect that developers run rdma-core tests and
fixed/extend them prior to the submission.

Thanks

Quoted 1 line

Bob Pearson

Quoted 10 lines

Nope. I don't have real RNICs at home to test. But (see my note to Zhu) the
non extended APIs do set the access flags correctly and the extended test
case does not. The wr_bind_mw() function can't fix this for the test case.
It has to set the access flags when it creates the MR and it didn't. It is
possible that mlx5 doesn't check the bind access flag but that seems unlikely.

mlx5 devices support MW 1 & 2 and kernel checks that only these types can be accepted from the user space. This is why mlx5 doesn't need to check access flags again.

    903 static int ib_uverbs_alloc_mw(struct uverbs_attr_bundle *attrs)
    904 {

....

Quoted 8 lines

    927         if (cmd.mw_type != IB_MW_TYPE_1 && cmd.mw_type != IB_MW_TYPE_2) {
    928                 ret = -EINVAL;
    929                 goto err_put;
    930         }

Thanks

I see that mlx5 checks the access flags in userspace only if MW_DEBUG is turned on (in set_bind_wr()).

I guess that's for the sake of performance, as it's part of the data path.

Quoted 2 lines
Quoted 2 lines

Bob

#10 lore raw
Quoted 97 lines · Edward Srouji
Quoted 95 lines · Robert B

External email: Use caution opening links or attachments

Quoted 70 lines · Robert B
Quoted 40 lines · Robert B
Quoted 27 lines · Edward Srouji
Quoted 25 lines · Leon Romanovsky
Quoted 24 lines · Robert B
Quoted 23 lines · Leon Romanovsky
Quoted 22 lines · Robert B

sorry/this time without the HTML.

======================================================================

ERROR: test_qp_ex_rc_bind_mw (tests.test_qpex.QpExTestCase) Verify bind memory window operation using the new post_send API.
----------------------------------------------------------------------

Traceback (most recent call last):
    File "/home/rpearson/src/rdma-core/tests/test_qpex.py", line 292, in
test_qp_ex_rc_bind_mw
      u.poll_cq(server.cq)
    File "/home/rpearson/src/rdma-core/tests/utils.py", line 538, in poll_cq
      raise PyverbsRDMAError('Completion status is {s}'. pyverbs.pyverbs_error.PyverbsRDMAError: Completion status is Memory window
bind error. Errno: 6, No such device or address

Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

This test attempts to bind a type 2 MW to an MR that does not have bind mw
access set and expects the test to succeed.

You're right, looks like a test bug. I'll send a fix upstream.

Can you please confirm that this solves your issue:

Well I get further. I am hitting a seg fault in python at

        client.qp.wr_rdma_write(new_key, server.mr.buf)

Quoted 8 lines

in test_qp_ex_rc_bind_mw.

I'm trying to track it down. I'm not very familiar with python and don't know how to run the test under gdb.

Thanks for the fix.

Bob

OK got it. In the setup for the test you write

    class QpExRCBindMw(RCResources):
        def create_qps(self):
            create_qp_ex(self, e.IBV_QPT_RC, e.IBV_QP_EX_WITH_BIND_MW)

        def create_mr(self):
            self.mr = u.create_custom_mr(self,
e.IBV_ACCESS_REMOTE_WRITE |
                            e.IBV_ACCESS_MW_BIND)

which asks for qp_ex->wr_bind_mw() to be set but later in the test you write

    client.qp.wr_rdma_write(new_key, server.mr.buf)

which calls qp_ex->wr_rdma_write() which is not set causing the seg fault. I think you should have written

Quoted 6 lines

            create_qp_ex(self, e.IBV_QPT_RC, e.IBV_QP_EX_WITH_BIND_MW | e.IBV_QP_EX_WITH_RDMA_WRITE)

since you need both extended QP operations.

Bob

With this patch the test is now running correctly

tests/test_qpex.py +0 -0
index 20288d45..0316bfcb 100644
@@ -146,10 +146,12 @@ class QpExRCAtomicFetchAdd(RCResources):
 

 class QpExRCBindMw(RCResources):
     def create_qps(self):
-        create_qp_ex(self, e.IBV_QPT_RC, e.IBV_QP_EX_WITH_BIND_MW) +        create_qp_ex(self, e.IBV_QPT_RC, e.IBV_QP_EX_WITH_BIND_MW | +                       e.IBV_QP_EX_WITH_RDMA_WRITE)

Quoted 7 lines
Quoted 6 lines

     def create_mr(self):
-        self.mr = u.create_custom_mr(self, e.IBV_ACCESS_REMOTE_WRITE) +        self.mr = u.create_custom_mr(self, e.IBV_ACCESS_REMOTE_WRITE | +                       e.IBV_ACCESS_MW_BIND)

I've sent a fix patch for upstream (you can see at github).

Thanks!

Quoted 66 lines
Quoted 66 lines

 class QpExTestCase(RDMATestCase):

Quoted 64 lines
Quoted 64 lines
Quoted 64 lines
tests/test_qpex.py +0 -0
index 4b58260f..c2d67ee8 100644
@@ -149,7 +149,7 @@ class QpExRCBindMw(RCResources):
          create_qp_ex(self, e.IBV_QPT_RC, e.IBV_QP_EX_WITH_BIND_MW)
 

     def create_mr(self):
-        self.mr = u.create_custom_mr(self,
e.IBV_ACCESS_REMOTE_WRITE)
+        self.mr = u.create_custom_mr(self,
e.IBV_ACCESS_REMOTE_WRITE | e.IBV_ACCESS_MW_BIND)

Quoted 31 lines
Quoted 11 lines
Quoted 11 lines

Does the test break after your MW series? Or will it break not-merged
code yet?

Generally speaking, we expect that developers run rdma-core tests and
fixed/extend them prior to the submission.

Thanks

Quoted 1 line

Bob Pearson

Quoted 10 lines

Nope. I don't have real RNICs at home to test. But (see my note to Zhu) the
non extended APIs do set the access flags correctly and the extended test
case does not. The wr_bind_mw() function can't fix this for the test case.
It has to set the access flags when it creates the MR and it didn't. It is
possible that mlx5 doesn't check the bind access flag but that seems unlikely.

mlx5 devices support MW 1 & 2 and kernel checks that only these types can be accepted from the user space. This is why mlx5 doesn't need to check access flags again.

    903 static int ib_uverbs_alloc_mw(struct uverbs_attr_bundle *attrs)
    904 {

....

Quoted 8 lines

    927         if (cmd.mw_type != IB_MW_TYPE_1 && cmd.mw_type != IB_MW_TYPE_2) {
    928                 ret = -EINVAL;
    929                 goto err_put;
    930         }

Thanks

I see that mlx5 checks the access flags in userspace only if MW_DEBUG is turned on (in set_bind_wr()).

I guess that's for the sake of performance, as it's part of the data path.

Quoted 2 lines
Quoted 2 lines

Bob

#11 lore raw
In reply to Leon Romanovsky
Quoted 44 lines · Leon Romanovsky
Quoted 27 lines · Robert B
Quoted 26 lines · Leon Romanovsky
Quoted 15 lines · Robert B

sorry/this time without the HTML.

======================================================================
ERROR: test_qp_ex_rc_bind_mw (tests.test_qpex.QpExTestCase) Verify bind memory window operation using the new post_send API.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/rpearson/src/rdma-core/tests/test_qpex.py", line 292, in
test_qp_ex_rc_bind_mw
    u.poll_cq(server.cq)
  File "/home/rpearson/src/rdma-core/tests/utils.py", line 538, in poll_cq raise PyverbsRDMAError('Completion status is {s}'.
pyverbs.pyverbs_error.PyverbsRDMAError: Completion status is Memory window bind error. Errno: 6, No such device or address

Quoted 2 lines

This test attempts to bind a type 2 MW to an MR that does not have bind mw access set and expects the test to succeed.

Does the test break after your MW series? Or will it break not-merged code yet?

Generally speaking, we expect that developers run rdma-core tests and fixed/extend them prior to the submission.

Thanks

Quoted 7 lines
Quoted 1 line
Quoted 1 line

Bob Pearson

Nope. I don't have real RNICs at home to test. But (see my note to Zhu) the non extended APIs do set the access flags correctly and the extended test case does not. The wr_bind_mw() function can't fix this for the test case. It has to set the access flags when it creates the MR and it didn't. It is possible that mlx5 doesn't check the bind access flag but that seems unlikely.

mlx5 devices support MW 1 & 2 and kernel checks that only these types can be accepted from the user space. This is why mlx5 doesn't need to check access flags again.

903 static int ib_uverbs_alloc_mw(struct uverbs_attr_bundle *attrs) 904 {

....

Quoted 7 lines
    927         if (cmd.mw_type != IB_MW_TYPE_1 && cmd.mw_type != IB_MW_TYPE_2) {
    928                 ret = -EINVAL;
    929                 goto err_put;
    930         }

Thanks

You check the type in alloc_mw but you only check the MR access flags if MW_DEBUG is set which is not by default. So you would fail a negative test which we sort of currently have. The second bug in the test which we found this morning is not correctly setting the op flags in the create_qp_ex call. According to the man page only the extended operations set in the op flags are 'implemented' for that QP. Apparently mlx5 goes ahead and populates them. Makes more sense to me since the API as described is kind of overwrought and dumb.

Bob