RDMA/core: test_qpex.py attempts invalid MW bind operation
sorry/this time without the HTML.
======================================================================
ERROR: test_qp_ex_rc_bind_mw (tests.test_qpex.QpExTestCase)
Verify bind memory window operation using the new post_send API.
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/rpearson/src/rdma-core/tests/test_qpex.py", line 292, in test_qp_ex_rc_bind_mw u.poll_cq(server.cq)
File "/home/rpearson/src/rdma-core/tests/utils.py", line 538, in poll_cq
raise PyverbsRDMAError('Completion status is {s}'.
pyverbs.pyverbs_error.PyverbsRDMAError: Completion status is Memory window bind error. Errno: 6, No such device or address
This test attempts to bind a type 2 MW to an MR that does not have bind mw access set and expects the test to succeed.
Bob Pearson
Quoted 15 lines · Robert B
sorry/this time without the HTML.
======================================================================
ERROR:test_qp_ex_rc_bind_mw(tests.test_qpex.QpExTestCase) Verify bind memory window operation using the newpost_sendAPI.
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/rpearson/src/rdma-core/tests/test_qpex.py", line 292, intest_qp_ex_rc_bind_mw
u.poll_cq(server.cq)
File "/home/rpearson/src/rdma-core/tests/utils.py", line 538, inpoll_cqraise PyverbsRDMAError('Completion status is {s}'. pyverbs.pyverbs_error.PyverbsRDMAError: Completion status is Memory window bind error. Errno: 6, No such device or address
Quoted 2 lines
This test attempts to bind a type 2 MW to an MR that does not have bind mw access set and expects the test to succeed.
Does the test break after your MW series? Or will it break not-merged code yet?
Generally speaking, we expect that developers run rdma-core tests and fixed/extend them prior to the submission.
Thanks
Quoted 3 lines
Bob Pearson
Quoted 18 lines · Leon Romanovsky
Quoted 15 lines · Robert B
sorry/this time without the HTML.
======================================================================
ERROR:test_qp_ex_rc_bind_mw(tests.test_qpex.QpExTestCase) Verify bind memory window operation using the newpost_sendAPI.
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/rpearson/src/rdma-core/tests/test_qpex.py", line 292, intest_qp_ex_rc_bind_mw
u.poll_cq(server.cq)
File "/home/rpearson/src/rdma-core/tests/utils.py", line 538, inpoll_cqraise PyverbsRDMAError('Completion status is {s}'.
pyverbs.pyverbs_error.PyverbsRDMAError: Completion status is Memory window bind error. Errno: 6, No such device or addressQuoted 2 lines
This test attempts to bind a type 2 MW to an MR that does not have bind mw access set and expects the test to succeed.
Quoted 9 lines
Does the test break after your MW series? Or will it break not-merged code yet?
Generally speaking, we expect that developers run rdma-core tests and fixed/extend them prior to the submission.
Thanks
Quoted 1 line
Bob Pearson
Nope. I don't have real RNICs at home to test. But (see my note to Zhu) the non extended APIs do set the access flags correctly and the extended test case does not. The wr_bind_mw() function can't fix this for the test case. It has to set the access flags when it creates the MR and it didn't. It is possible that mlx5 doesn't check the bind access flag but that seems unlikely.
Bob
Quoted 28 lines · Robert B
Quoted 26 lines · Leon Romanovsky
Quoted 15 lines · Robert B
sorry/this time without the HTML.
======================================================================
ERROR:test_qp_ex_rc_bind_mw(tests.test_qpex.QpExTestCase) Verify bind memory window operation using the newpost_sendAPI.
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/rpearson/src/rdma-core/tests/test_qpex.py", line 292, intest_qp_ex_rc_bind_mw
u.poll_cq(server.cq)
File "/home/rpearson/src/rdma-core/tests/utils.py", line 538, inpoll_cqraise PyverbsRDMAError('Completion status is {s}'.
pyverbs.pyverbs_error.PyverbsRDMAError: Completion status is Memory window bind error. Errno: 6, No such device or addressQuoted 2 lines
This test attempts to bind a type 2 MW to an MR that does not have bind mw access set and expects the test to succeed.
Does the test break after your MW series? Or will it break not-merged code yet?
Generally speaking, we expect that developers run rdma-core tests and fixed/extend them prior to the submission.
Thanks
Quoted 8 lines
Quoted 1 line
Quoted 1 line
Bob Pearson
Nope. I don't have real RNICs at home to test. But (see my note to Zhu) the non extended APIs do set the access flags correctly and the extended test case does not. The
wr_bind_mw()function can't fix this for the test case. It has to set the access flags when it creates the MR and it didn't. It is possible that mlx5 doesn't check the bind access flag but that seems unlikely.
mlx5 devices support MW 1 & 2 and kernel checks that only these types can be accepted from the user space. This is why mlx5 doesn't need to check access flags again.
903 static int ib_uverbs_alloc_mw(struct uverbs_attr_bundle *attrs)
904 {....
927 if (cmd.mw_type != IB_MW_TYPE_1 && cmd.mw_type != IB_MW_TYPE_2) {
928 ret = -EINVAL;
929 goto err_put;
930 }Thanks
Quoted 3 lines
Bob
Quoted 18 lines · Leon Romanovsky
Quoted 17 lines · Robert B
Quoted 16 lines · Leon Romanovsky
Quoted 15 lines · Robert B
sorry/this time without the HTML.
======================================================================
ERROR:test_qp_ex_rc_bind_mw(tests.test_qpex.QpExTestCase) Verify bind memory window operation using the newpost_sendAPI.
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/rpearson/src/rdma-core/tests/test_qpex.py", line 292, intest_qp_ex_rc_bind_mw
u.poll_cq(server.cq)
File "/home/rpearson/src/rdma-core/tests/utils.py", line 538, inpoll_cqraise PyverbsRDMAError('Completion status is {s}'.
pyverbs.pyverbs_error.PyverbsRDMAError: Completion status is Memory window bind error. Errno: 6, No such device or address
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
This test attempts to bind a type 2 MW to an MR that does not have bind mw access set and expects the test to succeed.
You're right, looks like a test bug. I'll send a fix upstream.
Can you please confirm that this solves your issue:
tests/test_qpex.py +0 -0
@@ -149,7 +149,7 @@ class QpExRCBindMw(RCResources):
create_qp_ex(self, e.IBV_QPT_RC, e.IBV_QP_EX_WITH_BIND_MW)
def create_mr(self):
- self.mr = u.create_custom_mr(self, e.IBV_ACCESS_REMOTE_WRITE)
+ self.mr = u.create_custom_mr(self, e.IBV_ACCESS_REMOTE_WRITE | e.IBV_ACCESS_MW_BIND)
Quoted 24 lines
Quoted 8 lines
Quoted 8 lines
Does the test break after your MW series? Or will it break not-merged code yet?
Generally speaking, we expect that developers run rdma-core tests and fixed/extend them prior to the submission.
Thanks
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Bob Pearson
Nope. I don't have real RNICs at home to test. But (see my note to Zhu) the non extended APIs do set the access flags correctly and the extended test case does not. The
wr_bind_mw()function can't fix this for the test case. It has to set the access flags when it creates the MR and it didn't. It is possible that mlx5 doesn't check the bind access flag but that seems unlikely.mlx5 devices support MW 1 & 2 and kernel checks that only these types can be accepted from the user space. This is why mlx5 doesn't need to check access flags again.
903 static int
ib_uverbs_alloc_mw(struct uverbs_attr_bundle *attrs)904 {....
Quoted 7 lines
927 if (cmd.mw_type != IB_MW_TYPE_1 && cmd.mw_type != IB_MW_TYPE_2) { 928 ret = -EINVAL; 929 goto err_put; 930 }Thanks
I see that mlx5 checks the access flags in userspace only if MW_DEBUG is turned on (in set_bind_wr()).
I guess that's for the sake of performance, as it's part of the data path.
Quoted 2 lines
Quoted 2 lines
Bob
Quoted 22 lines · Edward Srouji
Quoted 20 lines · Leon Romanovsky
Quoted 19 lines · Robert B
Quoted 18 lines · Leon Romanovsky
Quoted 17 lines · Robert B
sorry/this time without the HTML.
======================================================================
ERROR:
test_qp_ex_rc_bind_mw(tests.test_qpex.QpExTestCase) Verify bind memory window operation using the newpost_sendAPI.
----------------------------------------------------------------------Traceback (most recent call last):
File "/home/rpearson/src/rdma-core/tests/test_qpex.py", line 292, intest_qp_ex_rc_bind_mw
u.poll_cq(server.cq)
File "/home/rpearson/src/rdma-core/tests/utils.py", line 538, inpoll_cqraise PyverbsRDMAError('Completion status is {s}'. pyverbs.pyverbs_error.PyverbsRDMAError: Completion status is Memory window bind error. Errno: 6, No such device or address
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
This test attempts to bind a type 2 MW to an MR that does not have bind mw access set and expects the test to succeed.
You're right, looks like a test bug. I'll send a fix upstream.
Can you please confirm that this solves your issue:
Well I get further. I am hitting a seg fault in python at
client.qp.wr_rdma_write(new_key, server.mr.buf)
in test_qp_ex_rc_bind_mw.
I'm trying to track it down. I'm not very familiar with python and don't know how to run the test under gdb.
Thanks for the fix.
Bob
Quoted 51 lines
tests/test_qpex.py +0
-0@@ -149,7 +149,7 @@ class QpExRCBindMw(RCResources): create_qp_ex(self, e.IBV_QPT_RC, e.IBV_QP_EX_WITH_BIND_MW)def
create_mr(self):
- self.mr =u.create_custom_mr(self, e.IBV_ACCESS_REMOTE_WRITE)+ self.mr =u.create_custom_mr(self, e.IBV_ACCESS_REMOTE_WRITE | e.IBV_ACCESS_MW_BIND)Quoted 24 lines
Quoted 8 lines
Quoted 8 lines
Does the test break after your MW series? Or will it break not-merged code yet?
Generally speaking, we expect that developers run rdma-core tests and fixed/extend them prior to the submission.
Thanks
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Bob Pearson
Nope. I don't have real RNICs at home to test. But (see my note to Zhu) the non extended APIs do set the access flags correctly and the extended test case does not. The
wr_bind_mw()function can't fix this for the test case. It has to set the access flags when it creates the MR and it didn't. It is possible that mlx5 doesn't check the bind access flag but that seems unlikely.mlx5 devices support MW 1 & 2 and kernel checks that only these types can be accepted from the user space. This is why mlx5 doesn't need to check access flags again.
903 static int
ib_uverbs_alloc_mw(struct uverbs_attr_bundle *attrs)904 {....
Quoted 7 lines
927 if (cmd.mw_type !=
IB_MW_TYPE_1&& cmd.mw_type !=IB_MW_TYPE_2) { 928 ret =-EINVAL;
929 gotoerr_put;
930 }Thanks
I see that mlx5 checks the access flags in userspace only if
MW_DEBUGis turned on (inset_bind_wr()).I guess that's for the sake of performance, as it's part of the data path.
Quoted 2 lines
Quoted 2 lines
Bob
Quoted 38 lines · Robert B
External email: Use caution opening links or attachments
Quoted 27 lines · Edward Srouji
Quoted 25 lines · Leon Romanovsky
Quoted 24 lines · Robert B
Quoted 23 lines · Leon Romanovsky
Quoted 22 lines · Robert B
sorry/this time without the HTML.
======================================================================
ERROR:
test_qp_ex_rc_bind_mw(tests.test_qpex.QpExTestCase) Verify bind memory window operation using the newpost_sendAPI.
----------------------------------------------------------------------Traceback (most recent call last):
File "/home/rpearson/src/rdma-core/tests/test_qpex.py", line 292, intest_qp_ex_rc_bind_mw
u.poll_cq(server.cq)
File "/home/rpearson/src/rdma-core/tests/utils.py", line 538, inpoll_cq
raise PyverbsRDMAError('Completion status is {s}'. pyverbs.pyverbs_error.PyverbsRDMAError: Completion status is Memory window
bind error. Errno: 6, No such device or addressQuoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
This test attempts to bind a type 2 MW to an MR that does not have bind mw
access set and expects the test to succeed.You're right, looks like a test bug. I'll send a fix upstream.
Can you please confirm that this solves your issue:
Quoted 9 lines
Well I get further. I am hitting a seg fault in python at
client.qp.wr_rdma_write(new_key, server.mr.buf)in
test_qp_ex_rc_bind_mw.I'm trying to track it down. I'm not very familiar with python and don't know how to run the test under gdb.
I don't see the issue on mlx devices / rxe.
You can use gdb for python using: "gdb python"
Then : "run <program>" e.g.: "run tests/run_tests.py -k test_qp_ex_rc_bind_mw" (if you ran the command from the rdma-core root dir).
You can set breakpoints for C functions as you regularly do. If you want to add a breakpoint to the python test, I suggest you do "import pdb; pdb.set_trace()" at the line you want to set the bp at.
Good luck.
Quoted 62 lines
Thanks for the fix.
Bob
Quoted 58 lines
tests/test_qpex.py +0
-0@@ -149,7 +149,7 @@ class QpExRCBindMw(RCResources): create_qp_ex(self, e.IBV_QPT_RC, e.IBV_QP_EX_WITH_BIND_MW)def
create_mr(self):
- self.mr =u.create_custom_mr(self, e.IBV_ACCESS_REMOTE_WRITE)+ self.mr =u.create_custom_mr(self, e.IBV_ACCESS_REMOTE_WRITE | e.IBV_ACCESS_MW_BIND)Quoted 28 lines
Quoted 9 lines
Quoted 9 lines
Does the test break after your MW series? Or will it break not-merged code yet?
Generally speaking, we expect that developers run rdma-core tests and fixed/extend them prior to the submission.
Thanks
Quoted 1 line
Bob Pearson
Quoted 10 lines
Nope. I don't have real RNICs at home to test. But (see my note to Zhu) the
non extended APIs do set the access flags correctly and the extended test
case does not. Thewr_bind_mw()function can't fix this for the test case.
It has to set the access flags when it creates the MR and it didn't. It is
possible that mlx5 doesn't check the bind access flag but that seems unlikely.mlx5 devices support MW 1 & 2 and kernel checks that only these types can be accepted from the user space. This is why mlx5 doesn't need to check access flags again.
903 static int
ib_uverbs_alloc_mw(struct uverbs_attr_bundle *attrs)904 {....
Quoted 8 lines
927 if (cmd.mw_type !=
IB_MW_TYPE_1&& cmd.mw_type !=IB_MW_TYPE_2) {
928 ret =-EINVAL;
929 gotoerr_put;
930 }Thanks
I see that mlx5 checks the access flags in userspace only if
MW_DEBUGis turned on (inset_bind_wr()).I guess that's for the sake of performance, as it's part of the data path.
Quoted 2 lines
Quoted 2 lines
Bob
Quoted 34 lines · Robert B
Quoted 22 lines · Edward Srouji
Quoted 20 lines · Leon Romanovsky
Quoted 19 lines · Robert B
Quoted 18 lines · Leon Romanovsky
Quoted 17 lines · Robert B
sorry/this time without the HTML.
======================================================================
ERROR:
test_qp_ex_rc_bind_mw(tests.test_qpex.QpExTestCase) Verify bind memory window operation using the newpost_sendAPI.
----------------------------------------------------------------------Traceback (most recent call last):
File "/home/rpearson/src/rdma-core/tests/test_qpex.py", line 292, intest_qp_ex_rc_bind_mw
u.poll_cq(server.cq)
File "/home/rpearson/src/rdma-core/tests/utils.py", line 538, inpoll_cqraise PyverbsRDMAError('Completion status is {s}'. pyverbs.pyverbs_error.PyverbsRDMAError: Completion status is Memory window bind error. Errno: 6, No such device or addressQuoted 6 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
This test attempts to bind a type 2 MW to an MR that does not have bind mw access set and expects the test to succeed.
You're right, looks like a test bug. I'll send a fix upstream.
Can you please confirm that this solves your issue:
Well I get further. I am hitting a seg fault in python at
client.qp.wr_rdma_write(new_key, server.mr.buf)
Quoted 7 lines
in
test_qp_ex_rc_bind_mw.I'm trying to track it down. I'm not very familiar with python and don't know how to run the test under gdb.
Thanks for the fix.
Bob
OK got it. In the setup for the test you write
class QpExRCBindMw(RCResources):
def create_qps(self):
create_qp_ex(self, e.IBV_QPT_RC, e.IBV_QP_EX_WITH_BIND_MW)
def create_mr(self):
self.mr = u.create_custom_mr(self, e.IBV_ACCESS_REMOTE_WRITE | e.IBV_ACCESS_MW_BIND)
which asks for qp_ex->wr_bind_mw() to be set but later in the test you write
client.qp.wr_rdma_write(new_key, server.mr.buf)
which calls qp_ex->wr_rdma_write() which is not set causing the seg fault. I think you should have written
create_qp_ex(self, e.IBV_QPT_RC, e.IBV_QP_EX_WITH_BIND_MW | e.IBV_QP_EX_WITH_RDMA_WRITE)
since you need both extended QP operations.
Bob
Quoted 52 lines
Quoted 52 lines
tests/test_qpex.py +0
-0@@ -149,7 +149,7 @@ class QpExRCBindMw(RCResources): create_qp_ex(self, e.IBV_QPT_RC, e.IBV_QP_EX_WITH_BIND_MW)def
create_mr(self):
- self.mr =u.create_custom_mr(self, e.IBV_ACCESS_REMOTE_WRITE)+ self.mr =u.create_custom_mr(self, e.IBV_ACCESS_REMOTE_WRITE | e.IBV_ACCESS_MW_BIND)Quoted 24 lines
Quoted 8 lines
Quoted 8 lines
Does the test break after your MW series? Or will it break not-merged code yet?
Generally speaking, we expect that developers run rdma-core tests and fixed/extend them prior to the submission.
Thanks
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Bob Pearson
Nope. I don't have real RNICs at home to test. But (see my note to Zhu) the non extended APIs do set the access flags correctly and the extended test case does not. The
wr_bind_mw()function can't fix this for the test case. It has to set the access flags when it creates the MR and it didn't. It is possible that mlx5 doesn't check the bind access flag but that seems unlikely.mlx5 devices support MW 1 & 2 and kernel checks that only these types can be accepted from the user space. This is why mlx5 doesn't need to check access flags again.
903 static int
ib_uverbs_alloc_mw(struct uverbs_attr_bundle *attrs)904 {....
Quoted 7 lines
927 if (cmd.mw_type !=
IB_MW_TYPE_1&& cmd.mw_type !=IB_MW_TYPE_2) { 928 ret =-EINVAL;
929 gotoerr_put;
930 }Thanks
I see that mlx5 checks the access flags in userspace only if
MW_DEBUGis turned on (inset_bind_wr()).I guess that's for the sake of performance, as it's part of the data path.
Quoted 2 lines
Quoted 2 lines
Bob
Quoted 58 lines · Robert B
Quoted 34 lines · Robert B
Quoted 22 lines · Edward Srouji
Quoted 20 lines · Leon Romanovsky
Quoted 19 lines · Robert B
Quoted 18 lines · Leon Romanovsky
Quoted 17 lines · Robert B
sorry/this time without the HTML.
======================================================================
ERROR:
test_qp_ex_rc_bind_mw(tests.test_qpex.QpExTestCase) Verify bind memory window operation using the newpost_sendAPI.
----------------------------------------------------------------------Traceback (most recent call last):
File "/home/rpearson/src/rdma-core/tests/test_qpex.py", line 292, intest_qp_ex_rc_bind_mw
u.poll_cq(server.cq)
File "/home/rpearson/src/rdma-core/tests/utils.py", line 538, inpoll_cqraise PyverbsRDMAError('Completion status is {s}'. pyverbs.pyverbs_error.PyverbsRDMAError: Completion status is Memory window bind error. Errno: 6, No such device or addressQuoted 6 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
This test attempts to bind a type 2 MW to an MR that does not have bind mw access set and expects the test to succeed.
You're right, looks like a test bug. I'll send a fix upstream.
Can you please confirm that this solves your issue:
Well I get further. I am hitting a seg fault in python at
client.qp.wr_rdma_write(new_key, server.mr.buf)Quoted 7 lines
in
test_qp_ex_rc_bind_mw.I'm trying to track it down. I'm not very familiar with python and don't know how to run the test under gdb.
Thanks for the fix.
Bob
OK got it. In the setup for the test you write
class QpExRCBindMw(RCResources):
defcreate_qps(self):
create_qp_ex(self, e.IBV_QPT_RC, e.IBV_QP_EX_WITH_BIND_MW)def
create_mr(self):
self.mr =u.create_custom_mr(self, e.IBV_ACCESS_REMOTE_WRITE | e.IBV_ACCESS_MW_BIND)which asks for
qp_ex->wr_bind_mw()to be set but later in the test you write
client.qp.wr_rdma_write(new_key, server.mr.buf)
Quoted 7 lines
which calls
qp_ex->wr_rdma_write()which is not set causing the seg fault. I think you should have written
create_qp_ex(self, e.IBV_QPT_RC, e.IBV_QP_EX_WITH_BIND_MW | e.IBV_QP_EX_WITH_RDMA_WRITE)since you need both extended QP operations.
Bob
With this patch the test is now running correctly
tests/test_qpex.py +0 -0
@@ -146,10 +146,12 @@ class QpExRCAtomicFetchAdd(RCResources):
class QpExRCBindMw(RCResources):
def create_qps(self):
- create_qp_ex(self, e.IBV_QPT_RC, e.IBV_QP_EX_WITH_BIND_MW) + create_qp_ex(self, e.IBV_QPT_RC, e.IBV_QP_EX_WITH_BIND_MW | + e.IBV_QP_EX_WITH_RDMA_WRITE)
def create_mr(self):
- self.mr = u.create_custom_mr(self, e.IBV_ACCESS_REMOTE_WRITE) + self.mr = u.create_custom_mr(self, e.IBV_ACCESS_REMOTE_WRITE | + e.IBV_ACCESS_MW_BIND)
class QpExTestCase(RDMATestCase):
Quoted 53 lines
Quoted 53 lines
Quoted 53 lines
tests/test_qpex.py +0
-0@@ -149,7 +149,7 @@ class QpExRCBindMw(RCResources): create_qp_ex(self, e.IBV_QPT_RC, e.IBV_QP_EX_WITH_BIND_MW)def
create_mr(self):
- self.mr =u.create_custom_mr(self, e.IBV_ACCESS_REMOTE_WRITE)+ self.mr =u.create_custom_mr(self, e.IBV_ACCESS_REMOTE_WRITE | e.IBV_ACCESS_MW_BIND)Quoted 24 lines
Quoted 8 lines
Quoted 8 lines
Does the test break after your MW series? Or will it break not-merged code yet?
Generally speaking, we expect that developers run rdma-core tests and fixed/extend them prior to the submission.
Thanks
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Bob Pearson
Nope. I don't have real RNICs at home to test. But (see my note to Zhu) the non extended APIs do set the access flags correctly and the extended test case does not. The
wr_bind_mw()function can't fix this for the test case. It has to set the access flags when it creates the MR and it didn't. It is possible that mlx5 doesn't check the bind access flag but that seems unlikely.mlx5 devices support MW 1 & 2 and kernel checks that only these types can be accepted from the user space. This is why mlx5 doesn't need to check access flags again.
903 static int
ib_uverbs_alloc_mw(struct uverbs_attr_bundle *attrs)904 {....
Quoted 7 lines
927 if (cmd.mw_type !=
IB_MW_TYPE_1&& cmd.mw_type !=IB_MW_TYPE_2) { 928 ret =-EINVAL;
929 gotoerr_put;
930 }Thanks
I see that mlx5 checks the access flags in userspace only if
MW_DEBUGis turned on (inset_bind_wr()).I guess that's for the sake of performance, as it's part of the data path.
Quoted 2 lines
Quoted 2 lines
Bob
Quoted 95 lines · Robert B
External email: Use caution opening links or attachments
Quoted 70 lines · Robert B
Quoted 40 lines · Robert B
Quoted 27 lines · Edward Srouji
Quoted 25 lines · Leon Romanovsky
Quoted 24 lines · Robert B
Quoted 23 lines · Leon Romanovsky
Quoted 22 lines · Robert B
sorry/this time without the HTML.
======================================================================
ERROR:
test_qp_ex_rc_bind_mw(tests.test_qpex.QpExTestCase) Verify bind memory window operation using the newpost_sendAPI.
----------------------------------------------------------------------Traceback (most recent call last):
File "/home/rpearson/src/rdma-core/tests/test_qpex.py", line 292, intest_qp_ex_rc_bind_mw
u.poll_cq(server.cq)
File "/home/rpearson/src/rdma-core/tests/utils.py", line 538, inpoll_cq
raise PyverbsRDMAError('Completion status is {s}'. pyverbs.pyverbs_error.PyverbsRDMAError: Completion status is Memory window
bind error. Errno: 6, No such device or addressQuoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
This test attempts to bind a type 2 MW to an MR that does not have bind mw
access set and expects the test to succeed.You're right, looks like a test bug. I'll send a fix upstream.
Can you please confirm that this solves your issue:
Well I get further. I am hitting a seg fault in python at
client.qp.wr_rdma_write(new_key, server.mr.buf)Quoted 8 lines
in
test_qp_ex_rc_bind_mw.I'm trying to track it down. I'm not very familiar with python and don't know how to run the test under gdb.
Thanks for the fix.
Bob
OK got it. In the setup for the test you write
class QpExRCBindMw(RCResources):
defcreate_qps(self):
create_qp_ex(self, e.IBV_QPT_RC, e.IBV_QP_EX_WITH_BIND_MW)def
create_mr(self):
self.mr = u.create_custom_mr(self,
e.IBV_ACCESS_REMOTE_WRITE |
e.IBV_ACCESS_MW_BIND)which asks for
qp_ex->wr_bind_mw()to be set but later in the test you write
client.qp.wr_rdma_write(new_key, server.mr.buf)which calls
qp_ex->wr_rdma_write()which is not set causing the seg fault. I think you should have writtenQuoted 6 lines
create_qp_ex(self, e.IBV_QPT_RC, e.IBV_QP_EX_WITH_BIND_MW | e.IBV_QP_EX_WITH_RDMA_WRITE)since you need both extended QP operations.
Bob
With this patch the test is now running correctly
tests/test_qpex.py +0
-0@@ -146,10 +146,12 @@ class QpExRCAtomicFetchAdd(RCResources):class QpExRCBindMw(RCResources):
defcreate_qps(self):
-create_qp_ex(self, e.IBV_QPT_RC, e.IBV_QP_EX_WITH_BIND_MW)+ create_qp_ex(self, e.IBV_QPT_RC, e.IBV_QP_EX_WITH_BIND_MW | + e.IBV_QP_EX_WITH_RDMA_WRITE)
Quoted 6 lines
def
create_mr(self):
- self.mr =u.create_custom_mr(self, e.IBV_ACCESS_REMOTE_WRITE)+ self.mr = u.create_custom_mr(self, e.IBV_ACCESS_REMOTE_WRITE | + e.IBV_ACCESS_MW_BIND)
I've sent a fix patch for upstream (you can see at github).
Quoted 65 lines
class QpExTestCase(RDMATestCase):
Quoted 63 lines
Quoted 63 lines
Quoted 63 lines
tests/test_qpex.py +0
-0@@ -149,7 +149,7 @@ class QpExRCBindMw(RCResources): create_qp_ex(self, e.IBV_QPT_RC, e.IBV_QP_EX_WITH_BIND_MW)def
create_mr(self):
- self.mr =u.create_custom_mr(self, e.IBV_ACCESS_REMOTE_WRITE)+ self.mr =u.create_custom_mr(self, e.IBV_ACCESS_REMOTE_WRITE | e.IBV_ACCESS_MW_BIND)Quoted 31 lines
Quoted 11 lines
Quoted 11 lines
Does the test break after your MW series? Or will it break not-merged
code yet?Generally speaking, we expect that developers run rdma-core tests and
fixed/extend them prior to the submission.Thanks
Quoted 1 line
Bob Pearson
Quoted 10 lines
Nope. I don't have real RNICs at home to test. But (see my note to Zhu) the
non extended APIs do set the access flags correctly and the extended test
case does not. Thewr_bind_mw()function can't fix this for the test case.
It has to set the access flags when it creates the MR and it didn't. It is
possible that mlx5 doesn't check the bind access flag but that seems unlikely.mlx5 devices support MW 1 & 2 and kernel checks that only these types can be accepted from the user space. This is why mlx5 doesn't need to check access flags again.
903 static int
ib_uverbs_alloc_mw(struct uverbs_attr_bundle *attrs)
904 {....
Quoted 8 lines
927 if (cmd.mw_type !=
IB_MW_TYPE_1&& cmd.mw_type !=IB_MW_TYPE_2) {
928 ret =-EINVAL;
929 gotoerr_put;
930 }Thanks
I see that mlx5 checks the access flags in userspace only if
MW_DEBUGis turned on (inset_bind_wr()).I guess that's for the sake of performance, as it's part of the data path.
Quoted 2 lines
Quoted 2 lines
Bob
Quoted 97 lines · Edward Srouji
Quoted 95 lines · Robert B
External email: Use caution opening links or attachments
Quoted 70 lines · Robert B
Quoted 40 lines · Robert B
Quoted 27 lines · Edward Srouji
Quoted 25 lines · Leon Romanovsky
Quoted 24 lines · Robert B
Quoted 23 lines · Leon Romanovsky
Quoted 22 lines · Robert B
sorry/this time without the HTML.
======================================================================
ERROR:
test_qp_ex_rc_bind_mw(tests.test_qpex.QpExTestCase) Verify bind memory window operation using the newpost_sendAPI.
----------------------------------------------------------------------Traceback (most recent call last):
File "/home/rpearson/src/rdma-core/tests/test_qpex.py", line 292, intest_qp_ex_rc_bind_mw
u.poll_cq(server.cq)
File "/home/rpearson/src/rdma-core/tests/utils.py", line 538, inpoll_cq
raise PyverbsRDMAError('Completion status is {s}'. pyverbs.pyverbs_error.PyverbsRDMAError: Completion status is Memory window
bind error. Errno: 6, No such device or addressQuoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
This test attempts to bind a type 2 MW to an MR that does not have bind mw
access set and expects the test to succeed.You're right, looks like a test bug. I'll send a fix upstream.
Can you please confirm that this solves your issue:
Well I get further. I am hitting a seg fault in python at
client.qp.wr_rdma_write(new_key, server.mr.buf)Quoted 8 lines
in
test_qp_ex_rc_bind_mw.I'm trying to track it down. I'm not very familiar with python and don't know how to run the test under gdb.
Thanks for the fix.
Bob
OK got it. In the setup for the test you write
class QpExRCBindMw(RCResources):
defcreate_qps(self):
create_qp_ex(self, e.IBV_QPT_RC, e.IBV_QP_EX_WITH_BIND_MW)def
create_mr(self):
self.mr = u.create_custom_mr(self,
e.IBV_ACCESS_REMOTE_WRITE |
e.IBV_ACCESS_MW_BIND)which asks for
qp_ex->wr_bind_mw()to be set but later in the test you write
client.qp.wr_rdma_write(new_key, server.mr.buf)which calls
qp_ex->wr_rdma_write()which is not set causing the seg fault. I think you should have writtenQuoted 6 lines
create_qp_ex(self, e.IBV_QPT_RC, e.IBV_QP_EX_WITH_BIND_MW | e.IBV_QP_EX_WITH_RDMA_WRITE)since you need both extended QP operations.
Bob
With this patch the test is now running correctly
tests/test_qpex.py +0
-0@@ -146,10 +146,12 @@ class QpExRCAtomicFetchAdd(RCResources):class QpExRCBindMw(RCResources):
defcreate_qps(self):
-create_qp_ex(self, e.IBV_QPT_RC, e.IBV_QP_EX_WITH_BIND_MW)+ create_qp_ex(self, e.IBV_QPT_RC, e.IBV_QP_EX_WITH_BIND_MW | + e.IBV_QP_EX_WITH_RDMA_WRITE)
Quoted 7 lines
Quoted 6 lines
def
create_mr(self):
- self.mr =u.create_custom_mr(self, e.IBV_ACCESS_REMOTE_WRITE)+ self.mr = u.create_custom_mr(self, e.IBV_ACCESS_REMOTE_WRITE | + e.IBV_ACCESS_MW_BIND)I've sent a fix patch for upstream (you can see at github).
Thanks!
Quoted 66 lines
Quoted 66 lines
class QpExTestCase(RDMATestCase):
Quoted 64 lines
Quoted 64 lines
Quoted 64 lines
tests/test_qpex.py +0
-0@@ -149,7 +149,7 @@ class QpExRCBindMw(RCResources): create_qp_ex(self, e.IBV_QPT_RC, e.IBV_QP_EX_WITH_BIND_MW)def
create_mr(self):
- self.mr = u.create_custom_mr(self,
e.IBV_ACCESS_REMOTE_WRITE)
+ self.mr = u.create_custom_mr(self,
e.IBV_ACCESS_REMOTE_WRITE | e.IBV_ACCESS_MW_BIND)Quoted 31 lines
Quoted 11 lines
Quoted 11 lines
Does the test break after your MW series? Or will it break not-merged
code yet?Generally speaking, we expect that developers run rdma-core tests and
fixed/extend them prior to the submission.Thanks
Quoted 1 line
Bob Pearson
Quoted 10 lines
Nope. I don't have real RNICs at home to test. But (see my note to Zhu) the
non extended APIs do set the access flags correctly and the extended test
case does not. Thewr_bind_mw()function can't fix this for the test case.
It has to set the access flags when it creates the MR and it didn't. It is
possible that mlx5 doesn't check the bind access flag but that seems unlikely.mlx5 devices support MW 1 & 2 and kernel checks that only these types can be accepted from the user space. This is why mlx5 doesn't need to check access flags again.
903 static int
ib_uverbs_alloc_mw(struct uverbs_attr_bundle *attrs)
904 {....
Quoted 8 lines
927 if (cmd.mw_type !=
IB_MW_TYPE_1&& cmd.mw_type !=IB_MW_TYPE_2) {
928 ret =-EINVAL;
929 gotoerr_put;
930 }Thanks
I see that mlx5 checks the access flags in userspace only if
MW_DEBUGis turned on (inset_bind_wr()).I guess that's for the sake of performance, as it's part of the data path.
Quoted 2 lines
Quoted 2 lines
Bob
Quoted 44 lines · Leon Romanovsky
Quoted 27 lines · Robert B
Quoted 26 lines · Leon Romanovsky
Quoted 15 lines · Robert B
sorry/this time without the HTML.
======================================================================
ERROR:test_qp_ex_rc_bind_mw(tests.test_qpex.QpExTestCase) Verify bind memory window operation using the newpost_sendAPI.
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/rpearson/src/rdma-core/tests/test_qpex.py", line 292, intest_qp_ex_rc_bind_mw
u.poll_cq(server.cq)
File "/home/rpearson/src/rdma-core/tests/utils.py", line 538, inpoll_cqraise PyverbsRDMAError('Completion status is {s}'.
pyverbs.pyverbs_error.PyverbsRDMAError: Completion status is Memory window bind error. Errno: 6, No such device or addressQuoted 2 lines
This test attempts to bind a type 2 MW to an MR that does not have bind mw access set and expects the test to succeed.
Does the test break after your MW series? Or will it break not-merged code yet?
Generally speaking, we expect that developers run rdma-core tests and fixed/extend them prior to the submission.
Thanks
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Bob Pearson
Nope. I don't have real RNICs at home to test. But (see my note to Zhu) the non extended APIs do set the access flags correctly and the extended test case does not. The
wr_bind_mw()function can't fix this for the test case. It has to set the access flags when it creates the MR and it didn't. It is possible that mlx5 doesn't check the bind access flag but that seems unlikely.mlx5 devices support MW 1 & 2 and kernel checks that only these types can be accepted from the user space. This is why mlx5 doesn't need to check access flags again.
903 static int
ib_uverbs_alloc_mw(struct uverbs_attr_bundle *attrs)904 {....
Quoted 7 lines
927 if (cmd.mw_type != IB_MW_TYPE_1 && cmd.mw_type != IB_MW_TYPE_2) { 928 ret = -EINVAL; 929 goto err_put; 930 }Thanks
You check the type in alloc_mw but you only check the MR access flags if MW_DEBUG is set which is not by default. So you would fail a negative test which we sort of currently have. The second bug in the test which we found this morning is not correctly setting the op flags in the create_qp_ex call. According to the man page only the extended operations set in the op flags are 'implemented' for that QP. Apparently mlx5 goes ahead and populates them. Makes more sense to me since the API as described is kind of overwrought and dumb.
Bob