Skip to contentSkip to conversation

Bug in pyverbs for test_qp_ex_rc_bind_mw

DISCUSSION Bob Pearson 3 participants 4 messages May 18, 2022 – May 19, 2022 1 message not in the archive InfiniBand

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw

-------- Forwarded Message --------
Subject: Re: Bug in pyverbs for test_qp_ex_rc_bind_mw
Date: Tue, 17 May 2022 22:41:08 -0500
From: Bob Pearson <[email protected]>
To: Jason Gunthorpe <[email protected]>, Zhu Yanjun <[email protected]>, Edward Srouji <[email protected]>, Leon Romanovsky <[email protected]>

Quoted 15 lines · Bob Pearson

test_qp_ex_rc_bind_mw has an error in that the new_rkey is computed from the old mr rkey and not the old mw rkey. The following lines

	mw = MW(server.pd, mw_type=e.IBV_MW_TYPE_2)
	...
	new_rkey = inc_rkey(server.mr.rkey)
	server.qp.wr_bind_mw(mw, new_rkey, bind_info)

will compute a new rkey with the same index as mr and a key portion that is one larger than mr modulo 256. This is passed to wr_bind_mw which expects a parameter with a new key portion of the mw (not the mr). The memory windows implementation in rxe generates a random initial rkey for mw and for bind_mw it checks that the new 8 bit key is different than the old key. Since the mr and mw are random wrt each other we expect that the new key will match the old key approx 1 out of 256 test runs which will cause an error which is just what I see.

Quoted 9 lines

The correct code should be

new_key = inc_rkey(<old mw.rkey>)

which will guarantee that it is always different than the previous key. The problem is I can't figure out how to compute the rkey from the mw or I would submit a patch.

Bob

If in test_qpex.py I type

print("mw = ", mw)
print("mr = ", self.server.mr)

I get

mw = MW:
Rkey : 12345678
Handle : 4
MW Type : IBV_MW_TYPE_2

mr = MR
lkey : 432134
rkey : 432134
length : 1024
buf : 9403912345678
handle : 2

The difference is the colon ':' after MW and caps.

I can refer to mr.rkey as self.server.mr.rkey no problem

but mw.Rkey doesn't work. Neither does mw.rkey or anything else I have thought of.

I hate python. Just hate it.

Bob

#2 lore raw
Quoted 45 lines · Bob Pearson

-------- Forwarded Message --------
Subject: Re: Bug in pyverbs for test_qp_ex_rc_bind_mw
Date: Tue, 17 May 2022 22:41:08 -0500
From: Bob Pearson <[email protected]>
To: Jason Gunthorpe <[email protected]>, Zhu Yanjun <[email protected]>, Edward Srouji <[email protected]>, Leon Romanovsky <[email protected]>

Quoted 15 lines · Bob Pearson

test_qp_ex_rc_bind_mw has an error in that the new_rkey is computed from the old mr rkey and not the old mw rkey. The following lines

	mw = MW(server.pd, mw_type=e.IBV_MW_TYPE_2)
	...
	new_rkey = inc_rkey(server.mr.rkey)
	server.qp.wr_bind_mw(mw, new_rkey, bind_info)

will compute a new rkey with the same index as mr and a key portion that is one larger than mr modulo 256. This is passed to wr_bind_mw which expects a parameter with a new key portion of the mw (not the mr). The memory windows implementation in rxe generates a random initial rkey for mw and for bind_mw it checks that the new 8 bit key is different than the old key. Since the mr and mw are random wrt each other we expect that the new key will match the old key approx 1 out of 256 test runs which will cause an error which is just what I see.

Quoted 9 lines

The correct code should be

new_key = inc_rkey(<old mw.rkey>)

which will guarantee that it is always different than the previous key. The problem is I can't figure out how to compute the rkey from the mw or I would submit a patch.

Bob

If in test_qpex.py I type

print("mw = ", mw)
print("mr = ", self.server.mr)

I get

mw = MW:
Rkey : 12345678
Handle : 4
MW Type : IBV_MW_TYPE_2

Quoted 8 lines

mr = MR
lkey : 432134
rkey : 432134
length : 1024
buf : 9403912345678
handle : 2

The difference is the colon ':' after MW and caps.

For the difference, just post a RP to make them identical: https://github.com/linux-rdma/rdma-core/pull/1175

Thanks
Zhijian

Quoted 8 lines

I can refer to mr.rkey as self.server.mr.rkey no problem

but mw.Rkey doesn't work. Neither does mw.rkey or anything else I have thought of.

I hate python. Just hate it.

Bob

#3 lore raw
Quoted 53 lines · On 5/19/2022 6:04 AM, [email protected] wrote:
Quoted 51 lines · Bob Pearson

-------- Forwarded Message --------
Subject: Re: Bug in pyverbs for test_qp_ex_rc_bind_mw
Date: Tue, 17 May 2022 22:41:08 -0500
From: Bob Pearson <[email protected]>
To: Jason Gunthorpe <[email protected]>, Zhu Yanjun <[email protected]>, Edward Srouji <[email protected]>, Leon Romanovsky <[email protected]>

Quoted 15 lines · Bob Pearson

test_qp_ex_rc_bind_mw has an error in that the new_rkey is computed from the old mr rkey and not the old mw rkey. The following lines

	mw = MW(server.pd, mw_type=e.IBV_MW_TYPE_2)
	...
	new_rkey = inc_rkey(server.mr.rkey)
	server.qp.wr_bind_mw(mw, new_rkey, bind_info)

will compute a new rkey with the same index as mr and a key portion that is one larger than mr modulo 256. This is passed to wr_bind_mw which expects a parameter with a new key portion of the mw (not the mr). The memory windows implementation in rxe generates a random initial rkey for mw and for bind_mw it checks that the new 8 bit key is different than the old key. Since the mr and mw are random wrt each other we expect that the new key will match the old key approx 1 out of 256 test runs which will cause an error which is just what I see.

Quoted 9 lines

The correct code should be

new_key = inc_rkey(<old mw.rkey>)

which will guarantee that it is always different than the previous key. The problem is I can't figure out how to compute the rkey from the mw or I would submit a patch.

Bob

If in test_qpex.py I type

print("mw = ", mw)
print("mr = ", self.server.mr)

I get

mw = MW:
Rkey : 12345678
Handle : 4
MW Type : IBV_MW_TYPE_2

mr = MR
lkey : 432134
rkey : 432134
length : 1024
buf : 9403912345678
handle : 2

Quoted 2 lines
Quoted 1 line

The difference is the colon ':' after MW and caps.

For the difference, just post a RP to make them identical: https://github.com/linux-rdma/rdma-core/pull/1175

Thanks for the adjustment.

Quoted 6 lines

Thanks
Zhijian

Quoted 3 lines

I can refer to mr.rkey as self.server.mr.rkey no problem

but mw.Rkey doesn't work. Neither does mw.rkey or anything else I have thought of.

mw.rkey should work... I noticed that you've already figured that out and sent a patch.

We expose the MW rkey in pyverbs (within the cython code). If you still have an issue with it please let me know.

Quoted 2 lines
Quoted 2 lines

I hate python. Just hate it.

Don't hate the game, hate the player :) (in this case).

Quoted 1 line
Quoted 1 line

Bob

#4 lore raw
Quoted 58 lines · Edward Srouji
Quoted 53 lines · On 5/19/2022 6:04 AM, [email protected] wrote:
Quoted 51 lines · Bob Pearson

-------- Forwarded Message --------
Subject: Re: Bug in pyverbs for test_qp_ex_rc_bind_mw
Date: Tue, 17 May 2022 22:41:08 -0500
From: Bob Pearson <[email protected]>
To: Jason Gunthorpe <[email protected]>, Zhu Yanjun <[email protected]>, Edward Srouji <[email protected]>, Leon Romanovsky <[email protected]>

Quoted 15 lines · Bob Pearson

test_qp_ex_rc_bind_mw has an error in that the new_rkey is computed from the old mr rkey and not the old mw rkey. The following lines

    mw = MW(server.pd, mw_type=e.IBV_MW_TYPE_2)
    ...
    new_rkey = inc_rkey(server.mr.rkey)
    server.qp.wr_bind_mw(mw, new_rkey, bind_info)

will compute a new rkey with the same index as mr and a key portion that is one larger than mr modulo 256. This is passed to wr_bind_mw which expects a parameter with a new key portion of the mw (not the mr). The memory windows implementation in rxe generates a random initial rkey for mw and for bind_mw it checks that the new 8 bit key is different than the old key. Since the mr and mw are random wrt each other we expect that the new key will match the old key approx 1 out of 256 test runs which will cause an error which is just what I see.

Quoted 9 lines

The correct code should be

    new_key = inc_rkey(<old mw.rkey>)

which will guarantee that it is always different than the previous key. The problem is I can't figure out how to compute the rkey from the mw or I would submit a patch.

Bob

If in test_qpex.py I type

print("mw = ", mw)
print("mr = ", self.server.mr)

I get

mw = MW:
Rkey        : 12345678
Handle        : 4
MW Type        : IBV_MW_TYPE_2

mr = MR
lkey        : 432134
rkey        : 432134
length        : 1024
buf        : 9403912345678
handle        : 2

Quoted 2 lines
Quoted 1 line

The difference is the colon ':' after MW and caps.

For the difference, just post a RP to make them identical: https://github.com/linux-rdma/rdma-core/pull/1175

Thanks for the adjustment.

Quoted 8 lines
Quoted 6 lines

Thanks
Zhijian

Quoted 3 lines

I can refer to mr.rkey as self.server.mr.rkey no problem

but mw.Rkey doesn't work. Neither does mw.rkey or anything else I have thought of.

mw.rkey should work... I noticed that you've already figured that out and sent a patch.

I can't believe I didn't try that. It was 1 AM and I was tired. A nights sleep resolved the issue and reading the pyx code.

Bob

Quoted 8 lines

We expose the MW rkey in pyverbs (within the cython code). If you still have an issue with it please let me know.

Quoted 2 lines
Quoted 2 lines

I hate python. Just hate it.

Don't hate the game, hate the player :) (in this case).

Quoted 1 line
Quoted 1 line

Bob