Skip to contentSkip to conversation

modify srq pyverbs test (new?) failing for the rxe driver

DISCUSSION Bob Pearson 2 participants 4 messages Oct 24, 2023 – Oct 26, 2023 1 message not in the archive InfiniBand

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw
Quoted 10 lines · Bob Pearson

Edward,

There is a test which may be new since this failure didn't happen before. Modify srq tries to set max_wr to 4094 but the rxe driver rounds this up to 2^n-1 or 4095. My understanding of the IBA spec is that queue sizes can be set larger than requested. Also this same test tries to change max_sge using the same MAX_WR mask bit. There is no mention (as far as I can recall) in the IBA spec of being able to change the max_sge setting.

Quoted 3 lines

Is this really the correct behavior?

Bob Pearson

Just for reference

11.2.3.3 MODIFY SHARED RECEIVE QUEUE
Description:
Modifies the attributes of an SRQ for the specified HCA. If any of the modify attributes are invalid, none of the attributes shall be modified.
Input Modifiers:
•HCA handle.
•SRQ handle.
•The SRQ attributes to modify and their new values. The SRQ at- tributes that can be modified after the SRQ has been created are:
•The maximum number of outstanding Work Requests the Consumer expects to submit to the Shared Receive Queue, if resizing of the SRQ is supported by the HCA.

[It does *not* include max_sge.]

•SRQ Limit. If the SRQ Limit is greater than zero, then it shall be armed upon returning from this verb.
Output Modifiers:
•The actual number of outstanding Work Requests supported on the Shared Receive Queue. If an error is not returned, this is guaranteed to be greater than or equal to the number requested. (This may require the Consumer to increase the size of the CQ.)

[Unfortunately the rdma verbs API does not support returning the new value of max_wr. You have to call ib_query_srq to get the new value which does *not* have to equal the requested size.]

•Verb Results:
•Operation completed successfully. •Insufficient resources to complete request. •Invalid HCA handle.
•Invalid SRQ handle.
•SRQ is in the Error State.
•HCA does not support resizing SRQ.
•Maximum number of Work Requests requested exceeds HCA capability.
•SRQ Limit exceeds maximum number of Work Requests al- lowed on the SRQ.
•More outstanding entries on WQ than size specified. •HCA does not support SRQ.

[The test_modify_srq just seems incorrect.]

#2 lore raw

You're right.
I've patched the test and pushed it Upstream (see PR link below [1]). Please notify me if you're still having an issue after applying the patch.

Quoted 62 lines · Bob Pearson

External email: Use caution opening links or attachments

Quoted 10 lines · Bob Pearson

Edward,

There is a test which may be new since this failure didn't happen before. Modify srq tries to set max_wr to 4094 but the rxe driver rounds this up to 2^n-1 or 4095. My understanding of the IBA spec is that queue sizes can be set larger than requested. Also this same test tries to change max_sge using the same MAX_WR mask bit. There is no mention (as far as I can recall) in the IBA spec of being able to change the max_sge setting.

Quoted 3 lines

Is this really the correct behavior?

Bob Pearson

Just for reference

11.2.3.3 MODIFY SHARED RECEIVE QUEUE
Description:
  Modifies the attributes of an SRQ for the specified HCA. If any of the modify attributes are invalid, none of the attributes shall be modified.
  Input Modifiers:
    •HCA handle.
    •SRQ handle.
    •The SRQ attributes to modify and their new values. The SRQ at- tributes that can be modified after the SRQ has been created are:
      •The maximum number of outstanding Work Requests the Consumer expects to submit to the Shared Receive Queue, if resizing of the SRQ is supported by the HCA.

[It does *not* include max_sge.]

      •SRQ Limit. If the SRQ Limit is greater than zero, then it shall be armed upon returning from this verb.
  Output Modifiers:
    •The actual number of outstanding Work Requests supported on the Shared Receive Queue. If an error is not returned, this is guaranteed to be greater than or equal to the number requested. (This may require the Consumer to increase the size of the CQ.)

[Unfortunately the rdma verbs API does not support returning the new value of max_wr. You have to call ib_query_srq to get the new value which does *not* have to equal the requested size.]

    •Verb Results:
    •Operation completed successfully. •Insufficient resources to complete request. •Invalid HCA handle.
    •Invalid SRQ handle.
    •SRQ is in the Error State.
    •HCA does not support resizing SRQ.
    •Maximum number of Work Requests requested exceeds HCA capability.
    •SRQ Limit exceeds maximum number of Work Requests al- lowed on the SRQ.
    •More outstanding entries on WQ than size specified. •HCA does not support SRQ.

Quoted 1 line

[The test_modify_srq just seems incorrect.]

[1] https://github.com/linux-rdma/rdma-core/pull/1402

Thanks.

#3 lore raw
Quoted 3 lines · Edward Srouji

You're right.
I've patched the test and pushed it Upstream (see PR link below [1]). Please notify me if you're still having an issue after applying the patch.

Will do. Thanks!

Bob

Quoted 19 lines · Bob Pearson
Quoted 17 lines

External email: Use caution opening links or attachments

Quoted 13 lines · Bob Pearson

Edward,

There is a test which may be new since this failure didn't happen before. Modify srq tries to set max_wr to 4094 but the rxe driver rounds this up to 2^n-1 or 4095. My understanding of the IBA spec is that queue sizes can be set larger than requested. Also this same test tries to change max_sge using the same MAX_WR mask bit. There is no mention (as far as I can recall) in the IBA spec of being able to change the max_sge setting.

Is this really the correct behavior?

Bob Pearson

#4 lore raw
In reply to Edward Srouji
Quoted 4 lines · Edward Srouji

You're right.
I've patched the test and pushed it Upstream (see PR link below [1]). Please notify me if you're still having an issue after applying the patch.

Works fine now. Thanks.

Bob