modify srq pyverbs test (new?) failing for the rxe driver
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 10 lines · Bob Pearson
Edward,
There is a test which may be new since this failure didn't happen before. Modify srq tries to set
max_wrto 4094 but the rxe driver rounds this up to 2^n-1 or 4095. My understanding of the IBA spec is that queue sizes can be set larger than requested. Also this same test tries to changemax_sgeusing the sameMAX_WRmask bit. There is no mention (as far as I can recall) in the IBA spec of being able to change themax_sgesetting.
Quoted 3 lines
Is this really the correct behavior?
Bob Pearson
Just for reference
11.2.3.3 MODIFY SHARED RECEIVE QUEUE
Description:
Modifies the attributes of an SRQ for the specified HCA. If any of the modify attributes are invalid, none of the attributes shall be modified.
Input Modifiers:
•HCA handle.
•SRQ handle.
•The SRQ attributes to modify and their new values. The SRQ at- tributes that can be modified after the SRQ has been created are:
•The maximum number of outstanding Work Requests the Consumer expects to submit to the Shared Receive Queue, if resizing of the SRQ is supported by the HCA.
[It does *not* include max_sge.]
•SRQ Limit. If the SRQ Limit is greater than zero, then it shall be armed upon returning from this verb.
Output Modifiers:
•The actual number of outstanding Work Requests supported on the Shared Receive Queue. If an error is not returned, this is guaranteed to be greater than or equal to the number requested. (This may require the Consumer to increase the size of the CQ.)
[Unfortunately the rdma verbs API does not support returning the new value of max_wr. You have to call ib_query_srq to get the new value which does *not* have to equal the requested size.]
•Verb Results:
•Operation completed successfully. •Insufficient resources to complete request. •Invalid HCA handle.
•Invalid SRQ handle.
•SRQ is in the Error State.
•HCA does not support resizing SRQ.
•Maximum number of Work Requests requested exceeds HCA capability.
•SRQ Limit exceeds maximum number of Work Requests al- lowed on the SRQ.
•More outstanding entries on WQ than size specified. •HCA does not support SRQ.
[The test_modify_srq just seems incorrect.]
You're right.
I've patched the test and pushed it Upstream (see PR link below [1]). Please notify me if you're still having an issue after applying the patch.
Quoted 62 lines · Bob Pearson
External email: Use caution opening links or attachments
Quoted 10 lines · Bob Pearson
Edward,
There is a test which may be new since this failure didn't happen before. Modify srq tries to set
max_wrto 4094 but the rxe driver rounds this up to 2^n-1 or 4095. My understanding of the IBA spec is that queue sizes can be set larger than requested. Also this same test tries to changemax_sgeusing the sameMAX_WRmask bit. There is no mention (as far as I can recall) in the IBA spec of being able to change themax_sgesetting.Quoted 3 lines
Is this really the correct behavior?
Bob Pearson
Just for reference
11.2.3.3 MODIFY SHARED RECEIVE QUEUE
Description:
Modifies the attributes of an SRQ for the specified HCA. If any of the modify attributes are invalid, none of the attributes shall be modified.
Input Modifiers:
•HCA handle.
•SRQ handle.
•The SRQ attributes to modify and their new values. The SRQ at- tributes that can be modified after the SRQ has been created are:
•The maximum number of outstanding Work Requests the Consumer expects to submit to the Shared Receive Queue, if resizing of the SRQ is supported by the HCA.[It does *not* include
max_sge.]•SRQ Limit. If the SRQ Limit is greater than zero, then it shall be armed upon returning from this verb.
Output Modifiers:
•The actual number of outstanding Work Requests supported on the Shared Receive Queue. If an error is not returned, this is guaranteed to be greater than or equal to the number requested. (This may require the Consumer to increase the size of the CQ.)[Unfortunately the rdma verbs API does not support returning the new value of
max_wr. You have to callib_query_srqto get the new value which does *not* have to equal the requested size.]•Verb Results:
•Operation completed successfully. •Insufficient resources to complete request. •Invalid HCA handle.
•Invalid SRQ handle.
•SRQ is in the Error State.
•HCA does not support resizing SRQ.
•Maximum number of Work Requests requested exceeds HCA capability.
•SRQ Limit exceeds maximum number of Work Requests al- lowed on the SRQ.
•More outstanding entries on WQ than size specified. •HCA does not support SRQ.
Quoted 1 line
[The
test_modify_srqjust seems incorrect.]
[1] https://github.com/linux-rdma/rdma-core/pull/1402
Thanks.
Quoted 3 lines · Edward Srouji
You're right.
I've patched the test and pushed it Upstream (see PR link below [1]). Please notify me if you're still having an issue after applying the patch.
Will do. Thanks!
Bob
Quoted 19 lines · Bob Pearson
Quoted 17 lines
External email: Use caution opening links or attachments
Quoted 13 lines · Bob Pearson
Edward,
There is a test which may be new since this failure didn't happen before. Modify srq tries to set
max_wrto 4094 but the rxe driver rounds this up to 2^n-1 or 4095. My understanding of the IBA spec is that queue sizes can be set larger than requested. Also this same test tries to changemax_sgeusing the sameMAX_WRmask bit. There is no mention (as far as I can recall) in the IBA spec of being able to change themax_sgesetting.Is this really the correct behavior?
Bob Pearson
Quoted 4 lines · Edward Srouji
You're right.
I've patched the test and pushed it Upstream (see PR link below [1]). Please notify me if you're still having an issue after applying the patch.
Works fine now. Thanks.
Bob