Adding private data and private data len as argument to rdma_disconnect()
Hi,
I just wanted to float an idea for an enhancement and get opinions.
There is a provision in ib_cm layer API ib_send_cm_dreq() to send a private_data along with the DREQ.
This private_data is helpful in situations where the receiver of DREQ may want to understand the reason for the DREQ and do some action on the basis of that reason.
We have come across issues where it was critical for the RDS code to understand the reason behind a DREQ and recover/tweak some parameters.
Now, rdma_cm layer has a wrapper around ib_send_cm_dreq() i.e rdma_disconnect(). rdma_disconnect() is used by the consumers of rdma_cm. rdma_disconnect() does not have an argument that accepts private data. Due to this, consumers are not able to use this feature.
In case if we add the arguments private_data and private_data_length to rds_disconnect(), the only challenge would be to add those to the callers of it (NULL and 0).
Please let me know your thoughts on this.
Thanks
Anand
Hi,
A kind reminder.
Thanks,
Anand
Quoted 26 lines · Anand Ashok Khoje
Hi,
I just wanted to float an idea for an enhancement and get opinions.
There is a provision in
ib_cmlayer APIib_send_cm_dreq()to send aprivate_dataalong with the DREQ.
Thisprivate_datais helpful in situations where the receiver of DREQ may want to understand the reason for the DREQ and do some action on the basis of that reason.
We have come across issues where it was critical for the RDS code to understand the reason behind a DREQ and recover/tweak some parameters.Now,
rdma_cmlayer has a wrapper aroundib_send_cm_dreq()i.erdma_disconnect().rdma_disconnect()is used by the consumers ofrdma_cm.rdma_disconnect()does not have an argument that accepts private data. Due to this, consumers are not able to use this feature.In case if we add the arguments
private_dataandprivate_data_lengthtords_disconnect(), the only challenge would be to add those to the callers of it (NULL and 0).Please let me know your thoughts on this.
Thanks
Anand
Quoted 44 lines
-----Original Message-----
From: Anand Ashok Khoje <[email protected]>
Sent: Friday, 13 May 2022 06:10
To: Linux RDMA <[email protected]>; Jason Gunthorpe <[email protected]>; Doug Ledford <[email protected]>
Cc: Rama Nichanamatlu <[email protected]>; Manjunath Patil <[email protected]>
Subject: [EXTERNAL] Re: [RFC] Adding private data and private data len as argument tordma_disconnect()Hi,
A kind reminder.
Thanks,
Anand
Quoted 23 lines · Anand Ashok Khoje
Hi,
I just wanted to float an idea for an enhancement and get opinions.
There is a provision in
ib_cmlayer APIib_send_cm_dreq()to send aprivate_dataalong with the DREQ.
Thisprivate_datais helpful in situations where the receiver of DREQ may want
to understand the reason for the DREQ and do some action on the basis of that
reason.
We have come across issues where it was critical for the RDS code to understand
the reason behind a DREQ and recover/tweak some parameters.Now,
rdma_cmlayer has a wrapper aroundib_send_cm_dreq()i.erdma_disconnect().rdma_disconnect()is used by the consumers ofrdma_cm.rdma_disconnect()does
not have an argument that accepts private data. Due to this, consumers are not
able to use this feature.
Quoted 11 lines
Quoted 1 line
In case if we add the arguments
private_dataandprivate_data_lengthto
Quoted 1 line
rds_disconnect(), the only challenge would be to add those to thecallers
Quoted 7 lines
of it (NULL and 0).
Please let me know your thoughts on this.
Thanks
Anand
Not all providers support the transfer of private data in control messages after connection establishment - rdma_reject()/rdma_accept() being the last opportunity to send private data in connections lifetime for e.g. iWarp connections. Maybe that is why it is not exposed at the disconnect API call? Would RoCE support it?
I think you also ask for extending the CM event infrastructure to carry private data in DISCONNECTED events?
Thanks,
Bernard
Quoted 6 lines · Bernard Metzler
Not all providers support the transfer of private data in control messages after connection establishment - rdma_reject()/rdma_accept() being the last opportunity to send private data in connections lifetime for e.g. iWarp connections. Maybe that is why it is not exposed at the disconnect API call? Would RoCE support it?
RoCE and IB have a place to put it, IB does not.
The lack of support in iWarp is problematic, these APIs are supposed to be fairly high level things.
We clearly can't just extend the existing rdma_disconnect() due to this, and some kind of negotiation would be needed so the ULP can learn if the extra data is supported at all.
Jason
Quoted 10 lines · Jason Gunthorpe
Quoted 6 lines · Bernard Metzler
Not all providers support the transfer of private data in control messages after connection establishment - rdma_reject()/rdma_accept() being the last opportunity to send private data in connections lifetime for e.g. iWarp connections. Maybe that is why it is not exposed at the disconnect API call? Would RoCE support it?
RoCE and IB have a place to put it, IB does not.
I think you meant "iWARP does not", but that's partially incorrect. The Terminate message in iWARP carries a payload which is provided by the layer which caused the termination.
The "layer" 4-bit field and RDMA/DDP/LLP ETypes in:
https://datatracker.ietf.org/doc/html/rfc5040#section-4.8
It has been discussed before to add additional values to iWARP, to carry exactly this kind of "syndrome" data. The discussion could certainly be had again!
Quoted 2 lines
The lack of support in iWarp is problematic, these APIs are supposed to be fairly high level things.
Right. Especially we do not want to define a "protocol-in-protocol" which allows arbitary data to be exchanged here. It needs an interoperable, and testable, specification. For all RDMA protocols, which all ULPs can rely on.
Quoted 3 lines
We clearly can't just extend the existing
rdma_disconnect()due to this, and some kind of negotiation would be needed so the ULP can learn if the extra data is supported at all.
Yep.
Tom.
Quoted 3 lines
Jason