Skip to contentSkip to conversation

Adding private data and private data len as argument to rdma_disconnect()

RFC Anand Ashok Khoje 4 participants 5 messages May 5, 2022 – May 13, 2022 InfiniBand

lore
Share & tools
lore
#0 lore raw

Hi,

I just wanted to float an idea for an enhancement and get opinions.

There is a provision in ib_cm layer API ib_send_cm_dreq() to send a private_data along with the DREQ.
This private_data is helpful in situations where the receiver of DREQ may want to understand the reason for the DREQ and do some action on the basis of that reason.
We have come across issues where it was critical for the RDS code to understand the reason behind a DREQ and recover/tweak some parameters.

Now, rdma_cm layer has a wrapper around ib_send_cm_dreq() i.e rdma_disconnect(). rdma_disconnect() is used by the consumers of rdma_cm. rdma_disconnect() does not have an argument that accepts private data. Due to this, consumers are not able to use this feature.

In case if we add the arguments private_data and private_data_length to rds_disconnect(), the only challenge would be to add those to the callers of it (NULL and 0).

Please let me know your thoughts on this.

Thanks
Anand

#1 lore raw

Hi,

A kind reminder.

Thanks,

Anand

Quoted 26 lines · Anand Ashok Khoje

Hi,

I just wanted to float an idea for an enhancement and get opinions.

There is a provision in ib_cm layer API ib_send_cm_dreq() to send a private_data along with the DREQ.
This private_data is helpful in situations where the receiver of DREQ may want to understand the reason for the DREQ and do some action on the basis of that reason.
We have come across issues where it was critical for the RDS code to understand the reason behind a DREQ and recover/tweak some parameters.

Now, rdma_cm layer has a wrapper around ib_send_cm_dreq() i.e rdma_disconnect(). rdma_disconnect() is used by the consumers of rdma_cm. rdma_disconnect() does not have an argument that accepts private data. Due to this, consumers are not able to use this feature.

In case if we add the arguments private_data and private_data_length to rds_disconnect(), the only challenge would be to add those to the callers of it (NULL and 0).

Please let me know your thoughts on this.

Thanks
Anand

#2 lore raw
Quoted 44 lines

-----Original Message-----
From: Anand Ashok Khoje <[email protected]>
Sent: Friday, 13 May 2022 06:10
To: Linux RDMA <[email protected]>; Jason Gunthorpe <[email protected]>; Doug Ledford <[email protected]>
Cc: Rama Nichanamatlu <[email protected]>; Manjunath Patil <[email protected]>
Subject: [EXTERNAL] Re: [RFC] Adding private data and private data len as argument to rdma_disconnect()

Hi,

A kind reminder.

Thanks,

Anand

Quoted 23 lines · Anand Ashok Khoje

Hi,

I just wanted to float an idea for an enhancement and get opinions.

There is a provision in ib_cm layer API ib_send_cm_dreq() to send a private_data along with the DREQ.
This private_data is helpful in situations where the receiver of DREQ may want
to understand the reason for the DREQ and do some action on the basis of that
reason.
We have come across issues where it was critical for the RDS code to understand
the reason behind a DREQ and recover/tweak some parameters.

Now, rdma_cm layer has a wrapper around ib_send_cm_dreq() i.e rdma_disconnect().
rdma_disconnect() is used by the consumers of rdma_cm. rdma_disconnect() does
not have an argument that accepts private data. Due to this, consumers are not
able to use this feature.

Quoted 11 lines
Quoted 1 line

In case if we add the arguments private_data and private_data_length

to

Quoted 1 line

rds_disconnect(), the only challenge would be to add those to the

callers

Quoted 7 lines

of it (NULL and 0).

Please let me know your thoughts on this.

Thanks
Anand

Not all providers support the transfer of private data in control messages after connection establishment - rdma_reject()/rdma_accept() being the last opportunity to send private data in connections lifetime for e.g. iWarp connections. Maybe that is why it is not exposed at the disconnect API call? Would RoCE support it?

I think you also ask for extending the CM event infrastructure to carry private data in DISCONNECTED events?

Thanks,
Bernard

#3 lore raw
Quoted 6 lines · Bernard Metzler

Not all providers support the transfer of private data in control messages after connection establishment - rdma_reject()/rdma_accept() being the last opportunity to send private data in connections lifetime for e.g. iWarp connections. Maybe that is why it is not exposed at the disconnect API call? Would RoCE support it?

RoCE and IB have a place to put it, IB does not.

The lack of support in iWarp is problematic, these APIs are supposed to be fairly high level things.

We clearly can't just extend the existing rdma_disconnect() due to this, and some kind of negotiation would be needed so the ULP can learn if the extra data is supported at all.

Jason

#4 lore raw
Quoted 10 lines · Jason Gunthorpe
Quoted 6 lines · Bernard Metzler

Not all providers support the transfer of private data in control messages after connection establishment - rdma_reject()/rdma_accept() being the last opportunity to send private data in connections lifetime for e.g. iWarp connections. Maybe that is why it is not exposed at the disconnect API call? Would RoCE support it?

RoCE and IB have a place to put it, IB does not.

I think you meant "iWARP does not", but that's partially incorrect. The Terminate message in iWARP carries a payload which is provided by the layer which caused the termination.

The "layer" 4-bit field and RDMA/DDP/LLP ETypes in:
https://datatracker.ietf.org/doc/html/rfc5040#section-4.8

It has been discussed before to add additional values to iWARP, to carry exactly this kind of "syndrome" data. The discussion could certainly be had again!

Quoted 2 lines

The lack of support in iWarp is problematic, these APIs are supposed to be fairly high level things.

Right. Especially we do not want to define a "protocol-in-protocol" which allows arbitary data to be exchanged here. It needs an interoperable, and testable, specification. For all RDMA protocols, which all ULPs can rely on.

Quoted 3 lines

We clearly can't just extend the existing rdma_disconnect() due to this, and some kind of negotiation would be needed so the ULP can learn if the extra data is supported at all.

Yep.

Tom.

Quoted 3 lines

Jason