Introduce mlx5 data direct placement (DDP)

From: Leon Romanovsky <[email protected]>

Hi,

This series from Edward introduces mlx5 data direct placement (DDP) feature.

This feature allows WRs on the receiver side of the QP to be consumed out of order, permitting the sender side to transmit messages without guaranteeing arrival order on the receiver side.

When enabled, the completion ordering of WRs remains in-order, regardless of the Receive WRs consumption order.

RDMA Read and RDMA Atomic operations on the responder side continue to be executed in-order, while the ordering of data placement for RDMA Write and Send operations is not guaranteed.

Thanks

Edward Srouji (2):
net/mlx5: Introduce data placement ordering bits RDMA/mlx5: Support OOO RX WQE consumption

  drivers/infiniband/hw/mlx5/main.c    |  8 +++++
  drivers/infiniband/hw/mlx5/mlx5_ib.h |  1 +
  drivers/infiniband/hw/mlx5/qp.c      | 51 +++++++++++++++++++++++++---
  include/linux/mlx5/mlx5_ifc.h        | 24 +++++++++----
  include/uapi/rdma/mlx5-abi.h         |  5 +++
  5 files changed, 78 insertions(+), 11 deletions(-)

External email: Use caution opening links or attachments

Quoted 14 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

Hi,

This series from Edward introduces mlx5 data direct placement (DDP) feature.

This feature allows WRs on the receiver side of the QP to be consumed out of order, permitting the sender side to transmit messages without guaranteeing arrival order on the receiver side.

When enabled, the completion ordering of WRs remains in-order, regardless of the Receive WRs consumption order.

Quoted 3 lines

RDMA Read and RDMA Atomic operations on the responder side continue to be executed in-order, while the ordering of data placement for RDMA Write and Send operations is not guaranteed.

It is an interesting feature. If I got this feature correctly, this feature permits the user consumes the data out of order when RDMA Write and Send operations. But its completiong ordering is still in order.

Any scenario that this feature can be applied and what benefits will be got from this feature?

I am just curious about this. Normally the users will consume the data in order. In what scenario, the user will consume the data out of order?

Thanks,
Zhu Yanjun

Quoted 14 lines

Thanks

Edward Srouji (2):
   net/mlx5: Introduce data placement ordering bits RDMA/mlx5: Support OOO RX WQE consumption

  drivers/infiniband/hw/mlx5/main.c    |  8 +++++ drivers/infiniband/hw/mlx5/mlx5_ib.h |  1 + drivers/infiniband/hw/mlx5/qp.c      | 51 +++++++++++++++++++++++++--- include/linux/mlx5/mlx5_ifc.h        | 24 +++++++++---- include/uapi/rdma/mlx5-abi.h         |  5 +++

  5 files changed, 78 insertions(+), 11 deletions(-)

Quoted 18 lines · Zhu Yanjun

External email: Use caution opening links or attachments

Quoted 14 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

Hi,

This series from Edward introduces mlx5 data direct placement (DDP) feature.

This feature allows WRs on the receiver side of the QP to be consumed out of order, permitting the sender side to transmit messages without guaranteeing arrival order on the receiver side.

When enabled, the completion ordering of WRs remains in-order, regardless of the Receive WRs consumption order.

Quoted 8 lines

Quoted 3 lines

RDMA Read and RDMA Atomic operations on the responder side continue to be executed in-order, while the ordering of data placement for RDMA Write and Send operations is not guaranteed.

It is an interesting feature. If I got this feature correctly, this feature permits the user consumes the data out of order when RDMA Write and Send operations. But its completiong ordering is still in order.

Correct.

Quoted 6 lines

Any scenario that this feature can be applied and what benefits will be got from this feature?

I am just curious about this. Normally the users will consume the data in order. In what scenario, the user will consume the data out of order?

One of the main benefits of this feature is achieving higher bandwidth (BW) by allowing responders to receive packets out of order (OOO).

For example, this can be utilized in devices that support multi-plane functionality, as introduced in the "Multi-plane support for mlx5" series [1]. When mlx5 multi-plane is supported, a single logical mlx5 port aggregates multiple physical plane ports. In this scenario, the requester can "spray" packets across the multiple physical plane ports without guaranteeing packet order, either on the wire or on the receiver (responder) side.

With this approach, no barriers or fences are required to ensure in-order packet reception, which optimizes the data path for performance. This can result in better BW, theoretically achieving line-rate performance equivalent to the sum of the maximum BW of all physical plane ports, with only one QP.

[1] https://lore.kernel.org/lkml/[email protected]/

Quoted 19 lines

Thanks,
Zhu Yanjun

Quoted 15 lines

Thanks

Edward Srouji (2):
   net/mlx5: Introduce data placement ordering bits RDMA/mlx5: Support OOO RX WQE consumption

  drivers/infiniband/hw/mlx5/main.c    |  8 +++++ drivers/infiniband/hw/mlx5/mlx5_ib.h |  1 + drivers/infiniband/hw/mlx5/qp.c      | 51
+++++++++++++++++++++++++---
  include/linux/mlx5/mlx5_ifc.h        | 24 +++++++++---- include/uapi/rdma/mlx5-abi.h         |  5 +++

  5 files changed, 78 insertions(+), 11 deletions(-)

External email: Use caution opening links or attachments

Quoted 51 lines · Edward Srouji

Quoted 18 lines · Zhu Yanjun

External email: Use caution opening links or attachments

Quoted 14 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

Hi,

This series from Edward introduces mlx5 data direct placement (DDP) feature.

This feature allows WRs on the receiver side of the QP to be consumed out of order, permitting the sender side to transmit messages without guaranteeing arrival order on the receiver side.

When enabled, the completion ordering of WRs remains in-order, regardless of the Receive WRs consumption order.

Quoted 8 lines

Quoted 3 lines

RDMA Read and RDMA Atomic operations on the responder side continue to be executed in-order, while the ordering of data placement for RDMA Write and Send operations is not guaranteed.

It is an interesting feature. If I got this feature correctly, this feature permits the user consumes the data out of order when RDMA Write and Send operations. But its completiong ordering is still in order.

Correct.

Quoted 6 lines

Any scenario that this feature can be applied and what benefits will be got from this feature?

I am just curious about this. Normally the users will consume the data in order. In what scenario, the user will consume the data out of order?

One of the main benefits of this feature is achieving higher bandwidth (BW) by allowing
responders to receive packets out of order (OOO).

For example, this can be utilized in devices that support multi-plane functionality,
as introduced in the "Multi-plane support for mlx5" series [1]. When mlx5 multi-plane
is supported, a single logical mlx5 port aggregates multiple physical plane ports.
In this scenario, the requester can "spray" packets across the multiple physical
plane ports without guaranteeing packet order, either on the wire or on the receiver
(responder) side.

Quoted 7 lines

With this approach, no barriers or fences are required to ensure in-order packet
reception, which optimizes the data path for performance. This can result in better
BW, theoretically achieving line-rate performance equivalent to the sum of
the maximum BW of all physical plane ports, with only one QP.

Thanks a lot for your quick reply. Without ensuring in-order packet reception, this does optimize the data path for performance.

I agree with you.

But how does the receiver get the correct packets from the out-of-order packets efficiently?

The method is implemented in Software or Hardware?

I am just interested in this feature and want to know more about this.

Thanks,

Zhu Yanjun

Quoted 21 lines

[1] https://lore.kernel.org/lkml/[email protected]/

Quoted 19 lines

Thanks,
Zhu Yanjun

Quoted 15 lines

Thanks

Edward Srouji (2):
   net/mlx5: Introduce data placement ordering bits RDMA/mlx5: Support OOO RX WQE consumption

  drivers/infiniband/hw/mlx5/main.c    |  8 +++++ drivers/infiniband/hw/mlx5/mlx5_ib.h |  1 + drivers/infiniband/hw/mlx5/qp.c      | 51
+++++++++++++++++++++++++---
  include/linux/mlx5/mlx5_ifc.h        | 24 +++++++++---- include/uapi/rdma/mlx5-abi.h         |  5 +++

  5 files changed, 78 insertions(+), 11 deletions(-)

Signature · 2 lines

Best Regards,
Yanjun.Zhu

Quoted 66 lines · Zhu Yanjun

External email: Use caution opening links or attachments

Quoted 51 lines · Edward Srouji

Quoted 18 lines · Zhu Yanjun

External email: Use caution opening links or attachments

Quoted 14 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

Hi,

This series from Edward introduces mlx5 data direct placement (DDP) feature.

This feature allows WRs on the receiver side of the QP to be consumed out of order, permitting the sender side to transmit messages without guaranteeing arrival order on the receiver side.

When enabled, the completion ordering of WRs remains in-order, regardless of the Receive WRs consumption order.

Quoted 8 lines

Quoted 3 lines

RDMA Read and RDMA Atomic operations on the responder side continue to be executed in-order, while the ordering of data placement for RDMA Write and Send operations is not guaranteed.

It is an interesting feature. If I got this feature correctly, this feature permits the user consumes the data out of order when RDMA Write and Send operations. But its completiong ordering is still in order.

Correct.

Quoted 6 lines

Any scenario that this feature can be applied and what benefits will be got from this feature?

I am just curious about this. Normally the users will consume the data in order. In what scenario, the user will consume the data out of order?

One of the main benefits of this feature is achieving higher bandwidth (BW) by allowing
responders to receive packets out of order (OOO).

For example, this can be utilized in devices that support multi-plane functionality,
as introduced in the "Multi-plane support for mlx5" series [1]. When mlx5 multi-plane
is supported, a single logical mlx5 port aggregates multiple physical plane ports.
In this scenario, the requester can "spray" packets across the multiple physical
plane ports without guaranteeing packet order, either on the wire or on the receiver
(responder) side.

Quoted 7 lines

With this approach, no barriers or fences are required to ensure in-order packet
reception, which optimizes the data path for performance. This can result in better
BW, theoretically achieving line-rate performance equivalent to the sum of
the maximum BW of all physical plane ports, with only one QP.

Thanks a lot for your quick reply. Without ensuring in-order packet reception, this does optimize the data path for performance.

Quoted 6 lines

I agree with you.

But how does the receiver get the correct packets from the out-of-order packets efficiently?

The method is implemented in Software or Hardware?

The packets have new field that is used by the HW to understand the correct message order (similar to PSN).

Once the packets arrive OOO to the receiver side, the data is scattered directly (hence the DDP - "Direct Data Placement" name) by the HW.

So the efficiency is achieved by the HW, as it also saves the required context and metadata so it can deliver the correct completion to the user (in-order) once we have some WQEs that can be considered an "in-order window" and be delivered to the user.

The SW/Applications may receive OOO WR_IDs though (because the first CQE may have consumed Recv WQE of any index on the receiver side), and it's their responsibility to handle it from this point, if it's required.

Quoted 33 lines

I am just interested in this feature and want to know more about this.

Thanks,

Zhu Yanjun

Quoted 21 lines

[1] https://lore.kernel.org/lkml/[email protected]/

Quoted 19 lines

Thanks,
Zhu Yanjun

Quoted 15 lines

Thanks

Edward Srouji (2):
   net/mlx5: Introduce data placement ordering bits RDMA/mlx5: Support OOO RX WQE consumption

  drivers/infiniband/hw/mlx5/main.c    |  8 +++++ drivers/infiniband/hw/mlx5/mlx5_ib.h |  1 + drivers/infiniband/hw/mlx5/qp.c      | 51
+++++++++++++++++++++++++---
  include/linux/mlx5/mlx5_ifc.h        | 24 +++++++++---- include/uapi/rdma/mlx5-abi.h         |  5 +++

  5 files changed, 78 insertions(+), 11 deletions(-)

Signature · 2 lines

Best Regards,
Yanjun.Zhu

External email: Use caution opening links or attachments

Quoted 83 lines · Edward Srouji

Quoted 67 lines · Zhu Yanjun

External email: Use caution opening links or attachments

Quoted 52 lines · Edward Srouji

Quoted 18 lines · Zhu Yanjun

External email: Use caution opening links or attachments

Quoted 14 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

Hi,

This series from Edward introduces mlx5 data direct placement (DDP) feature.

This feature allows WRs on the receiver side of the QP to be consumed out of order, permitting the sender side to transmit messages without guaranteeing arrival order on the receiver side.

When enabled, the completion ordering of WRs remains in-order, regardless of the Receive WRs consumption order.

Quoted 8 lines

Quoted 3 lines

RDMA Read and RDMA Atomic operations on the responder side continue to be executed in-order, while the ordering of data placement for RDMA Write and Send operations is not guaranteed.

It is an interesting feature. If I got this feature correctly, this feature permits the user consumes the data out of order when RDMA Write and Send operations. But its completiong ordering is still in order.

Correct.

Quoted 7 lines

Any scenario that this feature can be applied and what benefits will be got from this feature?

I am just curious about this. Normally the users will consume the data in order. In what scenario, the user will consume the data out of order?

One of the main benefits of this feature is achieving higher bandwidth (BW) by allowing
responders to receive packets out of order (OOO).

For example, this can be utilized in devices that support multi-plane functionality,
as introduced in the "Multi-plane support for mlx5" series [1]. When mlx5 multi-plane
is supported, a single logical mlx5 port aggregates multiple physical plane ports.
In this scenario, the requester can "spray" packets across the multiple physical
plane ports without guaranteeing packet order, either on the wire or on the receiver
(responder) side.

Quoted 7 lines

With this approach, no barriers or fences are required to ensure in-order packet
reception, which optimizes the data path for performance. This can result in better
BW, theoretically achieving line-rate performance equivalent to the sum of
the maximum BW of all physical plane ports, with only one QP.

Thanks a lot for your quick reply. Without ensuring in-order packet reception, this does optimize the data path for performance.

Quoted 6 lines

I agree with you.

But how does the receiver get the correct packets from the out-of-order packets efficiently?

The method is implemented in Software or Hardware?

The packets have new field that is used by the HW to understand the correct message order (similar to PSN).

Once the packets arrive OOO to the receiver side, the data is scattered directly (hence the DDP - "Direct Data Placement" name) by the HW.

Quoted 8 lines

So the efficiency is achieved by the HW, as it also saves the required context and metadata so it can deliver the correct completion to the user (in-order) once we have some WQEs that can be considered an "in-order window" and be delivered to the user.

The SW/Applications may receive OOO WR_IDs though (because the first CQE may have consumed Recv WQE of any index on the receiver side), and it's their responsibility to handle it from this point, if it's required.

Got it. It seems that all the functionalities are implemented in HW. The SW only receives OOO WR_IDs. Thanks a lot. Perhaps it is helpful to RDMA LAG devices. It should enhance the performance^_^

BTW, do you have any performance data with this feature?

Best Regards,
Zhu Yanjun

Quoted 33 lines

Quoted 33 lines

I am just interested in this feature and want to know more about this.

Thanks,

Zhu Yanjun

Quoted 21 lines

[1] https://lore.kernel.org/lkml/[email protected]/

Quoted 19 lines

Thanks,
Zhu Yanjun

Quoted 15 lines

Thanks

Edward Srouji (2):
   net/mlx5: Introduce data placement ordering bits RDMA/mlx5: Support OOO RX WQE consumption

  drivers/infiniband/hw/mlx5/main.c    |  8 +++++ drivers/infiniband/hw/mlx5/mlx5_ib.h |  1 + drivers/infiniband/hw/mlx5/qp.c      | 51
+++++++++++++++++++++++++---
  include/linux/mlx5/mlx5_ifc.h        | 24 +++++++++---- include/uapi/rdma/mlx5-abi.h         |  5 +++

  5 files changed, 78 insertions(+), 11 deletions(-)

Signature · 2 lines

Best Regards,
Yanjun.Zhu

Quoted 100 lines · Zhu Yanjun

External email: Use caution opening links or attachments

Quoted 83 lines · Edward Srouji

Quoted 67 lines · Zhu Yanjun

External email: Use caution opening links or attachments

Quoted 52 lines · Edward Srouji

Quoted 18 lines · Zhu Yanjun

External email: Use caution opening links or attachments

Quoted 14 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

Hi,

This series from Edward introduces mlx5 data direct placement (DDP) feature.

This feature allows WRs on the receiver side of the QP to be consumed out of order, permitting the sender side to transmit messages without guaranteeing arrival order on the receiver side.

When enabled, the completion ordering of WRs remains in-order, regardless of the Receive WRs consumption order.

Quoted 8 lines

Quoted 3 lines

RDMA Read and RDMA Atomic operations on the responder side continue to be executed in-order, while the ordering of data placement for RDMA Write and Send operations is not guaranteed.

It is an interesting feature. If I got this feature correctly, this feature permits the user consumes the data out of order when RDMA Write and Send operations. But its completiong ordering is still in order.

Correct.

Quoted 7 lines

Any scenario that this feature can be applied and what benefits will be got from this feature?

I am just curious about this. Normally the users will consume the data in order. In what scenario, the user will consume the data out of order?

One of the main benefits of this feature is achieving higher bandwidth (BW) by allowing
responders to receive packets out of order (OOO).

For example, this can be utilized in devices that support multi-plane functionality,
as introduced in the "Multi-plane support for mlx5" series [1]. When mlx5 multi-plane
is supported, a single logical mlx5 port aggregates multiple physical plane ports.
In this scenario, the requester can "spray" packets across the multiple physical
plane ports without guaranteeing packet order, either on the wire or on the receiver
(responder) side.

Quoted 7 lines

With this approach, no barriers or fences are required to ensure in-order packet
reception, which optimizes the data path for performance. This can result in better
BW, theoretically achieving line-rate performance equivalent to the sum of
the maximum BW of all physical plane ports, with only one QP.

Thanks a lot for your quick reply. Without ensuring in-order packet reception, this does optimize the data path for performance.

Quoted 6 lines

I agree with you.

But how does the receiver get the correct packets from the out-of-order packets efficiently?

The method is implemented in Software or Hardware?

The packets have new field that is used by the HW to understand the correct message order (similar to PSN).

Once the packets arrive OOO to the receiver side, the data is scattered directly (hence the DDP - "Direct Data Placement" name) by the HW.

Quoted 8 lines

So the efficiency is achieved by the HW, as it also saves the required context and metadata so it can deliver the correct completion to the user (in-order) once we have some WQEs that can be considered an "in-order window" and be delivered to the user.

The SW/Applications may receive OOO WR_IDs though (because the first CQE may have consumed Recv WQE of any index on the receiver side), and it's their responsibility to handle it from this point, if it's required.

Got it. It seems that all the functionalities are implemented in HW. The SW only receives OOO WR_IDs. Thanks a lot. Perhaps it is helpful to RDMA LAG devices. It should enhance the performance^_^

Quoted 1 line

BTW, do you have any performance data with this feature?

Not yet. We tested it functionality wise for now.

But we should be able to measure its performance soon :).

Quoted 40 lines

Best Regards,
Zhu Yanjun

Quoted 33 lines

Quoted 33 lines

I am just interested in this feature and want to know more about this.

Thanks,

Zhu Yanjun

Quoted 21 lines

[1] https://lore.kernel.org/lkml/[email protected]/

Quoted 19 lines

Thanks,
Zhu Yanjun

Quoted 15 lines

Thanks

Edward Srouji (2):
   net/mlx5: Introduce data placement ordering bits RDMA/mlx5: Support OOO RX WQE consumption

  drivers/infiniband/hw/mlx5/main.c    |  8 +++++ drivers/infiniband/hw/mlx5/mlx5_ib.h |  1 + drivers/infiniband/hw/mlx5/qp.c      | 51
+++++++++++++++++++++++++---
  include/linux/mlx5/mlx5_ifc.h        | 24 +++++++++---- include/uapi/rdma/mlx5-abi.h         |  5 +++

  5 files changed, 78 insertions(+), 11 deletions(-)

Signature · 2 lines

Best Regards,
Yanjun.Zhu

External email: Use caution opening links or attachments

Quoted 111 lines · Edward Srouji

Quoted 109 lines · Zhu Yanjun

External email: Use caution opening links or attachments

Quoted 96 lines · Edward Srouji

Quoted 74 lines · Zhu Yanjun

External email: Use caution opening links or attachments

Quoted 59 lines · Edward Srouji

Quoted 25 lines · Zhu Yanjun

External email: Use caution opening links or attachments

Quoted 13 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

Hi,

This series from Edward introduces mlx5 data direct placement (DDP) feature.

This feature allows WRs on the receiver side of the QP to be consumed
out of order, permitting the sender side to transmit messages without
guaranteeing arrival order on the receiver side.

Quoted 7 lines

When enabled, the completion ordering of WRs remains in-order, regardless of the Receive WRs consumption order.

RDMA Read and RDMA Atomic operations on the responder side continue to
be executed in-order, while the ordering of data placement for RDMA Write and Send operations is not guaranteed.

Quoted 5 lines

It is an interesting feature. If I got this feature correctly, this feature permits the user consumes the data out of order when RDMA Write
and Send operations. But its completiong ordering is still in order.

Correct.

Quoted 9 lines

Any scenario that this feature can be applied and what benefits will be
got from this feature?

I am just curious about this. Normally the users will consume the data
in order. In what scenario, the user will consume the data out of order?

One of the main benefits of this feature is achieving higher bandwidth
(BW) by allowing
responders to receive packets out of order (OOO).

For example, this can be utilized in devices that support multi-plane functionality,
as introduced in the "Multi-plane support for mlx5" series [1]. When mlx5 multi-plane
is supported, a single logical mlx5 port aggregates multiple physical plane ports.
In this scenario, the requester can "spray" packets across the multiple physical
plane ports without guaranteeing packet order, either on the wire or on the receiver
(responder) side.

Quoted 7 lines

With this approach, no barriers or fences are required to ensure in-order packet
reception, which optimizes the data path for performance. This can result in better
BW, theoretically achieving line-rate performance equivalent to the sum of
the maximum BW of all physical plane ports, with only one QP.

Thanks a lot for your quick reply. Without ensuring in-order packet reception, this does optimize the data path for performance.

Quoted 7 lines

I agree with you.

But how does the receiver get the correct packets from the out-of-order
packets efficiently?

The method is implemented in Software or Hardware?

The packets have new field that is used by the HW to understand the correct message order (similar to PSN).

Once the packets arrive OOO to the receiver side, the data is scattered directly (hence the DDP - "Direct Data Placement" name) by the HW.

So the efficiency is achieved by the HW, as it also saves the required context and metadata so it can deliver the correct completion to the user (in-order) once we have some WQEs that can be considered an "in-order window" and be delivered to the user.

Quoted 4 lines

The SW/Applications may receive OOO WR_IDs though (because the first CQE
may have consumed Recv WQE of any index on the receiver side), and it's their responsibility to handle it from this point, if it's required.

Got it. It seems that all the functionalities are implemented in HW. The SW only receives OOO WR_IDs. Thanks a lot. Perhaps it is helpful to RDMA LAG devices. It should enhance the performance^_^

Quoted 5 lines

Quoted 1 line

BTW, do you have any performance data with this feature?

Not yet. We tested it functionality wise for now.

But we should be able to measure its performance soon :).

Thanks a lot. It is an interesting feature. If performance reports, please share them with us.

IMO, perhaps this feature can be used in random read/write devices, for example, hard disk?

Just my idea. Not sure if you have applied this feature with hard disk or not.

Best Regards,

Zhu Yanjun

Quoted 41 lines

Quoted 41 lines

Best Regards,
Zhu Yanjun

Quoted 34 lines

Quoted 34 lines

I am just interested in this feature and want to know more about this.

Thanks,

Zhu Yanjun

Quoted 22 lines

[1]
https://lore.kernel.org/lkml/[email protected]/

Quoted 19 lines

Thanks,
Zhu Yanjun

Quoted 15 lines

Thanks

Edward Srouji (2):
   net/mlx5: Introduce data placement ordering bits RDMA/mlx5: Support OOO RX WQE consumption

  drivers/infiniband/hw/mlx5/main.c    |  8 +++++ drivers/infiniband/hw/mlx5/mlx5_ib.h |  1 + drivers/infiniband/hw/mlx5/qp.c      | 51
+++++++++++++++++++++++++---
  include/linux/mlx5/mlx5_ifc.h        | 24 +++++++++---- include/uapi/rdma/mlx5-abi.h         |  5 +++

  5 files changed, 78 insertions(+), 11 deletions(-)

Signature · 2 lines

Best Regards,
Yanjun.Zhu

Signature · 2 lines

Best Regards,
Yanjun.Zhu

-----Original Message-----
From: Edward Srouji <[email protected]>
Sent: Thursday, September 5, 2024 2:23 PM
To: Zhu Yanjun <[email protected]>; Leon Romanovsky <[email protected]>; Jason Gunthorpe <[email protected]>
Cc: Leon Romanovsky <[email protected]>; [email protected]; [email protected]; [email protected]; Saeed Mahameed <[email protected]>; Tariq Toukan <[email protected]>; Yishai Hadas <[email protected]>
Subject: [EXTERNAL] Re: [PATCH rdma-next 0/2] Introduce mlx5 data direct placement (DDP)

Quoted 69 lines · Zhu Yanjun

External email: Use caution opening links or attachments

Quoted 52 lines · Edward Srouji

Quoted 18 lines · Zhu Yanjun

External email: Use caution opening links or attachments

Quoted 14 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

Hi,

This series from Edward introduces mlx5 data direct placement (DDP) feature.

This feature allows WRs on the receiver side of the QP to be consumed out of order, permitting the sender side to transmit messages without guaranteeing arrival order on the receiver side.

When enabled, the completion ordering of WRs remains in-order, regardless of the Receive WRs consumption order.

Quoted 8 lines

Quoted 3 lines

RDMA Read and RDMA Atomic operations on the responder side continue to be executed in-order, while the ordering of data placement for RDMA Write and Send operations is not guaranteed.

It is an interesting feature. If I got this feature correctly, this feature permits the user consumes the data out of order when RDMA Write and Send operations. But its completiong ordering is still in order.

Correct.

Quoted 7 lines

Any scenario that this feature can be applied and what benefits will be got from this feature?

I am just curious about this. Normally the users will consume the data in order. In what scenario, the user will consume the data out of order?

One of the main benefits of this feature is achieving higher bandwidth (BW) by allowing
responders to receive packets out of order (OOO).

For example, this can be utilized in devices that support multi-plane functionality,
as introduced in the "Multi-plane support for mlx5" series [1]. When mlx5 multi-plane
is supported, a single logical mlx5 port aggregates multiple physical plane ports.
In this scenario, the requester can "spray" packets across the multiple physical
plane ports without guaranteeing packet order, either on the wire or on the receiver
(responder) side.

Quoted 7 lines

With this approach, no barriers or fences are required to ensure in-order packet
reception, which optimizes the data path for performance. This can result in better
BW, theoretically achieving line-rate performance equivalent to the sum of
the maximum BW of all physical plane ports, with only one QP.

Thanks a lot for your quick reply. Without ensuring in-order packet reception, this does optimize the data path for performance.

I agree with you.

Quoted 4 lines

But how does the receiver get the correct packets from the out-of-order packets efficiently?

The method is implemented in Software or Hardware?

The packets have new field that is used by the HW to understand the correct message order (similar to PSN).

Once the packets arrive OOO to the receiver side, the data is scattered directly (hence the DDP - "Direct Data Placement" name) by the HW.

So the efficiency is achieved by the HW, as it also saves the required context and metadata so it can deliver the correct completion to the user (in-order) once we have some WQEs that can be considered an "in-order window" and be delivered to the user.

The SW/Applications may receive OOO WR_IDs though (because the first CQE may have consumed Recv WQE of any index on the receiver side), and it's their responsibility to handle it from this point, if it's required.

Quoted 9 lines

I am just interested in this feature and want to know more about this.

Thanks,

Zhu Yanjun

Quoted 2 lines

[1] INVALID URI REMOVED

3A__lore.kernel.org_lkml_cover.1718553901.git.leon- 40kernel.org_&d=DwIDaQ&c=BSDicqBQBDjDI9RkVyTcHQ&r=4ynb4Sj_4MUcZXbhvovE4tYSb qxyOwdSiLedP4yO55g&m=v7mstcYLoga4Ed_laSGpqjuQbnScgHCiflwmA4TzvXgi9x64qGYB4C ZGFrxQviQF&s=a-4dG1bvzL3dPsLsCSkubdHg_9eDKHIt-rEGQdaXvgU&e=

Quoted 23 lines

Quoted 10 lines

Quoted 10 lines

Thanks,
Zhu Yanjun

Quoted 7 lines

Thanks

Edward Srouji (2):
   net/mlx5: Introduce data placement ordering bits RDMA/mlx5: Support OOO RX WQE consumption

Quoted 9 lines

Quoted 9 lines

Quoted 8 lines

  drivers/infiniband/hw/mlx5/main.c    |  8 +++++ drivers/infiniband/hw/mlx5/mlx5_ib.h |  1 + drivers/infiniband/hw/mlx5/qp.c      | 51
+++++++++++++++++++++++++---
  include/linux/mlx5/mlx5_ifc.h        | 24 +++++++++---- include/uapi/rdma/mlx5-abi.h         |  5 +++

  5 files changed, 78 insertions(+), 11 deletions(-)

--
Best Regards,
Yanjun.Zhu

From: Leon Romanovsky <[email protected]>

Hi,

This series from Edward introduces mlx5 data direct placement (DDP) feature.

[...]

From: Leon Romanovsky <[email protected]>

Hi,

This series from Edward introduces mlx5 data direct placement (DDP) feature.

This feature allows WRs on the receiver side of the QP to be consumed out of order, permitting the sender side to transmit messages without guaranteeing arrival order on the receiver side.

When enabled, the completion ordering of WRs remains in-order, regardless of the Receive WRs consumption order.

RDMA Read and RDMA Atomic operations on the responder side continue to be executed in-order, while the ordering of data placement for RDMA Write and Send operations is not guaranteed.

Thanks

Edward Srouji (2):
net/mlx5: Introduce data placement ordering bits

RDMA/mlx5: Support OOO RX WQE consumption

 drivers/infiniband/hw/mlx5/main.c    |  8 +++++
 drivers/infiniband/hw/mlx5/mlx5_ib.h |  1 +
 drivers/infiniband/hw/mlx5/qp.c      | 51 +++++++++++++++++++++++++---
 include/linux/mlx5/mlx5_ifc.h        | 24 +++++++++----
 include/uapi/rdma/mlx5-abi.h         |  5 +++
 5 files changed, 78 insertions(+), 11 deletions(-)

Signature · 1 line

2.46.0

Jakub,

We applied this series to RDMA and first patch generates merge conflicts in include/linux/mlx5/mlx5_ifc.h between netdev and RDMA trees.

Can you please pull shared mlx5-next branch to avoid it?

Quoted 6 lines · On Mon, 4 Nov 2024 10:27:10 +0200 Leon Romanovsky wrote:

Jakub,

We applied this series to RDMA and first patch generates merge conflicts in include/linux/mlx5/mlx5_ifc.h between netdev and RDMA trees.

Can you please pull shared mlx5-next branch to avoid it?

Sorry I don't have the context, the thread looks 2 months old. If you'd like us to pull something please sense a pull request targeting net-next...