Add packet pacing support for IB verbs

From: Bodong Wang <bodong-VPRAkNaXOzVWk0Htik3J/[email protected]>

Add new member rate_limit to ib_qp_attr, it shows the packet pacing rate

in Kbps, 0 means unlimited.

IB_QP_RATE_LIMIT is added to ib_attr_mask, and it could be used by RAW

QPs when changing QP state from RTR to RTS, RTS to RTS.

Signed-off-by Bodong Wang <bodong-VPRAkNaXOzVWk0Htik3J/[email protected]>

Reviewed-by Matan Barak <matanb-VPRAkNaXOzVWk0Htik3J/[email protected]>

Signed-off-by Leon Romanovsky <[email protected]>

---
 drivers/infiniband/core/verbs.c | 2 ++
 include/rdma/ib_verbs.h         | 2 ++
 2 files changed, 4 insertions(+)

drivers/infiniband/core/verbs.c +2 -0

index 8368764..3e688b3 100644
@@ -1014,6 +1014,7 @@ static const struct {
 						 IB_QP_QKEY),
 				 [IB_QPT_GSI] = (IB_QP_CUR_STATE		|
 						 IB_QP_QKEY),
+				 [IB_QPT_RAW_PACKET] = IB_QP_RATE_LIMIT,
 			 }
 		}
 	},

@@ -1047,6 +1048,7 @@ static const struct {
 						IB_QP_QKEY),
 				[IB_QPT_GSI] = (IB_QP_CUR_STATE			|
 						IB_QP_QKEY),
+				[IB_QPT_RAW_PACKET] = IB_QP_RATE_LIMIT,
 			}
 		},
 		[IB_QPS_SQD]   = {

include/rdma/ib_verbs.h +2 -0

index 5ad43a4..a065361 100644
@@ -1102,6 +1102,7 @@ enum ib_qp_attr_mask {
 	IB_QP_RESERVED2			= (1<<22),
 	IB_QP_RESERVED3			= (1<<23),
 	IB_QP_RESERVED4			= (1<<24),
+	IB_QP_RATE_LIMIT		= (1<<25),
 };
 
 enum ib_qp_state {

@@ -1151,6 +1152,7 @@ struct ib_qp_attr {
 	u8			rnr_retry;
 	u8			alt_port_num;
 	u8			alt_timeout;
+	u32			rate_limit;
 };
 
 enum ib_wr_opcode {

Signature · 6 lines

2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Two (extremely) minor suggestions inline.

Yuval

Quoted 3 lines · Leon Romanovsky

From: Bodong Wang <bodong-VPRAkNaXOzVWk0Htik3J/[email protected]>

Add new member rate_limit to ib_qp_attr, it shows the packet pacing rate

Suggesting to replace with:
Add new member rate_limit to ib_qp_attr which holds the packet pacing rate

Quoted 3 lines

in Kbps, 0 means unlimited.

IB_QP_RATE_LIMIT is added to ib_attr_mask, and it could be used by RAW

Suggesting to replace with:
IB_QP_RATE_LIMIT is added to ib_attr_mask and could be used by RAW

Quoted 5 lines

QPs when changing QP state from RTR to RTS, RTS to RTS.

Signed-off-by Bodong Wang <bodong-VPRAkNaXOzVWk0Htik3J/[email protected]>

Reviewed-by Matan Barak <matanb-VPRAkNaXOzVWk0Htik3J/[email protected]>

Signed-off-by Leon Romanovsky <[email protected]>

include/rdma/ib_verbs.h struct ib_qp_attr { · line 1156

Quoted 7 lines

  enum ib_qp_state {

include/rdma/ib_verbs.h +1 -0

@@ -1151,6 +1152,7 @@ struct ib_qp_attr {
  	u8			rnr_retry;
  	u8			alt_port_num;
  	u8			alt_timeout;
+	u32			rate_limit;
  };

We already have ib_qp_attr::ib_ah_attr::static_rate, and that accounts for both the primary and alternate paths. We should not add a conflicting rate_limit field. Either use static_rate as defined by the spec, or replace/update it, with corresponding changes to how it is used in conjunction with SM data.

When sending from a 10G host to a 1G host, it is easy to overrun the receiver,
leading to packet loss and traffic backing off. Similar problems occur when
a 10G host sends data to a sub-10G virtual circuit, or a 40G host sending
to a 10G host. Packet pacing could control packet injection rate and reduces
network congestion to maximize throughput & minimize network latency.

Quoted 9 lines

When sending from a 10G host to a 1G host, it is easy to overrun the receiver,
leading to packet loss and traffic backing off. Similar problems occur when
a 10G host sends data to a sub-10G virtual circuit, or a 40G host sending
to a 10G host. Packet pacing could control packet injection rate and reduces
network congestion to maximize throughput & minimize network latency.

Why isn't the path record data and existing mechanisms sufficient to handle this?

Quoted 1 line

Quoted 1 line

When sending from a 10G host to a 1G host, it is easy to overrun

the

Quoted 2 lines

Quoted 2 lines

receiver,
leading to packet loss and traffic backing off. Similar problems

occur

Quoted 4 lines

Quoted 4 lines

when
a 10G host sends data to a sub-10G virtual circuit, or a 40G host sending
to a 10G host. Packet pacing could control packet injection rate

and

Quoted 2 lines

Quoted 2 lines

reduces
network congestion to maximize throughput & minimize network

latency.

Quoted 1 line

Quoted 1 line

Why isn't the path record data and existing mechanisms sufficient to

handle this?

Quoted 1 line

Packet pacing allows different combinations of traffic shaping: per- CPU,
per-flow and their combinations with better and steady QoS requirements without involving subnet management.

Quoted 19 lines · Sean

Quoted 1 line

Quoted 1 line

When sending from a 10G host to a 1G host, it is easy to overrun

the

Quoted 2 lines

Quoted 2 lines

receiver,
leading to packet loss and traffic backing off. Similar problems

occur

Quoted 4 lines

Quoted 4 lines

when
a 10G host sends data to a sub-10G virtual circuit, or a 40G host sending
to a 10G host. Packet pacing could control packet injection rate

and

Quoted 2 lines

Quoted 2 lines

reduces
network congestion to maximize throughput & minimize network

latency.

Quoted 2 lines

Why isn't the path record data and existing mechanisms sufficient to

handle this?

Quoted 1 line

Quoted 4 lines

Packet pacing allows different combinations of traffic shaping: per- CPU,
per-flow and their combinations with better and steady QoS requirements without involving subnet management.

The patch adds this as a QP attribute, and we already have a rate for that. I still don't see why the standard mechanisms are insufficient or couldn't be adapted.

Quoted 20 lines · Sean

Quoted 20 lines · Sean

Quoted 1 line

Quoted 1 line

When sending from a 10G host to a 1G host, it is easy to overrun

the

Quoted 2 lines

Quoted 2 lines

receiver,
leading to packet loss and traffic backing off. Similar problems

occur

Quoted 4 lines

Quoted 4 lines

when
a 10G host sends data to a sub-10G virtual circuit, or a 40G host sending
to a 10G host. Packet pacing could control packet injection rate

and

Quoted 2 lines

Quoted 2 lines

reduces
network congestion to maximize throughput & minimize network

latency.

Quoted 1 line

Why isn't the path record data and existing mechanisms sufficient to

handle this?
Packet pacing allows different combinations of traffic shaping: per- CPU,
per-flow and their combinations with better and steady QoS requirements without involving subnet management.

Quoted 1 line

The patch adds this as a QP attribute, and we already have a rate for that. I still don't see why the standard mechanisms are insufficient or couldn't be adapted.

I'll let to Bodong to elaborate on it more, but as far as I see, the AH attribute is relevant to UD QP only, while the packet pacing is intended for all types of QPs.

Thanks

I'll let to Bodong to elaborate on it more, but as far as I see, the AH attribute is relevant to UD QP only, while the packet pacing is intended for all types of QPs.

When sending from a 10G host to a 1G host, it is easy to overrun the receiver, leading to packet loss and traffic backing off. Similar problems occur when a 10G host sends data to a sub-10G virtual circuit, or a 40G host sending to a 10G host. Packet pacing could control packet injection rate and reduces network congestion to maximize throughput & minimize network latency.

Packet pacing is a rate limiting and shaping for a QP (SQ for RAW QP), set and change the rate is done by modifying QP. This series of patch made the following high level changes:
1. Report rate limit capabilities through user data. Reported capabilities
include: The maximum and minimum rate limit in kbps supported by packet pacing; Bitmap showing which QP types are supported by packet pacing operation.
2. Extend modify QP interface for growing attributes. Add rate limit support to the extended interface.
3. Enable mlx5-based hardware to be able to update the rate limit for RAW QP packet.

Available in the "topic/packet_pacing" topic branch of this git repo:
git://git.kernel.org/pub/scm/linux/kernel/git/leon/linux-rdma.git

Or for browsing:
https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=topic/packet_pacing

Thanks,
Bodong & Leon

Bodong Wang (4):
IB/mlx5: Report mlx5 packet pacing capabilities when querying device IB/core: Support rate limit for packet pacing
IB/uverbs: Extend modify_qp and support packet pacing IB/mlx5: Update the rate limit according to user setting for RAW QP

 drivers/infiniband/core/uverbs.h      |   1 +
 drivers/infiniband/core/uverbs_cmd.c  | 178 +++++++++++++++++++++-------------
 drivers/infiniband/core/uverbs_main.c |   1 +
 drivers/infiniband/core/verbs.c       |   2 +
 drivers/infiniband/hw/mlx5/main.c     |  16 ++-
 drivers/infiniband/hw/mlx5/mlx5_ib.h  |   1 +
 drivers/infiniband/hw/mlx5/qp.c       |  71 ++++++++++++--
 include/rdma/ib_verbs.h               |   2 +
 include/uapi/rdma/ib_user_verbs.h     |  12 +++
 include/uapi/rdma/mlx5-abi.h          |  13 +++
 10 files changed, 219 insertions(+), 78 deletions(-)

--
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html