Skip to contentSkip to conversation

Add packet pacing support for IB verbs

PATCH In review Leon Romanovsky 1 participant 5 messages Aug 28, 2016 – Sep 7, 2016 1 message not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover Add packet pacing support for IB verbs

When sending from a 10G host to a 1G host, it is easy to overrun the receiver, leading to packet loss and traffic backing off. Similar problems occur when a 10G host sends data to a sub-10G virtual circuit, or a 40G host sending to a 10G host. Packet pacing could control packet injection rate and reduces network congestion to maximize throughput & minimize network latency.

Packet pacing is a rate limiting and shaping for a QP (SQ for RAW QP), set and change the rate is done by modifying QP. This series of patch made the following high level changes:
1. Report rate limit capabilities through user data. Reported capabilities
include: The maximum and minimum rate limit in kbps supported by packet pacing; Bitmap showing which QP types are supported by packet pacing operation.
2. Extend modify QP interface for growing attributes. Add rate limit support to the extended interface.
3. Enable mlx5-based hardware to be able to update the rate limit for RAW QP packet.

Available in the "topic/packet_pacing" topic branch of this git repo:
git://git.kernel.org/pub/scm/linux/kernel/git/leon/linux-rdma.git

Or for browsing:
https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=topic/packet_pacing

Thanks,
Bodong & Leon

Bodong Wang (3):
IB/mlx5: Report mlx5 packet pacing capabilities when querying device IB/core: Support rate limit for packet pacing
IB/uverbs: Extend modify_qp and support packet pacing

 drivers/infiniband/core/uverbs.h      |   1 +
 drivers/infiniband/core/uverbs_cmd.c  | 178 +++++++++++++++++++++-------------
 drivers/infiniband/core/uverbs_main.c |   1 +
 drivers/infiniband/core/verbs.c       |   2 +
 drivers/infiniband/hw/mlx5/main.c     |  12 +++
 drivers/infiniband/hw/mlx5/user.h     |  13 +++
 include/rdma/ib_verbs.h               |   2 +
 include/uapi/rdma/ib_user_verbs.h     |  12 +++
 8 files changed, 152 insertions(+), 69 deletions(-)

--
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 1472383631-3494-1-git-send-email-leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org

#2 lore raw

PATCH 1/3 IB/mlx5: Report mlx5 packet pacing capabilities when querying device

From: Bodong Wang <bodong-VPRAkNaXOzVWk0Htik3J/[email protected]>

Enable mlx5 based hardware to report packet pacing capabilities from kernel to user space. Packet pacing allows to limit the rate to any number between the maximum and minimum, based on user settings.

The capabilities are exposed to user space through query_device by uhw. The following capabilities are reported:

1. The maximum and minimum rate limit in kbps supported by packet pacing.
2. Bitmap showing which QP types are supported by packet pacing operation.

---
 drivers/infiniband/hw/mlx5/main.c | 12 ++++++++++++
 drivers/infiniband/hw/mlx5/user.h | 13 +++++++++++++
 2 files changed, 25 insertions(+)
drivers/infiniband/hw/mlx5/main.c +12 -0
index f02a975..26a4c85 100644
@@ -594,6 +594,18 @@ static int mlx5_ib_query_device(struct ib_device *ibdev,
 	if (!mlx5_core_is_pf(mdev))
 		props->device_cap_flags |= IB_DEVICE_VIRTUAL_FUNCTION;
 
+	if (field_avail(typeof(resp), packet_pacing_caps, uhw->outlen)	&&
+	    MLX5_CAP_QOS(mdev, packet_pacing)				&&
+	    MLX5_CAP_GEN(mdev, qos)) {
+		resp.packet_pacing_caps.qp_rate_limit_max =
+			MLX5_CAP_QOS(mdev, packet_pacing_max_rate);
+		resp.packet_pacing_caps.qp_rate_limit_min =
+			MLX5_CAP_QOS(mdev, packet_pacing_min_rate);
+		resp.packet_pacing_caps.supported_qpts |=
+			1 << IB_QPT_RAW_PACKET;
+		resp.response_length += sizeof(resp.packet_pacing_caps);
+	}
+
 	if (uhw->outlen) {
 		err = ib_copy_to_udata(uhw, &resp, resp.response_length);
 
drivers/infiniband/hw/mlx5/user.h +13 -0
index 188dac4..071a9cc 100644
@@ -121,10 +121,23 @@ struct mlx5_ib_tso_caps {
 	__u32 supported_qpts;
 };
 
+struct mlx5_packet_pacing_caps {
+	__u32 qp_rate_limit_min;
+	__u32 qp_rate_limit_max; /* In kpbs */
+
+	/* Corresponding bit will be set if qp type from
+	 * 'enum ib_qp_type' is supported, e.g.
+	 * supported_qpts |= 1 << IB_QPT_RAW_PACKET
+	 */
+	__u32 supported_qpts;
+	__u32 reserved;
+};
+
 struct mlx5_ib_query_device_resp {
 	__u32	comp_mask;
 	__u32	response_length;
 	struct	mlx5_ib_tso_caps tso_caps;
+	struct	mlx5_packet_pacing_caps packet_pacing_caps;
 };
 
 struct mlx5_ib_create_cq {
Signature · 6 lines
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#3 lore raw

PATCH 2/3 IB/core: Support rate limit for packet pacing

In reply to a message not in the archive

From: Bodong Wang <bodong-VPRAkNaXOzVWk0Htik3J/[email protected]>

Add new member rate_limit to ib_qp_attr, it shows the packet pacing rate in Kbps, 0 means unlimited.

IB_QP_RATE_LIMIT is added to ib_attr_mask, and it could be used by RAW QPs when changing QP state from RTR to RTS, RTS to RTS.

---
 drivers/infiniband/core/verbs.c | 2 ++
 include/rdma/ib_verbs.h         | 2 ++
 2 files changed, 4 insertions(+)
drivers/infiniband/core/verbs.c +2 -0
index f2b776e..1221215 100644
@@ -992,6 +992,7 @@ static const struct {
 						 IB_QP_QKEY),
 				 [IB_QPT_GSI] = (IB_QP_CUR_STATE		|
 						 IB_QP_QKEY),
+				 [IB_QPT_RAW_PACKET] = IB_QP_RATE_LIMIT,
 			 }
 		}
 	},
@@ -1025,6 +1026,7 @@ static const struct {
 						IB_QP_QKEY),
 				[IB_QPT_GSI] = (IB_QP_CUR_STATE			|
 						IB_QP_QKEY),
+				[IB_QPT_RAW_PACKET] = IB_QP_RATE_LIMIT,
 			}
 		},
 		[IB_QPS_SQD]   = {
include/rdma/ib_verbs.h +2 -0
index 8e90dd2..b27f233 100644
@@ -1088,6 +1088,7 @@ enum ib_qp_attr_mask {
 	IB_QP_RESERVED2			= (1<<22),
 	IB_QP_RESERVED3			= (1<<23),
 	IB_QP_RESERVED4			= (1<<24),
+	IB_QP_RATE_LIMIT		= (1<<25),
 };
 
 enum ib_qp_state {
@@ -1137,6 +1138,7 @@ struct ib_qp_attr {
 	u8			rnr_retry;
 	u8			alt_port_num;
 	u8			alt_timeout;
+	u32			rate_limit;
 };
 
 enum ib_wr_opcode {
Signature · 6 lines
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#4 lore raw

PATCH 3/3 IB/uverbs: Extend modify_qp and support packet pacing

In reply to a message not in the archive

From: Bodong Wang <bodong-VPRAkNaXOzVWk0Htik3J/[email protected]>

- Modify_qp is extended to support more attributes. Existing applications are not affected when calling modify_qp. New applications could call modify_qp_ex to use the extended fields.
- New member rate_limit is added to the modify_qp extended structure, users can modify it through modify_qp to control packet packing.

---
 drivers/infiniband/core/uverbs.h      |   1 +
 drivers/infiniband/core/uverbs_cmd.c  | 178 +++++++++++++++++++++-------------
 drivers/infiniband/core/uverbs_main.c |   1 +
 include/uapi/rdma/ib_user_verbs.h     |  12 +++
 4 files changed, 123 insertions(+), 69 deletions(-)
drivers/infiniband/core/uverbs.h +1 -0
index df26a74..455034a 100644
@@ -289,5 +289,6 @@ IB_UVERBS_DECLARE_EX_CMD(modify_wq);
 IB_UVERBS_DECLARE_EX_CMD(destroy_wq);
 IB_UVERBS_DECLARE_EX_CMD(create_rwq_ind_table);
 IB_UVERBS_DECLARE_EX_CMD(destroy_rwq_ind_table);
+IB_UVERBS_DECLARE_EX_CMD(modify_qp);
 
 #endif /* UVERBS_H */
drivers/infiniband/core/uverbs_cmd.c +109 -69

209 lines not on this page · raw message

drivers/infiniband/core/uverbs_main.c +1 -0
index 0012fa5..80839c3 100644
@@ -137,6 +137,7 @@ static int (*uverbs_ex_cmd_table[])(struct ib_uverbs_file *file,
 	[IB_USER_VERBS_EX_CMD_DESTROY_WQ]       = ib_uverbs_ex_destroy_wq,
 	[IB_USER_VERBS_EX_CMD_CREATE_RWQ_IND_TBL] = ib_uverbs_ex_create_rwq_ind_table,
 	[IB_USER_VERBS_EX_CMD_DESTROY_RWQ_IND_TBL] = ib_uverbs_ex_destroy_rwq_ind_table,
+	[IB_USER_VERBS_EX_CMD_MODIFY_QP]        = ib_uverbs_ex_modify_qp,
 };
 
 static void ib_uverbs_add_one(struct ib_device *device);
include/uapi/rdma/ib_user_verbs.h +12 -0
index 7f035f4b..4aa532d 100644
@@ -93,6 +93,7 @@ enum {
 	IB_USER_VERBS_EX_CMD_QUERY_DEVICE = IB_USER_VERBS_CMD_QUERY_DEVICE,
 	IB_USER_VERBS_EX_CMD_CREATE_CQ = IB_USER_VERBS_CMD_CREATE_CQ,
 	IB_USER_VERBS_EX_CMD_CREATE_QP = IB_USER_VERBS_CMD_CREATE_QP,
+	IB_USER_VERBS_EX_CMD_MODIFY_QP = IB_USER_VERBS_CMD_MODIFY_QP,
 	IB_USER_VERBS_EX_CMD_CREATE_FLOW = IB_USER_VERBS_CMD_THRESHOLD,
 	IB_USER_VERBS_EX_CMD_DESTROY_FLOW,
 	IB_USER_VERBS_EX_CMD_CREATE_WQ,
@@ -670,9 +671,20 @@ struct ib_uverbs_modify_qp {
 	__u64 driver_data[0];
 };
 
+struct ib_uverbs_ex_modify_qp {
+	struct ib_uverbs_modify_qp base;
+	__u32	rate_limit;
+	__u32	reserved;
+};
+
 struct ib_uverbs_modify_qp_resp {
 };
 
+struct ib_uverbs_ex_modify_qp_resp {
+	__u32  comp_mask;
+	__u32  response_length;
+};
+
 struct ib_uverbs_destroy_qp {
 	__u64 response;
 	__u32 qp_handle;
Signature · 6 lines
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#5 lore raw
In reply to a message not in the archive

On Sun, Aug 28, 2016 at 02:27:08PM +0300, Leon Romanovsky wrote:

<...>

Quoted 4 lines

Bodong Wang (3):
IB/mlx5: Report mlx5 packet pacing capabilities when querying device IB/core: Support rate limit for packet pacing
IB/uverbs: Extend modify_qp and support packet pacing

Hi Doug,

Please drop this series.
We will respin it later on.

Thanks