Enable flow steering on IPoIB UD QP
Subsystems (4)
v2 cover Enable flow steering on IPoIB UD QP
Hi,
This is v2 of patch set from Yishai,
Changelog:
v1->v2:
* Use 'source_qpn' as part of QP creation without mandating association of an overlay with underlay QP.
v0->v1:
* Improved commit message
----
Quoted 1 line
From Yishai:
Enable accelerating IPoIB traffic from user applications
This series comes to allow user space applications to accelerate send and receive traffic which is typically handled by IPoIB ULP. On the receive flow, the user space application will use flow steering rules to direct the ingress IPoIB UD QP traffic to its own user space UD QP. On the send flow, the user space application will send from its own UD QP with UD AH, while the packet on the wire will have the source QPN of the IPoIB UD QP, which is the network interface advertised L2 hardware address.
This will be achieved by extending QP creation of IB UD QP to get the source QP number of the matching IPoIB UD QP.
In order to create such a UD QP needs to provide the source QPN of the IPoIB network interface.
The QPN should be achieved by using some networking tool as of ifconfig, ip link show on the associated interface. For example, when IPoIB is used, the QPN is the value of 3 bytes from the hardware address starting after the first byte as defined by rfc4391 [1]
Current mlx5 driver implements this API for the IPoIB ULP.
Thanks
[1] https://tools.ietf.org/html/rfc4391#page-11
---
Available in the "topic/underlay_qp_v2" topic branch of this git repo:git://git.kernel.org/pub/scm/linux/kernel/git/leon/linux-rdma.git
Or for browsing:
https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=topic/underlay_qp_v2
Possible merge conflicts resolution is available in:
https://git.kernel.org/cgit/linux/kernel/git/leon/linux-rdma.git/log/?h=testing/queue-next
Yishai Hadas (6):
IB/core: Enable QP creation with a given source QP number IB/uverbs: Enable QP creation with a given source QP number IB/mlx5: Add support for QP with a given source QPN IB/mlx5: Add multicast flow steering support for underlay QP net/mlx5: Report enhanced capabilities for IPoIB IB/mlx5: Report RX checksum capabilities for IPoIB
drivers/infiniband/core/uverbs_cmd.c | 17 +++++-- drivers/infiniband/hw/mlx5/main.c | 47 +++++++++++++----- drivers/infiniband/hw/mlx5/mlx5_ib.h | 2 + drivers/infiniband/hw/mlx5/qp.c | 72 ++++++++++++++++++++++------ drivers/net/ethernet/mellanox/mlx5/core/fw.c | 6 +++ include/linux/mlx5/device.h | 6 ++- include/rdma/ib_verbs.h | 2 + include/uapi/rdma/ib_user_verbs.h | 2 +- 8 files changed, 122 insertions(+), 32 deletions(-)
--
2.12.2
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 20170608131511.21040-1-leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org
PATCH v2 1/6 IB/core: Enable QP creation with a given source QP number
From: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/[email protected]>
Enable QP creation with a given source QP number. The created QP will use the source QPN as its wire QP number.
This comes as a pre-patch for downstream patches in this series to allow user space applications to accelerate traffic which is typically handled by IPoIB ULP.
--- include/rdma/ib_verbs.h | 2 ++ 1 file changed, 2 insertions(+)
include/rdma/ib_verbs.h +2 -0
@@ -1059,6 +1059,7 @@ enum ib_qp_create_flags {IB_QP_CREATE_USE_GFP_NOIO =1 <<7 ,IB_QP_CREATE_SCATTER_FCS =1 <<8 ,IB_QP_CREATE_CVLAN_STRIPPING =1 <<9 , +IB_QP_CREATE_SOURCE_QPN =1 <<10 ,/* reserve bits 26-31 for low level drivers' internal use */ IB_QP_CREATE_RESERVED_START =1 <<26 , IB_QP_CREATE_RESERVED_END = 1 << 31,
@@ -1086,6 +1087,7 @@ struct ib_qp_init_attr {*/ u8 port_num;struct ib_rwq_ind_table *rwq_ind_tbl; +u32 source_qpn; };struct ib_qp_open_attr {
Signature · 6 lines
2.12.2 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH v2 2/6 IB/uverbs: Enable QP creation with a given source QP number
In reply to a message not in the archiveFrom: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/[email protected]>
Enable QP creation with a given source QP number, the created QP will use the source QPN as its wire QP number.
To create such a QP, root privileges (i.e. CAP_NET_RAW) are required from the user application.
This comes as a pre-patch for downstream patches in this series to allow user space applications to accelerate traffic which is typically handled by IPoIB ULP.
--- drivers/infiniband/core/uverbs_cmd.c | 17 ++++++++++++++--- include/uapi/rdma/ib_user_verbs.h | 2 +- 2 files changed, 15 insertions(+), 4 deletions(-)
drivers/infiniband/core/uverbs_cmd.c +14 -3
@@ -1385,8 +1385,9 @@ static int create_qp(struct ib_uverbs_file *file, attr.rwq_ind_tbl = ind_tbl; } - if ((cmd_sz >= offsetof(typeof(*cmd), reserved1) + -sizeof (cmd->reserved1)) && cmd->reserved1) { +if (cmd_sz >sizeof (*cmd) && + !ib_is_udata_cleared (ucore,sizeof (*cmd), + cmd_sz -sizeof (*cmd))) { ret = -EOPNOTSUPP ; goto err_put; }
@@ -1484,11 +1485,21 @@ static int create_qp(struct ib_uverbs_file *file,IB_QP_CREATE_MANAGED_SEND |IB_QP_CREATE_MANAGED_RECV |IB_QP_CREATE_SCATTER_FCS | - IB_QP_CREATE_CVLAN_STRIPPING)) { +IB_QP_CREATE_CVLAN_STRIPPING | + IB_QP_CREATE_SOURCE_QPN)) { ret = -EINVAL; goto err_put; } +if (attr.create_flags &IB_QP_CREATE_SOURCE_QPN ) { +if (!capable (CAP_NET_RAW )) { + ret = -EPERM ; + goto err_put; + } + + attr.source_qpn = cmd->source_qpn; + } + buf = (void *)cmd +sizeof (*cmd); if (cmd_sz > sizeof(*cmd)) if (!(buf[0] == 0 && !memcmp(buf, buf + 1,
include/uapi/rdma/ib_user_verbs.h +1 -1
@@ -578,7 +578,7 @@ struct ib_uverbs_ex_create_qp {__u32 comp_mask;__u32 create_flags;__u32 rwq_ind_tbl_handle; -__u32 reserved1; +__u32 source_qpn; };struct ib_uverbs_open_qp {
Signature · 6 lines
2.12.2 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH v2 3/6 IB/mlx5: Add support for QP with a given source QPN
In reply to a message not in the archiveFrom: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/[email protected]>
Allow user space applications to accelerate send and receive traffic which is typically handled by IPoIB ULP by creating a UD QP with a given source QPN of the IPoIB UD QP.
UD QP with a given source QPN should basically be similar to RAW QP from point of view of its created resources.
However,
- Its TIS should point to the source QPN.
- Modify can be done only on its state as the transport attributes are managed by its source QP.
This patch manages below:
- Creating/destroying/modifying UD QP with a given source QPN.
--- drivers/infiniband/hw/mlx5/mlx5_ib.h | 2 + drivers/infiniband/hw/mlx5/qp.c | 72 ++++++++++++++++++++++++++++-------- 2 files changed, 59 insertions(+), 15 deletions(-)
drivers/infiniband/hw/mlx5/mlx5_ib.h +2 -0
@@ -378,6 +378,7 @@ struct mlx5_ib_qp {struct list_head cq_recv_list;struct list_head cq_send_list;u32 rate_limit; +u32 underlay_qpn; };struct mlx5_ib_cq_buf {
@@ -399,6 +400,7 @@ enum mlx5_ib_qp_flags {MLX5_IB_QP_CAP_SCATTER_FCS =1 <<7 ,MLX5_IB_QP_RSS =1 <<8 ,MLX5_IB_QP_CVLAN_STRIPPING =1 <<9 , + MLX5_IB_QP_UNDERLAY = 1 << 10, }; struct mlx5_umr_wr {
drivers/infiniband/hw/mlx5/qp.c +57 -15
178 lines not on this page · raw message
Signature · 6 lines
2.12.2 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 20170608131511.21040-4-leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org
Re: [PATCH rdma-next V2 3/6] IB/mlx5: Add support for QP with a given source QPN
Quoted 6 lines · Leon Romanovsky
From: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/[email protected]>
Allow user space applications to accelerate send and receive traffic which is typically handled by IPoIB ULP by creating a UD QP with a given source QPN of the IPoIB UD QP.
Quoted 7 lines
UD QP with a given source QPN should basically be similar to RAW QP from point of view of its created resources.
However,
- Its TIS should point to the source QPN.
- Modify can be done only on its state as the transport attributes are managed by its source QP.
I thought the discussion conclusion was to get rid of this confusing linkage from the API?
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 20170608163020.GA16857-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org
Re: [PATCH rdma-next V2 3/6] IB/mlx5: Add support for QP with a given source QPN
Quoted 3 lines · Jason Gunthorpe
I thought the discussion conclusion was to get rid of this confusing linkage from the API?
We agreed that from API point of view as introduced in the IB patches there is no linkage, however, each vendor can implement it according to its hardware. --
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 0ee2641a-4308-6b35-cd0a-e1a5b8b84a6a-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org
Re: [PATCH rdma-next V2 3/6] IB/mlx5: Add support for QP with a given source QPN
Quoted 9 lines · Yishai Hadas
Quoted 3 lines · Jason Gunthorpe
I thought the discussion conclusion was to get rid of this confusing linkage from the API?
We agreed that from API point of view as introduced in the IB patches there is no linkage, however, each vendor can implement it according to its hardware.
That doesn't even make sense. An API is an API, if the API does not require linkage then the hardware can't either.
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 20170612160257.GA24829-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org
Re: [PATCH rdma-next V2 3/6] IB/mlx5: Add support for QP with a given source QPN
Quoted 6 lines · Jason Gunthorpe
Quoted 2 lines · Yishai Hadas
On 6/8/2017 7:30 PM, Jason Gunthorpe wrote:
That doesn't even make sense. An API is an API, if the API does not require linkage then the hardware can't either.
The API is well defined now and doesn't require the linkage. But our current HW implementation requires creation of the internal underlay object in order to support this.
Alex
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): CAFgAxU88cfMu7fe9J4Tu=8OFwnND6UfCKF=xi7ZaNHStFpQJog-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org
Re: [PATCH rdma-next V2 3/6] IB/mlx5: Add support for QP with a given source QPN
Quoted 6 lines · Alex Rosenbaum
Quoted 4 lines · Jason Gunthorpe
Quoted 2 lines · Yishai Hadas
On 6/8/2017 7:30 PM, Jason Gunthorpe wrote:
Quoted 6 lines
Quoted 2 lines
That doesn't even make sense. An API is an API, if the API does not require linkage then the hardware can't either.
The API is well defined now and doesn't require the linkage. But our current HW implementation requires creation of the internal underlay object in order to support this.
Well, so long as mlx5 takes the pkey/etc from the qp that has source_qpn set, and allows the source_qpn to be set to a QP that does not already exist, I have no objection..
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH v2 4/6 IB/mlx5: Add multicast flow steering support for underlay QP
In reply to a message not in the archiveFrom: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/[email protected]>
In order to add multicast flow steering support, there is need to block the attaching of mcg flow for underlay QP, recognize multicast IB_FLOW_SPEC_IPV4 based on its IP and enable creating/destroying flow for IB layer.
--- drivers/infiniband/hw/mlx5/main.c | 43 ++++++++++++++++++++++++++++----------- 1 file changed, 31 insertions(+), 12 deletions(-)
drivers/infiniband/hw/mlx5/main.c +31 -12
@@ -2044,21 +2044,32 @@ static int parse_flow_attr(struct mlx5_core_dev *mdev, u32 *match_c,*/ static bool flow_is_multicast_only (struct ib_flow_attr *ib_attr) { -struct ib_flow_spec_eth *eth_spec; +union ib_flow_spec *flow_spec;if (ib_attr->type !=IB_FLOW_ATTR_NORMAL || - ib_attr->size <sizeof (struct ib_flow_attr ) + -sizeof (struct ib_flow_spec_eth ) || ib_attr->num_of_specs <1 )return false; - eth_spec = (struct ib_flow_spec_eth *)(ib_attr +1 ); -if (eth_spec->type !=IB_FLOW_SPEC_ETH || - eth_spec->size !=sizeof (*eth_spec)) + flow_spec = (union ib_flow_spec *)(ib_attr +1 ); +if (flow_spec->type ==IB_FLOW_SPEC_IPV4 ) { +struct ib_flow_spec_ipv4 *ipv4_spec; + + ipv4_spec = (struct ib_flow_spec_ipv4 *)flow_spec; +if (ipv4_is_multicast (ipv4_spec->val.dst_ip)) +return true; +return false; + } + +if (flow_spec->type ==IB_FLOW_SPEC_ETH ) { +struct ib_flow_spec_eth *eth_spec; + + eth_spec = (struct ib_flow_spec_eth *)flow_spec; +return is_multicast_ether_addr (eth_spec->mask.dst_mac) && +is_multicast_ether_addr (eth_spec->val.dst_mac); + } -return is_multicast_ether_addr (eth_spec->mask.dst_mac) && -is_multicast_ether_addr (eth_spec->val.dst_mac); +return false; } static bool is_valid_ethertype(struct mlx5_core_dev *mdev,
@@ -2536,8 +2547,14 @@ static struct ib_flow *mlx5_ib_create_flow(struct ib_qp *qp,static int mlx5_ib_mcg_attach (struct ib_qp *ibqp,union ib_gid *gid,u16 lid) {struct mlx5_ib_dev *dev =to_mdev (ibqp->device); +struct mlx5_ib_qp *mqp =to_mqp (ibqp);int err; +if (mqp->flags &MLX5_IB_QP_UNDERLAY ) { +mlx5_ib_dbg (dev,"Attaching a multi cast group to underlay QP is not supported\n" ); +return -EOPNOTSUPP ; + } + err =mlx5_core_attach_mcg (dev->mdev, gid, ibqp->qp_num);if (err) mlx5_ib_warn(dev, "failed attaching QPN 0x%x, MGID %pI6\n",
@@ -3728,18 +3745,20 @@ static void *mlx5_ib_add(struct mlx5_core_dev *mdev) (1ull <<IB_USER_VERBS_CMD_CLOSE_XRCD ); } + dev->ib_dev.create_flow = mlx5_ib_create_flow; + dev->ib_dev.destroy_flow = mlx5_ib_destroy_flow; + dev->ib_dev.uverbs_ex_cmd_mask |= + (1ull <<IB_USER_VERBS_EX_CMD_CREATE_FLOW ) | + (1ull <<IB_USER_VERBS_EX_CMD_DESTROY_FLOW ); +if (mlx5_ib_port_link_layer (&dev->ib_dev,1 ) ==IB_LINK_LAYER_ETHERNET ) { - dev->ib_dev.create_flow = mlx5_ib_create_flow; - dev->ib_dev.destroy_flow = mlx5_ib_destroy_flow; dev->ib_dev.create_wq = mlx5_ib_create_wq; dev->ib_dev.modify_wq = mlx5_ib_modify_wq; dev->ib_dev.destroy_wq = mlx5_ib_destroy_wq; dev->ib_dev.create_rwq_ind_table = mlx5_ib_create_rwq_ind_table; dev->ib_dev.destroy_rwq_ind_table = mlx5_ib_destroy_rwq_ind_table; dev->ib_dev.uverbs_ex_cmd_mask |= - (1ull <<IB_USER_VERBS_EX_CMD_CREATE_FLOW ) | - (1ull <<IB_USER_VERBS_EX_CMD_DESTROY_FLOW ) | (1ull <<IB_USER_VERBS_EX_CMD_CREATE_WQ ) | (1ull <<IB_USER_VERBS_EX_CMD_MODIFY_WQ ) | (1ull << IB_USER_VERBS_EX_CMD_DESTROY_WQ) |
Signature · 6 lines
2.12.2 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH v2 5/6 net/mlx5: Report enhanced capabilities for IPoIB
In reply to a message not in the archiveFrom: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/[email protected]>
Report 'ipoib_enhanced_offloads' capabilities from
the core layer, it will be used in the next patch from this series.
--- drivers/net/ethernet/mellanox/mlx5/core/fw.c | 6 ++++++ include/linux/mlx5/device.h | 6 +++++- 2 files changed, 11 insertions(+), 1 deletion(-)
drivers/net/ethernet/mellanox/mlx5/core/fw.c +6 -0
@@ -119,6 +119,12 @@ int mlx5_query_hca_caps(struct mlx5_core_dev *dev)return err; } +if (MLX5_CAP_GEN (dev, ipoib_enhanced_offloads)) { + err =mlx5_core_get_caps (dev,MLX5_CAP_IPOIB_ENHANCED_OFFLOADS ); +if (err) +return err; + } +if (MLX5_CAP_GEN (dev, pg)) { err =mlx5_core_get_caps (dev,MLX5_CAP_ODP );if (err)
include/linux/mlx5/device.h +5 -1
@@ -966,7 +966,7 @@ enum mlx5_cap_type {MLX5_CAP_ATOMIC ,MLX5_CAP_ROCE ,MLX5_CAP_IPOIB_OFFLOADS , -MLX5_CAP_EOIB_OFFLOADS , +MLX5_CAP_IPOIB_ENHANCED_OFFLOADS ,MLX5_CAP_FLOW_TABLE ,MLX5_CAP_ESWITCH_FLOW_TABLE , MLX5_CAP_ESWITCH,
@@ -1008,6 +1008,10 @@ enum mlx5_mcam_feature_groups {MLX5_GET (per_protocol_networking_offload_caps,\ mdev->caps.hca_max[MLX5_CAP_ETHERNET_OFFLOADS], cap) +#define MLX5_CAP_IPOIB_ENHANCED (mdev, cap) \ + MLX5_GET(per_protocol_networking_offload_caps,\ + mdev->caps.hca_cur[MLX5_CAP_IPOIB_ENHANCED_OFFLOADS], cap) +#define MLX5_CAP_ROCE (mdev, cap) \ MLX5_GET(roce_cap, mdev->caps.hca_cur[MLX5_CAP_ROCE], cap)
Signature · 6 lines
2.12.2 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH v2 6/6 IB/mlx5: Report RX checksum capabilities for IPoIB
In reply to a message not in the archiveFrom: Yishai Hadas <yishaih-VPRAkNaXOzVWk0Htik3J/[email protected]>
Report RX checksum capabilities when 'ipoib_enhanced_offloads' capabilities are set and support checksum.
--- drivers/infiniband/hw/mlx5/main.c | 4 ++++ 1 file changed, 4 insertions(+)
drivers/infiniband/hw/mlx5/main.c +4 -0
@@ -693,6 +693,10 @@ static int mlx5_ib_query_device(struct ib_device *ibdev, props->device_cap_flags |=IB_DEVICE_UD_TSO ; } +if (MLX5_CAP_GEN (mdev, ipoib_enhanced_offloads) && +MLX5_CAP_IPOIB_ENHANCED (mdev, csum_cap)) + props->device_cap_flags |=IB_DEVICE_UD_IP_CSUM ; +if (MLX5_CAP_GEN (dev->mdev, eth_net_offloads) &&MLX5_CAP_ETH (dev->mdev, scatter_fcs)) {/* Legacy bit to support old userspace libraries */
Signature · 6 lines
2.12.2 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 44 lines · Leon Romanovsky
Hi,
This is v2 of patch set from Yishai,
Changelog:
v1->v2:
* Use 'source_qpn' as part of QP creation without mandating
association
of an overlay with underlay QP.
v0->v1:
* Improved commit message----
From Yishai:Enable accelerating IPoIB traffic from user applications
This series comes to allow user space applications to accelerate send and receive traffic which is typically handled by IPoIB ULP. On the receive flow, the user space application will use flow steering rules to
direct the ingress IPoIB UD QP traffic to its own user space UD QP. On
the send flow, the user space application will send from its own UD QP
with UD AH, while the packet on the wire will have the source QPN of the
IPoIB UD QP, which is the network interface advertised L2 hardware address.This will be achieved by extending QP creation of IB UD QP to get the source QP number of the matching IPoIB UD QP.
In order to create such a UD QP needs to provide the source QPN of the
IPoIB network interface.The QPN should be achieved by using some networking tool as of ifconfig,
ip link show on the associated interface. For example, when IPoIB is used, the QPN is the value of 3 bytes from the hardware address starting
after the first byte as defined by rfc4391 [1]
Quoted 4 lines
Current mlx5 driver implements this API for the IPoIB ULP.
Thanks
Thanks, series applied.
Signature · 8 lines
Doug Ledford <[email protected]> GPG KeyID: B826A3330E572FDD Key fingerprint = AE6B 1BDA 122B 23B4 265B 1274 B826 A333 0E57 2FDD -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html