Add support for multicast in RoCEv2
cover Add support for multicast in RoCEv2
This series from Noa enables multicast traffic over RoCE v2.
To allow this, several changes were required.
First of all, there is a need to validate MLID when link layer is Infiniband. This check is mandatory according to Infiniband spec but the MLID should be ignored for RoCE according to RoCEv2 Annex (Annex 17, A17.5.6).
Second, there is need to cange the mapping of IPv4 multicast addresses to MGIDs to be according to RoCEv2 Annex (A17.4.1.1 L3 Addresses): GID = ::ffff:<IPv4 Address>
And third, multicast addresses need to be mapped to multicast MAC instead of searching for it in the GID table.
Thanks
Noa Osherovich (3):
IB/core: Fix the validations of a multicast LID in attach or detach
operations
IB/core: Set RoCEv2 MGID according to spec
IB/core: Add support for RoCEv2 multicast
drivers/infiniband/core/cma.c | 13 +++++---- drivers/infiniband/core/verbs.c | 62 +++++++++++++++++++++++++++++++++++++---- include/rdma/ib_addr.h | 8 +++++- 3 files changed, 70 insertions(+), 13 deletions(-)
--
2.12.2
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 20170612081404.17553-1-leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org
PATCH 1/3 IB/core: Fix the validations of a multicast LID in attach or detach operations
From: Noa Osherovich <noaos-VPRAkNaXOzVWk0Htik3J/[email protected]>
RoCE Annex (A16.9.10/11) declares that during attach (detach) QP to a multicast group, if the QP is associated with a RoCE port, the multicast group MLID is unused and is ignored.
During attach or detach multicast, when the QP is associated with a port, it is enough to check the port's link layer and validate the LID only if it is Infiniband. Otherwise, avoid validating the multicast LID.
--- drivers/infiniband/core/verbs.c | 44 +++++++++++++++++++++++++++++++++++++---- 1 file changed, 40 insertions(+), 4 deletions(-)
drivers/infiniband/core/verbs.c +40 -4
@@ -1519,6 +1519,44 @@ EXPORT_SYMBOL(ib_dealloc_fmr);/* Multicast groups */ +static bool is_valid_mcast_lid (struct ib_qp *qp,u16 lid) +{ +struct ib_qp_init_attr init_attr = {}; +struct ib_qp_attr attr = {}; +int num_eth_ports =0 ; +int port; + +/* If QP state >= init, it is assigned to a port and we can check this +* port only. +*/ +if (!ib_query_qp (qp, &attr,IB_QP_STATE |IB_QP_PORT , &init_attr)) { +if (attr.qp_state >=IB_QPS_INIT ) { +if (qp->device->get_link_layer (qp->device, attr.port_num) != +IB_LINK_LAYER_INFINIBAND ) +return true; + goto lid_check; + } + } + +/* Can't get a quick answer, iterate over all ports */ +for (port =0 ; port < qp->device->phys_port_cnt; port++) +if (qp->device->get_link_layer (qp->device, port) != +IB_LINK_LAYER_INFINIBAND ) + num_eth_ports++; + +/* If we have at lease one Ethernet port, RoCE annex declares that +* multicast LID should be ignored. We can't tell at this step if the +* QP belongs to an IB or Ethernet port. +*/ +if (num_eth_ports) +return true; + +/* If all the ports are IB, we can check according to IB spec. */ +lid_check: +return !(lid <be16_to_cpu (IB_MULTICAST_LID_BASE ) || + lid ==be16_to_cpu (IB_LID_PERMISSIVE )); +} +int ib_attach_mcast (struct ib_qp *qp,union ib_gid *gid,u16 lid) {int ret;
@@ -1526,8 +1564,7 @@ int ib_attach_mcast(struct ib_qp *qp, union ib_gid *gid, u16 lid)if (!qp->device->attach_mcast)return -ENOSYS ;if (gid->raw[0 ] !=0xff || qp->qp_type !=IB_QPT_UD || - lid <be16_to_cpu (IB_MULTICAST_LID_BASE ) || - lid ==be16_to_cpu (IB_LID_PERMISSIVE )) + !is_valid_mcast_lid (qp, lid))return -EINVAL ; ret = qp->device->attach_mcast (qp, gid, lid);
@@ -1544,8 +1581,7 @@ int ib_detach_mcast(struct ib_qp *qp, union ib_gid *gid, u16 lid)if (!qp->device->detach_mcast)return -ENOSYS ;if (gid->raw[0 ] !=0xff || qp->qp_type !=IB_QPT_UD || - lid <be16_to_cpu (IB_MULTICAST_LID_BASE ) || - lid ==be16_to_cpu (IB_LID_PERMISSIVE )) + !is_valid_mcast_lid (qp, lid))return -EINVAL ; ret = qp->device->detach_mcast (qp, gid, lid);
--
2.12.2
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 20170612081404.17553-2-leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org
RE: [PATCH rdma-next 1/3] IB/core: Fix the validations of a multicast LID in attach or detach operations
Quoted 32 lines
drivers/infiniband/core/verbs.c +8
-0+static bool is_valid_mcast_lid (struct ib_qp *qp,u16 lid) +{ +struct ib_qp_init_attr init_attr = {}; +struct ib_qp_attr attr = {}; +int num_eth_ports =0 ; +int port; + +/* If QP state >= init, it is assigned to a port and we can check this
drivers/infiniband/core/verbs.c +3
-0+* port only. +*/ + if (!ib_query_qp(qp, &attr, IB_QP_STATE | IB_QP_PORT,
&init_attr)) {(quoted diff) +2
-0+ if (attr.qp_state >= IB_QPS_INIT) { + if (qp->device->get_link_layer(qp->device,attr.port_num) !=
drivers/infiniband/core/verbs.c +13
-0+ IB_LINK_LAYER_INFINIBAND) +return true; + goto lid_check; + } + } + +/* Can't get a quick answer, iterate over all ports */ +for (port =0 ; port < qp->device->phys_port_cnt; port++) +if (qp->device->get_link_layer (qp->device, port) != +IB_LINK_LAYER_INFINIBAND ) + num_eth_ports++; + +/* If we have at lease one Ethernet port, RoCE annex declares that
(quoted diff) +1
-0+ * multicast LID should be ignored. We can't tell at this stepif the
drivers/infiniband/core/verbs.c EXPORT_SYMBOL(ib_dealloc_fmr); · line 1552
Quoted 4 lines
drivers/infiniband/core/verbs.c +4
-0+* QP belongs to an IB or Ethernet port. +*/ +if (num_eth_ports) +return true;
Reporting that a lid *might* be valid is a really weak check. Can we require that RoCE QPs be in the INIT state prior to joining, or use the mgid, or something to make the validation check work? Do we even know if the qp is a roce qp?
Quoted 32 lines
(quoted diff) +2
-0+ + /* If all the ports are IB, we can check according to IB spec.*/
drivers/infiniband/core/verbs.c +6
-2+lid_check: +return !(lid <be16_to_cpu (IB_MULTICAST_LID_BASE ) || + lid ==be16_to_cpu (IB_LID_PERMISSIVE )); +} +int ib_attach_mcast (struct ib_qp *qp,union ib_gid *gid,u16 lid) {int ret;@@ -1526,8 +1564,7 @@ int ib_attach_mcast(struct ib_qp *qp, union ib_gid *gid, u16 lid)if (!qp->device->attach_mcast)return -ENOSYS ;if (gid->raw[0 ] !=0xff || qp->qp_type !=IB_QPT_UD || - lid <be16_to_cpu (IB_MULTICAST_LID_BASE ) || - lid ==be16_to_cpu (IB_LID_PERMISSIVE )) + !is_valid_mcast_lid (qp, lid))return -EINVAL ;drivers/infiniband/core/verbs.c +1
-2ret = qp->device->attach_mcast (qp, gid, lid);@@ -1544,8 +1581,7 @@ int ib_detach_mcast(struct ib_qp *qp, union ib_gid *gid, u16 lid)if (!qp->device->detach_mcast)return -ENOSYS ;if (gid->raw[0 ] !=0xff || qp->qp_type !=IB_QPT_UD || - lid <be16_to_cpu (IB_MULTICAST_LID_BASE ) || - lid ==be16_to_cpu (IB_LID_PERMISSIVE )) + !is_valid_mcast_lid (qp, lid))return -EINVAL ;
Quoted 2 lines
ret =
qp->device->detach_mcast(qp, gid, lid);
--
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 1828884A29C6694DAF28B7E6B8A82373AB142896-P5GAC/[email protected]
Re: [PATCH rdma-next 1/3] IB/core: Fix the validations of a multicast LID in attach or detach operations
Quoted 30 lines · Sean
Quoted 30 lines
drivers/infiniband/core/verbs.c +8
-0+static bool is_valid_mcast_lid (struct ib_qp *qp,u16 lid) +{ +struct ib_qp_init_attr init_attr = {}; +struct ib_qp_attr attr = {}; +int num_eth_ports =0 ; +int port; + +/* If QP state >= init, it is assigned to a port and we can check this
drivers/infiniband/core/verbs.c +3
-0+* port only. +*/ + if (!ib_query_qp(qp, &attr, IB_QP_STATE | IB_QP_PORT,
&init_attr)) {(quoted diff) +2
-0+ if (attr.qp_state >= IB_QPS_INIT) { + if (qp->device->get_link_layer(qp->device,attr.port_num) !=
drivers/infiniband/core/verbs.c +13
-0+ IB_LINK_LAYER_INFINIBAND) +return true; + goto lid_check; + } + } + +/* Can't get a quick answer, iterate over all ports */ +for (port =0 ; port < qp->device->phys_port_cnt; port++) +if (qp->device->get_link_layer (qp->device, port) != +IB_LINK_LAYER_INFINIBAND ) + num_eth_ports++; + +/* If we have at lease one Ethernet port, RoCE annex declares that
Quoted 8 lines
drivers/infiniband/core/verbs.c EXPORT_SYMBOL(ib_dealloc_fmr); · line 1552
Quoted 6 lines
(quoted diff) +1
-0+ * multicast LID should be ignored. We can't tell at this stepif the
drivers/infiniband/core/verbs.c +4
-0+* QP belongs to an IB or Ethernet port. +*/ +if (num_eth_ports) +return true;Reporting that a lid *might* be valid is a really weak check. Can we require that RoCE QPs be in the INIT state prior to joining, or use the mgid, or something to make the validation check work? Do we even know if the qp is a roce qp?
It would be possible if ah existed (ib_qp_attr->rdma_ah_attr->rdma_ah_attr_type), but it isn't filled for IB_QPT_UD in mlx5.
Quoted 36 lines
Quoted 36 lines
(quoted diff) +2
-0+ + /* If all the ports are IB, we can check according to IB spec.*/
drivers/infiniband/core/verbs.c +6
-2+lid_check: +return !(lid <be16_to_cpu (IB_MULTICAST_LID_BASE ) || + lid ==be16_to_cpu (IB_LID_PERMISSIVE )); +} +int ib_attach_mcast (struct ib_qp *qp,union ib_gid *gid,u16 lid) {int ret;@@ -1526,8 +1564,7 @@ int ib_attach_mcast(struct ib_qp *qp, union ib_gid *gid, u16 lid)if (!qp->device->attach_mcast)return -ENOSYS ;if (gid->raw[0 ] !=0xff || qp->qp_type !=IB_QPT_UD || - lid <be16_to_cpu (IB_MULTICAST_LID_BASE ) || - lid ==be16_to_cpu (IB_LID_PERMISSIVE )) + !is_valid_mcast_lid (qp, lid))return -EINVAL ;drivers/infiniband/core/verbs.c +1
-2ret = qp->device->attach_mcast (qp, gid, lid);@@ -1544,8 +1581,7 @@ int ib_detach_mcast(struct ib_qp *qp, union ib_gid *gid, u16 lid)if (!qp->device->detach_mcast)return -ENOSYS ;if (gid->raw[0 ] !=0xff || qp->qp_type !=IB_QPT_UD || - lid <be16_to_cpu (IB_MULTICAST_LID_BASE ) || - lid ==be16_to_cpu (IB_LID_PERMISSIVE )) + !is_valid_mcast_lid (qp, lid))return -EINVAL ;ret =
qp->device->detach_mcast(qp, gid, lid);
--
Re: [PATCH rdma-next 1/3] IB/core: Fix the validations of a multicast LID in attach or detach operations
In reply to a message not in the archiveQuoted 30 lines · Sean
Quoted 30 lines
drivers/infiniband/core/verbs.c +8
-0+static bool is_valid_mcast_lid (struct ib_qp *qp,u16 lid) +{ +struct ib_qp_init_attr init_attr = {}; +struct ib_qp_attr attr = {}; +int num_eth_ports =0 ; +int port; + +/* If QP state >= init, it is assigned to a port and we can check this
drivers/infiniband/core/verbs.c +3
-0+* port only. +*/ + if (!ib_query_qp(qp, &attr, IB_QP_STATE | IB_QP_PORT,
&init_attr)) {(quoted diff) +2
-0+ if (attr.qp_state >= IB_QPS_INIT) { + if (qp->device->get_link_layer(qp->device,attr.port_num) !=
drivers/infiniband/core/verbs.c +13
-0+ IB_LINK_LAYER_INFINIBAND) +return true; + goto lid_check; + } + } + +/* Can't get a quick answer, iterate over all ports */ +for (port =0 ; port < qp->device->phys_port_cnt; port++) +if (qp->device->get_link_layer (qp->device, port) != +IB_LINK_LAYER_INFINIBAND ) + num_eth_ports++; + +/* If we have at lease one Ethernet port, RoCE annex declares that
Quoted 10 lines
drivers/infiniband/core/verbs.c EXPORT_SYMBOL(ib_dealloc_fmr); · line 1552
Quoted 6 lines
+ * multicast LID should be ignored. We can't tell at this step
if thedrivers/infiniband/core/verbs.c +4
-0+* QP belongs to an IB or Ethernet port. +*/ +if (num_eth_ports) +return true;Reporting that a lid *might* be valid is a really weak check. Can we require that RoCE QPs be in the INIT state prior to joining, or use the mgid, or something to make the validation check work? Do we even know if the qp is a roce qp?
This is not a join flow but attach flow. AFAIK, attaching a QP to a mcast group can be when QP is in RESET state. This patch does the best it can to give a valid answer but when QP is in RESET state and at least one of the ports is Eth port then we can't invalidate a MLID. --
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH 2/3 IB/core: Set RoCEv2 MGID according to spec
In reply to a message not in the archiveFrom: Noa Osherovich <noaos-VPRAkNaXOzVWk0Htik3J/[email protected]>
RoCEv2 Annex states that for RoCEv2 over IPv4, the corresponding IPv4 address is encoded into the GID according to the following rule:
GID= :ffff:<IPv4 address>
Remove the 0xff0e prefix for RoCEv2 packets with IPv4 and leave it zeroed and change rdma_is_multicast_addr() to consider the new logic.
--- drivers/infiniband/core/cma.c | 13 +++++++------ drivers/infiniband/core/verbs.c | 10 ++++++---- include/rdma/ib_addr.h | 8 +++++++- 3 files changed, 20 insertions(+), 11 deletions(-)
drivers/infiniband/core/cma.c +7 -6
@@ -4022,7 +4022,8 @@ static void iboe_mcast_work_handler(struct work_struct *work)kfree (mw); } -static void cma_iboe_set_mgid (struct sockaddr *addr,union ib_gid *mgid) +static void cma_iboe_set_mgid (struct sockaddr *addr,union ib_gid *mgid, +enum ib_gid_type gid_type) {struct sockaddr_in *sin = (struct sockaddr_in *)addr;struct sockaddr_in6 *sin6 = (struct sockaddr_in6 *)addr;
@@ -4032,8 +4033,8 @@ static void cma_iboe_set_mgid(struct sockaddr *addr, union ib_gid *mgid) }else if (addr->sa_family ==AF_INET6 ) {memcpy (mgid, &sin6->sin6_addr,sizeof *mgid); }else { - mgid->raw[0 ] =0xff ; - mgid->raw[1 ] =0x0e ; + mgid->raw[0 ] = (gid_type ==IB_GID_TYPE_IB ) ?0xff :0 ; + mgid->raw[1 ] = (gid_type ==IB_GID_TYPE_IB ) ?0x0e :0 ; mgid->raw[2 ] =0 ; mgid->raw[3 ] =0 ; mgid->raw[4 ] =0 ;
@@ -4074,7 +4075,9 @@ static int cma_iboe_join_multicast(struct rdma_id_private *id_priv, goto out1; } -cma_iboe_set_mgid (addr, &mc->multicast.ib->rec.mgid); + gid_type = id_priv->cma_dev->default_gid_type[id_priv->id.port_num - +rdma_start_port (id_priv->cma_dev->device)]; +cma_iboe_set_mgid (addr, &mc->multicast.ib->rec.mgid, gid_type); mc->multicast.ib->rec.pkey =cpu_to_be16 (0xffff );if (id_priv->id.ps ==RDMA_PS_UDP )
@@ -4090,8 +4093,6 @@ static int cma_iboe_join_multicast(struct rdma_id_private *id_priv, mc->multicast.ib->rec.hop_limit =1 ; mc->multicast.ib->rec.mtu =iboe_get_mtu (ndev->mtu); - gid_type = id_priv->cma_dev->default_gid_type[id_priv->id.port_num - -rdma_start_port (id_priv->cma_dev->device)];if (addr->sa_family ==AF_INET ) {if (gid_type ==IB_GID_TYPE_ROCE_UDP_ENCAP ) { mc->multicast.ib->rec.hop_limit =IPV6_DEFAULT_HOPLIMIT ;
drivers/infiniband/core/verbs.c +6 -4
@@ -1563,8 +1563,9 @@ int ib_attach_mcast(struct ib_qp *qp, union ib_gid *gid, u16 lid)if (!qp->device->attach_mcast)return -ENOSYS ; -if (gid->raw[0 ] !=0xff || qp->qp_type !=IB_QPT_UD || - !is_valid_mcast_lid (qp, lid)) + +if (!rdma_is_multicast_addr ((struct in6_addr *)gid->raw) || + qp->qp_type !=IB_QPT_UD || !is_valid_mcast_lid (qp, lid))return -EINVAL ; ret = qp->device->attach_mcast (qp, gid, lid);
@@ -1580,8 +1581,9 @@ int ib_detach_mcast(struct ib_qp *qp, union ib_gid *gid, u16 lid)if (!qp->device->detach_mcast)return -ENOSYS ; -if (gid->raw[0 ] !=0xff || qp->qp_type !=IB_QPT_UD || - !is_valid_mcast_lid (qp, lid)) + +if (!rdma_is_multicast_addr ((struct in6_addr *)gid->raw) || + qp->qp_type !=IB_QPT_UD || !is_valid_mcast_lid (qp, lid))return -EINVAL ; ret = qp->device->detach_mcast (qp, gid, lid);
include/rdma/ib_addr.h +7 -1
@@ -302,7 +302,13 @@ static inline void rdma_get_ll_mac(struct in6_addr *addr, u8 *mac)static inline int rdma_is_multicast_addr (struct in6_addr *addr) { -return addr->s6_addr[0 ] ==0xff ; +u32 ipv4_addr; + +if (addr->s6_addr[0 ] ==0xff ) +return 1 ; + +memcpy (&ipv4_addr, addr->s6_addr +12 ,4 ); +return (ipv6_addr_v4mapped (addr) &&ipv4_is_multicast (ipv4_addr)); }static inline void rdma_get_mcast_mac (struct in6_addr *addr,u8 *mac)
--
2.12.2
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH 3/3 IB/core: Add support for RoCEv2 multicast
In reply to a message not in the archiveFrom: Noa Osherovich <noaos-VPRAkNaXOzVWk0Htik3J/[email protected]>
When creating address handle from multicast GID, set MAC according to the appropriate formula instead of searching for it in the GID table:
- For IPv4 multicast GID use ip_eth_mc_map().
- For IPv6 multicast GID use ipv6_eth_mc_map().
--- drivers/infiniband/core/verbs.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)
drivers/infiniband/core/verbs.c +12 -0
@@ -1220,6 +1220,18 @@ int ib_resolve_eth_dmac(struct ib_device *device,if (rdma_link_local_addr ((struct in6_addr *)grh->dgid.raw)) {rdma_get_ll_mac ((struct in6_addr *)grh->dgid.raw, ah_attr->roce.dmac); +return 0 ; + } +if (rdma_is_multicast_addr ((struct in6_addr *)ah_attr->grh.dgid.raw)) { +if (ipv6_addr_v4mapped ((struct in6_addr *)ah_attr->grh.dgid.raw)) { +__be32 addr =0 ; + +memcpy (&addr, ah_attr->grh.dgid.raw +12 ,4 ); +ip_eth_mc_map (addr, (char *)ah_attr->roce.dmac); + }else { +ipv6_eth_mc_map ((struct in6_addr *)ah_attr->grh.dgid.raw, + (char *)ah_attr->roce.dmac); + } }else {union ib_gid sgid;struct ib_gid_attr sgid_attr;
--
2.12.2
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 24 lines · Leon Romanovsky
This series from Noa enables multicast traffic over RoCE v2.
To allow this, several changes were required.
First of all, there is a need to validate MLID when link layer is Infiniband. This check is mandatory according to Infiniband spec but the MLID should be ignored for RoCE according to RoCEv2 Annex (Annex 17, A17.5.6).
Second, there is need to cange the mapping of IPv4 multicast addresses to MGIDs to be according to RoCEv2 Annex (A17.4.1.1 L3 Addresses): GID = ::ffff:<IPv4 Address>
And third, multicast addresses need to be mapped to multicast MAC instead of searching for it in the GID table.
Thanks
Noa Osherovich (3):
IB/core: Fix the validations of a multicast LID in attach or detach
operations
IB/core: Set RoCEv2 MGID according to spec
IB/core: Add support for RoCEv2 multicast
Quoted 5 lines
drivers/infiniband/core/cma.c | 13 +++++---- drivers/infiniband/core/verbs.c | 62 +++++++++++++++++++++++++++++++++++++---- include/rdma/ib_addr.h | 8 +++++- 3 files changed, 70 insertions(+), 13 deletions(-)
Thanks, series applied.
Signature · 8 lines
Doug Ledford <[email protected]> GPG KeyID: B826A3330E572FDD Key fingerprint = AE6B 1BDA 122B 23B4 265B 1274 B826 A333 0E57 2FDD -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html