RDMA core, mlx5 fixes and refactoring for 4.15
Subsystems (6)
cover RDMA core, mlx5 fixes and refactoring for 4.15
1) Introduce and use rdma_create_user_ah
Currently there is inconsistency in resolving destination mac address in ib core and provider driver as below.
1. Provider drivers are using different API to resolve dmac.
2. Provider drivers are resolving destination mac address for user space consumers, but don't have to resolve for kernel consumers.
This leads ah_attr->roce.dmac being input field for kernel consumers and treated as output field for user space consumers.
Without proper API, umad module miss resolving destination mac address while creating address handle.
Additionally mlx4 driver resolves incorrect mac address for v4 mapped v6 destination multicast gid.
This patchset allows uverbs modules to invoke ib_core API that resolves destination mac address for user consumers.
This patchset avoids above issues by ib_core resolving destination mac address for roce for user as well as kernel consumers.
It uses ib_resolve_eth_dmac() API which resolves destination mac address for unicast, multicast, v4 mapped v6 addresses, and also link local gid. This eliminates code duplication in provider drivers and also brings uniformity by always resolving dmac in ib_core for all type of GIDs.
This allows to eliminate exporting low level API ib_resolve_eth_dmac(). So its removed as export symbol until a good use case arise.
2) Three simple fixes for IB/core.
3) Patch for mlx5 posted to the mailing list.
The patches are available in the git repository at:
git.kernel.org/pub/scm/linux/kernel/git/leon/linux-rdma.git tags/rdma-next-2017-10-16
Thanks
---------------------------------------
Jérémy Lefaure (1):
IB/mlx5: Use ARRAY_SIZE
Parav Pandit (4):
IB: Let ib_core resolve destination mac address IB/core: Fix unable to change lifespan entry for hw_counters IB/core: Fix use workqueue without WQ_MEM_RECLAIM IB/core: Take into account optional UDR, XRC headers and mandatory
ICRC for RoCE MTU
drivers/infiniband/core/addr.c | 2 +- drivers/infiniband/core/sysfs.c | 16 +++++++++++++++- drivers/infiniband/core/verbs.c | 8 +++++--- drivers/infiniband/hw/bnxt_re/ib_verbs.c | 8 -------- drivers/infiniband/hw/hns/hns_roce_ah.c | 14 +------------- drivers/infiniband/hw/mlx4/ah.c | 8 +++----- drivers/infiniband/hw/mlx5/ah.c | 4 ---- drivers/infiniband/hw/mlx5/odp.c | 6 +++--- drivers/infiniband/hw/ocrdma/ocrdma_ah.c | 15 --------------- include/rdma/ib_addr.h | 7 ++++--- include/rdma/ib_pack.h | 19 +++++++++++-------- include/rdma/ib_verbs.h | 2 -- 12 files changed, 43 insertions(+), 66 deletions(-)
--
2.14.2
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 20171016052812.28802-1-leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org
PATCH 1/5 IB: Let ib_core resolve destination mac address
From: Parav Pandit <parav-VPRAkNaXOzVWk0Htik3J/[email protected]>
Since IB/core resolves the destination mac address for user and kernel consumers, avoid resolving in multiple provider drivers.
Only ib_core resolves DMAC now, therefore resolve_eth_dmac is removed as exported symbol.
--- drivers/infiniband/core/verbs.c | 8 +++++--- drivers/infiniband/hw/bnxt_re/ib_verbs.c | 8 -------- drivers/infiniband/hw/hns/hns_roce_ah.c | 14 +------------- drivers/infiniband/hw/mlx4/ah.c | 8 +++----- drivers/infiniband/hw/mlx5/ah.c | 4 ---- drivers/infiniband/hw/ocrdma/ocrdma_ah.c | 15 --------------- include/rdma/ib_verbs.h | 2 -- 7 files changed, 9 insertions(+), 50 deletions(-)
drivers/infiniband/core/verbs.c +5 -3
@@ -53,6 +53,9 @@#include "core_priv.h" +static int ib_resolve_eth_dmac (struct ib_device *device, +struct rdma_ah_attr *ah_attr); +static const char *const ib_events[] = { [IB_EVENT_CQ_ERR ] ="CQ error" , [IB_EVENT_QP_FATAL ] ="QP fatal error" ,
@@ -1257,8 +1260,8 @@ int ib_modify_qp_is_ok(enum ib_qp_state cur_state, enum ib_qp_state next_state, }EXPORT_SYMBOL (ib_modify_qp_is_ok); -int ib_resolve_eth_dmac (struct ib_device *device, -struct rdma_ah_attr *ah_attr) +static int ib_resolve_eth_dmac (struct ib_device *device, +struct rdma_ah_attr *ah_attr) {int ret =0 ;struct ib_global_route *grh;
@@ -1317,7 +1320,6 @@ int ib_resolve_eth_dmac(struct ib_device *device, out:return ret; } -EXPORT_SYMBOL (ib_resolve_eth_dmac);/** * ib_modify_qp_with_udata - Modifies the attributes for the specified QP.
drivers/infiniband/hw/bnxt_re/ib_verbs.c +0 -8
@@ -729,14 +729,6 @@ struct ib_ah *bnxt_re_create_ah(struct ib_pd *ib_pd, ah->qplib_ah.nw_type =CMDQ_CREATE_AH_TYPE_V1 ;break ; } - rc =rdma_addr_find_l2_eth_by_grh (&sgid, &grh->dgid, - ah_attr->roce.dmac, &vlan_tag, - &sgid_attr.ndev->ifindex, -NULL ); -if (rc) { -dev_err (rdev_to_dev (rdev),"Failed to get dmac\n" ); - goto fail; - } }memcpy (ah->qplib_ah.dmac, ah_attr->roce.dmac,ETH_ALEN );
drivers/infiniband/hw/hns/hns_roce_ah.c +1 -13
@@ -48,7 +48,6 @@ struct ib_ah *hns_roce_create_ah(struct ib_pd *ibpd,struct ib_gid_attr gid_attr;struct hns_roce_ah *ah;u16 vlan_tag =0xffff ; -struct in6_addr in6;const struct ib_global_route *grh =rdma_ah_read_grh (ah_attr);union ib_gid sgid;int ret;
@@ -58,18 +57,7 @@ struct ib_ah *hns_roce_create_ah(struct ib_pd *ibpd,return ERR_PTR (-ENOMEM );/* Get mac address */ -memcpy (&in6, grh->dgid.raw,sizeof (grh->dgid.raw)); -if (rdma_is_multicast_addr (&in6)) { -rdma_get_mcast_mac (&in6, ah->av.mac); - }else { -u8 *dmac =rdma_ah_retrieve_dmac (ah_attr); - -if (!dmac) { -kfree (ah); -return ERR_PTR (-EINVAL ); - } -memcpy (ah->av.mac, dmac,ETH_ALEN ); - } +memcpy (ah->av.mac, ah_attr->roce.dmac,ETH_ALEN );/* Get source gid */ ret = ib_get_cached_gid(ibpd->device, rdma_ah_get_port_num(ah_attr),
drivers/infiniband/hw/mlx4/ah.c +3 -5
@@ -92,12 +92,10 @@ static struct ib_ah *create_iboe_ah(struct ib_pd *pd,int ret;memcpy (&in6, grh->dgid.raw,sizeof (in6)); -if (rdma_is_multicast_addr (&in6)) { +if (rdma_is_multicast_addr (&in6)) is_mcast =1 ; -rdma_get_mcast_mac (&in6, ah->av.eth.mac); - }else { -memcpy (ah->av.eth.mac, ah_attr->roce.dmac,ETH_ALEN ); - } + +memcpy (ah->av.eth.mac, ah_attr->roce.dmac,ETH_ALEN ); ret =ib_get_cached_gid (pd->device,rdma_ah_get_port_num (ah_attr), grh->sgid_index, &sgid, &gid_attr);if (ret)
drivers/infiniband/hw/mlx5/ah.c +0 -4
@@ -89,10 +89,6 @@ struct ib_ah *mlx5_ib_create_ah(struct ib_pd *pd, struct rdma_ah_attr *ah_attr, resp.response_length = min_resp_len; - err =ib_resolve_eth_dmac (pd->device, ah_attr); -if (err) -return ERR_PTR (err); -memcpy (resp.dmac, ah_attr->roce.dmac,ETH_ALEN ); err =ib_copy_to_udata (udata, &resp, resp.response_length);if (err)
drivers/infiniband/hw/ocrdma/ocrdma_ah.c +0 -15
@@ -201,21 +201,6 @@ struct ib_ah *ocrdma_create_ah(struct ib_pd *ibpd, struct rdma_ah_attr *attr,/* Get network header type for this GID */ ah->hdr_type =ib_gid_to_network_type (sgid_attr.gid_type, &sgid); -if ((pd->uctx) && - (!rdma_is_multicast_addr ((struct in6_addr *)grh->dgid.raw)) && - (!rdma_link_local_addr ((struct in6_addr *)grh->dgid.raw))) { - status =rdma_addr_find_l2_eth_by_grh (&sgid, &grh->dgid, - attr->roce.dmac, - &vlan_tag, - &sgid_attr.ndev->ifindex, -NULL ); -if (status) { -pr_err ("%s(): Failed to resolve dmac from gid." -"status = %d\n" , __func__, status); - goto av_conf_err; - } - } - status =set_av_attr (dev, ah, attr, &sgid, pd->id, &isvlan, vlan_tag);if (status) goto av_conf_err;
include/rdma/ib_verbs.h +0 -2
@@ -3622,8 +3622,6 @@ void ib_drain_rq(struct ib_qp *qp);void ib_drain_sq (struct ib_qp *qp);void ib_drain_qp (struct ib_qp *qp); -int ib_resolve_eth_dmac (struct ib_device *device, -struct rdma_ah_attr *ah_attr);int ib_get_eth_speed (struct ib_device *dev,u8 port_num,u8 *speed,u8 *width);static inline u8 *rdma_ah_retrieve_dmac (struct rdma_ah_attr *attr)
Signature · 6 lines
2.14.2 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH 2/5 IB/core: Fix unable to change lifespan entry for hw_counters
In reply to a message not in the archiveFrom: Parav Pandit <parav-VPRAkNaXOzVWk0Htik3J/[email protected]>
This patch fixes the case where 'lifespan' entry of the hw_counters is not writable. Currently write callback is not exposed for for the hw_counters sysfs operation. Due to this, modifying lifespan value results into permission denied error in below example.
echo 10 > /sys/class/infiniband/mlx5_0/ports/1/hw_counters/lifespan -bash: /sys/class/infiniband/mlx5_0/ports/1/hw_counters/lifespan:
Permission denied
This patch adds the hook to modify any attribute which implements store() operation.
--- drivers/infiniband/core/sysfs.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-)
drivers/infiniband/core/sysfs.c +15 -1
@@ -108,8 +108,22 @@ static ssize_t port_attr_show(struct kobject *kobj,return port_attr->show (p, port_attr, buf); } +static ssize_t port_attr_store (struct kobject *kobj, +struct attribute *attr, +const char *buf,size_t count) +{ +struct port_attribute *port_attr = + container_of(attr, struct port_attribute, attr); + struct ib_port *p = container_of(kobj, struct ib_port, kobj); + +if (!port_attr->store) +return -EIO ; +return port_attr->store (p, port_attr, buf, count); +} +static const struct sysfs_ops port_sysfs_ops = { - .show = port_attr_show + .show = port_attr_show, + .store = port_attr_store }; static ssize_t gid_attr_show(struct kobject *kobj,
Signature · 6 lines
2.14.2 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH 3/5 IB/core: Fix use workqueue without WQ_MEM_RECLAIM
In reply to a message not in the archiveFrom: Parav Pandit <parav-VPRAkNaXOzVWk0Htik3J/[email protected]>
The IB/core provides address resolution service and invokes callback handler when address resolve request completes of requester in worker thread context.
Such caller might allocate or free memory in callback handler depending on the completion status to make further progress or to terminate a connection. Most ULPs resolve route which involves allocating route entry and path record elements in callback event handler.
It has been noticed that WQ_MEM_RECLAIM flag should not be used for workers that tend to allocate memory in this [1] thread discussion.
In order to mitigate this situation, WQ_MEM_RECLAIM flag was dropped for other such WQs in this [2] patch.
Similar problem might arise with address resolution path, though its not yet noticed. The ib_addr workqueue is not memory reclaim path due to its nature of invoking callback that might allocate memory or don't free any memory under memory pressure.
[1] https://www.spinics.net/lists/linux-rdma/msg53239.html
[2] https://www.spinics.net/lists/linux-rdma/msg53416.html
--- drivers/infiniband/core/addr.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/infiniband/core/addr.c +1 -1
@@ -852,7 +852,7 @@ static struct notifier_block nb = {int addr_init (void ) { - addr_wq =alloc_ordered_workqueue ("ib_addr" ,WQ_MEM_RECLAIM ); + addr_wq =alloc_ordered_workqueue ("ib_addr" ,0 );if (!addr_wq)return -ENOMEM ;
Signature · 6 lines
2.14.2 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH 4/5 IB/core: Take into account optional UDR, XRC headers and mandatory ICRC for RoCE MTU
In reply to a message not in the archiveFrom: Parav Pandit <parav-VPRAkNaXOzVWk0Htik3J/[email protected]>
This fix considers optional UDP, XRC header sizes and mandatory ICRC 4 bytes in calculation of path MTU, by considering additional 8 bytes, path MTU calculation is more accurate.
--- include/rdma/ib_addr.h | 7 ++++--- include/rdma/ib_pack.h | 19 +++++++++++-------- 2 files changed, 15 insertions(+), 11 deletions(-)
include/rdma/ib_addr.h +4 -3
@@ -245,10 +245,11 @@ static inline void rdma_addr_set_dgid(struct rdma_dev_addr *dev_addr, union ib_gstatic inline enum ib_mtu iboe_get_mtu (int mtu) {/* -* reduce IB headers from effective IBoE MTU. 28 stands for -* atomic header which is the biggest possible header after BTH +* Reduce IB headers from effective IBoE MTU. */ - mtu = mtu -IB_GRH_BYTES -IB_BTH_BYTES -28 ; + mtu = mtu - (IB_GRH_BYTES +IB_UDP_BYTES +IB_BTH_BYTES + +IB_EXT_XRC_BYTES +IB_EXT_ATOMICETH_BYTES + +IB_ICRC_BYTES );if (mtu >=ib_mtu_enum_to_int (IB_MTU_4096 ))return IB_MTU_4096 ;
include/rdma/ib_pack.h +11 -8
@@ -37,14 +37,17 @@#include <uapi/linux/if_ether.h> enum { -IB_LRH_BYTES =8 , -IB_ETH_BYTES =14 , -IB_VLAN_BYTES =4 , -IB_GRH_BYTES =40 , -IB_IP4_BYTES =20 , -IB_UDP_BYTES =8 , -IB_BTH_BYTES =12 , -IB_DETH_BYTES =8 +IB_LRH_BYTES =8 , +IB_ETH_BYTES =14 , +IB_VLAN_BYTES =4 , +IB_GRH_BYTES =40 , +IB_IP4_BYTES =20 , +IB_UDP_BYTES =8 , +IB_BTH_BYTES =12 , +IB_DETH_BYTES =8 , +IB_EXT_ATOMICETH_BYTES =28 , +IB_EXT_XRC_BYTES =4 , +IB_ICRC_BYTES =4 };struct ib_field {
Signature · 6 lines
2.14.2 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
PATCH 5/5 IB/mlx5: Use ARRAY_SIZE
In reply to a message not in the archiveFrom: Jérémy Lefaure <[email protected]>
Using the ARRAY_SIZE macro improves the readability of the code.
Found with Coccinelle with the following semantic patch:
@r depends on (org || report)@
type T;
T[] E;
position p;
@@
(
(sizeof(E)@p /sizeof(*E))
|
(sizeof(E)@p /sizeof(E[...]))
|
(sizeof(E)@p /sizeof(T))
)
--- drivers/infiniband/hw/mlx5/odp.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
drivers/infiniband/hw/mlx5/odp.c +3 -3
@@ -32,6 +32,7 @@#include <rdma/ib_umem.h> #include <rdma/ib_umem_odp.h> +#include <linux/kernel.h> #include "mlx5_ib.h" #include "cmd.h"
@@ -929,9 +930,8 @@ static int mlx5_ib_mr_initiator_pfault_handler(return -EFAULT ; } -if (unlikely (opcode >=sizeof (mlx5_ib_odp_opcode_cap) / -sizeof (mlx5_ib_odp_opcode_cap[0 ]) || - !(transport_caps & mlx5_ib_odp_opcode_cap[opcode]))) { +if (unlikely (opcode >=ARRAY_SIZE (mlx5_ib_odp_opcode_cap) || + !(transport_caps & mlx5_ib_odp_opcode_cap[opcode]))) {mlx5_ib_err (dev,"ODP fault on QP of an unsupported opcode 0x%x\n" , opcode);return -EFAULT ;
Signature · 6 lines
2.14.2 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 10 lines · Leon Romanovsky
Jérémy Lefaure (1):
IB/mlx5: UseARRAY_SIZEParav Pandit (4):
IB: Letib_coreresolve destination mac address IB/core: Fix unable to change lifespan entry forhw_countersIB/core: Fix use workqueue withoutWQ_MEM_RECLAIMIB/core: Take into account optional UDR, XRC headers and mandatory
ICRC for RoCE MTU
Doug,
Sorry, please drop this series, I missed one patch, it should be 6 patches and not 5.
Thanks
A message that is not in the archive (sent off-list or not yet received): 20171016054225.GC2106-U/[email protected]
Quoted 13 lines · Leon Romanovsky
Quoted 11 lines · Leon Romanovsky
Jérémy Lefaure (1):
IB/mlx5: UseARRAY_SIZEParav Pandit (4):
IB: Letib_coreresolve destination mac address IB/core: Fix unable to change lifespan entry forhw_countersIB/core: Fix use workqueue withoutWQ_MEM_RECLAIMIB/core: Take into account optional UDR, XRC headers and
mandatory
ICRC for RoCE MTU
Quoted 3 lines
Doug,
Sorry, please drop this series, I missed one patch, it should be 6 patches and not 5.
Done.
Signature · 8 lines
Doug Ledford <[email protected]> GPG KeyID: B826A3330E572FDD Key fingerprint = AE6B 1BDA 122B 23B4 265B 1274 B826 A333 0E57 2FDD -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html