Fixes for v5.3

PATCH In review Leon Romanovsky 5 participants 14 messages Aug 15, 2019 – Aug 20, 2019

Subsystems (2)

InfiniBand Mellanox MLX5 IB driver

lore

Share & tools

lore

Leon Romanovsky

author

2019-08-15 08:38 UTC #0 lore raw

cover Fixes for v5.3

From: Leon Romanovsky <[email protected]>

Hi,

This is a collection of new fixes for v5.3, everything here is fixing kernel crash or visible bug with one exception: patch #5 "IB/mlx5:
Consolidate use_umr checks into single function". That patch is nice to have improvement to implement rest of the series.

Thanks

Ido Kalir (1):
IB/core: Fix NULL pointer dereference when bind QP to counter

Jason Gunthorpe (1):
RDMA/mlx5: Fix MR npages calculation for IB_ACCESS_HUGETLB

Leon Romanovsky (2):
RDMA/counters: Properly implement PID checks RDMA/restrack: Rewrite PID namespace check to be reliable

Moni Shoua (4):
IB/mlx5: Consolidate use_umr checks into single function IB/mlx5: Report and handle ODP support properly IB/mlx5: Fix MR re-registration flow to use UMR properly IB/mlx5: Block MR WR if UMR is not possible

 drivers/infiniband/core/counters.c   | 10 ++++------
 drivers/infiniband/core/nldev.c      |  3 +--
 drivers/infiniband/core/restrack.c   | 15 +++++++--------
 drivers/infiniband/core/umem.c       |  7 +------
 drivers/infiniband/hw/mlx5/main.c    |  6 +++---
 drivers/infiniband/hw/mlx5/mem.c     |  5 +++--
 drivers/infiniband/hw/mlx5/mlx5_ib.h | 14 ++++++++++++++
 drivers/infiniband/hw/mlx5/mr.c      |  7 +++----
 drivers/infiniband/hw/mlx5/odp.c     | 17 +++++++++--------
 drivers/infiniband/hw/mlx5/qp.c      | 24 +++++++++++++++++++-----
 include/rdma/restrack.h              |  3 +--
 11 files changed, 65 insertions(+), 46 deletions(-)

--
2.20.1

Leon Romanovsky

author

2019-08-15 08:38 UTC #1 lore raw

PATCH 1/8 IB/core: Fix NULL pointer dereference when bind QP to counter

From: Ido Kalir <[email protected]>

If QP is not visible to the pid, then we try to decrease its reference count and return from the function before the QP pointer is initialized. This lead to NULL pointer dereference.
Fix it by pass directly the res to the rdma_restract_put as arg instead of &qp->res.

This fixes below call trace:

[ 5845.110329] BUG: kernel NULL pointer dereference, address:
00000000000000dc
[ 5845.120482] Oops: 0002 [#1] SMP PTI
[ 5845.129119] RIP: 0010:rdma_restrack_put+0x5/0x30 [ib_core]
[ 5845.169450] Call Trace:
[ 5845.170544]  rdma_counter_get_qp+0x5c/0x70 [ib_core]
[ 5845.172074]  rdma_counter_bind_qpn_alloc+0x6f/0x1a0 [ib_core]
[ 5845.173731]  nldev_stat_set_doit+0x314/0x330 [ib_core]
[ 5845.175279]  rdma_nl_rcv_msg+0xeb/0x1d0 [ib_core]
[ 5845.176772]  ? __kmalloc_node_track_caller+0x20b/0x2b0
[ 5845.178321]  rdma_nl_rcv+0xcb/0x120 [ib_core]
[ 5845.179753]  netlink_unicast+0x179/0x220
[ 5845.181066]  netlink_sendmsg+0x2d8/0x3d0
[ 5845.182338]  sock_sendmsg+0x30/0x40
[ 5845.183544]  __sys_sendto+0xdc/0x160
[ 5845.184832]  ? syscall_trace_enter+0x1f8/0x2e0
[ 5845.186209]  ? __audit_syscall_exit+0x1d9/0x280
[ 5845.187584]  __x64_sys_sendto+0x24/0x30
[ 5845.188867]  do_syscall_64+0x48/0x120
[ 5845.190097]  entry_SYSCALL_64_after_hwframe+0x44/0xa9

Fixes 1bd8e0a9d0fd1 ("RDMA/counter: Allow manual mode configuration support")

Signed-off-by Ido Kalir <[email protected]>

Signed-off-by Leon Romanovsky <[email protected]>

---
 drivers/infiniband/core/counters.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

drivers/infiniband/core/counters.c +1 -1

index b79890739a2c..955d061af06a 100644
@@ -424,7 +424,7 @@ static struct ib_qp *rdma_counter_get_qp(struct ib_device *dev, u32 qp_num)
 	return qp;
 
 err:
-	rdma_restrack_put(&qp->res);
+	rdma_restrack_put(res);
 	return NULL;
 }

Signature · 1 line

2.20.1

Leon Romanovsky

author

2019-08-15 08:38 UTC #2 lore raw

PATCH 2/8 RDMA/counters: Properly implement PID checks

In reply to Leon Romanovsky

From: Leon Romanovsky <[email protected]>

"Auto" configuration mode is called for visible in that PID namespace and it ensures that all counters and QPs are coexist in the same namespace and belong to same PID.

Fixes 99fa331dc862 ("RDMA/counter: Add "auto" configuration mode support")

Reviewed-by Mark Zhang <[email protected]>

Signed-off-by Leon Romanovsky <[email protected]>

---
 drivers/infiniband/core/counters.c | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)

drivers/infiniband/core/counters.c +3 -5

index 955d061af06a..af8c85d18e62 100644
@@ -149,13 +149,11 @@ static bool auto_mode_match(struct ib_qp *qp, struct rdma_counter *counter,
 	struct auto_mode_param *param = &counter->mode.param;
 	bool match = true;
 
-	if (rdma_is_kernel_res(&counter->res) != rdma_is_kernel_res(&qp->res))
+	if (!rdma_is_visible_in_pid_ns(&qp->res))
 		return false;
 
-	/* Ensure that counter belong to right PID */
-	if (!rdma_is_kernel_res(&counter->res) &&
-	    !rdma_is_kernel_res(&qp->res) &&
-	    (task_pid_vnr(counter->res.task) != current->pid))
+	/* Ensure that counter belongs to the right PID */
+	if (task_pid_nr(counter->res.task) != task_pid_nr(qp->res.task))
 		return false;
 
 	if (auto_mask & RDMA_COUNTER_MASK_QP_TYPE)

Signature · 1 line

2.20.1

Leon Romanovsky

author

2019-08-15 08:38 UTC #3 lore raw

PATCH 3/8 RDMA/restrack: Rewrite PID namespace check to be reliable

In reply to Leon Romanovsky

From: Leon Romanovsky <[email protected]>

task_active_pid_ns() is wrong API to check PID namespace because it posses some restrictions and return PID namespace where the process was allocated. It created mismatches with current namespace, which can be different.

Rewrite whole rdma_is_visible_in_pid_ns() logic to provide reliable results without any relation to allocated PID namespace.

Fixes 8be565e65fa9 ("RDMA/nldev: Factor out the PID namespace check")

Fixes 6a6c306a09b5 ("RDMA/restrack: Make is_visible_in_pid_ns() as an API")

Reviewed-by Mark Zhang <[email protected]>

Signed-off-by Leon Romanovsky <[email protected]>

---
 drivers/infiniband/core/nldev.c    |  3 +--
 drivers/infiniband/core/restrack.c | 15 +++++++--------
 include/rdma/restrack.h            |  3 +--
 3 files changed, 9 insertions(+), 12 deletions(-)

drivers/infiniband/core/nldev.c +1 -2

index 87d40d1ecdde..020c26976558 100644
@@ -382,8 +382,7 @@ static int fill_res_info(struct sk_buff *msg, struct ib_device *device)
 	for (i = 0; i < RDMA_RESTRACK_MAX; i++) {
 		if (!names[i])
 			continue;
-		curr = rdma_restrack_count(device, i,
-					   task_active_pid_ns(current));
+		curr = rdma_restrack_count(device, i);
 		ret = fill_res_info_entry(msg, names[i], curr);
 		if (ret)
 			goto err;

drivers/infiniband/core/restrack.c +7 -8

index bddff426ee0f..a07665f7ef8c 100644
@@ -107,10 +107,8 @@ void rdma_restrack_clean(struct ib_device *dev)
  * rdma_restrack_count() - the current usage of specific object
  * @dev:  IB device
  * @type: actual type of object to operate
- * @ns:   PID namespace
  */
-int rdma_restrack_count(struct ib_device *dev, enum rdma_restrack_type type,
-			struct pid_namespace *ns)
+int rdma_restrack_count(struct ib_device *dev, enum rdma_restrack_type type)
 {
 	struct rdma_restrack_root *rt = &dev->res[type];
 	struct rdma_restrack_entry *e;

@@ -119,10 +117,9 @@ int rdma_restrack_count(struct ib_device *dev, enum rdma_restrack_type type,
 
 	xa_lock(&rt->xa);
 	xas_for_each(&xas, e, U32_MAX) {
-		if (ns == &init_pid_ns ||
-		    (!rdma_is_kernel_res(e) &&
-		     ns == task_active_pid_ns(e->task)))
-			cnt++;
+		if (!rdma_is_visible_in_pid_ns(e))
+			continue;
+		cnt++;
 	}
 	xa_unlock(&rt->xa);
 	return cnt;

@@ -360,5 +357,7 @@ bool rdma_is_visible_in_pid_ns(struct rdma_restrack_entry *res)
 	 */
 	if (rdma_is_kernel_res(res))
 		return task_active_pid_ns(current) == &init_pid_ns;
-	return task_active_pid_ns(current) == task_active_pid_ns(res->task);
+
+	/* PID 0 means that resource is not found in current namespace */
+	return task_pid_vnr(res->task);
 }

include/rdma/restrack.h +1 -2

index b0fc6b26bdf5..83df1ec6664e 100644
@@ -105,8 +105,7 @@ struct rdma_restrack_entry {
 };
 
 int rdma_restrack_count(struct ib_device *dev,
-			enum rdma_restrack_type type,
-			struct pid_namespace *ns);
+			enum rdma_restrack_type type);
 
 void rdma_restrack_kadd(struct rdma_restrack_entry *res);
 void rdma_restrack_uadd(struct rdma_restrack_entry *res);

Signature · 1 line

2.20.1

Leon Romanovsky

author

2019-08-15 08:39 UTC #4 lore raw

PATCH 4/8 RDMA/mlx5: Fix MR npages calculation for IB_ACCESS_HUGETLB

In reply to Leon Romanovsky

From: Jason Gunthorpe <[email protected]>

When ODP is enabled with IB_ACCESS_HUGETLB then the required pages should be calculated based on the extent of the MR, which is rounded to the nearest huge page alignment.

Fixes d2183c6f1958 ("RDMA/umem: Move page_shift from ib_umem to ib_odp_umem")

Signed-off-by Jason Gunthorpe <[email protected]>

Signed-off-by Leon Romanovsky <[email protected]>

---
 drivers/infiniband/core/umem.c   | 7 +------
 drivers/infiniband/hw/mlx5/mem.c | 5 +++--
 2 files changed, 4 insertions(+), 8 deletions(-)

drivers/infiniband/core/umem.c +1 -6

index 08da840ed7ee..56553668256f 100644
@@ -379,14 +379,9 @@ EXPORT_SYMBOL(ib_umem_release);
 
 int ib_umem_page_count(struct ib_umem *umem)
 {
-	int i;
-	int n;
+	int i, n = 0;
 	struct scatterlist *sg;
 
-	if (umem->is_odp)
-		return ib_umem_num_pages(umem);
-
-	n = 0;
 	for_each_sg(umem->sg_head.sgl, sg, umem->nmap, i)
 		n += sg_dma_len(sg) >> PAGE_SHIFT;

drivers/infiniband/hw/mlx5/mem.c +3 -2

index fe1a76d8531c..a40e0abf2338 100644
@@ -57,9 +57,10 @@ void mlx5_ib_cont_pages(struct ib_umem *umem, u64 addr,
 	int entry;
 
 	if (umem->is_odp) {
-		unsigned int page_shift = to_ib_umem_odp(umem)->page_shift;
+		struct ib_umem_odp *odp = to_ib_umem_odp(umem);
+		unsigned int page_shift = odp->page_shift;
 
-		*ncont = ib_umem_page_count(umem);
+		*ncont = ib_umem_odp_num_pages(odp);
 		*count = *ncont << (page_shift - PAGE_SHIFT);
 		*shift = page_shift;
 		if (order)

Signature · 1 line

2.20.1

Leon Romanovsky

author

2019-08-15 08:38 UTC #5 lore raw

PATCH 5/8 IB/mlx5: Consolidate use_umr checks into single function

In reply to Leon Romanovsky

From: Moni Shoua <[email protected]>

Introduce helper function to unify various use_umr checks.

Signed-off-by Moni Shoua <[email protected]>

Signed-off-by Leon Romanovsky <[email protected]>

---
 drivers/infiniband/hw/mlx5/mlx5_ib.h | 14 ++++++++++++++
 drivers/infiniband/hw/mlx5/mr.c      |  4 +---
 2 files changed, 15 insertions(+), 3 deletions(-)

drivers/infiniband/hw/mlx5/mlx5_ib.h +14 -0

index f6a53455bf8b..9ae587b74b12 100644
@@ -1475,4 +1475,18 @@ int bfregn_to_uar_index(struct mlx5_ib_dev *dev,
 			bool dyn_bfreg);
 
 int mlx5_ib_qp_set_counter(struct ib_qp *qp, struct rdma_counter *counter);
+
+static inline bool mlx5_ib_can_use_umr(struct mlx5_ib_dev *dev,
+				       bool do_modify_atomic)
+{
+	if (MLX5_CAP_GEN(dev->mdev, umr_modify_entity_size_disabled))
+		return false;
+
+	if (do_modify_atomic &&
+	    MLX5_CAP_GEN(dev->mdev, atomic) &&
+	    MLX5_CAP_GEN(dev->mdev, umr_modify_atomic_disabled))
+		return false;
+
+	return true;
+}
 #endif /* MLX5_IB_H */

drivers/infiniband/hw/mlx5/mr.c +1 -3

index b74fad08412f..8bce65c03b84 100644
@@ -1293,9 +1293,7 @@ struct ib_mr *mlx5_ib_reg_user_mr(struct ib_pd *pd, u64 start, u64 length,
 	if (err < 0)
 		return ERR_PTR(err);
 
-	use_umr = !MLX5_CAP_GEN(dev->mdev, umr_modify_entity_size_disabled) &&
-		  (!MLX5_CAP_GEN(dev->mdev, umr_modify_atomic_disabled) ||
-		   !MLX5_CAP_GEN(dev->mdev, atomic));
+	use_umr = mlx5_ib_can_use_umr(dev, true);
 
 	if (order <= mr_cache_max_order(dev) && use_umr) {
 		mr = alloc_mr_from_cache(pd, umem, virt_addr, length, ncont,

Signature · 1 line

2.20.1

Leon Romanovsky

author

2019-08-15 08:39 UTC #6 lore raw

PATCH 6/8 IB/mlx5: Report and handle ODP support properly

In reply to Leon Romanovsky

From: Moni Shoua <[email protected]>

ODP depends on the several device capabilities, among them is the ability to send UMR WQEs with that modify atomic and entity size of the MR. Therefore, only if all conditions to send such a UMR WQE are met then driver can report that ODP is supported. Use this check of conditions in all places where driver needs to know about ODP support.

Also, implicit ODP support depends on ability of driver to send UMR WQEs for an indirect mkey. Therefore, verify that all conditions to do so are met when reporting support.

Fixes c8d75a980fab ("IB/mlx5: Respect new UMR capabilities")

Signed-off-by Moni Shoua <[email protected]>

Reviewed-by Guy Levi <[email protected]>

Signed-off-by Leon Romanovsky <[email protected]>

---
 drivers/infiniband/hw/mlx5/main.c |  6 +++---
 drivers/infiniband/hw/mlx5/odp.c  | 17 +++++++++--------
 2 files changed, 12 insertions(+), 11 deletions(-)

drivers/infiniband/hw/mlx5/main.c +3 -3

index e12a4404096b..0569bcab02d4 100644
@@ -1023,7 +1023,7 @@ static int mlx5_ib_query_device(struct ib_device *ibdev,
 	props->timestamp_mask = 0x7FFFFFFFFFFFFFFFULL;
 
 	if (IS_ENABLED(CONFIG_INFINIBAND_ON_DEMAND_PAGING)) {
-		if (MLX5_CAP_GEN(mdev, pg))
+		if (dev->odp_caps.general_caps & IB_ODP_SUPPORT)
 			props->device_cap_flags |= IB_DEVICE_ON_DEMAND_PAGING;
 		props->odp_caps = dev->odp_caps;
 	}

@@ -6139,6 +6139,8 @@ static int mlx5_ib_stage_init_init(struct mlx5_ib_dev *dev)
 		dev->port[i].roce.last_port_state = IB_PORT_DOWN;
 	}
 
+	mlx5_ib_internal_fill_odp_caps(dev);
+
 	err = mlx5_ib_init_multiport_master(dev);
 	if (err)
 		return err;

@@ -6563,8 +6565,6 @@ static void mlx5_ib_stage_dev_res_cleanup(struct mlx5_ib_dev *dev)
 
 static int mlx5_ib_stage_odp_init(struct mlx5_ib_dev *dev)
 {
-	mlx5_ib_internal_fill_odp_caps(dev);
-
 	return mlx5_ib_odp_init_one(dev);
 }

drivers/infiniband/hw/mlx5/odp.c +9 -8

index 1d257d1b3b0d..0a59912a4cef 100644
@@ -301,7 +301,8 @@ void mlx5_ib_internal_fill_odp_caps(struct mlx5_ib_dev *dev)
 
 	memset(caps, 0, sizeof(*caps));
 
-	if (!MLX5_CAP_GEN(dev->mdev, pg))
+	if (!MLX5_CAP_GEN(dev->mdev, pg) ||
+	    !mlx5_ib_can_use_umr(dev, true))
 		return;
 
 	caps->general_caps = IB_ODP_SUPPORT;

@@ -355,7 +356,8 @@ void mlx5_ib_internal_fill_odp_caps(struct mlx5_ib_dev *dev)
 
 	if (MLX5_CAP_GEN(dev->mdev, fixed_buffer_size) &&
 	    MLX5_CAP_GEN(dev->mdev, null_mkey) &&
-	    MLX5_CAP_GEN(dev->mdev, umr_extended_translation_offset))
+	    MLX5_CAP_GEN(dev->mdev, umr_extended_translation_offset) &&
+	    !MLX5_CAP_GEN(dev->mdev, umr_indirect_mkey_disabled))
 		caps->general_caps |= IB_ODP_SUPPORT_IMPLICIT;
 
 	return;

@@ -1622,8 +1624,10 @@ int mlx5_ib_odp_init_one(struct mlx5_ib_dev *dev)
 {
 	int ret = 0;
 
-	if (dev->odp_caps.general_caps & IB_ODP_SUPPORT)
-		ib_set_device_ops(&dev->ib_dev, &mlx5_ib_dev_odp_ops);
+	if (!(dev->odp_caps.general_caps & IB_ODP_SUPPORT))
+		return ret;
+
+	ib_set_device_ops(&dev->ib_dev, &mlx5_ib_dev_odp_ops);
 
 	if (dev->odp_caps.general_caps & IB_ODP_SUPPORT_IMPLICIT) {
 		ret = mlx5_cmd_null_mkey(dev->mdev, &dev->null_mkey);

@@ -1633,9 +1637,6 @@ int mlx5_ib_odp_init_one(struct mlx5_ib_dev *dev)
 		}
 	}
 
-	if (!MLX5_CAP_GEN(dev->mdev, pg))
-		return ret;
-
 	ret = mlx5_ib_create_pf_eq(dev, &dev->odp_pf_eq);
 
 	return ret;

@@ -1643,7 +1644,7 @@ int mlx5_ib_odp_init_one(struct mlx5_ib_dev *dev)
 
 void mlx5_ib_odp_cleanup_one(struct mlx5_ib_dev *dev)
 {
-	if (!MLX5_CAP_GEN(dev->mdev, pg))
+	if (!(dev->odp_caps.general_caps & IB_ODP_SUPPORT))
 		return;
 
 	mlx5_ib_destroy_pf_eq(dev, &dev->odp_pf_eq);

Signature · 1 line

2.20.1

Leon Romanovsky

author

2019-08-15 08:39 UTC #7 lore raw

PATCH 7/8 IB/mlx5: Fix MR re-registration flow to use UMR properly

In reply to Leon Romanovsky

From: Moni Shoua <[email protected]>

The UMR WQE in the MR re-registration flow requires that modify_atomic and modify_entity_size capabilities are enabled. Therefore, check that the these capabilities are present before going to umr flow and go through slow path if not.

Fixes c8d75a980fab ("IB/mlx5: Respect new UMR capabilities")

Signed-off-by Moni Shoua <[email protected]>

Reviewed-by Guy Levi <[email protected]>

Signed-off-by Leon Romanovsky <[email protected]>

---
 drivers/infiniband/hw/mlx5/mr.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

drivers/infiniband/hw/mlx5/mr.c +2 -1

index 8bce65c03b84..3401f5f6792e 100644
@@ -1446,7 +1446,8 @@ int mlx5_ib_rereg_user_mr(struct ib_mr *ib_mr, int flags, u64 start,
 			goto err;
 	}
 
-	if (flags & IB_MR_REREG_TRANS && !use_umr_mtt_update(mr, addr, len)) {
+	if (!mlx5_ib_can_use_umr(dev, true) ||
+	    (flags & IB_MR_REREG_TRANS && !use_umr_mtt_update(mr, addr, len))) {
 		/*
 		 * UMR can't be used - MKey needs to be replaced.
 		 */

Signature · 1 line

2.20.1

Leon Romanovsky

author

2019-08-15 08:39 UTC #8 lore raw

PATCH 8/8 IB/mlx5: Block MR WR if UMR is not possible

In reply to Leon Romanovsky

From: Moni Shoua <[email protected]>

Check conditions that are mandatory to post_send UMR WQEs.
1. Modifying page size.
2. Modifying remote atomic permissions if atomic access is required.

If either condition is not fulfilled then fail to post_send() flow.

Fixes c8d75a980fab ("IB/mlx5: Respect new UMR capabilities")

Signed-off-by Moni Shoua <[email protected]>

Reviewed-by Guy Levi <[email protected]>

Signed-off-by Leon Romanovsky <[email protected]>

---
 drivers/infiniband/hw/mlx5/qp.c | 24 +++++++++++++++++++-----
 1 file changed, 19 insertions(+), 5 deletions(-)

drivers/infiniband/hw/mlx5/qp.c +19 -5

index 379328b2598f..72869ff4a334 100644
@@ -4162,7 +4162,7 @@ static u64 get_xlt_octo(u64 bytes)
 	       MLX5_IB_UMR_OCTOWORD;
 }
 
-static __be64 frwr_mkey_mask(void)
+static __be64 frwr_mkey_mask(bool atomic)
 {
 	u64 result;

@@ -4175,10 +4175,12 @@ static __be64 frwr_mkey_mask(void)
 		MLX5_MKEY_MASK_LW		|
 		MLX5_MKEY_MASK_RR		|
 		MLX5_MKEY_MASK_RW		|
-		MLX5_MKEY_MASK_A		|
 		MLX5_MKEY_MASK_SMALL_FENCE	|
 		MLX5_MKEY_MASK_FREE;
 
+	if (atomic)
+		result |= MLX5_MKEY_MASK_A;
+
 	return cpu_to_be64(result);
 }

@@ -4204,7 +4206,7 @@ static __be64 sig_mkey_mask(void)
 }
 
 static void set_reg_umr_seg(struct mlx5_wqe_umr_ctrl_seg *umr,
-			    struct mlx5_ib_mr *mr, u8 flags)
+			    struct mlx5_ib_mr *mr, u8 flags, bool atomic)
 {
 	int size = (mr->ndescs + mr->meta_ndescs) * mr->desc_size;

@@ -4212,7 +4214,7 @@ static void set_reg_umr_seg(struct mlx5_wqe_umr_ctrl_seg *umr,
 
 	umr->flags = flags;
 	umr->xlt_octowords = cpu_to_be16(get_xlt_octo(size));
-	umr->mkey_mask = frwr_mkey_mask();
+	umr->mkey_mask = frwr_mkey_mask(atomic);
 }
 
 static void set_linv_umr_seg(struct mlx5_wqe_umr_ctrl_seg *umr)

@@ -4811,10 +4813,22 @@ static int set_reg_wr(struct mlx5_ib_qp *qp,
 {
 	struct mlx5_ib_mr *mr = to_mmr(wr->mr);
 	struct mlx5_ib_pd *pd = to_mpd(qp->ibqp.pd);
+	struct mlx5_ib_dev *dev = to_mdev(pd->ibpd.device);
 	int mr_list_size = (mr->ndescs + mr->meta_ndescs) * mr->desc_size;
 	bool umr_inline = mr_list_size <= MLX5_IB_SQ_UMR_INLINE_THRESHOLD;
+	bool atomic = wr->access & IB_ACCESS_REMOTE_ATOMIC;
 	u8 flags = 0;
 
+	if (!mlx5_ib_can_use_umr(dev, atomic)) {
+		mlx5_ib_warn(to_mdev(qp->ibqp.device),
+			     "Fast update of %s for MR is disabled\n",
+			     (MLX5_CAP_GEN(dev->mdev,
+					   umr_modify_entity_size_disabled)) ?
+				     "entity size" :
+				     "atomic access");
+		return -EINVAL;
+	}
+
 	if (unlikely(wr->wr.send_flags & IB_SEND_INLINE)) {
 		mlx5_ib_warn(to_mdev(qp->ibqp.device),
 			     "Invalid IB_SEND_INLINE send flag\n");

@@ -4826,7 +4840,7 @@ static int set_reg_wr(struct mlx5_ib_qp *qp,
 	if (umr_inline)
 		flags |= MLX5_UMR_INLINE;
 
-	set_reg_umr_seg(*seg, mr, flags);
+	set_reg_umr_seg(*seg, mr, flags, atomic);
 	*seg += sizeof(struct mlx5_wqe_umr_ctrl_seg);
 	*size += sizeof(struct mlx5_wqe_umr_ctrl_seg) / 16;
 	handle_post_send_edge(&qp->sq, seg, *size, cur_edge);

--
2.20.1

Jason Gunthorpe

2019-08-15 19:29 UTC #9 lore raw

In reply to Leon Romanovsky

Quoted 21 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>
Hi,
This is a collection of new fixes for v5.3, everything here is fixing kernel crash or visible bug with one exception: patch #5 "IB/mlx5:
Consolidate use_umr checks into single function". That patch is nice to have improvement to implement rest of the series.
Thanks
Ido Kalir (1):
IB/core: Fix NULL pointer dereference when bind QP to counter
Jason Gunthorpe (1):
RDMA/mlx5: Fix MR npages calculation for IB_ACCESS_HUGETLB
Leon Romanovsky (2):
RDMA/counters: Properly implement PID checks RDMA/restrack: Rewrite PID namespace check to be reliable

Quoted 5 lines

Moni Shoua (4):
IB/mlx5: Consolidate use_umr checks into single function IB/mlx5: Report and handle ODP support properly IB/mlx5: Fix MR re-registration flow to use UMR properly IB/mlx5: Block MR WR if UMR is not possible

I'm a little murky on what thes are fixing? Moni?

Jason

Guy Levi(SW)

2019-08-15 20:26 UTC #10 lore raw

Quoted 35 lines

-----Original Message-----
From: Jason Gunthorpe <[email protected]>
Sent: Thursday, August 15, 2019 10:30 PM
To: Leon Romanovsky <[email protected]>; Moni Shoua <[email protected]>
Cc: Doug Ledford <[email protected]>; Leon Romanovsky <[email protected]>; RDMA mailing list <[email protected]>; Guy Levi(SW) <[email protected]>; Ido Kalir <[email protected]>; Majd Dibbiny <[email protected]>; Mark Zhang <[email protected]>
Subject: Re: [PATCH rdma-rc 0/8] Fixes for v5.3
Quoted 21 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
Hi,
This is a collection of new fixes for v5.3, everything here is fixing kernel crash or visible bug with one exception: patch #5 "IB/mlx5:
Consolidate use_umr checks into single function". That patch is nice to have improvement to implement rest of the series.
Thanks
Ido Kalir (1):
IB/core: Fix NULL pointer dereference when bind QP to counter
Jason Gunthorpe (1):
RDMA/mlx5: Fix MR npages calculation for IB_ACCESS_HUGETLB
Leon Romanovsky (2):
RDMA/counters: Properly implement PID checks RDMA/restrack: Rewrite PID namespace check to be reliable

Quoted 7 lines

Quoted 5 lines
Moni Shoua (4):
IB/mlx5: Consolidate use_umr checks into single function IB/mlx5: Report and handle ODP support properly IB/mlx5: Fix MR re-registration flow to use UMR properly IB/mlx5: Block MR WR if UMR is not possible
I'm a little murky on what thes are fixing? Moni?

See original fixed patch.
Sometime the HW device disables some UMR abilities and hence driver should avoid UMR usage in some cases. This HW behavior can be seen today in Linux@windows environment with "untrusted VF".

Quoted 2 lines

Jason

Doug Ledford

2019-08-20 16:56 UTC #11 lore raw

In reply to Leon Romanovsky

Quoted 21 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>
Hi,
This is a collection of new fixes for v5.3, everything here is fixing kernel crash or visible bug with one exception: patch #5 "IB/mlx5:
Consolidate use_umr checks into single function". That patch is nice to have improvement to implement rest of the series.
Thanks
Ido Kalir (1):
IB/core: Fix NULL pointer dereference when bind QP to counter
Jason Gunthorpe (1):
RDMA/mlx5: Fix MR npages calculation for IB_ACCESS_HUGETLB
Leon Romanovsky (2):
RDMA/counters: Properly implement PID checks RDMA/restrack: Rewrite PID namespace check to be reliable

Quoted 5 lines

Moni Shoua (4):
IB/mlx5: Consolidate use_umr checks into single function IB/mlx5: Report and handle ODP support properly IB/mlx5: Fix MR re-registration flow to use UMR properly IB/mlx5: Block MR WR if UMR is not possible

Hi Leon,

I took everything except Jason's patch to for-rc. He had tagged his patch in patchworks as under review by himself, mainly because there are some conflicts with other hmm patches I think, so he wanted to process all the patches himself on a distinct branch to resolve the issues. Thanks.

Signature · 3 lines

Doug Ledford <[email protected]>
    GPG KeyID: B826A3330E572FDD
    Fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD

Jason Gunthorpe

2019-08-20 17:38 UTC #12 lore raw

Quoted 28 lines · Doug Ledford

Quoted 21 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
Hi,
This is a collection of new fixes for v5.3, everything here is fixing kernel crash or visible bug with one exception: patch #5 "IB/mlx5:
Consolidate use_umr checks into single function". That patch is nice to have improvement to implement rest of the series.
Thanks
Ido Kalir (1):
IB/core: Fix NULL pointer dereference when bind QP to counter
Jason Gunthorpe (1):
RDMA/mlx5: Fix MR npages calculation for IB_ACCESS_HUGETLB
Leon Romanovsky (2):
RDMA/counters: Properly implement PID checks RDMA/restrack: Rewrite PID namespace check to be reliable
Quoted 5 lines
Moni Shoua (4):
IB/mlx5: Consolidate use_umr checks into single function IB/mlx5: Report and handle ODP support properly IB/mlx5: Fix MR re-registration flow to use UMR properly IB/mlx5: Block MR WR if UMR is not possible

Quoted 7 lines

Hi Leon,
I took everything except Jason's patch to for-rc. He had tagged his patch in patchworks as under review by himself, mainly because there are some conflicts with other hmm patches I think, so he wanted to process all the patches himself on a distinct branch to resolve the issues. Thanks.

Ah, I thought that got grabbed alread

Can you take it but rebase it to be the first patch on top of v5.3-rc5 in the -rc branch?

Thanks,
Jason

Doug Ledford

2019-08-20 17:48 UTC #13 lore raw

Quoted 41 lines · Jason Gunthorpe

Quoted 32 lines · Doug Ledford
Quoted 23 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
Hi,
This is a collection of new fixes for v5.3, everything here is fixing
kernel crash or visible bug with one exception: patch #5 "IB/mlx5:
Consolidate use_umr checks into single function". That patch is nice
to have improvement to implement rest of the series.
Thanks
Ido Kalir (1):
IB/core: Fix NULL pointer dereference when bind QP to counter
Jason Gunthorpe (1):
RDMA/mlx5: Fix MR npages calculation for IB_ACCESS_HUGETLB
Leon Romanovsky (2):
RDMA/counters: Properly implement PID checks RDMA/restrack: Rewrite PID namespace check to be reliable
Quoted 5 lines
Moni Shoua (4):
IB/mlx5: Consolidate use_umr checks into single function IB/mlx5: Report and handle ODP support properly IB/mlx5: Fix MR re-registration flow to use UMR properly IB/mlx5: Block MR WR if UMR is not possible
Hi Leon,
Quoted 7 lines
I took everything except Jason's patch to for-rc. He had tagged his patch in patchworks as under review by himself, mainly because there are
some conflicts with other hmm patches I think, so he wanted to process
all the patches himself on a distinct branch to resolve the issues. Thanks.

Quoted 1 line

Ah, I thought that got grabbed alread

I assume you meant to when you bumped for-rc to -rc5? Evidently, it got missed ;-)

Quoted 2 lines

Can you take it but rebase it to be the first patch on top of v5.3-rc5 in the -rc branch?

Sure. You can get your branch head by digging this commit out from my wip/dl-for-rc, but just for ease, I also pushed this patch as the head out to the official for-rc branch.

Signature · 3 lines

Doug Ledford <[email protected]>
    GPG KeyID: B826A3330E572FDD
    Fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD