Skip to contentSkip to conversation

MR cache enhancement

PATCH In review Leon Romanovsky 4 participants 15 messages Dec 6, 2021 – Dec 9, 2021

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover MR cache enhancement

From: Leon Romanovsky <[email protected]>

Hi,

This series from Aharon refactors mlx5 MR cache management logic to speedup deregistration significantly.

Thanks

Aharon Landau (7):
RDMA/mlx5: Merge similar flows of allocating MR from the cache RDMA/mlx5: Replace cache list with Xarray
RDMA/mlx5: Store in the cache mkeys instead of mrs RDMA/mlx5: Change the cache structure to an RB-tree RDMA/mlx5: Reorder calls to pcie_relaxed_ordering_enabled() RDMA/mlx5: Delay the deregistration of a non-cache mkey RDMA/mlx5: Rename the mkey cache variables and functions

 drivers/infiniband/hw/mlx5/main.c    |   4 +-
 drivers/infiniband/hw/mlx5/mlx5_ib.h |  78 +--
 drivers/infiniband/hw/mlx5/mr.c      | 930 +++++++++++++++++----------
 drivers/infiniband/hw/mlx5/odp.c     |  72 ++-
 include/linux/mlx5/driver.h          |   7 +-
 5 files changed, 673 insertions(+), 418 deletions(-)
Signature · 1 line
2.33.1
#2 lore raw

PATCH 2/7 RDMA/mlx5: Replace cache list with Xarray

From: Aharon Landau <[email protected]>

The Xarray allows us to store the cached mkeys in memory efficient way and internal xa_lock is used to protect the indexes. It helps us to get rid of ent->lock as it is not required anymore.

Entries are reserved in the Xarray using xa_cmpxchg before calling to the upcoming callbacks to avoid allocations in interrupt context. The xa_cmpxchg can sleep when using GFP_KERNEL, so we call it in a loop to ensure one reserved entry for each process trying to reserve.

---
 drivers/infiniband/hw/mlx5/mlx5_ib.h |  11 +-
 drivers/infiniband/hw/mlx5/mr.c      | 206 +++++++++++++++------------
 2 files changed, 119 insertions(+), 98 deletions(-)
drivers/infiniband/hw/mlx5/mlx5_ib.h +3 -8
index 804748174752..d0224f468169 100644
@@ -754,11 +754,9 @@ struct umr_common {
 };
 
 struct mlx5_cache_ent {
-	struct list_head	head;
-	/* sync access to the cahce entry
-	 */
-	spinlock_t		lock;
-
+	struct xarray		mkeys;
+	unsigned long		stored;
+	unsigned long		reserved;
 
 	char                    name[4];
 	u32                     order;
@@ -770,8 +768,6 @@ struct mlx5_cache_ent {
 	u8 fill_to_high_water:1;
 
 	/*
-	 * - available_mrs is the length of list head, ie the number of MRs
-	 *   available for immediate allocation.
 	 * - total_mrs is available_mrs plus all in use MRs that could be
 	 *   returned to the cache.
 	 * - limit is the low water mark for available_mrs, 2* limit is the
@@ -779,7 +775,6 @@ struct mlx5_cache_ent {
 	 * - pending is the number of MRs currently being created
 	 */
 	u32 total_mrs;
-	u32 available_mrs;
 	u32 limit;
 	u32 pending;
 
drivers/infiniband/hw/mlx5/mr.c +116 -90

437 lines not on this page · raw message

Signature · 1 line
2.33.1
#7 lore raw

PATCH 3/7 RDMA/mlx5: Store in the cache mkeys instead of mrs

In reply to Leon Romanovsky

From: Aharon Landau <[email protected]>

Currently, the driver stores the entire mlx5_ib_mr struct in the cache, although the only use of the cached MR is the mkey. Store only the mkey in the cache.

---
 drivers/infiniband/hw/mlx5/mlx5_ib.h |  21 ++---
 drivers/infiniband/hw/mlx5/mr.c      | 135 +++++++++++++--------------
 2 files changed, 71 insertions(+), 85 deletions(-)
drivers/infiniband/hw/mlx5/mlx5_ib.h +7 -14
index d0224f468169..9b12e970ca01 100644
@@ -668,14 +668,6 @@ struct mlx5_ib_mr {
 
 	/* This is zero'd when the MR is allocated */
 	union {
-		/* Used only while the MR is in the cache */
-		struct {
-			u32 out[MLX5_ST_SZ_DW(create_mkey_out)];
-			struct mlx5_async_work cb_work;
-			/* Cache list element */
-			struct list_head list;
-		};
-
 		/* Used only by kernel MRs (umem == NULL) */
 		struct {
 			void *descs;
@@ -715,12 +707,6 @@ struct mlx5_ib_mr {
 	};
 };
 
-/* Zero the fields in the mr that are variant depending on usage */
-static inline void mlx5_clear_mr(struct mlx5_ib_mr *mr)
-{
-	memset(mr->out, 0, sizeof(*mr) - offsetof(struct mlx5_ib_mr, out));
-}
-
 static inline bool is_odp_mr(struct mlx5_ib_mr *mr)
 {
 	return IS_ENABLED(CONFIG_INFINIBAND_ON_DEMAND_PAGING) && mr->umem &&
@@ -786,6 +772,13 @@ struct mlx5_cache_ent {
 	struct delayed_work	dwork;
 };
 
+struct mlx5_async_create_mkey {
+	u32 out[MLX5_ST_SZ_DW(create_mkey_out)];
+	struct mlx5_async_work cb_work;
+	struct mlx5_cache_ent *ent;
+	u32 mkey;
+};
+
 struct mlx5_mr_cache {
 	struct workqueue_struct *wq;
 	struct mlx5_cache_ent	ent[MAX_MR_CACHE_ENTRIES];
drivers/infiniband/hw/mlx5/mr.c +64 -71

317 lines not on this page · raw message

Signature · 1 line
2.33.1
#9 lore raw

PATCH 4/7 RDMA/mlx5: Change the cache structure to an RB-tree

In reply to Leon Romanovsky

From: Aharon Landau <[email protected]>

Currently, the cache structure is a linear array held within mlx5_ib_dev. Therefore, limits to the number of entries.

The existing entries are dedicated to mkeys of size 2^x and with no access_flags and later in the series, we allow caching mkeys with different attributes.

In this patch, we change the cache structure to an RB-tree of Xarray of mkeys. The tree key is the mkc used to create the stored mkeys.

---
 drivers/infiniband/hw/mlx5/mlx5_ib.h |  28 +-
 drivers/infiniband/hw/mlx5/mr.c      | 435 +++++++++++++++++----------
 drivers/infiniband/hw/mlx5/odp.c     |  71 +++--
 include/linux/mlx5/driver.h          |   5 +-
 4 files changed, 340 insertions(+), 199 deletions(-)
drivers/infiniband/hw/mlx5/mlx5_ib.h +18 -10
index 9b12e970ca01..202d8fbc423d 100644
@@ -745,10 +745,7 @@ struct mlx5_cache_ent {
 	unsigned long		reserved;
 
 	char                    name[4];
-	u32                     order;
-	u32			xlt;
-	u32			access_mode;
-	u32			page;
+	int			ndescs;
 
 	u8 disabled:1;
 	u8 fill_to_high_water:1;
@@ -770,6 +767,9 @@ struct mlx5_cache_ent {
 	struct mlx5_ib_dev     *dev;
 	struct work_struct	work;
 	struct delayed_work	dwork;
+
+	struct rb_node		node;
+	void			*mkc;
 };
 
 struct mlx5_async_create_mkey {
@@ -781,9 +781,11 @@ struct mlx5_async_create_mkey {
 
 struct mlx5_mr_cache {
 	struct workqueue_struct *wq;
-	struct mlx5_cache_ent	ent[MAX_MR_CACHE_ENTRIES];
+	struct rb_root		cache_root;
+	struct mutex		cache_lock;
 	struct dentry		*root;
 	unsigned long		last_add;
+	bool			maintained_cache;
 };
 
 struct mlx5_ib_port_resources {
@@ -1330,9 +1332,12 @@ int mlx5_ib_get_cqe_size(struct ib_cq *ibcq);
 int mlx5_mr_cache_init(struct mlx5_ib_dev *dev);
 int mlx5_mr_cache_cleanup(struct mlx5_ib_dev *dev);
 
-struct mlx5_ib_mr *mlx5_mr_cache_alloc(struct mlx5_ib_dev *dev,
-				       struct mlx5_cache_ent *ent,
-				       int access_flags);
+int mlx5_acc_flags_to_ent_flags(struct mlx5_ib_dev *dev, int access_flags);
+void mlx5_set_cache_mkc(struct mlx5_ib_dev *dev, void *mkc, int access_flags,
+			unsigned int access_mode, unsigned int page_shift);
+struct mlx5_ib_mr *mlx5_mr_cache_alloc(struct mlx5_ib_dev *dev, int *in,
+				       int inlen, unsigned int ndescs,
+				       unsigned int access_mode, bool force);
 
 int mlx5_ib_check_mr_status(struct ib_mr *ibmr, u32 check_mask,
 			    struct ib_mr_status *mr_status);
@@ -1356,7 +1361,7 @@ int mlx5r_odp_create_eq(struct mlx5_ib_dev *dev, struct mlx5_ib_pf_eq *eq);
 void mlx5_ib_odp_cleanup_one(struct mlx5_ib_dev *ibdev);
 int __init mlx5_ib_odp_init(void);
 void mlx5_ib_odp_cleanup(void);
-void mlx5_odp_init_mr_cache_entry(struct mlx5_cache_ent *ent);
+int mlx5_odp_init_mr_cache_entry(struct mlx5_ib_dev *dev);
 void mlx5_odp_populate_xlt(void *xlt, size_t idx, size_t nentries,
 			   struct mlx5_ib_mr *mr, int flags);
 
@@ -1375,7 +1380,10 @@ static inline int mlx5r_odp_create_eq(struct mlx5_ib_dev *dev,
 static inline void mlx5_ib_odp_cleanup_one(struct mlx5_ib_dev *ibdev) {}
 static inline int mlx5_ib_odp_init(void) { return 0; }
 static inline void mlx5_ib_odp_cleanup(void)				    {}
-static inline void mlx5_odp_init_mr_cache_entry(struct mlx5_cache_ent *ent) {}
+static inline int mlx5_odp_init_mr_cache_entry(struct mlx5_ib_dev *dev)
+{
+	return 0;
+}
 static inline void mlx5_odp_populate_xlt(void *xlt, size_t idx, size_t nentries,
 					 struct mlx5_ib_mr *mr, int flags) {}
 
drivers/infiniband/hw/mlx5/mr.c +281 -154

606 lines not on this page · raw message

drivers/infiniband/hw/mlx5/odp.c +40 -31

127 lines not on this page · raw message

include/linux/mlx5/driver.h +1 -4
index a623ec635947..c33f71134136 100644
@@ -699,10 +699,7 @@ enum {
 };
 
 enum {
-	MR_CACHE_LAST_STD_ENTRY = 20,
-	MLX5_IMR_MTT_CACHE_ENTRY,
-	MLX5_IMR_KSM_CACHE_ENTRY,
-	MAX_MR_CACHE_ENTRIES
+	MAX_MR_CACHE_ENTRIES = 21,
 };
 
 struct mlx5_profile {
Signature · 1 line
2.33.1
#1 lore raw

PATCH 1/7 RDMA/mlx5: Merge similar flows of allocating MR from the cache

In reply to Leon Romanovsky

From: Aharon Landau <[email protected]>

When allocating an MR from the cache, the driver calls to get_cache_mr(), and in case of failure, retries with create_cache_mr(). This is the flow of mlx5_mr_cache_alloc(), so use it instead.

---
 drivers/infiniband/hw/mlx5/mlx5_ib.h |  3 +-
 drivers/infiniband/hw/mlx5/mr.c      | 51 +++++-----------------------
 drivers/infiniband/hw/mlx5/odp.c     | 11 ++++--
 3 files changed, 19 insertions(+), 46 deletions(-)
drivers/infiniband/hw/mlx5/mlx5_ib.h +2 -1
index e636e954f6bf..804748174752 100644
@@ -1343,7 +1343,8 @@ int mlx5_mr_cache_init(struct mlx5_ib_dev *dev);
 int mlx5_mr_cache_cleanup(struct mlx5_ib_dev *dev);
 
 struct mlx5_ib_mr *mlx5_mr_cache_alloc(struct mlx5_ib_dev *dev,
-				       unsigned int entry, int access_flags);
+				       struct mlx5_cache_ent *ent,
+				       int access_flags);
 
 int mlx5_ib_check_mr_status(struct ib_mr *ibmr, u32 check_mask,
 			    struct ib_mr_status *mr_status);
drivers/infiniband/hw/mlx5/mr.c +9 -42

82 lines not on this page · raw message

drivers/infiniband/hw/mlx5/odp.c +8 -3
index 91eb615b89ee..0972afc3e952 100644
@@ -407,6 +407,7 @@ static void mlx5_ib_page_fault_resume(struct mlx5_ib_dev *dev,
 static struct mlx5_ib_mr *implicit_get_child_mr(struct mlx5_ib_mr *imr,
 						unsigned long idx)
 {
+	struct mlx5_ib_dev *dev = mr_to_mdev(imr);
 	struct ib_umem_odp *odp;
 	struct mlx5_ib_mr *mr;
 	struct mlx5_ib_mr *ret;
@@ -418,13 +419,14 @@ static struct mlx5_ib_mr *implicit_get_child_mr(struct mlx5_ib_mr *imr,
 	if (IS_ERR(odp))
 		return ERR_CAST(odp);
 
-	mr = mlx5_mr_cache_alloc(
-		mr_to_mdev(imr), MLX5_IMR_MTT_CACHE_ENTRY, imr->access_flags);
+	mr = mlx5_mr_cache_alloc(dev, &dev->cache.ent[MLX5_IMR_MTT_CACHE_ENTRY],
+				 imr->access_flags);
 	if (IS_ERR(mr)) {
 		ib_umem_odp_release(odp);
 		return mr;
 	}
 
+	mr->access_flags = imr->access_flags;
 	mr->ibmr.pd = imr->ibmr.pd;
 	mr->ibmr.device = &mr_to_mdev(imr)->ib_dev;
 	mr->umem = &odp->umem;
@@ -493,12 +495,15 @@ struct mlx5_ib_mr *mlx5_ib_alloc_implicit_mr(struct mlx5_ib_pd *pd,
 	if (IS_ERR(umem_odp))
 		return ERR_CAST(umem_odp);
 
-	imr = mlx5_mr_cache_alloc(dev, MLX5_IMR_KSM_CACHE_ENTRY, access_flags);
+	imr = mlx5_mr_cache_alloc(dev,
+				  &dev->cache.ent[MLX5_IMR_KSM_CACHE_ENTRY],
+				  access_flags);
 	if (IS_ERR(imr)) {
 		ib_umem_odp_release(umem_odp);
 		return imr;
 	}
 
+	imr->access_flags = access_flags;
 	imr->ibmr.pd = &pd->ibpd;
 	imr->ibmr.iova = 0;
 	imr->umem = &umem_odp->umem;
Signature · 1 line
2.33.1
#12 lore raw

PATCH 6/7 RDMA/mlx5: Delay the deregistration of a non-cache mkey

In reply to Leon Romanovsky

From: Aharon Landau <[email protected]>

When restarting an application with many non-cached mkeys, all the mkeys will be destroyed and then recreated.

This process takes a long time (about 20 seconds for deregistration and 28 seconds for registration of 100,000 MRs).

To shorten the restart runtime, insert the mkeys temporarily into the cache and schedule a delayed work to destroy them later. If there is no fitting entry to these mkeys, create a temporary entry that fits them.

If 30 seconds have passed and no user reclaimed the temporarily cached mkeys, the scheduled work will destroy the mkeys and the temporary entries.

When restarting an application, the mkeys will still be in the cache when trying to reg them again, therefore, the registration will be faster (4 seconds for deregistration and 5 seconds or registration of 100,000 MRs).

---
 drivers/infiniband/hw/mlx5/mlx5_ib.h |   3 +
 drivers/infiniband/hw/mlx5/mr.c      | 157 ++++++++++++++++++++++++---
 2 files changed, 146 insertions(+), 14 deletions(-)
drivers/infiniband/hw/mlx5/mlx5_ib.h +3 -0
index 202d8fbc423d..7dcc9e69a649 100644
@@ -638,6 +638,7 @@ struct mlx5_ib_mkey {
 	u32 key;
 	enum mlx5_mkey_type type;
 	unsigned int ndescs;
+	unsigned int access_mode;
 	struct wait_queue_head wait;
 	refcount_t usecount;
 };
@@ -747,6 +748,7 @@ struct mlx5_cache_ent {
 	char                    name[4];
 	int			ndescs;
 
+	u8 is_tmp:1;
 	u8 disabled:1;
 	u8 fill_to_high_water:1;
 
@@ -786,6 +788,7 @@ struct mlx5_mr_cache {
 	struct dentry		*root;
 	unsigned long		last_add;
 	bool			maintained_cache;
+	struct delayed_work	remove_ent_dwork;
 };
 
 struct mlx5_ib_port_resources {
drivers/infiniband/hw/mlx5/mr.c +143 -14

229 lines not on this page · raw message

Signature · 1 line
2.33.1
#13 lore raw

PATCH 7/7 RDMA/mlx5: Rename the mkey cache variables and functions

In reply to Leon Romanovsky

From: Aharon Landau <[email protected]>

After replacing the MR cache with an Mkey cache, rename the variables and functions to fit the new meaning.

---
 drivers/infiniband/hw/mlx5/main.c    |  4 +-
 drivers/infiniband/hw/mlx5/mlx5_ib.h | 26 ++++----
 drivers/infiniband/hw/mlx5/mr.c      | 88 ++++++++++++++--------------
 drivers/infiniband/hw/mlx5/odp.c     |  8 +--
 include/linux/mlx5/driver.h          |  4 +-
 5 files changed, 66 insertions(+), 64 deletions(-)
drivers/infiniband/hw/mlx5/main.c +2 -2
index 5ec8bd2f0b2f..74f32b563109 100644
@@ -4034,7 +4034,7 @@ static void mlx5_ib_stage_pre_ib_reg_umr_cleanup(struct mlx5_ib_dev *dev)
 {
 	int err;
 
-	err = mlx5_mr_cache_cleanup(dev);
+	err = mlx5_mkey_cache_cleanup(dev);
 	if (err)
 		mlx5_ib_warn(dev, "mr cache cleanup failed\n");
 
@@ -4131,7 +4131,7 @@ static int mlx5_ib_stage_post_ib_reg_umr_init(struct mlx5_ib_dev *dev)
 	dev->umrc.pd = pd;
 
 	sema_init(&dev->umrc.sem, MAX_UMR_WR);
-	ret = mlx5_mr_cache_init(dev);
+	ret = mlx5_mkey_cache_init(dev);
 	if (ret) {
 		mlx5_ib_warn(dev, "mr cache init failed %d\n", ret);
 		goto error_4;
drivers/infiniband/hw/mlx5/mlx5_ib.h +13 -13
index 7dcc9e69a649..54a5c4bc2919 100644
@@ -753,13 +753,13 @@ struct mlx5_cache_ent {
 	u8 fill_to_high_water:1;
 
 	/*
-	 * - total_mrs is available_mrs plus all in use MRs that could be
+	 * - total_mkeys is stored mkeys plus all in use mkeys that could be
 	 *   returned to the cache.
-	 * - limit is the low water mark for available_mrs, 2* limit is the
+	 * - limit is the low water mark for available_mkeys, 2 * limit is the
 	 *   upper water mark.
-	 * - pending is the number of MRs currently being created
+	 * - pending is the number of mkeys currently being created
 	 */
-	u32 total_mrs;
+	u32 total_mkeys;
 	u32 limit;
 	u32 pending;
 
@@ -781,7 +781,7 @@ struct mlx5_async_create_mkey {
 	u32 mkey;
 };
 
-struct mlx5_mr_cache {
+struct mlx5_mkey_cache {
 	struct workqueue_struct *wq;
 	struct rb_root		cache_root;
 	struct mutex		cache_lock;
@@ -1085,7 +1085,7 @@ struct mlx5_ib_dev {
 	struct mlx5_ib_resources	devr;
 
 	atomic_t			mkey_var;
-	struct mlx5_mr_cache		cache;
+	struct mlx5_mkey_cache		cache;
 	struct timer_list		delay_timer;
 	/* Prevents soft lock on massive reg MRs */
 	struct mutex			slow_path_mutex;
@@ -1332,15 +1332,15 @@ void mlx5_ib_populate_pas(struct ib_umem *umem, size_t page_size, __be64 *pas,
 			  u64 access_flags);
 void mlx5_ib_copy_pas(u64 *old, u64 *new, int step, int num);
 int mlx5_ib_get_cqe_size(struct ib_cq *ibcq);
-int mlx5_mr_cache_init(struct mlx5_ib_dev *dev);
-int mlx5_mr_cache_cleanup(struct mlx5_ib_dev *dev);
+int mlx5_mkey_cache_init(struct mlx5_ib_dev *dev);
+int mlx5_mkey_cache_cleanup(struct mlx5_ib_dev *dev);
 
 int mlx5_acc_flags_to_ent_flags(struct mlx5_ib_dev *dev, int access_flags);
 void mlx5_set_cache_mkc(struct mlx5_ib_dev *dev, void *mkc, int access_flags,
 			unsigned int access_mode, unsigned int page_shift);
-struct mlx5_ib_mr *mlx5_mr_cache_alloc(struct mlx5_ib_dev *dev, int *in,
-				       int inlen, unsigned int ndescs,
-				       unsigned int access_mode, bool force);
+struct mlx5_ib_mr *mlx5_mkey_cache_alloc(struct mlx5_ib_dev *dev, int *in,
+					 int inlen, unsigned int ndescs,
+					 unsigned int access_mode, bool force);
 
 int mlx5_ib_check_mr_status(struct ib_mr *ibmr, u32 check_mask,
 			    struct ib_mr_status *mr_status);
@@ -1364,7 +1364,7 @@ int mlx5r_odp_create_eq(struct mlx5_ib_dev *dev, struct mlx5_ib_pf_eq *eq);
 void mlx5_ib_odp_cleanup_one(struct mlx5_ib_dev *ibdev);
 int __init mlx5_ib_odp_init(void);
 void mlx5_ib_odp_cleanup(void);
-int mlx5_odp_init_mr_cache_entry(struct mlx5_ib_dev *dev);
+int mlx5_odp_init_mkey_cache_entry(struct mlx5_ib_dev *dev);
 void mlx5_odp_populate_xlt(void *xlt, size_t idx, size_t nentries,
 			   struct mlx5_ib_mr *mr, int flags);
 
@@ -1383,7 +1383,7 @@ static inline int mlx5r_odp_create_eq(struct mlx5_ib_dev *dev,
 static inline void mlx5_ib_odp_cleanup_one(struct mlx5_ib_dev *ibdev) {}
 static inline int mlx5_ib_odp_init(void) { return 0; }
 static inline void mlx5_ib_odp_cleanup(void)				    {}
-static inline int mlx5_odp_init_mr_cache_entry(struct mlx5_ib_dev *dev)
+static inline int mlx5_odp_init_mkey_cache_entry(struct mlx5_ib_dev *dev)
 {
 	return 0;
 }
drivers/infiniband/hw/mlx5/mr.c +45 -43

292 lines not on this page · raw message

drivers/infiniband/hw/mlx5/odp.c +4 -4
index 3d86a448ec97..25328abaedc9 100644
@@ -434,8 +434,8 @@ static struct mlx5_ib_mr *implicit_get_child_mr(struct mlx5_ib_mr *imr,
 			   mlx5_acc_flags_to_ent_flags(dev, imr->access_flags),
 			   MLX5_MKC_ACCESS_MODE_MTT, PAGE_SHIFT);
 
-	mr = mlx5_mr_cache_alloc(dev, in, inlen, MLX5_IMR_MTT_ENTRIES,
-				 MLX5_MKC_ACCESS_MODE_MTT, true);
+	mr = mlx5_mkey_cache_alloc(dev, in, inlen, MLX5_IMR_MTT_ENTRIES,
+				   MLX5_MKC_ACCESS_MODE_MTT, true);
 	if (IS_ERR(mr)) {
 		ib_umem_odp_release(odp);
 		kfree(in);
@@ -528,8 +528,8 @@ struct mlx5_ib_mr *mlx5_ib_alloc_implicit_mr(struct mlx5_ib_pd *pd,
 			   mlx5_acc_flags_to_ent_flags(dev, access_flags),
 			   MLX5_MKC_ACCESS_MODE_KSM, PAGE_SHIFT);
 
-	imr = mlx5_mr_cache_alloc(dev, in, inlen, mlx5_imr_ksm_entries,
-				  MLX5_MKC_ACCESS_MODE_KSM, true);
+	imr = mlx5_mkey_cache_alloc(dev, in, inlen, mlx5_imr_ksm_entries,
+				    MLX5_MKC_ACCESS_MODE_KSM, true);
 	if (IS_ERR(imr)) {
 		ib_umem_odp_release(umem_odp);
 		kfree(in);
include/linux/mlx5/driver.h +2 -2
index c33f71134136..51b30c11116e 100644
@@ -699,7 +699,7 @@ enum {
 };
 
 enum {
-	MAX_MR_CACHE_ENTRIES = 21,
+	MAX_MKEY_CACHE_ENTRIES = 21,
 };
 
 struct mlx5_profile {
@@ -708,7 +708,7 @@ struct mlx5_profile {
 	struct {
 		int	size;
 		int	limit;
-	} mr_cache[MAX_MR_CACHE_ENTRIES];
+	} mr_cache[MAX_MKEY_CACHE_ENTRIES];
 };
 
 struct mlx5_hca_cap {
Signature · 1 line
2.33.1
#11 lore raw

PATCH 5/7 RDMA/mlx5: Reorder calls to pcie_relaxed_ordering_enabled()

In reply to Leon Romanovsky

From: Aharon Landau <[email protected]>

The mkc is the key for the mkey cache, hence, created in each attempt to get a cache mkey, while pcie_relaxed_ordering_enabled() is called during the setting of the mkc, but used only for cases where IB_ACCESS_RELAXED_ORDERING is set.

pcie_relaxed_ordering_enabled() is an expensive call (26 us). Reorder the code so the driver will call it only when it is needed.

---
 drivers/infiniband/hw/mlx5/mr.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
drivers/infiniband/hw/mlx5/mr.c +7 -7
index 6000acbedc73..ca6faf599cd3 100644
@@ -68,7 +68,6 @@ static void set_mkc_access_pd_addr_fields(void *mkc, int acc, u64 start_addr,
 					  struct ib_pd *pd)
 {
 	struct mlx5_ib_dev *dev = to_mdev(pd->device);
-	bool ro_pci_enabled = pcie_relaxed_ordering_enabled(dev->mdev->pdev);
 
 	MLX5_SET(mkc, mkc, a, !!(acc & IB_ACCESS_REMOTE_ATOMIC));
 	MLX5_SET(mkc, mkc, rw, !!(acc & IB_ACCESS_REMOTE_WRITE));
@@ -76,12 +75,13 @@ static void set_mkc_access_pd_addr_fields(void *mkc, int acc, u64 start_addr,
 	MLX5_SET(mkc, mkc, lw, !!(acc & IB_ACCESS_LOCAL_WRITE));
 	MLX5_SET(mkc, mkc, lr, 1);
 
-	if (MLX5_CAP_GEN(dev->mdev, relaxed_ordering_write))
-		MLX5_SET(mkc, mkc, relaxed_ordering_write,
-			 (acc & IB_ACCESS_RELAXED_ORDERING) && ro_pci_enabled);
-	if (MLX5_CAP_GEN(dev->mdev, relaxed_ordering_read))
-		MLX5_SET(mkc, mkc, relaxed_ordering_read,
-			 (acc & IB_ACCESS_RELAXED_ORDERING) && ro_pci_enabled);
+	if ((acc & IB_ACCESS_RELAXED_ORDERING) &&
+	    pcie_relaxed_ordering_enabled(dev->mdev->pdev)) {
+		if (MLX5_CAP_GEN(dev->mdev, relaxed_ordering_write))
+			MLX5_SET(mkc, mkc, relaxed_ordering_write, 1);
+		if (MLX5_CAP_GEN(dev->mdev, relaxed_ordering_read))
+			MLX5_SET(mkc, mkc, relaxed_ordering_read, 1);
+	}
 
 	MLX5_SET(mkc, mkc, pd, to_mpd(pd)->pdn);
 	MLX5_SET(mkc, mkc, qpn, 0xffffff);
Signature · 1 line
2.33.1
maintainer
#3 lore raw

Re: [PATCH rdma-next 2/7] RDMA/mlx5: Replace cache list with Xarray

In reply to Leon Romanovsky
Quoted 25 lines
drivers/infiniband/hw/mlx5/mr.c +6 -5
@@ -166,14 +169,14 @@ static void create_mkey_callback(int status, struct mlx5_async_work *context)
 
 	WRITE_ONCE(dev->cache.last_add, jiffies);
 
-	spin_lock_irqsave(&ent->lock, flags);
-	list_add_tail(&mr->list, &ent->head);
-	ent->available_mrs++;
+	xa_lock_irqsave(&ent->mkeys, flags);
+	xa_ent = __xa_store(&ent->mkeys, ent->stored++, mr, GFP_ATOMIC);
+	WARN_ON(xa_ent != NULL);
+	ent->pending--;
 	ent->total_mrs++;
 	/* If we are doing fill_to_high_water then keep going. */
 	queue_adjust_cache_locked(ent);
-	ent->pending--;
-	spin_unlock_irqrestore(&ent->lock, flags);
+	xa_unlock_irqrestore(&ent->mkeys, flags);
 }
 
 static struct mlx5_ib_mr *alloc_cache_mr(struct mlx5_cache_ent *ent, void *mkc)
@@ -196,6 +199,25 @@ static struct mlx5_ib_mr *alloc_cache_mr(struct mlx5_cache_ent *ent, void *mkc)
 	return mr;
 }
 
+static int _push_reserve_mkey(struct mlx5_cache_ent *ent)

drivers/infiniband/hw/mlx5/mr.c static struct mlx5_ib_mr *alloc_cache_mr(struct mlx5_cache_ent *ent, void *mkc) · line 212

Quoted 10 lines
drivers/infiniband/hw/mlx5/mr.c +10 -0
+{
+	unsigned long to_reserve;
+	int rc;
+
+	while (true) {
+		to_reserve = ent->reserved;
+		rc = xa_err(__xa_cmpxchg(&ent->mkeys, to_reserve, NULL,
+					 XA_ZERO_ENTRY, GFP_KERNEL));
+		if (rc)
+			return rc;

What about when old != NULL ?

drivers/infiniband/hw/mlx5/mr.c static struct mlx5_ib_mr *alloc_cache_mr(struct mlx5_cache_ent *ent, void *mkc) · line 214

Quoted 2 lines
drivers/infiniband/hw/mlx5/mr.c +2 -0
+		if (to_reserve != ent->reserved)
+			continue;

There is an edge case where where reserved could have shrunk alot while the lock was released, and xa_cmpxchg could succeed. The above if will protect things, however a ZERO_ENTRY will have been written to some weird place in the XA. It needs a

if (old == NULL) // ie we stored something someplace weird

    __xa_erase(&ent->mkeys, to_reserve)
Quoted 14 lines
drivers/infiniband/hw/mlx5/mr.c +6 -1
+		ent->reserved++;
+		break;
+	}
+	return 0;
+}
+
 /* Asynchronously schedule new MRs to be populated in the cache. */
 static int add_keys(struct mlx5_cache_ent *ent, unsigned int num)
 {
@@ -217,23 +239,32 @@ static int add_keys(struct mlx5_cache_ent *ent, unsigned int num)
 			err = -ENOMEM;
 			break;
 		}
-		spin_lock_irq(&ent->lock);

drivers/infiniband/hw/mlx5/mr.c static int add_keys(struct mlx5_cache_ent *ent, unsigned int num) · line 250

Quoted 10 lines
drivers/infiniband/hw/mlx5/mr.c +7 -1
+		xa_lock_irq(&ent->mkeys);
 		if (ent->pending >= MAX_PENDING_REG_MR) {
+			xa_unlock_irq(&ent->mkeys);
 			err = -EAGAIN;
-			spin_unlock_irq(&ent->lock);
+			kfree(mr);
+			break;
+		}
+
+		err = _push_reserve_mkey(ent);

The test of ent->pending is out of date now since this can drop the lock

It feels like pending and (reserved - stored) are really the same thing, so maybe just directly limit the number of reserved and test it after

Quoted 4 lines
drivers/infiniband/hw/mlx5/mr.c +0 -0
@@ -287,39 +318,37 @@ static void remove_cache_mr_locked(struct mlx5_cache_ent *ent)
 {
 	struct mlx5_ib_mr *mr;
 

drivers/infiniband/hw/mlx5/mr.c static void remove_cache_mr_locked(struct mlx5_cache_ent *ent) · line 325

Quoted 10 lines
drivers/infiniband/hw/mlx5/mr.c +3 -5
-	lockdep_assert_held(&ent->lock);
-	if (list_empty(&ent->head))
+	if (!ent->stored)
 		return;
-	mr = list_first_entry(&ent->head, struct mlx5_ib_mr, list);
-	list_del(&mr->list);
-	ent->available_mrs--;
 
+	mr = __xa_store(&ent->mkeys, --ent->stored, NULL, GFP_KERNEL);
+	WARN_ON(mr == NULL || xa_is_err(mr));

Add a if (reserved != stored) before the below?

Review static void remove_cache_mr_locked(struct mlx5_cache_ent *ent) · line 326

Quoted 1 line
(quoted diff) +1 -0
+	WARN_ON(__xa_erase(&ent->mkeys, --ent->reserved) != NULL);

Also please avoid writing WARN_ON(something with side effects)

  old = __xa_erase(&ent->mkeys, --ent->reserved);
  WARN_ON(old != NULL);

Same for all places

drivers/infiniband/hw/mlx5/mr.c static void remove_cache_mr_locked(struct mlx5_cache_ent *ent) · line 335

Quoted 3 lines
drivers/infiniband/hw/mlx5/mr.c +1 -0
 static int resize_available_mrs(struct mlx5_cache_ent *ent, unsigned int target,
 				bool limit_fill)
+	 __acquires(&ent->lock) __releases(&ent->lock)

Why?

drivers/infiniband/hw/mlx5/mr.c static void remove_cache_mr_locked(struct mlx5_cache_ent *ent) · line 309

Quoted 5 lines
drivers/infiniband/hw/mlx5/mr.c +0 -2
 {
 	int err;
 
-	lockdep_assert_held(&ent->lock);
-

Why?

Quoted 10 lines
drivers/infiniband/hw/mlx5/mr.c +1 -2
 static void clean_keys(struct mlx5_ib_dev *dev, int c)
 {
 	struct mlx5_mr_cache *cache = &dev->cache;
 	struct mlx5_cache_ent *ent = &cache->ent[c];
-	struct mlx5_ib_mr *tmp_mr;
 	struct mlx5_ib_mr *mr;
-	LIST_HEAD(del_list);
+	unsigned long index;
 
 	cancel_delayed_work(&ent->dwork);

drivers/infiniband/hw/mlx5/mr.c static void mlx5_mr_cache_free(struct mlx5_ib_dev *dev, struct mlx5_ib_mr *mr) · line 647

Quoted 10 lines
drivers/infiniband/hw/mlx5/mr.c +1 -9
-	while (1) {
-		spin_lock_irq(&ent->lock);
-		if (list_empty(&ent->head)) {
-			spin_unlock_irq(&ent->lock);
-			break;
-		}
-		mr = list_first_entry(&ent->head, struct mlx5_ib_mr, list);
-		list_move(&mr->list, &del_list);
-		ent->available_mrs--;
+	xa_for_each(&ent->mkeys, index, mr) {

This isn't quite the same thing, the above tolerates concurrent add, this does not.

It should be more like

while (ent->stored) {
mr = __xa_erase(stored--);

Quoted 4 lines
drivers/infiniband/hw/mlx5/mr.c +0 -0
@@ -1886,6 +1901,17 @@ mlx5_free_priv_descs(struct mlx5_ib_mr *mr)
 	}
 }
 

drivers/infiniband/hw/mlx5/mr.c mlx5_free_priv_descs(struct mlx5_ib_mr *mr) · line 1913

Quoted 10 lines
drivers/infiniband/hw/mlx5/mr.c +10 -0
+static int push_reserve_mkey(struct mlx5_cache_ent *ent)
+{
+	int ret;
+
+	xa_lock_irq(&ent->mkeys);
+	ret = _push_reserve_mkey(ent);
+	xa_unlock_irq(&ent->mkeys);
+
+	return ret;
+}

Put this close to _push_reserve_mkey() please

Jason

maintainer
#8 lore raw

Re: [PATCH rdma-next 3/7] RDMA/mlx5: Store in the cache mkeys instead of mrs

In reply to Leon Romanovsky

drivers/infiniband/hw/mlx5/mlx5_ib.h struct mlx5_ib_mr {

Quoted 16 lines · Leon Romanovsky

From: Aharon Landau <[email protected]>

Currently, the driver stores the entire mlx5_ib_mr struct in the cache, although the only use of the cached MR is the mkey. Store only the mkey in the cache.

 drivers/infiniband/hw/mlx5/mlx5_ib.h |  21 ++---
 drivers/infiniband/hw/mlx5/mr.c      | 135 +++++++++++++--------------
 2 files changed, 71 insertions(+), 85 deletions(-)
drivers/infiniband/hw/mlx5/mlx5_ib.h +1 -0
index d0224f468169..9b12e970ca01 100644
+++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h
@@ -668,14 +668,6 @@ struct mlx5_ib_mr {

Shouldn't this delete cache_ent too?

Jason

maintainer
#10 lore raw

Re: [PATCH rdma-next 4/7] RDMA/mlx5: Change the cache structure to an RB-tree

In reply to Leon Romanovsky
Quoted 1 line · Leon Romanovsky

+static struct mlx5_cache_ent *ent_search(struct mlx5_mr_cache *cache, void *mkc)

mlx5_cache_ent_find() ?

This search isn't really what I would expect, it should stop if it finds a (somewhat) larger and otherwise compatible entry instead of continuing to bin on power of two sizes.

drivers/infiniband/hw/mlx5/mr.c static void cache_work_func(struct work_struct *work) · line 559

Quoted 2 lines
drivers/infiniband/hw/mlx5/mr.c +2 -0
+	struct rb_node *node = cache->cache_root.rb_node;
+	int size = MLX5_ST_SZ_BYTES(mkc);

size_t not int

drivers/infiniband/hw/mlx5/mr.c static void cache_work_func(struct work_struct *work) · line 607

Quoted 3 lines
drivers/infiniband/hw/mlx5/mr.c +3 -0
+	mkc = MLX5_ADDR_OF(create_mkey_in, in, memory_key_mkey_entry);
+	if (dev->cache.maintained_cache && !force) {
+		int order;

unsigned int

drivers/infiniband/hw/mlx5/mr.c static void cache_work_func(struct work_struct *work) · line 617

Quoted 9 lines
drivers/infiniband/hw/mlx5/mr.c +9 -0
+		/*
+		 * Try to get an mkey from pool.
+		 */
+		order = order_base_2(ndescs) > 2 ? order_base_2(ndescs) : 2;
+		MLX5_SET(mkc, mkc, translations_octword_size,
+			 get_mkc_octo_size(access_mode, 1 << order));
+		mutex_lock(&dev->cache.cache_lock);
+		ent = ent_search(&dev->cache, mkc);
+		mutex_unlock(&dev->cache.cache_lock);

Why is it OK to drop the lock here? What is protecting the ent pointer against free?

drivers/infiniband/hw/mlx5/mr.c static void cache_work_func(struct work_struct *work) · line 620

Quoted 1 line
drivers/infiniband/hw/mlx5/mr.c +1 -0
+	if (ent && (ent->limit || force)) {

What locking protects ent->limit ?

drivers/infiniband/hw/mlx5/mr.c static void cache_work_func(struct work_struct *work) · line 631

Quoted 11 lines
drivers/infiniband/hw/mlx5/mr.c +11 -0
+		xa_lock_irq(&ent->mkeys);
+		if (!ent->stored) {
+			if (ent->limit) {
+				queue_adjust_cache_locked(ent);
+				ent->miss++;
+			}
+			xa_unlock_irq(&ent->mkeys);
+
+			err = mlx5_ib_create_mkey(dev, &mr->mmkey, in, inlen);
+			if (err)
+				goto err;

So if there is no entry we create a bigger one rounded up to pow2?

drivers/infiniband/hw/mlx5/mr.c static void cache_work_func(struct work_struct *work) · line 636

Quoted 5 lines
drivers/infiniband/hw/mlx5/mr.c +5 -0
+
+			WRITE_ONCE(ent->dev->cache.last_add, jiffies);
+			xa_lock_irq(&ent->mkeys);
+			ent->total_mrs++;
+			xa_unlock_irq(&ent->mkeys);

May as well optimize for the normal case, do the total_mrs++ before create_mkey while we still have the lock and undo it if it fails. It is just minor stat reporting right?

drivers/infiniband/hw/mlx5/mr.c static void cache_work_func(struct work_struct *work) · line 642

Quoted 6 lines
drivers/infiniband/hw/mlx5/mr.c +6 -0
+		} else {
+			xa_ent = __xa_store(&ent->mkeys, --ent->stored,
+					  NULL, GFP_KERNEL);
+			WARN_ON(xa_ent == NULL || xa_is_err(xa_ent));
+			WARN_ON(__xa_erase(&ent->mkeys, --ent->reserved) !=
+				NULL);

This whole bigger block want to be in its own function 'mlx5_ent_get_mkey()'

Then you can write it simpler without all the duplicated error handling and deep indenting

Quoted 21 lines
drivers/infiniband/hw/mlx5/mr.c +3 -15
 			queue_adjust_cache_locked(ent);
-			ent->miss++;
-		}
-		xa_unlock_irq(&ent->mkeys);
-		err = create_cache_mkey(ent, &mr->mmkey.key);
-		if (err) {
-			kfree(mr);
-			return ERR_PTR(err);
+			xa_unlock_irq(&ent->mkeys);
+			mr->mmkey.key = (u32)xa_to_value(xa_ent);
 		}
+		mr->cache_ent = ent;
 	} else {
-		mr = __xa_store(&ent->mkeys, --ent->stored, NULL,
-				GFP_KERNEL);
-		WARN_ON(mr == NULL || xa_is_err(mr));
-		WARN_ON(__xa_erase(&ent->mkeys, --ent->reserved) != NULL);
-		queue_adjust_cache_locked(ent);
-		xa_unlock_irq(&ent->mkeys);
-
-		mr->mmkey.key = (u32)xa_to_value(xa_ent);

drivers/infiniband/hw/mlx5/mr.c static void cache_work_func(struct work_struct *work) · line 658

Quoted 10 lines
drivers/infiniband/hw/mlx5/mr.c +9 -0
+		/*
+		 * Can not use a cache mkey.
+		 * Create an mkey with the exact needed size.
+		 */
+		MLX5_SET(mkc, mkc, translations_octword_size,
+			 get_mkc_octo_size(access_mode, ndescs));
+		err = mlx5_ib_create_mkey(dev, &mr->mmkey, in, inlen);
+		if (err)
+			goto err;
 	}

I think this needs to be broken to functions, it is hard to read with all these ifs and inlined locking

Quoted 17 lines
drivers/infiniband/hw/mlx5/mr.c +16 -0
+static int ent_insert(struct mlx5_mr_cache *cache, struct mlx5_cache_ent *ent)
+{
+	struct rb_node **new = &cache->cache_root.rb_node, *parent = NULL;
+	int size = MLX5_ST_SZ_BYTES(mkc);
+	struct mlx5_cache_ent *cur;
+	int cmp;
+
+	/* Figure out where to put new node */
+	while (*new) {
+		cur = container_of(*new, struct mlx5_cache_ent, node);
+		parent = *new;
+		cmp = memcmp(ent->mkc, cur->mkc, size);
+		if (cmp < 0)
+			new = &((*new)->rb_left);
+		else if (cmp > 0)
+			new = &((*new)->rb_right);

drivers/infiniband/hw/mlx5/mr.c static void delay_time_func(struct timer_list *t) · line 752

Quoted 10 lines
drivers/infiniband/hw/mlx5/mr.c +10 -0
+		else
+			return -EEXIST;
+	}
+
+	/* Add new node and rebalance tree. */
+	rb_link_node(&ent->node, parent, new);
+	rb_insert_color(&ent->node, &cache->cache_root);
+
+	return 0;
+}

This should be near the find

Quoted 26 lines
drivers/infiniband/hw/mlx5/mr.c +26 -0
+
+static struct mlx5_cache_ent *mlx5_ib_create_cache_ent(struct mlx5_ib_dev *dev,
+						       int order)
+{
+	struct mlx5_cache_ent *ent;
+	int ret;
+
+	ent = kzalloc(sizeof(*ent), GFP_KERNEL);
+	if (!ent)
+		return ERR_PTR(-ENOMEM);
+
+	ent->mkc = kzalloc(MLX5_ST_SZ_BYTES(mkc), GFP_KERNEL);
+	if (!ent->mkc) {
+		kfree(ent);
+		return ERR_PTR(-ENOMEM);
+	}
+
+	ent->ndescs = 1 << order;
+	mlx5_set_cache_mkc(dev, ent->mkc, 0, MLX5_MKC_ACCESS_MODE_MTT,
+			   PAGE_SHIFT);
+	MLX5_SET(mkc, ent->mkc, translations_octword_size,
+		 get_mkc_octo_size(MLX5_MKC_ACCESS_MODE_MTT, ent->ndescs));
+	mutex_lock(&dev->cache.cache_lock);
+	ret = ent_insert(&dev->cache, ent);
+	mutex_unlock(&dev->cache.cache_lock);
+	if (ret) {

drivers/infiniband/hw/mlx5/mr.c static void delay_time_func(struct timer_list *t) · line 788

Quoted 10 lines
drivers/infiniband/hw/mlx5/mr.c +10 -0
+		kfree(ent->mkc);
+		kfree(ent);
+		return ERR_PTR(ret);
+	}
+
+	xa_init_flags(&ent->mkeys, XA_FLAGS_ALLOC | XA_FLAGS_LOCK_IRQ);
+	ent->dev = dev;
+
+	INIT_WORK(&ent->work, cache_work_func);
+	INIT_DELAYED_WORK(&ent->dwork, delayed_cache_work_func);

And the ent should be fully setup before adding it to the cache

Jason

#4 lore raw

Re: [PATCH rdma-next 2/7] RDMA/mlx5: Replace cache list with Xarray

In reply to Jason Gunthorpe
Quoted 1 line · Jason Gunthorpe
Quoted 1 line
(quoted diff) +0 -0
@@ -166,14 +169,14 @@ static void create_mkey_callback(int status, struct mlx5_async_work *context)

<...>

Quoted 5 lines

drivers/infiniband/hw/mlx5/mr.c static void remove_cache_mr_locked(struct mlx5_cache_ent *ent) · line 335

Quoted 3 lines

static int resize_available_mrs(struct mlx5_cache_ent *ent, unsigned int target,

drivers/infiniband/hw/mlx5/mr.c +1 -0
 				bool limit_fill)
+	 __acquires(&ent->lock) __releases(&ent->lock)

Why?

This is mine, I got tons of complains from sparse [1] because of this commit [2].

[1] drivers/infiniband/hw/mlx5/mr.c:280:25: warning: context imbalance in 'resize_available_mkeys' - unexpected unlock
[2] https://lwn.net/Articles/109066/

Thanks

#5 lore raw

Re: [PATCH rdma-next 2/7] RDMA/mlx5: Replace cache list with Xarray

In reply to Jason Gunthorpe
Quoted 74 lines · Jason Gunthorpe
Quoted 25 lines
drivers/infiniband/hw/mlx5/mr.c +6 -5
@@ -166,14 +169,14 @@ static void create_mkey_callback(int status, struct mlx5_async_work *context)
 
  	WRITE_ONCE(dev->cache.last_add, jiffies);
 
-	spin_lock_irqsave(&ent->lock, flags);
-	list_add_tail(&mr->list, &ent->head);
-	ent->available_mrs++;
+	xa_lock_irqsave(&ent->mkeys, flags);
+	xa_ent = __xa_store(&ent->mkeys, ent->stored++, mr, GFP_ATOMIC);
+	WARN_ON(xa_ent != NULL);
+	ent->pending--;
  	ent->total_mrs++;
  	/* If we are doing fill_to_high_water then keep going. */
  	queue_adjust_cache_locked(ent);
-	ent->pending--;
-	spin_unlock_irqrestore(&ent->lock, flags);
+	xa_unlock_irqrestore(&ent->mkeys, flags);
  }
 
  static struct mlx5_ib_mr *alloc_cache_mr(struct mlx5_cache_ent *ent, void *mkc)
@@ -196,6 +199,25 @@ static struct mlx5_ib_mr *alloc_cache_mr(struct mlx5_cache_ent *ent, void *mkc)
  	return mr;
  }
 
+static int _push_reserve_mkey(struct mlx5_cache_ent *ent)

drivers/infiniband/hw/mlx5/mr.c static struct mlx5_ib_mr *alloc_cache_mr(struct mlx5_cache_ent *ent, void *mkc) · line 212

Quoted 10 lines
drivers/infiniband/hw/mlx5/mr.c +10 -0
+{
+	unsigned long to_reserve;
+	int rc;
+
+	while (true) {
+		to_reserve = ent->reserved;
+		rc = xa_err(__xa_cmpxchg(&ent->mkeys, to_reserve, NULL,
+					 XA_ZERO_ENTRY, GFP_KERNEL));
+		if (rc)
+			return rc;

What about when old != NULL ?

drivers/infiniband/hw/mlx5/mr.c static struct mlx5_ib_mr *alloc_cache_mr(struct mlx5_cache_ent *ent, void *mkc) · line 214

Quoted 2 lines
drivers/infiniband/hw/mlx5/mr.c +2 -0
+		if (to_reserve != ent->reserved)
+			continue;

There is an edge case where where reserved could have shrunk alot while the lock was released, and xa_cmpxchg could succeed. The above if will protect things, however a ZERO_ENTRY will have been written to some weird place in the XA. It needs a

if (old == NULL) // ie we stored something someplace weird

     __xa_erase(&ent->mkeys, to_reserve)
Quoted 14 lines
drivers/infiniband/hw/mlx5/mr.c +6 -1
+		ent->reserved++;
+		break;
+	}
+	return 0;
+}
+
  /* Asynchronously schedule new MRs to be populated in the cache. */
  static int add_keys(struct mlx5_cache_ent *ent, unsigned int num)
  {
@@ -217,23 +239,32 @@ static int add_keys(struct mlx5_cache_ent *ent, unsigned int num)
  			err = -ENOMEM;
  			break;
  		}
-		spin_lock_irq(&ent->lock);

drivers/infiniband/hw/mlx5/mr.c static int add_keys(struct mlx5_cache_ent *ent, unsigned int num) · line 250

Quoted 10 lines
drivers/infiniband/hw/mlx5/mr.c +7 -1
+		xa_lock_irq(&ent->mkeys);
  		if (ent->pending >= MAX_PENDING_REG_MR) {
+			xa_unlock_irq(&ent->mkeys);
  			err = -EAGAIN;
-			spin_unlock_irq(&ent->lock);
+			kfree(mr);
+			break;
+		}
+
+		err = _push_reserve_mkey(ent);

The test of ent->pending is out of date now since this can drop the lock

Quoted 3 lines

It feels like pending and (reserved - stored) are really the same thing, so maybe just directly limit the number of reserved and test it after

The mlx5_ib_dereg_mr is reserving entries as well. Should I limit create_mkey_cb due to pending deregs?

Quoted 4 lines
Quoted 4 lines
drivers/infiniband/hw/mlx5/mr.c +0 -0
@@ -287,39 +318,37 @@ static void remove_cache_mr_locked(struct mlx5_cache_ent *ent)
  {
  	struct mlx5_ib_mr *mr;
 
Quoted 10 lines

drivers/infiniband/hw/mlx5/mr.c static void remove_cache_mr_locked(struct mlx5_cache_ent *ent) · line 324

Quoted 9 lines
drivers/infiniband/hw/mlx5/mr.c +3 -5
-	lockdep_assert_held(&ent->lock);
-	if (list_empty(&ent->head))
+	if (!ent->stored)
  		return;
-	mr = list_first_entry(&ent->head, struct mlx5_ib_mr, list);
-	list_del(&mr->list);
-	ent->available_mrs--;
+	mr = __xa_store(&ent->mkeys, --ent->stored, NULL, GFP_KERNEL);
+	WARN_ON(mr == NULL || xa_is_err(mr));

Add a if (reserved != stored) before the below?

I initiated the xarray using XA_FLAGS_ALLOC, therefore, the __xa_store above will mark the entry as ZERO_ENTRY.

Quoted 66 lines

Review static void remove_cache_mr_locked(struct mlx5_cache_ent *ent) · line 325

Quoted 2 lines
(quoted diff) +1 -0
+	WARN_ON(__xa_erase(&ent->mkeys, --ent->reserved) != NULL);

Also please avoid writing WARN_ON(something with side effects)

   old = __xa_erase(&ent->mkeys, --ent->reserved);
   WARN_ON(old != NULL);

Same for all places

drivers/infiniband/hw/mlx5/mr.c static void remove_cache_mr_locked(struct mlx5_cache_ent *ent) · line 335

Quoted 3 lines
drivers/infiniband/hw/mlx5/mr.c +1 -0
  static int resize_available_mrs(struct mlx5_cache_ent *ent, unsigned int target,
  				bool limit_fill)
+	 __acquires(&ent->lock) __releases(&ent->lock)

Why?

drivers/infiniband/hw/mlx5/mr.c static void remove_cache_mr_locked(struct mlx5_cache_ent *ent) · line 309

Quoted 5 lines
drivers/infiniband/hw/mlx5/mr.c +0 -2
  {
  	int err;
 
-	lockdep_assert_held(&ent->lock);
-

Why?

Quoted 10 lines
drivers/infiniband/hw/mlx5/mr.c +1 -2
  static void clean_keys(struct mlx5_ib_dev *dev, int c)
  {
  	struct mlx5_mr_cache *cache = &dev->cache;
  	struct mlx5_cache_ent *ent = &cache->ent[c];
-	struct mlx5_ib_mr *tmp_mr;
  	struct mlx5_ib_mr *mr;
-	LIST_HEAD(del_list);
+	unsigned long index;
 
  	cancel_delayed_work(&ent->dwork);

drivers/infiniband/hw/mlx5/mr.c static void mlx5_mr_cache_free(struct mlx5_ib_dev *dev, struct mlx5_ib_mr *mr) · line 647

Quoted 10 lines
drivers/infiniband/hw/mlx5/mr.c +1 -9
-	while (1) {
-		spin_lock_irq(&ent->lock);
-		if (list_empty(&ent->head)) {
-			spin_unlock_irq(&ent->lock);
-			break;
-		}
-		mr = list_first_entry(&ent->head, struct mlx5_ib_mr, list);
-		list_move(&mr->list, &del_list);
-		ent->available_mrs--;
+	xa_for_each(&ent->mkeys, index, mr) {

This isn't quite the same thing, the above tolerates concurrent add, this does not.

It should be more like

while (ent->stored) {
    mr = __xa_erase(stored--);
Quoted 4 lines
drivers/infiniband/hw/mlx5/mr.c +0 -0
@@ -1886,6 +1901,17 @@ mlx5_free_priv_descs(struct mlx5_ib_mr *mr)
  	}
  }
 

drivers/infiniband/hw/mlx5/mr.c mlx5_free_priv_descs(struct mlx5_ib_mr *mr) · line 1913

Quoted 10 lines
drivers/infiniband/hw/mlx5/mr.c +10 -0
+static int push_reserve_mkey(struct mlx5_cache_ent *ent)
+{
+	int ret;
+
+	xa_lock_irq(&ent->mkeys);
+	ret = _push_reserve_mkey(ent);
+	xa_unlock_irq(&ent->mkeys);
+
+	return ret;
+}

Put this close to _push_reserve_mkey() please

Jason

maintainer
#14 lore raw
In reply to Leon Romanovsky
Quoted 9 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

Hi,

This series from Aharon refactors mlx5 MR cache management logic to speedup deregistration significantly.

Thanks

Quoted 8 lines

Aharon Landau (7):
RDMA/mlx5: Merge similar flows of allocating MR from the cache RDMA/mlx5: Replace cache list with Xarray
RDMA/mlx5: Store in the cache mkeys instead of mrs RDMA/mlx5: Change the cache structure to an RB-tree RDMA/mlx5: Reorder calls to pcie_relaxed_ordering_enabled() RDMA/mlx5: Delay the deregistration of a non-cache mkey RDMA/mlx5: Rename the mkey cache variables and functions

Let's me resubmit it after more deep review.

Thanks

maintainer
#6 lore raw

Re: [PATCH rdma-next 2/7] RDMA/mlx5: Replace cache list with Xarray

In reply to Aharon Landau
Quoted 5 lines · Aharon Landau
Quoted 3 lines

It feels like pending and (reserved - stored) are really the same thing, so maybe just directly limit the number of reserved and test it after

The mlx5_ib_dereg_mr is reserving entries as well. Should I limit create_mkey_cb due to pending deregs?

Sure, it is the same concept, deregs are going to refill things just as wass as new creations.

Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
drivers/infiniband/hw/mlx5/mr.c +0 -2
@@ -287,39 +318,37 @@ static void remove_cache_mr_locked(struct mlx5_cache_ent *ent)
  {
  	struct mlx5_ib_mr *mr;
-	lockdep_assert_held(&ent->lock);
-	if (list_empty(&ent->head))
Quoted 8 lines

drivers/infiniband/hw/mlx5/mr.c static void remove_cache_mr_locked(struct mlx5_cache_ent *ent) · line 324

Quoted 7 lines
drivers/infiniband/hw/mlx5/mr.c +3 -3
+	if (!ent->stored)
  		return;
-	mr = list_first_entry(&ent->head, struct mlx5_ib_mr, list);
-	list_del(&mr->list);
-	ent->available_mrs--;
+	mr = __xa_store(&ent->mkeys, --ent->stored, NULL, GFP_KERNEL);
+	WARN_ON(mr == NULL || xa_is_err(mr));

Add a if (reserved != stored) before the below?

Quoted 2 lines

I initiated the xarray using XA_FLAGS_ALLOC, therefore, the __xa_store above will mark the entry as ZERO_ENTRY.

Oh, do not use XA_FLAGS_ALLOC, this is not what is it for

Jason