Skip to contentSkip to conversation

Batch of mlx5 fixes for v6.10

PATCH Applied Leon Romanovsky 2 participants 13 messages May 28, 2024 – Jun 24, 2024

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover Batch of mlx5 fixes for v6.10

From: Leon Romanovsky <[email protected]>

Various fixes which I collected during the last few weeks.

Thanks

Jason Gunthorpe (3):
RDMA/mlx5: Remove extra unlock on error path RDMA/mlx5: Follow rb_key.ats when creating new mkeys RDMA/mlx5: Ensure created mkeys always have a populated rb_key

Leon Romanovsky (1):
RDMA/cache: Release GID table even if leak is detected

Patrisious Haddad (1):
RDMA/mlx5: Add check for srq max_sge attribute

Yishai Hadas (1):
RDMA/mlx5: Fix unwind flow as part of mlx5_ib_stage_init_init

 drivers/infiniband/core/cache.c   | 13 ++++---------
 drivers/infiniband/hw/mlx5/main.c |  4 ++--
 drivers/infiniband/hw/mlx5/mr.c   |  8 ++++----
 drivers/infiniband/hw/mlx5/srq.c  | 13 ++++++++-----
 4 files changed, 18 insertions(+), 20 deletions(-)
Signature · 1 line
2.45.1
#7 lore raw

PATCH 2/6 RDMA/mlx5: Remove extra unlock on error path

From: Jason Gunthorpe <[email protected]>

The below commit lifted the locking out of this function but left this error path unlock behind resulting in unbalanced locking. Remove the missed unlock too.

Fixes 627122280c87 ("RDMA/mlx5: Add work to remove temporary entries from the cache")
---
 drivers/infiniband/hw/mlx5/mr.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
drivers/infiniband/hw/mlx5/mr.c +1 -3
index ecc111ed5d86..38d2c743db87 100644
@@ -641,10 +641,8 @@ static int mlx5_cache_ent_insert(struct mlx5_mkey_cache *cache,
 			new = &((*new)->rb_left);
 		if (cmp < 0)
 			new = &((*new)->rb_right);
-		if (cmp == 0) {
-			mutex_unlock(&cache->rb_lock);
+		if (cmp == 0)
 			return -EEXIST;
-		}
 	}
 
 	/* Add new node and rebalance tree. */
Signature · 1 line
2.45.1
#8 lore raw

PATCH 3/6 RDMA/mlx5: Follow rb_key.ats when creating new mkeys

In reply to Leon Romanovsky

From: Jason Gunthorpe <[email protected]>

When a cache ent already exists but doesn't have any mkeys in it the cache will automatically create a new one based on the specification in the ent->rb_key.

ent->ats was missed when creating the new key and so ma_translation_mode was not being set even though the ent requires it.

Fixes 73d09b2fe833 ("RDMA/mlx5: Introduce mlx5r_cache_rb_key")
---
 drivers/infiniband/hw/mlx5/mr.c | 1 +
 1 file changed, 1 insertion(+)
drivers/infiniband/hw/mlx5/mr.c +1 -0
index 38d2c743db87..35dcb9d9e12a 100644
@@ -246,6 +246,7 @@ static void set_cache_mkc(struct mlx5_cache_ent *ent, void *mkc)
 	MLX5_SET(mkc, mkc, access_mode_1_0, ent->rb_key.access_mode & 0x3);
 	MLX5_SET(mkc, mkc, access_mode_4_2,
 		(ent->rb_key.access_mode >> 2) & 0x7);
+	MLX5_SET(mkc, mkc, ma_translation_mode, !!ent->rb_key.ats);
 
 	MLX5_SET(mkc, mkc, translations_octword_size,
 		 get_mkc_octo_size(ent->rb_key.access_mode,
Signature · 1 line
2.45.1
#9 lore raw

PATCH 4/6 RDMA/mlx5: Ensure created mkeys always have a populated rb_key

In reply to Leon Romanovsky

From: Jason Gunthorpe <[email protected]>

cachable and mmkey.rb_key together are used by mlx5_revoke_mr() to put the MR/mkey back into the cache. In all cases they should be set correctly.

alloc_cacheable_mr() was setting cachable but not filling rb_key, resulting in cache_ent_find_and_store() bucketing them all into a 0 length entry.

implicit_get_child_mr()/mlx5_ib_alloc_implicit_mr() failed to set cachable or rb_key at all, so the cache was not working at all for implicit ODP.

Fixes 8c1185fef68c ("RDMA/mlx5: Change check for cacheable mkeys")
Fixes dd1b913fb0d0 ("RDMA/mlx5: Cache all user cacheable mkeys on dereg MR flow")
---
 drivers/infiniband/hw/mlx5/mr.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/infiniband/hw/mlx5/mr.c +2 -1
index 35dcb9d9e12a..d3c1f63791a2 100644
@@ -718,6 +718,8 @@ static struct mlx5_ib_mr *_mlx5_mr_cache_alloc(struct mlx5_ib_dev *dev,
 	}
 	mr->mmkey.cache_ent = ent;
 	mr->mmkey.type = MLX5_MKEY_MR;
+	mr->mmkey.rb_key = ent->rb_key;
+	mr->mmkey.cacheable = true;
 	init_waitqueue_head(&mr->mmkey.wait);
 	return mr;
 }
@@ -1168,7 +1170,6 @@ static struct mlx5_ib_mr *alloc_cacheable_mr(struct ib_pd *pd,
 	mr->ibmr.pd = pd;
 	mr->umem = umem;
 	mr->page_shift = order_base_2(page_size);
-	mr->mmkey.cacheable = true;
 	set_mr_fields(dev, mr, umem->length, access_flags, iova);
 
 	return mr;
Signature · 1 line
2.45.1
#10 lore raw

PATCH 5/6 RDMA/mlx5: Fix unwind flow as part of mlx5_ib_stage_init_init

In reply to Leon Romanovsky

From: Yishai Hadas <[email protected]>

Fix unwind flow as part of mlx5_ib_stage_init_init to use the correct goto upon an error.

Fixes 758ce14aee82 ("RDMA/mlx5: Implement MACsec gid addition and deletion")
---
 drivers/infiniband/hw/mlx5/main.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/infiniband/hw/mlx5/main.c +2 -2
index 2366c46eebc8..43660c831b22 100644
@@ -3759,10 +3759,10 @@ static int mlx5_ib_stage_init_init(struct mlx5_ib_dev *dev)
 	spin_lock_init(&dev->dm.lock);
 	dev->dm.dev = mdev;
 	return 0;
-err:
-	mlx5r_macsec_dealloc_gids(dev);
 err_mp:
 	mlx5_ib_cleanup_multiport_master(dev);
+err:
+	mlx5r_macsec_dealloc_gids(dev);
 	return err;
 }
 
Signature · 1 line
2.45.1
#1 lore raw

PATCH 1/6 RDMA/cache: Release GID table even if leak is detected

In reply to Leon Romanovsky

From: Leon Romanovsky <[email protected]>

When the table is released, we nullify pointer to GID table, it means that in case GID entry leak is detected, we will leak table too.

Delete code that prevents table destruction.

Fixes b150c3862d21 ("IB/core: Introduce GID entry reference counts")
---
 drivers/infiniband/core/cache.c | 13 ++++---------
 1 file changed, 4 insertions(+), 9 deletions(-)
drivers/infiniband/core/cache.c +4 -9
index c02a96d3572a..aa62c8c7ca75 100644
@@ -794,7 +794,6 @@ static struct ib_gid_table *alloc_gid_table(int sz)
 static void release_gid_table(struct ib_device *device,
 			      struct ib_gid_table *table)
 {
-	bool leak = false;
 	int i;
 
 	if (!table)
@@ -803,15 +802,11 @@ static void release_gid_table(struct ib_device *device,
 	for (i = 0; i < table->sz; i++) {
 		if (is_gid_entry_free(table->data_vec[i]))
 			continue;
-		if (kref_read(&table->data_vec[i]->kref) > 1) {
-			dev_err(&device->dev,
-				"GID entry ref leak for index %d ref=%u\n", i,
-				kref_read(&table->data_vec[i]->kref));
-			leak = true;
-		}
+
+		dev_err(&device->dev,
+			"GID entry ref leak for index %d ref=%u\n", i,
+			kref_read(&table->data_vec[i]->kref));
 	}
-	if (leak)
-		return;
 
 	mutex_destroy(&table->lock);
 	kfree(table->data_vec);
Signature · 1 line
2.45.1
#11 lore raw

PATCH 6/6 RDMA/mlx5: Add check for srq max_sge attribute

In reply to Leon Romanovsky

From: Patrisious Haddad <[email protected]>

max_sge attribute is passed by the user, and is inserted and used unchecked, so verify that the value doesn't exceed maximum allowed value before using it.

Fixes e126ba97dba9 ("mlx5: Add driver for Mellanox Connect-IB adapters")
---
 drivers/infiniband/hw/mlx5/srq.c | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)
drivers/infiniband/hw/mlx5/srq.c +8 -5
index a056ea835da5..84be0c3d5699 100644
@@ -199,17 +199,20 @@ int mlx5_ib_create_srq(struct ib_srq *ib_srq,
 	int err;
 	struct mlx5_srq_attr in = {};
 	__u32 max_srq_wqes = 1 << MLX5_CAP_GEN(dev->mdev, log_max_srq_sz);
+	__u32 max_sge_sz =  MLX5_CAP_GEN(dev->mdev, max_wqe_sz_rq) /
+			    sizeof(struct mlx5_wqe_data_seg);
 
 	if (init_attr->srq_type != IB_SRQT_BASIC &&
 	    init_attr->srq_type != IB_SRQT_XRC &&
 	    init_attr->srq_type != IB_SRQT_TM)
 		return -EOPNOTSUPP;
 
-	/* Sanity check SRQ size before proceeding */
-	if (init_attr->attr.max_wr >= max_srq_wqes) {
-		mlx5_ib_dbg(dev, "max_wr %d, cap %d\n",
-			    init_attr->attr.max_wr,
-			    max_srq_wqes);
+	/* Sanity check SRQ and sge size before proceeding */
+	if (init_attr->attr.max_wr >= max_srq_wqes ||
+	    init_attr->attr.max_sge > max_sge_sz) {
+		mlx5_ib_dbg(dev, "max_wr %d,wr_cap %d,max_sge %d, sge_cap:%d\n",
+			    init_attr->attr.max_wr, max_srq_wqes,
+			    init_attr->attr.max_sge, max_sge_sz);
 		return -EINVAL;
 	}
 
Signature · 1 line
2.45.1
author Applied
#12 lore raw
In reply to Leon Romanovsky
Quoted 4 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

Various fixes which I collected during the last few weeks.

Quoted 8 lines

Thanks

Jason Gunthorpe (3):
RDMA/mlx5: Remove extra unlock on error path RDMA/mlx5: Follow rb_key.ats when creating new mkeys RDMA/mlx5: Ensure created mkeys always have a populated rb_key

[...]

Applied, thanks!

[1/6] RDMA/cache: Release GID table even if leak is detected

      https://git.kernel.org/rdma/rdma/c/3ac844148b9b80
[2/6] RDMA/mlx5: Remove extra unlock on error path
      https://git.kernel.org/rdma/rdma/c/36e1ea42751ce8
[3/6] RDMA/mlx5: Follow rb_key.ats when creating new mkeys
      https://git.kernel.org/rdma/rdma/c/0f28eefd6bdb6c
[4/6] RDMA/mlx5: Ensure created mkeys always have a populated rb_key
      https://git.kernel.org/rdma/rdma/c/7322d666dbd55f
[5/6] RDMA/mlx5: Fix unwind flow as part of mlx5_ib_stage_init_init
      https://git.kernel.org/rdma/rdma/c/6bb41bed95ec39
[6/6] RDMA/mlx5: Add check for srq max_sge attribute
      https://git.kernel.org/rdma/rdma/c/c405e9cac10239

Best regards,

Signature · 1 line
Leon Romanovsky <[email protected]>
maintainer
#2 lore raw

Re: [PATCH rdma-rc 1/6] RDMA/cache: Release GID table even if leak is detected

In reply to Leon Romanovsky
Quoted 6 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

When the table is released, we nullify pointer to GID table, it means that in case GID entry leak is detected, we will leak table too.

Delete code that prevents table destruction.

This converts a memory leak into a UAF, it doesn't seem like a good direction??

Jason

#3 lore raw

Re: [PATCH rdma-rc 1/6] RDMA/cache: Release GID table even if leak is detected

Quoted 9 lines · Jason Gunthorpe
Quoted 6 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

When the table is released, we nullify pointer to GID table, it means that in case GID entry leak is detected, we will leak table too.

Delete code that prevents table destruction.

This converts a memory leak into a UAF, it doesn't seem like a good direction??

Maybe we should convert dev_err() to be WARN_ON(). I didn't see any complains about GID entry leaks. It is debug print.

Thanks

Quoted 3 lines

Jason

maintainer
#4 lore raw

Re: [PATCH rdma-rc 1/6] RDMA/cache: Release GID table even if leak is detected

Quoted 7 lines · Leon Romanovsky
Quoted 6 lines · Jason Gunthorpe
Quoted 5 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

When the table is released, we nullify pointer to GID table, it means that in case GID entry leak is detected, we will leak table too.

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

Delete code that prevents table destruction.

This converts a memory leak into a UAF, it doesn't seem like a good direction??

Maybe we should convert dev_err() to be WARN_ON(). I didn't see any complains about GID entry leaks. It is debug print.

Yes WARN_ON is better

Jason

maintainer
#5 lore raw

Re: [PATCH rdma-rc 1/6] RDMA/cache: Release GID table even if leak is detected

In reply to Leon Romanovsky
Quoted 5 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

When the table is released, we nullify pointer to GID table, it means that in case GID entry leak is detected, we will leak table too.

Quoted 7 lines

Delete code that prevents table destruction.

Fixes b150c3862d21 ("IB/core: Introduce GID entry reference counts")
---
 drivers/infiniband/core/cache.c | 13 ++++---------
 1 file changed, 4 insertions(+), 9 deletions(-)

Since this is causing syzkaller failures and it really doesn't seem like stable material I'm dropping it from the rc branch and putting it in for-next. Lets see if we can fix the failures before the merge window.

Jason

#6 lore raw

Re: [PATCH rdma-rc 1/6] RDMA/cache: Release GID table even if leak is detected

Quoted 14 lines · Jason Gunthorpe
Quoted 5 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

When the table is released, we nullify pointer to GID table, it means that in case GID entry leak is detected, we will leak table too.

Quoted 7 lines

Delete code that prevents table destruction.

Fixes b150c3862d21 ("IB/core: Introduce GID entry reference counts")
---
 drivers/infiniband/core/cache.c | 13 ++++---------
 1 file changed, 4 insertions(+), 9 deletions(-)
Quoted 3 lines

Since this is causing syzkaller failures and it really doesn't seem like stable material I'm dropping it from the rc branch and putting it in for-next. Lets see if we can fix the failures before the merge window.

I don't see it in the for-next branch, can you please add it?

Thanks

Quoted 2 lines

Jason