FRMR pools fixes
Subsystems (2)
cover FRMR pools fixes
From: Michael Guralnik <[email protected]>
This series addresses several bugs in FRMR pool handling.
Patch 2 fixes incorrect masking of TPH-related bits in the FRMR pool key, which caused stale TPH values to be used when creating handles from an empty pool.
Patch 3 fixes set-pinned flow to use the pool key returned from the driver build_key callback instead of the raw key supplied by user.
Patch 8 extends the FRMR pools API with a new drop() operation. This allows drivers to update pool state on handle destruction when revocation fails, without incorrectly returning the handle to the pool.
The remaining patches fix error path handling, covering cases where memory allocation fails during queue expansion, and where handle creation or destruction operations return errors.
Michael Guralnik (9):
RDMA/mlx5: Fix mkey creation error flow rollback RDMA/mlx5: Fix TPH extraction in FRMR pool key RDMA/core: Fix skipped usage for driver built FRMR key RDMA/core: Fix FRMR aging push to queue error flow RDMA/core: Fix FRMR set pinned push error path RDMA/core: Avoid NULL dereference on FRMR bad usage RDMA/core: Fix FRMR handle leak on push failure RDMA/core: Add ib_frmr_pool_drop for unrecoverable handles RDMA/mlx5: Drop FRMR pool handle on UMR revoke failure
drivers/infiniband/core/frmr_pools.c | 104 +++++++++++++++++++-------- drivers/infiniband/hw/mlx5/mr.c | 31 +++++--- include/rdma/frmr_pools.h | 3 +- 3 files changed, 98 insertions(+), 40 deletions(-)
Signature · 1 line
2.52.0
PATCH 1/9 RDMA/mlx5: Fix mkey creation error flow rollback
From: Michael Guralnik <[email protected]>
Fix the indices of mkeys destroyed in case of an error in batch mkey creation.
--- drivers/infiniband/hw/mlx5/mr.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/infiniband/hw/mlx5/mr.c +1 -1
@@ -294,7 +294,7 @@ static int mlx5r_create_mkeys(struct ib_device *device, struct ib_frmr_key *key, free_in:kfree (in);if (err) -for (; i >0 ; i--) +for (i--; i >=0 ; i--)mlx5_core_destroy_mkey (dev->mdev, handles[i]);return err; }
Signature · 1 line
2.52.0
PATCH 2/9 RDMA/mlx5: Fix TPH extraction in FRMR pool key
In reply to Michael GurFrom: Michael Guralnik <[email protected]>
Fix reading the PH value from the FRMR pool key by shifting the pool key to the relevant bits.
--- drivers/infiniband/hw/mlx5/mr.c | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-)
drivers/infiniband/hw/mlx5/mr.c +9 -7
@@ -31,6 +31,7 @@* SOFTWARE. */ +#include <linux/bitfield.h> #include <linux/kref.h> #include <linux/random.h> #include <linux/debugfs.h>
@@ -163,9 +164,8 @@ static int get_unchangeable_access_flags(struct mlx5_ib_dev *dev,#define MLX5_FRMR_POOLS_KEY_VENDOR_KEY_SUPPORTED \ MLX5_FRMR_POOLS_KEY_ACCESS_MODE_KSM_MASK -#define MLX5_FRMR_POOLS_KERNEL_KEY_PH_SHIFT 16 -#define MLX5_FRMR_POOLS_KERNEL_KEY_PH_MASK 0xFF0000 -#define MLX5_FRMR_POOLS_KERNEL_KEY_ST_INDEX_MASK 0xFFFF +#define MLX5_FRMR_POOLS_KERNEL_KEY_PH_MASK GENMASK_ULL(23, 16) +#define MLX5_FRMR_POOLS_KERNEL_KEY_ST_INDEX_MASK GENMASK_ULL(15, 0)static struct mlx5_ib_mr * _mlx5_frmr_pool_alloc(struct mlx5_ib_dev *dev, struct ib_umem *umem,
@@ -194,7 +194,8 @@ _mlx5_frmr_pool_alloc(struct mlx5_ib_dev *dev, struct ib_umem *umem, ph ^=MLX5_IB_NO_PH ; mr->ibmr.frmr.key.kernel_vendor_key = - st_index | (ph <<MLX5_FRMR_POOLS_KERNEL_KEY_PH_SHIFT ); +FIELD_PREP (MLX5_FRMR_POOLS_KERNEL_KEY_ST_INDEX_MASK , st_index) | +FIELD_PREP (MLX5_FRMR_POOLS_KERNEL_KEY_PH_MASK , ph); err =ib_frmr_pool_pop (&dev->ib_dev, &mr->ibmr);if (err) {kfree (mr);
@@ -271,9 +272,10 @@ static int mlx5r_create_mkeys(struct ib_device *device, struct ib_frmr_key *key, get_mkc_octo_size(access_mode, key->num_dma_blocks));MLX5_SET (mkc, mkc, log_page_size,PAGE_SHIFT ); - st_index = key->kernel_vendor_key & -MLX5_FRMR_POOLS_KERNEL_KEY_ST_INDEX_MASK ; - ph = key->kernel_vendor_key &MLX5_FRMR_POOLS_KERNEL_KEY_PH_MASK ; + st_index =FIELD_GET (MLX5_FRMR_POOLS_KERNEL_KEY_ST_INDEX_MASK , + key->kernel_vendor_key); + ph =FIELD_GET (MLX5_FRMR_POOLS_KERNEL_KEY_PH_MASK , + key->kernel_vendor_key);if (ph) {/* Normalize ph: swap MLX5_IB_NO_PH for 0 */ if (ph ==MLX5_IB_NO_PH )
Signature · 1 line
2.52.0
PATCH 3/9 RDMA/core: Fix skipped usage for driver built FRMR key
In reply to Michael GurFrom: Michael Guralnik <[email protected]>
When creating FRMR handles following a netlink command to pin handles, use the key after driver callback instead of using the key passed directly from user.
--- drivers/infiniband/core/frmr_pools.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/infiniband/core/frmr_pools.c +1 -1
@@ -426,7 +426,7 @@ int ib_frmr_pools_set_pinned(struct ib_device *device, struct ib_frmr_key *key,if (!handles)return -ENOMEM ; - ret = pools->pool_ops->create_frmrs (device, key, handles, + ret = pools->pool_ops->create_frmrs (device, &driver_key, handles, needed_handles);if (ret) {kfree (handles);
Signature · 1 line
2.52.0
PATCH 4/9 RDMA/core: Fix FRMR aging push to queue error flow
In reply to Michael GurFrom: Michael Guralnik <[email protected]>
Aging pools with pinned handles requires moving handles from the active queue to a non-empty inactive queue that might fail on new page allocation, we are currently not handling the fault and leaking any mkey that fails the push.
Fix by Introducing push_queue_to_queue_locked() that fills the destination's partial tail page from the source and then splices the remaining source pages onto the destination, performing no allocation.
Replace the per-handle move loop in age_pinned_pool() and the open-coded splice in pool_aging_work() with calls to the helper. As the helper cannot fail under memory pressure, removing a class of GFP_ATOMIC allocations under the pool lock and simplifying the error flow.
--- drivers/infiniband/core/frmr_pools.c | 53 ++++++++++++++++++++-------- 1 file changed, 38 insertions(+), 15 deletions(-)
drivers/infiniband/core/frmr_pools.c +38 -15
@@ -97,13 +97,44 @@ static void destroy_all_handles_in_queue(struct ib_device *device, } } +/* +* Bulk-move all handles from @src into @dst without allocating new pages. +* If @dst has a partial tail page, fill it handle-by-handle from @src first +* to preserve the invariant that only the tail page is partial, then splice +* the remaining @src pages onto @dst. On return @src is empty. +* +* Caller must hold the lock protecting both queues. +*/ +static void splice_frmr_queue_locked (struct frmr_queue *dst, +struct frmr_queue *src) +{ +u32 free_in_tail = dst->ci %NUM_HANDLES_PER_PAGE ; +u32 handle; + +if (free_in_tail) { + free_in_tail =NUM_HANDLES_PER_PAGE - free_in_tail; +while (free_in_tail && src->ci) { + handle =pop_handle_from_queue_locked (src); +push_handle_to_queue_locked (dst, handle); + free_in_tail--; + } + } + +if (src->ci >0 ) { +list_splice_tail_init (&src->pages_list, &dst->pages_list); + dst->num_pages += src->num_pages; + dst->ci += src->ci; + src->num_pages =0 ; + src->ci =0 ; + } +} +static bool age_pinned_pool (struct ib_device *device,struct ib_frmr_pool *pool) {struct ib_frmr_pools *pools = device->frmr_pools;u32 total, to_destroy, destroyed =0 ;bool has_work = false;u32 *handles; -u32 handle;spin_lock (&pool->lock); total = pool->queue.ci + pool->inactive_queue.ci + pool->in_use;
@@ -112,7 +143,7 @@ static bool age_pinned_pool(struct ib_device *device, struct ib_frmr_pool *pool)return false; } - to_destroy = total - pool->pinned_handles; + to_destroy =min (total - pool->pinned_handles, pool->inactive_queue.ci); handles =kcalloc (to_destroy,sizeof (*handles),GFP_ATOMIC );if (!handles) {
@@ -121,15 +152,13 @@ static bool age_pinned_pool(struct ib_device *device, struct ib_frmr_pool *pool) }/* Destroy all excess handles in the inactive queue */ -while (pool->inactive_queue.ci && destroyed < to_destroy) { - handles[destroyed++] =pop_handle_from_queue_locked ( +for (; destroyed < to_destroy; destroyed++) + handles[destroyed] =pop_handle_from_queue_locked ( &pool->inactive_queue); - }/* Move all handles from regular queue to inactive queue */ -while (pool->queue.ci) { - handle =pop_handle_from_queue_locked (&pool->queue); -push_handle_to_queue_locked (&pool->inactive_queue, handle); +if (pool->queue.ci >0 ) { +splice_frmr_queue_locked (&pool->inactive_queue, &pool->queue); has_work = true; }
@@ -158,13 +187,7 @@ static void pool_aging_work(struct work_struct *work)/* Move all pages from regular queue to inactive queue */ spin_lock (&pool->lock);if (pool->queue.ci >0 ) { -list_splice_tail_init (&pool->queue.pages_list, - &pool->inactive_queue.pages_list); - pool->inactive_queue.num_pages = pool->queue.num_pages; - pool->inactive_queue.ci = pool->queue.ci; - - pool->queue.num_pages =0 ; - pool->queue.ci =0 ; +splice_frmr_queue_locked (&pool->inactive_queue, &pool->queue); has_work = true; }spin_unlock (&pool->lock);
Signature · 1 line
2.52.0
PATCH 5/9 RDMA/core: Fix FRMR set pinned push error path
In reply to Michael GurFrom: Michael Guralnik <[email protected]>
Add destruction of FRMR handles in case the push to the pool fails. This prevents resources leak in case pool page allocation fails.
--- drivers/infiniband/core/frmr_pools.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-)
drivers/infiniband/core/frmr_pools.c +8 -3
@@ -461,11 +461,16 @@ int ib_frmr_pools_set_pinned(struct ib_device *device, struct ib_frmr_key *key, ret =push_handle_to_queue_locked (&pool->queue, handles[i]);if (ret) - goto end; +break ; } - -end:spin_unlock (&pool->lock); + +if (ret) { +/* Destroy handles created but never pushed to the pool. */ + pools->pool_ops->destroy_frmrs (device, &handles[i], + needed_handles - i); + } +kfree (handles); schedule_aging:
Signature · 1 line
2.52.0
PATCH 5/9 RDMA/core: Fix FRMR set pinned push error path
From: Tao Cui <[email protected]>
Quoted 5 lines
Add destruction of FRMR handles in case the push to the pool fails. This prevents resources leak in case pool page allocation fails.
Fixes 020d189d16a6 ("RDMA/core: Add pinned handles to FRMR pools")Signed-off-by Michael Guralnik <[email protected]>
Thanks
--
Tao
PATCH 6/9 RDMA/core: Avoid NULL dereference on FRMR bad usage
In reply to Michael GurFrom: Michael Guralnik <[email protected]>
In case a driver calls FRMR pop operation without a successful init, return after triggering a warning to avoid the NULL dereference.
--- drivers/infiniband/core/frmr_pools.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/infiniband/core/frmr_pools.c +3 -1
@@ -529,7 +529,9 @@ int ib_frmr_pool_pop(struct ib_device *device, struct ib_mr *mr)struct ib_frmr_pools *pools = device->frmr_pools;struct ib_frmr_pool *pool; -WARN_ON_ONCE (!device->frmr_pools); +if (WARN_ON_ONCE (!pools)) +return -EINVAL ; + pool =ib_frmr_pool_find (pools, &mr->frmr.key);if (!pool) { pool =create_frmr_pool (device, &mr->frmr.key);
Signature · 1 line
2.52.0
PATCH 7/9 RDMA/core: Fix FRMR handle leak on push failure
In reply to Michael GurFrom: Michael Guralnik <[email protected]>
Failure to push a handle to the pool, caused by ENOMEM on queue page allocation, will trigger missing in_use counter update, skewing pool state indefinitely.
Fix that by moving the handling of handle destruction in such case into the FRMR code, ensuring the handle is either pushed to the pool or destroyed inside the same function.
Adjust mlx5_ib call site accordingly.
--- drivers/infiniband/core/frmr_pools.c | 19 +++++++++++-------- drivers/infiniband/hw/mlx5/mr.c | 5 +++-- include/rdma/frmr_pools.h | 2 +- 3 files changed, 15 insertions(+), 11 deletions(-)
drivers/infiniband/core/frmr_pools.c +11 -8
@@ -549,9 +549,8 @@ EXPORT_SYMBOL(ib_frmr_pool_pop);* @device: The device to push the FRMR handle to. * @mr: The MR containing the FRMR handle to push back to the pool. * -* Returns 0 on success, negative error code on failure. */ -int ib_frmr_pool_push (struct ib_device *device,struct ib_mr *mr) +void ib_frmr_pool_push (struct ib_device *device,struct ib_mr *mr) {struct ib_frmr_pool *pool = mr->frmr.pool;struct ib_frmr_pools *pools = device->frmr_pools;
@@ -559,19 +558,23 @@ int ib_frmr_pool_push(struct ib_device *device, struct ib_mr *mr)int ret;spin_lock (&pool->lock); + pool->in_use--; + ret =push_handle_to_queue_locked (&pool->queue, mr->frmr.handle); +/* Schedule aging every time an empty pool becomes non-empty */ -if (pool->queue.ci ==0 ) +if (!ret && pool->queue.ci ==1 ) schedule_aging = true; - ret =push_handle_to_queue_locked (&pool->queue, mr->frmr.handle); -if (ret ==0 ) - pool->in_use--;spin_unlock (&pool->lock); -if (ret ==0 && schedule_aging) +if (ret) { + pools->pool_ops->destroy_frmrs (device, &mr->frmr.handle,1 ); +return ; + } + +if (schedule_aging)queue_delayed_work (pools->aging_wq, &pool->aging_work,secs_to_jiffies (READ_ONCE (pools->aging_period_sec))); -return ret; }EXPORT_SYMBOL (ib_frmr_pool_push);
drivers/infiniband/hw/mlx5/mr.c +3 -2
@@ -1379,9 +1379,10 @@ static int mlx5r_handle_mkey_cleanup(struct mlx5_ib_mr *mr)bool is_odp =is_odp_mr (mr);int ret; -if (mr->ibmr.frmr.pool && !mlx5_umr_revoke_mr_with_lock (mr) && - !ib_frmr_pool_push (mr->ibmr.device, &mr->ibmr)) +if (mr->ibmr.frmr.pool && !mlx5_umr_revoke_mr_with_lock (mr)) { +ib_frmr_pool_push (mr->ibmr.device, &mr->ibmr);return 0 ; + }if (is_odp)mutex_lock (&to_ib_umem_odp (mr->umem)->umem_mutex);
include/rdma/frmr_pools.h +1 -1
@@ -34,6 +34,6 @@ int ib_frmr_pools_init(struct ib_device *device, const struct ib_frmr_pool_ops *pool_ops);void ib_frmr_pools_cleanup (struct ib_device *device);int ib_frmr_pool_pop (struct ib_device *device,struct ib_mr *mr); -int ib_frmr_pool_push (struct ib_device *device,struct ib_mr *mr); +void ib_frmr_pool_push (struct ib_device *device,struct ib_mr *mr);#endif /* FRMR_POOLS_H */
Signature · 1 line
2.52.0
PATCH 8/9 RDMA/core: Add ib_frmr_pool_drop for unrecoverable handles
In reply to Michael GurFrom: Michael Guralnik <[email protected]>
A driver that has popped a handle from an FRMR pool can hit failures that leave the handle in a state where it can't safely be returned for reuse. The driver destroys the handle itself, but the pool has no way to learn about it, so the in_use counter drifts upward.
Add ib_frmr_pool_drop to balance the pool's accounting in this case. Every pop is now balanced by exactly one push or drop.
--- drivers/infiniband/core/frmr_pools.c | 15 +++++++++++++++ include/rdma/frmr_pools.h | 1 + 2 files changed, 16 insertions(+)
drivers/infiniband/core/frmr_pools.c +15 -0
@@ -578,3 +578,18 @@ void ib_frmr_pool_push(struct ib_device *device, struct ib_mr *mr) }EXPORT_SYMBOL (ib_frmr_pool_push); + +/* +* Drop a handle previously popped from the pool without returning it for +* reuse. The caller is responsible for destroying the underlying hardware +* resource. +*/ +void ib_frmr_pool_drop (struct ib_mr *mr) +{ +struct ib_frmr_pool *pool = mr->frmr.pool; + +spin_lock (&pool->lock); + pool->in_use--; +spin_unlock (&pool->lock); +} +EXPORT_SYMBOL (ib_frmr_pool_drop);
include/rdma/frmr_pools.h +1 -0
@@ -35,5 +35,6 @@ int ib_frmr_pools_init(struct ib_device *device,void ib_frmr_pools_cleanup (struct ib_device *device);int ib_frmr_pool_pop (struct ib_device *device,struct ib_mr *mr);void ib_frmr_pool_push (struct ib_device *device,struct ib_mr *mr); +void ib_frmr_pool_drop (struct ib_mr *mr);#endif /* FRMR_POOLS_H */
Signature · 1 line
2.52.0
PATCH 9/9 RDMA/mlx5: Drop FRMR pool handle on UMR revoke failure
In reply to Michael GurFrom: Michael Guralnik <[email protected]>
When UMR revoke fails during MR cleanup, the handle is left in an unknown state and cannot be returned to the pool. The driver already destroys the mkey via the fallback path, but the pool's in_use counter is never decremented, drifting upward over time.
Call ib_frmr_pool_drop on the revoke-failure path so the pool's accounting stays consistent with the handles it has handed out.
--- drivers/infiniband/hw/mlx5/mr.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-)
drivers/infiniband/hw/mlx5/mr.c +9 -3
@@ -1379,9 +1379,11 @@ static int mlx5r_handle_mkey_cleanup(struct mlx5_ib_mr *mr)bool is_odp =is_odp_mr (mr);int ret; -if (mr->ibmr.frmr.pool && !mlx5_umr_revoke_mr_with_lock (mr)) { -ib_frmr_pool_push (mr->ibmr.device, &mr->ibmr); -return 0 ; +if (mr->ibmr.frmr.pool) { +if (!mlx5_umr_revoke_mr_with_lock (mr)) { +ib_frmr_pool_push (mr->ibmr.device, &mr->ibmr); +return 0 ; + } }if (is_odp)
@@ -1403,6 +1405,10 @@ static int mlx5r_handle_mkey_cleanup(struct mlx5_ib_mr *mr)dma_resv_unlock (to_ib_umem_dmabuf (mr->umem)->attach->dmabuf->resv); } + +if (mr->ibmr.frmr.pool && !ret) +ib_frmr_pool_drop (&mr->ibmr); +return ret; }
Signature · 1 line
2.52.0
Quoted 19 lines · Michael Gur
From: Michael Guralnik <[email protected]>
This series addresses several bugs in FRMR pool handling.
Patch 2 fixes incorrect masking of TPH-related bits in the FRMR pool key, which caused stale TPH values to be used when creating handles from an empty pool.
Patch 3 fixes set-pinned flow to use the pool key returned from the driver
build_keycallback instead of the raw key supplied by user.Patch 8 extends the FRMR pools API with a new
drop()operation. This allows drivers to update pool state on handle destruction when revocation fails, without incorrectly returning the handle to the pool.The remaining patches fix error path handling, covering cases where memory allocation fails during queue expansion, and where handle creation or destruction operations return errors.
Quoted 10 lines
Michael Guralnik (9):
RDMA/mlx5: Fix mkey creation error flow rollback RDMA/mlx5: Fix TPH extraction in FRMR pool key RDMA/core: Fix skipped usage for driver built FRMR key RDMA/core: Fix FRMR aging push to queue error flow RDMA/core: Fix FRMR set pinned push error path RDMA/core: Avoid NULL dereference on FRMR bad usage RDMA/core: Fix FRMR handle leak on push failure RDMA/core: Addib_frmr_pool_dropfor unrecoverable handles RDMA/mlx5: Drop FRMR pool handle on UMR revoke failure
These fixes seem OK and Sashiko found two more:
https://sashiko.dev/#/patchset/20260610000145.820592-1-michaelgur%40nvidia.com
Can you send a followup?
I'll pick this up tomorrow
Jason
Quoted 33 lines · Jason Gunthorpe
External email: Use caution opening links or attachments
Quoted 19 lines · Michael Gur
From: Michael Guralnik <[email protected]>
This series addresses several bugs in FRMR pool handling.
Patch 2 fixes incorrect masking of TPH-related bits in the FRMR pool key, which caused stale TPH values to be used when creating handles from an empty pool.
Patch 3 fixes set-pinned flow to use the pool key returned from the driver
build_keycallback instead of the raw key supplied by user.Patch 8 extends the FRMR pools API with a new
drop()operation. This allows drivers to update pool state on handle destruction when revocation fails, without incorrectly returning the handle to the pool.The remaining patches fix error path handling, covering cases where memory allocation fails during queue expansion, and where handle creation or destruction operations return errors.
Quoted 10 lines
Michael Guralnik (9):
RDMA/mlx5: Fix mkey creation error flow rollback RDMA/mlx5: Fix TPH extraction in FRMR pool key RDMA/core: Fix skipped usage for driver built FRMR key RDMA/core: Fix FRMR aging push to queue error flow RDMA/core: Fix FRMR set pinned push error path RDMA/core: Avoid NULL dereference on FRMR bad usage RDMA/core: Fix FRMR handle leak on push failure RDMA/core: Addib_frmr_pool_dropfor unrecoverable handles RDMA/mlx5: Drop FRMR pool handle on UMR revoke failure
Quoted 5 lines
These fixes seem OK and Sashiko found two more:
https://sashiko.dev/#/patchset/20260610000145.820592-1-michaelgur%40nvidia.com
Can you send a followup?
Sure, I have a few patches around umr and frmr pools I'm testing and was planning to send on the next cycle. I'll handle these as well in that series.
Quoted 1 line
I'll pick this up tomorrow
Thanks
Quoted 2 lines
Jason