Batch of mlx5 MR cache fixes
Subsystems (2)
cover Batch of mlx5 MR cache fixes
From: Leon Romanovsky <[email protected]>
Hi,
This is a batch of mlx5 MR cache fixes.
Thanks
Michael Guralnik (4):
RDMA/mlx5: Drop redundant work canceling from clean_keys() RDMA/mlx5: Fix counter update on MR cache mkey creation RDMA/mlx5: Limit usage of over-sized mkeys from the MR cache RDMA/mlx5: Fix MR cache temp entries cleanup
drivers/infiniband/hw/mlx5/mlx5_ib.h | 2 +- drivers/infiniband/hw/mlx5/mr.c | 99 +++++++++++++--------------- 2 files changed, 45 insertions(+), 56 deletions(-)
Signature · 1 line
2.46.0
PATCH 1/4 RDMA/mlx5: Drop redundant work canceling from clean_keys()
From: Michael Guralnik <[email protected]>
The canceling of dealyed work in clean_keys() is a leftover from years back and was added to prevent races in the cleanup process of MR cache. The cleanup process was rewritten a few years ago and the canceling of delayed work and flushing of workqueue was added before the call to clean_keys().
--- drivers/infiniband/hw/mlx5/mr.c | 1 - 1 file changed, 1 deletion(-)
drivers/infiniband/hw/mlx5/mr.c +0 -1
@@ -772,7 +772,6 @@ static void clean_keys(struct mlx5_ib_dev *dev, struct mlx5_cache_ent *ent) {u32 mkey; -cancel_delayed_work (&ent->dwork);spin_lock_irq (&ent->mkeys_queue.lock);while (ent->mkeys_queue.ci) { mkey =pop_mkey_locked (ent);
Signature · 1 line
2.46.0
PATCH 2/4 RDMA/mlx5: Fix counter update on MR cache mkey creation
In reply to Leon RomanovskyFrom: Michael Guralnik <[email protected]>
After an mkey is created, update the counter for pending mkeys before reshceduling the work that is filling the cache.
Rescheduling the work with a full MR cache entry and a wrong 'pending' counter will cause us to miss disabling the fill_to_high_water flag. Thus leaving the cache full but with an indication that it's still needs to be filled up to it's full size (2 * limit).
Next time an mkey will be taken from the cache, we'll unnecessarily continue the process of filling the cache to it's full size.
--- drivers/infiniband/hw/mlx5/mr.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/infiniband/hw/mlx5/mr.c +1 -1
@@ -214,9 +214,9 @@ static void create_mkey_callback(int status, struct mlx5_async_work *context)spin_lock_irqsave (&ent->mkeys_queue.lock, flags);push_mkey_locked (ent, mkey_out->mkey); + ent->pending--;/* If we are doing fill_to_high_water then keep going. */ queue_adjust_cache_locked (ent); - ent->pending--;spin_unlock_irqrestore (&ent->mkeys_queue.lock, flags);kfree (mkey_out); }
Signature · 1 line
2.46.0
PATCH 3/4 RDMA/mlx5: Limit usage of over-sized mkeys from the MR cache
In reply to Leon RomanovskyFrom: Michael Guralnik <[email protected]>
When searching the MR cache for suitable cache entries, don't use mkeys larger than twice the size required for the MR.
This should ensure the usage of mkeys closer to the minimal required size and reduce memory waste.
On driver init we create entries for mkeys with clear attributes and powers of 2 sizes from 4 to the max supported size.
This solves the issue for anyone using mkeys that fit these requirements.
In the use case where an MR is registered with different attributes, like an access flag we can't UMR, we'll create a new cache entry to store it upon dereg.
Without this fix, any later registration with same attributes and smaller size will use the newly created cache entry and it's mkeys, disregarding the memory waste of using mkeys larger than required.
For example, one worst-case scenario can be when registering and deregstering a 1GB mkey with ATS enabled which will cause the creation of a new cache entry to hold those type of mkeys. A user registering a 4k MR with ATS will end up using the new cache entry and an mkey that can support a 1GB MR, thus wasting x250k memory than actually needed in the HW.
Additionally, allow all small registration to use the smallest size cache entry that is initialized on driver load even if size is larger than twice the required size.
--- drivers/infiniband/hw/mlx5/mr.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)
drivers/infiniband/hw/mlx5/mr.c +12 -2
@@ -49,6 +49,7 @@ enum { MAX_PENDING_REG_MR = 8, }; +#define MLX5_MR_CACHE_PERSISTENT_ENTRY_MIN_DESCS 4#define MLX5_UMR_ALIGN 2048 static void
@@ -662,6 +663,7 @@ mkey_cache_ent_from_rb_key(struct mlx5_ib_dev *dev, {struct rb_node *node = dev->cache.rb_root.rb_node;struct mlx5_cache_ent *cur, *smallest =NULL ; +u64 ndescs_limit;int cmp;/*
@@ -680,10 +682,18 @@ mkey_cache_ent_from_rb_key(struct mlx5_ib_dev *dev,return cur; } +/* +* Limit the usage of mkeys larger than twice the required size while +* also allowing the usage of smallest cache entry for small MRs. +*/ + ndescs_limit =max_t (u64, rb_key.ndescs *2 , +MLX5_MR_CACHE_PERSISTENT_ENTRY_MIN_DESCS ); +return (smallest && smallest->rb_key.access_mode == rb_key.access_mode && smallest->rb_key.access_flags == rb_key.access_flags && - smallest->rb_key.ats == rb_key.ats) ? + smallest->rb_key.ats == rb_key.ats && + smallest->rb_key.ndescs <= ndescs_limit) ? smallest :NULL ; }
@@ -964,7 +974,7 @@ int mlx5_mkey_cache_init(struct mlx5_ib_dev *dev)mlx5_mkey_cache_debugfs_init (dev);mutex_lock (&cache->rb_lock);for (i =0 ; i <=mkey_cache_max_order (dev); i++) { - rb_key.ndescs =1 << (i +2 ); + rb_key.ndescs =MLX5_MR_CACHE_PERSISTENT_ENTRY_MIN_DESCS << i; ent =mlx5r_cache_create_ent_locked (dev, rb_key, true);if (IS_ERR (ent)) { ret =PTR_ERR (ent);
Signature · 1 line
2.46.0
PATCH 4/4 RDMA/mlx5: Fix MR cache temp entries cleanup
In reply to Leon RomanovskyFrom: Michael Guralnik <[email protected]>
Fix the cleanup of the temp cache entries that are dynamically created in the MR cache.
The cleanup of the temp cache entries is currently scheduled only when a new entry is created. Since in the cleanup of the entries only the mkeys are destroyed and the cache entry stays in the cache, subsequent registrations might reuse the entry and it will eventually be filled with new mkeys without cleanup ever getting scheduled again.
On workloads that register and deregister MRs with a wide range of properties we see the cache ends up holding many cache entries, each holding the max number of mkeys that were ever used through it.
Additionally, as the cleanup work is scheduled to run over the whole cache, any mkey that is returned to the cache after the cleanup was scheduled will be held for less than the intended 30 seconds timeout.
Solve both issues by dropping the existing remove_ent_work and reusing the existing per-entry work to also handle the temp entries cleanup.
Schedule the work to run with a 30 seconds delay every time we push an mkey to a clean temp entry.
This ensures the cleanup runs on each entry only 30 seconds after the first mkey was pushed to an empty entry.
As we have already been distinguishing between persistent and temp entries when scheduling the cache_work_func, it is not being scheduled in any other flows for the temp entries.
Another benefit from moving to a per-entry cleanup is we now not required to hold the rb_tree mutex, thus enabling other flow to run concurrently.
--- drivers/infiniband/hw/mlx5/mlx5_ib.h | 2 +- drivers/infiniband/hw/mlx5/mr.c | 82 +++++++++++----------------- 2 files changed, 32 insertions(+), 52 deletions(-)
drivers/infiniband/hw/mlx5/mlx5_ib.h +1 -1
@@ -802,6 +802,7 @@ struct mlx5_cache_ent { u8 is_tmp:1; u8 disabled:1; u8 fill_to_high_water:1; + u8 tmp_cleanup_scheduled:1;/* * - limit is the low water mark for stored mkeys, 2* limit is the
@@ -833,7 +834,6 @@ struct mlx5_mkey_cache {struct mutex rb_lock;struct dentry *fs_root;unsigned long last_add; -struct delayed_work remove_ent_dwork; };struct mlx5_ib_port_resources {
drivers/infiniband/hw/mlx5/mr.c +31 -51
131 lines not on this page · raw message
Signature · 1 line
2.46.0
Leon Romanovsky
AppliedQuoted 9 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
Hi,
This is a batch of mlx5 MR cache fixes.
Thanks
[...]
Applied, thanks!
[1/4] RDMA/mlx5: Drop redundant work canceling from clean_keys()
https://git.kernel.org/rdma/rdma/c/291766e6420e72 [2/4] RDMA/mlx5: Fix counter update on MR cache mkey creation https://git.kernel.org/rdma/rdma/c/cf047a4d734c28 [3/4] RDMA/mlx5: Limit usage of over-sized mkeys from the MR cache https://git.kernel.org/rdma/rdma/c/24e58150b7c370 [4/4] RDMA/mlx5: Fix MR cache temp entries cleanup https://git.kernel.org/rdma/rdma/c/8a1e4024cc963c
Best regards,
Signature · 1 line
Leon Romanovsky <[email protected]>