Skip to content

vdpa/mlx5: Optimze MKEY operations

Dragos Tatulea kvm lkml virtualization 2 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v2 Latest7 patches 3 repliesIn review
  2. v17 patches 11 repliesSuperseded

Changes in v2 from the cover letter

- Swapped flex array usage for plain zero length array in first patch.
- Updated code to use Scope-Based Cleanup Helpers where appropriate (only second patch).
- Added macro define for MTT alignment in first patch.
- Improved commit messages/comments based on review comments.
- Removed extra newlines.
---

Patches in v2 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/7vdpa/mlx5: Create direct MKEYs in paralleldrivers/vdpa/mlx5/core/mr.c+98 -220 replies
2/7vdpa/mlx5: Delete direct MKEYs in paralleldrivers/vdpa/mlx5/core/mr.c+64 -00 replies
3/7vdpa/mlx5: Rename functiondrivers/vdpa/mlx5/core/mlx5_vdpa.h drivers/vdpa/mlx5/core/mr.c drivers/vdpa/mlx5/net/mlx5_vnet.c+6 -60 replies
4/7vdpa/mlx5: Extract mr members in own resource structdrivers/vdpa/mlx5/core/mlx5_vdpa.h drivers/vdpa/mlx5/core/mr.c drivers/vdpa/mlx5/core/resources.c +1 more+44 -410 replies
5/7vdpa/mlx5: Rename mr_mtx -> lockdrivers/vdpa/mlx5/core/mlx5_vdpa.h drivers/vdpa/mlx5/core/mr.c drivers/vdpa/mlx5/core/resources.c +1 more+16 -160 replies
6/7vdpa/mlx5: Introduce init/destroy for MR resourcesdrivers/vdpa/mlx5/core/mlx5_vdpa.h drivers/vdpa/mlx5/core/mr.c drivers/vdpa/mlx5/core/resources.c +1 more+26 -50 replies
7/7vdpa/mlx5: Postpone MR deletiondrivers/vdpa/mlx5/core/mlx5_vdpa.h drivers/vdpa/mlx5/core/mr.c drivers/vdpa/mlx5/net/mlx5_vnet.c+64 -50 replies

Reviews trailers per patch

All 7 patches have a Reviewed-by or Acked-by.

PatchCosmin RatiuEugenio Pérez
1/7R-b
2/7R-b A-b
3/7R-b
4/7R-b A-b
5/7R-b A-b
6/7R-b
7/7R-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v2 thread

Cover letter [PATCH vhost v2 0/7] vdpa/mlx5: Optimze MKEY operations

This series improves the time of .set_map() operations by parallelizing the MKEY creation and deletion for direct MKEYs. Looking at the top level MKEY creation/deletion functions, the following improvement can be seen:

|-------------------+-------------|
| operation         | improvement |
|-------------------+-------------|
| create_user_mr()  | 3-5x        |
| destroy_user_mr() | 8x          |
|-------------------+-------------|

The last part of the series introduces lazy MKEY deletion which postpones the MKEY deletion to a later point in a workqueue.

As this series and the previous ones were targeting live migration, we can also observe improvements on this front:

|-------------------+------------------+------------------|
| Stage             | Downtime #1 (ms) | Downtime #2 (ms) |
|-------------------+------------------+------------------|
| Baseline          | 3140             | 3630             |
| Parallel MKEY ops | 1200             | 2000             |
| Deferred deletion | 1014             | 1253             |
|-------------------+------------------+------------------|

Test configuration: 256 GB VM, 32 CPUs x 2 threads per core, 4 x mlx5 vDPA devices x 32 VQs (16 VQPs)

This series must be applied on top of the parallel VQ suspend/resume series [0].

[0] https://lore.kernel.org/all/[email protected]/

---
v2:
- Swapped flex array usage for plain zero length array in first patch.
- Updated code to use Scope-Based Cleanup Helpers where appropriate (only second patch).
- Added macro define for MTT alignment in first patch.
- Improved commit messages/comments based on review comments.
- Removed extra newlines.
---

Dragos Tatulea (7):
vdpa/mlx5: Create direct MKEYs in parallel vdpa/mlx5: Delete direct MKEYs in parallel vdpa/mlx5: Rename function
vdpa/mlx5: Extract mr members in own resource struct vdpa/mlx5: Rename mr_mtx -> lock
vdpa/mlx5: Introduce init/destroy for MR resources vdpa/mlx5: Postpone MR deletion

 drivers/vdpa/mlx5/core/mlx5_vdpa.h |  25 ++-
 drivers/vdpa/mlx5/core/mr.c        | 288 +++++++++++++++++++++++++----
 drivers/vdpa/mlx5/core/resources.c |   3 -
 drivers/vdpa/mlx5/net/mlx5_vnet.c  |  53 +++---
 4 files changed, 296 insertions(+), 73 deletions(-)
Signature · 1 line
2.45.1