vdpa/mlx5: Parallelize device suspend/resume
Get series with b4
b4 am [email protected]Versions newest first · linked by author and title
- v2 Latest10 patches 14 repliesIn review
- v17 patches 9 repliesSuperseded
Changes in v2 from the cover letter
- Changed to parallel VQ suspend/resume during CVQ MQ command. Support added in the last 2 patches.
- Made the fw async command more generic and moved it to resources.c. Did that because the following series (parallel mkey ops) needs this code as well.
Dropped Acked-by from Eugenio on modified patches.
- Fixed kfree -> kvfree.
- Removed extra newline caught during review.
- As discussed in the v1, the series can be pulled in completely in the vhost tree [0]. The mlx5_core patch was reviewed by Tariq who is also a maintainer for mlx5_core.
Patches in v2 No applies recorded yet
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/10 | net/mlx5: Support throttled commands from async APIdrivers/ | +16 | 3 replies | |
| 2/10 | vdpa/mlx5: Introduce error logging functiondrivers/ drivers/ | +17 | 0 replies | |
| 3/10 | vdpa/mlx5: Introduce async fw command wrapperdrivers/ drivers/ | +88 | 1 reply | |
| 4/10 | vdpa/mlx5: Use async API for vq query commanddrivers/ drivers/ | +78 | 1 reply | |
| 5/10 | vdpa/mlx5: Use async API for vq modify commandsdrivers/ | +106 | 1 reply | |
| 6/10 | vdpa/mlx5: Parallelize device suspenddrivers/ | +29 | 0 replies | |
| 7/10 | vdpa/mlx5: Parallelize device resumedrivers/ | +14 | 0 replies | |
| 8/10 | vdpa/mlx5: Keep notifiers during suspend but ignoredrivers/ | +4 | 0 replies | |
| 9/10 | vdpa/mlx5: Small improvement for change_num_qps()drivers/ | +11 | 1 reply | |
| 10/10 | vdpa/mlx5: Parallelize VQ suspend/resume for CVQ MQ commanddrivers/ | +12 | 1 reply |
Reviews trailers per patch
All 10 patches have a Reviewed-by or Acked-by.
| Patch | Tariq Toukan | Eugenio Pérez | Lei Yang |
|---|---|---|---|
| Series | T-b | ||
| 1/10 | R-b | ||
| 2/10 | R-b | A-b | |
| 3/10 | R-b | A-b | |
| 4/10 | R-b | A-b | |
| 5/10 | R-b | A-b | |
| 6/10 | R-b | A-b | |
| 7/10 | R-b | A-b | |
| 8/10 | R-b | A-b | |
| 9/10 | R-b | A-b | |
| 10/10 | R-b | A-b |
Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 3 reviewers.
Cover letter [PATCH vhost v2 00/10] vdpa/mlx5: Parallelize device suspend/resume
This series parallelizes the mlx5_vdpa device suspend and resume operations through the firmware async API. The purpose is to reduce live migration downtime.
The series starts with changing the VQ suspend and resume commands to the async API. After that, the switch is made to issue multiple commands of the same type in parallel.
Then, the an additional improvement is added: keep the notifiers enabled during suspend but make it a NOP. Upon resume make sure that the link state is forwarded. This shaves around 30ms per device constant time.
Finally, use parallel VQ suspend and resume during the CVQ MQ command.
For 1 vDPA device x 32 VQs (16 VQPs), on a large VM (256 GB RAM, 32 CPUs x 2 threads per core), the improvements are:
+-------------------+--------+--------+-----------+ | operation | Before | After | Reduction | |-------------------+--------+--------+-----------| | mlx5_vdpa_suspend | 37 ms | 2.5 ms | 14x | | mlx5_vdpa_resume | 16 ms | 5 ms | 3x | +-------------------+--------+--------+-----------+
---
v2:
- Changed to parallel VQ suspend/resume during CVQ MQ command. Support added in the last 2 patches.
- Made the fw async command more generic and moved it to resources.c. Did that because the following series (parallel mkey ops) needs this code as well.
Dropped Acked-by from Eugenio on modified patches.
- Fixed kfree -> kvfree.
- Removed extra newline caught during review.
- As discussed in the v1, the series can be pulled in completely in the vhost tree [0]. The mlx5_core patch was reviewed by Tariq who is also a maintainer for mlx5_core.
Dragos Tatulea (10):
net/mlx5: Support throttled commands from async API vdpa/mlx5: Introduce error logging function vdpa/mlx5: Introduce async fw command wrapper vdpa/mlx5: Use async API for vq query command vdpa/mlx5: Use async API for vq modify commands vdpa/mlx5: Parallelize device suspend
vdpa/mlx5: Parallelize device resume
vdpa/mlx5: Keep notifiers during suspend but ignore vdpa/mlx5: Small improvement for change_num_qps() vdpa/mlx5: Parallelize VQ suspend/resume for CVQ MQ command
drivers/net/ethernet/mellanox/mlx5/core/cmd.c | 21 +- drivers/vdpa/mlx5/core/mlx5_vdpa.h | 22 + drivers/vdpa/mlx5/core/resources.c | 73 ++++ drivers/vdpa/mlx5/net/mlx5_vnet.c | 396 +++++++++++------- 4 files changed, 361 insertions(+), 151 deletions(-)
Signature · 1 line
2.45.1