vdpa/mlx5: Parallelize device suspend/resume
Subsystems (5)
Dragos Tatulea
maintainerv2 cover vdpa/mlx5: Parallelize device suspend/resume
This series parallelizes the mlx5_vdpa device suspend and resume operations through the firmware async API. The purpose is to reduce live migration downtime.
The series starts with changing the VQ suspend and resume commands to the async API. After that, the switch is made to issue multiple commands of the same type in parallel.
Then, the an additional improvement is added: keep the notifiers enabled during suspend but make it a NOP. Upon resume make sure that the link state is forwarded. This shaves around 30ms per device constant time.
Finally, use parallel VQ suspend and resume during the CVQ MQ command.
For 1 vDPA device x 32 VQs (16 VQPs), on a large VM (256 GB RAM, 32 CPUs x 2 threads per core), the improvements are:
+-------------------+--------+--------+-----------+ | operation | Before | After | Reduction | |-------------------+--------+--------+-----------| | mlx5_vdpa_suspend | 37 ms | 2.5 ms | 14x | | mlx5_vdpa_resume | 16 ms | 5 ms | 3x | +-------------------+--------+--------+-----------+
---
v2:
- Changed to parallel VQ suspend/resume during CVQ MQ command. Support added in the last 2 patches.
- Made the fw async command more generic and moved it to resources.c. Did that because the following series (parallel mkey ops) needs this code as well.
Dropped Acked-by from Eugenio on modified patches.
- Fixed kfree -> kvfree.
- Removed extra newline caught during review.
- As discussed in the v1, the series can be pulled in completely in the vhost tree [0]. The mlx5_core patch was reviewed by Tariq who is also a maintainer for mlx5_core.
Dragos Tatulea (10):
net/mlx5: Support throttled commands from async API vdpa/mlx5: Introduce error logging function vdpa/mlx5: Introduce async fw command wrapper vdpa/mlx5: Use async API for vq query command vdpa/mlx5: Use async API for vq modify commands vdpa/mlx5: Parallelize device suspend
vdpa/mlx5: Parallelize device resume
vdpa/mlx5: Keep notifiers during suspend but ignore vdpa/mlx5: Small improvement for change_num_qps() vdpa/mlx5: Parallelize VQ suspend/resume for CVQ MQ command
drivers/net/ethernet/mellanox/mlx5/core/cmd.c | 21 +- drivers/vdpa/mlx5/core/mlx5_vdpa.h | 22 + drivers/vdpa/mlx5/core/resources.c | 73 ++++ drivers/vdpa/mlx5/net/mlx5_vnet.c | 396 +++++++++++------- 4 files changed, 361 insertions(+), 151 deletions(-)
Signature · 1 line
2.45.1
Dragos Tatulea
maintainerPATCH v2 1/10 net/mlx5: Support throttled commands from async API
Currently, commands that qualify as throttled can't be used via the async API. That's due to the fact that the throttle semaphore can sleep but the async API can't.
This patch allows throttling in the async API by using the tentative variant of the semaphore and upon failure (semaphore at 0) returns EBUSY to signal to the caller that they need to wait for the completion of previously issued commands.
Furthermore, make sure that the semaphore is released in the callback.
--- drivers/net/ethernet/mellanox/mlx5/core/cmd.c | 21 ++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/cmd.c +16 -5
@@ -1882,10 +1882,12 @@ static int cmd_exec(struct mlx5_core_dev *dev, void *in, int in_size, void *out, throttle_op =mlx5_cmd_is_throttle_opcode (opcode);if (throttle_op) { -/* atomic context may not sleep */ -if (callback) -return -EINVAL ; -down (&dev->cmd.vars.throttle_sem); +if (callback) { +if (down_trylock (&dev->cmd.vars.throttle_sem)) +return -EBUSY ; + }else { +down (&dev->cmd.vars.throttle_sem); + } } pages_queue =is_manage_pages (in);
@@ -2091,10 +2093,19 @@ static void mlx5_cmd_exec_cb_handler(int status, void *_work) {struct mlx5_async_work *work = _work;struct mlx5_async_ctx *ctx; +struct mlx5_core_dev *dev; +u16 opcode; ctx = work->ctx; - status =cmd_status_err (ctx->dev, status, work->opcode, work->op_mod, work->out); + dev = ctx->dev; + opcode = work->opcode; + status =cmd_status_err (dev, status, work->opcode, work->op_mod, work->out); work->user_callback (status, work); +/* Can't access "work" from this point on. It could have been freed in +* the callback. +*/ +if (mlx5_cmd_is_throttle_opcode (opcode)) +up (&dev->cmd.vars.throttle_sem);if (atomic_dec_and_test (&ctx->num_inflight))complete (&ctx->inflight_done); }
Signature · 1 line
2.45.1
Dragos Tatulea
maintainerRe: [PATCH mlx5-vhost v2 01/10] net/mlx5: Support throttled commands from async API
Quoted 9 lines · Dragos Tatulea
Currently, commands that qualify as throttled can't be used via the async API. That's due to the fact that the throttle semaphore can sleep but the async API can't.
This patch allows throttling in the async API by using the tentative variant of the semaphore and upon failure (semaphore at 0) returns EBUSY to signal to the caller that they need to wait for the completion of previously issued commands.
Quoted 5 lines
Furthermore, make sure that the semaphore is released in the callback.
Signed-off-by Dragos Tatulea <[email protected]>Cc Leon Romanovsky <[email protected]>Reviewed-by Tariq Toukan <[email protected]>
Same reminder as in v1: Tariq is the maintainer for mlx5 so his review also counts as Acked-by.
Thanks,
Dragos
Quoted 46 lines
--- drivers/net/ethernet/mellanox/mlx5/core/cmd.c | 21 ++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-)drivers/net/ethernet/mellanox/mlx5/core/cmd.c +16
-5@@ -1882,10 +1882,12 @@ static int cmd_exec(struct mlx5_core_dev *dev, void *in, int in_size, void *out, throttle_op =mlx5_cmd_is_throttle_opcode (opcode);if (throttle_op) { -/* atomic context may not sleep */ -if (callback) -return -EINVAL ; -down (&dev->cmd.vars.throttle_sem); +if (callback) { +if (down_trylock (&dev->cmd.vars.throttle_sem)) +return -EBUSY ; + }else { +down (&dev->cmd.vars.throttle_sem); + } } pages_queue =is_manage_pages (in);@@ -2091,10 +2093,19 @@ static void mlx5_cmd_exec_cb_handler(int status, void *_work) {struct mlx5_async_work *work = _work;struct mlx5_async_ctx *ctx; +struct mlx5_core_dev *dev; +u16 opcode; ctx = work->ctx; - status =cmd_status_err (ctx->dev, status, work->opcode, work->op_mod, work->out); + dev = ctx->dev; + opcode = work->opcode; + status =cmd_status_err (dev, status, work->opcode, work->op_mod, work->out); work->user_callback (status, work); +/* Can't access "work" from this point on. It could have been freed in +* the callback. +*/ +if (mlx5_cmd_is_throttle_opcode (opcode)) +up (&dev->cmd.vars.throttle_sem);if (atomic_dec_and_test (&ctx->num_inflight))complete (&ctx->inflight_done); }
Eugenio Perez Martin
maintainerRe: [PATCH mlx5-vhost v2 01/10] net/mlx5: Support throttled commands from async API
Quoted 13 lines · Dragos Tatulea
Quoted 9 lines · Dragos Tatulea
Currently, commands that qualify as throttled can't be used via the async API. That's due to the fact that the throttle semaphore can sleep but the async API can't.
This patch allows throttling in the async API by using the tentative variant of the semaphore and upon failure (semaphore at 0) returns EBUSY to signal to the caller that they need to wait for the completion of previously issued commands.
Quoted 8 lines
Quoted 5 lines
Furthermore, make sure that the semaphore is released in the callback.
Signed-off-by Dragos Tatulea <[email protected]>Cc Leon Romanovsky <[email protected]>Reviewed-by Tariq Toukan <[email protected]>Same reminder as in v1: Tariq is the maintainer for mlx5 so his review also counts as Acked-by.
Not sure if it was the case when you send the mail, but this series is
already in the maintainer's branch:
* https://git.kernel.org/pub/scm/linux/kernel/git/mst/vhost.git/
* https://git.kernel.org/pub/scm/linux/kernel/git/mst/vhost.git/commit/?h=vhost&id=691fd851e1bc8ec043798e1ab337305e6291cd6b
Thanks!
Dragos Tatulea
maintainerRe: [PATCH mlx5-vhost v2 01/10] net/mlx5: Support throttled commands from async API
Quoted 19 lines · Eugenio Perez Martin
Quoted 18 lines · Dragos Tatulea
Quoted 14 lines · Dragos Tatulea
Currently, commands that qualify as throttled can't be used via the async API. That's due to the fact that the throttle semaphore can sleep but the async API can't.
This patch allows throttling in the async API by using the tentative variant of the semaphore and upon failure (semaphore at 0) returns EBUSY to signal to the caller that they need to wait for the completion of previously issued commands.
Furthermore, make sure that the semaphore is released in the callback.
Signed-off-by Dragos Tatulea <[email protected]>Cc Leon Romanovsky <[email protected]>Reviewed-by Tariq Toukan <[email protected]>
Quoted 8 lines
Quoted 3 lines
Same reminder as in v1: Tariq is the maintainer for mlx5 so his review also counts as Acked-by.
Not sure if it was the case when you send the mail, but this series is
already in the maintainer's branch:
* https://git.kernel.org/pub/scm/linux/kernel/git/mst/vhost.git/
* https://git.kernel.org/pub/scm/linux/kernel/git/mst/vhost.git/commit/?h=vhost&id=691fd851e1bc8ec043798e1ab337305e6291cd6b
It wasn't. Thanks for the notice!
Thanks,
Dragos
Dragos Tatulea
maintainerPATCH v2 2/10 vdpa/mlx5: Introduce error logging function
In reply to Dragos Tatuleamlx5_vdpa_err() was missing. This patch adds it and uses it in the necessary places.
--- drivers/vdpa/mlx5/core/mlx5_vdpa.h | 5 +++++ drivers/vdpa/mlx5/net/mlx5_vnet.c | 24 ++++++++++++------------ 2 files changed, 17 insertions(+), 12 deletions(-)
drivers/vdpa/mlx5/core/mlx5_vdpa.h +5 -0
@@ -135,6 +135,11 @@ int mlx5_vdpa_update_cvq_iotlb(struct mlx5_vdpa_dev *mvdev,int mlx5_vdpa_create_dma_mr (struct mlx5_vdpa_dev *mvdev);int mlx5_vdpa_reset_mr (struct mlx5_vdpa_dev *mvdev,unsigned int asid); +#define mlx5_vdpa_err (__dev, format, ...) \ + dev_err((__dev)->mdev->device, "%s:%d:(pid %d) error: " format, __func__, __LINE__, \ + current->pid, ##__VA_ARGS__) + +#define mlx5_vdpa_warn (__dev, format, ...) \ dev_warn((__dev)->mdev->device, "%s:%d:(pid %d) warning: " format, __func__, __LINE__, \ current->pid, ##__VA_ARGS__)
drivers/vdpa/mlx5/net/mlx5_vnet.c +12 -12
@@ -1538,13 +1538,13 @@ static int suspend_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mv err =modify_virtqueue_state (ndev, mvq,MLX5_VIRTIO_NET_Q_OBJECT_STATE_SUSPEND );if (err) { -mlx5_vdpa_warn (&ndev->mvdev,"modify to suspend failed, err: %d\n" , err); +mlx5_vdpa_err (&ndev->mvdev,"modify to suspend failed, err: %d\n" , err);return err; } err =query_virtqueue (ndev, mvq, &attr);if (err) { -mlx5_vdpa_warn (&ndev->mvdev,"failed to query virtqueue, err: %d\n" , err); +mlx5_vdpa_err (&ndev->mvdev,"failed to query virtqueue, err: %d\n" , err);return err; }
@@ -1585,7 +1585,7 @@ static int resume_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvq*/ err =modify_virtqueue (ndev, mvq,0 );if (err) { -mlx5_vdpa_warn (&ndev->mvdev, +mlx5_vdpa_err (&ndev->mvdev,"modify vq properties failed for vq %u, err: %d\n" , mvq->index, err);return err;
@@ -1600,15 +1600,15 @@ static int resume_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvqcase MLX5_VIRTIO_NET_Q_OBJECT_STATE_RDY :return 0 ;default : -mlx5_vdpa_warn (&ndev->mvdev,"resume vq %u called from bad state %d\n" , +mlx5_vdpa_err (&ndev->mvdev,"resume vq %u called from bad state %d\n" , mvq->index, mvq->fw_state);return -EINVAL ; } err =modify_virtqueue_state (ndev, mvq,MLX5_VIRTIO_NET_Q_OBJECT_STATE_RDY );if (err) -mlx5_vdpa_warn (&ndev->mvdev,"modify to resume failed for vq %u, err: %d\n" , - mvq->index, err); +mlx5_vdpa_err (&ndev->mvdev,"modify to resume failed for vq %u, err: %d\n" , + mvq->index, err);return err; }
@@ -2002,13 +2002,13 @@ static int setup_steering(struct mlx5_vdpa_net *ndev) ns =mlx5_get_flow_namespace (ndev->mvdev.mdev,MLX5_FLOW_NAMESPACE_BYPASS );if (!ns) { -mlx5_vdpa_warn (&ndev->mvdev,"failed to get flow namespace\n" ); +mlx5_vdpa_err (&ndev->mvdev,"failed to get flow namespace\n" );return -EOPNOTSUPP ; } ndev->rxft =mlx5_create_auto_grouped_flow_table (ns, &ft_attr);if (IS_ERR (ndev->rxft)) { -mlx5_vdpa_warn (&ndev->mvdev,"failed to create flow table\n" ); +mlx5_vdpa_err (&ndev->mvdev,"failed to create flow table\n" );return PTR_ERR (ndev->rxft); }mlx5_vdpa_add_rx_flow_table (ndev);
@@ -2530,7 +2530,7 @@ static int mlx5_vdpa_get_vq_state(struct vdpa_device *vdev, u16 idx, struct vdpa err =query_virtqueue (ndev, mvq, &attr);if (err) { -mlx5_vdpa_warn (mvdev,"failed to query virtqueue\n" ); +mlx5_vdpa_err (mvdev,"failed to query virtqueue\n" );return err; } state->split.avail_index = attr.used_index;
@@ -3189,7 +3189,7 @@ static int mlx5_vdpa_compat_reset(struct vdpa_device *vdev, u32 flags)if ((flags &VDPA_RESET_F_CLEAN_MAP ) &&MLX5_CAP_GEN (mvdev->mdev, umem_uid_0)) {if (mlx5_vdpa_create_dma_mr (mvdev)) -mlx5_vdpa_warn (mvdev,"create MR failed\n" ); +mlx5_vdpa_err (mvdev,"create MR failed\n" ); }if (vq_reset)setup_vq_resources (ndev, false);
@@ -3244,7 +3244,7 @@ static int set_map_data(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, new_mr =mlx5_vdpa_create_mr (mvdev, iotlb);if (IS_ERR (new_mr)) { err =PTR_ERR (new_mr); -mlx5_vdpa_warn (mvdev,"create map failed(%d)\n" , err); +mlx5_vdpa_err (mvdev,"create map failed(%d)\n" , err);return err; } }else {
@@ -3257,7 +3257,7 @@ static int set_map_data(struct mlx5_vdpa_dev *mvdev, struct vhost_iotlb *iotlb, }else { err =mlx5_vdpa_change_map (mvdev, new_mr, asid);if (err) { -mlx5_vdpa_warn (mvdev,"change map failed(%d)\n" , err); +mlx5_vdpa_err (mvdev,"change map failed(%d)\n" , err); goto out_err; } }
Signature · 1 line
2.45.1
Dragos Tatulea
maintainerPATCH v2 3/10 vdpa/mlx5: Introduce async fw command wrapper
In reply to Dragos TatuleaIntroduce a new function mlx5_vdpa_exec_async_cmds() which wraps the mlx5_core async firmware command API in a way that will be used to parallelize certain operation in this driver.
The wrapper deals with the case when mlx5_cmd_exec_cb() returns EBUSY due to the command being throttled.
--- drivers/vdpa/mlx5/core/mlx5_vdpa.h | 15 ++++++ drivers/vdpa/mlx5/core/resources.c | 73 ++++++++++++++++++++++++++++++ 2 files changed, 88 insertions(+)
drivers/vdpa/mlx5/core/mlx5_vdpa.h +15 -0
@@ -105,6 +105,18 @@ struct mlx5_vdpa_dev {bool suspended; }; +struct mlx5_vdpa_async_cmd { +int err; +struct mlx5_async_work cb_work; +struct completion cmd_done; + +void *in; +size_t inlen; + +void *out; +size_t outlen; +}; +int mlx5_vdpa_create_tis (struct mlx5_vdpa_dev *mvdev,void *in,u32 *tisn);void mlx5_vdpa_destroy_tis (struct mlx5_vdpa_dev *mvdev,u32 tisn);int mlx5_vdpa_create_rqt (struct mlx5_vdpa_dev *mvdev,void *in,int inlen,u32 *rqtn);
@@ -134,6 +146,9 @@ int mlx5_vdpa_update_cvq_iotlb(struct mlx5_vdpa_dev *mvdev, unsigned int asid);int mlx5_vdpa_create_dma_mr (struct mlx5_vdpa_dev *mvdev);int mlx5_vdpa_reset_mr (struct mlx5_vdpa_dev *mvdev,unsigned int asid); +int mlx5_vdpa_exec_async_cmds (struct mlx5_vdpa_dev *mvdev, +struct mlx5_vdpa_async_cmd *cmds, +int num_cmds);#define mlx5_vdpa_err (__dev, format, ...) \ dev_err((__dev)->mdev->device, "%s:%d:(pid %d) error: " format, __func__, __LINE__, \
drivers/vdpa/mlx5/core/resources.c +73 -0
76 lines not on this page · raw message
Signature · 1 line
2.45.1
Re: [PATCH vhost v2 03/10] vdpa/mlx5: Introduce async fw command wrapper
Quoted 11 lines · Dragos Tatulea
Introduce a new function
mlx5_vdpa_exec_async_cmds()which wraps themlx5_coreasync firmware command API in a way that will be used to parallelize certain operation in this driver.The wrapper deals with the case when
mlx5_cmd_exec_cb()returns EBUSY due to the command being throttled.Signed-off-by Dragos Tatulea <[email protected]>Reviewed-by Tariq Toukan <[email protected]>
Quoted 122 lines
--- drivers/vdpa/mlx5/core/mlx5_vdpa.h | 15 ++++++ drivers/vdpa/mlx5/core/resources.c | 73 ++++++++++++++++++++++++++++++ 2 files changed, 88 insertions(+)drivers/vdpa/mlx5/core/mlx5_vdpa.h +15
-0@@ -105,6 +105,18 @@ struct mlx5_vdpa_dev {bool suspended; }; +struct mlx5_vdpa_async_cmd { +int err; +struct mlx5_async_work cb_work; +struct completion cmd_done; + +void *in; +size_t inlen; + +void *out; +size_t outlen; +}; +int mlx5_vdpa_create_tis (struct mlx5_vdpa_dev *mvdev,void *in,u32 *tisn);void mlx5_vdpa_destroy_tis (struct mlx5_vdpa_dev *mvdev,u32 tisn);int mlx5_vdpa_create_rqt (struct mlx5_vdpa_dev *mvdev,void *in,int inlen,u32 *rqtn);@@ -134,6 +146,9 @@ int mlx5_vdpa_update_cvq_iotlb(struct mlx5_vdpa_dev *mvdev, unsigned int asid);int mlx5_vdpa_create_dma_mr (struct mlx5_vdpa_dev *mvdev);int mlx5_vdpa_reset_mr (struct mlx5_vdpa_dev *mvdev,unsigned int asid); +int mlx5_vdpa_exec_async_cmds (struct mlx5_vdpa_dev *mvdev, +struct mlx5_vdpa_async_cmd *cmds, +int num_cmds);#define mlx5_vdpa_err (__dev, format, ...) \ dev_err((__dev)->mdev->device, "%s:%d:(pid %d) error: " format, __func__, __LINE__, \drivers/vdpa/mlx5/core/resources.c +73
-1@@ -321,3 +321,76 @@ void mlx5_vdpa_free_resources(struct mlx5_vdpa_dev *mvdev)mutex_destroy (&mvdev->mr_mtx); res->valid = false; } + +static void virtqueue_cmd_callback (int status,struct mlx5_async_work *context) +{ +struct mlx5_vdpa_async_cmd *cmd = + container_of(context, struct mlx5_vdpa_async_cmd, cb_work); + + cmd->err =mlx5_cmd_check (context->ctx->dev, status, cmd->in, cmd->out); +complete (&cmd->cmd_done); +} + +static int issue_async_cmd (struct mlx5_vdpa_dev *mvdev, +struct mlx5_vdpa_async_cmd *cmds, +int issued, +int *completed) + +{ +struct mlx5_vdpa_async_cmd *cmd = &cmds[issued]; +int err; + +retry: + err =mlx5_cmd_exec_cb (&mvdev->async_ctx, + cmd->in, cmd->inlen, + cmd->out, cmd->outlen, + virtqueue_cmd_callback, + &cmd->cb_work); +if (err == -EBUSY ) { +if (*completed < issued) { +/* Throttled by own commands: wait for oldest completion. */ +wait_for_completion (&cmds[*completed].cmd_done); + (*completed)++; + + goto retry; + }else { +/* Throttled by external commands: switch to sync api. */ + err =mlx5_cmd_exec (mvdev->mdev, + cmd->in, cmd->inlen, + cmd->out, cmd->outlen); +if (!err) + (*completed)++; + } + } + +return err; +} + +int mlx5_vdpa_exec_async_cmds (struct mlx5_vdpa_dev *mvdev, +struct mlx5_vdpa_async_cmd *cmds, +int num_cmds) +{ +int completed =0 ; +int issued =0 ; +int err =0 ; + +for (int i =0 ; i < num_cmds; i++) +init_completion (&cmds[i].cmd_done); + +while (issued < num_cmds) { + + err =issue_async_cmd (mvdev, cmds, issued, &completed); +if (err) { +mlx5_vdpa_err (mvdev,"error issuing command %d of %d: %d\n" , + issued, num_cmds, err); +break ; + } + + issued++; + } + +while (completed < issued) +wait_for_completion (&cmds[completed++].cmd_done); + +return err; +} --2.45.1
Dragos Tatulea
maintainerPATCH v2 4/10 vdpa/mlx5: Use async API for vq query command
In reply to Dragos TatuleaSwitch firmware vq query command to be issued via the async API to allow future parallelization.
For now the command is still serial but the infrastructure is there to issue commands in parallel, including ratelimiting the number of issued async commands to firmware.
A later patch will switch to issuing more commands at a time.
--- drivers/vdpa/mlx5/core/mlx5_vdpa.h | 2 + drivers/vdpa/mlx5/net/mlx5_vnet.c | 101 ++++++++++++++++++++++------- 2 files changed, 78 insertions(+), 25 deletions(-)
drivers/vdpa/mlx5/core/mlx5_vdpa.h +2 -0
@@ -103,6 +103,8 @@ struct mlx5_vdpa_dev {struct workqueue_struct *wq;unsigned int group2asid[MLX5_VDPA_NUMVQ_GROUPS ];bool suspended; + +struct mlx5_async_ctx async_ctx; };struct mlx5_vdpa_async_cmd {
drivers/vdpa/mlx5/net/mlx5_vnet.c +76 -25
149 lines not on this page · raw message
Signature · 1 line
2.45.1
Re: [PATCH vhost v2 04/10] vdpa/mlx5: Use async API for vq query command
Quoted 4 lines · Dragos Tatulea
Switch firmware vq query command to be issued via the async API to allow future parallelization.
Quoted 8 lines
For now the command is still serial but the infrastructure is there to issue commands in parallel, including ratelimiting the number of issued async commands to firmware.
A later patch will switch to issuing more commands at a time.
Signed-off-by Dragos Tatulea <[email protected]>Reviewed-by Tariq Toukan <[email protected]>
Quoted 180 lines
--- drivers/vdpa/mlx5/core/mlx5_vdpa.h | 2 + drivers/vdpa/mlx5/net/mlx5_vnet.c | 101 ++++++++++++++++++++++------- 2 files changed, 78 insertions(+), 25 deletions(-)drivers/vdpa/mlx5/core/mlx5_vdpa.h +2
-0@@ -103,6 +103,8 @@ struct mlx5_vdpa_dev {struct workqueue_struct *wq;unsigned int group2asid[MLX5_VDPA_NUMVQ_GROUPS ];bool suspended; + +struct mlx5_async_ctx async_ctx; };struct mlx5_vdpa_async_cmd {drivers/vdpa/mlx5/net/mlx5_vnet.c +76
-26@@ -1184,40 +1184,87 @@ struct mlx5_virtq_attr {u16 used_index; }; -static int query_virtqueue (struct mlx5_vdpa_net *ndev,struct mlx5_vdpa_virtqueue *mvq, -struct mlx5_virtq_attr *attr) -{ -int outlen =MLX5_ST_SZ_BYTES (query_virtio_net_q_out); -u32 in[MLX5_ST_SZ_DW (query_virtio_net_q_in)] = {}; -void *out; -void *obj_context; -void *cmd_hdr; -int err; - - out =kzalloc (outlen,GFP_KERNEL ); -if (!out) -return -ENOMEM ; +struct mlx5_virtqueue_query_mem { +u8 in[MLX5_ST_SZ_BYTES (query_virtio_net_q_in)]; +u8 out[MLX5_ST_SZ_BYTES (query_virtio_net_q_out)]; +}; - cmd_hdr =MLX5_ADDR_OF (query_virtio_net_q_in, in, general_obj_in_cmd_hdr); +static void fill_query_virtqueue_cmd (struct mlx5_vdpa_net *ndev, +struct mlx5_vdpa_virtqueue *mvq, +struct mlx5_virtqueue_query_mem *cmd) +{ +void *cmd_hdr =MLX5_ADDR_OF (query_virtio_net_q_in, cmd->in, general_obj_in_cmd_hdr);MLX5_SET (general_obj_in_cmd_hdr, cmd_hdr, opcode,MLX5_CMD_OP_QUERY_GENERAL_OBJECT );MLX5_SET (general_obj_in_cmd_hdr, cmd_hdr, obj_type,MLX5_OBJ_TYPE_VIRTIO_NET_Q );MLX5_SET (general_obj_in_cmd_hdr, cmd_hdr, obj_id, mvq->virtq_id);MLX5_SET (general_obj_in_cmd_hdr, cmd_hdr, uid, ndev->mvdev.res.uid); - err =mlx5_cmd_exec (ndev->mvdev.mdev, in,sizeof (in), out, outlen); -if (err) - goto err_cmd; +} + +static void query_virtqueue_end (struct mlx5_vdpa_net *ndev, +struct mlx5_virtqueue_query_mem *cmd, +struct mlx5_virtq_attr *attr) +{ +void *obj_context =MLX5_ADDR_OF (query_virtio_net_q_out, cmd->out, obj_context); - obj_context =MLX5_ADDR_OF (query_virtio_net_q_out, out, obj_context);memset (attr,0 ,sizeof (*attr)); attr->state =MLX5_GET (virtio_net_q_object, obj_context, state); attr->available_index =MLX5_GET (virtio_net_q_object, obj_context, hw_available_index); attr->used_index =MLX5_GET (virtio_net_q_object, obj_context, hw_used_index); -kfree (out); -return 0 ; +} -err_cmd: -kfree (out); +static int query_virtqueues (struct mlx5_vdpa_net *ndev, +int start_vq, +int num_vqs, +struct mlx5_virtq_attr *attrs) +{ +struct mlx5_vdpa_dev *mvdev = &ndev->mvdev; +struct mlx5_virtqueue_query_mem *cmd_mem; +struct mlx5_vdpa_async_cmd *cmds; +int err =0 ; + +WARN (start_vq + num_vqs > mvdev->max_vqs,"query vq range invalid [%d, %d), max_vqs: %u\n" , + start_vq, start_vq + num_vqs, mvdev->max_vqs); + + cmds =kvcalloc (num_vqs,sizeof (*cmds),GFP_KERNEL ); + cmd_mem =kvcalloc (num_vqs,sizeof (*cmd_mem),GFP_KERNEL ); +if (!cmds || !cmd_mem) { + err = -ENOMEM ; + goto done; + } + +for (int i =0 ; i < num_vqs; i++) { + cmds[i].in = &cmd_mem[i].in; + cmds[i].inlen =sizeof (cmd_mem[i].in); + cmds[i].out = &cmd_mem[i].out; + cmds[i].outlen =sizeof (cmd_mem[i].out); +fill_query_virtqueue_cmd (ndev, &ndev->vqs[start_vq + i], &cmd_mem[i]); + } + + err =mlx5_vdpa_exec_async_cmds (&ndev->mvdev, cmds, num_vqs); +if (err) { +mlx5_vdpa_err (mvdev,"error issuing query cmd for vq range [%d, %d): %d\n" , + start_vq, start_vq + num_vqs, err); + goto done; + } + +for (int i =0 ; i < num_vqs; i++) { +struct mlx5_vdpa_async_cmd *cmd = &cmds[i]; +int vq_idx = start_vq + i; + +if (cmd->err) { +mlx5_vdpa_err (mvdev,"query vq %d failed, err: %d\n" , vq_idx, err); +if (!err) + err = cmd->err; +continue ; + } + +query_virtqueue_end (ndev, &cmd_mem[i], &attrs[i]); + } + +done: +kvfree (cmd_mem); +kvfree (cmds);return err; }@@ -1542,7 +1589,7 @@ static int suspend_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvreturn err; } - err =query_virtqueue (ndev, mvq, &attr); + err =query_virtqueues (ndev, mvq->index,1 , &attr);if (err) {mlx5_vdpa_err (&ndev->mvdev,"failed to query virtqueue, err: %d\n" , err);return err;@@ -2528,7 +2575,7 @@ static int mlx5_vdpa_get_vq_state(struct vdpa_device *vdev, u16 idx, struct vdpareturn 0 ; } - err =query_virtqueue (ndev, mvq, &attr); + err =query_virtqueues (ndev, mvq->index,1 , &attr);if (err) {mlx5_vdpa_err (mvdev,"failed to query virtqueue\n" );return err;@@ -2879,7 +2926,7 @@ static int save_channel_info(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtquint err;if (mvq->initialized) { - err =query_virtqueue (ndev, mvq, &attr); + err =query_virtqueues (ndev, mvq->index,1 , &attr);if (err)return err; }@@ -3854,6 +3901,8 @@ static int mlx5_vdpa_dev_add(struct vdpa_mgmt_dev *v_mdev, const char *name, ndev->rqt_size =1 ; } +mlx5_cmd_init_async_ctx (mdev, &mvdev->async_ctx); + ndev->mvdev.mlx_features = device_features; mvdev->vdev.dma_dev = &mdev->pdev->dev; err =mlx5_vdpa_alloc_resources (&ndev->mvdev);@@ -3935,6 +3984,8 @@ static void mlx5_vdpa_dev_del(struct vdpa_mgmt_dev *v_mdev, struct vdpa_device * mvdev->wq =NULL ;destroy_workqueue (wq); mgtdev->ndev =NULL ; + +mlx5_cmd_cleanup_async_ctx (&mvdev->async_ctx); }static const struct vdpa_mgmtdev_ops mdev_ops = { --2.45.1
Dragos Tatulea
maintainerPATCH v2 5/10 vdpa/mlx5: Use async API for vq modify commands
In reply to Dragos TatuleaSwitch firmware vq modify command to be issued via the async API to allow future parallelization. The new refactored function applies the modify on a range of vqs and waits for their execution to complete.
For now the command is still used in a serial fashion. A later patch will switch to modifying multiple vqs in parallel.
--- drivers/vdpa/mlx5/net/mlx5_vnet.c | 154 ++++++++++++++++++++---------- 1 file changed, 106 insertions(+), 48 deletions(-)
drivers/vdpa/mlx5/net/mlx5_vnet.c +106 -48
@@ -1189,6 +1189,11 @@ struct mlx5_virtqueue_query_mem {u8 out[MLX5_ST_SZ_BYTES (query_virtio_net_q_out)]; }; +struct mlx5_virtqueue_modify_mem { +u8 in[MLX5_ST_SZ_BYTES (modify_virtio_net_q_in)]; +u8 out[MLX5_ST_SZ_BYTES (modify_virtio_net_q_out)]; +}; +static void fill_query_virtqueue_cmd (struct mlx5_vdpa_net *ndev,struct mlx5_vdpa_virtqueue *mvq,struct mlx5_virtqueue_query_mem *cmd)
@@ -1298,51 +1303,30 @@ static bool modifiable_virtqueue_fields(struct mlx5_vdpa_virtqueue *mvq)return true; } -static int modify_virtqueue (struct mlx5_vdpa_net *ndev, -struct mlx5_vdpa_virtqueue *mvq, -int state) +static void fill_modify_virtqueue_cmd (struct mlx5_vdpa_net *ndev, +struct mlx5_vdpa_virtqueue *mvq, +int state, +struct mlx5_virtqueue_modify_mem *cmd) { -int inlen =MLX5_ST_SZ_BYTES (modify_virtio_net_q_in); -u32 out[MLX5_ST_SZ_DW (modify_virtio_net_q_out)] = {};struct mlx5_vdpa_dev *mvdev = &ndev->mvdev;struct mlx5_vdpa_mr *desc_mr =NULL ;struct mlx5_vdpa_mr *vq_mr =NULL ; -bool state_change = false;void *obj_context;void *cmd_hdr;void *vq_ctx; -void *in; -int err; - -if (mvq->fw_state ==MLX5_VIRTIO_NET_Q_OBJECT_NONE ) -return 0 ; - -if (!modifiable_virtqueue_fields (mvq)) -return -EINVAL ; - in =kzalloc (inlen,GFP_KERNEL ); -if (!in) -return -ENOMEM ; - - cmd_hdr =MLX5_ADDR_OF (modify_virtio_net_q_in, in, general_obj_in_cmd_hdr); + cmd_hdr =MLX5_ADDR_OF (modify_virtio_net_q_in, cmd->in, general_obj_in_cmd_hdr);MLX5_SET (general_obj_in_cmd_hdr, cmd_hdr, opcode,MLX5_CMD_OP_MODIFY_GENERAL_OBJECT );MLX5_SET (general_obj_in_cmd_hdr, cmd_hdr, obj_type,MLX5_OBJ_TYPE_VIRTIO_NET_Q );MLX5_SET (general_obj_in_cmd_hdr, cmd_hdr, obj_id, mvq->virtq_id);MLX5_SET (general_obj_in_cmd_hdr, cmd_hdr, uid, ndev->mvdev.res.uid); - obj_context =MLX5_ADDR_OF (modify_virtio_net_q_in, in, obj_context); + obj_context =MLX5_ADDR_OF (modify_virtio_net_q_in, cmd->in, obj_context); vq_ctx =MLX5_ADDR_OF (virtio_net_q_object, obj_context, virtio_q_context); -if (mvq->modified_fields &MLX5_VIRTQ_MODIFY_MASK_STATE ) { -if (!is_valid_state_change (mvq->fw_state, state,is_resumable (ndev))) { - err = -EINVAL ; - goto done; - } - +if (mvq->modified_fields &MLX5_VIRTQ_MODIFY_MASK_STATE )MLX5_SET (virtio_net_q_object, obj_context, state, state); - state_change = true; - }if (mvq->modified_fields &MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_ADDRS ) {MLX5_SET64 (virtio_q, vq_ctx, desc_addr, mvq->desc_addr);
@@ -1388,38 +1372,36 @@ static int modify_virtqueue(struct mlx5_vdpa_net *ndev, }MLX5_SET64 (virtio_net_q_object, obj_context, modify_field_select, mvq->modified_fields); - err =mlx5_cmd_exec (ndev->mvdev.mdev, in, inlen, out,sizeof (out)); -if (err) - goto done; +} -if (state_change) - mvq->fw_state = state; +static void modify_virtqueue_end (struct mlx5_vdpa_net *ndev, +struct mlx5_vdpa_virtqueue *mvq, +int state) +{ +struct mlx5_vdpa_dev *mvdev = &ndev->mvdev;if (mvq->modified_fields &MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_MKEY ) { +unsigned int asid = mvdev->group2asid[MLX5_VDPA_DATAVQ_GROUP ]; +struct mlx5_vdpa_mr *vq_mr = mvdev->mr[asid]; +mlx5_vdpa_put_mr (mvdev, mvq->vq_mr);mlx5_vdpa_get_mr (mvdev, vq_mr); mvq->vq_mr = vq_mr; }if (mvq->modified_fields &MLX5_VIRTQ_MODIFY_MASK_DESC_GROUP_MKEY ) { +unsigned int asid = mvdev->group2asid[MLX5_VDPA_DATAVQ_DESC_GROUP ]; +struct mlx5_vdpa_mr *desc_mr = mvdev->mr[asid]; +mlx5_vdpa_put_mr (mvdev, mvq->desc_mr);mlx5_vdpa_get_mr (mvdev, desc_mr); mvq->desc_mr = desc_mr; } - mvq->modified_fields =0 ; - -done: -kfree (in); -return err; -} +if (mvq->modified_fields &MLX5_VIRTQ_MODIFY_MASK_STATE ) + mvq->fw_state = state; -static int modify_virtqueue_state (struct mlx5_vdpa_net *ndev, -struct mlx5_vdpa_virtqueue *mvq, -unsigned int state) -{ - mvq->modified_fields |=MLX5_VIRTQ_MODIFY_MASK_STATE ; -return modify_virtqueue (ndev, mvq, state); + mvq->modified_fields =0 ; }static int counter_set_alloc (struct mlx5_vdpa_net *ndev,struct mlx5_vdpa_virtqueue *mvq)
@@ -1572,6 +1554,82 @@ static int setup_vq(struct mlx5_vdpa_net *ndev,return err; } +static int modify_virtqueues (struct mlx5_vdpa_net *ndev,int start_vq,int num_vqs,int state) +{ +struct mlx5_vdpa_dev *mvdev = &ndev->mvdev; +struct mlx5_virtqueue_modify_mem *cmd_mem; +struct mlx5_vdpa_async_cmd *cmds; +int err =0 ; + +WARN (start_vq + num_vqs > mvdev->max_vqs,"modify vq range invalid [%d, %d), max_vqs: %u\n" , + start_vq, start_vq + num_vqs, mvdev->max_vqs); + + cmds =kvcalloc (num_vqs,sizeof (*cmds),GFP_KERNEL ); + cmd_mem =kvcalloc (num_vqs,sizeof (*cmd_mem),GFP_KERNEL ); +if (!cmds || !cmd_mem) { + err = -ENOMEM ; + goto done; + } + +for (int i =0 ; i < num_vqs; i++) { +struct mlx5_vdpa_async_cmd *cmd = &cmds[i]; +struct mlx5_vdpa_virtqueue *mvq; +int vq_idx = start_vq + i; + + mvq = &ndev->vqs[vq_idx]; + +if (!modifiable_virtqueue_fields (mvq)) { + err = -EINVAL ; + goto done; + } + +if (mvq->fw_state != state) { +if (!is_valid_state_change (mvq->fw_state, state,is_resumable (ndev))) { + err = -EINVAL ; + goto done; + } + + mvq->modified_fields |=MLX5_VIRTQ_MODIFY_MASK_STATE ; + } + + cmd->in = &cmd_mem[i].in; + cmd->inlen =sizeof (cmd_mem[i].in); + cmd->out = &cmd_mem[i].out; + cmd->outlen =sizeof (cmd_mem[i].out); +fill_modify_virtqueue_cmd (ndev, mvq, state, &cmd_mem[i]); + } + + err =mlx5_vdpa_exec_async_cmds (&ndev->mvdev, cmds, num_vqs); +if (err) { +mlx5_vdpa_err (mvdev,"error issuing modify cmd for vq range [%d, %d)\n" , + start_vq, start_vq + num_vqs); + goto done; + } + +for (int i =0 ; i < num_vqs; i++) { +struct mlx5_vdpa_async_cmd *cmd = &cmds[i]; +struct mlx5_vdpa_virtqueue *mvq; +int vq_idx = start_vq + i; + + mvq = &ndev->vqs[vq_idx]; + +if (cmd->err) { +mlx5_vdpa_err (mvdev,"modify vq %d failed, state: %d -> %d, err: %d\n" , + vq_idx, mvq->fw_state, state, err); +if (!err) + err = cmd->err; +continue ; + } + +modify_virtqueue_end (ndev, mvq, state); + } + +done: +kvfree (cmd_mem); +kvfree (cmds); +return err; +} +static int suspend_vq (struct mlx5_vdpa_net *ndev,struct mlx5_vdpa_virtqueue *mvq) {struct mlx5_virtq_attr attr;
@@ -1583,7 +1641,7 @@ static int suspend_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvif (mvq->fw_state !=MLX5_VIRTIO_NET_Q_OBJECT_STATE_RDY )return 0 ; - err =modify_virtqueue_state (ndev, mvq,MLX5_VIRTIO_NET_Q_OBJECT_STATE_SUSPEND ); + err =modify_virtqueues (ndev, mvq->index,1 ,MLX5_VIRTIO_NET_Q_OBJECT_STATE_SUSPEND );if (err) {mlx5_vdpa_err (&ndev->mvdev,"modify to suspend failed, err: %d\n" , err);return err;
@@ -1630,7 +1688,7 @@ static int resume_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvq/* Due to a FW quirk we need to modify the VQ fields first then change state. * This should be fixed soon. After that, a single command can be used. */ - err =modify_virtqueue (ndev, mvq,0 ); + err =modify_virtqueues (ndev, mvq->index,1 , mvq->fw_state);if (err) { mlx5_vdpa_err(&ndev->mvdev, "modify vq properties failed for vq %u, err: %d\n",
@@ -1652,7 +1710,7 @@ static int resume_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvqreturn -EINVAL ; } - err =modify_virtqueue_state (ndev, mvq,MLX5_VIRTIO_NET_Q_OBJECT_STATE_RDY ); + err =modify_virtqueues (ndev, mvq->index,1 ,MLX5_VIRTIO_NET_Q_OBJECT_STATE_RDY );if (err)mlx5_vdpa_err (&ndev->mvdev,"modify to resume failed for vq %u, err: %d\n" , mvq->index, err);
Signature · 1 line
2.45.1
Re: [PATCH vhost v2 05/10] vdpa/mlx5: Use async API for vq modify commands
Quoted 10 lines · Dragos Tatulea
Switch firmware vq modify command to be issued via the async API to allow future parallelization. The new refactored function applies the modify on a range of vqs and waits for their execution to complete.
For now the command is still used in a serial fashion. A later patch will switch to modifying multiple vqs in parallel.
Signed-off-by Dragos Tatulea <[email protected]>Reviewed-by Tariq Toukan <[email protected]>
Quoted 246 lines
--- drivers/vdpa/mlx5/net/mlx5_vnet.c | 154 ++++++++++++++++++++---------- 1 file changed, 106 insertions(+), 48 deletions(-)drivers/vdpa/mlx5/net/mlx5_vnet.c +106
-49@@ -1189,6 +1189,11 @@ struct mlx5_virtqueue_query_mem {u8 out[MLX5_ST_SZ_BYTES (query_virtio_net_q_out)]; }; +struct mlx5_virtqueue_modify_mem { +u8 in[MLX5_ST_SZ_BYTES (modify_virtio_net_q_in)]; +u8 out[MLX5_ST_SZ_BYTES (modify_virtio_net_q_out)]; +}; +static void fill_query_virtqueue_cmd (struct mlx5_vdpa_net *ndev,struct mlx5_vdpa_virtqueue *mvq,struct mlx5_virtqueue_query_mem *cmd)@@ -1298,51 +1303,30 @@ static bool modifiable_virtqueue_fields(struct mlx5_vdpa_virtqueue *mvq)return true; } -static int modify_virtqueue (struct mlx5_vdpa_net *ndev, -struct mlx5_vdpa_virtqueue *mvq, -int state) +static void fill_modify_virtqueue_cmd (struct mlx5_vdpa_net *ndev, +struct mlx5_vdpa_virtqueue *mvq, +int state, +struct mlx5_virtqueue_modify_mem *cmd) { -int inlen =MLX5_ST_SZ_BYTES (modify_virtio_net_q_in); -u32 out[MLX5_ST_SZ_DW (modify_virtio_net_q_out)] = {};struct mlx5_vdpa_dev *mvdev = &ndev->mvdev;struct mlx5_vdpa_mr *desc_mr =NULL ;struct mlx5_vdpa_mr *vq_mr =NULL ; -bool state_change = false;void *obj_context;void *cmd_hdr;void *vq_ctx; -void *in; -int err; - -if (mvq->fw_state ==MLX5_VIRTIO_NET_Q_OBJECT_NONE ) -return 0 ; - -if (!modifiable_virtqueue_fields (mvq)) -return -EINVAL ; - in =kzalloc (inlen,GFP_KERNEL ); -if (!in) -return -ENOMEM ; - - cmd_hdr =MLX5_ADDR_OF (modify_virtio_net_q_in, in, general_obj_in_cmd_hdr); + cmd_hdr =MLX5_ADDR_OF (modify_virtio_net_q_in, cmd->in, general_obj_in_cmd_hdr);MLX5_SET (general_obj_in_cmd_hdr, cmd_hdr, opcode,MLX5_CMD_OP_MODIFY_GENERAL_OBJECT );MLX5_SET (general_obj_in_cmd_hdr, cmd_hdr, obj_type,MLX5_OBJ_TYPE_VIRTIO_NET_Q );MLX5_SET (general_obj_in_cmd_hdr, cmd_hdr, obj_id, mvq->virtq_id);MLX5_SET (general_obj_in_cmd_hdr, cmd_hdr, uid, ndev->mvdev.res.uid); - obj_context =MLX5_ADDR_OF (modify_virtio_net_q_in, in, obj_context); + obj_context =MLX5_ADDR_OF (modify_virtio_net_q_in, cmd->in, obj_context); vq_ctx =MLX5_ADDR_OF (virtio_net_q_object, obj_context, virtio_q_context); -if (mvq->modified_fields &MLX5_VIRTQ_MODIFY_MASK_STATE ) { -if (!is_valid_state_change (mvq->fw_state, state,is_resumable (ndev))) { - err = -EINVAL ; - goto done; - } - +if (mvq->modified_fields &MLX5_VIRTQ_MODIFY_MASK_STATE )MLX5_SET (virtio_net_q_object, obj_context, state, state); - state_change = true; - }if (mvq->modified_fields &MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_ADDRS ) {MLX5_SET64 (virtio_q, vq_ctx, desc_addr, mvq->desc_addr);@@ -1388,38 +1372,36 @@ static int modify_virtqueue(struct mlx5_vdpa_net *ndev, }MLX5_SET64 (virtio_net_q_object, obj_context, modify_field_select, mvq->modified_fields); - err =mlx5_cmd_exec (ndev->mvdev.mdev, in, inlen, out,sizeof (out)); -if (err) - goto done; +} -if (state_change) - mvq->fw_state = state; +static void modify_virtqueue_end (struct mlx5_vdpa_net *ndev, +struct mlx5_vdpa_virtqueue *mvq, +int state) +{ +struct mlx5_vdpa_dev *mvdev = &ndev->mvdev;if (mvq->modified_fields &MLX5_VIRTQ_MODIFY_MASK_VIRTIO_Q_MKEY ) { +unsigned int asid = mvdev->group2asid[MLX5_VDPA_DATAVQ_GROUP ]; +struct mlx5_vdpa_mr *vq_mr = mvdev->mr[asid]; +mlx5_vdpa_put_mr (mvdev, mvq->vq_mr);mlx5_vdpa_get_mr (mvdev, vq_mr); mvq->vq_mr = vq_mr; }if (mvq->modified_fields &MLX5_VIRTQ_MODIFY_MASK_DESC_GROUP_MKEY ) { +unsigned int asid = mvdev->group2asid[MLX5_VDPA_DATAVQ_DESC_GROUP ]; +struct mlx5_vdpa_mr *desc_mr = mvdev->mr[asid]; +mlx5_vdpa_put_mr (mvdev, mvq->desc_mr);mlx5_vdpa_get_mr (mvdev, desc_mr); mvq->desc_mr = desc_mr; } - mvq->modified_fields =0 ; - -done: -kfree (in); -return err; -} +if (mvq->modified_fields &MLX5_VIRTQ_MODIFY_MASK_STATE ) + mvq->fw_state = state; -static int modify_virtqueue_state (struct mlx5_vdpa_net *ndev, -struct mlx5_vdpa_virtqueue *mvq, -unsigned int state) -{ - mvq->modified_fields |=MLX5_VIRTQ_MODIFY_MASK_STATE ; -return modify_virtqueue (ndev, mvq, state); + mvq->modified_fields =0 ; }static int counter_set_alloc (struct mlx5_vdpa_net *ndev,struct mlx5_vdpa_virtqueue *mvq)@@ -1572,6 +1554,82 @@ static int setup_vq(struct mlx5_vdpa_net *ndev,return err; } +static int modify_virtqueues (struct mlx5_vdpa_net *ndev,int start_vq,int num_vqs,int state) +{ +struct mlx5_vdpa_dev *mvdev = &ndev->mvdev; +struct mlx5_virtqueue_modify_mem *cmd_mem; +struct mlx5_vdpa_async_cmd *cmds; +int err =0 ; + +WARN (start_vq + num_vqs > mvdev->max_vqs,"modify vq range invalid [%d, %d), max_vqs: %u\n" , + start_vq, start_vq + num_vqs, mvdev->max_vqs); + + cmds =kvcalloc (num_vqs,sizeof (*cmds),GFP_KERNEL ); + cmd_mem =kvcalloc (num_vqs,sizeof (*cmd_mem),GFP_KERNEL ); +if (!cmds || !cmd_mem) { + err = -ENOMEM ; + goto done; + } + +for (int i =0 ; i < num_vqs; i++) { +struct mlx5_vdpa_async_cmd *cmd = &cmds[i]; +struct mlx5_vdpa_virtqueue *mvq; +int vq_idx = start_vq + i; + + mvq = &ndev->vqs[vq_idx]; + +if (!modifiable_virtqueue_fields (mvq)) { + err = -EINVAL ; + goto done; + } + +if (mvq->fw_state != state) { +if (!is_valid_state_change (mvq->fw_state, state,is_resumable (ndev))) { + err = -EINVAL ; + goto done; + } + + mvq->modified_fields |=MLX5_VIRTQ_MODIFY_MASK_STATE ; + } + + cmd->in = &cmd_mem[i].in; + cmd->inlen =sizeof (cmd_mem[i].in); + cmd->out = &cmd_mem[i].out; + cmd->outlen =sizeof (cmd_mem[i].out); +fill_modify_virtqueue_cmd (ndev, mvq, state, &cmd_mem[i]); + } + + err =mlx5_vdpa_exec_async_cmds (&ndev->mvdev, cmds, num_vqs); +if (err) { +mlx5_vdpa_err (mvdev,"error issuing modify cmd for vq range [%d, %d)\n" , + start_vq, start_vq + num_vqs); + goto done; + } + +for (int i =0 ; i < num_vqs; i++) { +struct mlx5_vdpa_async_cmd *cmd = &cmds[i]; +struct mlx5_vdpa_virtqueue *mvq; +int vq_idx = start_vq + i; + + mvq = &ndev->vqs[vq_idx]; + +if (cmd->err) { +mlx5_vdpa_err (mvdev,"modify vq %d failed, state: %d -> %d, err: %d\n" , + vq_idx, mvq->fw_state, state, err); +if (!err) + err = cmd->err; +continue ; + } + +modify_virtqueue_end (ndev, mvq, state); + } + +done: +kvfree (cmd_mem); +kvfree (cmds); +return err; +} +static int suspend_vq (struct mlx5_vdpa_net *ndev,struct mlx5_vdpa_virtqueue *mvq) {struct mlx5_virtq_attr attr;@@ -1583,7 +1641,7 @@ static int suspend_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvif (mvq->fw_state !=MLX5_VIRTIO_NET_Q_OBJECT_STATE_RDY )return 0 ; - err =modify_virtqueue_state (ndev, mvq,MLX5_VIRTIO_NET_Q_OBJECT_STATE_SUSPEND ); + err =modify_virtqueues (ndev, mvq->index,1 ,MLX5_VIRTIO_NET_Q_OBJECT_STATE_SUSPEND );if (err) {mlx5_vdpa_err (&ndev->mvdev,"modify to suspend failed, err: %d\n" , err);return err;@@ -1630,7 +1688,7 @@ static int resume_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvq/* Due to a FW quirk we need to modify the VQ fields first then change state. * This should be fixed soon. After that, a single command can be used. */ - err =modify_virtqueue (ndev, mvq,0 ); + err =modify_virtqueues (ndev, mvq->index,1 , mvq->fw_state);if (err) { mlx5_vdpa_err(&ndev->mvdev, "modify vq properties failed for vq %u, err: %d\n",@@ -1652,7 +1710,7 @@ static int resume_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvqreturn -EINVAL ; } - err =modify_virtqueue_state (ndev, mvq,MLX5_VIRTIO_NET_Q_OBJECT_STATE_RDY ); + err =modify_virtqueues (ndev, mvq->index,1 ,MLX5_VIRTIO_NET_Q_OBJECT_STATE_RDY );if (err)mlx5_vdpa_err (&ndev->mvdev,"modify to resume failed for vq %u, err: %d\n" , mvq->index, err); --2.45.1
Dragos Tatulea
maintainerPATCH v2 6/10 vdpa/mlx5: Parallelize device suspend
In reply to Dragos TatuleaCurrently device suspend works on vqs serially. Building up on previous changes that converted vq operations to the async api, this patch parallelizes the device suspend:
1) Suspend all active vqs parallel.
2) Query suspended vqs in parallel.
For 1 vDPA device x 32 VQs (16 VQPs) attached to a large VM (256 GB RAM, 32 CPUs x 2 threads per core), the device suspend time is reduced from ~37 ms to ~13 ms.
A later patch will remove the link unregister operation which will make it even faster.
--- drivers/vdpa/mlx5/net/mlx5_vnet.c | 56 ++++++++++++++++--------------- 1 file changed, 29 insertions(+), 27 deletions(-)
drivers/vdpa/mlx5/net/mlx5_vnet.c +29 -27
@@ -1630,49 +1630,51 @@ static int modify_virtqueues(struct mlx5_vdpa_net *ndev, int start_vq, int num_vreturn err; } -static int suspend_vq (struct mlx5_vdpa_net *ndev,struct mlx5_vdpa_virtqueue *mvq) +static int suspend_vqs (struct mlx5_vdpa_net *ndev,int start_vq,int num_vqs) { -struct mlx5_virtq_attr attr; +struct mlx5_vdpa_virtqueue *mvq; +struct mlx5_virtq_attr *attrs; +int vq_idx, i;int err; +if (start_vq >= ndev->cur_num_vqs) +return -EINVAL ; + + mvq = &ndev->vqs[start_vq];if (!mvq->initialized)return 0 ;if (mvq->fw_state !=MLX5_VIRTIO_NET_Q_OBJECT_STATE_RDY )return 0 ; - err =modify_virtqueues (ndev, mvq->index,1 ,MLX5_VIRTIO_NET_Q_OBJECT_STATE_SUSPEND ); -if (err) { -mlx5_vdpa_err (&ndev->mvdev,"modify to suspend failed, err: %d\n" , err); -return err; - } - - err =query_virtqueues (ndev, mvq->index,1 , &attr); -if (err) { -mlx5_vdpa_err (&ndev->mvdev,"failed to query virtqueue, err: %d\n" , err); + err =modify_virtqueues (ndev, start_vq, num_vqs,MLX5_VIRTIO_NET_Q_OBJECT_STATE_SUSPEND ); +if (err)return err; - } - - mvq->avail_idx = attr.available_index; - mvq->used_idx = attr.used_index; - -return 0 ; -} -static int suspend_vqs (struct mlx5_vdpa_net *ndev) -{ -int err =0 ; -int i; + attrs =kcalloc (num_vqs,sizeof (struct mlx5_virtq_attr ),GFP_KERNEL ); +if (!attrs) +return -ENOMEM ; -for (i =0 ; i < ndev->cur_num_vqs; i++) { -int local_err =suspend_vq (ndev, &ndev->vqs[i]); + err =query_virtqueues (ndev, start_vq, num_vqs, attrs); +if (err) + goto done; - err = local_err ? local_err : err; +for (i =0 , vq_idx = start_vq; i < num_vqs; i++, vq_idx++) { + mvq = &ndev->vqs[vq_idx]; + mvq->avail_idx = attrs[i].available_index; + mvq->used_idx = attrs[i].used_index; } +done: +kfree (attrs);return err; } +static int suspend_vq (struct mlx5_vdpa_net *ndev,struct mlx5_vdpa_virtqueue *mvq) +{ +return suspend_vqs (ndev, mvq->index,1 ); +} +static int resume_vq (struct mlx5_vdpa_net *ndev,struct mlx5_vdpa_virtqueue *mvq) {int err;
@@ -3053,7 +3055,7 @@ static int mlx5_vdpa_change_map(struct mlx5_vdpa_dev *mvdev,bool teardown = !is_resumable (ndev);int err; -suspend_vqs (ndev); +suspend_vqs (ndev,0 , ndev->cur_num_vqs);if (teardown) { err =save_channels_info (ndev);if (err)
@@ -3606,7 +3608,7 @@ static int mlx5_vdpa_suspend(struct vdpa_device *vdev)down_write (&ndev->reslock);unregister_link_notifier (ndev); - err =suspend_vqs (ndev); + err =suspend_vqs (ndev,0 , ndev->cur_num_vqs);mlx5_vdpa_cvq_suspend (mvdev); mvdev->suspended = true;up_write (&ndev->reslock);
Signature · 1 line
2.45.1
Dragos Tatulea
maintainerPATCH v2 7/10 vdpa/mlx5: Parallelize device resume
In reply to Dragos TatuleaCurrently device resume works on vqs serially. Building up on previous changes that converted vq operations to the async api, this patch parallelizes the device resume.
For 1 vDPA device x 32 VQs (16 VQPs) attached to a large VM (256 GB RAM, 32 CPUs x 2 threads per core), the device resume time is reduced from ~16 ms to ~4.5 ms.
--- drivers/vdpa/mlx5/net/mlx5_vnet.c | 40 +++++++++++-------------------- 1 file changed, 14 insertions(+), 26 deletions(-)
drivers/vdpa/mlx5/net/mlx5_vnet.c +14 -26
@@ -1675,10 +1675,15 @@ static int suspend_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvreturn suspend_vqs (ndev, mvq->index,1 ); } -static int resume_vq (struct mlx5_vdpa_net *ndev,struct mlx5_vdpa_virtqueue *mvq) +static int resume_vqs (struct mlx5_vdpa_net *ndev,int start_vq,int num_vqs) { +struct mlx5_vdpa_virtqueue *mvq;int err; +if (start_vq >= ndev->mvdev.max_vqs) +return -EINVAL ; + + mvq = &ndev->vqs[start_vq];if (!mvq->initialized)return 0 ;
@@ -1690,13 +1695,9 @@ static int resume_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvq/* Due to a FW quirk we need to modify the VQ fields first then change state. * This should be fixed soon. After that, a single command can be used. */ - err =modify_virtqueues (ndev, mvq->index,1 , mvq->fw_state); -if (err) { -mlx5_vdpa_err (&ndev->mvdev, -"modify vq properties failed for vq %u, err: %d\n" , - mvq->index, err); + err =modify_virtqueues (ndev, start_vq, num_vqs, mvq->fw_state); +if (err)return err; - }break ;case MLX5_VIRTIO_NET_Q_OBJECT_STATE_SUSPEND :if (!is_resumable (ndev)) {
@@ -1712,25 +1713,12 @@ static int resume_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvqreturn -EINVAL ; } - err =modify_virtqueues (ndev, mvq->index,1 ,MLX5_VIRTIO_NET_Q_OBJECT_STATE_RDY ); -if (err) -mlx5_vdpa_err (&ndev->mvdev,"modify to resume failed for vq %u, err: %d\n" , - mvq->index, err); - -return err; +return modify_virtqueues (ndev, start_vq, num_vqs,MLX5_VIRTIO_NET_Q_OBJECT_STATE_RDY ); } -static int resume_vqs (struct mlx5_vdpa_net *ndev) +static int resume_vq (struct mlx5_vdpa_net *ndev,struct mlx5_vdpa_virtqueue *mvq) { -int err =0 ; - -for (int i =0 ; i < ndev->cur_num_vqs; i++) { -int local_err =resume_vq (ndev, &ndev->vqs[i]); - - err = local_err ? local_err : err; - } - -return err; +return resume_vqs (ndev, mvq->index,1 ); }static void teardown_vq (struct mlx5_vdpa_net *ndev,struct mlx5_vdpa_virtqueue *mvq)
@@ -3080,7 +3068,7 @@ static int mlx5_vdpa_change_map(struct mlx5_vdpa_dev *mvdev,return err; } -resume_vqs (ndev); +resume_vqs (ndev,0 , ndev->cur_num_vqs);return 0 ; }
@@ -3204,7 +3192,7 @@ static void mlx5_vdpa_set_status(struct vdpa_device *vdev, u8 status)teardown_vq_resources (ndev);if (ndev->setup) { - err =resume_vqs (ndev); + err =resume_vqs (ndev,0 , ndev->cur_num_vqs);if (err) {mlx5_vdpa_warn (mvdev,"failed to resume VQs\n" ); goto err_driver;
@@ -3628,7 +3616,7 @@ static int mlx5_vdpa_resume(struct vdpa_device *vdev)down_write (&ndev->reslock); mvdev->suspended = false; - err =resume_vqs (ndev); + err =resume_vqs (ndev,0 , ndev->cur_num_vqs);register_link_notifier (ndev);up_write (&ndev->reslock);
Signature · 1 line
2.45.1
Dragos Tatulea
maintainerPATCH v2 8/10 vdpa/mlx5: Keep notifiers during suspend but ignore
In reply to Dragos TatuleaUnregistering notifiers is a costly operation. Instead of removing the notifiers during device suspend and adding them back at resume, simply ignore the call when the device is suspended.
At resume time call queue_link_work() to make sure that the device state is propagated in case there were changes.
For 1 vDPA device x 32 VQs (16 VQPs) attached to a large VM (256 GB RAM, 32 CPUs x 2 threads per core), the device suspend time is reduced from ~13 ms to ~2.5 ms.
--- drivers/vdpa/mlx5/net/mlx5_vnet.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/vdpa/mlx5/net/mlx5_vnet.c +4 -2
@@ -2850,6 +2850,9 @@ static int event_handler(struct notifier_block *nb, unsigned long event, void *pstruct mlx5_eqe *eqe = param;int ret =NOTIFY_DONE ; +if (ndev->mvdev.suspended) +return NOTIFY_DONE ; +if (event ==MLX5_EVENT_TYPE_PORT_CHANGE ) {switch (eqe->sub_type) {case MLX5_PORT_CHANGE_SUBTYPE_DOWN :
@@ -3595,7 +3598,6 @@ static int mlx5_vdpa_suspend(struct vdpa_device *vdev)mlx5_vdpa_info (mvdev,"suspending device\n" );down_write (&ndev->reslock); -unregister_link_notifier (ndev); err =suspend_vqs (ndev,0 , ndev->cur_num_vqs);mlx5_vdpa_cvq_suspend (mvdev); mvdev->suspended = true;
@@ -3617,7 +3619,7 @@ static int mlx5_vdpa_resume(struct vdpa_device *vdev)down_write (&ndev->reslock); mvdev->suspended = false; err =resume_vqs (ndev,0 , ndev->cur_num_vqs); -register_link_notifier (ndev); +queue_link_work (ndev);up_write (&ndev->reslock);return err;
Signature · 1 line
2.45.1
Dragos Tatulea
maintainerPATCH v2 9/10 vdpa/mlx5: Small improvement for change_num_qps()
In reply to Dragos Tatuleachange_num_qps() has a lot of multiplications by 2 to convert the number of VQ pairs to number of VQs. This patch simplifies the code by doing the VQP -> VQ count conversion at the beginning in a variable.
--- drivers/vdpa/mlx5/net/mlx5_vnet.c | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-)
drivers/vdpa/mlx5/net/mlx5_vnet.c +11 -10
@@ -2219,16 +2219,17 @@ static virtio_net_ctrl_ack handle_ctrl_mac(struct mlx5_vdpa_dev *mvdev, u8 cmd)static int change_num_qps (struct mlx5_vdpa_dev *mvdev,int newqps) {struct mlx5_vdpa_net *ndev =to_mlx5_vdpa_ndev (mvdev); -int cur_qps = ndev->cur_num_vqs /2 ; +int cur_vqs = ndev->cur_num_vqs; +int new_vqs = newqps *2 ;int err;int i; -if (cur_qps > newqps) { - err =modify_rqt (ndev,2 * newqps); +if (cur_vqs > new_vqs) { + err =modify_rqt (ndev, new_vqs);if (err)return err; -for (i = ndev->cur_num_vqs -1 ; i >=2 * newqps; i--) { +for (i = cur_vqs -1 ; i >= new_vqs; i--) {struct mlx5_vdpa_virtqueue *mvq = &ndev->vqs[i];if (is_resumable (ndev))
@@ -2237,27 +2238,27 @@ static int change_num_qps(struct mlx5_vdpa_dev *mvdev, int newqps)teardown_vq (ndev, mvq); } - ndev->cur_num_vqs =2 * newqps; + ndev->cur_num_vqs = new_vqs; }else { - ndev->cur_num_vqs =2 * newqps; -for (i = cur_qps *2 ; i <2 * newqps; i++) { + ndev->cur_num_vqs = new_vqs; +for (i = cur_vqs; i < new_vqs; i++) {struct mlx5_vdpa_virtqueue *mvq = &ndev->vqs[i]; err = mvq->initialized ?resume_vq (ndev, mvq) :setup_vq (ndev, mvq, true);if (err) goto clean_added; } - err =modify_rqt (ndev,2 * newqps); + err =modify_rqt (ndev, new_vqs);if (err) goto clean_added; }return 0 ; clean_added: -for (--i; i >=2 * cur_qps; --i) +for (--i; i >= cur_vqs; --i)teardown_vq (ndev, &ndev->vqs[i]); - ndev->cur_num_vqs =2 * cur_qps; + ndev->cur_num_vqs = cur_vqs;return err; }
Signature · 1 line
2.45.1
Re: [PATCH vhost v2 09/10] vdpa/mlx5: Small improvement for change_num_qps()
Quoted 8 lines · Dragos Tatulea
change_num_qps()has a lot of multiplications by 2 to convert the number of VQ pairs to number of VQs. This patch simplifies the code by doing the VQP -> VQ count conversion at the beginning in a variable.Signed-off-by Dragos Tatulea <[email protected]>Reviewed-by Tariq Toukan <[email protected]>
Quoted 67 lines
--- drivers/vdpa/mlx5/net/mlx5_vnet.c | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-)drivers/vdpa/mlx5/net/mlx5_vnet.c +11
-11@@ -2219,16 +2219,17 @@ static virtio_net_ctrl_ack handle_ctrl_mac(struct mlx5_vdpa_dev *mvdev, u8 cmd)static int change_num_qps (struct mlx5_vdpa_dev *mvdev,int newqps) {struct mlx5_vdpa_net *ndev =to_mlx5_vdpa_ndev (mvdev); -int cur_qps = ndev->cur_num_vqs /2 ; +int cur_vqs = ndev->cur_num_vqs; +int new_vqs = newqps *2 ;int err;int i; -if (cur_qps > newqps) { - err =modify_rqt (ndev,2 * newqps); +if (cur_vqs > new_vqs) { + err =modify_rqt (ndev, new_vqs);if (err)return err; -for (i = ndev->cur_num_vqs -1 ; i >=2 * newqps; i--) { +for (i = cur_vqs -1 ; i >= new_vqs; i--) {struct mlx5_vdpa_virtqueue *mvq = &ndev->vqs[i];if (is_resumable (ndev))@@ -2237,27 +2238,27 @@ static int change_num_qps(struct mlx5_vdpa_dev *mvdev, int newqps)teardown_vq (ndev, mvq); } - ndev->cur_num_vqs =2 * newqps; + ndev->cur_num_vqs = new_vqs; }else { - ndev->cur_num_vqs =2 * newqps; -for (i = cur_qps *2 ; i <2 * newqps; i++) { + ndev->cur_num_vqs = new_vqs; +for (i = cur_vqs; i < new_vqs; i++) {struct mlx5_vdpa_virtqueue *mvq = &ndev->vqs[i]; err = mvq->initialized ?resume_vq (ndev, mvq) :setup_vq (ndev, mvq, true);if (err) goto clean_added; } - err =modify_rqt (ndev,2 * newqps); + err =modify_rqt (ndev, new_vqs);if (err) goto clean_added; }return 0 ; clean_added: -for (--i; i >=2 * cur_qps; --i) +for (--i; i >= cur_vqs; --i)teardown_vq (ndev, &ndev->vqs[i]); - ndev->cur_num_vqs =2 * cur_qps; + ndev->cur_num_vqs = cur_vqs;return err; } --2.45.1
Dragos Tatulea
maintainerPATCH v2 10/10 vdpa/mlx5: Parallelize VQ suspend/resume for CVQ MQ command
In reply to Dragos Tatuleachange_num_qps() is still suspending/resuming VQs one by one. This change switches to parallel suspend/resume.
When increasing the number of queues the flow has changed a bit for
simplicity: the setup_vq() function will always be called before resume_vqs(). If the VQ is initialized, setup_vq() will exit early. If the VQ is not initialized, setup_vq() will create it and resume_vqs() will resume it.
--- drivers/vdpa/mlx5/net/mlx5_vnet.c | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-)
drivers/vdpa/mlx5/net/mlx5_vnet.c +12 -10
@@ -2229,25 +2229,27 @@ static int change_num_qps(struct mlx5_vdpa_dev *mvdev, int newqps)if (err)return err; -for (i = cur_vqs -1 ; i >= new_vqs; i--) { -struct mlx5_vdpa_virtqueue *mvq = &ndev->vqs[i]; - -if (is_resumable (ndev)) -suspend_vq (ndev, mvq); -else -teardown_vq (ndev, mvq); +if (is_resumable (ndev)) { +suspend_vqs (ndev, new_vqs, cur_vqs - new_vqs); + }else { +for (i = new_vqs; i < cur_vqs; i++) +teardown_vq (ndev, &ndev->vqs[i]); } ndev->cur_num_vqs = new_vqs; }else { ndev->cur_num_vqs = new_vqs; -for (i = cur_vqs; i < new_vqs; i++) { -struct mlx5_vdpa_virtqueue *mvq = &ndev->vqs[i]; - err = mvq->initialized ?resume_vq (ndev, mvq) :setup_vq (ndev, mvq, true); +for (i = cur_vqs; i < new_vqs; i++) { + err =setup_vq (ndev, &ndev->vqs[i], false);if (err) goto clean_added; } + + err =resume_vqs (ndev, cur_vqs, new_vqs - cur_vqs); +if (err) + goto clean_added; + err =modify_rqt (ndev, new_vqs);if (err) goto clean_added;
Signature · 1 line
2.45.1
Re: [PATCH vhost v2 10/10] vdpa/mlx5: Parallelize VQ suspend/resume for CVQ MQ command
Quoted 4 lines · Dragos Tatulea
change_num_qps()is still suspending/resuming VQs one by one. This change switches to parallel suspend/resume.
Quoted 8 lines
When increasing the number of queues the flow has changed a bit for
simplicity: thesetup_vq()function will always be called beforeresume_vqs(). If the VQ is initialized,setup_vq()will exit early. If the VQ is not initialized,setup_vq()will create it andresume_vqs()will resume it.Signed-off-by Dragos Tatulea <[email protected]>Reviewed-by Tariq Toukan <[email protected]>
Quoted 49 lines
--- drivers/vdpa/mlx5/net/mlx5_vnet.c | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-)drivers/vdpa/mlx5/net/mlx5_vnet.c +12
-11@@ -2229,25 +2229,27 @@ static int change_num_qps(struct mlx5_vdpa_dev *mvdev, int newqps)if (err)return err; -for (i = cur_vqs -1 ; i >= new_vqs; i--) { -struct mlx5_vdpa_virtqueue *mvq = &ndev->vqs[i]; - -if (is_resumable (ndev)) -suspend_vq (ndev, mvq); -else -teardown_vq (ndev, mvq); +if (is_resumable (ndev)) { +suspend_vqs (ndev, new_vqs, cur_vqs - new_vqs); + }else { +for (i = new_vqs; i < cur_vqs; i++) +teardown_vq (ndev, &ndev->vqs[i]); } ndev->cur_num_vqs = new_vqs; }else { ndev->cur_num_vqs = new_vqs; -for (i = cur_vqs; i < new_vqs; i++) { -struct mlx5_vdpa_virtqueue *mvq = &ndev->vqs[i]; - err = mvq->initialized ?resume_vq (ndev, mvq) :setup_vq (ndev, mvq, true); +for (i = cur_vqs; i < new_vqs; i++) { + err =setup_vq (ndev, &ndev->vqs[i], false);if (err) goto clean_added; } + + err =resume_vqs (ndev, cur_vqs, new_vqs - cur_vqs); +if (err) + goto clean_added; + err =modify_rqt (ndev, new_vqs);if (err) goto clean_added; --2.45.1
A message that is not in the archive (sent off-list or not yet received): CAJaqyWfwkNUYcMWwG4LthhYEquUYDJPRvHeyh9C_R-ioeFYuXw@mail.gmail.com
Hi Dragos
QE tested this series with mellanox nic, it failed with [1] when booting guest, and host dmesg also will print messages [2]. This bug can be reproduced boot guest with vhost-vdpa device.
[1] qemu) qemu-kvm: vhost VQ 1 ring restore failed: -1: Operation not permitted (1)
qemu-kvm: vhost VQ 0 ring restore failed: -1: Operation not permitted (1)
qemu-kvm: unable to start vhost net: 5: falling back on userspace virtio
qemu-kvm: vhost_set_features failed: Device or resource busy (16)
qemu-kvm: unable to start vhost net: 16: falling back on userspace virtio
[2] Host dmesg:
[ 1406.187977] mlx5_core 0000:0d:00.2: mlx5_vdpa_compat_reset:3267:(pid 8506): performing device reset [ 1406.189221] mlx5_core 0000:0d:00.2: mlx5_vdpa_compat_reset:3267:(pid 8506): performing device reset [ 1406.190354] mlx5_core 0000:0d:00.2: mlx5_vdpa_show_mr_leaks:573:(pid 8506) warning: mkey still alive after resource delete: mr: 000000000c5ccca2, mkey: 0x40000000, refcount: 2 [ 1471.538487] mlx5_core 0000:0d:00.2: cb_timeout_handler:938:(pid 428): cmd[13]: MODIFY_GENERAL_OBJECT(0xa01) Async, timeout. Will cause a leak of a command resource [ 1471.539486] mlx5_core 0000:0d:00.2: cb_timeout_handler:938:(pid 428): cmd[12]: MODIFY_GENERAL_OBJECT(0xa01) Async, timeout. Will cause a leak of a command resource [ 1471.540351] mlx5_core 0000:0d:00.2: modify_virtqueues:1617:(pid 8511) error: modify vq 0 failed, state: 0 -> 0, err: 0 [ 1471.541433] mlx5_core 0000:0d:00.2: modify_virtqueues:1617:(pid 8511) error: modify vq 1 failed, state: 0 -> 0, err: -110 [ 1471.542388] mlx5_core 0000:0d:00.2: mlx5_vdpa_set_status:3203:(pid 8511) warning: failed to resume VQs [ 1471.549778] mlx5_core 0000:0d:00.2: mlx5_vdpa_show_mr_leaks:573:(pid 8511) warning: mkey still alive after resource delete: mr: 000000000c5ccca2, mkey: 0x40000000, refcount: 2 [ 1512.929854] mlx5_core 0000:0d:00.2: mlx5_vdpa_compat_reset:3267:(pid 8565): performing device reset [ 1513.100290] mlx5_core 0000:0d:00.2:
mlx5_vdpa_show_mr_leaks:573:(pid 8565) warning: mkey still alive after resource delete: mr: 000000000c5ccca2, mkey: 0x40000000, refcount: 2
Thanks
Lei
Quoted 61 lines
This series parallelizes the
mlx5_vdpadevice suspend and resume operations through the firmware async API. The purpose is to reduce live migration downtime.The series starts with changing the VQ suspend and resume commands to the async API. After that, the switch is made to issue multiple commands of the same type in parallel.
Then, the an additional improvement is added: keep the notifiers enabled during suspend but make it a NOP. Upon resume make sure that the link state is forwarded. This shaves around 30ms per device constant time.
Finally, use parallel VQ suspend and resume during the CVQ MQ command.
For 1 vDPA device x 32 VQs (16 VQPs), on a large VM (256 GB RAM, 32 CPUs x 2 threads per core), the improvements are:
+-------------------+--------+--------+-----------+ | operation | Before | After | Reduction | |-------------------+--------+--------+-----------| | mlx5_vdpa_suspend | 37 ms | 2.5 ms | 14x | | mlx5_vdpa_resume | 16 ms | 5 ms | 3x | +-------------------+--------+--------+-----------+---
v2:
- Changed to parallel VQ suspend/resume during CVQ MQ command. Support added in the last 2 patches.
- Made the fw async command more generic and moved it toresources.c. Did that because the following series (parallel mkey ops) needs this code as well.
Dropped Acked-by from Eugenio on modified patches.
- Fixed kfree -> kvfree.
- Removed extra newline caught during review.
- As discussed in the v1, the series can be pulled in completely in the vhost tree [0]. Themlx5_corepatch was reviewed by Tariq who is also a maintainer formlx5_core.Dragos Tatulea (10):
net/mlx5: Support throttled commands from async API vdpa/mlx5: Introduce error logging function vdpa/mlx5: Introduce async fw command wrapper vdpa/mlx5: Use async API for vq query command vdpa/mlx5: Use async API for vq modify commands vdpa/mlx5: Parallelize device suspend
vdpa/mlx5: Parallelize device resume
vdpa/mlx5: Keep notifiers during suspend but ignore vdpa/mlx5: Small improvement forchange_num_qps()vdpa/mlx5: Parallelize VQ suspend/resume for CVQ MQ commanddrivers/net/ethernet/mellanox/mlx5/core/cmd.c | 21 +- drivers/vdpa/mlx5/core/mlx5_vdpa.h | 22 + drivers/vdpa/mlx5/core/resources.c | 73 ++++ drivers/vdpa/mlx5/net/mlx5_vnet.c | 396 +++++++++++------- 4 files changed, 361 insertions(+), 151 deletions(-)--
2.45.1
Dragos Tatulea
maintainerHi Lei,
Quoted 39 lines · Lei Yang
Hi Dragos
QE tested this series with mellanox nic, it failed with [1] when booting guest, and host dmesg also will print messages [2]. This bug can be reproduced boot guest with vhost-vdpa device.
[1] qemu) qemu-kvm: vhost VQ 1 ring restore failed: -1: Operation not permitted (1)
qemu-kvm: vhost VQ 0 ring restore failed: -1: Operation not permitted (1)
qemu-kvm: unable to start vhost net: 5: falling back on userspace virtio
qemu-kvm:vhost_set_featuresfailed: Device or resource busy (16)
qemu-kvm: unable to start vhost net: 16: falling back on userspace virtio[2] Host dmesg:
[ 1406.187977] mlx5_core 0000:0d:00.2: mlx5_vdpa_compat_reset:3267:(pid 8506): performing device reset [ 1406.189221] mlx5_core 0000:0d:00.2: mlx5_vdpa_compat_reset:3267:(pid 8506): performing device reset [ 1406.190354] mlx5_core 0000:0d:00.2: mlx5_vdpa_show_mr_leaks:573:(pid 8506) warning: mkey still alive after resource delete: mr: 000000000c5ccca2, mkey: 0x40000000, refcount: 2 [ 1471.538487] mlx5_core 0000:0d:00.2: cb_timeout_handler:938:(pid 428): cmd[13]: MODIFY_GENERAL_OBJECT(0xa01) Async, timeout. Will cause a leak of a command resource [ 1471.539486] mlx5_core 0000:0d:00.2: cb_timeout_handler:938:(pid 428): cmd[12]: MODIFY_GENERAL_OBJECT(0xa01) Async, timeout. Will cause a leak of a command resource [ 1471.540351] mlx5_core 0000:0d:00.2: modify_virtqueues:1617:(pid 8511) error: modify vq 0 failed, state: 0 -> 0, err: 0 [ 1471.541433] mlx5_core 0000:0d:00.2: modify_virtqueues:1617:(pid 8511) error: modify vq 1 failed, state: 0 -> 0, err: -110 [ 1471.542388] mlx5_core 0000:0d:00.2: mlx5_vdpa_set_status:3203:(pid 8511) warning: failed to resume VQs [ 1471.549778] mlx5_core 0000:0d:00.2: mlx5_vdpa_show_mr_leaks:573:(pid 8511) warning: mkey still alive after resource delete: mr: 000000000c5ccca2, mkey: 0x40000000, refcount: 2 [ 1512.929854] mlx5_core 0000:0d:00.2: mlx5_vdpa_compat_reset:3267:(pid 8565): performing device reset [ 1513.100290] mlx5_core 0000:0d:00.2:
Quoted 3 lines
mlx5_vdpa_show_mr_leaks:573:(pid 8565) warning: mkey still alive after resource delete: mr:
000000000c5ccca2, mkey:0x40000000, refcount: 2
Can you provide more details about the qemu version and the vdpa device options used?
Also, which FW version are you using? There is a relevant bug in FW 22.41.1000 which was fixed in the latest FW (22.42.1000). Did you encounter any FW syndromes in the host dmesg log?
Thanks,
Dragos
Lei Yang
Tested-byQuoted 46 lines · Dragos Tatulea
Hi Lei,
Quoted 42 lines · Lei Yang
Hi Dragos
QE tested this series with mellanox nic, it failed with [1] when booting guest, and host dmesg also will print messages [2]. This bug can be reproduced boot guest with vhost-vdpa device.
[1] qemu) qemu-kvm: vhost VQ 1 ring restore failed: -1: Operation not permitted (1)
qemu-kvm: vhost VQ 0 ring restore failed: -1: Operation not permitted (1)
qemu-kvm: unable to start vhost net: 5: falling back on userspace virtio
qemu-kvm:vhost_set_featuresfailed: Device or resource busy (16)
qemu-kvm: unable to start vhost net: 16: falling back on userspace virtio[2] Host dmesg:
[ 1406.187977] mlx5_core 0000:0d:00.2: mlx5_vdpa_compat_reset:3267:(pid 8506): performing device reset [ 1406.189221] mlx5_core 0000:0d:00.2: mlx5_vdpa_compat_reset:3267:(pid 8506): performing device reset [ 1406.190354] mlx5_core 0000:0d:00.2: mlx5_vdpa_show_mr_leaks:573:(pid 8506) warning: mkey still alive after resource delete: mr: 000000000c5ccca2, mkey: 0x40000000, refcount: 2 [ 1471.538487] mlx5_core 0000:0d:00.2: cb_timeout_handler:938:(pid 428): cmd[13]: MODIFY_GENERAL_OBJECT(0xa01) Async, timeout. Will cause a leak of a command resource [ 1471.539486] mlx5_core 0000:0d:00.2: cb_timeout_handler:938:(pid 428): cmd[12]: MODIFY_GENERAL_OBJECT(0xa01) Async, timeout. Will cause a leak of a command resource [ 1471.540351] mlx5_core 0000:0d:00.2: modify_virtqueues:1617:(pid 8511) error: modify vq 0 failed, state: 0 -> 0, err: 0 [ 1471.541433] mlx5_core 0000:0d:00.2: modify_virtqueues:1617:(pid 8511) error: modify vq 1 failed, state: 0 -> 0, err: -110 [ 1471.542388] mlx5_core 0000:0d:00.2: mlx5_vdpa_set_status:3203:(pid 8511) warning: failed to resume VQs [ 1471.549778] mlx5_core 0000:0d:00.2: mlx5_vdpa_show_mr_leaks:573:(pid 8511) warning: mkey still alive after resource delete: mr: 000000000c5ccca2, mkey: 0x40000000, refcount: 2 [ 1512.929854] mlx5_core 0000:0d:00.2: mlx5_vdpa_compat_reset:3267:(pid 8565): performing device reset [ 1513.100290] mlx5_core 0000:0d:00.2:mlx5_vdpa_show_mr_leaks:573:(pid 8565) warning: mkey still alive after resource delete: mr:
000000000c5ccca2, mkey:0x40000000, refcount: 2
Hi Dragos
Quoted 6 lines
Can you provide more details about the qemu version and the vdpa device options used?
Also, which FW version are you using? There is a relevant bug in FW 22.41.1000 which was fixed in the latest FW (22.42.1000). Did you encounter any FW syndromes in the host dmesg log?
This problem has gone when I updated the firmware version to 22.42.1000, and I tested it with regression tests using mellanox nic, everything works well.
Quoted 4 lines
Thanks,
Dragos
Dragos Tatulea
maintainerQuoted 57 lines · Lei Yang
Quoted 46 lines · Dragos Tatulea
Hi Lei,
Quoted 42 lines · Lei Yang
Hi Dragos
QE tested this series with mellanox nic, it failed with [1] when booting guest, and host dmesg also will print messages [2]. This bug can be reproduced boot guest with vhost-vdpa device.
[1] qemu) qemu-kvm: vhost VQ 1 ring restore failed: -1: Operation not permitted (1)
qemu-kvm: vhost VQ 0 ring restore failed: -1: Operation not permitted (1)
qemu-kvm: unable to start vhost net: 5: falling back on userspace virtio
qemu-kvm:vhost_set_featuresfailed: Device or resource busy (16)
qemu-kvm: unable to start vhost net: 16: falling back on userspace virtio[2] Host dmesg:
[ 1406.187977] mlx5_core 0000:0d:00.2: mlx5_vdpa_compat_reset:3267:(pid 8506): performing device reset [ 1406.189221] mlx5_core 0000:0d:00.2: mlx5_vdpa_compat_reset:3267:(pid 8506): performing device reset [ 1406.190354] mlx5_core 0000:0d:00.2: mlx5_vdpa_show_mr_leaks:573:(pid 8506) warning: mkey still alive after resource delete: mr: 000000000c5ccca2, mkey: 0x40000000, refcount: 2 [ 1471.538487] mlx5_core 0000:0d:00.2: cb_timeout_handler:938:(pid 428): cmd[13]: MODIFY_GENERAL_OBJECT(0xa01) Async, timeout. Will cause a leak of a command resource [ 1471.539486] mlx5_core 0000:0d:00.2: cb_timeout_handler:938:(pid 428): cmd[12]: MODIFY_GENERAL_OBJECT(0xa01) Async, timeout. Will cause a leak of a command resource [ 1471.540351] mlx5_core 0000:0d:00.2: modify_virtqueues:1617:(pid 8511) error: modify vq 0 failed, state: 0 -> 0, err: 0 [ 1471.541433] mlx5_core 0000:0d:00.2: modify_virtqueues:1617:(pid 8511) error: modify vq 1 failed, state: 0 -> 0, err: -110 [ 1471.542388] mlx5_core 0000:0d:00.2: mlx5_vdpa_set_status:3203:(pid 8511) warning: failed to resume VQs [ 1471.549778] mlx5_core 0000:0d:00.2: mlx5_vdpa_show_mr_leaks:573:(pid 8511) warning: mkey still alive after resource delete: mr: 000000000c5ccca2, mkey: 0x40000000, refcount: 2 [ 1512.929854] mlx5_core 0000:0d:00.2: mlx5_vdpa_compat_reset:3267:(pid 8565): performing device reset [ 1513.100290] mlx5_core 0000:0d:00.2:mlx5_vdpa_show_mr_leaks:573:(pid 8565) warning: mkey still alive after resource delete: mr:
000000000c5ccca2, mkey:0x40000000, refcount: 2Hi Dragos
Quoted 6 lines
Can you provide more details about the qemu version and the vdpa device options used?
Also, which FW version are you using? There is a relevant bug in FW 22.41.1000 which was fixed in the latest FW (22.42.1000). Did you encounter any FW syndromes in the host dmesg log?
Quoted 5 lines
This problem has gone when I updated the firmware version to 22.42.1000, and I tested it with regression tests using mellanox nic, everything works well.
Tested-by Lei Yang <[email protected]>
Good to hear. Thanks for the quick reaction.
Thanks,
Dragos
Eugenio Perez Martin
maintainerQuoted 61 lines · Dragos Tatulea
Quoted 57 lines · Lei Yang
Quoted 46 lines · Dragos Tatulea
Hi Lei,
Quoted 42 lines · Lei Yang
Hi Dragos
QE tested this series with mellanox nic, it failed with [1] when booting guest, and host dmesg also will print messages [2]. This bug can be reproduced boot guest with vhost-vdpa device.
[1] qemu) qemu-kvm: vhost VQ 1 ring restore failed: -1: Operation not permitted (1)
qemu-kvm: vhost VQ 0 ring restore failed: -1: Operation not permitted (1)
qemu-kvm: unable to start vhost net: 5: falling back on userspace virtio
qemu-kvm:vhost_set_featuresfailed: Device or resource busy (16)
qemu-kvm: unable to start vhost net: 16: falling back on userspace virtio[2] Host dmesg:
[ 1406.187977] mlx5_core 0000:0d:00.2: mlx5_vdpa_compat_reset:3267:(pid 8506): performing device reset [ 1406.189221] mlx5_core 0000:0d:00.2: mlx5_vdpa_compat_reset:3267:(pid 8506): performing device reset [ 1406.190354] mlx5_core 0000:0d:00.2: mlx5_vdpa_show_mr_leaks:573:(pid 8506) warning: mkey still alive after resource delete: mr: 000000000c5ccca2, mkey: 0x40000000, refcount: 2 [ 1471.538487] mlx5_core 0000:0d:00.2: cb_timeout_handler:938:(pid 428): cmd[13]: MODIFY_GENERAL_OBJECT(0xa01) Async, timeout. Will cause a leak of a command resource [ 1471.539486] mlx5_core 0000:0d:00.2: cb_timeout_handler:938:(pid 428): cmd[12]: MODIFY_GENERAL_OBJECT(0xa01) Async, timeout. Will cause a leak of a command resource [ 1471.540351] mlx5_core 0000:0d:00.2: modify_virtqueues:1617:(pid 8511) error: modify vq 0 failed, state: 0 -> 0, err: 0 [ 1471.541433] mlx5_core 0000:0d:00.2: modify_virtqueues:1617:(pid 8511) error: modify vq 1 failed, state: 0 -> 0, err: -110 [ 1471.542388] mlx5_core 0000:0d:00.2: mlx5_vdpa_set_status:3203:(pid 8511) warning: failed to resume VQs [ 1471.549778] mlx5_core 0000:0d:00.2: mlx5_vdpa_show_mr_leaks:573:(pid 8511) warning: mkey still alive after resource delete: mr: 000000000c5ccca2, mkey: 0x40000000, refcount: 2 [ 1512.929854] mlx5_core 0000:0d:00.2: mlx5_vdpa_compat_reset:3267:(pid 8565): performing device reset [ 1513.100290] mlx5_core 0000:0d:00.2:mlx5_vdpa_show_mr_leaks:573:(pid 8565) warning: mkey still alive after resource delete: mr:
000000000c5ccca2, mkey:0x40000000, refcount: 2Hi Dragos
Quoted 6 lines
Can you provide more details about the qemu version and the vdpa device options used?
Also, which FW version are you using? There is a relevant bug in FW 22.41.1000 which was fixed in the latest FW (22.42.1000). Did you encounter any FW syndromes in the host dmesg log?
Quoted 7 lines
Quoted 5 lines
This problem has gone when I updated the firmware version to 22.42.1000, and I tested it with regression tests using mellanox nic, everything works well.
Tested-by Lei Yang <[email protected]>Good to hear. Thanks for the quick reaction.
Is it possible to add a check so it doesn't use the async fashion in old FW?
Dragos Tatulea
maintainerQuoted 66 lines · Eugenio Perez Martin
Quoted 65 lines · Dragos Tatulea
Quoted 61 lines · Lei Yang
Quoted 46 lines · Dragos Tatulea
Hi Lei,
Quoted 42 lines · Lei Yang
Hi Dragos
QE tested this series with mellanox nic, it failed with [1] when booting guest, and host dmesg also will print messages [2]. This bug can be reproduced boot guest with vhost-vdpa device.
[1] qemu) qemu-kvm: vhost VQ 1 ring restore failed: -1: Operation not permitted (1)
qemu-kvm: vhost VQ 0 ring restore failed: -1: Operation not permitted (1)
qemu-kvm: unable to start vhost net: 5: falling back on userspace virtio
qemu-kvm:vhost_set_featuresfailed: Device or resource busy (16)
qemu-kvm: unable to start vhost net: 16: falling back on userspace virtio[2] Host dmesg:
[ 1406.187977] mlx5_core 0000:0d:00.2: mlx5_vdpa_compat_reset:3267:(pid 8506): performing device reset [ 1406.189221] mlx5_core 0000:0d:00.2: mlx5_vdpa_compat_reset:3267:(pid 8506): performing device reset [ 1406.190354] mlx5_core 0000:0d:00.2: mlx5_vdpa_show_mr_leaks:573:(pid 8506) warning: mkey still alive after resource delete: mr: 000000000c5ccca2, mkey: 0x40000000, refcount: 2 [ 1471.538487] mlx5_core 0000:0d:00.2: cb_timeout_handler:938:(pid 428): cmd[13]: MODIFY_GENERAL_OBJECT(0xa01) Async, timeout. Will cause a leak of a command resource [ 1471.539486] mlx5_core 0000:0d:00.2: cb_timeout_handler:938:(pid 428): cmd[12]: MODIFY_GENERAL_OBJECT(0xa01) Async, timeout. Will cause a leak of a command resource [ 1471.540351] mlx5_core 0000:0d:00.2: modify_virtqueues:1617:(pid 8511) error: modify vq 0 failed, state: 0 -> 0, err: 0 [ 1471.541433] mlx5_core 0000:0d:00.2: modify_virtqueues:1617:(pid 8511) error: modify vq 1 failed, state: 0 -> 0, err: -110 [ 1471.542388] mlx5_core 0000:0d:00.2: mlx5_vdpa_set_status:3203:(pid 8511) warning: failed to resume VQs [ 1471.549778] mlx5_core 0000:0d:00.2: mlx5_vdpa_show_mr_leaks:573:(pid 8511) warning: mkey still alive after resource delete: mr: 000000000c5ccca2, mkey: 0x40000000, refcount: 2 [ 1512.929854] mlx5_core 0000:0d:00.2: mlx5_vdpa_compat_reset:3267:(pid 8565): performing device reset [ 1513.100290] mlx5_core 0000:0d:00.2:mlx5_vdpa_show_mr_leaks:573:(pid 8565) warning: mkey still alive after resource delete: mr:
000000000c5ccca2, mkey:0x40000000, refcount: 2Hi Dragos
Quoted 6 lines
Can you provide more details about the qemu version and the vdpa device options used?
Also, which FW version are you using? There is a relevant bug in FW 22.41.1000 which was fixed in the latest FW (22.42.1000). Did you encounter any FW syndromes in the host dmesg log?
This problem has gone when I updated the firmware version to 22.42.1000, and I tested it with regression tests using mellanox nic, everything works well.
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Tested-by Lei Yang <[email protected]>Good to hear. Thanks for the quick reaction.
Is it possible to add a check so it doesn't use the async fashion in old FW?
Unfortunately not, it would have been there otherwise.
Note that this affects only FW version 22.41.1000. Older versions are not affected because VQ resume is not supported.
Thanks,
Dragos