Fix failure to hot add memory
Subsystems (4)
cover Fix failure to hot add memory
Hi Michael,
The following two patches are a fixing a failure to update the hardware with the updated used index. This results in a failure to to hot add memory to the guest which results in a memory map update and teardown and re-create of the resources.
The first patch just removes unnecessary code. The second on is the actual fix.
Eli Cohen (2):
vdpa/mlx5: Avoid unnecessary query virtqueue vdpa/mlx5: Restore the hardware used index after change map
drivers/vdpa/mlx5/net/mlx5_vnet.c | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-)
Signature · 1 line
2.29.2
PATCH 1/2 vdpa/mlx5: Avoid unnecessary query virtqueue
suspend_vq should only suspend the VQ on not save the current available index. This is done when a change of map occurs when the driver calls save_channel_info().
--- drivers/vdpa/mlx5/net/mlx5_vnet.c | 8 -------- 1 file changed, 8 deletions(-)
drivers/vdpa/mlx5/net/mlx5_vnet.c +0 -8
@@ -1148,8 +1148,6 @@ static int setup_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvq)static void suspend_vq (struct mlx5_vdpa_net *ndev,struct mlx5_vdpa_virtqueue *mvq) { -struct mlx5_virtq_attr attr; -if (!mvq->initialized)return ;
@@ -1158,12 +1156,6 @@ static void suspend_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mif (modify_virtqueue (ndev, mvq,MLX5_VIRTIO_NET_Q_OBJECT_STATE_SUSPEND ))mlx5_vdpa_warn (&ndev->mvdev,"modify to suspend failed\n" ); - -if (query_virtqueue (ndev, mvq, &attr)) { -mlx5_vdpa_warn (&ndev->mvdev,"failed to query virtqueue\n" ); -return ; - } - mvq->avail_idx = attr.available_index; }static void suspend_vqs (struct mlx5_vdpa_net *ndev)
Signature · 1 line
2.29.2
Re: [PATCH 1/2] vdpa/mlx5: Avoid unnecessary query virtqueue
Quoted 5 lines · Eli Cohen
suspend_vqshould only suspend the VQ on not save the current available index. This is done when a change of map occurs when the driver callssave_channel_info().Signed-off-by Eli Cohen <[email protected]>
Quoted 20 lines
--- drivers/vdpa/mlx5/net/mlx5_vnet.c | 8 -------- 1 file changed, 8 deletions(-)drivers/vdpa/mlx5/net/mlx5_vnet.c +0
-2@@ -1148,8 +1148,6 @@ static int setup_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvq)static void suspend_vq (struct mlx5_vdpa_net *ndev,struct mlx5_vdpa_virtqueue *mvq) { -struct mlx5_virtq_attr attr; -if (!mvq->initialized)return ;@@ -1158,12 +1156,6 @@ static void suspend_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mif (modify_virtqueue (ndev, mvq,MLX5_VIRTIO_NET_Q_OBJECT_STATE_SUSPEND ))
drivers/vdpa/mlx5/net/mlx5_vnet.c static void suspend_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *m · line 1161
Quoted 10 lines
drivers/vdpa/mlx5/net/mlx5_vnet.c +0
-6mlx5_vdpa_warn (&ndev->mvdev,"modify to suspend failed\n" ); - -if (query_virtqueue (ndev, mvq, &attr)) { -mlx5_vdpa_warn (&ndev->mvdev,"failed to query virtqueue\n" ); -return ; - } - mvq->avail_idx = attr.available_index; }static void suspend_vqs (struct mlx5_vdpa_net *ndev)
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH 1/2] vdpa/mlx5: Avoid unnecessary query virtqueue
In reply to Eli CohenQuoted 4 lines · Eli Cohen
suspend_vqshould only suspend the VQ on not save the current available index. This is done when a change of map occurs when the driver callssave_channel_info().
Hmmm, suspend_vq() is also called by teardown_vq(), the latter of which doesn't save the available index as save_channel_info() doesn't get called in that path at all. How does it handle the case that aget_vq_state() is called from userspace (e.g. QEMU) while the hardware VQ object was torn down, but userspace still wants to access the queue index?
Refer to https://lore.kernel.org/netdev/[email protected]/
vhost VQ 0 ring restore failed: -1: Resource temporarily unavailable (11) vhost VQ 1 ring restore failed: -1: Resource temporarily unavailable (11)
QEMU will complain with the above warning while VM is being rebooted or shut down.
Looks to me either the kernel driver should cover this requirement, or the userspace has to bear the burden in saving the index and not call into kernel if VQ is destroyed.
-Siwei
Quoted 35 lines
Signed-off-by Eli Cohen <[email protected]>--- drivers/vdpa/mlx5/net/mlx5_vnet.c | 8 -------- 1 file changed, 8 deletions(-)drivers/vdpa/mlx5/net/mlx5_vnet.c +0
-9@@ -1148,8 +1148,6 @@ static int setup_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvq)static void suspend_vq (struct mlx5_vdpa_net *ndev,struct mlx5_vdpa_virtqueue *mvq) { -struct mlx5_virtq_attr attr; -if (!mvq->initialized)return ;@@ -1158,12 +1156,6 @@ static void suspend_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mif (modify_virtqueue (ndev, mvq,MLX5_VIRTIO_NET_Q_OBJECT_STATE_SUSPEND ))mlx5_vdpa_warn (&ndev->mvdev,"modify to suspend failed\n" ); - -if (query_virtqueue (ndev, mvq, &attr)) { -mlx5_vdpa_warn (&ndev->mvdev,"failed to query virtqueue\n" ); -return ; - } - mvq->avail_idx = attr.available_index; }static void suspend_vqs (struct mlx5_vdpa_net *ndev) --2.29.2
Re: [PATCH 1/2] vdpa/mlx5: Avoid unnecessary query virtqueue
Quoted 19 lines · Si-Wei Liu
Quoted 4 lines · Eli Cohen
suspend_vqshould only suspend the VQ on not save the current available index. This is done when a change of map occurs when the driver callssave_channel_info().Hmmm,
suspend_vq()is also called byteardown_vq(), the latter of which doesn't save the available index assave_channel_info()doesn't get called in that path at all. How does it handle the case thataget_vq_state()is called from userspace (e.g. QEMU) while the hardware VQ object was torn down, but userspace still wants to access the queue index?Refer to https://lore.kernel.org/netdev/[email protected]/
vhost VQ 0 ring restore failed: -1: Resource temporarily unavailable (11) vhost VQ 1 ring restore failed: -1: Resource temporarily unavailable (11)
Quoted 6 lines
QEMU will complain with the above warning while VM is being rebooted or shut down.
Looks to me either the kernel driver should cover this requirement, or the userspace has to bear the burden in saving the index and not call into kernel if VQ is destroyed.
Actually, the userspace doesn't have the insights whether virt queue will be destroyed if just changing the device status via set_status(). Looking at other vdpa driver in tree i.e. ifcvf it doesn't behave like so. Hence this still looks to me to be Mellanox specifics and mlx5_vdpa implementation detail that shouldn't expose to userspace.
Quoted 39 lines
-Siwei
Quoted 35 lines
Signed-off-by Eli Cohen <[email protected]>--- drivers/vdpa/mlx5/net/mlx5_vnet.c | 8 -------- 1 file changed, 8 deletions(-)drivers/vdpa/mlx5/net/mlx5_vnet.c +0
-9@@ -1148,8 +1148,6 @@ static int setup_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvq)static void suspend_vq (struct mlx5_vdpa_net *ndev,struct mlx5_vdpa_virtqueue *mvq) { -struct mlx5_virtq_attr attr; -if (!mvq->initialized)return ;@@ -1158,12 +1156,6 @@ static void suspend_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mif (modify_virtqueue (ndev, mvq,MLX5_VIRTIO_NET_Q_OBJECT_STATE_SUSPEND ))mlx5_vdpa_warn (&ndev->mvdev,"modify to suspend failed\n" ); - -if (query_virtqueue (ndev, mvq, &attr)) { -mlx5_vdpa_warn (&ndev->mvdev,"failed to query virtqueue\n" ); -return ; - } - mvq->avail_idx = attr.available_index; }static void suspend_vqs (struct mlx5_vdpa_net *ndev) --2.29.2
Re: [PATCH 1/2] vdpa/mlx5: Avoid unnecessary query virtqueue
Quoted 20 lines · Si-Wei Liu
Quoted 19 lines · Si-Wei Liu
Quoted 3 lines · Eli Cohen
suspend_vqshould only suspend the VQ on not save the current available index. This is done when a change of map occurs when the driver callssave_channel_info().Hmmm,
suspend_vq()is also called byteardown_vq(), the latter of which doesn't save the available index assave_channel_info()doesn't get called in that path at all. How does it handle the case thataget_vq_state()is called from userspace (e.g. QEMU) while the hardware VQ object was torn down, but userspace still wants to access the queue index?Refer to https://lore.kernel.org/netdev/[email protected]/
vhost VQ 0 ring restore failed: -1: Resource temporarily unavailable (11) vhost VQ 1 ring restore failed: -1: Resource temporarily unavailable (11)
QEMU will complain with the above warning while VM is being rebooted or shut down.
Quoted 8 lines
Quoted 3 lines
Looks to me either the kernel driver should cover this requirement, or the userspace has to bear the burden in saving the index and not call into kernel if VQ is destroyed.
Actually, the userspace doesn't have the insights whether virt queue will be destroyed if just changing the device status via
set_status(). Looking at other vdpa driver in tree i.e. ifcvf it doesn't behave like so. Hence this still looks to me to be Mellanox specifics andmlx5_vdpaimplementation detail that shouldn't expose to userspace.
So I think we can simply drop this patch?
Thanks
Quoted 33 lines
Quoted 33 lines
-Siwei
Quoted 30 lines
Signed-off-by Eli Cohen <[email protected]>--- drivers/vdpa/mlx5/net/mlx5_vnet.c | 8 -------- 1 file changed, 8 deletions(-)drivers/vdpa/mlx5/net/mlx5_vnet.c +0
-8@@ -1148,8 +1148,6 @@ static int setup_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvq)static void suspend_vq (struct mlx5_vdpa_net *ndev,struct mlx5_vdpa_virtqueue *mvq) { -struct mlx5_virtq_attr attr; -if (!mvq->initialized)return ;@@ -1158,12 +1156,6 @@ static void suspend_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mif (modify_virtqueue (ndev, mvq,MLX5_VIRTIO_NET_Q_OBJECT_STATE_SUSPEND ))mlx5_vdpa_warn (&ndev->mvdev,"modify to suspend failed\n" ); - -if (query_virtqueue (ndev, mvq, &attr)) { -mlx5_vdpa_warn (&ndev->mvdev,"failed to query virtqueue\n" ); -return ; - } - mvq->avail_idx = attr.available_index; }
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
drivers/vdpa/mlx5/net/mlx5_vnet.c +0
-1static void suspend_vqs (struct mlx5_vdpa_net *ndev) --2.29.2
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH 1/2] vdpa/mlx5: Avoid unnecessary query virtqueue
Quoted 26 lines · Jason Wang
Quoted 23 lines · Si-Wei Liu
Quoted 22 lines · Si-Wei Liu
Quoted 3 lines · Eli Cohen
suspend_vqshould only suspend the VQ on not save the current available index. This is done when a change of map occurs when the driver callssave_channel_info().Hmmm,
suspend_vq()is also called byteardown_vq(), the latter of which doesn't save the available index assave_channel_info()doesn't get called in that path at all. How does it handle the case thataget_vq_state()is called from userspace (e.g. QEMU) while the hardware VQ object was torn down, but userspace still wants to access the queue index?Refer to https://lore.kernel.org/netdev/[email protected]/
vhost VQ 0 ring restore failed: -1: Resource temporarily unavailable (11) vhost VQ 1 ring restore failed: -1: Resource temporarily unavailable (11)
QEMU will complain with the above warning while VM is being rebooted or shut down.
Looks to me either the kernel driver should cover this requirement, or the userspace has to bear the burden in saving the index and not call into kernel if VQ is destroyed.
Quoted 8 lines
Quoted 5 lines
Actually, the userspace doesn't have the insights whether virt queue will be destroyed if just changing the device status via
set_status(). Looking at other vdpa driver in tree i.e. ifcvf it doesn't behave like so. Hence this still looks to me to be Mellanox specifics andmlx5_vdpaimplementation detail that shouldn't expose to userspace.So I think we can simply drop this patch?
Yep, I think so. To be honest I don't know why it has anything to do with the memory hotplug issue.
-Siwei
Quoted 42 lines
Thanks
Quoted 37 lines
Quoted 37 lines
-Siwei
Quoted 34 lines
Signed-off-by Eli Cohen <[email protected]>--- drivers/vdpa/mlx5/net/mlx5_vnet.c | 8 -------- 1 file changed, 8 deletions(-)drivers/vdpa/mlx5/net/mlx5_vnet.c +0
-9@@ -1148,8 +1148,6 @@ static int setup_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvq)static void suspend_vq (struct mlx5_vdpa_net *ndev,struct mlx5_vdpa_virtqueue *mvq) { -struct mlx5_virtq_attr attr; -if (!mvq->initialized)return ;@@ -1158,12 +1156,6 @@ static void suspend_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mif (modify_virtqueue (ndev, mvq,MLX5_VIRTIO_NET_Q_OBJECT_STATE_SUSPEND ))mlx5_vdpa_warn (&ndev->mvdev,"modify to suspend failed\n" ); - -if (query_virtqueue (ndev, mvq, &attr)) { -mlx5_vdpa_warn (&ndev->mvdev,"failed to query virtqueue\n" ); -return ; - } - mvq->avail_idx = attr.available_index; }static void suspend_vqs (struct mlx5_vdpa_net *ndev) --2.29.2
Re: [PATCH 1/2] vdpa/mlx5: Avoid unnecessary query virtqueue
Quoted 32 lines · Si-Wei Liu
Quoted 31 lines · Jason Wang
Quoted 28 lines · Si-Wei Liu
Quoted 16 lines · Si-Wei Liu
Quoted 3 lines · Eli Cohen
suspend_vqshould only suspend the VQ on not save the current available index. This is done when a change of map occurs when the driver callssave_channel_info().Hmmm,
suspend_vq()is also called byteardown_vq(), the latter of which doesn't save the available index assave_channel_info()doesn't get called in that path at all. How does it handle the case thataget_vq_state()is called from userspace (e.g. QEMU) while the hardware VQ object was torn down, but userspace still wants to access the queue index?Refer to https://lore.kernel.org/netdev/[email protected]/
vhost VQ 0 ring restore failed: -1: Resource temporarily unavailable (11) vhost VQ 1 ring restore failed: -1: Resource temporarily unavailable (11)
Quoted 6 lines
QEMU will complain with the above warning while VM is being rebooted or shut down.
Looks to me either the kernel driver should cover this requirement, or the userspace has to bear the burden in saving the index and not call into kernel if VQ is destroyed.
Actually, the userspace doesn't have the insights whether virt queue will be destroyed if just changing the device status via
set_status(). Looking at other vdpa driver in tree i.e. ifcvf it doesn't behave like so. Hence this still looks to me to be Mellanox specifics andmlx5_vdpaimplementation detail that shouldn't expose to userspace.
Quoted 3 lines
Quoted 1 line
So I think we can simply drop this patch?
Yep, I think so. To be honest I don't know why it has anything to do with the memory hotplug issue.
Eli may know more, my understanding is that, during memory hotplut, qemu need to propagate new memory mappings via set_map(). For mellanox, it means it needs to rebuild memory keys, so the virtqueue needs to be suspended.
Thanks
Quoted 39 lines
-Siwei
Quoted 36 lines
Thanks
Quoted 33 lines
Quoted 33 lines
-Siwei
Quoted 30 lines
Signed-off-by Eli Cohen <[email protected]>--- drivers/vdpa/mlx5/net/mlx5_vnet.c | 8 -------- 1 file changed, 8 deletions(-)drivers/vdpa/mlx5/net/mlx5_vnet.c +0
-8@@ -1148,8 +1148,6 @@ static int setup_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvq)static void suspend_vq (struct mlx5_vdpa_net *ndev,struct mlx5_vdpa_virtqueue *mvq) { -struct mlx5_virtq_attr attr; -if (!mvq->initialized)return ;@@ -1158,12 +1156,6 @@ static void suspend_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mif (modify_virtqueue (ndev, mvq,MLX5_VIRTIO_NET_Q_OBJECT_STATE_SUSPEND ))mlx5_vdpa_warn (&ndev->mvdev,"modify to suspend failed\n" ); - -if (query_virtqueue (ndev, mvq, &attr)) { -mlx5_vdpa_warn (&ndev->mvdev,"failed to query virtqueue\n" ); -return ; - } - mvq->avail_idx = attr.available_index; }
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
drivers/vdpa/mlx5/net/mlx5_vnet.c +0
-1static void suspend_vqs (struct mlx5_vdpa_net *ndev) --2.29.2
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH 1/2] vdpa/mlx5: Avoid unnecessary query virtqueue
Quoted 34 lines · Jason Wang
Quoted 32 lines · Si-Wei Liu
Quoted 31 lines · Jason Wang
Quoted 28 lines · Si-Wei Liu
Quoted 16 lines · Si-Wei Liu
Quoted 3 lines · Eli Cohen
suspend_vqshould only suspend the VQ on not save the current available index. This is done when a change of map occurs when the driver callssave_channel_info().Hmmm,
suspend_vq()is also called byteardown_vq(), the latter of which doesn't save the available index assave_channel_info()doesn't get called in that path at all. How does it handle the case thataget_vq_state()is called from userspace (e.g. QEMU) while the hardware VQ object was torn down, but userspace still wants to access the queue index?Refer to https://lore.kernel.org/netdev/[email protected]/
vhost VQ 0 ring restore failed: -1: Resource temporarily unavailable (11) vhost VQ 1 ring restore failed: -1: Resource temporarily unavailable (11)
Quoted 6 lines
QEMU will complain with the above warning while VM is being rebooted or shut down.
Looks to me either the kernel driver should cover this requirement, or the userspace has to bear the burden in saving the index and not call into kernel if VQ is destroyed.
Actually, the userspace doesn't have the insights whether virt queue will be destroyed if just changing the device status via
set_status(). Looking at other vdpa driver in tree i.e. ifcvf it doesn't behave like so. Hence this still looks to me to be Mellanox specifics andmlx5_vdpaimplementation detail that shouldn't expose to userspace.
Quoted 9 lines
Quoted 3 lines
Quoted 1 line
So I think we can simply drop this patch?
Yep, I think so. To be honest I don't know why it has anything to do with the memory hotplug issue.
Eli may know more, my understanding is that, during memory hotplut, qemu need to propagate new memory mappings via
set_map(). For mellanox, it means it needs to rebuild memory keys, so the virtqueue needs to be suspended.
I think Siwei was asking why the first patch was related to the hotplug issue.
But you're correct. When memory is added, I get a new memory map. This requires me to build a new memory key object which covers the new memory map. Since the virtqueue objects are referencing this memory key, I need to destroy them and build new ones referncing the new memory key.
Quoted 47 lines
Thanks
Quoted 43 lines
-Siwei
Quoted 40 lines
Thanks
Quoted 37 lines
Quoted 37 lines
-Siwei
Quoted 34 lines
Signed-off-by Eli Cohen <[email protected]>--- drivers/vdpa/mlx5/net/mlx5_vnet.c | 8 -------- 1 file changed, 8 deletions(-)drivers/vdpa/mlx5/net/mlx5_vnet.c +0
-9@@ -1148,8 +1148,6 @@ static int setup_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvq)static void suspend_vq (struct mlx5_vdpa_net *ndev,struct mlx5_vdpa_virtqueue *mvq) { -struct mlx5_virtq_attr attr; -if (!mvq->initialized)return ;@@ -1158,12 +1156,6 @@ static void suspend_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mif (modify_virtqueue (ndev, mvq,MLX5_VIRTIO_NET_Q_OBJECT_STATE_SUSPEND ))mlx5_vdpa_warn (&ndev->mvdev,"modify to suspend failed\n" ); - -if (query_virtqueue (ndev, mvq, &attr)) { -mlx5_vdpa_warn (&ndev->mvdev,"failed to query virtqueue\n" ); -return ; - } - mvq->avail_idx = attr.available_index; }static void suspend_vqs (struct mlx5_vdpa_net *ndev) --2.29.2
Re: [PATCH 1/2] vdpa/mlx5: Avoid unnecessary query virtqueue
Thanks Eli and Jason for clarifications. See inline.
Quoted 41 lines · Eli Cohen
Quoted 39 lines · Jason Wang
Quoted 35 lines · Si-Wei Liu
Quoted 25 lines · Jason Wang
Quoted 23 lines · Si-Wei Liu
Quoted 22 lines · Si-Wei Liu
Quoted 3 lines · Eli Cohen
suspend_vqshould only suspend the VQ on not save the current available index. This is done when a change of map occurs when the driver callssave_channel_info().Hmmm,
suspend_vq()is also called byteardown_vq(), the latter of which doesn't save the available index assave_channel_info()doesn't get called in that path at all. How does it handle the case thataget_vq_state()is called from userspace (e.g. QEMU) while the hardware VQ object was torn down, but userspace still wants to access the queue index?Refer to https://lore.kernel.org/netdev/[email protected]/
vhost VQ 0 ring restore failed: -1: Resource temporarily unavailable (11) vhost VQ 1 ring restore failed: -1: Resource temporarily unavailable (11)
QEMU will complain with the above warning while VM is being rebooted or shut down.
Looks to me either the kernel driver should cover this requirement, or the userspace has to bear the burden in saving the index and not call into kernel if VQ is destroyed.
Quoted 7 lines
Quoted 5 lines
Actually, the userspace doesn't have the insights whether virt queue will be destroyed if just changing the device status via
set_status(). Looking at other vdpa driver in tree i.e. ifcvf it doesn't behave like so. Hence this still looks to me to be Mellanox specifics andmlx5_vdpaimplementation detail that shouldn't expose to userspace.So I think we can simply drop this patch?
Yep, I think so. To be honest I don't know why it has anything to do with the memory hotplug issue.
Quoted 7 lines
Quoted 4 lines
Eli may know more, my understanding is that, during memory hotplut, qemu need to propagate new memory mappings via
set_map(). For mellanox, it means it needs to rebuild memory keys, so the virtqueue needs to be suspended.I think Siwei was asking why the first patch was related to the hotplug issue.
I was thinking how consistency is assured when saving/restoring this h/w avail_index against the one in the virtq memory, particularly in the region_add/.region_del() context (e.g. the hotplug case). Problem is I don't see explicit memory barrier when guest thread updates the avail_index, how does the device make sure the h/w avail_index is uptodate while guest may race with updating the virtq's avail_index in the mean while? Maybe I completely miss something obvious?
Thanks,
-Siwei
Quoted 53 lines
But you're correct. When memory is added, I get a new memory map. This requires me to build a new memory key object which covers the new memory map. Since the virtqueue objects are referencing this memory key, I need to destroy them and build new ones referncing the new memory key.
Quoted 47 lines
Thanks
Quoted 43 lines
-Siwei
Quoted 40 lines
Thanks
Quoted 37 lines
Quoted 37 lines
-Siwei
Quoted 34 lines
Signed-off-by Eli Cohen <[email protected]>--- drivers/vdpa/mlx5/net/mlx5_vnet.c | 8 -------- 1 file changed, 8 deletions(-)drivers/vdpa/mlx5/net/mlx5_vnet.c +0
-9@@ -1148,8 +1148,6 @@ static int setup_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvq)static void suspend_vq (struct mlx5_vdpa_net *ndev,struct mlx5_vdpa_virtqueue *mvq) { -struct mlx5_virtq_attr attr; -if (!mvq->initialized)return ;@@ -1158,12 +1156,6 @@ static void suspend_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mif (modify_virtqueue (ndev, mvq,MLX5_VIRTIO_NET_Q_OBJECT_STATE_SUSPEND ))mlx5_vdpa_warn (&ndev->mvdev,"modify to suspend failed\n" ); - -if (query_virtqueue (ndev, mvq, &attr)) { -mlx5_vdpa_warn (&ndev->mvdev,"failed to query virtqueue\n" ); -return ; - } - mvq->avail_idx = attr.available_index; }static void suspend_vqs (struct mlx5_vdpa_net *ndev) --2.29.2
Re: [PATCH 1/2] vdpa/mlx5: Avoid unnecessary query virtqueue
Quoted 52 lines · Si-Wei Liu
Thanks Eli and Jason for clarifications. See inline.
Quoted 41 lines · Eli Cohen
Quoted 39 lines · Jason Wang
Quoted 35 lines · Si-Wei Liu
Quoted 25 lines · Jason Wang
Quoted 23 lines · Si-Wei Liu
Quoted 22 lines · Si-Wei Liu
Quoted 3 lines · Eli Cohen
suspend_vqshould only suspend the VQ on not save the current available index. This is done when a change of map occurs when the driver callssave_channel_info().Hmmm,
suspend_vq()is also called byteardown_vq(), the latter of which doesn't save the available index assave_channel_info()doesn't get called in that path at all. How does it handle the case thataget_vq_state()is called from userspace (e.g. QEMU) while the hardware VQ object was torn down, but userspace still wants to access the queue index?Refer to https://lore.kernel.org/netdev/[email protected]/
vhost VQ 0 ring restore failed: -1: Resource temporarily unavailable (11) vhost VQ 1 ring restore failed: -1: Resource temporarily unavailable (11)
QEMU will complain with the above warning while VM is being rebooted or shut down.
Looks to me either the kernel driver should cover this requirement, or the userspace has to bear the burden in saving the index and not call into kernel if VQ is destroyed.
Quoted 7 lines
Quoted 5 lines
Actually, the userspace doesn't have the insights whether virt queue will be destroyed if just changing the device status via
set_status(). Looking at other vdpa driver in tree i.e. ifcvf it doesn't behave like so. Hence this still looks to me to be Mellanox specifics andmlx5_vdpaimplementation detail that shouldn't expose to userspace.So I think we can simply drop this patch?
Yep, I think so. To be honest I don't know why it has anything to do with the memory hotplug issue.
Quoted 7 lines
Quoted 4 lines
Eli may know more, my understanding is that, during memory hotplut, qemu need to propagate new memory mappings via
set_map(). For mellanox, it means it needs to rebuild memory keys, so the virtqueue needs to be suspended.I think Siwei was asking why the first patch was related to the hotplug issue.
Quoted 7 lines
I was thinking how consistency is assured when saving/restoring this h/w
avail_indexagainst the one in the virtq memory, particularly in the region_add/.region_del() context (e.g. the hotplug case). Problem is I don't see explicit memory barrier when guest thread updates theavail_index, how does the device make sure the h/wavail_indexis uptodate while guest may race with updating the virtq'savail_indexin the mean while? Maybe I completely miss something obvious?
If you're asking about syncronization upon hot plug of memory, the hardware always goes to read the available index from memory when a new hardware object is associted with a virtqueue. You can argue then that you don't need to restore the available index and you may be right but this is the currect inteface to the firmware.
If you're asking on generally how sync is assured when the guest updates the available index, can you please send a pointer to the code where you see the update without a memory barrier?
Quoted 57 lines
Thanks,
-SiweiQuoted 53 lines
But you're correct. When memory is added, I get a new memory map. This requires me to build a new memory key object which covers the new memory map. Since the virtqueue objects are referencing this memory key, I need to destroy them and build new ones referncing the new memory key.
Quoted 47 lines
Thanks
Quoted 43 lines
-Siwei
Quoted 40 lines
Thanks
Quoted 37 lines
Quoted 37 lines
-Siwei
Quoted 34 lines
Signed-off-by Eli Cohen <[email protected]>--- drivers/vdpa/mlx5/net/mlx5_vnet.c | 8 -------- 1 file changed, 8 deletions(-)drivers/vdpa/mlx5/net/mlx5_vnet.c +0
-9@@ -1148,8 +1148,6 @@ static int setup_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvq)static void suspend_vq (struct mlx5_vdpa_net *ndev,struct mlx5_vdpa_virtqueue *mvq) { -struct mlx5_virtq_attr attr; -if (!mvq->initialized)return ;@@ -1158,12 +1156,6 @@ static void suspend_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mif (modify_virtqueue (ndev, mvq,MLX5_VIRTIO_NET_Q_OBJECT_STATE_SUSPEND ))mlx5_vdpa_warn (&ndev->mvdev,"modify to suspend failed\n" ); - -if (query_virtqueue (ndev, mvq, &attr)) { -mlx5_vdpa_warn (&ndev->mvdev,"failed to query virtqueue\n" ); -return ; - } - mvq->avail_idx = attr.available_index; }static void suspend_vqs (struct mlx5_vdpa_net *ndev) --2.29.2
Re: [PATCH 1/2] vdpa/mlx5: Avoid unnecessary query virtqueue
Quoted 80 lines · Eli Cohen
Quoted 52 lines · Si-Wei Liu
Thanks Eli and Jason for clarifications. See inline.
Quoted 41 lines · Eli Cohen
Quoted 39 lines · Jason Wang
Quoted 35 lines · Si-Wei Liu
Quoted 25 lines · Jason Wang
Quoted 23 lines · Si-Wei Liu
Quoted 22 lines · Si-Wei Liu
Quoted 3 lines · Eli Cohen
suspend_vqshould only suspend the VQ on not save the current available index. This is done when a change of map occurs when the driver callssave_channel_info().Hmmm,
suspend_vq()is also called byteardown_vq(), the latter of which doesn't save the available index assave_channel_info()doesn't get called in that path at all. How does it handle the case thataget_vq_state()is called from userspace (e.g. QEMU) while the hardware VQ object was torn down, but userspace still wants to access the queue index?Refer to https://lore.kernel.org/netdev/[email protected]/
vhost VQ 0 ring restore failed: -1: Resource temporarily unavailable (11) vhost VQ 1 ring restore failed: -1: Resource temporarily unavailable (11)
QEMU will complain with the above warning while VM is being rebooted or shut down.
Looks to me either the kernel driver should cover this requirement, or the userspace has to bear the burden in saving the index and not call into kernel if VQ is destroyed.
Quoted 7 lines
Quoted 5 lines
Actually, the userspace doesn't have the insights whether virt queue will be destroyed if just changing the device status via
set_status(). Looking at other vdpa driver in tree i.e. ifcvf it doesn't behave like so. Hence this still looks to me to be Mellanox specifics andmlx5_vdpaimplementation detail that shouldn't expose to userspace.So I think we can simply drop this patch?
Yep, I think so. To be honest I don't know why it has anything to do with the memory hotplug issue.
Quoted 7 lines
Quoted 4 lines
Eli may know more, my understanding is that, during memory hotplut, qemu need to propagate new memory mappings via
set_map(). For mellanox, it means it needs to rebuild memory keys, so the virtqueue needs to be suspended.I think Siwei was asking why the first patch was related to the hotplug issue.
Quoted 7 lines
I was thinking how consistency is assured when saving/restoring this h/w
avail_indexagainst the one in the virtq memory, particularly in the region_add/.region_del() context (e.g. the hotplug case). Problem is I don't see explicit memory barrier when guest thread updates theavail_index, how does the device make sure the h/wavail_indexis uptodate while guest may race with updating the virtq'savail_indexin the mean while? Maybe I completely miss something obvious?DKIM-Signature: v1; arsa-sha256; crelaxed/relaxed; dnvidia.com; sn1; t 12257780; bhHnB0z4VEKwRS3WGY8d836MJgxu5Eln/jbFZlNXVxc08; hX-PGP-Universal:Date:From:To:CC:Subject:Message-ID:References: MIME-Version:Content-Type:Content-Disposition: Content-Transfer-Encoding:In-Reply-To:User-Agent:X-Originating-IP: X-ClientProxiedBy; bgGmb8+rcn3/rKzKQ/7QzSnghWzZ+FAU0XntsRZYGQ66sFvT7zsYPHogG3LIWNY77t wNHPw7GCJrNaH3nEXPbOp0FMOZw4Kv4W7UPuYPobbLeTkvuPAidjB8dM42vz+1X61t 9IVQT9X4hnAxRjI5CqZOo41GS4Tl1X+ykGoA+VE80BR/R/+nQ3tXDVULfppzeB+vu3 TWnnpaZ2GyoNyPlMiyVRkHdXzDVgA4uQHxwHn7otGK5J4lzyu8KrFyQtiP+f6hfu5v crJkYS8e9A+rfzUmKWuyHcKcmhPhAVJ4XdpzZcDXXlMHVxG7nR1o88xttC6D1+oNIP 9xHI3DkNBpEuAIf you're asking about syncronization upon hot plug of memory, the hardware always goes to read the available index from memory when a new hardware object is associted with a virtqueue. You can argue then that you don't need to restore the available index and you may be right but this is the currect inteface to the firmware.
Quoted 3 lines
If you're asking on generally how sync is assured when the guest updates the available index, can you please send a pointer to the code where you see the update without a memory barrier?
This is a snippet of virtqueue_add_split() where avail_index gets updated by guest:
/* Put entry in available array (but don't update avail->idx until they
* do sync). */
avail = vq->split.avail_idx_shadow & (vq->split.vring.num - 1);
vq->split.vring.avail->ring[avail] = cpu_to_virtio16(_vq->vdev, head);
/* Descriptors and available array need to be set before we expose the
* new available array entries. */
virtio_wmb(vq->weak_barriers);
vq->split.avail_idx_shadow++;
vq->split.vring.avail->idx = cpu_to_virtio16(_vq->vdev,
vq->split.avail_idx_shadow);
vq->num_added++;There's memory barrier to make sure the update to descriptor and available ring is seen before writing to the avail->idx, but there seems no gurantee that this update would flush to the memory immmedately either before or after the mlx5-vdpa is suspened and gets the old avail_index value stashed somewhere. In this case, how does the hardware ensure the consistency between the guest virtio and host mlx5-vdpa? Or, it completly relies on guest to update the avail_index once the next buffer is available, so that the index will be in sync again?
Thanks,
-Siwei
Quoted 58 lines
Quoted 58 lines
Thanks,
-SiweiQuoted 53 lines
But you're correct. When memory is added, I get a new memory map. This requires me to build a new memory key object which covers the new memory map. Since the virtqueue objects are referencing this memory key, I need to destroy them and build new ones referncing the new memory key.
Quoted 47 lines
Thanks
Quoted 43 lines
-Siwei
Quoted 40 lines
Thanks
Quoted 37 lines
Quoted 37 lines
-Siwei
Quoted 34 lines
Signed-off-by Eli Cohen <[email protected]>--- drivers/vdpa/mlx5/net/mlx5_vnet.c | 8 -------- 1 file changed, 8 deletions(-)drivers/vdpa/mlx5/net/mlx5_vnet.c +0
-9@@ -1148,8 +1148,6 @@ static int setup_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvq)static void suspend_vq (struct mlx5_vdpa_net *ndev,struct mlx5_vdpa_virtqueue *mvq) { -struct mlx5_virtq_attr attr; -if (!mvq->initialized)return ;@@ -1158,12 +1156,6 @@ static void suspend_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mif (modify_virtqueue (ndev, mvq,MLX5_VIRTIO_NET_Q_OBJECT_STATE_SUSPEND ))mlx5_vdpa_warn (&ndev->mvdev,"modify to suspend failed\n" ); - -if (query_virtqueue (ndev, mvq, &attr)) { -mlx5_vdpa_warn (&ndev->mvdev,"failed to query virtqueue\n" ); -return ; - } - mvq->avail_idx = attr.available_index; }static void suspend_vqs (struct mlx5_vdpa_net *ndev) --2.29.2
Re: [PATCH 1/2] vdpa/mlx5: Avoid unnecessary query virtqueue
Quoted 92 lines · Si-Wei Liu
Quoted 72 lines · Eli Cohen
Quoted 45 lines · Si-Wei Liu
Thanks Eli and Jason for clarifications. See inline.
Quoted 36 lines · Eli Cohen
Quoted 35 lines · Jason Wang
Quoted 33 lines · Si-Wei Liu
Quoted 24 lines · Jason Wang
Quoted 23 lines · Si-Wei Liu
Quoted 22 lines · Si-Wei Liu
Quoted 3 lines · Eli Cohen
suspend_vqshould only suspend the VQ on not save the current available index. This is done when a change of map occurs when the driver callssave_channel_info().Hmmm,
suspend_vq()is also called byteardown_vq(), the latter of which doesn't save the available index assave_channel_info()doesn't get called in that path at all. How does it handle the case thataget_vq_state()is called from userspace (e.g. QEMU) while the hardware VQ object was torn down, but userspace still wants to access the queue index?Refer to https://lore.kernel.org/netdev/[email protected]/
vhost VQ 0 ring restore failed: -1: Resource temporarily unavailable (11) vhost VQ 1 ring restore failed: -1: Resource temporarily unavailable (11)
QEMU will complain with the above warning while VM is being rebooted or shut down.
Looks to me either the kernel driver should cover this requirement, or the userspace has to bear the burden in saving the index and not call into kernel if VQ is destroyed.
Quoted 6 lines
Quoted 5 lines
Actually, the userspace doesn't have the insights whether virt queue will be destroyed if just changing the device status via
set_status(). Looking at other vdpa driver in tree i.e. ifcvf it doesn't behave like so. Hence this still looks to me to be Mellanox specifics andmlx5_vdpaimplementation detail that shouldn't expose to userspace.So I think we can simply drop this patch?
Yep, I think so. To be honest I don't know why it has anything to do with the memory hotplug issue.
Quoted 6 lines
Quoted 4 lines
Eli may know more, my understanding is that, during memory hotplut, qemu need to propagate new memory mappings via
set_map(). For mellanox, it means it needs to rebuild memory keys, so the virtqueue needs to be suspended.I think Siwei was asking why the first patch was related to the hotplug issue.
Quoted 7 lines
I was thinking how consistency is assured when saving/restoring this h/w
avail_indexagainst the one in the virtq memory, particularly in the region_add/.region_del() context (e.g. the hotplug case). Problem is I don't see explicit memory barrier when guest thread updates theavail_index, how does the device make sure the h/wavail_indexis uptodate while guest may race with updating the virtq'savail_indexin the mean while? Maybe I completely miss something obvious?DKIM-Signature: v1; arsa-sha256; crelaxed/relaxed; dnvidia.com; sn1; t 12257780; bhHnB0z4VEKwRS3WGY8d836MJgxu5Eln/jbFZlNXVxc08; hX-PGP-Universal:Date:From:To:CC:Subject:Message-ID:References: MIME-Version:Content-Type:Content-Disposition: Content-Transfer-Encoding:In-Reply-To:User-Agent:X-Originating-IP: X-ClientProxiedBy; bgGmb8+rcn3/rKzKQ/7QzSnghWzZ+FAU0XntsRZYGQ66sFvT7zsYPHogG3LIWNY77t wNHPw7GCJrNaH3nEXPbOp0FMOZw4Kv4W7UPuYPobbLeTkvuPAidjB8dM42vz+1X61t 9IVQT9X4hnAxRjI5CqZOo41GS4Tl1X+ykGoA+VE80BR/R/+nQ3tXDVULfppzeB+vu3 TWnnpaZ2GyoNyPlMiyVRkHdXzDVgA4uQHxwHn7otGK5J4lzyu8KrFyQtiP+f6hfu5v crJkYS8e9A+rfzUmKWuyHcKcmhPhAVJ4XdpzZcDXXlMHVxG7nR1o88xttC6D1+oNIP 9xHI3DkNBpEuAIf you're asking about syncronization upon hot plug of memory, the hardware always goes to read the available index from memory when a new hardware object is associted with a virtqueue. You can argue then that you don't need to restore the available index and you may be right but this is the currect inteface to the firmware.
Quoted 3 lines
If you're asking on generally how sync is assured when the guest updates the available index, can you please send a pointer to the code where you see the update without a memory barrier?
This is a snippet of
virtqueue_add_split()whereavail_indexgets updated by guest:/* Put entry in available array (but don't update avail->idx until they * do sync). */ avail = vq->split.avail_idx_shadow & (vq->split.vring.num - 1); vq->split.vring.avail->ring[avail] = cpu_to_virtio16(_vq->vdev, head); /* Descriptors and available array need to be set before we expose the * new available array entries. */ virtio_wmb(vq->weak_barriers); vq->split.avail_idx_shadow++; vq->split.vring.avail->idx = cpu_to_virtio16(_vq->vdev, vq->split.avail_idx_shadow); vq->num_added++;
Quoted 9 lines
There's memory barrier to make sure the update to descriptor and available ring is seen before writing to the
avail->idx, but there seems no gurantee that this update would flush to the memory immmedately either before or after the mlx5-vdpa is suspened and gets the oldavail_indexvalue stashed somewhere. In this case, how does the hardware ensure the consistency between the guest virtio and host mlx5-vdpa? Or, it completly relies on guest to update theavail_indexonce the next buffer is available, so that the index will be in sync again?
I'm not sure I get the question but notice that the driver should check and notify virtqueue when device want a notification. So there's a virtio_wmb() e.g in:
static bool virtqueue_kick_prepare_split(struct virtqueue *_vq) {
struct vring_virtqueue *vq = to_vvq(_vq);
u16 new, old;
bool needs_kick;
START_USE(vq);
/* We need to expose available array entries before checking avail
* event. */
virtio_mb(vq->weak_barries);
old = vq->split.avail_idx_shadow - vq->num_added; new = vq->split.avail_idx_shadow; vq->num_added = 0;
(See the comment above virtio_mb()). So the avail idx is guaranteed to be committed to memroy(cache hierarchy) before the check of notification. I think we sync through this.
Thanks
Quoted 53 lines
Thanks,
-SiweiQuoted 49 lines
Quoted 49 lines
Thanks,
-SiweiQuoted 46 lines
But you're correct. When memory is added, I get a new memory map. This requires me to build a new memory key object which covers the new memory map. Since the virtqueue objects are referencing this memory key, I need to destroy them and build new ones referncing the new memory key.
Quoted 41 lines
Thanks
Quoted 38 lines
-Siwei
Quoted 36 lines
Thanks
Quoted 33 lines
Quoted 33 lines
-Siwei
Quoted 30 lines
Signed-off-by Eli Cohen <[email protected]>--- drivers/vdpa/mlx5/net/mlx5_vnet.c | 8 -------- 1 file changed, 8 deletions(-)drivers/vdpa/mlx5/net/mlx5_vnet.c +0
-8@@ -1148,8 +1148,6 @@ static int setup_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvq)static void suspend_vq (struct mlx5_vdpa_net *ndev,struct mlx5_vdpa_virtqueue *mvq) { -struct mlx5_virtq_attr attr; -if (!mvq->initialized)return ;@@ -1158,12 +1156,6 @@ static void suspend_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mif (modify_virtqueue (ndev, mvq,MLX5_VIRTIO_NET_Q_OBJECT_STATE_SUSPEND ))mlx5_vdpa_warn (&ndev->mvdev,"modify to suspend failed\n" ); - -if (query_virtqueue (ndev, mvq, &attr)) { -mlx5_vdpa_warn (&ndev->mvdev,"failed to query virtqueue\n" ); -return ; - } - mvq->avail_idx = attr.available_index; }
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
drivers/vdpa/mlx5/net/mlx5_vnet.c +0
-1static void suspend_vqs (struct mlx5_vdpa_net *ndev) --2.29.2
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH 1/2] vdpa/mlx5: Avoid unnecessary query virtqueue
Quoted 125 lines · Jason Wang
Quoted 92 lines · Si-Wei Liu
Quoted 72 lines · Eli Cohen
Quoted 45 lines · Si-Wei Liu
Thanks Eli and Jason for clarifications. See inline.
Quoted 36 lines · Eli Cohen
Quoted 35 lines · Jason Wang
Quoted 33 lines · Si-Wei Liu
Quoted 24 lines · Jason Wang
Quoted 23 lines · Si-Wei Liu
Quoted 22 lines · Si-Wei Liu
Quoted 3 lines · Eli Cohen
suspend_vqshould only suspend the VQ on not save the current available index. This is done when a change of map occurs when the driver callssave_channel_info().Hmmm,
suspend_vq()is also called byteardown_vq(), the latter of which doesn't save the available index assave_channel_info()doesn't get called in that path at all. How does it handle the case thataget_vq_state()is called from userspace (e.g. QEMU) while the hardware VQ object was torn down, but userspace still wants to access the queue index?Refer to https://lore.kernel.org/netdev/[email protected]/
vhost VQ 0 ring restore failed: -1: Resource temporarily unavailable (11) vhost VQ 1 ring restore failed: -1: Resource temporarily unavailable (11)
QEMU will complain with the above warning while VM is being rebooted or shut down.
Looks to me either the kernel driver should cover this requirement, or the userspace has to bear the burden in saving the index and not call into kernel if VQ is destroyed.
Quoted 6 lines
Quoted 5 lines
Actually, the userspace doesn't have the insights whether virt queue will be destroyed if just changing the device status via
set_status(). Looking at other vdpa driver in tree i.e. ifcvf it doesn't behave like so. Hence this still looks to me to be Mellanox specifics andmlx5_vdpaimplementation detail that shouldn't expose to userspace.So I think we can simply drop this patch?
Yep, I think so. To be honest I don't know why it has anything to do with the memory hotplug issue.
Quoted 6 lines
Quoted 4 lines
Eli may know more, my understanding is that, during memory hotplut, qemu need to propagate new memory mappings via
set_map(). For mellanox, it means it needs to rebuild memory keys, so the virtqueue needs to be suspended.I think Siwei was asking why the first patch was related to the hotplug issue.
Quoted 7 lines
I was thinking how consistency is assured when saving/restoring this h/w
avail_indexagainst the one in the virtq memory, particularly in the region_add/.region_del() context (e.g. the hotplug case). Problem is I don't see explicit memory barrier when guest thread updates theavail_index, how does the device make sure the h/wavail_indexis uptodate while guest may race with updating the virtq'savail_indexin the mean while? Maybe I completely miss something obvious?DKIM-Signature: v1; arsa-sha256; crelaxed/relaxed; dnvidia.com; sn1; t 12257780; bhHnB0z4VEKwRS3WGY8d836MJgxu5Eln/jbFZlNXVxc08; hX-PGP-Universal:Date:From:To:CC:Subject:Message-ID:References: MIME-Version:Content-Type:Content-Disposition: Content-Transfer-Encoding:In-Reply-To:User-Agent:X-Originating-IP: X-ClientProxiedBy; bgGmb8+rcn3/rKzKQ/7QzSnghWzZ+FAU0XntsRZYGQ66sFvT7zsYPHogG3LIWNY77t wNHPw7GCJrNaH3nEXPbOp0FMOZw4Kv4W7UPuYPobbLeTkvuPAidjB8dM42vz+1X61t 9IVQT9X4hnAxRjI5CqZOo41GS4Tl1X+ykGoA+VE80BR/R/+nQ3tXDVULfppzeB+vu3 TWnnpaZ2GyoNyPlMiyVRkHdXzDVgA4uQHxwHn7otGK5J4lzyu8KrFyQtiP+f6hfu5v crJkYS8e9A+rfzUmKWuyHcKcmhPhAVJ4XdpzZcDXXlMHVxG7nR1o88xttC6D1+oNIP 9xHI3DkNBpEuAIf you're asking about syncronization upon hot plug of memory, the hardware always goes to read the available index from memory when a new hardware object is associted with a virtqueue. You can argue then that you don't need to restore the available index and you may be right but this is the currect inteface to the firmware.
Quoted 3 lines
If you're asking on generally how sync is assured when the guest updates the available index, can you please send a pointer to the code where you see the update without a memory barrier?
This is a snippet of
virtqueue_add_split()whereavail_indexgets updated by guest:/* Put entry in available array (but don't update avail->idx until they * do sync). */ avail = vq->split.avail_idx_shadow & (vq->split.vring.num - 1); vq->split.vring.avail->ring[avail] = cpu_to_virtio16(_vq->vdev, head); /* Descriptors and available array need to be set before we expose the * new available array entries. */ virtio_wmb(vq->weak_barriers); vq->split.avail_idx_shadow++; vq->split.vring.avail->idx = cpu_to_virtio16(_vq->vdev, vq->split.avail_idx_shadow); vq->num_added++;Quoted 9 lines
There's memory barrier to make sure the update to descriptor and available ring is seen before writing to the
avail->idx, but there seems no gurantee that this update would flush to the memory immmedately either before or after the mlx5-vdpa is suspened and gets the oldavail_indexvalue stashed somewhere. In this case, how does the hardware ensure the consistency between the guest virtio and host mlx5-vdpa? Or, it completly relies on guest to update theavail_indexonce the next buffer is available, so that the index will be in sync again?I'm not sure I get the question but notice that the driver should check and notify virtqueue when device want a notification. So there's a
virtio_wmb()e.g in:static bool
virtqueue_kick_prepare_split(struct virtqueue *_vq){ struct vring_virtqueue *vq = to_vvq(_vq); u16 new, old; bool needs_kick; START_USE(vq); /* We need to expose available array entries before checking avail * event. */ virtio_mb(vq->weak_barries); old = vq->split.avail_idx_shadow - vq->num_added; new = vq->split.avail_idx_shadow; vq->num_added = 0;
Quoted 3 lines
(See the comment above
virtio_mb()). So the avail idx is guaranteed to be committed to memroy(cache hierarchy) before the check of notification. I think we sync through this.
Thanks for pointing it out! Indeed, the avail index is synced before kicking the device. However, even so I think the race might still be possible, see below:
mlx5_vdpa device virtio driver
-------------------------------------------------------------------------
virtqueue_add_split
(bumped up avail_idx1 to avail_idx2, but update not yet committed to mem)
(hot plug memory...)mlx5_vdpa_set_map
mlx5_vdpa_change_map
suspend_vqs
suspend_vq
(avail_idx1 was saved)
save_channels_info
:
: virtqueue_kick_prepare_split
: (avail_idx2 committed to memory)
restore_channels_info
setup_driver
...
create_virtqueue
(saved avail_idx1
getting restored;
whereas current
avail_idx2 in
memory is ignored)
:
:
virtqueue_notify
vp_notifymlx5_vdpa_kick_vq
(device processing up to
avail_idx1 rather than
avail_idx2?)
According to Eli, the vdpa firmware does not load the latest value, i.e. avail_idx2 from memory when re-creating the virtqueue object. Instead it starts with a stale avail_idx1 that was saved in mlx5_vdpa driver private structure before the memory update is made. That is the way how the current firmware interface is designed. The thing I'm not sure though is if the firmware/device will resync and get to the latest avail_idx2 from memory while processing the kick request with mlx5_vdpa_kick_vq? If not, a few avail entries will be missing in the kick, which could cause odd behavior or implicit failure.
But if the firmware will resync on kick_vq (and get_vq_state), I think I completely lost the point of saving and restoring avail_idx when changing the memory map...
Thanks,
-Siwei
Quoted 62 lines
Thanks
Quoted 57 lines
Thanks,
-SiweiQuoted 53 lines
Quoted 53 lines
Thanks,
-SiweiQuoted 50 lines
But you're correct. When memory is added, I get a new memory map. This requires me to build a new memory key object which covers the new memory map. Since the virtqueue objects are referencing this memory key, I need to destroy them and build new ones referncing the new memory key.
Quoted 45 lines
Thanks
Quoted 42 lines
-Siwei
Quoted 40 lines
Thanks
Quoted 37 lines
Quoted 37 lines
-Siwei
Quoted 34 lines
Signed-off-by Eli Cohen <[email protected]>--- drivers/vdpa/mlx5/net/mlx5_vnet.c | 8 -------- 1 file changed, 8 deletions(-)drivers/vdpa/mlx5/net/mlx5_vnet.c +0
-9@@ -1148,8 +1148,6 @@ static int setup_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvq)static void suspend_vq (struct mlx5_vdpa_net *ndev,struct mlx5_vdpa_virtqueue *mvq) { -struct mlx5_virtq_attr attr; -if (!mvq->initialized)return ;@@ -1158,12 +1156,6 @@ static void suspend_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mif (modify_virtqueue (ndev, mvq,MLX5_VIRTIO_NET_Q_OBJECT_STATE_SUSPEND ))mlx5_vdpa_warn (&ndev->mvdev,"modify to suspend failed\n" ); - -if (query_virtqueue (ndev, mvq, &attr)) { -mlx5_vdpa_warn (&ndev->mvdev,"failed to query virtqueue\n" ); -return ; - } - mvq->avail_idx = attr.available_index; }static void suspend_vqs (struct mlx5_vdpa_net *ndev) --2.29.2
Re: [PATCH 1/2] vdpa/mlx5: Avoid unnecessary query virtqueue
Quoted 149 lines · Si-Wei Liu
Quoted 122 lines · Jason Wang
Quoted 91 lines · Si-Wei Liu
Quoted 71 lines · Eli Cohen
Quoted 44 lines · Si-Wei Liu
Thanks Eli and Jason for clarifications. See inline.
Quoted 33 lines · Eli Cohen
Quoted 32 lines · Jason Wang
Quoted 31 lines · Si-Wei Liu
Quoted 30 lines · Jason Wang
Quoted 20 lines · Si-Wei Liu
Quoted 19 lines · Si-Wei Liu
Quoted 3 lines · Eli Cohen
suspend_vqshould only suspend the VQ on not save the current available index. This is done when a change of map occurs when the driver callssave_channel_info().Hmmm,
suspend_vq()is also called byteardown_vq(), the latter of which doesn't save the available index assave_channel_info()doesn't get called in that path at all. How does it handle the case thataget_vq_state()is called from userspace (e.g. QEMU) while the hardware VQ object was torn down, but userspace still wants to access the queue index?Refer to https://lore.kernel.org/netdev/[email protected]/
vhost VQ 0 ring restore failed: -1: Resource temporarily unavailable (11) vhost VQ 1 ring restore failed: -1: Resource temporarily unavailable (11)
QEMU will complain with the above warning while VM is being rebooted or shut down.
Quoted 8 lines
Quoted 3 lines
Looks to me either the kernel driver should cover this requirement, or the userspace has to bear the burden in saving the index and not call into kernel if VQ is destroyed.
Actually, the userspace doesn't have the insights whether virt queue will be destroyed if just changing the device status via
set_status(). Looking at other vdpa driver in tree i.e. ifcvf it doesn't behave like so. Hence this still looks to me to be Mellanox specifics andmlx5_vdpaimplementation detail that shouldn't expose to userspace.So I think we can simply drop this patch?
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Yep, I think so. To be honest I don't know why it has anything to do with the memory hotplug issue.
Eli may know more, my understanding is that, during memory hotplut, qemu need to propagate new memory mappings via
set_map(). For mellanox, it means it needs to rebuild memory keys, so the virtqueue needs to be suspended.I think Siwei was asking why the first patch was related to the hotplug issue.
Quoted 7 lines
I was thinking how consistency is assured when saving/restoring this h/w
avail_indexagainst the one in the virtq memory, particularly in the region_add/.region_del() context (e.g. the hotplug case). Problem is I don't see explicit memory barrier when guest thread updates theavail_index, how does the device make sure the h/wavail_indexis uptodate while guest may race with updating the virtq'savail_indexin the mean while? Maybe I completely miss something obvious?DKIM-Signature: v1; arsa-sha256; crelaxed/relaxed; dnvidia.com; sn1; t 12257780; bhHnB0z4VEKwRS3WGY8d836MJgxu5Eln/jbFZlNXVxc08; hX-PGP-Universal:Date:From:To:CC:Subject:Message-ID:References: MIME-Version:Content-Type:Content-Disposition: Content-Transfer-Encoding:In-Reply-To:User-Agent:X-Originating-IP: X-ClientProxiedBy; bgGmb8+rcn3/rKzKQ/7QzSnghWzZ+FAU0XntsRZYGQ66sFvT7zsYPHogG3LIWNY77t wNHPw7GCJrNaH3nEXPbOp0FMOZw4Kv4W7UPuYPobbLeTkvuPAidjB8dM42vz+1X61t 9IVQT9X4hnAxRjI5CqZOo41GS4Tl1X+ykGoA+VE80BR/R/+nQ3tXDVULfppzeB+vu3 TWnnpaZ2GyoNyPlMiyVRkHdXzDVgA4uQHxwHn7otGK5J4lzyu8KrFyQtiP+f6hfu5v crJkYS8e9A+rfzUmKWuyHcKcmhPhAVJ4XdpzZcDXXlMHVxG7nR1o88xttC6D1+oNIP 9xHI3DkNBpEuAIf you're asking about syncronization upon hot plug of memory, the hardware always goes to read the available index from memory when a new hardware object is associted with a virtqueue. You can argue then that you don't need to restore the available index and you may be right but this is the currect inteface to the firmware.
Quoted 3 lines
If you're asking on generally how sync is assured when the guest updates the available index, can you please send a pointer to the code where you see the update without a memory barrier?
This is a snippet of
virtqueue_add_split()whereavail_indexgets updated by guest:/* Put entry in available array (but don't update avail->idx until they * do sync). */ avail = vq->split.avail_idx_shadow & (vq->split.vring.num - 1); vq->split.vring.avail->ring[avail] = cpu_to_virtio16(_vq->vdev, head); /* Descriptors and available array need to be set before we expose the * new available array entries. */ virtio_wmb(vq->weak_barriers); vq->split.avail_idx_shadow++; vq->split.vring.avail->idx = cpu_to_virtio16(_vq->vdev, vq->split.avail_idx_shadow); vq->num_added++;Quoted 9 lines
There's memory barrier to make sure the update to descriptor and available ring is seen before writing to the
avail->idx, but there seems no gurantee that this update would flush to the memory immmedately either before or after the mlx5-vdpa is suspened and gets the oldavail_indexvalue stashed somewhere. In this case, how does the hardware ensure the consistency between the guest virtio and host mlx5-vdpa? Or, it completly relies on guest to update theavail_indexonce the next buffer is available, so that the index will be in sync again?I'm not sure I get the question but notice that the driver should check and notify virtqueue when device want a notification. So there's a
virtio_wmb()e.g in:static bool
virtqueue_kick_prepare_split(struct virtqueue *_vq){ struct vring_virtqueue *vq = to_vvq(_vq); u16 new, old; bool needs_kick; START_USE(vq); /* We need to expose available array entries before checking avail * event. */ virtio_mb(vq->weak_barries); old = vq->split.avail_idx_shadow - vq->num_added; new = vq->split.avail_idx_shadow; vq->num_added = 0;Quoted 3 lines
(See the comment above
virtio_mb()). So the avail idx is guaranteed to be committed to memroy(cache hierarchy) before the check of notification. I think we sync through this.Thanks for pointing it out! Indeed, the avail index is synced before kicking the device. However, even so I think the race might still be possible, see below:
mlx5_vdpadevice virtio driver
-------------------------------------------------------------------------
virtqueue_add_split
(bumped upavail_idx1toavail_idx2, but update not yet committed to mem)(hot plug memory...)
mlx5_vdpa_set_map
mlx5_vdpa_change_map
suspend_vqs
suspend_vq
(avail_idx1was saved)
save_channels_info
:
:virtqueue_kick_prepare_split
: (avail_idx2committed to memory)
restore_channels_info
Quoted 8 lines
setup_driver
...
create_virtqueue
(savedavail_idx1
getting restored;
whereas current
avail_idx2in
memory is ignored)
The index is not ignored, the device is expected to read descriptors from avail_idx1 until avail_idx2.
Quoted 10 lines
:
:
virtqueue_notify
vp_notifymlx5_vdpa_kick_vq
(device processing up to
avail_idx1rather than
avail_idx2?)
Quoted 2 lines
According to Eli, the vdpa firmware does not load the latest value, i.e.
avail_idx2from memory when re-creating the virtqueue object.
During sync it's not needed but after the device start to work, it needs to read descriptor until the available index.
Quoted 6 lines
Instead it starts with a stale
avail_idx1that was saved inmlx5_vdpadriver private structure before the memory update is made. That is the way how the current firmware interface is designed. The thing I'm not sure though is if thefirmware/devicewill resync and get to the latestavail_idx2from memory while processing the kick request withmlx5_vdpa_kick_vq?
I think the point is that, the device must have an internal avail index, and the device is only expected to sync this internal avail index.
Quoted 6 lines
If not, a few avail entries will be missing in the kick, which could cause odd behavior or implicit failure.
But if the firmware will resync on
kick_vq(andget_vq_state), I think I completely lost the point of saving and restoringavail_idxwhen changing the memory map...
The memory map changing is usually transparent to guest driver but only the device. Maybe you can refer vhost codes or qemu codes for the device logic.
Thanks
Quoted 59 lines
Thanks,
-SiweiQuoted 55 lines
Thanks
Quoted 52 lines
Thanks,
-SiweiQuoted 49 lines
Quoted 49 lines
Thanks,
-SiweiQuoted 46 lines
But you're correct. When memory is added, I get a new memory map. This requires me to build a new memory key object which covers the new memory map. Since the virtqueue objects are referencing this memory key, I need to destroy them and build new ones referncing the new memory key.
Quoted 41 lines
Thanks
Quoted 38 lines
-Siwei
Quoted 36 lines
Thanks
Quoted 33 lines
Quoted 33 lines
-Siwei
Quoted 30 lines
Signed-off-by Eli Cohen <[email protected]>--- drivers/vdpa/mlx5/net/mlx5_vnet.c | 8 -------- 1 file changed, 8 deletions(-)drivers/vdpa/mlx5/net/mlx5_vnet.c +0
-8@@ -1148,8 +1148,6 @@ static int setup_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvq)static void suspend_vq (struct mlx5_vdpa_net *ndev,struct mlx5_vdpa_virtqueue *mvq) { -struct mlx5_virtq_attr attr; -if (!mvq->initialized)return ;@@ -1158,12 +1156,6 @@ static void suspend_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mif (modify_virtqueue (ndev, mvq,MLX5_VIRTIO_NET_Q_OBJECT_STATE_SUSPEND ))mlx5_vdpa_warn (&ndev->mvdev,"modify to suspend failed\n" ); - -if (query_virtqueue (ndev, mvq, &attr)) { -mlx5_vdpa_warn (&ndev->mvdev,"failed to query virtqueue\n" ); -return ; - } - mvq->avail_idx = attr.available_index; }
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
drivers/vdpa/mlx5/net/mlx5_vnet.c +0
-1static void suspend_vqs (struct mlx5_vdpa_net *ndev) --2.29.2
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH 1/2] vdpa/mlx5: Avoid unnecessary query virtqueue
In reply to Si-Wei LiuQuoted 171 lines · Si-Wei Liu
Quoted 125 lines · Jason Wang
Quoted 92 lines · Si-Wei Liu
Quoted 72 lines · Eli Cohen
Quoted 45 lines · Si-Wei Liu
Thanks Eli and Jason for clarifications. See inline.
Quoted 36 lines · Eli Cohen
Quoted 35 lines · Jason Wang
Quoted 33 lines · Si-Wei Liu
Quoted 24 lines · Jason Wang
Quoted 23 lines · Si-Wei Liu
Quoted 22 lines · Si-Wei Liu
Quoted 3 lines · Eli Cohen
suspend_vqshould only suspend the VQ on not save the current available index. This is done when a change of map occurs when the driver callssave_channel_info().Hmmm,
suspend_vq()is also called byteardown_vq(), the latter of which doesn't save the available index assave_channel_info()doesn't get called in that path at all. How does it handle the case thataget_vq_state()is called from userspace (e.g. QEMU) while the hardware VQ object was torn down, but userspace still wants to access the queue index?Refer to https://lore.kernel.org/netdev/[email protected]/
vhost VQ 0 ring restore failed: -1: Resource temporarily unavailable (11) vhost VQ 1 ring restore failed: -1: Resource temporarily unavailable (11)
QEMU will complain with the above warning while VM is being rebooted or shut down.
Looks to me either the kernel driver should cover this requirement, or the userspace has to bear the burden in saving the index and not call into kernel if VQ is destroyed.
Quoted 6 lines
Quoted 5 lines
Actually, the userspace doesn't have the insights whether virt queue will be destroyed if just changing the device status via
set_status(). Looking at other vdpa driver in tree i.e. ifcvf it doesn't behave like so. Hence this still looks to me to be Mellanox specifics andmlx5_vdpaimplementation detail that shouldn't expose to userspace.So I think we can simply drop this patch?
Yep, I think so. To be honest I don't know why it has anything to do with the memory hotplug issue.
Quoted 6 lines
Quoted 4 lines
Eli may know more, my understanding is that, during memory hotplut, qemu need to propagate new memory mappings via
set_map(). For mellanox, it means it needs to rebuild memory keys, so the virtqueue needs to be suspended.I think Siwei was asking why the first patch was related to the hotplug issue.
Quoted 7 lines
I was thinking how consistency is assured when saving/restoring this h/w
avail_indexagainst the one in the virtq memory, particularly in the region_add/.region_del() context (e.g. the hotplug case). Problem is I don't see explicit memory barrier when guest thread updates theavail_index, how does the device make sure the h/wavail_indexis uptodate while guest may race with updating the virtq'savail_indexin the mean while? Maybe I completely miss something obvious?DKIM-Signature: v1; arsa-sha256; crelaxed/relaxed; dnvidia.com; sn1; t 12257780; bhHnB0z4VEKwRS3WGY8d836MJgxu5Eln/jbFZlNXVxc08; hX-PGP-Universal:Date:From:To:CC:Subject:Message-ID:References: MIME-Version:Content-Type:Content-Disposition: Content-Transfer-Encoding:In-Reply-To:User-Agent:X-Originating-IP: X-ClientProxiedBy; bgGmb8+rcn3/rKzKQ/7QzSnghWzZ+FAU0XntsRZYGQ66sFvT7zsYPHogG3LIWNY77t wNHPw7GCJrNaH3nEXPbOp0FMOZw4Kv4W7UPuYPobbLeTkvuPAidjB8dM42vz+1X61t 9IVQT9X4hnAxRjI5CqZOo41GS4Tl1X+ykGoA+VE80BR/R/+nQ3tXDVULfppzeB+vu3 TWnnpaZ2GyoNyPlMiyVRkHdXzDVgA4uQHxwHn7otGK5J4lzyu8KrFyQtiP+f6hfu5v crJkYS8e9A+rfzUmKWuyHcKcmhPhAVJ4XdpzZcDXXlMHVxG7nR1o88xttC6D1+oNIP 9xHI3DkNBpEuAIf you're asking about syncronization upon hot plug of memory, the hardware always goes to read the available index from memory when a new hardware object is associted with a virtqueue. You can argue then that you don't need to restore the available index and you may be right but this is the currect inteface to the firmware.
Quoted 3 lines
If you're asking on generally how sync is assured when the guest updates the available index, can you please send a pointer to the code where you see the update without a memory barrier?
This is a snippet of
virtqueue_add_split()whereavail_indexgets updated by guest:/* Put entry in available array (but don't update avail->idx until they * do sync). */ avail = vq->split.avail_idx_shadow & (vq->split.vring.num - 1); vq->split.vring.avail->ring[avail] = cpu_to_virtio16(_vq->vdev, head); /* Descriptors and available array need to be set before we expose the * new available array entries. */ virtio_wmb(vq->weak_barriers); vq->split.avail_idx_shadow++; vq->split.vring.avail->idx = cpu_to_virtio16(_vq->vdev, vq->split.avail_idx_shadow); vq->num_added++;Quoted 9 lines
There's memory barrier to make sure the update to descriptor and available ring is seen before writing to the
avail->idx, but there seems no gurantee that this update would flush to the memory immmedately either before or after the mlx5-vdpa is suspened and gets the oldavail_indexvalue stashed somewhere. In this case, how does the hardware ensure the consistency between the guest virtio and host mlx5-vdpa? Or, it completly relies on guest to update theavail_indexonce the next buffer is available, so that the index will be in sync again?I'm not sure I get the question but notice that the driver should check and notify virtqueue when device want a notification. So there's a
virtio_wmb()e.g in:static bool
virtqueue_kick_prepare_split(struct virtqueue *_vq){ struct vring_virtqueue *vq = to_vvq(_vq); u16 new, old; bool needs_kick; START_USE(vq); /* We need to expose available array entries before checking avail * event. */ virtio_mb(vq->weak_barries); old = vq->split.avail_idx_shadow - vq->num_added; new = vq->split.avail_idx_shadow; vq->num_added = 0;Quoted 3 lines
(See the comment above
virtio_mb()). So the avail idx is guaranteed to be committed to memroy(cache hierarchy) before the check of notification. I think we sync through this.Thanks for pointing it out! Indeed, the avail index is synced before kicking the device. However, even so I think the race might still be possible, see below:
mlx5_vdpadevice virtio driver
-------------------------------------------------------------------------
virtqueue_add_split
(bumped upavail_idx1toavail_idx2, but update not yet committed to mem)(hot plug memory...)
mlx5_vdpa_set_map
mlx5_vdpa_change_map
suspend_vqs
suspend_vq
(avail_idx1was saved)
save_channels_info
:
:virtqueue_kick_prepare_split
: (avail_idx2committed to memory)
restore_channels_info
setup_driver
...
create_virtqueue
(savedavail_idx1
getting restored;
whereas current
avail_idx2in
memory is ignored)
:
:
virtqueue_notify
vp_notifymlx5_vdpa_kick_vq
(device processing up to
avail_idx1rather than
avail_idx2?)
Quoted 2 lines
According to Eli, the vdpa firmware does not load the latest value, i.e.
avail_idx2from memory when re-creating the virtqueue object.
I said exactly the opposite. The hardware always goes to read the available index from memory. The requirement to configure it when creating a new object is still a requirement defined by the interface so I must not violate interface requirments.
Quoted 77 lines
Instead it starts with a stale
avail_idx1that was saved inmlx5_vdpadriver private structure before the memory update is made. That is the way how the current firmware interface is designed. The thing I'm not sure though is if thefirmware/devicewill resync and get to the latestavail_idx2from memory while processing the kick request withmlx5_vdpa_kick_vq? If not, a few avail entries will be missing in the kick, which could cause odd behavior or implicit failure.But if the firmware will resync on
kick_vq(andget_vq_state), I think I completely lost the point of saving and restoringavail_idxwhen changing the memory map...Thanks,
-SiweiQuoted 62 lines
Thanks
Quoted 57 lines
Thanks,
-SiweiQuoted 53 lines
Quoted 53 lines
Thanks,
-SiweiQuoted 50 lines
But you're correct. When memory is added, I get a new memory map. This requires me to build a new memory key object which covers the new memory map. Since the virtqueue objects are referencing this memory key, I need to destroy them and build new ones referncing the new memory key.
Quoted 45 lines
Thanks
Quoted 42 lines
-Siwei
Quoted 40 lines
Thanks
Quoted 37 lines
Quoted 37 lines
-Siwei
Quoted 34 lines
Signed-off-by Eli Cohen <[email protected]>--- drivers/vdpa/mlx5/net/mlx5_vnet.c | 8 -------- 1 file changed, 8 deletions(-)drivers/vdpa/mlx5/net/mlx5_vnet.c +0
-9@@ -1148,8 +1148,6 @@ static int setup_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvq)static void suspend_vq (struct mlx5_vdpa_net *ndev,struct mlx5_vdpa_virtqueue *mvq) { -struct mlx5_virtq_attr attr; -if (!mvq->initialized)return ;@@ -1158,12 +1156,6 @@ static void suspend_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mif (modify_virtqueue (ndev, mvq,MLX5_VIRTIO_NET_Q_OBJECT_STATE_SUSPEND ))mlx5_vdpa_warn (&ndev->mvdev,"modify to suspend failed\n" ); - -if (query_virtqueue (ndev, mvq, &attr)) { -mlx5_vdpa_warn (&ndev->mvdev,"failed to query virtqueue\n" ); -return ; - } - mvq->avail_idx = attr.available_index; }static void suspend_vqs (struct mlx5_vdpa_net *ndev) --2.29.2
Re: [PATCH 1/2] vdpa/mlx5: Avoid unnecessary query virtqueue
In reply to Si-Wei LiuQuoted 34 lines · Si-Wei Liu
Quoted 33 lines · Jason Wang
Quoted 28 lines · Si-Wei Liu
Quoted 16 lines · Si-Wei Liu
Quoted 3 lines · Eli Cohen
suspend_vqshould only suspend the VQ on not save the current available index. This is done when a change of map occurs when the driver callssave_channel_info().Hmmm,
suspend_vq()is also called byteardown_vq(), the latter of which doesn't save the available index assave_channel_info()doesn't get called in that path at all. How does it handle the case thataget_vq_state()is called from userspace (e.g. QEMU) while the hardware VQ object was torn down, but userspace still wants to access the queue index?Refer to https://lore.kernel.org/netdev/[email protected]/
vhost VQ 0 ring restore failed: -1: Resource temporarily unavailable (11) vhost VQ 1 ring restore failed: -1: Resource temporarily unavailable (11)
Quoted 6 lines
QEMU will complain with the above warning while VM is being rebooted or shut down.
Looks to me either the kernel driver should cover this requirement, or the userspace has to bear the burden in saving the index and not call into kernel if VQ is destroyed.
Actually, the userspace doesn't have the insights whether virt queue will be destroyed if just changing the device status via
set_status(). Looking at other vdpa driver in tree i.e. ifcvf it doesn't behave like so. Hence this still looks to me to be Mellanox specifics andmlx5_vdpaimplementation detail that shouldn't expose to userspace.
Quoted 4 lines
Quoted 1 line
So I think we can simply drop this patch?
Yep, I think so. To be honest I don't know why it has anything to do with the memory hotplug issue.
No relation. That's why I put them in two different patches. Only the second one is the fix as I stated in the cover letter.
Anyway, let's just take the second patch.
Michael, do you need me to send PATCH 2 again as a single patch or can you just take it?
Quoted 45 lines
-Siwei
Quoted 42 lines
Thanks
Quoted 37 lines
Quoted 37 lines
-Siwei
Quoted 34 lines
Signed-off-by Eli Cohen <[email protected]>--- drivers/vdpa/mlx5/net/mlx5_vnet.c | 8 -------- 1 file changed, 8 deletions(-)drivers/vdpa/mlx5/net/mlx5_vnet.c +0
-9@@ -1148,8 +1148,6 @@ static int setup_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvq)static void suspend_vq (struct mlx5_vdpa_net *ndev,struct mlx5_vdpa_virtqueue *mvq) { -struct mlx5_virtq_attr attr; -if (!mvq->initialized)return ;@@ -1158,12 +1156,6 @@ static void suspend_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mif (modify_virtqueue (ndev, mvq,MLX5_VIRTIO_NET_Q_OBJECT_STATE_SUSPEND ))mlx5_vdpa_warn (&ndev->mvdev,"modify to suspend failed\n" ); - -if (query_virtqueue (ndev, mvq, &attr)) { -mlx5_vdpa_warn (&ndev->mvdev,"failed to query virtqueue\n" ); -return ; - } - mvq->avail_idx = attr.available_index; }static void suspend_vqs (struct mlx5_vdpa_net *ndev) --2.29.2
Michael S. Tsirkin
maintainerRe: [PATCH 1/2] vdpa/mlx5: Avoid unnecessary query virtqueue
Quoted 40 lines · Eli Cohen
Quoted 34 lines · Si-Wei Liu
Quoted 33 lines · Jason Wang
Quoted 28 lines · Si-Wei Liu
Quoted 16 lines · Si-Wei Liu
Quoted 3 lines · Eli Cohen
suspend_vqshould only suspend the VQ on not save the current available index. This is done when a change of map occurs when the driver callssave_channel_info().Hmmm,
suspend_vq()is also called byteardown_vq(), the latter of which doesn't save the available index assave_channel_info()doesn't get called in that path at all. How does it handle the case thataget_vq_state()is called from userspace (e.g. QEMU) while the hardware VQ object was torn down, but userspace still wants to access the queue index?Refer to https://lore.kernel.org/netdev/[email protected]/
vhost VQ 0 ring restore failed: -1: Resource temporarily unavailable (11) vhost VQ 1 ring restore failed: -1: Resource temporarily unavailable (11)
Quoted 6 lines
QEMU will complain with the above warning while VM is being rebooted or shut down.
Looks to me either the kernel driver should cover this requirement, or the userspace has to bear the burden in saving the index and not call into kernel if VQ is destroyed.
Actually, the userspace doesn't have the insights whether virt queue will be destroyed if just changing the device status via
set_status(). Looking at other vdpa driver in tree i.e. ifcvf it doesn't behave like so. Hence this still looks to me to be Mellanox specifics andmlx5_vdpaimplementation detail that shouldn't expose to userspace.Quoted 4 lines
Quoted 1 line
So I think we can simply drop this patch?
Yep, I think so. To be honest I don't know why it has anything to do with the memory hotplug issue.
Quoted 7 lines
No relation. That's why I put them in two different patches. Only the second one is the fix as I stated in the cover letter.
Anyway, let's just take the second patch.
Michael, do you need me to send PATCH 2 again as a single patch or can you just take it?
Pls post fixes separately. Thanks!
Quoted 41 lines
Quoted 41 lines
-Siwei
Quoted 37 lines
Thanks
Quoted 33 lines
Quoted 33 lines
-Siwei
Quoted 30 lines
Signed-off-by Eli Cohen <[email protected]>--- drivers/vdpa/mlx5/net/mlx5_vnet.c | 8 -------- 1 file changed, 8 deletions(-)drivers/vdpa/mlx5/net/mlx5_vnet.c +0
-8@@ -1148,8 +1148,6 @@ static int setup_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvq)static void suspend_vq (struct mlx5_vdpa_net *ndev,struct mlx5_vdpa_virtqueue *mvq) { -struct mlx5_virtq_attr attr; -if (!mvq->initialized)return ;@@ -1158,12 +1156,6 @@ static void suspend_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mif (modify_virtqueue (ndev, mvq,MLX5_VIRTIO_NET_Q_OBJECT_STATE_SUSPEND ))mlx5_vdpa_warn (&ndev->mvdev,"modify to suspend failed\n" ); - -if (query_virtqueue (ndev, mvq, &attr)) { -mlx5_vdpa_warn (&ndev->mvdev,"failed to query virtqueue\n" ); -return ; - } - mvq->avail_idx = attr.available_index; }
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
drivers/vdpa/mlx5/net/mlx5_vnet.c +0
-1static void suspend_vqs (struct mlx5_vdpa_net *ndev) --2.29.2
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH 1/2] vdpa/mlx5: Avoid unnecessary query virtqueue
In reply to Jason WangQuoted 25 lines · Jason Wang
Quoted 23 lines · Si-Wei Liu
Quoted 22 lines · Si-Wei Liu
Quoted 3 lines · Eli Cohen
suspend_vqshould only suspend the VQ on not save the current available index. This is done when a change of map occurs when the driver callssave_channel_info().Hmmm,
suspend_vq()is also called byteardown_vq(), the latter of which doesn't save the available index assave_channel_info()doesn't get called in that path at all. How does it handle the case thataget_vq_state()is called from userspace (e.g. QEMU) while the hardware VQ object was torn down, but userspace still wants to access the queue index?Refer to https://lore.kernel.org/netdev/[email protected]/
vhost VQ 0 ring restore failed: -1: Resource temporarily unavailable (11) vhost VQ 1 ring restore failed: -1: Resource temporarily unavailable (11)
QEMU will complain with the above warning while VM is being rebooted or shut down.
Looks to me either the kernel driver should cover this requirement, or the userspace has to bear the burden in saving the index and not call into kernel if VQ is destroyed.
Quoted 9 lines
Quoted 5 lines
Actually, the userspace doesn't have the insights whether virt queue will be destroyed if just changing the device status via
set_status(). Looking at other vdpa driver in tree i.e. ifcvf it doesn't behave like so. Hence this still looks to me to be Mellanox specifics andmlx5_vdpaimplementation detail that shouldn't expose to userspace.So I think we can simply drop this patch?
Yes, I agree. Let's just avoid it.
Quoted 41 lines
Thanks
Quoted 37 lines
Quoted 37 lines
-Siwei
Quoted 34 lines
Signed-off-by Eli Cohen <[email protected]>--- drivers/vdpa/mlx5/net/mlx5_vnet.c | 8 -------- 1 file changed, 8 deletions(-)drivers/vdpa/mlx5/net/mlx5_vnet.c +0
-9@@ -1148,8 +1148,6 @@ static int setup_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvq)static void suspend_vq (struct mlx5_vdpa_net *ndev,struct mlx5_vdpa_virtqueue *mvq) { -struct mlx5_virtq_attr attr; -if (!mvq->initialized)return ;@@ -1158,12 +1156,6 @@ static void suspend_vq(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mif (modify_virtqueue (ndev, mvq,MLX5_VIRTIO_NET_Q_OBJECT_STATE_SUSPEND ))mlx5_vdpa_warn (&ndev->mvdev,"modify to suspend failed\n" ); - -if (query_virtqueue (ndev, mvq, &attr)) { -mlx5_vdpa_warn (&ndev->mvdev,"failed to query virtqueue\n" ); -return ; - } - mvq->avail_idx = attr.available_index; }static void suspend_vqs (struct mlx5_vdpa_net *ndev) --2.29.2
PATCH 2/2 vdpa/mlx5: Restore the hardware used index after change map
In reply to Eli CohenWhen a change of memory map occurs, the hardware resources are destroyed and then re-created again with the new memory map. In such case, we need to restore the hardware available and used indices. The driver failed to restore the used index which is added here.
Fixes 1a86b377aa21 ("vdpa/mlx5: Add VDPA driver for supported mlx5 devices")
--- drivers/vdpa/mlx5/net/mlx5_vnet.c | 7 +++++++ 1 file changed, 7 insertions(+)
drivers/vdpa/mlx5/net/mlx5_vnet.c +7 -0
@@ -87,6 +87,7 @@ struct mlx5_vq_restore_info {u64 device_addr;u64 driver_addr;u16 avail_index; +u16 used_index;bool ready;struct vdpa_callback cb;bool restore;
@@ -121,6 +122,7 @@ struct mlx5_vdpa_virtqueue {u32 virtq_id;struct mlx5_vdpa_net *ndev;u16 avail_idx; +u16 used_idx;int fw_state;/* keep last in the struct */
@@ -804,6 +806,7 @@ static int create_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtque obj_context =MLX5_ADDR_OF (create_virtio_net_q_in, in, obj_context);MLX5_SET (virtio_net_q_object, obj_context, hw_available_index, mvq->avail_idx); +MLX5_SET (virtio_net_q_object, obj_context, hw_used_index, mvq->used_idx);MLX5_SET (virtio_net_q_object, obj_context, queue_feature_bit_mask_12_3,get_features_12_3 (ndev->mvdev.actual_features)); vq_ctx =MLX5_ADDR_OF (virtio_net_q_object, obj_context, virtio_q_context);
@@ -1022,6 +1025,7 @@ static int connect_qps(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mstruct mlx5_virtq_attr {u8 state;u16 available_index; +u16 used_index; }; static int query_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvq,
@@ -1052,6 +1056,7 @@ static int query_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueumemset (attr,0 ,sizeof (*attr)); attr->state =MLX5_GET (virtio_net_q_object, obj_context, state); attr->available_index =MLX5_GET (virtio_net_q_object, obj_context, hw_available_index); + attr->used_index =MLX5_GET (virtio_net_q_object, obj_context, hw_used_index);kfree (out);return 0 ;
@@ -1602,6 +1607,7 @@ static int save_channel_info(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqureturn err; ri->avail_index = attr.available_index; + ri->used_index = attr.used_index; ri->ready = mvq->ready; ri->num_ent = mvq->num_ent; ri->desc_addr = mvq->desc_addr;
@@ -1646,6 +1652,7 @@ static void restore_channels_info(struct mlx5_vdpa_net *ndev)continue ; mvq->avail_idx = ri->avail_index; + mvq->used_idx = ri->used_index; mvq->ready = ri->ready; mvq->num_ent = ri->num_ent; mvq->desc_addr = ri->desc_addr;
Signature · 1 line
2.29.2
Re: [PATCH 2/2] vdpa/mlx5: Restore the hardware used index after change map
Quoted 7 lines · Eli Cohen
When a change of memory map occurs, the hardware resources are destroyed and then re-created again with the new memory map. In such case, we need to restore the hardware available and used indices. The driver failed to restore the used index which is added here.
Fixes
1a86b377aa21("vdpa/mlx5: Add VDPA driver for supported mlx5 devices")Signed-off-by Eli Cohen <[email protected]>
A question. Does this mean after a vq is suspended, the hw used index is not equal to vq used index?
Thanks
Quoted 57 lines
--- drivers/vdpa/mlx5/net/mlx5_vnet.c | 7 +++++++ 1 file changed, 7 insertions(+)drivers/vdpa/mlx5/net/mlx5_vnet.c +6
-0@@ -87,6 +87,7 @@ struct mlx5_vq_restore_info {u64 device_addr;u64 driver_addr;u16 avail_index; +u16 used_index;bool ready;struct vdpa_callback cb;bool restore;@@ -121,6 +122,7 @@ struct mlx5_vdpa_virtqueue {u32 virtq_id;struct mlx5_vdpa_net *ndev;u16 avail_idx; +u16 used_idx;int fw_state;/* keep last in the struct */ @@ -804,6 +806,7 @@ static int create_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtque obj_context =MLX5_ADDR_OF (create_virtio_net_q_in, in, obj_context);MLX5_SET (virtio_net_q_object, obj_context, hw_available_index, mvq->avail_idx); +MLX5_SET (virtio_net_q_object, obj_context, hw_used_index, mvq->used_idx);MLX5_SET (virtio_net_q_object, obj_context, queue_feature_bit_mask_12_3,get_features_12_3 (ndev->mvdev.actual_features)); vq_ctx =MLX5_ADDR_OF (virtio_net_q_object, obj_context, virtio_q_context);@@ -1022,6 +1025,7 @@ static int connect_qps(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mstruct mlx5_virtq_attr {u8 state;u16 available_index; +u16 used_index; }; static int query_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvq,@@ -1052,6 +1056,7 @@ static int query_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueumemset (attr,0 ,sizeof (*attr)); attr->state =MLX5_GET (virtio_net_q_object, obj_context, state); attr->available_index =MLX5_GET (virtio_net_q_object, obj_context, hw_available_index); + attr->used_index =MLX5_GET (virtio_net_q_object, obj_context, hw_used_index);kfree (out);return 0 ;@@ -1602,6 +1607,7 @@ static int save_channel_info(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqureturn err; ri->avail_index = attr.available_index; + ri->used_index = attr.used_index; ri->ready = mvq->ready; ri->num_ent = mvq->num_ent; ri->desc_addr = mvq->desc_addr;
drivers/vdpa/mlx5/net/mlx5_vnet.c static void restore_channels_info(struct mlx5_vdpa_net *ndev) · line 1658
Quoted 7 lines
drivers/vdpa/mlx5/net/mlx5_vnet.c +1
-0@@ -1646,6 +1652,7 @@ static void restore_channels_info(struct mlx5_vdpa_net *ndev)continue ; mvq->avail_idx = ri->avail_index; + mvq->used_idx = ri->used_index; mvq->ready = ri->ready; mvq->num_ent = ri->num_ent; mvq->desc_addr = ri->desc_addr;
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH 2/2] vdpa/mlx5: Restore the hardware used index after change map
Quoted 7 lines · Jason Wang
Quoted 5 lines · Eli Cohen
When a change of memory map occurs, the hardware resources are destroyed and then re-created again with the new memory map. In such case, we need to restore the hardware available and used indices. The driver failed to restore the used index which is added here.
Quoted 6 lines
Quoted 2 lines
Fixes
1a86b377aa21("vdpa/mlx5: Add VDPA driver for supported mlx5 devices")Signed-off-by Eli Cohen <[email protected]>A question. Does this mean after a vq is suspended, the hw used index is not equal to vq used index?
Surely there is just one "Used index" for a VQ. What I was trying to say is that after the VQ is suspended, I read the used index by querying the hardware. The read result is the used index that the hardware wrote to memory. After the I create the new hardware object, I need to tell it what is the used index (and the available index) as a way to sync it with the existing VQ.
This sync is especially important when a change of map occurs while the VQ was already used (hence the indices are likely to be non zero). This can be triggered by hot adding memory after the VQs have been used.
Quoted 63 lines
Thanks
Quoted 58 lines
--- drivers/vdpa/mlx5/net/mlx5_vnet.c | 7 +++++++ 1 file changed, 7 insertions(+)drivers/vdpa/mlx5/net/mlx5_vnet.c +7
-0@@ -87,6 +87,7 @@ struct mlx5_vq_restore_info {u64 device_addr;u64 driver_addr;u16 avail_index; +u16 used_index;bool ready;struct vdpa_callback cb;bool restore;@@ -121,6 +122,7 @@ struct mlx5_vdpa_virtqueue {u32 virtq_id;struct mlx5_vdpa_net *ndev;u16 avail_idx; +u16 used_idx;int fw_state;/* keep last in the struct */ @@ -804,6 +806,7 @@ static int create_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtque obj_context =MLX5_ADDR_OF (create_virtio_net_q_in, in, obj_context);MLX5_SET (virtio_net_q_object, obj_context, hw_available_index, mvq->avail_idx); +MLX5_SET (virtio_net_q_object, obj_context, hw_used_index, mvq->used_idx);MLX5_SET (virtio_net_q_object, obj_context, queue_feature_bit_mask_12_3,get_features_12_3 (ndev->mvdev.actual_features)); vq_ctx =MLX5_ADDR_OF (virtio_net_q_object, obj_context, virtio_q_context);@@ -1022,6 +1025,7 @@ static int connect_qps(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mstruct mlx5_virtq_attr {u8 state;u16 available_index; +u16 used_index; }; static int query_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvq,@@ -1052,6 +1056,7 @@ static int query_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueumemset (attr,0 ,sizeof (*attr)); attr->state =MLX5_GET (virtio_net_q_object, obj_context, state); attr->available_index =MLX5_GET (virtio_net_q_object, obj_context, hw_available_index); + attr->used_index =MLX5_GET (virtio_net_q_object, obj_context, hw_used_index);kfree (out);return 0 ;@@ -1602,6 +1607,7 @@ static int save_channel_info(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqureturn err; ri->avail_index = attr.available_index; + ri->used_index = attr.used_index; ri->ready = mvq->ready; ri->num_ent = mvq->num_ent; ri->desc_addr = mvq->desc_addr;@@ -1646,6 +1652,7 @@ static void restore_channels_info(struct mlx5_vdpa_net *ndev)continue ; mvq->avail_idx = ri->avail_index; + mvq->used_idx = ri->used_index; mvq->ready = ri->ready; mvq->num_ent = ri->num_ent; mvq->desc_addr = ri->desc_addr;
Re: [PATCH 2/2] vdpa/mlx5: Restore the hardware used index after change map
Quoted 10 lines · Eli Cohen
Quoted 9 lines · Jason Wang
Quoted 7 lines · Eli Cohen
When a change of memory map occurs, the hardware resources are destroyed and then re-created again with the new memory map. In such case, we need to restore the hardware available and used indices. The driver failed to restore the used index which is added here.
Fixes
1a86b377aa21("vdpa/mlx5: Add VDPA driver for supported mlx5 devices")Signed-off-by Eli Cohen <[email protected]>
Quoted 6 lines
Quoted 2 lines
A question. Does this mean after a vq is suspended, the hw used index is not equal to vq used index?
Surely there is just one "Used index" for a VQ. What I was trying to say is that after the VQ is suspended, I read the used index by querying the hardware. The read result is the used index that the hardware wrote to memory.
Just to make sure I understand here. So it looks to me we had two index. The first is the used index which is stored in the memory/virtqueue, the second is the one that is stored by the device.
Quoted 3 lines
After the I create the new hardware object, I need to tell it
what is the used index (and the available index) as a way to sync it with the existing VQ.
For avail index I understand that the hardware index is not synced with the avail index stored in the memory/virtqueue. The question is used index, if the hardware one is not synced with the one in the virtqueue. It means after vq is suspended, some requests is not completed by the hardware (e.g the buffer were not put to used ring).
This may have implications to live migration, it means those unaccomplished requests needs to be migrated to the destination and resubmitted to the device. This looks not easy.
Thanks
Quoted 58 lines
This sync is especially important when a change of map occurs while the VQ was already used (hence the indices are likely to be non zero). This can be triggered by hot adding memory after the VQs have been used.
Quoted 53 lines
Thanks
Quoted 50 lines
--- drivers/vdpa/mlx5/net/mlx5_vnet.c | 7 +++++++ 1 file changed, 7 insertions(+)drivers/vdpa/mlx5/net/mlx5_vnet.c +6
-0@@ -87,6 +87,7 @@ struct mlx5_vq_restore_info {u64 device_addr;u64 driver_addr;u16 avail_index; +u16 used_index;bool ready;struct vdpa_callback cb;bool restore;@@ -121,6 +122,7 @@ struct mlx5_vdpa_virtqueue {u32 virtq_id;struct mlx5_vdpa_net *ndev;u16 avail_idx; +u16 used_idx;int fw_state;/* keep last in the struct */ @@ -804,6 +806,7 @@ static int create_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtque obj_context =MLX5_ADDR_OF (create_virtio_net_q_in, in, obj_context);MLX5_SET (virtio_net_q_object, obj_context, hw_available_index, mvq->avail_idx); +MLX5_SET (virtio_net_q_object, obj_context, hw_used_index, mvq->used_idx);MLX5_SET (virtio_net_q_object, obj_context, queue_feature_bit_mask_12_3,get_features_12_3 (ndev->mvdev.actual_features)); vq_ctx =MLX5_ADDR_OF (virtio_net_q_object, obj_context, virtio_q_context);@@ -1022,6 +1025,7 @@ static int connect_qps(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mstruct mlx5_virtq_attr {u8 state;u16 available_index; +u16 used_index; }; static int query_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvq,@@ -1052,6 +1056,7 @@ static int query_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueumemset (attr,0 ,sizeof (*attr)); attr->state =MLX5_GET (virtio_net_q_object, obj_context, state); attr->available_index =MLX5_GET (virtio_net_q_object, obj_context, hw_available_index); + attr->used_index =MLX5_GET (virtio_net_q_object, obj_context, hw_used_index);kfree (out);return 0 ;@@ -1602,6 +1607,7 @@ static int save_channel_info(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqureturn err; ri->avail_index = attr.available_index; + ri->used_index = attr.used_index; ri->ready = mvq->ready; ri->num_ent = mvq->num_ent;
Quoted 8 lines
Quoted 8 lines
drivers/vdpa/mlx5/net/mlx5_vnet.c static void restore_channels_info(struct mlx5_vdpa_net *ndev) · line 1657
Quoted 8 lines
drivers/vdpa/mlx5/net/mlx5_vnet.c +1
-0ri->desc_addr = mvq->desc_addr;@@ -1646,6 +1652,7 @@ static void restore_channels_info(struct mlx5_vdpa_net *ndev)continue ; mvq->avail_idx = ri->avail_index; + mvq->used_idx = ri->used_index; mvq->ready = ri->ready; mvq->num_ent = ri->num_ent; mvq->desc_addr = ri->desc_addr;
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH 2/2] vdpa/mlx5: Restore the hardware used index after change map
Quoted 20 lines · Jason Wang
Quoted 10 lines · Eli Cohen
Quoted 9 lines · Jason Wang
Quoted 7 lines · Eli Cohen
When a change of memory map occurs, the hardware resources are destroyed and then re-created again with the new memory map. In such case, we need to restore the hardware available and used indices. The driver failed to restore the used index which is added here.
Fixes
1a86b377aa21("vdpa/mlx5: Add VDPA driver for supported mlx5 devices")Signed-off-by Eli Cohen <[email protected]>Quoted 6 lines
Quoted 2 lines
A question. Does this mean after a vq is suspended, the hw used index is not equal to vq used index?
Surely there is just one "Used index" for a VQ. What I was trying to say is that after the VQ is suspended, I read the used index by querying the hardware. The read result is the used index that the hardware wrote to memory.
Quoted 4 lines
Just to make sure I understand here. So it looks to me we had two index. The first is the used index which is stored in the memory/virtqueue, the second is the one that is stored by the device.
It is the structures defined in the virtio spec in 2.6.6 for the available ring and 2.6.8 for the used ring. As you know these the available ring is written to by the driver and read by the device. The opposite happens for the used index.
The reason I need to restore the last known indices is for the new hardware objects to sync on the last state and take over from there.
Quoted 86 lines
Quoted 4 lines
After the I create the new hardware object, I need to tell it
what is the used index (and the available index) as a way to sync it with the existing VQ.For avail index I understand that the hardware index is not synced with the avail index stored in the memory/virtqueue. The question is used index, if the hardware one is not synced with the one in the virtqueue. It means after vq is suspended, some requests is not completed by the hardware (e.g the buffer were not put to used ring).
This may have implications to live migration, it means those unaccomplished requests needs to be migrated to the destination and resubmitted to the device. This looks not easy.
Thanks
Quoted 66 lines
This sync is especially important when a change of map occurs while the VQ was already used (hence the indices are likely to be non zero). This can be triggered by hot adding memory after the VQs have been used.
Quoted 61 lines
Thanks
Quoted 58 lines
--- drivers/vdpa/mlx5/net/mlx5_vnet.c | 7 +++++++ 1 file changed, 7 insertions(+)drivers/vdpa/mlx5/net/mlx5_vnet.c +7
-0@@ -87,6 +87,7 @@ struct mlx5_vq_restore_info {u64 device_addr;u64 driver_addr;u16 avail_index; +u16 used_index;bool ready;struct vdpa_callback cb;bool restore;@@ -121,6 +122,7 @@ struct mlx5_vdpa_virtqueue {u32 virtq_id;struct mlx5_vdpa_net *ndev;u16 avail_idx; +u16 used_idx;int fw_state;/* keep last in the struct */ @@ -804,6 +806,7 @@ static int create_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtque obj_context =MLX5_ADDR_OF (create_virtio_net_q_in, in, obj_context);MLX5_SET (virtio_net_q_object, obj_context, hw_available_index, mvq->avail_idx); +MLX5_SET (virtio_net_q_object, obj_context, hw_used_index, mvq->used_idx);MLX5_SET (virtio_net_q_object, obj_context, queue_feature_bit_mask_12_3,get_features_12_3 (ndev->mvdev.actual_features)); vq_ctx =MLX5_ADDR_OF (virtio_net_q_object, obj_context, virtio_q_context);@@ -1022,6 +1025,7 @@ static int connect_qps(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mstruct mlx5_virtq_attr {u8 state;u16 available_index; +u16 used_index; }; static int query_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvq,@@ -1052,6 +1056,7 @@ static int query_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueumemset (attr,0 ,sizeof (*attr)); attr->state =MLX5_GET (virtio_net_q_object, obj_context, state); attr->available_index =MLX5_GET (virtio_net_q_object, obj_context, hw_available_index); + attr->used_index =MLX5_GET (virtio_net_q_object, obj_context, hw_used_index);kfree (out);return 0 ;@@ -1602,6 +1607,7 @@ static int save_channel_info(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqureturn err; ri->avail_index = attr.available_index; + ri->used_index = attr.used_index; ri->ready = mvq->ready; ri->num_ent = mvq->num_ent; ri->desc_addr = mvq->desc_addr;@@ -1646,6 +1652,7 @@ static void restore_channels_info(struct mlx5_vdpa_net *ndev)continue ; mvq->avail_idx = ri->avail_index; + mvq->used_idx = ri->used_index; mvq->ready = ri->ready; mvq->num_ent = ri->num_ent; mvq->desc_addr = ri->desc_addr;
Re: [PATCH 2/2] vdpa/mlx5: Restore the hardware used index after change map
Quoted 18 lines · Eli Cohen
Quoted 17 lines · Jason Wang
Quoted 15 lines · Eli Cohen
Quoted 10 lines · Jason Wang
Quoted 7 lines · Eli Cohen
When a change of memory map occurs, the hardware resources are destroyed and then re-created again with the new memory map. In such case, we need to restore the hardware available and used indices. The driver failed to restore the used index which is added here.
Fixes
1a86b377aa21("vdpa/mlx5: Add VDPA driver for supported mlx5 devices")Signed-off-by Eli Cohen <[email protected]>A question. Does this mean after a vq is suspended, the hw used index is not equal to vq used index?
Surely there is just one "Used index" for a VQ. What I was trying to say is that after the VQ is suspended, I read the used index by querying the hardware. The read result is the used index that the hardware wrote to memory.
Quoted 8 lines
Quoted 4 lines
Just to make sure I understand here. So it looks to me we had two index. The first is the used index which is stored in the memory/virtqueue, the second is the one that is stored by the device.
It is the structures defined in the virtio spec in 2.6.6 for the available ring and 2.6.8 for the used ring. As you know these the available ring is written to by the driver and read by the device. The opposite happens for the used index.
Right, so for used index it was wrote by device. And the device should have an internal used index value that is used to write to the used ring. And the code is used to sync the device internal used index if I understand this correctly.
Quoted 2 lines
The reason I need to restore the last known indices is for the new hardware objects to sync on the last state and take over from there.
Right, after the vq suspending, the questions are:
1) is hardware internal used index might not be the same with the used index in the virtqueue?
or
2) can we simply sync the virtqueue's used index to the hardware's used index?
Thanks
Quoted 75 lines
Quoted 75 lines
Quoted 4 lines
After the I create the new hardware object, I need to tell it
what is the used index (and the available index) as a way to sync it with the existing VQ.For avail index I understand that the hardware index is not synced with the avail index stored in the memory/virtqueue. The question is used index, if the hardware one is not synced with the one in the virtqueue. It means after vq is suspended, some requests is not completed by the hardware (e.g the buffer were not put to used ring).
This may have implications to live migration, it means those unaccomplished requests needs to be migrated to the destination and resubmitted to the device. This looks not easy.
Thanks
Quoted 57 lines
This sync is especially important when a change of map occurs while the VQ was already used (hence the indices are likely to be non zero). This can be triggered by hot adding memory after the VQs have been used.
Quoted 53 lines
Thanks
Quoted 50 lines
--- drivers/vdpa/mlx5/net/mlx5_vnet.c | 7 +++++++ 1 file changed, 7 insertions(+)drivers/vdpa/mlx5/net/mlx5_vnet.c +6
-0@@ -87,6 +87,7 @@ struct mlx5_vq_restore_info {u64 device_addr;u64 driver_addr;u16 avail_index; +u16 used_index;bool ready;struct vdpa_callback cb;bool restore;@@ -121,6 +122,7 @@ struct mlx5_vdpa_virtqueue {u32 virtq_id;struct mlx5_vdpa_net *ndev;u16 avail_idx; +u16 used_idx;int fw_state;/* keep last in the struct */ @@ -804,6 +806,7 @@ static int create_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtque obj_context =MLX5_ADDR_OF (create_virtio_net_q_in, in, obj_context);MLX5_SET (virtio_net_q_object, obj_context, hw_available_index, mvq->avail_idx); +MLX5_SET (virtio_net_q_object, obj_context, hw_used_index, mvq->used_idx);MLX5_SET (virtio_net_q_object, obj_context, queue_feature_bit_mask_12_3,get_features_12_3 (ndev->mvdev.actual_features)); vq_ctx =MLX5_ADDR_OF (virtio_net_q_object, obj_context, virtio_q_context);@@ -1022,6 +1025,7 @@ static int connect_qps(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mstruct mlx5_virtq_attr {u8 state;u16 available_index; +u16 used_index; }; static int query_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvq,@@ -1052,6 +1056,7 @@ static int query_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueumemset (attr,0 ,sizeof (*attr)); attr->state =MLX5_GET (virtio_net_q_object, obj_context, state); attr->available_index =MLX5_GET (virtio_net_q_object, obj_context, hw_available_index); + attr->used_index =MLX5_GET (virtio_net_q_object, obj_context, hw_used_index);kfree (out);return 0 ;@@ -1602,6 +1607,7 @@ static int save_channel_info(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqureturn err; ri->avail_index = attr.available_index; + ri->used_index = attr.used_index; ri->ready = mvq->ready; ri->num_ent = mvq->num_ent;
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
drivers/vdpa/mlx5/net/mlx5_vnet.c static void restore_channels_info(struct mlx5_vdpa_net *ndev) · line 1657
Quoted 8 lines
drivers/vdpa/mlx5/net/mlx5_vnet.c +1
-0ri->desc_addr = mvq->desc_addr;@@ -1646,6 +1652,7 @@ static void restore_channels_info(struct mlx5_vdpa_net *ndev)continue ; mvq->avail_idx = ri->avail_index; + mvq->used_idx = ri->used_index; mvq->ready = ri->ready; mvq->num_ent = ri->num_ent; mvq->desc_addr = ri->desc_addr;
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH 2/2] vdpa/mlx5: Restore the hardware used index after change map
Quoted 36 lines · Jason Wang
Quoted 18 lines · Eli Cohen
Quoted 17 lines · Jason Wang
Quoted 15 lines · Eli Cohen
Quoted 10 lines · Jason Wang
Quoted 7 lines · Eli Cohen
When a change of memory map occurs, the hardware resources are destroyed and then re-created again with the new memory map. In such case, we need to restore the hardware available and used indices. The driver failed to restore the used index which is added here.
Fixes
1a86b377aa21("vdpa/mlx5: Add VDPA driver for supported mlx5 devices")Signed-off-by Eli Cohen <[email protected]>A question. Does this mean after a vq is suspended, the hw used index is not equal to vq used index?
Surely there is just one "Used index" for a VQ. What I was trying to say is that after the VQ is suspended, I read the used index by querying the hardware. The read result is the used index that the hardware wrote to memory.
Quoted 8 lines
Quoted 4 lines
Just to make sure I understand here. So it looks to me we had two index. The first is the used index which is stored in the memory/virtqueue, the second is the one that is stored by the device.
It is the structures defined in the virtio spec in 2.6.6 for the available ring and 2.6.8 for the used ring. As you know these the available ring is written to by the driver and read by the device. The opposite happens for the used index.
Right, so for used index it was wrote by device. And the device should have an internal used index value that is used to write to the used ring. And the code is used to sync the device internal used index if I understand this correctly.
Quoted 9 lines
Quoted 2 lines
The reason I need to restore the last known indices is for the new hardware objects to sync on the last state and take over from there.
Right, after the vq suspending, the questions are:
1) is hardware internal used index might not be the same with the used index in the virtqueue?
Generally the answer is no because the hardware is the only one writing it. New objects start up with the initial value configured to them upon creation. This value was zero before this change.
You could argue that since the hardware has access to virtqueue memory, it could just read the value from there but it does not.
Quoted 5 lines
or
2) can we simply sync the virtqueue's used index to the hardware's used index?
Theoretically it could be done but that's not how the hardware works. One reason is that is not supposed to read from that area. But it is really hardware implementation detail.
Quoted 84 lines
Quoted 83 lines
Quoted 83 lines
Quoted 4 lines
After the I create the new hardware object, I need to tell it
what is the used index (and the available index) as a way to sync it with the existing VQ.For avail index I understand that the hardware index is not synced with the avail index stored in the memory/virtqueue. The question is used index, if the hardware one is not synced with the one in the virtqueue. It means after vq is suspended, some requests is not completed by the hardware (e.g the buffer were not put to used ring).
This may have implications to live migration, it means those unaccomplished requests needs to be migrated to the destination and resubmitted to the device. This looks not easy.
Thanks
Quoted 65 lines
This sync is especially important when a change of map occurs while the VQ was already used (hence the indices are likely to be non zero). This can be triggered by hot adding memory after the VQs have been used.
Quoted 61 lines
Thanks
Quoted 58 lines
--- drivers/vdpa/mlx5/net/mlx5_vnet.c | 7 +++++++ 1 file changed, 7 insertions(+)drivers/vdpa/mlx5/net/mlx5_vnet.c +7
-0@@ -87,6 +87,7 @@ struct mlx5_vq_restore_info {u64 device_addr;u64 driver_addr;u16 avail_index; +u16 used_index;bool ready;struct vdpa_callback cb;bool restore;@@ -121,6 +122,7 @@ struct mlx5_vdpa_virtqueue {u32 virtq_id;struct mlx5_vdpa_net *ndev;u16 avail_idx; +u16 used_idx;int fw_state;/* keep last in the struct */ @@ -804,6 +806,7 @@ static int create_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtque obj_context =MLX5_ADDR_OF (create_virtio_net_q_in, in, obj_context);MLX5_SET (virtio_net_q_object, obj_context, hw_available_index, mvq->avail_idx); +MLX5_SET (virtio_net_q_object, obj_context, hw_used_index, mvq->used_idx);MLX5_SET (virtio_net_q_object, obj_context, queue_feature_bit_mask_12_3,get_features_12_3 (ndev->mvdev.actual_features)); vq_ctx =MLX5_ADDR_OF (virtio_net_q_object, obj_context, virtio_q_context);@@ -1022,6 +1025,7 @@ static int connect_qps(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mstruct mlx5_virtq_attr {u8 state;u16 available_index; +u16 used_index; }; static int query_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvq,@@ -1052,6 +1056,7 @@ static int query_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueumemset (attr,0 ,sizeof (*attr)); attr->state =MLX5_GET (virtio_net_q_object, obj_context, state); attr->available_index =MLX5_GET (virtio_net_q_object, obj_context, hw_available_index); + attr->used_index =MLX5_GET (virtio_net_q_object, obj_context, hw_used_index);kfree (out);return 0 ;@@ -1602,6 +1607,7 @@ static int save_channel_info(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqureturn err; ri->avail_index = attr.available_index; + ri->used_index = attr.used_index; ri->ready = mvq->ready; ri->num_ent = mvq->num_ent; ri->desc_addr = mvq->desc_addr;@@ -1646,6 +1652,7 @@ static void restore_channels_info(struct mlx5_vdpa_net *ndev)continue ; mvq->avail_idx = ri->avail_index; + mvq->used_idx = ri->used_index; mvq->ready = ri->ready; mvq->num_ent = ri->num_ent; mvq->desc_addr = ri->desc_addr;
Jason Wang
Acked-byRe: [PATCH 2/2] vdpa/mlx5: Restore the hardware used index after change map
Quoted 39 lines · Eli Cohen
Quoted 38 lines · Jason Wang
Quoted 17 lines · Eli Cohen
Quoted 16 lines · Jason Wang
Quoted 15 lines · Eli Cohen
Quoted 10 lines · Jason Wang
Quoted 7 lines · Eli Cohen
When a change of memory map occurs, the hardware resources are destroyed and then re-created again with the new memory map. In such case, we need to restore the hardware available and used indices. The driver failed to restore the used index which is added here.
Fixes
1a86b377aa21("vdpa/mlx5: Add VDPA driver for supported mlx5 devices")Signed-off-by Eli Cohen <[email protected]>A question. Does this mean after a vq is suspended, the hw used index is not equal to vq used index?
Surely there is just one "Used index" for a VQ. What I was trying to say is that after the VQ is suspended, I read the used index by querying the hardware. The read result is the used index that the hardware wrote to memory.
Quoted 8 lines
Quoted 4 lines
Just to make sure I understand here. So it looks to me we had two index. The first is the used index which is stored in the memory/virtqueue, the second is the one that is stored by the device.
It is the structures defined in the virtio spec in 2.6.6 for the available ring and 2.6.8 for the used ring. As you know these the available ring is written to by the driver and read by the device. The opposite happens for the used index.
Right, so for used index it was wrote by device. And the device should have an internal used index value that is used to write to the used ring. And the code is used to sync the device internal used index if I understand this correctly.
Quoted 2 lines
The reason I need to restore the last known indices is for the new hardware objects to sync on the last state and take over from there.
Right, after the vq suspending, the questions are:
Quoted 8 lines
Quoted 3 lines
1) is hardware internal used index might not be the same with the used index in the virtqueue?
Generally the answer is no because the hardware is the only one writing it. New objects start up with the initial value configured to them upon creation. This value was zero before this change.
You could argue that since the hardware has access to virtqueue memory, it could just read the value from there but it does not.
I see.
Quoted 9 lines
Quoted 6 lines
or
2) can we simply sync the virtqueue's used index to the hardware's used index?
Theoretically it could be done but that's not how the hardware works. One reason is that is not supposed to read from that area. But it is really hardware implementation detail.
I get you now.
So
Thanks
Quoted 73 lines
Quoted 73 lines
Quoted 73 lines
Quoted 73 lines
Quoted 3 lines
After the I create the new hardware object, I need to tell it
what is the used index (and the available index) as a way to sync it with the existing VQ.For avail index I understand that the hardware index is not synced with the avail index stored in the memory/virtqueue. The question is used index, if the hardware one is not synced with the one in the virtqueue. It means after vq is suspended, some requests is not completed by the hardware (e.g the buffer were not put to used ring).
This may have implications to live migration, it means those unaccomplished requests needs to be migrated to the destination and resubmitted to the device. This looks not easy.
Thanks
Quoted 57 lines
This sync is especially important when a change of map occurs while the VQ was already used (hence the indices are likely to be non zero). This can be triggered by hot adding memory after the VQs have been used.
Quoted 53 lines
Thanks
Quoted 50 lines
--- drivers/vdpa/mlx5/net/mlx5_vnet.c | 7 +++++++ 1 file changed, 7 insertions(+)drivers/vdpa/mlx5/net/mlx5_vnet.c +6
-0@@ -87,6 +87,7 @@ struct mlx5_vq_restore_info {u64 device_addr;u64 driver_addr;u16 avail_index; +u16 used_index;bool ready;struct vdpa_callback cb;bool restore;@@ -121,6 +122,7 @@ struct mlx5_vdpa_virtqueue {u32 virtq_id;struct mlx5_vdpa_net *ndev;u16 avail_idx; +u16 used_idx;int fw_state;/* keep last in the struct */ @@ -804,6 +806,7 @@ static int create_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtque obj_context =MLX5_ADDR_OF (create_virtio_net_q_in, in, obj_context);MLX5_SET (virtio_net_q_object, obj_context, hw_available_index, mvq->avail_idx); +MLX5_SET (virtio_net_q_object, obj_context, hw_used_index, mvq->used_idx);MLX5_SET (virtio_net_q_object, obj_context, queue_feature_bit_mask_12_3,get_features_12_3 (ndev->mvdev.actual_features)); vq_ctx =MLX5_ADDR_OF (virtio_net_q_object, obj_context, virtio_q_context);@@ -1022,6 +1025,7 @@ static int connect_qps(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mstruct mlx5_virtq_attr {u8 state;u16 available_index; +u16 used_index; }; static int query_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueue *mvq,@@ -1052,6 +1056,7 @@ static int query_virtqueue(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqueumemset (attr,0 ,sizeof (*attr)); attr->state =MLX5_GET (virtio_net_q_object, obj_context, state); attr->available_index =MLX5_GET (virtio_net_q_object, obj_context, hw_available_index); + attr->used_index =MLX5_GET (virtio_net_q_object, obj_context, hw_used_index);kfree (out);return 0 ;@@ -1602,6 +1607,7 @@ static int save_channel_info(struct mlx5_vdpa_net *ndev, struct mlx5_vdpa_virtqureturn err; ri->avail_index = attr.available_index; + ri->used_index = attr.used_index; ri->ready = mvq->ready; ri->num_ent = mvq->num_ent;
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
drivers/vdpa/mlx5/net/mlx5_vnet.c static void restore_channels_info(struct mlx5_vdpa_net *ndev) · line 1657
Quoted 8 lines
drivers/vdpa/mlx5/net/mlx5_vnet.c +1
-0ri->desc_addr = mvq->desc_addr;@@ -1646,6 +1652,7 @@ static void restore_channels_info(struct mlx5_vdpa_net *ndev)continue ; mvq->avail_idx = ri->avail_index; + mvq->used_idx = ri->used_index; mvq->ready = ri->ready; mvq->num_ent = ri->num_ent; mvq->desc_addr = ri->desc_addr;
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization