virtio_mem: break device on remove
Subsystems (2)
Michael S. Tsirkin
maintainerPATCH 1/1 virtio_mem: break device on remove
A common pattern for device reset is currently:vdev->config->reset(vdev);
.. cleanup ..
reset prevents new interrupts from arriving and waits for interrupt handlers to finish.
However if - as is common - the handler queues a work request which is flushed during the cleanup stage, we have code adding buffers / trying to get buffers while device is reset. Not good.
This was reproduced by running
modprobe virtio_console
modprobe -r virtio_console
in a loop, and this reasoning seems to apply to virtio mem though I could not reproduce it there.
Fix this up by calling virtio_break_device + flush before reset.
--- drivers/virtio/virtio_mem.c | 2 ++ 1 file changed, 2 insertions(+)
drivers/virtio/virtio_mem.c +2 -0
@@ -2888,6 +2888,8 @@ static void virtio_mem_remove(struct virtio_device *vdev)virtio_mem_deinit_hotplug (vm);/* reset the device and cleanup the queues */ +virtio_break_device (vdev); +flush_work (&vm->wq);virtio_reset_device (vdev); vdev->config->del_vqs (vdev);
Signature · 1 line
MST
Quoted 29 lines · Michael S. Tsirkin
A common pattern for device reset is currently:
vdev->config->reset(vdev);
.. cleanup ..reset prevents new interrupts from arriving and waits for interrupt handlers to finish.
However if - as is common - the handler queues a work request which is flushed during the cleanup stage, we have code adding buffers / trying to get buffers while device is reset. Not good.
This was reproduced by running
modprobevirtio_console
modprobe -rvirtio_console
in a loop, and this reasoning seems to apply to virtio mem though I could not reproduce it there.Fix this up by calling
virtio_break_device+ flush before reset.Signed-off-by Michael S. Tsirkin <[email protected]>--- drivers/virtio/virtio_mem.c | 2 ++ 1 file changed, 2 insertions(+)drivers/virtio/virtio_mem.c +0
-0
drivers/virtio/virtio_mem.c static void virtio_mem_remove(struct virtio_device *vdev) · line 2892
Quoted 5 lines
drivers/virtio/virtio_mem.c +2
-0@@ -2888,6 +2888,8 @@ static void virtio_mem_remove(struct virtio_device *vdev)virtio_mem_deinit_hotplug (vm);/* reset the device and cleanup the queues */ +virtio_break_device (vdev); +flush_work (&vm->wq);
We set vm->removing to true and call cancel_work_sync() in virtio_mem_deinit_hotplug(). Isn't is sufficient?
Thanks
Quoted 3 lines
drivers/virtio/virtio_mem.c +0
-0virtio_reset_device (vdev); vdev->config->del_vqs (vdev);
Michael S. Tsirkin
maintainerQuoted 40 lines · Jason Wang
Quoted 29 lines · Michael S. Tsirkin
A common pattern for device reset is currently:
vdev->config->reset(vdev);
.. cleanup ..reset prevents new interrupts from arriving and waits for interrupt handlers to finish.
However if - as is common - the handler queues a work request which is flushed during the cleanup stage, we have code adding buffers / trying to get buffers while device is reset. Not good.
This was reproduced by running
modprobevirtio_console
modprobe -rvirtio_console
in a loop, and this reasoning seems to apply to virtio mem though I could not reproduce it there.Fix this up by calling
virtio_break_device+ flush before reset.Signed-off-by Michael S. Tsirkin <[email protected]>--- drivers/virtio/virtio_mem.c | 2 ++ 1 file changed, 2 insertions(+)drivers/virtio/virtio_mem.c +0
-0drivers/virtio/virtio_mem.c static void virtio_mem_remove(struct virtio_device *vdev) · line 2891
Quoted 4 lines
drivers/virtio/virtio_mem.c +2
-0@@ -2888,6 +2888,8 @@ static void virtio_mem_remove(struct virtio_device *vdev)virtio_mem_deinit_hotplug (vm);/* reset the device and cleanup the queues */ +virtio_break_device (vdev); +flush_work (&vm->wq);We set
vm->removingto true and callcancel_work_sync()invirtio_mem_deinit_hotplug(). Isn't is sufficient?
Quoted 1 line
Thanks
Hmm I think you are right. David, I will drop this for now. Up to you to consider whether some central capability will be helpful as a replacement for the virtio-mem specific "removing" flag.
Quoted 3 lines
Quoted 3 lines
drivers/virtio/virtio_mem.c +0
-0virtio_reset_device (vdev); vdev->config->del_vqs (vdev);
Quoted 41 lines · Michael S. Tsirkin
Quoted 40 lines · Jason Wang
Quoted 29 lines · Michael S. Tsirkin
A common pattern for device reset is currently:
vdev->config->reset(vdev);
.. cleanup ..reset prevents new interrupts from arriving and waits for interrupt handlers to finish.
However if - as is common - the handler queues a work request which is flushed during the cleanup stage, we have code adding buffers / trying to get buffers while device is reset. Not good.
This was reproduced by running
modprobevirtio_console
modprobe -rvirtio_console
in a loop, and this reasoning seems to apply to virtio mem though I could not reproduce it there.Fix this up by calling
virtio_break_device+ flush before reset.Signed-off-by Michael S. Tsirkin <[email protected]>--- drivers/virtio/virtio_mem.c | 2 ++ 1 file changed, 2 insertions(+)drivers/virtio/virtio_mem.c +0
-0drivers/virtio/virtio_mem.c static void virtio_mem_remove(struct virtio_device *vdev) · line 2891
Quoted 4 lines
drivers/virtio/virtio_mem.c +2
-0@@ -2888,6 +2888,8 @@ static void virtio_mem_remove(struct virtio_device *vdev)virtio_mem_deinit_hotplug (vm);/* reset the device and cleanup the queues */ +virtio_break_device (vdev); +flush_work (&vm->wq);We set
vm->removingto true and callcancel_work_sync()invirtio_mem_deinit_hotplug(). Isn't is sufficient?
Quoted 6 lines
Quoted 1 line
Thanks
Hmm I think you are right. David, I will drop this for now. Up to you to consider whether some central capability will be helpful as a replacement for the virtio-mem specific "removing" flag.
It's all a bit tricky because we also have to handle pending timers and pending memory onlining/offlining operations in a controlled way. Maybe we could convert to virtio_break_device() and use the &dev->vqs_list_lock as a replacement for the removal_lock. However, I'm not 100% sure if it's nice to use that lock from drivers/virtio/virtio_mem.c directly.
Signature · 3 lines
Thanks, David / dhildenb
Michael S. Tsirkin
maintainerQuoted 49 lines · David Hildenbrand
Quoted 41 lines · Michael S. Tsirkin
Quoted 40 lines · Jason Wang
Quoted 29 lines · Michael S. Tsirkin
A common pattern for device reset is currently:
vdev->config->reset(vdev);
.. cleanup ..reset prevents new interrupts from arriving and waits for interrupt handlers to finish.
However if - as is common - the handler queues a work request which is flushed during the cleanup stage, we have code adding buffers / trying to get buffers while device is reset. Not good.
This was reproduced by running
modprobevirtio_console
modprobe -rvirtio_console
in a loop, and this reasoning seems to apply to virtio mem though I could not reproduce it there.Fix this up by calling
virtio_break_device+ flush before reset.Signed-off-by Michael S. Tsirkin <[email protected]>--- drivers/virtio/virtio_mem.c | 2 ++ 1 file changed, 2 insertions(+)drivers/virtio/virtio_mem.c +0
-0drivers/virtio/virtio_mem.c static void virtio_mem_remove(struct virtio_device *vdev) · line 2891
Quoted 4 lines
drivers/virtio/virtio_mem.c +2
-0@@ -2888,6 +2888,8 @@ static void virtio_mem_remove(struct virtio_device *vdev)virtio_mem_deinit_hotplug (vm);/* reset the device and cleanup the queues */ +virtio_break_device (vdev); +flush_work (&vm->wq);We set
vm->removingto true and callcancel_work_sync()invirtio_mem_deinit_hotplug(). Isn't is sufficient?Quoted 6 lines
Quoted 1 line
Thanks
Hmm I think you are right. David, I will drop this for now. Up to you to consider whether some central capability will be helpful as a replacement for the virtio-mem specific "removing" flag.
Quoted 6 lines
It's all a bit tricky because we also have to handle pending timers and pending memory onlining/offlining operations in a controlled way. Maybe we could convert to
virtio_break_device()and use the&dev->vqs_list_lockas a replacement for theremoval_lock. However, I'm not 100% sure if it's nice to use that lock fromdrivers/virtio/virtio_mem.cdirectly.
We could add an API if you like. Or maybe it makes sense to add a separate one that lets you find out that removal started. Need to figure out how to handle suspend too then ...
Generally we have these checks that device is not going away sprinkled over all drivers and I don't like it, but
it's not easy to build a sane API to handle it, especially for high speed things when we can't take locks because performance.
Quoted 4 lines
Signature · 3 lines
Thanks, David / dhildenb
Quoted 57 lines · Michael S. Tsirkin
Quoted 49 lines · David Hildenbrand
Quoted 41 lines · Michael S. Tsirkin
Quoted 40 lines · Jason Wang
Quoted 29 lines · Michael S. Tsirkin
A common pattern for device reset is currently:
vdev->config->reset(vdev);
.. cleanup ..reset prevents new interrupts from arriving and waits for interrupt handlers to finish.
However if - as is common - the handler queues a work request which is flushed during the cleanup stage, we have code adding buffers / trying to get buffers while device is reset. Not good.
This was reproduced by running
modprobevirtio_console
modprobe -rvirtio_console
in a loop, and this reasoning seems to apply to virtio mem though I could not reproduce it there.Fix this up by calling
virtio_break_device+ flush before reset.Signed-off-by Michael S. Tsirkin <[email protected]>--- drivers/virtio/virtio_mem.c | 2 ++ 1 file changed, 2 insertions(+)drivers/virtio/virtio_mem.c +0
-0drivers/virtio/virtio_mem.c static void virtio_mem_remove(struct virtio_device *vdev) · line 2891
Quoted 4 lines
drivers/virtio/virtio_mem.c +2
-0@@ -2888,6 +2888,8 @@ static void virtio_mem_remove(struct virtio_device *vdev)virtio_mem_deinit_hotplug (vm);/* reset the device and cleanup the queues */ +virtio_break_device (vdev); +flush_work (&vm->wq);We set
vm->removingto true and callcancel_work_sync()invirtio_mem_deinit_hotplug(). Isn't is sufficient?Quoted 6 lines
Quoted 1 line
Thanks
Hmm I think you are right. David, I will drop this for now. Up to you to consider whether some central capability will be helpful as a replacement for the virtio-mem specific "removing" flag.
Quoted 6 lines
It's all a bit tricky because we also have to handle pending timers and pending memory onlining/offlining operations in a controlled way. Maybe we could convert to
virtio_break_device()and use the&dev->vqs_list_lockas a replacement for theremoval_lock. However, I'm not 100% sure if it's nice to use that lock fromdrivers/virtio/virtio_mem.cdirectly.
Quoted 7 lines
We could add an API if you like. Or maybe it makes sense to add a separate one that lets you find out that removal started. Need to figure out how to handle suspend too then ...
Generally we have these checks that device is not going away sprinkled over all drivers and I don't like it, but
it's not easy to build a sane API to handle it, especially for high speed things when we can't take locks because performance.
The interesting case might be how to handle virtio_mem_retry(), whereby we conditionally queue work if !removing.
Having that said, in an ideal world we could deny removal requests for virtio_mem completely if there is still any memory added to the system ...
Signature · 3 lines
Thanks, David / dhildenb
Michael S. Tsirkin
maintainerQuoted 66 lines · David Hildenbrand
Quoted 57 lines · Michael S. Tsirkin
Quoted 49 lines · David Hildenbrand
Quoted 41 lines · Michael S. Tsirkin
Quoted 40 lines · Jason Wang
Quoted 29 lines · Michael S. Tsirkin
A common pattern for device reset is currently:
vdev->config->reset(vdev);
.. cleanup ..reset prevents new interrupts from arriving and waits for interrupt handlers to finish.
However if - as is common - the handler queues a work request which is flushed during the cleanup stage, we have code adding buffers / trying to get buffers while device is reset. Not good.
This was reproduced by running
modprobevirtio_console
modprobe -rvirtio_console
in a loop, and this reasoning seems to apply to virtio mem though I could not reproduce it there.Fix this up by calling
virtio_break_device+ flush before reset.Signed-off-by Michael S. Tsirkin <[email protected]>--- drivers/virtio/virtio_mem.c | 2 ++ 1 file changed, 2 insertions(+)drivers/virtio/virtio_mem.c +0
-0drivers/virtio/virtio_mem.c static void virtio_mem_remove(struct virtio_device *vdev) · line 2891
Quoted 4 lines
drivers/virtio/virtio_mem.c +2
-0@@ -2888,6 +2888,8 @@ static void virtio_mem_remove(struct virtio_device *vdev)virtio_mem_deinit_hotplug (vm);/* reset the device and cleanup the queues */ +virtio_break_device (vdev); +flush_work (&vm->wq);We set
vm->removingto true and callcancel_work_sync()invirtio_mem_deinit_hotplug(). Isn't is sufficient?Quoted 6 lines
Quoted 1 line
Thanks
Hmm I think you are right. David, I will drop this for now. Up to you to consider whether some central capability will be helpful as a replacement for the virtio-mem specific "removing" flag.
Quoted 6 lines
It's all a bit tricky because we also have to handle pending timers and pending memory onlining/offlining operations in a controlled way. Maybe we could convert to
virtio_break_device()and use the&dev->vqs_list_lockas a replacement for theremoval_lock. However, I'm not 100% sure if it's nice to use that lock fromdrivers/virtio/virtio_mem.cdirectly.Quoted 7 lines
We could add an API if you like. Or maybe it makes sense to add a separate one that lets you find out that removal started. Need to figure out how to handle suspend too then ...
Generally we have these checks that device is not going away sprinkled over all drivers and I don't like it, but
it's not easy to build a sane API to handle it, especially for high speed things when we can't take locks because performance.
Quoted 11 lines
The interesting case might be how to handle
virtio_mem_retry(), whereby we conditionally queue work if !removing.Having that said, in an ideal world we could deny removal requests for
virtio_memcompletely if there is still any memory added to the system ...Signature · 3 lines
Thanks, David / dhildenb
removal requests might come from guest admin.
Signature · 1 line
MST