virtio_balloon: quiesce balloon work on device shutdown
Subsystems (2)
v2 cover virtio_balloon: quiesce balloon work on device shutdown
Since commit 8bd2fa086a04 ("virtio: break and reset virtio devices on device_shutdown()") the virtio bus breaks and resets every virtio device during device_shutdown(), i.e. on reboot and kexec. virtio_balloon has no .shutdown of its own, so that generic path runs while the balloon's asynchronous work is still armed: the free page reporting worker, the inflate/deflate and stats workers, the OOM notifier and the free page shrinker.
Once the device has been broken, virtqueue_add_inbuf() in virtballoon_free_page_report() returns -EIO and trips its WARN_ON_ONCE(). On a kernel booted with panic_on_warn that turns an ordinary reboot into a fatal panic in the middle of device_shutdown(), so the machine never reaches the new kernel. The inflate/deflate and OOM paths do not warn but are no better off: they call wait_event(vb->acked, ...) and would block forever on a queue that can no longer complete.
This was hit in the field as an intermittent failure of a virtualization cluster upgrade: guest storage nodes were rebooted via kexec into the new kernel, and the ones whose free page reporting happened to run during device_shutdown() panicked (the guests run with panic_on_warn) and never came back, stalling the rolling upgrade. The crash dump showed the WARN at virtio_balloon.c:216 in a page_reporting kworker, with all the balloon virtqueues already broken.
Validated by churning balloon inflate/deflate from the host while kexec-rebooting the guest in a loop under panic_on_warn: the unpatched kernel reproduces the WARN within a couple of cycles, while the patched kernel survives many consecutive kexec cycles cleanly (12/12 in the final run, 0 WARNs). checkpatch is clean across the series.
Changes in v2:
- Add a virtio_device_shutdown() core helper and call it from the balloon .shutdown handler instead of open-coding break + synchronize_cbs + reset (David Hildenbrand).
- New patch: make tell_host() warn and bail instead of hanging if a buffer add ever fails (David Hildenbrand); kept as a separate patch (Michael S. Tsirkin).
v1: https://lore.kernel.org/all/[email protected]
Denis V. Lunev (4):
virtio: add virtio_device_shutdown() helper
virtio_balloon: factor out virtballoon_quiesce()
virtio_balloon: quiesce balloon work before device shutdown
virtio_balloon: warn on failed buffer add in tell_host()
drivers/virtio/virtio.c | 41 ++++++++++++++++++++++----------- drivers/virtio/virtio_balloon.c | 40 ++++++++++++++++++++++++-------- include/linux/virtio.h | 1 + 3 files changed, 59 insertions(+), 23 deletions(-)
Signature · 1 line
2.53.0
PATCH v2 1/4 virtio: add virtio_device_shutdown() helper
The generic virtio bus .shutdown handler, virtio_dev_shutdown(), breaks and resets a device once it has established that the driver has no .shutdown of its own. A driver that does implement .shutdown, to quiesce its own activity first, still needs the same break and reset afterwards and would otherwise have to open code it.
Factor the break + synchronize_cbs + reset sequence out of virtio_dev_shutdown() into an exported virtio_device_shutdown() helper so such drivers can reuse it instead of duplicating the core logic.
No functional change.
--- drivers/virtio/virtio.c | 41 +++++++++++++++++++++++++++-------------- include/linux/virtio.h | 1 + 2 files changed, 28 insertions(+), 14 deletions(-)
drivers/virtio/virtio.c +27 -14
@@ -401,6 +401,32 @@ static const struct cpumask *virtio_irq_get_affinity(struct device *_d,return dev->config->get_vq_affinity (dev, irq_vec); } +/** +* virtio_device_shutdown - break and reset a device on shutdown +* @dev: the device +* +* Drivers with their own .shutdown method should quiesce their activity and +* then call this to stop the device the way the generic shutdown path does. +*/ +void virtio_device_shutdown (struct virtio_device *dev) +{ +/* +* Some devices get wedged if you kick them after they are +* reset. Mark all vqs as broken to make sure we don't. +*/ +virtio_break_device (dev); +/* +* Guarantee that any callback will see vq->broken as true. +*/ +virtio_synchronize_cbs (dev); +/* +* As IOMMUs are reset on shutdown, this will block device access to memory. +* Some devices get wedged if this happens, so reset to make sure it does not. +*/ + dev->config->reset (dev); +} +EXPORT_SYMBOL_GPL (virtio_device_shutdown); +static void virtio_dev_shutdown (struct device *_d) {struct virtio_device *dev =dev_to_virtio (_d);
@@ -419,20 +445,7 @@ static void virtio_dev_shutdown(struct device *_d)return ; } -/* -* Some devices get wedged if you kick them after they are -* reset. Mark all vqs as broken to make sure we don't. -*/ -virtio_break_device (dev); -/* -* Guarantee that any callback will see vq->broken as true. -*/ -virtio_synchronize_cbs (dev); -/* -* As IOMMUs are reset on shutdown, this will block device access to memory. -* Some devices get wedged if this happens, so reset to make sure it does not. -*/ - dev->config->reset (dev); +virtio_device_shutdown (dev); }static int virtio_dev_num_vf (struct device *dev)
include/linux/virtio.h +1 -0
@@ -213,6 +213,7 @@ int virtio_device_freeze(struct virtio_device *dev);int virtio_device_restore (struct virtio_device *dev);#endif void virtio_reset_device (struct virtio_device *dev); +void virtio_device_shutdown (struct virtio_device *dev);int virtio_device_reset_prepare (struct virtio_device *dev);int virtio_device_reset_done (struct virtio_device *dev);
Signature · 1 line
2.53.0
Re: [PATCH v2 1/4] virtio: add virtio_device_shutdown() helper
Quoted 6 lines · Denis V. Lunev
The generic virtio bus .shutdown handler,
virtio_dev_shutdown(), breaks and resets a device once it has established that the driver has no .shutdown of its own. A driver that does implement .shutdown, to quiesce its own activity first, still needs the same break and reset afterwards and would otherwise have to open code it.
Quoted 8 lines
Factor the break +
synchronize_cbs+ reset sequence out ofvirtio_dev_shutdown()into an exportedvirtio_device_shutdown()helper so such drivers can reuse it instead of duplicating the core logic.No functional change.
Signed-off-by Denis V. Lunev <[email protected]>---
Signature · 3 lines
Cheers, David
PATCH v2 2/4 virtio_balloon: factor out virtballoon_quiesce()
In reply to Denis V. Lunevvirtballoon_remove() stops all of the balloon's asynchronous work (the free page reporting worker, the inflate/deflate and stats workers, the OOM notifier and the free page shrinker) before tearing the device down. A following change needs the same teardown from a .shutdown handler, so move it into a virtballoon_quiesce() helper.
No functional change.
--- drivers/virtio/virtio_balloon.c | 27 ++++++++++++++++++++------- 1 file changed, 20 insertions(+), 7 deletions(-)
drivers/virtio/virtio_balloon.c +20 -7
@@ -1098,26 +1098,39 @@ static void remove_common(struct virtio_balloon *vb) vb->vdev->config->del_vqs (vb->vdev); } -static void virtballoon_remove (struct virtio_device *vdev) +/* +* Stop all asynchronous balloon work. The device must still be alive so that +* in-flight requests can drain via the host before it is reset or freed. +*/ +static void virtballoon_quiesce (struct virtio_balloon *vb) { -struct virtio_balloon *vb = vdev->priv; +struct virtio_device *vdev = vb->vdev; -if (virtio_has_feature (vb->vdev,VIRTIO_BALLOON_F_REPORTING )) +if (virtio_has_feature (vdev,VIRTIO_BALLOON_F_REPORTING ))page_reporting_unregister (&vb->pr_dev_info); -if (virtio_has_feature (vb->vdev,VIRTIO_BALLOON_F_DEFLATE_ON_OOM )) +if (virtio_has_feature (vdev,VIRTIO_BALLOON_F_DEFLATE_ON_OOM ))unregister_oom_notifier (&vb->oom_nb); -if (virtio_has_feature (vb->vdev,VIRTIO_BALLOON_F_FREE_PAGE_HINT )) +if (virtio_has_feature (vdev,VIRTIO_BALLOON_F_FREE_PAGE_HINT ))virtio_balloon_unregister_shrinker (vb); +spin_lock_irq (&vb->stop_update_lock); vb->stop_update = true;spin_unlock_irq (&vb->stop_update_lock);cancel_work_sync (&vb->update_balloon_size_work);cancel_work_sync (&vb->update_balloon_stats_work); -if (virtio_has_feature (vdev,VIRTIO_BALLOON_F_FREE_PAGE_HINT )) { +if (virtio_has_feature (vdev,VIRTIO_BALLOON_F_FREE_PAGE_HINT ))cancel_work_sync (&vb->report_free_page_work); +} + +static void virtballoon_remove (struct virtio_device *vdev) +{ +struct virtio_balloon *vb = vdev->priv; + +virtballoon_quiesce (vb); + +if (virtio_has_feature (vdev,VIRTIO_BALLOON_F_FREE_PAGE_HINT ))destroy_workqueue (vb->balloon_wq); - }remove_common (vb);mutex_destroy (&vb->balloon_lock);
Signature · 1 line
2.53.0
Re: [PATCH v2 2/4] virtio_balloon: factor out virtballoon_quiesce()
Quoted 9 lines · Denis V. Lunev
virtballoon_remove()stops all of the balloon's asynchronous work (the free page reporting worker, the inflate/deflate and stats workers, the OOM notifier and the free page shrinker) before tearing the device down. A following change needs the same teardown from a .shutdown handler, so move it into avirtballoon_quiesce()helper.No functional change.
Signed-off-by Denis V. Lunev <[email protected]>
Signature · 3 lines
Cheers, David
PATCH v2 3/4 virtio_balloon: quiesce balloon work before device shutdown
In reply to Denis V. LunevCommit 8bd2fa086a04 ("virtio: break and reset virtio devices on device_shutdown()") added a generic virtio bus .shutdown handler that breaks and resets every virtio device during device_shutdown(), i.e. on reboot and kexec.
virtio_balloon provides no .shutdown of its own, so that generic path runs while the balloon's asynchronous work is still armed. Once the device has been broken, virtqueue_add_inbuf() in virtballoon_free_page_report() returns -EIO and trips its WARN_ON_ONCE(). On a kernel booted with panic_on_warn that turns an ordinary reboot, for example a kexec based upgrade, into a fatal panic in the middle of device_shutdown(), so the machine never reaches the new kernel.
Relaxing that single WARN_ON_ONCE() would only hide the symptom: the inflate/deflate and OOM paths do not warn, they call wait_event(vb->acked, ...) and would instead block forever on a broken queue that can no longer complete. The device has to be quiesced, not just kept quiet.
Add a .shutdown handler that quiesces the balloon via the shared virtballoon_quiesce() helper while the device is still alive, and only then breaks and resets it via virtio_device_shutdown(). Unlike virtballoon_remove() the balloon workqueue is not destroyed, as shutdown does not free the device and cancel_work_sync() together with stop_update already prevent any further work from being queued.
--- drivers/virtio/virtio_balloon.c | 7 +++++++ 1 file changed, 7 insertions(+)
drivers/virtio/virtio_balloon.c +7 -0
@@ -1137,6 +1137,12 @@ static void virtballoon_remove(struct virtio_device *vdev)kfree (vb); } +static void virtballoon_shutdown (struct virtio_device *vdev) +{ +virtballoon_quiesce (vdev->priv); +virtio_device_shutdown (vdev); +} +#ifdef CONFIG_PM_SLEEP static int virtballoon_freeze (struct virtio_device *vdev) {
@@ -1202,6 +1208,7 @@ static struct virtio_driver virtio_balloon_driver = { .validate = virtballoon_validate, .probe = virtballoon_probe, .remove = virtballoon_remove, + .shutdown = virtballoon_shutdown, .config_changed = virtballoon_changed, #ifdef CONFIG_PM_SLEEP .freeze = virtballoon_freeze,
Signature · 1 line
2.53.0
Re: [PATCH v2 3/4] virtio_balloon: quiesce balloon work before device shutdown
Quoted 38 lines · Denis V. Lunev
Commit
8bd2fa086a04("virtio: break and reset virtio devices ondevice_shutdown()") added a generic virtio bus .shutdown handler that breaks and resets every virtio device duringdevice_shutdown(), i.e. on reboot and kexec.
virtio_balloonprovides no .shutdown of its own, so that generic path runs while the balloon's asynchronous work is still armed. Once the device has been broken,virtqueue_add_inbuf()invirtballoon_free_page_report()returns-EIOand trips itsWARN_ON_ONCE(). On a kernel booted withpanic_on_warnthat turns an ordinary reboot, for example a kexec based upgrade, into a fatal panic in the middle ofdevice_shutdown(), so the machine never reaches the new kernel.Relaxing that single
WARN_ON_ONCE()would only hide the symptom: the inflate/deflate and OOM paths do not warn, they callwait_event(vb->acked, ...)and would instead block forever on a broken queue that can no longer complete. The device has to be quiesced, not just kept quiet.Add a .shutdown handler that quiesces the balloon via the shared
virtballoon_quiesce()helper while the device is still alive, and only then breaks and resets it viavirtio_device_shutdown(). Unlikevirtballoon_remove()the balloon workqueue is not destroyed, as shutdown does not free the device andcancel_work_sync()together withstop_updatealready prevent any further work from being queued.Fixes 8bd2fa086a04 ("virtio: break and reset virtio devices on device_shutdown()")Signed-off-by Denis V. Lunev <[email protected]>--- drivers/virtio/virtio_balloon.c | 7 +++++++ 1 file changed, 7 insertions(+)drivers/virtio/virtio_balloon.c +0
-0
drivers/virtio/virtio_balloon.c static void virtballoon_remove(struct virtio_device *vdev) · line 1144
Quoted 8 lines
drivers/virtio/virtio_balloon.c +5
-0@@ -1137,6 +1137,12 @@ static void virtballoon_remove(struct virtio_device *vdev)kfree (vb); } +static void virtballoon_shutdown (struct virtio_device *vdev) +{ +virtballoon_quiesce (vdev->priv); +virtio_device_shutdown (vdev); +}
I'm curious why virtio_gpu_shutdown() doesn't need that (did not look into the details).
Signature · 3 lines
Cheers, David
Re: [PATCH v2 3/4] virtio_balloon: quiesce balloon work before device shutdown
Quoted 50 lines · David Hildenbrand (Arm)
Quoted 38 lines · Denis V. Lunev
Commit
8bd2fa086a04("virtio: break and reset virtio devices ondevice_shutdown()") added a generic virtio bus .shutdown handler that breaks and resets every virtio device duringdevice_shutdown(), i.e. on reboot and kexec.
virtio_balloonprovides no .shutdown of its own, so that generic path runs while the balloon's asynchronous work is still armed. Once the device has been broken,virtqueue_add_inbuf()invirtballoon_free_page_report()returns-EIOand trips itsWARN_ON_ONCE(). On a kernel booted withpanic_on_warnthat turns an ordinary reboot, for example a kexec based upgrade, into a fatal panic in the middle ofdevice_shutdown(), so the machine never reaches the new kernel.Relaxing that single
WARN_ON_ONCE()would only hide the symptom: the inflate/deflate and OOM paths do not warn, they callwait_event(vb->acked, ...)and would instead block forever on a broken queue that can no longer complete. The device has to be quiesced, not just kept quiet.Add a .shutdown handler that quiesces the balloon via the shared
virtballoon_quiesce()helper while the device is still alive, and only then breaks and resets it viavirtio_device_shutdown(). Unlikevirtballoon_remove()the balloon workqueue is not destroyed, as shutdown does not free the device andcancel_work_sync()together withstop_updatealready prevent any further work from being queued.Fixes 8bd2fa086a04 ("virtio: break and reset virtio devices on device_shutdown()")Signed-off-by Denis V. Lunev <[email protected]>--- drivers/virtio/virtio_balloon.c | 7 +++++++ 1 file changed, 7 insertions(+)drivers/virtio/virtio_balloon.c +0
-0drivers/virtio/virtio_balloon.c static void virtballoon_remove(struct virtio_device *vdev) · line 1144
Quoted 8 lines
drivers/virtio/virtio_balloon.c +5
-0@@ -1137,6 +1137,12 @@ static void virtballoon_remove(struct virtio_device *vdev)kfree (vb); } +static void virtballoon_shutdown (struct virtio_device *vdev) +{ +virtballoon_quiesce (vdev->priv); +virtio_device_shutdown (vdev); +}I'm curious why
virtio_gpu_shutdown()doesn't need that (did not look into the details).
Quoted 2 lines
Reviewed-by David Hildenbrand (Arm) <[email protected]>
I would spend more time with other drivers once we will done with this. I have strong candidate - virtio-mem.
Den
David Hildenbrand (Arm)
maintainerRe: [PATCH v2 3/4] virtio_balloon: quiesce balloon work before device shutdown
Quoted 51 lines · Denis V. Lunev
Quoted 50 lines · David Hildenbrand (Arm)
Quoted 38 lines · Denis V. Lunev
Commit
8bd2fa086a04("virtio: break and reset virtio devices ondevice_shutdown()") added a generic virtio bus .shutdown handler that breaks and resets every virtio device duringdevice_shutdown(), i.e. on reboot and kexec.
virtio_balloonprovides no .shutdown of its own, so that generic path runs while the balloon's asynchronous work is still armed. Once the device has been broken,virtqueue_add_inbuf()invirtballoon_free_page_report()returns-EIOand trips itsWARN_ON_ONCE(). On a kernel booted withpanic_on_warnthat turns an ordinary reboot, for example a kexec based upgrade, into a fatal panic in the middle ofdevice_shutdown(), so the machine never reaches the new kernel.Relaxing that single
WARN_ON_ONCE()would only hide the symptom: the inflate/deflate and OOM paths do not warn, they callwait_event(vb->acked, ...)and would instead block forever on a broken queue that can no longer complete. The device has to be quiesced, not just kept quiet.Add a .shutdown handler that quiesces the balloon via the shared
virtballoon_quiesce()helper while the device is still alive, and only then breaks and resets it viavirtio_device_shutdown(). Unlikevirtballoon_remove()the balloon workqueue is not destroyed, as shutdown does not free the device andcancel_work_sync()together withstop_updatealready prevent any further work from being queued.Fixes 8bd2fa086a04 ("virtio: break and reset virtio devices on device_shutdown()")Signed-off-by Denis V. Lunev <[email protected]>--- drivers/virtio/virtio_balloon.c | 7 +++++++ 1 file changed, 7 insertions(+)drivers/virtio/virtio_balloon.c +0
-0drivers/virtio/virtio_balloon.c static void virtballoon_remove(struct virtio_device *vdev) · line 1144
Quoted 8 lines
drivers/virtio/virtio_balloon.c +5
-0@@ -1137,6 +1137,12 @@ static void virtballoon_remove(struct virtio_device *vdev)kfree (vb); } +static void virtballoon_shutdown (struct virtio_device *vdev) +{ +virtballoon_quiesce (vdev->priv); +virtio_device_shutdown (vdev); +}I'm curious why
virtio_gpu_shutdown()doesn't need that (did not look into the details).
Quoted 4 lines
Quoted 2 lines
Reviewed-by David Hildenbrand (Arm) <[email protected]>I would spend more time with other drivers once we will done with this. I have strong candidate - virtio-mem.
Heh, I briefly checked and it should handle it better I think.
If virtqueue_add_sgs() fails, it propagates the error (-EIO?) back to the main loop where we end up in
switch (rc) {
...
default:
/* Unknown error, mark as broken */
dev_err(&vm->vdev->dev, ...
vm->broken = true;
}And just stop.
But I didn't actually look into the details.
Signature · 3 lines
Cheers, David
PATCH v2 4/4 virtio_balloon: warn on failed buffer add in tell_host()
In reply to Denis V. Lunevtell_host() ignores the return value of virtqueue_add_outbuf() and goes on to kick the queue and wait_event() for the host's ack. The comment claims "We should always be able to add one buffer to an empty queue", but that does not hold once the virtqueue has been broken (e.g. on device shutdown): the add then fails with -EIO and the following wait_event() would block forever on a buffer the host can never return.
Warn and bail out on failure, mirroring virtballoon_free_page_report().
--- drivers/virtio/virtio_balloon.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/virtio/virtio_balloon.c +4 -2
@@ -184,16 +184,18 @@ static void tell_host(struct virtio_balloon *vb, struct virtqueue *vq) {struct scatterlist sg;unsigned int len; +int err;sg_init_one (&sg, vb->pfns,sizeof (vb->pfns[0 ]) * vb->num_pfns);/* We should always be able to add one buffer to an empty queue. */ -virtqueue_add_outbuf (vq, &sg,1 , vb,GFP_KERNEL ); + err =virtqueue_add_outbuf (vq, &sg,1 , vb,GFP_KERNEL ); +if (WARN_ON_ONCE (err)) +return ;virtqueue_kick (vq);/* When host has read buffer, this completes via balloon_ack */ wait_event (vb->acked,virtqueue_get_buf (vq, &len)); - } static int virtballoon_free_page_report(struct page_reporting_dev_info *pr_dev_info,
Signature · 1 line
2.53.0
David Hildenbrand (Arm)
maintainerRe: [PATCH v2 4/4] virtio_balloon: warn on failed buffer add in tell_host()
Quoted 28 lines · Denis V. Lunev
tell_host()ignores the return value ofvirtqueue_add_outbuf()and goes on to kick the queue andwait_event()for the host's ack. The comment claims "We should always be able to add one buffer to an empty queue", but that does not hold once the virtqueue has been broken (e.g. on device shutdown): the add then fails with-EIOand the followingwait_event()would block forever on a buffer the host can never return.Warn and bail out on failure, mirroring
virtballoon_free_page_report().Suggested-by David Hildenbrand <[email protected]>Signed-off-by Denis V. Lunev <[email protected]>--- drivers/virtio/virtio_balloon.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)drivers/virtio/virtio_balloon.c +1
-0@@ -184,16 +184,18 @@ static void tell_host(struct virtio_balloon *vb, struct virtqueue *vq) {struct scatterlist sg;unsigned int len; +int err;sg_init_one (&sg, vb->pfns,sizeof (vb->pfns[0 ]) * vb->num_pfns);/* We should always be able to add one buffer to an empty queue. */
drivers/virtio/virtio_balloon.c static void tell_host(struct virtio_balloon *vb, struct virtqueue *vq) · line 199
Quoted 10 lines
drivers/virtio/virtio_balloon.c +3
-2-virtqueue_add_outbuf (vq, &sg,1 , vb,GFP_KERNEL ); + err =virtqueue_add_outbuf (vq, &sg,1 , vb,GFP_KERNEL ); +if (WARN_ON_ONCE (err)) +return ;virtqueue_kick (vq);/* When host has read buffer, this completes via balloon_ack */ wait_event (vb->acked,virtqueue_get_buf (vq, &len)); - }
We have another uncheck instance in stats_handle_request(), what about that one?
Signature · 3 lines
Cheers, David
[PATCH v2 5/4] virtio_balloon: warn on failed buffer add in stats_handle_request()
Like tell_host(), stats_handle_request() ignores the return value of virtqueue_add_outbuf() and kicks the queue regardless. The same "we should always be able to add one buffer to an empty queue" assumption does not hold once the virtqueue has been broken (e.g. on device shutdown), where the add fails with -EIO. Unlike tell_host() it does not wait_event() afterwards so it cannot hang, but it still kicks a queue with nothing queued.
Warn and bail out on failure, mirroring tell_host() and virtballoon_free_page_report().
--- drivers/virtio/virtio_balloon.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/virtio/virtio_balloon.c +4 -1
@@ -445,6 +445,7 @@ static void stats_handle_request(struct virtio_balloon *vb)struct virtqueue *vq;struct scatterlist sg;unsigned int len, num_stats; +int err; num_stats =update_balloon_stats (vb);
@@ -452,7 +453,9 @@ static void stats_handle_request(struct virtio_balloon *vb)if (!virtqueue_get_buf (vq, &len))return ;sg_init_one (&sg, vb->stats,sizeof (vb->stats[0 ]) * num_stats); -virtqueue_add_outbuf (vq, &sg,1 , vb,GFP_KERNEL ); + err =virtqueue_add_outbuf (vq, &sg,1 , vb,GFP_KERNEL ); +if (WARN_ON_ONCE (err)) +return ;virtqueue_kick (vq); }
Signature · 1 line
2.53.0
Re: [PATCH v2 5/4] virtio_balloon: warn on failed buffer add in stats_handle_request()
Quoted 30 lines · Denis V. Lunev
Like
tell_host(),stats_handle_request()ignores the return value ofvirtqueue_add_outbuf()and kicks the queue regardless. The same "we should always be able to add one buffer to an empty queue" assumption does not hold once the virtqueue has been broken (e.g. on device shutdown), where the add fails with-EIO. Unliketell_host()it does notwait_event()afterwards so it cannot hang, but it still kicks a queue with nothing queued.Warn and bail out on failure, mirroring
tell_host()andvirtballoon_free_page_report().Suggested-by David Hildenbrand <[email protected]>Signed-off-by Denis V. Lunev <[email protected]>--- drivers/virtio/virtio_balloon.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)drivers/virtio/virtio_balloon.c +1
-0@@ -445,6 +445,7 @@ static void stats_handle_request(struct virtio_balloon *vb)struct virtqueue *vq;struct scatterlist sg;unsigned int len, num_stats; +int err; num_stats =update_balloon_stats (vb);
drivers/virtio/virtio_balloon.c static void stats_handle_request(struct virtio_balloon *vb) · line 461
Quoted 10 lines
drivers/virtio/virtio_balloon.c +3
-1@@ -452,7 +453,9 @@ static void stats_handle_request(struct virtio_balloon *vb)if (!virtqueue_get_buf (vq, &len))return ;sg_init_one (&sg, vb->stats,sizeof (vb->stats[0 ]) * num_stats); -virtqueue_add_outbuf (vq, &sg,1 , vb,GFP_KERNEL ); + err =virtqueue_add_outbuf (vq, &sg,1 , vb,GFP_KERNEL ); +if (WARN_ON_ONCE (err)) +return ;virtqueue_kick (vq); }
Although I would just squash #4 and #5.
Signature · 3 lines
Cheers, David
Re: [PATCH v2 5/4] virtio_balloon: warn on failed buffer add in stats_handle_request()
Quoted 43 lines · David Hildenbrand (Arm)
Quoted 30 lines · Denis V. Lunev
Like
tell_host(),stats_handle_request()ignores the return value ofvirtqueue_add_outbuf()and kicks the queue regardless. The same "we should always be able to add one buffer to an empty queue" assumption does not hold once the virtqueue has been broken (e.g. on device shutdown), where the add fails with-EIO. Unliketell_host()it does notwait_event()afterwards so it cannot hang, but it still kicks a queue with nothing queued.Warn and bail out on failure, mirroring
tell_host()andvirtballoon_free_page_report().Suggested-by David Hildenbrand <[email protected]>Signed-off-by Denis V. Lunev <[email protected]>--- drivers/virtio/virtio_balloon.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)drivers/virtio/virtio_balloon.c +1
-0@@ -445,6 +445,7 @@ static void stats_handle_request(struct virtio_balloon *vb)struct virtqueue *vq;struct scatterlist sg;unsigned int len, num_stats; +int err; num_stats =update_balloon_stats (vb);drivers/virtio/virtio_balloon.c static void stats_handle_request(struct virtio_balloon *vb) · line 461
Quoted 10 lines
drivers/virtio/virtio_balloon.c +3
-1@@ -452,7 +453,9 @@ static void stats_handle_request(struct virtio_balloon *vb)if (!virtqueue_get_buf (vq, &len))return ;sg_init_one (&sg, vb->stats,sizeof (vb->stats[0 ]) * num_stats); -virtqueue_add_outbuf (vq, &sg,1 , vb,GFP_KERNEL ); + err =virtqueue_add_outbuf (vq, &sg,1 , vb,GFP_KERNEL ); +if (WARN_ON_ONCE (err)) +return ;virtqueue_kick (vq); }Reviewed-by David Hildenbrand (Arm) <[email protected]>
Quoted 2 lines
Although I would just squash #4 and #5.
Sure thing. Does this way to avoid re-submit if that is possible.
Den
Quoted 48 lines · Denis V. Lunev
This email originated from an IP that might not be authorized by the domain it was sent from. Do not click links or open attachments unless it is an email you expected to receive. Since commit
8bd2fa086a04("virtio: break and reset virtio devices ondevice_shutdown()") the virtio bus breaks and resets every virtio device duringdevice_shutdown(), i.e. on reboot and kexec.virtio_balloonhas no .shutdown of its own, so that generic path runs while the balloon's asynchronous work is still armed: the free page reporting worker, the inflate/deflate and stats workers, the OOM notifier and the free page shrinker.Once the device has been broken,
virtqueue_add_inbuf()invirtballoon_free_page_report()returns-EIOand trips itsWARN_ON_ONCE(). On a kernel booted withpanic_on_warnthat turns an ordinary reboot into a fatal panic in the middle ofdevice_shutdown(), so the machine never reaches the new kernel. The inflate/deflate and OOM paths do not warn but are no better off: they callwait_event(vb->acked, ...)and would block forever on a queue that can no longer complete.This was hit in the field as an intermittent failure of a virtualization cluster upgrade: guest storage nodes were rebooted via kexec into the new kernel, and the ones whose free page reporting happened to run during
device_shutdown()panicked (the guests run withpanic_on_warn) and never came back, stalling the rolling upgrade. The crash dump showed the WARN at virtio_balloon.c:216 in apage_reportingkworker, with all the balloon virtqueues already broken.Validated by churning balloon inflate/deflate from the host while kexec-rebooting the guest in a loop under
panic_on_warn: the unpatched kernel reproduces the WARN within a couple of cycles, while the patched kernel survives many consecutive kexec cycles cleanly (12/12 in the final run, 0 WARNs). checkpatch is clean across the series.Changes in v2:
- Add avirtio_device_shutdown()core helper and call it from the balloon .shutdown handler instead of open-coding break +synchronize_cbs+ reset (David Hildenbrand).
- New patch: maketell_host()warn and bail instead of hanging if a buffer add ever fails (David Hildenbrand); kept as a separate patch (Michael S. Tsirkin).v1: https://lore.kernel.org/all/[email protected]
Denis V. Lunev (4):
virtio: addvirtio_device_shutdown()helper
virtio_balloon: factor outvirtballoon_quiesce()
virtio_balloon: quiesce balloon work before device shutdown
virtio_balloon: warn on failed buffer add intell_host()
Quoted 5 lines
drivers/virtio/virtio.c | 41 ++++++++++++++++++++++----------- drivers/virtio/virtio_balloon.c | 40 ++++++++++++++++++++++++-------- include/linux/virtio.h | 1 + 3 files changed, 59 insertions(+), 23 deletions(-)
Hi, David!
Is this good to go in? I have not seen the confirmation that the series is taken into your tree.
Thank you in advance,
Den
David Hildenbrand (Arm)
maintainerQuoted 49 lines · Denis V. Lunev
Quoted 48 lines · Denis V. Lunev
This email originated from an IP that might not be authorized by the domain it was sent from. Do not click links or open attachments unless it is an email you expected to receive. Since commit
8bd2fa086a04("virtio: break and reset virtio devices ondevice_shutdown()") the virtio bus breaks and resets every virtio device duringdevice_shutdown(), i.e. on reboot and kexec.virtio_balloonhas no .shutdown of its own, so that generic path runs while the balloon's asynchronous work is still armed: the free page reporting worker, the inflate/deflate and stats workers, the OOM notifier and the free page shrinker.Once the device has been broken,
virtqueue_add_inbuf()invirtballoon_free_page_report()returns-EIOand trips itsWARN_ON_ONCE(). On a kernel booted withpanic_on_warnthat turns an ordinary reboot into a fatal panic in the middle ofdevice_shutdown(), so the machine never reaches the new kernel. The inflate/deflate and OOM paths do not warn but are no better off: they callwait_event(vb->acked, ...)and would block forever on a queue that can no longer complete.This was hit in the field as an intermittent failure of a virtualization cluster upgrade: guest storage nodes were rebooted via kexec into the new kernel, and the ones whose free page reporting happened to run during
device_shutdown()panicked (the guests run withpanic_on_warn) and never came back, stalling the rolling upgrade. The crash dump showed the WARN at virtio_balloon.c:216 in apage_reportingkworker, with all the balloon virtqueues already broken.Validated by churning balloon inflate/deflate from the host while kexec-rebooting the guest in a loop under
panic_on_warn: the unpatched kernel reproduces the WARN within a couple of cycles, while the patched kernel survives many consecutive kexec cycles cleanly (12/12 in the final run, 0 WARNs). checkpatch is clean across the series.Changes in v2:
- Add avirtio_device_shutdown()core helper and call it from the balloon .shutdown handler instead of open-coding break +synchronize_cbs+ reset (David Hildenbrand).
- New patch: maketell_host()warn and bail instead of hanging if a buffer add ever fails (David Hildenbrand); kept as a separate patch (Michael S. Tsirkin).v1: https://lore.kernel.org/all/[email protected]
Denis V. Lunev (4):
virtio: addvirtio_device_shutdown()helper
virtio_balloon: factor outvirtballoon_quiesce()
virtio_balloon: quiesce balloon work before device shutdown
virtio_balloon: warn on failed buffer add intell_host()
Quoted 6 lines
Quoted 5 lines
drivers/virtio/virtio.c | 41 ++++++++++++++++++++++----------- drivers/virtio/virtio_balloon.c | 40 ++++++++++++++++++++++++-------- include/linux/virtio.h | 1 + 3 files changed, 59 insertions(+), 23 deletions(-)Hi, David!
Hi! :)
Quoted 3 lines
Is this good to go in? I have not seen the confirmation that the series is taken into your tree.
I don't have a tree (yet), and once I have one it will likely be more mm focused :)
@MST, I think this is good to go!
Signature · 3 lines
Cheers, David
Michael S. Tsirkin
maintainerQuoted 60 lines · David Hildenbrand (Arm)
Quoted 49 lines · Denis V. Lunev
Quoted 48 lines · Denis V. Lunev
This email originated from an IP that might not be authorized by the domain it was sent from. Do not click links or open attachments unless it is an email you expected to receive. Since commit
8bd2fa086a04("virtio: break and reset virtio devices ondevice_shutdown()") the virtio bus breaks and resets every virtio device duringdevice_shutdown(), i.e. on reboot and kexec.virtio_balloonhas no .shutdown of its own, so that generic path runs while the balloon's asynchronous work is still armed: the free page reporting worker, the inflate/deflate and stats workers, the OOM notifier and the free page shrinker.Once the device has been broken,
virtqueue_add_inbuf()invirtballoon_free_page_report()returns-EIOand trips itsWARN_ON_ONCE(). On a kernel booted withpanic_on_warnthat turns an ordinary reboot into a fatal panic in the middle ofdevice_shutdown(), so the machine never reaches the new kernel. The inflate/deflate and OOM paths do not warn but are no better off: they callwait_event(vb->acked, ...)and would block forever on a queue that can no longer complete.This was hit in the field as an intermittent failure of a virtualization cluster upgrade: guest storage nodes were rebooted via kexec into the new kernel, and the ones whose free page reporting happened to run during
device_shutdown()panicked (the guests run withpanic_on_warn) and never came back, stalling the rolling upgrade. The crash dump showed the WARN at virtio_balloon.c:216 in apage_reportingkworker, with all the balloon virtqueues already broken.Validated by churning balloon inflate/deflate from the host while kexec-rebooting the guest in a loop under
panic_on_warn: the unpatched kernel reproduces the WARN within a couple of cycles, while the patched kernel survives many consecutive kexec cycles cleanly (12/12 in the final run, 0 WARNs). checkpatch is clean across the series.Changes in v2:
- Add avirtio_device_shutdown()core helper and call it from the balloon .shutdown handler instead of open-coding break +synchronize_cbs+ reset (David Hildenbrand).
- New patch: maketell_host()warn and bail instead of hanging if a buffer add ever fails (David Hildenbrand); kept as a separate patch (Michael S. Tsirkin).v1: https://lore.kernel.org/all/[email protected]
Denis V. Lunev (4):
virtio: addvirtio_device_shutdown()helper
virtio_balloon: factor outvirtballoon_quiesce()
virtio_balloon: quiesce balloon work before device shutdown
virtio_balloon: warn on failed buffer add intell_host()Quoted 6 lines
Quoted 5 lines
drivers/virtio/virtio.c | 41 ++++++++++++++++++++++----------- drivers/virtio/virtio_balloon.c | 40 ++++++++++++++++++++++++-------- include/linux/virtio.h | 1 + 3 files changed, 59 insertions(+), 23 deletions(-)Hi, David!
Hi! :)
Quoted 1 line
Quoted 11 lines
Quoted 2 lines
Is this good to go in? I have not seen the confirmation that the series is taken into your tree.
I don't have a tree (yet), and once I have one it will likely be more mm focused :)
@MST, I think this is good to go!
Signature · 3 lines
Cheers, David
Indeed, it's just a bugfix and I'm trying to get features into qemu now before their freeze. I'll work on linux end of next week.
Quoted 63 lines · Michael S. Tsirkin
Quoted 62 lines · David Hildenbrand (Arm)
Quoted 49 lines · Denis V. Lunev
Quoted 48 lines · Denis V. Lunev
This email originated from an IP that might not be authorized by the domain it was sent from. Do not click links or open attachments unless it is an email you expected to receive. Since commit
8bd2fa086a04("virtio: break and reset virtio devices ondevice_shutdown()") the virtio bus breaks and resets every virtio device duringdevice_shutdown(), i.e. on reboot and kexec.virtio_balloonhas no .shutdown of its own, so that generic path runs while the balloon's asynchronous work is still armed: the free page reporting worker, the inflate/deflate and stats workers, the OOM notifier and the free page shrinker.Once the device has been broken,
virtqueue_add_inbuf()invirtballoon_free_page_report()returns-EIOand trips itsWARN_ON_ONCE(). On a kernel booted withpanic_on_warnthat turns an ordinary reboot into a fatal panic in the middle ofdevice_shutdown(), so the machine never reaches the new kernel. The inflate/deflate and OOM paths do not warn but are no better off: they callwait_event(vb->acked, ...)and would block forever on a queue that can no longer complete.This was hit in the field as an intermittent failure of a virtualization cluster upgrade: guest storage nodes were rebooted via kexec into the new kernel, and the ones whose free page reporting happened to run during
device_shutdown()panicked (the guests run withpanic_on_warn) and never came back, stalling the rolling upgrade. The crash dump showed the WARN at virtio_balloon.c:216 in apage_reportingkworker, with all the balloon virtqueues already broken.Validated by churning balloon inflate/deflate from the host while kexec-rebooting the guest in a loop under
panic_on_warn: the unpatched kernel reproduces the WARN within a couple of cycles, while the patched kernel survives many consecutive kexec cycles cleanly (12/12 in the final run, 0 WARNs). checkpatch is clean across the series.Changes in v2:
- Add avirtio_device_shutdown()core helper and call it from the balloon .shutdown handler instead of open-coding break +synchronize_cbs+ reset (David Hildenbrand).
- New patch: maketell_host()warn and bail instead of hanging if a buffer add ever fails (David Hildenbrand); kept as a separate patch (Michael S. Tsirkin).v1: https://lore.kernel.org/all/[email protected]
Denis V. Lunev (4):
virtio: addvirtio_device_shutdown()helper
virtio_balloon: factor outvirtballoon_quiesce()
virtio_balloon: quiesce balloon work before device shutdown
virtio_balloon: warn on failed buffer add intell_host()Quoted 6 lines
Quoted 5 lines
drivers/virtio/virtio.c | 41 ++++++++++++++++++++++----------- drivers/virtio/virtio_balloon.c | 40 ++++++++++++++++++++++++-------- include/linux/virtio.h | 1 + 3 files changed, 59 insertions(+), 23 deletions(-)Hi, David!
Hi! :)
Quoted 2 lines
Is this good to go in? I have not seen the confirmation that the series is taken into your tree.
I don't have a tree (yet), and once I have one it will likely be more mm focused :)
Quoted 9 lines
Quoted 6 lines
@MST, I think this is good to go!
Signature · 3 lines
Cheers, DavidIndeed, it's just a bugfix and I'm trying to get features into qemu now before their freeze. I'll work on linux end of next week.
Hi, Michael!
Have you had a chance to take series into your tree, i.e. should I continue to track this submission?
Thank you in advance,
Den
Michael S. Tsirkin
maintainerQuoted 73 lines · Denis V. Lunev
Quoted 63 lines · Michael S. Tsirkin
Quoted 62 lines · David Hildenbrand (Arm)
Quoted 49 lines · Denis V. Lunev
Quoted 48 lines · Denis V. Lunev
This email originated from an IP that might not be authorized by the domain it was sent from. Do not click links or open attachments unless it is an email you expected to receive. Since commit
8bd2fa086a04("virtio: break and reset virtio devices ondevice_shutdown()") the virtio bus breaks and resets every virtio device duringdevice_shutdown(), i.e. on reboot and kexec.virtio_balloonhas no .shutdown of its own, so that generic path runs while the balloon's asynchronous work is still armed: the free page reporting worker, the inflate/deflate and stats workers, the OOM notifier and the free page shrinker.Once the device has been broken,
virtqueue_add_inbuf()invirtballoon_free_page_report()returns-EIOand trips itsWARN_ON_ONCE(). On a kernel booted withpanic_on_warnthat turns an ordinary reboot into a fatal panic in the middle ofdevice_shutdown(), so the machine never reaches the new kernel. The inflate/deflate and OOM paths do not warn but are no better off: they callwait_event(vb->acked, ...)and would block forever on a queue that can no longer complete.This was hit in the field as an intermittent failure of a virtualization cluster upgrade: guest storage nodes were rebooted via kexec into the new kernel, and the ones whose free page reporting happened to run during
device_shutdown()panicked (the guests run withpanic_on_warn) and never came back, stalling the rolling upgrade. The crash dump showed the WARN at virtio_balloon.c:216 in apage_reportingkworker, with all the balloon virtqueues already broken.Validated by churning balloon inflate/deflate from the host while kexec-rebooting the guest in a loop under
panic_on_warn: the unpatched kernel reproduces the WARN within a couple of cycles, while the patched kernel survives many consecutive kexec cycles cleanly (12/12 in the final run, 0 WARNs). checkpatch is clean across the series.Changes in v2:
- Add avirtio_device_shutdown()core helper and call it from the balloon .shutdown handler instead of open-coding break +synchronize_cbs+ reset (David Hildenbrand).
- New patch: maketell_host()warn and bail instead of hanging if a buffer add ever fails (David Hildenbrand); kept as a separate patch (Michael S. Tsirkin).v1: https://lore.kernel.org/all/[email protected]
Denis V. Lunev (4):
virtio: addvirtio_device_shutdown()helper
virtio_balloon: factor outvirtballoon_quiesce()
virtio_balloon: quiesce balloon work before device shutdown
virtio_balloon: warn on failed buffer add intell_host()Quoted 6 lines
Quoted 5 lines
drivers/virtio/virtio.c | 41 ++++++++++++++++++++++----------- drivers/virtio/virtio_balloon.c | 40 ++++++++++++++++++++++++-------- include/linux/virtio.h | 1 + 3 files changed, 59 insertions(+), 23 deletions(-)Hi, David!
Hi! :)
Quoted 2 lines
Is this good to go in? I have not seen the confirmation that the series is taken into your tree.
I don't have a tree (yet), and once I have one it will likely be more mm focused :)
Quoted 9 lines
Quoted 6 lines
@MST, I think this is good to go!
Signature · 3 lines
Cheers, DavidIndeed, it's just a bugfix and I'm trying to get features into qemu now before their freeze. I'll work on linux end of next week.
Quoted 7 lines
Hi, Michael!
Have you had a chance to take series into your tree, i.e. should I continue to track this submission?
Thank you in advance,
Den
Hi!
It's on my queue and will be in the next Linux release, don't worry.
Signature · 1 line
MST