Skip to contentSkip to conversation

virtio_balloon: quiesce balloon work on device shutdown

PATCH In review Denis V. Lunev 4 participants 19 messages Jun 24 – Jul 13

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v2 cover virtio_balloon: quiesce balloon work on device shutdown

Since commit 8bd2fa086a04 ("virtio: break and reset virtio devices on device_shutdown()") the virtio bus breaks and resets every virtio device during device_shutdown(), i.e. on reboot and kexec. virtio_balloon has no .shutdown of its own, so that generic path runs while the balloon's asynchronous work is still armed: the free page reporting worker, the inflate/deflate and stats workers, the OOM notifier and the free page shrinker.

Once the device has been broken, virtqueue_add_inbuf() in virtballoon_free_page_report() returns -EIO and trips its WARN_ON_ONCE(). On a kernel booted with panic_on_warn that turns an ordinary reboot into a fatal panic in the middle of device_shutdown(), so the machine never reaches the new kernel. The inflate/deflate and OOM paths do not warn but are no better off: they call wait_event(vb->acked, ...) and would block forever on a queue that can no longer complete.

This was hit in the field as an intermittent failure of a virtualization cluster upgrade: guest storage nodes were rebooted via kexec into the new kernel, and the ones whose free page reporting happened to run during device_shutdown() panicked (the guests run with panic_on_warn) and never came back, stalling the rolling upgrade. The crash dump showed the WARN at virtio_balloon.c:216 in a page_reporting kworker, with all the balloon virtqueues already broken.

Validated by churning balloon inflate/deflate from the host while kexec-rebooting the guest in a loop under panic_on_warn: the unpatched kernel reproduces the WARN within a couple of cycles, while the patched kernel survives many consecutive kexec cycles cleanly (12/12 in the final run, 0 WARNs). checkpatch is clean across the series.

Changes in v2:
- Add a virtio_device_shutdown() core helper and call it from the balloon .shutdown handler instead of open-coding break + synchronize_cbs + reset (David Hildenbrand).
- New patch: make tell_host() warn and bail instead of hanging if a buffer add ever fails (David Hildenbrand); kept as a separate patch (Michael S. Tsirkin).

v1: https://lore.kernel.org/all/[email protected]

Denis V. Lunev (4):
virtio: add virtio_device_shutdown() helper
virtio_balloon: factor out virtballoon_quiesce()
virtio_balloon: quiesce balloon work before device shutdown
virtio_balloon: warn on failed buffer add in tell_host()

 drivers/virtio/virtio.c         | 41 ++++++++++++++++++++++-----------
 drivers/virtio/virtio_balloon.c | 40 ++++++++++++++++++++++++--------
 include/linux/virtio.h          |  1 +
 3 files changed, 59 insertions(+), 23 deletions(-)
Signature · 1 line
2.53.0
#1 lore raw

PATCH v2 1/4 virtio: add virtio_device_shutdown() helper

The generic virtio bus .shutdown handler, virtio_dev_shutdown(), breaks and resets a device once it has established that the driver has no .shutdown of its own. A driver that does implement .shutdown, to quiesce its own activity first, still needs the same break and reset afterwards and would otherwise have to open code it.

Factor the break + synchronize_cbs + reset sequence out of virtio_dev_shutdown() into an exported virtio_device_shutdown() helper so such drivers can reuse it instead of duplicating the core logic.

No functional change.

---
 drivers/virtio/virtio.c | 41 +++++++++++++++++++++++++++--------------
 include/linux/virtio.h  |  1 +
 2 files changed, 28 insertions(+), 14 deletions(-)
drivers/virtio/virtio.c +27 -14
index 299fa83be1d5..75bb4ffe3b87 100644
@@ -401,6 +401,32 @@ static const struct cpumask *virtio_irq_get_affinity(struct device *_d,
 	return dev->config->get_vq_affinity(dev, irq_vec);
 }
 
+/**
+ * virtio_device_shutdown - break and reset a device on shutdown
+ * @dev: the device
+ *
+ * Drivers with their own .shutdown method should quiesce their activity and
+ * then call this to stop the device the way the generic shutdown path does.
+ */
+void virtio_device_shutdown(struct virtio_device *dev)
+{
+	/*
+	 * Some devices get wedged if you kick them after they are
+	 * reset. Mark all vqs as broken to make sure we don't.
+	 */
+	virtio_break_device(dev);
+	/*
+	 * Guarantee that any callback will see vq->broken as true.
+	 */
+	virtio_synchronize_cbs(dev);
+	/*
+	 * As IOMMUs are reset on shutdown, this will block device access to memory.
+	 * Some devices get wedged if this happens, so reset to make sure it does not.
+	 */
+	dev->config->reset(dev);
+}
+EXPORT_SYMBOL_GPL(virtio_device_shutdown);
+
 static void virtio_dev_shutdown(struct device *_d)
 {
 	struct virtio_device *dev = dev_to_virtio(_d);
@@ -419,20 +445,7 @@ static void virtio_dev_shutdown(struct device *_d)
 		return;
 	}
 
-	/*
-	 * Some devices get wedged if you kick them after they are
-	 * reset. Mark all vqs as broken to make sure we don't.
-	 */
-	virtio_break_device(dev);
-	/*
-	 * Guarantee that any callback will see vq->broken as true.
-	 */
-	virtio_synchronize_cbs(dev);
-	/*
-	 * As IOMMUs are reset on shutdown, this will block device access to memory.
-	 * Some devices get wedged if this happens, so reset to make sure it does not.
-	 */
-	dev->config->reset(dev);
+	virtio_device_shutdown(dev);
 }
 
 static int virtio_dev_num_vf(struct device *dev)
include/linux/virtio.h +1 -0
index bf089e51970e..66184828fdd0 100644
@@ -213,6 +213,7 @@ int virtio_device_freeze(struct virtio_device *dev);
 int virtio_device_restore(struct virtio_device *dev);
 #endif
 void virtio_reset_device(struct virtio_device *dev);
+void virtio_device_shutdown(struct virtio_device *dev);
 int virtio_device_reset_prepare(struct virtio_device *dev);
 int virtio_device_reset_done(struct virtio_device *dev);
 
Signature · 1 line
2.53.0
maintainer Reviewed-by on v2 1/4
#2 lore raw

Re: [PATCH v2 1/4] virtio: add virtio_device_shutdown() helper

Quoted 6 lines · Denis V. Lunev

The generic virtio bus .shutdown handler, virtio_dev_shutdown(), breaks and resets a device once it has established that the driver has no .shutdown of its own. A driver that does implement .shutdown, to quiesce its own activity first, still needs the same break and reset afterwards and would otherwise have to open code it.

Quoted 8 lines

Factor the break + synchronize_cbs + reset sequence out of virtio_dev_shutdown() into an exported virtio_device_shutdown() helper so such drivers can reuse it instead of duplicating the core logic.

No functional change.

---

Signature · 3 lines
Cheers,

David
#3 lore raw

PATCH v2 2/4 virtio_balloon: factor out virtballoon_quiesce()

In reply to Denis V. Lunev

virtballoon_remove() stops all of the balloon's asynchronous work (the free page reporting worker, the inflate/deflate and stats workers, the OOM notifier and the free page shrinker) before tearing the device down. A following change needs the same teardown from a .shutdown handler, so move it into a virtballoon_quiesce() helper.

No functional change.

---
 drivers/virtio/virtio_balloon.c | 27 ++++++++++++++++++++-------
 1 file changed, 20 insertions(+), 7 deletions(-)
drivers/virtio/virtio_balloon.c +20 -7
index 088b3a0e6ce6..5b02d9191ac6 100644
@@ -1098,26 +1098,39 @@ static void remove_common(struct virtio_balloon *vb)
 	vb->vdev->config->del_vqs(vb->vdev);
 }
 
-static void virtballoon_remove(struct virtio_device *vdev)
+/*
+ * Stop all asynchronous balloon work. The device must still be alive so that
+ * in-flight requests can drain via the host before it is reset or freed.
+ */
+static void virtballoon_quiesce(struct virtio_balloon *vb)
 {
-	struct virtio_balloon *vb = vdev->priv;
+	struct virtio_device *vdev = vb->vdev;
 
-	if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_REPORTING))
+	if (virtio_has_feature(vdev, VIRTIO_BALLOON_F_REPORTING))
 		page_reporting_unregister(&vb->pr_dev_info);
-	if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_DEFLATE_ON_OOM))
+	if (virtio_has_feature(vdev, VIRTIO_BALLOON_F_DEFLATE_ON_OOM))
 		unregister_oom_notifier(&vb->oom_nb);
-	if (virtio_has_feature(vb->vdev, VIRTIO_BALLOON_F_FREE_PAGE_HINT))
+	if (virtio_has_feature(vdev, VIRTIO_BALLOON_F_FREE_PAGE_HINT))
 		virtio_balloon_unregister_shrinker(vb);
+
 	spin_lock_irq(&vb->stop_update_lock);
 	vb->stop_update = true;
 	spin_unlock_irq(&vb->stop_update_lock);
 	cancel_work_sync(&vb->update_balloon_size_work);
 	cancel_work_sync(&vb->update_balloon_stats_work);
 
-	if (virtio_has_feature(vdev, VIRTIO_BALLOON_F_FREE_PAGE_HINT)) {
+	if (virtio_has_feature(vdev, VIRTIO_BALLOON_F_FREE_PAGE_HINT))
 		cancel_work_sync(&vb->report_free_page_work);
+}
+
+static void virtballoon_remove(struct virtio_device *vdev)
+{
+	struct virtio_balloon *vb = vdev->priv;
+
+	virtballoon_quiesce(vb);
+
+	if (virtio_has_feature(vdev, VIRTIO_BALLOON_F_FREE_PAGE_HINT))
 		destroy_workqueue(vb->balloon_wq);
-	}
 
 	remove_common(vb);
 	mutex_destroy(&vb->balloon_lock);
Signature · 1 line
2.53.0
maintainer Reviewed-by on v2 2/4
#4 lore raw

Re: [PATCH v2 2/4] virtio_balloon: factor out virtballoon_quiesce()

Quoted 9 lines · Denis V. Lunev

virtballoon_remove() stops all of the balloon's asynchronous work (the free page reporting worker, the inflate/deflate and stats workers, the OOM notifier and the free page shrinker) before tearing the device down. A following change needs the same teardown from a .shutdown handler, so move it into a virtballoon_quiesce() helper.

No functional change.

Signature · 3 lines
Cheers,

David
#5 lore raw

PATCH v2 3/4 virtio_balloon: quiesce balloon work before device shutdown

In reply to Denis V. Lunev

Commit 8bd2fa086a04 ("virtio: break and reset virtio devices on device_shutdown()") added a generic virtio bus .shutdown handler that breaks and resets every virtio device during device_shutdown(), i.e. on reboot and kexec.

virtio_balloon provides no .shutdown of its own, so that generic path runs while the balloon's asynchronous work is still armed. Once the device has been broken, virtqueue_add_inbuf() in virtballoon_free_page_report() returns -EIO and trips its WARN_ON_ONCE(). On a kernel booted with panic_on_warn that turns an ordinary reboot, for example a kexec based upgrade, into a fatal panic in the middle of device_shutdown(), so the machine never reaches the new kernel.

Relaxing that single WARN_ON_ONCE() would only hide the symptom: the inflate/deflate and OOM paths do not warn, they call wait_event(vb->acked, ...) and would instead block forever on a broken queue that can no longer complete. The device has to be quiesced, not just kept quiet.

Add a .shutdown handler that quiesces the balloon via the shared virtballoon_quiesce() helper while the device is still alive, and only then breaks and resets it via virtio_device_shutdown(). Unlike virtballoon_remove() the balloon workqueue is not destroyed, as shutdown does not free the device and cancel_work_sync() together with stop_update already prevent any further work from being queued.

Fixes 8bd2fa086a04 ("virtio: break and reset virtio devices on device_shutdown()")
---
 drivers/virtio/virtio_balloon.c | 7 +++++++
 1 file changed, 7 insertions(+)
drivers/virtio/virtio_balloon.c +7 -0
index 5b02d9191ac6..26fc3c40d5b2 100644
@@ -1137,6 +1137,12 @@ static void virtballoon_remove(struct virtio_device *vdev)
 	kfree(vb);
 }
 
+static void virtballoon_shutdown(struct virtio_device *vdev)
+{
+	virtballoon_quiesce(vdev->priv);
+	virtio_device_shutdown(vdev);
+}
+
 #ifdef CONFIG_PM_SLEEP
 static int virtballoon_freeze(struct virtio_device *vdev)
 {
@@ -1202,6 +1208,7 @@ static struct virtio_driver virtio_balloon_driver = {
 	.validate =	virtballoon_validate,
 	.probe =	virtballoon_probe,
 	.remove =	virtballoon_remove,
+	.shutdown =	virtballoon_shutdown,
 	.config_changed = virtballoon_changed,
 #ifdef CONFIG_PM_SLEEP
 	.freeze	=	virtballoon_freeze,
Signature · 1 line
2.53.0
maintainer Reviewed-by on v2 3/4
#6 lore raw

Re: [PATCH v2 3/4] virtio_balloon: quiesce balloon work before device shutdown

Quoted 38 lines · Denis V. Lunev

Commit 8bd2fa086a04 ("virtio: break and reset virtio devices on device_shutdown()") added a generic virtio bus .shutdown handler that breaks and resets every virtio device during device_shutdown(), i.e. on reboot and kexec.

virtio_balloon provides no .shutdown of its own, so that generic path runs while the balloon's asynchronous work is still armed. Once the device has been broken, virtqueue_add_inbuf() in virtballoon_free_page_report() returns -EIO and trips its WARN_ON_ONCE(). On a kernel booted with panic_on_warn that turns an ordinary reboot, for example a kexec based upgrade, into a fatal panic in the middle of device_shutdown(), so the machine never reaches the new kernel.

Relaxing that single WARN_ON_ONCE() would only hide the symptom: the inflate/deflate and OOM paths do not warn, they call wait_event(vb->acked, ...) and would instead block forever on a broken queue that can no longer complete. The device has to be quiesced, not just kept quiet.

Add a .shutdown handler that quiesces the balloon via the shared virtballoon_quiesce() helper while the device is still alive, and only then breaks and resets it via virtio_device_shutdown(). Unlike virtballoon_remove() the balloon workqueue is not destroyed, as shutdown does not free the device and cancel_work_sync() together with stop_update already prevent any further work from being queued.

Fixes 8bd2fa086a04 ("virtio: break and reset virtio devices on device_shutdown()")
---
 drivers/virtio/virtio_balloon.c | 7 +++++++
 1 file changed, 7 insertions(+)
drivers/virtio/virtio_balloon.c +0 -0
index 5b02d9191ac6..26fc3c40d5b2 100644

drivers/virtio/virtio_balloon.c static void virtballoon_remove(struct virtio_device *vdev) · line 1144

Quoted 8 lines
drivers/virtio/virtio_balloon.c +5 -0
@@ -1137,6 +1137,12 @@ static void virtballoon_remove(struct virtio_device *vdev)
 	kfree(vb);
 }
 
+static void virtballoon_shutdown(struct virtio_device *vdev)
+{
+	virtballoon_quiesce(vdev->priv);
+	virtio_device_shutdown(vdev);
+}

I'm curious why virtio_gpu_shutdown() doesn't need that (did not look into the details).

Signature · 3 lines
Cheers,

David
#7 lore raw

Re: [PATCH v2 3/4] virtio_balloon: quiesce balloon work before device shutdown

Quoted 50 lines · David Hildenbrand (Arm)
Quoted 38 lines · Denis V. Lunev

Commit 8bd2fa086a04 ("virtio: break and reset virtio devices on device_shutdown()") added a generic virtio bus .shutdown handler that breaks and resets every virtio device during device_shutdown(), i.e. on reboot and kexec.

virtio_balloon provides no .shutdown of its own, so that generic path runs while the balloon's asynchronous work is still armed. Once the device has been broken, virtqueue_add_inbuf() in virtballoon_free_page_report() returns -EIO and trips its WARN_ON_ONCE(). On a kernel booted with panic_on_warn that turns an ordinary reboot, for example a kexec based upgrade, into a fatal panic in the middle of device_shutdown(), so the machine never reaches the new kernel.

Relaxing that single WARN_ON_ONCE() would only hide the symptom: the inflate/deflate and OOM paths do not warn, they call wait_event(vb->acked, ...) and would instead block forever on a broken queue that can no longer complete. The device has to be quiesced, not just kept quiet.

Add a .shutdown handler that quiesces the balloon via the shared virtballoon_quiesce() helper while the device is still alive, and only then breaks and resets it via virtio_device_shutdown(). Unlike virtballoon_remove() the balloon workqueue is not destroyed, as shutdown does not free the device and cancel_work_sync() together with stop_update already prevent any further work from being queued.

Fixes 8bd2fa086a04 ("virtio: break and reset virtio devices on device_shutdown()")
---
 drivers/virtio/virtio_balloon.c | 7 +++++++
 1 file changed, 7 insertions(+)
drivers/virtio/virtio_balloon.c +0 -0
index 5b02d9191ac6..26fc3c40d5b2 100644

drivers/virtio/virtio_balloon.c static void virtballoon_remove(struct virtio_device *vdev) · line 1144

Quoted 8 lines
drivers/virtio/virtio_balloon.c +5 -0
@@ -1137,6 +1137,12 @@ static void virtballoon_remove(struct virtio_device *vdev)
 	kfree(vb);
 }
 
+static void virtballoon_shutdown(struct virtio_device *vdev)
+{
+	virtballoon_quiesce(vdev->priv);
+	virtio_device_shutdown(vdev);
+}

I'm curious why virtio_gpu_shutdown() doesn't need that (did not look into the details).

Quoted 2 lines

I would spend more time with other drivers once we will done with this. I have strong candidate - virtio-mem.

Den

#8 lore raw

Re: [PATCH v2 3/4] virtio_balloon: quiesce balloon work before device shutdown

Quoted 51 lines · Denis V. Lunev
Quoted 50 lines · David Hildenbrand (Arm)
Quoted 38 lines · Denis V. Lunev

Commit 8bd2fa086a04 ("virtio: break and reset virtio devices on device_shutdown()") added a generic virtio bus .shutdown handler that breaks and resets every virtio device during device_shutdown(), i.e. on reboot and kexec.

virtio_balloon provides no .shutdown of its own, so that generic path runs while the balloon's asynchronous work is still armed. Once the device has been broken, virtqueue_add_inbuf() in virtballoon_free_page_report() returns -EIO and trips its WARN_ON_ONCE(). On a kernel booted with panic_on_warn that turns an ordinary reboot, for example a kexec based upgrade, into a fatal panic in the middle of device_shutdown(), so the machine never reaches the new kernel.

Relaxing that single WARN_ON_ONCE() would only hide the symptom: the inflate/deflate and OOM paths do not warn, they call wait_event(vb->acked, ...) and would instead block forever on a broken queue that can no longer complete. The device has to be quiesced, not just kept quiet.

Add a .shutdown handler that quiesces the balloon via the shared virtballoon_quiesce() helper while the device is still alive, and only then breaks and resets it via virtio_device_shutdown(). Unlike virtballoon_remove() the balloon workqueue is not destroyed, as shutdown does not free the device and cancel_work_sync() together with stop_update already prevent any further work from being queued.

Fixes 8bd2fa086a04 ("virtio: break and reset virtio devices on device_shutdown()")
---
 drivers/virtio/virtio_balloon.c | 7 +++++++
 1 file changed, 7 insertions(+)
drivers/virtio/virtio_balloon.c +0 -0
index 5b02d9191ac6..26fc3c40d5b2 100644

drivers/virtio/virtio_balloon.c static void virtballoon_remove(struct virtio_device *vdev) · line 1144

Quoted 8 lines
drivers/virtio/virtio_balloon.c +5 -0
@@ -1137,6 +1137,12 @@ static void virtballoon_remove(struct virtio_device *vdev)
 	kfree(vb);
 }
 
+static void virtballoon_shutdown(struct virtio_device *vdev)
+{
+	virtballoon_quiesce(vdev->priv);
+	virtio_device_shutdown(vdev);
+}

I'm curious why virtio_gpu_shutdown() doesn't need that (did not look into the details).

Quoted 4 lines
Quoted 2 lines

I would spend more time with other drivers once we will done with this. I have strong candidate - virtio-mem.

Heh, I briefly checked and it should handle it better I think.

If virtqueue_add_sgs() fails, it propagates the error (-EIO?) back to the main loop where we end up in

switch (rc) {
	...
	default:
	/* Unknown error, mark as broken */
	dev_err(&vm->vdev->dev, ...
	vm->broken = true;
}

And just stop.

But I didn't actually look into the details.

Signature · 3 lines
Cheers,

David
#9 lore raw

PATCH v2 4/4 virtio_balloon: warn on failed buffer add in tell_host()

In reply to Denis V. Lunev

tell_host() ignores the return value of virtqueue_add_outbuf() and goes on to kick the queue and wait_event() for the host's ack. The comment claims "We should always be able to add one buffer to an empty queue", but that does not hold once the virtqueue has been broken (e.g. on device shutdown): the add then fails with -EIO and the following wait_event() would block forever on a buffer the host can never return.

Warn and bail out on failure, mirroring virtballoon_free_page_report().

---
 drivers/virtio/virtio_balloon.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/virtio/virtio_balloon.c +4 -2
index 26fc3c40d5b2..0866a8781f0b 100644
@@ -184,16 +184,18 @@ static void tell_host(struct virtio_balloon *vb, struct virtqueue *vq)
 {
 	struct scatterlist sg;
 	unsigned int len;
+	int err;
 
 	sg_init_one(&sg, vb->pfns, sizeof(vb->pfns[0]) * vb->num_pfns);
 
 	/* We should always be able to add one buffer to an empty queue. */
-	virtqueue_add_outbuf(vq, &sg, 1, vb, GFP_KERNEL);
+	err = virtqueue_add_outbuf(vq, &sg, 1, vb, GFP_KERNEL);
+	if (WARN_ON_ONCE(err))
+		return;
 	virtqueue_kick(vq);
 
 	/* When host has read buffer, this completes via balloon_ack */
 	wait_event(vb->acked, virtqueue_get_buf(vq, &len));
-
 }
 
 static int virtballoon_free_page_report(struct page_reporting_dev_info *pr_dev_info,
Signature · 1 line
2.53.0
#10 lore raw

Re: [PATCH v2 4/4] virtio_balloon: warn on failed buffer add in tell_host()

Quoted 28 lines · Denis V. Lunev

tell_host() ignores the return value of virtqueue_add_outbuf() and goes on to kick the queue and wait_event() for the host's ack. The comment claims "We should always be able to add one buffer to an empty queue", but that does not hold once the virtqueue has been broken (e.g. on device shutdown): the add then fails with -EIO and the following wait_event() would block forever on a buffer the host can never return.

Warn and bail out on failure, mirroring virtballoon_free_page_report().

---
 drivers/virtio/virtio_balloon.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/virtio/virtio_balloon.c +1 -0
index 26fc3c40d5b2..0866a8781f0b 100644
@@ -184,16 +184,18 @@ static void tell_host(struct virtio_balloon *vb, struct virtqueue *vq)
 {
 	struct scatterlist sg;
 	unsigned int len;
+	int err;
 
 	sg_init_one(&sg, vb->pfns, sizeof(vb->pfns[0]) * vb->num_pfns);
 
 	/* We should always be able to add one buffer to an empty queue. */

drivers/virtio/virtio_balloon.c static void tell_host(struct virtio_balloon *vb, struct virtqueue *vq) · line 199

Quoted 10 lines
drivers/virtio/virtio_balloon.c +3 -2
-	virtqueue_add_outbuf(vq, &sg, 1, vb, GFP_KERNEL);
+	err = virtqueue_add_outbuf(vq, &sg, 1, vb, GFP_KERNEL);
+	if (WARN_ON_ONCE(err))
+		return;
 	virtqueue_kick(vq);
 
 	/* When host has read buffer, this completes via balloon_ack */
 	wait_event(vb->acked, virtqueue_get_buf(vq, &len));
-
 }

We have another uncheck instance in stats_handle_request(), what about that one?

Signature · 3 lines
Cheers,

David
#11 lore raw

[PATCH v2 5/4] virtio_balloon: warn on failed buffer add in stats_handle_request()

Like tell_host(), stats_handle_request() ignores the return value of virtqueue_add_outbuf() and kicks the queue regardless. The same "we should always be able to add one buffer to an empty queue" assumption does not hold once the virtqueue has been broken (e.g. on device shutdown), where the add fails with -EIO. Unlike tell_host() it does not wait_event() afterwards so it cannot hang, but it still kicks a queue with nothing queued.

Warn and bail out on failure, mirroring tell_host() and virtballoon_free_page_report().

---
 drivers/virtio/virtio_balloon.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/virtio/virtio_balloon.c +4 -1
index 0866a8781f0b..454bbb77331d 100644
@@ -445,6 +445,7 @@ static void stats_handle_request(struct virtio_balloon *vb)
 	struct virtqueue *vq;
 	struct scatterlist sg;
 	unsigned int len, num_stats;
+	int err;
 
 	num_stats = update_balloon_stats(vb);
 
@@ -452,7 +453,9 @@ static void stats_handle_request(struct virtio_balloon *vb)
 	if (!virtqueue_get_buf(vq, &len))
 		return;
 	sg_init_one(&sg, vb->stats, sizeof(vb->stats[0]) * num_stats);
-	virtqueue_add_outbuf(vq, &sg, 1, vb, GFP_KERNEL);
+	err = virtqueue_add_outbuf(vq, &sg, 1, vb, GFP_KERNEL);
+	if (WARN_ON_ONCE(err))
+		return;
 	virtqueue_kick(vq);
 }
 
Signature · 1 line
2.53.0
maintainer Reviewed-by
#12 lore raw

Re: [PATCH v2 5/4] virtio_balloon: warn on failed buffer add in stats_handle_request()

Quoted 30 lines · Denis V. Lunev

Like tell_host(), stats_handle_request() ignores the return value of virtqueue_add_outbuf() and kicks the queue regardless. The same "we should always be able to add one buffer to an empty queue" assumption does not hold once the virtqueue has been broken (e.g. on device shutdown), where the add fails with -EIO. Unlike tell_host() it does not wait_event() afterwards so it cannot hang, but it still kicks a queue with nothing queued.

Warn and bail out on failure, mirroring tell_host() and virtballoon_free_page_report().

---
 drivers/virtio/virtio_balloon.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/virtio/virtio_balloon.c +1 -0
index 0866a8781f0b..454bbb77331d 100644
@@ -445,6 +445,7 @@ static void stats_handle_request(struct virtio_balloon *vb)
 	struct virtqueue *vq;
 	struct scatterlist sg;
 	unsigned int len, num_stats;
+	int err;
 
 	num_stats = update_balloon_stats(vb);
 

drivers/virtio/virtio_balloon.c static void stats_handle_request(struct virtio_balloon *vb) · line 461

Quoted 10 lines
drivers/virtio/virtio_balloon.c +3 -1
@@ -452,7 +453,9 @@ static void stats_handle_request(struct virtio_balloon *vb)
 	if (!virtqueue_get_buf(vq, &len))
 		return;
 	sg_init_one(&sg, vb->stats, sizeof(vb->stats[0]) * num_stats);
-	virtqueue_add_outbuf(vq, &sg, 1, vb, GFP_KERNEL);
+	err = virtqueue_add_outbuf(vq, &sg, 1, vb, GFP_KERNEL);
+	if (WARN_ON_ONCE(err))
+		return;
 	virtqueue_kick(vq);
 }
 

Although I would just squash #4 and #5.

Signature · 3 lines
Cheers,

David
#13 lore raw

Re: [PATCH v2 5/4] virtio_balloon: warn on failed buffer add in stats_handle_request()

Quoted 43 lines · David Hildenbrand (Arm)
Quoted 30 lines · Denis V. Lunev

Like tell_host(), stats_handle_request() ignores the return value of virtqueue_add_outbuf() and kicks the queue regardless. The same "we should always be able to add one buffer to an empty queue" assumption does not hold once the virtqueue has been broken (e.g. on device shutdown), where the add fails with -EIO. Unlike tell_host() it does not wait_event() afterwards so it cannot hang, but it still kicks a queue with nothing queued.

Warn and bail out on failure, mirroring tell_host() and virtballoon_free_page_report().

---
 drivers/virtio/virtio_balloon.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/virtio/virtio_balloon.c +1 -0
index 0866a8781f0b..454bbb77331d 100644
@@ -445,6 +445,7 @@ static void stats_handle_request(struct virtio_balloon *vb)
 	struct virtqueue *vq;
 	struct scatterlist sg;
 	unsigned int len, num_stats;
+	int err;
 
 	num_stats = update_balloon_stats(vb);
 

drivers/virtio/virtio_balloon.c static void stats_handle_request(struct virtio_balloon *vb) · line 461

Quoted 10 lines
drivers/virtio/virtio_balloon.c +3 -1
@@ -452,7 +453,9 @@ static void stats_handle_request(struct virtio_balloon *vb)
 	if (!virtqueue_get_buf(vq, &len))
 		return;
 	sg_init_one(&sg, vb->stats, sizeof(vb->stats[0]) * num_stats);
-	virtqueue_add_outbuf(vq, &sg, 1, vb, GFP_KERNEL);
+	err = virtqueue_add_outbuf(vq, &sg, 1, vb, GFP_KERNEL);
+	if (WARN_ON_ONCE(err))
+		return;
 	virtqueue_kick(vq);
 }
 
Quoted 2 lines

Although I would just squash #4 and #5.

Sure thing. Does this way to avoid re-submit if that is possible.

Den

#14 lore raw
In reply to Denis V. Lunev
Quoted 48 lines · Denis V. Lunev

This email originated from an IP that might not be authorized by the domain it was sent from. Do not click links or open attachments unless it is an email you expected to receive. Since commit 8bd2fa086a04 ("virtio: break and reset virtio devices on device_shutdown()") the virtio bus breaks and resets every virtio device during device_shutdown(), i.e. on reboot and kexec. virtio_balloon has no .shutdown of its own, so that generic path runs while the balloon's asynchronous work is still armed: the free page reporting worker, the inflate/deflate and stats workers, the OOM notifier and the free page shrinker.

Once the device has been broken, virtqueue_add_inbuf() in virtballoon_free_page_report() returns -EIO and trips its WARN_ON_ONCE(). On a kernel booted with panic_on_warn that turns an ordinary reboot into a fatal panic in the middle of device_shutdown(), so the machine never reaches the new kernel. The inflate/deflate and OOM paths do not warn but are no better off: they call wait_event(vb->acked, ...) and would block forever on a queue that can no longer complete.

This was hit in the field as an intermittent failure of a virtualization cluster upgrade: guest storage nodes were rebooted via kexec into the new kernel, and the ones whose free page reporting happened to run during device_shutdown() panicked (the guests run with panic_on_warn) and never came back, stalling the rolling upgrade. The crash dump showed the WARN at virtio_balloon.c:216 in a page_reporting kworker, with all the balloon virtqueues already broken.

Validated by churning balloon inflate/deflate from the host while kexec-rebooting the guest in a loop under panic_on_warn: the unpatched kernel reproduces the WARN within a couple of cycles, while the patched kernel survives many consecutive kexec cycles cleanly (12/12 in the final run, 0 WARNs). checkpatch is clean across the series.

Changes in v2:
- Add a virtio_device_shutdown() core helper and call it from the balloon .shutdown handler instead of open-coding break + synchronize_cbs + reset (David Hildenbrand).
- New patch: make tell_host() warn and bail instead of hanging if a buffer add ever fails (David Hildenbrand); kept as a separate patch (Michael S. Tsirkin).

v1: https://lore.kernel.org/all/[email protected]

Denis V. Lunev (4):
virtio: add virtio_device_shutdown() helper
virtio_balloon: factor out virtballoon_quiesce()
virtio_balloon: quiesce balloon work before device shutdown
virtio_balloon: warn on failed buffer add in tell_host()

Quoted 5 lines
 drivers/virtio/virtio.c         | 41 ++++++++++++++++++++++-----------
 drivers/virtio/virtio_balloon.c | 40 ++++++++++++++++++++++++--------
 include/linux/virtio.h          |  1 +
 3 files changed, 59 insertions(+), 23 deletions(-)

Hi, David!

Is this good to go in? I have not seen the confirmation that the series is taken into your tree.

Thank you in advance,
    Den

#15 lore raw
Quoted 49 lines · Denis V. Lunev
Quoted 48 lines · Denis V. Lunev

This email originated from an IP that might not be authorized by the domain it was sent from. Do not click links or open attachments unless it is an email you expected to receive. Since commit 8bd2fa086a04 ("virtio: break and reset virtio devices on device_shutdown()") the virtio bus breaks and resets every virtio device during device_shutdown(), i.e. on reboot and kexec. virtio_balloon has no .shutdown of its own, so that generic path runs while the balloon's asynchronous work is still armed: the free page reporting worker, the inflate/deflate and stats workers, the OOM notifier and the free page shrinker.

Once the device has been broken, virtqueue_add_inbuf() in virtballoon_free_page_report() returns -EIO and trips its WARN_ON_ONCE(). On a kernel booted with panic_on_warn that turns an ordinary reboot into a fatal panic in the middle of device_shutdown(), so the machine never reaches the new kernel. The inflate/deflate and OOM paths do not warn but are no better off: they call wait_event(vb->acked, ...) and would block forever on a queue that can no longer complete.

This was hit in the field as an intermittent failure of a virtualization cluster upgrade: guest storage nodes were rebooted via kexec into the new kernel, and the ones whose free page reporting happened to run during device_shutdown() panicked (the guests run with panic_on_warn) and never came back, stalling the rolling upgrade. The crash dump showed the WARN at virtio_balloon.c:216 in a page_reporting kworker, with all the balloon virtqueues already broken.

Validated by churning balloon inflate/deflate from the host while kexec-rebooting the guest in a loop under panic_on_warn: the unpatched kernel reproduces the WARN within a couple of cycles, while the patched kernel survives many consecutive kexec cycles cleanly (12/12 in the final run, 0 WARNs). checkpatch is clean across the series.

Changes in v2:
- Add a virtio_device_shutdown() core helper and call it from the balloon .shutdown handler instead of open-coding break + synchronize_cbs + reset (David Hildenbrand).
- New patch: make tell_host() warn and bail instead of hanging if a buffer add ever fails (David Hildenbrand); kept as a separate patch (Michael S. Tsirkin).

v1: https://lore.kernel.org/all/[email protected]

Denis V. Lunev (4):
virtio: add virtio_device_shutdown() helper
virtio_balloon: factor out virtballoon_quiesce()
virtio_balloon: quiesce balloon work before device shutdown
virtio_balloon: warn on failed buffer add in tell_host()

Quoted 6 lines
Quoted 5 lines
 drivers/virtio/virtio.c         | 41 ++++++++++++++++++++++-----------
 drivers/virtio/virtio_balloon.c | 40 ++++++++++++++++++++++++--------
 include/linux/virtio.h          |  1 +
 3 files changed, 59 insertions(+), 23 deletions(-)

Hi, David!

Hi! :)

Quoted 3 lines

Is this good to go in? I have not seen the confirmation that the series is taken into your tree.

I don't have a tree (yet), and once I have one it will likely be more mm focused :)

@MST, I think this is good to go!

Signature · 3 lines
Cheers,

David
#16 lore raw
Quoted 60 lines · David Hildenbrand (Arm)
Quoted 49 lines · Denis V. Lunev
Quoted 48 lines · Denis V. Lunev

This email originated from an IP that might not be authorized by the domain it was sent from. Do not click links or open attachments unless it is an email you expected to receive. Since commit 8bd2fa086a04 ("virtio: break and reset virtio devices on device_shutdown()") the virtio bus breaks and resets every virtio device during device_shutdown(), i.e. on reboot and kexec. virtio_balloon has no .shutdown of its own, so that generic path runs while the balloon's asynchronous work is still armed: the free page reporting worker, the inflate/deflate and stats workers, the OOM notifier and the free page shrinker.

Once the device has been broken, virtqueue_add_inbuf() in virtballoon_free_page_report() returns -EIO and trips its WARN_ON_ONCE(). On a kernel booted with panic_on_warn that turns an ordinary reboot into a fatal panic in the middle of device_shutdown(), so the machine never reaches the new kernel. The inflate/deflate and OOM paths do not warn but are no better off: they call wait_event(vb->acked, ...) and would block forever on a queue that can no longer complete.

This was hit in the field as an intermittent failure of a virtualization cluster upgrade: guest storage nodes were rebooted via kexec into the new kernel, and the ones whose free page reporting happened to run during device_shutdown() panicked (the guests run with panic_on_warn) and never came back, stalling the rolling upgrade. The crash dump showed the WARN at virtio_balloon.c:216 in a page_reporting kworker, with all the balloon virtqueues already broken.

Validated by churning balloon inflate/deflate from the host while kexec-rebooting the guest in a loop under panic_on_warn: the unpatched kernel reproduces the WARN within a couple of cycles, while the patched kernel survives many consecutive kexec cycles cleanly (12/12 in the final run, 0 WARNs). checkpatch is clean across the series.

Changes in v2:
- Add a virtio_device_shutdown() core helper and call it from the balloon .shutdown handler instead of open-coding break + synchronize_cbs + reset (David Hildenbrand).
- New patch: make tell_host() warn and bail instead of hanging if a buffer add ever fails (David Hildenbrand); kept as a separate patch (Michael S. Tsirkin).

v1: https://lore.kernel.org/all/[email protected]

Denis V. Lunev (4):
virtio: add virtio_device_shutdown() helper
virtio_balloon: factor out virtballoon_quiesce()
virtio_balloon: quiesce balloon work before device shutdown
virtio_balloon: warn on failed buffer add in tell_host()

Quoted 6 lines
Quoted 5 lines
 drivers/virtio/virtio.c         | 41 ++++++++++++++++++++++-----------
 drivers/virtio/virtio_balloon.c | 40 ++++++++++++++++++++++++--------
 include/linux/virtio.h          |  1 +
 3 files changed, 59 insertions(+), 23 deletions(-)

Hi, David!

Hi! :)

Quoted 1 line
Quoted 11 lines
Quoted 2 lines

Is this good to go in? I have not seen the confirmation that the series is taken into your tree.

I don't have a tree (yet), and once I have one it will likely be more mm focused :)

@MST, I think this is good to go!

Signature · 3 lines
Cheers,

David

Indeed, it's just a bugfix and I'm trying to get features into qemu now before their freeze. I'll work on linux end of next week.

#17 lore raw
Quoted 63 lines · Michael S. Tsirkin
Quoted 62 lines · David Hildenbrand (Arm)
Quoted 49 lines · Denis V. Lunev
Quoted 48 lines · Denis V. Lunev

This email originated from an IP that might not be authorized by the domain it was sent from. Do not click links or open attachments unless it is an email you expected to receive. Since commit 8bd2fa086a04 ("virtio: break and reset virtio devices on device_shutdown()") the virtio bus breaks and resets every virtio device during device_shutdown(), i.e. on reboot and kexec. virtio_balloon has no .shutdown of its own, so that generic path runs while the balloon's asynchronous work is still armed: the free page reporting worker, the inflate/deflate and stats workers, the OOM notifier and the free page shrinker.

Once the device has been broken, virtqueue_add_inbuf() in virtballoon_free_page_report() returns -EIO and trips its WARN_ON_ONCE(). On a kernel booted with panic_on_warn that turns an ordinary reboot into a fatal panic in the middle of device_shutdown(), so the machine never reaches the new kernel. The inflate/deflate and OOM paths do not warn but are no better off: they call wait_event(vb->acked, ...) and would block forever on a queue that can no longer complete.

This was hit in the field as an intermittent failure of a virtualization cluster upgrade: guest storage nodes were rebooted via kexec into the new kernel, and the ones whose free page reporting happened to run during device_shutdown() panicked (the guests run with panic_on_warn) and never came back, stalling the rolling upgrade. The crash dump showed the WARN at virtio_balloon.c:216 in a page_reporting kworker, with all the balloon virtqueues already broken.

Validated by churning balloon inflate/deflate from the host while kexec-rebooting the guest in a loop under panic_on_warn: the unpatched kernel reproduces the WARN within a couple of cycles, while the patched kernel survives many consecutive kexec cycles cleanly (12/12 in the final run, 0 WARNs). checkpatch is clean across the series.

Changes in v2:
- Add a virtio_device_shutdown() core helper and call it from the balloon .shutdown handler instead of open-coding break + synchronize_cbs + reset (David Hildenbrand).
- New patch: make tell_host() warn and bail instead of hanging if a buffer add ever fails (David Hildenbrand); kept as a separate patch (Michael S. Tsirkin).

v1: https://lore.kernel.org/all/[email protected]

Denis V. Lunev (4):
virtio: add virtio_device_shutdown() helper
virtio_balloon: factor out virtballoon_quiesce()
virtio_balloon: quiesce balloon work before device shutdown
virtio_balloon: warn on failed buffer add in tell_host()

Quoted 6 lines
Quoted 5 lines
 drivers/virtio/virtio.c         | 41 ++++++++++++++++++++++-----------
 drivers/virtio/virtio_balloon.c | 40 ++++++++++++++++++++++++--------
 include/linux/virtio.h          |  1 +
 3 files changed, 59 insertions(+), 23 deletions(-)

Hi, David!

Hi! :)

Quoted 2 lines

Is this good to go in? I have not seen the confirmation that the series is taken into your tree.

I don't have a tree (yet), and once I have one it will likely be more mm focused :)

Quoted 9 lines
Quoted 6 lines

@MST, I think this is good to go!

Signature · 3 lines
Cheers,

David

Indeed, it's just a bugfix and I'm trying to get features into qemu now before their freeze. I'll work on linux end of next week.

Hi, Michael!

Have you had a chance to take series into your tree, i.e. should I continue to track this submission?

Thank you in advance,
    Den

#18 lore raw
Quoted 73 lines · Denis V. Lunev
Quoted 63 lines · Michael S. Tsirkin
Quoted 62 lines · David Hildenbrand (Arm)
Quoted 49 lines · Denis V. Lunev
Quoted 48 lines · Denis V. Lunev

This email originated from an IP that might not be authorized by the domain it was sent from. Do not click links or open attachments unless it is an email you expected to receive. Since commit 8bd2fa086a04 ("virtio: break and reset virtio devices on device_shutdown()") the virtio bus breaks and resets every virtio device during device_shutdown(), i.e. on reboot and kexec. virtio_balloon has no .shutdown of its own, so that generic path runs while the balloon's asynchronous work is still armed: the free page reporting worker, the inflate/deflate and stats workers, the OOM notifier and the free page shrinker.

Once the device has been broken, virtqueue_add_inbuf() in virtballoon_free_page_report() returns -EIO and trips its WARN_ON_ONCE(). On a kernel booted with panic_on_warn that turns an ordinary reboot into a fatal panic in the middle of device_shutdown(), so the machine never reaches the new kernel. The inflate/deflate and OOM paths do not warn but are no better off: they call wait_event(vb->acked, ...) and would block forever on a queue that can no longer complete.

This was hit in the field as an intermittent failure of a virtualization cluster upgrade: guest storage nodes were rebooted via kexec into the new kernel, and the ones whose free page reporting happened to run during device_shutdown() panicked (the guests run with panic_on_warn) and never came back, stalling the rolling upgrade. The crash dump showed the WARN at virtio_balloon.c:216 in a page_reporting kworker, with all the balloon virtqueues already broken.

Validated by churning balloon inflate/deflate from the host while kexec-rebooting the guest in a loop under panic_on_warn: the unpatched kernel reproduces the WARN within a couple of cycles, while the patched kernel survives many consecutive kexec cycles cleanly (12/12 in the final run, 0 WARNs). checkpatch is clean across the series.

Changes in v2:
- Add a virtio_device_shutdown() core helper and call it from the balloon .shutdown handler instead of open-coding break + synchronize_cbs + reset (David Hildenbrand).
- New patch: make tell_host() warn and bail instead of hanging if a buffer add ever fails (David Hildenbrand); kept as a separate patch (Michael S. Tsirkin).

v1: https://lore.kernel.org/all/[email protected]

Denis V. Lunev (4):
virtio: add virtio_device_shutdown() helper
virtio_balloon: factor out virtballoon_quiesce()
virtio_balloon: quiesce balloon work before device shutdown
virtio_balloon: warn on failed buffer add in tell_host()

Quoted 6 lines
Quoted 5 lines
 drivers/virtio/virtio.c         | 41 ++++++++++++++++++++++-----------
 drivers/virtio/virtio_balloon.c | 40 ++++++++++++++++++++++++--------
 include/linux/virtio.h          |  1 +
 3 files changed, 59 insertions(+), 23 deletions(-)

Hi, David!

Hi! :)

Quoted 2 lines

Is this good to go in? I have not seen the confirmation that the series is taken into your tree.

I don't have a tree (yet), and once I have one it will likely be more mm focused :)

Quoted 9 lines
Quoted 6 lines

@MST, I think this is good to go!

Signature · 3 lines
Cheers,

David

Indeed, it's just a bugfix and I'm trying to get features into qemu now before their freeze. I'll work on linux end of next week.

Quoted 7 lines

Hi, Michael!

Have you had a chance to take series into your tree, i.e. should I continue to track this submission?

Thank you in advance,
    Den

Hi!
It's on my queue and will be in the next Linux release, don't worry.

Signature · 1 line
MST