Skip to content

virtio_balloon: quiesce balloon work on device shutdown

Denis V. Lunev lkml virtualization 2 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v2 Latest4 patches 14 repliesIn review
  2. v12 patches 8 repliesSuperseded

Changes in v2 from the cover letter

- Add a virtio_device_shutdown() core helper and call it from the balloon .shutdown handler instead of open-coding break + synchronize_cbs + reset (David Hildenbrand).
- New patch: make tell_host() warn and bail instead of hanging if a buffer add ever fails (David Hildenbrand); kept as a separate patch (Michael S. Tsirkin).

Patches in v2 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/4virtio: add virtio_device_shutdown() helperdrivers/virtio/virtio.c include/linux/virtio.h+28 -141 reply
2/4virtio_balloon: factor out virtballoon_quiesce()drivers/virtio/virtio_balloon.c+20 -71 reply
3/4virtio_balloon: quiesce balloon work before device shutdowndrivers/virtio/virtio_balloon.c+7 -03 replies
4/4virtio_balloon: warn on failed buffer add in tell_host()drivers/virtio/virtio_balloon.c+4 -24 replies

Reviews trailers per patch

All 4 patches have a Reviewed-by or Acked-by.

PatchDavid Hildenbrand (Arm)
1/4R-b
2/4R-b
3/4R-b
4/4R-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v2 thread

Cover letter [PATCH v2 0/4] virtio_balloon: quiesce balloon work on device shutdown

Since commit 8bd2fa086a04 ("virtio: break and reset virtio devices on device_shutdown()") the virtio bus breaks and resets every virtio device during device_shutdown(), i.e. on reboot and kexec. virtio_balloon has no .shutdown of its own, so that generic path runs while the balloon's asynchronous work is still armed: the free page reporting worker, the inflate/deflate and stats workers, the OOM notifier and the free page shrinker.

Once the device has been broken, virtqueue_add_inbuf() in virtballoon_free_page_report() returns -EIO and trips its WARN_ON_ONCE(). On a kernel booted with panic_on_warn that turns an ordinary reboot into a fatal panic in the middle of device_shutdown(), so the machine never reaches the new kernel. The inflate/deflate and OOM paths do not warn but are no better off: they call wait_event(vb->acked, ...) and would block forever on a queue that can no longer complete.

This was hit in the field as an intermittent failure of a virtualization cluster upgrade: guest storage nodes were rebooted via kexec into the new kernel, and the ones whose free page reporting happened to run during device_shutdown() panicked (the guests run with panic_on_warn) and never came back, stalling the rolling upgrade. The crash dump showed the WARN at virtio_balloon.c:216 in a page_reporting kworker, with all the balloon virtqueues already broken.

Validated by churning balloon inflate/deflate from the host while kexec-rebooting the guest in a loop under panic_on_warn: the unpatched kernel reproduces the WARN within a couple of cycles, while the patched kernel survives many consecutive kexec cycles cleanly (12/12 in the final run, 0 WARNs). checkpatch is clean across the series.

Changes in v2:
- Add a virtio_device_shutdown() core helper and call it from the balloon .shutdown handler instead of open-coding break + synchronize_cbs + reset (David Hildenbrand).
- New patch: make tell_host() warn and bail instead of hanging if a buffer add ever fails (David Hildenbrand); kept as a separate patch (Michael S. Tsirkin).

v1: https://lore.kernel.org/all/[email protected]

Denis V. Lunev (4):
virtio: add virtio_device_shutdown() helper
virtio_balloon: factor out virtballoon_quiesce()
virtio_balloon: quiesce balloon work before device shutdown
virtio_balloon: warn on failed buffer add in tell_host()

 drivers/virtio/virtio.c         | 41 ++++++++++++++++++++++-----------
 drivers/virtio/virtio_balloon.c | 40 ++++++++++++++++++++++++--------
 include/linux/virtio.h          |  1 +
 3 files changed, 59 insertions(+), 23 deletions(-)
Signature · 1 line
2.53.0