Skip to contentSkip to conversation

virtio_blk: Fix disk deletion hang on device surprise removal

PATCH Superseded Parav Pandit 2 participants 4 messages May 26, 2025 – May 27, 2025

Subsystems (4)

v2 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 virtio_blk: Fix disk deletion hang on device surprise removal

When the PCI device is surprise-removed, requests may not complete on the device as the VQ is marked as broken. As a result, disk deletion hangs.

Fix it by aborting the requests when the VQ is broken.

With this fix now fio completes swiftly.
An alternative of IO timeout has been considered, however when the driver knows about unresponsive block device, swiftly clearing them enables users and upper layers to react quickly.

Verified with multiple device unplug iterations with pending requests in virtio used ring and some pending with the device.

---
v1->v2:
- Addressed comments from Stephan
- fixed spelling to 'waiting'
- Addressed comments from Michael
- Dropped checking broken vq from queue_rq() and queue_rqs() because it is checked in lower layer routines in virtio core

v0->v1:
- Fixed comments from Stefan to rename a cleanup function
- Improved logic for handling any outstanding requests in bio layer
- improved cancel callback to sync with ongoing done()

---
 drivers/block/virtio_blk.c | 83 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 83 insertions(+)
drivers/block/virtio_blk.c +83 -0
index 7cffea01d868..12f31e6c00cb 100644
@@ -1554,6 +1554,87 @@ static int virtblk_probe(struct virtio_device *vdev)
 	return err;
 }
 
+static bool virtblk_request_cancel(struct request *rq, void *data)
+{
+	struct virtblk_req *vbr = blk_mq_rq_to_pdu(rq);
+	struct virtio_blk *vblk = data;
+	struct virtio_blk_vq *vq;
+	unsigned long flags;
+
+	vq = &vblk->vqs[rq->mq_hctx->queue_num];
+
+	spin_lock_irqsave(&vq->lock, flags);
+
+	vbr->in_hdr.status = VIRTIO_BLK_S_IOERR;
+	if (blk_mq_request_started(rq) && !blk_mq_request_completed(rq))
+		blk_mq_complete_request(rq);
+
+	spin_unlock_irqrestore(&vq->lock, flags);
+	return true;
+}
+
+static void virtblk_broken_device_cleanup(struct virtio_blk *vblk)
+{
+	struct request_queue *q = vblk->disk->queue;
+
+	if (!virtqueue_is_broken(vblk->vqs[0].vq))
+		return;
+
+	/* Start freezing the queue, so that new requests keeps waiting at the
+	 * door of bio_queue_enter(). We cannot fully freeze the queue because
+	 * freezed queue is an empty queue and there are pending requests, so
+	 * only start freezing it.
+	 */
+	blk_freeze_queue_start(q);
+
+	/* When quiescing completes, all ongoing dispatches have completed
+	 * and no new dispatch will happen towards the driver.
+	 * This ensures that later when cancel is attempted, then are not
+	 * getting processed by the queue_rq() or queue_rqs() handlers.
+	 */
+	blk_mq_quiesce_queue(q);
+
+	/*
+	 * Synchronize with any ongoing VQ callbacks, effectively quiescing
+	 * the device and preventing it from completing further requests
+	 * to the block layer. Any outstanding, incomplete requests will be
+	 * completed by virtblk_request_cancel().
+	 */
+	virtio_synchronize_cbs(vblk->vdev);
+
+	/* At this point, no new requests can enter the queue_rq() and
+	 * completion routine will not complete any new requests either for the
+	 * broken vq. Hence, it is safe to cancel all requests which are
+	 * started.
+	 */
+	blk_mq_tagset_busy_iter(&vblk->tag_set, virtblk_request_cancel, vblk);
+	blk_mq_tagset_wait_completed_request(&vblk->tag_set);
+
+	/* All pending requests are cleaned up. Time to resume so that disk
+	 * deletion can be smooth. Start the HW queues so that when queue is
+	 * unquiesced requests can again enter the driver.
+	 */
+	blk_mq_start_stopped_hw_queues(q, true);
+
+	/* Unquiescing will trigger dispatching any pending requests to the
+	 * driver which has crossed bio_queue_enter() to the driver.
+	 */
+	blk_mq_unquiesce_queue(q);
+
+	/* Wait for all pending dispatches to terminate which may have been
+	 * initiated after unquiescing.
+	 */
+	blk_mq_freeze_queue_wait(q);
+
+	/* Mark the disk dead so that once queue unfreeze, the requests
+	 * waiting at the door of bio_queue_enter() can be aborted right away.
+	 */
+	blk_mark_disk_dead(vblk->disk);
+
+	/* Unfreeze the queue so that any waiting requests will be aborted. */
+	blk_mq_unfreeze_queue_nomemrestore(q);
+}
+
 static void virtblk_remove(struct virtio_device *vdev)
 {
 	struct virtio_blk *vblk = vdev->priv;
@@ -1561,6 +1642,8 @@ static void virtblk_remove(struct virtio_device *vdev)
 	/* Make sure no work handler is accessing the device. */
 	flush_work(&vblk->config_work);
 
+	virtblk_broken_device_cleanup(vblk);
+
 	del_gendisk(vblk->disk);
 	blk_mq_free_tag_set(&vblk->tag_set);
 
Signature · 1 line
2.34.1
#1 lore raw
Quoted 14 lines · Parav Pandit

When the PCI device is surprise-removed, requests may not complete on the device as the VQ is marked as broken. As a result, disk deletion hangs.

Fix it by aborting the requests when the VQ is broken.

With this fix now fio completes swiftly.
An alternative of IO timeout has been considered, however when the driver knows about unresponsive block device, swiftly clearing them enables users and upper layers to react quickly.

Verified with multiple device unplug iterations with pending requests in virtio used ring and some pending with the device.

Quoted 7 lines

Thanks!
I like the patch. Yes something to improve:

Quoted 29 lines

---
v1->v2:
- Addressed comments from Stephan
- fixed spelling to 'waiting'
- Addressed comments from Michael
- Dropped checking broken vq from queue_rq() and queue_rqs() because it is checked in lower layer routines in virtio core

v0->v1:
- Fixed comments from Stefan to rename a cleanup function
- Improved logic for handling any outstanding requests in bio layer
- improved cancel callback to sync with ongoing done()

---
 drivers/block/virtio_blk.c | 83 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 83 insertions(+)
drivers/block/virtio_blk.c +2 -0
index 7cffea01d868..12f31e6c00cb 100644
@@ -1554,6 +1554,87 @@ static int virtblk_probe(struct virtio_device *vdev)
 	return err;
 }
 
+static bool virtblk_request_cancel(struct request *rq, void *data)
 
 
+{

drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device *vdev) · line 1570

Quoted 10 lines
drivers/block/virtio_blk.c +10 -0
+	struct virtblk_req *vbr = blk_mq_rq_to_pdu(rq);
+	struct virtio_blk *vblk = data;
+	struct virtio_blk_vq *vq;
+	unsigned long flags;
+
+	vq = &vblk->vqs[rq->mq_hctx->queue_num];
+
+	spin_lock_irqsave(&vq->lock, flags);
+
+	vbr->in_hdr.status = VIRTIO_BLK_S_IOERR;

My undertanding is that this is only safe to call when device is not accessing the vq anymore? Right? otherwise device can overwrite the status?

But I am not sure I understand what guarantees this. Is there an assumption here that if vq is broken and we are on remove path then device is already gone? It seems to hold but
I'd prefer something that makes this guarantee at the API level.

drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device *vdev) · line 1581

Quoted 13 lines
drivers/block/virtio_blk.c +13 -0
+	if (blk_mq_request_started(rq) && !blk_mq_request_completed(rq))
+		blk_mq_complete_request(rq);
+
+	spin_unlock_irqrestore(&vq->lock, flags);
+	return true;
+}
+
+static void virtblk_broken_device_cleanup(struct virtio_blk *vblk)
+{
+	struct request_queue *q = vblk->disk->queue;
+
+	if (!virtqueue_is_broken(vblk->vqs[0].vq))
+		return;

can marking vqs broken be in progress such that 0
is already broken but another one is not, yet?
if not pls add a comment explainging why.

Quoted 10 lines
drivers/block/virtio_blk.c +10 -0
+
+	/* Start freezing the queue, so that new requests keeps waiting at the
+	 * door of bio_queue_enter(). We cannot fully freeze the queue because
+	 * freezed queue is an empty queue and there are pending requests, so
+	 * only start freezing it.
+	 */
+	blk_freeze_queue_start(q);
+
+	/* When quiescing completes, all ongoing dispatches have completed
+	 * and no new dispatch will happen towards the driver.

drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device *vdev) · line 1601

Quoted 10 lines
drivers/block/virtio_blk.c +10 -0
+	 * This ensures that later when cancel is attempted, then are not
+	 * getting processed by the queue_rq() or queue_rqs() handlers.
+	 */
+	blk_mq_quiesce_queue(q);
+
+	/*
+	 * Synchronize with any ongoing VQ callbacks, effectively quiescing
+	 * the device and preventing it from completing further requests
+	 * to the block layer. Any outstanding, incomplete requests will be
+	 * completed by virtblk_request_cancel().

I think what you really mean is:
finish processing in callbacks, that might have started before vqs were marked as broken.

Quoted 54 lines
drivers/block/virtio_blk.c +38 -0
+	 */
+	virtio_synchronize_cbs(vblk->vdev);
 
 
 
 
+
+	/* At this point, no new requests can enter the queue_rq() and
+	 * completion routine will not complete any new requests either for the
+	 * broken vq. Hence, it is safe to cancel all requests which are
+	 * started.
+	 */
+	blk_mq_tagset_busy_iter(&vblk->tag_set, virtblk_request_cancel, vblk);
+	blk_mq_tagset_wait_completed_request(&vblk->tag_set);
+
+	/* All pending requests are cleaned up. Time to resume so that disk
+	 * deletion can be smooth. Start the HW queues so that when queue is
+	 * unquiesced requests can again enter the driver.
+	 */
+	blk_mq_start_stopped_hw_queues(q, true);
+
+	/* Unquiescing will trigger dispatching any pending requests to the
+	 * driver which has crossed bio_queue_enter() to the driver.
+	 */
+	blk_mq_unquiesce_queue(q);
+
+	/* Wait for all pending dispatches to terminate which may have been
+	 * initiated after unquiescing.
+	 */
+	blk_mq_freeze_queue_wait(q);
+
+	/* Mark the disk dead so that once queue unfreeze, the requests
+	 * waiting at the door of bio_queue_enter() can be aborted right away.
+	 */
+	blk_mark_disk_dead(vblk->disk);
+
+	/* Unfreeze the queue so that any waiting requests will be aborted. */
+	blk_mq_unfreeze_queue_nomemrestore(q);
+}
+
 static void virtblk_remove(struct virtio_device *vdev)
 {
 	struct virtio_blk *vblk = vdev->priv;
@@ -1561,6 +1642,8 @@ static void virtblk_remove(struct virtio_device *vdev)
 	/* Make sure no work handler is accessing the device. */
 	flush_work(&vblk->config_work);
 
+	virtblk_broken_device_cleanup(vblk);
+
 	del_gendisk(vblk->disk);
 	blk_mq_free_tag_set(&vblk->tag_set);
 
Signature · 1 line
2.34.1
#2 lore raw
Quoted 64 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, May 27, 2025 1:45 AM

Quoted 16 lines · Parav Pandit

When the PCI device is surprise-removed, requests may not complete on the device as the VQ is marked as broken. As a result, disk deletion hangs.

Fix it by aborting the requests when the VQ is broken.

With this fix now fio completes swiftly.
An alternative of IO timeout has been considered, however when the driver knows about unresponsive block device, swiftly clearing them enables users and upper layers to react quickly.

Verified with multiple device unplug iterations with pending requests in virtio used ring and some pending with the device.

Fixes 43bb40c5b926 ("virtio_pci: Support surprise removal of virtio

pci device")

Quoted 8 lines

Closes:
https://lore.kernel.org/virtualization/c45dd68698cd47238c55fb73ca9b474
[email protected]/

Thanks!
I like the patch. Yes something to improve:

drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device

Quoted 23 lines

---
v1->v2:
- Addressed comments from Stephan
- fixed spelling to 'waiting'
- Addressed comments from Michael
- Dropped checking broken vq from queue_rq() and queue_rqs() because it is checked in lower layer routines in virtio core

v0->v1:
- Fixed comments from Stefan to rename a cleanup function
- Improved logic for handling any outstanding requests in bio layer
- improved cancel callback to sync with ongoing done()

---
 drivers/block/virtio_blk.c | 83
++++++++++++++++++++++++++++++++++++++
 1 file changed, 83 insertions(+)
drivers/block/virtio_blk.c +0 -0
index 7cffea01d868..12f31e6c00cb 100644
@@ -1554,6 +1554,87 @@ static int virtblk_probe(struct virtio_device

*vdev)

Quoted 4 lines
drivers/block/virtio_blk.c +1 -0
 	return err;
 }
 
+static bool virtblk_request_cancel(struct request *rq, void *data)
Quoted 15 lines

drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device *vdev) · line 1568

Quoted 11 lines
drivers/block/virtio_blk.c +11 -0
+{
+	struct virtblk_req *vbr = blk_mq_rq_to_pdu(rq);
+	struct virtio_blk *vblk = data;
+	struct virtio_blk_vq *vq;
+	unsigned long flags;
+
+	vq = &vblk->vqs[rq->mq_hctx->queue_num];
+
+	spin_lock_irqsave(&vq->lock, flags);
+
+	vbr->in_hdr.status = VIRTIO_BLK_S_IOERR;

My undertanding is that this is only safe to call when device is not accessing the vq anymore? Right? otherwise device can overwrite the status?

Even if the device is accessing the VQ (such as descriptors) and its memory, and if we free and unmap the request, it is a problem. So the contract is, when the device has reported that the transport is broken, after that it must not touch the VQ.

The in_hdr.status is updated either here or in the done() handler. And both are protected by spin lock, so they don't step on each other.

Quoted 3 lines

But I am not sure I understand what guarantees this.
Is there an assumption here that if vq is broken and we are on remove path then device is already gone?

True, the assumption is that device must not touch VQ or memory pointed by VQ when it has reported that device is broken.

Quoted 3 lines

It seems to hold but
I'd prefer something that makes this guarantee at the API level.

Not sure how we can guarantee that from the device. It's the assumption from the driver in following the pci spec.

Quoted 39 lines
Quoted 4 lines
drivers/block/virtio_blk.c +2 -0
+	if (blk_mq_request_started(rq) && !blk_mq_request_completed(rq))
+		blk_mq_complete_request(rq);

drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device *vdev) · line 1581

Quoted 10 lines
drivers/block/virtio_blk.c +10 -0
+
+	spin_unlock_irqrestore(&vq->lock, flags);
+	return true;
+}
+
+static void virtblk_broken_device_cleanup(struct virtio_blk *vblk) {
+	struct request_queue *q = vblk->disk->queue;
+
+	if (!virtqueue_is_broken(vblk->vqs[0].vq))
+		return;

can marking vqs broken be in progress such that 0 is already broken but another one is not, yet?
if not pls add a comment explainging why.

drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device · line 1582

Quoted 2 lines
drivers/block/virtio_blk.c +2 -0
+
+	/* Start freezing the queue, so that new requests keeps waiting at

the

drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device · line 1583

Quoted 1 line
drivers/block/virtio_blk.c +1 -0
+	 * door of bio_queue_enter(). We cannot fully freeze the queue

because

drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device · line 1584

Quoted 1 line
drivers/block/virtio_blk.c +1 -0
+	 * freezed queue is an empty queue and there are pending requests,

so

Quoted 13 lines
drivers/block/virtio_blk.c +13 -0
+	 * only start freezing it.
+	 */
+	blk_freeze_queue_start(q);
+
+	/* When quiescing completes, all ongoing dispatches have completed
+	 * and no new dispatch will happen towards the driver.
+	 * This ensures that later when cancel is attempted, then are not
+	 * getting processed by the queue_rq() or queue_rqs() handlers.
+	 */
+	blk_mq_quiesce_queue(q);
+
+	/*
+	 * Synchronize with any ongoing VQ callbacks, effectively quiescing
Quoted 9 lines

drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device *vdev) · line 1601

Quoted 3 lines
drivers/block/virtio_blk.c +3 -0
+	 * the device and preventing it from completing further requests
+	 * to the block layer. Any outstanding, incomplete requests will be
+	 * completed by virtblk_request_cancel().

I think what you really mean is:
finish processing in callbacks, that might have started before vqs were marked as broken.

Yes, I will remove the word "effectively quiescing the device", because that is not done by pure sw code here. It is quiescing the async interrupt handler side.
How about a below rephrase:

/*
* Synchronize with any ongoing VQ callbacks which may be started before VQs are marked broken,
* preventing it from completing further requests to the block layer. Any outstanding,
* incomplete requests will be completed by virtblk_request_cancel().
*/

Quoted 60 lines

drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device

Quoted 11 lines
drivers/block/virtio_blk.c +5 -0
+	 */
+	virtio_synchronize_cbs(vblk->vdev);
 
 
 
 
+
+	/* At this point, no new requests can enter the queue_rq() and
+	 * completion routine will not complete any new requests either for

the

drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device *vdev)

Quoted 4 lines
drivers/block/virtio_blk.c +4 -0
+	 * broken vq. Hence, it is safe to cancel all requests which are
+	 * started.
+	 */
+	blk_mq_tagset_busy_iter(&vblk->tag_set, virtblk_request_cancel,

vblk);

drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device *vdev)

Quoted 4 lines
drivers/block/virtio_blk.c +4 -0
+	blk_mq_tagset_wait_completed_request(&vblk->tag_set);
+
+	/* All pending requests are cleaned up. Time to resume so that disk
+	 * deletion can be smooth. Start the HW queues so that when queue

is

Quoted 6 lines
drivers/block/virtio_blk.c +6 -0
+	 * unquiesced requests can again enter the driver.
+	 */
+	blk_mq_start_stopped_hw_queues(q, true);
+
+	/* Unquiescing will trigger dispatching any pending requests to the
+	 * driver which has crossed bio_queue_enter() to the driver.

drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device *vdev)

Quoted 10 lines
drivers/block/virtio_blk.c +10 -0
+	 */
+	blk_mq_unquiesce_queue(q);
+
+	/* Wait for all pending dispatches to terminate which may have been
+	 * initiated after unquiescing.
+	 */
+	blk_mq_freeze_queue_wait(q);
+
+	/* Mark the disk dead so that once queue unfreeze, the requests
+	 * waiting at the door of bio_queue_enter() can be aborted right

away.

drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device *vdev)

Quoted 4 lines
drivers/block/virtio_blk.c +4 -0
+	 */
+	blk_mark_disk_dead(vblk->disk);
+
+	/* Unfreeze the queue so that any waiting requests will be aborted.

*/

Quoted 16 lines
drivers/block/virtio_blk.c +3 -0
+	blk_mq_unfreeze_queue_nomemrestore(q);
+}
+
 static void virtblk_remove(struct virtio_device *vdev)  {
 	struct virtio_blk *vblk = vdev->priv; @@ -1561,6 +1642,8 @@ static

void virtblk_remove(struct virtio_device *vdev)

drivers/block/virtio_blk.c +2 -1
 	/* Make sure no work handler is accessing the device. */
 	flush_work(&vblk->config_work);
 
+	virtblk_broken_device_cleanup(vblk);
+
 	del_gendisk(vblk->disk);
 	blk_mq_free_tag_set(&vblk->tag_set);
 
--

2.34.1

#3 lore raw
In reply to Michael S. Tsirkin
Quoted 4 lines

From: Michael S. Tsirkin <[email protected]>
Sent: Tuesday, May 27, 2025 1:45 AM

On Mon, May 26, 2025 at 02:40:33PM +0000, Parav Pandit wrote:

[..]

Quoted 8 lines
Quoted 8 lines
drivers/block/virtio_blk.c +8 -0
+	if (blk_mq_request_started(rq) && !blk_mq_request_completed(rq))
+		blk_mq_complete_request(rq);
+
+	spin_unlock_irqrestore(&vq->lock, flags);
+	return true;
+}
+
+static void virtblk_broken_device_cleanup(struct virtio_blk *vblk) {
Quoted 9 lines

drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device *vdev) · line 1581

Quoted 4 lines
drivers/block/virtio_blk.c +4 -0
+	struct request_queue *q = vblk->disk->queue;
+
+	if (!virtqueue_is_broken(vblk->vqs[0].vq))
+		return;

can marking vqs broken be in progress such that 0 is already broken but another one is not, yet?
if not pls add a comment explainging why.

I missed to answer this previous email.

It should not happen during surprise removal case, because surprise removal and remove() is serialized operation listed below,

virtio_pci_remove()
virtio_break_device()
mark N Vqs as broken..
unregister_virtio_device()
virtioblk_remove()

If one attempts to unbind the pci device and once the remove() crosses the broken check, and after that if the pci device is removed (before existing virtblk_remove finishes, then yes, same hang can occur.
Such thing requires an additional time out handling, which should be handled separate patch as new functionality. Since timeout handling would be additional, it won't replace the existing known surprise removal case anyway. I anticipate that it will likely reuse pieces from the proposed cleanup() function.