Skip to contentSkip to conversation

virtio_blk: Fix disk deletion hang on device surprise removal

PATCH In review Parav Pandit 5 participants 24 messages Jun 2, 2025 – Jun 27, 2025

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

PATCH v5 1/1 virtio_blk: Fix disk deletion hang on device surprise removal

When the PCI device is surprise removed, requests may not complete the device as the VQ is marked as broken. Due to this, the disk deletion hangs.

Fix it by aborting the requests when the VQ is broken.

With this fix now fio completes swiftly.
An alternative of IO timeout has been considered, however when the driver knows about unresponsive block device, swiftly clearing them enables users and upper layers to react quickly.

Verified with multiple device unplug iterations with pending requests in virtio used ring and some pending with the device.

---
v4->v5:
- fixed comment style where comment to start with one empty line at start
- Addressed comments from Alok
- fixed typo in broken vq check
v3->v4:
- Addressed comments from Michael
- renamed virtblk_request_cancel() to virtblk_complete_request_with_ioerr()
- Added comments for virtblk_complete_request_with_ioerr()
- Renamed virtblk_broken_device_cleanup() to virtblk_cleanup_broken_device()
- Added comments for virtblk_cleanup_broken_device()
- Moved the broken vq check in virtblk_remove()
- Fixed comment style to have first empty line
- replaced freezed to frozen
- Fixed comments rephrased

v2->v3:
- Addressed comments from Michael
- updated comment for synchronizing with callbacks

v1->v2:
- Addressed comments from Stephan
- fixed spelling to 'waiting'
- Addressed comments from Michael
- Dropped checking broken vq from queue_rq() and queue_rqs() because it is checked in lower layer routines in virtio core

v0->v1:
- Fixed comments from Stefan to rename a cleanup function
- Improved logic for handling any outstanding requests in bio layer
- improved cancel callback to sync with ongoing done()

---
 drivers/block/virtio_blk.c | 95 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 95 insertions(+)
drivers/block/virtio_blk.c +95 -0
index 7cffea01d868..c5e383c0ac48 100644
@@ -1554,6 +1554,98 @@ static int virtblk_probe(struct virtio_device *vdev)
 	return err;
 }
 
+/*
+ * If the vq is broken, device will not complete requests.
+ * So we do it for the device.
+ */
+static bool virtblk_complete_request_with_ioerr(struct request *rq, void *data)
+{
+	struct virtblk_req *vbr = blk_mq_rq_to_pdu(rq);
+	struct virtio_blk *vblk = data;
+	struct virtio_blk_vq *vq;
+	unsigned long flags;
+
+	vq = &vblk->vqs[rq->mq_hctx->queue_num];
+
+	spin_lock_irqsave(&vq->lock, flags);
+
+	vbr->in_hdr.status = VIRTIO_BLK_S_IOERR;
+	if (blk_mq_request_started(rq) && !blk_mq_request_completed(rq))
+		blk_mq_complete_request(rq);
+
+	spin_unlock_irqrestore(&vq->lock, flags);
+	return true;
+}
+
+/*
+ * If the device is broken, it will not use any buffers and waiting
+ * for that to happen is pointless. We'll do the cleanup in the driver,
+ * completing all requests for the device.
+ */
+static void virtblk_cleanup_broken_device(struct virtio_blk *vblk)
+{
+	struct request_queue *q = vblk->disk->queue;
+
+	/*
+	 * Start freezing the queue, so that new requests keeps waiting at the
+	 * door of bio_queue_enter(). We cannot fully freeze the queue because
+	 * frozen queue is an empty queue and there are pending requests, so
+	 * only start freezing it.
+	 */
+	blk_freeze_queue_start(q);
+
+	/*
+	 * When quiescing completes, all ongoing dispatches have completed
+	 * and no new dispatch will happen towards the driver.
+	 */
+	blk_mq_quiesce_queue(q);
+
+	/*
+	 * Synchronize with any ongoing VQ callbacks that may have started
+	 * before the VQs were marked as broken. Any outstanding requests
+	 * will be completed by virtblk_complete_request_with_ioerr().
+	 */
+	virtio_synchronize_cbs(vblk->vdev);
+
+	/*
+	 * At this point, no new requests can enter the queue_rq() and
+	 * completion routine will not complete any new requests either for the
+	 * broken vq. Hence, it is safe to cancel all requests which are
+	 * started.
+	 */
+	blk_mq_tagset_busy_iter(&vblk->tag_set,
+				virtblk_complete_request_with_ioerr, vblk);
+	blk_mq_tagset_wait_completed_request(&vblk->tag_set);
+
+	/*
+	 * All pending requests are cleaned up. Time to resume so that disk
+	 * deletion can be smooth. Start the HW queues so that when queue is
+	 * unquiesced requests can again enter the driver.
+	 */
+	blk_mq_start_stopped_hw_queues(q, true);
+
+	/*
+	 * Unquiescing will trigger dispatching any pending requests to the
+	 * driver which has crossed bio_queue_enter() to the driver.
+	 */
+	blk_mq_unquiesce_queue(q);
+
+	/*
+	 * Wait for all pending dispatches to terminate which may have been
+	 * initiated after unquiescing.
+	 */
+	blk_mq_freeze_queue_wait(q);
+
+	/*
+	 * Mark the disk dead so that once we unfreeze the queue, requests
+	 * waiting at the door of bio_queue_enter() can be aborted right away.
+	 */
+	blk_mark_disk_dead(vblk->disk);
+
+	/* Unfreeze the queue so that any waiting requests will be aborted. */
+	blk_mq_unfreeze_queue_nomemrestore(q);
+}
+
 static void virtblk_remove(struct virtio_device *vdev)
 {
 	struct virtio_blk *vblk = vdev->priv;
@@ -1561,6 +1653,9 @@ static void virtblk_remove(struct virtio_device *vdev)
 	/* Make sure no work handler is accessing the device. */
 	flush_work(&vblk->config_work);
 
+	if (virtqueue_is_broken(vblk->vqs[0].vq))
+		virtblk_cleanup_broken_device(vblk);
+
 	del_gendisk(vblk->disk);
 	blk_mq_free_tag_set(&vblk->tag_set);
 
Signature · 1 line
2.34.1
Reviewed-by on v5 1/1
#1 lore raw
Quoted 14 lines · Parav Pandit

When the PCI device is surprise removed, requests may not complete the device as the VQ is marked as broken. Due to this, the disk deletion hangs.

Fix it by aborting the requests when the VQ is broken.

With this fix now fio completes swiftly.
An alternative of IO timeout has been considered, however when the driver knows about unresponsive block device, swiftly clearing them enables users and upper layers to react quickly.

Verified with multiple device unplug iterations with pending requests in virtio used ring and some pending with the device.

Quoted 6 lines
Fixes 43bb40c5b926 ("virtio_pci: Support surprise removal of virtio pci device")
Closes https://urldefense.com/v3/__https://lore.kernel.org/virtualization/ [email protected]/__;!!ACWV5N9M2RV99hQ! OKusQ3sGH7nkVTy9MUDtaIS17UDtjabP3Lby9jTBdT3Ur38XLl2_DOFo0eVzx_dNeh16lD3Ss6ItE9eG$ Reviewed-by: Max Gurtovoy<[email protected]>

look good to me.

Thanks,
Alok

#2 lore raw
In reply to Parav Pandit

Hi Michael, Stefan,

Quoted 37 lines

From: Parav Pandit <[email protected]>
Sent: 02 June 2025 08:15 AM
To: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Cc: [email protected]; NBU-Contact-Li Rongqing (EXTERNAL) <[email protected]>; Chaitanya Kulkarni <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Parav Pandit <[email protected]>; Max Gurtovoy <[email protected]>; Israel Rukshin <[email protected]>
Subject: [PATCH v5] virtio_blk: Fix disk deletion hang on device surprise removal

When the PCI device is surprise removed, requests may not complete the device as the VQ is marked as broken. Due to this, the disk deletion hangs.

Fix it by aborting the requests when the VQ is broken.

With this fix now fio completes swiftly.
An alternative of IO timeout has been considered, however when the driver knows about unresponsive block device, swiftly clearing them enables users and upper layers to react quickly.

Verified with multiple device unplug iterations with pending requests in virtio used ring and some pending with the device.

Fixes 43bb40c5b926 ("virtio_pci: Support surprise removal of virtio pci

device")

Closes:
https://lore.kernel.org/virtualization/c45dd68698cd47238c55fb73ca9b4741
@baidu.com/

Quoted 5 lines

---
v4->v5:
- fixed comment style where comment to start with one empty line at start
- Addressed comments from Alok
- fixed typo in broken vq check

Did you get a chance to review this version where I fixed all the comments you proposed?

[..]

reviewer Reviewed-by on v5 1/1
#3 lore raw
In reply to Parav Pandit
Quoted 3 lines · Parav Pandit

When the PCI device is surprise removed, requests may not complete the device as the VQ is marked as broken. Due to this, the disk deletion hangs.

There are loops in the core virtio driver code that expect device register reads to eventually return 0:
drivers/virtio/virtio_pci_modern.c:vp_reset() drivers/virtio/virtio_pci_modern_dev.c:vp_modern_set_queue_reset()

Is there a hang if these loops are hit when a device has been surprise removed? I'm trying to understand whether surprise removal is fully supported or whether this patch is one step in that direction.

Apart from that, I'm happy with the virtio_blk.c aspects of the patch:

Quoted 173 lines

Fix it by aborting the requests when the VQ is broken.

With this fix now fio completes swiftly.
An alternative of IO timeout has been considered, however when the driver knows about unresponsive block device, swiftly clearing them enables users and upper layers to react quickly.

Verified with multiple device unplug iterations with pending requests in virtio used ring and some pending with the device.

---
v4->v5:
- fixed comment style where comment to start with one empty line at start
- Addressed comments from Alok
- fixed typo in broken vq check
v3->v4:
- Addressed comments from Michael
- renamed virtblk_request_cancel() to virtblk_complete_request_with_ioerr()
- Added comments for virtblk_complete_request_with_ioerr()
- Renamed virtblk_broken_device_cleanup() to virtblk_cleanup_broken_device()
- Added comments for virtblk_cleanup_broken_device()
- Moved the broken vq check in virtblk_remove()
- Fixed comment style to have first empty line
- replaced freezed to frozen
- Fixed comments rephrased

v2->v3:
- Addressed comments from Michael
- updated comment for synchronizing with callbacks

v1->v2:
- Addressed comments from Stephan
- fixed spelling to 'waiting'
- Addressed comments from Michael
- Dropped checking broken vq from queue_rq() and queue_rqs() because it is checked in lower layer routines in virtio core

v0->v1:
- Fixed comments from Stefan to rename a cleanup function
- Improved logic for handling any outstanding requests in bio layer
- improved cancel callback to sync with ongoing done()

---
 drivers/block/virtio_blk.c | 95 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 95 insertions(+)
drivers/block/virtio_blk.c +95 -0
index 7cffea01d868..c5e383c0ac48 100644
@@ -1554,6 +1554,98 @@ static int virtblk_probe(struct virtio_device *vdev)
 	return err;
 }
 
+/*
+ * If the vq is broken, device will not complete requests.
+ * So we do it for the device.
+ */
+static bool virtblk_complete_request_with_ioerr(struct request *rq, void *data)
+{
+	struct virtblk_req *vbr = blk_mq_rq_to_pdu(rq);
+	struct virtio_blk *vblk = data;
+	struct virtio_blk_vq *vq;
+	unsigned long flags;
+
+	vq = &vblk->vqs[rq->mq_hctx->queue_num];
+
+	spin_lock_irqsave(&vq->lock, flags);
+
+	vbr->in_hdr.status = VIRTIO_BLK_S_IOERR;
+	if (blk_mq_request_started(rq) && !blk_mq_request_completed(rq))
+		blk_mq_complete_request(rq);
+
+	spin_unlock_irqrestore(&vq->lock, flags);
+	return true;
+}
+
+/*
+ * If the device is broken, it will not use any buffers and waiting
+ * for that to happen is pointless. We'll do the cleanup in the driver,
+ * completing all requests for the device.
+ */
+static void virtblk_cleanup_broken_device(struct virtio_blk *vblk)
+{
+	struct request_queue *q = vblk->disk->queue;
+
+	/*
+	 * Start freezing the queue, so that new requests keeps waiting at the
+	 * door of bio_queue_enter(). We cannot fully freeze the queue because
+	 * frozen queue is an empty queue and there are pending requests, so
+	 * only start freezing it.
+	 */
+	blk_freeze_queue_start(q);
+
+	/*
+	 * When quiescing completes, all ongoing dispatches have completed
+	 * and no new dispatch will happen towards the driver.
+	 */
+	blk_mq_quiesce_queue(q);
+
+	/*
+	 * Synchronize with any ongoing VQ callbacks that may have started
+	 * before the VQs were marked as broken. Any outstanding requests
+	 * will be completed by virtblk_complete_request_with_ioerr().
+	 */
+	virtio_synchronize_cbs(vblk->vdev);
+
+	/*
+	 * At this point, no new requests can enter the queue_rq() and
+	 * completion routine will not complete any new requests either for the
+	 * broken vq. Hence, it is safe to cancel all requests which are
+	 * started.
+	 */
+	blk_mq_tagset_busy_iter(&vblk->tag_set,
+				virtblk_complete_request_with_ioerr, vblk);
+	blk_mq_tagset_wait_completed_request(&vblk->tag_set);
+
+	/*
+	 * All pending requests are cleaned up. Time to resume so that disk
+	 * deletion can be smooth. Start the HW queues so that when queue is
+	 * unquiesced requests can again enter the driver.
+	 */
+	blk_mq_start_stopped_hw_queues(q, true);
+
+	/*
+	 * Unquiescing will trigger dispatching any pending requests to the
+	 * driver which has crossed bio_queue_enter() to the driver.
+	 */
+	blk_mq_unquiesce_queue(q);
+
+	/*
+	 * Wait for all pending dispatches to terminate which may have been
+	 * initiated after unquiescing.
+	 */
+	blk_mq_freeze_queue_wait(q);
+
+	/*
+	 * Mark the disk dead so that once we unfreeze the queue, requests
+	 * waiting at the door of bio_queue_enter() can be aborted right away.
+	 */
+	blk_mark_disk_dead(vblk->disk);
+
+	/* Unfreeze the queue so that any waiting requests will be aborted. */
+	blk_mq_unfreeze_queue_nomemrestore(q);
+}
+
 static void virtblk_remove(struct virtio_device *vdev)
 {
 	struct virtio_blk *vblk = vdev->priv;
@@ -1561,6 +1653,9 @@ static void virtblk_remove(struct virtio_device *vdev)
 	/* Make sure no work handler is accessing the device. */
 	flush_work(&vblk->config_work);
 
+	if (virtqueue_is_broken(vblk->vqs[0].vq))
+		virtblk_cleanup_broken_device(vblk);
+
 	del_gendisk(vblk->disk);
 	blk_mq_free_tag_set(&vblk->tag_set);
 
Signature · 1 line
2.34.1
#4 lore raw
Quoted 10 lines

From: Stefan Hajnoczi <[email protected]>
Sent: 25 June 2025 12:26 AM

Quoted 3 lines · Parav Pandit

When the PCI device is surprise removed, requests may not complete the device as the VQ is marked as broken. Due to this, the disk deletion hangs.

Quoted 9 lines

There are loops in the core virtio driver code that expect device register reads to eventually return 0:
drivers/virtio/virtio_pci_modern.c:vp_reset() drivers/virtio/virtio_pci_modern_dev.c:vp_modern_set_queue_reset()

Is there a hang if these loops are hit when a device has been surprise removed? I'm trying to understand whether surprise removal is fully supported or whether this patch is one step in that direction.

In one of the previous replies I answered to Michael, but don't have the link handy. It is not fully supported by this patch. It will hang.

This patch restores driver back to the same state what it was before the fixes tag patch.
The virtio stack level work is needed to support surprise removal, including the reset flow you rightly pointed.

Quoted 3 lines

Apart from that, I'm happy with the virtio_blk.c aspects of the patch:

Thanks.

Quoted 182 lines

drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device

Quoted 67 lines

Fix it by aborting the requests when the VQ is broken.

With this fix now fio completes swiftly.
An alternative of IO timeout has been considered, however when the driver knows about unresponsive block device, swiftly clearing them enables users and upper layers to react quickly.

Verified with multiple device unplug iterations with pending requests in virtio used ring and some pending with the device.

Fixes 43bb40c5b926 ("virtio_pci: Support surprise removal of virtio

pci device")

Closes:
https://lore.kernel.org/virtualization/c45dd68698cd47238c55fb73ca9b474
[email protected]/

---
v4->v5:
- fixed comment style where comment to start with one empty line at start
- Addressed comments from Alok
- fixed typo in broken vq check
v3->v4:
- Addressed comments from Michael
- renamed virtblk_request_cancel() to virtblk_complete_request_with_ioerr()
- Added comments for virtblk_complete_request_with_ioerr()
- Renamed virtblk_broken_device_cleanup() to virtblk_cleanup_broken_device()
- Added comments for virtblk_cleanup_broken_device()
- Moved the broken vq check in virtblk_remove()
- Fixed comment style to have first empty line
- replaced freezed to frozen
- Fixed comments rephrased

v2->v3:
- Addressed comments from Michael
- updated comment for synchronizing with callbacks

v1->v2:
- Addressed comments from Stephan
- fixed spelling to 'waiting'
- Addressed comments from Michael
- Dropped checking broken vq from queue_rq() and queue_rqs() because it is checked in lower layer routines in virtio core

v0->v1:
- Fixed comments from Stefan to rename a cleanup function
- Improved logic for handling any outstanding requests in bio layer
- improved cancel callback to sync with ongoing done()

---
 drivers/block/virtio_blk.c | 95
++++++++++++++++++++++++++++++++++++++
 1 file changed, 95 insertions(+)
drivers/block/virtio_blk.c +0 -0
index 7cffea01d868..c5e383c0ac48 100644
@@ -1554,6 +1554,98 @@ static int virtblk_probe(struct virtio_device

*vdev)

Quoted 28 lines
drivers/block/virtio_blk.c +25 -0
 	return err;
 }
 
+/*
+ * If the vq is broken, device will not complete requests.
+ * So we do it for the device.
+ */
+static bool virtblk_complete_request_with_ioerr(struct request *rq,
+void *data) {
+	struct virtblk_req *vbr = blk_mq_rq_to_pdu(rq);
+	struct virtio_blk *vblk = data;
+	struct virtio_blk_vq *vq;
+	unsigned long flags;
+
+	vq = &vblk->vqs[rq->mq_hctx->queue_num];
+
+	spin_lock_irqsave(&vq->lock, flags);
+
+	vbr->in_hdr.status = VIRTIO_BLK_S_IOERR;
+	if (blk_mq_request_started(rq) && !blk_mq_request_completed(rq))
+		blk_mq_complete_request(rq);
+
+	spin_unlock_irqrestore(&vq->lock, flags);
+	return true;
+}
+
+/*
+ * If the device is broken, it will not use any buffers and waiting

drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device *vdev)

Quoted 10 lines
drivers/block/virtio_blk.c +10 -0
+ * for that to happen is pointless. We'll do the cleanup in the
+driver,
+ * completing all requests for the device.
+ */
+static void virtblk_cleanup_broken_device(struct virtio_blk *vblk) {
+	struct request_queue *q = vblk->disk->queue;
+
+	/*
+	 * Start freezing the queue, so that new requests keeps waiting at the
+	 * door of bio_queue_enter(). We cannot fully freeze the queue

because

Quoted 11 lines
drivers/block/virtio_blk.c +11 -0
+	 * frozen queue is an empty queue and there are pending requests, so
+	 * only start freezing it.
+	 */
+	blk_freeze_queue_start(q);
+
+	/*
+	 * When quiescing completes, all ongoing dispatches have completed
+	 * and no new dispatch will happen towards the driver.
+	 */
+	blk_mq_quiesce_queue(q);
+

drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device *vdev)

Quoted 10 lines
drivers/block/virtio_blk.c +10 -0
+	/*
+	 * Synchronize with any ongoing VQ callbacks that may have started
+	 * before the VQs were marked as broken. Any outstanding requests
+	 * will be completed by virtblk_complete_request_with_ioerr().
+	 */
+	virtio_synchronize_cbs(vblk->vdev);
+
+	/*
+	 * At this point, no new requests can enter the queue_rq() and
+	 * completion routine will not complete any new requests either for

the

drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device *vdev)

Quoted 10 lines
drivers/block/virtio_blk.c +10 -0
+	 * broken vq. Hence, it is safe to cancel all requests which are
+	 * started.
+	 */
+	blk_mq_tagset_busy_iter(&vblk->tag_set,
+				virtblk_complete_request_with_ioerr, vblk);
+	blk_mq_tagset_wait_completed_request(&vblk->tag_set);
+
+	/*
+	 * All pending requests are cleaned up. Time to resume so that disk
+	 * deletion can be smooth. Start the HW queues so that when queue

is

Quoted 9 lines
drivers/block/virtio_blk.c +9 -0
+	 * unquiesced requests can again enter the driver.
+	 */
+	blk_mq_start_stopped_hw_queues(q, true);
+
+	/*
+	 * Unquiescing will trigger dispatching any pending requests to the
+	 * driver which has crossed bio_queue_enter() to the driver.
+	 */
+	blk_mq_unquiesce_queue(q);

drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device *vdev)

Quoted 10 lines
drivers/block/virtio_blk.c +10 -0
+
+	/*
+	 * Wait for all pending dispatches to terminate which may have been
+	 * initiated after unquiescing.
+	 */
+	blk_mq_freeze_queue_wait(q);
+
+	/*
+	 * Mark the disk dead so that once we unfreeze the queue, requests
+	 * waiting at the door of bio_queue_enter() can be aborted right

away.

Quoted 22 lines
drivers/block/virtio_blk.c +7 -0
+	 */
+	blk_mark_disk_dead(vblk->disk);
+
+	/* Unfreeze the queue so that any waiting requests will be aborted. */
+	blk_mq_unfreeze_queue_nomemrestore(q);
+}
+
 static void virtblk_remove(struct virtio_device *vdev)  {
 	struct virtio_blk *vblk = vdev->priv; @@ -1561,6 +1653,9 @@ static
 void virtblk_remove(struct virtio_device *vdev)
drivers/block/virtio_blk.c +3 -1
 	/* Make sure no work handler is accessing the device. */
 	flush_work(&vblk->config_work);
 
+	if (virtqueue_is_broken(vblk->vqs[0].vq))
+		virtblk_cleanup_broken_device(vblk);
+
 	del_gendisk(vblk->disk);
 	blk_mq_free_tag_set(&vblk->tag_set);
 
--

2.34.1

#5 lore raw
Quoted 19 lines · Parav Pandit
Quoted 10 lines

From: Stefan Hajnoczi <[email protected]>
Sent: 25 June 2025 12:26 AM

Quoted 3 lines · Parav Pandit

When the PCI device is surprise removed, requests may not complete the device as the VQ is marked as broken. Due to this, the disk deletion hangs.

Quoted 9 lines

There are loops in the core virtio driver code that expect device register reads to eventually return 0:
drivers/virtio/virtio_pci_modern.c:vp_reset() drivers/virtio/virtio_pci_modern_dev.c:vp_modern_set_queue_reset()

Is there a hang if these loops are hit when a device has been surprise removed? I'm trying to understand whether surprise removal is fully supported or whether this patch is one step in that direction.

Quoted 5 lines

In one of the previous replies I answered to Michael, but don't have the link handy. It is not fully supported by this patch. It will hang.

This patch restores driver back to the same state what it was before the fixes tag patch.
The virtio stack level work is needed to support surprise removal, including the reset flow you rightly pointed.

Have plans to do that?

Quoted 187 lines
Quoted 3 lines

Apart from that, I'm happy with the virtio_blk.c aspects of the patch:

Thanks.

Quoted 182 lines

drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device

Quoted 67 lines

Fix it by aborting the requests when the VQ is broken.

With this fix now fio completes swiftly.
An alternative of IO timeout has been considered, however when the driver knows about unresponsive block device, swiftly clearing them enables users and upper layers to react quickly.

Verified with multiple device unplug iterations with pending requests in virtio used ring and some pending with the device.

Fixes 43bb40c5b926 ("virtio_pci: Support surprise removal of virtio

pci device")

Closes:
https://lore.kernel.org/virtualization/c45dd68698cd47238c55fb73ca9b474
[email protected]/

---
v4->v5:
- fixed comment style where comment to start with one empty line at start
- Addressed comments from Alok
- fixed typo in broken vq check
v3->v4:
- Addressed comments from Michael
- renamed virtblk_request_cancel() to virtblk_complete_request_with_ioerr()
- Added comments for virtblk_complete_request_with_ioerr()
- Renamed virtblk_broken_device_cleanup() to virtblk_cleanup_broken_device()
- Added comments for virtblk_cleanup_broken_device()
- Moved the broken vq check in virtblk_remove()
- Fixed comment style to have first empty line
- replaced freezed to frozen
- Fixed comments rephrased

v2->v3:
- Addressed comments from Michael
- updated comment for synchronizing with callbacks

v1->v2:
- Addressed comments from Stephan
- fixed spelling to 'waiting'
- Addressed comments from Michael
- Dropped checking broken vq from queue_rq() and queue_rqs() because it is checked in lower layer routines in virtio core

v0->v1:
- Fixed comments from Stefan to rename a cleanup function
- Improved logic for handling any outstanding requests in bio layer
- improved cancel callback to sync with ongoing done()

---
 drivers/block/virtio_blk.c | 95
++++++++++++++++++++++++++++++++++++++
 1 file changed, 95 insertions(+)
drivers/block/virtio_blk.c +0 -0
index 7cffea01d868..c5e383c0ac48 100644
@@ -1554,6 +1554,98 @@ static int virtblk_probe(struct virtio_device

*vdev)

Quoted 28 lines
drivers/block/virtio_blk.c +25 -0
 	return err;
 }
 
+/*
+ * If the vq is broken, device will not complete requests.
+ * So we do it for the device.
+ */
+static bool virtblk_complete_request_with_ioerr(struct request *rq,
+void *data) {
+	struct virtblk_req *vbr = blk_mq_rq_to_pdu(rq);
+	struct virtio_blk *vblk = data;
+	struct virtio_blk_vq *vq;
+	unsigned long flags;
+
+	vq = &vblk->vqs[rq->mq_hctx->queue_num];
+
+	spin_lock_irqsave(&vq->lock, flags);
+
+	vbr->in_hdr.status = VIRTIO_BLK_S_IOERR;
+	if (blk_mq_request_started(rq) && !blk_mq_request_completed(rq))
+		blk_mq_complete_request(rq);
+
+	spin_unlock_irqrestore(&vq->lock, flags);
+	return true;
+}
+
+/*
+ * If the device is broken, it will not use any buffers and waiting

drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device *vdev)

Quoted 10 lines
drivers/block/virtio_blk.c +10 -0
+ * for that to happen is pointless. We'll do the cleanup in the
+driver,
+ * completing all requests for the device.
+ */
+static void virtblk_cleanup_broken_device(struct virtio_blk *vblk) {
+	struct request_queue *q = vblk->disk->queue;
+
+	/*
+	 * Start freezing the queue, so that new requests keeps waiting at the
+	 * door of bio_queue_enter(). We cannot fully freeze the queue

because

Quoted 11 lines
drivers/block/virtio_blk.c +11 -0
+	 * frozen queue is an empty queue and there are pending requests, so
+	 * only start freezing it.
+	 */
+	blk_freeze_queue_start(q);
+
+	/*
+	 * When quiescing completes, all ongoing dispatches have completed
+	 * and no new dispatch will happen towards the driver.
+	 */
+	blk_mq_quiesce_queue(q);
+

drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device *vdev)

Quoted 10 lines
drivers/block/virtio_blk.c +10 -0
+	/*
+	 * Synchronize with any ongoing VQ callbacks that may have started
+	 * before the VQs were marked as broken. Any outstanding requests
+	 * will be completed by virtblk_complete_request_with_ioerr().
+	 */
+	virtio_synchronize_cbs(vblk->vdev);
+
+	/*
+	 * At this point, no new requests can enter the queue_rq() and
+	 * completion routine will not complete any new requests either for

the

drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device *vdev)

Quoted 10 lines
drivers/block/virtio_blk.c +10 -0
+	 * broken vq. Hence, it is safe to cancel all requests which are
+	 * started.
+	 */
+	blk_mq_tagset_busy_iter(&vblk->tag_set,
+				virtblk_complete_request_with_ioerr, vblk);
+	blk_mq_tagset_wait_completed_request(&vblk->tag_set);
+
+	/*
+	 * All pending requests are cleaned up. Time to resume so that disk
+	 * deletion can be smooth. Start the HW queues so that when queue

is

Quoted 9 lines
drivers/block/virtio_blk.c +9 -0
+	 * unquiesced requests can again enter the driver.
+	 */
+	blk_mq_start_stopped_hw_queues(q, true);
+
+	/*
+	 * Unquiescing will trigger dispatching any pending requests to the
+	 * driver which has crossed bio_queue_enter() to the driver.
+	 */
+	blk_mq_unquiesce_queue(q);

drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device *vdev)

Quoted 10 lines
drivers/block/virtio_blk.c +10 -0
+
+	/*
+	 * Wait for all pending dispatches to terminate which may have been
+	 * initiated after unquiescing.
+	 */
+	blk_mq_freeze_queue_wait(q);
+
+	/*
+	 * Mark the disk dead so that once we unfreeze the queue, requests
+	 * waiting at the door of bio_queue_enter() can be aborted right

away.

Quoted 22 lines
drivers/block/virtio_blk.c +7 -0
+	 */
+	blk_mark_disk_dead(vblk->disk);
+
+	/* Unfreeze the queue so that any waiting requests will be aborted. */
+	blk_mq_unfreeze_queue_nomemrestore(q);
+}
+
 static void virtblk_remove(struct virtio_device *vdev)  {
 	struct virtio_blk *vblk = vdev->priv; @@ -1561,6 +1653,9 @@ static
 void virtblk_remove(struct virtio_device *vdev)
drivers/block/virtio_blk.c +3 -1
 	/* Make sure no work handler is accessing the device. */
 	flush_work(&vblk->config_work);
 
+	if (virtqueue_is_broken(vblk->vqs[0].vq))
+		virtblk_cleanup_broken_device(vblk);
+
 	del_gendisk(vblk->disk);
 	blk_mq_free_tag_set(&vblk->tag_set);
 
--

2.34.1

#6 lore raw
Quoted 27 lines

From: Michael S. Tsirkin <[email protected]>
Sent: 25 June 2025 12:37 AM

Quoted 15 lines · Parav Pandit
Quoted 15 lines

From: Stefan Hajnoczi <[email protected]>
Sent: 25 June 2025 12:26 AM

Quoted 3 lines · Parav Pandit

When the PCI device is surprise removed, requests may not complete the device as the VQ is marked as broken. Due to this, the disk deletion hangs.

There are loops in the core virtio driver code that expect device register reads to eventually return 0:
drivers/virtio/virtio_pci_modern.c:vp_reset() drivers/virtio/virtio_pci_modern_dev.c:vp_modern_set_queue_reset()

Quoted 5 lines
Quoted 4 lines

Is there a hang if these loops are hit when a device has been surprise removed? I'm trying to understand whether surprise removal is fully supported or whether this patch is one step in that direction.

In one of the previous replies I answered to Michael, but don't have the link

handy.

Quoted 9 lines
Quoted 3 lines

It is not fully supported by this patch. It will hang.

This patch restores driver back to the same state what it was before the fixes

tag patch.

Quoted 1 line

The virtio stack level work is needed to support surprise removal, including

the reset flow you rightly pointed.

Have plans to do that?

Didn't give enough thoughts on it yet.

Quoted 193 lines
Quoted 193 lines
Quoted 3 lines

Apart from that, I'm happy with the virtio_blk.c aspects of the patch:

Thanks.

Quoted 188 lines
Quoted 69 lines

Fix it by aborting the requests when the VQ is broken.

With this fix now fio completes swiftly.
An alternative of IO timeout has been considered, however when the driver knows about unresponsive block device, swiftly clearing them enables users and upper layers to react quickly.

Verified with multiple device unplug iterations with pending requests in virtio used ring and some pending with the device.

Fixes 43bb40c5b926 ("virtio_pci: Support surprise removal of

virtio pci device")

Closes:
https://lore.kernel.org/virtualization/c45dd68698cd47238c55fb73ca9
b474
[email protected]/

---
v4->v5:
- fixed comment style where comment to start with one empty line at start
- Addressed comments from Alok
- fixed typo in broken vq check
v3->v4:
- Addressed comments from Michael
- renamed virtblk_request_cancel() to virtblk_complete_request_with_ioerr()
- Added comments for virtblk_complete_request_with_ioerr()
- Renamed virtblk_broken_device_cleanup() to virtblk_cleanup_broken_device()
- Added comments for virtblk_cleanup_broken_device()
- Moved the broken vq check in virtblk_remove()
- Fixed comment style to have first empty line
- replaced freezed to frozen
- Fixed comments rephrased

v2->v3:
- Addressed comments from Michael
- updated comment for synchronizing with callbacks

v1->v2:
- Addressed comments from Stephan
- fixed spelling to 'waiting'
- Addressed comments from Michael
- Dropped checking broken vq from queue_rq() and queue_rqs() because it is checked in lower layer routines in virtio core

v0->v1:
- Fixed comments from Stefan to rename a cleanup function
- Improved logic for handling any outstanding requests in bio layer
- improved cancel callback to sync with ongoing done()

---
 drivers/block/virtio_blk.c | 95
++++++++++++++++++++++++++++++++++++++
 1 file changed, 95 insertions(+)
(quoted diff) +0 -0

b/drivers/block/virtio_blk.c index 7cffea01d868..c5e383c0ac48 100644

drivers/block/virtio_blk.c +0 -0
@@ -1554,6 +1554,98 @@ static int virtblk_probe(struct
Quoted 1 line

virtio_device

*vdev)

Quoted 29 lines
drivers/block/virtio_blk.c +26 -0
 	return err;
 }
 
+/*
+ * If the vq is broken, device will not complete requests.
+ * So we do it for the device.
+ */
+static bool virtblk_complete_request_with_ioerr(struct request
+*rq, void *data) {
+	struct virtblk_req *vbr = blk_mq_rq_to_pdu(rq);
+	struct virtio_blk *vblk = data;
+	struct virtio_blk_vq *vq;
+	unsigned long flags;
+
+	vq = &vblk->vqs[rq->mq_hctx->queue_num];
+
+	spin_lock_irqsave(&vq->lock, flags);
+
+	vbr->in_hdr.status = VIRTIO_BLK_S_IOERR;
+	if (blk_mq_request_started(rq) && !blk_mq_request_completed(rq))
+		blk_mq_complete_request(rq);
+
+	spin_unlock_irqrestore(&vq->lock, flags);
+	return true;
+}
+
+/*
+ * If the device is broken, it will not use any buffers and
+waiting

drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device *vdev)

Quoted 10 lines
drivers/block/virtio_blk.c +10 -0
+ * for that to happen is pointless. We'll do the cleanup in the
+driver,
+ * completing all requests for the device.
+ */
+static void virtblk_cleanup_broken_device(struct virtio_blk *vblk) {
+	struct request_queue *q = vblk->disk->queue;
+
+	/*
+	 * Start freezing the queue, so that new requests keeps waiting at the
+	 * door of bio_queue_enter(). We cannot fully freeze the queue

because

Quoted 12 lines
drivers/block/virtio_blk.c +12 -0
+	 * frozen queue is an empty queue and there are pending requests, so
+	 * only start freezing it.
+	 */
+	blk_freeze_queue_start(q);
+
+	/*
+	 * When quiescing completes, all ongoing dispatches have completed
+	 * and no new dispatch will happen towards the driver.
+	 */
+	blk_mq_quiesce_queue(q);
+
+	/*

drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device *vdev)

Quoted 10 lines
drivers/block/virtio_blk.c +10 -0
+	 * Synchronize with any ongoing VQ callbacks that may have started
+	 * before the VQs were marked as broken. Any outstanding requests
+	 * will be completed by virtblk_complete_request_with_ioerr().
+	 */
+	virtio_synchronize_cbs(vblk->vdev);
+
+	/*
+	 * At this point, no new requests can enter the queue_rq() and
+	 * completion routine will not complete any new requests either
+for

the

drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device *vdev)

Quoted 11 lines
drivers/block/virtio_blk.c +11 -0
+	 * broken vq. Hence, it is safe to cancel all requests which are
+	 * started.
+	 */
+	blk_mq_tagset_busy_iter(&vblk->tag_set,
+				virtblk_complete_request_with_ioerr, vblk);
+	blk_mq_tagset_wait_completed_request(&vblk->tag_set);
+
+	/*
+	 * All pending requests are cleaned up. Time to resume so that disk
+	 * deletion can be smooth. Start the HW queues so that when
+queue

is

Quoted 9 lines
drivers/block/virtio_blk.c +9 -0
+	 * unquiesced requests can again enter the driver.
+	 */
+	blk_mq_start_stopped_hw_queues(q, true);
+
+	/*
+	 * Unquiescing will trigger dispatching any pending requests to the
+	 * driver which has crossed bio_queue_enter() to the driver.
+	 */
+	blk_mq_unquiesce_queue(q);

drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device *vdev)

Quoted 10 lines
drivers/block/virtio_blk.c +10 -0
+
+	/*
+	 * Wait for all pending dispatches to terminate which may have been
+	 * initiated after unquiescing.
+	 */
+	blk_mq_freeze_queue_wait(q);
+
+	/*
+	 * Mark the disk dead so that once we unfreeze the queue, requests
+	 * waiting at the door of bio_queue_enter() can be aborted right

away.

Quoted 22 lines
drivers/block/virtio_blk.c +7 -0
+	 */
+	blk_mark_disk_dead(vblk->disk);
+
+	/* Unfreeze the queue so that any waiting requests will be aborted. */
+	blk_mq_unfreeze_queue_nomemrestore(q);
+}
+
 static void virtblk_remove(struct virtio_device *vdev)  {
 	struct virtio_blk *vblk = vdev->priv; @@ -1561,6 +1653,9 @@

static void virtblk_remove(struct virtio_device *vdev)

drivers/block/virtio_blk.c +3 -1
 	/* Make sure no work handler is accessing the device. */
 	flush_work(&vblk->config_work);
 
+	if (virtqueue_is_broken(vblk->vqs[0].vq))
+		virtblk_cleanup_broken_device(vblk);
+
 	del_gendisk(vblk->disk);
 	blk_mq_free_tag_set(&vblk->tag_set);
 
--

2.34.1

#7 lore raw
Quoted 29 lines · Parav Pandit
Quoted 29 lines

From: Michael S. Tsirkin <[email protected]>
Sent: 25 June 2025 12:37 AM

Quoted 15 lines · Parav Pandit
Quoted 15 lines

From: Stefan Hajnoczi <[email protected]>
Sent: 25 June 2025 12:26 AM

Quoted 3 lines · Parav Pandit

When the PCI device is surprise removed, requests may not complete the device as the VQ is marked as broken. Due to this, the disk deletion hangs.

There are loops in the core virtio driver code that expect device register reads to eventually return 0:
drivers/virtio/virtio_pci_modern.c:vp_reset() drivers/virtio/virtio_pci_modern_dev.c:vp_modern_set_queue_reset()

Quoted 5 lines
Quoted 4 lines

Is there a hang if these loops are hit when a device has been surprise removed? I'm trying to understand whether surprise removal is fully supported or whether this patch is one step in that direction.

In one of the previous replies I answered to Michael, but don't have the link

handy.

Quoted 2 lines

It is not fully supported by this patch. It will hang.

Quoted 8 lines
Quoted 7 lines
Quoted 1 line

This patch restores driver back to the same state what it was before the fixes

tag patch.

Quoted 1 line

The virtio stack level work is needed to support surprise removal, including

the reset flow you rightly pointed.

Have plans to do that?

Didn't give enough thoughts on it yet.

This one is kind of pointless then? It just fixes the specific race window that your test harness happens to hit?

Maybe it's better to wait until someone does a comprehensive fix..

Quoted 193 lines
Quoted 193 lines
Quoted 193 lines
Quoted 3 lines

Apart from that, I'm happy with the virtio_blk.c aspects of the patch:

Thanks.

Quoted 188 lines
Quoted 69 lines

Fix it by aborting the requests when the VQ is broken.

With this fix now fio completes swiftly.
An alternative of IO timeout has been considered, however when the driver knows about unresponsive block device, swiftly clearing them enables users and upper layers to react quickly.

Verified with multiple device unplug iterations with pending requests in virtio used ring and some pending with the device.

Fixes 43bb40c5b926 ("virtio_pci: Support surprise removal of

virtio pci device")

Closes:
https://lore.kernel.org/virtualization/c45dd68698cd47238c55fb73ca9
b474
[email protected]/

---
v4->v5:
- fixed comment style where comment to start with one empty line at start
- Addressed comments from Alok
- fixed typo in broken vq check
v3->v4:
- Addressed comments from Michael
- renamed virtblk_request_cancel() to virtblk_complete_request_with_ioerr()
- Added comments for virtblk_complete_request_with_ioerr()
- Renamed virtblk_broken_device_cleanup() to virtblk_cleanup_broken_device()
- Added comments for virtblk_cleanup_broken_device()
- Moved the broken vq check in virtblk_remove()
- Fixed comment style to have first empty line
- replaced freezed to frozen
- Fixed comments rephrased

v2->v3:
- Addressed comments from Michael
- updated comment for synchronizing with callbacks

v1->v2:
- Addressed comments from Stephan
- fixed spelling to 'waiting'
- Addressed comments from Michael
- Dropped checking broken vq from queue_rq() and queue_rqs() because it is checked in lower layer routines in virtio core

v0->v1:
- Fixed comments from Stefan to rename a cleanup function
- Improved logic for handling any outstanding requests in bio layer
- improved cancel callback to sync with ongoing done()

---
 drivers/block/virtio_blk.c | 95
++++++++++++++++++++++++++++++++++++++
 1 file changed, 95 insertions(+)
(quoted diff) +0 -0

b/drivers/block/virtio_blk.c index 7cffea01d868..c5e383c0ac48 100644

drivers/block/virtio_blk.c +0 -0
@@ -1554,6 +1554,98 @@ static int virtblk_probe(struct
Quoted 1 line

virtio_device

*vdev)

Quoted 29 lines
drivers/block/virtio_blk.c +26 -0
 	return err;
 }
 
+/*
+ * If the vq is broken, device will not complete requests.
+ * So we do it for the device.
+ */
+static bool virtblk_complete_request_with_ioerr(struct request
+*rq, void *data) {
+	struct virtblk_req *vbr = blk_mq_rq_to_pdu(rq);
+	struct virtio_blk *vblk = data;
+	struct virtio_blk_vq *vq;
+	unsigned long flags;
+
+	vq = &vblk->vqs[rq->mq_hctx->queue_num];
+
+	spin_lock_irqsave(&vq->lock, flags);
+
+	vbr->in_hdr.status = VIRTIO_BLK_S_IOERR;
+	if (blk_mq_request_started(rq) && !blk_mq_request_completed(rq))
+		blk_mq_complete_request(rq);
+
+	spin_unlock_irqrestore(&vq->lock, flags);
+	return true;
+}
+
+/*
+ * If the device is broken, it will not use any buffers and
+waiting

drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device *vdev)

Quoted 10 lines
drivers/block/virtio_blk.c +10 -0
+ * for that to happen is pointless. We'll do the cleanup in the
+driver,
+ * completing all requests for the device.
+ */
+static void virtblk_cleanup_broken_device(struct virtio_blk *vblk) {
+	struct request_queue *q = vblk->disk->queue;
+
+	/*
+	 * Start freezing the queue, so that new requests keeps waiting at the
+	 * door of bio_queue_enter(). We cannot fully freeze the queue

because

Quoted 12 lines
drivers/block/virtio_blk.c +12 -0
+	 * frozen queue is an empty queue and there are pending requests, so
+	 * only start freezing it.
+	 */
+	blk_freeze_queue_start(q);
+
+	/*
+	 * When quiescing completes, all ongoing dispatches have completed
+	 * and no new dispatch will happen towards the driver.
+	 */
+	blk_mq_quiesce_queue(q);
+
+	/*

drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device *vdev)

Quoted 10 lines
drivers/block/virtio_blk.c +10 -0
+	 * Synchronize with any ongoing VQ callbacks that may have started
+	 * before the VQs were marked as broken. Any outstanding requests
+	 * will be completed by virtblk_complete_request_with_ioerr().
+	 */
+	virtio_synchronize_cbs(vblk->vdev);
+
+	/*
+	 * At this point, no new requests can enter the queue_rq() and
+	 * completion routine will not complete any new requests either
+for

the

drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device *vdev)

Quoted 11 lines
drivers/block/virtio_blk.c +11 -0
+	 * broken vq. Hence, it is safe to cancel all requests which are
+	 * started.
+	 */
+	blk_mq_tagset_busy_iter(&vblk->tag_set,
+				virtblk_complete_request_with_ioerr, vblk);
+	blk_mq_tagset_wait_completed_request(&vblk->tag_set);
+
+	/*
+	 * All pending requests are cleaned up. Time to resume so that disk
+	 * deletion can be smooth. Start the HW queues so that when
+queue

is

Quoted 9 lines
drivers/block/virtio_blk.c +9 -0
+	 * unquiesced requests can again enter the driver.
+	 */
+	blk_mq_start_stopped_hw_queues(q, true);
+
+	/*
+	 * Unquiescing will trigger dispatching any pending requests to the
+	 * driver which has crossed bio_queue_enter() to the driver.
+	 */
+	blk_mq_unquiesce_queue(q);

drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device *vdev)

Quoted 10 lines
drivers/block/virtio_blk.c +10 -0
+
+	/*
+	 * Wait for all pending dispatches to terminate which may have been
+	 * initiated after unquiescing.
+	 */
+	blk_mq_freeze_queue_wait(q);
+
+	/*
+	 * Mark the disk dead so that once we unfreeze the queue, requests
+	 * waiting at the door of bio_queue_enter() can be aborted right

away.

Quoted 22 lines
drivers/block/virtio_blk.c +7 -0
+	 */
+	blk_mark_disk_dead(vblk->disk);
+
+	/* Unfreeze the queue so that any waiting requests will be aborted. */
+	blk_mq_unfreeze_queue_nomemrestore(q);
+}
+
 static void virtblk_remove(struct virtio_device *vdev)  {
 	struct virtio_blk *vblk = vdev->priv; @@ -1561,6 +1653,9 @@

static void virtblk_remove(struct virtio_device *vdev)

drivers/block/virtio_blk.c +3 -1
 	/* Make sure no work handler is accessing the device. */
 	flush_work(&vblk->config_work);
 
+	if (virtqueue_is_broken(vblk->vqs[0].vq))
+		virtblk_cleanup_broken_device(vblk);
+
 	del_gendisk(vblk->disk);
 	blk_mq_free_tag_set(&vblk->tag_set);
 
--

2.34.1

#8 lore raw
Quoted 42 lines

From: Michael S. Tsirkin <[email protected]>
Sent: 25 June 2025 01:24 AM

Quoted 22 lines · Parav Pandit
Quoted 22 lines

From: Michael S. Tsirkin <[email protected]>
Sent: 25 June 2025 12:37 AM

Quoted 16 lines · Parav Pandit
Quoted 16 lines

From: Stefan Hajnoczi <[email protected]>
Sent: 25 June 2025 12:26 AM

Quoted 3 lines · Parav Pandit

When the PCI device is surprise removed, requests may not complete the device as the VQ is marked as broken. Due to this, the disk deletion hangs.

There are loops in the core virtio driver code that expect device register reads to eventually return 0:
drivers/virtio/virtio_pci_modern.c:vp_reset() drivers/virtio/virtio_pci_modern_dev.c:vp_modern_set_queue_reset ()

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Is there a hang if these loops are hit when a device has been surprise removed? I'm trying to understand whether surprise removal is fully supported or whether this patch is one step in that

direction.

Quoted 11 lines
Quoted 11 lines
Quoted 3 lines
Quoted 1 line

In one of the previous replies I answered to Michael, but don't have the link

handy.

Quoted 4 lines

It is not fully supported by this patch. It will hang.

This patch restores driver back to the same state what it was before the fixes

tag patch.

Quoted 2 lines

The virtio stack level work is needed to support surprise removal, including

Quoted 9 lines
Quoted 5 lines
Quoted 4 lines

the reset flow you rightly pointed.

Have plans to do that?

Didn't give enough thoughts on it yet.

This one is kind of pointless then? It just fixes the specific race window that your test harness happens to hit?

It was reported by Li from Baidu, whose tests failed. I missed to tag "reported-by" in v5. I had it until v4. :(

Quoted 3 lines

Maybe it's better to wait until someone does a comprehensive fix..

Oh, I was under impression is that you wanted to step forward in discussion of v4.
If you prefer a comprehensive support across layers of virtio, I suggest you should revert the cited patch in fixes tag.

Otherwise, it is in degraded state as virtio never supported surprise removal. By reverting the cited patch (or with this fix), the requests and disk deletion will not hang.

Please let me know if I should re-send to revert the patch listed in fixes tag.

Quoted 210 lines
Quoted 89 lines
Quoted 89 lines
Quoted 89 lines
Quoted 3 lines

Apart from that, I'm happy with the virtio_blk.c aspects of the patch:

Thanks.

Quoted 84 lines
Quoted 70 lines

Fix it by aborting the requests when the VQ is broken.

With this fix now fio completes swiftly.
An alternative of IO timeout has been considered, however when the driver knows about unresponsive block device, swiftly clearing them enables users and upper layers to react quickly.

Verified with multiple device unplug iterations with pending requests in virtio used ring and some pending with the device.

Fixes 43bb40c5b926 ("virtio_pci: Support surprise removal of

virtio pci device")

Closes:
https://lore.kernel.org/virtualization/c45dd68698cd47238c55fb7
3ca9
b474
[email protected]/

---
v4->v5:
- fixed comment style where comment to start with one empty line at start
- Addressed comments from Alok
- fixed typo in broken vq check
v3->v4:
- Addressed comments from Michael
- renamed virtblk_request_cancel() to virtblk_complete_request_with_ioerr()
- Added comments for virtblk_complete_request_with_ioerr()
- Renamed virtblk_broken_device_cleanup() to virtblk_cleanup_broken_device()
- Added comments for virtblk_cleanup_broken_device()
- Moved the broken vq check in virtblk_remove()
- Fixed comment style to have first empty line
- replaced freezed to frozen
- Fixed comments rephrased

v2->v3:
- Addressed comments from Michael
- updated comment for synchronizing with callbacks

v1->v2:
- Addressed comments from Stephan
- fixed spelling to 'waiting'
- Addressed comments from Michael
- Dropped checking broken vq from queue_rq() and queue_rqs() because it is checked in lower layer routines in virtio core

v0->v1:
- Fixed comments from Stefan to rename a cleanup function
- Improved logic for handling any outstanding requests in bio layer
- improved cancel callback to sync with ongoing done()

---
 drivers/block/virtio_blk.c | 95
++++++++++++++++++++++++++++++++++++++
 1 file changed, 95 insertions(+)
(quoted diff) +0 -0

b/drivers/block/virtio_blk.c index 7cffea01d868..c5e383c0ac48 100644

drivers/block/virtio_blk.c +0 -0
@@ -1554,6 +1554,98 @@ static int virtblk_probe(struct
Quoted 1 line

virtio_device

*vdev)

Quoted 12 lines
drivers/block/virtio_blk.c +9 -0
 	return err;
 }
 
+/*
+ * If the vq is broken, device will not complete requests.
+ * So we do it for the device.
+ */
+static bool virtblk_complete_request_with_ioerr(struct
+request *rq, void *data) {
+	struct virtblk_req *vbr = blk_mq_rq_to_pdu(rq);
+	struct virtio_blk *vblk = data;
+	struct virtio_blk_vq *vq;
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device *vdev)

Quoted 8 lines
drivers/block/virtio_blk.c +8 -0
+	unsigned long flags;
+
+	vq = &vblk->vqs[rq->mq_hctx->queue_num];
+
+	spin_lock_irqsave(&vq->lock, flags);
+
+	vbr->in_hdr.status = VIRTIO_BLK_S_IOERR;
+	if (blk_mq_request_started(rq) &&

!blk_mq_request_completed(rq))

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
drivers/block/virtio_blk.c +10 -0
+		blk_mq_complete_request(rq);
+
+	spin_unlock_irqrestore(&vq->lock, flags);
+	return true;
+}
+
+/*
+ * If the device is broken, it will not use any buffers and
+waiting
+ * for that to happen is pointless. We'll do the cleanup in
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device *vdev)

Quoted 8 lines
drivers/block/virtio_blk.c +8 -0
+the driver,
+ * completing all requests for the device.
+ */
+static void virtblk_cleanup_broken_device(struct virtio_blk *vblk) {
+	struct request_queue *q = vblk->disk->queue;
+
+	/*
+	 * Start freezing the queue, so that new requests keeps

waiting at the

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

drivers/block/virtio_blk.c

Quoted 2 lines
drivers/block/virtio_blk.c +2 -0
+	 * door of bio_queue_enter(). We cannot fully freeze the
+queue

because

drivers/block/virtio_blk.c

Quoted 1 line
drivers/block/virtio_blk.c +1 -0
+	 * frozen queue is an empty queue and there are pending

requests, so

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device *vdev)

Quoted 6 lines
drivers/block/virtio_blk.c +6 -0
+	 * only start freezing it.
+	 */
+	blk_freeze_queue_start(q);
+
+	/*
+	 * When quiescing completes, all ongoing dispatches have

completed

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device *vdev)

Quoted 6 lines
drivers/block/virtio_blk.c +6 -0
+	 * and no new dispatch will happen towards the driver.
+	 */
+	blk_mq_quiesce_queue(q);
+
+	/*
+	 * Synchronize with any ongoing VQ callbacks that may have

started

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

drivers/block/virtio_blk.c

Quoted 1 line
drivers/block/virtio_blk.c +1 -0
+	 * before the VQs were marked as broken. Any outstanding

requests

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

drivers/block/virtio_blk.c

Quoted 1 line
drivers/block/virtio_blk.c +1 -0
+	 * will be completed by

virtblk_complete_request_with_ioerr().

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device *vdev)

Quoted 5 lines
drivers/block/virtio_blk.c +5 -0
+	 */
+	virtio_synchronize_cbs(vblk->vdev);
+
+	/*
+	 * At this point, no new requests can enter the queue_rq()

and

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

drivers/block/virtio_blk.c

Quoted 2 lines
drivers/block/virtio_blk.c +2 -0
+	 * completion routine will not complete any new requests
+either for

the

drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device *vdev)

Quoted 5 lines
drivers/block/virtio_blk.c +5 -0
+	 * broken vq. Hence, it is safe to cancel all requests which are
+	 * started.
+	 */
+	blk_mq_tagset_busy_iter(&vblk->tag_set,
+				virtblk_complete_request_with_ioerr,

vblk);

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device *vdev)

Quoted 4 lines
drivers/block/virtio_blk.c +4 -0
+	blk_mq_tagset_wait_completed_request(&vblk->tag_set);
+
+	/*
+	 * All pending requests are cleaned up. Time to resume so

that disk

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

drivers/block/virtio_blk.c

Quoted 2 lines
drivers/block/virtio_blk.c +2 -0
+	 * deletion can be smooth. Start the HW queues so that when
+queue

is

drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device *vdev)

Quoted 6 lines
drivers/block/virtio_blk.c +6 -0
+	 * unquiesced requests can again enter the driver.
+	 */
+	blk_mq_start_stopped_hw_queues(q, true);
+
+	/*
+	 * Unquiescing will trigger dispatching any pending requests

to the

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device *vdev)

Quoted 6 lines
drivers/block/virtio_blk.c +6 -0
+	 * driver which has crossed bio_queue_enter() to the driver.
+	 */
+	blk_mq_unquiesce_queue(q);
+
+	/*
+	 * Wait for all pending dispatches to terminate which may

have been

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device *vdev)

Quoted 6 lines
drivers/block/virtio_blk.c +6 -0
+	 * initiated after unquiescing.
+	 */
+	blk_mq_freeze_queue_wait(q);
+
+	/*
+	 * Mark the disk dead so that once we unfreeze the queue,

requests

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

drivers/block/virtio_blk.c

Quoted 2 lines
drivers/block/virtio_blk.c +2 -0
+	 * waiting at the door of bio_queue_enter() can be aborted
+right

away.

drivers/block/virtio_blk.c static int virtblk_probe(struct virtio_device *vdev)

Quoted 4 lines
drivers/block/virtio_blk.c +4 -0
+	 */
+	blk_mark_disk_dead(vblk->disk);
+
+	/* Unfreeze the queue so that any waiting requests will be

aborted. */

Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
drivers/block/virtio_blk.c +3 -0
+	blk_mq_unfreeze_queue_nomemrestore(q);
+}
+
 static void virtblk_remove(struct virtio_device *vdev)  {
 	struct virtio_blk *vblk = vdev->priv; @@ -1561,6 +1653,9 @@

static void virtblk_remove(struct virtio_device *vdev)

drivers/block/virtio_blk.c +3 -1
 	/* Make sure no work handler is accessing the device. */
 	flush_work(&vblk->config_work);
 
+	if (virtqueue_is_broken(vblk->vqs[0].vq))
+		virtblk_cleanup_broken_device(vblk);
+
 	del_gendisk(vblk->disk);
 	blk_mq_free_tag_set(&vblk->tag_set);
 
--

2.34.1

#9 lore raw
Quoted 54 lines · Parav Pandit
Quoted 42 lines

From: Michael S. Tsirkin <[email protected]>
Sent: 25 June 2025 01:24 AM

Quoted 22 lines · Parav Pandit
Quoted 22 lines

From: Michael S. Tsirkin <[email protected]>
Sent: 25 June 2025 12:37 AM

Quoted 16 lines · Parav Pandit
Quoted 16 lines

From: Stefan Hajnoczi <[email protected]>
Sent: 25 June 2025 12:26 AM

Quoted 3 lines · Parav Pandit

When the PCI device is surprise removed, requests may not complete the device as the VQ is marked as broken. Due to this, the disk deletion hangs.

There are loops in the core virtio driver code that expect device register reads to eventually return 0:
drivers/virtio/virtio_pci_modern.c:vp_reset() drivers/virtio/virtio_pci_modern_dev.c:vp_modern_set_queue_reset ()

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Is there a hang if these loops are hit when a device has been surprise removed? I'm trying to understand whether surprise removal is fully supported or whether this patch is one step in that

direction.

Quoted 11 lines
Quoted 11 lines
Quoted 3 lines
Quoted 1 line

In one of the previous replies I answered to Michael, but don't have the link

handy.

Quoted 4 lines

It is not fully supported by this patch. It will hang.

This patch restores driver back to the same state what it was before the fixes

tag patch.

Quoted 2 lines

The virtio stack level work is needed to support surprise removal, including

Quoted 9 lines
Quoted 5 lines
Quoted 4 lines

the reset flow you rightly pointed.

Have plans to do that?

Didn't give enough thoughts on it yet.

This one is kind of pointless then? It just fixes the specific race window that your test harness happens to hit?

It was reported by Li from Baidu, whose tests failed. I missed to tag "reported-by" in v5. I had it until v4. :(

Quoted 8 lines
Quoted 3 lines

Maybe it's better to wait until someone does a comprehensive fix..

Oh, I was under impression is that you wanted to step forward in discussion of v4.
If you prefer a comprehensive support across layers of virtio, I suggest you should revert the cited patch in fixes tag.

Otherwise, it is in degraded state as virtio never supported surprise removal. By reverting the cited patch (or with this fix), the requests and disk deletion will not hang.

But they will hung in virtio core on reset, will they not? The tests just do not happen to trigger this?

Quoted 212 lines

Please let me know if I should re-send to revert the patch listed in fixes tag.

Quoted 210 lines
Quoted 89 lines
Quoted 89 lines
Quoted 89 lines
Quoted 3 lines

Apart from that, I'm happy with the virtio_blk.c aspects of the patch:

Thanks.

Quoted 84 lines
Quoted 70 lines

Fix it by aborting the requests when the VQ is broken.

With this fix now fio completes swiftly.
An alternative of IO timeout has been considered, however when the driver knows about unresponsive block device, swiftly clearing them enables users and upper layers to react quickly.

Verified with multiple device unplug iterations with pending requests in virtio used ring and some pending with the device.

Fixes 43bb40c5b926 ("virtio_pci: Support surprise removal of

virtio pci device")

Closes:
https://lore.kernel.org/virtualization/c45dd68698cd47238c55fb7
3ca9
b474
[email protected]/

---
v4->v5:
- fixed comment style where comment to start with one empty line at start
- Addressed comments from Alok
- fixed typo in broken vq check
v3->v4:
- Addressed comments from Michael
- renamed virtblk_request_cancel() to virtblk_complete_request_with_ioerr()
- Added comments for virtblk_complete_request_with_ioerr()
- Renamed virtblk_broken_device_cleanup() to virtblk_cleanup_broken_device()
- Added comments for virtblk_cleanup_broken_device()
- Moved the broken vq check in virtblk_remove()
- Fixed comment style to have first empty line
- replaced freezed to frozen
- Fixed comments rephrased

v2->v3:
- Addressed comments from Michael
- updated comment for synchronizing with callbacks

v1->v2:
- Addressed comments from Stephan
- fixed spelling to 'waiting'
- Addressed comments from Michael
- Dropped checking broken vq from queue_rq() and queue_rqs() because it is checked in lower layer routines in virtio core

v0->v1:
- Fixed comments from Stefan to rename a cleanup function
- Improved logic for handling any outstanding requests in bio layer
- improved cancel callback to sync with ongoing done()

---
 drivers/block/virtio_blk.c | 95
++++++++++++++++++++++++++++++++++++++
 1 file changed, 95 insertions(+)
(quoted diff) +0 -0

b/drivers/block/virtio_blk.c index 7cffea01d868..c5e383c0ac48 100644

drivers/block/virtio_blk.c +0 -0
@@ -1554,6 +1554,98 @@ static int virtblk_probe(struct
Quoted 1 line

virtio_device

*vdev)

Quoted 12 lines
drivers/block/virtio_blk.c +9 -0
 	return err;
 }
 
+/*
+ * If the vq is broken, device will not complete requests.
+ * So we do it for the device.
+ */
+static bool virtblk_complete_request_with_ioerr(struct
+request *rq, void *data) {
+	struct virtblk_req *vbr = blk_mq_rq_to_pdu(rq);
+	struct virtio_blk *vblk = data;
+	struct virtio_blk_vq *vq;
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

drivers/block/virtio_blk.c

Quoted 8 lines
drivers/block/virtio_blk.c +8 -0
+	unsigned long flags;
+
+	vq = &vblk->vqs[rq->mq_hctx->queue_num];
+
+	spin_lock_irqsave(&vq->lock, flags);
+
+	vbr->in_hdr.status = VIRTIO_BLK_S_IOERR;
+	if (blk_mq_request_started(rq) &&

!blk_mq_request_completed(rq))

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
drivers/block/virtio_blk.c +10 -0
+		blk_mq_complete_request(rq);
+
+	spin_unlock_irqrestore(&vq->lock, flags);
+	return true;
+}
+
+/*
+ * If the device is broken, it will not use any buffers and
+waiting
+ * for that to happen is pointless. We'll do the cleanup in
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

drivers/block/virtio_blk.c

Quoted 8 lines
drivers/block/virtio_blk.c +8 -0
+the driver,
+ * completing all requests for the device.
+ */
+static void virtblk_cleanup_broken_device(struct virtio_blk *vblk) {
+	struct request_queue *q = vblk->disk->queue;
+
+	/*
+	 * Start freezing the queue, so that new requests keeps

waiting at the

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

drivers/block/virtio_blk.c

Quoted 2 lines
drivers/block/virtio_blk.c +2 -0
+	 * door of bio_queue_enter(). We cannot fully freeze the
+queue

because

drivers/block/virtio_blk.c

Quoted 1 line
drivers/block/virtio_blk.c +1 -0
+	 * frozen queue is an empty queue and there are pending

requests, so

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

drivers/block/virtio_blk.c

Quoted 6 lines
drivers/block/virtio_blk.c +6 -0
+	 * only start freezing it.
+	 */
+	blk_freeze_queue_start(q);
+
+	/*
+	 * When quiescing completes, all ongoing dispatches have

completed

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

drivers/block/virtio_blk.c

Quoted 6 lines
drivers/block/virtio_blk.c +6 -0
+	 * and no new dispatch will happen towards the driver.
+	 */
+	blk_mq_quiesce_queue(q);
+
+	/*
+	 * Synchronize with any ongoing VQ callbacks that may have

started

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

drivers/block/virtio_blk.c

Quoted 1 line
drivers/block/virtio_blk.c +1 -0
+	 * before the VQs were marked as broken. Any outstanding

requests

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

drivers/block/virtio_blk.c

Quoted 1 line
drivers/block/virtio_blk.c +1 -0
+	 * will be completed by

virtblk_complete_request_with_ioerr().

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

drivers/block/virtio_blk.c

Quoted 5 lines
drivers/block/virtio_blk.c +5 -0
+	 */
+	virtio_synchronize_cbs(vblk->vdev);
+
+	/*
+	 * At this point, no new requests can enter the queue_rq()

and

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

drivers/block/virtio_blk.c

Quoted 2 lines
drivers/block/virtio_blk.c +2 -0
+	 * completion routine will not complete any new requests
+either for

the

drivers/block/virtio_blk.c

Quoted 5 lines
drivers/block/virtio_blk.c +5 -0
+	 * broken vq. Hence, it is safe to cancel all requests which are
+	 * started.
+	 */
+	blk_mq_tagset_busy_iter(&vblk->tag_set,
+				virtblk_complete_request_with_ioerr,

vblk);

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

drivers/block/virtio_blk.c

Quoted 4 lines
drivers/block/virtio_blk.c +4 -0
+	blk_mq_tagset_wait_completed_request(&vblk->tag_set);
+
+	/*
+	 * All pending requests are cleaned up. Time to resume so

that disk

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

drivers/block/virtio_blk.c

Quoted 2 lines
drivers/block/virtio_blk.c +2 -0
+	 * deletion can be smooth. Start the HW queues so that when
+queue

is

drivers/block/virtio_blk.c

Quoted 6 lines
drivers/block/virtio_blk.c +6 -0
+	 * unquiesced requests can again enter the driver.
+	 */
+	blk_mq_start_stopped_hw_queues(q, true);
+
+	/*
+	 * Unquiescing will trigger dispatching any pending requests

to the

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

drivers/block/virtio_blk.c

Quoted 6 lines
drivers/block/virtio_blk.c +6 -0
+	 * driver which has crossed bio_queue_enter() to the driver.
+	 */
+	blk_mq_unquiesce_queue(q);
+
+	/*
+	 * Wait for all pending dispatches to terminate which may

have been

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

drivers/block/virtio_blk.c

Quoted 6 lines
drivers/block/virtio_blk.c +6 -0
+	 * initiated after unquiescing.
+	 */
+	blk_mq_freeze_queue_wait(q);
+
+	/*
+	 * Mark the disk dead so that once we unfreeze the queue,

requests

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

drivers/block/virtio_blk.c

Quoted 2 lines
drivers/block/virtio_blk.c +2 -0
+	 * waiting at the door of bio_queue_enter() can be aborted
+right

away.

drivers/block/virtio_blk.c

Quoted 4 lines
drivers/block/virtio_blk.c +4 -0
+	 */
+	blk_mark_disk_dead(vblk->disk);
+
+	/* Unfreeze the queue so that any waiting requests will be

aborted. */

Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
drivers/block/virtio_blk.c +3 -0
+	blk_mq_unfreeze_queue_nomemrestore(q);
+}
+
 static void virtblk_remove(struct virtio_device *vdev)  {
 	struct virtio_blk *vblk = vdev->priv; @@ -1561,6 +1653,9 @@

static void virtblk_remove(struct virtio_device *vdev)

drivers/block/virtio_blk.c +3 -1
 	/* Make sure no work handler is accessing the device. */
 	flush_work(&vblk->config_work);
 
+	if (virtqueue_is_broken(vblk->vqs[0].vq))
+		virtblk_cleanup_broken_device(vblk);
+
 	del_gendisk(vblk->disk);
 	blk_mq_free_tag_set(&vblk->tag_set);
 
--

2.34.1

#10 lore raw
Quoted 67 lines

From: Michael S. Tsirkin <[email protected]>
Sent: 25 June 2025 04:34 PM

Quoted 53 lines · Parav Pandit
Quoted 44 lines

From: Michael S. Tsirkin <[email protected]>
Sent: 25 June 2025 01:24 AM

Quoted 23 lines · Parav Pandit
Quoted 23 lines

From: Michael S. Tsirkin <[email protected]>
Sent: 25 June 2025 12:37 AM

Quoted 17 lines · Parav Pandit
Quoted 17 lines

From: Stefan Hajnoczi <[email protected]>
Sent: 25 June 2025 12:26 AM

Quoted 3 lines · Parav Pandit

When the PCI device is surprise removed, requests may not complete the device as the VQ is marked as broken. Due to this, the disk deletion hangs.

There are loops in the core virtio driver code that expect device register reads to eventually return 0:
drivers/virtio/virtio_pci_modern.c:vp_reset() drivers/virtio/virtio_pci_modern_dev.c:vp_modern_set_queue_r eset
()

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Is there a hang if these loops are hit when a device has been surprise removed? I'm trying to understand whether surprise removal is fully supported or whether this patch is one step in that

direction.

Quoted 11 lines
Quoted 11 lines
Quoted 3 lines
Quoted 1 line

In one of the previous replies I answered to Michael, but don't have the link

handy.

Quoted 4 lines

It is not fully supported by this patch. It will hang.

This patch restores driver back to the same state what it was before the fixes

tag patch.

Quoted 2 lines

The virtio stack level work is needed to support surprise removal, including

Quoted 9 lines
Quoted 5 lines
Quoted 4 lines

the reset flow you rightly pointed.

Have plans to do that?

Didn't give enough thoughts on it yet.

This one is kind of pointless then? It just fixes the specific race window that your test harness happens to hit?

Quoted 7 lines

It was reported by Li from Baidu, whose tests failed. I missed to tag "reported-by" in v5. I had it until v4. :(

Quoted 3 lines

Maybe it's better to wait until someone does a comprehensive fix..

Oh, I was under impression is that you wanted to step forward in discussion

of v4.

Quoted 1 line

If you prefer a comprehensive support across layers of virtio, I suggest you

Quoted 9 lines

should revert the cited patch in fixes tag.

Quoted 3 lines

Otherwise, it is in degraded state as virtio never supported surprise removal. By reverting the cited patch (or with this fix), the requests and disk deletion

will not hang.

But they will hung in virtio core on reset, will they not? The tests just do not happen to trigger this?

It will hang if it a true surprise removal which no device did so far because it never worked. (or did, but always hung that no one reported yet)

I am familiar with 2 or more PCI devices who reports surprise removal, which do not complete the requests but yet allows device reset flow. This is because device is still there on the PCI bus. Only via side band signals device removal was reported.

But I agree that for full support, virtio all layer changes would be needed as new functionality (without fixes tag :) ).

Quoted 226 lines
Quoted 3 lines

Please let me know if I should re-send to revert the patch listed in fixes tag.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

Apart from that, I'm happy with the virtio_blk.c aspects of the

patch:

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 2 lines

Thanks.

Quoted 7 lines
Quoted 7 lines

Fix it by aborting the requests when the VQ is broken.

With this fix now fio completes swiftly.
An alternative of IO timeout has been considered, however when the driver knows about unresponsive block device, swiftly clearing them enables users and upper layers to react

quickly.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Verified with multiple device unplug iterations with pending requests in virtio used ring and some pending with the

device.

Quoted 86 lines
Quoted 86 lines
Quoted 77 lines
Quoted 77 lines
Quoted 77 lines
Quoted 77 lines
Quoted 63 lines
Fixes 43bb40c5b926 ("virtio_pci: Support surprise removal

of virtio pci device")

Closes:
https://lore.kernel.org/virtualization/c45dd68698cd47238c5
5fb7
3ca9
b474
[email protected]/

---
v4->v5:
- fixed comment style where comment to start with one empty line at start
- Addressed comments from Alok
- fixed typo in broken vq check
v3->v4:
- Addressed comments from Michael
- renamed virtblk_request_cancel() to virtblk_complete_request_with_ioerr()
- Added comments for virtblk_complete_request_with_ioerr()
- Renamed virtblk_broken_device_cleanup() to virtblk_cleanup_broken_device()
- Added comments for virtblk_cleanup_broken_device()
- Moved the broken vq check in virtblk_remove()
- Fixed comment style to have first empty line
- replaced freezed to frozen
- Fixed comments rephrased

v2->v3:
- Addressed comments from Michael
- updated comment for synchronizing with callbacks

v1->v2:
- Addressed comments from Stephan
- fixed spelling to 'waiting'
- Addressed comments from Michael
- Dropped checking broken vq from queue_rq() and queue_rqs() because it is checked in lower layer routines in virtio
core

v0->v1:
- Fixed comments from Stefan to rename a cleanup function
- Improved logic for handling any outstanding requests in bio layer
- improved cancel callback to sync with ongoing done()

---
 drivers/block/virtio_blk.c | 95
++++++++++++++++++++++++++++++++++++++
 1 file changed, 95 insertions(+)
(quoted diff) +0 -0

b/drivers/block/virtio_blk.c index
7cffea01d868..c5e383c0ac48
100644

drivers/block/virtio_blk.c +0 -0
@@ -1554,6 +1554,98 @@ static int virtblk_probe(struct
Quoted 1 line

virtio_device

*vdev)

Quoted 12 lines
drivers/block/virtio_blk.c +9 -0
 	return err;
 }
 
+/*
+ * If the vq is broken, device will not complete requests.
+ * So we do it for the device.
+ */
+static bool virtblk_complete_request_with_ioerr(struct
+request *rq, void *data) {
+	struct virtblk_req *vbr = blk_mq_rq_to_pdu(rq);
+	struct virtio_blk *vblk = data;
+	struct virtio_blk_vq *vq;
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

drivers/block/virtio_blk.c

Quoted 8 lines
drivers/block/virtio_blk.c +8 -0
+	unsigned long flags;
+
+	vq = &vblk->vqs[rq->mq_hctx->queue_num];
+
+	spin_lock_irqsave(&vq->lock, flags);
+
+	vbr->in_hdr.status = VIRTIO_BLK_S_IOERR;
+	if (blk_mq_request_started(rq) &&

!blk_mq_request_completed(rq))

Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
drivers/block/virtio_blk.c +6 -0
+		blk_mq_complete_request(rq);
+
+	spin_unlock_irqrestore(&vq->lock, flags);
+	return true;
+}
+
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

drivers/block/virtio_blk.c

Quoted 8 lines
drivers/block/virtio_blk.c +8 -0
+/*
+ * If the device is broken, it will not use any buffers
+and waiting
+ * for that to happen is pointless. We'll do the cleanup
+in the driver,
+ * completing all requests for the device.
+ */
+static void virtblk_cleanup_broken_device(struct virtio_blk *vblk)

{

Quoted 105 lines
Quoted 105 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

drivers/block/virtio_blk.c

Quoted 4 lines
drivers/block/virtio_blk.c +4 -0
+	struct request_queue *q = vblk->disk->queue;
+
+	/*
+	 * Start freezing the queue, so that new requests keeps

waiting at the

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

drivers/block/virtio_blk.c

Quoted 2 lines
drivers/block/virtio_blk.c +2 -0
+	 * door of bio_queue_enter(). We cannot fully freeze the
+queue

because

drivers/block/virtio_blk.c

Quoted 1 line
drivers/block/virtio_blk.c +1 -0
+	 * frozen queue is an empty queue and there are pending

requests, so

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

drivers/block/virtio_blk.c

Quoted 6 lines
drivers/block/virtio_blk.c +6 -0
+	 * only start freezing it.
+	 */
+	blk_freeze_queue_start(q);
+
+	/*
+	 * When quiescing completes, all ongoing dispatches have

completed

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

drivers/block/virtio_blk.c

Quoted 7 lines
drivers/block/virtio_blk.c +7 -0
+	 * and no new dispatch will happen towards the driver.
+	 */
+	blk_mq_quiesce_queue(q);
+
+	/*
+	 * Synchronize with any ongoing VQ callbacks that may
+have

started

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

drivers/block/virtio_blk.c

Quoted 1 line
drivers/block/virtio_blk.c +1 -0
+	 * before the VQs were marked as broken. Any outstanding

requests

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

drivers/block/virtio_blk.c

Quoted 1 line
drivers/block/virtio_blk.c +1 -0
+	 * will be completed by

virtblk_complete_request_with_ioerr().

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

drivers/block/virtio_blk.c

Quoted 6 lines
drivers/block/virtio_blk.c +6 -0
+	 */
+	virtio_synchronize_cbs(vblk->vdev);
+
+	/*
+	 * At this point, no new requests can enter the
+queue_rq()

and

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

drivers/block/virtio_blk.c

Quoted 2 lines
drivers/block/virtio_blk.c +2 -0
+	 * completion routine will not complete any new requests
+either for

the

drivers/block/virtio_blk.c

Quoted 5 lines
drivers/block/virtio_blk.c +5 -0
+	 * broken vq. Hence, it is safe to cancel all requests which are
+	 * started.
+	 */
+	blk_mq_tagset_busy_iter(&vblk->tag_set,
+				virtblk_complete_request_with_ioerr,

vblk);

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

drivers/block/virtio_blk.c

Quoted 5 lines
drivers/block/virtio_blk.c +5 -0
+	blk_mq_tagset_wait_completed_request(&vblk->tag_set);
+
+	/*
+	 * All pending requests are cleaned up. Time to resume
+so

that disk

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

drivers/block/virtio_blk.c

Quoted 2 lines
drivers/block/virtio_blk.c +2 -0
+	 * deletion can be smooth. Start the HW queues so that
+when queue

is

drivers/block/virtio_blk.c

Quoted 7 lines
drivers/block/virtio_blk.c +7 -0
+	 * unquiesced requests can again enter the driver.
+	 */
+	blk_mq_start_stopped_hw_queues(q, true);
+
+	/*
+	 * Unquiescing will trigger dispatching any pending
+requests

to the

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

drivers/block/virtio_blk.c

Quoted 7 lines
drivers/block/virtio_blk.c +7 -0
+	 * driver which has crossed bio_queue_enter() to the driver.
+	 */
+	blk_mq_unquiesce_queue(q);
+
+	/*
+	 * Wait for all pending dispatches to terminate which
+may

have been

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

drivers/block/virtio_blk.c

Quoted 7 lines
drivers/block/virtio_blk.c +7 -0
+	 * initiated after unquiescing.
+	 */
+	blk_mq_freeze_queue_wait(q);
+
+	/*
+	 * Mark the disk dead so that once we unfreeze the
+queue,

requests

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

drivers/block/virtio_blk.c

Quoted 2 lines
drivers/block/virtio_blk.c +2 -0
+	 * waiting at the door of bio_queue_enter() can be
+aborted right

away.

drivers/block/virtio_blk.c

Quoted 5 lines
drivers/block/virtio_blk.c +5 -0
+	 */
+	blk_mark_disk_dead(vblk->disk);
+
+	/* Unfreeze the queue so that any waiting requests will
+be

aborted. */

Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
drivers/block/virtio_blk.c +3 -0
+	blk_mq_unfreeze_queue_nomemrestore(q);
+}
+
 static void virtblk_remove(struct virtio_device *vdev)  {
 	struct virtio_blk *vblk = vdev->priv; @@ -1561,6 +1653,9

@@ static void virtblk_remove(struct virtio_device *vdev)

drivers/block/virtio_blk.c +3 -1
 	/* Make sure no work handler is accessing the device. */
 	flush_work(&vblk->config_work);
 
+	if (virtqueue_is_broken(vblk->vqs[0].vq))
+		virtblk_cleanup_broken_device(vblk);
+
 	del_gendisk(vblk->disk);
 	blk_mq_free_tag_set(&vblk->tag_set);
 
--

2.34.1

#11 lore raw
Quoted 73 lines · Parav Pandit
Quoted 73 lines

From: Michael S. Tsirkin <[email protected]>
Sent: 25 June 2025 04:34 PM

Quoted 53 lines · Parav Pandit
Quoted 44 lines

From: Michael S. Tsirkin <[email protected]>
Sent: 25 June 2025 01:24 AM

Quoted 23 lines · Parav Pandit
Quoted 23 lines

From: Michael S. Tsirkin <[email protected]>
Sent: 25 June 2025 12:37 AM

Quoted 17 lines · Parav Pandit
Quoted 17 lines

From: Stefan Hajnoczi <[email protected]>
Sent: 25 June 2025 12:26 AM

Quoted 3 lines · Parav Pandit

When the PCI device is surprise removed, requests may not complete the device as the VQ is marked as broken. Due to this, the disk deletion hangs.

There are loops in the core virtio driver code that expect device register reads to eventually return 0:
drivers/virtio/virtio_pci_modern.c:vp_reset() drivers/virtio/virtio_pci_modern_dev.c:vp_modern_set_queue_r eset
()

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Is there a hang if these loops are hit when a device has been surprise removed? I'm trying to understand whether surprise removal is fully supported or whether this patch is one step in that

direction.

Quoted 11 lines
Quoted 11 lines
Quoted 3 lines
Quoted 1 line

In one of the previous replies I answered to Michael, but don't have the link

handy.

Quoted 4 lines

It is not fully supported by this patch. It will hang.

This patch restores driver back to the same state what it was before the fixes

tag patch.

Quoted 2 lines

The virtio stack level work is needed to support surprise removal, including

Quoted 9 lines
Quoted 5 lines
Quoted 4 lines

the reset flow you rightly pointed.

Have plans to do that?

Didn't give enough thoughts on it yet.

This one is kind of pointless then? It just fixes the specific race window that your test harness happens to hit?

Quoted 7 lines

It was reported by Li from Baidu, whose tests failed. I missed to tag "reported-by" in v5. I had it until v4. :(

Quoted 3 lines

Maybe it's better to wait until someone does a comprehensive fix..

Oh, I was under impression is that you wanted to step forward in discussion

of v4.

Quoted 1 line

If you prefer a comprehensive support across layers of virtio, I suggest you

should revert the cited patch in fixes tag.

Quoted 3 lines

Otherwise, it is in degraded state as virtio never supported surprise removal. By reverting the cited patch (or with this fix), the requests and disk deletion

will not hang.

Quoted 8 lines
Quoted 3 lines

But they will hung in virtio core on reset, will they not? The tests just do not happen to trigger this?

It will hang if it a true surprise removal which no device did so far because it never worked. (or did, but always hung that no one reported yet)

I am familiar with 2 or more PCI devices who reports surprise removal, which do not complete the requests but yet allows device reset flow. This is because device is still there on the PCI bus. Only via side band signals device removal was reported.

So why do we care about it so much? I think it's great this patch exists, for example it makes it easier to test surprise removal and find more bugs. But is it better to just have it hang unconditionally? Are we now making a commitment that it's working - one we don't seem to intend to implement?

Quoted 1 line

But I agree that for full support, virtio all layer changes would be needed as new functionality (without fixes tag :) ).

Or with a fixes tag - lots of people just use it as a signal to mean "where can this be reasonably backported to".

Quoted 226 lines
Quoted 226 lines
Quoted 3 lines

Please let me know if I should re-send to revert the patch listed in fixes tag.

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

Apart from that, I'm happy with the virtio_blk.c aspects of the

patch:

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 2 lines

Thanks.

Quoted 7 lines
Quoted 7 lines

Fix it by aborting the requests when the VQ is broken.

With this fix now fio completes swiftly.
An alternative of IO timeout has been considered, however when the driver knows about unresponsive block device, swiftly clearing them enables users and upper layers to react

quickly.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Verified with multiple device unplug iterations with pending requests in virtio used ring and some pending with the

device.

Quoted 86 lines
Quoted 86 lines
Quoted 77 lines
Quoted 77 lines
Quoted 77 lines
Quoted 77 lines
Quoted 63 lines
Fixes 43bb40c5b926 ("virtio_pci: Support surprise removal

of virtio pci device")

Closes:
https://lore.kernel.org/virtualization/c45dd68698cd47238c5
5fb7
3ca9
b474
[email protected]/

---
v4->v5:
- fixed comment style where comment to start with one empty line at start
- Addressed comments from Alok
- fixed typo in broken vq check
v3->v4:
- Addressed comments from Michael
- renamed virtblk_request_cancel() to virtblk_complete_request_with_ioerr()
- Added comments for virtblk_complete_request_with_ioerr()
- Renamed virtblk_broken_device_cleanup() to virtblk_cleanup_broken_device()
- Added comments for virtblk_cleanup_broken_device()
- Moved the broken vq check in virtblk_remove()
- Fixed comment style to have first empty line
- replaced freezed to frozen
- Fixed comments rephrased

v2->v3:
- Addressed comments from Michael
- updated comment for synchronizing with callbacks

v1->v2:
- Addressed comments from Stephan
- fixed spelling to 'waiting'
- Addressed comments from Michael
- Dropped checking broken vq from queue_rq() and queue_rqs() because it is checked in lower layer routines in virtio
core

v0->v1:
- Fixed comments from Stefan to rename a cleanup function
- Improved logic for handling any outstanding requests in bio layer
- improved cancel callback to sync with ongoing done()

---
 drivers/block/virtio_blk.c | 95
++++++++++++++++++++++++++++++++++++++
 1 file changed, 95 insertions(+)
(quoted diff) +0 -0

b/drivers/block/virtio_blk.c index
7cffea01d868..c5e383c0ac48
100644

drivers/block/virtio_blk.c +0 -0
@@ -1554,6 +1554,98 @@ static int virtblk_probe(struct
Quoted 1 line

virtio_device

*vdev)

Quoted 12 lines
drivers/block/virtio_blk.c +9 -0
 	return err;
 }
 
+/*
+ * If the vq is broken, device will not complete requests.
+ * So we do it for the device.
+ */
+static bool virtblk_complete_request_with_ioerr(struct
+request *rq, void *data) {
+	struct virtblk_req *vbr = blk_mq_rq_to_pdu(rq);
+	struct virtio_blk *vblk = data;
+	struct virtio_blk_vq *vq;
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

drivers/block/virtio_blk.c

Quoted 8 lines
drivers/block/virtio_blk.c +8 -0
+	unsigned long flags;
+
+	vq = &vblk->vqs[rq->mq_hctx->queue_num];
+
+	spin_lock_irqsave(&vq->lock, flags);
+
+	vbr->in_hdr.status = VIRTIO_BLK_S_IOERR;
+	if (blk_mq_request_started(rq) &&

!blk_mq_request_completed(rq))

Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
drivers/block/virtio_blk.c +6 -0
+		blk_mq_complete_request(rq);
+
+	spin_unlock_irqrestore(&vq->lock, flags);
+	return true;
+}
+
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

drivers/block/virtio_blk.c

Quoted 8 lines
drivers/block/virtio_blk.c +8 -0
+/*
+ * If the device is broken, it will not use any buffers
+and waiting
+ * for that to happen is pointless. We'll do the cleanup
+in the driver,
+ * completing all requests for the device.
+ */
+static void virtblk_cleanup_broken_device(struct virtio_blk *vblk)

{

Quoted 105 lines
Quoted 105 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

drivers/block/virtio_blk.c

Quoted 4 lines
drivers/block/virtio_blk.c +4 -0
+	struct request_queue *q = vblk->disk->queue;
+
+	/*
+	 * Start freezing the queue, so that new requests keeps

waiting at the

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

drivers/block/virtio_blk.c

Quoted 2 lines
drivers/block/virtio_blk.c +2 -0
+	 * door of bio_queue_enter(). We cannot fully freeze the
+queue

because

drivers/block/virtio_blk.c

Quoted 1 line
drivers/block/virtio_blk.c +1 -0
+	 * frozen queue is an empty queue and there are pending

requests, so

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

drivers/block/virtio_blk.c

Quoted 6 lines
drivers/block/virtio_blk.c +6 -0
+	 * only start freezing it.
+	 */
+	blk_freeze_queue_start(q);
+
+	/*
+	 * When quiescing completes, all ongoing dispatches have

completed

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

drivers/block/virtio_blk.c

Quoted 7 lines
drivers/block/virtio_blk.c +7 -0
+	 * and no new dispatch will happen towards the driver.
+	 */
+	blk_mq_quiesce_queue(q);
+
+	/*
+	 * Synchronize with any ongoing VQ callbacks that may
+have

started

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

drivers/block/virtio_blk.c

Quoted 1 line
drivers/block/virtio_blk.c +1 -0
+	 * before the VQs were marked as broken. Any outstanding

requests

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

drivers/block/virtio_blk.c

Quoted 1 line
drivers/block/virtio_blk.c +1 -0
+	 * will be completed by

virtblk_complete_request_with_ioerr().

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

drivers/block/virtio_blk.c

Quoted 6 lines
drivers/block/virtio_blk.c +6 -0
+	 */
+	virtio_synchronize_cbs(vblk->vdev);
+
+	/*
+	 * At this point, no new requests can enter the
+queue_rq()

and

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

drivers/block/virtio_blk.c

Quoted 2 lines
drivers/block/virtio_blk.c +2 -0
+	 * completion routine will not complete any new requests
+either for

the

drivers/block/virtio_blk.c

Quoted 5 lines
drivers/block/virtio_blk.c +5 -0
+	 * broken vq. Hence, it is safe to cancel all requests which are
+	 * started.
+	 */
+	blk_mq_tagset_busy_iter(&vblk->tag_set,
+				virtblk_complete_request_with_ioerr,

vblk);

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

drivers/block/virtio_blk.c

Quoted 5 lines
drivers/block/virtio_blk.c +5 -0
+	blk_mq_tagset_wait_completed_request(&vblk->tag_set);
+
+	/*
+	 * All pending requests are cleaned up. Time to resume
+so

that disk

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

drivers/block/virtio_blk.c

Quoted 2 lines
drivers/block/virtio_blk.c +2 -0
+	 * deletion can be smooth. Start the HW queues so that
+when queue

is

drivers/block/virtio_blk.c

Quoted 7 lines
drivers/block/virtio_blk.c +7 -0
+	 * unquiesced requests can again enter the driver.
+	 */
+	blk_mq_start_stopped_hw_queues(q, true);
+
+	/*
+	 * Unquiescing will trigger dispatching any pending
+requests

to the

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

drivers/block/virtio_blk.c

Quoted 7 lines
drivers/block/virtio_blk.c +7 -0
+	 * driver which has crossed bio_queue_enter() to the driver.
+	 */
+	blk_mq_unquiesce_queue(q);
+
+	/*
+	 * Wait for all pending dispatches to terminate which
+may

have been

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

drivers/block/virtio_blk.c

Quoted 7 lines
drivers/block/virtio_blk.c +7 -0
+	 * initiated after unquiescing.
+	 */
+	blk_mq_freeze_queue_wait(q);
+
+	/*
+	 * Mark the disk dead so that once we unfreeze the
+queue,

requests

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

drivers/block/virtio_blk.c

Quoted 2 lines
drivers/block/virtio_blk.c +2 -0
+	 * waiting at the door of bio_queue_enter() can be
+aborted right

away.

drivers/block/virtio_blk.c

Quoted 5 lines
drivers/block/virtio_blk.c +5 -0
+	 */
+	blk_mark_disk_dead(vblk->disk);
+
+	/* Unfreeze the queue so that any waiting requests will
+be

aborted. */

Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
drivers/block/virtio_blk.c +3 -0
+	blk_mq_unfreeze_queue_nomemrestore(q);
+}
+
 static void virtblk_remove(struct virtio_device *vdev)  {
 	struct virtio_blk *vblk = vdev->priv; @@ -1561,6 +1653,9

@@ static void virtblk_remove(struct virtio_device *vdev)

drivers/block/virtio_blk.c +3 -1
 	/* Make sure no work handler is accessing the device. */
 	flush_work(&vblk->config_work);
 
+	if (virtqueue_is_broken(vblk->vqs[0].vq))
+		virtblk_cleanup_broken_device(vblk);
+
 	del_gendisk(vblk->disk);
 	blk_mq_free_tag_set(&vblk->tag_set);
 
--

2.34.1

#12 lore raw
Quoted 92 lines

From: Michael S. Tsirkin <[email protected]>
Sent: 25 June 2025 05:15 PM

Quoted 22 lines · Parav Pandit
Quoted 22 lines

From: Michael S. Tsirkin <[email protected]>
Sent: 25 June 2025 04:34 PM

Quoted 17 lines · Parav Pandit
Quoted 17 lines

From: Michael S. Tsirkin <[email protected]>
Sent: 25 June 2025 01:24 AM

Quoted 12 lines · Parav Pandit
Quoted 12 lines

From: Michael S. Tsirkin <[email protected]>
Sent: 25 June 2025 12:37 AM

Quoted 6 lines · Parav Pandit
Quoted 6 lines

From: Stefan Hajnoczi <[email protected]>
Sent: 25 June 2025 12:26 AM

On Mon, Jun 02, 2025 at 02:44:33AM +0000, Parav Pandit

Quoted 43 lines · wrote:
Quoted 43 lines
Quoted 43 lines
Quoted 28 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 3 lines

When the PCI device is surprise removed, requests may not complete the device as the VQ is marked as broken. Due to this, the disk deletion hangs.

There are loops in the core virtio driver code that expect device register reads to eventually return 0:
drivers/virtio/virtio_pci_modern.c:vp_reset() drivers/virtio/virtio_pci_modern_dev.c:vp_modern_set_que ue_r
eset
()

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Is there a hang if these loops are hit when a device has been surprise removed? I'm trying to understand whether surprise removal is fully supported or whether this patch is one step in that

direction.

Quoted 11 lines
Quoted 11 lines
Quoted 3 lines
Quoted 1 line

In one of the previous replies I answered to Michael, but don't have the link

handy.

Quoted 4 lines

It is not fully supported by this patch. It will hang.

This patch restores driver back to the same state what it was before the fixes

tag patch.

Quoted 2 lines

The virtio stack level work is needed to support surprise removal, including

Quoted 9 lines
Quoted 5 lines
Quoted 4 lines

the reset flow you rightly pointed.

Have plans to do that?

Didn't give enough thoughts on it yet.

This one is kind of pointless then? It just fixes the specific race window that your test harness happens to hit?

It was reported by Li from Baidu, whose tests failed. I missed to tag "reported-by" in v5. I had it until v4. :(

Quoted 3 lines

Maybe it's better to wait until someone does a comprehensive fix..

Quoted 8 lines
Quoted 8 lines
Quoted 2 lines

Oh, I was under impression is that you wanted to step forward in discussion

of v4.

Quoted 2 lines

If you prefer a comprehensive support across layers of virtio, I suggest you

should revert the cited patch in fixes tag.

Quoted 2 lines

Otherwise, it is in degraded state as virtio never supported surprise

removal.

Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

By reverting the cited patch (or with this fix), the requests and disk deletion

will not hang.

But they will hung in virtio core on reset, will they not? The tests just do not happen to trigger this?

It will hang if it a true surprise removal which no device did so far because it

never worked.

Quoted 3 lines

(or did, but always hung that no one reported yet)

I am familiar with 2 or more PCI devices who reports surprise removal,

Quoted 9 lines

which do not complete the requests but yet allows device reset flow.

Quoted 1 line

This is because device is still there on the PCI bus. Only via side band signals

device removal was reported.

So why do we care about it so much? I think it's great this patch exists, for example it makes it easier to test surprise removal and find more bugs. But is it better to just have it hang unconditionally? Are we now making a commitment that it's working - one we don't seem to intend to implement?

The patch improves the situation from its current state. But as you posted, more changes in pci layer are needed. I didn't audit where else it may be needed.

vp_reset() may need to return the status back of successful/failure reset.
Otherwise during probe(), vp_reset() aborts the reset and attempts to load the driver for removed device.

I guess suspend() callback also infinitely waits during freezing the queue also needs adaptation.

Quoted 7 lines
Quoted 1 line

But I agree that for full support, virtio all layer changes would be needed as

new functionality (without fixes tag :) ).

Or with a fixes tag - lots of people just use it as a signal to mean "where can this be reasonably backported to".

Yes, I think the fix for the older kernels is needed, hence I cced stable too.

Quoted 248 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Please let me know if I should re-send to revert the patch listed in fixes

tag.

Quoted 62 lines
Quoted 62 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Apart from that, I'm happy with the virtio_blk.c aspects of the

patch:

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

Thanks.

Quoted 1 line
Quoted 1 line
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Fix it by aborting the requests when the VQ is broken.

With this fix now fio completes swiftly.
An alternative of IO timeout has been considered, however when the driver knows about unresponsive block device, swiftly clearing them enables users and upper layers to react

quickly.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Verified with multiple device unplug iterations with pending requests in virtio used ring and some pending with the

device.

Quoted 40 lines
Quoted 40 lines
Quoted 40 lines
Quoted 40 lines
Quoted 40 lines
Quoted 40 lines
Quoted 40 lines
Fixes 43bb40c5b926 ("virtio_pci: Support surprise

removal of virtio pci device")

Closes:
https://lore.kernel.org/virtualization/c45dd68698cd472
38c5
5fb7
3ca9
b474
[email protected]/

---
v4->v5:
- fixed comment style where comment to start with one empty line at start
- Addressed comments from Alok
- fixed typo in broken vq check
v3->v4:
- Addressed comments from Michael
- renamed virtblk_request_cancel() to virtblk_complete_request_with_ioerr()
- Added comments for virtblk_complete_request_with_ioerr()
- Renamed virtblk_broken_device_cleanup() to virtblk_cleanup_broken_device()
- Added comments for virtblk_cleanup_broken_device()
- Moved the broken vq check in virtblk_remove()
- Fixed comment style to have first empty line
- replaced freezed to frozen
- Fixed comments rephrased

v2->v3:
- Addressed comments from Michael
- updated comment for synchronizing with callbacks

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

v1->v2:
- Addressed comments from Stephan
- fixed spelling to 'waiting'
- Addressed comments from Michael
- Dropped checking broken vq from queue_rq() and

queue_rqs()

Quoted 81 lines
Quoted 81 lines
Quoted 45 lines
Quoted 45 lines
Quoted 36 lines
Quoted 36 lines
Quoted 36 lines
Quoted 36 lines
Quoted 21 lines

because it is checked in lower layer routines in
virtio core

v0->v1:
- Fixed comments from Stefan to rename a cleanup function
- Improved logic for handling any outstanding requests in bio layer
- improved cancel callback to sync with ongoing done()

---
 drivers/block/virtio_blk.c | 95
++++++++++++++++++++++++++++++++++++++
 1 file changed, 95 insertions(+)
(quoted diff) +0 -0

b/drivers/block/virtio_blk.c index
7cffea01d868..c5e383c0ac48
100644

drivers/block/virtio_blk.c +0 -0
@@ -1554,6 +1554,98 @@ static int virtblk_probe(struct
Quoted 1 line

virtio_device

*vdev)

Quoted 13 lines
drivers/block/virtio_blk.c +10 -0
 	return err;
 }
 
+/*
+ * If the vq is broken, device will not complete requests.
+ * So we do it for the device.
+ */
+static bool
+virtblk_complete_request_with_ioerr(struct
+request *rq, void *data) {
+	struct virtblk_req *vbr = blk_mq_rq_to_pdu(rq);
+	struct virtio_blk *vblk = data;
+	struct virtio_blk_vq *vq;
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

drivers/block/virtio_blk.c

Quoted 8 lines
drivers/block/virtio_blk.c +8 -0
+	unsigned long flags;
+
+	vq = &vblk->vqs[rq->mq_hctx->queue_num];
+
+	spin_lock_irqsave(&vq->lock, flags);
+
+	vbr->in_hdr.status = VIRTIO_BLK_S_IOERR;
+	if (blk_mq_request_started(rq) &&

!blk_mq_request_completed(rq))

Quoted 15 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
drivers/block/virtio_blk.c +7 -0
+		blk_mq_complete_request(rq);
+
+	spin_unlock_irqrestore(&vq->lock, flags);
+	return true;
+}
+
+/*
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

drivers/block/virtio_blk.c

Quoted 8 lines

+ * If the device is broken, it will not use any

drivers/block/virtio_blk.c +7 -0
+buffers and waiting
+ * for that to happen is pointless. We'll do the
+cleanup in the driver,
+ * completing all requests for the device.
+ */
+static void virtblk_cleanup_broken_device(struct
+virtio_blk *vblk)

{

Quoted 20 lines
Quoted 20 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

drivers/block/virtio_blk.c

Quoted 5 lines
drivers/block/virtio_blk.c +5 -0
+	struct request_queue *q = vblk->disk->queue;
+
+	/*
+	 * Start freezing the queue, so that new requests
+keeps

waiting at the

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

drivers/block/virtio_blk.c

Quoted 2 lines
drivers/block/virtio_blk.c +2 -0
+	 * door of bio_queue_enter(). We cannot fully freeze
+the queue

because

drivers/block/virtio_blk.c

Quoted 2 lines
drivers/block/virtio_blk.c +2 -0
+	 * frozen queue is an empty queue and there are
+pending

requests, so

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

drivers/block/virtio_blk.c

Quoted 7 lines
drivers/block/virtio_blk.c +7 -0
+	 * only start freezing it.
+	 */
+	blk_freeze_queue_start(q);
+
+	/*
+	 * When quiescing completes, all ongoing dispatches
+have

completed

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

drivers/block/virtio_blk.c

Quoted 1 line
drivers/block/virtio_blk.c +1 -0
+	 * and no new dispatch will happen towards the

driver.

Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

drivers/block/virtio_blk.c

Quoted 6 lines
drivers/block/virtio_blk.c +6 -0
+	 */
+	blk_mq_quiesce_queue(q);
+
+	/*
+	 * Synchronize with any ongoing VQ callbacks that
+may have

started

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

drivers/block/virtio_blk.c

Quoted 2 lines
drivers/block/virtio_blk.c +2 -0
+	 * before the VQs were marked as broken. Any
+outstanding

requests

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

drivers/block/virtio_blk.c

Quoted 1 line
drivers/block/virtio_blk.c +1 -0
+	 * will be completed by

virtblk_complete_request_with_ioerr().

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

drivers/block/virtio_blk.c

Quoted 6 lines
drivers/block/virtio_blk.c +6 -0
+	 */
+	virtio_synchronize_cbs(vblk->vdev);
+
+	/*
+	 * At this point, no new requests can enter the
+queue_rq()

and

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

drivers/block/virtio_blk.c

Quoted 2 lines
drivers/block/virtio_blk.c +2 -0
+	 * completion routine will not complete any new
+requests either for

the

drivers/block/virtio_blk.c

Quoted 1 line
drivers/block/virtio_blk.c +1 -0
+	 * broken vq. Hence, it is safe to cancel all requests

which are

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

drivers/block/virtio_blk.c

Quoted 4 lines
drivers/block/virtio_blk.c +4 -0
+	 * started.
+	 */
+	blk_mq_tagset_busy_iter(&vblk->tag_set,
+

virtblk_complete_request_with_ioerr,

Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines

vblk);

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

drivers/block/virtio_blk.c

Quoted 6 lines
drivers/block/virtio_blk.c +6 -0
+
+blk_mq_tagset_wait_completed_request(&vblk->tag_set);
+
+	/*
+	 * All pending requests are cleaned up. Time to
+resume so

that disk

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

drivers/block/virtio_blk.c

Quoted 2 lines
drivers/block/virtio_blk.c +2 -0
+	 * deletion can be smooth. Start the HW queues so
+that when queue

is

drivers/block/virtio_blk.c

Quoted 7 lines
drivers/block/virtio_blk.c +7 -0
+	 * unquiesced requests can again enter the driver.
+	 */
+	blk_mq_start_stopped_hw_queues(q, true);
+
+	/*
+	 * Unquiescing will trigger dispatching any pending
+requests

to the

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

drivers/block/virtio_blk.c

Quoted 1 line
drivers/block/virtio_blk.c +1 -0
+	 * driver which has crossed bio_queue_enter() to the

driver.

Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

drivers/block/virtio_blk.c

Quoted 6 lines
drivers/block/virtio_blk.c +6 -0
+	 */
+	blk_mq_unquiesce_queue(q);
+
+	/*
+	 * Wait for all pending dispatches to terminate
+which may

have been

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

drivers/block/virtio_blk.c

Quoted 7 lines
drivers/block/virtio_blk.c +7 -0
+	 * initiated after unquiescing.
+	 */
+	blk_mq_freeze_queue_wait(q);
+
+	/*
+	 * Mark the disk dead so that once we unfreeze the
+queue,

requests

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

drivers/block/virtio_blk.c

Quoted 2 lines
drivers/block/virtio_blk.c +2 -0
+	 * waiting at the door of bio_queue_enter() can be
+aborted right

away.

drivers/block/virtio_blk.c

Quoted 5 lines
drivers/block/virtio_blk.c +5 -0
+	 */
+	blk_mark_disk_dead(vblk->disk);
+
+	/* Unfreeze the queue so that any waiting requests
+will be

aborted. */

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

drivers/block/virtio_blk.c

Quoted 6 lines
drivers/block/virtio_blk.c +4 -0
+	blk_mq_unfreeze_queue_nomemrestore(q);
+}
+
 static void virtblk_remove(struct virtio_device *vdev)  {
 	struct virtio_blk *vblk = vdev->priv; @@ -1561,6
+1653,9 @@ static void virtblk_remove(struct virtio_device

*vdev)

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

/* Make sure no work handler is accessing the device.

*/

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
drivers/block/virtio_blk.c +3 -1
 	flush_work(&vblk->config_work);
 
+	if (virtqueue_is_broken(vblk->vqs[0].vq))
+		virtblk_cleanup_broken_device(vblk);
+
 	del_gendisk(vblk->disk);
 	blk_mq_free_tag_set(&vblk->tag_set);
 
--

2.34.1

#13 lore raw
Quoted 101 lines · Parav Pandit
Quoted 92 lines

From: Michael S. Tsirkin <[email protected]>
Sent: 25 June 2025 05:15 PM

Quoted 22 lines · Parav Pandit
Quoted 22 lines

From: Michael S. Tsirkin <[email protected]>
Sent: 25 June 2025 04:34 PM

Quoted 17 lines · Parav Pandit
Quoted 17 lines

From: Michael S. Tsirkin <[email protected]>
Sent: 25 June 2025 01:24 AM

Quoted 12 lines · Parav Pandit
Quoted 12 lines

From: Michael S. Tsirkin <[email protected]>
Sent: 25 June 2025 12:37 AM

Quoted 6 lines · Parav Pandit
Quoted 6 lines

From: Stefan Hajnoczi <[email protected]>
Sent: 25 June 2025 12:26 AM

On Mon, Jun 02, 2025 at 02:44:33AM +0000, Parav Pandit

Quoted 43 lines · wrote:
Quoted 43 lines
Quoted 43 lines
Quoted 28 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 3 lines

When the PCI device is surprise removed, requests may not complete the device as the VQ is marked as broken. Due to this, the disk deletion hangs.

There are loops in the core virtio driver code that expect device register reads to eventually return 0:
drivers/virtio/virtio_pci_modern.c:vp_reset() drivers/virtio/virtio_pci_modern_dev.c:vp_modern_set_que ue_r
eset
()

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Is there a hang if these loops are hit when a device has been surprise removed? I'm trying to understand whether surprise removal is fully supported or whether this patch is one step in that

direction.

Quoted 11 lines
Quoted 11 lines
Quoted 3 lines
Quoted 1 line

In one of the previous replies I answered to Michael, but don't have the link

handy.

Quoted 4 lines

It is not fully supported by this patch. It will hang.

This patch restores driver back to the same state what it was before the fixes

tag patch.

Quoted 2 lines

The virtio stack level work is needed to support surprise removal, including

Quoted 9 lines
Quoted 5 lines
Quoted 4 lines

the reset flow you rightly pointed.

Have plans to do that?

Didn't give enough thoughts on it yet.

This one is kind of pointless then? It just fixes the specific race window that your test harness happens to hit?

It was reported by Li from Baidu, whose tests failed. I missed to tag "reported-by" in v5. I had it until v4. :(

Quoted 3 lines

Maybe it's better to wait until someone does a comprehensive fix..

Quoted 8 lines
Quoted 8 lines
Quoted 2 lines

Oh, I was under impression is that you wanted to step forward in discussion

of v4.

Quoted 2 lines

If you prefer a comprehensive support across layers of virtio, I suggest you

should revert the cited patch in fixes tag.

Quoted 2 lines

Otherwise, it is in degraded state as virtio never supported surprise

removal.

Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

By reverting the cited patch (or with this fix), the requests and disk deletion

will not hang.

But they will hung in virtio core on reset, will they not? The tests just do not happen to trigger this?

It will hang if it a true surprise removal which no device did so far because it

never worked.

Quoted 3 lines

(or did, but always hung that no one reported yet)

I am familiar with 2 or more PCI devices who reports surprise removal,

Quoted 9 lines

which do not complete the requests but yet allows device reset flow.

Quoted 1 line

This is because device is still there on the PCI bus. Only via side band signals

device removal was reported.

So why do we care about it so much? I think it's great this patch exists, for example it makes it easier to test surprise removal and find more bugs. But is it better to just have it hang unconditionally? Are we now making a commitment that it's working - one we don't seem to intend to implement?

Quoted 6 lines

The patch improves the situation from its current state. But as you posted, more changes in pci layer are needed. I didn't audit where else it may be needed.

vp_reset() may need to return the status back of successful/failure reset.
Otherwise during probe(), vp_reset() aborts the reset and attempts to load the driver for removed device.

yes however this is not at all different that hotunplug right after reset.

Quoted 1 line

I guess suspend() callback also infinitely waits during freezing the queue also needs adaptation.

Which callback is that I don't understand.

Quoted 257 lines
Quoted 7 lines
Quoted 1 line

But I agree that for full support, virtio all layer changes would be needed as

new functionality (without fixes tag :) ).

Or with a fixes tag - lots of people just use it as a signal to mean "where can this be reasonably backported to".

Yes, I think the fix for the older kernels is needed, hence I cced stable too.

Quoted 248 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Please let me know if I should re-send to revert the patch listed in fixes

tag.

Quoted 62 lines
Quoted 62 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Apart from that, I'm happy with the virtio_blk.c aspects of the

patch:

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

Thanks.

Quoted 1 line
Quoted 1 line
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Fix it by aborting the requests when the VQ is broken.

With this fix now fio completes swiftly.
An alternative of IO timeout has been considered, however when the driver knows about unresponsive block device, swiftly clearing them enables users and upper layers to react

quickly.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Verified with multiple device unplug iterations with pending requests in virtio used ring and some pending with the

device.

Quoted 40 lines
Quoted 40 lines
Quoted 40 lines
Quoted 40 lines
Quoted 40 lines
Quoted 40 lines
Quoted 40 lines
Fixes 43bb40c5b926 ("virtio_pci: Support surprise

removal of virtio pci device")

Closes:
https://lore.kernel.org/virtualization/c45dd68698cd472
38c5
5fb7
3ca9
b474
[email protected]/

---
v4->v5:
- fixed comment style where comment to start with one empty line at start
- Addressed comments from Alok
- fixed typo in broken vq check
v3->v4:
- Addressed comments from Michael
- renamed virtblk_request_cancel() to virtblk_complete_request_with_ioerr()
- Added comments for virtblk_complete_request_with_ioerr()
- Renamed virtblk_broken_device_cleanup() to virtblk_cleanup_broken_device()
- Added comments for virtblk_cleanup_broken_device()
- Moved the broken vq check in virtblk_remove()
- Fixed comment style to have first empty line
- replaced freezed to frozen
- Fixed comments rephrased

v2->v3:
- Addressed comments from Michael
- updated comment for synchronizing with callbacks

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

v1->v2:
- Addressed comments from Stephan
- fixed spelling to 'waiting'
- Addressed comments from Michael
- Dropped checking broken vq from queue_rq() and

queue_rqs()

Quoted 81 lines
Quoted 81 lines
Quoted 45 lines
Quoted 45 lines
Quoted 36 lines
Quoted 36 lines
Quoted 36 lines
Quoted 36 lines
Quoted 21 lines

because it is checked in lower layer routines in
virtio core

v0->v1:
- Fixed comments from Stefan to rename a cleanup function
- Improved logic for handling any outstanding requests in bio layer
- improved cancel callback to sync with ongoing done()

---
 drivers/block/virtio_blk.c | 95
++++++++++++++++++++++++++++++++++++++
 1 file changed, 95 insertions(+)
(quoted diff) +0 -0

b/drivers/block/virtio_blk.c index
7cffea01d868..c5e383c0ac48
100644

drivers/block/virtio_blk.c +0 -0
@@ -1554,6 +1554,98 @@ static int virtblk_probe(struct
Quoted 1 line

virtio_device

*vdev)

Quoted 13 lines
drivers/block/virtio_blk.c +10 -0
 	return err;
 }
 
+/*
+ * If the vq is broken, device will not complete requests.
+ * So we do it for the device.
+ */
+static bool
+virtblk_complete_request_with_ioerr(struct
+request *rq, void *data) {
+	struct virtblk_req *vbr = blk_mq_rq_to_pdu(rq);
+	struct virtio_blk *vblk = data;
+	struct virtio_blk_vq *vq;
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

drivers/block/virtio_blk.c

Quoted 8 lines
drivers/block/virtio_blk.c +8 -0
+	unsigned long flags;
+
+	vq = &vblk->vqs[rq->mq_hctx->queue_num];
+
+	spin_lock_irqsave(&vq->lock, flags);
+
+	vbr->in_hdr.status = VIRTIO_BLK_S_IOERR;
+	if (blk_mq_request_started(rq) &&

!blk_mq_request_completed(rq))

Quoted 15 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
drivers/block/virtio_blk.c +7 -0
+		blk_mq_complete_request(rq);
+
+	spin_unlock_irqrestore(&vq->lock, flags);
+	return true;
+}
+
+/*
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

drivers/block/virtio_blk.c

Quoted 8 lines

+ * If the device is broken, it will not use any

drivers/block/virtio_blk.c +7 -0
+buffers and waiting
+ * for that to happen is pointless. We'll do the
+cleanup in the driver,
+ * completing all requests for the device.
+ */
+static void virtblk_cleanup_broken_device(struct
+virtio_blk *vblk)

{

Quoted 20 lines
Quoted 20 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

drivers/block/virtio_blk.c

Quoted 5 lines
drivers/block/virtio_blk.c +5 -0
+	struct request_queue *q = vblk->disk->queue;
+
+	/*
+	 * Start freezing the queue, so that new requests
+keeps

waiting at the

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

drivers/block/virtio_blk.c

Quoted 2 lines
drivers/block/virtio_blk.c +2 -0
+	 * door of bio_queue_enter(). We cannot fully freeze
+the queue

because

drivers/block/virtio_blk.c

Quoted 2 lines
drivers/block/virtio_blk.c +2 -0
+	 * frozen queue is an empty queue and there are
+pending

requests, so

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

drivers/block/virtio_blk.c

Quoted 7 lines
drivers/block/virtio_blk.c +7 -0
+	 * only start freezing it.
+	 */
+	blk_freeze_queue_start(q);
+
+	/*
+	 * When quiescing completes, all ongoing dispatches
+have

completed

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

drivers/block/virtio_blk.c

Quoted 1 line
drivers/block/virtio_blk.c +1 -0
+	 * and no new dispatch will happen towards the

driver.

Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

drivers/block/virtio_blk.c

Quoted 6 lines
drivers/block/virtio_blk.c +6 -0
+	 */
+	blk_mq_quiesce_queue(q);
+
+	/*
+	 * Synchronize with any ongoing VQ callbacks that
+may have

started

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

drivers/block/virtio_blk.c

Quoted 2 lines
drivers/block/virtio_blk.c +2 -0
+	 * before the VQs were marked as broken. Any
+outstanding

requests

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

drivers/block/virtio_blk.c

Quoted 1 line
drivers/block/virtio_blk.c +1 -0
+	 * will be completed by

virtblk_complete_request_with_ioerr().

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

drivers/block/virtio_blk.c

Quoted 6 lines
drivers/block/virtio_blk.c +6 -0
+	 */
+	virtio_synchronize_cbs(vblk->vdev);
+
+	/*
+	 * At this point, no new requests can enter the
+queue_rq()

and

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

drivers/block/virtio_blk.c

Quoted 2 lines
drivers/block/virtio_blk.c +2 -0
+	 * completion routine will not complete any new
+requests either for

the

drivers/block/virtio_blk.c

Quoted 1 line
drivers/block/virtio_blk.c +1 -0
+	 * broken vq. Hence, it is safe to cancel all requests

which are

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

drivers/block/virtio_blk.c

Quoted 4 lines
drivers/block/virtio_blk.c +4 -0
+	 * started.
+	 */
+	blk_mq_tagset_busy_iter(&vblk->tag_set,
+

virtblk_complete_request_with_ioerr,

Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines

vblk);

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

drivers/block/virtio_blk.c

Quoted 6 lines
drivers/block/virtio_blk.c +6 -0
+
+blk_mq_tagset_wait_completed_request(&vblk->tag_set);
+
+	/*
+	 * All pending requests are cleaned up. Time to
+resume so

that disk

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

drivers/block/virtio_blk.c

Quoted 2 lines
drivers/block/virtio_blk.c +2 -0
+	 * deletion can be smooth. Start the HW queues so
+that when queue

is

drivers/block/virtio_blk.c

Quoted 7 lines
drivers/block/virtio_blk.c +7 -0
+	 * unquiesced requests can again enter the driver.
+	 */
+	blk_mq_start_stopped_hw_queues(q, true);
+
+	/*
+	 * Unquiescing will trigger dispatching any pending
+requests

to the

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

drivers/block/virtio_blk.c

Quoted 1 line
drivers/block/virtio_blk.c +1 -0
+	 * driver which has crossed bio_queue_enter() to the

driver.

Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

drivers/block/virtio_blk.c

Quoted 6 lines
drivers/block/virtio_blk.c +6 -0
+	 */
+	blk_mq_unquiesce_queue(q);
+
+	/*
+	 * Wait for all pending dispatches to terminate
+which may

have been

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

drivers/block/virtio_blk.c

Quoted 7 lines
drivers/block/virtio_blk.c +7 -0
+	 * initiated after unquiescing.
+	 */
+	blk_mq_freeze_queue_wait(q);
+
+	/*
+	 * Mark the disk dead so that once we unfreeze the
+queue,

requests

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

drivers/block/virtio_blk.c

Quoted 2 lines
drivers/block/virtio_blk.c +2 -0
+	 * waiting at the door of bio_queue_enter() can be
+aborted right

away.

drivers/block/virtio_blk.c

Quoted 5 lines
drivers/block/virtio_blk.c +5 -0
+	 */
+	blk_mark_disk_dead(vblk->disk);
+
+	/* Unfreeze the queue so that any waiting requests
+will be

aborted. */

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

drivers/block/virtio_blk.c

Quoted 6 lines
drivers/block/virtio_blk.c +4 -0
+	blk_mq_unfreeze_queue_nomemrestore(q);
+}
+
 static void virtblk_remove(struct virtio_device *vdev)  {
 	struct virtio_blk *vblk = vdev->priv; @@ -1561,6
+1653,9 @@ static void virtblk_remove(struct virtio_device

*vdev)

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

/* Make sure no work handler is accessing the device.

*/

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
drivers/block/virtio_blk.c +3 -1
 	flush_work(&vblk->config_work);
 
+	if (virtqueue_is_broken(vblk->vqs[0].vq))
+		virtblk_cleanup_broken_device(vblk);
+
 	del_gendisk(vblk->disk);
 	blk_mq_free_tag_set(&vblk->tag_set);
 
--

2.34.1

#14 lore raw
Quoted 119 lines

From: Michael S. Tsirkin <[email protected]>
Sent: 26 June 2025 12:52 AM

Quoted 23 lines · Parav Pandit
Quoted 23 lines

From: Michael S. Tsirkin <[email protected]>
Sent: 25 June 2025 05:15 PM

Quoted 18 lines · Parav Pandit
Quoted 18 lines

From: Michael S. Tsirkin <[email protected]>
Sent: 25 June 2025 04:34 PM

Quoted 13 lines · Parav Pandit
Quoted 13 lines

From: Michael S. Tsirkin <[email protected]>
Sent: 25 June 2025 01:24 AM

Quoted 8 lines · Parav Pandit
Quoted 8 lines

From: Michael S. Tsirkin <[email protected]>
Sent: 25 June 2025 12:37 AM

Quoted 2 lines · Parav Pandit
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

From: Stefan Hajnoczi <[email protected]>
Sent: 25 June 2025 12:26 AM

On Mon, Jun 02, 2025 at 02:44:33AM +0000, Parav Pandit

Quoted 2 lines · wrote:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

When the PCI device is surprise removed, requests may not complete the device as the VQ is marked as

broken.

Quoted 67 lines
Quoted 67 lines
Quoted 44 lines
Quoted 44 lines
Quoted 44 lines
Quoted 27 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 1 line

Due to this, the disk deletion hangs.

There are loops in the core virtio driver code that expect device register reads to eventually return 0:
drivers/virtio/virtio_pci_modern.c:vp_reset() drivers/virtio/virtio_pci_modern_dev.c:vp_modern_set _que
ue_r
eset
()

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Is there a hang if these loops are hit when a device has been surprise removed? I'm trying to understand whether surprise removal is fully supported or whether this patch is one step in that

direction.

Quoted 11 lines
Quoted 11 lines
Quoted 3 lines
Quoted 1 line

In one of the previous replies I answered to Michael, but don't have the link

handy.

Quoted 4 lines

It is not fully supported by this patch. It will hang.

This patch restores driver back to the same state what it was before the fixes

tag patch.

Quoted 2 lines

The virtio stack level work is needed to support
surprise removal, including

Quoted 9 lines
Quoted 5 lines
Quoted 4 lines

the reset flow you rightly pointed.

Have plans to do that?

Didn't give enough thoughts on it yet.

This one is kind of pointless then? It just fixes the specific race window that your test harness happens to hit?

It was reported by Li from Baidu, whose tests failed. I missed to tag "reported-by" in v5. I had it until v4. :(

Quoted 3 lines

Maybe it's better to wait until someone does a comprehensive fix..

Oh, I was under impression is that you wanted to step forward in discussion

Quoted 7 lines
Quoted 7 lines

of v4.

Quoted 2 lines

If you prefer a comprehensive support across layers of virtio, I suggest you

should revert the cited patch in fixes tag.

Quoted 3 lines

Otherwise, it is in degraded state as virtio never supported surprise

removal.

Quoted 9 lines
Quoted 7 lines
Quoted 2 lines

By reverting the cited patch (or with this fix), the requests and disk deletion

will not hang.

But they will hung in virtio core on reset, will they not? The tests just do not happen to trigger this?

It will hang if it a true surprise removal which no device did so far because it

never worked.

Quoted 4 lines

(or did, but always hung that no one reported yet)

I am familiar with 2 or more PCI devices who reports surprise removal,

which do not complete the requests but yet allows device reset flow.

Quoted 8 lines
Quoted 8 lines
Quoted 2 lines

This is because device is still there on the PCI bus. Only via side band signals

device removal was reported.

So why do we care about it so much? I think it's great this patch exists, for example it makes it easier to test surprise removal and find more bugs. But is it better to just have it hang unconditionally? Are we now making a commitment that it's working -

one we don't seem to intend to implement?

Quoted 5 lines
Quoted 1 line

The patch improves the situation from its current state. But as you posted, more changes in pci layer are needed. I didn't audit where else it may be needed.

Quoted 6 lines
Quoted 2 lines

vp_reset() may need to return the status back of successful/failure reset. Otherwise during probe(), vp_reset() aborts the reset and attempts to load

the driver for removed device.

yes however this is not at all different that hotunplug right after reset.

For hotunplug after reset, we likely need a timeout handler.
Because block driver running inside the remove() callback waiting for the IO, may not get notified from driver core to synchronize ongoing remove().

Quoted 4 lines
Quoted 1 line

I guess suspend() callback also infinitely waits during freezing the queue

also needs adaptation.

Which callback is that I don't understand.

virtblk_freeze() at [1].

[1] https://elixir.bootlin.com/linux/v6.15.3/source/drivers/block/virtio_blk.c#L1622

Quoted 268 lines
Quoted 268 lines
Quoted 10 lines
Quoted 4 lines

But I agree that for full support, virtio all layer changes would be needed as

new functionality (without fixes tag :) ).

Or with a fixes tag - lots of people just use it as a signal to mean "where can this be reasonably backported to".

Yes, I think the fix for the older kernels is needed, hence I cced stable too.

Quoted 256 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Please let me know if I should re-send to revert the patch listed in fixes

tag.

Quoted 64 lines
Quoted 64 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Apart from that, I'm happy with the virtio_blk.c aspects of the

patch:

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

Thanks.

Quoted 1 line
Quoted 1 line
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Fix it by aborting the requests when the VQ is broken.

With this fix now fio completes swiftly. An alternative of IO timeout has been considered, however when the driver knows about unresponsive block device, swiftly clearing them enables users and upper layers to react

quickly.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Verified with multiple device unplug iterations with pending requests in virtio used ring and some pending with the

device.

Quoted 42 lines
Quoted 42 lines
Quoted 42 lines
Quoted 42 lines
Quoted 42 lines
Quoted 42 lines
Quoted 42 lines
Fixes 43bb40c5b926 ("virtio_pci: Support surprise

removal of virtio pci device")

Closes:
https://lore.kernel.org/virtualization/c45dd68698c
d472
38c5
5fb7
3ca9
b474
[email protected]/

---
v4->v5:
- fixed comment style where comment to start with one empty line at start
- Addressed comments from Alok
- fixed typo in broken vq check
v3->v4:
- Addressed comments from Michael
- renamed virtblk_request_cancel() to virtblk_complete_request_with_ioerr()
- Added comments for virtblk_complete_request_with_ioerr()
- Renamed virtblk_broken_device_cleanup() to virtblk_cleanup_broken_device()
- Added comments for virtblk_cleanup_broken_device()
- Moved the broken vq check in virtblk_remove()
- Fixed comment style to have first empty line
- replaced freezed to frozen
- Fixed comments rephrased

v2->v3:
- Addressed comments from Michael
- updated comment for synchronizing with callbacks

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

v1->v2:
- Addressed comments from Stephan
- fixed spelling to 'waiting'
- Addressed comments from Michael
- Dropped checking broken vq from queue_rq() and

queue_rqs()

Quoted 82 lines
Quoted 82 lines
Quoted 46 lines
Quoted 46 lines
Quoted 37 lines
Quoted 37 lines
Quoted 37 lines
Quoted 37 lines
Quoted 22 lines

because it is checked in lower layer routines in
virtio core

v0->v1:
- Fixed comments from Stefan to rename a cleanup function
- Improved logic for handling any outstanding requests in bio layer
- improved cancel callback to sync with ongoing done()

---
 drivers/block/virtio_blk.c | 95
++++++++++++++++++++++++++++++++++++++
 1 file changed, 95 insertions(+)
(quoted diff) +0 -0

b/drivers/block/virtio_blk.c index
7cffea01d868..c5e383c0ac48
100644

drivers/block/virtio_blk.c +0 -0
@@ -1554,6 +1554,98 @@ static int
Quoted 1 line

virtblk_probe(struct virtio_device

*vdev)

Quoted 13 lines
drivers/block/virtio_blk.c +10 -0
 	return err;
 }
 
+/*
+ * If the vq is broken, device will not complete requests.
+ * So we do it for the device.
+ */
+static bool
+virtblk_complete_request_with_ioerr(struct
+request *rq, void *data) {
+	struct virtblk_req *vbr = blk_mq_rq_to_pdu(rq);
+	struct virtio_blk *vblk = data;
+	struct virtio_blk_vq *vq;
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

drivers/block/virtio_blk.c

Quoted 8 lines
drivers/block/virtio_blk.c +8 -0
+	unsigned long flags;
+
+	vq = &vblk->vqs[rq->mq_hctx->queue_num];
+
+	spin_lock_irqsave(&vq->lock, flags);
+
+	vbr->in_hdr.status = VIRTIO_BLK_S_IOERR;
+	if (blk_mq_request_started(rq) &&

!blk_mq_request_completed(rq))

Quoted 15 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
drivers/block/virtio_blk.c +7 -0
+		blk_mq_complete_request(rq);
+
+	spin_unlock_irqrestore(&vq->lock, flags);
+	return true;
+}
+
+/*
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

drivers/block/virtio_blk.c

Quoted 8 lines

+ * If the device is broken, it will not use any

drivers/block/virtio_blk.c +7 -0
+buffers and waiting
+ * for that to happen is pointless. We'll do the
+cleanup in the driver,
+ * completing all requests for the device.
+ */
+static void virtblk_cleanup_broken_device(struct
+virtio_blk *vblk)

{

Quoted 20 lines
Quoted 20 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

drivers/block/virtio_blk.c

Quoted 5 lines
drivers/block/virtio_blk.c +5 -0
+	struct request_queue *q = vblk->disk->queue;
+
+	/*
+	 * Start freezing the queue, so that new
+requests keeps

waiting at the

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

drivers/block/virtio_blk.c

Quoted 2 lines
drivers/block/virtio_blk.c +2 -0
+	 * door of bio_queue_enter(). We cannot fully
+freeze the queue

because

drivers/block/virtio_blk.c

Quoted 2 lines
drivers/block/virtio_blk.c +2 -0
+	 * frozen queue is an empty queue and there are
+pending

requests, so

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

drivers/block/virtio_blk.c

Quoted 7 lines
drivers/block/virtio_blk.c +7 -0
+	 * only start freezing it.
+	 */
+	blk_freeze_queue_start(q);
+
+	/*
+	 * When quiescing completes, all ongoing
+dispatches have

completed

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

drivers/block/virtio_blk.c

Quoted 1 line
drivers/block/virtio_blk.c +1 -0
+	 * and no new dispatch will happen towards the

driver.

Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

drivers/block/virtio_blk.c

Quoted 6 lines
drivers/block/virtio_blk.c +6 -0
+	 */
+	blk_mq_quiesce_queue(q);
+
+	/*
+	 * Synchronize with any ongoing VQ callbacks
+that may have

started

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

drivers/block/virtio_blk.c

Quoted 2 lines
drivers/block/virtio_blk.c +2 -0
+	 * before the VQs were marked as broken. Any
+outstanding

requests

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

drivers/block/virtio_blk.c

Quoted 1 line
drivers/block/virtio_blk.c +1 -0
+	 * will be completed by

virtblk_complete_request_with_ioerr().

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

drivers/block/virtio_blk.c

Quoted 6 lines
drivers/block/virtio_blk.c +6 -0
+	 */
+	virtio_synchronize_cbs(vblk->vdev);
+
+	/*
+	 * At this point, no new requests can enter the
+queue_rq()

and

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

drivers/block/virtio_blk.c

Quoted 2 lines
drivers/block/virtio_blk.c +2 -0
+	 * completion routine will not complete any new
+requests either for

the

drivers/block/virtio_blk.c

Quoted 2 lines
drivers/block/virtio_blk.c +2 -0
+	 * broken vq. Hence, it is safe to cancel all
+requests

which are

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

drivers/block/virtio_blk.c

Quoted 4 lines
drivers/block/virtio_blk.c +4 -0
+	 * started.
+	 */
+	blk_mq_tagset_busy_iter(&vblk->tag_set,
+

virtblk_complete_request_with_ioerr,

Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines

vblk);

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

drivers/block/virtio_blk.c

Quoted 7 lines
drivers/block/virtio_blk.c +7 -0
+
+blk_mq_tagset_wait_completed_request(&vblk->tag_s
+et);
+
+	/*
+	 * All pending requests are cleaned up. Time to
+resume so

that disk

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

drivers/block/virtio_blk.c

Quoted 2 lines
drivers/block/virtio_blk.c +2 -0
+	 * deletion can be smooth. Start the HW queues
+so that when queue

is

drivers/block/virtio_blk.c

Quoted 7 lines
drivers/block/virtio_blk.c +7 -0
+	 * unquiesced requests can again enter the driver.
+	 */
+	blk_mq_start_stopped_hw_queues(q, true);
+
+	/*
+	 * Unquiescing will trigger dispatching any
+pending requests

to the

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

drivers/block/virtio_blk.c

Quoted 2 lines
drivers/block/virtio_blk.c +2 -0
+	 * driver which has crossed bio_queue_enter() to
+the

driver.

Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

drivers/block/virtio_blk.c

Quoted 6 lines
drivers/block/virtio_blk.c +6 -0
+	 */
+	blk_mq_unquiesce_queue(q);
+
+	/*
+	 * Wait for all pending dispatches to terminate
+which may

have been

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

drivers/block/virtio_blk.c

Quoted 7 lines
drivers/block/virtio_blk.c +7 -0
+	 * initiated after unquiescing.
+	 */
+	blk_mq_freeze_queue_wait(q);
+
+	/*
+	 * Mark the disk dead so that once we unfreeze
+the queue,

requests

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

drivers/block/virtio_blk.c

Quoted 2 lines
drivers/block/virtio_blk.c +2 -0
+	 * waiting at the door of bio_queue_enter() can
+be aborted right

away.

drivers/block/virtio_blk.c

Quoted 5 lines
drivers/block/virtio_blk.c +5 -0
+	 */
+	blk_mark_disk_dead(vblk->disk);
+
+	/* Unfreeze the queue so that any waiting
+requests will be

aborted. */

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

drivers/block/virtio_blk.c

Quoted 7 lines
drivers/block/virtio_blk.c +5 -0
+	blk_mq_unfreeze_queue_nomemrestore(q);
+}
+
 static void virtblk_remove(struct virtio_device *vdev)  {
 	struct virtio_blk *vblk = vdev->priv; @@ -1561,6
+1653,9 @@ static void virtblk_remove(struct
+virtio_device

*vdev)

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

/* Make sure no work handler is accessing the device.

*/

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
drivers/block/virtio_blk.c +3 -1
 	flush_work(&vblk->config_work);
 
+	if (virtqueue_is_broken(vblk->vqs[0].vq))
+		virtblk_cleanup_broken_device(vblk);
+
 	del_gendisk(vblk->disk);
 	blk_mq_free_tag_set(&vblk->tag_set);
 
--

2.34.1

#15 lore raw
Quoted 119 lines · Parav Pandit
Quoted 119 lines

From: Michael S. Tsirkin <[email protected]>
Sent: 26 June 2025 12:52 AM

Quoted 23 lines · Parav Pandit
Quoted 23 lines

From: Michael S. Tsirkin <[email protected]>
Sent: 25 June 2025 05:15 PM

Quoted 18 lines · Parav Pandit
Quoted 18 lines

From: Michael S. Tsirkin <[email protected]>
Sent: 25 June 2025 04:34 PM

Quoted 13 lines · Parav Pandit
Quoted 13 lines

From: Michael S. Tsirkin <[email protected]>
Sent: 25 June 2025 01:24 AM

Quoted 8 lines · Parav Pandit
Quoted 8 lines

From: Michael S. Tsirkin <[email protected]>
Sent: 25 June 2025 12:37 AM

Quoted 2 lines · Parav Pandit
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

From: Stefan Hajnoczi <[email protected]>
Sent: 25 June 2025 12:26 AM

On Mon, Jun 02, 2025 at 02:44:33AM +0000, Parav Pandit

Quoted 2 lines · wrote:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

When the PCI device is surprise removed, requests
may not complete the device as the VQ is marked as

broken.

Quoted 67 lines
Quoted 67 lines
Quoted 44 lines
Quoted 44 lines
Quoted 44 lines
Quoted 27 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 1 line

Due to this, the disk deletion hangs.

There are loops in the core virtio driver code that expect device register reads to eventually return 0:
drivers/virtio/virtio_pci_modern.c:vp_reset() drivers/virtio/virtio_pci_modern_dev.c:vp_modern_set _que
ue_r
eset
()

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Is there a hang if these loops are hit when a device has been surprise removed? I'm trying to understand whether surprise removal is fully supported or whether this patch is one step in that

direction.

Quoted 11 lines
Quoted 11 lines
Quoted 3 lines
Quoted 1 line

In one of the previous replies I answered to Michael, but don't have the link

handy.

Quoted 4 lines

It is not fully supported by this patch. It will hang.

This patch restores driver back to the same state what it was before the fixes

tag patch.

Quoted 2 lines

The virtio stack level work is needed to support
surprise removal, including

Quoted 9 lines
Quoted 5 lines
Quoted 4 lines

the reset flow you rightly pointed.

Have plans to do that?

Didn't give enough thoughts on it yet.

This one is kind of pointless then? It just fixes the specific race window that your test harness happens to hit?

It was reported by Li from Baidu, whose tests failed. I missed to tag "reported-by" in v5. I had it until v4. :(

Quoted 3 lines

Maybe it's better to wait until someone does a comprehensive fix..

Oh, I was under impression is that you wanted to step forward in discussion

Quoted 7 lines
Quoted 7 lines

of v4.

Quoted 2 lines

If you prefer a comprehensive support across layers of virtio, I suggest you

should revert the cited patch in fixes tag.

Quoted 3 lines

Otherwise, it is in degraded state as virtio never supported surprise

removal.

Quoted 9 lines
Quoted 7 lines
Quoted 2 lines

By reverting the cited patch (or with this fix), the requests and disk deletion

will not hang.

But they will hung in virtio core on reset, will they not? The tests just do not happen to trigger this?

It will hang if it a true surprise removal which no device did so far because it

never worked.

Quoted 4 lines

(or did, but always hung that no one reported yet)

I am familiar with 2 or more PCI devices who reports surprise removal,

which do not complete the requests but yet allows device reset flow.

Quoted 8 lines
Quoted 8 lines
Quoted 2 lines

This is because device is still there on the PCI bus. Only via side band signals

device removal was reported.

So why do we care about it so much? I think it's great this patch exists, for example it makes it easier to test surprise removal and find more bugs. But is it better to just have it hang unconditionally? Are we now making a commitment that it's working -

one we don't seem to intend to implement?

Quoted 5 lines
Quoted 1 line

The patch improves the situation from its current state. But as you posted, more changes in pci layer are needed. I didn't audit where else it may be needed.

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

vp_reset() may need to return the status back of successful/failure reset. Otherwise during probe(), vp_reset() aborts the reset and attempts to load

the driver for removed device.

yes however this is not at all different that hotunplug right after reset.

For hotunplug after reset, we likely need a timeout handler.
Because block driver running inside the remove() callback waiting for the IO, may not get notified from driver core to synchronize ongoing remove().

Notified of what? So is the scenario that graceful remove starts, and meanwhile a surprise removal happens?

Quoted 277 lines
Quoted 5 lines
Quoted 2 lines

I guess suspend() callback also infinitely waits during freezing the queue

also needs adaptation.

Which callback is that I don't understand.

virtblk_freeze() at [1].

[1] https://elixir.bootlin.com/linux/v6.15.3/source/drivers/block/virtio_blk.c#L1622

Quoted 268 lines
Quoted 268 lines
Quoted 10 lines
Quoted 4 lines

But I agree that for full support, virtio all layer changes would be needed as

new functionality (without fixes tag :) ).

Or with a fixes tag - lots of people just use it as a signal to mean "where can this be reasonably backported to".

Yes, I think the fix for the older kernels is needed, hence I cced stable too.

Quoted 256 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Please let me know if I should re-send to revert the patch listed in fixes

tag.

Quoted 64 lines
Quoted 64 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Apart from that, I'm happy with the virtio_blk.c aspects of the

patch:

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

Thanks.

Quoted 1 line
Quoted 1 line
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Fix it by aborting the requests when the VQ is broken.

With this fix now fio completes swiftly.
An alternative of IO timeout has been considered,
however when the driver knows about unresponsive
block device, swiftly clearing them enables users
and upper layers to react

quickly.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Verified with multiple device unplug iterations
with pending requests in virtio used ring and some
pending with the

device.

Quoted 42 lines
Quoted 42 lines
Quoted 42 lines
Quoted 42 lines
Quoted 42 lines
Quoted 42 lines
Quoted 42 lines

Fixes: 43bb40c5b926 ("virtio_pci: Support surprise
removal of virtio pci device")
Cc: [email protected]
Reported-by: Li RongQing <[email protected]>
Closes:
https://lore.kernel.org/virtualization/c45dd68698c
d472
38c5
5fb7
3ca9
b474
[email protected]/
Reviewed-by: Max Gurtovoy <[email protected]>
Reviewed-by: Israel Rukshin <[email protected]>
Signed-off-by: Parav Pandit <[email protected]>

---
v4->v5:
- fixed comment style where comment to start with
one empty line at start
- Addressed comments from Alok
- fixed typo in broken vq check
v3->v4:
- Addressed comments from Michael
- renamed virtblk_request_cancel() to
virtblk_complete_request_with_ioerr()
- Added comments for
virtblk_complete_request_with_ioerr()
- Renamed virtblk_broken_device_cleanup() to
virtblk_cleanup_broken_device()
- Added comments for
virtblk_cleanup_broken_device()
- Moved the broken vq check in virtblk_remove()
- Fixed comment style to have first empty line
- replaced freezed to frozen
- Fixed comments rephrased

v2->v3:
- Addressed comments from Michael
- updated comment for synchronizing with callbacks

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

v1->v2:
- Addressed comments from Stephan
- fixed spelling to 'waiting'
- Addressed comments from Michael
- Dropped checking broken vq from queue_rq() and

queue_rqs()

Quoted 82 lines
Quoted 82 lines
Quoted 46 lines
Quoted 46 lines
Quoted 37 lines
Quoted 37 lines
Quoted 37 lines
Quoted 37 lines
Quoted 23 lines

because it is checked in lower layer routines in
virtio core

v0->v1:
- Fixed comments from Stefan to rename a cleanup
function
- Improved logic for handling any outstanding requests
in bio layer
- improved cancel callback to sync with ongoing
done()
---
drivers/block/virtio_blk.c | 95
++++++++++++++++++++++++++++++++++++++
1 file changed, 95 insertions(+)

diff --git a/drivers/block/virtio_blk.c
b/drivers/block/virtio_blk.c index
7cffea01d868..c5e383c0ac48
100644
--- a/drivers/block/virtio_blk.c
+++ b/drivers/block/virtio_blk.c
@@ -1554,6 +1554,98 @@ static int
virtblk_probe(struct virtio_device

*vdev)

Quoted 13 lines

return err;
}

+/*
+ * If the vq is broken, device will not complete requests.
+ * So we do it for the device.
+ */
+static bool
+virtblk_complete_request_with_ioerr(struct
+request *rq, void *data) {
+ struct virtblk_req *vbr = blk_mq_rq_to_pdu(rq);
+ struct virtio_blk *vblk = data;
+ struct virtio_blk_vq *vq;

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

+ unsigned long flags;
+
+ vq = &vblk->vqs[rq->mq_hctx->queue_num];
+
+ spin_lock_irqsave(&vq->lock, flags);
+
+ vbr->in_hdr.status = VIRTIO_BLK_S_IOERR;
+ if (blk_mq_request_started(rq) &&

!blk_mq_request_completed(rq))

Quoted 15 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

+ blk_mq_complete_request(rq);
+
+ spin_unlock_irqrestore(&vq->lock, flags);
+ return true;
+}
+
+/*

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

+ * If the device is broken, it will not use any
+buffers and waiting
+ * for that to happen is pointless. We'll do the
+cleanup in the driver,
+ * completing all requests for the device.
+ */
+static void virtblk_cleanup_broken_device(struct
+virtio_blk *vblk)

{

Quoted 20 lines
Quoted 20 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

+ struct request_queue *q = vblk->disk->queue;
+
+ /*
+ * Start freezing the queue, so that new
+requests keeps

waiting at the

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

+ * door of bio_queue_enter(). We cannot fully
+freeze the queue

because

Quoted 2 lines

+ * frozen queue is an empty queue and there are
+pending

requests, so

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

+ * only start freezing it.
+ */
+ blk_freeze_queue_start(q);
+
+ /*
+ * When quiescing completes, all ongoing
+dispatches have

completed

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

+ * and no new dispatch will happen towards the

driver.

Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

+ */
+ blk_mq_quiesce_queue(q);
+
+ /*
+ * Synchronize with any ongoing VQ callbacks
+that may have

started

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

+ * before the VQs were marked as broken. Any
+outstanding

requests

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

+ * will be completed by

virtblk_complete_request_with_ioerr().

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

+ */
+ virtio_synchronize_cbs(vblk->vdev);
+
+ /*
+ * At this point, no new requests can enter the
+queue_rq()

and

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

+ * completion routine will not complete any new
+requests either for

the

Quoted 2 lines

+ * broken vq. Hence, it is safe to cancel all
+requests

which are

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

+ * started.
+ */
+ blk_mq_tagset_busy_iter(&vblk->tag_set,
+

virtblk_complete_request_with_ioerr,

Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines

vblk);

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

+
+blk_mq_tagset_wait_completed_request(&vblk->tag_s
+et);
+
+ /*
+ * All pending requests are cleaned up. Time to
+resume so

that disk

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 2 lines

+ * deletion can be smooth. Start the HW queues
+so that when queue

is

Quoted 7 lines

+ * unquiesced requests can again enter the driver.
+ */
+ blk_mq_start_stopped_hw_queues(q, true);
+
+ /*
+ * Unquiescing will trigger dispatching any
+pending requests

to the

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

+ * driver which has crossed bio_queue_enter() to
+the

driver.

Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

+ */
+ blk_mq_unquiesce_queue(q);
+
+ /*
+ * Wait for all pending dispatches to terminate
+which may

have been

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

+ * initiated after unquiescing.
+ */
+ blk_mq_freeze_queue_wait(q);
+
+ /*
+ * Mark the disk dead so that once we unfreeze
+the queue,

requests

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines

+ * waiting at the door of bio_queue_enter() can
+be aborted right

away.

Quoted 5 lines

+ */
+ blk_mark_disk_dead(vblk->disk);
+
+ /* Unfreeze the queue so that any waiting
+requests will be

aborted. */

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

+ blk_mq_unfreeze_queue_nomemrestore(q);
+}
+
static void virtblk_remove(struct virtio_device *vdev) {
struct virtio_blk *vblk = vdev->priv; @@ -1561,6
+1653,9 @@ static void virtblk_remove(struct
+virtio_device

*vdev)

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

/* Make sure no work handler is accessing the device.

*/

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

flush_work(&vblk->config_work);

+ if (virtqueue_is_broken(vblk->vqs[0].vq))
+ virtblk_cleanup_broken_device(vblk);
+
del_gendisk(vblk->disk);
blk_mq_free_tag_set(&vblk->tag_set);

--
2.34.1

#16 lore raw
Quoted 135 lines

From: Michael S. Tsirkin <[email protected]>
Sent: 26 June 2025 11:34 AM

Quoted 27 lines · Parav Pandit
Quoted 27 lines

From: Michael S. Tsirkin <[email protected]>
Sent: 26 June 2025 12:52 AM

Quoted 21 lines · Parav Pandit
Quoted 21 lines

From: Michael S. Tsirkin <[email protected]>
Sent: 25 June 2025 05:15 PM

Quoted 16 lines · Parav Pandit
Quoted 16 lines

From: Michael S. Tsirkin <[email protected]>
Sent: 25 June 2025 04:34 PM

Quoted 11 lines · Parav Pandit
Quoted 11 lines

From: Michael S. Tsirkin <[email protected]>
Sent: 25 June 2025 01:24 AM

Quoted 6 lines · Parav Pandit
Quoted 6 lines

From: Michael S. Tsirkin <[email protected]>
Sent: 25 June 2025 12:37 AM

On Tue, Jun 24, 2025 at 07:01:44PM +0000, Parav Pandit

Quoted 47 lines · wrote:
Quoted 47 lines
Quoted 11 lines
Quoted 11 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

From: Stefan Hajnoczi <[email protected]>
Sent: 25 June 2025 12:26 AM

On Mon, Jun 02, 2025 at 02:44:33AM +0000, Parav
Pandit

Quoted 3 lines · wrote:
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

When the PCI device is surprise removed,
requests may not complete the device as the VQ
is marked as

broken.

Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines

Due to this, the disk deletion hangs.

There are loops in the core virtio driver code
that expect device register reads to eventually return 0:
drivers/virtio/virtio_pci_modern.c:vp_reset()
drivers/virtio/virtio_pci_modern_dev.c:vp_modern
_set
_que
ue_r
eset
()

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Is there a hang if these loops are hit when a
device has been surprise removed? I'm trying to
understand whether surprise removal is fully
supported or whether this patch is one step in
that

direction.

Quoted 16 lines
Quoted 6 lines
Quoted 3 lines
Quoted 1 line

In one of the previous replies I answered to
Michael, but don't have the link

handy.

Quoted 2 lines

It is not fully supported by this patch. It will hang.

Quoted 9 lines
Quoted 2 lines

This patch restores driver back to the same state
what it was before the fixes

tag patch.

Quoted 2 lines

The virtio stack level work is needed to support
surprise removal, including

the reset flow you rightly pointed.

Have plans to do that?

Didn't give enough thoughts on it yet.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

This one is kind of pointless then? It just fixes the specific race window that your test harness happens to hit?

It was reported by Li from Baidu, whose tests failed. I missed to tag "reported-by" in v5. I had it until v4. :(

Quoted 1 line

Maybe it's better to wait until someone does a comprehensive

fix..

Quoted 47 lines
Quoted 47 lines
Quoted 29 lines
Quoted 29 lines
Quoted 11 lines
Quoted 11 lines
Quoted 4 lines
Quoted 2 lines

Oh, I was under impression is that you wanted to step forward in discussion

of v4.

Quoted 2 lines

If you prefer a comprehensive support across layers of virtio, I suggest you

should revert the cited patch in fixes tag.

Quoted 3 lines

Otherwise, it is in degraded state as virtio never supported surprise

removal.

Quoted 9 lines
Quoted 7 lines
Quoted 2 lines

By reverting the cited patch (or with this fix), the requests and disk deletion

will not hang.

But they will hung in virtio core on reset, will they not? The tests just do not happen to trigger this?

It will hang if it a true surprise removal which no device did so far because it

never worked.

Quoted 4 lines

(or did, but always hung that no one reported yet)

I am familiar with 2 or more PCI devices who reports surprise removal,

which do not complete the requests but yet allows device reset flow.

Quoted 2 lines

This is because device is still there on the PCI bus. Only via side band signals

Quoted 7 lines
Quoted 7 lines

device removal was reported.

So why do we care about it so much? I think it's great this patch exists, for example it makes it easier to test surprise removal and find more bugs. But is it better to just have it hang unconditionally? Are we now making a commitment that it's working -

one we don't seem to intend to implement?

Quoted 8 lines
Quoted 1 line

The patch improves the situation from its current state. But as you posted, more changes in pci layer are needed. I didn't audit where else it may be needed.

vp_reset() may need to return the status back of successful/failure reset. Otherwise during probe(), vp_reset() aborts the reset and attempts to load

the driver for removed device.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

yes however this is not at all different that hotunplug right after reset.

For hotunplug after reset, we likely need a timeout handler. Because block driver running inside the remove() callback waiting for the IO,

may not get notified from driver core to synchronize ongoing remove().

Notified of what?

Notification that surprise-removal occurred.

Quoted 3 lines

So is the scenario that graceful remove starts, and meanwhile a surprise removal happens?

Right.

Quoted 294 lines
Quoted 294 lines
Quoted 6 lines
Quoted 3 lines

I guess suspend() callback also infinitely waits during freezing the queue

also needs adaptation.

Which callback is that I don't understand.

virtblk_freeze() at [1].

[1]
https://elixir.bootlin.com/linux/v6.15.3/source/drivers/block/virtio_b
lk.c#L1622

Quoted 282 lines
Quoted 282 lines
Quoted 10 lines
Quoted 4 lines

But I agree that for full support, virtio all layer changes would be needed as

new functionality (without fixes tag :) ).

Or with a fixes tag - lots of people just use it as a signal to mean "where can this be reasonably backported to".

Yes, I think the fix for the older kernels is needed, hence I cced stable too.

Quoted 270 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Please let me know if I should re-send to revert the patch listed in fixes

tag.

Quoted 72 lines
Quoted 72 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Apart from that, I'm happy with the virtio_blk.c
aspects of the

patch:

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

Reviewed-by: Stefan Hajnoczi
<[email protected]>

Thanks.

Quoted 1 line
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Fix it by aborting the requests when the VQ is broken.

With this fix now fio completes swiftly.
An alternative of IO timeout has been
considered, however when the driver knows
about unresponsive block device, swiftly
clearing them enables users and upper layers
to react

quickly.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Verified with multiple device unplug
iterations with pending requests in virtio
used ring and some pending with the

device.

Quoted 48 lines
Quoted 48 lines
Quoted 48 lines
Quoted 48 lines
Quoted 48 lines
Quoted 48 lines

Fixes: 43bb40c5b926 ("virtio_pci: Support
surprise removal of virtio pci device")
Cc: [email protected]
Reported-by: Li RongQing
<[email protected]>
Closes:
https://lore.kernel.org/virtualization/c45dd68
698c
d472
38c5
5fb7
3ca9
b474
[email protected]/
Reviewed-by: Max Gurtovoy
<[email protected]>
Reviewed-by: Israel Rukshin
<[email protected]>
Signed-off-by: Parav Pandit <[email protected]>

---
v4->v5:
- fixed comment style where comment to start
with one empty line at start
- Addressed comments from Alok
- fixed typo in broken vq check
v3->v4:
- Addressed comments from Michael
- renamed virtblk_request_cancel() to
virtblk_complete_request_with_ioerr()
- Added comments for
virtblk_complete_request_with_ioerr()
- Renamed virtblk_broken_device_cleanup() to
virtblk_cleanup_broken_device()
- Added comments for
virtblk_cleanup_broken_device()
- Moved the broken vq check in
virtblk_remove()
- Fixed comment style to have first empty line
- replaced freezed to frozen
- Fixed comments rephrased

v2->v3:
- Addressed comments from Michael
- updated comment for synchronizing with
callbacks

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

v1->v2:
- Addressed comments from Stephan
- fixed spelling to 'waiting'
- Addressed comments from Michael
- Dropped checking broken vq from queue_rq()
and

queue_rqs()

Quoted 83 lines
Quoted 83 lines
Quoted 47 lines
Quoted 47 lines
Quoted 38 lines
Quoted 38 lines
Quoted 38 lines
Quoted 38 lines

because it is checked in lower layer
routines in virtio core

v0->v1:
- Fixed comments from Stefan to rename a
cleanup function
- Improved logic for handling any outstanding requests
in bio layer
- improved cancel callback to sync with
ongoing
done()
---
drivers/block/virtio_blk.c | 95
++++++++++++++++++++++++++++++++++++++
1 file changed, 95 insertions(+)

diff --git a/drivers/block/virtio_blk.c
b/drivers/block/virtio_blk.c index
7cffea01d868..c5e383c0ac48
100644
--- a/drivers/block/virtio_blk.c
+++ b/drivers/block/virtio_blk.c
@@ -1554,6 +1554,98 @@ static int
virtblk_probe(struct virtio_device
*vdev)
return err;
}

+/*
+ * If the vq is broken, device will not complete requests.
+ * So we do it for the device.
+ */
+static bool
+virtblk_complete_request_with_ioerr(struct
+request *rq, void *data) {
+ struct virtblk_req *vbr = blk_mq_rq_to_pdu(rq);
+ struct virtio_blk *vblk = data;
+ struct virtio_blk_vq *vq;

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

+ unsigned long flags;
+
+ vq = &vblk->vqs[rq->mq_hctx->queue_num];
+
+ spin_lock_irqsave(&vq->lock, flags);
+
+ vbr->in_hdr.status = VIRTIO_BLK_S_IOERR;
+ if (blk_mq_request_started(rq) &&

!blk_mq_request_completed(rq))

Quoted 15 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

+ blk_mq_complete_request(rq);
+
+ spin_unlock_irqrestore(&vq->lock, flags);
+ return true; }
+
+/*
+ * If the device is broken, it will not use

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

+any buffers and waiting
+ * for that to happen is pointless. We'll do
+the cleanup in the driver,
+ * completing all requests for the device.
+ */
+static void
+virtblk_cleanup_broken_device(struct
+virtio_blk *vblk)

{

Quoted 20 lines
Quoted 20 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

+ struct request_queue *q = vblk->disk->queue;
+
+ /*
+ * Start freezing the queue, so that new
+requests keeps

waiting at the

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

+ * door of bio_queue_enter(). We cannot
+fully freeze the queue
because
+ * frozen queue is an empty queue and there
+are pending

requests, so

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

+ * only start freezing it.
+ */
+ blk_freeze_queue_start(q);
+
+ /*
+ * When quiescing completes, all ongoing
+dispatches have

completed

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

+ * and no new dispatch will happen towards
+the

driver.

Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

+ */
+ blk_mq_quiesce_queue(q);
+
+ /*
+ * Synchronize with any ongoing VQ callbacks
+that may have

started

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

+ * before the VQs were marked as broken. Any
+outstanding

requests

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

+ * will be completed by

virtblk_complete_request_with_ioerr().

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

+ */
+ virtio_synchronize_cbs(vblk->vdev);
+
+ /*
+ * At this point, no new requests can enter
+the
+queue_rq()

and

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

+ * completion routine will not complete any
+new requests either for
the
+ * broken vq. Hence, it is safe to cancel
+all requests

which are

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

+ * started.
+ */
+ blk_mq_tagset_busy_iter(&vblk->tag_set,
+

virtblk_complete_request_with_ioerr,

Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines

vblk);

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

+
+blk_mq_tagset_wait_completed_request(&vblk->t
+ag_s
+et);
+
+ /*
+ * All pending requests are cleaned up. Time
+to resume so

that disk

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

+ * deletion can be smooth. Start the HW
+queues so that when queue
is
+ * unquiesced requests can again enter the driver.
+ */
+ blk_mq_start_stopped_hw_queues(q, true);
+
+ /*
+ * Unquiescing will trigger dispatching any
+pending requests

to the

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

+ * driver which has crossed
+bio_queue_enter() to the

driver.

Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
Quoted 24 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

+ */
+ blk_mq_unquiesce_queue(q);
+
+ /*
+ * Wait for all pending dispatches to
+terminate which may

have been

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

+ * initiated after unquiescing.
+ */
+ blk_mq_freeze_queue_wait(q);
+
+ /*
+ * Mark the disk dead so that once we
+unfreeze the queue,

requests

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

+ * waiting at the door of bio_queue_enter()
+can be aborted right
away.
+ */
+ blk_mark_disk_dead(vblk->disk);
+
+ /* Unfreeze the queue so that any waiting
+requests will be

aborted. */

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

+ blk_mq_unfreeze_queue_nomemrestore(q);
+}
+
static void virtblk_remove(struct virtio_device *vdev) {
struct virtio_blk *vblk = vdev->priv; @@
-1561,6
+1653,9 @@ static void virtblk_remove(struct
+virtio_device

*vdev)

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

/* Make sure no work handler is accessing the device.

*/

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

flush_work(&vblk->config_work);

+ if (virtqueue_is_broken(vblk->vqs[0].vq))
+ virtblk_cleanup_broken_device(vblk);
+
del_gendisk(vblk->disk);
blk_mq_free_tag_set(&vblk->tag_set);

--
2.34.1

#17 lore raw
Quoted 7 lines · Parav Pandit
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

yes however this is not at all different that hotunplug right after reset.

For hotunplug after reset, we likely need a timeout handler. Because block driver running inside the remove() callback waiting for the IO,

may not get notified from driver core to synchronize ongoing remove().

Quoted 7 lines
Quoted 1 line

Notified of what?

Notification that surprise-removal occurred.

Quoted 3 lines

So is the scenario that graceful remove starts, and meanwhile a surprise removal happens?

Right.

where is it stuck then? can you explain?

#18 lore raw
Quoted 26 lines

From: Michael S. Tsirkin <[email protected]>
Sent: 26 June 2025 12:04 PM
To: Parav Pandit <[email protected]>
Cc: Stefan Hajnoczi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; NBU-Contact-Li Rongqing (EXTERNAL) <[email protected]>; Chaitanya Kulkarni <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Max Gurtovoy <[email protected]>; Israel Rukshin <[email protected]>
Subject: Re: [PATCH v5] virtio_blk: Fix disk deletion hang on device surprise removal

Quoted 11 lines · Parav Pandit
Quoted 9 lines
Quoted 5 lines
Quoted 2 lines

yes however this is not at all different that hotunplug right after reset.

For hotunplug after reset, we likely need a timeout handler. Because block driver running inside the remove() callback waiting for the IO,

may not get notified from driver core to synchronize ongoing remove().

Notified of what?

Notification that surprise-removal occurred.

Quoted 7 lines
Quoted 4 lines
Quoted 3 lines

So is the scenario that graceful remove starts, and meanwhile a surprise removal happens?

Right.

where is it stuck then? can you explain?

I am not sure I understood the question.

Let me try:
Following scenario will hang even with the current fix:

Say,
1. the graceful removal is ongoing in the remove() callback, where disk deletion del_gendisk() is ongoing, which waits for the requests to complete,

2. Now few requests are yet to complete, and surprise removal started.

At this point, virtio block driver will not get notified by the driver core layer, because it is likely serializing remove() happening by user/driver unload and PCI hotplug driver-initiated device removal. So vblk driver doesn't know that device is removed, block layer is waiting for requests completions to arrive which it never gets.
So del_gendisk() gets stuck.

This needs some kind of timeout handling to improve the situation to make removal more robust.

Did I answer or I didn't understand the question?

#19 lore raw
Quoted 44 lines · Parav Pandit
Quoted 26 lines

From: Michael S. Tsirkin <[email protected]>
Sent: 26 June 2025 12:04 PM
To: Parav Pandit <[email protected]>
Cc: Stefan Hajnoczi <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; NBU-Contact-Li Rongqing (EXTERNAL) <[email protected]>; Chaitanya Kulkarni <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Max Gurtovoy <[email protected]>; Israel Rukshin <[email protected]>
Subject: Re: [PATCH v5] virtio_blk: Fix disk deletion hang on device surprise removal

Quoted 11 lines · Parav Pandit
Quoted 9 lines
Quoted 5 lines
Quoted 2 lines

yes however this is not at all different that hotunplug right after reset.

For hotunplug after reset, we likely need a timeout handler. Because block driver running inside the remove() callback waiting for the IO,

may not get notified from driver core to synchronize ongoing remove().

Notified of what?

Notification that surprise-removal occurred.

Quoted 7 lines
Quoted 4 lines
Quoted 3 lines

So is the scenario that graceful remove starts, and meanwhile a surprise removal happens?

Right.

where is it stuck then? can you explain?

I am not sure I understood the question.

Let me try:
Following scenario will hang even with the current fix:

Say,
1. the graceful removal is ongoing in the remove() callback, where disk deletion del_gendisk() is ongoing, which waits for the requests to complete,

2. Now few requests are yet to complete, and surprise removal started.

Quoted 7 lines

At this point, virtio block driver will not get notified by the driver core layer, because it is likely serializing remove() happening by user/driver unload and PCI hotplug driver-initiated device removal. So vblk driver doesn't know that device is removed, block layer is waiting for requests completions to arrive which it never gets.
So del_gendisk() gets stuck.

This needs some kind of timeout handling to improve the situation to make removal more robust.

Did I answer or I didn't understand the question?

You did, thanks! How do other drivers handle this? The issue seems generic.

Signature · 1 line
MST
#20 lore raw
Quoted 2 lines · Michael S. Tsirkin

You did, thanks! How do other drivers handle this? The issue seems generic.

They implement blk_mq_ops' ".timeout" callback, which appears to be missing in virtio_mq_ops.

#21 lore raw
In reply to Stefan Hajnoczi
Quoted 5 lines · Stefan Hajnoczi
Quoted 3 lines · Parav Pandit

When the PCI device is surprise removed, requests may not complete the device as the VQ is marked as broken. Due to this, the disk deletion hangs.

Quoted 7 lines

There are loops in the core virtio driver code that expect device register reads to eventually return 0:
drivers/virtio/virtio_pci_modern.c:vp_reset() drivers/virtio/virtio_pci_modern_dev.c:vp_modern_set_queue_reset()

Is there a hang if these loops are hit when a device has been surprise removed?

Probably, yes. We can't really check broken status there though I think - reset is called with a broken device normally.

Quoted 2 lines

I'm trying to understand whether surprise removal is fully supported or whether this patch is one step in that direction.

I think it's a step.

Quoted 176 lines

Apart from that, I'm happy with the virtio_blk.c aspects of the patch:

Quoted 173 lines

Fix it by aborting the requests when the VQ is broken.

With this fix now fio completes swiftly.
An alternative of IO timeout has been considered, however when the driver knows about unresponsive block device, swiftly clearing them enables users and upper layers to react quickly.

Verified with multiple device unplug iterations with pending requests in virtio used ring and some pending with the device.

---
v4->v5:
- fixed comment style where comment to start with one empty line at start
- Addressed comments from Alok
- fixed typo in broken vq check
v3->v4:
- Addressed comments from Michael
- renamed virtblk_request_cancel() to virtblk_complete_request_with_ioerr()
- Added comments for virtblk_complete_request_with_ioerr()
- Renamed virtblk_broken_device_cleanup() to virtblk_cleanup_broken_device()
- Added comments for virtblk_cleanup_broken_device()
- Moved the broken vq check in virtblk_remove()
- Fixed comment style to have first empty line
- replaced freezed to frozen
- Fixed comments rephrased

v2->v3:
- Addressed comments from Michael
- updated comment for synchronizing with callbacks

v1->v2:
- Addressed comments from Stephan
- fixed spelling to 'waiting'
- Addressed comments from Michael
- Dropped checking broken vq from queue_rq() and queue_rqs() because it is checked in lower layer routines in virtio core

v0->v1:
- Fixed comments from Stefan to rename a cleanup function
- Improved logic for handling any outstanding requests in bio layer
- improved cancel callback to sync with ongoing done()

---
 drivers/block/virtio_blk.c | 95 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 95 insertions(+)
drivers/block/virtio_blk.c +95 -0
index 7cffea01d868..c5e383c0ac48 100644
@@ -1554,6 +1554,98 @@ static int virtblk_probe(struct virtio_device *vdev)
 	return err;
 }
 
+/*
+ * If the vq is broken, device will not complete requests.
+ * So we do it for the device.
+ */
+static bool virtblk_complete_request_with_ioerr(struct request *rq, void *data)
+{
+	struct virtblk_req *vbr = blk_mq_rq_to_pdu(rq);
+	struct virtio_blk *vblk = data;
+	struct virtio_blk_vq *vq;
+	unsigned long flags;
+
+	vq = &vblk->vqs[rq->mq_hctx->queue_num];
+
+	spin_lock_irqsave(&vq->lock, flags);
+
+	vbr->in_hdr.status = VIRTIO_BLK_S_IOERR;
+	if (blk_mq_request_started(rq) && !blk_mq_request_completed(rq))
+		blk_mq_complete_request(rq);
+
+	spin_unlock_irqrestore(&vq->lock, flags);
+	return true;
+}
+
+/*
+ * If the device is broken, it will not use any buffers and waiting
+ * for that to happen is pointless. We'll do the cleanup in the driver,
+ * completing all requests for the device.
+ */
+static void virtblk_cleanup_broken_device(struct virtio_blk *vblk)
+{
+	struct request_queue *q = vblk->disk->queue;
+
+	/*
+	 * Start freezing the queue, so that new requests keeps waiting at the
+	 * door of bio_queue_enter(). We cannot fully freeze the queue because
+	 * frozen queue is an empty queue and there are pending requests, so
+	 * only start freezing it.
+	 */
+	blk_freeze_queue_start(q);
+
+	/*
+	 * When quiescing completes, all ongoing dispatches have completed
+	 * and no new dispatch will happen towards the driver.
+	 */
+	blk_mq_quiesce_queue(q);
+
+	/*
+	 * Synchronize with any ongoing VQ callbacks that may have started
+	 * before the VQs were marked as broken. Any outstanding requests
+	 * will be completed by virtblk_complete_request_with_ioerr().
+	 */
+	virtio_synchronize_cbs(vblk->vdev);
+
+	/*
+	 * At this point, no new requests can enter the queue_rq() and
+	 * completion routine will not complete any new requests either for the
+	 * broken vq. Hence, it is safe to cancel all requests which are
+	 * started.
+	 */
+	blk_mq_tagset_busy_iter(&vblk->tag_set,
+				virtblk_complete_request_with_ioerr, vblk);
+	blk_mq_tagset_wait_completed_request(&vblk->tag_set);
+
+	/*
+	 * All pending requests are cleaned up. Time to resume so that disk
+	 * deletion can be smooth. Start the HW queues so that when queue is
+	 * unquiesced requests can again enter the driver.
+	 */
+	blk_mq_start_stopped_hw_queues(q, true);
+
+	/*
+	 * Unquiescing will trigger dispatching any pending requests to the
+	 * driver which has crossed bio_queue_enter() to the driver.
+	 */
+	blk_mq_unquiesce_queue(q);
+
+	/*
+	 * Wait for all pending dispatches to terminate which may have been
+	 * initiated after unquiescing.
+	 */
+	blk_mq_freeze_queue_wait(q);
+
+	/*
+	 * Mark the disk dead so that once we unfreeze the queue, requests
+	 * waiting at the door of bio_queue_enter() can be aborted right away.
+	 */
+	blk_mark_disk_dead(vblk->disk);
+
+	/* Unfreeze the queue so that any waiting requests will be aborted. */
+	blk_mq_unfreeze_queue_nomemrestore(q);
+}
+
 static void virtblk_remove(struct virtio_device *vdev)
 {
 	struct virtio_blk *vblk = vdev->priv;
@@ -1561,6 +1653,9 @@ static void virtblk_remove(struct virtio_device *vdev)
 	/* Make sure no work handler is accessing the device. */
 	flush_work(&vblk->config_work);
 
+	if (virtqueue_is_broken(vblk->vqs[0].vq))
+		virtblk_cleanup_broken_device(vblk);
+
 	del_gendisk(vblk->disk);
 	blk_mq_free_tag_set(&vblk->tag_set);
 
Signature · 1 line
2.34.1
#22 lore raw
In reply to Stefan Hajnoczi
Quoted 10 lines · Stefan Hajnoczi
Quoted 3 lines · Parav Pandit

When the PCI device is surprise removed, requests may not complete the device as the VQ is marked as broken. Due to this, the disk deletion hangs.

There are loops in the core virtio driver code that expect device register reads to eventually return 0:
drivers/virtio/virtio_pci_modern.c:vp_reset() drivers/virtio/virtio_pci_modern_dev.c:vp_modern_set_queue_reset()

Quoted 6 lines

Is there a hang if these loops are hit when a device has been surprise removed? I'm trying to understand whether surprise removal is fully supported or whether this patch is one step in that direction.

Apart from that, I'm happy with the virtio_blk.c aspects of the patch:

Is this as simple as this?

-->

---

drivers/virtio/virtio_pci_modern.c +5 -1
index 7182f43ed055..df983fa9046a 100644
@@ -555,8 +555,12 @@ static void vp_reset(struct virtio_device *vdev)
 	 * This will flush out the status write, and flush in device writes,
 	 * including MSI-X interrupts, if any.
 	 */
-	while (vp_modern_get_status(mdev))
+	while (vp_modern_get_status(mdev)) {
+		/* If device is removed meanwhile, it will never respond. */
+		if (!pci_device_is_present(vp_dev->pci_dev))
+			break;
 		msleep(1);
+	}
 
 	vp_modern_avq_cleanup(vdev);
 
drivers/virtio/virtio_pci_modern_dev.c +10 -2
index 0d3dbfaf4b23..7177ce0d63be 100644
@@ -523,11 +523,19 @@ void vp_modern_set_queue_reset(struct virtio_pci_modern_device *mdev, u16 index)
 	vp_iowrite16(index, &cfg->cfg.queue_select);
 	vp_iowrite16(1, &cfg->queue_reset);
 
-	while (vp_ioread16(&cfg->queue_reset))
+	while (vp_ioread16(&cfg->queue_reset)) {
+		/* If device is removed meanwhile, it will never respond. */
+		if (!pci_device_is_present(vp_dev->pci_dev))
+			break;
 		msleep(1);
+	}
 
-	while (vp_ioread16(&cfg->cfg.queue_enable))
+	while (vp_ioread16(&cfg->cfg.queue_enable)) {
+		/* If device is removed meanwhile, it will never respond. */
+		if (!pci_device_is_present(vp_dev->pci_dev))
+			break;
 		msleep(1);
+	}
 }
 EXPORT_SYMBOL_GPL(vp_modern_set_queue_reset);
 
#23 lore raw
Quoted 16 lines

From: Michael S. Tsirkin <[email protected]>
Sent: 25 June 2025 06:09 PM

Quoted 10 lines · Stefan Hajnoczi
Quoted 3 lines · Parav Pandit

When the PCI device is surprise removed, requests may not complete the device as the VQ is marked as broken. Due to this, the disk deletion hangs.

There are loops in the core virtio driver code that expect device register reads to eventually return 0:
drivers/virtio/virtio_pci_modern.c:vp_reset() drivers/virtio/virtio_pci_modern_dev.c:vp_modern_set_queue_reset()

Quoted 9 lines
Quoted 6 lines

Is there a hang if these loops are hit when a device has been surprise removed? I'm trying to understand whether surprise removal is fully supported or whether this patch is one step in that direction.

Apart from that, I'm happy with the virtio_blk.c aspects of the patch:

Is this as simple as this?

Didn't audit the code where else the changes needed.
I had similar change a while ago, but I also recall hitting an assert in the pci layer somewhere during the vp_reset() flow.

Do not have lab access today. Will check back tomorrow and update.

Quoted 54 lines

-->

---

drivers/virtio/virtio_pci_modern.c +5 -1
index 7182f43ed055..df983fa9046a 100644
@@ -555,8 +555,12 @@ static void vp_reset(struct virtio_device *vdev)
 	 * This will flush out the status write, and flush in device writes,
 	 * including MSI-X interrupts, if any.
 	 */
-	while (vp_modern_get_status(mdev))
+	while (vp_modern_get_status(mdev)) {
+		/* If device is removed meanwhile, it will never respond. */
+		if (!pci_device_is_present(vp_dev->pci_dev))
+			break;
 		msleep(1);
+	}
 
 	vp_modern_avq_cleanup(vdev);
 
drivers/virtio/virtio_pci_modern_dev.c +10 -2
index 0d3dbfaf4b23..7177ce0d63be 100644
@@ -523,11 +523,19 @@ void vp_modern_set_queue_reset(struct
 virtio_pci_modern_device *mdev, u16 index)
 	vp_iowrite16(index, &cfg->cfg.queue_select);
 	vp_iowrite16(1, &cfg->queue_reset);
 
-	while (vp_ioread16(&cfg->queue_reset))
+	while (vp_ioread16(&cfg->queue_reset)) {
+		/* If device is removed meanwhile, it will never respond. */
+		if (!pci_device_is_present(vp_dev->pci_dev))
+			break;
 		msleep(1);
+	}
 
-	while (vp_ioread16(&cfg->cfg.queue_enable))
+	while (vp_ioread16(&cfg->cfg.queue_enable)) {
+		/* If device is removed meanwhile, it will never respond. */
+		if (!pci_device_is_present(vp_dev->pci_dev))
+			break;
 		msleep(1);
+	}
 }
 EXPORT_SYMBOL_GPL(vp_modern_set_queue_reset);