Revert "virtio_pci: Support surprise removal of virtio pci
RFC 1/1 Revert "virtio_pci: Support surprise removal of virtio pci
Revert "virtio_pci: Support surprise removal of virtio pci device"
This reverts commit 43bb40c5b92659966bdf4bfe584fde0a3575a049.
Marking the device as broken will cause the uncompleted IO request on virtio-blk since virtblk_done will not continue when it find the broken virtqueu at last it will cause the failure of removal of virtio-blk device because of uncompleted IO request
The correct fix for the issue that commit 43bb40c5b9 ("virtio_pci:
Support surprise removal of virtio pci device") tried to fix is that virtio backend always complete the IO request as soon as possible
--- drivers/virtio/virtio_pci_common.c | 7 ------- 1 file changed, 7 deletions(-)
drivers/virtio/virtio_pci_common.c +0 -7
@@ -584,13 +584,6 @@ static void virtio_pci_remove(struct pci_dev *pci_dev)struct virtio_pci_device *vp_dev =pci_get_drvdata (pci_dev);struct device *dev =get_device (&vp_dev->vdev.dev); -/* -* Device is marked broken on surprise removal so that virtio upper -* layers can abort any ongoing operation. -*/ -if (!pci_device_is_present (pci_dev)) -virtio_break_device (&vp_dev->vdev); -pci_disable_sriov (pci_dev);unregister_virtio_device (&vp_dev->vdev);
--
2.9.4
Quoted 8 lines
From: Li,Rongqing <[email protected]>
Sent: Thursday, January 11, 2024 2:45 PMRevert "
virtio_pci: Support surprise removal of virtio pci device"This reverts commit
43bb40c5b92659966bdf4bfe584fde0a3575a049.
Quoted 9 lines
Marking the device as broken will cause the uncompleted IO request on virtio-blk since
virtblk_donewill not continue when it find the broken virtqueu at last it will cause the failure of removal of virtio-blk device because of uncompleted IO requestThe correct fix for the issue that commit
43bb40c5b9("virtio_pci:
Support surprise removal of virtio pci device") tried to fix is that virtio backend always complete the IO request as soon as possible
This can never be guaranteed by a device and by pci spec given it is surprise removal by definition.
For Linux virtio blk is not the only block device which has surprised removal. Nvme blk driver has supported surprise removal for several years now. I am sure nbd and others would have too given its network.
I am not familiar with the blk driver stack.
So please explore with blk community of how to complete an aborted/never completed io which never completed by the device. Since the blk driver knows exactly that the device is removed, it can 100% complete the io to upper layer with the error in deterministic way.
Quoted 27 lines
Signed-off-by Li RongQing <[email protected]>--- drivers/virtio/virtio_pci_common.c | 7 ------- 1 file changed, 7 deletions(-)drivers/virtio/virtio_pci_common.c +0
-7@@ -584,13 +584,6 @@ static void virtio_pci_remove(struct pci_dev *pci_dev)struct virtio_pci_device *vp_dev =pci_get_drvdata (pci_dev);struct device *dev =get_device (&vp_dev->vdev.dev); -/* -* Device is marked broken on surprise removal so that virtio upper -* layers can abort any ongoing operation. -*/ -if (!pci_device_is_present (pci_dev)) -virtio_break_device (&vp_dev->vdev); -pci_disable_sriov (pci_dev);unregister_virtio_device(&vp_dev->vdev);--
2.9.4
Quoted 36 lines
-----Original Message-----
From: Parav Pandit <[email protected]>
Sent: Thursday, January 11, 2024 6:18 PM
To: Li,Rongqing <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Cc: songyang23 <[email protected]>; liubokai <[email protected]>; Song,Zhan <[email protected]>
Subject: RE: [RFC] Revert "virtio_pci: Support surprise removal of virtio pciQuoted 7 lines
From: Li,Rongqing <[email protected]>
Sent: Thursday, January 11, 2024 2:45 PMRevert "
virtio_pci: Support surprise removal of virtio pci device"This reverts commit
43bb40c5b92659966bdf4bfe584fde0a3575a049.Quoted 9 lines
Marking the device as broken will cause the uncompleted IO request on virtio-blk since
virtblk_donewill not continue when it find the broken virtqueu at last it will cause the failure of removal of virtio-blk device because of uncompleted IO requestThe correct fix for the issue that commit
43bb40c5b9("virtio_pci:
Support surprise removal of virtio pci device") tried to fix is that virtio backend always complete the IO request as soon as possibleThis can never be guaranteed by a device and by pci spec given it is surprise removal by definition.
For Linux virtio blk is not the only block device which has surprised removal. Nvme blk driver has supported surprise removal for several years now. I am sure nbd and others would have too given its network.
Quoted 6 lines
I am not familiar with the blk driver stack.
So please explore with blk community of how to complete an aborted/never completed io which never completed by the device.
Since the blk driver knows exactly that the device is removed, it can 100% complete the io to upper layer with the error in deterministic way.
Without the commit 43bb40c5b92("virtio_pci: Support surprise removal of virtio pci device"), My virtio-blk device can be removed successfully, with this commit, and if we use fio to send io requests the virtio-blk, The removal always failed.
Once virt queue is marked as broken, io request can not be finished since lots of virtio functions will check If the vq is broken, if broken, exit directly, like virtqueue_get_buf_ctx_packed
static void *virtqueue_get_buf_ctx_packed(struct virtqueue *_vq,
unsigned int *len,
void **ctx)
{
struct vring_virtqueue *vq = to_vvq(_vq);
u16 last_used, id, last_used_idx;
bool used_wrap_counter;
void *ret;
START_USE(vq);
if (unlikely(vq->broken)) {
END_USE(vq);
return NULL;
} Nvme is different from virtio, it can ensure that all requests are flushed to completion, like below patch:
commit 1d39e6928cbd0eb737c51545210b5186d5551ba1
Author: Keith Busch <[email protected]> Date: Wed Jun 6 08:13:08 2018 -0600
nvme-pci: unquiesce dead controller queues
This patch ensures the nvme namsepace request queues are not quiesced on a surprise removal. It's possible the queues were previously killed in a failed reset, so the queues need to be unquiesced to ensure all requests are flushed to completion.
-Li
Quoted 49 lines
From: Li,Rongqing <[email protected]>
Sent: Thursday, January 11, 2024 5:18 PMQuoted 7 lines
-----Original Message-----
From: Parav Pandit <[email protected]>
Sent: Thursday, January 11, 2024 6:18 PM
To: Li,Rongqing <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Cc: songyang23 <[email protected]>; liubokaiQuoted 28 lines
Song,Zhan <[email protected]>
Subject: RE: [RFC] Revert "virtio_pci: Support surprise removal of virtio pciQuoted 7 lines
From: Li,Rongqing <[email protected]>
Sent: Thursday, January 11, 2024 2:45 PMRevert "
virtio_pci: Support surprise removal of virtio pci device"This reverts commit
43bb40c5b92659966bdf4bfe584fde0a3575a049.Quoted 9 lines
Marking the device as broken will cause the uncompleted IO request on virtio-blk since
virtblk_donewill not continue when it find the broken virtqueu at last it will cause the failure of removal of virtio-blk device because of uncompleted IO requestThe correct fix for the issue that commit
43bb40c5b9("virtio_pci:
Support surprise removal of virtio pci device") tried to fix is that virtio backend always complete the IO request as soon as possibleThis can never be guaranteed by a device and by pci spec given it is surprise removal by definition.
For Linux virtio blk is not the only block device which has surprised removal. Nvme blk driver has supported surprise removal for several years now. I am sure nbd and others would have too given its network.
Quoted 6 lines
I am not familiar with the blk driver stack.
So please explore with blk community of how to complete an aborted/never completed io which never completed by the device. Since the blk driver knows exactly that the device is removed, it can 100% complete the io to upper layer with the error in deterministic way.
Quoted 5 lines
Without the commit 43bb40c5b92("virtio_pci: Support surprise removal of virtio pci device"), My virtio-blk device can be removed successfully, with this commit, and if we use fio to send io requests the virtio-blk, The removal always failed.
How can it complete when the device is not going complete the io ever?
Quoted 22 lines
Once virt queue is marked as broken, io request can not be finished since lots of virtio functions
will check If the vq is broken, if broken, exit directly, likevirtqueue_get_buf_ctx_packedstatic void *virtqueue_get_buf_ctx_packed(struct virtqueue
*_vq,
unsigned int *len,
void **ctx){ struct vring_virtqueue *vq = to_vvq(_vq); u16 last_used, id, last_used_idx; bool used_wrap_counter; void *ret; START_USE(vq); if (unlikely(vq->broken)) { END_USE(vq); return NULL; }
Quoted 3 lines
Nvme is different from virtio, it can ensure that all requests are flushed to completion, like below patch:
Who is "it" in your above comment? Is it Driver or device?
In below commit who (driver/device) flushes the completions? If it is device, then it is not surprise removal, right?
Quoted 19 lines
commit
1d39e6928cbd0eb737c51545210b5186d5551ba1Author: Keith Busch <[email protected]> Date: Wed Jun 6 08:13:08 2018 -0600nvme-pci: unquiesce dead controller queues
This patch ensures the nvme namsepace request queues are not quiesced on a surprise removal. It's possible the queues were previously killed in a failed reset, so the queues need to be unquiesced to ensure all requests are flushed to completion.
Signed-off-by Keith Busch <[email protected]>Reviewed-by Sagi Grimberg <[email protected]>Reviewed-by Johannes Thumshirn <[email protected]>Signed-off-by Christoph Hellwig <[email protected]>Signed-off-by Jens Axboe <[email protected]>-Li
Quoted 62 lines
-----Original Message-----
From: Parav Pandit <[email protected]>
Sent: Thursday, January 11, 2024 7:55 PM
To: Li,Rongqing <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Cc: songyang23 <[email protected]>; liubokai <[email protected]>; Song,Zhan <[email protected]>
Subject: RE: [RFC] Revert "virtio_pci: Support surprise removal of virtio pciQuoted 49 lines
From: Li,Rongqing <[email protected]>
Sent: Thursday, January 11, 2024 5:18 PMQuoted 7 lines
-----Original Message-----
From: Parav Pandit <[email protected]>
Sent: Thursday, January 11, 2024 6:18 PM
To: Li,Rongqing <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Cc: songyang23 <[email protected]>; liubokaiQuoted 35 lines
Song,Zhan <[email protected]>
Subject: RE: [RFC] Revert "virtio_pci: Support surprise removal of virtio pciQuoted 12 lines
From: Li,Rongqing <[email protected]>
Sent: Thursday, January 11, 2024 2:45 PMRevert "
virtio_pci: Support surprise removal of virtio pci device"This reverts commit
43bb40c5b92659966bdf4bfe584fde0a3575a049.Marking the device as broken will cause the uncompleted IO request on virtio-blk since
virtblk_donewill not continue when it find the broken virtqueu at last it will cause the failure of removal of virtio-blk device because of uncompleted IO requestQuoted 5 lines
The correct fix for the issue that commit
43bb40c5b9("virtio_pci:
Support surprise removal of virtio pci device") tried to fix is that virtio backend always complete the IO request as soon as possibleThis can never be guaranteed by a device and by pci spec given it is surprise removal by definition.
For Linux virtio blk is not the only block device which has surprised removal. Nvme blk driver has supported surprise removal for several years now. I am sure nbd and others would have too given its network.
I am not familiar with the blk driver stack.
So please explore with blk community of how to complete an aborted/never completed io which never completed by the device. Since the blk driver knows exactly that the device is removed, it can 100% complete the io to upper layer with the error in deterministic way.
Quoted 7 lines
Quoted 5 lines
Without the commit 43bb40c5b92("virtio_pci: Support surprise removal of virtio pci device"), My virtio-blk device can be removed successfully, with this commit, and if we use fio to send io requests the virtio-blk, The removal always failed.
How can it complete when the device is not going complete the io ever?
Now this issue is that the virtio backend (device) has completed the io request, but virtio-blk Can not get the completed io request, since the virtqueue is marked as broken.
-Li
Quoted 49 lines
Quoted 21 lines
Once virt queue is marked as broken, io request can not be finished since lots of virtio functions will check If the vq is broken, if broken, exit directly, like
virtqueue_get_buf_ctx_packedstatic void *virtqueue_get_buf_ctx_packed(struct virtqueue
*_vq,
unsigned int *len,
void **ctx){ struct vring_virtqueue *vq = to_vvq(_vq); u16 last_used, id, last_used_idx; bool used_wrap_counter; void *ret; START_USE(vq); if (unlikely(vq->broken)) { END_USE(vq); return NULL; }Quoted 3 lines
Nvme is different from virtio, it can ensure that all requests are flushed to completion, like below patch:
Who is "it" in your above comment? Is it Driver or device?
In below commit who (driver/device) flushes the completions? If it is device, then it is not surprise removal, right?
Quoted 7 lines
commit
1d39e6928cbd0eb737c51545210b5186d5551ba1Author: Keith Busch <[email protected]> Date: Wed Jun 6 08:13:08 2018 -0600nvme-pci: unquiesce dead controller queues
This patch ensures the nvme namsepace request queues are not
quiesced
Quoted 12 lines
on a surprise removal. It's possible the queues were previously killed in a failed reset, so the queues need to be unquiesced to ensure all requests are flushed to completion.
Signed-off-by Keith Busch <[email protected]>Reviewed-by Sagi Grimberg <[email protected]>Reviewed-by Johannes Thumshirn <[email protected]>Signed-off-by Christoph Hellwig <[email protected]>Signed-off-by Jens Axboe <[email protected]>-Li
Quoted 74 lines
From: Li,Rongqing <[email protected]>
Sent: Thursday, January 11, 2024 5:28 PMQuoted 7 lines
-----Original Message-----
From: Parav Pandit <[email protected]>
Sent: Thursday, January 11, 2024 7:55 PM
To: Li,Rongqing <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Cc: songyang23 <[email protected]>; liubokaiQuoted 43 lines
Song,Zhan <[email protected]>
Subject: RE: [RFC] Revert "virtio_pci: Support surprise removal of virtio pciQuoted 38 lines
From: Li,Rongqing <[email protected]>
Sent: Thursday, January 11, 2024 5:18 PMQuoted 7 lines
-----Original Message-----
From: Parav Pandit <[email protected]>
Sent: Thursday, January 11, 2024 6:18 PM
To: Li,Rongqing <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Cc: songyang23 <[email protected]>; liubokaiQuoted 26 lines
Song,Zhan <[email protected]>
Subject: RE: [RFC] Revert "virtio_pci: Support surprise removal of virtio pciQuoted 12 lines
From: Li,Rongqing <[email protected]>
Sent: Thursday, January 11, 2024 2:45 PMRevert "
virtio_pci: Support surprise removal of virtio pci device"This reverts commit
43bb40c5b92659966bdf4bfe584fde0a3575a049.Marking the device as broken will cause the uncompleted IO request on virtio-blk since
virtblk_donewill not continue when it find the broken virtqueu at last it will cause the failure of removal of virtio-blk device because of uncompleted IO requestQuoted 5 lines
The correct fix for the issue that commit
43bb40c5b9("virtio_pci:
Support surprise removal of virtio pci device") tried to fix is that virtio backend always complete the IO request as soon as possibleThis can never be guaranteed by a device and by pci spec given it is surprise removal by definition.
For Linux virtio blk is not the only block device which has surprised
removal.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Nvme blk driver has supported surprise removal for several years now. I am sure nbd and others would have too given its network.
I am not familiar with the blk driver stack.
So please explore with blk community of how to complete an aborted/never completed io which never completed by the device. Since the blk driver knows exactly that the device is removed, it can 100% complete the io to upper layer with the error in deterministicway.
Quoted 8 lines
Quoted 8 lines
Quoted 1 line
Without the commit 43bb40c5b92("virtio_pci: Support surprise removal of virtio pci device"), My virtio-blk device can be removed successfully, with this commit, and if we use fio to send io requests the virtio-blk, The removal always failed.
Quoted 6 lines
Quoted 2 lines
How can it complete when the device is not going complete the io ever?
Now this issue is that the virtio backend (device) has completed the io request, but virtio-blk Can not get the completed io request, since the virtqueue is marked as broken.
Your test was just a timing luck when the io was completed, it is not always the case.
Hence, vblk driver should flush all the pending requests with error to upper layers for which the ios have not yet processed after it was broken because when surprise removal request was received from that point onwards there is no communication with device.
Quoted 52 lines
-Li
Quoted 49 lines
Quoted 21 lines
Once virt queue is marked as broken, io request can not be finished since lots of virtio functions will check If the vq is broken, if broken, exit directly, like
virtqueue_get_buf_ctx_packedstatic void *virtqueue_get_buf_ctx_packed(struct virtqueue
*_vq,
unsigned int *len,
void **ctx){ struct vring_virtqueue *vq = to_vvq(_vq); u16 last_used, id, last_used_idx; bool used_wrap_counter; void *ret; START_USE(vq); if (unlikely(vq->broken)) { END_USE(vq); return NULL; }Quoted 3 lines
Nvme is different from virtio, it can ensure that all requests are flushed to completion, like below patch:
Who is "it" in your above comment? Is it Driver or device?
In below commit who (driver/device) flushes the completions? If it is device, then it is not surprise removal, right?
Quoted 7 lines
commit
1d39e6928cbd0eb737c51545210b5186d5551ba1Author: Keith Busch <[email protected]> Date: Wed Jun 6 08:13:08 2018 -0600nvme-pci: unquiesce dead controller queues
This patch ensures the nvme namsepace request queues are not
quiesced
Quoted 12 lines
on a surprise removal. It's possible the queues were previously killed in a failed reset, so the queues need to be unquiesced to ensure all requests are flushed to completion.
Signed-off-by Keith Busch <[email protected]>Reviewed-by Sagi Grimberg <[email protected]>Reviewed-by Johannes Thumshirn <[email protected]>Signed-off-by Christoph Hellwig <[email protected]>Signed-off-by Jens Axboe <[email protected]>-Li
Michael S. Tsirkin
maintainerQuoted 24 lines · Li,Rongqing
Revert "
virtio_pci: Support surprise removal of virtio pci device"This reverts commit
43bb40c5b92659966bdf4bfe584fde0a3575a049.Marking the device as broken will cause the uncompleted IO request on virtio-blk since
virtblk_donewill not continue when it find the broken virtqueu at last it will cause the failure of removal of virtio-blk device because of uncompleted IO requestThe correct fix for the issue that commit
43bb40c5b9("virtio_pci:
Support surprise removal of virtio pci device") tried to fix is that virtio backend always complete the IO request as soon as possibleSigned-off-by Li RongQing <[email protected]>--- drivers/virtio/virtio_pci_common.c | 7 ------- 1 file changed, 7 deletions(-)drivers/virtio/virtio_pci_common.c +0
-0@@ -584,13 +584,6 @@ static void virtio_pci_remove(struct pci_dev *pci_dev)struct virtio_pci_device *vp_dev =pci_get_drvdata (pci_dev);
drivers/virtio/virtio_pci_common.c static void virtio_pci_remove(struct pci_dev *pci_dev) · line 587
Quoted 10 lines
drivers/virtio/virtio_pci_common.c +0
-7struct device *dev =get_device (&vp_dev->vdev.dev); -/* -* Device is marked broken on surprise removal so that virtio upper -* layers can abort any ongoing operation. -*/ -if (!pci_device_is_present (pci_dev)) -virtio_break_device (&vp_dev->vdev); -pci_disable_sriov (pci_dev);
Maybe break device is not the right thing to do. Indeed, request buffers which have been already used are not completed by driver.
Just removing this is not good since you can then get more requests and the device is gone.
Quoted 3 lines
drivers/virtio/virtio_pci_common.c +0
-1unregister_virtio_device (&vp_dev->vdev); --2.9.4
Quoted 21 lines
drivers/virtio/virtio_pci_common.c static void virtio_pci_remove(struct pci_dev
Quoted 7 lines
drivers/virtio/virtio_pci_common.c +0
-0@@ -584,13 +584,6 @@ static void virtio_pci_remove(struct pci_dev
*pci_dev)drivers/virtio/virtio_pci_common.c static void virtio_pci_remove(struct pci_dev *pci_dev) · line 587
Quoted 11 lines
drivers/virtio/virtio_pci_common.c +0
-7struct virtio_pci_device *vp_dev =pci_get_drvdata (pci_dev);struct device *dev =get_device (&vp_dev->vdev.dev); -/* -* Device is marked broken on surprise removal so that virtio upper -* layers can abort any ongoing operation. -*/ -if (!pci_device_is_present (pci_dev)) -virtio_break_device (&vp_dev->vdev); -pci_disable_sriov (pci_dev);
Quoted 2 lines
Maybe break device is not the right thing to do. Indeed, request buffers which have been already used are not completed by driver.
True, break device makes virtio-blk no longer work for surprise removal
And seems no simple method can work for all devices.
-Li
Quoted 29 lines
From: Li,Rongqing <[email protected]>
Sent: Friday, January 12, 2024 7:59 AMQuoted 22 lines
drivers/virtio/virtio_pci_common.c static void virtio_pci_remove(struct pci_dev
Quoted 6 lines
drivers/virtio/virtio_pci_common.c +0
-0@@ -584,13 +584,6 @@ static void virtio_pci_remove(struct pci_dev
*pci_dev)drivers/virtio/virtio_pci_common.c static void virtio_pci_remove(struct pci_dev *pci_dev) · line 587
Quoted 11 lines
drivers/virtio/virtio_pci_common.c +0
-7struct virtio_pci_device *vp_dev =pci_get_drvdata (pci_dev);struct device *dev =get_device (&vp_dev->vdev.dev); -/* -* Device is marked broken on surprise removal so that virtio upper -* layers can abort any ongoing operation. -*/ -if (!pci_device_is_present (pci_dev)) -virtio_break_device (&vp_dev->vdev); -pci_disable_sriov (pci_dev);Maybe break device is not the right thing to do. Indeed, request buffers which have been already used are not completed by driver.
We can possibly have a virtio_device level flag to indicate device dead/not responsive.
Quoted 3 lines
True, break device makes virtio-blk no longer work for surprise removal
And seems no simple method can work for all devices.
When the VQ is broken or when virtio_device is marked dead, virtblk_remove() should a. flush all the outstanding requests
b. inform the upper layer to not issue more requests
c. for transient requests arriving on MQ, fail them in the request queue callback.