Revert "virtio_pci: Support surprise removal of virtio pci device"
Subsystems (3)
PATCH 1/1 Revert "virtio_pci: Support surprise removal of virtio pci device"
This reverts commit 43bb40c5b926 ("virtio_pci: Support surprise removal of virtio pci device").
Virtio drivers and PCI devices have never fully supported true surprise (aka hot unplug) removal. Drivers historically continued processing and waiting for pending I/O and even continued synchronous device reset during surprise removal. Devices have also continued completing I/Os, doing DMA and allowing device reset after surprise removal to support such drivers.
Supporting it correctly would require a new device capability and driver negotiation in the virtio specification to safely stop
I/O and free queue memory. Failure to do so either breaks all the existing drivers with call trace listed in the commit or crashes the host on continuing the DMA. Hence, until such specification and devices are invented, restore the previous behavior of treating surprise removal as graceful removal to avoid regressions and maintain system stability same as before the
commit 43bb40c5b926 ("virtio_pci: Support surprise removal of virtio pci device").
As explained above, previous analysis of solving this only in driver was incomplete and non-reliable at [1] and at [2]; Hence reverting commit 43bb40c5b926 ("virtio_pci: Support surprise removal of virtio pci device") is still the best stand to restore failures of virtio net and block devices.
[1] https://lore.kernel.org/virtualization/CY8PR12MB719506CC5613EB100BC6C638DCBD2@CY8PR12MB7195.namprd12.prod.outlook.com/#t
[2] https://lore.kernel.org/virtualization/[email protected]/
--- drivers/virtio/virtio_pci_common.c | 7 ------- 1 file changed, 7 deletions(-)
drivers/virtio/virtio_pci_common.c +0 -7
@@ -747,13 +747,6 @@ static void virtio_pci_remove(struct pci_dev *pci_dev)struct virtio_pci_device *vp_dev =pci_get_drvdata (pci_dev);struct device *dev =get_device (&vp_dev->vdev.dev); -/* -* Device is marked broken on surprise removal so that virtio upper -* layers can abort any ongoing operation. -*/ -if (!pci_device_is_present (pci_dev)) -virtio_break_device (&vp_dev->vdev); -pci_disable_sriov (pci_dev);unregister_virtio_device (&vp_dev->vdev);
Signature · 1 line
2.26.2
Michael S. Tsirkin
maintainerQuoted 10 lines · Parav Pandit
This reverts commit
43bb40c5b926("virtio_pci: Support surprise removal of virtio pci device").Virtio drivers and PCI devices have never fully supported true surprise (aka hot unplug) removal. Drivers historically continued processing and waiting for pending I/O and even continued synchronous device reset during surprise removal. Devices have also continued completing I/Os, doing DMA and allowing device reset after surprise removal to support such drivers.
Supporting it correctly would require a new device capability
If a device is removed, it is removed. Windows drivers supported this since forever and it's just a Linux bug that it does not handle all the cases. This is not something you can handle with a capability.
Quoted 5 lines
and
driver negotiation in the virtio specification to safely stop I/O and free queue memory. Failure to do so either breaks all the existing drivers with call trace listed in the commit or crashes the host on continuing the DMA.
If the device is gone, then DMA does not continue.
IIUC what is going on for you, is that you have developed a surprise removal emulation that pretends to remove the device but actually the device is doing DMA. So of course things break then.
Quoted 6 lines
Hence, until such specification and devices
are invented, restore the previous behavior of treating surprise removal as graceful removal to avoid regressions and maintain system stability same as before the
commit43bb40c5b926("virtio_pci: Support surprise removal of virtio pci device").
Quoted 7 lines
As explained above, previous analysis of solving this only in driver was incomplete and non-reliable at [1] and at [2]; Hence reverting commit
43bb40c5b926("virtio_pci: Support surprise removal of virtio pci device") is still the best stand to restore failures of virtio net and block devices.
I can only repeat what I said then, this is not how we do kernel development.
Quoted 1 line
[2] https://lore.kernel.org/virtualization/[email protected]/
What was missing here, is handling corner cases. So let us please try to handle them.
Here is how I would try to do it:
- add a new driver callback
- start a periodic timer task in virtio core on remove
- in the timer, probe that the device is still present. if not, invoke a driver callback
- cancel the task on device reset
If you do not have the time, let me know and I will try to look into it.
Quoted 29 lines
Fixes 43bb40c5b926 ("virtio_pci: Support surprise removal of virtio pci device")Reported-by [email protected]Signed-off-by Parav Pandit <[email protected]>--- drivers/virtio/virtio_pci_common.c | 7 ------- 1 file changed, 7 deletions(-)drivers/virtio/virtio_pci_common.c +0
-7@@ -747,13 +747,6 @@ static void virtio_pci_remove(struct pci_dev *pci_dev)struct virtio_pci_device *vp_dev =pci_get_drvdata (pci_dev);struct device *dev =get_device (&vp_dev->vdev.dev); -/* -* Device is marked broken on surprise removal so that virtio upper -* layers can abort any ongoing operation. -*/ -if (!pci_device_is_present (pci_dev)) -virtio_break_device (&vp_dev->vdev); -pci_disable_sriov (pci_dev);unregister_virtio_device (&vp_dev->vdev);Signature · 1 line
2.26.2
Quoted 14 lines
From: Michael S. Tsirkin <[email protected]>
Sent: 22 August 2025 03:52 PMQuoted 1 line · Parav Pandit
This reverts commit
43bb40c5b926("virtio_pci: Support surprise removal ofvirtio pci device").
Quoted 8 lines
Virtio drivers and PCI devices have never fully supported true surprise (aka hot unplug) removal. Drivers historically continued processing and waiting for pending I/O and even continued synchronous device reset during surprise removal. Devices have also continued completing I/Os, doing DMA and allowing device reset after surprise removal to support such drivers.
Quoted 3 lines
Quoted 1 line
Supporting it correctly would require a new device capability
If a device is removed, it is removed.
This is how it was implemented and none of the virtio drivers supported it. So vendors had stepped away from such device implementation. (not just us).
Quoted 2 lines
Windows drivers supported this since
forever and it's just a Linux bug that it does not handle all the cases.
Internal tests showed the opposite that hotplug driver + virtio didn't handle the surprise removal either.
Can you please share the starting version of the Windows driver that actually does not wait for the CVQ commands and outstanding requests for block etc drivers?
We have even seen window driver that corrupts the CVQ command buffer when CVQ command times out!
Quoted 3 lines
This is not
something you can handle with a capability.
The sad truth in my view is that capability is needed so that device can know how to do _sane_ stop as driver told exactly to do so.
For sure if device stops the DMA, the device is unusable until kernel 6.20 or whatever new kernel arrives.
Quoted 4 lines
Quoted 8 lines
Quoted 5 lines
and
driver negotiation in the virtio specification to safely stop I/O and free queue memory. Failure to do so either breaks all the existing drivers with call trace listed in the commit or crashes the host on continuing the DMA.If the device is gone, then DMA does not continue.
This is exactly how we built the device.
And device behaved correctly, it resulted in the call trace on multiple driver hands in net, block area. Commit 43bb40c5b926 partially fixed it.
However, device does not know when to behave correctly vs behave as driver expects.
And to do this reliability a capability is needed.
Quoted 4 lines
IIUC what is going on for you, is that you have developed a surprise removal emulation that pretends to remove the device but actually the device is doing DMA. So of course things break then.
Not really. The device was built correctly to do surprise removal and stop the DMA. That caused the call traces. One of them listed in 43bb40c5b926.
So to maintain backward compatibility, device had to graceful removal as expected by these drivers.
Quoted 13 lines
Quoted 4 lines
Hence, until such specification and devices are invented, restore the previous behavior of treating surprise removal as graceful removal to avoid regressions and maintain system stability same as before the commit
43bb40c5b926("virtio_pci: Support surprise removal of virtio pcidevice").
Quoted 8 lines
As explained above, previous analysis of solving this only in driver was incomplete and non-reliable at [1] and at [2]; Hence reverting commit
43bb40c5b926("virtio_pci: Support surprise removal of virtio pci device") is still the best stand to restore failures of virtio net and block devices.
Quoted 8 lines
Quoted 2 lines
[1]
https://lore.kernel.org/virtualization/CY8PR12MB719506CC5613EB100BC6C6
Quoted 1 line
I can only repeat what I said then, this is not how we do kernel development.
Kernel development does not promote breaking users.
Here users are affected with this incomplete kernel behaviour.
Virtio subsystem (multiple drivers) including marked stable kernel lacks the contract on when to expect DMA completions and when not.
Quoted 7 lines
Quoted 3 lines
[2]
https://lore.kernel.org/virtualization/20250602024358.57114-1-parav@nv
idia.com/What was missing here, is handling corner cases. So let us please try to handle them.
Quoted 8 lines
Here is how I would try to do it:
- add a new driver callback
- start a periodic timer task in virtio core on remove
- in the timer, probe that the device is still present. if not, invoke a driver callback
- cancel the task on device reset
Multiple users request to restore the stability before adding the surprise removal support in reliable way. Lets be practical.
Each net, blk, fs, console, crypto has different way of flushing the pending requests.
It is unlikely to find a single stable kernel where all the virtio devices would have support for above.
So until above grant work is added to new kernel, user deserves to have stability restored in stable kernels.
Hence this request to revert multiple times from the users.
Quoted 36 lines
If you do not have the time, let me know and I will try to look into it.
drivers/virtio/virtio_pci_common.c static void virtio_pci_remove(struct pci_dev
Quoted 18 lines
Fixes 43bb40c5b926 ("virtio_pci: Support surprise removal of virtiopci device")
Reported-by [email protected]Closes:
https://lore.kernel.org/virtualization/c45dd68698cd47238c55fb73ca9b474
[email protected]/Signed-off-by Parav Pandit <[email protected]>--- drivers/virtio/virtio_pci_common.c | 7 ------- 1 file changed, 7 deletions(-)drivers/virtio/virtio_pci_common.c +0
-0@@ -747,13 +747,6 @@ static void virtio_pci_remove(struct pci_dev
*pci_dev)Quoted 15 lines
drivers/virtio/virtio_pci_common.c +0
-8struct virtio_pci_device *vp_dev =pci_get_drvdata (pci_dev);struct device *dev =get_device (&vp_dev->vdev.dev); -/* -* Device is marked broken on surprise removal so that virtio upper -* layers can abort any ongoing operation. -*/ -if (!pci_device_is_present (pci_dev)) -virtio_break_device (&vp_dev->vdev); -pci_disable_sriov (pci_dev);unregister_virtio_device (&vp_dev->vdev); --2.26.2
Michael S. Tsirkin
maintainerQuoted 14 lines · Parav Pandit
Quoted 14 lines
From: Michael S. Tsirkin <[email protected]>
Sent: 22 August 2025 03:52 PMQuoted 1 line · Parav Pandit
This reverts commit
43bb40c5b926("virtio_pci: Support surprise removal ofvirtio pci device").
Quoted 8 lines
Virtio drivers and PCI devices have never fully supported true surprise (aka hot unplug) removal. Drivers historically continued processing and waiting for pending I/O and even continued synchronous device reset during surprise removal. Devices have also continued completing I/Os, doing DMA and allowing device reset after surprise removal to support such drivers.
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Supporting it correctly would require a new device capability
If a device is removed, it is removed.
This is how it was implemented and none of the virtio drivers supported it. So vendors had stepped away from such device implementation. (not just us).
If the slot does not have a mechanical interlock, I can pull the device out. It's not up to a device implementation.
Signature · 1 line
MST
Quoted 22 lines
From: Michael S. Tsirkin <[email protected]>
Sent: 22 August 2025 06:34 PMQuoted 17 lines · Parav Pandit
Quoted 17 lines
From: Michael S. Tsirkin <[email protected]>
Sent: 22 August 2025 03:52 PMQuoted 2 lines · Parav Pandit
This reverts commit
43bb40c5b926("virtio_pci: Support surprise removal ofvirtio pci device").
Quoted 9 lines
Virtio drivers and PCI devices have never fully supported true surprise (aka hot unplug) removal. Drivers historically continued processing and waiting for pending I/O and even continued synchronous device reset during surprise removal. Devices have also continued completing I/Os, doing DMA and allowing device reset after surprise removal to support such drivers.
Supporting it correctly would require a new device capability
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
If a device is removed, it is removed.
This is how it was implemented and none of the virtio drivers supported it. So vendors had stepped away from such device implementation. (not just us).
If the slot does not have a mechanical interlock, I can pull the device out. It's not up to a device implementation.
Sure yes, stack is not there yet to support it.
Each of the virtio device drivers are not there yet.
Lets build that infra, let device indicate it and it will be smooth ride for driver and device.
Michael S. Tsirkin
maintainerQuoted 28 lines · Parav Pandit
Quoted 28 lines
From: Michael S. Tsirkin <[email protected]>
Sent: 22 August 2025 06:34 PMQuoted 21 lines · Parav Pandit
Quoted 15 lines
From: Michael S. Tsirkin <[email protected]>
Sent: 22 August 2025 03:52 PMQuoted 2 lines · Parav Pandit
This reverts commit
43bb40c5b926("virtio_pci: Support surprise removal ofvirtio pci device").
Quoted 8 lines
Virtio drivers and PCI devices have never fully supported true surprise (aka hot unplug) removal. Drivers historically continued processing and waiting for pending I/O and even continued synchronous device reset during surprise removal. Devices have also continued completing I/Os, doing DMA and allowing device reset after surprise removal to support such drivers.
Quoted 3 lines
Quoted 1 line
Supporting it correctly would require a new device capability
If a device is removed, it is removed.
This is how it was implemented and none of the virtio drivers supported it. So vendors had stepped away from such device implementation. (not just us).
Quoted 6 lines
Quoted 2 lines
If the slot does not have a mechanical interlock, I can pull the device out. It's not up to a device implementation.
Sure yes, stack is not there yet to support it.
Each of the virtio device drivers are not there yet.
Lets build that infra, let device indicate it and it will be smooth ride for driver and device.
There is simply no way for the device to "support" for surprise removal, or lack such support thereof. The support is up to the slot, not the device. Any pci compliant device can be placed in a slot that allows surprise removal and that is all. The user can then remove the device. Software can then either recover gracefully - it should - or hang or crash - it does sometimes, now. The patch you are trying to revert is an attempt to move some use-cases from the 1st to the 2nd category.
But what is going on now, as far as I could tell, is that someone developed a surprise removal emulation that does not actually remove the device, and is using that for testing the code in linux that
supports surprise removal. That weird emulation
seems to lead to all kind of weird issues. You answer is to remove the existing code and tell your testing team "we do not support surprise removal".
But just go ahead and tell this to them straight away. You do not need this patch for this.
Or better still, let's fix the issues please.
Signature · 1 line
MST
Quoted 41 lines
From: Michael S. Tsirkin <[email protected]>
Sent: 22 August 2025 07:30 PMQuoted 28 lines · Parav Pandit
Quoted 28 lines
From: Michael S. Tsirkin <[email protected]>
Sent: 22 August 2025 06:34 PMQuoted 21 lines · Parav Pandit
Quoted 15 lines
From: Michael S. Tsirkin <[email protected]>
Sent: 22 August 2025 03:52 PMQuoted 2 lines · Parav Pandit
This reverts commit
43bb40c5b926("virtio_pci: Support surprise removal ofvirtio pci device").
Quoted 8 lines
Virtio drivers and PCI devices have never fully supported true surprise (aka hot unplug) removal. Drivers historically continued processing and waiting for pending I/O and even continued synchronous device reset during surprise removal. Devices have also continued completing I/Os, doing DMA and allowing device reset after surprise removal to support such drivers.
Quoted 3 lines
Quoted 1 line
Supporting it correctly would require a new device capability
If a device is removed, it is removed.
This is how it was implemented and none of the virtio drivers supported it. So vendors had stepped away from such device implementation. (not just us).
Quoted 6 lines
Quoted 2 lines
If the slot does not have a mechanical interlock, I can pull the device out. It's not up to a device implementation.
Sure yes, stack is not there yet to support it.
Each of the virtio device drivers are not there yet.
Lets build that infra, let device indicate it and it will be smooth ride for driverand device.
Quoted 9 lines
There is simply no way for the device to "support" for surprise removal, or lack such support thereof.
The support is up to the slot, not the device. Any pci
compliant device can be placed in a slot that allows surprise removal and that is all. The user can then remove the device.
Software can then either recover gracefully - it should - or hang or crash - it does sometimes, now. The patch you are trying to revert is an attempt to move some use-cases from the 1st to the 2nd category.
It is the driver (and not the device) who needs to tell the device that it will do sane cleanup and not wait infinitely.
Quoted 3 lines
But what is going on now, as far as I could tell, is that someone developed a surprise removal emulation that does not actually remove the device, and is using that for testing the code in linux that supports surprise removal.
Nop. Your analysis is incorrect.
And I explained you that already.
The device implementation supports correct implementation where device stops all the dma and also does not support register access. And no single virtio driver supported that.
On a surprised removed device, driver expects I/Os to complete and this is beyond a 'bug fix' watermark.
Quoted 5 lines
That
weird emulation seems to lead to all kind of weird issues. You answer is to remove the existing code and tell your testing team "we do not support surprise removal".
He he, it is no the device, it is the driver that does not support surprise removal as you can see in your proposed patches and other sw changes.
Quoted 3 lines
But just go ahead and tell this to them straight away. You do not need this patch for this.
It is needed until infrastructure in multiple subsystem is built.
Quoted 3 lines
Or better still, let's fix the issues please.
The implementation is more than a fix category for stable kernels.
Hence, what is asked is to do proper implementation for future kernels and until that point restore the bad use experience.
Quoted 3 lines
--
MST
Michael S. Tsirkin
maintainerQuoted 50 lines · Parav Pandit
Quoted 41 lines
From: Michael S. Tsirkin <[email protected]>
Sent: 22 August 2025 07:30 PMQuoted 28 lines · Parav Pandit
Quoted 28 lines
From: Michael S. Tsirkin <[email protected]>
Sent: 22 August 2025 06:34 PMQuoted 21 lines · Parav Pandit
Quoted 15 lines
From: Michael S. Tsirkin <[email protected]>
Sent: 22 August 2025 03:52 PMQuoted 2 lines · Parav Pandit
This reverts commit
43bb40c5b926("virtio_pci: Support surprise removal ofvirtio pci device").
Quoted 8 lines
Virtio drivers and PCI devices have never fully supported true surprise (aka hot unplug) removal. Drivers historically continued processing and waiting for pending I/O and even continued synchronous device reset during surprise removal. Devices have also continued completing I/Os, doing DMA and allowing device reset after surprise removal to support such drivers.
Quoted 3 lines
Quoted 1 line
Supporting it correctly would require a new device capability
If a device is removed, it is removed.
This is how it was implemented and none of the virtio drivers supported it. So vendors had stepped away from such device implementation. (not just us).
Quoted 6 lines
Quoted 2 lines
If the slot does not have a mechanical interlock, I can pull the device out. It's not up to a device implementation.
Sure yes, stack is not there yet to support it.
Each of the virtio device drivers are not there yet.
Lets build that infra, let device indicate it and it will be smooth ride for driverand device.
Quoted 9 lines
There is simply no way for the device to "support" for surprise removal, or lack such support thereof.
The support is up to the slot, not the device. Any pci
compliant device can be placed in a slot that allows surprise removal and that is all. The user can then remove the device.
Software can then either recover gracefully - it should - or hang or crash - it does sometimes, now. The patch you are trying to revert is an attempt to move some use-cases from the 1st to the 2nd category.
Quoted 1 line
It is the driver (and not the device) who needs to tell the device that it will do sane cleanup and not wait infinitely.
You can invent a way for driver to tell the device that it is not broken. But even if the driver does not do it, nothing at all prevents users from removing the device.
Quoted 8 lines
Quoted 3 lines
But what is going on now, as far as I could tell, is that someone developed a surprise removal emulation that does not actually remove the device, and is using that for testing the code in linux that supports surprise removal.
Nop. Your analysis is incorrect.
And I explained you that already.
The device implementation supports correct implementation where device stops all the dma and also does not support register access. And no single virtio driver supported that.
Quoted 8 lines
On a surprised removed device, driver expects I/Os to complete and this is beyond a 'bug fix' watermark.
Quoted 5 lines
That
weird emulation seems to lead to all kind of weird issues. You answer is to remove the existing code and tell your testing team "we do not support surprise removal".He he, it is no the device, it is the driver that does not support surprise removal as you can see in your proposed patches and other sw changes.
Then fix the driver. Or don't, for that matter, if you lack the time.
Quoted 4 lines
Quoted 3 lines
But just go ahead and tell this to them straight away. You do not need this patch for this.
It is needed until infrastructure in multiple subsystem is built.
What I do not understand, is what good does the revert do. Sorry.
Quoted 5 lines
Quoted 3 lines
Or better still, let's fix the issues please.
The implementation is more than a fix category for stable kernels.
Hence, what is asked is to do proper implementation for future kernels and until that point restore the bad use experience.
I am not at all interested in discussing ease of backporting fixes before they are developed.
Not how we do kernel development, sorry.
Quoted 3 lines
Quoted 3 lines
--
MST
Quoted 67 lines
From: Michael S. Tsirkin <[email protected]>
Sent: 24 August 2025 08:03 PM
To: Parav Pandit <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Max Gurtovoy <[email protected]>; NBU- Contact-Li Rongqing (EXTERNAL) <[email protected]>
Subject: Re: [PATCH] Revert "virtio_pci: Support surprise removal of virtio pci device"Quoted 14 lines · Parav Pandit
Quoted 14 lines
From: Michael S. Tsirkin <[email protected]>
Sent: 22 August 2025 07:30 PMQuoted 9 lines · Parav Pandit
Quoted 9 lines
From: Michael S. Tsirkin <[email protected]>
Sent: 22 August 2025 06:34 PMQuoted 4 lines · Parav Pandit
Quoted 4 lines
From: Michael S. Tsirkin <[email protected]>
Sent: 22 August 2025 03:52 PMOn Fri, Aug 22, 2025 at 12:17:06PM +0300, Parav Pandit wrote:
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
This reverts commit
43bb40c5b926("virtio_pci: Support surprise removal ofvirtio pci device").
Quoted 5 lines
Virtio drivers and PCI devices have never fully supported true surprise (aka hot unplug) removal. Drivers historically continued processing and waiting for pending I/O and even continued synchronous device reset during surprise
removal.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Devices have also continued completing I/Os, doing DMA and allowing device reset after surprise removal to support such
drivers.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines
Supporting it correctly would require a new device capability
If a device is removed, it is removed.
This is how it was implemented and none of the virtio drivers
supported it.
Quoted 22 lines
Quoted 22 lines
Quoted 11 lines
Quoted 6 lines
Quoted 2 lines
So vendors had stepped away from such device implementation. (not just us).
If the slot does not have a mechanical interlock, I can pull the device out. It's not up to a device implementation.
Sure yes, stack is not there yet to support it. Each of the virtio device drivers are not there yet. Lets build that infra, let device indicate it and it will be smooth ride for driver
and device.
There is simply no way for the device to "support" for surprise removal, or lack such support thereof.
The support is up to the slot, not the device. Any pci compliant device can be placed in a slot that allows surprise removal and that is all. The user can then remove the device.
Software can then either recover gracefully - it should - or hang or crash - it does sometimes, now. The patch you are trying to revert is an attempt to move some use-cases from the 1st to the 2nd category.
Quoted 6 lines
Quoted 1 line
It is the driver (and not the device) who needs to tell the device that it will
do sane cleanup and not wait infinitely.
You can invent a way for driver to tell the device that it is not broken. But even if the driver does not do it, nothing at all prevents users from removing the device.
Sure. Vendors will implement the device accordingly based on the driver's config.
If driver tells that it is ready for surprise removal, the device will implement functionality accordingly. And user can do anything it wants.
But user will have indication and knowledge from the device, when not to break the system.
Quoted 20 lines
Quoted 5 lines
Quoted 5 lines
But what is going on now, as far as I could tell, is that someone developed a surprise removal emulation that does not actually remove the device, and is using that for testing the code in linux that supports
surprise removal.
Quoted 3 lines
Nop. Your analysis is incorrect.
And I explained you that already.
The device implementation supports correct implementation where devicestops all the dma and also does not support register access.
Quoted 3 lines
And no single virtio driver supported that.
On a surprised removed device, driver expects I/Os to complete and this is
beyond a 'bug fix' watermark.
Quoted 6 lines
Quoted 6 lines
That
weird emulation seems to lead to all kind of weird issues. You answer is to remove the existing code and tell your testing team "we do not support surprise removal".
Quoted 5 lines
Quoted 1 line
He he, it is no the device, it is the driver that does not support surprise
removal as you can see in your proposed patches and other sw changes.
Then fix the driver. Or don't, for that matter, if you lack the time.
I explained, it is not a fix. It is a completely new implementation of the infrastructure that spans, virtio, pci or core subsystem and each individual device types.
Quoted 7 lines
Quoted 4 lines
Quoted 3 lines
But just go ahead and tell this to them straight away. You do not need this patch for this.
It is needed until infrastructure in multiple subsystem is built.
What I do not understand, is what good does the revert do. Sorry.
Let me explain.
It prevents the issue of vblk requests being stuck due to broken VQ. It prevents the vnet driver start_xmit() to be not stuck on skb completions. And virtio stack works exactly the way it was working before the commit.
Quoted 11 lines
Quoted 5 lines
Quoted 3 lines
Or better still, let's fix the issues please.
The implementation is more than a fix category for stable kernels. Hence, what is asked is to do proper implementation for future kernels and
until that point restore the bad use experience.
I am not at all interested in discussing ease of backporting fixes before they are developed.
Your attribution of "fixing a problem" for the missing new implementation that span across multiple different subsystem (virtio generic, block net) is not accurate. I understand that such confusion arises from your previous email regarding physical removal of the device...
Hope I explained that its virtual system where user has not removed the card physically.
Driver is expecting requests to complete and attempting device reset too...
Quoted 2 lines
Not how we do kernel development, sorry.
The ask is to revert a patch that broke the virtio stack.
No one is asking to backport new implementation and new spec to old kernels.
Michael S. Tsirkin
maintainerQuoted 5 lines · Parav Pandit
Quoted 2 lines
What I do not understand, is what good does the revert do. Sorry.
Let me explain.
It prevents the issue of vblk requests being stuck due to broken VQ. It prevents the vnet driverstart_xmit()to be not stuck on skb completions.
This is the part I don't get. In what scenario, before 43bb40c5b9265 start_xmit is not stuck, but after 43bb40c5b9265 it is stuck?
Once the device is gone, it is not using any buffers at all.
Signature · 1 line
MST
Quoted 11 lines · Michael S. Tsirkin
Quoted 5 lines · Parav Pandit
Quoted 2 lines
What I do not understand, is what good does the revert do. Sorry.
Let me explain.
It prevents the issue of vblk requests being stuck due to broken VQ. It prevents the vnet driverstart_xmit()to be not stuck on skb completions.This is the part I don't get. In what scenario, before
43bb40c5b9265start_xmitis not stuck, but after43bb40c5b9265it is stuck?Once the device is gone, it is not using any buffers at all.
What I also don't understand: virtio-ccw does exactly the same thing (virtio_break_device(), added in 2014), and it supports surprise removal _only_, yet I don't remember seeing bug reports?
Quoted 13 lines
From: Cornelia Huck <[email protected]>
Sent: 27 August 2025 05:04 PMQuoted 7 lines · Michael S. Tsirkin
Quoted 5 lines · Parav Pandit
Quoted 2 lines
What I do not understand, is what good does the revert do. Sorry.
Let me explain.
It prevents the issue of vblk requests being stuck due to broken VQ. It prevents the vnet driverstart_xmit()to be not stuck on skb completions.
Quoted 8 lines
Quoted 4 lines
This is the part I don't get. In what scenario, before
43bb40c5b9265start_xmitis not stuck, but after43bb40c5b9265it is stuck?Once the device is gone, it is not using any buffers at all.
What I also don't understand: virtio-ccw does exactly the same thing (
virtio_break_device(), added in 2014), and it supports surprise removal _only_, yet I don't remember seeing bug reports?
I suspect that stress testing may not have happened for ccw with active vblk Ios and outstanding transmit pkt and cvq commands. Hard to say as we don't have ccw hw or systems.
Quoted 15 lines · Parav Pandit
Quoted 15 lines
From: Cornelia Huck <[email protected]>
Sent: 27 August 2025 05:04 PMQuoted 10 lines · Michael S. Tsirkin
Quoted 5 lines · Parav Pandit
Quoted 2 lines
What I do not understand, is what good does the revert do. Sorry.
Let me explain.
It prevents the issue of vblk requests being stuck due to broken VQ. It prevents the vnet driverstart_xmit()to be not stuck on skb completions.This is the part I don't get. In what scenario, before
43bb40c5b9265start_xmitis not stuck, but after43bb40c5b9265it is stuck?
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Once the device is gone, it is not using any buffers at all.
What I also don't understand: virtio-ccw does exactly the same thing (
virtio_break_device(), added in 2014), and it supports surprise removal _only_, yet I don't remember seeing bug reports?I suspect that stress testing may not have happened for ccw with active vblk Ios and outstanding transmit pkt and cvq commands. Hard to say as we don't have ccw hw or systems.
cc:ing linux-s390 list. I'd be surprised if nobody ever tested surprise removal on a loaded system in the last 11 years.
Michael S. Tsirkin
maintainerQuoted 23 lines · Cornelia Huck
Quoted 21 lines · Parav Pandit
Quoted 20 lines
From: Cornelia Huck <[email protected]>
Sent: 27 August 2025 05:04 PMQuoted 11 lines · Michael S. Tsirkin
Quoted 5 lines · Parav Pandit
Quoted 2 lines
What I do not understand, is what good does the revert do. Sorry.
Let me explain.
It prevents the issue of vblk requests being stuck due to broken VQ. It prevents the vnet driverstart_xmit()to be not stuck on skb completions.This is the part I don't get. In what scenario, before
43bb40c5b9265start_xmitis not stuck, but after43bb40c5b9265it is stuck?Once the device is gone, it is not using any buffers at all.
What I also don't understand: virtio-ccw does exactly the same thing (
virtio_break_device(), added in 2014), and it supports surprise removal _only_, yet I don't remember seeing bug reports?
Quoted 5 lines
Quoted 2 lines
I suspect that stress testing may not have happened for ccw with active vblk Ios and outstanding transmit pkt and cvq commands. Hard to say as we don't have ccw hw or systems.
cc:ing linux-s390 list. I'd be surprised if nobody ever tested surprise removal on a loaded system in the last 11 years.
As it became very clear from follow up discussion, the issue is nothing to do with virtio, it is with a broken hypervisor that allows device to DMA into guest memory while also telling the guest that the device has been removed.
I guess s390 is just not broken like this.
Signature · 1 line
MST
Quoted 31 lines · Michael S. Tsirkin
Quoted 23 lines · Cornelia Huck
Quoted 21 lines · Parav Pandit
Quoted 20 lines
From: Cornelia Huck <[email protected]>
Sent: 27 August 2025 05:04 PMQuoted 11 lines · Michael S. Tsirkin
Quoted 5 lines · Parav Pandit
Quoted 2 lines
What I do not understand, is what good does the revert do. Sorry.
Let me explain.
It prevents the issue of vblk requests being stuck due to broken VQ. It prevents the vnet driverstart_xmit()to be not stuck on skb completions.This is the part I don't get. In what scenario, before
43bb40c5b9265start_xmitis not stuck, but after43bb40c5b9265it is stuck?Once the device is gone, it is not using any buffers at all.
What I also don't understand: virtio-ccw does exactly the same thing (
virtio_break_device(), added in 2014), and it supports surprise removal _only_, yet I don't remember seeing bug reports?Quoted 5 lines
Quoted 2 lines
I suspect that stress testing may not have happened for ccw with active vblk Ios and outstanding transmit pkt and cvq commands. Hard to say as we don't have ccw hw or systems.
cc:ing linux-s390 list. I'd be surprised if nobody ever tested surprise removal on a loaded system in the last 11 years.
Quoted 6 lines
As it became very clear from follow up discussion, the issue is nothing to do with virtio, it is with a broken hypervisor that allows device to DMA into guest memory while also telling the guest that the device has been removed.
I guess s390 is just not broken like this.
Ah good, I missed that -- that indeed sounds broken, and needs to be fixed there.
Quoted 45 lines
From: Cornelia Huck <[email protected]>
Sent: 28 August 2025 05:52 PMQuoted 6 lines · Michael S. Tsirkin
Quoted 5 lines · Cornelia Huck
Quoted 3 lines · Parav Pandit
Quoted 3 lines
From: Cornelia Huck <[email protected]>
Sent: 27 August 2025 05:04 PMQuoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines · Michael S. Tsirkin
Quoted 7 lines · Parav Pandit
Quoted 5 lines
Quoted 2 lines
What I do not understand, is what good does the revert do. Sorry.
Let me explain.
It prevents the issue of vblk requests being stuck due to broken VQ. It prevents the vnet driverstart_xmit()to be not stuck on skbcompletions.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
This is the part I don't get. In what scenario, before
43bb40c5b9265start_xmitis not stuck, but after43bb40c5b9265it isstuck?
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Once the device is gone, it is not using any buffers at all.
What I also don't understand: virtio-ccw does exactly the same thing (
virtio_break_device(), added in 2014), and it supports surprise removal _only_, yet I don't remember seeing bug reports?I suspect that stress testing may not have happened for ccw with active
vblk Ios and outstanding transmit pkt and cvq commands.
Quoted 11 lines
Quoted 4 lines
Quoted 1 line
Hard to say as we don't have ccw hw or systems.
cc:ing linux-s390 list. I'd be surprised if nobody ever tested surprise removal on a loaded system in the last 11 years.
As it became very clear from follow up discussion, the issue is nothing to do with virtio, it is with a broken hypervisor that allows device to DMA into guest memory while also telling the guest that the device has been removed.
Quoted 4 lines
Quoted 1 line
I guess s390 is just not broken like this.
Ah good, I missed that -- that indeed sounds broken, and needs to be fixed there.
Nop. This is not the issue. You missed this focused on fixing the device.
The fact is: the driver is expecting the IOs and CVQ commands and DMA to succeed even after device is removed. The driver is expecting the device reset to also succeed.
Stefan already pointed out this in the vblk driver patches.
This is why you see call traces on del_gendisk(), CVQ commands.
Again, it is the broken drivers not the device.
Device can stop the DMA and stop responding to the requests and kernel 6.X will continue to hang as long as it has cited commit.
Michael S. Tsirkin
maintainerQuoted 51 lines · Parav Pandit
Quoted 45 lines
From: Cornelia Huck <[email protected]>
Sent: 28 August 2025 05:52 PMQuoted 6 lines · Michael S. Tsirkin
Quoted 5 lines · Cornelia Huck
Quoted 3 lines · Parav Pandit
Quoted 3 lines
From: Cornelia Huck <[email protected]>
Sent: 27 August 2025 05:04 PMQuoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines · Michael S. Tsirkin
Quoted 7 lines · Parav Pandit
Quoted 5 lines
Quoted 2 lines
What I do not understand, is what good does the revert do. Sorry.
Let me explain.
It prevents the issue of vblk requests being stuck due to broken VQ. It prevents the vnet driverstart_xmit()to be not stuck on skbcompletions.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
This is the part I don't get. In what scenario, before
43bb40c5b9265start_xmitis not stuck, but after43bb40c5b9265it isstuck?
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Once the device is gone, it is not using any buffers at all.
What I also don't understand: virtio-ccw does exactly the same thing (
virtio_break_device(), added in 2014), and it supports surprise removal _only_, yet I don't remember seeing bug reports?I suspect that stress testing may not have happened for ccw with active
vblk Ios and outstanding transmit pkt and cvq commands.
Quoted 11 lines
Quoted 4 lines
Quoted 1 line
Hard to say as we don't have ccw hw or systems.
cc:ing linux-s390 list. I'd be surprised if nobody ever tested surprise removal on a loaded system in the last 11 years.
As it became very clear from follow up discussion, the issue is nothing to do with virtio, it is with a broken hypervisor that allows device to DMA into guest memory while also telling the guest that the device has been removed.
Quoted 4 lines
Quoted 1 line
I guess s390 is just not broken like this.
Ah good, I missed that -- that indeed sounds broken, and needs to be fixed there.
Nop. This is not the issue. You missed this focused on fixing the device.
Quoted 7 lines
The fact is: the driver is expecting the IOs and CVQ commands and DMA to succeed even after device is removed. The driver is expecting the device reset to also succeed.
Stefan already pointed out this in the vblk driver patches.
This is why you see call traces ondel_gendisk(), CVQ commands.Again, it is the broken drivers not the device.
Device can stop the DMA and stop responding to the requests and kernel 6.X will continue to hang as long as it has cited commit.
Parav, the issues you cite are real but unrelated and will hang anyway with or without the commit.
All you have to do is pull out the device while e.g. a command is in the process of being submitted.
All the commit you want to revert does, is in some instances instead of just hanging it will make queue as broken and release memory. Since you device is not really gone and keeps DMAing into memory, guest memory gets corrupted.
But your argument that the issue is that the fix is "incomplete" is bogus - when we make the fix complete it will become even worse for this broken devices.
Signature · 1 line
MST
Quoted 68 lines
From: Michael S. Tsirkin <[email protected]>
Sent: 28 August 2025 06:31 PMQuoted 13 lines · Parav Pandit
Quoted 13 lines
From: Cornelia Huck <[email protected]>
Sent: 28 August 2025 05:52 PMQuoted 7 lines · Michael S. Tsirkin
Quoted 6 lines · Cornelia Huck
Quoted 4 lines · Parav Pandit
Quoted 4 lines
From: Cornelia Huck <[email protected]>
Sent: 27 August 2025 05:04 PMOn Wed, Aug 27 2025, "Michael S. Tsirkin" <[email protected]>
Quoted 3 lines · wrote:
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines · Parav Pandit
Quoted 1 line
Quoted 1 line
What I do not understand, is what good does the revert do.
Sorry.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Let me explain.
It prevents the issue of vblk requests being stuck due to brokenVQ.
Quoted 29 lines
Quoted 29 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
It prevents the vnet driver
start_xmit()to be not stuck on skbcompletions.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
This is the part I don't get. In what scenario, before
43bb40c5b9265start_xmitis not stuck, but after43bb40c5b9265it isstuck?
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 6 lines
Quoted 2 lines
Once the device is gone, it is not using any buffers at all.
What I also don't understand: virtio-ccw does exactly the same thing (
virtio_break_device(), added in 2014), and it supports surprise removal _only_, yet I don't remember seeing bug reports?I suspect that stress testing may not have happened for ccw with active
vblk Ios and outstanding transmit pkt and cvq commands.
Quoted 11 lines
Quoted 4 lines
Quoted 1 line
Hard to say as we don't have ccw hw or systems.
cc:ing linux-s390 list. I'd be surprised if nobody ever tested surprise removal on a loaded system in the last 11 years.
As it became very clear from follow up discussion, the issue is nothing to do with virtio, it is with a broken hypervisor that allows device to DMA into guest memory while also telling the guest that the device has been removed.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
I guess s390 is just not broken like this.
Ah good, I missed that -- that indeed sounds broken, and needs to be fixed there.
Nop. This is not the issue. You missed this focused on fixing the device.
The fact is: the driver is expecting the IOs and CVQ commands and DMA to
succeed even after device is removed.
Quoted 4 lines
The driver is expecting the device reset to also succeed. Stefan already pointed out this in the vblk driver patches. This is why you see call traces on
del_gendisk(), CVQ commands.
Quoted 8 lines
Quoted 2 lines
Again, it is the broken drivers not the device.
Device can stop the DMA and stop responding to the requests and kernel6.X will continue to hang as long as it has cited commit.
Parav, the issues you cite are real but unrelated and will hang anyway with or without the commit.
How is it unrelated?
If it is going to hang anyway (in your view), and you proposed different callback etc as brand-new feature to Linux kernel, what is the objection to revert it? As you pointed out it will be in multiple subsystems (net, block, pci) etc, why not do the proper work?
Reverting at least helps those stable kernels to operate smoothly as before.
Quoted 8 lines
All you have to do is pull out the device while e.g. a command is in the process of being submitted.
All the commit you want to revert does, is in some instances instead of just hanging it will make queue as broken and release memory. Since you device is not really gone and keeps DMAing into memory, guest memory gets corrupted.
Nop. This is not the case.
What is "some instance"?
The virtio block driver is expecting the IOs to be completed without the cited commit. As you listed cross subsystem callbacks, such infrastructure != fix.
So to make things clear, as discussed.
1. have proper kernel infrastructure in placed as you outlined the design using callback
2. have the spec update to make sure drivers negotiate its readiness for surprise removal and do not expect to access the device.
Until that point, restore the stability of stable kernels.
Quoted 5 lines
But your argument that the issue is that the fix is "incomplete" is bogus -
when we make the fix complete it will become even worse for this broken devices.
I explained you that the device was doing the right thing and that is why exactly the call trace in the cited patch showed up.
Again quoting "broken device" is wrong.
The drivers are broken trying to reset the removed device. And the ask is to do proper feature negotiation to get to that point.
When the reasonable workaround is suggested in previous email, you opted to not respond to it? This is not how broken user experience is restored for stable kernel.
Are you waiting for the test results if it works?
If so, then yes, it makes sense.
I will of course test and submit proper v2.