virtio short improvements
cover virtio short improvements
Hi,
This series contains small improvements for virtio pci driver. The main idea support surprise removal of virtio pci device.
Patches 1 to 3 prepare the code to handle surprise removal by marking the device as broken in patch-4.
Patch summary:
patch-1: ensures that compiler optimization doesn't occur on vq->broken bit
patch-2: maintains the mirror sequence on VQ delete and VQ create
patch-3: protects vqs list for simultaneous access from reader and a writer
patch-4: handles surprise removal of virtio pci device which avoids
call trace and system lockup
Parav Pandit (4):
virtio: Improve vq->broken access to avoid any compiler optimization
virtio: Keep vring_del_virtqueue() mirror of VQ create
virtio: Protect vqs list access
virtio_pci: Support surprise removal of virtio pci device
drivers/virtio/virtio.c | 1 + drivers/virtio/virtio_pci_common.c | 7 +++++++ drivers/virtio/virtio_ring.c | 17 ++++++++++++++--- include/linux/virtio.h | 1 + 4 files changed, 23 insertions(+), 3 deletions(-)
Signature · 6 lines
2.27.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
PATCH 1/4 virtio: Improve vq->broken access to avoid any compiler optimization
Currently vq->broken field is read by virtqueue_is_broken() in busy loop in one context by virtnet_send_command().
vq->broken is set to true in other process context by virtio_break_device(). Reader and writer are accessing it without any synchronization. This may lead to a compiler optimization which may result to optimize reading vq->broken only once.
Hence, force reading vq->broken on each invocation of virtqueue_is_broken() and ensure that its update is visible.
--- drivers/virtio/virtio_ring.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/virtio/virtio_ring.c +4 -2
@@ -2373,7 +2373,7 @@ bool virtqueue_is_broken(struct virtqueue *_vq) {struct vring_virtqueue *vq =to_vvq (_vq); -return vq->broken; +return READ_ONCE (vq->broken); }EXPORT_SYMBOL_GPL (virtqueue_is_broken);
@@ -2387,7 +2387,9 @@ void virtio_break_device(struct virtio_device *dev)list_for_each_entry (_vq, &dev->vqs, list) {struct vring_virtqueue *vq =to_vvq (_vq); - vq->broken = true; + +/* Pairs with READ_ONCE() in virtqueue_is_broken(). */ +smp_store_release (&vq->broken, true); } }EXPORT_SYMBOL_GPL (virtio_break_device);
Signature · 6 lines
2.27.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Michael S. Tsirkin
maintainerRe: [PATCH 1/4] virtio: Improve vq->broken access to avoid any compiler optimization
Quoted 8 lines · Parav Pandit
Currently
vq->brokenfield is read byvirtqueue_is_broken()in busy loop in one context byvirtnet_send_command().
vq->brokenis set to true in other process context byvirtio_break_device(). Reader and writer are accessing it without any synchronization. This may lead to a compiler optimization which may result to optimize readingvq->brokenonly once.
Quoted 4 lines
Hence, force reading
vq->brokenon each invocation ofvirtqueue_is_broken()and ensure that its update is visible.Fixes e2dcdfe95c0b ("virtio: virtio_break_device() to mark all virtqueues broken.")
This is all theoretical right? virtqueue_get_buf is not inlined so compiler generally assumes any vq field can change.
I'm inclined to not include a Fixes
tag then. And please do change subject to say "theoretical" to make that clear to people.
Quoted 19 lines
Signed-off-by Parav Pandit <[email protected]>--- drivers/virtio/virtio_ring.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)drivers/virtio/virtio_ring.c +1
-1@@ -2373,7 +2373,7 @@ bool virtqueue_is_broken(struct virtqueue *_vq) {struct vring_virtqueue *vq =to_vvq (_vq); -return vq->broken; +return READ_ONCE (vq->broken); }EXPORT_SYMBOL_GPL (virtqueue_is_broken);
drivers/virtio/virtio_ring.c void virtio_break_device(struct virtio_device *dev) · line 2392
Quoted 7 lines
drivers/virtio/virtio_ring.c +3
-1@@ -2387,7 +2387,9 @@ void virtio_break_device(struct virtio_device *dev)list_for_each_entry (_vq, &dev->vqs, list) {struct vring_virtqueue *vq =to_vvq (_vq); - vq->broken = true; + +/* Pairs with READ_ONCE() in virtqueue_is_broken(). */ +smp_store_release (&vq->broken, true);
A bit puzzled here. Why do we need release semantics here? IUC store_release does not generally pair with READ_ONCE - does it?
The commit log does not describe this either.
Quoted 5 lines
drivers/virtio/virtio_ring.c +0
-0} }EXPORT_SYMBOL_GPL (virtio_break_device);Signature · 1 line
2.27.0
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
RE: [PATCH 1/4] virtio: Improve vq->broken access to avoid any compiler optimization
Quoted 17 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Sunday, July 18, 2021 2:09 AMQuoted 11 lines · Parav Pandit
Currently
vq->brokenfield is read byvirtqueue_is_broken()in busy loop in one context byvirtnet_send_command().
vq->brokenis set to true in other process context byvirtio_break_device(). Reader and writer are accessing it without any synchronization. This may lead to a compiler optimization which may result to optimize readingvq->brokenonly once.Hence, force reading
vq->brokenon each invocation ofvirtqueue_is_broken()and ensure that its update is visible.
Quoted 6 lines
Quoted 2 lines
Fixes e2dcdfe95c0b ("virtio: virtio_break_device() to mark allvirtqueues broken.")
This is all theoretical right?
virtqueue_get_bufis not inlined so compiler generally assumes any vq field can change.
Broken bit checking cannot rely on some other kernel API for correctness.
So it possibly not hitting this case now, but we shouldn't depend other APIs usage to ensure correctness.
Quoted 5 lines
I'm inclined to not include a Fixes
tag then. And please do change subject to say "theoretical" to make that clear to people.
I do not have any strong opinion on fixes tag. But virtio_broken_queue() API should be self-contained; for that I am not sure if this just theoretical. Do you mean theoretical, because we haven't hit this bug?
Quoted 21 lines
Quoted 21 lines
Signed-off-by Parav Pandit <[email protected]>--- drivers/virtio/virtio_ring.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)(quoted diff) +0
-0
b/drivers/virtio/virtio_ring.cindex89bfe46a8a7f..7f379fe7d78d100644drivers/virtio/virtio_ring.c +1
-1@@ -2373,7 +2373,7 @@ bool virtqueue_is_broken(struct virtqueue *_vq) {struct vring_virtqueue *vq =to_vvq (_vq); -return vq->broken; +return READ_ONCE (vq->broken); }EXPORT_SYMBOL_GPL (virtqueue_is_broken);@@ -2387,7 +2387,9 @@ void virtio_break_device(struct virtio_device *dev)
Quoted 10 lines
drivers/virtio/virtio_ring.c void virtio_break_device(struct virtio_device *dev) · line 2392
Quoted 6 lines
drivers/virtio/virtio_ring.c +3
-1list_for_each_entry (_vq, &dev->vqs, list) {struct vring_virtqueue *vq =to_vvq (_vq); - vq->broken = true; + +/* Pairs with READ_ONCE() in virtqueue_is_broken(). */ +smp_store_release (&vq->broken, true);A bit puzzled here. Why do we need release semantics here? IUC
store_releasedoes not generally pair withREAD_ONCE- does it?
It does; paired at few places, such as,
(a) in uverbs_main.c, default_async_file
(b) in netlink.c, cb_table
(c) fs/dcache.c i_dir_seq,
However, in this scenario, WRITE_ONCE() is enough. So I will simplify and use that in v1.
Quoted 1 line
The commit log does not describe this either.
In commit log I mentioned that "ensure that update is visible". I think a better commit log is, to say: "ensure that broken bit is written". I will send v2 with
(a) updated comment
(b) replacing smp.. with WRITE_ONCE()
(c) dropping the fixes tag.
Quoted 6 lines
Quoted 6 lines
drivers/virtio/virtio_ring.c +0
-0} }
EXPORT_SYMBOL_GPL(virtio_break_device);
--
2.27.0
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Michael S. Tsirkin
maintainerRe: [PATCH 1/4] virtio: Improve vq->broken access to avoid any compiler optimization
Quoted 27 lines · Parav Pandit
Quoted 17 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Sunday, July 18, 2021 2:09 AMQuoted 11 lines · Parav Pandit
Currently
vq->brokenfield is read byvirtqueue_is_broken()in busy loop in one context byvirtnet_send_command().
vq->brokenis set to true in other process context byvirtio_break_device(). Reader and writer are accessing it without any synchronization. This may lead to a compiler optimization which may result to optimize readingvq->brokenonly once.Hence, force reading
vq->brokenon each invocation ofvirtqueue_is_broken()and ensure that its update is visible.Quoted 6 lines
Quoted 2 lines
Fixes e2dcdfe95c0b ("virtio: virtio_break_device() to mark allvirtqueues broken.")
This is all theoretical right?
virtqueue_get_bufis not inlined so compiler generally assumes any vq field can change.Broken bit checking cannot rely on some other kernel API for correctness.
So it possibly not hitting this case now, but we shouldn't depend other APIs usage to ensure correctness.Quoted 1 line
Quoted 6 lines
Quoted 4 lines
I'm inclined to not include a Fixes
tag then. And please do change subject to say "theoretical" to make that clear to people.I do not have any strong opinion on fixes tag. But
virtio_broken_queue()API should be self-contained; for that I am not sure if this just theoretical. Do you mean theoretical, because we haven't hit this bug?
Because with existing code I don't believe existing compilers are clever enough to optimize this away.
Quoted 37 lines
Quoted 21 lines
Quoted 21 lines
Signed-off-by Parav Pandit <[email protected]>--- drivers/virtio/virtio_ring.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)(quoted diff) +0
-0
b/drivers/virtio/virtio_ring.cindex89bfe46a8a7f..7f379fe7d78d100644drivers/virtio/virtio_ring.c +1
-1@@ -2373,7 +2373,7 @@ bool virtqueue_is_broken(struct virtqueue *_vq) {struct vring_virtqueue *vq =to_vvq (_vq); -return vq->broken; +return READ_ONCE (vq->broken); }EXPORT_SYMBOL_GPL (virtqueue_is_broken);@@ -2387,7 +2387,9 @@ void virtio_break_device(struct virtio_device *dev)Quoted 10 lines
drivers/virtio/virtio_ring.c void virtio_break_device(struct virtio_device *dev) · line 2392
Quoted 6 lines
drivers/virtio/virtio_ring.c +3
-1list_for_each_entry (_vq, &dev->vqs, list) {struct vring_virtqueue *vq =to_vvq (_vq); - vq->broken = true; + +/* Pairs with READ_ONCE() in virtqueue_is_broken(). */ +smp_store_release (&vq->broken, true);A bit puzzled here. Why do we need release semantics here? IUC
store_releasedoes not generally pair withREAD_ONCE- does it?It does; paired at few places, such as,
(a) in
uverbs_main.c,default_async_file
(b) innetlink.c,cb_table
(c)fs/dcache.ci_dir_seq,
Quoted 6 lines
However, in this scenario,
WRITE_ONCE()is enough. So I will simplify and use that in v1.Quoted 1 line
The commit log does not describe this either.
In commit log I mentioned that "ensure that update is visible". I think a better commit log is, to say: "ensure that broken bit is written".
"is read repeatedly" maybe.
Quoted 11 lines
I will send v2 with
(a) updated comment
(b) replacing smp.. withWRITE_ONCE()
(c) dropping the fixes tag.Quoted 6 lines
Quoted 6 lines
drivers/virtio/virtio_ring.c +0
-0} }
EXPORT_SYMBOL_GPL(virtio_break_device);
--
2.27.0
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
RE: [PATCH 1/4] virtio: Improve vq->broken access to avoid any compiler optimization
Quoted 38 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Monday, July 19, 2021 5:35 PMQuoted 17 lines · Parav Pandit
Quoted 17 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Sunday, July 18, 2021 2:09 AMQuoted 11 lines · Parav Pandit
Currently
vq->brokenfield is read byvirtqueue_is_broken()in busy loop in one context byvirtnet_send_command().
vq->brokenis set to true in other process context byvirtio_break_device(). Reader and writer are accessing it without any synchronization. This may lead to a compiler optimization which may result to optimize readingvq->brokenonly once.Hence, force reading
vq->brokenon each invocation ofvirtqueue_is_broken()and ensure that its update is visible.Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Fixes e2dcdfe95c0b ("virtio: virtio_break_device() to mark allvirtqueues broken.")
This is all theoretical right?
virtqueue_get_bufis not inlined so compiler generally assumes any vq field can change.Broken bit checking cannot rely on some other kernel API for correctness. So it possibly not hitting this case now, but we shouldn't depend other APIs
usage to ensure correctness.
Quoted 6 lines
Quoted 6 lines
I'm inclined to not include a Fixes
tag then. And please do change subject to say "theoretical" to make that clear to people.
Quoted 6 lines
Quoted 1 line
I do not have any strong opinion on fixes tag. But
virtio_broken_queue()API should be self-contained; for that I am not sure if this just theoretical.
Quoted 1 line
Do you mean theoretical, because we haven't hit this bug?
Because with existing code I don't believe existing compilers are clever enough to optimize this away.
Ok. got it. I will mention in the commit log.
Quoted 41 lines
Quoted 34 lines
Quoted 24 lines
Quoted 24 lines
Signed-off-by Parav Pandit <[email protected]>--- drivers/virtio/virtio_ring.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)(quoted diff) +0
-0
b/drivers/virtio/virtio_ring.cindex89bfe46a8a7f..7f379fe7d78d100644drivers/virtio/virtio_ring.c +1
-1@@ -2373,7 +2373,7 @@ bool virtqueue_is_broken(struct virtqueue *_vq) {struct vring_virtqueue *vq =to_vvq (_vq); -return vq->broken; +return READ_ONCE (vq->broken); }EXPORT_SYMBOL_GPL (virtqueue_is_broken);@@ -2387,7 +2387,9 @@ void virtio_break_device(struct
virtio_device
*dev)Quoted 10 lines
drivers/virtio/virtio_ring.c void virtio_break_device(struct virtio_device *dev) · line 2392
Quoted 6 lines
drivers/virtio/virtio_ring.c +3
-1list_for_each_entry (_vq, &dev->vqs, list) {struct vring_virtqueue *vq =to_vvq (_vq); - vq->broken = true; + +/* Pairs with READ_ONCE() in virtqueue_is_broken(). */ +smp_store_release (&vq->broken, true);A bit puzzled here. Why do we need release semantics here? IUC
store_releasedoes not generally pair withREAD_ONCE- does it?Quoted 7 lines
It does; paired at few places, such as,
(a) in
uverbs_main.c,default_async_file
(b) innetlink.c,cb_table
(c)fs/dcache.ci_dir_seq,However, in this scenario,
WRITE_ONCE()is enough. So I will simplify and
Quoted 8 lines
use that in v1.
Quoted 5 lines
Quoted 3 lines
The commit log does not describe this either.
In commit log I mentioned that "ensure that update is visible". I think a better commit log is, to say: "ensure that broken bit is written".
"is read repeatedly" maybe.
I updated it to below in v2.
" Hence, force reading vq->broken on each invocation ofvirtqueue_is_broken() and also force writing it so that such update is visible to the readers."
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
PATCH 2/4 virtio: Keep vring_del_virtqueue() mirror of VQ create
In reply to Parav Pandit via VirtualizationKeep the vring_del_virtqueue() mirror of the create routines. i.e. to delete list entry first as it is added last during the create routine.
--- drivers/virtio/virtio_ring.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/virtio/virtio_ring.c +2 -1
@@ -2291,6 +2291,8 @@ void vring_del_virtqueue(struct virtqueue *_vq) {struct vring_virtqueue *vq =to_vvq (_vq); +list_del (&_vq->list); +if (vq->we_own_ring) {if (vq->packed_ring) { vring_free_queue(vq->vq.vdev,
@@ -2321,7 +2323,6 @@ void vring_del_virtqueue(struct virtqueue *_vq)kfree (vq->split.desc_state);kfree (vq->split.desc_extra); } -list_del (&_vq->list);kfree (vq); }EXPORT_SYMBOL_GPL (vring_del_virtqueue);
Signature · 6 lines
2.27.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
PATCH 3/4 virtio: Protect vqs list access
In reply to Parav Pandit via VirtualizationVQs may be accessed to mark the device broken while they are created/destroyed. Hence protect the access to the vqs list.
--- drivers/virtio/virtio.c | 1 + drivers/virtio/virtio_ring.c | 8 ++++++++ include/linux/virtio.h | 1 + 3 files changed, 10 insertions(+)
drivers/virtio/virtio.c +1 -0
@@ -355,6 +355,7 @@ int register_virtio_device(struct virtio_device *dev)virtio_add_status (dev,VIRTIO_CONFIG_S_ACKNOWLEDGE );INIT_LIST_HEAD (&dev->vqs); +rwlock_init (&dev->vqs_list_lock);/* * device_add() causes the bus infrastructure to look for a matching
drivers/virtio/virtio_ring.c +8 -0
@@ -1755,7 +1755,9 @@ static struct virtqueue *vring_create_virtqueue_packed(cpu_to_le16 (vq->packed.event_flags_shadow); } +write_lock (&vdev->vqs_list_lock);list_add_tail (&vq->vq.list, &vdev->vqs); +write_unlock (&vdev->vqs_list_lock);return &vq->vq; err_desc_extra:
@@ -2229,7 +2231,9 @@ struct virtqueue *__vring_new_virtqueue(unsigned int index,memset (vq->split.desc_state,0 , vring.num *sizeof (struct vring_desc_state_split )); +write_lock (&vdev->vqs_list_lock);list_add_tail (&vq->vq.list, &vdev->vqs); +write_unlock (&vdev->vqs_list_lock);return &vq->vq; err_extra:
@@ -2291,7 +2295,9 @@ void vring_del_virtqueue(struct virtqueue *_vq) {struct vring_virtqueue *vq =to_vvq (_vq); +write_lock (&vq->vq.vdev->vqs_list_lock);list_del (&_vq->list); +write_unlock (&vq->vq.vdev->vqs_list_lock);if (vq->we_own_ring) {if (vq->packed_ring) {
@@ -2386,12 +2392,14 @@ void virtio_break_device(struct virtio_device *dev) {struct virtqueue *_vq; +read_lock (&dev->vqs_list_lock);list_for_each_entry (_vq, &dev->vqs, list) {struct vring_virtqueue *vq =to_vvq (_vq);/* Pairs with READ_ONCE() in virtqueue_is_broken(). */ smp_store_release (&vq->broken, true); } +read_unlock (&dev->vqs_list_lock); }EXPORT_SYMBOL_GPL (virtio_break_device);
include/linux/virtio.h +1 -0
@@ -115,6 +115,7 @@ struct virtio_device {const struct virtio_config_ops *config;const struct vringh_config_ops *vringh_config;struct list_head vqs; +rwlock_t vqs_list_lock;u64 features;void *priv; };
Signature · 6 lines
2.27.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Michael S. Tsirkin
maintainerRe: [PATCH 3/4] virtio: Protect vqs list access
Quoted 16 lines · Parav Pandit
VQs may be accessed to mark the device broken while they are created/destroyed. Hence protect the access to the vqs list.
Fixes e2dcdfe95c0b ("virtio: virtio_break_device() to mark all virtqueues broken.")Signed-off-by Parav Pandit <[email protected]>--- drivers/virtio/virtio.c | 1 + drivers/virtio/virtio_ring.c | 8 ++++++++ include/linux/virtio.h | 1 + 3 files changed, 10 insertions(+)drivers/virtio/virtio.c +0
-0
drivers/virtio/virtio.c int register_virtio_device(struct virtio_device *dev) · line 361
Quoted 7 lines
drivers/virtio/virtio.c +1
-0@@ -355,6 +355,7 @@ int register_virtio_device(struct virtio_device *dev)virtio_add_status (dev,VIRTIO_CONFIG_S_ACKNOWLEDGE );INIT_LIST_HEAD (&dev->vqs); +rwlock_init (&dev->vqs_list_lock);/* * device_add() causes the bus infrastructure to look for a matching
Let's just use a simple spinlock. I don't think we are worried about scaling the breaking of devices to multiple CPUs.
Quoted 63 lines
drivers/virtio/virtio_ring.c +8
-0@@ -1755,7 +1755,9 @@ static struct virtqueue *vring_create_virtqueue_packed(cpu_to_le16 (vq->packed.event_flags_shadow); } +write_lock (&vdev->vqs_list_lock);list_add_tail (&vq->vq.list, &vdev->vqs); +write_unlock (&vdev->vqs_list_lock);return &vq->vq; err_desc_extra:@@ -2229,7 +2231,9 @@ struct virtqueue *__vring_new_virtqueue(unsigned int index,memset (vq->split.desc_state,0 , vring.num *sizeof (struct vring_desc_state_split )); +write_lock (&vdev->vqs_list_lock);list_add_tail (&vq->vq.list, &vdev->vqs); +write_unlock (&vdev->vqs_list_lock);return &vq->vq; err_extra:@@ -2291,7 +2295,9 @@ void vring_del_virtqueue(struct virtqueue *_vq) {struct vring_virtqueue *vq =to_vvq (_vq); +write_lock (&vq->vq.vdev->vqs_list_lock);list_del (&_vq->list); +write_unlock (&vq->vq.vdev->vqs_list_lock);if (vq->we_own_ring) {if (vq->packed_ring) {@@ -2386,12 +2392,14 @@ void virtio_break_device(struct virtio_device *dev) {struct virtqueue *_vq; +read_lock (&dev->vqs_list_lock);list_for_each_entry (_vq, &dev->vqs, list) {struct vring_virtqueue *vq =to_vvq (_vq);/* Pairs with READ_ONCE() in virtqueue_is_broken(). */ smp_store_release (&vq->broken, true); } +read_unlock (&dev->vqs_list_lock); }EXPORT_SYMBOL_GPL (virtio_break_device);include/linux/virtio.h +1
-0@@ -115,6 +115,7 @@ struct virtio_device {const struct virtio_config_ops *config;const struct vringh_config_ops *vringh_config;struct list_head vqs; +rwlock_t vqs_list_lock;u64 features;void *priv; };Signature · 1 line
2.27.0
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
RE: [PATCH 3/4] virtio: Protect vqs list access
Quoted 24 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Sunday, July 18, 2021 2:11 AMQuoted 19 lines · Parav Pandit
VQs may be accessed to mark the device broken while they are created/destroyed. Hence protect the access to the vqs list.
Fixes e2dcdfe95c0b ("virtio: virtio_break_device() to mark allvirtqueues broken.")
Signed-off-by Parav Pandit <[email protected]>--- drivers/virtio/virtio.c | 1 + drivers/virtio/virtio_ring.c | 8 ++++++++ include/linux/virtio.h | 1 + 3 files changed, 10 insertions(+)drivers/virtio/virtio.c index renamed from drivers/virtio/virtio.c +0
-0
4b15c00c0a0a..a0d81e35ec4b100644drivers/virtio/virtio.c +0
-0@@ -355,6 +355,7 @@ int register_virtio_device(struct virtio_device *dev)virtio_add_status (dev,VIRTIO_CONFIG_S_ACKNOWLEDGE );
Quoted 9 lines
drivers/virtio/virtio.c int register_virtio_device(struct virtio_device *dev) · line 361
Quoted 5 lines
drivers/virtio/virtio.c +1
-0INIT_LIST_HEAD (&dev->vqs); +rwlock_init (&dev->vqs_list_lock);/* * device_add() causes the bus infrastructure to look for a matching Let's just use a simple spinlock. I don't think we are worried about scaling the breaking of devices to multiple CPUs.
It wasn't the scaling, just wanted to have the clarity on list access. But I realized that rwlock is bigger size too. So yeah, will simplify to spinlock in v2.
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
PATCH 4/4 virtio_pci: Support surprise removal of virtio pci device
In reply to Parav Pandit via VirtualizationWhen a virtio pci device undergo surprise removal (aka async removaln in PCIe spec), mark the device is broken so that any upper layer drivers can abort any outstanding operation.
When a virtio net pci device undergo surprise removal which is used by a NetworkManager, a below call trace was observed.
kernel:watchdog: BUG: soft lockup - CPU#1 stuck for 26s! [kworker/1:1:27059]
watchdog: BUG: soft lockup - CPU#1 stuck for 52s! [kworker/1:1:27059] CPU: 1 PID: 27059 Comm: kworker/1:1 Tainted: G S W I L 5.13.0-hotplug+ #8 Hardware name: Dell Inc. PowerEdge R640/0H28RR, BIOS 2.9.4 11/06/2020 Workqueue: events linkwatch_event RIP: 0010:virtnet_send_command+0xfc/0x150 [virtio_net] Call Trace: virtnet_set_rx_mode+0xcf/0x2a7 [virtio_net] ? __hw_addr_create_ex+0x85/0xc0 __dev_mc_add+0x72/0x80 igmp6_group_added+0xa7/0xd0 ipv6_mc_up+0x3c/0x60 ipv6_find_idev+0x36/0x80 addrconf_add_dev+0x1e/0xa0 addrconf_dev_config+0x71/0x130 addrconf_notify+0x1f5/0xb40 ? rtnl_is_locked+0x11/0x20 ? __switch_to_asm+0x42/0x70 ? finish_task_switch+0xaf/0x2c0 ? raw_notifier_call_chain+0x3e/0x50 raw_notifier_call_chain+0x3e/0x50 netdev_state_change+0x67/0x90 linkwatch_do_dev+0x3c/0x50 __linkwatch_run_queue+0xd2/0x220 linkwatch_event+0x21/0x30 process_one_work+0x1c8/0x370 worker_thread+0x30/0x380 ? process_one_work+0x370/0x370 kthread+0x118/0x140 ? set_kthread_struct+0x40/0x40 ret_from_fork+0x1f/0x30
Hence, add the ability to abort the command on surprise removal which prevents infinite loop and system lockup.
--- drivers/virtio/virtio_pci_common.c | 7 +++++++ 1 file changed, 7 insertions(+)
drivers/virtio/virtio_pci_common.c +7 -0
@@ -576,6 +576,13 @@ static void virtio_pci_remove(struct pci_dev *pci_dev)struct virtio_pci_device *vp_dev =pci_get_drvdata (pci_dev);struct device *dev =get_device (&vp_dev->vdev.dev); +/* +* Device is marked broken on surprise removal so that virtio upper +* layers can abort any ongoing operation. +*/ +if (!pci_device_is_present (pci_dev)) +virtio_break_device (&vp_dev->vdev); +pci_disable_sriov (pci_dev);unregister_virtio_device (&vp_dev->vdev);
Signature · 6 lines
2.27.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Michael S. Tsirkin
maintainerRe: [PATCH 4/4] virtio_pci: Support surprise removal of virtio pci device
Quoted 1 line · Parav Pandit
When a virtio pci device undergo surprise removal (aka async removaln in
typo
Quoted 38 lines
PCIe spec), mark the device is broken so that any upper layer drivers can abort any outstanding operation.
When a virtio net pci device undergo surprise removal which is used by a NetworkManager, a below call trace was observed.
kernel:watchdog: BUG: soft lockup - CPU#1 stuck for 26s! [kworker/1:1:27059]
watchdog: BUG: soft lockup - CPU#1 stuck for 52s! [kworker/1:1:27059] CPU: 1 PID: 27059 Comm: kworker/1:1 Tainted: G S W I L 5.13.0-hotplug+ #8 Hardware name: Dell Inc. PowerEdge R640/0H28RR, BIOS 2.9.4 11/06/2020 Workqueue: events linkwatch_event RIP: 0010:virtnet_send_command+0xfc/0x150 [virtio_net] Call Trace: virtnet_set_rx_mode+0xcf/0x2a7 [virtio_net] ? __hw_addr_create_ex+0x85/0xc0 __dev_mc_add+0x72/0x80 igmp6_group_added+0xa7/0xd0 ipv6_mc_up+0x3c/0x60 ipv6_find_idev+0x36/0x80 addrconf_add_dev+0x1e/0xa0 addrconf_dev_config+0x71/0x130 addrconf_notify+0x1f5/0xb40 ? rtnl_is_locked+0x11/0x20 ? __switch_to_asm+0x42/0x70 ? finish_task_switch+0xaf/0x2c0 ? raw_notifier_call_chain+0x3e/0x50 raw_notifier_call_chain+0x3e/0x50 netdev_state_change+0x67/0x90 linkwatch_do_dev+0x3c/0x50 __linkwatch_run_queue+0xd2/0x220 linkwatch_event+0x21/0x30 process_one_work+0x1c8/0x370 worker_thread+0x30/0x380 ? process_one_work+0x370/0x370 kthread+0x118/0x140 ? set_kthread_struct+0x40/0x40 ret_from_fork+0x1f/0x30
Quoted 4 lines
Hence, add the ability to abort the command on surprise removal which prevents infinite loop and system lockup.
Signed-off-by Parav Pandit <[email protected]>
OK that's nice, but I suspect this is not enough. For example we need to also fix up all config space reads to handle all-ones patterns specially.
E.g.
/* After writing 0 to device_status, the driver MUST wait for a read of
* device_status to return 0 before reinitializing the device.
* This will flush out the status write, and flush in device writes,
* including MSI-X interrupts, if any.
*/
while (vp_modern_get_status(mdev))
msleep(1);
lots of code in drivers needs to be fixed too.
I guess we need to annotate drivers somehow to mark up whether they support surprise removal? And maybe find a way to let host know?
Quoted 24 lines
--- drivers/virtio/virtio_pci_common.c | 7 +++++++ 1 file changed, 7 insertions(+)drivers/virtio/virtio_pci_common.c +7
-0@@ -576,6 +576,13 @@ static void virtio_pci_remove(struct pci_dev *pci_dev)struct virtio_pci_device *vp_dev =pci_get_drvdata (pci_dev);struct device *dev =get_device (&vp_dev->vdev.dev); +/* +* Device is marked broken on surprise removal so that virtio upper +* layers can abort any ongoing operation. +*/ +if (!pci_device_is_present (pci_dev)) +virtio_break_device (&vp_dev->vdev); +pci_disable_sriov (pci_dev);unregister_virtio_device (&vp_dev->vdev);Signature · 1 line
2.27.0
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
RE: [PATCH 4/4] virtio_pci: Support surprise removal of virtio pci device
Quoted 10 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Sunday, July 18, 2021 2:17 AMQuoted 2 lines · Parav Pandit
When a virtio pci device undergo surprise removal (aka async removaln in
typo
Fixing it. Checkpatch.pl and codespell, both didn't catch it. 😊
Quoted 15 lines
OK that's nice, but I suspect this is not enough.
For example we need to also fix up all config space reads to handle all-ones patterns specially.E.g.
/* After writing 0 to
device_status, the driver MUST wait for a read of
*device_statusto return 0 before reinitializing the device.
* This will flush out the status write, and flush in device writes,
* including MSI-X interrupts, if any.
*/
while (vp_modern_get_status(mdev))
msleep(1);
Quoted 1 line
lots of code in drivers needs to be fixed too.
Above one particularly known to us in the hot plug area.
Above fix is needed to close the race of hot plug and unplug happening in parallel, which is occurring today, but less common. It is in my todo list to fix it.
Will take care of it in near future in other series.
Quoted 3 lines
I guess we need to annotate drivers somehow to mark up whether they support surprise removal? And maybe find a way to let host know?
What is the benefit of it? Who can make use of that information?
In virtio pci case, it is similar improvement to what nvme pci driver went through few years back to support hot plug/unplug. Lets complete this of fixes to make it little more robust like nvme.
Quoted 26 lines
drivers/virtio/virtio_pci_common.c static void virtio_pci_remove(struct pci_dev
Quoted 13 lines
--- drivers/virtio/virtio_pci_common.c | 7 +++++++ 1 file changed, 7 insertions(+)drivers/virtio/virtio_pci_common.c +0
-0@@ -576,6 +576,13 @@ static void virtio_pci_remove(struct pci_dev
*pci_dev)Quoted 12 lines
drivers/virtio/virtio_pci_common.c +7
-0struct virtio_pci_device *vp_dev =pci_get_drvdata (pci_dev);struct device *dev =get_device (&vp_dev->vdev.dev); +/* +* Device is marked broken on surprise removal so that virtio upper +* layers can abort any ongoing operation. +*/ +if (!pci_device_is_present (pci_dev)) +virtio_break_device (&vp_dev->vdev); +pci_disable_sriov (pci_dev);
Quoted 3 lines
Quoted 3 lines
drivers/virtio/virtio_pci_common.c +0
-1unregister_virtio_device (&vp_dev->vdev); --2.27.0
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Michael S. Tsirkin
maintainerRe: [PATCH 4/4] virtio_pci: Support surprise removal of virtio pci device
Quoted 34 lines · Parav Pandit
Quoted 10 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Sunday, July 18, 2021 2:17 AMQuoted 2 lines · Parav Pandit
When a virtio pci device undergo surprise removal (aka async removaln in
typo
Fixing it. Checkpatch.pl and codespell, both didn't catch it. 😊
Quoted 15 lines
OK that's nice, but I suspect this is not enough.
For example we need to also fix up all config space reads to handle all-ones patterns specially.E.g.
/* After writing 0 to
device_status, the driver MUST wait for a read of
*device_statusto return 0 before reinitializing the device.
* This will flush out the status write, and flush in device writes,
* including MSI-X interrupts, if any.
*/
while (vp_modern_get_status(mdev))
msleep(1);Quoted 1 line
lots of code in drivers needs to be fixed too.
Above one particularly known to us in the hot plug area.
Above fix is needed to close the race of hot plug and unplug happening in parallel, which is occurring today, but less common. It is in my todo list to fix it.
Will take care of it in near future in other series.Quoted 1 line
Quoted 3 lines
Quoted 2 lines
I guess we need to annotate drivers somehow to mark up whether they support surprise removal? And maybe find a way to let host know?
What is the benefit of it? Who can make use of that information?
For example, host could avoid removing devices by hot removal if guest is not ready for it. Or libosinfo could use that to tell users what to do.
Quoted 2 lines
In virtio pci case, it is similar improvement to what nvme pci driver went through few years back to support hot plug/unplug. Lets complete this of fixes to make it little more robust like nvme.
At least please mention in commit log that it's incomplete.
Quoted 26 lines
Quoted 26 lines
drivers/virtio/virtio_pci_common.c static void virtio_pci_remove(struct pci_dev
Quoted 13 lines
--- drivers/virtio/virtio_pci_common.c | 7 +++++++ 1 file changed, 7 insertions(+)drivers/virtio/virtio_pci_common.c +0
-0@@ -576,6 +576,13 @@ static void virtio_pci_remove(struct pci_dev
*pci_dev)Quoted 12 lines
drivers/virtio/virtio_pci_common.c +7
-0struct virtio_pci_device *vp_dev =pci_get_drvdata (pci_dev);struct device *dev =get_device (&vp_dev->vdev.dev); +/* +* Device is marked broken on surprise removal so that virtio upper +* layers can abort any ongoing operation. +*/ +if (!pci_device_is_present (pci_dev)) +virtio_break_device (&vp_dev->vdev); +pci_disable_sriov (pci_dev);
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines
drivers/virtio/virtio_pci_common.c +0
-1unregister_virtio_device (&vp_dev->vdev); --2.27.0
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
RE: [PATCH 4/4] virtio_pci: Support surprise removal of virtio pci device
Quoted 41 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Monday, July 19, 2021 5:37 PMQuoted 27 lines · Parav Pandit
Quoted 10 lines
From: Michael S. Tsirkin <[email protected]>
Sent: Sunday, July 18, 2021 2:17 AMQuoted 2 lines · Parav Pandit
When a virtio pci device undergo surprise removal (aka async removaln in
typo
Fixing it. Checkpatch.pl and codespell, both didn't catch it. 😊
Quoted 15 lines
OK that's nice, but I suspect this is not enough.
For example we need to also fix up all config space reads to handle all-ones patterns specially.E.g.
/* After writing 0 to
device_status, the driver MUST wait for a read of
*device_statusto return 0 before reinitializing the device.
* This will flush out the status write, and flush in device writes,
* including MSI-X interrupts, if any.
*/
while (vp_modern_get_status(mdev))
msleep(1);Quoted 3 lines
Quoted 1 line
lots of code in drivers needs to be fixed too.
Above one particularly known to us in the hot plug area. Above fix is needed to close the race of hot plug and unplug happening in
parallel, which is occurring today, but less common.
Quoted 4 lines
It is in my todo list to fix it.
Will take care of it in near future in other series.Quoted 1 line
Quoted 6 lines
Quoted 3 lines
Quoted 2 lines
I guess we need to annotate drivers somehow to mark up whether they support surprise removal? And maybe find a way to let host know?
What is the benefit of it? Who can make use of that information?
For example, host could avoid removing devices by hot removal if guest is not ready for it. Or libosinfo could use that to tell users what to do.
Not sure how to achieve it. Because this is decided by the pci hot plug driver at bit early stage. And pci hot plug slot doesn't know what is going to arrive in it.
Quoted 6 lines
Quoted 2 lines
In virtio pci case, it is similar improvement to what nvme pci driver went
through few years back to support hot plug/unplug.
Quoted 1 line
Lets complete this of fixes to make it little more robust like nvme.
At least please mention in commit log that it's incomplete.
This fix is self-contained. I don’t see mentioning about other bugs in this commit log. But sure yes, I will mentioned in the cover letter that more improvements will be done subsequently. Thanks.
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH 4/4] virtio_pci: Support surprise removal of virtio pci device
In reply to Michael S. TsirkinQuoted 62 lines · Michael S. Tsirkin
Quoted 1 line · Parav Pandit
When a virtio pci device undergo surprise removal (aka async removaln in
typo
Quoted 38 lines
PCIe spec), mark the device is broken so that any upper layer drivers can abort any outstanding operation.
When a virtio net pci device undergo surprise removal which is used by a NetworkManager, a below call trace was observed.
kernel:watchdog: BUG: soft lockup - CPU#1 stuck for 26s! [kworker/1:1:27059]
watchdog: BUG: soft lockup - CPU#1 stuck for 52s! [kworker/1:1:27059] CPU: 1 PID: 27059 Comm: kworker/1:1 Tainted: G S W I L 5.13.0-hotplug+ #8 Hardware name: Dell Inc. PowerEdge R640/0H28RR, BIOS 2.9.4 11/06/2020 Workqueue: events linkwatch_event RIP: 0010:virtnet_send_command+0xfc/0x150 [virtio_net] Call Trace: virtnet_set_rx_mode+0xcf/0x2a7 [virtio_net] ? __hw_addr_create_ex+0x85/0xc0 __dev_mc_add+0x72/0x80 igmp6_group_added+0xa7/0xd0 ipv6_mc_up+0x3c/0x60 ipv6_find_idev+0x36/0x80 addrconf_add_dev+0x1e/0xa0 addrconf_dev_config+0x71/0x130 addrconf_notify+0x1f5/0xb40 ? rtnl_is_locked+0x11/0x20 ? __switch_to_asm+0x42/0x70 ? finish_task_switch+0xaf/0x2c0 ? raw_notifier_call_chain+0x3e/0x50 raw_notifier_call_chain+0x3e/0x50 netdev_state_change+0x67/0x90 linkwatch_do_dev+0x3c/0x50 __linkwatch_run_queue+0xd2/0x220 linkwatch_event+0x21/0x30 process_one_work+0x1c8/0x370 worker_thread+0x30/0x380 ? process_one_work+0x370/0x370 kthread+0x118/0x140 ? set_kthread_struct+0x40/0x40 ret_from_fork+0x1f/0x30Quoted 4 lines
Hence, add the ability to abort the command on surprise removal which prevents infinite loop and system lockup.
Signed-off-by Parav Pandit <[email protected]>OK that's nice, but I suspect this is not enough. For example we need to also fix up all config space reads to handle all-ones patterns specially.
E.g.
/* After writing 0 to
device_status, the driver MUST wait for a read of
*device_statusto return 0 before reinitializing the device.
* This will flush out the status write, and flush in device writes,
* including MSI-X interrupts, if any.
*/
while (vp_modern_get_status(mdev))
msleep(1);
Quoted 5 lines
lots of code in drivers needs to be fixed too.
I guess we need to annotate drivers somehow to mark up whether they support surprise removal? And maybe find a way to let host know?
I'm wondering whether virtio-pci surprise removal would need more support in drivers than virtio-ccw surprise removal; given that virtio-ccw *only* supports surprise removal and I don't remember any problem reports, the situation is probably not that bad.
Is surprise removal of block devices still a big problem? We have some support for (non-virtio) ccw devices (e.g. dasd) via a 'disconnected' state that was designed to mitigate problems with block devices that are suddenly gone.
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization