vhost: Clear the pending messages on vhost_init_device_iotlb()
Subsystems (5)
RFC 1/1 vhost: Clear the pending messages on vhost_init_device_iotlb()
When the vhost iotlb is used along with a guest virtual iommu and the guest gets rebooted, some MISS messages may have been recorded just before the reboot and spuriously executed by the virtual iommu after the reboot. Despite the device iotlb gets re-initialized, the messages are not cleared. Fix that by calling vhost_clear_msg() at the end of vhost_init_device_iotlb().
--- drivers/vhost/vhost.c | 1 + 1 file changed, 1 insertion(+)
drivers/vhost/vhost.c +1 -0
@@ -1751,6 +1751,7 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled) }vhost_iotlb_free (oiotlb); +vhost_clear_msg (d);return 0 ; }
Signature · 6 lines
2.37.3 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Michael S. Tsirkin
maintainerQuoted 17 lines · Eric Auger
When the vhost iotlb is used along with a guest virtual iommu and the guest gets rebooted, some MISS messages may have been recorded just before the reboot and spuriously executed by the virtual iommu after the reboot. Despite the device iotlb gets re-initialized, the messages are not cleared. Fix that by calling
vhost_clear_msg()at the end ofvhost_init_device_iotlb().Signed-off-by Eric Auger <[email protected]>--- drivers/vhost/vhost.c | 1 + 1 file changed, 1 insertion(+)drivers/vhost/vhost.c +0
-0
drivers/vhost/vhost.c int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled) · line 1757
Quoted 7 lines
drivers/vhost/vhost.c +1
-0@@ -1751,6 +1751,7 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled) }vhost_iotlb_free (oiotlb); +vhost_clear_msg (d);return 0 ; }
Hmm. Can't messages meanwhile get processes and affect the new iotlb?
Quoted 2 lines
Signature · 1 line
2.37.3
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Hi Michael,
Quoted 20 lines · Michael S. Tsirkin
Quoted 19 lines · Eric Auger
When the vhost iotlb is used along with a guest virtual iommu and the guest gets rebooted, some MISS messages may have been recorded just before the reboot and spuriously executed by the virtual iommu after the reboot. Despite the device iotlb gets re-initialized, the messages are not cleared. Fix that by calling
vhost_clear_msg()at the end ofvhost_init_device_iotlb().Signed-off-by Eric Auger <[email protected]>--- drivers/vhost/vhost.c | 1 + 1 file changed, 1 insertion(+)drivers/vhost/vhost.c +0
-0@@ -1751,6 +1751,7 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled) }
Quoted 7 lines
drivers/vhost/vhost.c int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled) · line 1757
Quoted 5 lines
drivers/vhost/vhost.c +1
-0vhost_iotlb_free (oiotlb); +vhost_clear_msg (d);return 0 ; }Hmm. Can't messages meanwhile get processes and affect the new iotlb?
Isn't the msg processing stopped at the moment this function is called (VHOST_SET_FEATURES)?
Thanks
Eric
Quoted 4 lines
Quoted 4 lines
Signature · 2 lines
-- 2.37.3
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Michael S. Tsirkin
maintainerQuoted 29 lines · Eric Auger
Hi Michael,
Quoted 20 lines · Michael S. Tsirkin
Quoted 19 lines · Eric Auger
When the vhost iotlb is used along with a guest virtual iommu and the guest gets rebooted, some MISS messages may have been recorded just before the reboot and spuriously executed by the virtual iommu after the reboot. Despite the device iotlb gets re-initialized, the messages are not cleared. Fix that by calling
vhost_clear_msg()at the end ofvhost_init_device_iotlb().Signed-off-by Eric Auger <[email protected]>--- drivers/vhost/vhost.c | 1 + 1 file changed, 1 insertion(+)drivers/vhost/vhost.c +0
-0@@ -1751,6 +1751,7 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled) }Quoted 7 lines
drivers/vhost/vhost.c int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled) · line 1757
Quoted 5 lines
drivers/vhost/vhost.c +1
-0vhost_iotlb_free (oiotlb); +vhost_clear_msg (d);return 0 ; }Hmm. Can't messages meanwhile get processes and affect the new iotlb?
Quoted 6 lines
Isn't the msg processing stopped at the moment this function is called (
VHOST_SET_FEATURES)?Thanks
Eric
It's pretty late here I'm not sure. You tell me what prevents it.
BTW vhost_init_device_iotlb gets enabled parameter but ignores it, we really should drop that.
Also, it looks like if features are set with VIRTIO_F_ACCESS_PLATFORM and then cleared, iotlb is not properly cleared - bug?
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Signature · 2 lines
-- 2.37.3
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Quoted 31 lines · Michael S. Tsirkin
Quoted 29 lines · Eric Auger
Hi Michael,
Quoted 27 lines · Michael S. Tsirkin
Quoted 17 lines · Eric Auger
When the vhost iotlb is used along with a guest virtual iommu and the guest gets rebooted, some MISS messages may have been recorded just before the reboot and spuriously executed by the virtual iommu after the reboot. Despite the device iotlb gets re-initialized, the messages are not cleared. Fix that by calling
vhost_clear_msg()at the end ofvhost_init_device_iotlb().Signed-off-by Eric Auger <[email protected]>--- drivers/vhost/vhost.c | 1 + 1 file changed, 1 insertion(+)drivers/vhost/vhost.c +0
-0drivers/vhost/vhost.c int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled) · line 1757
Quoted 7 lines
drivers/vhost/vhost.c +1
-0@@ -1751,6 +1751,7 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled) }vhost_iotlb_free (oiotlb); +vhost_clear_msg (d);return 0 ; }Hmm. Can't messages meanwhile get processes and affect the new iotlb?
Quoted 8 lines
Quoted 6 lines
Isn't the msg processing stopped at the moment this function is called (
VHOST_SET_FEATURES)?Thanks
Eric
It's pretty late here I'm not sure. You tell me what prevents it.
So the proposed code assumes that Qemu doesn't process device IOTLB before VHOST_SET_FEAETURES. Consider there's no reset in the general vhost uAPI, I wonder if it's better to move the clear to device code like VHOST_NET_SET_BACKEND. So we can clear it per vq?
Quoted 3 lines
BTW
vhost_init_device_iotlbgets enabled parameter but ignores it, we really should drop that.
Yes.
Quoted 3 lines
Also, it looks like if features are set with
VIRTIO_F_ACCESS_PLATFORMand then cleared, iotlb is not properly cleared - bug?
Not sure, old IOTLB may still work. But for safety, we need to disable device IOTLB in this case.
Thanks
Quoted 7 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
--
2.37.3
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Quoted 34 lines · Jason Wang
Quoted 33 lines · Michael S. Tsirkin
Quoted 32 lines · Eric Auger
Hi Michael,
Quoted 20 lines · Michael S. Tsirkin
Quoted 19 lines · Eric Auger
When the vhost iotlb is used along with a guest virtual iommu and the guest gets rebooted, some MISS messages may have been recorded just before the reboot and spuriously executed by the virtual iommu after the reboot. Despite the device iotlb gets re-initialized, the messages are not cleared. Fix that by calling
vhost_clear_msg()at the end ofvhost_init_device_iotlb().Signed-off-by Eric Auger <[email protected]>--- drivers/vhost/vhost.c | 1 + 1 file changed, 1 insertion(+)drivers/vhost/vhost.c +0
-0@@ -1751,6 +1751,7 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled) }Quoted 7 lines
drivers/vhost/vhost.c int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled) · line 1757
Quoted 5 lines
drivers/vhost/vhost.c +1
-0vhost_iotlb_free (oiotlb); +vhost_clear_msg (d);return 0 ; }Hmm. Can't messages meanwhile get processes and affect the new iotlb?
Isn't the msg processing stopped at the moment this function is called (
VHOST_SET_FEATURES)?
Quoted 8 lines
Quoted 4 lines
Quoted 3 lines
Thanks
Eric
It's pretty late here I'm not sure. You tell me what prevents it.
So the proposed code assumes that Qemu doesn't process device IOTLB before
VHOST_SET_FEAETURES. Consider there's no reset in the general vhost uAPI, I wonder if it's better to move the clear to device code likeVHOST_NET_SET_BACKEND. So we can clear it per vq?
OK I will look at this alternative
Quoted 4 lines
Quoted 3 lines
BTW
vhost_init_device_iotlbgets enabled parameter but ignores it, we really should drop that.Yes.
Yes I saw that too. I will send a patch.
Quoted 5 lines
Quoted 3 lines
Also, it looks like if features are set with
VIRTIO_F_ACCESS_PLATFORMand then cleared, iotlb is not properly cleared - bug?Not sure, old IOTLB may still work. But for safety, we need to disable device IOTLB in this case.
OK
Thanks
Eric
Quoted 7 lines
Thanks
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
--
2.37.3
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Michael S. Tsirkin
maintainerQuoted 37 lines · Jason Wang
Quoted 36 lines · Michael S. Tsirkin
Quoted 34 lines · Eric Auger
Hi Michael,
Quoted 20 lines · Michael S. Tsirkin
Quoted 19 lines · Eric Auger
When the vhost iotlb is used along with a guest virtual iommu and the guest gets rebooted, some MISS messages may have been recorded just before the reboot and spuriously executed by the virtual iommu after the reboot. Despite the device iotlb gets re-initialized, the messages are not cleared. Fix that by calling
vhost_clear_msg()at the end ofvhost_init_device_iotlb().Signed-off-by Eric Auger <[email protected]>--- drivers/vhost/vhost.c | 1 + 1 file changed, 1 insertion(+)drivers/vhost/vhost.c +0
-0@@ -1751,6 +1751,7 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled) }Quoted 7 lines
drivers/vhost/vhost.c int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled) · line 1757
Quoted 5 lines
drivers/vhost/vhost.c +1
-0vhost_iotlb_free (oiotlb); +vhost_clear_msg (d);return 0 ; }Hmm. Can't messages meanwhile get processes and affect the new iotlb?
Isn't the msg processing stopped at the moment this function is called (
VHOST_SET_FEATURES)?Thanks
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
Eric
It's pretty late here I'm not sure. You tell me what prevents it.
So the proposed code assumes that Qemu doesn't process device IOTLB before
VHOST_SET_FEAETURES. Consider there's no reset in the general vhost uAPI, I wonder if it's better to move the clear to device code likeVHOST_NET_SET_BACKEND. So we can clear it per vq?
Hmm this makes no sense to me. iommu sits between backend and frontend. Tying one to another is going to backfire.
I'm thinking more along the lines of doing everything under iotlb_lock.
Quoted 7 lines
Quoted 3 lines
BTW
vhost_init_device_iotlbgets enabled parameter but ignores it, we really should drop that.Yes.
Quoted 1 line
Quoted 15 lines
Quoted 2 lines
Also, it looks like if features are set with
VIRTIO_F_ACCESS_PLATFORMand then cleared, iotlb is not properly cleared - bug?Not sure, old IOTLB may still work. But for safety, we need to disable device IOTLB in this case.
Thanks
Quoted 7 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
--
2.37.3
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Quoted 43 lines · Michael S. Tsirkin
Quoted 41 lines · Jason Wang
Quoted 39 lines · Michael S. Tsirkin
Quoted 29 lines · Eric Auger
Hi Michael,
Quoted 20 lines · Michael S. Tsirkin
Quoted 19 lines · Eric Auger
When the vhost iotlb is used along with a guest virtual iommu and the guest gets rebooted, some MISS messages may have been recorded just before the reboot and spuriously executed by the virtual iommu after the reboot. Despite the device iotlb gets re-initialized, the messages are not cleared. Fix that by calling
vhost_clear_msg()at the end ofvhost_init_device_iotlb().Signed-off-by Eric Auger <[email protected]>--- drivers/vhost/vhost.c | 1 + 1 file changed, 1 insertion(+)drivers/vhost/vhost.c +0
-0@@ -1751,6 +1751,7 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled) }Quoted 7 lines
drivers/vhost/vhost.c int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled) · line 1757
Quoted 5 lines
drivers/vhost/vhost.c +1
-0vhost_iotlb_free (oiotlb); +vhost_clear_msg (d);return 0 ; }Hmm. Can't messages meanwhile get processes and affect the new iotlb?
Quoted 6 lines
Isn't the msg processing stopped at the moment this function is called (
VHOST_SET_FEATURES)?Thanks
Eric
It's pretty late here I'm not sure. You tell me what prevents it.
Quoted 7 lines
Quoted 4 lines
So the proposed code assumes that Qemu doesn't process device IOTLB before
VHOST_SET_FEAETURES. Consider there's no reset in the general vhost uAPI, I wonder if it's better to move the clear to device code likeVHOST_NET_SET_BACKEND. So we can clear it per vq?Hmm this makes no sense to me. iommu sits between backend and frontend. Tying one to another is going to backfire.
I think we need to emulate what real devices are doing. Device should clear the page fault message during reset, so the driver won't read anything after reset. But we don't have a per device stop or reset message for vhost-net. That's why the VHOST_NET_SET_BACKEND came into my mind.
Quoted 3 lines
I'm thinking more along the lines of doing everything under
iotlb_lock.
I think the problem is we need to find a proper place to clear the message. So I don't get how iotlb_lock can help: the message could be still read from user space after the backend is set to NULL.
Thanks
Quoted 22 lines
Quoted 22 lines
Quoted 6 lines
BTW
vhost_init_device_iotlbgets enabled parameter but ignores it, we really should drop that.Yes.
Quoted 3 lines
Also, it looks like if features are set with
VIRTIO_F_ACCESS_PLATFORMand then cleared, iotlb is not properly cleared - bug?Not sure, old IOTLB may still work. But for safety, we need to disable device IOTLB in this case.
Thanks
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
--
2.37.3
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Michael S. Tsirkin
maintainerQuoted 49 lines · Jason Wang
Quoted 48 lines · Michael S. Tsirkin
Quoted 45 lines · Jason Wang
Quoted 31 lines · Michael S. Tsirkin
Quoted 29 lines · Eric Auger
Hi Michael,
Quoted 27 lines · Michael S. Tsirkin
Quoted 17 lines · Eric Auger
When the vhost iotlb is used along with a guest virtual iommu and the guest gets rebooted, some MISS messages may have been recorded just before the reboot and spuriously executed by the virtual iommu after the reboot. Despite the device iotlb gets re-initialized, the messages are not cleared. Fix that by calling
vhost_clear_msg()at the end ofvhost_init_device_iotlb().Signed-off-by Eric Auger <[email protected]>--- drivers/vhost/vhost.c | 1 + 1 file changed, 1 insertion(+)drivers/vhost/vhost.c +0
-0drivers/vhost/vhost.c int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled) · line 1757
Quoted 7 lines
drivers/vhost/vhost.c +1
-0@@ -1751,6 +1751,7 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled) }vhost_iotlb_free (oiotlb); +vhost_clear_msg (d);return 0 ; }Hmm. Can't messages meanwhile get processes and affect the new iotlb?
Quoted 8 lines
Quoted 6 lines
Isn't the msg processing stopped at the moment this function is called (
VHOST_SET_FEATURES)?Thanks
Eric
It's pretty late here I'm not sure. You tell me what prevents it.
So the proposed code assumes that Qemu doesn't process device IOTLB before
VHOST_SET_FEAETURES. Consider there's no reset in the general vhost uAPI, I wonder if it's better to move the clear to device code likeVHOST_NET_SET_BACKEND. So we can clear it per vq?
Quoted 8 lines
Quoted 2 lines
Hmm this makes no sense to me. iommu sits between backend and frontend. Tying one to another is going to backfire.
I think we need to emulate what real devices are doing. Device should clear the page fault message during reset, so the driver won't read anything after reset. But we don't have a per device stop or reset message for vhost-net. That's why the
VHOST_NET_SET_BACKENDcame into my mind.
That's not a reset message. Userspace can switch backends at will. I guess we could check when backend is set to -1.
It's a hack but might work.
Quoted 9 lines
Quoted 3 lines
I'm thinking more along the lines of doing everything under
iotlb_lock.I think the problem is we need to find a proper place to clear the message. So I don't get how
iotlb_lockcan help: the message could be still read from user space after the backend is set to NULL.Thanks
Well I think the real problem is this.
vhost_net_set_features does:
if ((features & (1ULL << VIRTIO_F_ACCESS_PLATFORM))) {
if (vhost_init_device_iotlb(&n->dev, true))
goto out_unlock;
}so we get a new iotlb each time features are set.
But features can be changes while device is running. E.g.
VHOST_F_LOG_ALL
Let's just say this hack of reusing feature bits for backend was not my brightest idea :(
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 6 lines
BTW
vhost_init_device_iotlbgets enabled parameter but ignores it, we really should drop that.Yes.
Quoted 3 lines
Also, it looks like if features are set with
VIRTIO_F_ACCESS_PLATFORMand then cleared, iotlb is not properly cleared - bug?Not sure, old IOTLB may still work. But for safety, we need to disable device IOTLB in this case.
Thanks
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
--
2.37.3
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Hi Michael, Jason,
Quoted 75 lines · Michael S. Tsirkin
Quoted 44 lines · Jason Wang
Quoted 43 lines · Michael S. Tsirkin
Quoted 42 lines · Jason Wang
Quoted 30 lines · Michael S. Tsirkin
Quoted 29 lines · Eric Auger
Hi Michael,
Quoted 27 lines · Michael S. Tsirkin
Quoted 17 lines · Eric Auger
When the vhost iotlb is used along with a guest virtual iommu and the guest gets rebooted, some MISS messages may have been recorded just before the reboot and spuriously executed by the virtual iommu after the reboot. Despite the device iotlb gets re-initialized, the messages are not cleared. Fix that by calling
vhost_clear_msg()at the end ofvhost_init_device_iotlb().Signed-off-by Eric Auger <[email protected]>--- drivers/vhost/vhost.c | 1 + 1 file changed, 1 insertion(+)drivers/vhost/vhost.c +0
-0drivers/vhost/vhost.c int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled) · line 1757
Quoted 7 lines
drivers/vhost/vhost.c +1
-0@@ -1751,6 +1751,7 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled) }vhost_iotlb_free (oiotlb); +vhost_clear_msg (d);return 0 ; }Hmm. Can't messages meanwhile get processes and affect the new iotlb?
Quoted 7 lines
Quoted 6 lines
Isn't the msg processing stopped at the moment this function is called (
VHOST_SET_FEATURES)?Thanks
Eric
It's pretty late here I'm not sure. You tell me what prevents it.
So the proposed code assumes that Qemu doesn't process device IOTLB before
VHOST_SET_FEAETURES. Consider there's no reset in the general vhost uAPI, I wonder if it's better to move the clear to device code likeVHOST_NET_SET_BACKEND. So we can clear it per vq?Quoted 7 lines
Quoted 2 lines
Hmm this makes no sense to me. iommu sits between backend and frontend. Tying one to another is going to backfire.
I think we need to emulate what real devices are doing. Device should clear the page fault message during reset, so the driver won't read anything after reset. But we don't have a per device stop or reset message for vhost-net. That's why the
VHOST_NET_SET_BACKENDcame into my mind.That's not a reset message. Userspace can switch backends at will. I guess we could check when backend is set to -1.
It's a hack but might work.Quoted 7 lines
Quoted 2 lines
I'm thinking more along the lines of doing everything under
iotlb_lock.I think the problem is we need to find a proper place to clear the message. So I don't get how
iotlb_lockcan help: the message could be still read from user space after the backend is set to NULL.Thanks
Well I think the real problem is this.
vhost_net_set_featuresdoes:if ((features & (1ULL << VIRTIO_F_ACCESS_PLATFORM))) { if (vhost_init_device_iotlb(&n->dev, true)) goto out_unlock; }so we get a new iotlb each time features are set.
Quoted 8 lines
But features can be changes while device is running. E.g.
VHOST_F_LOG_ALLLet's just say this hack of reusing feature bits for backend was not my brightest idea :(
Isn't vhost_init_device_iotlb() racy then, as d->iotlb is first updated with niotlb and later d->vqs[i]->iotlb is updated with niotlb. What does garantee this is done atomically?
Shouldn't we hold the dev->mutex to make all the sequence atomic and include vhost_clear_msg()? Can't the vhost_clear_msg() take the dev lock?
Thanks
Eric
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 7 lines
BTW
vhost_init_device_iotlbgets enabled parameter but ignores it, we really should drop that.Yes.
Quoted 2 lines
Also, it looks like if features are set with
VIRTIO_F_ACCESS_PLATFORMand then cleared, iotlb is not properly cleared - bug?Not sure, old IOTLB may still work. But for safety, we need to disable device IOTLB in this case.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Thanks
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
--
2.37.3
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Quoted 84 lines · Eric Auger
Hi Michael, Jason,
Quoted 80 lines · Michael S. Tsirkin
Quoted 44 lines · Jason Wang
Quoted 43 lines · Michael S. Tsirkin
Quoted 42 lines · Jason Wang
Quoted 30 lines · Michael S. Tsirkin
Quoted 29 lines · Eric Auger
Hi Michael,
Quoted 27 lines · Michael S. Tsirkin
Quoted 17 lines · Eric Auger
When the vhost iotlb is used along with a guest virtual iommu and the guest gets rebooted, some MISS messages may have been recorded just before the reboot and spuriously executed by the virtual iommu after the reboot. Despite the device iotlb gets re-initialized, the messages are not cleared. Fix that by calling
vhost_clear_msg()at the end ofvhost_init_device_iotlb().Signed-off-by Eric Auger <[email protected]>--- drivers/vhost/vhost.c | 1 + 1 file changed, 1 insertion(+)drivers/vhost/vhost.c +0
-0drivers/vhost/vhost.c int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled) · line 1757
Quoted 7 lines
drivers/vhost/vhost.c +1
-0@@ -1751,6 +1751,7 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled) }vhost_iotlb_free (oiotlb); +vhost_clear_msg (d);return 0 ; }Hmm. Can't messages meanwhile get processes and affect the new iotlb?
Quoted 7 lines
Quoted 6 lines
Isn't the msg processing stopped at the moment this function is called (
VHOST_SET_FEATURES)?Thanks
Eric
It's pretty late here I'm not sure. You tell me what prevents it.
So the proposed code assumes that Qemu doesn't process device IOTLB before
VHOST_SET_FEAETURES. Consider there's no reset in the general vhost uAPI, I wonder if it's better to move the clear to device code likeVHOST_NET_SET_BACKEND. So we can clear it per vq?Quoted 7 lines
Quoted 2 lines
Hmm this makes no sense to me. iommu sits between backend and frontend. Tying one to another is going to backfire.
I think we need to emulate what real devices are doing. Device should clear the page fault message during reset, so the driver won't read anything after reset. But we don't have a per device stop or reset message for vhost-net. That's why the
VHOST_NET_SET_BACKENDcame into my mind.That's not a reset message. Userspace can switch backends at will. I guess we could check when backend is set to -1.
It's a hack but might work.Quoted 7 lines
Quoted 2 lines
I'm thinking more along the lines of doing everything under
iotlb_lock.I think the problem is we need to find a proper place to clear the message. So I don't get how
iotlb_lockcan help: the message could be still read from user space after the backend is set to NULL.Thanks
Well I think the real problem is this.
vhost_net_set_featuresdoes:if ((features & (1ULL << VIRTIO_F_ACCESS_PLATFORM))) { if (vhost_init_device_iotlb(&n->dev, true)) goto out_unlock; }so we get a new iotlb each time features are set.
But features can be changes while device is running. E.g.
VHOST_F_LOG_ALL
Quoted 8 lines
Quoted 3 lines
Let's just say this hack of reusing feature bits for backend was not my brightest idea :(
Isn't
vhost_init_device_iotlb()racy then, asd->iotlbis first updated with niotlb and later d->vqs[i]->iotlb is updated with niotlb. What does garantee this is done atomically?Shouldn't we hold the
dev->mutexto make all the sequence atomic and includevhost_clear_msg()? Can't thevhost_clear_msg()take the dev lock?
It depends on where we want to place the vhost_clear_msg(), e.g in most of the device ioctl, the dev->mutex has been held.
Thanks
Quoted 26 lines
Thanks
Eric
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 7 lines
BTW
vhost_init_device_iotlbgets enabled parameter but ignores it, we really should drop that.Yes.
Quoted 2 lines
Also, it looks like if features are set with
VIRTIO_F_ACCESS_PLATFORMand then cleared, iotlb is not properly cleared - bug?Not sure, old IOTLB may still work. But for safety, we need to disable device IOTLB in this case.
Thanks
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
--
2.37.3
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Hi Jason,
Quoted 87 lines · Jason Wang
Quoted 86 lines · Eric Auger
Hi Michael, Jason,
Quoted 83 lines · Michael S. Tsirkin
Quoted 44 lines · Jason Wang
Quoted 43 lines · Michael S. Tsirkin
Quoted 42 lines · Jason Wang
Quoted 30 lines · Michael S. Tsirkin
Quoted 29 lines · Eric Auger
Hi Michael,
Quoted 27 lines · Michael S. Tsirkin
Quoted 17 lines · Eric Auger
When the vhost iotlb is used along with a guest virtual iommu and the guest gets rebooted, some MISS messages may have been recorded just before the reboot and spuriously executed by the virtual iommu after the reboot. Despite the device iotlb gets re-initialized, the messages are not cleared. Fix that by calling
vhost_clear_msg()at the end ofvhost_init_device_iotlb().Signed-off-by Eric Auger <[email protected]>--- drivers/vhost/vhost.c | 1 + 1 file changed, 1 insertion(+)drivers/vhost/vhost.c +0
-0drivers/vhost/vhost.c int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled) · line 1757
Quoted 7 lines
drivers/vhost/vhost.c +1
-0@@ -1751,6 +1751,7 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled) }vhost_iotlb_free (oiotlb); +vhost_clear_msg (d);return 0 ; }Hmm. Can't messages meanwhile get processes and affect the new iotlb?
Quoted 7 lines
Quoted 6 lines
Isn't the msg processing stopped at the moment this function is called (
VHOST_SET_FEATURES)?Thanks
Eric
It's pretty late here I'm not sure. You tell me what prevents it.
So the proposed code assumes that Qemu doesn't process device IOTLB before
VHOST_SET_FEAETURES. Consider there's no reset in the general vhost uAPI, I wonder if it's better to move the clear to device code likeVHOST_NET_SET_BACKEND. So we can clear it per vq?Quoted 7 lines
Quoted 2 lines
Hmm this makes no sense to me. iommu sits between backend and frontend. Tying one to another is going to backfire.
I think we need to emulate what real devices are doing. Device should clear the page fault message during reset, so the driver won't read anything after reset. But we don't have a per device stop or reset message for vhost-net. That's why the
VHOST_NET_SET_BACKENDcame into my mind.That's not a reset message. Userspace can switch backends at will. I guess we could check when backend is set to -1.
It's a hack but might work.Quoted 7 lines
Quoted 2 lines
I'm thinking more along the lines of doing everything under
iotlb_lock.I think the problem is we need to find a proper place to clear the message. So I don't get how
iotlb_lockcan help: the message could be still read from user space after the backend is set to NULL.Thanks
Well I think the real problem is this.
vhost_net_set_featuresdoes:if ((features & (1ULL << VIRTIO_F_ACCESS_PLATFORM))) { if (vhost_init_device_iotlb(&n->dev, true)) goto out_unlock; }so we get a new iotlb each time features are set.
But features can be changes while device is running. E.g.
VHOST_F_LOG_ALLLet's just say this hack of reusing feature bits for backend was not my brightest idea :(
Quoted 6 lines
Quoted 4 lines
Isn't
vhost_init_device_iotlb()racy then, asd->iotlbis first updated with niotlb and later d->vqs[i]->iotlb is updated with niotlb. What does garantee this is done atomically?Shouldn't we hold the
dev->mutexto make all the sequence atomic and includevhost_clear_msg()? Can't thevhost_clear_msg()take the dev lock?It depends on where we want to place the
vhost_clear_msg(), e.g in most of the device ioctl, thedev->mutexhas been held.
OK, I will double check and respin accordingly
Eric
Quoted 23 lines
Thanks
Quoted 20 lines
Thanks
Eric
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Quoted 5 lines
BTW
vhost_init_device_iotlbgets enabled parameter but ignores it, we really should drop that.Yes.
Quoted 2 lines
Also, it looks like if features are set with
VIRTIO_F_ACCESS_PLATFORMand then cleared, iotlb is not properly cleared - bug?Not sure, old IOTLB may still work. But for safety, we need to disable device IOTLB in this case.
Thanks
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
--
2.37.3
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Quoted 60 lines · Michael S. Tsirkin
Quoted 49 lines · Jason Wang
Quoted 48 lines · Michael S. Tsirkin
Quoted 45 lines · Jason Wang
Quoted 31 lines · Michael S. Tsirkin
Quoted 29 lines · Eric Auger
Hi Michael,
Quoted 27 lines · Michael S. Tsirkin
Quoted 17 lines · Eric Auger
When the vhost iotlb is used along with a guest virtual iommu and the guest gets rebooted, some MISS messages may have been recorded just before the reboot and spuriously executed by the virtual iommu after the reboot. Despite the device iotlb gets re-initialized, the messages are not cleared. Fix that by calling
vhost_clear_msg()at the end ofvhost_init_device_iotlb().Signed-off-by Eric Auger <[email protected]>--- drivers/vhost/vhost.c | 1 + 1 file changed, 1 insertion(+)drivers/vhost/vhost.c +0
-0drivers/vhost/vhost.c int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled) · line 1757
Quoted 7 lines
drivers/vhost/vhost.c +1
-0@@ -1751,6 +1751,7 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled) }vhost_iotlb_free (oiotlb); +vhost_clear_msg (d);return 0 ; }Hmm. Can't messages meanwhile get processes and affect the new iotlb?
Quoted 8 lines
Quoted 6 lines
Isn't the msg processing stopped at the moment this function is called (
VHOST_SET_FEATURES)?Thanks
Eric
It's pretty late here I'm not sure. You tell me what prevents it.
So the proposed code assumes that Qemu doesn't process device IOTLB before
VHOST_SET_FEAETURES. Consider there's no reset in the general vhost uAPI, I wonder if it's better to move the clear to device code likeVHOST_NET_SET_BACKEND. So we can clear it per vq?Quoted 8 lines
Quoted 2 lines
Hmm this makes no sense to me. iommu sits between backend and frontend. Tying one to another is going to backfire.
I think we need to emulate what real devices are doing. Device should clear the page fault message during reset, so the driver won't read anything after reset. But we don't have a per device stop or reset message for vhost-net. That's why the
VHOST_NET_SET_BACKENDcame into my mind.
Quoted 3 lines
That's not a reset message. Userspace can switch backends at will. I guess we could check when backend is set to -1.
It's a hack but might work.
Yes, that's what I meant actually.
Quoted 15 lines
Quoted 10 lines
Quoted 4 lines
I'm thinking more along the lines of doing everything under
iotlb_lock.I think the problem is we need to find a proper place to clear the message. So I don't get how
iotlb_lockcan help: the message could be still read from user space after the backend is set to NULL.Thanks
Well I think the real problem is this.
vhost_net_set_featuresdoes:
Quoted 7 lines
if ((features & (1ULL << VIRTIO_F_ACCESS_PLATFORM))) { if (vhost_init_device_iotlb(&n->dev, true)) goto out_unlock; }so we get a new iotlb each time features are set.
Right, but this looks like another independent issue that needs to be fixed.
Quoted 9 lines
But features can be changes while device is running. E.g.
VHOST_F_LOG_ALLLet's just say this hack of reusing feature bits for backend was not my brightest idea :(
Probably :)
Thanks
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 10 lines
BTW
vhost_init_device_iotlbgets enabled parameter but ignores it, we really should drop that.Yes.
Quoted 3 lines
Also, it looks like if features are set with
VIRTIO_F_ACCESS_PLATFORMand then cleared, iotlb is not properly cleared - bug?Not sure, old IOTLB may still work. But for safety, we need to disable device IOTLB in this case.
Thanks
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
--
2.37.3
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization