Skip to contentSkip to conversation

vhost: Clear the pending messages on vhost_init_device_iotlb()

RFC In review Eric Auger 3 participants 13 messages Nov 7, 2022 – Nov 9, 2022

Subsystems (5)
lore
Share & tools
lore

Eric Auger

author
#0 lore raw

RFC 1/1 vhost: Clear the pending messages on vhost_init_device_iotlb()

When the vhost iotlb is used along with a guest virtual iommu and the guest gets rebooted, some MISS messages may have been recorded just before the reboot and spuriously executed by the virtual iommu after the reboot. Despite the device iotlb gets re-initialized, the messages are not cleared. Fix that by calling vhost_clear_msg() at the end of vhost_init_device_iotlb().

---
 drivers/vhost/vhost.c | 1 +
 1 file changed, 1 insertion(+)
drivers/vhost/vhost.c +1 -0
index 40097826cff0..422a1fdee0ca 100644
@@ -1751,6 +1751,7 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled)
 	}
 
 	vhost_iotlb_free(oiotlb);
+	vhost_clear_msg(d);
 
 	return 0;
 }
Signature · 6 lines
2.37.3

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
#1 lore raw
Quoted 17 lines · Eric Auger

When the vhost iotlb is used along with a guest virtual iommu and the guest gets rebooted, some MISS messages may have been recorded just before the reboot and spuriously executed by the virtual iommu after the reboot. Despite the device iotlb gets re-initialized, the messages are not cleared. Fix that by calling vhost_clear_msg() at the end of vhost_init_device_iotlb().

---
 drivers/vhost/vhost.c | 1 +
 1 file changed, 1 insertion(+)
drivers/vhost/vhost.c +0 -0
index 40097826cff0..422a1fdee0ca 100644

drivers/vhost/vhost.c int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled) · line 1757

Quoted 7 lines
drivers/vhost/vhost.c +1 -0
@@ -1751,6 +1751,7 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled)
 	}
 
 	vhost_iotlb_free(oiotlb);
+	vhost_clear_msg(d);
 
 	return 0;
 }

Hmm. Can't messages meanwhile get processes and affect the new iotlb?

Quoted 2 lines
Signature · 1 line
2.37.3

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

author
#2 lore raw

Hi Michael,

Quoted 20 lines · Michael S. Tsirkin
Quoted 19 lines · Eric Auger

When the vhost iotlb is used along with a guest virtual iommu and the guest gets rebooted, some MISS messages may have been recorded just before the reboot and spuriously executed by the virtual iommu after the reboot. Despite the device iotlb gets re-initialized, the messages are not cleared. Fix that by calling vhost_clear_msg() at the end of vhost_init_device_iotlb().

---
 drivers/vhost/vhost.c | 1 +
 1 file changed, 1 insertion(+)
drivers/vhost/vhost.c +0 -0
index 40097826cff0..422a1fdee0ca 100644
@@ -1751,6 +1751,7 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled)
 	}
 
Quoted 7 lines

drivers/vhost/vhost.c int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled) · line 1757

Quoted 5 lines
drivers/vhost/vhost.c +1 -0
 	vhost_iotlb_free(oiotlb);
+	vhost_clear_msg(d);
 
 	return 0;
 }

Hmm. Can't messages meanwhile get processes and affect the new iotlb?

Isn't the msg processing stopped at the moment this function is called (VHOST_SET_FEATURES)?

Thanks

Eric

Quoted 4 lines
Quoted 4 lines
Signature · 2 lines
-- 
2.37.3

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#3 lore raw
Quoted 29 lines · Eric Auger

Hi Michael,

Quoted 20 lines · Michael S. Tsirkin
Quoted 19 lines · Eric Auger

When the vhost iotlb is used along with a guest virtual iommu and the guest gets rebooted, some MISS messages may have been recorded just before the reboot and spuriously executed by the virtual iommu after the reboot. Despite the device iotlb gets re-initialized, the messages are not cleared. Fix that by calling vhost_clear_msg() at the end of vhost_init_device_iotlb().

---
 drivers/vhost/vhost.c | 1 +
 1 file changed, 1 insertion(+)
drivers/vhost/vhost.c +0 -0
index 40097826cff0..422a1fdee0ca 100644
@@ -1751,6 +1751,7 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled)
 	}
 
Quoted 7 lines

drivers/vhost/vhost.c int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled) · line 1757

Quoted 5 lines
drivers/vhost/vhost.c +1 -0
 	vhost_iotlb_free(oiotlb);
+	vhost_clear_msg(d);
 
 	return 0;
 }

Hmm. Can't messages meanwhile get processes and affect the new iotlb?

Quoted 6 lines

Isn't the msg processing stopped at the moment this function is called (VHOST_SET_FEATURES)?

Thanks

Eric

It's pretty late here I'm not sure. You tell me what prevents it.

BTW vhost_init_device_iotlb gets enabled parameter but ignores it, we really should drop that.

Also, it looks like if features are set with VIRTIO_F_ACCESS_PLATFORM and then cleared, iotlb is not properly cleared - bug?

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Signature · 2 lines
-- 
2.37.3

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#4 lore raw
Quoted 31 lines · Michael S. Tsirkin
Quoted 29 lines · Eric Auger

Hi Michael,

Quoted 27 lines · Michael S. Tsirkin
Quoted 17 lines · Eric Auger

When the vhost iotlb is used along with a guest virtual iommu and the guest gets rebooted, some MISS messages may have been recorded just before the reboot and spuriously executed by the virtual iommu after the reboot. Despite the device iotlb gets re-initialized, the messages are not cleared. Fix that by calling vhost_clear_msg() at the end of vhost_init_device_iotlb().

---
 drivers/vhost/vhost.c | 1 +
 1 file changed, 1 insertion(+)
drivers/vhost/vhost.c +0 -0
index 40097826cff0..422a1fdee0ca 100644

drivers/vhost/vhost.c int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled) · line 1757

Quoted 7 lines
drivers/vhost/vhost.c +1 -0
@@ -1751,6 +1751,7 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled)
   }
 
   vhost_iotlb_free(oiotlb);
+  vhost_clear_msg(d);
 
   return 0;
 }

Hmm. Can't messages meanwhile get processes and affect the new iotlb?

Quoted 8 lines
Quoted 6 lines

Isn't the msg processing stopped at the moment this function is called (VHOST_SET_FEATURES)?

Thanks

Eric

It's pretty late here I'm not sure. You tell me what prevents it.

So the proposed code assumes that Qemu doesn't process device IOTLB before VHOST_SET_FEAETURES. Consider there's no reset in the general vhost uAPI, I wonder if it's better to move the clear to device code like VHOST_NET_SET_BACKEND. So we can clear it per vq?

Quoted 3 lines

BTW vhost_init_device_iotlb gets enabled parameter but ignores it, we really should drop that.

Yes.

Quoted 3 lines

Also, it looks like if features are set with VIRTIO_F_ACCESS_PLATFORM and then cleared, iotlb is not properly cleared - bug?

Not sure, old IOTLB may still work. But for safety, we need to disable device IOTLB in this case.

Thanks

Quoted 7 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

--
2.37.3

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

author
#5 lore raw
Quoted 34 lines · Jason Wang
Quoted 33 lines · Michael S. Tsirkin
Quoted 32 lines · Eric Auger

Hi Michael,

Quoted 20 lines · Michael S. Tsirkin
Quoted 19 lines · Eric Auger

When the vhost iotlb is used along with a guest virtual iommu and the guest gets rebooted, some MISS messages may have been recorded just before the reboot and spuriously executed by the virtual iommu after the reboot. Despite the device iotlb gets re-initialized, the messages are not cleared. Fix that by calling vhost_clear_msg() at the end of vhost_init_device_iotlb().

---
 drivers/vhost/vhost.c | 1 +
 1 file changed, 1 insertion(+)
drivers/vhost/vhost.c +0 -0
index 40097826cff0..422a1fdee0ca 100644
@@ -1751,6 +1751,7 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled)
   }
 
Quoted 7 lines

drivers/vhost/vhost.c int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled) · line 1757

Quoted 5 lines
drivers/vhost/vhost.c +1 -0
   vhost_iotlb_free(oiotlb);
+  vhost_clear_msg(d);
 
   return 0;
 }

Hmm. Can't messages meanwhile get processes and affect the new iotlb?

Isn't the msg processing stopped at the moment this function is called (VHOST_SET_FEATURES)?

Quoted 8 lines
Quoted 4 lines
Quoted 3 lines

Thanks

Eric

It's pretty late here I'm not sure. You tell me what prevents it.

So the proposed code assumes that Qemu doesn't process device IOTLB before VHOST_SET_FEAETURES. Consider there's no reset in the general vhost uAPI, I wonder if it's better to move the clear to device code like VHOST_NET_SET_BACKEND. So we can clear it per vq?

OK I will look at this alternative

Quoted 4 lines
Quoted 3 lines

BTW vhost_init_device_iotlb gets enabled parameter but ignores it, we really should drop that.

Yes.

Yes I saw that too. I will send a patch.

Quoted 5 lines
Quoted 3 lines

Also, it looks like if features are set with VIRTIO_F_ACCESS_PLATFORM and then cleared, iotlb is not properly cleared - bug?

Not sure, old IOTLB may still work. But for safety, we need to disable device IOTLB in this case.

OK

Thanks

Eric

Quoted 7 lines

Thanks

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

--
2.37.3

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#6 lore raw
In reply to Jason Wang
Quoted 37 lines · Jason Wang
Quoted 36 lines · Michael S. Tsirkin
Quoted 34 lines · Eric Auger

Hi Michael,

Quoted 20 lines · Michael S. Tsirkin
Quoted 19 lines · Eric Auger

When the vhost iotlb is used along with a guest virtual iommu and the guest gets rebooted, some MISS messages may have been recorded just before the reboot and spuriously executed by the virtual iommu after the reboot. Despite the device iotlb gets re-initialized, the messages are not cleared. Fix that by calling vhost_clear_msg() at the end of vhost_init_device_iotlb().

---
 drivers/vhost/vhost.c | 1 +
 1 file changed, 1 insertion(+)
drivers/vhost/vhost.c +0 -0
index 40097826cff0..422a1fdee0ca 100644
@@ -1751,6 +1751,7 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled)
   }
 
Quoted 7 lines

drivers/vhost/vhost.c int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled) · line 1757

Quoted 5 lines
drivers/vhost/vhost.c +1 -0
   vhost_iotlb_free(oiotlb);
+  vhost_clear_msg(d);
 
   return 0;
 }

Hmm. Can't messages meanwhile get processes and affect the new iotlb?

Isn't the msg processing stopped at the moment this function is called (VHOST_SET_FEATURES)?

Thanks

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

Eric

It's pretty late here I'm not sure. You tell me what prevents it.

So the proposed code assumes that Qemu doesn't process device IOTLB before VHOST_SET_FEAETURES. Consider there's no reset in the general vhost uAPI, I wonder if it's better to move the clear to device code like VHOST_NET_SET_BACKEND. So we can clear it per vq?

Hmm this makes no sense to me. iommu sits between backend and frontend. Tying one to another is going to backfire.

I'm thinking more along the lines of doing everything under iotlb_lock.

Quoted 7 lines
Quoted 3 lines

BTW vhost_init_device_iotlb gets enabled parameter but ignores it, we really should drop that.

Yes.

Quoted 1 line
Quoted 15 lines
Quoted 2 lines

Also, it looks like if features are set with VIRTIO_F_ACCESS_PLATFORM and then cleared, iotlb is not properly cleared - bug?

Not sure, old IOTLB may still work. But for safety, we need to disable device IOTLB in this case.

Thanks

Quoted 7 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

--
2.37.3

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#7 lore raw
Quoted 43 lines · Michael S. Tsirkin
Quoted 41 lines · Jason Wang
Quoted 39 lines · Michael S. Tsirkin
Quoted 29 lines · Eric Auger

Hi Michael,

Quoted 20 lines · Michael S. Tsirkin
Quoted 19 lines · Eric Auger

When the vhost iotlb is used along with a guest virtual iommu and the guest gets rebooted, some MISS messages may have been recorded just before the reboot and spuriously executed by the virtual iommu after the reboot. Despite the device iotlb gets re-initialized, the messages are not cleared. Fix that by calling vhost_clear_msg() at the end of vhost_init_device_iotlb().

---
 drivers/vhost/vhost.c | 1 +
 1 file changed, 1 insertion(+)
drivers/vhost/vhost.c +0 -0
index 40097826cff0..422a1fdee0ca 100644
@@ -1751,6 +1751,7 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled)
   }
 
Quoted 7 lines

drivers/vhost/vhost.c int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled) · line 1757

Quoted 5 lines
drivers/vhost/vhost.c +1 -0
   vhost_iotlb_free(oiotlb);
+  vhost_clear_msg(d);
 
   return 0;
 }

Hmm. Can't messages meanwhile get processes and affect the new iotlb?

Quoted 6 lines

Isn't the msg processing stopped at the moment this function is called (VHOST_SET_FEATURES)?

Thanks

Eric

It's pretty late here I'm not sure. You tell me what prevents it.

Quoted 7 lines
Quoted 4 lines

So the proposed code assumes that Qemu doesn't process device IOTLB before VHOST_SET_FEAETURES. Consider there's no reset in the general vhost uAPI, I wonder if it's better to move the clear to device code like VHOST_NET_SET_BACKEND. So we can clear it per vq?

Hmm this makes no sense to me. iommu sits between backend and frontend. Tying one to another is going to backfire.

I think we need to emulate what real devices are doing. Device should clear the page fault message during reset, so the driver won't read anything after reset. But we don't have a per device stop or reset message for vhost-net. That's why the VHOST_NET_SET_BACKEND came into my mind.

Quoted 3 lines

I'm thinking more along the lines of doing everything under iotlb_lock.

I think the problem is we need to find a proper place to clear the message. So I don't get how iotlb_lock can help: the message could be still read from user space after the backend is set to NULL.

Thanks

Quoted 22 lines
Quoted 22 lines
Quoted 6 lines

BTW vhost_init_device_iotlb gets enabled parameter but ignores it, we really should drop that.

Yes.

Quoted 3 lines

Also, it looks like if features are set with VIRTIO_F_ACCESS_PLATFORM and then cleared, iotlb is not properly cleared - bug?

Not sure, old IOTLB may still work. But for safety, we need to disable device IOTLB in this case.

Thanks

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

--
2.37.3

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#8 lore raw
Quoted 49 lines · Jason Wang
Quoted 48 lines · Michael S. Tsirkin
Quoted 45 lines · Jason Wang
Quoted 31 lines · Michael S. Tsirkin
Quoted 29 lines · Eric Auger

Hi Michael,

Quoted 27 lines · Michael S. Tsirkin
Quoted 17 lines · Eric Auger

When the vhost iotlb is used along with a guest virtual iommu and the guest gets rebooted, some MISS messages may have been recorded just before the reboot and spuriously executed by the virtual iommu after the reboot. Despite the device iotlb gets re-initialized, the messages are not cleared. Fix that by calling vhost_clear_msg() at the end of vhost_init_device_iotlb().

---
 drivers/vhost/vhost.c | 1 +
 1 file changed, 1 insertion(+)
drivers/vhost/vhost.c +0 -0
index 40097826cff0..422a1fdee0ca 100644

drivers/vhost/vhost.c int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled) · line 1757

Quoted 7 lines
drivers/vhost/vhost.c +1 -0
@@ -1751,6 +1751,7 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled)
   }
 
   vhost_iotlb_free(oiotlb);
+  vhost_clear_msg(d);
 
   return 0;
 }

Hmm. Can't messages meanwhile get processes and affect the new iotlb?

Quoted 8 lines
Quoted 6 lines

Isn't the msg processing stopped at the moment this function is called (VHOST_SET_FEATURES)?

Thanks

Eric

It's pretty late here I'm not sure. You tell me what prevents it.

So the proposed code assumes that Qemu doesn't process device IOTLB before VHOST_SET_FEAETURES. Consider there's no reset in the general vhost uAPI, I wonder if it's better to move the clear to device code like VHOST_NET_SET_BACKEND. So we can clear it per vq?

Quoted 8 lines
Quoted 2 lines

Hmm this makes no sense to me. iommu sits between backend and frontend. Tying one to another is going to backfire.

I think we need to emulate what real devices are doing. Device should clear the page fault message during reset, so the driver won't read anything after reset. But we don't have a per device stop or reset message for vhost-net. That's why the VHOST_NET_SET_BACKEND came into my mind.

That's not a reset message. Userspace can switch backends at will. I guess we could check when backend is set to -1.
It's a hack but might work.

Quoted 9 lines
Quoted 3 lines

I'm thinking more along the lines of doing everything under iotlb_lock.

I think the problem is we need to find a proper place to clear the message. So I don't get how iotlb_lock can help: the message could be still read from user space after the backend is set to NULL.

Thanks

Well I think the real problem is this.

vhost_net_set_features does:

        if ((features & (1ULL << VIRTIO_F_ACCESS_PLATFORM))) {
                if (vhost_init_device_iotlb(&n->dev, true))
                        goto out_unlock;
        }

so we get a new iotlb each time features are set.

But features can be changes while device is running. E.g.
VHOST_F_LOG_ALL

Let's just say this hack of reusing feature bits for backend was not my brightest idea :(

Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 6 lines

BTW vhost_init_device_iotlb gets enabled parameter but ignores it, we really should drop that.

Yes.

Quoted 3 lines

Also, it looks like if features are set with VIRTIO_F_ACCESS_PLATFORM and then cleared, iotlb is not properly cleared - bug?

Not sure, old IOTLB may still work. But for safety, we need to disable device IOTLB in this case.

Thanks

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

--
2.37.3

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

author
#9 lore raw

Hi Michael, Jason,

Quoted 75 lines · Michael S. Tsirkin
Quoted 44 lines · Jason Wang
Quoted 43 lines · Michael S. Tsirkin
Quoted 42 lines · Jason Wang
Quoted 30 lines · Michael S. Tsirkin
Quoted 29 lines · Eric Auger

Hi Michael,

Quoted 27 lines · Michael S. Tsirkin
Quoted 17 lines · Eric Auger

When the vhost iotlb is used along with a guest virtual iommu and the guest gets rebooted, some MISS messages may have been recorded just before the reboot and spuriously executed by the virtual iommu after the reboot. Despite the device iotlb gets re-initialized, the messages are not cleared. Fix that by calling vhost_clear_msg() at the end of vhost_init_device_iotlb().

---
 drivers/vhost/vhost.c | 1 +
 1 file changed, 1 insertion(+)
drivers/vhost/vhost.c +0 -0
index 40097826cff0..422a1fdee0ca 100644

drivers/vhost/vhost.c int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled) · line 1757

Quoted 7 lines
drivers/vhost/vhost.c +1 -0
@@ -1751,6 +1751,7 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled)
   }
 
   vhost_iotlb_free(oiotlb);
+  vhost_clear_msg(d);
 
   return 0;
 }

Hmm. Can't messages meanwhile get processes and affect the new iotlb?

Quoted 7 lines
Quoted 6 lines

Isn't the msg processing stopped at the moment this function is called (VHOST_SET_FEATURES)?

Thanks

Eric

It's pretty late here I'm not sure. You tell me what prevents it.

So the proposed code assumes that Qemu doesn't process device IOTLB before VHOST_SET_FEAETURES. Consider there's no reset in the general vhost uAPI, I wonder if it's better to move the clear to device code like VHOST_NET_SET_BACKEND. So we can clear it per vq?

Quoted 7 lines
Quoted 2 lines

Hmm this makes no sense to me. iommu sits between backend and frontend. Tying one to another is going to backfire.

I think we need to emulate what real devices are doing. Device should clear the page fault message during reset, so the driver won't read anything after reset. But we don't have a per device stop or reset message for vhost-net. That's why the VHOST_NET_SET_BACKEND came into my mind.

That's not a reset message. Userspace can switch backends at will. I guess we could check when backend is set to -1.
It's a hack but might work.

Quoted 7 lines
Quoted 2 lines

I'm thinking more along the lines of doing everything under iotlb_lock.

I think the problem is we need to find a proper place to clear the message. So I don't get how iotlb_lock can help: the message could be still read from user space after the backend is set to NULL.

Thanks

Well I think the real problem is this.

vhost_net_set_features does:

        if ((features & (1ULL << VIRTIO_F_ACCESS_PLATFORM))) {
                if (vhost_init_device_iotlb(&n->dev, true))
                        goto out_unlock;
        }

so we get a new iotlb each time features are set.

Quoted 8 lines

But features can be changes while device is running. E.g.
VHOST_F_LOG_ALL

Let's just say this hack of reusing feature bits for backend was not my brightest idea :(

Isn't vhost_init_device_iotlb() racy then, as d->iotlb is first updated with niotlb and later d->vqs[i]->iotlb is updated with niotlb. What does garantee this is done atomically?

Shouldn't we hold the dev->mutex to make all the sequence atomic and include vhost_clear_msg()?  Can't the vhost_clear_msg() take the dev lock?

Thanks

Eric

Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
Quoted 7 lines

BTW vhost_init_device_iotlb gets enabled parameter but ignores it, we really should drop that.

Yes.

Quoted 2 lines

Also, it looks like if features are set with VIRTIO_F_ACCESS_PLATFORM and then cleared, iotlb is not properly cleared - bug?

Not sure, old IOTLB may still work. But for safety, we need to disable device IOTLB in this case.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Thanks

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

--
2.37.3

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#12 lore raw
In reply to Michael S. Tsirkin
Quoted 60 lines · Michael S. Tsirkin
Quoted 49 lines · Jason Wang
Quoted 48 lines · Michael S. Tsirkin
Quoted 45 lines · Jason Wang
Quoted 31 lines · Michael S. Tsirkin
Quoted 29 lines · Eric Auger

Hi Michael,

Quoted 27 lines · Michael S. Tsirkin
Quoted 17 lines · Eric Auger

When the vhost iotlb is used along with a guest virtual iommu and the guest gets rebooted, some MISS messages may have been recorded just before the reboot and spuriously executed by the virtual iommu after the reboot. Despite the device iotlb gets re-initialized, the messages are not cleared. Fix that by calling vhost_clear_msg() at the end of vhost_init_device_iotlb().

---
 drivers/vhost/vhost.c | 1 +
 1 file changed, 1 insertion(+)
drivers/vhost/vhost.c +0 -0
index 40097826cff0..422a1fdee0ca 100644

drivers/vhost/vhost.c int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled) · line 1757

Quoted 7 lines
drivers/vhost/vhost.c +1 -0
@@ -1751,6 +1751,7 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled)
   }
 
   vhost_iotlb_free(oiotlb);
+  vhost_clear_msg(d);
 
   return 0;
 }

Hmm. Can't messages meanwhile get processes and affect the new iotlb?

Quoted 8 lines
Quoted 6 lines

Isn't the msg processing stopped at the moment this function is called (VHOST_SET_FEATURES)?

Thanks

Eric

It's pretty late here I'm not sure. You tell me what prevents it.

So the proposed code assumes that Qemu doesn't process device IOTLB before VHOST_SET_FEAETURES. Consider there's no reset in the general vhost uAPI, I wonder if it's better to move the clear to device code like VHOST_NET_SET_BACKEND. So we can clear it per vq?

Quoted 8 lines
Quoted 2 lines

Hmm this makes no sense to me. iommu sits between backend and frontend. Tying one to another is going to backfire.

I think we need to emulate what real devices are doing. Device should clear the page fault message during reset, so the driver won't read anything after reset. But we don't have a per device stop or reset message for vhost-net. That's why the VHOST_NET_SET_BACKEND came into my mind.

Quoted 3 lines

That's not a reset message. Userspace can switch backends at will. I guess we could check when backend is set to -1.
It's a hack but might work.

Yes, that's what I meant actually.

Quoted 15 lines
Quoted 10 lines
Quoted 4 lines

I'm thinking more along the lines of doing everything under iotlb_lock.

I think the problem is we need to find a proper place to clear the message. So I don't get how iotlb_lock can help: the message could be still read from user space after the backend is set to NULL.

Thanks

Well I think the real problem is this.

vhost_net_set_features does:

Quoted 7 lines
        if ((features & (1ULL << VIRTIO_F_ACCESS_PLATFORM))) {
                if (vhost_init_device_iotlb(&n->dev, true))
                        goto out_unlock;
        }

so we get a new iotlb each time features are set.

Right, but this looks like another independent issue that needs to be fixed.

Quoted 9 lines

But features can be changes while device is running. E.g.
VHOST_F_LOG_ALL

Let's just say this hack of reusing feature bits for backend was not my brightest idea :(

Probably :)

Thanks

Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 10 lines

BTW vhost_init_device_iotlb gets enabled parameter but ignores it, we really should drop that.

Yes.

Quoted 3 lines

Also, it looks like if features are set with VIRTIO_F_ACCESS_PLATFORM and then cleared, iotlb is not properly cleared - bug?

Not sure, old IOTLB may still work. But for safety, we need to disable device IOTLB in this case.

Thanks

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

--
2.37.3

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

#10 lore raw
In reply to Eric Auger
Quoted 84 lines · Eric Auger

Hi Michael, Jason,

Quoted 80 lines · Michael S. Tsirkin
Quoted 44 lines · Jason Wang
Quoted 43 lines · Michael S. Tsirkin
Quoted 42 lines · Jason Wang
Quoted 30 lines · Michael S. Tsirkin
Quoted 29 lines · Eric Auger

Hi Michael,

Quoted 27 lines · Michael S. Tsirkin
Quoted 17 lines · Eric Auger

When the vhost iotlb is used along with a guest virtual iommu and the guest gets rebooted, some MISS messages may have been recorded just before the reboot and spuriously executed by the virtual iommu after the reboot. Despite the device iotlb gets re-initialized, the messages are not cleared. Fix that by calling vhost_clear_msg() at the end of vhost_init_device_iotlb().

---
 drivers/vhost/vhost.c | 1 +
 1 file changed, 1 insertion(+)
drivers/vhost/vhost.c +0 -0
index 40097826cff0..422a1fdee0ca 100644

drivers/vhost/vhost.c int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled) · line 1757

Quoted 7 lines
drivers/vhost/vhost.c +1 -0
@@ -1751,6 +1751,7 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled)
   }
 
   vhost_iotlb_free(oiotlb);
+  vhost_clear_msg(d);
 
   return 0;
 }

Hmm. Can't messages meanwhile get processes and affect the new iotlb?

Quoted 7 lines
Quoted 6 lines

Isn't the msg processing stopped at the moment this function is called (VHOST_SET_FEATURES)?

Thanks

Eric

It's pretty late here I'm not sure. You tell me what prevents it.

So the proposed code assumes that Qemu doesn't process device IOTLB before VHOST_SET_FEAETURES. Consider there's no reset in the general vhost uAPI, I wonder if it's better to move the clear to device code like VHOST_NET_SET_BACKEND. So we can clear it per vq?

Quoted 7 lines
Quoted 2 lines

Hmm this makes no sense to me. iommu sits between backend and frontend. Tying one to another is going to backfire.

I think we need to emulate what real devices are doing. Device should clear the page fault message during reset, so the driver won't read anything after reset. But we don't have a per device stop or reset message for vhost-net. That's why the VHOST_NET_SET_BACKEND came into my mind.

That's not a reset message. Userspace can switch backends at will. I guess we could check when backend is set to -1.
It's a hack but might work.

Quoted 7 lines
Quoted 2 lines

I'm thinking more along the lines of doing everything under iotlb_lock.

I think the problem is we need to find a proper place to clear the message. So I don't get how iotlb_lock can help: the message could be still read from user space after the backend is set to NULL.

Thanks

Well I think the real problem is this.

vhost_net_set_features does:

        if ((features & (1ULL << VIRTIO_F_ACCESS_PLATFORM))) {
                if (vhost_init_device_iotlb(&n->dev, true))
                        goto out_unlock;
        }

so we get a new iotlb each time features are set.

But features can be changes while device is running. E.g.
VHOST_F_LOG_ALL

Quoted 8 lines
Quoted 3 lines

Let's just say this hack of reusing feature bits for backend was not my brightest idea :(

Isn't vhost_init_device_iotlb() racy then, as d->iotlb is first updated with niotlb and later d->vqs[i]->iotlb is updated with niotlb. What does garantee this is done atomically?

Shouldn't we hold the dev->mutex to make all the sequence atomic and include vhost_clear_msg()? Can't the vhost_clear_msg() take the dev lock?

It depends on where we want to place the vhost_clear_msg(), e.g in most of the device ioctl, the dev->mutex has been held.

Thanks

Quoted 26 lines

Thanks

Eric

Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 20 lines
Quoted 7 lines

BTW vhost_init_device_iotlb gets enabled parameter but ignores it, we really should drop that.

Yes.

Quoted 2 lines

Also, it looks like if features are set with VIRTIO_F_ACCESS_PLATFORM and then cleared, iotlb is not properly cleared - bug?

Not sure, old IOTLB may still work. But for safety, we need to disable device IOTLB in this case.

Thanks

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

--
2.37.3

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

author
#11 lore raw

Hi Jason,

Quoted 87 lines · Jason Wang
Quoted 86 lines · Eric Auger

Hi Michael, Jason,

Quoted 83 lines · Michael S. Tsirkin
Quoted 44 lines · Jason Wang
Quoted 43 lines · Michael S. Tsirkin
Quoted 42 lines · Jason Wang
Quoted 30 lines · Michael S. Tsirkin
Quoted 29 lines · Eric Auger

Hi Michael,

Quoted 27 lines · Michael S. Tsirkin
Quoted 17 lines · Eric Auger

When the vhost iotlb is used along with a guest virtual iommu and the guest gets rebooted, some MISS messages may have been recorded just before the reboot and spuriously executed by the virtual iommu after the reboot. Despite the device iotlb gets re-initialized, the messages are not cleared. Fix that by calling vhost_clear_msg() at the end of vhost_init_device_iotlb().

---
 drivers/vhost/vhost.c | 1 +
 1 file changed, 1 insertion(+)
drivers/vhost/vhost.c +0 -0
index 40097826cff0..422a1fdee0ca 100644

drivers/vhost/vhost.c int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled) · line 1757

Quoted 7 lines
drivers/vhost/vhost.c +1 -0
@@ -1751,6 +1751,7 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled)
   }
 
   vhost_iotlb_free(oiotlb);
+  vhost_clear_msg(d);
 
   return 0;
 }

Hmm. Can't messages meanwhile get processes and affect the new iotlb?

Quoted 7 lines
Quoted 6 lines

Isn't the msg processing stopped at the moment this function is called (VHOST_SET_FEATURES)?

Thanks

Eric

It's pretty late here I'm not sure. You tell me what prevents it.

So the proposed code assumes that Qemu doesn't process device IOTLB before VHOST_SET_FEAETURES. Consider there's no reset in the general vhost uAPI, I wonder if it's better to move the clear to device code like VHOST_NET_SET_BACKEND. So we can clear it per vq?

Quoted 7 lines
Quoted 2 lines

Hmm this makes no sense to me. iommu sits between backend and frontend. Tying one to another is going to backfire.

I think we need to emulate what real devices are doing. Device should clear the page fault message during reset, so the driver won't read anything after reset. But we don't have a per device stop or reset message for vhost-net. That's why the VHOST_NET_SET_BACKEND came into my mind.

That's not a reset message. Userspace can switch backends at will. I guess we could check when backend is set to -1.
It's a hack but might work.

Quoted 7 lines
Quoted 2 lines

I'm thinking more along the lines of doing everything under iotlb_lock.

I think the problem is we need to find a proper place to clear the message. So I don't get how iotlb_lock can help: the message could be still read from user space after the backend is set to NULL.

Thanks

Well I think the real problem is this.

vhost_net_set_features does:

        if ((features & (1ULL << VIRTIO_F_ACCESS_PLATFORM))) {
                if (vhost_init_device_iotlb(&n->dev, true))
                        goto out_unlock;
        }

so we get a new iotlb each time features are set.

But features can be changes while device is running. E.g.
VHOST_F_LOG_ALL

Let's just say this hack of reusing feature bits for backend was not my brightest idea :(

Quoted 6 lines
Quoted 4 lines

Isn't vhost_init_device_iotlb() racy then, as d->iotlb is first updated with niotlb and later d->vqs[i]->iotlb is updated with niotlb. What does garantee this is done atomically?

Shouldn't we hold the dev->mutex to make all the sequence atomic and include vhost_clear_msg()? Can't the vhost_clear_msg() take the dev lock?

It depends on where we want to place the vhost_clear_msg(), e.g in most of the device ioctl, the dev->mutex has been held.

OK, I will double check and respin accordingly

Eric

Quoted 23 lines

Thanks

Quoted 20 lines

Thanks

Eric

Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Quoted 5 lines

BTW vhost_init_device_iotlb gets enabled parameter but ignores it, we really should drop that.

Yes.

Quoted 2 lines

Also, it looks like if features are set with VIRTIO_F_ACCESS_PLATFORM and then cleared, iotlb is not properly cleared - bug?

Not sure, old IOTLB may still work. But for safety, we need to disable device IOTLB in this case.

Thanks

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

--
2.37.3

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization