Skip to contentSkip to conversation

vhost: Clean iotlb cache on vring stop

PATCH Applied Eugenio Pérez 3 participants 10 messages Aug 27, 2021 – Sep 28, 2021

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 vhost: Clean iotlb cache on vring stop

Old IOVA cache entries are left when there is a change on virtio driver in VM. In case that all these old entries have iova addresses lesser than new iova entries, vhost code will need to iterate all the cache to find the new ones. In case of just a new iova entry needed for the new translations, this condition will last forever.

This has been observed in virtio-net to testpmd's vfio-pci driver transition, reducing the performance from more than 10Mpps to less than 0.07Mpps if the hugepage address was higher than the networking buffers. Since all new buffers are contained in this new gigantic page, vhost needs to scan IOTLB_CACHE_SIZE - 1 for each translation at worst.

---
 lib/vhost/vhost_user.c | 2 ++
 1 file changed, 2 insertions(+)
lib/vhost/vhost_user.c +2 -0
index 29a4c9af60..7de48f5333 100644
@@ -2113,6 +2113,8 @@ vhost_user_get_vring_base(struct virtio_net **pdev,
 	msg->size = sizeof(msg->payload.state);
 	msg->fd_num = 0;
 
+	vhost_user_iotlb_flush_all(vq);
+
 	vring_invalidate(dev, vq);
 
 	return RTE_VHOST_MSG_RESULT_REPLY;
Signature · 1 line
2.27.0
#1 lore raw

Hi Eugenio,

Quoted 8 lines

-----Original Message-----
From: Eugenio Pérez <[email protected]>
Sent: Saturday, August 28, 2021 12:13 AM
To: Maxime Coquelin <[email protected]>; Xia, Chenbo <[email protected]>
Cc: [email protected]; Pei Zhang <[email protected]>; Jason Wang <[email protected]>
Subject: [PATCH] vhost: Clean iotlb cache on vring stop

Clean -> clean

Quoted 7 lines

Old IOVA cache entries are left when there is a change on virtio driver in VM. In case that all these old entries have iova addresses lesser than new iova entries, vhost code will need to iterate all the cache to find the new ones. In case of just a new iova entry needed for the new translations, this condition will last forever.

Quoted 5 lines

This has been observed in virtio-net to testpmd's vfio-pci driver transition, reducing the performance from more than 10Mpps to less than 0.07Mpps if the hugepage address was higher than the networking buffers. Since all new buffers are contained in this new gigantic page, vhost needs to scan IOTLB_CACHE_SIZE - 1 for each translation at worst.

I'm curious why QEMU will not invalidate iotlb when virtio-net driver is removed (dma region should be unmapped).

And since the perf drop is huge, why not cc to stable and add fix tag?

Thanks,
Chenbo

Quoted 22 lines
---
 lib/vhost/vhost_user.c | 2 ++
 1 file changed, 2 insertions(+)
lib/vhost/vhost_user.c +2 -1
index 29a4c9af60..7de48f5333 100644
@@ -2113,6 +2113,8 @@ vhost_user_get_vring_base(struct virtio_net **pdev,
 	msg->size = sizeof(msg->payload.state);
 	msg->fd_num = 0;
 
+	vhost_user_iotlb_flush_all(vq);
+
 	vring_invalidate(dev, vq);
 
 	return RTE_VHOST_MSG_RESULT_REPLY;
--

2.27.0

#2 lore raw
Quoted 3 lines · Chenbo

Hi Eugenio,

Hi Chenbo,

Quoted 11 lines
Quoted 8 lines

-----Original Message-----
From: Eugenio Pérez <[email protected]>
Sent: Saturday, August 28, 2021 12:13 AM
To: Maxime Coquelin <[email protected]>; Xia, Chenbo <[email protected]>
Cc: [email protected]; Pei Zhang <[email protected]>; Jason Wang <[email protected]>
Subject: [PATCH] vhost: Clean iotlb cache on vring stop

Clean -> clean

Is that something I need to send a new revision for, or is it ok to apply on the maintainer side?

Quoted 7 lines
Quoted 7 lines

Old IOVA cache entries are left when there is a change on virtio driver in VM. In case that all these old entries have iova addresses lesser than new iova entries, vhost code will need to iterate all the cache to find the new ones. In case of just a new iova entry needed for the new translations, this condition will last forever.

Quoted 9 lines
Quoted 5 lines

This has been observed in virtio-net to testpmd's vfio-pci driver transition, reducing the performance from more than 10Mpps to less than 0.07Mpps if the hugepage address was higher than the networking buffers. Since all new buffers are contained in this new gigantic page, vhost needs to scan IOTLB_CACHE_SIZE - 1 for each translation at worst.

I'm curious why QEMU will not invalidate iotlb when virtio-net driver is removed (dma region should be unmapped).

I'm going to investigate this more, but qemu iommu notifier callback (vhost_iommu_unmap_notify) is never called through all the test. Also, guest kernel code calls dma_unmap_page for each buffer and vqs, but it never generates an iotlb flush.

Or do you mean that qemu should also flush all iotlb entries on vhost device stop?

Quoted 2 lines

And since the perf drop is huge, why not cc to stable and add fix tag?

I was not sure if it was worth it to backport, but I would say that the issue can be reproduced with enough bad luck. Since translations have always been saved in a linked list:

Fixes d012d1f293f4 ("vhost: add IOTLB helper functions")

Same question as before, if no changes to the code are needed for the patch, do I need to send a second revision?

Thanks!

Quoted 26 lines

Thanks,
Chenbo

Quoted 22 lines
---
 lib/vhost/vhost_user.c | 2 ++
 1 file changed, 2 insertions(+)
lib/vhost/vhost_user.c +2 -1
index 29a4c9af60..7de48f5333 100644
@@ -2113,6 +2113,8 @@ vhost_user_get_vring_base(struct virtio_net **pdev,
      msg->size = sizeof(msg->payload.state);
      msg->fd_num = 0;
 
+     vhost_user_iotlb_flush_all(vq);
+
      vring_invalidate(dev, vq);
 
      return RTE_VHOST_MSG_RESULT_REPLY;
--

2.27.0

#3 lore raw

Hi Eugenio,

Quoted 48 lines

-----Original Message-----
From: Eugenio Perez Martin <[email protected]>
Sent: Tuesday, August 31, 2021 2:10 AM
To: Xia, Chenbo <[email protected]>
Cc: Maxime Coquelin <[email protected]>; [email protected]; Pei Zhang <[email protected]>; Jason Wang <[email protected]>
Subject: Re: [PATCH] vhost: Clean iotlb cache on vring stop

Quoted 3 lines · Chenbo

Hi Eugenio,

Hi Chenbo,

Quoted 11 lines
Quoted 8 lines

-----Original Message-----
From: Eugenio Pérez <[email protected]>
Sent: Saturday, August 28, 2021 12:13 AM
To: Maxime Coquelin <[email protected]>; Xia, Chenbo <[email protected]>
Cc: [email protected]; Pei Zhang <[email protected]>; Jason Wang <[email protected]>
Subject: [PATCH] vhost: Clean iotlb cache on vring stop

Clean -> clean

Is that something I need to send a new revision for, or is it ok to apply on the maintainer side?

Quoted 7 lines
Quoted 7 lines

Old IOVA cache entries are left when there is a change on virtio driver in VM. In case that all these old entries have iova addresses lesser than new iova entries, vhost code will need to iterate all the cache to find the new ones. In case of just a new iova entry needed for the new translations, this condition will last forever.

Quoted 7 lines
Quoted 5 lines

This has been observed in virtio-net to testpmd's vfio-pci driver transition, reducing the performance from more than 10Mpps to less than 0.07Mpps if the hugepage address was higher than the networking buffers. Since all new buffers are contained in this new gigantic page, vhost needs to scan IOTLB_CACHE_SIZE - 1 for each translation at worst.

I'm curious why QEMU will not invalidate iotlb when virtio-net driver is

removed

Quoted 2 lines

(dma region should be unmapped).

Quoted 7 lines

I'm going to investigate this more, but qemu iommu notifier callback (vhost_iommu_unmap_notify) is never called through all the test. Also, guest kernel code calls dma_unmap_page for each buffer and vqs, but it never generates an iotlb flush.

Or do you mean that qemu should also flush all iotlb entries on vhost device stop?

I think as you said, the driver unmapped all entries, so theoretically the iotlb entries should all be invalidated. The iotlb invalidation should be triggered by iommu entry unmapping. Make sense to you?

Quoted 4 lines
Quoted 3 lines

And since the perf drop is huge, why not cc to stable and add fix tag?

Quoted 8 lines

I was not sure if it was worth it to backport, but I would say that the issue can be reproduced with enough bad luck. Since translations have always been saved in a linked list:

Fixes d012d1f293f4 ("vhost: add IOTLB helper functions")

Same question as before, if no changes to the code are needed for the patch, do I need to send a second revision?

I think it sounds a bug to me. So no matter how often it could be reproduced, it's worth to backport. I suggest to send a v2 with the title fixed and Fixes & cc stable tag added so that it will show in stable mailing list and save Maxime's effort when applying.

Thanks,
Chenbo

Quoted 30 lines

Thanks!

Quoted 26 lines

Thanks,
Chenbo

Quoted 22 lines
---
 lib/vhost/vhost_user.c | 2 ++
 1 file changed, 2 insertions(+)
lib/vhost/vhost_user.c +2 -1
index 29a4c9af60..7de48f5333 100644
@@ -2113,6 +2113,8 @@ vhost_user_get_vring_base(struct virtio_net **pdev,
      msg->size = sizeof(msg->payload.state);
      msg->fd_num = 0;
 
+     vhost_user_iotlb_flush_all(vq);
+
      vring_invalidate(dev, vq);
 
      return RTE_VHOST_MSG_RESULT_REPLY;
--

2.27.0

#4 lore raw
Quoted 56 lines · Chenbo

Hi Eugenio,

Quoted 53 lines

-----Original Message-----
From: Eugenio Perez Martin <[email protected]>
Sent: Tuesday, August 31, 2021 2:10 AM
To: Xia, Chenbo <[email protected]>
Cc: Maxime Coquelin <[email protected]>; [email protected]; Pei Zhang <[email protected]>; Jason Wang <[email protected]>
Subject: Re: [PATCH] vhost: Clean iotlb cache on vring stop

Quoted 3 lines · Chenbo

Hi Eugenio,

Hi Chenbo,

Quoted 11 lines
Quoted 8 lines

-----Original Message-----
From: Eugenio Pérez <[email protected]>
Sent: Saturday, August 28, 2021 12:13 AM
To: Maxime Coquelin <[email protected]>; Xia, Chenbo <[email protected]>
Cc: [email protected]; Pei Zhang <[email protected]>; Jason Wang <[email protected]>
Subject: [PATCH] vhost: Clean iotlb cache on vring stop

Clean -> clean

Is that something I need to send a new revision for, or is it ok to apply on the maintainer side?

Quoted 7 lines
Quoted 7 lines

Old IOVA cache entries are left when there is a change on virtio driver in VM. In case that all these old entries have iova addresses lesser than new iova entries, vhost code will need to iterate all the cache to find the new ones. In case of just a new iova entry needed for the new translations, this condition will last forever.

Quoted 7 lines
Quoted 5 lines

This has been observed in virtio-net to testpmd's vfio-pci driver transition, reducing the performance from more than 10Mpps to less than 0.07Mpps if the hugepage address was higher than the networking buffers. Since all new buffers are contained in this new gigantic page, vhost needs to scan IOTLB_CACHE_SIZE - 1 for each translation at worst.

I'm curious why QEMU will not invalidate iotlb when virtio-net driver is

removed

Quoted 2 lines

(dma region should be unmapped).

I'm going to investigate this more, but qemu iommu notifier callback (vhost_iommu_unmap_notify) is never called through all the test. Also, guest kernel code calls dma_unmap_page for each buffer and vqs, but it never generates an iotlb flush.

Quoted 7 lines
Quoted 2 lines

Or do you mean that qemu should also flush all iotlb entries on vhost device stop?

I think as you said, the driver unmapped all entries, so theoretically the iotlb entries should all be invalidated. The iotlb invalidation should be triggered by iommu entry unmapping. Make sense to you?

It totally makes sense. I just wanted to make sure at what level you meant the unmap should happen: If you meant qemu, then we need more code. If you meant the virtio-net/ring kernel driver, then there is a bug/misunderstanding on what is happening.

I think the iotlb unmapping is not happening because the entries are still valid, although not needed with vfio/guest's testpmd. If that were not true we would see way more errors with virtio-net+iommu because of invalid translations:

* virtio-net adds a buffer to virtio_ring. virtio-ring dma map does not generate a iotlb map: Qemu will wait for each device to request the missing pages.
* The device will miss the translation: It requests it, and saves it in the list for the future.
* virtio-ring is done with the buffer: It unmaps them. After a while, kernel needs to dma_map that page again. This new mapping does not generate a page miss as far as I see.
* Since this new mapping is already on the device's iotlb cache, the device will not issue a page miss. That means the page must be mapped to the same GPA as before: If not, the network would fail.

With this assumption, it is useless to iotlb-unmap them: The device will need to request them again if they're cleared from its cache. But all of this is my guess and I could be wrong :).

Quoted 10 lines
Quoted 10 lines
Quoted 3 lines

And since the perf drop is huge, why not cc to stable and add fix tag?

I was not sure if it was worth it to backport, but I would say that the issue can be reproduced with enough bad luck. Since translations have always been saved in a linked list:

Fixes d012d1f293f4 ("vhost: add IOTLB helper functions")
Quoted 8 lines
Quoted 2 lines

Same question as before, if no changes to the code are needed for the patch, do I need to send a second revision?

I think it sounds a bug to me. So no matter how often it could be reproduced, it's worth to backport. I suggest to send a v2 with the title fixed and Fixes & cc stable tag added so that it will show in stable mailing list and save Maxime's effort when applying.

Sure, I will do it.

Thanks!

Quoted 34 lines

Thanks,
Chenbo

Quoted 30 lines

Thanks!

Quoted 26 lines

Thanks,
Chenbo

Quoted 22 lines
---
 lib/vhost/vhost_user.c | 2 ++
 1 file changed, 2 insertions(+)
lib/vhost/vhost_user.c +2 -1
index 29a4c9af60..7de48f5333 100644
@@ -2113,6 +2113,8 @@ vhost_user_get_vring_base(struct virtio_net **pdev,
      msg->size = sizeof(msg->payload.state);
      msg->fd_num = 0;
 
+     vhost_user_iotlb_flush_all(vq);
+
      vring_invalidate(dev, vq);
 
      return RTE_VHOST_MSG_RESULT_REPLY;
--

2.27.0

Reviewed-by on v1 1/1
#5 lore raw
In reply to Eugenio Pérez
Quoted 24 lines · Eugenio Pérez

Old IOVA cache entries are left when there is a change on virtio driver in VM. In case that all these old entries have iova addresses lesser than new iova entries, vhost code will need to iterate all the cache to find the new ones. In case of just a new iova entry needed for the new translations, this condition will last forever.

This has been observed in virtio-net to testpmd's vfio-pci driver transition, reducing the performance from more than 10Mpps to less than 0.07Mpps if the hugepage address was higher than the networking buffers. Since all new buffers are contained in this new gigantic page, vhost needs to scan IOTLB_CACHE_SIZE - 1 for each translation at worst.

---
  lib/vhost/vhost_user.c | 2 ++
  1 file changed, 2 insertions(+)
lib/vhost/vhost_user.c +0 -0
index 29a4c9af60..7de48f5333 100644

lib/vhost/vhost_user.c vhost_user_get_vring_base(struct virtio_net **pdev, · line 2120

Quoted 8 lines
lib/vhost/vhost_user.c +2 -0
@@ -2113,6 +2113,8 @@ vhost_user_get_vring_base(struct virtio_net **pdev,
  	msg->size = sizeof(msg->payload.state);
  	msg->fd_num = 0;
 
+	vhost_user_iotlb_flush_all(vq);
+
  	vring_invalidate(dev, vq);
 
  	return RTE_VHOST_MSG_RESULT_REPLY;

Thanks,
Maxime

#6 lore raw

Hi Maxime,

Quoted 37 lines

-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Thursday, September 23, 2021 4:34 PM
To: Eugenio Pérez <[email protected]>; Xia, Chenbo <[email protected]>
Cc: [email protected]; Pei Zhang <[email protected]>; Jason Wang <[email protected]>
Subject: Re: [PATCH] vhost: Clean iotlb cache on vring stop

Quoted 26 lines · Eugenio Pérez

Old IOVA cache entries are left when there is a change on virtio driver in VM. In case that all these old entries have iova addresses lesser than new iova entries, vhost code will need to iterate all the cache to find the new ones. In case of just a new iova entry needed for the new translations, this condition will last forever.

This has been observed in virtio-net to testpmd's vfio-pci driver transition, reducing the performance from more than 10Mpps to less than 0.07Mpps if the hugepage address was higher than the networking buffers. Since all new buffers are contained in this new gigantic page, vhost needs to scan IOTLB_CACHE_SIZE - 1 for each translation at worst.

---
  lib/vhost/vhost_user.c | 2 ++
  1 file changed, 2 insertions(+)
lib/vhost/vhost_user.c +0 -0
index 29a4c9af60..7de48f5333 100644
@@ -2113,6 +2113,8 @@ vhost_user_get_vring_base(struct virtio_net **pdev,
  	msg->size = sizeof(msg->payload.state);
  	msg->fd_num = 0;
 
Quoted 8 lines

lib/vhost/vhost_user.c vhost_user_get_vring_base(struct virtio_net **pdev, · line 2120

Quoted 6 lines
lib/vhost/vhost_user.c +2 -0
+	vhost_user_iotlb_flush_all(vq);
+
  	vring_invalidate(dev, vq);
 
  	return RTE_VHOST_MSG_RESULT_REPLY;

So you will fix title and add fix tag when applying, right?

/Chenbo

Quoted 3 lines

Thanks,
Maxime

#7 lore raw
Quoted 43 lines · Chenbo

Hi Maxime,

Quoted 41 lines

-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Thursday, September 23, 2021 4:34 PM
To: Eugenio Pérez <[email protected]>; Xia, Chenbo <[email protected]>
Cc: [email protected]; Pei Zhang <[email protected]>; Jason Wang <[email protected]>
Subject: Re: [PATCH] vhost: Clean iotlb cache on vring stop

Quoted 30 lines · Eugenio Pérez

Old IOVA cache entries are left when there is a change on virtio driver in VM. In case that all these old entries have iova addresses lesser than new iova entries, vhost code will need to iterate all the cache to find the new ones. In case of just a new iova entry needed for the new translations, this condition will last forever.

This has been observed in virtio-net to testpmd's vfio-pci driver transition, reducing the performance from more than 10Mpps to less than 0.07Mpps if the hugepage address was higher than the networking buffers. Since all new buffers are contained in this new gigantic page, vhost needs to scan IOTLB_CACHE_SIZE - 1 for each translation at worst.

---
   lib/vhost/vhost_user.c | 2 ++
   1 file changed, 2 insertions(+)
lib/vhost/vhost_user.c +2 -0
index 29a4c9af60..7de48f5333 100644
@@ -2113,6 +2113,8 @@ vhost_user_get_vring_base(struct virtio_net **pdev,
   	msg->size = sizeof(msg->payload.state);
   	msg->fd_num = 0;
 
+	vhost_user_iotlb_flush_all(vq);
+
   	vring_invalidate(dev, vq);
 
Quoted 6 lines
Quoted 4 lines

lib/vhost/vhost_user.c vhost_user_get_vring_base(struct virtio_net **pdev, · line 2120

Quoted 2 lines
lib/vhost/vhost_user.c +0 -0
   	return RTE_VHOST_MSG_RESULT_REPLY;

So you will fix title and add fix tag when applying, right?

Yes, I will. Thanks for the heads-up!

Maxime

Quoted 6 lines

/Chenbo

Quoted 3 lines

Thanks,
Maxime

Reviewed-by
#8 lore raw
Quoted 54 lines

-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Thursday, September 23, 2021 4:50 PM
To: Xia, Chenbo <[email protected]>; Eugenio Pérez <[email protected]>
Cc: [email protected]; Pei Zhang <[email protected]>; Jason Wang <[email protected]>
Subject: Re: [PATCH] vhost: Clean iotlb cache on vring stop

Quoted 43 lines · Chenbo

Hi Maxime,

Quoted 41 lines

-----Original Message-----
From: Maxime Coquelin <[email protected]>
Sent: Thursday, September 23, 2021 4:34 PM
To: Eugenio Pérez <[email protected]>; Xia, Chenbo <[email protected]>
Cc: [email protected]; Pei Zhang <[email protected]>; Jason Wang <[email protected]>
Subject: Re: [PATCH] vhost: Clean iotlb cache on vring stop

Quoted 30 lines · Eugenio Pérez

Old IOVA cache entries are left when there is a change on virtio driver in VM. In case that all these old entries have iova addresses lesser than new iova entries, vhost code will need to iterate all the cache to find the new ones. In case of just a new iova entry needed for the new translations, this condition will last forever.

This has been observed in virtio-net to testpmd's vfio-pci driver transition, reducing the performance from more than 10Mpps to less than 0.07Mpps if the hugepage address was higher than the networking buffers. Since all new buffers are contained in this new gigantic page, vhost needs to scan IOTLB_CACHE_SIZE - 1 for each translation at worst.

---
   lib/vhost/vhost_user.c | 2 ++
   1 file changed, 2 insertions(+)
lib/vhost/vhost_user.c +2 -0
index 29a4c9af60..7de48f5333 100644
@@ -2113,6 +2113,8 @@ vhost_user_get_vring_base(struct virtio_net **pdev,
   	msg->size = sizeof(msg->payload.state);
   	msg->fd_num = 0;
 
+	vhost_user_iotlb_flush_all(vq);
+
   	vring_invalidate(dev, vq);
 
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

lib/vhost/vhost_user.c vhost_user_get_vring_base(struct virtio_net **pdev, · line 2120

Quoted 2 lines
lib/vhost/vhost_user.c +0 -0
   	return RTE_VHOST_MSG_RESULT_REPLY;

So you will fix title and add fix tag when applying, right?

Yes, I will. Thanks for the heads-up!

Cool! With both fixed:

Quoted 9 lines

Maxime

Quoted 6 lines

/Chenbo

Quoted 3 lines

Thanks,
Maxime

#9 lore raw
In reply to Eugenio Pérez
Quoted 12 lines · Eugenio Pérez

Old IOVA cache entries are left when there is a change on virtio driver in VM. In case that all these old entries have iova addresses lesser than new iova entries, vhost code will need to iterate all the cache to find the new ones. In case of just a new iova entry needed for the new translations, this condition will last forever.

This has been observed in virtio-net to testpmd's vfio-pci driver transition, reducing the performance from more than 10Mpps to less than 0.07Mpps if the hugepage address was higher than the networking buffers. Since all new buffers are contained in this new gigantic page, vhost needs to scan IOTLB_CACHE_SIZE - 1 for each translation at worst.

Quoted 6 lines
---
  lib/vhost/vhost_user.c | 2 ++
  1 file changed, 2 insertions(+)

Applied to dpdk-next-virtio/main with changes suggested by Chenbo.

Thanks,
Maxime