Skip to contentSkip to conversation

hw/virtio/vhost: Disable IOTLB callbacks when IOMMU gets disabled

PATCH In review Eric Auger 7 participants 34 messages Jan 20, 2025 – Feb 20, 2025

lore
Share & tools
lore

Eric Auger

author
#0 lore raw

PATCH 1/1 hw/virtio/vhost: Disable IOTLB callbacks when IOMMU gets disabled

When a guest exposed with a vhost device and protected by an intel IOMMU gets rebooted, we sometimes observe a spurious warning:

Fail to lookup the translated address ffffe000

We observe that the IOMMU gets disabled through a write to the global command register (CMAR_GCMD.TE) before the vhost device gets stopped. When this warning happens it can be observed an inflight IOTLB miss occurs after the IOMMU disable and before the vhost stop. In that case a flat translation occurs and the check in vhost_memory_region_lookup() fails.

Let's disable the IOTLB callbacks when all IOMMU MRs have been unregistered.

---
 hw/virtio/vhost.c | 4 ++++
 1 file changed, 4 insertions(+)
hw/virtio/vhost.c +4 -0
index 6aa72fd434..128c2ab094 100644
@@ -931,6 +931,10 @@ static void vhost_iommu_region_del(MemoryListener *listener,
             break;
         }
     }
+    if (QLIST_EMPTY(&dev->iommu_list) &&
+        dev->vhost_ops->vhost_set_iotlb_callback) {
+        dev->vhost_ops->vhost_set_iotlb_callback(dev, false);
+    }
 }
 
 void vhost_toggle_device_iotlb(VirtIODevice *vdev)
Signature · 1 line
2.47.1
#1 lore raw
Quoted 26 lines · Eric Auger

When a guest exposed with a vhost device and protected by an intel IOMMU gets rebooted, we sometimes observe a spurious warning:

Fail to lookup the translated address ffffe000

We observe that the IOMMU gets disabled through a write to the global command register (CMAR_GCMD.TE) before the vhost device gets stopped. When this warning happens it can be observed an inflight IOTLB miss occurs after the IOMMU disable and before the vhost stop. In that case a flat translation occurs and the check in vhost_memory_region_lookup() fails.

Let's disable the IOTLB callbacks when all IOMMU MRs have been unregistered.

---
 hw/virtio/vhost.c | 4 ++++
 1 file changed, 4 insertions(+)
hw/virtio/vhost.c +0 -0
index 6aa72fd434..128c2ab094 100644

hw/virtio/vhost.c static void vhost_iommu_region_del(MemoryListener *listener, · line 937

Quoted 7 lines
hw/virtio/vhost.c +4 -0
@@ -931,6 +931,10 @@ static void vhost_iommu_region_del(MemoryListener *listener,
             break;
         }
     }
+    if (QLIST_EMPTY(&dev->iommu_list) &&
+        dev->vhost_ops->vhost_set_iotlb_callback) {
+        dev->vhost_ops->vhost_set_iotlb_callback(dev, false);
+    }

So the current code assumes:

1) IOMMU is enabled before vhost starts
2) IOMMU is disabled after vhost stops

This patch seems to fix 2) but not 1). Do we need to deal with the IOMMU enabled after vhost starts?

Thanks

Quoted 6 lines
hw/virtio/vhost.c +0 -1
 }
 
 void vhost_toggle_device_iotlb(VirtIODevice *vdev)
--

2.47.1

author
#2 lore raw

Hi Jason,

Quoted 35 lines · Jason Wang
Quoted 25 lines · Eric Auger

When a guest exposed with a vhost device and protected by an intel IOMMU gets rebooted, we sometimes observe a spurious warning:

Fail to lookup the translated address ffffe000

We observe that the IOMMU gets disabled through a write to the global command register (CMAR_GCMD.TE) before the vhost device gets stopped. When this warning happens it can be observed an inflight IOTLB miss occurs after the IOMMU disable and before the vhost stop. In that case a flat translation occurs and the check in vhost_memory_region_lookup() fails.

Let's disable the IOTLB callbacks when all IOMMU MRs have been unregistered.

---
 hw/virtio/vhost.c | 4 ++++
 1 file changed, 4 insertions(+)
hw/virtio/vhost.c +0 -0
index 6aa72fd434..128c2ab094 100644

hw/virtio/vhost.c static void vhost_iommu_region_del(MemoryListener *listener, · line 937

Quoted 7 lines
hw/virtio/vhost.c +4 -0
@@ -931,6 +931,10 @@ static void vhost_iommu_region_del(MemoryListener *listener,
             break;
         }
     }
+    if (QLIST_EMPTY(&dev->iommu_list) &&
+        dev->vhost_ops->vhost_set_iotlb_callback) {
+        dev->vhost_ops->vhost_set_iotlb_callback(dev, false);
+    }

So the current code assumes:

Quoted 5 lines

1) IOMMU is enabled before vhost starts
2) IOMMU is disabled after vhost stops

This patch seems to fix 2) but not 1). Do we need to deal with the IOMMU enabled after vhost starts?

This patch handles the case where the IOMMU is disabled *before* vhost stops (not 2). This is what I concretely observe on guest reboot.

But maybe I misunderstood your comments/questions?

Thanks

Eric

Quoted 9 lines

Thanks

Quoted 6 lines
hw/virtio/vhost.c +0 -1
 }
 
 void vhost_toggle_device_iotlb(VirtIODevice *vdev)
--

2.47.1

author
#3 lore raw
In reply to Jason Wang

Hi Jason,

Quoted 35 lines · Jason Wang
Quoted 25 lines · Eric Auger

When a guest exposed with a vhost device and protected by an intel IOMMU gets rebooted, we sometimes observe a spurious warning:

Fail to lookup the translated address ffffe000

We observe that the IOMMU gets disabled through a write to the global command register (CMAR_GCMD.TE) before the vhost device gets stopped. When this warning happens it can be observed an inflight IOTLB miss occurs after the IOMMU disable and before the vhost stop. In that case a flat translation occurs and the check in vhost_memory_region_lookup() fails.

Let's disable the IOTLB callbacks when all IOMMU MRs have been unregistered.

---
 hw/virtio/vhost.c | 4 ++++
 1 file changed, 4 insertions(+)
hw/virtio/vhost.c +0 -0
index 6aa72fd434..128c2ab094 100644

hw/virtio/vhost.c static void vhost_iommu_region_del(MemoryListener *listener, · line 937

Quoted 7 lines
hw/virtio/vhost.c +4 -0
@@ -931,6 +931,10 @@ static void vhost_iommu_region_del(MemoryListener *listener,
             break;
         }
     }
+    if (QLIST_EMPTY(&dev->iommu_list) &&
+        dev->vhost_ops->vhost_set_iotlb_callback) {
+        dev->vhost_ops->vhost_set_iotlb_callback(dev, false);
+    }

So the current code assumes:

Quoted 5 lines

1) IOMMU is enabled before vhost starts
2) IOMMU is disabled after vhost stops

This patch seems to fix 2) but not 1). Do we need to deal with the IOMMU enabled after vhost starts?

sorry I initially misunderstood the above comment. Indeed in the reboot case assumption 2) happens to be wrong. However what I currently do is:
stop listening to iotlb miss requests from the kernel because my understanding is those requests are just spurious ones, generate warnings and we do not care since we are rebooting the system.

However I do not claim this could handle the case where the IOMMU MR would be turned off and then turned on. I think in that case we should also flush the kernel IOTLB and this is not taken care of in this patch. Is it a relevant use case?

wrt removing assumption 1) and allow IOMMU enabled after vhost start. Is that a valid use case as the virtio driver is using the dma api?

Eric

Quoted 9 lines

Thanks

Quoted 6 lines
hw/virtio/vhost.c +0 -1
 }
 
 void vhost_toggle_device_iotlb(VirtIODevice *vdev)
--

2.47.1

#4 lore raw
Quoted 52 lines · Eric Auger

Hi Jason,

Quoted 35 lines · Jason Wang
Quoted 25 lines · Eric Auger

When a guest exposed with a vhost device and protected by an intel IOMMU gets rebooted, we sometimes observe a spurious warning:

Fail to lookup the translated address ffffe000

We observe that the IOMMU gets disabled through a write to the global command register (CMAR_GCMD.TE) before the vhost device gets stopped. When this warning happens it can be observed an inflight IOTLB miss occurs after the IOMMU disable and before the vhost stop. In that case a flat translation occurs and the check in vhost_memory_region_lookup() fails.

Let's disable the IOTLB callbacks when all IOMMU MRs have been unregistered.

---
 hw/virtio/vhost.c | 4 ++++
 1 file changed, 4 insertions(+)
hw/virtio/vhost.c +0 -0
index 6aa72fd434..128c2ab094 100644

hw/virtio/vhost.c static void vhost_iommu_region_del(MemoryListener *listener, · line 937

Quoted 7 lines
hw/virtio/vhost.c +4 -0
@@ -931,6 +931,10 @@ static void vhost_iommu_region_del(MemoryListener *listener,
             break;
         }
     }
+    if (QLIST_EMPTY(&dev->iommu_list) &&
+        dev->vhost_ops->vhost_set_iotlb_callback) {
+        dev->vhost_ops->vhost_set_iotlb_callback(dev, false);
+    }

So the current code assumes:

Quoted 5 lines

1) IOMMU is enabled before vhost starts
2) IOMMU is disabled after vhost stops

This patch seems to fix 2) but not 1). Do we need to deal with the IOMMU enabled after vhost starts?

sorry I initially misunderstood the above comment. Indeed in the reboot case assumption 2) happens to be wrong. However what I currently do is:
stop listening to iotlb miss requests from the kernel because my understanding is those requests are just spurious ones, generate warnings and we do not care since we are rebooting the system.

Quoted 4 lines

However I do not claim this could handle the case where the IOMMU MR would be turned off and then turned on. I think in that case we should also flush the kernel IOTLB and this is not taken care of in this patch. Is it a relevant use case?

Not sure.

Quoted 3 lines

wrt removing assumption 1) and allow IOMMU enabled after vhost start. Is that a valid use case as the virtio driver is using the dma api?

It should not be but we can't assume the behaviour of the guest. It could be buggy or even malicious.

Btw, we had the following codes while handling te:

/* Handle Translation Enable/Disable */
static void vtd_handle_gcmd_te(IntelIOMMUState *s, bool en)

{
    if (s->dmar_enabled == en) {
        return;
    }

    trace_vtd_dmar_enable(en);

...

    vtd_reset_caches(s);
    vtd_address_space_refresh_all(s);
}

vtd_address_space_refresh_all() will basically disable the iommu memory region. It looks not sufficient to trigger the region_del callback, maybe we should delete the region or introduce listener callback?

Thanks

Quoted 14 lines

Eric

Quoted 9 lines

Thanks

Quoted 6 lines

}

void vhost_toggle_device_iotlb(VirtIODevice *vdev)
--
2.47.1

author
#5 lore raw

Hi Jason,

Quoted 55 lines · Jason Wang
Quoted 50 lines · Eric Auger

Hi Jason,

Quoted 35 lines · Jason Wang
Quoted 25 lines · Eric Auger

When a guest exposed with a vhost device and protected by an intel IOMMU gets rebooted, we sometimes observe a spurious warning:

Fail to lookup the translated address ffffe000

We observe that the IOMMU gets disabled through a write to the global command register (CMAR_GCMD.TE) before the vhost device gets stopped. When this warning happens it can be observed an inflight IOTLB miss occurs after the IOMMU disable and before the vhost stop. In that case a flat translation occurs and the check in vhost_memory_region_lookup() fails.

Let's disable the IOTLB callbacks when all IOMMU MRs have been unregistered.

---
 hw/virtio/vhost.c | 4 ++++
 1 file changed, 4 insertions(+)
hw/virtio/vhost.c +0 -0
index 6aa72fd434..128c2ab094 100644

hw/virtio/vhost.c static void vhost_iommu_region_del(MemoryListener *listener, · line 937

Quoted 7 lines
hw/virtio/vhost.c +4 -0
@@ -931,6 +931,10 @@ static void vhost_iommu_region_del(MemoryListener *listener,
             break;
         }
     }
+    if (QLIST_EMPTY(&dev->iommu_list) &&
+        dev->vhost_ops->vhost_set_iotlb_callback) {
+        dev->vhost_ops->vhost_set_iotlb_callback(dev, false);
+    }

So the current code assumes:

Quoted 5 lines

1) IOMMU is enabled before vhost starts
2) IOMMU is disabled after vhost stops

This patch seems to fix 2) but not 1). Do we need to deal with the IOMMU enabled after vhost starts?

sorry I initially misunderstood the above comment. Indeed in the reboot case assumption 2) happens to be wrong. However what I currently do is:
stop listening to iotlb miss requests from the kernel because my understanding is those requests are just spurious ones, generate warnings and we do not care since we are rebooting the system.

Quoted 4 lines

However I do not claim this could handle the case where the IOMMU MR would be turned off and then turned on. I think in that case we should also flush the kernel IOTLB and this is not taken care of in this patch. Is it a relevant use case?

Quoted 6 lines

Not sure.

Quoted 2 lines

wrt removing assumption 1) and allow IOMMU enabled after vhost start. Is that a valid use case as the virtio driver is using the dma api?

It should not be but we can't assume the behaviour of the guest. It could be buggy or even malicious.

agreed

Quoted 14 lines

Btw, we had the following codes while handling te:

/* Handle Translation Enable/Disable */
static void vtd_handle_gcmd_te(IntelIOMMUState *s, bool en)

{
    if (s->dmar_enabled == en) {
        return;
    }

    trace_vtd_dmar_enable(en);

...

Quoted 8 lines
    vtd_reset_caches(s);
    vtd_address_space_refresh_all(s);
}

vtd_address_space_refresh_all() will basically disable the iommu memory region. It looks not sufficient to trigger the region_del callback, maybe we should delete the region or introduce listener callback?

This is exactly the code path which is entered in my use case.

vtd_address_space_refresh_all(s) induces the vhost_iommu_region_del. But given the current implement of this latter the IOTLB callback is not unset and the kernel IOTLB is not refreshed. Also as I pointed out the hdev->mem->regions are not updated? shouldn't they. Can you explain what they correspond to?

Thanks

Eric

Quoted 14 lines

Thanks

Quoted 11 lines

Eric

Quoted 8 lines

Thanks

Quoted 6 lines

}

void vhost_toggle_device_iotlb(VirtIODevice *vdev)
--
2.47.1

#6 lore raw
Quoted 86 lines · Eric Auger

Hi Jason,

Quoted 55 lines · Jason Wang
Quoted 50 lines · Eric Auger

Hi Jason,

Quoted 35 lines · Jason Wang
Quoted 25 lines · Eric Auger

When a guest exposed with a vhost device and protected by an intel IOMMU gets rebooted, we sometimes observe a spurious warning:

Fail to lookup the translated address ffffe000

We observe that the IOMMU gets disabled through a write to the global command register (CMAR_GCMD.TE) before the vhost device gets stopped. When this warning happens it can be observed an inflight IOTLB miss occurs after the IOMMU disable and before the vhost stop. In that case a flat translation occurs and the check in vhost_memory_region_lookup() fails.

Let's disable the IOTLB callbacks when all IOMMU MRs have been unregistered.

---
 hw/virtio/vhost.c | 4 ++++
 1 file changed, 4 insertions(+)
hw/virtio/vhost.c +0 -0
index 6aa72fd434..128c2ab094 100644

hw/virtio/vhost.c static void vhost_iommu_region_del(MemoryListener *listener, · line 937

Quoted 7 lines
hw/virtio/vhost.c +4 -0
@@ -931,6 +931,10 @@ static void vhost_iommu_region_del(MemoryListener *listener,
             break;
         }
     }
+    if (QLIST_EMPTY(&dev->iommu_list) &&
+        dev->vhost_ops->vhost_set_iotlb_callback) {
+        dev->vhost_ops->vhost_set_iotlb_callback(dev, false);
+    }

So the current code assumes:

Quoted 5 lines

1) IOMMU is enabled before vhost starts
2) IOMMU is disabled after vhost stops

This patch seems to fix 2) but not 1). Do we need to deal with the IOMMU enabled after vhost starts?

sorry I initially misunderstood the above comment. Indeed in the reboot case assumption 2) happens to be wrong. However what I currently do is:
stop listening to iotlb miss requests from the kernel because my understanding is those requests are just spurious ones, generate warnings and we do not care since we are rebooting the system.

Quoted 4 lines

However I do not claim this could handle the case where the IOMMU MR would be turned off and then turned on. I think in that case we should also flush the kernel IOTLB and this is not taken care of in this patch. Is it a relevant use case?

Quoted 6 lines

Not sure.

Quoted 2 lines

wrt removing assumption 1) and allow IOMMU enabled after vhost start. Is that a valid use case as the virtio driver is using the dma api?

It should not be but we can't assume the behaviour of the guest. It could be buggy or even malicious.

agreed

Quoted 18 lines

Btw, we had the following codes while handling te:

/* Handle Translation Enable/Disable */
static void vtd_handle_gcmd_te(IntelIOMMUState *s, bool en)

{
    if (s->dmar_enabled == en) {
        return;
    }

    trace_vtd_dmar_enable(en);

...

    vtd_reset_caches(s);
    vtd_address_space_refresh_all(s);
}
Quoted 8 lines
Quoted 4 lines

vtd_address_space_refresh_all() will basically disable the iommu memory region. It looks not sufficient to trigger the region_del callback, maybe we should delete the region or introduce listener callback?

This is exactly the code path which is entered in my use case.

vtd_address_space_refresh_all(s) induces the vhost_iommu_region_del. But given the current implement of this latter the IOTLB callback is not unset and the kernel IOTLB is not refreshed. Also as I pointed out the hdev->mem->regions are not updated? shouldn't they. Can you explain what they correspond to?

Adding Peter for more ideas.

I think it's better to find a way to trigger the listener here, probably:

1) add/delete the memory regions instead of enable/disable

or

2) introduce new listener ops that can be triggered when a region is enabled or disabled

Thanks

Quoted 20 lines

Thanks

Eric

Quoted 14 lines

Thanks

Quoted 11 lines

Eric

Quoted 8 lines

Thanks

Quoted 6 lines

}

void vhost_toggle_device_iotlb(VirtIODevice *vdev)
--
2.47.1

author
#7 lore raw

Hi Jason,

Quoted 87 lines · Jason Wang
Quoted 86 lines · Eric Auger

Hi Jason,

Quoted 54 lines · Jason Wang
Quoted 49 lines · Eric Auger

Hi Jason,

Quoted 35 lines · Jason Wang
Quoted 25 lines · Eric Auger

When a guest exposed with a vhost device and protected by an intel IOMMU gets rebooted, we sometimes observe a spurious warning:

Fail to lookup the translated address ffffe000

We observe that the IOMMU gets disabled through a write to the global command register (CMAR_GCMD.TE) before the vhost device gets stopped. When this warning happens it can be observed an inflight IOTLB miss occurs after the IOMMU disable and before the vhost stop. In that case a flat translation occurs and the check in vhost_memory_region_lookup() fails.

Let's disable the IOTLB callbacks when all IOMMU MRs have been unregistered.

---
 hw/virtio/vhost.c | 4 ++++
 1 file changed, 4 insertions(+)
hw/virtio/vhost.c +0 -0
index 6aa72fd434..128c2ab094 100644

hw/virtio/vhost.c static void vhost_iommu_region_del(MemoryListener *listener, · line 937

Quoted 7 lines
hw/virtio/vhost.c +4 -0
@@ -931,6 +931,10 @@ static void vhost_iommu_region_del(MemoryListener *listener,
             break;
         }
     }
+    if (QLIST_EMPTY(&dev->iommu_list) &&
+        dev->vhost_ops->vhost_set_iotlb_callback) {
+        dev->vhost_ops->vhost_set_iotlb_callback(dev, false);
+    }

So the current code assumes:

Quoted 5 lines

1) IOMMU is enabled before vhost starts
2) IOMMU is disabled after vhost stops

This patch seems to fix 2) but not 1). Do we need to deal with the IOMMU enabled after vhost starts?

sorry I initially misunderstood the above comment. Indeed in the reboot case assumption 2) happens to be wrong. However what I currently do is:
stop listening to iotlb miss requests from the kernel because my understanding is those requests are just spurious ones, generate warnings and we do not care since we are rebooting the system.

Quoted 4 lines

However I do not claim this could handle the case where the IOMMU MR would be turned off and then turned on. I think in that case we should also flush the kernel IOTLB and this is not taken care of in this patch. Is it a relevant use case?

Quoted 6 lines

Not sure.

Quoted 2 lines

wrt removing assumption 1) and allow IOMMU enabled after vhost start. Is that a valid use case as the virtio driver is using the dma api?

It should not be but we can't assume the behaviour of the guest. It could be buggy or even malicious.

agreed

Quoted 21 lines

Btw, we had the following codes while handling te:

/* Handle Translation Enable/Disable */
static void vtd_handle_gcmd_te(IntelIOMMUState *s, bool en)

{
    if (s->dmar_enabled == en) {
        return;
    }

    trace_vtd_dmar_enable(en);

...

    vtd_reset_caches(s);
    vtd_address_space_refresh_all(s);
}

vtd_address_space_refresh_all() will basically disable the iommu memory region. It looks not sufficient to trigger the region_del callback, maybe we should delete the region or introduce listener callback?

Quoted 8 lines
Quoted 3 lines

This is exactly the code path which is entered in my use case.

vtd_address_space_refresh_all(s) induces the vhost_iommu_region_del. But given the current implement of this latter the IOTLB callback is not unset and the kernel IOTLB is not refreshed. Also as I pointed out the hdev->mem->regions are not updated? shouldn't they. Can you explain what they correspond to?

Adding Peter for more ideas.

I think it's better to find a way to trigger the listener here, probably:

1) add/delete the memory regions instead of enable/disable

sorry I don't understand what you mean. The vhost_iommu_region_del call stack is provided below [1]. Write to the intel iommu GCMD TE bit induces a call to vhost_iommu_region_del. This happens before the vhost_dev_stop whose call stack is provided below [2] and originates from a bus reset.

We may have inflight IOTLB miss requests happening between both.

If this happens, vhost_device_iotlb_miss() fails because the IOVA is not translated anymore by the IOMMU and the iotlb.translated_addr returned by address_space_get_iotlb_entry() is not within the registered vhost_memory_regions looked up in vhost_memory_region_lookup(), hence the "Fail to lookup the translated address" message.

It sounds wrong that vhost keeps on using IOVAs that are not translated anymore. It looks we have a reset ordering issue and this patch is just removing the sympton and not the cause.

At the moment I don't really get what is initiating the intel iommu TE bit write. This is a guest action but is it initiated from a misordered qemu event?

It could also relate to
[PATCH 0/4] intel_iommu: Reset vIOMMU after all the rest of devices
https://lore.kernel.org/all/?q=s%3Aintel_iommu%3A+Reset+vIOMMU+after+all+the+rest+of+devices

Thanks

Eric

[1]
#0  vhost_iommu_region_del (listener=0x55555708a388, section=0x7ffff62bf030) at ../hw/virtio/vhost.c:927
#1  0x0000555555c851b4 in address_space_update_topology_pass (as=0x55555822e4f0, old_view=0x7fffe8e81850, new_view=0x55555723dfa0, adding=false) at ../system/memory.c:977
#2  0x0000555555c857a0 in address_space_set_flatview (as=0x55555822e4f0) at ../system/memory.c:1079
#3  0x0000555555c85986 in memory_region_transaction_commit () at ../system/memory.c:1132
#4  0x0000555555c89f25 in memory_region_set_enabled (mr=0x555557dc0d30, enabled=false) at ../system/memory.c:2686
#5  0x0000555555b5edb1 in vtd_switch_address_space (as=0x555557dc0c60) at ../hw/i386/intel_iommu.c:1735
#6  0x0000555555b5ee6f in vtd_switch_address_space_all (s=0x555557db1500) at ../hw/i386/intel_iommu.c:1779
#7  0x0000555555b64533 in vtd_address_space_refresh_all (s=0x555557db1500) at ../hw/i386/intel_iommu.c:4006
#8  0x0000555555b60770 in vtd_handle_gcmd_te (s=0x555557db1500, en=false) at ../hw/i386/intel_iommu.c:2419
#9  0x0000555555b60885 in vtd_handle_gcmd_write (s=0x555557db1500) at ../hw/i386/intel_iommu.c:2449
#10 0x0000555555b61db2 in vtd_mem_write (opaque=0x555557db1500, addr=24, val=100663296, size=4) at ../hw/i386/intel_iommu.c:2991
#11 0x0000555555c833e0 in memory_region_write_accessor (mr=0x555557db1840, addr=24, value=0x7ffff62bf3d8, size=4, shift=0, mask=4294967295, attrs=...) at ../system/memory.c:497
#12 0x0000555555c83711 in access_with_adjusted_size (addr=24, value=0x7ffff62bf3d8, size=4, access_size_min=4, access_size_max=8, access_fn=0x555555c832ea <memory_region_write_accessor>, mr=0x555557db1840, attrs=...) at ../system/memory.c:573
#13 0x0000555555c8698b in memory_region_dispatch_write (mr=0x555557db1840, addr=24, data=100663296, op=MO_32, attrs=...) at ../system/memory.c:1521
#14 0x0000555555c95619 in flatview_write_continue_step (attrs=..., buf=0x7ffff7fbb028 "", len=4, mr_addr=24, l=0x7ffff62bf4c0, mr=0x555557db1840) at ../system/physmem.c:2803
#15 0x0000555555c956eb in flatview_write_continue (fv=0x7fffe852d370, addr=4275634200, attrs=..., ptr=0x7ffff7fbb028, len=4, mr_addr=24, l=4, mr=0x555557db1840) at ../system/physmem.c:2833
#16 0x0000555555c957f9 in flatview_write (fv=0x7fffe852d370, addr=4275634200, attrs=..., buf=0x7ffff7fbb028, len=4) at ../system/physmem.c:2864
#17 0x0000555555c95c39 in address_space_write (as=0x555556ff1720 <address_space_memory>, addr=4275634200, attrs=..., buf=0x7ffff7fbb028, len=4) at ../system/physmem.c:2984
#18 0x0000555555c95cb1 in address_space_rw (as=0x555556ff1720 <address_space_memory>, addr=4275634200, attrs=..., buf=0x7ffff7fbb028, len=4, is_write=true) at ../system/physmem.c:2994
#19 0x0000555555ceeb56 in kvm_cpu_exec (cpu=0x55555727e950) at ../accel/kvm/kvm-all.c:3188
#20 0x0000555555cf1ea6 in kvm_vcpu_thread_fn (arg=0x55555727e950) at ../accel/kvm/kvm-accel-ops.c:50
#21 0x0000555555f6de20 in qemu_thread_start (args=0x555557288960) at ../util/qemu-thread-posix.c:541
#22 0x00007ffff7289e92 in start_thread () at /lib64/libc.so.6

[2]
#0  vhost_dev_stop (hdev=0x55555708a2c0, vdev=0x555558234cb0, vrings=false) at ../hw/virtio/vhost.c:2222
#1  0x0000555555990266 in vhost_net_stop_one (net=0x55555708a2c0, dev=0x555558234cb0) at ../hw/net/vhost_net.c:350
#2  0x00005555559906ea in vhost_net_stop (dev=0x555558234cb0, ncs=0x55555826f968, data_queue_pairs=1, cvq=0) at ../hw/net/vhost_net.c:462 #3  0x0000555555c1ad0a in virtio_net_vhost_status (n=0x555558234cb0, status=0 '\000') at ../hw/net/virtio-net.c:318
#4  0x0000555555c1afaf in virtio_net_set_status (vdev=0x555558234cb0, status=0 '\000') at ../hw/net/virtio-net.c:393
#5  0x0000555555c591bd in virtio_set_status (vdev=0x555558234cb0, val=0 '\000') at ../hw/virtio/virtio.c:2242
#6  0x0000555555c595eb in virtio_reset (opaque=0x555558234cb0) at ../hw/virtio/virtio.c:2325
#7  0x0000555555a0d1e1 in virtio_bus_reset (bus=0x555558234c30) at ../hw/virtio/virtio-bus.c:109
#8  0x0000555555a13d51 in virtio_pci_reset (qdev=0x55555822c830) at ../hw/virtio/virtio-pci.c:2282
#9  0x0000555555a14016 in virtio_pci_bus_reset_hold (obj=0x55555822c830, type=RESET_TYPE_COLD) at ../hw/virtio/virtio-pci.c:2322
#10 0x0000555555d08831 in resettable_phase_hold (obj=0x55555822c830, opaque=0x0, type=RESET_TYPE_COLD) at ../hw/core/resettable.c:180 #11 0x0000555555d00fc5 in bus_reset_child_foreach (obj=0x555557ffa3c0, cb=0x555555d086d5 <resettable_phase_hold>, opaque=0x0, type=RESET_TYPE_COLD) at ../hw/core/bus.c:97
#12 0x0000555555d084d8 in resettable_child_foreach (rc=0x555557146f20, obj=0x555557ffa3c0, cb=0x555555d086d5 <resettable_phase_hold>, opaque=0x0, type=RESET_TYPE_COLD) at ../hw/core/resettable.c:92 #13 0x0000555555d08792 in resettable_phase_hold (obj=0x555557ffa3c0, opaque=0x0, type=RESET_TYPE_COLD) at ../hw/core/resettable.c:169 #14 0x0000555555d0543b in device_reset_child_foreach (obj=0x555557ff98e0, cb=0x555555d086d5 <resettable_phase_hold>, opaque=0x0, type=RESET_TYPE_COLD) at ../hw/core/qdev.c:275
#15 0x0000555555d084d8 in resettable_child_foreach (rc=0x55555715a090, obj=0x555557ff98e0, cb=0x555555d086d5 <resettable_phase_hold>, opaque=0x0, type=RESET_TYPE_COLD) at ../hw/core/resettable.c:92 #16 0x0000555555d08792 in resettable_phase_hold (obj=0x555557ff98e0, opaque=0x0, type=RESET_TYPE_COLD) at ../hw/core/resettable.c:169 #17 0x0000555555d00fc5 in bus_reset_child_foreach (obj=0x555557445120, cb=0x555555d086d5 <resettable_phase_hold>, opaque=0x0, type=RESET_TYPE_COLD) at ../hw/core/bus.c:97
#18 0x0000555555d084d8 in resettable_child_foreach (rc=0x555557146f20, obj=0x555557445120, cb=0x555555d086d5 <resettable_phase_hold>, opaque=0x0, type=RESET_TYPE_COLD) at ../hw/core/resettable.c:92 #19 0x0000555555d08792 in resettable_phase_hold (obj=0x555557445120, opaque=0x0, type=RESET_TYPE_COLD) at ../hw/core/resettable.c:169 #20 0x0000555555d0543b in device_reset_child_foreach (obj=0x5555572d0a00, cb=0x555555d086d5 <resettable_phase_hold>, opaque=0x0, type=RESET_TYPE_COLD) at ../hw/core/qdev.c:275
#21 0x0000555555d084d8 in resettable_child_foreach (rc=0x5555570cf410, obj=0x5555572d0a00, cb=0x555555d086d5 <resettable_phase_hold>, opaque=0x0, type=RESET_TYPE_COLD) at ../hw/core/resettable.c:92 #22 0x0000555555d08792 in resettable_phase_hold (obj=0x5555572d0a00, opaque=0x0, type=RESET_TYPE_COLD) at ../hw/core/resettable.c:169 #23 0x0000555555d00fc5 in bus_reset_child_foreach (obj=0x55555723d270, cb=0x555555d086d5 <resettable_phase_hold>, opaque=0x0, type=RESET_TYPE_COLD) at ../hw/core/bus.c:97
#24 0x0000555555d084d8 in resettable_child_foreach (rc=0x5555571bfde0, obj=0x55555723d270, cb=0x555555d086d5 <resettable_phase_hold>, opaque=0x0, type=RESET_TYPE_COLD) at ../hw/core/resettable.c:92 #25 0x0000555555d08792 in resettable_phase_hold (obj=0x55555723d270, opaque=0x0, type=RESET_TYPE_COLD) at ../hw/core/resettable.c:169 #26 0x0000555555d06d6d in resettable_container_child_foreach (obj=0x55555724a8a0, cb=0x555555d086d5 <resettable_phase_hold>, opaque=0x0, type=RESET_TYPE_COLD) at ../hw/core/resetcontainer.c:54 #27 0x0000555555d084d8 in resettable_child_foreach (rc=0x5555572180f0, obj=0x55555724a8a0, cb=0x555555d086d5 <resettable_phase_hold>, opaque=0x0, type=RESET_TYPE_COLD) at ../hw/core/resettable.c:92 #28 0x0000555555d08792 in resettable_phase_hold (obj=0x55555724a8a0, opaque=0x0, type=RESET_TYPE_COLD) at ../hw/core/resettable.c:169 #29 0x0000555555d0838d in resettable_assert_reset (obj=0x55555724a8a0, type=RESET_TYPE_COLD) at ../hw/core/resettable.c:58
#30 0x0000555555d082e5 in resettable_reset (obj=0x55555724a8a0, type=RESET_TYPE_COLD) at ../hw/core/resettable.c:45
#31 0x00005555558db5c8 in qemu_devices_reset (reason=SHUTDOWN_CAUSE_GUEST_RESET) at ../hw/core/reset.c:179 #32 0x0000555555b6f5b2 in pc_machine_reset (machine=0x555557234490, reason=SHUTDOWN_CAUSE_GUEST_RESET) at ../hw/i386/pc.c:1877
#33 0x0000555555a5a0a2 in qemu_system_reset (reason=SHUTDOWN_CAUSE_GUEST_RESET) at ../system/runstate.c:516 #34 0x0000555555a5a891 in main_loop_should_exit (status=0x7fffffffd574) at ../system/runstate.c:792
#35 0x0000555555a5a992 in qemu_main_loop () at ../system/runstate.c:825 #36 0x0000555555e9cced in qemu_default_main () at ../system/main.c:37 #37 0x0000555555e9cd2a in main (argc=58, argv=0x7fffffffd6d8) at ../system/main.c:48

Quoted 25 lines

or

2) introduce new listener ops that can be triggered when a region is enabled or disabled

Thanks

Quoted 17 lines

Thanks

Eric

Quoted 13 lines

Thanks

Quoted 11 lines

Eric

Quoted 8 lines

Thanks

Quoted 6 lines

}

void vhost_toggle_device_iotlb(VirtIODevice *vdev)
--
2.47.1

#8 lore raw
Quoted 97 lines · Eric Auger

Hi Jason,

Quoted 92 lines · Jason Wang
Quoted 82 lines · Eric Auger

Hi Jason,

Quoted 54 lines · Jason Wang
Quoted 49 lines · Eric Auger

Hi Jason,

Quoted 35 lines · Jason Wang
Quoted 25 lines · Eric Auger

When a guest exposed with a vhost device and protected by an intel IOMMU gets rebooted, we sometimes observe a spurious warning:

Fail to lookup the translated address ffffe000

We observe that the IOMMU gets disabled through a write to the global command register (CMAR_GCMD.TE) before the vhost device gets stopped. When this warning happens it can be observed an inflight IOTLB miss occurs after the IOMMU disable and before the vhost stop. In that case a flat translation occurs and the check in vhost_memory_region_lookup() fails.

Let's disable the IOTLB callbacks when all IOMMU MRs have been unregistered.

---
 hw/virtio/vhost.c | 4 ++++
 1 file changed, 4 insertions(+)
hw/virtio/vhost.c +0 -0
index 6aa72fd434..128c2ab094 100644

hw/virtio/vhost.c static void vhost_iommu_region_del(MemoryListener *listener, · line 937

Quoted 7 lines
hw/virtio/vhost.c +4 -0
@@ -931,6 +931,10 @@ static void vhost_iommu_region_del(MemoryListener *listener,
             break;
         }
     }
+    if (QLIST_EMPTY(&dev->iommu_list) &&
+        dev->vhost_ops->vhost_set_iotlb_callback) {
+        dev->vhost_ops->vhost_set_iotlb_callback(dev, false);
+    }

So the current code assumes:

Quoted 5 lines

1) IOMMU is enabled before vhost starts
2) IOMMU is disabled after vhost stops

This patch seems to fix 2) but not 1). Do we need to deal with the IOMMU enabled after vhost starts?

sorry I initially misunderstood the above comment. Indeed in the reboot case assumption 2) happens to be wrong. However what I currently do is:
stop listening to iotlb miss requests from the kernel because my understanding is those requests are just spurious ones, generate warnings and we do not care since we are rebooting the system.

Quoted 4 lines

However I do not claim this could handle the case where the IOMMU MR would be turned off and then turned on. I think in that case we should also flush the kernel IOTLB and this is not taken care of in this patch. Is it a relevant use case?

Quoted 6 lines

Not sure.

Quoted 2 lines

wrt removing assumption 1) and allow IOMMU enabled after vhost start. Is that a valid use case as the virtio driver is using the dma api?

It should not be but we can't assume the behaviour of the guest. It could be buggy or even malicious.

agreed

Quoted 17 lines

Btw, we had the following codes while handling te:

/* Handle Translation Enable/Disable */
static void vtd_handle_gcmd_te(IntelIOMMUState *s, bool en)

{
    if (s->dmar_enabled == en) {
        return;
    }

    trace_vtd_dmar_enable(en);

...

    vtd_reset_caches(s);
    vtd_address_space_refresh_all(s);
}
Quoted 7 lines
Quoted 4 lines

vtd_address_space_refresh_all() will basically disable the iommu memory region. It looks not sufficient to trigger the region_del callback, maybe we should delete the region or introduce listener callback?

This is exactly the code path which is entered in my use case.

vtd_address_space_refresh_all(s) induces the vhost_iommu_region_del. But given the current implement of this latter the IOTLB callback is not unset and the kernel IOTLB is not refreshed. Also as I pointed out the hdev->mem->regions are not updated? shouldn't they. Can you explain what they correspond to?

Adding Peter for more ideas.

Quoted 8 lines
Quoted 3 lines

I think it's better to find a way to trigger the listener here, probably:

1) add/delete the memory regions instead of enable/disable

sorry I don't understand what you mean. The vhost_iommu_region_del call stack is provided below [1]. Write to the intel iommu GCMD TE bit induces a call to vhost_iommu_region_del. This happens before the vhost_dev_stop whose call stack is provided below [2] and originates from a bus reset.

Right, I see.

Quoted 9 lines

We may have inflight IOTLB miss requests happening between both.

If this happens, vhost_device_iotlb_miss() fails because the IOVA is not translated anymore by the IOMMU and the iotlb.translated_addr returned by address_space_get_iotlb_entry() is not within the registered vhost_memory_regions looked up in vhost_memory_region_lookup(), hence the "Fail to lookup the translated address" message.

Quoted 3 lines

It sounds wrong that vhost keeps on using IOVAs that are not translated anymore. It looks we have a reset ordering issue and this patch is just removing the sympton and not the cause.

Exactly.

Quoted 4 lines

At the moment I don't really get what is initiating the intel iommu TE bit write. This is a guest action but is it initiated from a misordered qemu event?

It could also be a guest bug which reset the vIOMMU before the device. Do we have a calltrace in the guest?

Adding more vtd maintiners for more thoughts.

Thanks

Quoted 185 lines

It could also relate to
[PATCH 0/4] intel_iommu: Reset vIOMMU after all the rest of devices
https://lore.kernel.org/all/?q=s%3Aintel_iommu%3A+Reset+vIOMMU+after+all+the+rest+of+devices

Thanks

Eric

[1]
#0 vhost_iommu_region_del (listener=0x55555708a388, section=0x7ffff62bf030) at ../hw/virtio/vhost.c:927
#1 0x0000555555c851b4 in address_space_update_topology_pass (as=0x55555822e4f0, old_view=0x7fffe8e81850, new_view=0x55555723dfa0, adding=false) at ../system/memory.c:977
#2 0x0000555555c857a0 in address_space_set_flatview (as=0x55555822e4f0) at ../system/memory.c:1079
#3 0x0000555555c85986 in memory_region_transaction_commit () at ../system/memory.c:1132
#4 0x0000555555c89f25 in memory_region_set_enabled (mr=0x555557dc0d30, enabled=false) at ../system/memory.c:2686
#5 0x0000555555b5edb1 in vtd_switch_address_space (as=0x555557dc0c60) at ../hw/i386/intel_iommu.c:1735
#6 0x0000555555b5ee6f in vtd_switch_address_space_all (s=0x555557db1500) at ../hw/i386/intel_iommu.c:1779
#7 0x0000555555b64533 in vtd_address_space_refresh_all (s=0x555557db1500) at ../hw/i386/intel_iommu.c:4006
#8 0x0000555555b60770 in vtd_handle_gcmd_te (s=0x555557db1500, en=false) at ../hw/i386/intel_iommu.c:2419
#9 0x0000555555b60885 in vtd_handle_gcmd_write (s=0x555557db1500) at ../hw/i386/intel_iommu.c:2449
#10 0x0000555555b61db2 in vtd_mem_write (opaque=0x555557db1500, addr=24, val=100663296, size=4) at ../hw/i386/intel_iommu.c:2991
#11 0x0000555555c833e0 in memory_region_write_accessor (mr=0x555557db1840, addr=24, value=0x7ffff62bf3d8, size=4, shift=0, mask=4294967295, attrs=...) at ../system/memory.c:497
#12 0x0000555555c83711 in access_with_adjusted_size

    (addr=24, value=0x7ffff62bf3d8, size=4, access_size_min=4,
access_size_max=8, access_fn=0x555555c832ea
<memory_region_write_accessor>, mr=0x555557db1840, attrs=...)
    at ../system/memory.c:573

#13 0x0000555555c8698b in memory_region_dispatch_write (mr=0x555557db1840, addr=24, data=100663296, op=MO_32, attrs=...) at ../system/memory.c:1521
#14 0x0000555555c95619 in flatview_write_continue_step (attrs=..., buf=0x7ffff7fbb028 "", len=4, mr_addr=24, l=0x7ffff62bf4c0, mr=0x555557db1840) at ../system/physmem.c:2803
#15 0x0000555555c956eb in flatview_write_continue (fv=0x7fffe852d370, addr=4275634200, attrs=..., ptr=0x7ffff7fbb028, len=4, mr_addr=24, l=4, mr=0x555557db1840) at ../system/physmem.c:2833
#16 0x0000555555c957f9 in flatview_write (fv=0x7fffe852d370, addr=4275634200, attrs=..., buf=0x7ffff7fbb028, len=4) at ../system/physmem.c:2864
#17 0x0000555555c95c39 in address_space_write (as=0x555556ff1720 <address_space_memory>, addr=4275634200, attrs=..., buf=0x7ffff7fbb028, len=4) at ../system/physmem.c:2984
#18 0x0000555555c95cb1 in address_space_rw (as=0x555556ff1720 <address_space_memory>, addr=4275634200, attrs=..., buf=0x7ffff7fbb028, len=4, is_write=true) at ../system/physmem.c:2994
#19 0x0000555555ceeb56 in kvm_cpu_exec (cpu=0x55555727e950) at ../accel/kvm/kvm-all.c:3188
#20 0x0000555555cf1ea6 in kvm_vcpu_thread_fn (arg=0x55555727e950) at ../accel/kvm/kvm-accel-ops.c:50
#21 0x0000555555f6de20 in qemu_thread_start (args=0x555557288960) at ../util/qemu-thread-posix.c:541
#22 0x00007ffff7289e92 in start_thread () at /lib64/libc.so.6

[2]
#0 vhost_dev_stop (hdev=0x55555708a2c0, vdev=0x555558234cb0, vrings=false) at ../hw/virtio/vhost.c:2222
#1 0x0000555555990266 in vhost_net_stop_one (net=0x55555708a2c0, dev=0x555558234cb0) at ../hw/net/vhost_net.c:350
#2 0x00005555559906ea in vhost_net_stop (dev=0x555558234cb0, ncs=0x55555826f968, data_queue_pairs=1, cvq=0) at ../hw/net/vhost_net.c:462 #3 0x0000555555c1ad0a in virtio_net_vhost_status (n=0x555558234cb0, status=0 '\000') at ../hw/net/virtio-net.c:318
#4 0x0000555555c1afaf in virtio_net_set_status (vdev=0x555558234cb0, status=0 '\000') at ../hw/net/virtio-net.c:393
#5 0x0000555555c591bd in virtio_set_status (vdev=0x555558234cb0, val=0 '\000') at ../hw/virtio/virtio.c:2242
#6 0x0000555555c595eb in virtio_reset (opaque=0x555558234cb0) at ../hw/virtio/virtio.c:2325
#7 0x0000555555a0d1e1 in virtio_bus_reset (bus=0x555558234c30) at ../hw/virtio/virtio-bus.c:109
#8 0x0000555555a13d51 in virtio_pci_reset (qdev=0x55555822c830) at ../hw/virtio/virtio-pci.c:2282
#9 0x0000555555a14016 in virtio_pci_bus_reset_hold (obj=0x55555822c830, type=RESET_TYPE_COLD) at ../hw/virtio/virtio-pci.c:2322
#10 0x0000555555d08831 in resettable_phase_hold (obj=0x55555822c830, opaque=0x0, type=RESET_TYPE_COLD) at ../hw/core/resettable.c:180 #11 0x0000555555d00fc5 in bus_reset_child_foreach (obj=0x555557ffa3c0, cb=0x555555d086d5 <resettable_phase_hold>, opaque=0x0, type=RESET_TYPE_COLD) at ../hw/core/bus.c:97
#12 0x0000555555d084d8 in resettable_child_foreach (rc=0x555557146f20, obj=0x555557ffa3c0, cb=0x555555d086d5 <resettable_phase_hold>, opaque=0x0, type=RESET_TYPE_COLD) at ../hw/core/resettable.c:92 #13 0x0000555555d08792 in resettable_phase_hold (obj=0x555557ffa3c0, opaque=0x0, type=RESET_TYPE_COLD) at ../hw/core/resettable.c:169 #14 0x0000555555d0543b in device_reset_child_foreach (obj=0x555557ff98e0, cb=0x555555d086d5 <resettable_phase_hold>, opaque=0x0, type=RESET_TYPE_COLD) at ../hw/core/qdev.c:275
#15 0x0000555555d084d8 in resettable_child_foreach (rc=0x55555715a090, obj=0x555557ff98e0, cb=0x555555d086d5 <resettable_phase_hold>, opaque=0x0, type=RESET_TYPE_COLD) at ../hw/core/resettable.c:92 #16 0x0000555555d08792 in resettable_phase_hold (obj=0x555557ff98e0, opaque=0x0, type=RESET_TYPE_COLD) at ../hw/core/resettable.c:169 #17 0x0000555555d00fc5 in bus_reset_child_foreach (obj=0x555557445120, cb=0x555555d086d5 <resettable_phase_hold>, opaque=0x0, type=RESET_TYPE_COLD) at ../hw/core/bus.c:97
#18 0x0000555555d084d8 in resettable_child_foreach (rc=0x555557146f20, obj=0x555557445120, cb=0x555555d086d5 <resettable_phase_hold>, opaque=0x0, type=RESET_TYPE_COLD) at ../hw/core/resettable.c:92 #19 0x0000555555d08792 in resettable_phase_hold (obj=0x555557445120, opaque=0x0, type=RESET_TYPE_COLD) at ../hw/core/resettable.c:169 #20 0x0000555555d0543b in device_reset_child_foreach (obj=0x5555572d0a00, cb=0x555555d086d5 <resettable_phase_hold>, opaque=0x0, type=RESET_TYPE_COLD) at ../hw/core/qdev.c:275
#21 0x0000555555d084d8 in resettable_child_foreach (rc=0x5555570cf410, obj=0x5555572d0a00, cb=0x555555d086d5 <resettable_phase_hold>, opaque=0x0, type=RESET_TYPE_COLD) at ../hw/core/resettable.c:92 #22 0x0000555555d08792 in resettable_phase_hold (obj=0x5555572d0a00, opaque=0x0, type=RESET_TYPE_COLD) at ../hw/core/resettable.c:169 #23 0x0000555555d00fc5 in bus_reset_child_foreach (obj=0x55555723d270, cb=0x555555d086d5 <resettable_phase_hold>, opaque=0x0, type=RESET_TYPE_COLD) at ../hw/core/bus.c:97
#24 0x0000555555d084d8 in resettable_child_foreach (rc=0x5555571bfde0, obj=0x55555723d270, cb=0x555555d086d5 <resettable_phase_hold>, opaque=0x0, type=RESET_TYPE_COLD) at ../hw/core/resettable.c:92 #25 0x0000555555d08792 in resettable_phase_hold (obj=0x55555723d270, opaque=0x0, type=RESET_TYPE_COLD) at ../hw/core/resettable.c:169 #26 0x0000555555d06d6d in resettable_container_child_foreach (obj=0x55555724a8a0, cb=0x555555d086d5 <resettable_phase_hold>, opaque=0x0, type=RESET_TYPE_COLD) at ../hw/core/resetcontainer.c:54 #27 0x0000555555d084d8 in resettable_child_foreach (rc=0x5555572180f0, obj=0x55555724a8a0, cb=0x555555d086d5 <resettable_phase_hold>, opaque=0x0, type=RESET_TYPE_COLD) at ../hw/core/resettable.c:92 #28 0x0000555555d08792 in resettable_phase_hold (obj=0x55555724a8a0, opaque=0x0, type=RESET_TYPE_COLD) at ../hw/core/resettable.c:169 #29 0x0000555555d0838d in resettable_assert_reset (obj=0x55555724a8a0, type=RESET_TYPE_COLD) at ../hw/core/resettable.c:58
#30 0x0000555555d082e5 in resettable_reset (obj=0x55555724a8a0, type=RESET_TYPE_COLD) at ../hw/core/resettable.c:45
#31 0x00005555558db5c8 in qemu_devices_reset (reason=SHUTDOWN_CAUSE_GUEST_RESET) at ../hw/core/reset.c:179 #32 0x0000555555b6f5b2 in pc_machine_reset (machine=0x555557234490, reason=SHUTDOWN_CAUSE_GUEST_RESET) at ../hw/i386/pc.c:1877
#33 0x0000555555a5a0a2 in qemu_system_reset (reason=SHUTDOWN_CAUSE_GUEST_RESET) at ../system/runstate.c:516 #34 0x0000555555a5a891 in main_loop_should_exit (status=0x7fffffffd574) at ../system/runstate.c:792
#35 0x0000555555a5a992 in qemu_main_loop () at ../system/runstate.c:825 #36 0x0000555555e9cced in qemu_default_main () at ../system/main.c:37 #37 0x0000555555e9cd2a in main (argc=58, argv=0x7fffffffd6d8) at ../system/main.c:48

Quoted 25 lines

or

2) introduce new listener ops that can be triggered when a region is enabled or disabled

Thanks

Quoted 17 lines

Thanks

Eric

Quoted 13 lines

Thanks

Quoted 11 lines

Eric

Quoted 8 lines

Thanks

Quoted 6 lines

}

void vhost_toggle_device_iotlb(VirtIODevice *vdev)
--
2.47.1

#9 lore raw
In reply to Eric Auger
Quoted 127 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Subject: Re: [PATCH] hw/virtio/vhost: Disable IOTLB callbacks when IOMMU gets disabled

Hi Jason,

Quoted 6 lines · Jason Wang
Quoted 5 lines · Eric Auger

Hi Jason,

Quoted 1 line · Jason Wang

On Wed, Jan 22, 2025 at 12:25 AM Eric Auger <[email protected]>

Quoted 4 lines · wrote:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Hi Jason,

Quoted 1 line · Jason Wang

On Tue, Jan 21, 2025 at 1:33 AM Eric Auger <[email protected]>

Quoted 7 lines · wrote:
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

When a guest exposed with a vhost device and protected by an intel IOMMU gets rebooted, we sometimes observe a spurious warning:

Fail to lookup the translated address ffffe000

We observe that the IOMMU gets disabled through a write to the global command register (CMAR_GCMD.TE) before the vhost device gets

stopped.

Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines

When this warning happens it can be observed an inflight IOTLB miss occurs after the IOMMU disable and before the vhost stop. In that case a flat translation occurs and the check in vhost_memory_region_lookup() fails.

Let's disable the IOTLB callbacks when all IOMMU MRs have been unregistered.

---
 hw/virtio/vhost.c | 4 ++++
 1 file changed, 4 insertions(+)
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

hw/virtio/vhost.c static void

Quoted 5 lines
hw/virtio/vhost.c +0 -0
index 6aa72fd434..128c2ab094 100644
@@ -931,6 +931,10 @@ static void

vhost_iommu_region_del(MemoryListener *listener,

Quoted 48 lines
Quoted 48 lines
Quoted 24 lines
Quoted 20 lines
Quoted 7 lines

hw/virtio/vhost.c static void · line 937

Quoted 7 lines
hw/virtio/vhost.c +4 -0
             break;
         }
     }
+    if (QLIST_EMPTY(&dev->iommu_list) &&
+        dev->vhost_ops->vhost_set_iotlb_callback) {
+        dev->vhost_ops->vhost_set_iotlb_callback(dev, false);
+    }
Quoted 7 lines

So the current code assumes:

1) IOMMU is enabled before vhost starts
2) IOMMU is disabled after vhost stops

This patch seems to fix 2) but not 1). Do we need to deal with the IOMMU enabled after vhost starts?

sorry I initially misunderstood the above comment. Indeed in the reboot case assumption 2) happens to be wrong. However what I currently do is:
stop listening to iotlb miss requests from the kernel because my understanding is those requests are just spurious ones, generate warnings and we do not care since we are rebooting the system.

Quoted 4 lines

However I do not claim this could handle the case where the IOMMU MR would be turned off and then turned on. I think in that case we should also flush the kernel IOTLB and this is not taken care of in this patch. Is it a relevant use case?

Quoted 6 lines

Not sure.

Quoted 2 lines

wrt removing assumption 1) and allow IOMMU enabled after vhost start. Is that a valid use case as the virtio driver is using the dma api?

It should not be but we can't assume the behaviour of the guest. It could be buggy or even malicious.

agreed

Quoted 17 lines

Btw, we had the following codes while handling te:

/* Handle Translation Enable/Disable */
static void vtd_handle_gcmd_te(IntelIOMMUState *s, bool en)

{
    if (s->dmar_enabled == en) {
        return;
    }

    trace_vtd_dmar_enable(en);

...

    vtd_reset_caches(s);
    vtd_address_space_refresh_all(s);
}
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

vtd_address_space_refresh_all() will basically disable the iommu memory region. It looks not sufficient to trigger the region_del callback, maybe we should delete the region or introduce listener callback?

This is exactly the code path which is entered in my use case.

vtd_address_space_refresh_all(s) induces the vhost_iommu_region_del. But

given the current implement of this latter the IOTLB callback is not unset and the kernel IOTLB is not refreshed. Also as I pointed out the hdev->mem->regions are not updated? shouldn't they. Can you explain what they correspond to?

Quoted 5 lines

Adding Peter for more ideas.

I think it's better to find a way to trigger the listener here, probably:

1) add/delete the memory regions instead of enable/disable

sorry I don't understand what you mean. The vhost_iommu_region_del call stack is provided below [1]. Write to the intel iommu GCMD TE bit induces a call to vhost_iommu_region_del. This happens before the vhost_dev_stop whose call stack is provided below [2] and originates from a bus reset.

We may have inflight IOTLB miss requests happening between both.

If this happens, vhost_device_iotlb_miss() fails because the IOVA is not translated anymore by the IOMMU and the iotlb.translated_addr returned by address_space_get_iotlb_entry() is not within the registered vhost_memory_regions looked up in vhost_memory_region_lookup(), hence the "Fail to lookup the translated address" message.

Quoted 7 lines

It sounds wrong that vhost keeps on using IOVAs that are not translated anymore. It looks we have a reset ordering issue and this patch is just removing the sympton and not the cause.

At the moment I don't really get what is initiating the intel iommu TE bit write. This is a guest action but is it initiated from a misordered qemu event?

During reboot, native_machine_shutdown() calls x86_platform.iommu_shutdown() to disable iommu before reset. So Peter's patch will not address your issue.

Before iommu shutdown, device_shutdown() is called to shutdown all devices. Not clear why vhost is still active.

Thanks
Zhenzhong

Quoted 5 lines

It could also relate to
[PATCH 0/4] intel_iommu: Reset vIOMMU after all the rest of devices
https://lore.kernel.org/all/?q=s%3Aintel_iommu%3A+Reset+vIOMMU+after+all+
the+rest+of+devices

#10 lore raw
Quoted 136 lines · Zhenzhong
Quoted 128 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Subject: Re: [PATCH] hw/virtio/vhost: Disable IOTLB callbacks when IOMMU gets disabled

Hi Jason,

Quoted 6 lines · Jason Wang
Quoted 5 lines · Eric Auger

Hi Jason,

Quoted 1 line · Jason Wang

On Wed, Jan 22, 2025 at 12:25 AM Eric Auger <[email protected]>

Quoted 4 lines · wrote:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Hi Jason,

Quoted 1 line · Jason Wang

On Tue, Jan 21, 2025 at 1:33 AM Eric Auger <[email protected]>

Quoted 7 lines · wrote:
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

When a guest exposed with a vhost device and protected by an intel IOMMU gets rebooted, we sometimes observe a spurious warning:

Fail to lookup the translated address ffffe000

We observe that the IOMMU gets disabled through a write to the global command register (CMAR_GCMD.TE) before the vhost device gets

stopped.

Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines

When this warning happens it can be observed an inflight IOTLB miss occurs after the IOMMU disable and before the vhost stop. In that case a flat translation occurs and the check in vhost_memory_region_lookup() fails.

Let's disable the IOTLB callbacks when all IOMMU MRs have been unregistered.

---
 hw/virtio/vhost.c | 4 ++++
 1 file changed, 4 insertions(+)
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

hw/virtio/vhost.c static void

Quoted 5 lines
hw/virtio/vhost.c +0 -0
index 6aa72fd434..128c2ab094 100644
@@ -931,6 +931,10 @@ static void

vhost_iommu_region_del(MemoryListener *listener,

Quoted 48 lines
Quoted 48 lines
Quoted 24 lines
Quoted 20 lines
Quoted 7 lines

hw/virtio/vhost.c static void · line 937

Quoted 7 lines
hw/virtio/vhost.c +4 -0
             break;
         }
     }
+    if (QLIST_EMPTY(&dev->iommu_list) &&
+        dev->vhost_ops->vhost_set_iotlb_callback) {
+        dev->vhost_ops->vhost_set_iotlb_callback(dev, false);
+    }
Quoted 7 lines

So the current code assumes:

1) IOMMU is enabled before vhost starts
2) IOMMU is disabled after vhost stops

This patch seems to fix 2) but not 1). Do we need to deal with the IOMMU enabled after vhost starts?

sorry I initially misunderstood the above comment. Indeed in the reboot case assumption 2) happens to be wrong. However what I currently do is:
stop listening to iotlb miss requests from the kernel because my understanding is those requests are just spurious ones, generate warnings and we do not care since we are rebooting the system.

Quoted 4 lines

However I do not claim this could handle the case where the IOMMU MR would be turned off and then turned on. I think in that case we should also flush the kernel IOTLB and this is not taken care of in this patch. Is it a relevant use case?

Quoted 6 lines

Not sure.

Quoted 2 lines

wrt removing assumption 1) and allow IOMMU enabled after vhost start. Is that a valid use case as the virtio driver is using the dma api?

It should not be but we can't assume the behaviour of the guest. It could be buggy or even malicious.

agreed

Quoted 17 lines

Btw, we had the following codes while handling te:

/* Handle Translation Enable/Disable */
static void vtd_handle_gcmd_te(IntelIOMMUState *s, bool en)

{
    if (s->dmar_enabled == en) {
        return;
    }

    trace_vtd_dmar_enable(en);

...

    vtd_reset_caches(s);
    vtd_address_space_refresh_all(s);
}
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

vtd_address_space_refresh_all() will basically disable the iommu memory region. It looks not sufficient to trigger the region_del callback, maybe we should delete the region or introduce listener callback?

This is exactly the code path which is entered in my use case.

vtd_address_space_refresh_all(s) induces the vhost_iommu_region_del. But

given the current implement of this latter the IOTLB callback is not unset and the kernel IOTLB is not refreshed. Also as I pointed out the hdev->mem->regions are not updated? shouldn't they. Can you explain what they correspond to?

Quoted 5 lines

Adding Peter for more ideas.

I think it's better to find a way to trigger the listener here, probably:

1) add/delete the memory regions instead of enable/disable

sorry I don't understand what you mean. The vhost_iommu_region_del call stack is provided below [1]. Write to the intel iommu GCMD TE bit induces a call to vhost_iommu_region_del. This happens before the vhost_dev_stop whose call stack is provided below [2] and originates from a bus reset.

We may have inflight IOTLB miss requests happening between both.

If this happens, vhost_device_iotlb_miss() fails because the IOVA is not translated anymore by the IOMMU and the iotlb.translated_addr returned by address_space_get_iotlb_entry() is not within the registered vhost_memory_regions looked up in vhost_memory_region_lookup(), hence the "Fail to lookup the translated address" message.

Quoted 7 lines

It sounds wrong that vhost keeps on using IOVAs that are not translated anymore. It looks we have a reset ordering issue and this patch is just removing the sympton and not the cause.

At the moment I don't really get what is initiating the intel iommu TE bit write. This is a guest action but is it initiated from a misordered qemu event?

Quoted 5 lines

During reboot, native_machine_shutdown() calls x86_platform.iommu_shutdown() to disable iommu before reset. So Peter's patch will not address your issue.

Before iommu shutdown, device_shutdown() is called to shutdown all devices. Not clear why vhost is still active.

It might be because neither virtio bus nor virtio-net provides a shutdown method.

There used to be requests to provide those to unbreak the kexec.

A quick try might be to provide a .driver.shutdown to virtio_net_driver structure and reset the device there as a start.

Thanks

Quoted 10 lines

Thanks
Zhenzhong

Quoted 5 lines

It could also relate to
[PATCH 0/4] intel_iommu: Reset vIOMMU after all the rest of devices
https://lore.kernel.org/all/?q=s%3Aintel_iommu%3A+Reset+vIOMMU+after+all+
the+rest+of+devices

#11 lore raw
Quoted 142 lines · Jason Wang
Quoted 139 lines · Zhenzhong
Quoted 128 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Subject: Re: [PATCH] hw/virtio/vhost: Disable IOTLB callbacks when IOMMU gets disabled

Hi Jason,

Quoted 6 lines · Jason Wang
Quoted 5 lines · Eric Auger

Hi Jason,

Quoted 1 line · Jason Wang

On Wed, Jan 22, 2025 at 12:25 AM Eric Auger <[email protected]>

Quoted 4 lines · wrote:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Hi Jason,

Quoted 1 line · Jason Wang

On Tue, Jan 21, 2025 at 1:33 AM Eric Auger <[email protected]>

Quoted 7 lines · wrote:
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

When a guest exposed with a vhost device and protected by an intel IOMMU gets rebooted, we sometimes observe a spurious warning:

Fail to lookup the translated address ffffe000

We observe that the IOMMU gets disabled through a write to the global command register (CMAR_GCMD.TE) before the vhost device gets

stopped.

Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines

When this warning happens it can be observed an inflight IOTLB miss occurs after the IOMMU disable and before the vhost stop. In that case a flat translation occurs and the check in vhost_memory_region_lookup() fails.

Let's disable the IOTLB callbacks when all IOMMU MRs have been unregistered.

---
 hw/virtio/vhost.c | 4 ++++
 1 file changed, 4 insertions(+)
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

hw/virtio/vhost.c static void

Quoted 5 lines
hw/virtio/vhost.c +0 -0
index 6aa72fd434..128c2ab094 100644
@@ -931,6 +931,10 @@ static void

vhost_iommu_region_del(MemoryListener *listener,

Quoted 48 lines
Quoted 48 lines
Quoted 24 lines
Quoted 20 lines
Quoted 7 lines

hw/virtio/vhost.c static void · line 937

Quoted 7 lines
hw/virtio/vhost.c +4 -0
             break;
         }
     }
+    if (QLIST_EMPTY(&dev->iommu_list) &&
+        dev->vhost_ops->vhost_set_iotlb_callback) {
+        dev->vhost_ops->vhost_set_iotlb_callback(dev, false);
+    }
Quoted 7 lines

So the current code assumes:

1) IOMMU is enabled before vhost starts
2) IOMMU is disabled after vhost stops

This patch seems to fix 2) but not 1). Do we need to deal with the IOMMU enabled after vhost starts?

sorry I initially misunderstood the above comment. Indeed in the reboot case assumption 2) happens to be wrong. However what I currently do is:
stop listening to iotlb miss requests from the kernel because my understanding is those requests are just spurious ones, generate warnings and we do not care since we are rebooting the system.

Quoted 4 lines

However I do not claim this could handle the case where the IOMMU MR would be turned off and then turned on. I think in that case we should also flush the kernel IOTLB and this is not taken care of in this patch. Is it a relevant use case?

Quoted 6 lines

Not sure.

Quoted 2 lines

wrt removing assumption 1) and allow IOMMU enabled after vhost start. Is that a valid use case as the virtio driver is using the dma api?

It should not be but we can't assume the behaviour of the guest. It could be buggy or even malicious.

agreed

Quoted 17 lines

Btw, we had the following codes while handling te:

/* Handle Translation Enable/Disable */
static void vtd_handle_gcmd_te(IntelIOMMUState *s, bool en)

{
    if (s->dmar_enabled == en) {
        return;
    }

    trace_vtd_dmar_enable(en);

...

    vtd_reset_caches(s);
    vtd_address_space_refresh_all(s);
}
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

vtd_address_space_refresh_all() will basically disable the iommu memory region. It looks not sufficient to trigger the region_del callback, maybe we should delete the region or introduce listener callback?

This is exactly the code path which is entered in my use case.

vtd_address_space_refresh_all(s) induces the vhost_iommu_region_del. But

given the current implement of this latter the IOTLB callback is not unset and the kernel IOTLB is not refreshed. Also as I pointed out the hdev->mem->regions are not updated? shouldn't they. Can you explain what they correspond to?

Quoted 5 lines

Adding Peter for more ideas.

I think it's better to find a way to trigger the listener here, probably:

1) add/delete the memory regions instead of enable/disable

sorry I don't understand what you mean. The vhost_iommu_region_del call stack is provided below [1]. Write to the intel iommu GCMD TE bit induces a call to vhost_iommu_region_del. This happens before the vhost_dev_stop whose call stack is provided below [2] and originates from a bus reset.

We may have inflight IOTLB miss requests happening between both.

If this happens, vhost_device_iotlb_miss() fails because the IOVA is not translated anymore by the IOMMU and the iotlb.translated_addr returned by address_space_get_iotlb_entry() is not within the registered vhost_memory_regions looked up in vhost_memory_region_lookup(), hence the "Fail to lookup the translated address" message.

Quoted 7 lines

It sounds wrong that vhost keeps on using IOVAs that are not translated anymore. It looks we have a reset ordering issue and this patch is just removing the sympton and not the cause.

At the moment I don't really get what is initiating the intel iommu TE bit write. This is a guest action but is it initiated from a misordered qemu event?

During reboot, native_machine_shutdown() calls x86_platform.iommu_shutdown() to disable iommu before reset. So Peter's patch will not address your issue.

Quoted 7 lines
Quoted 2 lines

Before iommu shutdown, device_shutdown() is called to shutdown all devices. Not clear why vhost is still active.

It might be because neither virtio bus nor virtio-net provides a shutdown method.

There used to be requests to provide those to unbreak the kexec.

More could be seen at https://issues.redhat.com/browse/RHEL-331

This looks exactly the same issue.

Thanks

Quoted 16 lines

A quick try might be to provide a .driver.shutdown to virtio_net_driver structure and reset the device there as a start.

Thanks

Quoted 10 lines

Thanks
Zhenzhong

Quoted 5 lines

It could also relate to
[PATCH 0/4] intel_iommu: Reset vIOMMU after all the rest of devices
https://lore.kernel.org/all/?q=s%3Aintel_iommu%3A+Reset+vIOMMU+after+all+
the+rest+of+devices

#12 lore raw
Quoted 154 lines

-----Original Message-----
From: Jason Wang <[email protected]>
Subject: Re: [PATCH] hw/virtio/vhost: Disable IOTLB callbacks when IOMMU gets disabled

Quoted 9 lines · Jason Wang
Quoted 6 lines · Zhenzhong
Quoted 6 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Subject: Re: [PATCH] hw/virtio/vhost: Disable IOTLB callbacks when IOMMU

gets

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

disabled

Hi Jason,

Quoted 1 line · Jason Wang

On Wed, Jan 22, 2025 at 3:55 PM Eric Auger <[email protected]>

Quoted 5 lines · wrote:
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Hi Jason,

Quoted 1 line · Jason Wang

On Wed, Jan 22, 2025 at 12:25 AM Eric Auger <[email protected]>

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines · wrote:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Hi Jason,

Quoted 1 line · Jason Wang

On Tue, Jan 21, 2025 at 1:33 AM Eric Auger <[email protected]>

Quoted 2 lines · wrote:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

When a guest exposed with a vhost device and protected by an intel IOMMU gets rebooted, we sometimes observe a spurious

warning:

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Fail to lookup the translated address ffffe000

We observe that the IOMMU gets disabled through a write to the

global

Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

command register (CMAR_GCMD.TE) before the vhost device gets

stopped.

Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines

When this warning happens it can be observed an inflight IOTLB miss occurs after the IOMMU disable and before the vhost stop. In that case a flat translation occurs and the check in vhost_memory_region_lookup() fails.

Let's disable the IOTLB callbacks when all IOMMU MRs have been unregistered.

---
 hw/virtio/vhost.c | 4 ++++
 1 file changed, 4 insertions(+)
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

hw/virtio/vhost.c static void

Quoted 5 lines
hw/virtio/vhost.c +0 -0
index 6aa72fd434..128c2ab094 100644
@@ -931,6 +931,10 @@ static void

vhost_iommu_region_del(MemoryListener *listener,

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
hw/virtio/vhost.c +4 -0
             break;
         }
     }
+    if (QLIST_EMPTY(&dev->iommu_list) &&
+        dev->vhost_ops->vhost_set_iotlb_callback) {
+        dev->vhost_ops->vhost_set_iotlb_callback(dev, false);
+    }
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

So the current code assumes:

1) IOMMU is enabled before vhost starts
2) IOMMU is disabled after vhost stops

This patch seems to fix 2) but not 1). Do we need to deal with the IOMMU enabled after vhost starts?

sorry I initially misunderstood the above comment. Indeed in the

reboot

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

case assumption 2) happens to be wrong. However what I currently do

is:

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

stop listening to iotlb miss requests from the kernel because my understanding is those requests are just spurious ones, generate warnings and we do not care since we are rebooting the system.

However I do not claim this could handle the case where the IOMMU

MR

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

would be turned off and then turned on. I think in that case we should also flush the kernel IOTLB and this is not taken care of in this patch. Is it a relevant use case?

Not sure.

Quoted 1 line

wrt removing assumption 1) and allow IOMMU enabled after vhost

start. Is

Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 3 lines
Quoted 1 line

that a valid use case as the virtio driver is using the dma api?

It should not be but we can't assume the behaviour of the guest. It could be buggy or even malicious.

agreed

Quoted 17 lines

Btw, we had the following codes while handling te:

/* Handle Translation Enable/Disable */
static void vtd_handle_gcmd_te(IntelIOMMUState *s, bool en)

{
    if (s->dmar_enabled == en) {
        return;
    }

    trace_vtd_dmar_enable(en);

...

    vtd_reset_caches(s);
    vtd_address_space_refresh_all(s);
}
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

vtd_address_space_refresh_all() will basically disable the iommu memory region. It looks not sufficient to trigger the region_del callback, maybe we should delete the region or introduce listener callback?

This is exactly the code path which is entered in my use case.

vtd_address_space_refresh_all(s) induces the vhost_iommu_region_del.

But

Quoted 1 line
Quoted 1 line
Quoted 1 line

given the current implement of this latter the IOTLB callback is not unset and

the

Quoted 1 line
Quoted 1 line
Quoted 1 line

kernel IOTLB is not refreshed. Also as I pointed out the hdev->mem->regions

are

Quoted 24 lines
Quoted 24 lines
Quoted 24 lines

not updated? shouldn't they. Can you explain what they correspond to?

Quoted 5 lines

Adding Peter for more ideas.

I think it's better to find a way to trigger the listener here, probably:

1) add/delete the memory regions instead of enable/disable

sorry I don't understand what you mean. The vhost_iommu_region_del call stack is provided below [1]. Write to the intel iommu GCMD TE bit induces a call to vhost_iommu_region_del. This happens before the vhost_dev_stop whose call stack is provided below [2] and originates from a bus reset.

We may have inflight IOTLB miss requests happening between both.

If this happens, vhost_device_iotlb_miss() fails because the IOVA is not translated anymore by the IOMMU and the iotlb.translated_addr returned by address_space_get_iotlb_entry() is not within the registered vhost_memory_regions looked up in vhost_memory_region_lookup(), hence the "Fail to lookup the translated address" message.

It sounds wrong that vhost keeps on using IOVAs that are not translated anymore. It looks we have a reset ordering issue and this patch is just removing the sympton and not the cause.

Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

At the moment I don't really get what is initiating the intel iommu TE bit write. This is a guest action but is it initiated from a misordered qemu event?

During reboot, native_machine_shutdown() calls

x86_platform.iommu_shutdown()

Quoted 7 lines
Quoted 4 lines

to disable iommu before reset. So Peter's patch will not address your issue.

Before iommu shutdown, device_shutdown() is called to shutdown all devices. Not clear why vhost is still active.

It might be because neither virtio bus nor virtio-net provides a shutdown method.

Oh, I see.

Quoted 6 lines
Quoted 2 lines

There used to be requests to provide those to unbreak the kexec.

More could be seen at https://issues.redhat.com/browse/RHEL-331

This looks exactly the same issue.

Have not access😊

Quoted 6 lines

Thanks

Quoted 3 lines

A quick try might be to provide a .driver.shutdown to virtio_net_driver structure and reset the device there as a start.

Make sense.

Thanks
Zhenzhong

#13 lore raw
Quoted 172 lines · Zhenzhong
Quoted 155 lines

-----Original Message-----
From: Jason Wang <[email protected]>
Subject: Re: [PATCH] hw/virtio/vhost: Disable IOTLB callbacks when IOMMU gets disabled

Quoted 9 lines · Jason Wang
Quoted 6 lines · Zhenzhong
Quoted 6 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Subject: Re: [PATCH] hw/virtio/vhost: Disable IOTLB callbacks when IOMMU

gets

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

disabled

Hi Jason,

Quoted 1 line · Jason Wang

On Wed, Jan 22, 2025 at 3:55 PM Eric Auger <[email protected]>

Quoted 5 lines · wrote:
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Hi Jason,

Quoted 1 line · Jason Wang

On Wed, Jan 22, 2025 at 12:25 AM Eric Auger <[email protected]>

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines · wrote:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Hi Jason,

Quoted 1 line · Jason Wang

On Tue, Jan 21, 2025 at 1:33 AM Eric Auger <[email protected]>

Quoted 2 lines · wrote:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

When a guest exposed with a vhost device and protected by an intel IOMMU gets rebooted, we sometimes observe a spurious

warning:

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Fail to lookup the translated address ffffe000

We observe that the IOMMU gets disabled through a write to the

global

Quoted 28 lines
Quoted 28 lines
Quoted 28 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

command register (CMAR_GCMD.TE) before the vhost device gets

stopped.

Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines

When this warning happens it can be observed an inflight IOTLB miss occurs after the IOMMU disable and before the vhost stop. In that case a flat translation occurs and the check in vhost_memory_region_lookup() fails.

Let's disable the IOTLB callbacks when all IOMMU MRs have been unregistered.

---
 hw/virtio/vhost.c | 4 ++++
 1 file changed, 4 insertions(+)
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

hw/virtio/vhost.c static void

Quoted 5 lines
hw/virtio/vhost.c +0 -0
index 6aa72fd434..128c2ab094 100644
@@ -931,6 +931,10 @@ static void

vhost_iommu_region_del(MemoryListener *listener,

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
hw/virtio/vhost.c +4 -0
             break;
         }
     }
+    if (QLIST_EMPTY(&dev->iommu_list) &&
+        dev->vhost_ops->vhost_set_iotlb_callback) {
+        dev->vhost_ops->vhost_set_iotlb_callback(dev, false);
+    }
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

So the current code assumes:

1) IOMMU is enabled before vhost starts
2) IOMMU is disabled after vhost stops

This patch seems to fix 2) but not 1). Do we need to deal with the IOMMU enabled after vhost starts?

sorry I initially misunderstood the above comment. Indeed in the

reboot

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

case assumption 2) happens to be wrong. However what I currently do

is:

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

stop listening to iotlb miss requests from the kernel because my understanding is those requests are just spurious ones, generate warnings and we do not care since we are rebooting the system.

However I do not claim this could handle the case where the IOMMU

MR

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

would be turned off and then turned on. I think in that case we should also flush the kernel IOTLB and this is not taken care of in this patch. Is it a relevant use case?

Not sure.

Quoted 1 line

wrt removing assumption 1) and allow IOMMU enabled after vhost

start. Is

Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 3 lines
Quoted 1 line

that a valid use case as the virtio driver is using the dma api?

It should not be but we can't assume the behaviour of the guest. It could be buggy or even malicious.

agreed

Quoted 17 lines

Btw, we had the following codes while handling te:

/* Handle Translation Enable/Disable */
static void vtd_handle_gcmd_te(IntelIOMMUState *s, bool en)

{
    if (s->dmar_enabled == en) {
        return;
    }

    trace_vtd_dmar_enable(en);

...

    vtd_reset_caches(s);
    vtd_address_space_refresh_all(s);
}
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

vtd_address_space_refresh_all() will basically disable the iommu memory region. It looks not sufficient to trigger the region_del callback, maybe we should delete the region or introduce listener callback?

This is exactly the code path which is entered in my use case.

vtd_address_space_refresh_all(s) induces the vhost_iommu_region_del.

But

Quoted 1 line
Quoted 1 line
Quoted 1 line

given the current implement of this latter the IOTLB callback is not unset and

the

Quoted 1 line
Quoted 1 line
Quoted 1 line

kernel IOTLB is not refreshed. Also as I pointed out the hdev->mem->regions

are

Quoted 24 lines
Quoted 24 lines
Quoted 24 lines

not updated? shouldn't they. Can you explain what they correspond to?

Quoted 5 lines

Adding Peter for more ideas.

I think it's better to find a way to trigger the listener here, probably:

1) add/delete the memory regions instead of enable/disable

sorry I don't understand what you mean. The vhost_iommu_region_del call stack is provided below [1]. Write to the intel iommu GCMD TE bit induces a call to vhost_iommu_region_del. This happens before the vhost_dev_stop whose call stack is provided below [2] and originates from a bus reset.

We may have inflight IOTLB miss requests happening between both.

If this happens, vhost_device_iotlb_miss() fails because the IOVA is not translated anymore by the IOMMU and the iotlb.translated_addr returned by address_space_get_iotlb_entry() is not within the registered vhost_memory_regions looked up in vhost_memory_region_lookup(), hence the "Fail to lookup the translated address" message.

It sounds wrong that vhost keeps on using IOVAs that are not translated anymore. It looks we have a reset ordering issue and this patch is just removing the sympton and not the cause.

Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

At the moment I don't really get what is initiating the intel iommu TE bit write. This is a guest action but is it initiated from a misordered qemu event?

During reboot, native_machine_shutdown() calls

x86_platform.iommu_shutdown()

Quoted 7 lines
Quoted 4 lines

to disable iommu before reset. So Peter's patch will not address your issue.

Before iommu shutdown, device_shutdown() is called to shutdown all devices. Not clear why vhost is still active.

It might be because neither virtio bus nor virtio-net provides a shutdown method.

Oh, I see.

Quoted 1 line
Quoted 1 line
Quoted 7 lines
Quoted 5 lines
Quoted 1 line

There used to be requests to provide those to unbreak the kexec.

More could be seen at https://issues.redhat.com/browse/RHEL-331

This looks exactly the same issue.

Have not access😊

It should work now.

Thanks

Quoted 12 lines
Quoted 7 lines

Thanks

Quoted 3 lines

A quick try might be to provide a .driver.shutdown to virtio_net_driver structure and reset the device there as a start.

Make sense.

Thanks
Zhenzhong

#14 lore raw
Quoted 198 lines

-----Original Message-----
From: Jason Wang <[email protected]>
Subject: Re: [PATCH] hw/virtio/vhost: Disable IOTLB callbacks when IOMMU gets disabled

Quoted 6 lines · Zhenzhong
Quoted 6 lines

-----Original Message-----
From: Jason Wang <[email protected]>
Subject: Re: [PATCH] hw/virtio/vhost: Disable IOTLB callbacks when IOMMU

gets

Quoted 4 lines
Quoted 4 lines

disabled

Quoted 1 line · Jason Wang
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines · Zhenzhong
Quoted 6 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Subject: Re: [PATCH] hw/virtio/vhost: Disable IOTLB callbacks when

IOMMU

Quoted 6 lines
Quoted 6 lines

gets

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

disabled

Hi Jason,

Quoted 8 lines
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines · Jason Wang
Quoted 1 line

On Wed, Jan 22, 2025 at 3:55 PM Eric Auger <[email protected]>

Quoted 5 lines · wrote:
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Hi Jason,

Quoted 1 line · Jason Wang

On Wed, Jan 22, 2025 at 12:25 AM Eric Auger

<[email protected]>

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines · wrote:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Hi Jason,

Quoted 1 line · Jason Wang

On Tue, Jan 21, 2025 at 1:33 AM Eric Auger

<[email protected]>

Quoted 10 lines
Quoted 10 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines · wrote:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

When a guest exposed with a vhost device and protected by an intel IOMMU gets rebooted, we sometimes observe a spurious

warning:

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Fail to lookup the translated address ffffe000

We observe that the IOMMU gets disabled through a write to the

global

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

command register (CMAR_GCMD.TE) before the vhost device

gets

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

stopped.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

When this warning happens it can be observed an inflight IOTLB miss occurs after the IOMMU disable and before the vhost stop.

In

Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

that case a flat translation occurs and the check in vhost_memory_region_lookup() fails.

Let's disable the IOTLB callbacks when all IOMMU MRs have been unregistered.

---
 hw/virtio/vhost.c | 4 ++++
 1 file changed, 4 insertions(+)
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

hw/virtio/vhost.c static void

Quoted 5 lines
hw/virtio/vhost.c +0 -0
index 6aa72fd434..128c2ab094 100644
@@ -931,6 +931,10 @@ static void

vhost_iommu_region_del(MemoryListener *listener,

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

hw/virtio/vhost.c static void · line 937

Quoted 7 lines
hw/virtio/vhost.c +4 -0
             break;
         }
     }
+    if (QLIST_EMPTY(&dev->iommu_list) &&
+        dev->vhost_ops->vhost_set_iotlb_callback) {
+        dev->vhost_ops->vhost_set_iotlb_callback(dev, false);
+    }

So the current code assumes:

Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

1) IOMMU is enabled before vhost starts
2) IOMMU is disabled after vhost stops

This patch seems to fix 2) but not 1). Do we need to deal with the IOMMU enabled after vhost starts?

sorry I initially misunderstood the above comment. Indeed in the

reboot

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

case assumption 2) happens to be wrong. However what I currently

do

Quoted 6 lines
Quoted 6 lines

is:

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

stop listening to iotlb miss requests from the kernel because my understanding is those requests are just spurious ones, generate warnings and we do not care since we are rebooting the system.

However I do not claim this could handle the case where the

IOMMU

Quoted 2 lines
Quoted 2 lines

MR

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

would be turned off and then turned on. I think in that case we

should

Quoted 27 lines
Quoted 27 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

also flush the kernel IOTLB and this is not taken care of in this patch. Is it a relevant use case?

Not sure.

Quoted 1 line

wrt removing assumption 1) and allow IOMMU enabled after vhost

start. Is

Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 3 lines
Quoted 1 line

that a valid use case as the virtio driver is using the dma api?

It should not be but we can't assume the behaviour of the guest. It could be buggy or even malicious.

agreed

Quoted 17 lines

Btw, we had the following codes while handling te:

/* Handle Translation Enable/Disable */
static void vtd_handle_gcmd_te(IntelIOMMUState *s, bool en)

{
    if (s->dmar_enabled == en) {
        return;
    }

    trace_vtd_dmar_enable(en);

...

    vtd_reset_caches(s);
    vtd_address_space_refresh_all(s);
}
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

vtd_address_space_refresh_all() will basically disable the iommu memory region. It looks not sufficient to trigger the region_del callback, maybe we should delete the region or introduce listener callback?

This is exactly the code path which is entered in my use case.

vtd_address_space_refresh_all(s) induces the

vhost_iommu_region_del.

Quoted 2 lines
Quoted 2 lines

But

Quoted 1 line
Quoted 1 line
Quoted 1 line

given the current implement of this latter the IOTLB callback is not unset

and

Quoted 11 lines
Quoted 2 lines

the

Quoted 1 line
Quoted 1 line
Quoted 1 line

kernel IOTLB is not refreshed. Also as I pointed out the hdev->mem-

regions

Quoted 8 lines

are

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

not updated? shouldn't they. Can you explain what they correspond to?

Quoted 5 lines

Adding Peter for more ideas.

I think it's better to find a way to trigger the listener here, probably:

1) add/delete the memory regions instead of enable/disable

sorry I don't understand what you mean. The vhost_iommu_region_del

call

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

stack is provided below [1]. Write to the intel iommu GCMD TE bit induces a call to vhost_iommu_region_del. This happens before the vhost_dev_stop whose call stack is provided below [2] and originates from a bus reset.

We may have inflight IOTLB miss requests happening between both.

If this happens, vhost_device_iotlb_miss() fails because the IOVA is not translated anymore by the IOMMU and the iotlb.translated_addr returned by address_space_get_iotlb_entry() is not within the registered vhost_memory_regions looked up in vhost_memory_region_lookup(),

hence

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines

the "Fail to lookup the translated address" message.

It sounds wrong that vhost keeps on using IOVAs that are not translated anymore. It looks we have a reset ordering issue and this patch is just removing the sympton and not the cause.

At the moment I don't really get what is initiating the intel iommu TE bit write. This is a guest action but is it initiated from a misordered qemu event?

Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line

During reboot, native_machine_shutdown() calls

x86_platform.iommu_shutdown()

Quoted 3 lines
Quoted 3 lines

to disable iommu before reset. So Peter's patch will not address your issue.

Before iommu shutdown, device_shutdown() is called to shutdown all

devices.

Quoted 10 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line

Not clear why vhost is still active.

It might be because neither virtio bus nor virtio-net provides a shutdown method.

Oh, I see.

Quoted 3 lines
Quoted 2 lines

There used to be requests to provide those to unbreak the kexec.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

More could be seen at https://issues.redhat.com/browse/RHEL-331

This looks exactly the same issue.

Have not access😊

It should work now.

Yes, exactly same issue.

Thanks
Zhenzhong

author
#15 lore raw
In reply to Jason Wang

Hi,

Quoted 139 lines · Jason Wang
Quoted 138 lines · Jason Wang
Quoted 136 lines · Zhenzhong
Quoted 126 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Subject: Re: [PATCH] hw/virtio/vhost: Disable IOTLB callbacks when IOMMU gets disabled

Hi Jason,

Quoted 6 lines · Jason Wang
Quoted 5 lines · Eric Auger

Hi Jason,

Quoted 1 line · Jason Wang

On Wed, Jan 22, 2025 at 12:25 AM Eric Auger <[email protected]>

Quoted 4 lines · wrote:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Hi Jason,

Quoted 1 line · Jason Wang

On Tue, Jan 21, 2025 at 1:33 AM Eric Auger <[email protected]>

Quoted 7 lines · wrote:
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

When a guest exposed with a vhost device and protected by an intel IOMMU gets rebooted, we sometimes observe a spurious warning:

Fail to lookup the translated address ffffe000

We observe that the IOMMU gets disabled through a write to the global command register (CMAR_GCMD.TE) before the vhost device gets

stopped.

Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines

When this warning happens it can be observed an inflight IOTLB miss occurs after the IOMMU disable and before the vhost stop. In that case a flat translation occurs and the check in vhost_memory_region_lookup() fails.

Let's disable the IOTLB callbacks when all IOMMU MRs have been unregistered.

---
 hw/virtio/vhost.c | 4 ++++
 1 file changed, 4 insertions(+)
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

hw/virtio/vhost.c static void

Quoted 5 lines
hw/virtio/vhost.c +0 -0
index 6aa72fd434..128c2ab094 100644
@@ -931,6 +931,10 @@ static void

vhost_iommu_region_del(MemoryListener *listener,

Quoted 48 lines
Quoted 48 lines
Quoted 24 lines
Quoted 20 lines
Quoted 7 lines

hw/virtio/vhost.c static void · line 937

Quoted 7 lines
hw/virtio/vhost.c +4 -0
             break;
         }
     }
+    if (QLIST_EMPTY(&dev->iommu_list) &&
+        dev->vhost_ops->vhost_set_iotlb_callback) {
+        dev->vhost_ops->vhost_set_iotlb_callback(dev, false);
+    }
Quoted 7 lines

So the current code assumes:

1) IOMMU is enabled before vhost starts
2) IOMMU is disabled after vhost stops

This patch seems to fix 2) but not 1). Do we need to deal with the IOMMU enabled after vhost starts?

sorry I initially misunderstood the above comment. Indeed in the reboot case assumption 2) happens to be wrong. However what I currently do is:
stop listening to iotlb miss requests from the kernel because my understanding is those requests are just spurious ones, generate warnings and we do not care since we are rebooting the system.

Quoted 4 lines

However I do not claim this could handle the case where the IOMMU MR would be turned off and then turned on. I think in that case we should also flush the kernel IOTLB and this is not taken care of in this patch. Is it a relevant use case?

Quoted 6 lines

Not sure.

Quoted 2 lines

wrt removing assumption 1) and allow IOMMU enabled after vhost start. Is that a valid use case as the virtio driver is using the dma api?

It should not be but we can't assume the behaviour of the guest. It could be buggy or even malicious.

agreed

Quoted 17 lines

Btw, we had the following codes while handling te:

/* Handle Translation Enable/Disable */
static void vtd_handle_gcmd_te(IntelIOMMUState *s, bool en)

{
    if (s->dmar_enabled == en) {
        return;
    }

    trace_vtd_dmar_enable(en);

...

    vtd_reset_caches(s);
    vtd_address_space_refresh_all(s);
}
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

vtd_address_space_refresh_all() will basically disable the iommu memory region. It looks not sufficient to trigger the region_del callback, maybe we should delete the region or introduce listener callback?

This is exactly the code path which is entered in my use case.

vtd_address_space_refresh_all(s) induces the vhost_iommu_region_del. But

given the current implement of this latter the IOTLB callback is not unset and the kernel IOTLB is not refreshed. Also as I pointed out the hdev->mem->regions are not updated? shouldn't they. Can you explain what they correspond to?

Quoted 5 lines

Adding Peter for more ideas.

I think it's better to find a way to trigger the listener here, probably:

1) add/delete the memory regions instead of enable/disable

sorry I don't understand what you mean. The vhost_iommu_region_del call stack is provided below [1]. Write to the intel iommu GCMD TE bit induces a call to vhost_iommu_region_del. This happens before the vhost_dev_stop whose call stack is provided below [2] and originates from a bus reset.
We may have inflight IOTLB miss requests happening between both.

If this happens, vhost_device_iotlb_miss() fails because the IOVA is not translated anymore by the IOMMU and the iotlb.translated_addr returned by address_space_get_iotlb_entry() is not within the registered vhost_memory_regions looked up in vhost_memory_region_lookup(), hence the "Fail to lookup the translated address" message.

Quoted 7 lines

It sounds wrong that vhost keeps on using IOVAs that are not translated anymore. It looks we have a reset ordering issue and this patch is just removing the sympton and not the cause.

At the moment I don't really get what is initiating the intel iommu TE bit write. This is a guest action but is it initiated from a misordered qemu event?

During reboot, native_machine_shutdown() calls x86_platform.iommu_shutdown() to disable iommu before reset. So Peter's patch will not address your issue.

Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

Before iommu shutdown, device_shutdown() is called to shutdown all devices. Not clear why vhost is still active.

It might be because neither virtio bus nor virtio-net provides a shutdown method.

There used to be requests to provide those to unbreak the kexec.

More could be seen at https://issues.redhat.com/browse/RHEL-331

Cool, we have a culprit now :-) I see the ticket is closed, I will reopen it. Are there known implementation challenges that caused the fix postponing or do we "just" need someone with free cycles to carry out the task. By the way FYI, we have other tickets opened related to vSMMU and virtio devices also happening during reboot.

Thanks

Eric

Quoted 16 lines

This looks exactly the same issue.

Thanks

Quoted 12 lines

A quick try might be to provide a .driver.shutdown to virtio_net_driver structure and reset the device there as a start.

Thanks

Quoted 7 lines

Thanks
Zhenzhong

Quoted 4 lines

It could also relate to
[PATCH 0/4] intel_iommu: Reset vIOMMU after all the rest of devices
https://lore.kernel.org/all/?q=s%3Aintel_iommu%3A+Reset+vIOMMU+after+all+
the+rest+of+devices

#16 lore raw
In reply to Jason Wang

Hello, Jason, Eric,

Quoted 7 lines · Jason Wang

It might be because neither virtio bus nor virtio-net provides a shutdown method.

There used to be requests to provide those to unbreak the kexec.

A quick try might be to provide a .driver.shutdown to virtio_net_driver structure and reset the device there as a start.

I didn't check virtio driver path, but if that's missing it's reasonable to support it indeed.

OTOH, even with that, vhost can still hit such DMA issue if it's a hard-reset, am I right? IOW, when using QMP command "system-reset". If my memory is correct, that's the problem I was working on the VFIO series, rather than a clean reboot. And that won't give guest driver chance to run anything, IIUC.

I am wildly suspecting a VT-d write to GCMD to disable it can also appear if there's a hard reset, then when bootloading the VM the bios (or whatever firmware at early stage) may want to make sure the VT-d device is completely off by writting to GCMD. But that's a pure guess.. and that may or may not matter much on how we fix this problem.

IOW, I suspect we need to fix both of them,

(a) for soft-reset, by making sure drivers properly quiesce DMAs proactively when VM gracefully shuts down.

(b) for hard-reset, by making sure QEMU reset in proper order.

One thing to mention is for problem (b) VFIO used to have an extra challenge on !FLR devices, I discussed it in patch 4's comment there. Quotting from patch 4 of series:

https://lore.kernel.org/all/[email protected]

* (1) Device depth-first reset hierachy doesn't yet work for vIOMMUs
* (reference: resettable_cold_reset_fn())
*
* Currently, vIOMMU devices are created as normal '-device'
* cmdlines. It means in many ways it has the same attributes with
* most of the rest devices, even if the rest devices should
* logically be under control of the vIOMMU unit.
*
* One side effect of it is vIOMMU devices will be currently put
* randomly under qdev tree hierarchy, which is the source of
* device reset ordering in current QEMU (depth-first traversal).
* It means vIOMMU now can be reset before some devices. For fully
* emulated devices that's not a problem, because the traversal
* holds BQL for the whole process. However it is a problem if DMA
* can happen without BQL, like VFIO, vDPA or remote device process.
*
* TODO: one ideal solution can be that we make vIOMMU the parent
* of the whole pci host bridge. Hence vIOMMU can be reset after
* all the devices are reset and quiesced.
*
* (2) Some devices register its own reset functions
*
* Even if above issue solved, if devices register its own reset
* functions for some reason via QEMU reset hooks, vIOMMU can still
* be reset before the device. One example is vfio_reset_handler()
* where FLR is not supported on the device.
*
* TODO: merge relevant reset functions into the device tree reset
* framework.

So maybe vhost doesn't have problem (2) listed above, and maybe it means it's still worthwhile thinking more about problem (1), which is to change the QOM tree to provide a correct topology representation when vIOMMU is
present: so far it should be still a pretty much orphaned object there.. if QEMU relies on QOM tree topology for reset order, we may need to move it to the right place sooner or later.

Thanks,

Signature · 1 line
Peter Xu
#17 lore raw
Quoted 12 lines · Peter Xu

Hello, Jason, Eric,

Quoted 7 lines · Jason Wang

It might be because neither virtio bus nor virtio-net provides a shutdown method.

There used to be requests to provide those to unbreak the kexec.

A quick try might be to provide a .driver.shutdown to virtio_net_driver structure and reset the device there as a start.

Quoted 8 lines

I didn't check virtio driver path, but if that's missing it's reasonable to support it indeed.

OTOH, even with that, vhost can still hit such DMA issue if it's a hard-reset, am I right? IOW, when using QMP command "system-reset". If my memory is correct, that's the problem I was working on the VFIO series, rather than a clean reboot. And that won't give guest driver chance to run anything, IIUC.

Yes.

Quoted 50 lines

I am wildly suspecting a VT-d write to GCMD to disable it can also appear if there's a hard reset, then when bootloading the VM the bios (or whatever firmware at early stage) may want to make sure the VT-d device is completely off by writting to GCMD. But that's a pure guess.. and that may or may not matter much on how we fix this problem.

IOW, I suspect we need to fix both of them,

(a) for soft-reset, by making sure drivers properly quiesce DMAs proactively when VM gracefully shuts down.

(b) for hard-reset, by making sure QEMU reset in proper order.

One thing to mention is for problem (b) VFIO used to have an extra challenge on !FLR devices, I discussed it in patch 4's comment there. Quotting from patch 4 of series:

https://lore.kernel.org/all/[email protected]

* (1) Device depth-first reset hierachy doesn't yet work for vIOMMUs
* (reference: resettable_cold_reset_fn())
*
* Currently, vIOMMU devices are created as normal '-device'
* cmdlines. It means in many ways it has the same attributes with
* most of the rest devices, even if the rest devices should
* logically be under control of the vIOMMU unit.
*
* One side effect of it is vIOMMU devices will be currently put
* randomly under qdev tree hierarchy, which is the source of
* device reset ordering in current QEMU (depth-first traversal).
* It means vIOMMU now can be reset before some devices. For fully
* emulated devices that's not a problem, because the traversal
* holds BQL for the whole process. However it is a problem if DMA
* can happen without BQL, like VFIO, vDPA or remote device process.
*
* TODO: one ideal solution can be that we make vIOMMU the parent
* of the whole pci host bridge. Hence vIOMMU can be reset after
* all the devices are reset and quiesced.
*
* (2) Some devices register its own reset functions
*
* Even if above issue solved, if devices register its own reset
* functions for some reason via QEMU reset hooks, vIOMMU can still
* be reset before the device. One example is vfio_reset_handler()
* where FLR is not supported on the device.
*
* TODO: merge relevant reset functions into the device tree reset
* framework.

Quoted 6 lines

So maybe vhost doesn't have problem (2) listed above, and maybe it means it's still worthwhile thinking more about problem (1), which is to change the QOM tree to provide a correct topology representation when vIOMMU is
present: so far it should be still a pretty much orphaned object there.. if QEMU relies on QOM tree topology for reset order, we may need to move it to the right place sooner or later.

Sounds like a non-trivial task, so for a hard reset, maybe we can proceed with Eric's proposal to deal with the reset before the device stops.

Thanks

Quoted 6 lines

Thanks,

--
Peter Xu

#18 lore raw
Quoted 82 lines · Jason Wang
Quoted 12 lines · Peter Xu

Hello, Jason, Eric,

Quoted 7 lines · Jason Wang

It might be because neither virtio bus nor virtio-net provides a shutdown method.

There used to be requests to provide those to unbreak the kexec.

A quick try might be to provide a .driver.shutdown to virtio_net_driver structure and reset the device there as a start.

Quoted 8 lines

I didn't check virtio driver path, but if that's missing it's reasonable to support it indeed.

OTOH, even with that, vhost can still hit such DMA issue if it's a hard-reset, am I right? IOW, when using QMP command "system-reset". If my memory is correct, that's the problem I was working on the VFIO series, rather than a clean reboot. And that won't give guest driver chance to run anything, IIUC.

Yes.

Quoted 50 lines

I am wildly suspecting a VT-d write to GCMD to disable it can also appear if there's a hard reset, then when bootloading the VM the bios (or whatever firmware at early stage) may want to make sure the VT-d device is completely off by writting to GCMD. But that's a pure guess.. and that may or may not matter much on how we fix this problem.

IOW, I suspect we need to fix both of them,

(a) for soft-reset, by making sure drivers properly quiesce DMAs proactively when VM gracefully shuts down.

(b) for hard-reset, by making sure QEMU reset in proper order.

One thing to mention is for problem (b) VFIO used to have an extra challenge on !FLR devices, I discussed it in patch 4's comment there. Quotting from patch 4 of series:

https://lore.kernel.org/all/[email protected]

* (1) Device depth-first reset hierachy doesn't yet work for vIOMMUs
* (reference: resettable_cold_reset_fn())
*
* Currently, vIOMMU devices are created as normal '-device'
* cmdlines. It means in many ways it has the same attributes with
* most of the rest devices, even if the rest devices should
* logically be under control of the vIOMMU unit.
*
* One side effect of it is vIOMMU devices will be currently put
* randomly under qdev tree hierarchy, which is the source of
* device reset ordering in current QEMU (depth-first traversal).
* It means vIOMMU now can be reset before some devices. For fully
* emulated devices that's not a problem, because the traversal
* holds BQL for the whole process. However it is a problem if DMA
* can happen without BQL, like VFIO, vDPA or remote device process.
*
* TODO: one ideal solution can be that we make vIOMMU the parent
* of the whole pci host bridge. Hence vIOMMU can be reset after
* all the devices are reset and quiesced.
*
* (2) Some devices register its own reset functions
*
* Even if above issue solved, if devices register its own reset
* functions for some reason via QEMU reset hooks, vIOMMU can still
* be reset before the device. One example is vfio_reset_handler()
* where FLR is not supported on the device.
*
* TODO: merge relevant reset functions into the device tree reset
* framework.

Quoted 6 lines

So maybe vhost doesn't have problem (2) listed above, and maybe it means it's still worthwhile thinking more about problem (1), which is to change the QOM tree to provide a correct topology representation when vIOMMU is
present: so far it should be still a pretty much orphaned object there.. if QEMU relies on QOM tree topology for reset order, we may need to move it to the right place sooner or later.

Quoted 3 lines

Sounds like a non-trivial task, so for a hard reset, maybe we can proceed with Eric's proposal to deal with the reset before the device stops.

Btw, I actually meant to break the assumption that vhost needs to be enabled/disabled after/before vIOMMU. This only works for virtio-net / vhost. From the view of vhost, it would work similar to _F_LOG_ALL (where there's no assumption on the order of enabling/disabling dirty page tracking and device start/stop).

Thanks

Quoted 9 lines

Thanks

Quoted 6 lines

Thanks,

--
Peter Xu

#19 lore raw
Quoted 83 lines · Jason Wang
Quoted 82 lines · Jason Wang
Quoted 12 lines · Peter Xu

Hello, Jason, Eric,

Quoted 7 lines · Jason Wang

It might be because neither virtio bus nor virtio-net provides a shutdown method.

There used to be requests to provide those to unbreak the kexec.

A quick try might be to provide a .driver.shutdown to virtio_net_driver structure and reset the device there as a start.

Quoted 8 lines

I didn't check virtio driver path, but if that's missing it's reasonable to support it indeed.

OTOH, even with that, vhost can still hit such DMA issue if it's a hard-reset, am I right? IOW, when using QMP command "system-reset". If my memory is correct, that's the problem I was working on the VFIO series, rather than a clean reboot. And that won't give guest driver chance to run anything, IIUC.

Yes.

Quoted 56 lines

I am wildly suspecting a VT-d write to GCMD to disable it can also appear if there's a hard reset, then when bootloading the VM the bios (or whatever firmware at early stage) may want to make sure the VT-d device is completely off by writting to GCMD. But that's a pure guess.. and that may or may not matter much on how we fix this problem.

IOW, I suspect we need to fix both of them,

(a) for soft-reset, by making sure drivers properly quiesce DMAs proactively when VM gracefully shuts down.

(b) for hard-reset, by making sure QEMU reset in proper order.

One thing to mention is for problem (b) VFIO used to have an extra challenge on !FLR devices, I discussed it in patch 4's comment there. Quotting from patch 4 of series:

https://lore.kernel.org/all/[email protected]

* (1) Device depth-first reset hierachy doesn't yet work for vIOMMUs
* (reference: resettable_cold_reset_fn())
*
* Currently, vIOMMU devices are created as normal '-device'
* cmdlines. It means in many ways it has the same attributes with
* most of the rest devices, even if the rest devices should
* logically be under control of the vIOMMU unit.
*
* One side effect of it is vIOMMU devices will be currently put
* randomly under qdev tree hierarchy, which is the source of
* device reset ordering in current QEMU (depth-first traversal).
* It means vIOMMU now can be reset before some devices. For fully
* emulated devices that's not a problem, because the traversal
* holds BQL for the whole process. However it is a problem if DMA
* can happen without BQL, like VFIO, vDPA or remote device process.
*
* TODO: one ideal solution can be that we make vIOMMU the parent
* of the whole pci host bridge. Hence vIOMMU can be reset after
* all the devices are reset and quiesced.
*
* (2) Some devices register its own reset functions
*
* Even if above issue solved, if devices register its own reset
* functions for some reason via QEMU reset hooks, vIOMMU can still
* be reset before the device. One example is vfio_reset_handler()
* where FLR is not supported on the device.
*
* TODO: merge relevant reset functions into the device tree reset
* framework.

So maybe vhost doesn't have problem (2) listed above, and maybe it means it's still worthwhile thinking more about problem (1), which is to change the QOM tree to provide a correct topology representation when vIOMMU is
present: so far it should be still a pretty much orphaned object there.. if QEMU relies on QOM tree topology for reset order, we may need to move it to the right place sooner or later.

Quoted 3 lines
Quoted 3 lines

Sounds like a non-trivial task, so for a hard reset, maybe we can proceed with Eric's proposal to deal with the reset before the device stops.

The major challenge when I was working on that (as far as I can still remember..): some devices are created at early stage of QEMU startup, which can happen before the vIOMMU is created and realized. Then it can be challenging to re-parent those devices to be childrens of the vIOMMU, or we may need a way to create vIOMMU always earlier than those..

Quoted 6 lines

Btw, I actually meant to break the assumption that vhost needs to be enabled/disabled after/before vIOMMU. This only works for virtio-net / vhost. From the view of vhost, it would work similar to _F_LOG_ALL (where there's no assumption on the order of enabling/disabling dirty page tracking and device start/stop).

Yes, we can go for a lightweight solution.

Thanks,

Signature · 1 line
Peter Xu
author
#20 lore raw
In reply to Duan, Zhenzhong

Hi Zhenzhong,

Quoted 129 lines · Zhenzhong
Quoted 129 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Subject: Re: [PATCH] hw/virtio/vhost: Disable IOTLB callbacks when IOMMU gets disabled

Hi Jason,

Quoted 6 lines · Jason Wang
Quoted 5 lines · Eric Auger

Hi Jason,

Quoted 1 line · Jason Wang

On Wed, Jan 22, 2025 at 12:25 AM Eric Auger <[email protected]>

Quoted 4 lines · wrote:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Hi Jason,

Quoted 1 line · Jason Wang

On Tue, Jan 21, 2025 at 1:33 AM Eric Auger <[email protected]>

Quoted 7 lines · wrote:
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

When a guest exposed with a vhost device and protected by an intel IOMMU gets rebooted, we sometimes observe a spurious warning:

Fail to lookup the translated address ffffe000

We observe that the IOMMU gets disabled through a write to the global command register (CMAR_GCMD.TE) before the vhost device gets

stopped.

Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines

When this warning happens it can be observed an inflight IOTLB miss occurs after the IOMMU disable and before the vhost stop. In that case a flat translation occurs and the check in vhost_memory_region_lookup() fails.

Let's disable the IOTLB callbacks when all IOMMU MRs have been unregistered.

---
 hw/virtio/vhost.c | 4 ++++
 1 file changed, 4 insertions(+)
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

hw/virtio/vhost.c static void

Quoted 5 lines
hw/virtio/vhost.c +0 -0
index 6aa72fd434..128c2ab094 100644
@@ -931,6 +931,10 @@ static void

vhost_iommu_region_del(MemoryListener *listener,

Quoted 48 lines
Quoted 48 lines
Quoted 24 lines
Quoted 20 lines
Quoted 7 lines

hw/virtio/vhost.c static void · line 937

Quoted 7 lines
hw/virtio/vhost.c +4 -0
             break;
         }
     }
+    if (QLIST_EMPTY(&dev->iommu_list) &&
+        dev->vhost_ops->vhost_set_iotlb_callback) {
+        dev->vhost_ops->vhost_set_iotlb_callback(dev, false);
+    }
Quoted 7 lines

So the current code assumes:

1) IOMMU is enabled before vhost starts
2) IOMMU is disabled after vhost stops

This patch seems to fix 2) but not 1). Do we need to deal with the IOMMU enabled after vhost starts?

sorry I initially misunderstood the above comment. Indeed in the reboot case assumption 2) happens to be wrong. However what I currently do is:
stop listening to iotlb miss requests from the kernel because my understanding is those requests are just spurious ones, generate warnings and we do not care since we are rebooting the system.

Quoted 4 lines

However I do not claim this could handle the case where the IOMMU MR would be turned off and then turned on. I think in that case we should also flush the kernel IOTLB and this is not taken care of in this patch. Is it a relevant use case?

Quoted 6 lines

Not sure.

Quoted 2 lines

wrt removing assumption 1) and allow IOMMU enabled after vhost start. Is that a valid use case as the virtio driver is using the dma api?

It should not be but we can't assume the behaviour of the guest. It could be buggy or even malicious.

agreed

Quoted 17 lines

Btw, we had the following codes while handling te:

/* Handle Translation Enable/Disable */
static void vtd_handle_gcmd_te(IntelIOMMUState *s, bool en)

{
    if (s->dmar_enabled == en) {
        return;
    }

    trace_vtd_dmar_enable(en);

...

    vtd_reset_caches(s);
    vtd_address_space_refresh_all(s);
}
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

vtd_address_space_refresh_all() will basically disable the iommu memory region. It looks not sufficient to trigger the region_del callback, maybe we should delete the region or introduce listener callback?

This is exactly the code path which is entered in my use case.

vtd_address_space_refresh_all(s) induces the vhost_iommu_region_del. But

given the current implement of this latter the IOTLB callback is not unset and the kernel IOTLB is not refreshed. Also as I pointed out the hdev->mem->regions are not updated? shouldn't they. Can you explain what they correspond to?

Quoted 5 lines

Adding Peter for more ideas.

I think it's better to find a way to trigger the listener here, probably:

1) add/delete the memory regions instead of enable/disable

sorry I don't understand what you mean. The vhost_iommu_region_del call stack is provided below [1]. Write to the intel iommu GCMD TE bit induces a call to vhost_iommu_region_del. This happens before the vhost_dev_stop whose call stack is provided below [2] and originates from a bus reset.
We may have inflight IOTLB miss requests happening between both.

If this happens, vhost_device_iotlb_miss() fails because the IOVA is not translated anymore by the IOMMU and the iotlb.translated_addr returned by address_space_get_iotlb_entry() is not within the registered vhost_memory_regions looked up in vhost_memory_region_lookup(), hence the "Fail to lookup the translated address" message.

It sounds wrong that vhost keeps on using IOVAs that are not translated anymore. It looks we have a reset ordering issue and this patch is just removing the sympton and not the cause.

Quoted 5 lines
Quoted 3 lines

At the moment I don't really get what is initiating the intel iommu TE bit write. This is a guest action but is it initiated from a misordered qemu event?

During reboot, native_machine_shutdown() calls x86_platform.iommu_shutdown() to disable iommu before reset. So Peter's patch will not address your issue.

Exactly I see the initial IOMMU disable comes from this guest code path (native_machine_shutdown). Indeed this is unrelated to Peter's series. Then I see that vIOMMU is reset and then vhost_dev_stop is called (this is related to Peter's series).

Quoted 3 lines

Before iommu shutdown, device_shutdown() is called to shutdown all devices. Not clear why vhost is still active.

That's the piece I was missing next.

Thanks!

Eric

Quoted 8 lines

Thanks
Zhenzhong

Quoted 4 lines

It could also relate to
[PATCH 0/4] intel_iommu: Reset vIOMMU after all the rest of devices
https://lore.kernel.org/all/?q=s%3Aintel_iommu%3A+Reset+vIOMMU+after+all+
the+rest+of+devices

#21 lore raw

Hi Eric,

Quoted 4 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Subject: Re: [PATCH] hw/virtio/vhost: Disable IOTLB callbacks when IOMMU gets disabled

...

Quoted 37 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

vtd_address_space_refresh_all() will basically disable the iommu memory region. It looks not sufficient to trigger the region_del callback, maybe we should delete the region or introduce listener callback?

This is exactly the code path which is entered in my use case.

vtd_address_space_refresh_all(s) induces the vhost_iommu_region_del. But

given the current implement of this latter the IOTLB callback is not unset and

the

Quoted 1 line
Quoted 1 line

kernel IOTLB is not refreshed. Also as I pointed out the hdev->mem->regions

are

Quoted 26 lines
Quoted 26 lines

not updated? shouldn't they. Can you explain what they correspond to?

Quoted 5 lines

Adding Peter for more ideas.

I think it's better to find a way to trigger the listener here, probably:

1) add/delete the memory regions instead of enable/disable

sorry I don't understand what you mean. The vhost_iommu_region_del call stack is provided below [1]. Write to the intel iommu GCMD TE bit induces a call to vhost_iommu_region_del. This happens before the vhost_dev_stop whose call stack is provided below [2] and originates from a bus reset.
We may have inflight IOTLB miss requests happening between both.

If this happens, vhost_device_iotlb_miss() fails because the IOVA is not translated anymore by the IOMMU and the iotlb.translated_addr returned by address_space_get_iotlb_entry() is not within the registered vhost_memory_regions looked up in vhost_memory_region_lookup(), hence the "Fail to lookup the translated address" message.

It sounds wrong that vhost keeps on using IOVAs that are not translated anymore. It looks we have a reset ordering issue and this patch is just removing the sympton and not the cause.

At the moment I don't really get what is initiating the intel iommu TE bit write. This is a guest action but is it initiated from a misordered qemu event?

Quoted 8 lines
Quoted 1 line

During reboot, native_machine_shutdown() calls

x86_platform.iommu_shutdown()

Quoted 1 line

to disable iommu before reset. So Peter's patch will not address your issue.

Exactly I see the initial IOMMU disable comes from this guest code path (native_machine_shutdown). Indeed this is unrelated to Peter's series. Then I see that vIOMMU is reset and then vhost_dev_stop is called (this is related to Peter's series).

Agree.

Thanks
Zhenzhong

author
#22 lore raw
In reply to Duan, Zhenzhong

Hi

Quoted 129 lines · Zhenzhong
Quoted 129 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Subject: Re: [PATCH] hw/virtio/vhost: Disable IOTLB callbacks when IOMMU gets disabled

Hi Jason,

Quoted 6 lines · Jason Wang
Quoted 5 lines · Eric Auger

Hi Jason,

Quoted 1 line · Jason Wang

On Wed, Jan 22, 2025 at 12:25 AM Eric Auger <[email protected]>

Quoted 4 lines · wrote:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Hi Jason,

Quoted 1 line · Jason Wang

On Tue, Jan 21, 2025 at 1:33 AM Eric Auger <[email protected]>

Quoted 7 lines · wrote:
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

When a guest exposed with a vhost device and protected by an intel IOMMU gets rebooted, we sometimes observe a spurious warning:

Fail to lookup the translated address ffffe000

We observe that the IOMMU gets disabled through a write to the global command register (CMAR_GCMD.TE) before the vhost device gets

stopped.

Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines

When this warning happens it can be observed an inflight IOTLB miss occurs after the IOMMU disable and before the vhost stop. In that case a flat translation occurs and the check in vhost_memory_region_lookup() fails.

Let's disable the IOTLB callbacks when all IOMMU MRs have been unregistered.

---
 hw/virtio/vhost.c | 4 ++++
 1 file changed, 4 insertions(+)
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

hw/virtio/vhost.c static void

Quoted 5 lines
hw/virtio/vhost.c +0 -0
index 6aa72fd434..128c2ab094 100644
@@ -931,6 +931,10 @@ static void

vhost_iommu_region_del(MemoryListener *listener,

Quoted 48 lines
Quoted 48 lines
Quoted 24 lines
Quoted 20 lines
Quoted 7 lines

hw/virtio/vhost.c static void · line 937

Quoted 7 lines
hw/virtio/vhost.c +4 -0
             break;
         }
     }
+    if (QLIST_EMPTY(&dev->iommu_list) &&
+        dev->vhost_ops->vhost_set_iotlb_callback) {
+        dev->vhost_ops->vhost_set_iotlb_callback(dev, false);
+    }
Quoted 7 lines

So the current code assumes:

1) IOMMU is enabled before vhost starts
2) IOMMU is disabled after vhost stops

This patch seems to fix 2) but not 1). Do we need to deal with the IOMMU enabled after vhost starts?

sorry I initially misunderstood the above comment. Indeed in the reboot case assumption 2) happens to be wrong. However what I currently do is:
stop listening to iotlb miss requests from the kernel because my understanding is those requests are just spurious ones, generate warnings and we do not care since we are rebooting the system.

Quoted 4 lines

However I do not claim this could handle the case where the IOMMU MR would be turned off and then turned on. I think in that case we should also flush the kernel IOTLB and this is not taken care of in this patch. Is it a relevant use case?

Quoted 6 lines

Not sure.

Quoted 2 lines

wrt removing assumption 1) and allow IOMMU enabled after vhost start. Is that a valid use case as the virtio driver is using the dma api?

It should not be but we can't assume the behaviour of the guest. It could be buggy or even malicious.

agreed

Quoted 17 lines

Btw, we had the following codes while handling te:

/* Handle Translation Enable/Disable */
static void vtd_handle_gcmd_te(IntelIOMMUState *s, bool en)

{
    if (s->dmar_enabled == en) {
        return;
    }

    trace_vtd_dmar_enable(en);

...

    vtd_reset_caches(s);
    vtd_address_space_refresh_all(s);
}
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

vtd_address_space_refresh_all() will basically disable the iommu memory region. It looks not sufficient to trigger the region_del callback, maybe we should delete the region or introduce listener callback?

This is exactly the code path which is entered in my use case.

vtd_address_space_refresh_all(s) induces the vhost_iommu_region_del. But

given the current implement of this latter the IOTLB callback is not unset and the kernel IOTLB is not refreshed. Also as I pointed out the hdev->mem->regions are not updated? shouldn't they. Can you explain what they correspond to?

Quoted 5 lines

Adding Peter for more ideas.

I think it's better to find a way to trigger the listener here, probably:

1) add/delete the memory regions instead of enable/disable

sorry I don't understand what you mean. The vhost_iommu_region_del call stack is provided below [1]. Write to the intel iommu GCMD TE bit induces a call to vhost_iommu_region_del. This happens before the vhost_dev_stop whose call stack is provided below [2] and originates from a bus reset.
We may have inflight IOTLB miss requests happening between both.

If this happens, vhost_device_iotlb_miss() fails because the IOVA is not translated anymore by the IOMMU and the iotlb.translated_addr returned by address_space_get_iotlb_entry() is not within the registered vhost_memory_regions looked up in vhost_memory_region_lookup(), hence the "Fail to lookup the translated address" message.

It sounds wrong that vhost keeps on using IOVAs that are not translated anymore. It looks we have a reset ordering issue and this patch is just removing the sympton and not the cause.

Quoted 8 lines
Quoted 3 lines

At the moment I don't really get what is initiating the intel iommu TE bit write. This is a guest action but is it initiated from a misordered qemu event?

During reboot, native_machine_shutdown() calls x86_platform.iommu_shutdown() to disable iommu before reset. So Peter's patch will not address your issue.

Before iommu shutdown, device_shutdown() is called to shutdown all devices. Not clear why vhost is still active.

I tested [PATCH] virtio: Remove virtio devices on device_shutdown()
https://lore.kernel.org/all/[email protected]/

and it fixes my issue

Eric

Quoted 8 lines

Thanks
Zhenzhong

Quoted 4 lines

It could also relate to
[PATCH 0/4] intel_iommu: Reset vIOMMU after all the rest of devices
https://lore.kernel.org/all/?q=s%3Aintel_iommu%3A+Reset+vIOMMU+after+all+
the+rest+of+devices

author
#24 lore raw

Hi Michael,

Quoted 10 lines · Michael S. Tsirkin
Quoted 6 lines · Eric Auger

I tested [PATCH] virtio: Remove virtio devices on device_shutdown()
https://lore.kernel.org/all/[email protected]/

and it fixes my issue

Eric

To make sure, we are dropping this in favor of the guest fix (which I intent to merge shortly), correct?

Yes this should be dropped in favour of the implementation of the shutdown callback on guest side. Did you send your patch, I did not see it.

however on qemu side, there is "[PATCH v3 0/5] Fix vIOMMU reset order" that needs to be merged to fix qmp system_reset.

Thanks

Eric

Quoted 1 line
#25 lore raw
Quoted 10 lines · Eric Auger

Hi Michael,

Quoted 6 lines · Michael S. Tsirkin
Quoted 5 lines · Eric Auger

I tested [PATCH] virtio: Remove virtio devices on device_shutdown()
https://lore.kernel.org/all/[email protected]/

and it fixes my issue

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

Eric

To make sure, we are dropping this in favor of the guest fix (which I intent to merge shortly), correct?

Yes this should be dropped in favour of the implementation of the shutdown callback on guest side. Did you send your patch, I did not see it.

Will send tomorrow.

Quoted 2 lines

however on qemu side, there is "[PATCH v3 0/5] Fix vIOMMU reset order" that needs to be merged to fix qmp system_reset.

Yes that one I'm testing.

Quoted 4 lines

Thanks

Eric

Quoted 1 line
#26 lore raw
In reply to Eric Auger
Quoted 25 lines · Eric Auger

When a guest exposed with a vhost device and protected by an intel IOMMU gets rebooted, we sometimes observe a spurious warning:

Fail to lookup the translated address ffffe000

We observe that the IOMMU gets disabled through a write to the global command register (CMAR_GCMD.TE) before the vhost device gets stopped. When this warning happens it can be observed an inflight IOTLB miss occurs after the IOMMU disable and before the vhost stop. In that case a flat translation occurs and the check in vhost_memory_region_lookup() fails.

Let's disable the IOTLB callbacks when all IOMMU MRs have been unregistered.

---
  hw/virtio/vhost.c | 4 ++++
  1 file changed, 4 insertions(+)
hw/virtio/vhost.c +0 -0
index 6aa72fd434..128c2ab094 100644

hw/virtio/vhost.c static void vhost_iommu_region_del(MemoryListener *listener, · line 940

Quoted 10 lines
hw/virtio/vhost.c +4 -0
@@ -931,6 +931,10 @@ static void vhost_iommu_region_del(MemoryListener *listener,
              break;
          }
      }
+    if (QLIST_EMPTY(&dev->iommu_list) &&
+        dev->vhost_ops->vhost_set_iotlb_callback) {
+        dev->vhost_ops->vhost_set_iotlb_callback(dev, false);
+    }
  }
 
  void vhost_toggle_device_iotlb(VirtIODevice *vdev)

I think you need the counterpart in vhost_iommu_region_del() (for instance if we have an add after a del that results in an empty list). But you cannot unconditionally enable it (for instance if vhost is not started)

Perhaps you should move the vhost_set_iotlb_callback() call from vhost_start()/vhost_stop() to vhost_iommu_region_add()/vhost_iommu_region_del()?

Thanks,
Laurent

#27 lore raw
Quoted 21 lines · Laurent Vivier
Quoted 20 lines · Eric Auger

When a guest exposed with a vhost device and protected by an intel IOMMU gets rebooted, we sometimes observe a spurious warning:

Fail to lookup the translated address ffffe000

We observe that the IOMMU gets disabled through a write to the global command register (CMAR_GCMD.TE) before the vhost device gets stopped. When this warning happens it can be observed an inflight IOTLB miss occurs after the IOMMU disable and before the vhost stop. In that case a flat translation occurs and the check in vhost_memory_region_lookup() fails.

Let's disable the IOTLB callbacks when all IOMMU MRs have been unregistered.

---
 hw/virtio/vhost.c | 4 ++++
 1 file changed, 4 insertions(+)
Quoted 16 lines
Quoted 5 lines
hw/virtio/vhost.c +0 -0
index 6aa72fd434..128c2ab094 100644

hw/virtio/vhost.c static void vhost_iommu_region_del(MemoryListener *listener, · line 939

Quoted 9 lines
hw/virtio/vhost.c +4 -0
@@ -931,6 +931,10 @@ static void vhost_iommu_region_del(MemoryListener *listener,
             break;
         }
     }
+    if (QLIST_EMPTY(&dev->iommu_list) &&
+        dev->vhost_ops->vhost_set_iotlb_callback) {
+        dev->vhost_ops->vhost_set_iotlb_callback(dev, false);
+    }
 }
 void vhost_toggle_device_iotlb(VirtIODevice *vdev)

I think you need the counterpart in vhost_iommu_region_del() (for

I guess you meant vhost_iommu_region_add(). I was going to comment exactly on that, I agree here.

Quoted 2 lines

instance if we have an add after a del that results in an empty list). But you cannot unconditionally enable it (for instance if vhost is not started)

Good point.

Quoted 2 lines

Perhaps you should move the vhost_set_iotlb_callback() call from vhost_start()/vhost_stop() to vhost_iommu_region_add()/vhost_iommu_region_del()?

I also like this idea.

Stefano

author
#28 lore raw

Hi Stefano,

Quoted 47 lines · Stefano Garzarella
Quoted 30 lines · Laurent Vivier
Quoted 29 lines · Eric Auger

When a guest exposed with a vhost device and protected by an intel IOMMU gets rebooted, we sometimes observe a spurious warning:

Fail to lookup the translated address ffffe000

We observe that the IOMMU gets disabled through a write to the global command register (CMAR_GCMD.TE) before the vhost device gets stopped. When this warning happens it can be observed an inflight IOTLB miss occurs after the IOMMU disable and before the vhost stop. In that case a flat translation occurs and the check in vhost_memory_region_lookup() fails.

Let's disable the IOTLB callbacks when all IOMMU MRs have been unregistered.

---
 hw/virtio/vhost.c | 4 ++++

 1 file changed, 4 insertions(+)
hw/virtio/vhost.c +0 -0
index 6aa72fd434..128c2ab094 100644
@@ -931,6 +931,10 @@ static void
 vhost_iommu_region_del(MemoryListener *listener,
              break;
          }
      }
Quoted 8 lines

hw/virtio/vhost.c static void vhost_iommu_region_del(MemoryListener *listener, · line 940

Quoted 6 lines
hw/virtio/vhost.c +4 -0
+    if (QLIST_EMPTY(&dev->iommu_list) &&
+        dev->vhost_ops->vhost_set_iotlb_callback) {
+        dev->vhost_ops->vhost_set_iotlb_callback(dev, false);
+    }
  }
  void vhost_toggle_device_iotlb(VirtIODevice *vdev)

I think you need the counterpart in vhost_iommu_region_del() (for

I guess you meant vhost_iommu_region_add(). I was going to comment exactly on that, I agree here.

Quoted 3 lines

instance if we have an add after a del that results in an empty list). But you cannot unconditionally enable it (for instance if vhost is not started)

Quoted 8 lines

Good point.

Quoted 4 lines

Perhaps you should move the vhost_set_iotlb_callback() call from vhost_start()/vhost_stop() to vhost_iommu_region_add()/vhost_iommu_region_del()?

I also like this idea.

OK makes sense. I will go in this direction.

Eric

Quoted 3 lines

Stefano

author
#29 lore raw
In reply to Laurent Vivier
Quoted 31 lines · Laurent Vivier
Quoted 30 lines · Eric Auger

When a guest exposed with a vhost device and protected by an intel IOMMU gets rebooted, we sometimes observe a spurious warning:

Fail to lookup the translated address ffffe000

We observe that the IOMMU gets disabled through a write to the global command register (CMAR_GCMD.TE) before the vhost device gets stopped. When this warning happens it can be observed an inflight IOTLB miss occurs after the IOMMU disable and before the vhost stop. In that case a flat translation occurs and the check in vhost_memory_region_lookup() fails.

Let's disable the IOTLB callbacks when all IOMMU MRs have been unregistered.

---
  hw/virtio/vhost.c | 4 ++++

  1 file changed, 4 insertions(+)
hw/virtio/vhost.c +1 -0
index 6aa72fd434..128c2ab094 100644
@@ -931,6 +931,10 @@ static void
 vhost_iommu_region_del(MemoryListener *listener,
               break;
           }
       }
+    if (QLIST_EMPTY(&dev->iommu_list) &&
Quoted 8 lines

hw/virtio/vhost.c static void vhost_iommu_region_del(MemoryListener *listener, · line 940

Quoted 5 lines
hw/virtio/vhost.c +3 -0
+        dev->vhost_ops->vhost_set_iotlb_callback) {
+        dev->vhost_ops->vhost_set_iotlb_callback(dev, false);
+    }
   }
     void vhost_toggle_device_iotlb(VirtIODevice *vdev)

I think you need the counterpart in vhost_iommu_region_del() (for instance if we have an add after a del that results in an empty list).

I guess you meant vhost_iommu_region_add()

Quoted 2 lines

But you cannot unconditionally enable it (for instance if vhost is not started)

agreed. I will further look at the control path.

Quoted 4 lines

Perhaps you should move the vhost_set_iotlb_callback() call from vhost_start()/vhost_stop() to vhost_iommu_region_add()/vhost_iommu_region_del()?

Interesting. I will study that.

Thanks!

Eric

Quoted 4 lines

Thanks,
Laurent

author
#30 lore raw
In reply to Laurent Vivier

Hi,

Quoted 33 lines · Laurent Vivier
Quoted 32 lines · Eric Auger

When a guest exposed with a vhost device and protected by an intel IOMMU gets rebooted, we sometimes observe a spurious warning:

Fail to lookup the translated address ffffe000

We observe that the IOMMU gets disabled through a write to the global command register (CMAR_GCMD.TE) before the vhost device gets stopped. When this warning happens it can be observed an inflight IOTLB miss occurs after the IOMMU disable and before the vhost stop. In that case a flat translation occurs and the check in vhost_memory_region_lookup() fails.

Let's disable the IOTLB callbacks when all IOMMU MRs have been unregistered.

---
  hw/virtio/vhost.c | 4 ++++

  1 file changed, 4 insertions(+)
hw/virtio/vhost.c +3 -0
index 6aa72fd434..128c2ab094 100644
@@ -931,6 +931,10 @@ static void
 vhost_iommu_region_del(MemoryListener *listener,
               break;
           }
       }
+    if (QLIST_EMPTY(&dev->iommu_list) &&
+        dev->vhost_ops->vhost_set_iotlb_callback) {
+        dev->vhost_ops->vhost_set_iotlb_callback(dev, false);
Quoted 8 lines

hw/virtio/vhost.c static void vhost_iommu_region_del(MemoryListener *listener, · line 940

Quoted 3 lines
hw/virtio/vhost.c +1 -0
+    }
   }
     void vhost_toggle_device_iotlb(VirtIODevice *vdev)

I think you need the counterpart in vhost_iommu_region_del() (for instance if we have an add after a del that results in an empty list). But you cannot unconditionally enable it (for instance if vhost is not started)

if we enter vhost_iommu_region_add(), this means that the iommu_listener has been registered in vhost_vdev_start() and thus vdev->vhost_started is set.

Quoted 4 lines

Perhaps you should move the vhost_set_iotlb_callback() call from vhost_start()/vhost_stop() to vhost_iommu_region_add()/vhost_iommu_region_del()?

I currently fail to understand whether we shouldn't keep listening to iotlb callbacks when the IOMMU gets disabled. In that case shouldn't we flush the kernel IOTLB and update the hdev->mem->regions to reflect the IOMMR MR disablement?

Thanks

Eric

Quoted 4 lines

Thanks,
Laurent

#31 lore raw
In reply to Eric Auger

Hi Eric,

Quoted 9 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Subject: [PATCH] hw/virtio/vhost: Disable IOTLB callbacks when IOMMU gets disabled

When a guest exposed with a vhost device and protected by an intel IOMMU gets rebooted, we sometimes observe a spurious warning:

Fail to lookup the translated address ffffe000

Do you see this print once during one time reboot?

Quoted 10 lines

We observe that the IOMMU gets disabled through a write to the global command register (CMAR_GCMD.TE) before the vhost device gets stopped. When this warning happens it can be observed an inflight IOTLB miss occurs after the IOMMU disable and before the vhost stop. In that case a flat translation occurs and the check in vhost_memory_region_lookup() fails.

Let's disable the IOTLB callbacks when all IOMMU MRs have been unregistered.

Try to understand the sequence, is it like below?

vhost vcpu

call into vtd_iommu_translate();

                                                                    set s->dmar_enabled = false;
                                                                    switch off iommu address space;
                                                                    disable vhost IOTLB callbacks;

check if !s->dmar_enabled;
return flat translation and trigger warning

Thanks
Zhenzhong

Quoted 24 lines

---
hw/virtio/vhost.c | 4 ++++

1 file changed, 4 insertions(+)
hw/virtio/vhost.c +4 -0
index 6aa72fd434..128c2ab094 100644
@@ -931,6 +931,10 @@ static void vhost_iommu_region_del(MemoryListener
 *listener,
            break;
        }
    }
+    if (QLIST_EMPTY(&dev->iommu_list) &&
+        dev->vhost_ops->vhost_set_iotlb_callback) {
+        dev->vhost_ops->vhost_set_iotlb_callback(dev, false);
+    }
 }
 

void vhost_toggle_device_iotlb(VirtIODevice *vdev) --
2.47.1

author
#32 lore raw

Hi Zhenzhong,

Quoted 7 lines · Zhenzhong

Hi Eric,

Quoted 5 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Subject: [PATCH] hw/virtio/vhost: Disable IOTLB callbacks when IOMMU gets disabled

Quoted 5 lines
Quoted 4 lines

When a guest exposed with a vhost device and protected by an intel IOMMU gets rebooted, we sometimes observe a spurious warning:

Fail to lookup the translated address ffffe000

Do you see this print once during one time reboot?

Actually this happens rarely on reboot. The reproducibility is of the order of magnitude of 1/10 for me. I use a vm with vhost net device + virtual intel iommu featuring a crontab job.
@reboot /usr/sbin/reboot

Quoted 8 lines
Quoted 8 lines

We observe that the IOMMU gets disabled through a write to the global command register (CMAR_GCMD.TE) before the vhost device gets stopped. When this warning happens it can be observed an inflight IOTLB miss occurs after the IOMMU disable and before the vhost stop. In that case a flat translation occurs and the check in vhost_memory_region_lookup() fails.

Quoted 7 lines
Quoted 2 lines

Let's disable the IOTLB callbacks when all IOMMU MRs have been unregistered.

Try to understand the sequence, is it like below?

vhost vcpu

call into vtd_iommu_translate();

No that's a kernel vhost translate request that normally tries to find out the translated address on kernel side in the IOTLB but since the data is not there it ends up asking for the translation to user space ...

Quoted 4 lines
                                                                    set s->dmar_enabled = false;
                                                                    switch off iommu address space;
                                                                    disable vhost IOTLB callbacks;

vtd_handle_gcmd_write/vtd_handle_gcmd_te/vtd_handle_gcmd_te which eventually calls vhost_iommu_region_del

Quoted 3 lines

check if !s->dmar_enabled;
return flat translation and trigger warning

vhost inflight translation reaches user space through vhost_device_iotlb_miss()

Quoted 27 lines

Thanks
Zhenzhong

Quoted 23 lines

---
hw/virtio/vhost.c | 4 ++++

1 file changed, 4 insertions(+)
hw/virtio/vhost.c +4 -0
index 6aa72fd434..128c2ab094 100644
@@ -931,6 +931,10 @@ static void vhost_iommu_region_del(MemoryListener
 *listener,
            break;
        }
    }
+    if (QLIST_EMPTY(&dev->iommu_list) &&
+        dev->vhost_ops->vhost_set_iotlb_callback) {
+        dev->vhost_ops->vhost_set_iotlb_callback(dev, false);
+    }
 }
 

void vhost_toggle_device_iotlb(VirtIODevice *vdev) --
2.47.1

#33 lore raw
Quoted 49 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Subject: Re: [PATCH] hw/virtio/vhost: Disable IOTLB callbacks when IOMMU gets disabled

Hi Zhenzhong,

Quoted 7 lines · Zhenzhong

Hi Eric,

Quoted 5 lines

-----Original Message-----
From: Eric Auger <[email protected]>
Subject: [PATCH] hw/virtio/vhost: Disable IOTLB callbacks when IOMMU gets disabled

Quoted 5 lines
Quoted 4 lines

When a guest exposed with a vhost device and protected by an intel IOMMU gets rebooted, we sometimes observe a spurious warning:

Fail to lookup the translated address ffffe000

Do you see this print once during one time reboot?

Actually this happens rarely on reboot. The reproducibility is of the order of magnitude of 1/10 for me. I use a vm with vhost net device + virtual intel iommu featuring a crontab job.
@reboot /usr/sbin/reboot

Quoted 8 lines
Quoted 8 lines

We observe that the IOMMU gets disabled through a write to the global command register (CMAR_GCMD.TE) before the vhost device gets stopped. When this warning happens it can be observed an inflight IOTLB miss occurs after the IOMMU disable and before the vhost stop. In that case a flat translation occurs and the check in vhost_memory_region_lookup() fails.

Quoted 7 lines
Quoted 2 lines

Let's disable the IOTLB callbacks when all IOMMU MRs have been unregistered.

Try to understand the sequence, is it like below?

vhost vcpu

call into vtd_iommu_translate();

No that's a kernel vhost translate request that normally tries to find out the translated address on kernel side in the IOTLB but since the data is not there it ends up asking for the translation to user space ...

Quoted 4 lines
                                                                    set s->dmar_enabled = false;
                                                                    switch off iommu address space;
                                                                    disable vhost IOTLB callbacks;
Quoted 7 lines

vtd_handle_gcmd_write/vtd_handle_gcmd_te/vtd_handle_gcmd_te which eventually calls vhost_iommu_region_del

Quoted 3 lines

check if !s->dmar_enabled;
return flat translation and trigger warning

vhost inflight translation reaches user space through vhost_device_iotlb_miss()

Understood, thanks Eric!

BRs.
Zhenzhong