Skip to contentSkip to conversation

intel-iommu: Document iova_tree

PATCH Superseded Peter Xu 3 participants 13 messages Dec 1, 2022 – Dec 6, 2022

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Peter Xu

author
#0 lore raw

PATCH 1/1 intel-iommu: Document iova_tree

It seems not super clear on when iova_tree is used, and why. Add a rich comment above iova_tree to track why we needed the iova_tree, and when we need it.

---
 include/hw/i386/intel_iommu.h | 30 +++++++++++++++++++++++++++++-
 1 file changed, 29 insertions(+), 1 deletion(-)
include/hw/i386/intel_iommu.h +29 -1
index 46d973e629..8d130ab2e3 100644
@@ -109,7 +109,35 @@ struct VTDAddressSpace {
     QLIST_ENTRY(VTDAddressSpace) next;
     /* Superset of notifier flags that this address space has */
     IOMMUNotifierFlag notifier_flags;
-    IOVATree *iova_tree;          /* Traces mapped IOVA ranges */
+    /*
+     * @iova_tree traces mapped IOVA ranges.
+     *
+     * The tree is not needed if no MAP notifiers is registered with
+     * current VTD address space, because all UNMAP (including iotlb or
+     * dev-iotlb) events can be transparently delivered to !MAP iommu
+     * notifiers.
+     *
+     * The tree OTOH is required for MAP typed iommu notifiers for a few
+     * reasons.
+     *
+     * Firstly, there's no way to identify whether an PSI event is MAP or
+     * UNMAP within the PSI message itself.  Without having prior knowledge
+     * of existing state vIOMMU doesn't know whether it should notify MAP
+     * or UNMAP for a PSI message it received.
+     *
+     * Secondly, PSI received from guest driver (or even a large PSI can
+     * grow into a DSI at least with Linux intel-iommu driver) can be
+     * larger in range than the newly mapped ranges for either MAP or UNMAP
+     * events. If it directly pass-throughs any such event it may confuse
+     * the registered drivers (e.g. vfio-pci) on either: (1) trying to map
+     * the same region more than once (for VFIO_IOMMU_MAP_DMA, -EEXIST will
+     * trigger), or (2) trying to UNMAP a range that is still partially
+     * mapped.  That accuracy is not required for UNMAP-only notifiers, but
+     * it is a must-to-have for MAP-inclusive notifiers, because the vIOMMU
+     * needs to make sure the shadow page table is always in sync with the
+     * guest IOMMU pgtables for a device.
+     */
+    IOVATree *iova_tree;
 };
 
 struct VTDIOTLBEntry {
Signature · 1 line
2.37.3
#1 lore raw

Hi Peter

Quoted 15 lines · Peter Xu

It seems not super clear on when iova_tree is used, and why. Add a rich comment above iova_tree to track why we needed the iova_tree, and when we need it.

---
 include/hw/i386/intel_iommu.h | 30 +++++++++++++++++++++++++++++-
 1 file changed, 29 insertions(+), 1 deletion(-)
include/hw/i386/intel_iommu.h +0 -0
index 46d973e629..8d130ab2e3 100644

include/hw/i386/intel_iommu.h struct VTDAddressSpace { · line 115

Quoted 8 lines
include/hw/i386/intel_iommu.h +4 -1
@@ -109,7 +109,35 @@ struct VTDAddressSpace {
     QLIST_ENTRY(VTDAddressSpace) next;
     /* Superset of notifier flags that this address space has */
     IOMMUNotifierFlag notifier_flags;
-    IOVATree *iova_tree;          /* Traces mapped IOVA ranges */
+    /*
+     * @iova_tree traces mapped IOVA ranges.
+     *
+     * The tree is not needed if no MAP notifiers is registered with

s/no MAP notifiers/no MAP notifier

include/hw/i386/intel_iommu.h struct VTDAddressSpace { · line 118

Quoted 3 lines
include/hw/i386/intel_iommu.h +3 -0
+     * current VTD address space, because all UNMAP (including iotlb or
+     * dev-iotlb) events can be transparently delivered to !MAP iommu
+     * notifiers.

because all UNMAP notifications (iotlb or dev-iotlb) can be triggered directly, as opposed to MAP notifications. (?)

include/hw/i386/intel_iommu.h struct VTDAddressSpace { · line 123

Quoted 5 lines
include/hw/i386/intel_iommu.h +5 -0
+     *
+     * The tree OTOH is required for MAP typed iommu notifiers for a few
+     * reasons.
+     *
+     * Firstly, there's no way to identify whether an PSI event is MAP or

maybe give the decryption of the 'PSI' and 'DSI" acronyms once ;-)

include/hw/i386/intel_iommu.h struct VTDAddressSpace { · line 131

Quoted 8 lines
include/hw/i386/intel_iommu.h +8 -0
+     * UNMAP within the PSI message itself.  Without having prior knowledge
+     * of existing state vIOMMU doesn't know whether it should notify MAP
+     * or UNMAP for a PSI message it received.
+     *
+     * Secondly, PSI received from guest driver (or even a large PSI can
+     * grow into a DSI at least with Linux intel-iommu driver) can be
+     * larger in range than the newly mapped ranges for either MAP or UNMAP
+     * events. If it directly pass-throughs any such event it may confuse

If it directly notifies the registered device with the unmodified range, it may confuse the drivers ../..

So the range of the MAP notification can be adapted based on the existing IOVA mappings.

include/hw/i386/intel_iommu.h struct VTDAddressSpace { · line 143

Quoted 12 lines
include/hw/i386/intel_iommu.h +9 -0
+     * the registered drivers (e.g. vfio-pci) on either: (1) trying to map
+     * the same region more than once (for VFIO_IOMMU_MAP_DMA, -EEXIST will
+     * trigger), or (2) trying to UNMAP a range that is still partially
+     * mapped.  That accuracy is not required for UNMAP-only notifiers, but
+     * it is a must-to-have for MAP-inclusive notifiers, because the vIOMMU
+     * needs to make sure the shadow page table is always in sync with the
+     * guest IOMMU pgtables for a device.
+     */
+    IOVATree *iova_tree;
 };
 
 struct VTDIOTLBEntry {

Thanks

Eric

author
#2 lore raw
Quoted 1 line · Eric Auger

Hi Peter

Hi, Eric,

Quoted 12 lines · Peter Xu
Quoted 10 lines

It seems not super clear on when iova_tree is used, and why. Add a rich comment above iova_tree to track why we needed the iova_tree, and when we need it.

---
 include/hw/i386/intel_iommu.h | 30 +++++++++++++++++++++++++++++-
 1 file changed, 29 insertions(+), 1 deletion(-)
Quoted 15 lines

include/hw/i386/intel_iommu.h struct VTDAddressSpace { · line 115

Quoted 13 lines
include/hw/i386/intel_iommu.h +4 -1
index 46d973e629..8d130ab2e3 100644
@@ -109,7 +109,35 @@ struct VTDAddressSpace {
     QLIST_ENTRY(VTDAddressSpace) next;
     /* Superset of notifier flags that this address space has */
     IOMMUNotifierFlag notifier_flags;
-    IOVATree *iova_tree;          /* Traces mapped IOVA ranges */
+    /*
+     * @iova_tree traces mapped IOVA ranges.
+     *
+     * The tree is not needed if no MAP notifiers is registered with

s/no MAP notifiers/no MAP notifier

Will fix.

Quoted 5 lines

include/hw/i386/intel_iommu.h struct VTDAddressSpace { · line 118

Quoted 3 lines
include/hw/i386/intel_iommu.h +3 -0
+     * current VTD address space, because all UNMAP (including iotlb or
+     * dev-iotlb) events can be transparently delivered to !MAP iommu
+     * notifiers.

because all UNMAP notifications (iotlb or dev-iotlb) can be triggered directly, as opposed to MAP notifications. (?)

What I wanted to say is any PSI or DSI messages we got from the guest can be transparently delivered to QEMU's iommu notifiers. I'm not sure "triggered directly" best describe the case here.

PSI: Page Selective Invalidations
DSI: Domain Selective Invalidations

Sorry to mention these terms again, but that's really what the "transparent delivery" means here - we get the PSI/DSI messages, then we notify with the same ranges in IOMMU notifiers. They're not the same concept but we do that transparently without changing the core of the messages.

Maybe I should spell out "!MAP" as "UNMAP-only"? Would that help?

Quoted 6 lines

include/hw/i386/intel_iommu.h struct VTDAddressSpace { · line 123

Quoted 5 lines
include/hw/i386/intel_iommu.h +5 -0
+     *
+     * The tree OTOH is required for MAP typed iommu notifiers for a few
+     * reasons.
+     *
+     * Firstly, there's no way to identify whether an PSI event is MAP or

maybe give the decryption of the 'PSI' and 'DSI" acronyms once ;-)

Please see above. :)

These are VT-d terms used in multiple places in the .[ch] files, I assume I'll just keep using them because otherwise I'll need to comment them everytime we use any PSI/DSI terms. It might become an overkill I'm afraid.

Quoted 10 lines

include/hw/i386/intel_iommu.h struct VTDAddressSpace { · line 131

Quoted 8 lines
include/hw/i386/intel_iommu.h +8 -0
+     * UNMAP within the PSI message itself.  Without having prior knowledge
+     * of existing state vIOMMU doesn't know whether it should notify MAP
+     * or UNMAP for a PSI message it received.
+     *
+     * Secondly, PSI received from guest driver (or even a large PSI can
+     * grow into a DSI at least with Linux intel-iommu driver) can be
+     * larger in range than the newly mapped ranges for either MAP or UNMAP
+     * events. If it directly pass-throughs any such event it may confuse

If it directly notifies the registered device with the unmodified range, it may confuse the drivers ../..

Will fix.

Quoted 2 lines

So the range of the MAP notification can be adapted based on the existing IOVA mappings.

Yes, e.g. the iova tree makes sure we don't map something again if it's mapped.

Thanks,

Quoted 17 lines

include/hw/i386/intel_iommu.h struct VTDAddressSpace { · line 143

Quoted 13 lines
include/hw/i386/intel_iommu.h +9 -0
+     * the registered drivers (e.g. vfio-pci) on either: (1) trying to map
+     * the same region more than once (for VFIO_IOMMU_MAP_DMA, -EEXIST will
+     * trigger), or (2) trying to UNMAP a range that is still partially
+     * mapped.  That accuracy is not required for UNMAP-only notifiers, but
+     * it is a must-to-have for MAP-inclusive notifiers, because the vIOMMU
+     * needs to make sure the shadow page table is always in sync with the
+     * guest IOMMU pgtables for a device.
+     */
+    IOVATree *iova_tree;
 };
 
 struct VTDIOTLBEntry {

Thanks

Eric

Signature · 1 line
Peter Xu
author
#3 lore raw
Quoted 77 lines · Peter Xu
Quoted 1 line · Eric Auger

Hi Peter

Hi, Eric,

Quoted 12 lines · Peter Xu
Quoted 10 lines

It seems not super clear on when iova_tree is used, and why. Add a rich comment above iova_tree to track why we needed the iova_tree, and when we need it.

---
 include/hw/i386/intel_iommu.h | 30 +++++++++++++++++++++++++++++-
 1 file changed, 29 insertions(+), 1 deletion(-)
Quoted 15 lines

include/hw/i386/intel_iommu.h struct VTDAddressSpace { · line 115

Quoted 13 lines
include/hw/i386/intel_iommu.h +4 -1
index 46d973e629..8d130ab2e3 100644
@@ -109,7 +109,35 @@ struct VTDAddressSpace {
     QLIST_ENTRY(VTDAddressSpace) next;
     /* Superset of notifier flags that this address space has */
     IOMMUNotifierFlag notifier_flags;
-    IOVATree *iova_tree;          /* Traces mapped IOVA ranges */
+    /*
+     * @iova_tree traces mapped IOVA ranges.
+     *
+     * The tree is not needed if no MAP notifiers is registered with

s/no MAP notifiers/no MAP notifier

Will fix.

Quoted 5 lines

include/hw/i386/intel_iommu.h struct VTDAddressSpace { · line 118

Quoted 3 lines
include/hw/i386/intel_iommu.h +3 -0
+     * current VTD address space, because all UNMAP (including iotlb or
+     * dev-iotlb) events can be transparently delivered to !MAP iommu
+     * notifiers.

because all UNMAP notifications (iotlb or dev-iotlb) can be triggered directly, as opposed to MAP notifications. (?)

What I wanted to say is any PSI or DSI messages we got from the guest can be transparently delivered to QEMU's iommu notifiers. I'm not sure "triggered directly" best describe the case here.

PSI: Page Selective Invalidations
DSI: Domain Selective Invalidations

Sorry to mention these terms again, but that's really what the "transparent delivery" means here - we get the PSI/DSI messages, then we notify with the same ranges in IOMMU notifiers. They're not the same concept but we do that transparently without changing the core of the messages.

Maybe I should spell out "!MAP" as "UNMAP-only"? Would that help?

Quoted 6 lines

include/hw/i386/intel_iommu.h struct VTDAddressSpace { · line 123

Quoted 5 lines
include/hw/i386/intel_iommu.h +5 -0
+     *
+     * The tree OTOH is required for MAP typed iommu notifiers for a few
+     * reasons.
+     *
+     * Firstly, there's no way to identify whether an PSI event is MAP or

maybe give the decryption of the 'PSI' and 'DSI" acronyms once ;-)

Please see above. :)

These are VT-d terms used in multiple places in the .[ch] files, I assume I'll just keep using them because otherwise I'll need to comment them everytime we use any PSI/DSI terms. It might become an overkill I'm afraid.

Quoted 9 lines
Quoted 8 lines
include/hw/i386/intel_iommu.h +8 -0
+     * UNMAP within the PSI message itself.  Without having prior knowledge
+     * of existing state vIOMMU doesn't know whether it should notify MAP
+     * or UNMAP for a PSI message it received.
+     *
+     * Secondly, PSI received from guest driver (or even a large PSI can
+     * grow into a DSI at least with Linux intel-iommu driver) can be
+     * larger in range than the newly mapped ranges for either MAP or UNMAP
+     * events. If it directly pass-throughs any such event it may confuse
Quoted 8 lines
Quoted 1 line

If it directly notifies the registered device with the unmodified range, it may confuse the drivers ../..

Will fix.

Quoted 2 lines

So the range of the MAP notification can be adapted based on the existing IOVA mappings.

Yes, e.g. the iova tree makes sure we don't map something again if it's mapped.

I figured maybe simpler I just attach a v2..

Thanks,

Signature · 1 line
Peter Xu

From 67594a4bfad481a2810b69e2b17e6399f39a18a2 Mon Sep 17 00:00:00 2001
From: Peter Xu <[email protected]>
Date: Thu, 1 Dec 2022 11:11:41 -0500
Subject: [PATCH] intel-iommu: Document iova_tree
Content-type: text/plain

It seems not super clear on when iova_tree is used, and why. Add a rich comment above iova_tree to track why we needed the iova_tree, and when we need it.

---
 include/hw/i386/intel_iommu.h | 37 ++++++++++++++++++++++++++++++++++-
 1 file changed, 36 insertions(+), 1 deletion(-)
include/hw/i386/intel_iommu.h +36 -1
index 46d973e629..d96da8cc75 100644
@@ -109,7 +109,42 @@ struct VTDAddressSpace {
     QLIST_ENTRY(VTDAddressSpace) next;
     /* Superset of notifier flags that this address space has */
     IOMMUNotifierFlag notifier_flags;
-    IOVATree *iova_tree;          /* Traces mapped IOVA ranges */
+    /*
+     * @iova_tree traces mapped IOVA ranges.
+     *
+     * The tree is not needed if no MAP notifier is registered with current
+     * VTD address space, because all UNMAP events the vIOMMU receives (for
+     * either iotlb or dev-iotlb) can be transparently delivered to iommu
+     * notifiers.
+     *
+     * The tree OTOH is required for MAP typed iommu notifiers for a few
+     * reasons.
+     *
+     * Firstly, there's no way to identify whether an PSI/DSI event is an
+     * MAP or UNMAP event within the message itself.  Without having prior
+     * knowledge of existing state vIOMMU doesn't know whether it should
+     * notify MAP or UNMAP for a PSI message it received when caching mode
+     * is enabled (for MAP notifiers).
+     *
+     * Secondly, PSI messages received from guest driver can be enlarged in
+     * range, covers but not limited to what the guest driver wanted to
+     * invalidate.  When the range to invalidates gets bigger than the
+     * limit of a PSI message, it can even become a DSI which will
+     * invalidate the whole domain.  If the vIOMMU directly notifies the
+     * registered device with the unmodified range, it may confuse the
+     * registered drivers (e.g. vfio-pci) on either:
+     *
+     *   (1) Trying to map the same region more than once (for
+     *       VFIO_IOMMU_MAP_DMA, -EEXIST will trigger), or,
+     *
+     *   (2) Trying to UNMAP a range that is still partially mapped.
+     *
+     * That accuracy is not required for UNMAP-only notifiers, but it is a
+     * must-to-have for notifiers registered with MAP events, because the
+     * vIOMMU needs to make sure the shadow page table is always in sync
+     * with the guest IOMMU pgtables for a device.
+     */
+    IOVATree *iova_tree;
 };
 
 struct VTDIOTLBEntry {
Signature · 1 line
2.37.3
#4 lore raw
In reply to Peter Xu

Hi Peter,

Quoted 31 lines · Peter Xu
Quoted 1 line · Eric Auger

Hi Peter

Hi, Eric,

Quoted 11 lines · Peter Xu
Quoted 10 lines

It seems not super clear on when iova_tree is used, and why. Add a rich comment above iova_tree to track why we needed the iova_tree, and when we need it.

---
 include/hw/i386/intel_iommu.h | 30 +++++++++++++++++++++++++++++-
 1 file changed, 29 insertions(+), 1 deletion(-)
Quoted 14 lines

include/hw/i386/intel_iommu.h struct VTDAddressSpace { · line 115

Quoted 13 lines
include/hw/i386/intel_iommu.h +4 -1
index 46d973e629..8d130ab2e3 100644
@@ -109,7 +109,35 @@ struct VTDAddressSpace {
     QLIST_ENTRY(VTDAddressSpace) next;
     /* Superset of notifier flags that this address space has */
     IOMMUNotifierFlag notifier_flags;
-    IOVATree *iova_tree;          /* Traces mapped IOVA ranges */
+    /*
+     * @iova_tree traces mapped IOVA ranges.
+     *
+     * The tree is not needed if no MAP notifiers is registered with

s/no MAP notifiers/no MAP notifier

Will fix.

Quoted 8 lines
Quoted 5 lines

include/hw/i386/intel_iommu.h struct VTDAddressSpace { · line 118

Quoted 3 lines
include/hw/i386/intel_iommu.h +3 -0
+     * current VTD address space, because all UNMAP (including iotlb or
+     * dev-iotlb) events can be transparently delivered to !MAP iommu
+     * notifiers.

because all UNMAP notifications (iotlb or dev-iotlb) can be triggered directly, as opposed to MAP notifications. (?)

What I wanted to say is any PSI or DSI messages we got from the guest can be transparently delivered to QEMU's iommu notifiers. I'm not sure "triggered directly" best describe the case here.

yes "transparently delivered" is OK. Or "guest invalidate commands can be directly passed to the IOMMU UNMAP notifiers without any further reshuffling". But that's nitpicking.

Quoted 10 lines

PSI: Page Selective Invalidations
DSI: Domain Selective Invalidations

Sorry to mention these terms again, but that's really what the "transparent delivery" means here - we get the PSI/DSI messages, then we notify with the same ranges in IOMMU notifiers. They're not the same concept but we do that transparently without changing the core of the messages.

Maybe I should spell out "!MAP" as "UNMAP-only"? Would that help?

yeah those are unmap notifiers if I am correct.

Quoted 8 lines
Quoted 7 lines

include/hw/i386/intel_iommu.h struct VTDAddressSpace { · line 123

Quoted 6 lines
include/hw/i386/intel_iommu.h +5 -0
+     *
+     * The tree OTOH is required for MAP typed iommu notifiers for a few
+     * reasons.
+     *
+     * Firstly, there's no way to identify whether an PSI event is MAP or

maybe give the decryption of the 'PSI' and 'DSI" acronyms once ;-)

Please see above. :)

ok thanks

Quoted 4 lines

These are VT-d terms used in multiple places in the .[ch] files, I assume I'll just keep using them because otherwise I'll need to comment them everytime we use any PSI/DSI terms. It might become an overkill I'm afraid.

OK maybe just using the full terminology once is enough.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
include/hw/i386/intel_iommu.h +8 -0
+     * UNMAP within the PSI message itself.  Without having prior knowledge
+     * of existing state vIOMMU doesn't know whether it should notify MAP
+     * or UNMAP for a PSI message it received.
+     *
+     * Secondly, PSI received from guest driver (or even a large PSI can
+     * grow into a DSI at least with Linux intel-iommu driver) can be
+     * larger in range than the newly mapped ranges for either MAP or UNMAP
+     * events. If it directly pass-throughs any such event it may confuse
Quoted 5 lines
Quoted 1 line

If it directly notifies the registered device with the unmodified range, it may confuse the drivers ../..

Will fix.

Quoted 1 line

So the range of the MAP notification can be adapted based on the existing IOVA mappings.

Yes, e.g. the iova tree makes sure we don't map something again if it's mapped.

OK

Thanks

Eric

Quoted 19 lines

Thanks,

Quoted 16 lines

include/hw/i386/intel_iommu.h struct VTDAddressSpace { · line 143

Quoted 12 lines
include/hw/i386/intel_iommu.h +9 -0
+     * the registered drivers (e.g. vfio-pci) on either: (1) trying to map
+     * the same region more than once (for VFIO_IOMMU_MAP_DMA, -EEXIST will
+     * trigger), or (2) trying to UNMAP a range that is still partially
+     * mapped.  That accuracy is not required for UNMAP-only notifiers, but
+     * it is a must-to-have for MAP-inclusive notifiers, because the vIOMMU
+     * needs to make sure the shadow page table is always in sync with the
+     * guest IOMMU pgtables for a device.
+     */
+    IOVATree *iova_tree;
 };
 
 struct VTDIOTLBEntry {

Thanks

Eric

author
#5 lore raw
Quoted 11 lines · Eric Auger
Quoted 8 lines
Quoted 5 lines

include/hw/i386/intel_iommu.h struct VTDAddressSpace { · line 118

Quoted 3 lines
include/hw/i386/intel_iommu.h +3 -0
+     * current VTD address space, because all UNMAP (including iotlb or
+     * dev-iotlb) events can be transparently delivered to !MAP iommu
+     * notifiers.

because all UNMAP notifications (iotlb or dev-iotlb) can be triggered directly, as opposed to MAP notifications. (?)

What I wanted to say is any PSI or DSI messages we got from the guest can be transparently delivered to QEMU's iommu notifiers. I'm not sure "triggered directly" best describe the case here.

yes "transparently delivered" is OK. Or "guest invalidate commands can be directly passed to the IOMMU UNMAP notifiers without any further reshuffling". But that's nitpicking.

Will do.

Quoted 17 lines
Quoted 10 lines

PSI: Page Selective Invalidations
DSI: Domain Selective Invalidations

Sorry to mention these terms again, but that's really what the "transparent delivery" means here - we get the PSI/DSI messages, then we notify with the same ranges in IOMMU notifiers. They're not the same concept but we do that transparently without changing the core of the messages.

Maybe I should spell out "!MAP" as "UNMAP-only"? Would that help?

yeah those are unmap notifiers if I am correct.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
include/hw/i386/intel_iommu.h +5 -0
+     *
+     * The tree OTOH is required for MAP typed iommu notifiers for a few
+     * reasons.
+     *
+     * Firstly, there's no way to identify whether an PSI event is MAP or
Quoted 8 lines
Quoted 2 lines
Quoted 1 line

maybe give the decryption of the 'PSI' and 'DSI" acronyms once ;-)

Please see above. :)

ok thanks

Quoted 4 lines

These are VT-d terms used in multiple places in the .[ch] files, I assume I'll just keep using them because otherwise I'll need to comment them everytime we use any PSI/DSI terms. It might become an overkill I'm afraid.

OK maybe just using the full terminology once is enough.

Ok, I'll add them.

Thanks Eric.

Signature · 1 line
Peter Xu
#6 lore raw
In reply to Peter Xu
Quoted 17 lines · Peter Xu

It seems not super clear on when iova_tree is used, and why. Add a rich comment above iova_tree to track why we needed the iova_tree, and when we need it.

---
 include/hw/i386/intel_iommu.h | 30 +++++++++++++++++++++++++++++-
 1 file changed, 29 insertions(+), 1 deletion(-)
include/hw/i386/intel_iommu.h +0 -0
index 46d973e629..8d130ab2e3 100644
@@ -109,7 +109,35 @@ struct VTDAddressSpace {
     QLIST_ENTRY(VTDAddressSpace) next;

include/hw/i386/intel_iommu.h struct VTDAddressSpace { · line 118

Quoted 10 lines
include/hw/i386/intel_iommu.h +7 -1
     /* Superset of notifier flags that this address space has */
     IOMMUNotifierFlag notifier_flags;
-    IOVATree *iova_tree;          /* Traces mapped IOVA ranges */
+    /*
+     * @iova_tree traces mapped IOVA ranges.
+     *
+     * The tree is not needed if no MAP notifiers is registered with
+     * current VTD address space, because all UNMAP (including iotlb or
+     * dev-iotlb) events can be transparently delivered to !MAP iommu
+     * notifiers.

So this means the UNMAP notifier doesn't need to be as accurate as MAP. (Should we document it in the notifier headers)?

For MAP[a, b] MAP[b, c] we can do a UNMAP[a. c].

include/hw/i386/intel_iommu.h struct VTDAddressSpace {

Quoted 13 lines
include/hw/i386/intel_iommu.h +13 -0
+     *
+     * The tree OTOH is required for MAP typed iommu notifiers for a few
+     * reasons.
+     *
+     * Firstly, there's no way to identify whether an PSI event is MAP or
+     * UNMAP within the PSI message itself.  Without having prior knowledge
+     * of existing state vIOMMU doesn't know whether it should notify MAP
+     * or UNMAP for a PSI message it received.
+     *
+     * Secondly, PSI received from guest driver (or even a large PSI can
+     * grow into a DSI at least with Linux intel-iommu driver) can be
+     * larger in range than the newly mapped ranges for either MAP or UNMAP
+     * events.

Yes, so I think we need a document that the UNMAP handler should be prepared for this.

Thanks

Quoted 16 lines

If it directly pass-throughs any such event it may confuse

include/hw/i386/intel_iommu.h +9 -1
+     * the registered drivers (e.g. vfio-pci) on either: (1) trying to map
+     * the same region more than once (for VFIO_IOMMU_MAP_DMA, -EEXIST will
+     * trigger), or (2) trying to UNMAP a range that is still partially
+     * mapped.  That accuracy is not required for UNMAP-only notifiers, but
+     * it is a must-to-have for MAP-inclusive notifiers, because the vIOMMU
+     * needs to make sure the shadow page table is always in sync with the
+     * guest IOMMU pgtables for a device.
+     */
+    IOVATree *iova_tree;
 };
 
 struct VTDIOTLBEntry {
--

2.37.3

author
#7 lore raw
Quoted 23 lines · Jason Wang
Quoted 22 lines · Peter Xu

It seems not super clear on when iova_tree is used, and why. Add a rich comment above iova_tree to track why we needed the iova_tree, and when we need it.

---
 include/hw/i386/intel_iommu.h | 30 +++++++++++++++++++++++++++++-
 1 file changed, 29 insertions(+), 1 deletion(-)
include/hw/i386/intel_iommu.h +2 -1
index 46d973e629..8d130ab2e3 100644
@@ -109,7 +109,35 @@ struct VTDAddressSpace {
     QLIST_ENTRY(VTDAddressSpace) next;
     /* Superset of notifier flags that this address space has */
     IOMMUNotifierFlag notifier_flags;
-    IOVATree *iova_tree;          /* Traces mapped IOVA ranges */
+    /*
+     * @iova_tree traces mapped IOVA ranges.
Quoted 8 lines

include/hw/i386/intel_iommu.h struct VTDAddressSpace { · line 118

Quoted 5 lines
include/hw/i386/intel_iommu.h +5 -0
+     *
+     * The tree is not needed if no MAP notifiers is registered with
+     * current VTD address space, because all UNMAP (including iotlb or
+     * dev-iotlb) events can be transparently delivered to !MAP iommu
+     * notifiers.

So this means the UNMAP notifier doesn't need to be as accurate as MAP. (Should we document it in the notifier headers)?

Yes.

Quoted 2 lines

For MAP[a, b] MAP[b, c] we can do a UNMAP[a. c].

IIUC a better way to say this is, for MAP[a, b] we can do an UNMAP[a-X, b+Y] as long as the range covers [a, b]?

Quoted 15 lines
Quoted 4 lines
include/hw/i386/intel_iommu.h +3 -0
+     *
+     * The tree OTOH is required for MAP typed iommu notifiers for a few
+     * reasons.

include/hw/i386/intel_iommu.h struct VTDAddressSpace { · line 131

Quoted 10 lines
include/hw/i386/intel_iommu.h +10 -0
+     *
+     * Firstly, there's no way to identify whether an PSI event is MAP or
+     * UNMAP within the PSI message itself.  Without having prior knowledge
+     * of existing state vIOMMU doesn't know whether it should notify MAP
+     * or UNMAP for a PSI message it received.
+     *
+     * Secondly, PSI received from guest driver (or even a large PSI can
+     * grow into a DSI at least with Linux intel-iommu driver) can be
+     * larger in range than the newly mapped ranges for either MAP or UNMAP
+     * events.
Quoted 2 lines

Yes, so I think we need a document that the UNMAP handler should be prepared for this.

How about I squash below into this same patch?

include/exec/memory.h +18 -0
index 91f8a2395a..c83bd11a68 100644
@@ -129,6 +129,24 @@ struct IOMMUTLBEntry {
 /*
  * Bitmap for different IOMMUNotifier capabilities. Each notifier can
  * register with one or multiple IOMMU Notifier capability bit(s).
+ *
+ * Normally there're two use cases for the notifiers:
+ *
+ *   (1) When the device needs accurate synchronizations of the vIOMMU page
+ *       tables, it needs to register with both MAP|UNMAP notifies (which
+ *       is defined as IOMMU_NOTIFIER_IOTLB_EVENTS below).  As long as MAP
+ *       events are registered, the notifications will be accurate but
+ *       there's overhead on synchronizing the guest vIOMMU page tables.
+ *
+ *   (2) When the device doesn't need accurate synchronizations of the
+ *       vIOMMU page tables (when the device can both cache translations
+ *       and requesting to translate dynamically during DMA process), it
+ *       needs to register only with UNMAP or DEVIOTLB_UNMAP notifies.
+ *       Note that in such working mode shadow page table is not used for
+ *       vIOMMU unit on this address space, so the UNMAP messages can be
+ *       actually larger than the real invalidations (just like how the
+ *       Linux IOMMU driver normally works, where an invalidation can be
+ *       enlarged as long as it still covers the target range).
  */
 typedef enum {
     IOMMU_NOTIFIER_NONE = 0,

Thanks,

Signature · 1 line
Peter Xu
#8 lore raw
Quoted 34 lines · Peter Xu
Quoted 23 lines · Jason Wang
Quoted 22 lines · Peter Xu

It seems not super clear on when iova_tree is used, and why. Add a rich comment above iova_tree to track why we needed the iova_tree, and when we need it.

---
 include/hw/i386/intel_iommu.h | 30 +++++++++++++++++++++++++++++-
 1 file changed, 29 insertions(+), 1 deletion(-)
include/hw/i386/intel_iommu.h +2 -1
index 46d973e629..8d130ab2e3 100644
@@ -109,7 +109,35 @@ struct VTDAddressSpace {
     QLIST_ENTRY(VTDAddressSpace) next;
     /* Superset of notifier flags that this address space has */
     IOMMUNotifierFlag notifier_flags;
-    IOVATree *iova_tree;          /* Traces mapped IOVA ranges */
+    /*
+     * @iova_tree traces mapped IOVA ranges.
Quoted 8 lines

include/hw/i386/intel_iommu.h struct VTDAddressSpace { · line 118

Quoted 5 lines
include/hw/i386/intel_iommu.h +5 -0
+     *
+     * The tree is not needed if no MAP notifiers is registered with
+     * current VTD address space, because all UNMAP (including iotlb or
+     * dev-iotlb) events can be transparently delivered to !MAP iommu
+     * notifiers.

So this means the UNMAP notifier doesn't need to be as accurate as MAP. (Should we document it in the notifier headers)?

Quoted 7 lines

Yes.

Quoted 2 lines

For MAP[a, b] MAP[b, c] we can do a UNMAP[a. c].

IIUC a better way to say this is, for MAP[a, b] we can do an UNMAP[a-X, b+Y] as long as the range covers [a, b]?

Right.

Quoted 16 lines
Quoted 16 lines
Quoted 15 lines
include/hw/i386/intel_iommu.h +13 -0
+     *
+     * The tree OTOH is required for MAP typed iommu notifiers for a few
+     * reasons.
+     *
+     * Firstly, there's no way to identify whether an PSI event is MAP or
+     * UNMAP within the PSI message itself.  Without having prior knowledge
+     * of existing state vIOMMU doesn't know whether it should notify MAP
+     * or UNMAP for a PSI message it received.
+     *
+     * Secondly, PSI received from guest driver (or even a large PSI can
+     * grow into a DSI at least with Linux intel-iommu driver) can be
+     * larger in range than the newly mapped ranges for either MAP or UNMAP
+     * events.
Quoted 4 lines
Quoted 2 lines

Yes, so I think we need a document that the UNMAP handler should be prepared for this.

How about I squash below into this same patch?

Looks good to me.

Thanks

Quoted 36 lines
include/exec/memory.h +18 -0
index 91f8a2395a..c83bd11a68 100644
@@ -129,6 +129,24 @@ struct IOMMUTLBEntry {
 /*
  * Bitmap for different IOMMUNotifier capabilities. Each notifier can
  * register with one or multiple IOMMU Notifier capability bit(s).
+ *
+ * Normally there're two use cases for the notifiers:
+ *
+ *   (1) When the device needs accurate synchronizations of the vIOMMU page
+ *       tables, it needs to register with both MAP|UNMAP notifies (which
+ *       is defined as IOMMU_NOTIFIER_IOTLB_EVENTS below).  As long as MAP
+ *       events are registered, the notifications will be accurate but
+ *       there's overhead on synchronizing the guest vIOMMU page tables.
+ *
+ *   (2) When the device doesn't need accurate synchronizations of the
+ *       vIOMMU page tables (when the device can both cache translations
+ *       and requesting to translate dynamically during DMA process), it
+ *       needs to register only with UNMAP or DEVIOTLB_UNMAP notifies.
+ *       Note that in such working mode shadow page table is not used for
+ *       vIOMMU unit on this address space, so the UNMAP messages can be
+ *       actually larger than the real invalidations (just like how the
+ *       Linux IOMMU driver normally works, where an invalidation can be
+ *       enlarged as long as it still covers the target range).
  */
 typedef enum {
     IOMMU_NOTIFIER_NONE = 0,

Thanks,

--
Peter Xu

#9 lore raw
In reply to Peter Xu

Hi Peter,

Quoted 58 lines · Peter Xu
Quoted 21 lines · Jason Wang
Quoted 20 lines · Peter Xu

It seems not super clear on when iova_tree is used, and why. Add a rich comment above iova_tree to track why we needed the iova_tree, and when we need it.

---
 include/hw/i386/intel_iommu.h | 30 +++++++++++++++++++++++++++++-
 1 file changed, 29 insertions(+), 1 deletion(-)
include/hw/i386/intel_iommu.h +1 -1
index 46d973e629..8d130ab2e3 100644
@@ -109,7 +109,35 @@ struct VTDAddressSpace {
     QLIST_ENTRY(VTDAddressSpace) next;
     /* Superset of notifier flags that this address space has */
     IOMMUNotifierFlag notifier_flags;
-    IOVATree *iova_tree;          /* Traces mapped IOVA ranges */
+    /*
Quoted 8 lines

include/hw/i386/intel_iommu.h struct VTDAddressSpace { · line 118

Quoted 6 lines
include/hw/i386/intel_iommu.h +6 -0
+     * @iova_tree traces mapped IOVA ranges.
+     *
+     * The tree is not needed if no MAP notifiers is registered with
+     * current VTD address space, because all UNMAP (including iotlb or
+     * dev-iotlb) events can be transparently delivered to !MAP iommu
+     * notifiers.

So this means the UNMAP notifier doesn't need to be as accurate as MAP. (Should we document it in the notifier headers)?

Yes.

Quoted 1 line

For MAP[a, b] MAP[b, c] we can do a UNMAP[a. c].

IIUC a better way to say this is, for MAP[a, b] we can do an UNMAP[a-X, b+Y] as long as the range covers [a, b]?

Quoted 13 lines

include/hw/i386/intel_iommu.h struct VTDAddressSpace { · line 131

Quoted 13 lines
include/hw/i386/intel_iommu.h +13 -0
+     *
+     * The tree OTOH is required for MAP typed iommu notifiers for a few
+     * reasons.
+     *
+     * Firstly, there's no way to identify whether an PSI event is MAP or
+     * UNMAP within the PSI message itself.  Without having prior knowledge
+     * of existing state vIOMMU doesn't know whether it should notify MAP
+     * or UNMAP for a PSI message it received.
+     *
+     * Secondly, PSI received from guest driver (or even a large PSI can
+     * grow into a DSI at least with Linux intel-iommu driver) can be
+     * larger in range than the newly mapped ranges for either MAP or UNMAP
+     * events.
Quoted 2 lines

Yes, so I think we need a document that the UNMAP handler should be prepared for this.

How about I squash below into this same patch?

include/exec/memory.h +0 -0
index 91f8a2395a..c83bd11a68 100644

include/exec/memory.h struct IOMMUTLBEntry { · line 135

Quoted 7 lines
include/exec/memory.h +4 -0
@@ -129,6 +129,24 @@ struct IOMMUTLBEntry {
 /*
  * Bitmap for different IOMMUNotifier capabilities. Each notifier can
  * register with one or multiple IOMMU Notifier capability bit(s).
+ *
+ * Normally there're two use cases for the notifiers:
+ *
+ *   (1) When the device needs accurate synchronizations of the vIOMMU page

accurate synchronizations sound too vague & subjective to me.

include/exec/memory.h struct IOMMUTLBEntry { · line 143

Quoted 8 lines
include/exec/memory.h +8 -0
+ *       tables, it needs to register with both MAP|UNMAP notifies (which
+ *       is defined as IOMMU_NOTIFIER_IOTLB_EVENTS below).  As long as MAP
+ *       events are registered, the notifications will be accurate but
+ *       there's overhead on synchronizing the guest vIOMMU page tables.
+ *
+ *   (2) When the device doesn't need accurate synchronizations of the
+ *       vIOMMU page tables (when the device can both cache translations
+ *       and requesting to translate dynamically during DMA process), it

s/requesting/request

include/exec/memory.h struct IOMMUTLBEntry { · line 144

Quoted 1 line
include/exec/memory.h +1 -0
+ *       needs to register only with UNMAP or DEVIOTLB_UNMAP notifies.

would be nice to clarify the distinction between both then

include/exec/memory.h struct IOMMUTLBEntry { · line 146

Quoted 2 lines
include/exec/memory.h +2 -0
+ *       Note that in such working mode shadow page table is not used for
+ *       vIOMMU unit on this address space, so the UNMAP messages can be

I do not catch 'is not used for vIOMMU unit on this address space'

Quoted 9 lines
include/exec/memory.h +3 -0
+ *       actually larger than the real invalidations (just like how the
+ *       Linux IOMMU driver normally works, where an invalidation can be
+ *       enlarged as long as it still covers the target range).
  */
 typedef enum {
     IOMMU_NOTIFIER_NONE = 0,

Thanks,

Thanks

Eric

author
#10 lore raw
Quoted 55 lines · Eric Auger

Hi Peter,

Quoted 53 lines · Peter Xu
Quoted 21 lines · Jason Wang
Quoted 20 lines · Peter Xu

It seems not super clear on when iova_tree is used, and why. Add a rich comment above iova_tree to track why we needed the iova_tree, and when we need it.

---
 include/hw/i386/intel_iommu.h | 30 +++++++++++++++++++++++++++++-
 1 file changed, 29 insertions(+), 1 deletion(-)
include/hw/i386/intel_iommu.h +1 -1
index 46d973e629..8d130ab2e3 100644
@@ -109,7 +109,35 @@ struct VTDAddressSpace {
     QLIST_ENTRY(VTDAddressSpace) next;
     /* Superset of notifier flags that this address space has */
     IOMMUNotifierFlag notifier_flags;
-    IOVATree *iova_tree;          /* Traces mapped IOVA ranges */
+    /*
Quoted 8 lines

include/hw/i386/intel_iommu.h struct VTDAddressSpace { · line 118

Quoted 6 lines
include/hw/i386/intel_iommu.h +6 -0
+     * @iova_tree traces mapped IOVA ranges.
+     *
+     * The tree is not needed if no MAP notifiers is registered with
+     * current VTD address space, because all UNMAP (including iotlb or
+     * dev-iotlb) events can be transparently delivered to !MAP iommu
+     * notifiers.

So this means the UNMAP notifier doesn't need to be as accurate as MAP. (Should we document it in the notifier headers)?

Yes.

Quoted 1 line

For MAP[a, b] MAP[b, c] we can do a UNMAP[a. c].

IIUC a better way to say this is, for MAP[a, b] we can do an UNMAP[a-X, b+Y] as long as the range covers [a, b]?

Quoted 13 lines

include/hw/i386/intel_iommu.h struct VTDAddressSpace { · line 131

Quoted 13 lines
include/hw/i386/intel_iommu.h +13 -0
+     *
+     * The tree OTOH is required for MAP typed iommu notifiers for a few
+     * reasons.
+     *
+     * Firstly, there's no way to identify whether an PSI event is MAP or
+     * UNMAP within the PSI message itself.  Without having prior knowledge
+     * of existing state vIOMMU doesn't know whether it should notify MAP
+     * or UNMAP for a PSI message it received.
+     *
+     * Secondly, PSI received from guest driver (or even a large PSI can
+     * grow into a DSI at least with Linux intel-iommu driver) can be
+     * larger in range than the newly mapped ranges for either MAP or UNMAP
+     * events.
Quoted 2 lines

Yes, so I think we need a document that the UNMAP handler should be prepared for this.

How about I squash below into this same patch?

Quoted 13 lines

include/exec/memory.h struct IOMMUTLBEntry { · line 135

Quoted 12 lines
include/exec/memory.h +4 -0
index 91f8a2395a..c83bd11a68 100644
@@ -129,6 +129,24 @@ struct IOMMUTLBEntry {
 /*
  * Bitmap for different IOMMUNotifier capabilities. Each notifier can
  * register with one or multiple IOMMU Notifier capability bit(s).
+ *
+ * Normally there're two use cases for the notifiers:
+ *
+ *   (1) When the device needs accurate synchronizations of the vIOMMU page

accurate synchronizations sound too vague & subjective to me.

Suggestions?

Quoted 8 lines

include/exec/memory.h struct IOMMUTLBEntry { · line 143

Quoted 8 lines
include/exec/memory.h +8 -0
+ *       tables, it needs to register with both MAP|UNMAP notifies (which
+ *       is defined as IOMMU_NOTIFIER_IOTLB_EVENTS below).  As long as MAP
+ *       events are registered, the notifications will be accurate but
+ *       there's overhead on synchronizing the guest vIOMMU page tables.
+ *
+ *   (2) When the device doesn't need accurate synchronizations of the
+ *       vIOMMU page tables (when the device can both cache translations
+ *       and requesting to translate dynamically during DMA process), it
Quoted 6 lines

s/requesting/request

include/exec/memory.h struct IOMMUTLBEntry { · line 144

Quoted 1 line
include/exec/memory.h +1 -0
+ *       needs to register only with UNMAP or DEVIOTLB_UNMAP notifies.

would be nice to clarify the distinction between both then

include/exec/memory.h struct IOMMUTLBEntry { · line 146

Quoted 2 lines
include/exec/memory.h +2 -0
+ *       Note that in such working mode shadow page table is not used for
+ *       vIOMMU unit on this address space, so the UNMAP messages can be

I do not catch 'is not used for vIOMMU unit on this address space'

How about: "Note that in this working mode the vIOMMU will not maintain a shadowed page table for the address space, and the UNMAP messages can be.."?

Quoted 13 lines
Quoted 9 lines
include/exec/memory.h +3 -0
+ *       actually larger than the real invalidations (just like how the
+ *       Linux IOMMU driver normally works, where an invalidation can be
+ *       enlarged as long as it still covers the target range).
  */
 typedef enum {
     IOMMU_NOTIFIER_NONE = 0,

Thanks,

Thanks

Eric

Signature · 1 line
Peter Xu
#11 lore raw
Quoted 69 lines · Peter Xu
Quoted 55 lines · Eric Auger

Hi Peter,

Quoted 53 lines · Peter Xu
Quoted 21 lines · Jason Wang
Quoted 20 lines · Peter Xu

It seems not super clear on when iova_tree is used, and why. Add a rich comment above iova_tree to track why we needed the iova_tree, and when we need it.

---
 include/hw/i386/intel_iommu.h | 30 +++++++++++++++++++++++++++++-
 1 file changed, 29 insertions(+), 1 deletion(-)
include/hw/i386/intel_iommu.h +1 -1
index 46d973e629..8d130ab2e3 100644
@@ -109,7 +109,35 @@ struct VTDAddressSpace {
     QLIST_ENTRY(VTDAddressSpace) next;
     /* Superset of notifier flags that this address space has */
     IOMMUNotifierFlag notifier_flags;
-    IOVATree *iova_tree;          /* Traces mapped IOVA ranges */
+    /*
Quoted 8 lines

include/hw/i386/intel_iommu.h struct VTDAddressSpace { · line 118

Quoted 6 lines
include/hw/i386/intel_iommu.h +6 -0
+     * @iova_tree traces mapped IOVA ranges.
+     *
+     * The tree is not needed if no MAP notifiers is registered with
+     * current VTD address space, because all UNMAP (including iotlb or
+     * dev-iotlb) events can be transparently delivered to !MAP iommu
+     * notifiers.

So this means the UNMAP notifier doesn't need to be as accurate as MAP. (Should we document it in the notifier headers)?

Yes.

Quoted 1 line

For MAP[a, b] MAP[b, c] we can do a UNMAP[a. c].

IIUC a better way to say this is, for MAP[a, b] we can do an UNMAP[a-X, b+Y] as long as the range covers [a, b]?

Quoted 13 lines

include/hw/i386/intel_iommu.h struct VTDAddressSpace { · line 131

Quoted 13 lines
include/hw/i386/intel_iommu.h +13 -0
+     *
+     * The tree OTOH is required for MAP typed iommu notifiers for a few
+     * reasons.
+     *
+     * Firstly, there's no way to identify whether an PSI event is MAP or
+     * UNMAP within the PSI message itself.  Without having prior knowledge
+     * of existing state vIOMMU doesn't know whether it should notify MAP
+     * or UNMAP for a PSI message it received.
+     *
+     * Secondly, PSI received from guest driver (or even a large PSI can
+     * grow into a DSI at least with Linux intel-iommu driver) can be
+     * larger in range than the newly mapped ranges for either MAP or UNMAP
+     * events.
Quoted 2 lines

Yes, so I think we need a document that the UNMAP handler should be prepared for this.

How about I squash below into this same patch?

Quoted 13 lines

include/exec/memory.h struct IOMMUTLBEntry { · line 135

Quoted 12 lines
include/exec/memory.h +4 -0
index 91f8a2395a..c83bd11a68 100644
@@ -129,6 +129,24 @@ struct IOMMUTLBEntry {
 /*
  * Bitmap for different IOMMUNotifier capabilities. Each notifier can
  * register with one or multiple IOMMU Notifier capability bit(s).
+ *
+ * Normally there're two use cases for the notifiers:
+ *
+ *   (1) When the device needs accurate synchronizations of the vIOMMU page

accurate synchronizations sound too vague & subjective to me.

Quoted 1 line

Suggestions?

Well I would say:
when the notified device maintains a shadow page table and must to be notified on each guest MAP (page table entry creation) and UNMAP (invalidation) events (VFIO). Both notifications must be accurate so that the shadow page table is fully in sync with the guest view.

Quoted 9 lines
Quoted 9 lines

include/exec/memory.h struct IOMMUTLBEntry { · line 143

Quoted 9 lines
include/exec/memory.h +8 -0
+ *       tables, it needs to register with both MAP|UNMAP notifies (which
+ *       is defined as IOMMU_NOTIFIER_IOTLB_EVENTS below).  As long as MAP
+ *       events are registered, the notifications will be accurate but
+ *       there's overhead on synchronizing the guest vIOMMU page tables.
+ *
+ *   (2) When the device doesn't need accurate synchronizations of the
+ *       vIOMMU page tables (when the device can both cache translations
+ *       and requesting to translate dynamically during DMA process), it

when the notified device maintains a cache of IOMMU translations (IOTLB) and is able to fill that cache by requesting translations from the vIOMMU through a protocol similar to ATS. In that case the notified device only needs to register an UNMAP notifier. In that case the unmap notifications are allower to be wider than the strict necessary.

However the problem is since you need to satisfy the VFIO use case, how do you detect when you are allowed to invalidate more that the strict necessary?

Eric

Quoted 22 lines
Quoted 6 lines

s/requesting/request

include/exec/memory.h struct IOMMUTLBEntry { · line 144

Quoted 1 line
include/exec/memory.h +1 -0
+ *       needs to register only with UNMAP or DEVIOTLB_UNMAP notifies.

would be nice to clarify the distinction between both then

include/exec/memory.h struct IOMMUTLBEntry { · line 146

Quoted 2 lines
include/exec/memory.h +2 -0
+ *       Note that in such working mode shadow page table is not used for
+ *       vIOMMU unit on this address space, so the UNMAP messages can be

I do not catch 'is not used for vIOMMU unit on this address space'

How about: "Note that in this working mode the vIOMMU will not maintain a shadowed page table for the address space, and the UNMAP messages can be.."?

Quoted 13 lines
Quoted 9 lines
include/exec/memory.h +3 -0
+ *       actually larger than the real invalidations (just like how the
+ *       Linux IOMMU driver normally works, where an invalidation can be
+ *       enlarged as long as it still covers the target range).
  */
 typedef enum {
     IOMMU_NOTIFIER_NONE = 0,

Thanks,

Thanks

Eric

author
#12 lore raw
Quoted 72 lines · Eric Auger
Quoted 69 lines · Peter Xu
Quoted 68 lines · Eric Auger

Hi Peter,

Quoted 58 lines · Peter Xu
Quoted 21 lines · Jason Wang
Quoted 20 lines · Peter Xu

It seems not super clear on when iova_tree is used, and why. Add a rich comment above iova_tree to track why we needed the iova_tree, and when we need it.

---
 include/hw/i386/intel_iommu.h | 30 +++++++++++++++++++++++++++++-
 1 file changed, 29 insertions(+), 1 deletion(-)
include/hw/i386/intel_iommu.h +1 -1
index 46d973e629..8d130ab2e3 100644
@@ -109,7 +109,35 @@ struct VTDAddressSpace {
     QLIST_ENTRY(VTDAddressSpace) next;
     /* Superset of notifier flags that this address space has */
     IOMMUNotifierFlag notifier_flags;
-    IOVATree *iova_tree;          /* Traces mapped IOVA ranges */
+    /*
Quoted 8 lines

include/hw/i386/intel_iommu.h struct VTDAddressSpace { · line 118

Quoted 6 lines
include/hw/i386/intel_iommu.h +6 -0
+     * @iova_tree traces mapped IOVA ranges.
+     *
+     * The tree is not needed if no MAP notifiers is registered with
+     * current VTD address space, because all UNMAP (including iotlb or
+     * dev-iotlb) events can be transparently delivered to !MAP iommu
+     * notifiers.

So this means the UNMAP notifier doesn't need to be as accurate as MAP. (Should we document it in the notifier headers)?

Yes.

Quoted 1 line

For MAP[a, b] MAP[b, c] we can do a UNMAP[a. c].

IIUC a better way to say this is, for MAP[a, b] we can do an UNMAP[a-X, b+Y] as long as the range covers [a, b]?

Quoted 13 lines

include/hw/i386/intel_iommu.h struct VTDAddressSpace { · line 131

Quoted 13 lines
include/hw/i386/intel_iommu.h +13 -0
+     *
+     * The tree OTOH is required for MAP typed iommu notifiers for a few
+     * reasons.
+     *
+     * Firstly, there's no way to identify whether an PSI event is MAP or
+     * UNMAP within the PSI message itself.  Without having prior knowledge
+     * of existing state vIOMMU doesn't know whether it should notify MAP
+     * or UNMAP for a PSI message it received.
+     *
+     * Secondly, PSI received from guest driver (or even a large PSI can
+     * grow into a DSI at least with Linux intel-iommu driver) can be
+     * larger in range than the newly mapped ranges for either MAP or UNMAP
+     * events.
Quoted 2 lines

Yes, so I think we need a document that the UNMAP handler should be prepared for this.

How about I squash below into this same patch?

include/exec/memory.h +0 -0
index 91f8a2395a..c83bd11a68 100644

include/exec/memory.h struct IOMMUTLBEntry { · line 135

Quoted 7 lines
include/exec/memory.h +4 -0
@@ -129,6 +129,24 @@ struct IOMMUTLBEntry {
 /*
  * Bitmap for different IOMMUNotifier capabilities. Each notifier can
  * register with one or multiple IOMMU Notifier capability bit(s).
+ *
+ * Normally there're two use cases for the notifiers:
+ *
+ *   (1) When the device needs accurate synchronizations of the vIOMMU page

accurate synchronizations sound too vague & subjective to me.

Quoted 3 lines
Quoted 1 line

Suggestions?

Well I would say:
when the notified device maintains a shadow page table and must to be

s/to//

Quoted 3 lines

notified on each guest MAP (page table entry creation) and UNMAP (invalidation) events (VFIO). Both notifications must be accurate so that the shadow page table is fully in sync with the guest view.

Thanks, I'll try to squash this into the new version.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

include/exec/memory.h struct IOMMUTLBEntry { · line 143

Quoted 9 lines
include/exec/memory.h +8 -0
+ *       tables, it needs to register with both MAP|UNMAP notifies (which
+ *       is defined as IOMMU_NOTIFIER_IOTLB_EVENTS below).  As long as MAP
+ *       events are registered, the notifications will be accurate but
+ *       there's overhead on synchronizing the guest vIOMMU page tables.
+ *
+ *   (2) When the device doesn't need accurate synchronizations of the
+ *       vIOMMU page tables (when the device can both cache translations
+ *       and requesting to translate dynamically during DMA process), it
Quoted 5 lines

when the notified device maintains a cache of IOMMU translations (IOTLB) and is able to fill that cache by requesting translations from the vIOMMU through a protocol similar to ATS. In that case the notified device only needs to register an UNMAP notifier. In that case the unmap notifications are allower to be wider than the strict necessary.

Same here.

Quoted 4 lines

However the problem is since you need to satisfy the VFIO use case, how do you detect when you are allowed to invalidate more that the strict necessary?

We detect that by checking whether the vtd_as has map notifier registered. Please feel free to have a look at all sites of vtd_as_has_map_notifier(). We maintain the iova tree only for MAP case currently.

Signature · 1 line
Peter Xu