Skip to contentSkip to conversation

docs: Add '-device intel-iommu' entry

PATCH Superseded Peter Xu 4 participants 8 messages Jun 11, 2021 – Jul 8, 2021

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Peter Xu

author
#0 lore raw

PATCH 1/1 docs: Add '-device intel-iommu' entry

The parameters of intel-iommu device are non-trivial to understand. Add an entry for it so that people can reference to it when using.

There're actually a few more options there, but I hide them explicitly because they shouldn't be used by normal QEMU users.

Cc Chao Yang <[email protected]>
Cc Lei Yang <[email protected]>
Cc Jing Zhao <[email protected]>
Cc Jason Wang <[email protected]>
Cc Michael S. Tsirkin <[email protected]>
Cc Alex Williamson <[email protected]>
---
 qemu-options.hx | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)
qemu-options.hx +32 -0
index 14258784b3a..4bb04243907 100644
@@ -926,6 +926,38 @@ SRST
 
 ``-device pci-ipmi-bt,bmc=id``
     Like the KCS interface, but defines a BT interface on the PCI bus.
+
+``-device intel-iommu[,option=...]``
+    This is only supported by ``-machine q35``, which will enable Intel VT-d
+    emulation within the guest.  It supports below options:
+
+    ``intremap=on|off`` (default: auto)
+        This enables interrupt remapping feature in the guest.  It's required
+        to enable complete x2apic.  Currently it only supports kvm
+        kernel-irqchip modes ``off`` or ``split``.  Full kernel-irqchip is not
+        yet supported.
+
+    ``caching-mode=on|off`` (default: off)
+        This enables caching mode for the VT-d emulated device.  When
+        caching-mode is enabled, each guest DMA buffer mapping will generate an
+        IOTLB invalidation from the guest IOMMU driver to the vIOMMU device in
+        a synchronous way.  It is required for ``-device vfio-pci`` to work
+        with the VT-d device, because host assigned devices requires to setup
+        the DMA mapping on the host before guest DMA starts.
+
+    ``device-iotlb=on|off`` (default: off)
+        This enables device-iotlb capability for the emulated VT-d device.  So
+        far virtio/vhost should be the only real user for this parameter,
+        paired with ats=on configured for the device.
+
+    ``aw-bits=39|48`` (default: 39)
+        This decides the address width of IOVA address space.  The address
+        space has 39 bits width for 3-level IOMMU page tables, and 48 bits for
+        4-level IOMMU page tables.
+
+    Please also refer to the wiki page for general scenarios of VT-d
+    emulation in QEMU: https://wiki.qemu.org/Features/VT-d.
+
 ERST
 
 DEF("name", HAS_ARG, QEMU_OPTION_name,
Signature · 1 line
2.31.1
author
#1 lore raw
Quoted 13 lines · Peter Xu

The parameters of intel-iommu device are non-trivial to understand. Add an entry for it so that people can reference to it when using.

There're actually a few more options there, but I hide them explicitly because they shouldn't be used by normal QEMU users.

Cc Chao Yang <[email protected]>
Cc Lei Yang <[email protected]>
Cc Jing Zhao <[email protected]>
Cc Jason Wang <[email protected]>
Cc Michael S. Tsirkin <[email protected]>
Cc Alex Williamson <[email protected]>
Quoted 3 lines
---
 qemu-options.hx | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)

Ping..

Signature · 1 line
Peter Xu
Reviewed-by on v1 1/1
#2 lore raw
In reply to Peter Xu
Quoted 22 lines · Peter Xu

The parameters of intel-iommu device are non-trivial to understand. Add an entry for it so that people can reference to it when using.

There're actually a few more options there, but I hide them explicitly because they shouldn't be used by normal QEMU users.

Cc Chao Yang <[email protected]>
Cc Lei Yang <[email protected]>
Cc Jing Zhao <[email protected]>
Cc Jason Wang <[email protected]>
Cc Michael S. Tsirkin <[email protected]>
Cc Alex Williamson <[email protected]>
---
  qemu-options.hx | 32 ++++++++++++++++++++++++++++++++
  1 file changed, 32 insertions(+)
qemu-options.hx +0 -0
index 14258784b3a..4bb04243907 100644

qemu-options.hx SRST · line 935

Quoted 10 lines
qemu-options.hx +7 -0
@@ -926,6 +926,38 @@ SRST
 
  ``-device pci-ipmi-bt,bmc=id``
      Like the KCS interface, but defines a BT interface on the PCI bus.
+
+``-device intel-iommu[,option=...]``
+    This is only supported by ``-machine q35``, which will enable Intel VT-d
+    emulation within the guest.  It supports below options:
+
+    ``intremap=on|off`` (default: auto)
+        This enables interrupt remapping feature in the guest.

I think we don't need "in the guest" here.

Other than this:

Thanks

Quoted 29 lines
qemu-options.hx +25 -0
  It's required
+        to enable complete x2apic.  Currently it only supports kvm
+        kernel-irqchip modes ``off`` or ``split``.  Full kernel-irqchip is not
+        yet supported.
+
+    ``caching-mode=on|off`` (default: off)
+        This enables caching mode for the VT-d emulated device.  When
+        caching-mode is enabled, each guest DMA buffer mapping will generate an
+        IOTLB invalidation from the guest IOMMU driver to the vIOMMU device in
+        a synchronous way.  It is required for ``-device vfio-pci`` to work
+        with the VT-d device, because host assigned devices requires to setup
+        the DMA mapping on the host before guest DMA starts.
+
+    ``device-iotlb=on|off`` (default: off)
+        This enables device-iotlb capability for the emulated VT-d device.  So
+        far virtio/vhost should be the only real user for this parameter,
+        paired with ats=on configured for the device.
+
+    ``aw-bits=39|48`` (default: 39)
+        This decides the address width of IOVA address space.  The address
+        space has 39 bits width for 3-level IOMMU page tables, and 48 bits for
+        4-level IOMMU page tables.
+
+    Please also refer to the wiki page for general scenarios of VT-d
+    emulation in QEMU: https://wiki.qemu.org/Features/VT-d.
+
  ERST
 

DEF("name", HAS_ARG, QEMU_OPTION_name,

author
#3 lore raw
Quoted 28 lines · Jason Wang
Quoted 26 lines · Peter Xu

The parameters of intel-iommu device are non-trivial to understand. Add an entry for it so that people can reference to it when using.

There're actually a few more options there, but I hide them explicitly because they shouldn't be used by normal QEMU users.

Cc Chao Yang <[email protected]>
Cc Lei Yang <[email protected]>
Cc Jing Zhao <[email protected]>
Cc Jason Wang <[email protected]>
Cc Michael S. Tsirkin <[email protected]>
Cc Alex Williamson <[email protected]>
---
  qemu-options.hx | 32 ++++++++++++++++++++++++++++++++
  1 file changed, 32 insertions(+)
qemu-options.hx +2 -0
index 14258784b3a..4bb04243907 100644
@@ -926,6 +926,38 @@ SRST
  ``-device pci-ipmi-bt,bmc=id``
      Like the KCS interface, but defines a BT interface on the PCI bus.
+
+``-device intel-iommu[,option=...]``
Quoted 8 lines

qemu-options.hx SRST

Quoted 5 lines
qemu-options.hx +5 -0
+    This is only supported by ``-machine q35``, which will enable Intel VT-d
+    emulation within the guest.  It supports below options:
+
+    ``intremap=on|off`` (default: auto)
+        This enables interrupt remapping feature in the guest.

I think we don't need "in the guest" here.

Sure.

Quoted 4 lines

Other than this:

Thanks,

Signature · 1 line
Peter Xu
#4 lore raw
In reply to Peter Xu

Hi Peter,

Quoted 25 lines · Peter Xu

The parameters of intel-iommu device are non-trivial to understand. Add an entry for it so that people can reference to it when using.

There're actually a few more options there, but I hide them explicitly because they shouldn't be used by normal QEMU users.

Cc Chao Yang <[email protected]>
Cc Lei Yang <[email protected]>
Cc Jing Zhao <[email protected]>
Cc Jason Wang <[email protected]>
Cc Michael S. Tsirkin <[email protected]>
Cc Alex Williamson <[email protected]>
---
 qemu-options.hx | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)
qemu-options.hx +0 -0
index 14258784b3a..4bb04243907 100644
@@ -926,6 +926,38 @@ SRST
 
 ``-device pci-ipmi-bt,bmc=id``
     Like the KCS interface, but defines a BT interface on the PCI bus.

qemu-options.hx SRST · line 938

Quoted 10 lines
qemu-options.hx +10 -0
+
+``-device intel-iommu[,option=...]``
+    This is only supported by ``-machine q35``, which will enable Intel VT-d
+    emulation within the guest.  It supports below options:
+
+    ``intremap=on|off`` (default: auto)
+        This enables interrupt remapping feature in the guest.  It's required
+        to enable complete x2apic.  Currently it only supports kvm
+        kernel-irqchip modes ``off`` or ``split``.  Full kernel-irqchip is not
+        yet supported.

maybe explain how the default is chosen (based on kernel-irqhcip mode?)

qemu-options.hx SRST · line 950

Quoted 12 lines
qemu-options.hx +12 -0
+
+    ``caching-mode=on|off`` (default: off)
+        This enables caching mode for the VT-d emulated device.  When
+        caching-mode is enabled, each guest DMA buffer mapping will generate an
+        IOTLB invalidation from the guest IOMMU driver to the vIOMMU device in
+        a synchronous way.  It is required for ``-device vfio-pci`` to work
+        with the VT-d device, because host assigned devices requires to setup
+        the DMA mapping on the host before guest DMA starts.
+
+    ``device-iotlb=on|off`` (default: off)
+        This enables device-iotlb capability for the emulated VT-d device.  So
+        far virtio/vhost should be the only real user for this parameter,

I thought if was needed for vhost. What is the usage for virtio?

qemu-options.hx SRST · line 951

Quoted 1 line
qemu-options.hx +1 -0
+        paired with ats=on configured for the device.

What is not obvious to me is whether there is a kind of diagnosis whether device-iotlb and caching mode settings are consistent with vfio/vhost usage. Does qemu exit if there is inconsistency?

qemu-options.hx SRST · line 963

Quoted 13 lines
qemu-options.hx +9 -0
+
+    ``aw-bits=39|48`` (default: 39)
+        This decides the address width of IOVA address space.  The address
+        space has 39 bits width for 3-level IOMMU page tables, and 48 bits for
+        4-level IOMMU page tables.
+
+    Please also refer to the wiki page for general scenarios of VT-d
+    emulation in QEMU: https://wiki.qemu.org/Features/VT-d.
+
 ERST
 
 DEF("name", HAS_ARG, QEMU_OPTION_name,

Besides, it is quite useful.

Thanks

Eric

author
#5 lore raw
Quoted 1 line · Eric Auger

Hi Peter,

Hi, Eric,

Quoted 25 lines · Peter Xu
Quoted 23 lines

The parameters of intel-iommu device are non-trivial to understand. Add an entry for it so that people can reference to it when using.

There're actually a few more options there, but I hide them explicitly because they shouldn't be used by normal QEMU users.

Cc Chao Yang <[email protected]>
Cc Lei Yang <[email protected]>
Cc Jing Zhao <[email protected]>
Cc Jason Wang <[email protected]>
Cc Michael S. Tsirkin <[email protected]>
Cc Alex Williamson <[email protected]>
---
 qemu-options.hx | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)
qemu-options.hx +0 -0
index 14258784b3a..4bb04243907 100644
@@ -926,6 +926,38 @@ SRST
 
Quoted 13 lines

qemu-options.hx SRST · line 938

Quoted 12 lines
qemu-options.hx +10 -0
 ``-device pci-ipmi-bt,bmc=id``
     Like the KCS interface, but defines a BT interface on the PCI bus.
+
+``-device intel-iommu[,option=...]``
+    This is only supported by ``-machine q35``, which will enable Intel VT-d
+    emulation within the guest.  It supports below options:
+
+    ``intremap=on|off`` (default: auto)
+        This enables interrupt remapping feature in the guest.  It's required
+        to enable complete x2apic.  Currently it only supports kvm
+        kernel-irqchip modes ``off`` or ``split``.  Full kernel-irqchip is not
+        yet supported.

maybe explain how the default is chosen (based on kernel-irqhcip mode?)

Will do.

Quoted 5 lines
Quoted 5 lines
qemu-options.hx +5 -0
+
+    ``caching-mode=on|off`` (default: off)
+        This enables caching mode for the VT-d emulated device.  When
+        caching-mode is enabled, each guest DMA buffer mapping will generate an
+        IOTLB invalidation from the guest IOMMU driver to the vIOMMU device in
Quoted 8 lines

qemu-options.hx SRST · line 950

Quoted 7 lines
qemu-options.hx +7 -0
+        a synchronous way.  It is required for ``-device vfio-pci`` to work
+        with the VT-d device, because host assigned devices requires to setup
+        the DMA mapping on the host before guest DMA starts.
+
+    ``device-iotlb=on|off`` (default: off)
+        This enables device-iotlb capability for the emulated VT-d device.  So
+        far virtio/vhost should be the only real user for this parameter,

I thought if was needed for vhost. What is the usage for virtio?

I used "virtio/vhost" because vhost is only one type of virtio backend, so normally they should be used in pair. Also, the "ats=on" parameter should be an option to virtio frontend rather than vhost, hence I mentioned both.

Quoted 4 lines

qemu-options.hx SRST · line 951

Quoted 1 line
qemu-options.hx +1 -0
+        paired with ats=on configured for the device.

What is not obvious to me is whether there is a kind of diagnosis whether device-iotlb and caching mode settings are consistent with vfio/vhost usage. Does qemu exit if there is inconsistency?

Caching mode check is done in vtd_machine_done_notify_one().

I think there's no such check for vhost, however iirc vhost should work with/without ats=on; I did't check deeper.

Quoted 11 lines
Quoted 11 lines
qemu-options.hx +9 -0
+
+    ``aw-bits=39|48`` (default: 39)
+        This decides the address width of IOVA address space.  The address
+        space has 39 bits width for 3-level IOMMU page tables, and 48 bits for
+        4-level IOMMU page tables.
+
+    Please also refer to the wiki page for general scenarios of VT-d
+    emulation in QEMU: https://wiki.qemu.org/Features/VT-d.
+
 ERST
 
Quoted 3 lines

qemu-options.hx SRST · line 963

Quoted 2 lines
qemu-options.hx +0 -0
 DEF("name", HAS_ARG, QEMU_OPTION_name,

Besides, it is quite useful.

Thanks for taking a look!

Signature · 1 line
Peter Xu
Reviewed-by
#6 lore raw

Hi Peter,

Quoted 60 lines · Peter Xu
Quoted 1 line · Eric Auger

Hi Peter,

Hi, Eric,

Quoted 25 lines · Peter Xu
Quoted 23 lines

The parameters of intel-iommu device are non-trivial to understand. Add an entry for it so that people can reference to it when using.

There're actually a few more options there, but I hide them explicitly because they shouldn't be used by normal QEMU users.

Cc Chao Yang <[email protected]>
Cc Lei Yang <[email protected]>
Cc Jing Zhao <[email protected]>
Cc Jason Wang <[email protected]>
Cc Michael S. Tsirkin <[email protected]>
Cc Alex Williamson <[email protected]>
---
 qemu-options.hx | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)
qemu-options.hx +0 -0
index 14258784b3a..4bb04243907 100644
@@ -926,6 +926,38 @@ SRST
 
Quoted 13 lines

qemu-options.hx SRST · line 938

Quoted 12 lines
qemu-options.hx +10 -0
 ``-device pci-ipmi-bt,bmc=id``
     Like the KCS interface, but defines a BT interface on the PCI bus.
+
+``-device intel-iommu[,option=...]``
+    This is only supported by ``-machine q35``, which will enable Intel VT-d
+    emulation within the guest.  It supports below options:
+
+    ``intremap=on|off`` (default: auto)
+        This enables interrupt remapping feature in the guest.  It's required
+        to enable complete x2apic.  Currently it only supports kvm
+        kernel-irqchip modes ``off`` or ``split``.  Full kernel-irqchip is not
+        yet supported.

maybe explain how the default is chosen (based on kernel-irqhcip mode?)

Will do.

Quoted 5 lines
Quoted 5 lines
qemu-options.hx +5 -0
+
+    ``caching-mode=on|off`` (default: off)
+        This enables caching mode for the VT-d emulated device.  When
+        caching-mode is enabled, each guest DMA buffer mapping will generate an
+        IOTLB invalidation from the guest IOMMU driver to the vIOMMU device in
Quoted 8 lines

qemu-options.hx SRST · line 950

Quoted 7 lines
qemu-options.hx +7 -0
+        a synchronous way.  It is required for ``-device vfio-pci`` to work
+        with the VT-d device, because host assigned devices requires to setup
+        the DMA mapping on the host before guest DMA starts.
+
+    ``device-iotlb=on|off`` (default: off)
+        This enables device-iotlb capability for the emulated VT-d device.  So
+        far virtio/vhost should be the only real user for this parameter,

I thought if was needed for vhost. What is the usage for virtio?

Quoted 3 lines

I used "virtio/vhost" because vhost is only one type of virtio backend, so normally they should be used in pair. Also, the "ats=on" parameter should be an option to virtio frontend rather than vhost, hence I mentioned both.

OK makes sense.

Quoted 10 lines
Quoted 5 lines

qemu-options.hx SRST · line 951

Quoted 2 lines
qemu-options.hx +1 -0
+        paired with ats=on configured for the device.

What is not obvious to me is whether there is a kind of diagnosis whether device-iotlb and caching mode settings are consistent with vfio/vhost usage. Does qemu exit if there is inconsistency?

Caching mode check is done in vtd_machine_done_notify_one().

I think there's no such check for vhost, however iirc vhost should work with/without ats=on; I did't check deeper.

OK

Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
qemu-options.hx +9 -0
+
+    ``aw-bits=39|48`` (default: 39)
+        This decides the address width of IOVA address space.  The address
+        space has 39 bits width for 3-level IOMMU page tables, and 48 bits for
+        4-level IOMMU page tables.
+
+    Please also refer to the wiki page for general scenarios of VT-d
+    emulation in QEMU: https://wiki.qemu.org/Features/VT-d.
+
 ERST
 
Quoted 6 lines
Quoted 3 lines

qemu-options.hx SRST · line 963

Quoted 2 lines
qemu-options.hx +0 -0
 DEF("name", HAS_ARG, QEMU_OPTION_name,

Besides, it is quite useful.

Thanks for taking a look!

Besides,

Eric

Reviewed-by on v1 1/1
#7 lore raw
In reply to Peter Xu
Quoted 10 lines

From: Peter Xu < [email protected] >
Sent: Saturday, June 12, 2021 2:55 AM

The parameters of intel-iommu device are non-trivial to understand. Add an
entry for it so that people can reference to it when using.

There're actually a few more options there, but I hide them explicitly because
they shouldn't be used by normal QEMU users.

yes, it's a good start.

Looks good to me.

Regards,
Yi Liu

Quoted 60 lines
Cc Chao Yang <[email protected]>
Cc Lei Yang <[email protected]>
Cc Jing Zhao <[email protected]>
Cc Jason Wang <[email protected]>
Cc Michael S. Tsirkin <[email protected]>
Cc Alex Williamson <[email protected]>
---
 qemu-options.hx | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)
qemu-options.hx +14 -0
index 14258784b3a..4bb04243907 100644
@@ -926,6 +926,38 @@ SRST
 
 ``-device pci-ipmi-bt,bmc=id``
     Like the KCS interface, but defines a BT interface on the PCI bus.
+
+``-device intel-iommu[,option=...]``
+    This is only supported by ``-machine q35``, which will enable Intel VT-d
+    emulation within the guest.  It supports below options:
+
+    ``intremap=on|off`` (default: auto)
+        This enables interrupt remapping feature in the guest.  It's required
+        to enable complete x2apic.  Currently it only supports kvm
+        kernel-irqchip modes ``off`` or ``split``.  Full kernel-irqchip is not
+        yet supported.
+
+    ``caching-mode=on|off`` (default: off)
+        This enables caching mode for the VT-d emulated device.  When
+        caching-mode is enabled, each guest DMA buffer mapping will

generate an
+ IOTLB invalidation from the guest IOMMU driver to the vIOMMU
device in

qemu-options.hx +17 -1
+        a synchronous way.  It is required for ``-device vfio-pci`` to work
+        with the VT-d device, because host assigned devices requires to setup
+        the DMA mapping on the host before guest DMA starts.
+
+    ``device-iotlb=on|off`` (default: off)
+        This enables device-iotlb capability for the emulated VT-d device.  So
+        far virtio/vhost should be the only real user for this parameter,
+        paired with ats=on configured for the device.
+
+    ``aw-bits=39|48`` (default: 39)
+        This decides the address width of IOVA address space.  The address
+        space has 39 bits width for 3-level IOMMU page tables, and 48 bits for
+        4-level IOMMU page tables.
+
+    Please also refer to the wiki page for general scenarios of VT-d
+    emulation in QEMU: https://wiki.qemu.org/Features/VT-d.
+
 ERST
 
 DEF("name", HAS_ARG, QEMU_OPTION_name,
--

2.31.1