Skip to contentSkip to conversation

vSMMU initialization

RFC In review Baptiste Reynal 3 participants 15 messages Jun 12, 2015 – Jul 15, 2015 6 messages not in the archive

lore
Share & tools
lore
#0 lore raw

cover vSMMU initialization

The ARM SMMU has support for 2-stages address translations, allowing a virtual address to be translated at two levels:
- Stage 1 translates a virtual address (VA) into an intermediate physical address (IPA)
- Stage 2 translates an IPA into a physical address (PA)

Will Deacon introduced a virtual SMMU interface for KVM, which gives a virtual machine the possibility to use an IOMMU with native drivers. While the VM will program the first stage of translation (stage 1), the interface will program the second (stage 2) on the physical SMMU.

This RFC brings support for the vSMMU on QEMU-side, introducing the code for vSMMU initialization and the device tree operations needed to attach devices to it.

The main purpose of such vSMMU device is to allow VFIO devices to benefit from an IOMMU on the guest side, for userspace drivers or nested VFIO.

It is an early state of development so:
- Interrupts are not handled for now
- Only one device can be bound to the vSMMU

I'm open to any comments or suggestions.

Baptiste Reynal (6):
headers sync
hw/core/platform-bus: initialization notifier hw/core/platform-bus: add base_address field hw/vfio: vsmmu device
hw/arm/sysbus-fdt: enable vsmmu dynamic instantiation hw/arm/sysbus-fdt: add smmu masters in device tree

 hw/arm/sysbus-fdt.c             |  78 ++++++++++++++++++++
 hw/arm/virt.c                   |   4 +
 hw/core/platform-bus.c          |  13 ++++
 hw/vfio/Makefile.objs           |   1 +
 hw/vfio/common.c                |   8 +-
 hw/vfio/platform.c              |   2 +
 hw/vfio/smmu.c                  | 157 ++++++++++++++++++++++++++++++++++++++++
 include/hw/platform-bus.h       |   3 +
 include/hw/vfio/vfio-platform.h |   1 +
 include/hw/vfio/vfio-smmu.h     |  50 +++++++++++++
 linux-headers/linux/kvm.h       |  15 ++++
 11 files changed, 330 insertions(+), 2 deletions(-)
 create mode 100644 hw/vfio/smmu.c
 create mode 100644 include/hw/vfio/vfio-smmu.h
Signature · 1 line
2.4.3

A message that is not in the archive (sent off-list or not yet received): 1434118810-28219-1-git-send-email-b.reynal-lrHrjnjw1UfHK3s98zE1ajGjJy/[email protected]

#2 lore raw

RFC 1/6 headers sync

---
 linux-headers/linux/kvm.h | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
linux-headers/linux/kvm.h +15 -0
index fad9e5c..d48d7b1 100644
@@ -1003,6 +1003,19 @@ struct kvm_device_attr {
 #define   KVM_DEV_VFIO_GROUP_ADD			1
 #define   KVM_DEV_VFIO_GROUP_DEL			2
 
+#define  KVM_DEV_ARM_SMMU_V2_CFG		1
+#define   KVM_DEV_ARM_SMMU_V2_CFG_INIT			1
+#define   KVM_DEV_ARM_SMMU_V2_CFG_IRQ			2
+#define   KVM_DEV_ARM_SMMU_V2_CFG_SIZE			3
+#define  KVM_DEV_ARM_SMMU_V2_VFIO		2
+#define   KVM_DEV_ARM_SMMU_V2_VFIO_GROUP_ADD		1
+#define   KVM_DEV_ARM_SMMU_V2_VFIO_GROUP_DEL		2
+
+struct arm_smmu_v2_vfio_group_sid {
+	__u32	fd;	/* file descriptor of the VFIO group */
+	__u16	sid;
+};
+
 enum kvm_device_type {
 	KVM_DEV_TYPE_FSL_MPIC_20	= 1,
 #define KVM_DEV_TYPE_FSL_MPIC_20	KVM_DEV_TYPE_FSL_MPIC_20
@@ -1018,6 +1031,8 @@ enum kvm_device_type {
 #define KVM_DEV_TYPE_FLIC		KVM_DEV_TYPE_FLIC
 	KVM_DEV_TYPE_ARM_VGIC_V3,
 #define KVM_DEV_TYPE_ARM_VGIC_V3	KVM_DEV_TYPE_ARM_VGIC_V3
+	KVM_DEV_TYPE_ARM_SMMU_V2,
+#define KVM_DEV_TYPE_ARM_SMMU_V2	KVM_DEV_TYPE_ARM_SMMU_V2
 	KVM_DEV_TYPE_MAX,
 };
 
Signature · 1 line
2.4.3
#3 lore raw

RFC 2/6 hw/core/platform-bus: initialization notifier

In reply to a message not in the archive

Add a platform_bus_link_done_notifier for devices to be notified once the platform bus is initialized. Address and IRQs for platform devices are unknown at device realization, we need to notify the device once the platform bus gathered all devices, i.e. when those resources are ready. This is required for some device initialization (e.g. the vSMMU base address needs to be set on the KVM device).

---
 hw/core/platform-bus.c    | 13 +++++++++++++
 include/hw/platform-bus.h |  1 +
 2 files changed, 14 insertions(+)
hw/core/platform-bus.c +13 -0
index 0f052b3..8642dfc 100644
@@ -24,6 +24,18 @@
 #include "exec/address-spaces.h"
 #include "sysemu/sysemu.h"
 
+static NotifierList platform_bus_link_done_notifiers =
+    NOTIFIER_LIST_INITIALIZER(platform_bus_link_done_notifiers);
+
+void qemu_add_platform_bus_link_done_notifier(Notifier *notify)
+{
+    notifier_list_add(&platform_bus_link_done_notifiers, notify);
+}
+
+static void qemu_run_platform_bus_link_done_notifiers(void)
+{
+    notifier_list_notify(&platform_bus_link_done_notifiers, NULL);
+}
 
 /*
  * Returns the PlatformBus IRQ number for a SysBusDevice irq number or -1 if
@@ -196,6 +208,7 @@ static void platform_bus_init_notify(Notifier *notifier, void *data)
     plaform_bus_refresh_irqs(pb);
 
     foreach_dynamic_sysbus_device(link_sysbus_device, pb);
+    qemu_run_platform_bus_link_done_notifiers();
 }
 
 static void platform_bus_realize(DeviceState *dev, Error **errp)
include/hw/platform-bus.h +1 -0
index bd42b83..3c3f96b 100644
@@ -53,5 +53,6 @@ int platform_bus_get_irqn(PlatformBusDevice *platform_bus, SysBusDevice *sbdev,
                           int n);
 hwaddr platform_bus_get_mmio_addr(PlatformBusDevice *pbus, SysBusDevice *sbdev,
                                   int n);
+void qemu_add_platform_bus_link_done_notifier(Notifier *notify);
 
 #endif /* !HW_PLATFORM_BUS_H */
Signature · 1 line
2.4.3
#4 lore raw

RFC 3/6 hw/core/platform-bus: add base_address field

In reply to a message not in the archive

Add base_address field to platform_bus structure and initialize it.

As the platform devices addresses are relative to the platform bus, we need it to get the absolute address of a device.

---
 hw/arm/virt.c             | 4 ++++
 include/hw/platform-bus.h | 2 ++
 2 files changed, 6 insertions(+)
hw/arm/virt.c +4 -0
index 1b1cc71..33361c5 100644
@@ -740,6 +740,7 @@ static void create_platform_bus(VirtBoardInfo *vbi, qemu_irq *pic)
 {
     DeviceState *dev;
     SysBusDevice *s;
+    PlatformBusDevice *pbus;
     int i;
     ARMPlatformBusFDTParams *fdt_params = g_new(ARMPlatformBusFDTParams, 1);
     MemoryRegion *sysmem = get_system_memory();
@@ -766,6 +767,9 @@ static void create_platform_bus(VirtBoardInfo *vbi, qemu_irq *pic)
         platform_bus_params.platform_bus_size);
     qdev_init_nofail(dev);
     s = SYS_BUS_DEVICE(dev);
+    pbus = PLATFORM_BUS_DEVICE(dev);
+
+    pbus->base_address = vbi->memmap[VIRT_PLATFORM_BUS].base;
 
     for (i = 0; i < platform_bus_params.platform_bus_num_irqs; i++) {
         int irqn = platform_bus_params.platform_bus_first_irq + i;
include/hw/platform-bus.h +2 -0
index 3c3f96b..13efc82 100644
@@ -41,6 +41,8 @@ struct PlatformBusDevice {
     bool done_gathering;
 
     /*< public >*/
+    hwaddr base_address;
+
     uint32_t mmio_size;
     MemoryRegion mmio;
 
Signature · 1 line
2.4.3
#5 lore raw

RFC 4/6 hw/vfio: vsmmu device

In reply to a message not in the archive

This patches introduces support for ARM virtual SMMU, enabling two stages address translation.

The vSMMU device can be instantiated from the command line using following option:

-device vsmmu,x-group=1

---
 hw/vfio/Makefile.objs       |   1 +
 hw/vfio/common.c            |   8 ++-
 hw/vfio/platform.c          |   1 +
 hw/vfio/smmu.c              | 157 ++++++++++++++++++++++++++++++++++++++++++++
 include/hw/vfio/vfio-smmu.h |  50 ++++++++++++++
 5 files changed, 215 insertions(+), 2 deletions(-)
 create mode 100644 hw/vfio/smmu.c
 create mode 100644 include/hw/vfio/vfio-smmu.h
hw/vfio/Makefile.objs +1 -0
index d540c9d..4b91901 100644
@@ -3,4 +3,5 @@ obj-$(CONFIG_SOFTMMU) += common.o
 obj-$(CONFIG_PCI) += pci.o
 obj-$(CONFIG_SOFTMMU) += platform.o
 obj-$(CONFIG_SOFTMMU) += calxeda-xgmac.o
+obj-$(CONFIG_SOFTMMU) += smmu.o
 endif
hw/vfio/common.c +6 -2
index b1045da..a3ee72f 100644
@@ -676,8 +676,12 @@ static int vfio_connect_container(VFIOGroup *group, AddressSpace *as)
             goto free_container_exit;
         }
 
-        ret = ioctl(fd, VFIO_SET_IOMMU,
-                    v2 ? VFIO_TYPE1v2_IOMMU : VFIO_TYPE1_IOMMU);
+        if (ioctl(fd, VFIO_CHECK_EXTENSION, VFIO_TYPE1_NESTING_IOMMU)) {
+            ret = ioctl(fd, VFIO_SET_IOMMU, VFIO_TYPE1_NESTING_IOMMU);
+        } else {
+            ret = ioctl(fd, VFIO_SET_IOMMU,
+			    v2 ? VFIO_TYPE1v2_IOMMU : VFIO_TYPE1_IOMMU);
+        }
         if (ret) {
             error_report("vfio: failed to set iommu for container: %m");
             ret = -errno;
hw/vfio/platform.c +1 -0
index 9382bb7..6192458 100644
@@ -475,6 +475,7 @@ static int vfio_base_device_init(VFIODevice *vbasedev)
         error_report("vfio: failed to get group %d", groupid);
         return -ENOENT;
     }
+    vbasedev->group = group;
 
     g_snprintf(path, sizeof(path), "%s", vbasedev->name);
 
hw/vfio/smmu.c new file +157 -0

157 lines not on this page · raw message

include/hw/vfio/vfio-smmu.h new file +50 -0

50 lines not on this page · raw message

Signature · 1 line
2.4.3
#6 lore raw

RFC 5/6 hw/arm/sysbus-fdt: enable vsmmu dynamic instantiation

In reply to a message not in the archive

This patch adds the device tree node creation for the vSMMU.

---
 hw/arm/sysbus-fdt.c | 43 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 43 insertions(+)
hw/arm/sysbus-fdt.c +43 -0
index 3038b94..67f4425 100644
@@ -26,6 +26,10 @@
 #include "sysemu/device_tree.h"
 #include "hw/platform-bus.h"
 #include "sysemu/sysemu.h"
+#include "hw/vfio/vfio-platform.h"
+#include "hw/vfio/vfio-smmu.h"
+
+#include <libfdt.h>
 
 /*
  * internal struct that contains the information to create dynamic
@@ -53,8 +57,47 @@ typedef struct NodeCreationPair {
     int (*add_fdt_node_fn)(SysBusDevice *sbdev, void *opaque);
 } NodeCreationPair;
 
+VFIOSmmuDevice *vsmmu = NULL;
+
+static int add_arm_smmu_fdt_node(SysBusDevice *sbdev, void *opaque)
+{
+    PlatformBusFDTData *data = opaque;
+    vsmmu = VFIO_SMMU_DEVICE(sbdev);
+    void *fdt = data->fdt;
+    const char *nodename;
+    uint32_t reg_attr[4], irq_attr[3];
+    uint32_t gint = 0;
+
+    nodename = g_strdup_printf("/smmu@%" PRIx64, vsmmu->base);
+
+    qemu_fdt_add_subnode(fdt, nodename);
+    qemu_fdt_setprop(fdt, nodename, "compatible", "arm,smmu-v2", 12);
+
+    reg_attr[0] = 0;
+    reg_attr[1] = cpu_to_be32(vsmmu->base);
+    reg_attr[2] = 0;
+    reg_attr[3] = cpu_to_be32(vsmmu->size);
+
+    qemu_fdt_setprop(fdt, nodename, "reg",
+            reg_attr, 4*sizeof(uint32_t));
+
+    /* Add dummy interrupt */
+    irq_attr[0] = 0;
+    irq_attr[1] = 0;
+    irq_attr[2] = 0;
+
+    qemu_fdt_setprop(fdt, nodename, "interrupts",
+            irq_attr, 3*sizeof(uint32_t));
+
+    qemu_fdt_setprop(fdt, nodename, "#global-interrupts", &gint,
+            sizeof(uint32_t));
+
+    return 0;
+}
+
 /* list of supported dynamic sysbus devices */
 static const NodeCreationPair add_fdt_node_functions[] = {
+    {TYPE_VFIO_SMMU, add_arm_smmu_fdt_node},
     {"", NULL}, /* last element */
 };
 
Signature · 1 line
2.4.3
#7 lore raw

RFC 6/6 hw/arm/sysbus-fdt: add smmu masters in device tree

In reply to a message not in the archive

This patch adds a function to bind a device as an SMMU master in the device tree. Currently, only one device is supported.

A device can be set as SMMU master by using the "x-iommu" argument on the command line:

-device pl330,x-iommu=true

---
 hw/arm/sysbus-fdt.c             | 35 +++++++++++++++++++++++++++++++++++
 hw/vfio/platform.c              |  1 +
 include/hw/vfio/vfio-platform.h |  1 +
 3 files changed, 37 insertions(+)
hw/arm/sysbus-fdt.c +35 -0
index 67f4425..356f644 100644
@@ -101,6 +101,38 @@ static const NodeCreationPair add_fdt_node_functions[] = {
     {"", NULL}, /* last element */
 };
 
+static int add_smmu_master(SysBusDevice *sbdev, void *opaque)
+{
+    if (object_dynamic_cast(OBJECT(sbdev), TYPE_VFIO_PLATFORM) &&
+	    VFIO_PLATFORM_DEVICE(sbdev)->iommu_master) {
+        int master[2];
+        char *smmu_nodename, *master_nodename;
+        uint64_t mmio_base;
+        PlatformBusFDTData *data = opaque;
+        PlatformBusDevice *pbus = data->pbus;
+        void *fdt = data->fdt;
+        VFIOPlatformDevice *vdev = VFIO_PLATFORM_DEVICE(sbdev);
+        VFIODevice *vbasedev = &vdev->vbasedev;
+        const char *parent_node = data->pbus_node_name;
+
+        smmu_nodename = g_strdup_printf("/smmu@%" PRIx64, vsmmu->base);
+        mmio_base = platform_bus_get_mmio_addr(pbus, sbdev, 0);
+
+        master_nodename = g_strdup_printf("%s/%s@%" PRIx64, parent_node,
+                vbasedev->name,
+                mmio_base);
+
+        master[0] = cpu_to_be32(qemu_fdt_get_phandle(fdt, master_nodename));
+        master[1] = cpu_to_be32(vbasedev->group->groupid);
+
+        qemu_fdt_setprop(fdt, smmu_nodename, "mmu-masters",
+                master, 2*sizeof(int));
+
+    }
+
+    return 0;
+}
+
 /**
  * add_fdt_node - add the device tree node of a dynamic sysbus device
  *
@@ -193,6 +225,9 @@ static void add_all_platform_bus_fdt_nodes(ARMPlatformBusFDTParams *fdt_params)
 
     /* Loop through all dynamic sysbus devices and create their node */
     foreach_dynamic_sysbus_device(add_fdt_node, &data);
+    if (vsmmu) {
+        foreach_dynamic_sysbus_device(add_smmu_master, &data);
+    }
 
     g_free(node);
 }
hw/vfio/platform.c +1 -0
index 6192458..d4ea297 100644
@@ -585,6 +585,7 @@ static Property vfio_platform_dev_properties[] = {
     DEFINE_PROP_BOOL("x-mmap", VFIOPlatformDevice, vbasedev.allow_mmap, true),
     DEFINE_PROP_UINT32("mmap-timeout-ms", VFIOPlatformDevice,
                        mmap_timeout, 1100),
+    DEFINE_PROP_BOOL("x-iommu", VFIOPlatformDevice, iommu_master, false),
     DEFINE_PROP_END_OF_LIST(),
 };
 
include/hw/vfio/vfio-platform.h +1 -0
index 26b2ad6..8f5d980 100644
@@ -57,6 +57,7 @@ typedef struct VFIOPlatformDevice {
     uint32_t mmap_timeout; /* delay to re-enable mmaps after interrupt */
     QEMUTimer *mmap_timer; /* allows fast-path resume after IRQ hit */
     QemuMutex intp_mutex; /* protect the intp_list IRQ state */
+    bool iommu_master; /* is the device a master to the vSMMU ? */
 } VFIOPlatformDevice;
 
 typedef struct VFIOPlatformDeviceClass {
Signature · 1 line
2.4.3
#8 lore raw
In reply to a message not in the archive
Quoted 10 lines · Baptiste Reynal

The ARM SMMU has support for 2-stages address translations, allowing a virtual address to be translated at two levels:
- Stage 1 translates a virtual address (VA) into an intermediate physical address (IPA)
- Stage 2 translates an IPA into a physical address (PA)

Will Deacon introduced a virtual SMMU interface for KVM, which gives a virtual machine the possibility to use an IOMMU with native drivers. While the VM will program the first stage of translation (stage 1), the interface will program the second (stage 2) on the physical SMMU.

Please note that I have no plans to merge the kernel-side of this at the moment. It was merely an exploratory tool to see what a non-PV vSMMU implementation might look like and certainly not intended to be used in anger.

Will

A message that is not in the archive (sent off-list or not yet received): [email protected]

#10 lore raw

Hi Will,

Quoted 21 lines

-----Original Message-----
From: iommu-bounces-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org [mailto:iommu- bounces-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org] On Behalf Of Will Deacon
Sent: Friday, June 12, 2015 7:53 PM
To: Baptiste Reynal
Cc: iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org; tech-lrHrjnjw1UfHK3s98zE1ajGjJy/[email protected]; [email protected]
Subject: Re: [RFC 0/6] vSMMU initialization

Quoted 10 lines · Baptiste Reynal

The ARM SMMU has support for 2-stages address translations, allowing a virtual address to be translated at two levels:
- Stage 1 translates a virtual address (VA) into an intermediate physical address (IPA)
- Stage 2 translates an IPA into a physical address (PA)

Will Deacon introduced a virtual SMMU interface for KVM, which gives a virtual machine the possibility to use an IOMMU with native drivers. While the VM will program the first stage of translation (stage 1), the interface will program the second (stage 2) on the physical SMMU.

Quoted 4 lines

Please note that I have no plans to merge the kernel-side of this at the moment. It was merely an exploratory tool to see what a non-PV vSMMU implementation might look like and certainly not intended to be used in anger.

How do you see the context fault reporting work for the PV interface? Currently the vSMMU interface does seem quiet restrictive and it may simplify things by having PV iommu interface. But, do you see this even true in case of SMMUv3? Just wondering if we can give more control with respect memory transaction attributes to the guest. Also, would it make sense to give guest control of the fault handling attributes i.e. fault/terminate model.

-Varun

A message that is not in the archive (sent off-list or not yet received): BN1PR0301MB06274ACB3A193799F40BE9F4EA9B0-RQSpjbwlmjSD1ymB6+i1+JwN6zqB+hSMnBOFsp37pqbUKgpGm//BTAC/[email protected]

#12 lore raw
Quoted 1 line · Varun Sethi

Hi Will,

Hi Varun,

Quoted 12 lines
Quoted 12 lines · Baptiste Reynal
Quoted 10 lines

The ARM SMMU has support for 2-stages address translations, allowing a virtual address to be translated at two levels:
- Stage 1 translates a virtual address (VA) into an intermediate physical address (IPA)
- Stage 2 translates an IPA into a physical address (PA)

Will Deacon introduced a virtual SMMU interface for KVM, which gives a virtual machine the possibility to use an IOMMU with native drivers. While the VM will program the first stage of translation (stage 1), the interface will program the second (stage 2) on the physical SMMU.

Quoted 5 lines
Quoted 4 lines

Please note that I have no plans to merge the kernel-side of this at the moment. It was merely an exploratory tool to see what a non-PV vSMMU implementation might look like and certainly not intended to be used in anger.

How do you see the context fault reporting work for the PV interface?

We could have an interrupt, for the PV IOMMU and have the hypervisor inject that, no?

Quoted 3 lines

Currently the vSMMU interface does seem quiet restrictive and it may simplify things by having PV iommu interface. But, do you see this even true in case of SMMUv3?

I think SMMUv3 is *far* more amenable to the vSMMU approach, largely because it moves many of the data structures into memory, but also because it has support for things like ATS and PRI, so sharing page tables with the CPU becomes a real possibility (and is something that doesn't work well with a PV model).

Quoted 3 lines

Just wondering if we can give more control with respect memory transaction attributes to the guest. Also, would it make sense to give guest control of the fault handling attributes i.e. fault/terminate model.

I'd like to see the basics prototyped before we start trying to design for these more advanced use-cases. I'm confident there are plenty of things we haven't even considered at the moment.

Will

A message that is not in the archive (sent off-list or not yet received): [email protected]

#14 lore raw
Quoted 17 lines · Will Deacon
Quoted 1 line · Varun Sethi

Hi Will,

Hi Varun,

Quoted 12 lines
Quoted 12 lines · Baptiste Reynal
Quoted 10 lines

The ARM SMMU has support for 2-stages address translations, allowing a virtual address to be translated at two levels:
- Stage 1 translates a virtual address (VA) into an intermediate physical address (IPA)
- Stage 2 translates an IPA into a physical address (PA)

Will Deacon introduced a virtual SMMU interface for KVM, which gives a virtual machine the possibility to use an IOMMU with native drivers. While the VM will program the first stage of translation (stage 1), the interface will program the second (stage 2) on the physical SMMU.

Quoted 12 lines
Quoted 5 lines
Quoted 4 lines

Please note that I have no plans to merge the kernel-side of this at the moment. It was merely an exploratory tool to see what a non-PV vSMMU implementation might look like and certainly not intended to be used in anger.

How do you see the context fault reporting work for the PV interface?

We could have an interrupt, for the PV IOMMU and have the hypervisor inject that, no?

Quoted 3 lines

Currently the vSMMU interface does seem quiet restrictive and it may simplify things by having PV iommu interface. But, do you see this even true in case of SMMUv3?

Varun, may I know what do you mean by "more restrictive" ? Do you have in mind any use case which should apply to the PV interface and not the vSMMU ?

Quoted 7 lines

I think SMMUv3 is *far* more amenable to the vSMMU approach, largely because it moves many of the data structures into memory, but also because it has support for things like ATS and PRI, so sharing page tables with the CPU becomes a real possibility (and is something that doesn't work well with a PV model).

Quoted 7 lines
Quoted 3 lines

Just wondering if we can give more control with respect memory transaction attributes to the guest. Also, would it make sense to give guest control of the fault handling attributes i.e. fault/terminate model.

I'd like to see the basics prototyped before we start trying to design for these more advanced use-cases. I'm confident there are plenty of things we haven't even considered at the moment.

Hi Will,

Quoted 1 line

From my current understanding, vSMMU and PV interface seems

complementary and have different target. While the PV interface targets broad compatibility with hardware and page table abstraction, the vSMMU relies on specific hardware capabilities for better performances (with dual-stage support and future ATS/PRI). As no PV interface exists for now, we decided to focus our effort on the vSMMU. Unless PV interface is strictly needed, we'd like to continue with the implementation of the vSMMU.

In my opinion, both solutions are complementary and can co-exist once someone shows interest for the PV.

Quoted 2 lines

Will

Best regards,
Baptiste

A message that is not in the archive (sent off-list or not yet received): CAN9JPjF1E0Dg_KAQcc4C+jKDuOHEATVhN+1NPa66jLrP8RieKA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org

#16 lore raw
Quoted 15 lines · Baptiste Reynal
Quoted 6 lines · Will Deacon

I think SMMUv3 is *far* more amenable to the vSMMU approach, largely because it moves many of the data structures into memory, but also because it has support for things like ATS and PRI, so sharing page tables with the CPU becomes a real possibility (and is something that doesn't work well with a PV model).

Quoted 7 lines
Quoted 3 lines

Just wondering if we can give more control with respect memory transaction attributes to the guest. Also, would it make sense to give guest control of the fault handling attributes i.e. fault/terminate model.

I'd like to see the basics prototyped before we start trying to design for these more advanced use-cases. I'm confident there are plenty of things we haven't even considered at the moment.

Quoted 8 lines

From my current understanding, vSMMU and PV interface seems complementary and have different target. While the PV interface targets broad compatibility with hardware and page table abstraction, the vSMMU relies on specific hardware capabilities for better performances (with dual-stage support and future ATS/PRI). As no PV interface exists for now, we decided to focus our effort on the vSMMU. Unless PV interface is strictly needed, we'd like to continue with the implementation of the vSMMU.

On the contrary, I'm not going to merge vSMMU code unless there are strong technical reasons against PV. I don't see your performance argument (I would actually expect the vSMMU to be *slower* with SMMUv2) and I really don't want fragmentation where user ABI is concerned.

Also, your argument about focussing on vSMMU because PV doesn't exist yet doesn't make sense. Mainline doesn't support either of these for ARM.

Quoted 2 lines

In my opinion, both solutions are complementary and can co-exist once someone shows interest for the PV.

I think you have this the wrong way around. We should start with PV (one portable interface) and only add vSMMU interfaces where there are good reasons to do so.

Will

#17 lore raw
In reply to a message not in the archive

Hi Baptiste,

Quoted 37 lines

-----Original Message-----
From: Baptiste Reynal [mailto:b.reynal-lrHrjnjw1UfHK3s98zE1ajGjJy/[email protected]]
Sent: Wednesday, July 15, 2015 7:08 PM
To: Will Deacon
Cc: Sethi Varun-B16395; iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org; tech-lrHrjnjw1UfHK3s98zE1ajGjJy/[email protected]; [email protected]
Subject: Re: [RFC 0/6] vSMMU initialization

Quoted 14 lines · Will Deacon
Quoted 1 line · Varun Sethi

Hi Will,

Hi Varun,

Quoted 9 lines
Quoted 9 lines · Baptiste Reynal
Quoted 8 lines

The ARM SMMU has support for 2-stages address translations, allowing a virtual address to be translated at two levels:
- Stage 1 translates a virtual address (VA) into an intermediate physical address (IPA)
- Stage 2 translates an IPA into a physical address (PA)

Will Deacon introduced a virtual SMMU interface for KVM, which gives a virtual machine the possibility to use an IOMMU with native

drivers.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

While the VM will program the first stage of translation (stage 1), the interface will program the second (stage 2) on the physical

SMMU.

Quoted 10 lines
Quoted 6 lines
Quoted 5 lines

Please note that I have no plans to merge the kernel-side of this at the moment. It was merely an exploratory tool to see what a non-PV vSMMU implementation might look like and certainly not intended to be used in anger.

How do you see the context fault reporting work for the PV interface?

We could have an interrupt, for the PV IOMMU and have the hypervisor inject that, no?

Quoted 8 lines
Quoted 3 lines
Quoted 3 lines

Currently the vSMMU interface does seem quiet restrictive and it may simplify things by having PV iommu interface. But, do you see this even true in case of SMMUv3?

Varun, may I know what do you mean by "more restrictive" ? Do you have in mind any use case which should apply to the PV interface and not the vSMMU ?

[varun] What I meant was that vSMMU allows for setting up of the stage 1 translation, but doesn't specifically allow access to other SMMU hardware functionality. We can certainly extend the vSMMU interface for providing additional functionality to the guest VM. I do agree with Will, that for extending the vSMMU interface prototyping is necessary. We need to come up with specific use cases for that. For controlling stage 1 translation PV interface may be a bit simpler, but then we would need a generic interface to bind to most IOMMUs in the host.

-Varun

#18 lore raw
In reply to a message not in the archive

Hi Will,

Quoted 27 lines

-----Original Message-----
From: Will Deacon [mailto:[email protected]]
Sent: Tuesday, July 14, 2015 4:34 PM
To: Sethi Varun-B16395
Cc: Baptiste Reynal; iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org; tech-lrHrjnjw1UfHK3s98zE1ajGjJy/[email protected]; [email protected]
Subject: Re: [RFC 0/6] vSMMU initialization

Quoted 1 line · Varun Sethi

Hi Will,

Hi Varun,

Quoted 9 lines
Quoted 9 lines · Baptiste Reynal
Quoted 8 lines

The ARM SMMU has support for 2-stages address translations, allowing a virtual address to be translated at two levels:
- Stage 1 translates a virtual address (VA) into an intermediate physical address (IPA)
- Stage 2 translates an IPA into a physical address (PA)

Will Deacon introduced a virtual SMMU interface for KVM, which gives a virtual machine the possibility to use an IOMMU with native

drivers.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

While the VM will program the first stage of translation (stage 1), the interface will program the second (stage 2) on the physical

SMMU.

Quoted 1 line
Quoted 1 line
Quoted 9 lines
Quoted 5 lines
Quoted 4 lines

Please note that I have no plans to merge the kernel-side of this at the moment. It was merely an exploratory tool to see what a non-PV vSMMU implementation might look like and certainly not intended to be used in anger.

How do you see the context fault reporting work for the PV interface?

We could have an interrupt, for the PV IOMMU and have the hypervisor inject that, no?

Can you please elaborate on the PV IOMMU interface. I want to understand how context fault information would be communicated to the guest.

-Varun

A message that is not in the archive (sent off-list or not yet received): BN1PR0301MB0627944BCFB295564CA1F7AAEA9A0-RQSpjbwlmjSD1ymB6+i1+JwN6zqB+hSMnBOFsp37pqbUKgpGm//BTAC/[email protected]

#20 lore raw
Quoted 18 lines · Varun Sethi
Quoted 18 lines
Quoted 9 lines
Quoted 9 lines · Baptiste Reynal
Quoted 8 lines

The ARM SMMU has support for 2-stages address translations, allowing a virtual address to be translated at two levels:
- Stage 1 translates a virtual address (VA) into an intermediate physical address (IPA)
- Stage 2 translates an IPA into a physical address (PA)

Will Deacon introduced a virtual SMMU interface for KVM, which gives a virtual machine the possibility to use an IOMMU with native

drivers.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

While the VM will program the first stage of translation (stage 1), the interface will program the second (stage 2) on the physical

SMMU.

Quoted 5 lines
Quoted 5 lines

Please note that I have no plans to merge the kernel-side of this at the moment. It was merely an exploratory tool to see what a non-PV vSMMU implementation might look like and certainly not intended to be used in anger.

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

How do you see the context fault reporting work for the PV interface?

We could have an interrupt, for the PV IOMMU and have the hypervisor inject that, no?

Can you please elaborate on the PV IOMMU interface. I want to understand how context fault information would be communicated to the guest.

I replied to this the other day!

My assumption is that we'd have an irq and some memory region to describe the fault in as general a way as possible. Whether that memory region looks like MMIO registers or something like a virtio ring buffer is an implementation detail to be resolved by prototyping.

Will