Skip to contentSkip to conversation

VFIO no-iommu

RFC PATCH Superseded Alex Williamson 7 participants 21 messages Oct 9, 2015 – Oct 12, 2015 VFIO driver

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

cover VFIO no-iommu

Recent patches for UIO have been attempting to add MSI/X support, which unfortunately implies DMA support, which users have been enabling anyway, but was never intended for UIO. VFIO on the other hand expects an IOMMU to provide isolation of devices, but provides a much more complete device interface, which already supports full MSI/X support. There's really no way to support userspace drivers with DMA capable devices without an IOMMU to protect the host, but we can at least think about doing it in a way that properly taints the kernel and avoids creating new code duplicating existing code, that does have a supportable use case.

The diffstat is only so large because I moved vfio.c to vfio_core.c so I could more easily keep the module named vfio.ko while keeping the bulk of the no-iommu support in a separate file that can be optionally compiled. We're really looking at a couple hundred lines of mostly stub code. The VFIO_NOIOMMU_IOMMU could certainly be expanded to do page pinning and virt_to_bus() translation, but I didn't want to complicate anything yet.

I've only compiled this and tested loading the module with the new no-iommu mode enabled, I haven't actually tried to port a DPDK driver to it, though it ought to be a pretty obvious mix of the existing UIO and VFIO versions (set the IOMMU, but avoid using it for mapping, use however bus translations are done w/ UIO). The core vfio device file is still /dev/vfio/vfio, but all the groups become /dev/vfio-noiommu/$GROUP.

It should be obvious, but I always feel obligated to state that this does not and will not ever enable device assignment to virtual machines on non-IOMMU capable platforms.

I'm curious what IOMMU folks think of this. This hack is really only possible because we don't use iommu_ops for regular DMA, so we can hijack it fairly safely. I believe that's intended to change though, so this may not be practical long term. Thanks,

Alex

---

Alex Williamson (2):
vfio: Move vfio.c vfio_core.c
vfio: Include no-iommu mode

 drivers/vfio/Kconfig        |   15 
 drivers/vfio/Makefile       |    4 
 drivers/vfio/vfio.c         | 1640 ------------------------------------------
 drivers/vfio/vfio_core.c    | 1680 +++++++++++++++++++++++++++++++++++++++++++
 drivers/vfio/vfio_noiommu.c |  185 +++++
 drivers/vfio/vfio_private.h |   31 +
 include/uapi/linux/vfio.h   |    2 
 7 files changed, 1917 insertions(+), 1640 deletions(-)
 delete mode 100644 drivers/vfio/vfio.c
 create mode 100644 drivers/vfio/vfio_core.c
 create mode 100644 drivers/vfio/vfio_noiommu.c
 create mode 100644 drivers/vfio/vfio_private.h
#1 lore raw

RFC PATCH 1/2 vfio: Move vfio.c vfio_core.c

This allows us to more easily create a "vfio" module that includes multiple files. No code change, rename and Makefile update only.

---
 drivers/vfio/Makefile    |    1 
 drivers/vfio/vfio.c      | 1640 ----------------------------------------------
 drivers/vfio/vfio_core.c | 1640 ++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 1641 insertions(+), 1640 deletions(-)
 delete mode 100644 drivers/vfio/vfio.c
 create mode 100644 drivers/vfio/vfio_core.c
drivers/vfio/Makefile +1 -0
index 7b8a31f..e8fa248 100644
@@ -1,4 +1,5 @@
 vfio_virqfd-y := virqfd.o
+vfio-y := vfio_core.o
 
 obj-$(CONFIG_VFIO) += vfio.o
 obj-$(CONFIG_VFIO_VIRQFD) += vfio_virqfd.o
drivers/vfio/vfio.c deleted +0 -1640

1,640 lines not on this page · raw message

drivers/vfio/vfio_core.c new file +1640 -0

1,355 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Part of the diff is not shown here. View the raw message.

#2 lore raw

Re: [RFC PATCH 1/2] vfio: Move vfio.c vfio_core.c

Quoted 11 lines · Alex Williamson

This allows us to more easily create a "vfio" module that includes multiple files. No code change, rename and Makefile update only.

---
 drivers/vfio/Makefile    |    1 
 drivers/vfio/vfio.c      | 1640 ----------------------------------------------
 drivers/vfio/vfio_core.c | 1640 ++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 1641 insertions(+), 1640 deletions(-)
 delete mode 100644 drivers/vfio/vfio.c
 create mode 100644 drivers/vfio/vfio_core.c

Use -S to git format-patch so we can see the rename, not a delete/add patch please.

thanks,

greg k-h

#3 lore raw

RFC PATCH 2/2 vfio: Include no-iommu mode

In reply to Alex Williamson

There is really no way to safely give a user full access to a PCI without an IOMMU to protect the host from errant DMA. There is also no way to provide DMA translation, for use cases such as devices assignment to virtual machines. However, there are still those users that want userspace drivers under those conditions. The UIO driver exists for this use case, but does not provide the degree of device access and programming that VFIO has. In an effort to avoid code duplication, this introduces a No-IOMMU mode for VFIO.

This mode requires enabling CONFIG_VFIO_NOIOMMU and loading the vfio module with the option "enable_unsafe_pci_noiommu_mode". This should make it very clear that this mode is not safe. In this mode, there is no support for unprivileged users, CAP_SYS_ADMIN is required for access to the necessary dev files. Mixing no-iommu and secure VFIO is also unsupported, as are any VFIO IOMMU backends other than the vfio-noiommu backend. Furthermore, unsafe group files are relocated to /dev/vfio-noiommu/. Upon successful loading in this mode, the kernel is tainted due to the dummy IOMMU put in place. Unloading of the module in this mode is also unsupported and will BUG due to the lack of support for unregistering an IOMMU for a bus type.

---
 drivers/vfio/Kconfig        |   15 +++
 drivers/vfio/Makefile       |    3 +
 drivers/vfio/vfio_core.c    |   40 +++++++++
 drivers/vfio/vfio_noiommu.c |  185 +++++++++++++++++++++++++++++++++++++++++++
 drivers/vfio/vfio_private.h |   31 +++++++
 include/uapi/linux/vfio.h   |    2 
 6 files changed, 276 insertions(+)
 create mode 100644 drivers/vfio/vfio_noiommu.c
 create mode 100644 drivers/vfio/vfio_private.h
drivers/vfio/Kconfig +15 -0
index 4540179..929de0d 100644
@@ -31,5 +31,20 @@ menuconfig VFIO
 
 	  If you don't know what to do here, say N.
 
+menuconfig VFIO_NOIOMMU
+	bool "VFIO No IOMMU support"
+	depends on VFIO
+	help
+	  VFIO is built on the ability to isolate devices using the IOMMU.
+	  Only with an IOMMU can userspace access to DMA capable devices be
+	  considered secure.  VFIO No IOMMU support enables a dummy IOMMU
+	  for the purpose of re-using the VFIO infrastructure in a non-secure
+	  mode.  Use of this mode will result in an unsupportable kernel and
+	  will therefore taint the kernel.  Device assignment to virtual
+	  machines is also not possible with this mode since the dummy IOMMU
+	  cannot provide DMA translation.
+
+	  If you don't know what to do here, say N.
+
 source "drivers/vfio/pci/Kconfig"
 source "drivers/vfio/platform/Kconfig"
drivers/vfio/Makefile +3 -0
index e8fa248..9736449 100644
@@ -1,5 +1,8 @@
 vfio_virqfd-y := virqfd.o
 vfio-y := vfio_core.o
+ifdef CONFIG_VFIO_NOIOMMU
+vfio-y += vfio_noiommu.o
+endif
 
 obj-$(CONFIG_VFIO) += vfio.o
 obj-$(CONFIG_VFIO_VIRQFD) += vfio_virqfd.o
drivers/vfio/vfio_core.c +40 -0
index 563c510..495a490 100644
@@ -34,10 +34,18 @@
 #include <linux/vfio.h>
 #include <linux/wait.h>
 
+#include "vfio_private.h"
+
 #define DRIVER_VERSION	"0.3"
 #define DRIVER_AUTHOR	"Alex Williamson <[email protected]>"
 #define DRIVER_DESC	"VFIO - User Level meta-driver"
 
+static bool pci_noiommu __read_mostly;
+#ifdef CONFIG_VFIO_NOIOMMU
+module_param_named(enable_unsafe_pci_noiommu_mode, pci_noiommu, bool, S_IRUGO);
+MODULE_PARM_DESC(enable_unsafe_pci_noiommu_mode, "Enable UNSAFE, no-IOMMU mode for PCI.  This mode provides no device isolation, no DMA translation, no host kernel protection, cannot be used for device assignment to virtual machines, requires root permissions, and will taint the kernel.  If you do not know what this is for, step away. (default: false)");
+#endif
+
 static struct vfio {
 	struct class			*class;
 	struct list_head		iommu_drivers_list;
@@ -101,6 +109,9 @@ int vfio_register_iommu_driver(const struct vfio_iommu_driver_ops *ops)
 {
 	struct vfio_iommu_driver *driver, *tmp;
 
+	if (pci_noiommu && strcmp(ops->name, "vfio-noiommu"))
+		return -EPERM;
+
 	driver = kzalloc(sizeof(*driver), GFP_KERNEL);
 	if (!driver)
 		return -ENOMEM;
@@ -998,6 +1009,9 @@ static int vfio_fops_open(struct inode *inode, struct file *filep)
 {
 	struct vfio_container *container;
 
+	if (pci_noiommu && !capable(CAP_SYS_ADMIN))
+		return -EPERM;
+
 	container = kzalloc(sizeof(*container), GFP_KERNEL);
 	if (!container)
 		return -ENOMEM;
@@ -1221,6 +1235,9 @@ static int vfio_group_get_device_fd(struct vfio_group *group, char *buf)
 	    !group->container->iommu_driver || !vfio_group_viable(group))
 		return -EINVAL;
 
+	if (pci_noiommu && !capable(CAP_SYS_ADMIN))
+		return -EPERM;
+
 	device = vfio_device_get_from_name(group, buf);
 	if (!device)
 		return -ENODEV;
@@ -1347,6 +1364,9 @@ static int vfio_group_fops_open(struct inode *inode, struct file *filep)
 	struct vfio_group *group;
 	int opened;
 
+	if (pci_noiommu && !capable(CAP_SYS_ADMIN))
+		return -EPERM;
+
 	group = vfio_group_get_from_minor(iminor(inode));
 	if (!group)
 		return -ENODEV;
@@ -1507,6 +1527,9 @@ struct vfio_group *vfio_group_get_external_user(struct file *filep)
 {
 	struct vfio_group *group = filep->private_data;
 
+	if (pci_noiommu)
+		return ERR_PTR(-EPERM);
+
 	if (filep->f_op != &vfio_group_fops)
 		return ERR_PTR(-EINVAL);
 
@@ -1549,6 +1572,9 @@ EXPORT_SYMBOL_GPL(vfio_external_check_extension);
  */
 static char *vfio_devnode(struct device *dev, umode_t *mode)
 {
+	if (pci_noiommu)
+		return kasprintf(GFP_KERNEL, "vfio-noiommu/%s", dev_name(dev));
+
 	return kasprintf(GFP_KERNEL, "vfio/%s", dev_name(dev));
 }
 
@@ -1564,6 +1590,12 @@ static int __init vfio_init(void)
 {
 	int ret;
 
+	if (pci_noiommu) {
+		ret = vfio_noiommu_iommu_init(&pci_bus_type);
+		if (ret)
+			return ret;
+	}
+
 	idr_init(&vfio.group_idr);
 	mutex_init(&vfio.group_lock);
 	mutex_init(&vfio.iommu_drivers_lock);
@@ -1605,6 +1637,9 @@ static int __init vfio_init(void)
 	request_module_nowait("vfio_iommu_type1");
 	request_module_nowait("vfio_iommu_spapr_tce");
 
+	if (pci_noiommu)
+		vfio_noiommu_init();
+
 	return 0;
 
 err_cdev_add:
@@ -1621,6 +1656,11 @@ static void __exit vfio_cleanup(void)
 {
 	WARN_ON(!list_empty(&vfio.group_list));
 
+	if (pci_noiommu) {
+		vfio_noiommu_cleanup();
+		vfio_noiommu_iommu_cleanup(&pci_bus_type);
+	}
+
 	idr_destroy(&vfio.group_idr);
 	cdev_del(&vfio.group_cdev);
 	unregister_chrdev_region(vfio.group_devt, MINORMASK);
drivers/vfio/vfio_noiommu.c new file +185 -0

185 lines not on this page · raw message

drivers/vfio/vfio_private.h new file +31 -0
new file mode 100644
index 0000000..6fe073d
@@ -0,0 +1,31 @@
+/*
+ * Copyright (C) 2015 Red Hat, Inc.  All rights reserved
+ * Author: Alex Williamson <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ */
+
+#ifndef VFIO_PRIVATE_H
+#define VFIO_PRIVATE_H
+
+#ifdef CONFIG_VFIO_NOIOMMU
+int vfio_noiommu_iommu_init(struct bus_type *bus);
+void vfio_noiommu_iommu_cleanup(struct bus_type *bus);
+int vfio_noiommu_init(void);
+void vfio_noiommu_cleanup(void);
+#else
+static inline int vfio_noiommu_iommu_init(struct bus_type *bus)
+{
+	return -ENODEV;
+}
+static inline void vfio_noiommu_iommu_cleanup(struct bus_type *bus) { }
+static inline int vfio_noiommu_init(void)
+{
+	return -ENODEV;
+}
+static inline void vfio_noiommu_cleanup(void) { }
+#endif
+#endif /* VFIO_PRIVATE_H */
+
include/uapi/linux/vfio.h +2 -0
index 9fd7b5d..c221614 100644
@@ -38,6 +38,8 @@
 
 #define VFIO_SPAPR_TCE_v2_IOMMU		7
 
+#define VFIO_NOIOMMU_IOMMU		8
+
 /*
  * The IOCTL interface is designed for extensibility by embedding the
  * structure length (argsz) and flags into structures passed between
#4 lore raw

Re: [RFC PATCH 2/2] vfio: Include no-iommu mode

Quoted 9 lines · Alex Williamson

There is really no way to safely give a user full access to a PCI without an IOMMU to protect the host from errant DMA. There is also no way to provide DMA translation, for use cases such as devices assignment to virtual machines. However, there are still those users that want userspace drivers under those conditions. The UIO driver exists for this use case, but does not provide the degree of device access and programming that VFIO has. In an effort to avoid code duplication, this introduces a No-IOMMU mode for VFIO.

Quoted 5 lines

This mode requires enabling CONFIG_VFIO_NOIOMMU and loading the vfio module with the option "enable_unsafe_pci_noiommu_mode". This should make it very clear that this mode is not safe. In this mode, there is no support for unprivileged users, CAP_SYS_ADMIN is required for access to the necessary dev files.

CAP_SYS_RAWIO seems a better match (in particular, it allows access to /dev/mem, which is the same thing).

Quoted 7 lines

Mixing no-iommu and secure VFIO is
also unsupported, as are any VFIO IOMMU backends other than the vfio-noiommu backend. Furthermore, unsafe group files are relocated to /dev/vfio-noiommu/. Upon successful loading in this mode, the kernel is tainted due to the dummy IOMMU put in place. Unloading of the module in this mode is also unsupported and will BUG due to the lack of support for unregistering an IOMMU for a bus type.

I did not see an API for detecting whether memory translation is provided or not. We can have the caller guess this by looking at the device name, or by requiring the user to specify this, but I think it's cleaner to provide programmatic access to this attribute.

#5 lore raw

Re: [RFC PATCH 2/2] vfio: Include no-iommu mode

Quoted 8 lines · Avi Kivity
Quoted 7 lines

Mixing no-iommu and secure VFIO is
also unsupported, as are any VFIO IOMMU backends other than the vfio-noiommu backend. Furthermore, unsafe group files are relocated to /dev/vfio-noiommu/. Upon successful loading in this mode, the kernel is tainted due to the dummy IOMMU put in place. Unloading of the module in this mode is also unsupported and will BUG due to the lack of support for unregistering an IOMMU for a bus type.

Quoted 4 lines

I did not see an API for detecting whether memory translation is provided or not. We can have the caller guess this by looking at the device name, or by requiring the user to specify this, but I think it's cleaner to provide programmatic access to this attribute.

It seems that caller can just check for VFIO_NOIOMMU_IOMMU.

Isn't this why it's there?

VFIO_IOMMU_MAP_DMA, VFIO_IOMMU_ENABLE and VFIO_IOMMU_DISABLE will probably also fail ...

Signature · 1 line
MST
#6 lore raw

Re: [RFC PATCH 2/2] vfio: Include no-iommu mode

Quoted 8 lines · Michael S. Tsirkin
Quoted 7 lines · Avi Kivity
Quoted 7 lines

Mixing no-iommu and secure VFIO is
also unsupported, as are any VFIO IOMMU backends other than the vfio-noiommu backend. Furthermore, unsafe group files are relocated to /dev/vfio-noiommu/. Upon successful loading in this mode, the kernel is tainted due to the dummy IOMMU put in place. Unloading of the module in this mode is also unsupported and will BUG due to the lack of support for unregistering an IOMMU for a bus type.

Quoted 7 lines
Quoted 4 lines

I did not see an API for detecting whether memory translation is provided or not. We can have the caller guess this by looking at the device name, or by requiring the user to specify this, but I think it's cleaner to provide programmatic access to this attribute.

It seems that caller can just check for VFIO_NOIOMMU_IOMMU.

Isn't this why it's there?

That's just means the capability is there, not that it's active.

But since you must pass the same value to open(), you already know that you're using noiommu.

Quoted 3 lines

VFIO_IOMMU_MAP_DMA, VFIO_IOMMU_ENABLE and VFIO_IOMMU_DISABLE will probably also fail ...

Don't you have to call MAP_DMA to pin the memory?

#7 lore raw

Re: [RFC PATCH 2/2] vfio: Include no-iommu mode

Quoted 15 lines · Avi Kivity
Quoted 12 lines · Michael S. Tsirkin
Quoted 11 lines · Avi Kivity
Quoted 7 lines

Mixing no-iommu and secure VFIO is
also unsupported, as are any VFIO IOMMU backends other than the vfio-noiommu backend. Furthermore, unsafe group files are relocated to /dev/vfio-noiommu/. Upon successful loading in this mode, the kernel is tainted due to the dummy IOMMU put in place. Unloading of the module in this mode is also unsupported and will BUG due to the lack of support for unregistering an IOMMU for a bus type.

I did not see an API for detecting whether memory translation is provided or not. We can have the caller guess this by looking at the device name, or by requiring the user to specify this, but I think it's cleaner to provide programmatic access to this attribute.

Quoted 5 lines
Quoted 3 lines

It seems that caller can just check for VFIO_NOIOMMU_IOMMU.

Isn't this why it's there?

That's just means the capability is there, not that it's active.

Well it's currently exactly the same.
I guess you can check the return value of VFIO_SET_IOMMU as well.

Quoted 8 lines

But since you must pass the same value to open(), you already know that you're using noiommu.

Quoted 3 lines

VFIO_IOMMU_MAP_DMA, VFIO_IOMMU_ENABLE and VFIO_IOMMU_DISABLE will probably also fail ...

Don't you have to call MAP_DMA to pin the memory?

Well check it out - the patch in question doesn't implement this ioctl. In fact it doesn't implement anything except CHECK_EXTENSION.

And this makes sense to me: MAP_DMA
maps a virtual address to io address, and that doesn't work for the dummy iommu.

You can pin memory using many other ways, including mlock and hugetlbfs.

Signature · 1 line
MST
#8 lore raw

Re: [RFC PATCH 2/2] vfio: Include no-iommu mode

Quoted 12 lines · Michael S. Tsirkin
Quoted 8 lines

But since you must pass the same value to open(), you already know that you're using noiommu.

Quoted 3 lines

VFIO_IOMMU_MAP_DMA, VFIO_IOMMU_ENABLE and VFIO_IOMMU_DISABLE will probably also fail ...

Don't you have to call MAP_DMA to pin the memory?

Well check it out - the patch in question doesn't implement this ioctl. In fact it doesn't implement anything except CHECK_EXTENSION.

Quoted 7 lines

And this makes sense to me: MAP_DMA
maps a virtual address to io address, and that doesn't work for the dummy iommu.

You can pin memory using many other ways, including mlock and hugetlbfs.

mlock() does not pin memory.

--
Gleb.

#9 lore raw

Re: [RFC PATCH 2/2] vfio: Include no-iommu mode

In reply to Avi Kivity
Quoted 11 lines · Avi Kivity
Quoted 9 lines · Alex Williamson

There is really no way to safely give a user full access to a PCI without an IOMMU to protect the host from errant DMA. There is also no way to provide DMA translation, for use cases such as devices assignment to virtual machines. However, there are still those users that want userspace drivers under those conditions. The UIO driver exists for this use case, but does not provide the degree of device access and programming that VFIO has. In an effort to avoid code duplication, this introduces a No-IOMMU mode for VFIO.

Quoted 8 lines
Quoted 5 lines

This mode requires enabling CONFIG_VFIO_NOIOMMU and loading the vfio module with the option "enable_unsafe_pci_noiommu_mode". This should make it very clear that this mode is not safe. In this mode, there is no support for unprivileged users, CAP_SYS_ADMIN is required for access to the necessary dev files.

CAP_SYS_RAWIO seems a better match (in particular, it allows access to /dev/mem, which is the same thing).

Sure, that seems reasonable.

Quoted 8 lines
Quoted 7 lines

Mixing no-iommu and secure VFIO is
also unsupported, as are any VFIO IOMMU backends other than the vfio-noiommu backend. Furthermore, unsafe group files are relocated to /dev/vfio-noiommu/. Upon successful loading in this mode, the kernel is tainted due to the dummy IOMMU put in place. Unloading of the module in this mode is also unsupported and will BUG due to the lack of support for unregistering an IOMMU for a bus type.

Quoted 4 lines

I did not see an API for detecting whether memory translation is provided or not. We can have the caller guess this by looking at the device name, or by requiring the user to specify this, but I think it's cleaner to provide programmatic access to this attribute.

The VFIO user can probe and needs to set the IOMMU model in use before they can access a device file descriptor. In this mode, the VFIO_NOIOMMU_IOMMU is the only model available, which as proposed here provides no translation, and in fact no mapping ioctls. Thanks,

Alex

#10 lore raw

Re: [RFC PATCH 2/2] vfio: Include no-iommu mode

In reply to Alex Williamson
Quoted 21 lines · On Fri, 09 Oct 2015 12:41:10 -0600 Alex Williamson…

There is really no way to safely give a user full access to a PCI without an IOMMU to protect the host from errant DMA. There is also no way to provide DMA translation, for use cases such as devices assignment to virtual machines. However, there are still those users that want userspace drivers under those conditions. The UIO driver exists for this use case, but does not provide the degree of device access and programming that VFIO has. In an effort to avoid code duplication, this introduces a No-IOMMU mode for VFIO.

This mode requires enabling CONFIG_VFIO_NOIOMMU and loading the vfio module with the option "enable_unsafe_pci_noiommu_mode". This should make it very clear that this mode is not safe. In this mode, there is no support for unprivileged users, CAP_SYS_ADMIN is required for access to the necessary dev files. Mixing no-iommu and secure VFIO is also unsupported, as are any VFIO IOMMU backends other than the vfio-noiommu backend. Furthermore, unsafe group files are relocated to /dev/vfio-noiommu/. Upon successful loading in this mode, the kernel is tainted due to the dummy IOMMU put in place. Unloading of the module in this mode is also unsupported and will BUG due to the lack of support for unregistering an IOMMU for a bus type.

Quoted 1 line

Will this work for distro's where chaning kernel command line options is really not that practical. We need to boot with one command line and then decide to use IOMMU (or not) later on during the service startup of the dataplane application. Recent experience is that IOMMU's are broken on many platforms so the only way to make a DPDK application it to write a test program that can be used to check if VFIO+IOMMU works first.

Also, although you think the long option will set the bar high enough it probably will not satisfy anyone. It is annoying enough, that I would just carry a patch to remove it the silly requirement. And the the people who believe
all user mode DMA is evil won't be satisfied either.

But I really like having the same consistent API for handling device access with IOMMU and when IOMMU will/won't work.

#11 lore raw

Re: [RFC PATCH 2/2] vfio: Include no-iommu mode

Quoted 26 lines · Stephen Hemminger
Quoted 21 lines · On Fri, 09 Oct 2015 12:41:10 -0600 Alex Williamson…

There is really no way to safely give a user full access to a PCI without an IOMMU to protect the host from errant DMA. There is also no way to provide DMA translation, for use cases such as devices assignment to virtual machines. However, there are still those users that want userspace drivers under those conditions. The UIO driver exists for this use case, but does not provide the degree of device access and programming that VFIO has. In an effort to avoid code duplication, this introduces a No-IOMMU mode for VFIO.

This mode requires enabling CONFIG_VFIO_NOIOMMU and loading the vfio module with the option "enable_unsafe_pci_noiommu_mode". This should make it very clear that this mode is not safe. In this mode, there is no support for unprivileged users, CAP_SYS_ADMIN is required for access to the necessary dev files. Mixing no-iommu and secure VFIO is also unsupported, as are any VFIO IOMMU backends other than the vfio-noiommu backend. Furthermore, unsafe group files are relocated to /dev/vfio-noiommu/. Upon successful loading in this mode, the kernel is tainted due to the dummy IOMMU put in place. Unloading of the module in this mode is also unsupported and will BUG due to the lack of support for unregistering an IOMMU for a bus type.

Quoted 1 line
Quoted 7 lines

Will this work for distro's where chaning kernel command line options is really not that practical. We need to boot with one command line and then decide to use IOMMU (or not) later on during the service startup of the dataplane application. Recent experience is that IOMMU's are broken on many platforms so the only way to make a DPDK application it to write a test program that can be used to check if VFIO+IOMMU works first.

There's no kernel command line dependency, if you find that VFIO+IOMMU doesn't work, unload the vfio modules and reload with the no-iommu option and try again. VFIO itself cannot simply fall back to no-iommu, that definitely needs to be a user opt-in. The userspace app though should easily be able to tell when the type1 model is not available and fall back to no-iommu.

Quoted 5 lines

Also, although you think the long option will set the bar high enough it probably will not satisfy anyone. It is annoying enough, that I would just carry a patch to remove it the silly requirement. And the the people who believe
all user mode DMA is evil won't be satisfied either.

I find that many users blindly follow howtos and only sometimes do they question the options if they sound scary enough. So yeah, I would intend to make the option upstream sound scary enough for people to think twice about using it and maybe even read the description. That still doesn't prevent pasting it into modprobe.d and forgetting about it.

I don't see non-IOMMU protected usermode DMA as evil, it's just unsupportable and I want to be able to immediately know that it's a possibility if I'm looking at a kernel bug.

Quoted 2 lines

But I really like having the same consistent API for handling device access with IOMMU and when IOMMU will/won't work.

We still need to hear from IOMMU folks, AIUI there are plans to implement dma_iops via iommu_ops, which would make squatting on iommu_ops much more difficult for a hack like this. Thanks,

Alex

#12 lore raw

Re: [RFC PATCH 2/2] vfio: Include no-iommu mode

Quoted 11 lines · Alex Williamson
Quoted 5 lines

Also, although you think the long option will set the bar high enough it probably will not satisfy anyone. It is annoying enough, that I would just carry a patch to remove it the silly requirement. And the the people who believe
all user mode DMA is evil won't be satisfied either.

I find that many users blindly follow howtos and only sometimes do they question the options if they sound scary enough. So yeah, I would intend to make the option upstream sound scary enough for people to think twice about using it and maybe even read the description. That still doesn't prevent pasting it into modprobe.d and forgetting about it.

I think we need to allow for packages to work with this. I.e. drop files into config directories rather than require editing of config files. I think this is mostly workable via modprobe.d.

(for our own use case, we don't require the extreme performance of many L2/L3 dpdk apps so we'll work with regular iommued vfio for bare-metal installations and ship prebuilt virtual machine images for clouds, so it doesn't matter that much to me).

#13 lore raw

Re: [RFC PATCH 2/2] vfio: Include no-iommu mode

In reply to Stephen Hemminger
Quoted 24 lines · Stephen Hemminger
Quoted 21 lines · On Fri, 09 Oct 2015 12:41:10 -0600 Alex Williamson…

There is really no way to safely give a user full access to a PCI without an IOMMU to protect the host from errant DMA. There is also no way to provide DMA translation, for use cases such as devices assignment to virtual machines. However, there are still those users that want userspace drivers under those conditions. The UIO driver exists for this use case, but does not provide the degree of device access and programming that VFIO has. In an effort to avoid code duplication, this introduces a No-IOMMU mode for VFIO.

This mode requires enabling CONFIG_VFIO_NOIOMMU and loading the vfio module with the option "enable_unsafe_pci_noiommu_mode". This should make it very clear that this mode is not safe. In this mode, there is no support for unprivileged users, CAP_SYS_ADMIN is required for access to the necessary dev files. Mixing no-iommu and secure VFIO is also unsupported, as are any VFIO IOMMU backends other than the vfio-noiommu backend. Furthermore, unsafe group files are relocated to /dev/vfio-noiommu/. Upon successful loading in this mode, the kernel is tainted due to the dummy IOMMU put in place. Unloading of the module in this mode is also unsupported and will BUG due to the lack of support for unregistering an IOMMU for a bus type.

Quoted 6 lines
Quoted 1 line

Will this work for distro's where chaning kernel command line options is really not that practical. We need to boot with one command line and then decide to use IOMMU (or not) later on during the service startup of the dataplane application.

On open? That's too late in my opinion. But maybe the flag can be tweaked so that it will probe for iommu, if there - do the right thing, but if that fails, enable the dummy one. And maybe defer tainting until device open.

Won't address the "old IOMMUs add performance overhead" usecase but I'm not very impressed by that in any case.

Quoted 4 lines

Recent experience is that IOMMU's
are broken on many platforms so the only way to make a DPDK application it to write a test program that can be used to check if VFIO+IOMMU works first.

In userspace? Well that's just piling up work-arounds. And assuming hardware is broken, who knows what's going on security-wise. These broken systems need to be identified and black-listed in kernel.

Quoted 3 lines

Also, although you think the long option will set the bar high enough it probably will not satisfy anyone. It is annoying enough, that I would just carry a patch to remove it the silly requirement.

That sounds reasonable. Anyone who can carry a kernel patch does not need the warning.

Quoted 4 lines

And the the people who believe
all user mode DMA is evil won't be satisfied either.
But I really like having the same consistent API for handling device access with IOMMU and when IOMMU will/won't work.

I agree that's good. Makes it easier to migrate applications to the safe configuration down the road. Thanks Alex!

Signature · 1 line
mST
#14 lore raw

Re: [RFC PATCH 2/2] vfio: Include no-iommu mode

Quoted 32 lines · Michael S. Tsirkin
Quoted 24 lines · Stephen Hemminger
Quoted 21 lines · On Fri, 09 Oct 2015 12:41:10 -0600 Alex Williamson…

There is really no way to safely give a user full access to a PCI without an IOMMU to protect the host from errant DMA. There is also no way to provide DMA translation, for use cases such as devices assignment to virtual machines. However, there are still those users that want userspace drivers under those conditions. The UIO driver exists for this use case, but does not provide the degree of device access and programming that VFIO has. In an effort to avoid code duplication, this introduces a No-IOMMU mode for VFIO.

This mode requires enabling CONFIG_VFIO_NOIOMMU and loading the vfio module with the option "enable_unsafe_pci_noiommu_mode". This should make it very clear that this mode is not safe. In this mode, there is no support for unprivileged users, CAP_SYS_ADMIN is required for access to the necessary dev files. Mixing no-iommu and secure VFIO is also unsupported, as are any VFIO IOMMU backends other than the vfio-noiommu backend. Furthermore, unsafe group files are relocated to /dev/vfio-noiommu/. Upon successful loading in this mode, the kernel is tainted due to the dummy IOMMU put in place. Unloading of the module in this mode is also unsupported and will BUG due to the lack of support for unregistering an IOMMU for a bus type.

Quoted 6 lines
Quoted 1 line

Will this work for distro's where chaning kernel command line options is really not that practical. We need to boot with one command line and then decide to use IOMMU (or not) later on during the service startup of the dataplane application.

Quoted 4 lines

On open? That's too late in my opinion. But maybe the flag can be tweaked so that it will probe for iommu, if there - do the right thing, but if that fails, enable the dummy one. And maybe defer tainting until device open.

The vfio mechanics are that a vfio bus driver, such as vfio-pci binds to a device. In the probe function, we check for an iommu group, which vfio-core then uses to create the vfio group. So there's nothing to open(), the iommu association needs to be made prior to even binding the device to vfio-pci. Probing for an iommu can also only be done on a per bus_type basis, which will likely eventually become a per bus instance to support heterogeneous iommus, so vfio can't simply determine that an iommu is not present globally. This is why the new module option includes the word "pci", so that it can probe for and attach the dummy iommu specifically on the pci_bus_type.

We can still consider if there are better points at which to initiate the fake iommu group. Trying to think through vfio-pci doing it on probe(), but it seems pretty ugly.

In this RFC, I specifically avoided making the vfio no-iommu iommu driver just another modular iommu backend, I wanted it to be tied to a vfio module option such that vfio behaves differently with open()s and certain ioctls. I think it would be really confusing to users if safe and unsafe modes could be used concurrently for different devices.

Quoted 2 lines

Won't address the "old IOMMUs add performance overhead" usecase but I'm not very impressed by that in any case.

Yep, me neither, certainly not for static mappings. There's a lot of FUD left over from latencies in the streaming DMA mapping paths where mappings are created and destroyed at a high rate. That has more to do with flushing mappings out of the hardware than with iotlb miss latency or actual translation, which is all that should be in play for most uses here.

Quoted 24 lines
Quoted 4 lines

Recent experience is that IOMMU's
are broken on many platforms so the only way to make a DPDK application it to write a test program that can be used to check if VFIO+IOMMU works first.

In userspace? Well that's just piling up work-arounds. And assuming hardware is broken, who knows what's going on security-wise. These broken systems need to be identified and black-listed in kernel.

Quoted 3 lines

Also, although you think the long option will set the bar high enough it probably will not satisfy anyone. It is annoying enough, that I would just carry a patch to remove it the silly requirement.

That sounds reasonable. Anyone who can carry a kernel patch does not need the warning.

Quoted 4 lines

And the the people who believe
all user mode DMA is evil won't be satisfied either.
But I really like having the same consistent API for handling device access with IOMMU and when IOMMU will/won't work.

I agree that's good. Makes it easier to migrate applications to the safe configuration down the road. Thanks Alex!

#15 lore raw

Re: [RFC PATCH 2/2] vfio: Include no-iommu mode

Quoted 33 lines · Alex Williamson
Quoted 32 lines · Michael S. Tsirkin
Quoted 30 lines · Stephen Hemminger
Quoted 21 lines · On Fri, 09 Oct 2015 12:41:10 -0600 Alex Williamson…

There is really no way to safely give a user full access to a PCI without an IOMMU to protect the host from errant DMA. There is also no way to provide DMA translation, for use cases such as devices assignment to virtual machines. However, there are still those users that want userspace drivers under those conditions. The UIO driver exists for this use case, but does not provide the degree of device access and programming that VFIO has. In an effort to avoid code duplication, this introduces a No-IOMMU mode for VFIO.

This mode requires enabling CONFIG_VFIO_NOIOMMU and loading the vfio module with the option "enable_unsafe_pci_noiommu_mode". This should make it very clear that this mode is not safe. In this mode, there is no support for unprivileged users, CAP_SYS_ADMIN is required for access to the necessary dev files. Mixing no-iommu and secure VFIO is also unsupported, as are any VFIO IOMMU backends other than the vfio-noiommu backend. Furthermore, unsafe group files are relocated to /dev/vfio-noiommu/. Upon successful loading in this mode, the kernel is tainted due to the dummy IOMMU put in place. Unloading of the module in this mode is also unsupported and will BUG due to the lack of support for unregistering an IOMMU for a bus type.

Quoted 1 line

Will this work for distro's where chaning kernel command line options is really not that practical. We need to boot with one command line and then decide to use IOMMU (or not) later on during the service startup of the dataplane application.

Quoted 4 lines
Quoted 4 lines

On open? That's too late in my opinion. But maybe the flag can be tweaked so that it will probe for iommu, if there - do the right thing, but if that fails, enable the dummy one. And maybe defer tainting until device open.

I forgot to address the tainting point; I think we were previously talking about tainting at the point where bus master is enabled, but I chose to do it much earlier here because the act of registering a dummy iommu_ops for a bus type is pretty much the point at which we have a good chance of breaking the system. I also considered that some devices can manipulate their config space registers using device specific registers (such as the example of GPUs that mirror config space in mmio). It's therefore not always possible to taint at the point where we think the user has done something bad. The best case would be tainting at the point where the device file descriptor is opened as you suggested, but we can't do that while we're exposing dummy iommu_ops to the whole bus type. Maybe another option would be to create vfio wrappers for the iommu callbacks to have the iommu facade more local to vfio.

My main requirements are that I do not want be disruptive to the existing vfio code or add a significant amount of code that needs to be maintained for the purpose of supporting a use mode that we don't really think is supportable. Thanks,

Alex

Quoted 63 lines

The vfio mechanics are that a vfio bus driver, such as vfio-pci binds to a device. In the probe function, we check for an iommu group, which vfio-core then uses to create the vfio group. So there's nothing to open(), the iommu association needs to be made prior to even binding the device to vfio-pci. Probing for an iommu can also only be done on a per bus_type basis, which will likely eventually become a per bus instance to support heterogeneous iommus, so vfio can't simply determine that an iommu is not present globally. This is why the new module option includes the word "pci", so that it can probe for and attach the dummy iommu specifically on the pci_bus_type.

We can still consider if there are better points at which to initiate the fake iommu group. Trying to think through vfio-pci doing it on probe(), but it seems pretty ugly.

In this RFC, I specifically avoided making the vfio no-iommu iommu driver just another modular iommu backend, I wanted it to be tied to a vfio module option such that vfio behaves differently with open()s and certain ioctls. I think it would be really confusing to users if safe and unsafe modes could be used concurrently for different devices.

Quoted 2 lines

Won't address the "old IOMMUs add performance overhead" usecase but I'm not very impressed by that in any case.

Yep, me neither, certainly not for static mappings. There's a lot of FUD left over from latencies in the streaming DMA mapping paths where mappings are created and destroyed at a high rate. That has more to do with flushing mappings out of the hardware than with iotlb miss latency or actual translation, which is all that should be in play for most uses here.

Quoted 16 lines
Quoted 4 lines

Recent experience is that IOMMU's
are broken on many platforms so the only way to make a DPDK application it to write a test program that can be used to check if VFIO+IOMMU works first.

In userspace? Well that's just piling up work-arounds. And assuming hardware is broken, who knows what's going on security-wise. These broken systems need to be identified and black-listed in kernel.

Quoted 3 lines

Also, although you think the long option will set the bar high enough it probably will not satisfy anyone. It is annoying enough, that I would just carry a patch to remove it the silly requirement.

That sounds reasonable. Anyone who can carry a kernel patch does not need the warning.

Quoted 8 lines
Quoted 4 lines

And the the people who believe
all user mode DMA is evil won't be satisfied either.
But I really like having the same consistent API for handling device access with IOMMU and when IOMMU will/won't work.

I agree that's good. Makes it easier to migrate applications to the safe configuration down the road. Thanks Alex!

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#16 lore raw
In reply to Alex Williamson

Hi Alex,
Thanks for the patch Alex. This would also require support in Qemu to expose the physical address to the VM. Are you looking at that part as well?

Regards
Varun

-----Original Message-----
From: iommu-bounces-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org [mailto:iommu-bounces-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org] On Behalf Of Alex Williamson
Sent: Saturday, October 10, 2015 12:11 AM
To: [email protected]
Cc: avi-VrcmuVmyx1hWk0Htik3J/[email protected]; avi-RmZWMc9puTNJc61us3aD9laTQe2KTcn/@public.gmane.org; gleb-VrcmuVmyx1hWk0Htik3J/[email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; gleb-RmZWMc9puTNJc61us3aD9laTQe2KTcn/@public.gmane.org; [email protected]; vladz-RmZWMc9puTNJc61us3aD9laTQe2KTcn/@public.gmane.org; iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org; hjk-vqZO0P4V72/[email protected]; [email protected]
Subject: [RFC PATCH 0/2] VFIO no-iommu

Recent patches for UIO have been attempting to add MSI/X support, which unfortunately implies DMA support, which users have been enabling anyway, but was never intended for UIO. VFIO on the other hand expects an IOMMU to provide isolation of devices, but provides a much more complete device interface, which already supports full MSI/X support. There's really no way to support userspace drivers with DMA capable devices without an IOMMU to protect the host, but we can at least think about doing it in a way that properly taints the kernel and avoids creating new code duplicating existing code, that does have a supportable use case.

The diffstat is only so large because I moved vfio.c to vfio_core.c so I could more easily keep the module named vfio.ko while keeping the bulk of the no-iommu support in a separate file that can be optionally compiled. We're really looking at a couple hundred lines of mostly stub code. The VFIO_NOIOMMU_IOMMU could certainly be expanded to do page pinning and virt_to_bus() translation, but I didn't want to complicate anything yet.

I've only compiled this and tested loading the module with the new no-iommu mode enabled, I haven't actually tried to port a DPDK driver to it, though it ought to be a pretty obvious mix of the existing UIO and VFIO versions (set the IOMMU, but avoid using it for mapping, use however bus translations are done w/ UIO). The core vfio device file is still /dev/vfio/vfio, but all the groups become /dev/vfio-noiommu/$GROUP.

It should be obvious, but I always feel obligated to state that this does not and will not ever enable device assignment to virtual machines on non-IOMMU capable platforms.

I'm curious what IOMMU folks think of this. This hack is really only possible because we don't use iommu_ops for regular DMA, so we can hijack it fairly safely. I believe that's intended to change though, so this may not be practical long term. Thanks,

Alex

---

Alex Williamson (2):
vfio: Move vfio.c vfio_core.c
vfio: Include no-iommu mode

 drivers/vfio/Kconfig        |   15 
 drivers/vfio/Makefile       |    4 
 drivers/vfio/vfio.c         | 1640 ------------------------------------------
 drivers/vfio/vfio_core.c    | 1680 +++++++++++++++++++++++++++++++++++++++++++
 drivers/vfio/vfio_noiommu.c |  185 +++++
 drivers/vfio/vfio_private.h |   31 +
 include/uapi/linux/vfio.h   |    2 
 7 files changed, 1917 insertions(+), 1640 deletions(-)  delete mode 100644 drivers/vfio/vfio.c  create mode 100644 drivers/vfio/vfio_core.c  create mode 100644 drivers/vfio/vfio_noiommu.c  create mode 100644 drivers/vfio/vfio_private.h _______________________________________________

iommu mailing list iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

#17 lore raw
Quoted 2 lines · Varun Sethi

Hi Alex,
Thanks for the patch Alex. This would also require support in Qemu to expose the physical address to the VM. Are you looking at that part as well?

Quoting from below:

It should be obvious, but I always feel obligated to state that this does not and will not ever enable device assignment to virtual machines on non-IOMMU capable platforms.

I have no intention whatsoever, nor would I encourage or support use of this mode for assignment to a virtual machine. VFIO has uses beyond QEMU. Thanks,

Alex

Quoted 37 lines

-----Original Message-----
From: iommu-bounces-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org [mailto:iommu-bounces-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org] On Behalf Of Alex Williamson
Sent: Saturday, October 10, 2015 12:11 AM
To: [email protected]
Cc: avi-VrcmuVmyx1hWk0Htik3J/[email protected]; avi-RmZWMc9puTNJc61us3aD9laTQe2KTcn/@public.gmane.org; gleb-VrcmuVmyx1hWk0Htik3J/[email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; gleb-RmZWMc9puTNJc61us3aD9laTQe2KTcn/@public.gmane.org; [email protected]; vladz-RmZWMc9puTNJc61us3aD9laTQe2KTcn/@public.gmane.org; iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org; hjk-vqZO0P4V72/[email protected]; [email protected]
Subject: [RFC PATCH 0/2] VFIO no-iommu

Recent patches for UIO have been attempting to add MSI/X support, which unfortunately implies DMA support, which users have been enabling anyway, but was never intended for UIO. VFIO on the other hand expects an IOMMU to provide isolation of devices, but provides a much more complete device interface, which already supports full MSI/X support. There's really no way to support userspace drivers with DMA capable devices without an IOMMU to protect the host, but we can at least think about doing it in a way that properly taints the kernel and avoids creating new code duplicating existing code, that does have a supportable use case.

The diffstat is only so large because I moved vfio.c to vfio_core.c so I could more easily keep the module named vfio.ko while keeping the bulk of the no-iommu support in a separate file that can be optionally compiled. We're really looking at a couple hundred lines of mostly stub code. The VFIO_NOIOMMU_IOMMU could certainly be expanded to do page pinning and virt_to_bus() translation, but I didn't want to complicate anything yet.

I've only compiled this and tested loading the module with the new no-iommu mode enabled, I haven't actually tried to port a DPDK driver to it, though it ought to be a pretty obvious mix of the existing UIO and VFIO versions (set the IOMMU, but avoid using it for mapping, use however bus translations are done w/ UIO). The core vfio device file is still /dev/vfio/vfio, but all the groups become /dev/vfio-noiommu/$GROUP.

It should be obvious, but I always feel obligated to state that this does not and will not ever enable device assignment to virtual machines on non-IOMMU capable platforms.

I'm curious what IOMMU folks think of this. This hack is really only possible because we don't use iommu_ops for regular DMA, so we can hijack it fairly safely. I believe that's intended to change though, so this may not be practical long term. Thanks,

Alex

---

Alex Williamson (2):
vfio: Move vfio.c vfio_core.c
vfio: Include no-iommu mode

 drivers/vfio/Kconfig        |   15 
 drivers/vfio/Makefile       |    4 
 drivers/vfio/vfio.c         | 1640 ------------------------------------------
 drivers/vfio/vfio_core.c    | 1680 +++++++++++++++++++++++++++++++++++++++++++
 drivers/vfio/vfio_noiommu.c |  185 +++++
 drivers/vfio/vfio_private.h |   31 +
 include/uapi/linux/vfio.h   |    2 
 7 files changed, 1917 insertions(+), 1640 deletions(-)  delete mode 100644 drivers/vfio/vfio.c  create mode 100644 drivers/vfio/vfio_core.c  create mode 100644 drivers/vfio/vfio_noiommu.c  create mode 100644 drivers/vfio/vfio_private.h _______________________________________________

iommu mailing list iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

#18 lore raw
In reply to Alex Williamson
Quoted 11 lines · Alex Williamson

Recent patches for UIO have been attempting to add MSI/X support, which unfortunately implies DMA support, which users have been enabling anyway, but was never intended for UIO. VFIO on the other hand expects an IOMMU to provide isolation of devices, but provides a much more complete device interface, which already supports full MSI/X support. There's really no way to support userspace drivers with DMA capable devices without an IOMMU to protect the host, but we can at least think about doing it in a way that properly taints the kernel and avoids creating new code duplicating existing code, that does have a supportable use case.

Quoted 7 lines

The diffstat is only so large because I moved vfio.c to vfio_core.c so I could more easily keep the module named vfio.ko while keeping the bulk of the no-iommu support in a separate file that can be optionally compiled. We're really looking at a couple hundred lines of mostly stub code. The VFIO_NOIOMMU_IOMMU could certainly be expanded to do page pinning and virt_to_bus() translation, but I didn't want to complicate anything yet.

I think it's already useful like this, since all current users seem happy enough to just use hugetlbfs to do pinning, and ignore translation.

Quoted 11 lines

I've only compiled this and tested loading the module with the new no-iommu mode enabled, I haven't actually tried to port a DPDK driver to it, though it ought to be a pretty obvious mix of the existing UIO and VFIO versions (set the IOMMU, but avoid using it for mapping, use however bus translations are done w/ UIO). The core vfio device file is still /dev/vfio/vfio, but all the groups become /dev/vfio-noiommu/$GROUP.

It should be obvious, but I always feel obligated to state that this does not and will not ever enable device assignment to virtual machines on non-IOMMU capable platforms.

In theory, it's kind of possible using paravirtualization.

Within guest, you'd make map_page retrieve the io address from the host and return that as dma_addr_t. The only question would be APIs that require more than one contigious page in IO space (e.g. I think alloc coherent is like this?).
Not a problem if host is using hugetlbfs, but if not, I guess we could add a hypercall and some Linux API on the host to trigger compaction on the host aggressively. MADV_CONTIGIOUS?

Quoted 26 lines

I'm curious what IOMMU folks think of this. This hack is really only possible because we don't use iommu_ops for regular DMA, so we can hijack it fairly safely. I believe that's intended to change though, so this may not be practical long term. Thanks,

Alex

---

Alex Williamson (2):
vfio: Move vfio.c vfio_core.c
vfio: Include no-iommu mode

 drivers/vfio/Kconfig        |   15 
 drivers/vfio/Makefile       |    4 
 drivers/vfio/vfio.c         | 1640 ------------------------------------------
 drivers/vfio/vfio_core.c    | 1680 +++++++++++++++++++++++++++++++++++++++++++
 drivers/vfio/vfio_noiommu.c |  185 +++++
 drivers/vfio/vfio_private.h |   31 +
 include/uapi/linux/vfio.h   |    2 
 7 files changed, 1917 insertions(+), 1640 deletions(-)
 delete mode 100644 drivers/vfio/vfio.c
 create mode 100644 drivers/vfio/vfio_core.c
 create mode 100644 drivers/vfio/vfio_noiommu.c
 create mode 100644 drivers/vfio/vfio_private.h
#19 lore raw
Quoted 38 lines · Michael S. Tsirkin
Quoted 11 lines · Alex Williamson

Recent patches for UIO have been attempting to add MSI/X support, which unfortunately implies DMA support, which users have been enabling anyway, but was never intended for UIO. VFIO on the other hand expects an IOMMU to provide isolation of devices, but provides a much more complete device interface, which already supports full MSI/X support. There's really no way to support userspace drivers with DMA capable devices without an IOMMU to protect the host, but we can at least think about doing it in a way that properly taints the kernel and avoids creating new code duplicating existing code, that does have a supportable use case.

Quoted 7 lines

The diffstat is only so large because I moved vfio.c to vfio_core.c so I could more easily keep the module named vfio.ko while keeping the bulk of the no-iommu support in a separate file that can be optionally compiled. We're really looking at a couple hundred lines of mostly stub code. The VFIO_NOIOMMU_IOMMU could certainly be expanded to do page pinning and virt_to_bus() translation, but I didn't want to complicate anything yet.

I think it's already useful like this, since all current users seem happy enough to just use hugetlbfs to do pinning, and ignore translation.

Quoted 11 lines

I've only compiled this and tested loading the module with the new no-iommu mode enabled, I haven't actually tried to port a DPDK driver to it, though it ought to be a pretty obvious mix of the existing UIO and VFIO versions (set the IOMMU, but avoid using it for mapping, use however bus translations are done w/ UIO). The core vfio device file is still /dev/vfio/vfio, but all the groups become /dev/vfio-noiommu/$GROUP.

It should be obvious, but I always feel obligated to state that this does not and will not ever enable device assignment to virtual machines on non-IOMMU capable platforms.

In theory, it's kind of possible using paravirtualization.

Quoted 7 lines

Within guest, you'd make map_page retrieve the io address from the host and return that as dma_addr_t. The only question would be APIs that require more than one contigious page in IO space (e.g. I think alloc coherent is like this?).
Not a problem if host is using hugetlbfs, but if not, I guess we could add a hypercall and some Linux API on the host to trigger compaction on the host aggressively. MADV_CONTIGIOUS?

Not that I see a good reason for that.
Just use an iommu.

Quoted 27 lines
Quoted 27 lines

I'm curious what IOMMU folks think of this. This hack is really only possible because we don't use iommu_ops for regular DMA, so we can hijack it fairly safely. I believe that's intended to change though, so this may not be practical long term. Thanks,

Alex

---

Alex Williamson (2):
vfio: Move vfio.c vfio_core.c
vfio: Include no-iommu mode

 drivers/vfio/Kconfig        |   15 
 drivers/vfio/Makefile       |    4 
 drivers/vfio/vfio.c         | 1640 ------------------------------------------
 drivers/vfio/vfio_core.c    | 1680 +++++++++++++++++++++++++++++++++++++++++++
 drivers/vfio/vfio_noiommu.c |  185 +++++
 drivers/vfio/vfio_private.h |   31 +
 include/uapi/linux/vfio.h   |    2 
 7 files changed, 1917 insertions(+), 1640 deletions(-)
 delete mode 100644 drivers/vfio/vfio.c
 create mode 100644 drivers/vfio/vfio_core.c
 create mode 100644 drivers/vfio/vfio_noiommu.c
 create mode 100644 drivers/vfio/vfio_private.h
#20 lore raw
Quoted 21 lines · Michael S. Tsirkin
Quoted 20 lines · Michael S. Tsirkin
Quoted 18 lines · Alex Williamson

Recent patches for UIO have been attempting to add MSI/X support, which unfortunately implies DMA support, which users have been enabling anyway, but was never intended for UIO. VFIO on the other hand expects an IOMMU to provide isolation of devices, but provides a much more complete device interface, which already supports full MSI/X support. There's really no way to support userspace drivers with DMA capable devices without an IOMMU to protect the host, but we can at least think about doing it in a way that properly taints the kernel and avoids creating new code duplicating existing code, that does have a supportable use case.

The diffstat is only so large because I moved vfio.c to vfio_core.c so I could more easily keep the module named vfio.ko while keeping the bulk of the no-iommu support in a separate file that can be optionally compiled. We're really looking at a couple hundred lines of mostly stub code. The VFIO_NOIOMMU_IOMMU could certainly be expanded to do page pinning and virt_to_bus() translation, but I didn't want to complicate anything yet.

Quoted 3 lines
Quoted 3 lines

I think it's already useful like this, since all current users seem happy enough to just use hugetlbfs to do pinning, and ignore translation.

That was sort of my thought too...

Quoted 22 lines
Quoted 14 lines
Quoted 11 lines

I've only compiled this and tested loading the module with the new no-iommu mode enabled, I haven't actually tried to port a DPDK driver to it, though it ought to be a pretty obvious mix of the existing UIO and VFIO versions (set the IOMMU, but avoid using it for mapping, use however bus translations are done w/ UIO). The core vfio device file is still /dev/vfio/vfio, but all the groups become /dev/vfio-noiommu/$GROUP.

It should be obvious, but I always feel obligated to state that this does not and will not ever enable device assignment to virtual machines on non-IOMMU capable platforms.

In theory, it's kind of possible using paravirtualization.

Quoted 7 lines

Within guest, you'd make map_page retrieve the io address from the host and return that as dma_addr_t. The only question would be APIs that require more than one contigious page in IO space (e.g. I think alloc coherent is like this?).
Not a problem if host is using hugetlbfs, but if not, I guess we could add a hypercall and some Linux API on the host to trigger compaction on the host aggressively. MADV_CONTIGIOUS?

Quoted 2 lines

Not that I see a good reason for that.
Just use an iommu.

Right, I think it boils down to how much code are we willing to maintain for an interface that we consider so dangerous and unsupportable that we immediately taint the kernel. This is partially why I stopped short of expanding the no-iommu interface to do page pinning or virt-to-bus translation. A few hundred lines of boilerplate stubs to enable re-use of code that is maintained for supportable interfaces is one thing. Building onto that with paravirtual IOMMU interfaces to a VM for something that ultimately cannot be supported is not something I want to participate in. Get an IOMMU. Thanks,

Alex