iommu: iommu_ops group interface
Subsystems (4)
cover iommu: iommu_ops group interface
IOMMUs can't always distiguish transactions from each individual device in a system. Sometimes this is by design (such as powerpc partitionable endpoints), other times by topology (PCIe-to-PCI bridges masking downstream devices). We call these sets of indistinguishable devices "groups".
In order to support secure userspace drivers, like vfio, we need an interface to expose the device-to-group relationship. This allows us to create policies ensuring that userspace controls all of the devices in the group before allowing individual device access.
This series implements the iommu_ops API interface and sysfs interface for exposing groups to userspace. This also includes the intel-iommu and amd-iommu backend implementations. It's intended that the vfio driver will make use of these interfaces to support generic device assignment for virtual machines. Seegit://github.com/awilliam/linux-vfio.git (vfio-ng) for a working example using this interface.
Patches based on Joerg's next branch to support per-bus iommu_ops.
Note the amd-iommu is untested, I'm still working on setting up an AMD-Vi capable system. Thanks,
Alex
---
Alex Williamson (4):
iommu: Add option to group multi-function devices
amd-iommu: Implement iommu_device_group
intel-iommu: Implement iommu_device_group
iommu: Add iommu_device_group callback and iommu_group sysfs entry
Documentation/kernel-parameters.txt | 4 ++ arch/ia64/include/asm/iommu.h | 2 + arch/ia64/kernel/pci-dma.c | 1 + arch/x86/include/asm/iommu.h | 1 + arch/x86/kernel/pci-dma.c | 11 ++++++ drivers/iommu/amd_iommu.c | 21 ++++++++++++ drivers/iommu/intel-iommu.c | 47 +++++++++++++++++++++++++++ drivers/iommu/iommu.c | 61 +++++++++++++++++++++++++++++++++++ include/linux/iommu.h | 7 ++++ 9 files changed, 154 insertions(+), 1 deletions(-)
PATCH 1/4 iommu: Add iommu_device_group callback and iommu_group sysfs entry
An IOMMU group is a set of devices for which the IOMMU cannot distinguish transactions. For PCI devices, a group often occurs when a PCI bridge is involved. Transactions from any device behind the bridge appear to be sourced from the bridge itself. We leave it to the IOMMU driver to define the grouping restraints for their platform.
Using this new interface, the group for a device can be retrieved using the iommu_device_group() callback. Users will compare the value returned against the value returned for other devices to determine whether they are part of the same group. Devices with no group are not translated by the IOMMU. There should be no expectations about the group numbers as they may be arbitrarily assigned by the IOMMU driver and may not be persistent across boots.
We also provide a sysfs interface to the group numbers here so that userspace can understand IOMMU dependencies between devices for managing safe, userspace drivers.
---
drivers/iommu/iommu.c | 61 +++++++++++++++++++++++++++++++++++++++++++++++++ include/linux/iommu.h | 7 ++++++ 2 files changed, 68 insertions(+), 0 deletions(-)
drivers/iommu/iommu.c +61 -0
@@ -25,8 +25,60 @@#include <linux/errno.h> #include <linux/iommu.h> +static ssize_t show_iommu_group (struct device *dev, +struct device_attribute *attr,char *buf) +{ +unsigned int groupid; + +if (iommu_device_group (dev, &groupid)) +return 0 ; + +return sprintf (buf,"%u" , groupid); +} +static DEVICE_ATTR(iommu_group, S_IRUGO, show_iommu_group, NULL); + +static int add_remove_iommu_group (struct device *dev,void *data) +{ +unsigned int groupid; +int add = *(int *)data; + +if (iommu_device_group (dev, &groupid) ==0 ) { +if (add) +return device_create_file (dev, &dev_attr_iommu_group); +else +device_remove_file (dev, &dev_attr_iommu_group); + } + +return 0 ; +} + +static int iommu_device_notifier (struct notifier_block *nb, +unsigned long action,void *data) +{ +struct device *dev = data; +int add; + +if (action ==BUS_NOTIFY_ADD_DEVICE ) { + add =1 ; +return add_remove_iommu_group (dev, &add); + }else if (action ==BUS_NOTIFY_DEL_DEVICE ) { + add =0 ; +return add_remove_iommu_group (dev, &add); + } + +return 0 ; +} + +static struct notifier_block iommu_device_nb = { + .notifier_call = iommu_device_notifier, +}; +static void iommu_bus_init (struct bus_type *bus,struct iommu_ops *ops) { +int add =1 ; + +bus_register_notifier (bus, &iommu_device_nb); +bus_for_each_dev (bus,NULL , &add, add_remove_iommu_group); }/**
@@ -186,3 +238,12 @@ int iommu_unmap(struct iommu_domain *domain, unsigned long iova, int gfp_order)return domain->ops->unmap (domain, iova, gfp_order); }EXPORT_SYMBOL_GPL (iommu_unmap); + +int iommu_device_group (struct device *dev,unsigned int *groupid) +{ +if (iommu_present (dev->bus) && dev->bus->iommu_ops->device_group) +return dev->bus->iommu_ops->device_group (dev, groupid); + +return -ENODEV ; +} +EXPORT_SYMBOL_GPL (iommu_device_group);
include/linux/iommu.h +7 -0
@@ -61,6 +61,7 @@ struct iommu_ops { unsigned long iova);int (*domain_has_cap)(struct iommu_domain *domain,unsigned long cap); +int (*device_group)(struct device *dev,unsigned int *groupid); };extern int bus_set_iommu (struct bus_type *bus,struct iommu_ops *ops);
@@ -81,6 +82,7 @@ extern int iommu_domain_has_cap(struct iommu_domain *domain, unsigned long cap);extern void iommu_set_fault_handler (struct iommu_domain *domain,iommu_fault_handler_t handler); +extern int iommu_device_group (struct device *dev,unsigned int *groupid);/** * report_iommu_fault() - report about an IOMMU fault to the IOMMU framework
@@ -179,6 +181,11 @@ static inline void iommu_set_fault_handler(struct iommu_domain *domain, { } +static inline int iommu_device_group (struct device *dev,unsigned int *groupid); +{ +return -ENODEV ; +} +#endif /* CONFIG_IOMMU_API */ #endif /* __LINUX_IOMMU_H */
Re: [PATCH 1/4] iommu: Add iommu_device_group callback and iommu_group sysfs entry
Quoted 15 lines · Alex Williamson
An IOMMU group is a set of devices for which the IOMMU cannot distinguish transactions. For PCI devices, a group often occurs when a PCI bridge is involved. Transactions from any device behind the bridge appear to be sourced from the bridge itself. We leave it to the IOMMU driver to define the grouping restraints for their platform.
Using this new interface, the group for a device can be retrieved using the
iommu_device_group()callback. Users will compare the value returned against the value returned for other devices to determine whether they are part of the same group. Devices with no group are not translated by the IOMMU. There should be no expectations about the group numbers as they may be arbitrarily assigned by the IOMMU driver and may not be persistent across boots.
Quoted 3 lines
We also provide a sysfs interface to the group numbers here so that userspace can understand IOMMU dependencies between devices for managing safe, userspace drivers.
Finally giving these patches a close read. Sorry it's been so long.
Quoted 17 lines
Signed-off-by Alex Williamson <[email protected]>---
drivers/iommu/iommu.c | 61 +++++++++++++++++++++++++++++++++++++++++++++++++ include/linux/iommu.h | 7 ++++++ 2 files changed, 68 insertions(+), 0 deletions(-)drivers/iommu/iommu.c +1
-0@@ -25,8 +25,60 @@#include <linux/errno.h> #include <linux/iommu.h> +static ssize_t show_iommu_group(struct device *dev,
drivers/iommu/iommu.c line 38
Quoted 10 lines
drivers/iommu/iommu.c +10
-0+ struct device_attribute *attr, char *buf) +{ + unsigned int groupid; + +if (iommu_device_group (dev, &groupid)) +return 0 ; + +return sprintf (buf,"%u" , groupid); +} +static DEVICE_ATTR(iommu_group, S_IRUGO, show_iommu_group, NULL);
Hrm. Assuming the group is is an unsigned int seems dangerous to me. More seriously, we really want these to be unique across the whole system, but they're allocated by the iommu driver which can't guarantee that if it's not the only one present. Seems to me it would be safer to have an actual iommu_group structure allocated for each group, and use the pointer to it as the ID to hand around (with NULL meaning "no iommu" / untranslated). The structure could contain a more human readable - or more relevant to platform documentation - ID where appropriate.
Quoted 4 lines
drivers/iommu/iommu.c +4
-0+static int add_remove_iommu_group (struct device *dev,void *data) +{ +unsigned int groupid; +int add = *(int *)data;
drivers/iommu/iommu.c line 53
Quoted 10 lines
drivers/iommu/iommu.c +10
-0+ +if (iommu_device_group (dev, &groupid) ==0 ) { +if (add) +return device_create_file (dev, &dev_attr_iommu_group); +else +device_remove_file (dev, &dev_attr_iommu_group); + } + +return 0 ; +}
Multiplexing add and remove together seems pointlessly obfuscated.
Quoted 13 lines
drivers/iommu/iommu.c +13
-0+static int iommu_device_notifier (struct notifier_block *nb, +unsigned long action,void *data) +{ +struct device *dev = data; +int add; + +if (action ==BUS_NOTIFY_ADD_DEVICE ) { + add =1 ; +return add_remove_iommu_group (dev, &add); + }else if (action ==BUS_NOTIFY_DEL_DEVICE ) { + add =0 ; +return add_remove_iommu_group (dev, &add); + }
drivers/iommu/iommu.c line 77
Quoted 10 lines
drivers/iommu/iommu.c +8
-0+ +return 0 ; +} + +static struct notifier_block iommu_device_nb = { + .notifier_call = iommu_device_notifier, +}; +static void iommu_bus_init (struct bus_type *bus,struct iommu_ops *ops) {
I don't know of any current examples, but I do worry that makin this per bus-type rather than bus-instance might bite us in the arse later on.
Quoted 53 lines
drivers/iommu/iommu.c +13
-0+int add =1 ; + +bus_register_notifier (bus, &iommu_device_nb); +bus_for_each_dev (bus,NULL , &add, add_remove_iommu_group); }/** @@ -186,3 +238,12 @@ int iommu_unmap(struct iommu_domain *domain, unsigned long iova, int gfp_order)return domain->ops->unmap (domain, iova, gfp_order); }EXPORT_SYMBOL_GPL (iommu_unmap); + +int iommu_device_group (struct device *dev,unsigned int *groupid) +{ +if (iommu_present (dev->bus) && dev->bus->iommu_ops->device_group) +return dev->bus->iommu_ops->device_group (dev, groupid); + +return -ENODEV ; +} +EXPORT_SYMBOL_GPL (iommu_device_group);include/linux/iommu.h +7
-0@@ -61,6 +61,7 @@ struct iommu_ops { unsigned long iova);int (*domain_has_cap)(struct iommu_domain *domain,unsigned long cap); +int (*device_group)(struct device *dev,unsigned int *groupid); };extern int bus_set_iommu (struct bus_type *bus,struct iommu_ops *ops);@@ -81,6 +82,7 @@ extern int iommu_domain_has_cap(struct iommu_domain *domain, unsigned long cap);extern void iommu_set_fault_handler (struct iommu_domain *domain,iommu_fault_handler_t handler); +extern int iommu_device_group (struct device *dev,unsigned int *groupid);/** * report_iommu_fault() - report about an IOMMU fault to the IOMMU framework @@ -179,6 +181,11 @@ static inline void iommu_set_fault_handler(struct iommu_domain *domain, { } +static inline int iommu_device_group (struct device *dev,unsigned int *groupid); +{ +return -ENODEV ; +} +#endif /* CONFIG_IOMMU_API */ #endif /* __LINUX_IOMMU_H */
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Re: [PATCH 1/4] iommu: Add iommu_device_group callback and iommu_group sysfs entry
Quoted 13 lines · David Gibson
drivers/iommu/iommu.c line 38
Quoted 11 lines
drivers/iommu/iommu.c +11
-0+static ssize_t show_iommu_group (struct device *dev, +struct device_attribute *attr,char *buf) +{ +unsigned int groupid; + +if (iommu_device_group (dev, &groupid)) +return 0 ; + +return sprintf (buf,"%u" , groupid); +} +static DEVICE_ATTR(iommu_group, S_IRUGO, show_iommu_group, NULL);Hrm. Assuming the group is is an unsigned int seems dangerous to me.
Quoted 8 lines
More seriously, we really want these to be unique across the whole system, but they're allocated by the iommu driver which can't guarantee that if it's not the only one present. Seems to me it would be safer to have an actual
iommu_groupstructure allocated for each group, and use the pointer to it as the ID to hand around (with NULL meaning "no iommu" / untranslated). The structure could contain a more human readable - or more relevant to platform documentation - ID where appropriate.
Don't forget that to keep sanity, we really want to expose the groups via sysfs (per-group dir with symlinks to the devices).
I'm working with Alexey on providing an in-kernel powerpc specific API to expose the PE stuff to whatever's going to interface to VFIO to create the groups, though we can eventually collapse that. The idea is that on non-PE capable brigdes (old style), I would make a single group per host bridge.
In addition, Alex, I noticed that you still have the domain stuff there, which is fine I suppose, we could make it a requirement on power that you only put a single group in a domain... but the API is still to put individual devices in a domain, not groups, and that somewhat sucks.
You could "fix" that by having some kind of ->domain_enable() or whatever that's used to "activate" the domain and verifies that it contains entire groups but that looks like a pointless way to complicate both the API and the implementation.
Cheers,
Ben.
Re: [PATCH 1/4] iommu: Add iommu_device_group callback and iommu_group sysfs entry
Quoted 15 lines · Benjamin Herrenschmidt
Quoted 13 lines · David Gibson
drivers/iommu/iommu.c line 38
Quoted 11 lines
drivers/iommu/iommu.c +11
-0+static ssize_t show_iommu_group (struct device *dev, +struct device_attribute *attr,char *buf) +{ +unsigned int groupid; + +if (iommu_device_group (dev, &groupid)) +return 0 ; + +return sprintf (buf,"%u" , groupid); +} +static DEVICE_ATTR(iommu_group, S_IRUGO, show_iommu_group, NULL);Hrm. Assuming the group is is an unsigned int seems dangerous to me.
Quoted 8 lines
Quoted 8 lines
More seriously, we really want these to be unique across the whole system, but they're allocated by the iommu driver which can't guarantee that if it's not the only one present. Seems to me it would be safer to have an actual
iommu_groupstructure allocated for each group, and use the pointer to it as the ID to hand around (with NULL meaning "no iommu" / untranslated). The structure could contain a more human readable - or more relevant to platform documentation - ID where appropriate.
Note that iommu drivers are registered per bus_type, so the unique pair is {bus_type, groupid}, which seems sufficient for vfio.
Quoted 8 lines
Don't forget that to keep sanity, we really want to expose the groups via sysfs (per-group dir with symlinks to the devices).
I'm working with Alexey on providing an in-kernel powerpc specific API to expose the PE stuff to whatever's going to interface to VFIO to create the groups, though we can eventually collapse that. The idea is that on non-PE capable brigdes (old style), I would make a single group per host bridge.
If your non-PE capable bridges aren't actually providing isolation, they should return -ENODEV for the group_device() callback, then vfio will ignore them.
Quoted 9 lines
In addition, Alex, I noticed that you still have the domain stuff there, which is fine I suppose, we could make it a requirement on power that you only put a single group in a domain... but the API is still to put individual devices in a domain, not groups, and that somewhat sucks.
You could "fix" that by having some kind of ->domain_enable() or whatever that's used to "activate" the domain and verifies that it contains entire groups but that looks like a pointless way to complicate both the API and the implementation.
Right, groups are currently just a way to identify dependent sets, not a unit of work. We can also have group membership change dynamically (hotplug slot behind a PCIe-to-PCI bridge), so there are cases where we might need to formally attach/detach a group element to a domain at some later point. This really hasn't felt like a stumbling point for vfio, at least on x86. Thanks,
Alex
Re: [PATCH 1/4] iommu: Add iommu_device_group callback and iommu_group sysfs entry
Quoted 12 lines · Alex Williamson
Note that iommu drivers are registered per
bus_type, so the unique pair is {bus_type, groupid}, which seems sufficient for vfio.Quoted 8 lines
Don't forget that to keep sanity, we really want to expose the groups via sysfs (per-group dir with symlinks to the devices).
I'm working with Alexey on providing an in-kernel powerpc specific API to expose the PE stuff to whatever's going to interface to VFIO to create the groups, though we can eventually collapse that. The idea is that on non-PE capable brigdes (old style), I would make a single group per host bridge.
Quoted 3 lines
If your non-PE capable bridges aren't actually providing isolation, they should return
-ENODEVfor thegroup_device()callback, then vfio will ignore them.
Why ignore them ? It's perfectly fine to consider everything below the host bridge as one group. There is isolation ... at the host bridge level.
Really groups should be a structure, not a magic number. We want to iterate them and their content, represent them via an API, etc... and so magic numbers means that anything under the hood will have to constantly convert between that and some kind of internal data structure.
I also somewhat dislike the bus_type as the anchor to the grouping system, but that's not necessarily as bad an issue for us to deal with.
Eventually what will happen on my side is that I will have a powerpc "generic" (ie. accross platforms) that allow to enumerate groups and retrieve the dma windows associated with them etc...
That API will use underlying function pointers provided by the PCI host bridge (for which we do have a data structure, struct pci_controller, like many other archs except I think x86 :-)
Any host platform that doesn't provide those pointers (ie. all of them initially) will get a default behaviour which is to group everything below a host bridge (since host bridges still have independent iommu windows, at least for us they all do).
On top of that we can implement a "backend" that provides those pointers for the p7ioc bridge used on the powernv platform, which will expose more fine grained groups based on our "partitionable endpoint" mechanism.
The grouping will have been decided early at boot time based on a mix of HW resources and bus topology, plus things like whether there is a PCI-X bridge etc... and will be initially immutable.
Ideally, we need to expose a subset of this API as a "generic" interface to allow generic code to iterate the groups and their content, and to construct the appropriate sysfs representation.
Quoted 10 lines
Quoted 9 lines
In addition, Alex, I noticed that you still have the domain stuff there, which is fine I suppose, we could make it a requirement on power that you only put a single group in a domain... but the API is still to put individual devices in a domain, not groups, and that somewhat sucks.
You could "fix" that by having some kind of ->domain_enable() or whatever that's used to "activate" the domain and verifies that it contains entire groups but that looks like a pointless way to complicate both the API and the implementation.
Quoted 6 lines
Right, groups are currently just a way to identify dependent sets, not a unit of work. We can also have group membership change dynamically (hotplug slot behind a PCIe-to-PCI bridge), so there are cases where we might need to formally attach/detach a group element to a domain at some later point. This really hasn't felt like a stumbling point for vfio, at least on x86. Thanks,
It doesn't matter much as long as we have a way to know that a group is "complete", ie that all devices of a group have been taken over by vfio and put into a domain, and block them from being lost. Only then can we actually "use" the group and start reconfiguring the iommu etc... for use by the guest.
Note that groups -will- contain briges eventually. We need to take that into account since bridges -usually- don't have an ordinary driver attached to them so there may be issues there with tracking whether they are taken over by vfio...
Cheers,
Ben.
Re: [PATCH 1/4] iommu: Add iommu_device_group callback and iommu_group sysfs entry
Quoted 18 lines · Benjamin Herrenschmidt
Quoted 12 lines · Alex Williamson
Note that iommu drivers are registered per
bus_type, so the unique pair is {bus_type, groupid}, which seems sufficient for vfio.Quoted 8 lines
Don't forget that to keep sanity, we really want to expose the groups via sysfs (per-group dir with symlinks to the devices).
I'm working with Alexey on providing an in-kernel powerpc specific API to expose the PE stuff to whatever's going to interface to VFIO to create the groups, though we can eventually collapse that. The idea is that on non-PE capable brigdes (old style), I would make a single group per host bridge.
Quoted 3 lines
If your non-PE capable bridges aren't actually providing isolation, they should return
-ENODEVfor thegroup_device()callback, then vfio will ignore them.
Quoted 8 lines
Why ignore them ? It's perfectly fine to consider everything below the host bridge as one group. There is isolation ... at the host bridge level.
Really groups should be a structure, not a magic number. We want to iterate them and their content, represent them via an API, etc... and so magic numbers means that anything under the hood will have to constantly convert between that and some kind of internal data structure.
Right. These have to be discoverable, so we need some kind of in-kernel object to represent them. Might as well use that everywhere, rather than just at higher levels.
Quoted 46 lines
I also somewhat dislike the
bus_typeas the anchor to the grouping system, but that's not necessarily as bad an issue for us to deal with.Eventually what will happen on my side is that I will have a powerpc "generic" (ie. accross platforms) that allow to enumerate groups and retrieve the dma windows associated with them etc...
That API will use underlying function pointers provided by the PCI host bridge (for which we do have a data structure, struct
pci_controller, like many other archs except I think x86 :-)Any host platform that doesn't provide those pointers (ie. all of them initially) will get a default behaviour which is to group everything below a host bridge (since host bridges still have independent iommu windows, at least for us they all do).
On top of that we can implement a "backend" that provides those pointers for the p7ioc bridge used on the powernv platform, which will expose more fine grained groups based on our "partitionable endpoint" mechanism.
The grouping will have been decided early at boot time based on a mix of HW resources and bus topology, plus things like whether there is a PCI-X bridge etc... and will be initially immutable.
Ideally, we need to expose a subset of this API as a "generic" interface to allow generic code to iterate the groups and their content, and to construct the appropriate sysfs representation.
Quoted 10 lines
Quoted 9 lines
In addition, Alex, I noticed that you still have the domain stuff there, which is fine I suppose, we could make it a requirement on power that you only put a single group in a domain... but the API is still to put individual devices in a domain, not groups, and that somewhat sucks.
You could "fix" that by having some kind of ->domain_enable() or whatever that's used to "activate" the domain and verifies that it contains entire groups but that looks like a pointless way to complicate both the API and the implementation.
Quoted 6 lines
Right, groups are currently just a way to identify dependent sets, not a unit of work. We can also have group membership change dynamically (hotplug slot behind a PCIe-to-PCI bridge), so there are cases where we might need to formally attach/detach a group element to a domain at some later point. This really hasn't felt like a stumbling point for vfio, at least on x86. Thanks,
Quoted 5 lines
It doesn't matter much as long as we have a way to know that a group is "complete", ie that all devices of a group have been taken over by vfio and put into a domain, and block them from being lost. Only then can we actually "use" the group and start reconfiguring the iommu etc... for use by the guest.
I think this is handled by later patches in the series.
Quoted 9 lines
Note that groups -will- contain briges eventually. We need to take that into account since bridges -usually- don't have an ordinary driver attached to them so there may be issues there with tracking whether they are taken over by vfio...
Cheers,
Ben.
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Re: [PATCH 1/4] iommu: Add iommu_device_group callback and iommu_group sysfs entry
In reply to Benjamin HerrenschmidtQuoted 8 lines · Benjamin Herrenschmidt
Quoted 6 lines · Alex Williamson
Note that iommu drivers are registered per
bus_type, so the unique pair is {bus_type, groupid}, which seems sufficient for vfio.Quoted 3 lines
Don't forget that to keep sanity, we really want to expose the groups via sysfs (per-group dir with symlinks to the devices).
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
I'm working with Alexey on providing an in-kernel powerpc specific API to expose the PE stuff to whatever's going to interface to VFIO to create the groups, though we can eventually collapse that. The idea is that on non-PE capable brigdes (old style), I would make a single group per host bridge.
If your non-PE capable bridges aren't actually providing isolation, they
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Quoted 6 lines
Quoted 2 lines
should return
-ENODEVfor thegroup_device()callback, then vfio will ignore them.Why ignore them ? It's perfectly fine to consider everything below the host bridge as one group. There is isolation ... at the host bridge level.
If there is isolation, yes, report a group. A host bridge by itself does not imply DMA isolation on most platforms. We could say devices without isolation are all in the same group, but then imagine a system with two PCI host bridges. Host bridge A has an iommu providing B:D.F isolation, host bridge B has no iommu and no isolation. Clearly devices behind A have groupids. However if we assign a single groupid for all the devices behind B, we subvert the isolation of devices behind A. To avoid that, devices that cannot be isolated should not have a groupid.
Quoted 4 lines
Really groups should be a structure, not a magic number. We want to iterate them and their content, represent them via an API, etc... and so magic numbers means that anything under the hood will have to constantly convert between that and some kind of internal data structure.
What would a group structure in core driver code contain? Who manages it? Don't forget, most users don't have hardware capable of making use of this and most users with the right hardware still aren't going to use it. Those group structures live in vfio and get allocated when vfio is loaded. VT-d and AMD-Vi can support this new groupid interface with zero additional data structures floating around.
As Chris noted, the choice of an unsigned int to efficiently represent groupids in PCI isn't just coincidence. If you can't make use of the same trick, surely it's not too difficult to enumerate your PEs and use that as the groupid.
Quoted 46 lines
I also somewhat dislike the
bus_typeas the anchor to the grouping system, but that's not necessarily as bad an issue for us to deal with.Eventually what will happen on my side is that I will have a powerpc "generic" (ie. accross platforms) that allow to enumerate groups and retrieve the dma windows associated with them etc...
That API will use underlying function pointers provided by the PCI host bridge (for which we do have a data structure, struct
pci_controller, like many other archs except I think x86 :-)Any host platform that doesn't provide those pointers (ie. all of them initially) will get a default behaviour which is to group everything below a host bridge (since host bridges still have independent iommu windows, at least for us they all do).
On top of that we can implement a "backend" that provides those pointers for the p7ioc bridge used on the powernv platform, which will expose more fine grained groups based on our "partitionable endpoint" mechanism.
The grouping will have been decided early at boot time based on a mix of HW resources and bus topology, plus things like whether there is a PCI-X bridge etc... and will be initially immutable.
Ideally, we need to expose a subset of this API as a "generic" interface to allow generic code to iterate the groups and their content, and to construct the appropriate sysfs representation.
Quoted 10 lines
Quoted 9 lines
In addition, Alex, I noticed that you still have the domain stuff there, which is fine I suppose, we could make it a requirement on power that you only put a single group in a domain... but the API is still to put individual devices in a domain, not groups, and that somewhat sucks.
You could "fix" that by having some kind of ->domain_enable() or whatever that's used to "activate" the domain and verifies that it contains entire groups but that looks like a pointless way to complicate both the API and the implementation.
Quoted 6 lines
Right, groups are currently just a way to identify dependent sets, not a unit of work. We can also have group membership change dynamically (hotplug slot behind a PCIe-to-PCI bridge), so there are cases where we might need to formally attach/detach a group element to a domain at some later point. This really hasn't felt like a stumbling point for vfio, at least on x86. Thanks,
Quoted 5 lines
It doesn't matter much as long as we have a way to know that a group is "complete", ie that all devices of a group have been taken over by vfio and put into a domain, and block them from being lost. Only then can we actually "use" the group and start reconfiguring the iommu etc... for use by the guest.
This is done. I call a group "viable" when all of the devices are bound to their vfio bus driver and ready for use. The API will not allow a user access to the iommu or device file descriptors unless the group is viable. Unbinding a device from the vfio bus driver will block as long as the group is in use (need to decide if we enable a system policy to let vfio kill the process to release a device). I'm also working on code that proactively attaches the vfio bus driver to a hot-added device if it's being added to an in-use group.
Quoted 4 lines
Note that groups -will- contain briges eventually. We need to take that into account since bridges -usually- don't have an ordinary driver attached to them so there may be issues there with tracking whether they are taken over by vfio...
I look forward to patches for this ;) Thanks,
Alex
Re: [PATCH 1/4] iommu: Add iommu_device_group callback and iommu_group sysfs entry
In reply to Alex WilliamsonQuoted 25 lines · Alex Williamson
Quoted 15 lines · Benjamin Herrenschmidt
Quoted 13 lines · David Gibson
drivers/iommu/iommu.c line 38
Quoted 11 lines
drivers/iommu/iommu.c +11
-0+static ssize_t show_iommu_group (struct device *dev, +struct device_attribute *attr,char *buf) +{ +unsigned int groupid; + +if (iommu_device_group (dev, &groupid)) +return 0 ; + +return sprintf (buf,"%u" , groupid); +} +static DEVICE_ATTR(iommu_group, S_IRUGO, show_iommu_group, NULL);Hrm. Assuming the group is is an unsigned int seems dangerous to me.
Quoted 8 lines
Quoted 8 lines
More seriously, we really want these to be unique across the whole system, but they're allocated by the iommu driver which can't guarantee that if it's not the only one present. Seems to me it would be safer to have an actual
iommu_groupstructure allocated for each group, and use the pointer to it as the ID to hand around (with NULL meaning "no iommu" / untranslated). The structure could contain a more human readable - or more relevant to platform documentation - ID where appropriate.
Quoted 2 lines
Note that iommu drivers are registered per
bus_type, so the unique pair is {bus_type, groupid}, which seems sufficient for vfio.
Hrm. That's.. far from obvious. And still breaks down if we have two separate iommus on the same bus type (e.g. two independent PCI host bridges with inbuilt IOMMUs).
Quoted 18 lines
Quoted 8 lines
Don't forget that to keep sanity, we really want to expose the groups via sysfs (per-group dir with symlinks to the devices).
I'm working with Alexey on providing an in-kernel powerpc specific API to expose the PE stuff to whatever's going to interface to VFIO to create the groups, though we can eventually collapse that. The idea is that on non-PE capable brigdes (old style), I would make a single group per host bridge.
If your non-PE capable bridges aren't actually providing isolation, they should return
-ENODEVfor thegroup_device()callback, then vfio will ignore them.Quoted 5 lines
In addition, Alex, I noticed that you still have the domain stuff there, which is fine I suppose, we could make it a requirement on power that you only put a single group in a domain... but the API is still to put individual devices in a domain, not groups, and that somewhat sucks.
Quoted 7 lines
Quoted 4 lines
You could "fix" that by having some kind of ->domain_enable() or whatever that's used to "activate" the domain and verifies that it contains entire groups but that looks like a pointless way to complicate both the API and the implementation.
Right, groups are currently just a way to identify dependent sets, not a unit of work.
I'm not quite sure what you mean by "unit of work". But assigning the groups as a unit generally makes more sense to me than assigning devices individually, but only being able to use them when the group is together. Particularly when things are hotplugged into groups.
Quoted 8 lines
We can also have group membership change dynamically
(hotplug slot behind a PCIe-to-PCI bridge), so there are cases where we might need to formally attach/detach a group element to a domain at some later point. This really hasn't felt like a stumbling point for vfio, at least on x86. Thanks,Alex
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Re: [PATCH 1/4] iommu: Add iommu_device_group callback and iommu_group sysfs entry
Quoted 26 lines · * David Gibson ([email protected]) wrote:
Quoted 25 lines · Alex Williamson
Quoted 23 lines · Benjamin Herrenschmidt
Quoted 21 lines · David Gibson
drivers/iommu/iommu.c line 38
Quoted 11 lines
drivers/iommu/iommu.c +11
-0+static ssize_t show_iommu_group (struct device *dev, +struct device_attribute *attr,char *buf) +{ +unsigned int groupid; + +if (iommu_device_group (dev, &groupid)) +return 0 ; + +return sprintf (buf,"%u" , groupid); +} +static DEVICE_ATTR(iommu_group, S_IRUGO, show_iommu_group, NULL);Hrm. Assuming the group is is an unsigned int seems dangerous to me. More seriously, we really want these to be unique across the whole system, but they're allocated by the iommu driver which can't guarantee that if it's not the only one present. Seems to me it would be safer to have an actual
iommu_groupstructure allocated for each group, and use the pointer to it as the ID to hand around (with NULL meaning "no iommu" / untranslated). The structure could contain a more human readable - or more relevant to platform documentation - ID where appropriate.
Quoted 6 lines
Quoted 2 lines
Note that iommu drivers are registered per
bus_type, so the unique pair is {bus_type, groupid}, which seems sufficient for vfio.Hrm. That's.. far from obvious. And still breaks down if we have two separate iommus on the same bus type (e.g. two independent PCI host bridges with inbuilt IOMMUs).
Happens to still work for Intel IOMMU on x86 the way Alex wrote the Intel VT-d patch in this series, as well as AMD IOMMU. The caveat for AMD IOMMU is that the groupid generation would break (as-is) once there's support for multiple PCI segments. This is not an inherent shortcoming of the groupid mechanism though, just a current limitation of AMD IOMMU's implementation. Alex overloaded B:D.F for those which is a convenient id since that maps to the device (or in the case of devices behind a PCIe-to-PCI bridge, the requestor ID of all devices behind the bridge, or "the group").
thanks,
-chris
Re: [PATCH 1/4] iommu: Add iommu_device_group callback and iommu_group sysfs entry
Quoted 34 lines · Chris Wright
Quoted 26 lines · * David Gibson ([email protected]) wrote:
Quoted 25 lines · Alex Williamson
Quoted 23 lines · Benjamin Herrenschmidt
Quoted 21 lines · David Gibson
drivers/iommu/iommu.c line 38
Quoted 11 lines
drivers/iommu/iommu.c +11
-0+static ssize_t show_iommu_group (struct device *dev, +struct device_attribute *attr,char *buf) +{ +unsigned int groupid; + +if (iommu_device_group (dev, &groupid)) +return 0 ; + +return sprintf (buf,"%u" , groupid); +} +static DEVICE_ATTR(iommu_group, S_IRUGO, show_iommu_group, NULL);Hrm. Assuming the group is is an unsigned int seems dangerous to me. More seriously, we really want these to be unique across the whole system, but they're allocated by the iommu driver which can't guarantee that if it's not the only one present. Seems to me it would be safer to have an actual
iommu_groupstructure allocated for each group, and use the pointer to it as the ID to hand around (with NULL meaning "no iommu" / untranslated). The structure could contain a more human readable - or more relevant to platform documentation - ID where appropriate.Quoted 6 lines
Quoted 2 lines
Note that iommu drivers are registered per
bus_type, so the unique pair is {bus_type, groupid}, which seems sufficient for vfio.Hrm. That's.. far from obvious. And still breaks down if we have two separate iommus on the same bus type (e.g. two independent PCI host bridges with inbuilt IOMMUs).
Quoted 9 lines
Happens to still work for Intel IOMMU on x86 the way Alex wrote the Intel VT-d patch in this series, as well as AMD IOMMU. The caveat for AMD IOMMU is that the groupid generation would break (as-is) once there's support for multiple PCI segments. This is not an inherent shortcoming of the groupid mechanism though, just a current limitation of AMD IOMMU's implementation. Alex overloaded B:D.F for those which is a convenient id since that maps to the device (or in the case of devices behind a PCIe-to-PCI bridge, the requestor ID of all devices behind the bridge, or "the group").
"Happens to still work" is not exactly a ringing endorsement.
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Re: [PATCH 1/4] iommu: Add iommu_device_group callback and iommu_group sysfs entry
Quoted 20 lines · * David Gibson ([email protected]) wrote:
Quoted 9 lines · Chris Wright
Quoted 7 lines · * David Gibson ([email protected]) wrote:
Quoted 2 lines · Alex Williamson
Note that iommu drivers are registered per
bus_type, so the unique pair is {bus_type, groupid}, which seems sufficient for vfio.Hrm. That's.. far from obvious. And still breaks down if we have two separate iommus on the same bus type (e.g. two independent PCI host bridges with inbuilt IOMMUs).
Quoted 9 lines
Happens to still work for Intel IOMMU on x86 the way Alex wrote the Intel VT-d patch in this series, as well as AMD IOMMU. The caveat for AMD IOMMU is that the groupid generation would break (as-is) once there's support for multiple PCI segments. This is not an inherent shortcoming of the groupid mechanism though, just a current limitation of AMD IOMMU's implementation. Alex overloaded B:D.F for those which is a convenient id since that maps to the device (or in the case of devices behind a PCIe-to-PCI bridge, the requestor ID of all devices behind the bridge, or "the group").
Quoted 1 line
"Happens to still work" is not exactly a ringing endorsement.
Heh. Put it another way. Generating the group ID is left up to the IOMMU. This will break down when there's a system with multiple IOMMU's on the same bus_type that don't have any awareness of one another. This is not the case for the existing series and x86 hw.
I'm not opposed to doing the allocation and ptr as id (taking care for possibility that PCI hotplug/unplug/replug could reuse the same memory for group id, however). Just pointing out that the current system works as is, and there's some value in it's simplicity (overloading ID == group structure + pretty printing ID in sysfs, for example).
thanks,
-chris
Re: [PATCH 1/4] iommu: Add iommu_device_group callback and iommu_group sysfs entry
Quoted 10 lines · Chris Wright
Heh. Put it another way. Generating the group ID is left up to the IOMMU. This will break down when there's a system with multiple IOMMU's on the same
bus_typethat don't have any awareness of one another. This is not the case for the existing series and x86 hw.I'm not opposed to doing the allocation and ptr as id (taking care for possibility that PCI hotplug/unplug/replug could reuse the same memory for group id, however). Just pointing out that the current system works as is, and there's some value in it's simplicity (overloading ID == group structure + pretty printing ID in sysfs, for example).
Well, ID can work even with multiple domains since we have domains numbers. bdfn is 16-bit, which leaves 16-bit for the domain number, which is sufficient.
So by encoding (domain << 16) | bdfn, we can get away with a 32-bit number... it just sucks.
Note that on pseries, I wouldn't use bdfn anyway, I would use my internal "PE#" which is also a number that I can constraint to 16-bits.
So I can work with a number as long as it's at least an unsigned int (32-bit), but I think it somewhat sucks, and will impose gratuituous number <-> structure conversions all over, but if we keep the whole group thing an iommu specific data structure, then let's stick to the number and move on with life.
We might get better results if we kept the number as
struct iommu_group_id {
u16 domain;
u16 group;
};(Or a union of that with an unsigned int)
That way the domain information is available generically (can be match with pci_domain_nr() for example), and sysfs can then be layed out as
/sys/bus/pci/groups/<domain>/<id>
Which is nicer than having enormous id's
Cheers,
Ben.
Re: [PATCH 1/4] iommu: Add iommu_device_group callback and iommu_group sysfs entry
Quoted 12 lines · Benjamin Herrenschmidt
Quoted 10 lines · Chris Wright
Heh. Put it another way. Generating the group ID is left up to the IOMMU. This will break down when there's a system with multiple IOMMU's on the same
bus_typethat don't have any awareness of one another. This is not the case for the existing series and x86 hw.I'm not opposed to doing the allocation and ptr as id (taking care for possibility that PCI hotplug/unplug/replug could reuse the same memory for group id, however). Just pointing out that the current system works as is, and there's some value in it's simplicity (overloading ID == group structure + pretty printing ID in sysfs, for example).
Quoted 6 lines
Well, ID can work even with multiple domains since we have domains numbers. bdfn is 16-bit, which leaves 16-bit for the domain number, which is sufficient.
So by encoding (domain << 16) | bdfn, we can get away with a 32-bit number... it just sucks.
Except that the iommu group structure is not supposed to be specific to PCI.
Quoted 29 lines
Note that on pseries, I wouldn't use bdfn anyway, I would use my internal "PE#" which is also a number that I can constraint to 16-bits.
So I can work with a number as long as it's at least an unsigned int (32-bit), but I think it somewhat sucks, and will impose gratuituous number <-> structure conversions all over, but if we keep the whole group thing an iommu specific data structure, then let's stick to the number and move on with life.
We might get better results if we kept the number as
struct iommu_group_id { u16 domain; u16 group; };(Or a union of that with an unsigned int)
That way the domain information is available generically (can be match with
pci_domain_nr()for example), and sysfs can then be layed out as/sys/bus/pci/groups/<domain>/<id>
Which is nicer than having enormous id's
Cheers,
Ben.
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Re: [PATCH 1/4] iommu: Add iommu_device_group callback and iommu_group sysfs entry
In reply to Benjamin HerrenschmidtQuoted 12 lines · * Benjamin Herrenschmidt ([email protected]) wrote:
Quoted 10 lines · Chris Wright
Heh. Put it another way. Generating the group ID is left up to the IOMMU. This will break down when there's a system with multiple IOMMU's on the same
bus_typethat don't have any awareness of one another. This is not the case for the existing series and x86 hw.I'm not opposed to doing the allocation and ptr as id (taking care for possibility that PCI hotplug/unplug/replug could reuse the same memory for group id, however). Just pointing out that the current system works as is, and there's some value in it's simplicity (overloading ID == group structure + pretty printing ID in sysfs, for example).
Quoted 6 lines
Well, ID can work even with multiple domains since we have domains numbers. bdfn is 16-bit, which leaves 16-bit for the domain number, which is sufficient.
So by encoding (domain << 16) | bdfn, we can get away with a 32-bit number... it just sucks.
Yup, that's just what Alex did for VT-d ;)
+ union {
+ struct {
+ u8 devfn; + u8 bus; + u16 segment;
+ } pci;
+ u32 group;
+ } id;
Just that the alias table used for AMD IOMMU to map bdf -> requestor ID is not multi-segment aware, so the id is only bdf of bridge.
Quoted 16 lines
Note that on pseries, I wouldn't use bdfn anyway, I would use my internal "PE#" which is also a number that I can constraint to 16-bits.
So I can work with a number as long as it's at least an unsigned int (32-bit), but I think it somewhat sucks, and will impose gratuituous number <-> structure conversions all over, but if we keep the whole group thing an iommu specific data structure, then let's stick to the number and move on with life.
We might get better results if we kept the number as
struct iommu_group_id { u16 domain; u16 group; };
Quoted 8 lines
(Or a union of that with an unsigned int)
That way the domain information is available generically (can be match with
pci_domain_nr()for example), and sysfs can then be layed out as/sys/bus/pci/groups/<domain>/<id>
Which is nicer than having enormous id's
Seems fine to me (although I missed /sys/bus/pci/groups/ introduction), except that I think the freescale folks aren't interested in PCI which is one reason why the thing is just an opaque id.
thanks,
-chris
Re: [PATCH 1/4] iommu: Add iommu_device_group callback and iommu_group sysfs entry
Quoted 31 lines · Chris Wright
Quoted 12 lines · * Benjamin Herrenschmidt ([email protected]) wrote:
Quoted 10 lines · Chris Wright
Heh. Put it another way. Generating the group ID is left up to the IOMMU. This will break down when there's a system with multiple IOMMU's on the same
bus_typethat don't have any awareness of one another. This is not the case for the existing series and x86 hw.I'm not opposed to doing the allocation and ptr as id (taking care for possibility that PCI hotplug/unplug/replug could reuse the same memory for group id, however). Just pointing out that the current system works as is, and there's some value in it's simplicity (overloading ID == group structure + pretty printing ID in sysfs, for example).
Quoted 6 lines
Well, ID can work even with multiple domains since we have domains numbers. bdfn is 16-bit, which leaves 16-bit for the domain number, which is sufficient.
So by encoding (domain << 16) | bdfn, we can get away with a 32-bit number... it just sucks.
Yup, that's just what Alex did for VT-d ;)
(quoted diff) +8
-0+ union { + struct { + u8 devfn; + u8 bus; + u16 segment; + } pci; + u32 group; + } id;
Quoted 11 lines
Just that the alias table used for AMD IOMMU to map bdf -> requestor ID is not multi-segment aware, so the id is only bdf of bridge.
Quoted 8 lines
Note that on pseries, I wouldn't use bdfn anyway, I would use my internal "PE#" which is also a number that I can constraint to 16-bits.
So I can work with a number as long as it's at least an unsigned int (32-bit), but I think it somewhat sucks, and will impose gratuituous number <-> structure conversions all over, but if we keep the whole group thing an iommu specific data structure, then let's stick to the number and move on with life.
I think you're over emphasizing these number <-> struct conversions. Nowhere in the iommu/driver core code is there a need for this. It's a simple struct device -> groupid translation. Even in the vfio code we only need to do lookups based on groupid rarely and never in anything resembling a performance path.
Quoted 12 lines
Quoted 12 lines
We might get better results if we kept the number as
struct iommu_group_id { u16 domain; u16 group; };(Or a union of that with an unsigned int)
That way the domain information is available generically (can be match with
pci_domain_nr()for example), and sysfs can then be layed out as
Quoted 7 lines
Quoted 3 lines
/sys/bus/pci/groups/<domain>/<id>
Which is nicer than having enormous id's
Seems fine to me (although I missed
/sys/bus/pci/groups/introduction), except that I think the freescale folks aren't interested in PCI which is one reason why the thing is just an opaque id.
I missed the /sys/bus/pci/groups/ introduction as well. Groupids are not PCI specific. FWIW, the current sysfs representation looks something like this:
$ cat /sys/bus/pci/devices/0000:01:10.6/iommu_group 384
$ ls -l /sys/devices/virtual/vfio/pci:384 total 0 -r--r--r-- 1 root root 4096 Nov 30 16:25 dev drwxr-xr-x 2 root root 0 Nov 30 16:25 devices drwxr-xr-x 2 root root 0 Nov 30 16:25 power lrwxrwxrwx 1 root root 0 Nov 30 16:24 subsystem -> ../../../../class/vfio -rw-r--r-- 1 root root 4096 Nov 30 16:24 uevent
$ cat /sys/devices/virtual/vfio/pci:384/dev 252:28
$ ls -l /dev/vfio/pci:384 crw-rw---- 1 root root 252, 28 Nov 30 16:24 /dev/vfio/pci:384
$ ls -l /sys/devices/virtual/vfio/pci:384/devices total 0 lrwxrwxrwx 1 root root 0 Nov 30 16:36 0000:01:10.0 -> ../../../../pci0000:00/0000:00:01.0/0000:01:10.0 lrwxrwxrwx 1 root root 0 Nov 30 16:25 0000:01:10.1 -> ../../../../pci0000:00/0000:00:01.0/0000:01:10.1 lrwxrwxrwx 1 root root 0 Nov 30 16:36 0000:01:10.2 -> ../../../../pci0000:00/0000:00:01.0/0000:01:10.2 lrwxrwxrwx 1 root root 0 Nov 30 16:25 0000:01:10.3 -> ../../../../pci0000:00/0000:00:01.0/0000:01:10.3 lrwxrwxrwx 1 root root 0 Nov 30 16:36 0000:01:10.4 -> ../../../../pci0000:00/0000:00:01.0/0000:01:10.4 lrwxrwxrwx 1 root root 0 Nov 30 16:25 0000:01:10.5 -> ../../../../pci0000:00/0000:00:01.0/0000:01:10.5 lrwxrwxrwx 1 root root 0 Nov 30 16:36 0000:01:10.6 -> ../../../../pci0000:00/0000:00:01.0/0000:01:10.6 lrwxrwxrwx 1 root root 0 Nov 30 16:25 0000:01:10.7 -> ../../../../pci0000:00/0000:00:01.0/0000:01:10.7
$ for i in $(find /sys/devices/virtual/vfio/pci:384/devices/ -type l); do cat $i/iommu_group; echo; done 384 384 384 384 384 384 384 384
I'm a little annoyed with the pci:384 notation, but that seems like how it works out best for using {bus_type, groupid}. Thanks,
Alex
RE: [PATCH 1/4] iommu: Add iommu_device_group callback and iommu_group sysfs entry
In reply to Chris WrightQuoted 11 lines
Quoted 7 lines
That way the domain information is available generically (can be match with
pci_domain_nr()for example), and sysfs can then be layed out as/sys/bus/pci/groups/<domain>/<id>
Which is nicer than having enormous id's
Seems fine to me (although I missed
/sys/bus/pci/groups/introduction), except that I think the freescale folks aren't interested in PCI which is one reason why the thing is just an opaque id.
We are interested in PCI, but non-PCI as well. All DMA devices, have a globally unique "logical I/O device number" (LIODN) which will map to a group # for us.
Stuart
Re: [PATCH 1/4] iommu: Add iommu_device_group callback and iommu_group sysfs entry
In reply to David GibsonQuoted 26 lines · David Gibson
Quoted 25 lines · Alex Williamson
Quoted 23 lines · Benjamin Herrenschmidt
Quoted 21 lines · David Gibson
drivers/iommu/iommu.c line 38
Quoted 11 lines
drivers/iommu/iommu.c +11
-0+static ssize_t show_iommu_group (struct device *dev, +struct device_attribute *attr,char *buf) +{ +unsigned int groupid; + +if (iommu_device_group (dev, &groupid)) +return 0 ; + +return sprintf (buf,"%u" , groupid); +} +static DEVICE_ATTR(iommu_group, S_IRUGO, show_iommu_group, NULL);Hrm. Assuming the group is is an unsigned int seems dangerous to me. More seriously, we really want these to be unique across the whole system, but they're allocated by the iommu driver which can't guarantee that if it's not the only one present. Seems to me it would be safer to have an actual
iommu_groupstructure allocated for each group, and use the pointer to it as the ID to hand around (with NULL meaning "no iommu" / untranslated). The structure could contain a more human readable - or more relevant to platform documentation - ID where appropriate.
Quoted 6 lines
Quoted 2 lines
Note that iommu drivers are registered per
bus_type, so the unique pair is {bus_type, groupid}, which seems sufficient for vfio.Hrm. That's.. far from obvious. And still breaks down if we have two separate iommus on the same bus type (e.g. two independent PCI host bridges with inbuilt IOMMUs).
See bus_set_iommu(struct bus_type *bus, struct iommu_ops *ops). It wasn't long ago that we had a global iommu_ops. Are heterogeneous iommus using different drivers on the same bus_type really a possibility on your system, or are we dealing with hypothetical situations? You'd currently need to multiplex iommu_ops callbacks between the drivers yourself.
Quoted 23 lines
Quoted 23 lines
Quoted 8 lines
Don't forget that to keep sanity, we really want to expose the groups via sysfs (per-group dir with symlinks to the devices).
I'm working with Alexey on providing an in-kernel powerpc specific API to expose the PE stuff to whatever's going to interface to VFIO to create the groups, though we can eventually collapse that. The idea is that on non-PE capable brigdes (old style), I would make a single group per host bridge.
If your non-PE capable bridges aren't actually providing isolation, they should return
-ENODEVfor thegroup_device()callback, then vfio will ignore them.Quoted 9 lines
In addition, Alex, I noticed that you still have the domain stuff there, which is fine I suppose, we could make it a requirement on power that you only put a single group in a domain... but the API is still to put individual devices in a domain, not groups, and that somewhat sucks.
You could "fix" that by having some kind of ->domain_enable() or whatever that's used to "activate" the domain and verifies that it contains entire groups but that looks like a pointless way to complicate both the API and the implementation.
Quoted 7 lines
Quoted 2 lines
Right, groups are currently just a way to identify dependent sets, not a unit of work.
I'm not quite sure what you mean by "unit of work". But assigning the groups as a unit generally makes more sense to me than assigning devices individually, but only being able to use them when the group is together. Particularly when things are hotplugged into groups.
iommu_attach_device() takes a device as a "unit of work", not a group. If we have a group in use by a userspace driver and a new physical device is hotplugged into the host and belongs to the group, we need the granularity of the device to be able to add it to the domain.
I'm not sure if we're getting into VM usage with "assigning" terminology above. You're free to architect qemu however you want on POWER to make groups be the assignable unit to a guest. On x86, an individual device is the assignable unit for a guest. Unassigned group devices will still be required to be held by vfio, they'll just be unused. Thanks,
Alex
Re: [PATCH 1/4] iommu: Add iommu_device_group callback and iommu_group sysfs entry
Quoted 5 lines · Alex Williamson
I'm not sure if we're getting into VM usage with "assigning" terminology above. You're free to architect qemu however you want on POWER to make groups be the assignable unit to a guest. On x86, an individual device is the assignable unit for a guest. Unassigned group devices will still be required to be held by vfio, they'll just be unused. Thanks,
I'm not sure I understand this.
BY DEFINITION, the smallest assignable unit is the group, surely? Isn't that the *point* of the groups? That the IOMMU cannot tell the difference between the devices in the group?
So in *practice*, surely you cannot assign just *one* device from a group? You can assign the while group, or nothing.
You might *pretend* to assign single devices, and we might try to cope with the weirdness that happens when you want *one* device of a group to be owned by one VM, another device in the same group to be owned by a second VM, and a third device from the same group to be driven by a native driver in the host. But why not just assign groups as a whole? Surely that makes most sense?
Btw, did we get a quirk for the Ricoh multi-function devices which all need to be in the same group because they do all their DMA from function zero? I think we need another similar quirk for a Marvell SATA controller which seems to do its AHCI DMA from its IDE function; see
https://bugzilla.redhat.com/757166
Signature · 1 line
dwmw2
Re: [PATCH 1/4] iommu: Add iommu_device_group callback and iommu_group sysfs entry
Quoted 16 lines · David Woodhouse
Quoted 5 lines · Alex Williamson
I'm not sure if we're getting into VM usage with "assigning" terminology above. You're free to architect qemu however you want on POWER to make groups be the assignable unit to a guest. On x86, an individual device is the assignable unit for a guest. Unassigned group devices will still be required to be held by vfio, they'll just be unused. Thanks,
I'm not sure I understand this.
BY DEFINITION, the smallest assignable unit is the group, surely? Isn't that the *point* of the groups? That the IOMMU cannot tell the difference between the devices in the group?
So in *practice*, surely you cannot assign just *one* device from a group? You can assign the while group, or nothing.
Quoted 6 lines
You might *pretend* to assign single devices, and we might try to cope with the weirdness that happens when you want *one* device of a group to be owned by one VM, another device in the same group to be owned by a second VM, and a third device from the same group to be driven by a native driver in the host. But why not just assign groups as a whole? Surely that makes most sense?
We've got multiple levels when we add qemu and guests into the mix. A group is the smallest assignable unit for kernel->userspace. In fact, vfio is constructed so that the user cannot do anything with a group until all the devices of the group are bound to the vfio bus drivers. Qemu, as a userspace driver, must therefore take ownership of the entire group. However, there's no requirement that a userspace driver must make use of all the devices in the group, so qemu is free to expose individual devices from the group to the guest. IMHO, it doesn't make sense to have a default model saying "I know you just wanted the nic, but it's in the same group as this graphics card, so surprise, you get both!". Obviously if a user does want to expose multiple devices from a group to a guest, we support that too.
Spitting groups among multiple VMs or between VM and native host drivers defeats the purpose of the group. Neither of these are allowed.
Quoted 5 lines
Btw, did we get a quirk for the Ricoh multi-function devices which all need to be in the same group because they do all their DMA from function zero? I think we need another similar quirk for a Marvell SATA controller which seems to do its AHCI DMA from its IDE function; see
https://bugzilla.redhat.com/757166
No, as I mentioned, groups are currently for iommu_ops, not dma_ops, though it makes sense that iommu drivers could use the group info or create common quirk infrastructure for handling broken devices like these. Thanks,
Alex
Re: [PATCH 1/4] iommu: Add iommu_device_group callback and iommu_group sysfs entry
Quoted 8 lines · Alex Williamson
We've got multiple levels when we add qemu and guests into the mix. A group is the smallest assignable unit for
kernel->userspace. In fact, vfio is constructed so that the user cannot do anything with a group until all the devices of the group are bound to the vfio bus drivers. Qemu, as a userspace driver, must therefore take ownership of the entire group. However, there's no requirement that a userspace driver must make use of all the devices in the group, so qemu is free to expose individual devices from the group to the guest.
Sure but that has nothing to do with your kernel->user API. As you said, you still need to take "ownership" of the entire group. If qemu chose to only present to the guest part of that group, it's qemu's problem.
Put it differently. One day, maybe, we'll finally get some clues and implement a proper way for admin tools to know what can or cannot be put into a guest. In any case, how would you present to the user (administrator) the groups ? By individual devices with odd behaviours / errors etc... when the said user/admin tries to put individual devices into different partitions while they belong in the same group ? Or by just exposing ... groups ?
So while those various tools will want to see what's in the group (to represent it to the user, to expose it to the guest, for other finer grained operations such as MSI setup, etc...) the basic unit of ownership is and remains the group, and I don't see how it makes sense to have your underlying iommu interface operate on anything else.
Quoted 5 lines
IMHO, it doesn't make
sense to have a default model saying "I know you just wanted the nic, but it's in the same group as this graphics card, so surprise, you get both!". Obviously if a user does want to expose multiple devices from a group to a guest, we support that too.
No you don't get it. That's the wrong way around. The user will be presented to start with with a group of nic+graphics card so that user knows from the beginning what's going on.
Because your other option is to put the nic in ... and suddenly have the graphic card go away from the host without showing up in a guest. That won't be any better than having it "just show up" in the guest without asking for it.
IE. In both cases, it's wrong. The only thing that makes sense is from the beginning, expose those two as a group so that the user has no other choice but put them both into the guest at once and have it represented to the user as such to begin with.
Quoted 9 lines
Spitting groups among multiple VMs or between VM and native host drivers defeats the purpose of the group. Neither of these are allowed.
Quoted 5 lines
Btw, did we get a quirk for the Ricoh multi-function devices which all need to be in the same group because they do all their DMA from function zero? I think we need another similar quirk for a Marvell SATA controller which seems to do its AHCI DMA from its IDE function; see
https://bugzilla.redhat.com/757166
Quoted 4 lines
No, as I mentioned, groups are currently for
iommu_ops, notdma_ops, though it makes sense that iommu drivers could use the group info or create common quirk infrastructure for handling broken devices like these. Thanks,
Which is why the group info in iommu_ops should just be a representation of something else under the hood that needs to be sorted out more clearly than what we are doing at the moment.
Cheers,
Ben.
Re: [PATCH 1/4] iommu: Add iommu_device_group callback and iommu_group sysfs entry
Quoted 15 lines · Benjamin Herrenschmidt
Quoted 8 lines · Alex Williamson
We've got multiple levels when we add qemu and guests into the mix. A group is the smallest assignable unit for
kernel->userspace. In fact, vfio is constructed so that the user cannot do anything with a group until all the devices of the group are bound to the vfio bus drivers. Qemu, as a userspace driver, must therefore take ownership of the entire group. However, there's no requirement that a userspace driver must make use of all the devices in the group, so qemu is free to expose individual devices from the group to the guest.Sure but that has nothing to do with your
kernel->userAPI. As you said, you still need to take "ownership" of the entire group. If qemu chose to only present to the guest part of that group, it's qemu's problem.
Quoted 7 lines
Put it differently. One day, maybe, we'll finally get some clues and implement a proper way for admin tools to know what can or cannot be put into a guest. In any case, how would you present to the user (administrator) the groups ? By individual devices with odd behaviours / errors etc... when the said user/admin tries to put individual devices into different partitions while they belong in the same group ? Or by just exposing ... groups ?
Why are we even talking about this here? We both agree on the kernel->user ownership model and it's qemu's decision what to do from there. x86 will typically have groups with a single device. When that's not the case, it really doesn't seem that hard for an admin tool to notify the user and ask them if they want to also assign the dependent device or leave it unused. The complexity vs "hey, why did this other device just appear on my guest and disappear from my host" or even "why won't this stupid tool let me separate these two devices" feels like a wash.
Quoted 5 lines
So while those various tools will want to see what's in the group (to represent it to the user, to expose it to the guest, for other finer grained operations such as MSI setup, etc...) the basic unit of ownership is and remains the group, and I don't see how it makes sense to have your underlying iommu interface operate on anything else.
Are we going to tear apart every aspect of the IOMMU API, or can we use some pieces of it as it exists and incrementally improve it? Yes, if the IOMMU really can't distinguish between devices in a group, it probably doesn't matter which of them you attach to a domain. However, groups of devices aren't a basic unit of work for the rest of the kernel and even if the iommu device decoders can't tell the difference between transactions from hardware, the iommu driver might care for accounting purposes.
Quoted 15 lines
Quoted 5 lines
IMHO, it doesn't make
sense to have a default model saying "I know you just wanted the nic, but it's in the same group as this graphics card, so surprise, you get both!". Obviously if a user does want to expose multiple devices from a group to a guest, we support that too.No you don't get it. That's the wrong way around. The user will be presented to start with with a group of nic+graphics card so that user knows from the beginning what's going on.
Because your other option is to put the nic in ... and suddenly have the graphic card go away from the host without showing up in a guest. That won't be any better than having it "just show up" in the guest without asking for it.
Quoted 4 lines
IE. In both cases, it's wrong. The only thing that makes sense is from the beginning, expose those two as a group so that the user has no other choice but put them both into the guest at once and have it represented to the user as such to begin with.
People have very different ideas about what's intuitive and how interfaces should work, just look at gnome3/unity versus the useful desktops we used to have. Neither of us are likely to write the piece of management code that does this and the design decisions we're making here enable either usage model.
Quoted 9 lines
Quoted 9 lines
Spitting groups among multiple VMs or between VM and native host drivers defeats the purpose of the group. Neither of these are allowed.
Quoted 5 lines
Btw, did we get a quirk for the Ricoh multi-function devices which all need to be in the same group because they do all their DMA from function zero? I think we need another similar quirk for a Marvell SATA controller which seems to do its AHCI DMA from its IDE function; see
https://bugzilla.redhat.com/757166
Quoted 8 lines
Quoted 4 lines
No, as I mentioned, groups are currently for
iommu_ops, notdma_ops, though it makes sense that iommu drivers could use the group info or create common quirk infrastructure for handling broken devices like these. Thanks,Which is why the group info in
iommu_opsshould just be a representation of something else under the hood that needs to be sorted out more clearly than what we are doing at the moment.
I will be happy to review your patches to transform groups from an identifier into an object and make them the pervasive unit of work in the iommu layer. The above problem can be solved by:
struct pci_dev *pci_quirk_dma_dev(struct pci_dev *pdev) {
if (some match code)
return lookup different device
return pdev
}
Which would get called by both the dma_ops code and iommu_device_group. Thanks,
Alex
Re: [PATCH 1/4] iommu: Add iommu_device_group callback and iommu_group sysfs entry
In reply to Alex WilliamsonQuoted 7 lines · Alex Williamson
Quoted 5 lines
Btw, did we get a quirk for the Ricoh multi-function devices which all need to be in the same group because they do all their DMA from function zero? I think we need another similar quirk for a Marvell SATA controller which seems to do its AHCI DMA from its IDE function; see
https://bugzilla.redhat.com/757166No, as I mentioned, groups are currently for
iommu_ops, notdma_ops,
That doesn't matter. There are multiple devices which do all their DMA from the same source-id, and which have to be handled together. It doesn't *matter* that it's a hardware bug in this case, and it's just because they're multiple PCI devices behind a PCIe bridge in the other case. The grouping code needs to handle it, whether it's for the benefit of iommu_ops, dma_ops, or both.
Quoted 3 lines
though it makes sense that iommu drivers could use the group info or create common quirk infrastructure for handling broken devices like these.
Well, I think we mostly have a consensus that dma_ops should be implemented *using* iommu_ops; I'd rather be trying to make things *consistent* between the two rather than building up the differences.
So I'd definitely go for your "use the group info" option. It is the *same* info, and needs the *same* quirks.
Signature · 1 line
dwmw2
Re: [PATCH 1/4] iommu: Add iommu_device_group callback and iommu_group sysfs entry
Quoted 20 lines · David Woodhouse
Quoted 7 lines · Alex Williamson
Quoted 5 lines
Btw, did we get a quirk for the Ricoh multi-function devices which all need to be in the same group because they do all their DMA from function zero? I think we need another similar quirk for a Marvell SATA controller which seems to do its AHCI DMA from its IDE function; see
https://bugzilla.redhat.com/757166No, as I mentioned, groups are currently for
iommu_ops, notdma_ops,That doesn't matter. There are multiple devices which do all their DMA from the same source-id, and which have to be handled together. It doesn't *matter* that it's a hardware bug in this case, and it's just because they're multiple PCI devices behind a PCIe bridge in the other case. The grouping code needs to handle it, whether it's for the benefit of
iommu_ops,dma_ops, or both.Quoted 3 lines
though it makes sense that iommu drivers could use the group info or create common quirk infrastructure for handling broken devices like these.
Quoted 6 lines
Well, I think we mostly have a consensus that
dma_opsshould be implemented *using*iommu_ops; I'd rather be trying to make things *consistent* between the two rather than building up the differences.So I'd definitely go for your "use the group info" option. It is the *same* info, and needs the *same* quirks.
The problem at this stage is that all implementations proposed for the group info (both Alex and David) are somewhat higher level stuff that is itself constructed from the underlying iommu implementation.
And this is probably the right thing to do to avoid breaking too much stuff at once.
I think the best approach for devices like that is to have a header quirk setting a new bit flag in pci_dev along the line of "dma_group_functions" which causes the platform iommu implementation to "do the right thing" and in turn create the isolation groups accordingly as well.
Later on, we might want to revisit but I think it's nasty enough to leave it as-is. The creation of isolation groups will have to answer to all sort of other platform specific constraints so I'd rather not try to put too much of that logic in generic code.
Cheers,
Ben.
Re: [PATCH 1/4] iommu: Add iommu_device_group callback and iommu_group sysfs entry
In reply to Alex WilliamsonQuoted 2 lines · Alex Williamson
See
bus_set_iommu(struct bus_type *bus, struct iommu_ops *ops). It wasn't long ago that we had a globaliommu_ops.
Right now, the dma_ops are per-device but the iommu populate/depopulate hooks are system global, installed by the platform code at boot time, on powerpc. We could try to "fit in" an iommu "driver" at that point.
In theory, you can connect separate IO chips on POWER server and in fact it happens on high end boxes, tho they tend to have a similar "PE#" mechanism, but with different constraints and resources capacities. However we aren't concerned with that initially for KVM as we are targetting only lower end machines with only one p7ioc for now.
But I can see more heterogenous setup showing up in embedded space..
Quoted 5 lines
Are heterogeneous
iommus using different drivers on the samebus_typereally a possibility on your system, or are we dealing with hypothetical situations? You'd currently need to multiplexiommu_opscallbacks between the drivers yourself.
We could.
Ben.
PATCH 2/4 intel-iommu: Implement iommu_device_group
In reply to Alex WilliamsonWe generally have BDF granularity for devices, so we just need to make sure devices aren't hidden behind PCIe-to-PCI bridges. We can then make up a group number that's simply the concatenated seg|bus|dev|fn so we don't have to track them (not that users should depend on that).
---
drivers/iommu/intel-iommu.c | 44 +++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 44 insertions(+), 0 deletions(-)
drivers/iommu/intel-iommu.c +44 -0
@@ -3901,6 +3901,49 @@ static int intel_iommu_domain_has_cap(struct iommu_domain *domain,return 0 ; } +/* Group numbers are arbitrary. Device with the same group number +* indicate the iommu cannot differentiate between them. To avoid +* tracking used groups we just use the seg|bus|devfn of the lowest +* level we're able to differentiate devices */ +static int intel_iommu_device_group (struct device *dev,unsigned int *groupid) +{ +struct pci_dev *pdev =to_pci_dev (dev); +struct pci_dev *bridge; +union { +struct { +u8 devfn; +u8 bus; +u16 segment; + } pci; +u32 group; + } id; + +if (iommu_no_mapping (dev)) +return -ENODEV ; + + id.pci.segment =pci_domain_nr (pdev->bus); + id.pci.bus = pdev->bus->number; + id.pci.devfn = pdev->devfn; + +if (!device_to_iommu (id.pci.segment, id.pci.bus, id.pci.devfn)) +return -ENODEV ; + + bridge =pci_find_upstream_pcie_bridge (pdev); +if (bridge) { +if (pci_is_pcie (bridge)) { + id.pci.bus = bridge->subordinate->number; + id.pci.devfn =0 ; + }else { + id.pci.bus = bridge->bus->number; + id.pci.devfn = bridge->devfn; + } + } + + *groupid = id.group; + +return 0 ; +} +static struct iommu_ops intel_iommu_ops = { .domain_init = intel_iommu_domain_init, .domain_destroy = intel_iommu_domain_destroy,
@@ -3910,6 +3953,7 @@ static struct iommu_ops intel_iommu_ops = { .unmap = intel_iommu_unmap, .iova_to_phys = intel_iommu_iova_to_phys, .domain_has_cap = intel_iommu_domain_has_cap, + .device_group = intel_iommu_device_group, };static void __devinit quirk_iommu_rwbf (struct pci_dev *dev)
Re: [PATCH 2/4] intel-iommu: Implement iommu_device_group
Quoted 7 lines · Alex Williamson
We generally have BDF granularity for devices, so we just need to make sure devices aren't hidden behind PCIe-to-PCI bridges. We can then make up a group number that's simply the concatenated seg|bus|dev|fn so we don't have to track them (not that users should depend on that).
Signed-off-by Alex Williamson <[email protected]>
David, does this look good to you? Can you send an Acked-by if its okay?
Signature · 5 lines
AMD Operating System Research Center Advanced Micro Devices GmbH Einsteinring 24 85609 Dornach General Managers: Alberto Bozzo, Andrew Bowd Registration: Dornach, Landkr. Muenchen; Registerger. Muenchen, HRB Nr. 43632
David Woodhouse
Acked-byRe: [PATCH 2/4] intel-iommu: Implement iommu_device_group
Quoted 10 lines · Joerg
Quoted 7 lines · Alex Williamson
We generally have BDF granularity for devices, so we just need to make sure devices aren't hidden behind PCIe-to-PCI bridges. We can then make up a group number that's simply the concatenated seg|bus|dev|fn so we don't have to track them (not that users should depend on that).
Signed-off-by Alex Williamson <[email protected]>David, does this look good to you? Can you send an Acked-by if its okay?
Looks good to me in principle, although I'd like to see it tested on one of those HP laptops with the dodgy Ricoh multi-function devices.
Signature · 1 line
dwmw2
PATCH 3/4 amd-iommu: Implement iommu_device_group
In reply to Alex WilliamsonJust use the amd_iommu_alias_table directly.
---
drivers/iommu/amd_iommu.c | 13 +++++++++++++ 1 files changed, 13 insertions(+), 0 deletions(-)
drivers/iommu/amd_iommu.c +13 -0
@@ -2773,6 +2773,18 @@ static int amd_iommu_domain_has_cap(struct iommu_domain *domain,return 0 ; } +static int amd_iommu_device_group (struct device *dev,unsigned int *groupid) +{ +struct iommu_dev_data *dev_data = dev->archdata.iommu; + +if (!dev_data) +return -ENODEV ; + + *groupid = amd_iommu_alias_table[dev_data->devid]; + +return 0 ; +} +static struct iommu_ops amd_iommu_ops = { .domain_init = amd_iommu_domain_init, .domain_destroy = amd_iommu_domain_destroy,
@@ -2782,6 +2794,7 @@ static struct iommu_ops amd_iommu_ops = { .unmap = amd_iommu_unmap, .iova_to_phys = amd_iommu_iova_to_phys, .domain_has_cap = amd_iommu_domain_has_cap, + .device_group = amd_iommu_device_group, };/*****************************************************************************
PATCH 4/4 iommu: Add option to group multi-function devices
In reply to Alex WilliamsonThe option iommu=group_mf indicates the that the iommu driver should expose all functions of a multi-function PCI device as the same iommu_device_group. This is useful for disallowing individual functions being exposed as independent devices to userspace as there are often hidden dependencies. Virtual functions are not affected by this option.
---
Documentation/kernel-parameters.txt | 4 +++- arch/ia64/include/asm/iommu.h | 2 ++ arch/ia64/kernel/pci-dma.c | 1 + arch/x86/include/asm/iommu.h | 1 + arch/x86/kernel/pci-dma.c | 11 +++++++++++ drivers/iommu/amd_iommu.c | 10 +++++++++- drivers/iommu/intel-iommu.c | 3 +++ 7 files changed, 30 insertions(+), 2 deletions(-)
Documentation/kernel-parameters.txt +3 -1
@@ -1036,7 +1036,9 @@ bytes respectively. Such letter suffixes can also be entirely omitted. nomerge forcesac soft - pt [x86, IA-64] + pt [x86, IA-64] + group_mf [x86, IA-64] + io7= [HW] IO7 for Marvel based alpha systems See comment before marvel_specify_io7 in
arch/ia64/include/asm/iommu.h +2 -0
@@ -11,8 +11,10 @@ extern int force_iommu, no_iommu;extern int iommu_detected;#ifdef CONFIG_DMAR extern int iommu_pass_through; +extern int iommu_group_mf;#else #define iommu_pass_through (0) +#define iommu_group_mf (0)#endif extern void iommu_dma_init (void );extern void machvec_init (const char *name);
arch/ia64/kernel/pci-dma.c +1 -0
@@ -33,6 +33,7 @@ int force_iommu __read_mostly;#endif int iommu_pass_through; +int iommu_group_mf;/* Dummy device used for NULL arguments (normally ISA). Better would be probably a smaller DMA mask, but this is bug-to-bug compatible
arch/x86/include/asm/iommu.h +1 -0
@@ -5,6 +5,7 @@ extern struct dma_map_ops nommu_dma_ops;extern int force_iommu, no_iommu;extern int iommu_detected;extern int iommu_pass_through; +extern int iommu_group_mf;/* 10 seconds */ #define DMAR_OPERATION_TIMEOUT ((cycles_t) tsc_khz*10*1000)
arch/x86/kernel/pci-dma.c +11 -0
@@ -44,6 +44,15 @@ int iommu_detected __read_mostly = 0;*/ int iommu_pass_through__read_mostly ; +/* +* Group multi-function PCI devices into a single device-group for the +* iommu_device_group interface. This tells the iommu driver to pretend +* it cannot distinguish between functions of a device, exposing only one +* group for the device. Useful for disallowing use of individual PCI +* functions from userspace drivers. +*/ +int iommu_group_mf__read_mostly ; +extern struct iommu_table_entry __iommu_table[], __iommu_table_end[];/* Dummy device used for NULL arguments (normally ISA). */
@@ -168,6 +177,8 @@ static __init int iommu_setup(char *p)#endif if (!strncmp (p,"pt" ,2 )) iommu_pass_through =1 ; +if (!strncmp (p,"group_mf" ,8 )) + iommu_group_mf =1 ;gart_parse_options (p);
drivers/iommu/amd_iommu.c +9 -1
@@ -2776,11 +2776,19 @@ static int amd_iommu_domain_has_cap(struct iommu_domain *domain,static int amd_iommu_device_group (struct device *dev,unsigned int *groupid) {struct iommu_dev_data *dev_data = dev->archdata.iommu; +struct pci_dev *pdev =to_pci_dev (dev); +u16 devid;if (!dev_data)return -ENODEV ; - *groupid = amd_iommu_alias_table[dev_data->devid]; +if (pdev->is_virtfn || !iommu_group_mf) + devid = dev_data->devid; +else + devid =calc_devid (pdev->bus->number, +PCI_DEVFN (PCI_SLOT (pdev->devfn),0 )); + + *groupid = amd_iommu_alias_table[devid];return 0 ; }
drivers/iommu/intel-iommu.c +3 -0
@@ -3939,6 +3939,9 @@ static int intel_iommu_device_group(struct device *dev, unsigned int *groupid) } } +if (!pdev->is_virtfn && iommu_group_mf) + id.pci.devfn =PCI_DEVFN (PCI_SLOT (id.pci.devfn),0 ); + *groupid = id.group;return 0 ;
Re: [PATCH 4/4] iommu: Add option to group multi-function devices
Quoted 5 lines · Alex Williamson
The option iommu=group_mf indicates the that the iommu driver should expose all functions of a multi-function PCI device as the same
iommu_device_group. This is useful for disallowing individual functions being exposed as independent devices to userspace as there are often hidden dependencies. Virtual functions are not affected by this option.
This seems ok as a testing/debugging hack. But in general, surely, whether to group the functions should be a device quirk.
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Quoted 5 lines · Alex Williamson
IOMMUs can't always distiguish transactions from each individual device in a system. Sometimes this is by design (such as powerpc partitionable endpoints), other times by topology (PCIe-to-PCI bridges masking downstream devices). We call these sets of indistinguishable devices "groups".
Other times it's because a multi-function PCIe device is broken and does all its DMA from function zero.... like some Ricoh devices seen in laptops. Can you handle quirks to "group" those too?
Signature · 4 lines
Sent with MeeGo's ActiveSync support. David Woodhouse Open Source Technology Centre [email protected] Intel Corporation
Quoted 10 lines · David
Quoted 5 lines · Alex Williamson
IOMMUs can't always distiguish transactions from each individual device in a system. Sometimes this is by design (such as powerpc partitionable endpoints), other times by topology (PCIe-to-PCI bridges masking downstream devices). We call these sets of indistinguishable devices "groups".
Other times it's because a multi-function PCIe device is broken and does all its DMA from function zero.... like some Ricoh devices seen in laptops. Can you handle quirks to "group" those too?
I could add:
drivers/pci/quirks.c: int pci_iommu_group_mf_quirk(struct pci_dev *)
That's used just like iommu_group_mf to always blacklist specific devices. Let me know VID/DID if you have them. Thanks,
Alex
Quoted 6 lines · Alex Williamson
drivers/pci/quirks.c: intpci_iommu_group_mf_quirk(struct pci_dev *)That's used just like
iommu_group_mfto always blacklist specific devices. Let me know VID/DID if you have them. Thanks,
Don't remember the ID offhand, but there are reports in Red Hat bugzilla of Ricoh multifunction devices on some (HP, I think) laptops which cause streams of DMA errors unless the sdhci driver is blacklisted...
Signature · 4 lines
Sent with MeeGo's ActiveSync support. David Woodhouse Open Source Technology Centre [email protected] Intel Corporation
Quoted 12 lines · Alex Williamson
Quoted 10 lines · David
Quoted 5 lines · Alex Williamson
IOMMUs can't always distiguish transactions from each individual device in a system. Sometimes this is by design (such as powerpc partitionable endpoints), other times by topology (PCIe-to-PCI bridges masking downstream devices). We call these sets of indistinguishable devices "groups".
Other times it's because a multi-function PCIe device is broken and does all its DMA from function zero.... like some Ricoh devices seen in laptops. Can you handle quirks to "group" those too?
Quoted 6 lines
I could add:
drivers/pci/quirks.c: intpci_iommu_group_mf_quirk(struct pci_dev *)That's used just like
iommu_group_mfto always blacklist specific devices. Let me know VID/DID if you have them. Thanks,
I guess just a bit and a quirk is easiest, something like below (quirking an 82576 just as an example). Thanks,
Alex
drivers/iommu/amd_iommu.c +3 -3
@@ -2782,11 +2782,11 @@ static int amd_iommu_device_group(struct device *dev, unsigned int *groupid)if (!dev_data)return -ENODEV ; -if (pdev->is_virtfn || !iommu_group_mf) - devid = dev_data->devid; -else +if (pdev->group_mf || (iommu_group_mf && !pdev->is_virtfn)) devid =calc_devid (pdev->bus->number,PCI_DEVFN (PCI_SLOT (pdev->devfn),0 )); +else + devid = dev_data->devid; *groupid = amd_iommu_alias_table[devid];
drivers/iommu/intel-iommu.c +1 -1
@@ -3939,7 +3939,7 @@ static int intel_iommu_device_group(struct device *dev, unsigned int *groupid) } } -if (!pdev->is_virtfn && iommu_group_mf) +if (pdev->group_mf || (iommu_group_mf && !pdev->is_virtfn)) id.pci.devfn =PCI_DEVFN (PCI_SLOT (id.pci.devfn),0 ); *groupid = id.group;
drivers/pci/quirks.c +6 -0
@@ -3009,3 +3009,9 @@ int pci_dev_specific_reset(struct pci_dev *dev, int probe)return -ENOTTY ; } + +static void __devinit quirk_iommu_group_mf (struct pci_dev * dev) +{ + dev->group_mf =1 ; +} +DECLARE_PCI_FIXUP_EARLY (PCI_VENDOR_ID_INTEL ,0x10c9 , quirk_iommu_group_mf);
include/linux/pci.h +1 -0
@@ -322,6 +322,7 @@ struct pci_dev { unsigned int is_hotplug_bridge:1; unsigned int __aer_firmware_first_valid:1; unsigned int __aer_firmware_first:1; + unsigned int group_mf:1; /* per device iommu=group_mf */pci_dev_flags_t dev_flags;atomic_t enable_cnt;/* pci_enable_device has been called */
Quoted 20 lines · Alex Williamson
IOMMUs can't always distiguish transactions from each individual device in a system. Sometimes this is by design (such as powerpc partitionable endpoints), other times by topology (PCIe-to-PCI bridges masking downstream devices). We call these sets of indistinguishable devices "groups".
In order to support secure userspace drivers, like vfio, we need an interface to expose the device-to-group relationship. This allows us to create policies ensuring that userspace controls all of the devices in the group before allowing individual device access.
This series implements the
iommu_opsAPI interface and sysfs interface for exposing groups to userspace. This also includes the intel-iommu and amd-iommu backend implementations. It's intended that the vfio driver will make use of these interfaces to support generic device assignment for virtual machines. Seegit://github.com/awilliam/linux-vfio.git(vfio-ng) for a working example using this interface.
Quoted 4 lines
Patches based on Joerg's next branch to support per-bus
iommu_ops.Note the amd-iommu is untested, I'm still working on setting up an AMD-Vi capable system. Thanks,
Now tested, even seems to work :)
Alex
Quoted 19 lines
---
Alex Williamson (4):
iommu: Add option to group multi-function devices
amd-iommu: Implementiommu_device_group
intel-iommu: Implementiommu_device_group
iommu: Addiommu_device_groupcallback andiommu_groupsysfs entryDocumentation/kernel-parameters.txt | 4 ++ arch/ia64/include/asm/iommu.h | 2 + arch/ia64/kernel/pci-dma.c | 1 + arch/x86/include/asm/iommu.h | 1 + arch/x86/kernel/pci-dma.c | 11 ++++++ drivers/iommu/amd_iommu.c | 21 ++++++++++++ drivers/iommu/intel-iommu.c | 47 +++++++++++++++++++++++++++ drivers/iommu/iommu.c | 61 +++++++++++++++++++++++++++++++++++ include/linux/iommu.h | 7 ++++ 9 files changed, 154 insertions(+), 1 deletions(-)
Quoted 7 lines · Alex Williamson
Alex Williamson (4):
iommu: Add option to group multi-function devices
amd-iommu: Implementiommu_device_group
intel-iommu: Implementiommu_device_group
iommu: Addiommu_device_groupcallback andiommu_groupsysfs entry
Quoted 10 lines
Documentation/kernel-parameters.txt | 4 ++ arch/ia64/include/asm/iommu.h | 2 + arch/ia64/kernel/pci-dma.c | 1 + arch/x86/include/asm/iommu.h | 1 + arch/x86/kernel/pci-dma.c | 11 ++++++ drivers/iommu/amd_iommu.c | 21 ++++++++++++ drivers/iommu/intel-iommu.c | 47 +++++++++++++++++++++++++++ drivers/iommu/iommu.c | 61 +++++++++++++++++++++++++++++++++++ include/linux/iommu.h | 7 ++++ 9 files changed, 154 insertions(+), 1 deletions(-)
Applied with some changes to patch 1. I basically split up the add_remove function into a add and a remove function. I think the code is cleaner that way.
Joerg
Signature · 5 lines
AMD Operating System Research Center Advanced Micro Devices GmbH Einsteinring 24 85609 Dornach General Managers: Alberto Bozzo, Andrew Bowd Registration: Dornach, Landkr. Muenchen; Registerger. Muenchen, HRB Nr. 43632