iommu: Reserved regions for IOVAs beyond dma_mask and iommu aperture
Subsystems (2)
cover iommu: Reserved regions for IOVAs beyond dma_mask and iommu aperture
VFIO currently exposes the usable IOVA regions through the VFIO_IOMMU_GET_INFO ioctl / VFIO_IOMMU_TYPE1_INFO_CAP_IOVA_RANGE capability. However it fails to take into account the dma_mask of the devices within the container. The top limit currently is defined by the iommu aperture.
So, for instance, if the IOMMU supports up to 48bits, it may give the impression the max IOVA is 48b while a device may have a dma_mask of 42b. So this API cannot really be used to compute the max usable IOVA.
This patch removes the IOVA region beyond the dma_mask's. As we start to expose this reserved region in the sysfs file /sys/kernel/iommu_groups/<n>/reserved_regions, we also need to handle the IOVA range beyond the IOMMU aperture to handle the case where the dma_mask would have a higher number of bits than the iommu max input address.
This is a change to the ABI as this reserved region was not yet exposed in sysfs /sys/kernel/iommu_groups/<n>/reserved_regions or through the VFIO ioctl. At VFIO level we increment the version of the VFIO_IOMMU_TYPE1_INFO_CAP_IOVA_RANGE capability to advertise that change.
Best Regards
Eric
This series can be found at:
https://github.com/eauger/qemu/tree/dma_mask_rfc
Eric Auger (3):
iommu: Fix merging in iommu_insert_resv_region
iommu: Account for dma_mask and iommu aperture in IOVA reserved
regions
vfio/type1: Increase the version of
VFIO_IOMMU_TYPE1_INFO_CAP_IOVA_RANGE
.../ABI/testing/sysfs-kernel-iommu_groups | 7 ++++ drivers/iommu/iommu.c | 41 ++++++++++++++++++- drivers/vfio/vfio_iommu_type1.c | 2 +- 3 files changed, 48 insertions(+), 2 deletions(-)
Signature · 1 line
2.21.3
RFC 1/3 iommu: Fix merging in iommu_insert_resv_region
We currently fail to merge a region into another one whose top address is ULLONG_MAX. This situation shouldn't have been encountered yet due to the nature of reserved regions being exposed but this would happen if we were to expose regions beyond the reach of dma_mask or beyond the reach of the iommu.
--- drivers/iommu/iommu.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/iommu/iommu.c +1 -1
@@ -423,7 +423,7 @@ static int iommu_insert_resv_region(struct iommu_resv_region *new, check_overlap: top_end = top->start + top->length -1 ; -if (iter->start > top_end +1 ) { +if (top_end !=ULLONG_MAX && iter->start > top_end +1 ) {list_move_tail (&iter->list, &stack); }else { top->length =max (top_end, iter_end) - top->start +1 ;
Signature · 1 line
2.21.3
RFC 2/3 iommu: Account for dma_mask and iommu aperture in IOVA reserved regions
In reply to Eric AugerVFIO currently exposes the usable IOVA regions through the VFIO_IOMMU_GET_INFO ioctl. However it fails to take into account the dma_mask of the devices within the container. The top limit currently is defined by the iommu aperture.
So, for instance, if the IOMMU supports up to 48bits, it may give the impression the max IOVA is 48b while a device may have a dma_mask of 42b. So this API cannot really be used to compute the max usable IOVA.
This patch removes the IOVA region beyond the dma_mask's. As we start to expose this reserved region in the sysfs file /sys/kernel/iommu_groups/<n>/reserved_regions, we also need to expose the IOVA range beyond the IOMMU aperture to handle the case where the dma_mask would have a higher number of bits than the iommu max input address. Those out-of-reach regions get the IOMMU_RESV_RESERVED type.
This is a change to the ABI as this reserved region was not yet exposed in sysfs /sys/kernel/iommu_groups/<n>/reserved_regions or through the VFIO ioctl. Document that change.
--- .../ABI/testing/sysfs-kernel-iommu_groups | 7 ++++ drivers/iommu/iommu.c | 39 +++++++++++++++++++ 2 files changed, 46 insertions(+)
Documentation/ABI/testing/sysfs-kernel-iommu_groups +7 -0
@@ -33,3 +33,10 @@ Description: In case an RMRR is used only by graphics or USB devices it is now exposed as "direct-relaxable" instead of "direct". In device assignment use case, for instance, those RMRR are considered to be relaxable and safe. + +What: /sys/kernel/iommu_groups/reserved_regions +Date: Sept 2020 +KernelVersion: v5.11 +Contact: Eric Auger <[email protected]> +Description: Regions beyond the device dma_mask and the iommu aperture + now are exposed as IOMMU_RESV_RESERVED reserved regions.
drivers/iommu/iommu.c +39 -0
@@ -2511,9 +2511,48 @@ EXPORT_SYMBOL_GPL(iommu_domain_set_attr);void iommu_get_resv_regions (struct device *dev,struct list_head *list) {const struct iommu_ops *ops = dev->bus->iommu_ops; +struct iommu_resv_region *region; +struct iommu_domain *domain; + + domain =iommu_get_domain_for_dev (dev); + +if (domain) { +struct iommu_domain_geometry geo; + +if (iommu_domain_get_attr (domain,DOMAIN_ATTR_GEOMETRY , &geo)) +return ; + +if (geo.aperture_end <ULLONG_MAX && geo.aperture_end != geo.aperture_start) { + region =iommu_alloc_resv_region (geo.aperture_end +1 , +ULLONG_MAX - geo.aperture_end, +0 ,IOMMU_RESV_RESERVED ); +if (!region) +return ; +list_add_tail (®ion->list, list); + } + +if (geo.aperture_start >0 ) { + region =iommu_alloc_resv_region (0 , geo.aperture_start, +0 ,IOMMU_RESV_RESERVED ); +if (!region) +return ; +list_add_tail (®ion->list, list); + } + }if (ops && ops->get_resv_regions) ops->get_resv_regions (dev, list); + +if (!dev->dma_mask || *dev->dma_mask ==ULLONG_MAX ) +return ; + + region =iommu_alloc_resv_region (*dev->dma_mask +1 , +ULLONG_MAX - *dev->dma_mask, +0 ,IOMMU_RESV_RESERVED ); +if (!region) +return ; + +list_add_tail (®ion->list, list); }void iommu_put_resv_regions (struct device *dev,struct list_head *list)
Signature · 6 lines
2.21.3 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC 2/3] iommu: Account for dma_mask and iommu aperture in IOVA reserved regions
Quoted 4 lines · Eric Auger
VFIO currently exposes the usable IOVA regions through the
VFIO_IOMMU_GET_INFOioctl. However it fails to take into account thedma_maskof the devices within the container. The top limit currently is defined by the iommu aperture.
Can we take a step back here? The dma_mask only has a meaning for the DMA API, and not the iommu API, it should have no relevance here.
More importantly if we are using vfio no dma_mask should be set to start with.
drivers/iommu/iommu.c EXPORT_SYMBOL_GPL(iommu_domain_set_attr); · line 2525
Quoted 1 line
drivers/iommu/iommu.c +1
-0+if (geo.aperture_end <ULLONG_MAX && geo.aperture_end != geo.aperture_start) {
Please avoid pointlessly overlong lines.
Re: [RFC 2/3] iommu: Account for dma_mask and iommu aperture in IOVA reserved regions
Hi Christoph,
Quoted 11 lines · Christoph Hellwig
Quoted 4 lines · Eric Auger
VFIO currently exposes the usable IOVA regions through the
VFIO_IOMMU_GET_INFOioctl. However it fails to take into account thedma_maskof the devices within the container. The top limit currently is defined by the iommu aperture.Can we take a step back here? The
dma_maskonly has a meaning for the DMA API, and not the iommu API, it should have no relevance here.More importantly if we are using vfio no
dma_maskshould be set to start with.
You will find more context in my reply to Alex.
Thanks
Eric
Quoted 5 lines
drivers/iommu/iommu.c EXPORT_SYMBOL_GPL(iommu_domain_set_attr); · line 2525
Quoted 2 lines
drivers/iommu/iommu.c +1
-0+if (geo.aperture_end <ULLONG_MAX && geo.aperture_end != geo.aperture_start) {Please avoid pointlessly overlong lines.
RFC 3/3 vfio/type1: Increase the version of VFIO_IOMMU_TYPE1_INFO_CAP_IOVA_RANGE
In reply to Eric AugerNow the IOVA regions beyond the dma_mask and the vfio aperture are removed from the usable IOVA ranges, the API becomes reliable to compute the max IOVA. Let's advertise this by using a new version for the capability.
--- drivers/vfio/vfio_iommu_type1.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/vfio/vfio_iommu_type1.c +1 -1
@@ -2541,7 +2541,7 @@ static int vfio_iommu_iova_add_cap(struct vfio_info_cap *caps,struct vfio_iommu_type1_info_cap_iova_range *iova_cap; header =vfio_info_cap_add (caps, size, -VFIO_IOMMU_TYPE1_INFO_CAP_IOVA_RANGE ,1 ); +VFIO_IOMMU_TYPE1_INFO_CAP_IOVA_RANGE ,2 );if (IS_ERR (header))return PTR_ERR (header);
Signature · 6 lines
2.21.3 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
Quoted 5 lines · On Mon, 28 Sep 2020 21:50:34 +0200 Eric Auger <[email protected]> wrote:
VFIO currently exposes the usable IOVA regions through the
VFIO_IOMMU_GET_INFOioctl /VFIO_IOMMU_TYPE1_INFO_CAP_IOVA_RANGEcapability. However it fails to take into account thedma_maskof the devices within the container. The top limit currently is defined by the iommu aperture.
I think that dma_mask is traditionally a DMA API interface for a device driver to indicate to the DMA layer which mappings are accessible to the device. On the other hand, vfio makes use of the IOMMU API where the driver is in userspace. That userspace driver has full control of the IOVA range of the device, therefore dma_mask is mostly irrelevant to vfio. I think the issue you're trying to tackle is that the IORT code is making use of the dma_mask to try to describe a DMA address limitation imposed by the PCI root bus, living between the endpoint device and the IOMMU. Therefore, if the IORT code is exposing a topology or system imposed device limitation, this seems much more akin to something like an MSI reserved range, where it's not necessarily the device or the IOMMU with the limitation, but something that sits between them.
Quoted 6 lines
So, for instance, if the IOMMU supports up to 48bits, it may give the impression the max IOVA is 48b while a device may have a
dma_maskof 42b. So this API cannot really be used to compute the max usable IOVA.This patch removes the IOVA region beyond the dma_mask's.
Rather it adds a reserved region accounting for the range above the device's dma_mask. I don't think the IOMMU API should be consuming dma_mask like this though. For example, what happens in pci_dma_configure() when there are no OF or ACPI DMA restrictions? It appears to me that the dma_mask from whatever previous driver had the device carries over to the new driver. That's generally ok for the DMA API because a driver is required to set the device's DMA mask. It doesn't make sense however to blindly consume that dma_mask and export it via an IOMMU API. For example I would expect to see different results depending on whether a host driver has been bound to a device. It seems the correct IOMMU API approach would be for the IORT code to specifically register reserved ranges for the device.
Quoted 5 lines
As we start to expose this reserved region in the sysfs file /sys/kernel/iommu_groups/<n>/reserved_regions, we also need to handle the IOVA range beyond the IOMMU aperture to handle the case where the
dma_maskwould have a higher number of bits than the iommu max input address.
Why? The IOMMU geometry already describes this and vfio combines both the IOMMU geometry and the device reserved regions when generating the IOVA ranges? Who is going to consume this information? Additionally it appears that reserved regions will report different information depending on whether a device is attached to a domain.
Quoted 5 lines
This is a change to the ABI as this reserved region was not yet exposed in sysfs /sys/kernel/iommu_groups/<n>/reserved_regions or through the VFIO ioctl. At VFIO level we increment the version of the
VFIO_IOMMU_TYPE1_INFO_CAP_IOVA_RANGEcapability to advertise that change.
Is this really an ABI change? The original entry for reserved regions includes:
Not necessarily all reserved regions are listed. This is typically used to output direct-mapped, MSI, non mappable regions.
I imagine the intention here was non-mappable relative to the IOMMU, but non-mappable to the device is essentially what we're including here.
I'm also concerned about bumping the vfio interface version for the IOVA range. We're not changing the interface, we're modifying the result, and even then only for a fraction of users. How many users are potentially broken by that change? Are we going to bump the version for everyone any time the result changes on any platform? Thanks,
Alex
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Hi all,
[also correcting some outdated email addresses + adding Lorenzo in cc]
Quoted 9 lines · Alex Williamson
Quoted 5 lines · On Mon, 28 Sep 2020 21:50:34 +0200 Eric Auger <[email protected]> wrote:
VFIO currently exposes the usable IOVA regions through the
VFIO_IOMMU_GET_INFOioctl /VFIO_IOMMU_TYPE1_INFO_CAP_IOVA_RANGEcapability. However it fails to take into account thedma_maskof the devices within the container. The top limit currently is defined by the iommu aperture.
Quoted 13 lines
I think that
dma_maskis traditionally a DMA API interface for a device driver to indicate to the DMA layer which mappings are accessible to the device. On the other hand, vfio makes use of the IOMMU API where the driver is in userspace. That userspace driver has full control of the IOVA range of the device, thereforedma_maskis mostly irrelevant to vfio. I think the issue you're trying to tackle is that the IORT code is making use of thedma_maskto try to describe a DMA address limitation imposed by the PCI root bus, living between the endpoint device and the IOMMU. Therefore, if the IORT code is exposing a topology or system imposed device limitation, this seems much more akin to something like an MSI reserved range, where it's not necessarily the device or the IOMMU with the limitation, but something that sits between them.
First I think I failed to explain the context. I worked on NVMe passthrough on ARM. The QEMU NVMe backend uses VFIO to program the physical device. The IOVA allocator there currently uses an IOVA range within [0x10000, 1ULL << 39]. This IOVA layout rather is arbitrary if I understand correctly. I noticed we rapidly get some VFIO MAP DMA failures because the allocated IOVA collide with the ARM MSI reserved IOVA window [0x8000000, 0x8100000]. Since 9b77e5c79840 ("vfio/type1:
Check reserved region conflict and update iova list"), such VFIO MAP DMA attempts to map IOVAs belonging to host reserved IOVA windows fail. So, by using the VFIO_IOMMU_GET_INFO ioctl / VFIO_IOMMU_TYPE1_INFO_CAP_IOVA_RANGE I can change the IOVA allocator to avoid allocating within this range and others. While working on this, I tried to automatically compute the min/max IOVAs and change the arbitrary [0x10000, 1ULL << 39]. My SMMUv2 supports up to 48b so naturally the max IOVA was computed as 1ULL << 48. The QEMU NVMe backend allocates at the bottom and at the top of the range. I noticed the use case was not working as soon as the top IOVA was more than 1ULL << 42. And then we noticed the dma_mask was set to 42 by using
cat /sys/bus/pci/devices/0005:01:00.0/dma_mask_bits. So my interpretation is the dma_mask was somehow containing the info the device couldn't handle IOVAs beyond a certain limit.
In my case the 42b limit is computed in iort_dma_setup() by acpi_dma_get_range(dev, &dmaaddr, &offset, &size);
Referring to the comment, it does "Evaluate DMA regions and return respectively DMA region start, offset and size in dma_addr, offset and size on parsing success". This parses the ACPI table, looking for ACPI companions with _DMA methods.
But as Alex mentioned, the IORT also allows to define limits on "the number of address bits, starting from the least significant bit that can be generated by a device when it accesses memory". See Named component node.Device Memory Address Size limit or PCI root complex node. Memory address size limit.
ret = acpi_dma_get_range(dev, &dmaaddr, &offset, &size);
if (ret == -ENODEV)
ret = dev_is_pci(dev) ? rc_dma_get_range(dev, &size)
: nc_dma_get_range(dev, &size);So eventually those info collected from the ACPI tables which do impact the usable IOVA range seem to be stored in the dma_mask, hence that proposal.
Quoted 10 lines
Quoted 7 lines
So, for instance, if the IOMMU supports up to 48bits, it may give the impression the max IOVA is 48b while a device may have a
dma_maskof 42b. So this API cannot really be used to compute the max usable IOVA.This patch removes the IOVA region beyond the dma_mask's.
Rather it adds a reserved region accounting for the range above the device's
dma_mask.
Yep. It adds new reserved regions in /sys/kernel/iommu_groups/<n>/reserved_regions and remove those from the usable regions exposed by VFIO GET_INFO.
I don't think the IOMMU API should be consuming
Quoted 2 lines
dma_masklike this though. For example, what happens inpci_dma_configure()when there are no OF or ACPI DMA restrictions?
My guess was that the dma_mask was set to the max range but I did not test it.
It
Quoted 8 lines
appears to me that the
dma_maskfrom whatever previous driver had the device carries over to the new driver. That's generally ok for the DMA API because a driver is required to set the device's DMA mask. It doesn't make sense however to blindly consume thatdma_maskand export it via an IOMMU API. For example I would expect to see different results depending on whether a host driver has been bound to a device. It seems the correct IOMMU API approach would be for the IORT code to specifically register reserved ranges for the device.
Is it only specific to IORT table? acpi_dma_get_range() in drivers/acpi/scan.c is generic.
Quoted 10 lines
Quoted 6 lines
As we start to expose this reserved region in the sysfs file /sys/kernel/iommu_groups/<n>/reserved_regions, we also need to handle the IOVA range beyond the IOMMU aperture to handle the case where the
dma_maskwould have a higher number of bits than the iommu max input address.Why? The IOMMU geometry already describes this and vfio combines both the IOMMU geometry and the device reserved regions when generating the IOVA ranges?
Yes VFIO layer does add the info about the topology but /sys/kernel/iommu_groups/<n>/reserved_regions, generated by the IOMMU code, does not. this latter only exposes reserved regions. Assume the dma_mask is 48b and the IOMMU aperture is 42b (assuming it is possible), if you only take into account the "dma_mask" limitation, the end-user will interpret this as: I can use up to 48b.
Who is going to consume this information? Additionally
Quoted 2 lines
it appears that reserved regions will report different information depending on whether a device is attached to a domain.
yes that's correct. Well at some point we decided to expose (some) reserved regions through sysfs. Only printing a reduced set of those also can be misleading, hence my attempt to be more comprehensive.
Quoted 7 lines
Quoted 6 lines
This is a change to the ABI as this reserved region was not yet exposed in sysfs /sys/kernel/iommu_groups/<n>/reserved_regions or through the VFIO ioctl. At VFIO level we increment the version of the
VFIO_IOMMU_TYPE1_INFO_CAP_IOVA_RANGEcapability to advertise that change.
Quoted 5 lines
Is this really an ABI change? The original entry for reserved regions includes:
Not necessarily all reserved regions are listed. This is typically used to output direct-mapped, MSI, non mappable regions.
I agree. That's not really a change in the ABI but I wanted to make things clear about the induced changes for the end-user. On the other end there will be a change in the number of reported resv regions.
Quoted 10 lines
I imagine the intention here was non-mappable relative to the IOMMU, but non-mappable to the device is essentially what we're including here.
I'm also concerned about bumping the vfio interface version for the IOVA range. We're not changing the interface, we're modifying the result, and even then only for a fraction of users. How many users are potentially broken by that change? Are we going to bump the version for everyone any time the result changes on any platform? Thanks,
The userspace needs to know if the GET_INFO is reliable to compute the min/max IOVAs. If we do not change the version, it cannot know and must assume it is not. Wasn't the version field meant for that somehow?
Thanks
Eric
Quoted 3 lines
Alex
Quoted 28 lines · On Tue, 29 Sep 2020 09:18:22 +0200 Auger Eric <[email protected]> wrote:
Hi all,
[also correcting some outdated email addresses + adding Lorenzo in cc]
Quoted 9 lines · Alex Williamson
Quoted 5 lines · On Mon, 28 Sep 2020 21:50:34 +0200 Eric Auger <[email protected]> wrote:
VFIO currently exposes the usable IOVA regions through the
VFIO_IOMMU_GET_INFOioctl /VFIO_IOMMU_TYPE1_INFO_CAP_IOVA_RANGEcapability. However it fails to take into account thedma_maskof the devices within the container. The top limit currently is defined by the iommu aperture.Quoted 13 lines
I think that
dma_maskis traditionally a DMA API interface for a device driver to indicate to the DMA layer which mappings are accessible to the device. On the other hand, vfio makes use of the IOMMU API where the driver is in userspace. That userspace driver has full control of the IOVA range of the device, thereforedma_maskis mostly irrelevant to vfio. I think the issue you're trying to tackle is that the IORT code is making use of thedma_maskto try to describe a DMA address limitation imposed by the PCI root bus, living between the endpoint device and the IOMMU. Therefore, if the IORT code is exposing a topology or system imposed device limitation, this seems much more akin to something like an MSI reserved range, where it's not necessarily the device or the IOMMU with the limitation, but something that sits between them.
Quoted 5 lines
First I think I failed to explain the context. I worked on NVMe passthrough on ARM. The QEMU NVMe backend uses VFIO to program the physical device. The IOVA allocator there currently uses an IOVA range within [
0x10000, 1ULL << 39]. This IOVA layout rather is arbitrary if I understand correctly.
39 bits is the minimum available on some VT-d systems, so it was probably considered a reasonable minimum address width to consider.
Quoted 9 lines
I noticed we rapidly get some VFIO MAP DMA
failures because the allocated IOVA collide with the ARM MSI reserved IOVA window [0x8000000,0x8100000]. Since9b77e5c79840("vfio/type1:
Check reserved region conflict and update iova list"), such VFIO MAP DMA attempts to map IOVAs belonging to host reserved IOVA windows fail. So, by using theVFIO_IOMMU_GET_INFOioctl /VFIO_IOMMU_TYPE1_INFO_CAP_IOVA_RANGEI can change the IOVA allocator to avoid allocating within this range and others. While working on this, I tried to automatically compute the min/max IOVAs and change the
Quoted 8 lines
arbitrary [
0x10000, 1ULL << 39]. My SMMUv2 supports up to 48b so naturally the max IOVA was computed as 1ULL << 48. The QEMU NVMe backend allocates at the bottom and at the top of the range. I noticed the use case was not working as soon as the top IOVA was more than 1ULL << 42. And then we noticed thedma_maskwas set to 42 by using
cat /sys/bus/pci/devices/0005:01:00.0/dma_mask_bits. So my interpretation is thedma_maskwas somehow containing the info the device couldn't handle IOVAs beyond a certain limit.
I see that there are both OF and ACPI hooks in pci_dma_configure() and both modify dev->dma_mask, which is what pci-sysfs is exposing here, but I'm not convinced this even does what it's intended to do. The driver core calls this via the bus->dma_configure callback before probing a driver, but then what happens when the driver calls pci_set_dma_mask()? This is just a wrapper for dma_set_mask() and I don't see anywhere that would take into account the existing dev->dma_mask. It seems for example that pci_dma_configure() could produce a 42 bit mask as we have here, then the driver could override that with anything that the dma_ops.dma_supported() callback finds acceptable, and I don't see any instances where the current dev->dma_mask is considered. Am I overlooking something?
Quoted 14 lines
In my case the 42b limit is computed in
iort_dma_setup()byacpi_dma_get_range(dev, &dmaaddr, &offset, &size);Referring to the comment, it does "Evaluate DMA regions and return respectively DMA region start, offset and size in
dma_addr, offset and size on parsing success". This parses the ACPI table, looking for ACPI companions with_DMAmethods.But as Alex mentioned, the IORT also allows to define limits on "the number of address bits, starting from the least significant bit that can be generated by a device when it accesses memory". See Named component node.Device Memory Address Size limit or PCI root complex node. Memory address size limit.
Quoted 8 lines
ret = acpi_dma_get_range(dev, &dmaaddr, &offset, &size); if (ret == -ENODEV) ret = dev_is_pci(dev) ? rc_dma_get_range(dev, &size) : nc_dma_get_range(dev, &size);So eventually those info collected from the ACPI tables which do impact the usable IOVA range seem to be stored in the
dma_mask, hence that proposal.
As above, it's not clear to me that anyone other than the driver and the dma_supported() callback on dma_ops have any input on the value of dma_mask, so I'm a little baffled by the current operation.
Quoted 14 lines
Quoted 9 lines
Quoted 6 lines
So, for instance, if the IOMMU supports up to 48bits, it may give the impression the max IOVA is 48b while a device may have a
dma_maskof 42b. So this API cannot really be used to compute the max usable IOVA.This patch removes the IOVA region beyond the dma_mask's.
Rather it adds a reserved region accounting for the range above the device's
dma_mask.Yep. It adds new reserved regions in /sys/kernel/iommu_groups/<n>/reserved_regions and remove those from the usable regions exposed by VFIO
GET_INFO.
Quoted 5 lines
I don't think the IOMMU API should be consuming
Quoted 2 lines
dma_masklike this though. For example, what happens inpci_dma_configure()when there are no OF or ACPI DMA restrictions?My guess was that the
dma_maskwas set to the max range but I did not test it.
Still, we're making use of a driver configured property for the purposes of using the DMA API and consuming it in the IOMMU API, specifically to satisfy a userspace driver where the in-kernel meta- driver can't make any assumptions about the device DMA mask. It's all rather convoluted.
Quoted 10 lines
It
Quoted 8 lines
appears to me that the
dma_maskfrom whatever previous driver had the device carries over to the new driver. That's generally ok for the DMA API because a driver is required to set the device's DMA mask. It doesn't make sense however to blindly consume thatdma_maskand export it via an IOMMU API. For example I would expect to see different results depending on whether a host driver has been bound to a device. It seems the correct IOMMU API approach would be for the IORT code to specifically register reserved ranges for the device.
Quoted 2 lines
Is it only specific to IORT table?
acpi_dma_get_range()indrivers/acpi/scan.cis generic.
Yes, anything trying to implement similar restrictions. It appears to me that platform code is stepping on a driver owned field used by dma_ops already here. Maybe reserved regions should be consumed by dma_ops to understand restrictions between the device and the IOMMU.
Quoted 9 lines
Quoted 9 lines
Quoted 5 lines
As we start to expose this reserved region in the sysfs file /sys/kernel/iommu_groups/<n>/reserved_regions, we also need to handle the IOVA range beyond the IOMMU aperture to handle the case where the
dma_maskwould have a higher number of bits than the iommu max input address.Why? The IOMMU geometry already describes this and vfio combines both the IOMMU geometry and the device reserved regions when generating the IOVA ranges?
Quoted 6 lines
Yes VFIO layer does add the info about the topology but /sys/kernel/iommu_groups/<n>/reserved_regions, generated by the IOMMU code, does not. this latter only exposes reserved regions. Assume the
dma_maskis 48b and the IOMMU aperture is 42b (assuming it is possible), if you only take into account the "dma_mask" limitation, the end-user will interpret this as: I can use up to 48b.
What end user? The DMA API is allocating within the address space of the IOMMU, so it will simply never encounter the issue. Within the IOMMU API we can already query the geometry of the IOMMU to know its width. It seems like reserved regions is trying to take on new responsibilities here.
Quoted 6 lines
Who is going to consume this information? Additionally
Quoted 2 lines
it appears that reserved regions will report different information depending on whether a device is attached to a domain.
yes that's correct. Well at some point we decided to expose (some) reserved regions through sysfs. Only printing a reduced set of those also can be misleading, hence my attempt to be more comprehensive.
Seems it further blurs the lines of what reserved regions is intended to report. Is it trying to replace the IOMMU API geometry interface?
Quoted 26 lines
Quoted 11 lines
Quoted 5 lines
This is a change to the ABI as this reserved region was not yet exposed in sysfs /sys/kernel/iommu_groups/<n>/reserved_regions or through the VFIO ioctl. At VFIO level we increment the version of the
VFIO_IOMMU_TYPE1_INFO_CAP_IOVA_RANGEcapability to advertise that change.Is this really an ABI change? The original entry for reserved regions includes:
Not necessarily all reserved regions are listed. This is typically used to output direct-mapped, MSI, non mappable regions.
I agree. That's not really a change in the ABI but I wanted to make things clear about the induced changes for the end-user. On the other end there will be a change in the number of reported resv regions.
Quoted 10 lines
I imagine the intention here was non-mappable relative to the IOMMU, but non-mappable to the device is essentially what we're including here.
I'm also concerned about bumping the vfio interface version for the IOVA range. We're not changing the interface, we're modifying the result, and even then only for a fraction of users. How many users are potentially broken by that change? Are we going to bump the version for everyone any time the result changes on any platform? Thanks,
Quoted 3 lines
The userspace needs to know if the
GET_INFOis reliable to compute the min/max IOVAs. If we do not change the version, it cannot know and must assume it is not. Wasn't the version field meant for that somehow?
Our IOVA RANGE is correct to the best of our knowledge, but I don't think we can or should rev the version every time we find something incorrect. For instance, we wouldn't do the same for a firmware induced range that was missing or incorrect. I think the version field is for creating a new data structure to replace or expand the existing one, not to simply say the returned data might be different. I understand the userspace dilemma, but this doesn't seem like the right solution. Thanks,
Alex
Hi Alex,
Quoted 31 lines · Alex Williamson
Quoted 28 lines · On Tue, 29 Sep 2020 09:18:22 +0200 Auger Eric <[email protected]> wrote:
Hi all,
[also correcting some outdated email addresses + adding Lorenzo in cc]
Quoted 22 lines · Alex Williamson
Quoted 5 lines · On Mon, 28 Sep 2020 21:50:34 +0200 Eric Auger <[email protected]> wrote:
VFIO currently exposes the usable IOVA regions through the
VFIO_IOMMU_GET_INFOioctl /VFIO_IOMMU_TYPE1_INFO_CAP_IOVA_RANGEcapability. However it fails to take into account thedma_maskof the devices within the container. The top limit currently is defined by the iommu aperture.I think that
dma_maskis traditionally a DMA API interface for a device driver to indicate to the DMA layer which mappings are accessible to the device. On the other hand, vfio makes use of the IOMMU API where the driver is in userspace. That userspace driver has full control of the IOVA range of the device, thereforedma_maskis mostly irrelevant to vfio. I think the issue you're trying to tackle is that the IORT code is making use of thedma_maskto try to describe a DMA address limitation imposed by the PCI root bus, living between the endpoint device and the IOMMU. Therefore, if the IORT code is exposing a topology or system imposed device limitation, this seems much more akin to something like an MSI reserved range, where it's not necessarily the device or the IOMMU with the limitation, but something that sits between them.
Quoted 8 lines
Quoted 5 lines
First I think I failed to explain the context. I worked on NVMe passthrough on ARM. The QEMU NVMe backend uses VFIO to program the physical device. The IOVA allocator there currently uses an IOVA range within [
0x10000, 1ULL << 39]. This IOVA layout rather is arbitrary if I understand correctly.39 bits is the minimum available on some VT-d systems, so it was probably considered a reasonable minimum address width to consider.
OK
Quoted 19 lines
Quoted 10 lines
I noticed we rapidly get some VFIO MAP DMA
failures because the allocated IOVA collide with the ARM MSI reserved IOVA window [0x8000000,0x8100000]. Since9b77e5c79840("vfio/type1:
Check reserved region conflict and update iova list"), such VFIO MAP DMA attempts to map IOVAs belonging to host reserved IOVA windows fail. So, by using theVFIO_IOMMU_GET_INFOioctl /VFIO_IOMMU_TYPE1_INFO_CAP_IOVA_RANGEI can change the IOVA allocator to avoid allocating within this range and others. While working on this, I tried to automatically compute the min/max IOVAs and change theQuoted 8 lines
arbitrary [
0x10000, 1ULL << 39]. My SMMUv2 supports up to 48b so naturally the max IOVA was computed as 1ULL << 48. The QEMU NVMe backend allocates at the bottom and at the top of the range. I noticed the use case was not working as soon as the top IOVA was more than 1ULL << 42. And then we noticed thedma_maskwas set to 42 by using
cat /sys/bus/pci/devices/0005:01:00.0/dma_mask_bits. So my interpretation is thedma_maskwas somehow containing the info the device couldn't handle IOVAs beyond a certain limit.
Quoted 12 lines
I see that there are both OF and ACPI hooks in
pci_dma_configure()and both modifydev->dma_mask, which is what pci-sysfs is exposing here, but I'm not convinced this even does what it's intended to do. The driver core calls this via thebus->dma_configurecallback before probing a driver, but then what happens when the driver callspci_set_dma_mask()? This is just a wrapper fordma_set_mask()and I don't see anywhere that would take into account the existingdev->dma_mask. It seems for example thatpci_dma_configure()could produce a 42 bit mask as we have here, then the driver could override that with anything that thedma_ops.dma_supported()callback finds acceptable, and I don't see any instances where the currentdev->dma_maskis considered. Am I overlooking something?
I don't see it either. So the dma_mask set by the driver would never be checked against the dma_mask limited found when parsing OF/ACPI?
Quoted 20 lines
Quoted 20 lines
In my case the 42b limit is computed in
iort_dma_setup()byacpi_dma_get_range(dev, &dmaaddr, &offset, &size);Referring to the comment, it does "Evaluate DMA regions and return respectively DMA region start, offset and size in
dma_addr, offset and size on parsing success". This parses the ACPI table, looking for ACPI companions with_DMAmethods.But as Alex mentioned, the IORT also allows to define limits on "the number of address bits, starting from the least significant bit that can be generated by a device when it accesses memory". See Named component node.Device Memory Address Size limit or PCI root complex node. Memory address size limit.
ret = acpi_dma_get_range(dev, &dmaaddr, &offset, &size); if (ret == -ENODEV) ret = dev_is_pci(dev) ? rc_dma_get_range(dev, &size) : nc_dma_get_range(dev, &size);
Quoted 7 lines
Quoted 3 lines
So eventually those info collected from the ACPI tables which do impact the usable IOVA range seem to be stored in the
dma_mask, hence that proposal.As above, it's not clear to me that anyone other than the driver and the
dma_supported()callback ondma_opshave any input on the value ofdma_mask, so I'm a little baffled by the current operation.
me too
Quoted 18 lines
Quoted 18 lines
Quoted 10 lines
Quoted 7 lines
So, for instance, if the IOMMU supports up to 48bits, it may give the impression the max IOVA is 48b while a device may have a
dma_maskof 42b. So this API cannot really be used to compute the max usable IOVA.This patch removes the IOVA region beyond the dma_mask's.
Rather it adds a reserved region accounting for the range above the device's
dma_mask.Yep. It adds new reserved regions in /sys/kernel/iommu_groups/<n>/reserved_regions and remove those from the usable regions exposed by VFIO
GET_INFO.I don't think the IOMMU API should be consuming
Quoted 2 lines
dma_masklike this though. For example, what happens inpci_dma_configure()when there are no OF or ACPI DMA restrictions?
Quoted 8 lines
Quoted 2 lines
My guess was that the
dma_maskwas set to the max range but I did not test it.Still, we're making use of a driver configured property for the purposes of using the DMA API and consuming it in the IOMMU API, specifically to satisfy a userspace driver where the in-kernel meta- driver can't make any assumptions about the device DMA mask. It's all rather convoluted.
I can't object here. Still we are missing missing consolidated info about max IOVA range. I interpreted the dma_mask as the info missing to get it consolidated but now we are not even sure it does the job and even I would hijack its original goal.
Quoted 27 lines
Quoted 10 lines
It
Quoted 8 lines
appears to me that the
dma_maskfrom whatever previous driver had the device carries over to the new driver. That's generally ok for the DMA API because a driver is required to set the device's DMA mask. It doesn't make sense however to blindly consume thatdma_maskand export it via an IOMMU API. For example I would expect to see different results depending on whether a host driver has been bound to a device. It seems the correct IOMMU API approach would be for the IORT code to specifically register reserved ranges for the device.Quoted 2 lines
Is it only specific to IORT table?
acpi_dma_get_range()indrivers/acpi/scan.cis generic.Yes, anything trying to implement similar restrictions. It appears to me that platform code is stepping on a driver owned field used by
dma_opsalready here. Maybe reserved regions should be consumed bydma_opsto understand restrictions between the device and the IOMMU.Quoted 9 lines
Quoted 9 lines
Quoted 5 lines
As we start to expose this reserved region in the sysfs file /sys/kernel/iommu_groups/<n>/reserved_regions, we also need to handle the IOVA range beyond the IOMMU aperture to handle the case where the
dma_maskwould have a higher number of bits than the iommu max input address.Why? The IOMMU geometry already describes this and vfio combines both the IOMMU geometry and the device reserved regions when generating the IOVA ranges?
Quoted 8 lines
Quoted 6 lines
Yes VFIO layer does add the info about the topology but /sys/kernel/iommu_groups/<n>/reserved_regions, generated by the IOMMU code, does not. this latter only exposes reserved regions. Assume the
dma_maskis 48b and the IOMMU aperture is 42b (assuming it is possible), if you only take into account the "dma_mask" limitation, the end-user will interpret this as: I can use up to 48b.What end user?
Anyone who reads /sys/kernel/iommu_groups/<n>/reserved_regions. A user application could use that info. Anyway I think I indeed wants this file to do more than it was originally meant for, enumerating *some* of the reserved regions. That's a bit frustrating though (same for the VFIO ioctl).
The DMA API is allocating within the address space of
Quoted 45 lines
the IOMMU, so it will simply never encounter the issue. Within the IOMMU API we can already query the geometry of the IOMMU to know its width. It seems like reserved regions is trying to take on new responsibilities here.
Quoted 6 lines
Who is going to consume this information? Additionally
Quoted 2 lines
it appears that reserved regions will report different information depending on whether a device is attached to a domain.
yes that's correct. Well at some point we decided to expose (some) reserved regions through sysfs. Only printing a reduced set of those also can be misleading, hence my attempt to be more comprehensive.
Seems it further blurs the lines of what reserved regions is intended to report. Is it trying to replace the IOMMU API geometry interface?
Quoted 26 lines
Quoted 11 lines
Quoted 5 lines
This is a change to the ABI as this reserved region was not yet exposed in sysfs /sys/kernel/iommu_groups/<n>/reserved_regions or through the VFIO ioctl. At VFIO level we increment the version of the
VFIO_IOMMU_TYPE1_INFO_CAP_IOVA_RANGEcapability to advertise that change.Is this really an ABI change? The original entry for reserved regions includes:
Not necessarily all reserved regions are listed. This is typically used to output direct-mapped, MSI, non mappable regions.
I agree. That's not really a change in the ABI but I wanted to make things clear about the induced changes for the end-user. On the other end there will be a change in the number of reported resv regions.
Quoted 10 lines
I imagine the intention here was non-mappable relative to the IOMMU, but non-mappable to the device is essentially what we're including here.
I'm also concerned about bumping the vfio interface version for the IOVA range. We're not changing the interface, we're modifying the result, and even then only for a fraction of users. How many users are potentially broken by that change? Are we going to bump the version for everyone any time the result changes on any platform? Thanks,
Quoted 3 lines
The userspace needs to know if the
GET_INFOis reliable to compute the min/max IOVAs. If we do not change the version, it cannot know and must assume it is not. Wasn't the version field meant for that somehow?
Quoted 8 lines
Our IOVA RANGE is correct to the best of our knowledge, but I don't think we can or should rev the version every time we find something incorrect. For instance, we wouldn't do the same for a firmware induced range that was missing or incorrect. I think the version field is for creating a new data structure to replace or expand the existing one, not to simply say the returned data might be different. I understand the userspace dilemma, but this doesn't seem like the right solution. Thanks,
understood
Thanks
Eric
Quoted 3 lines
Alex
[+Christoph]
Quoted 58 lines · Alex Williamson
Quoted 28 lines · On Tue, 29 Sep 2020 09:18:22 +0200 Auger Eric <[email protected]> wrote:
Hi all,
[also correcting some outdated email addresses + adding Lorenzo in cc]
Quoted 9 lines · Alex Williamson
Quoted 5 lines · On Mon, 28 Sep 2020 21:50:34 +0200 Eric Auger <[email protected]> wrote:
VFIO currently exposes the usable IOVA regions through the
VFIO_IOMMU_GET_INFOioctl /VFIO_IOMMU_TYPE1_INFO_CAP_IOVA_RANGEcapability. However it fails to take into account thedma_maskof the devices within the container. The top limit currently is defined by the iommu aperture.Quoted 13 lines
I think that
dma_maskis traditionally a DMA API interface for a device driver to indicate to the DMA layer which mappings are accessible to the device. On the other hand, vfio makes use of the IOMMU API where the driver is in userspace. That userspace driver has full control of the IOVA range of the device, thereforedma_maskis mostly irrelevant to vfio. I think the issue you're trying to tackle is that the IORT code is making use of thedma_maskto try to describe a DMA address limitation imposed by the PCI root bus, living between the endpoint device and the IOMMU. Therefore, if the IORT code is exposing a topology or system imposed device limitation, this seems much more akin to something like an MSI reserved range, where it's not necessarily the device or the IOMMU with the limitation, but something that sits between them.Quoted 5 lines
First I think I failed to explain the context. I worked on NVMe passthrough on ARM. The QEMU NVMe backend uses VFIO to program the physical device. The IOVA allocator there currently uses an IOVA range within [
0x10000, 1ULL << 39]. This IOVA layout rather is arbitrary if I understand correctly.39 bits is the minimum available on some VT-d systems, so it was probably considered a reasonable minimum address width to consider.
Quoted 9 lines
I noticed we rapidly get some VFIO MAP DMA
failures because the allocated IOVA collide with the ARM MSI reserved IOVA window [0x8000000,0x8100000]. Since9b77e5c79840("vfio/type1:
Check reserved region conflict and update iova list"), such VFIO MAP DMA attempts to map IOVAs belonging to host reserved IOVA windows fail. So, by using theVFIO_IOMMU_GET_INFOioctl /VFIO_IOMMU_TYPE1_INFO_CAP_IOVA_RANGEI can change the IOVA allocator to avoid allocating within this range and others. While working on this, I tried to automatically compute the min/max IOVAs and change theQuoted 8 lines
arbitrary [
0x10000, 1ULL << 39]. My SMMUv2 supports up to 48b so naturally the max IOVA was computed as 1ULL << 48. The QEMU NVMe backend allocates at the bottom and at the top of the range. I noticed the use case was not working as soon as the top IOVA was more than 1ULL << 42. And then we noticed thedma_maskwas set to 42 by using
cat /sys/bus/pci/devices/0005:01:00.0/dma_mask_bits. So my interpretation is thedma_maskwas somehow containing the info the device couldn't handle IOVAs beyond a certain limit.
Quoted 12 lines
I see that there are both OF and ACPI hooks in
pci_dma_configure()and both modifydev->dma_mask, which is what pci-sysfs is exposing here, but I'm not convinced this even does what it's intended to do. The driver core calls this via thebus->dma_configurecallback before probing a driver, but then what happens when the driver callspci_set_dma_mask()? This is just a wrapper fordma_set_mask()and I don't see anywhere that would take into account the existingdev->dma_mask. It seems for example thatpci_dma_configure()could produce a 42 bit mask as we have here, then the driver could override that with anything that thedma_ops.dma_supported()callback finds acceptable, and I don't see any instances where the currentdev->dma_maskis considered. Am I overlooking something?
I don't think so but Christoph and Robin can provide more input on this - it is a long story.
ACPI and OF bindings set a default dma_mask (and dev->bus_dma_limit), this does not prevent a driver from overriding the dev->dma_mask but DMA mapping code still takes into account the dev->bus_dma_limit.
This may help:
git log -p 03bfdc31176c
Thanks,
Lorenzo
Quoted 16 lines · Lorenzo Pieralisi
Quoted 4 lines
I see that there are both OF and ACPI hooks in
pci_dma_configure()and both modifydev->dma_mask, which is what pci-sysfs is exposing here, but I'm not convinced this even does what it's intended to do. The driver core calls this via thebus->dma_configurecallback beforeQuoted 8 lines
probing a driver, but then what happens when the driver calls
pci_set_dma_mask()? This is just a wrapper fordma_set_mask()and I don't see anywhere that would take into account the existingdev->dma_mask. It seems for example thatpci_dma_configure()could produce a 42 bit mask as we have here, then the driver could override that with anything that thedma_ops.dma_supported()callback finds acceptable, and I don't see any instances where the currentdev->dma_maskis considered. Am I overlooking something?I don't think so but Christoph and Robin can provide more input on this - it is a long story.
Quoted 7 lines
ACPI and OF bindings set a default
dma_mask(anddev->bus_dma_limit), this does not prevent a driver from overriding thedev->dma_maskbut DMA mapping code still takes into account thedev->bus_dma_limit.This may help:
git log -p
03bfdc31176c
This is at best a historic artefact. Bus drivers have no business messing with the DMA mask, dev->bus_dma_limit is the way to communicate addressing limits on the bus (or another interconnect closer to the CPU).
Hi all,
Quoted 17 lines · Christoph Hellwig
Quoted 16 lines · Lorenzo Pieralisi
Quoted 4 lines
I see that there are both OF and ACPI hooks in
pci_dma_configure()and both modifydev->dma_mask, which is what pci-sysfs is exposing here, but I'm not convinced this even does what it's intended to do. The driver core calls this via thebus->dma_configurecallback beforeQuoted 8 lines
probing a driver, but then what happens when the driver calls
pci_set_dma_mask()? This is just a wrapper fordma_set_mask()and I don't see anywhere that would take into account the existingdev->dma_mask. It seems for example thatpci_dma_configure()could produce a 42 bit mask as we have here, then the driver could override that with anything that thedma_ops.dma_supported()callback finds acceptable, and I don't see any instances where the currentdev->dma_maskis considered. Am I overlooking something?I don't think so but Christoph and Robin can provide more input on this - it is a long story.
Quoted 7 lines
Quoted 7 lines
ACPI and OF bindings set a default
dma_mask(anddev->bus_dma_limit), this does not prevent a driver from overriding thedev->dma_maskbut DMA mapping code still takes into account thedev->bus_dma_limit.This may help:
git log -p
03bfdc31176c
Thank you Lorenzo for the pointer.
Quoted 5 lines
This is at best a historic artefact. Bus drivers have no business messing with the DMA mask,
dev->bus_dma_limitis the way to communicate addressing limits on the bus (or another interconnect closer to the CPU).
Then could I envision to use the dev->bus_dma_limit instead of the dev->dma_mask?
Nevertheless, I would need a way to let the userspace know that the usable IOVA ranges reported by VFIO_IOMMU_TYPE1_INFO_CAP_IOVA_RANGE takes into account the addressing limits of the bus.
Thanks
Eric