Skip to contentSkip to conversation

iommu/intel: Exclude devices using RMRRs from IOMMU API domains

PATCH Superseded Alex Williamson 3 participants 5 messages May 14, 2014 – May 21, 2014 IOMMU

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 iommu/intel: Exclude devices using RMRRs from IOMMU API domains

The user of the IOMMU API domain expects to have full control of the IOVA space for the domain. RMRRs are fundamentally incompatible with that idea. We can neither map the RMRR into the IOMMU API domain, nor can we guarantee that the device won't continue DMA with the area described by the RMRR as part of the new domain. Therefore we must prevent such devices from being used by the IOMMU API.

---
 drivers/iommu/intel-iommu.c |   15 +++++++++++++++
 1 file changed, 15 insertions(+)
drivers/iommu/intel-iommu.c +15 -0
index f256ffc..eb77149 100644
@@ -4188,6 +4188,21 @@ static int intel_iommu_attach_device(struct iommu_domain *domain,
 	int addr_width;
 	u8 bus, devfn;
 
+	/*
+	 * With IOMMU API domains we don't have the freedom to insert RMRR
+	 * entries into the domain mapping, the IOMMU API user expects full
+	 * control of the IOVA space of the device.  We also have no ability
+	 * to shutdown whatever back channel operations occur through the
+	 * RMRR.  Therefore our only option is to prevent devices making use
+	 * of RMRRs from being used by the IOMMU API.  As usual we exempt
+	 * USB devices since their RMRR support is largely historical.
+	 */
+	if (device_has_rmrr(dev) && (!dev_is_pci(dev) ||
+	    (to_pci_dev(dev)->class >> 8) != PCI_CLASS_SERIAL_USB)) {
+		dev_warn(dev, "Device is ineligible for IOMMU domain attach due to platform RMRR requirement.  Contact your platform vendor.\n");
+		return -EPERM;
+	}
+
 	/* normally dev is not mapped */
 	if (unlikely(domain_context_mapped(dev))) {
 		struct dmar_domain *old_domain;
#1 lore raw
Quoted 6 lines · Alex Williamson

The user of the IOMMU API domain expects to have full control of the IOVA space for the domain. RMRRs are fundamentally incompatible with that idea. We can neither map the RMRR into the IOMMU API domain, nor can we guarantee that the device won't continue DMA with the area described by the RMRR as part of the new domain. Therefore we must prevent such devices from being used by the IOMMU API.

Ick, ick, ick. The more the ramifications of RMRRs become apparent, the more I wish we'd just done the Right Thing™ and declared that firmware SHALL NOT leave any device doing (IOMMU-visible) DMA after the OS takes over. That way, if they wanted this kind of abomination then they'd have to come up with a way of doing it differently. Hell, can't you do PCIe transactions which claim to be already translated, and thus just bypass the IOMMU?

OK, rant over...

Why can't we map the RMRR into the IOMMU API domain? If we're setting up a VM guest, that basically means we'd want to poke a hole in its memory map and mark the RMRR-afflicted range as reserved or absent. It's horrible, but *everything* about RMRRs is horrible. It's not impossible, and it would allow us to give these devices away to guests. Don't we sometimes *have* devices that we want to give to guests, that are afflicted with RMRRs?

There are discussions about RMRRs being (ab)used for more than their existing brain-damaged purpose. Where we have a peripheral device that will (mis)interpret certain address ranges as "local" rather than forwarding transactions up towards main memory, we need to ensure that such ranges are never used as virtual addresses. This has largely been an invisible problem until we found a device where the affected range matched the IOVA our DMA API uses by default. Using an RMRR has been proposed as a simple way to achieve that... which means that you end up not being able to assign *those* devices to IOMMU domains either.

I do suspect it's going to lead to complaints... but I'm just not sure I can bring myself to care. Sane designs don't require RMRRs. If someone comes to me and complains that their HP storage controller or whatever can't be assigned to a guest, I'm quite prepared to tell them to replace it with something non-broken. Will you back me up when it happens?

Signature · 1 line
dwmw2
#2 lore raw
Quoted 18 lines · David Woodhouse
Quoted 6 lines · Alex Williamson

The user of the IOMMU API domain expects to have full control of the IOVA space for the domain. RMRRs are fundamentally incompatible with that idea. We can neither map the RMRR into the IOMMU API domain, nor can we guarantee that the device won't continue DMA with the area described by the RMRR as part of the new domain. Therefore we must prevent such devices from being used by the IOMMU API.

Ick, ick, ick. The more the ramifications of RMRRs become apparent, the more I wish we'd just done the Right Thing™ and declared that firmware SHALL NOT leave any device doing (IOMMU-visible) DMA after the OS takes over. That way, if they wanted this kind of abomination then they'd have to come up with a way of doing it differently. Hell, can't you do PCIe transactions which claim to be already translated, and thus just bypass the IOMMU?

OK, rant over...

Quoted 7 lines

Why can't we map the RMRR into the IOMMU API domain? If we're setting up a VM guest, that basically means we'd want to poke a hole in its memory map and mark the RMRR-afflicted range as reserved or absent. It's horrible, but *everything* about RMRRs is horrible. It's not impossible, and it would allow us to give these devices away to guests. Don't we sometimes *have* devices that we want to give to guests, that are afflicted with RMRRs?

You're right, it is possible to assign devices with RMRRs, but in order to do so we'd need to expose the RMRR areas for a device beyond the inner workings of intel-iommu and mark those ranges as reserved in the guest. That alone makes hotplug of such devices into a guest impossible.

Enabling such a use case also potentially provides an untrusted guest with direct access to regions of platform memory that potentially allows for untold platform specific exploits.

We currently have no visibility to RMRRs from the IOMMU API, so we can't even attempt to do the above, nor can we guarantee that we have any ability to make a device discontinue use of an RMRR area when it is assigned to a VM domain. Therefore the only safe thing to do is prevent use of such devices by a VM domain.

Quoted 10 lines

There are discussions about RMRRs being (ab)used for more than their existing brain-damaged purpose. Where we have a peripheral device that will (mis)interpret certain address ranges as "local" rather than forwarding transactions up towards main memory, we need to ensure that such ranges are never used as virtual addresses. This has largely been an invisible problem until we found a device where the affected range matched the IOVA our DMA API uses by default. Using an RMRR has been proposed as a simple way to achieve that... which means that you end up not being able to assign *those* devices to IOMMU domains either.

Quoted 5 lines

I do suspect it's going to lead to complaints... but I'm just not sure I can bring myself to care. Sane designs don't require RMRRs. If someone comes to me and complains that their HP storage controller or whatever can't be assigned to a guest, I'm quite prepared to tell them to replace it with something non-broken. Will you back me up when it happens?

Exactly, I have a hard time bringing myself to care about supporting such devices. Vendors that proliferate RMRR usage need to be aware of the implications of RMRRs for all use cases of a device. First and foremost, we need to lock out devices with RMRRs because we have no ability to either honor or disable RMRRs when used by the IOMMU API. If vendors that rely on RMRRs want to make such devices assignable by providing interfaces to describe and map the area into a VM, or even a mechanism to disable the RMRR, more power to them. The current situation is simply unsafe and needs to be prevented. Thanks,

Alex

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

#3 lore raw
Quoted 59 lines · Alex Williamson
Quoted 18 lines · David Woodhouse
Quoted 6 lines · Alex Williamson

The user of the IOMMU API domain expects to have full control of the IOVA space for the domain. RMRRs are fundamentally incompatible with that idea. We can neither map the RMRR into the IOMMU API domain, nor can we guarantee that the device won't continue DMA with the area described by the RMRR as part of the new domain. Therefore we must prevent such devices from being used by the IOMMU API.

Ick, ick, ick. The more the ramifications of RMRRs become apparent, the more I wish we'd just done the Right Thing™ and declared that firmware SHALL NOT leave any device doing (IOMMU-visible) DMA after the OS takes over. That way, if they wanted this kind of abomination then they'd have to come up with a way of doing it differently. Hell, can't you do PCIe transactions which claim to be already translated, and thus just bypass the IOMMU?

OK, rant over...

Quoted 7 lines

Why can't we map the RMRR into the IOMMU API domain? If we're setting up a VM guest, that basically means we'd want to poke a hole in its memory map and mark the RMRR-afflicted range as reserved or absent. It's horrible, but *everything* about RMRRs is horrible. It's not impossible, and it would allow us to give these devices away to guests. Don't we sometimes *have* devices that we want to give to guests, that are afflicted with RMRRs?

You're right, it is possible to assign devices with RMRRs, but in order to do so we'd need to expose the RMRR areas for a device beyond the inner workings of intel-iommu and mark those ranges as reserved in the guest. That alone makes hotplug of such devices into a guest impossible.

Enabling such a use case also potentially provides an untrusted guest with direct access to regions of platform memory that potentially allows for untold platform specific exploits.

We currently have no visibility to RMRRs from the IOMMU API, so we can't even attempt to do the above, nor can we guarantee that we have any ability to make a device discontinue use of an RMRR area when it is assigned to a VM domain. Therefore the only safe thing to do is prevent use of such devices by a VM domain.

Quoted 10 lines

There are discussions about RMRRs being (ab)used for more than their existing brain-damaged purpose. Where we have a peripheral device that will (mis)interpret certain address ranges as "local" rather than forwarding transactions up towards main memory, we need to ensure that such ranges are never used as virtual addresses. This has largely been an invisible problem until we found a device where the affected range matched the IOVA our DMA API uses by default. Using an RMRR has been proposed as a simple way to achieve that... which means that you end up not being able to assign *those* devices to IOMMU domains either.

Quoted 5 lines

I do suspect it's going to lead to complaints... but I'm just not sure I can bring myself to care. Sane designs don't require RMRRs. If someone comes to me and complains that their HP storage controller or whatever can't be assigned to a guest, I'm quite prepared to tell them to replace it with something non-broken. Will you back me up when it happens?

Quoted 9 lines

Exactly, I have a hard time bringing myself to care about supporting such devices. Vendors that proliferate RMRR usage need to be aware of the implications of RMRRs for all use cases of a device. First and foremost, we need to lock out devices with RMRRs because we have no ability to either honor or disable RMRRs when used by the IOMMU API. If vendors that rely on RMRRs want to make such devices assignable by providing interfaces to describe and map the area into a VM, or even a mechanism to disable the RMRR, more power to them. The current situation is simply unsafe and needs to be prevented.

I care but I think this patch should go in until there is a better solution.

-- ljk

Quoted 9 lines

Thanks,

Alex

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

#4 lore raw
Quoted 11 lines · Linda Knippers
Quoted 10 lines

Exactly, I have a hard time bringing myself to care about supporting such devices. Vendors that proliferate RMRR usage need to be aware of the implications of RMRRs for all use cases of a device. First and foremost, we need to lock out devices with RMRRs because we have no ability to either honor or disable RMRRs when used by the IOMMU API. If vendors that rely on RMRRs want to make such devices assignable by providing interfaces to describe and map the area into a VM, or even a mechanism to disable the RMRR, more power to them. The current situation is simply unsafe and needs to be prevented.

Quoted 2 lines

I care but I think this patch should go in until there is a better solution.

I was quite happy with our earlier "solution" of just assuming that RMRRs were a boot-time thing only. Once a driver takes over the device, or it's given to a guest, we forget the RMRRs entirely. The hardware has no business still doing rogue DMA after we've taken control of it.

Again, it's only those who stupidly designed hardware who suffer. And life was a lot simpler for us.

Signature · 1 line
dwmw2