Skip to contentSkip to conversation

intel-iommu: Fix domain_ids exhaustion

PATCH Superseded Alex Williamson 3 participants 6 messages Feb 16, 2011 – Feb 23, 2011 PCI

lore
Share & tools
lore
#0 lore raw

v2 cover intel-iommu: Fix domain_ids exhaustion

When we unbind a device from a driver, we don't properly unlink the domain from the iommu, so we never free the domain id it was using. We're typically limited to something like 256 domain ids, so a loop of unbinding and rebinding a device can exhaust this pretty quickly. If we're assigning the device to a KVM guest, libvirt does exactly this each time the device is removed from the host driver or added back. When we do run out, we oops the kernel. Fix these.

v2:

We only want to call domain_exit() for domains automatically created via the dma ops path. VM and SI domains have their own life cycle and should not be destroyed here. With v1, if a device was unbound from pci-stub while assigned to a VM, the kernel would oops on the next call into iommu ops.

BTW, should we even be removing the device from the domain in the VM domain case? Drivers and VM domains are (unfortunately) orthogonal concepts here with the way KVM is currently wired. Thanks,

Alex

---

Alex Williamson (2):
intel-iommu: Fix get_domain_for_dev() error path
intel-iommu: Unlink domain from iommu

 drivers/pci/intel-iommu.c |   15 +++++++++++++--
 1 files changed, 13 insertions(+), 2 deletions(-)
#1 lore raw

PATCH v2 1/2 intel-iommu: Unlink domain from iommu

When we remove a device, we unlink the iommu from the domain, but we never do the reverse unlinking of the domain from the iommu. This means that we never clear iommu->domain_ids, eventually leading to resource exhaustion if we repeatedly bind and unbind a device to a driver. Also free empty domains to avoid a resource leak.

---

 drivers/pci/intel-iommu.c |   13 ++++++++++++-
 1 files changed, 12 insertions(+), 1 deletions(-)
drivers/pci/intel-iommu.c +12 -1
index 4789f8e..b670b06 100644
@@ -3260,9 +3260,15 @@ static int device_notifier(struct notifier_block *nb,
 	if (!domain)
 		return 0;
 
-	if (action == BUS_NOTIFY_UNBOUND_DRIVER && !iommu_pass_through)
+	if (action == BUS_NOTIFY_UNBOUND_DRIVER && !iommu_pass_through) {
 		domain_remove_one_dev_info(domain, pdev);
 
+		if (!(domain->flags & DOMAIN_FLAG_VIRTUAL_MACHINE) &&
+	    	    !(domain->flags & DOMAIN_FLAG_STATIC_IDENTITY) &&
+		    list_empty(&domain->devices))
+			domain_exit(domain);
+	}
+
 	return 0;
 }
 
@@ -3411,6 +3417,11 @@ static void domain_remove_one_dev_info(struct dmar_domain *domain,
 		domain->iommu_count--;
 		domain_update_iommu_cap(domain);
 		spin_unlock_irqrestore(&domain->iommu_lock, tmp_flags);
+
+		spin_lock_irqsave(&iommu->lock, tmp_flags);
+		clear_bit(domain->id, iommu->domain_ids);
+		iommu->domains[domain->id] = NULL;
+		spin_unlock_irqrestore(&iommu->lock, tmp_flags);
 	}
 
 	spin_unlock_irqrestore(&device_domain_lock, flags);
#2 lore raw

PATCH v2 2/2 intel-iommu: Fix get_domain_for_dev() error path

In reply to Alex Williamson

If we run out of domain_ids and fail iommu_attach_domain(), we fall into domain_exit() without having setup enough of the domain structure for this to do anything useful. In fact, it typically runs off into the weeds walking the bogus domain->devices list. Just free the domain.

---

 drivers/pci/intel-iommu.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
drivers/pci/intel-iommu.c +1 -1
index b670b06..b0343d1 100644
@@ -1835,7 +1835,7 @@ static struct dmar_domain *get_domain_for_dev(struct pci_dev *pdev, int gaw)
 
 	ret = iommu_attach_domain(domain, iommu);
 	if (ret) {
-		domain_exit(domain);
+		free_domain_mem(domain);
 		goto error;
 	}
 
Acked-by on the v2 series
#3 lore raw
In reply to Alex Williamson
Quoted 30 lines · Alex Williamson

When we unbind a device from a driver, we don't properly unlink the domain from the iommu, so we never free the domain id it was using. We're typically limited to something like 256 domain ids, so a loop of unbinding and rebinding a device can exhaust this pretty quickly. If we're assigning the device to a KVM guest, libvirt does exactly this each time the device is removed from the host driver or added back. When we do run out, we oops the kernel. Fix these.

v2:

We only want to call domain_exit() for domains automatically created via the dma ops path. VM and SI domains have their own life cycle and should not be destroyed here. With v1, if a device was unbound from pci-stub while assigned to a VM, the kernel would oops on the next call into iommu ops.

BTW, should we even be removing the device from the domain in the VM domain case? Drivers and VM domains are (unfortunately) orthogonal concepts here with the way KVM is currently wired. Thanks,

Alex

---

Alex Williamson (2):
intel-iommu: Fix get_domain_for_dev() error path
intel-iommu: Unlink domain from iommu

Quoted 6 lines
 drivers/pci/intel-iommu.c |   15 +++++++++++++--
 1 files changed, 13 insertions(+), 2 deletions(-)

--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#4 lore raw
In reply to Alex Williamson

Once merged, should it go to -stable? If so, maybe it would be a good idea to add that while collecting the Acked-By's...

Signature · 4 lines
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh
#5 lore raw
Quoted 2 lines · Henrique de Moraes Holschuh

Once merged, should it go to -stable? If so, maybe it would be a good idea to add that while collecting the Acked-By's...

Yeah, I think this would be a valid -stable candidate. I'm hoping we'll get an Ack from David here soon...

Alex