Intel IOMMU patch to reprocess RMRR info
PATCH v2 1/1 Intel IOMMU patch to reprocess RMRR info
When a 32bit PCI device is removed from the SI Domain, the RMRR information for this device becomes invalid and needs to be reprocessed to avoid DMA Read errors. These errors are evidenced by the Present bit being cleared in the device's context entry. Fixing this problem with an enhancement to process the RMRR info when the device is assigned to another domain. The Bus Master bit is cleared during the move to another domain and during the reprocessing of the RMRR info so no DMA can take place at this time.
----
PATCH v1: https://lkml.org/lkml/2012/6/15/204
drivers/iommu/intel-iommu.c | 47 ++++++++++++++++++++++++++++++++++++++++--
1 files changed, 44 insertions(+), 3 deletions(-)
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +44 -3
@@ -2706,11 +2706,39 @@ static int iommu_dummy(struct pci_dev *preturn pdev->dev.archdata.iommu ==DUMMY_DEVICE_DOMAIN_INFO ; } +static int reprocess_rmrr (struct device *dev) +{ +struct dmar_rmrr_unit *rmrr; +struct pci_dev *pdev; +int i, ret; + + pdev =to_pci_dev (dev); + +for_each_rmrr_units (rmrr) { +for (i =0 ; i < rmrr->devices_cnt; i++) { +/* +* Here we are just concerned with +* finding the one device that was +* removed from the si_domain and +* re-evaluating its RMRR info. +*/ +if (rmrr->devices[i] != pdev) +continue ; +pr_info ("IOMMU: Reprocess RMRR information for device %s.\n" , +pci_name (pdev)); + ret =iommu_prepare_rmrr_dev (rmrr, pdev); +if (ret) +pr_err ("IOMMU: Reprocessing RMRR reserved region for device failed" ); + } + } +return 0 ; +} +/* Check if the pdev needs to go through non-identity map and unmap process.*/ static int iommu_no_mapping (struct device *dev) {struct pci_dev *pdev; -int found; +int found, current_bus_master;if (unlikely (dev->bus != &pci_bus_type))return 1 ;
@@ -2731,9 +2759,22 @@ static int iommu_no_mapping(struct devic* 32 bit DMA is removed from si_domain and fall back * to non-identity mapping. */ -domain_remove_one_dev_info (si_domain, pdev);printk (KERN_INFO "32bit %s uses non-identity mapping\n" , -pci_name (pdev)); +pci_name (pdev)); +/* +* If a device gets this far we need to clear the Bus +* Master bit before we start moving devices from domain +* to domain. We will also reset the Bus Master bit +* after reprocessing the RMRR info. However, we only +* do both the clearing and setting if needed. +*/ + current_bus_master = pdev->is_busmaster; +if (current_bus_master) +pci_clear_master (pdev); +domain_remove_one_dev_info (si_domain, pdev); +reprocess_rmrr (dev); +if (current_bus_master) +pci_set_master (pdev);return 0 ; } }else {
A message that is not in the archive (sent off-list or not yet received): 20120918164955.12296.28799.sendpatchset-jP8EmR9A9vELnkn81s9yt/[email protected]
Quoted 74 lines · Tom Mingarelli
When a 32bit PCI device is removed from the SI Domain, the RMRR information for this device becomes invalid and needs to be reprocessed to avoid DMA Read errors. These errors are evidenced by the Present bit being cleared in the device's context entry. Fixing this problem with an enhancement to process the RMRR info when the device is assigned to another domain. The Bus Master bit is cleared during the move to another domain and during the reprocessing of the RMRR info so no DMA can take place at this time.
----
PATCH v1: https://lkml.org/lkml/2012/6/15/204
drivers/iommu/intel-iommu.c| 47 ++++++++++++++++++++++++++++++++++++++++--1 files changed, 44 insertions(+), 3 deletions(-)Signed-off-by Thomas Mingarelli<[email protected]>diff -up
./drivers/iommu/intel-iommu.c.ORIG./drivers/iommu/intel-iommu.cdrivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +37
-3@@ -2706,11 +2706,39 @@ static int iommu_dummy(struct pci_dev *preturn pdev->dev.archdata.iommu ==DUMMY_DEVICE_DOMAIN_INFO ; } +static int reprocess_rmrr (struct device *dev) +{ +struct dmar_rmrr_unit *rmrr; +struct pci_dev *pdev; +int i, ret; + + pdev =to_pci_dev (dev); + +for_each_rmrr_units (rmrr) { +for (i =0 ; i< rmrr->devices_cnt; i++) { +/* +* Here we are just concerned with +* finding the one device that was +* removed from the si_domain and +* re-evaluating its RMRR info. +*/ +if (rmrr->devices[i] != pdev) +continue ; +pr_info ("IOMMU: Reprocess RMRR information for device %s.\n" , +pci_name (pdev)); + ret =iommu_prepare_rmrr_dev (rmrr, pdev); +if (ret) +pr_err ("IOMMU: Reprocessing RMRR reserved region for device failed" ); + } + } +return 0 ; +} +/* Check if the pdev needs to go through non-identity map and unmap process.*/ static int iommu_no_mapping (struct device *dev) {struct pci_dev *pdev; -int found; +int found, current_bus_master;if (unlikely (dev->bus !=&pci_bus_type))return 1 ;@@ -2731,9 +2759,22 @@ static int iommu_no_mapping(struct devic* 32 bit DMA is removed from si_domain and fall back * to non-identity mapping. */ -domain_remove_one_dev_info (si_domain, pdev);printk (KERN_INFO "32bit %s uses non-identity mapping\n" , -pci_name (pdev)); +pci_name (pdev)); +/* +* If a device gets this far we need to clear the Bus +* Master bit before we start moving devices from domain +* to domain. We will also reset the Bus Master bit +* after reprocessing the RMRR info. However, we only +* do both the clearing and setting if needed. +*/
drivers/iommu/intel-iommu.c static int iommu_no_mapping(struct devic · line 2780
Quoted 10 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +7
-0+ current_bus_master = pdev->is_busmaster; +if (current_bus_master) +pci_clear_master (pdev); +domain_remove_one_dev_info (si_domain, pdev); +reprocess_rmrr (dev); +if (current_bus_master) +pci_set_master (pdev);return 0 ; } }else {
appears to have recommended changes from v1, so looks good wrt handling devices w/rmrr.
[adding David Woodhouse]
Quoted 28 lines · Tom Mingarelli
When a 32bit PCI device is removed from the SI Domain, the RMRR information for this device becomes invalid and needs to be reprocessed to avoid DMA Read errors. These errors are evidenced by the Present bit being cleared in the device's context entry. Fixing this problem with an enhancement to process the RMRR info when the device is assigned to another domain. The Bus Master bit is cleared during the move to another domain and during the reprocessing of the RMRR info so no DMA can take place at this time.
----
PATCH v1: https://lkml.org/lkml/2012/6/15/204
drivers/iommu/intel-iommu.c| 47 ++++++++++++++++++++++++++++++++++++++++--1 files changed, 44 insertions(+), 3 deletions(-)Signed-off-by Thomas Mingarelli <[email protected]>diff -up
./drivers/iommu/intel-iommu.c.ORIG./drivers/iommu/intel-iommu.cdrivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +6
-0@@ -2706,11 +2706,39 @@ static int iommu_dummy(struct pci_dev *preturn pdev->dev.archdata.iommu ==DUMMY_DEVICE_DOMAIN_INFO ; } +static int reprocess_rmrr (struct device *dev) +{ +struct dmar_rmrr_unit *rmrr; +struct pci_dev *pdev; +int i, ret; +
drivers/iommu/intel-iommu.c static int iommu_dummy(struct pci_dev *p · line 2724
Quoted 10 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +10
-0+ pdev =to_pci_dev (dev); + +for_each_rmrr_units (rmrr) { +for (i =0 ; i < rmrr->devices_cnt; i++) { +/* +* Here we are just concerned with +* finding the one device that was +* removed from the si_domain and +* re-evaluating its RMRR info. +*/
I'm still not sure why this comment is wrapped so tightly.
drivers/iommu/intel-iommu.c static int iommu_dummy(struct pci_dev *p · line 2731
Quoted 7 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +7
-0+if (rmrr->devices[i] != pdev) +continue ; +pr_info ("IOMMU: Reprocess RMRR information for device %s.\n" , +pci_name (pdev)); + ret =iommu_prepare_rmrr_dev (rmrr, pdev); +if (ret) +pr_err ("IOMMU: Reprocessing RMRR reserved region for device failed" );
This could be "if (iommu_prepare_rmrr...)" because...
drivers/iommu/intel-iommu.c static int iommu_dummy(struct pci_dev *p · line 2734
Quoted 3 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +3
-0+ } + } +return 0 ;
Why does return anything? Looks like it could be a void function since you're not returning the only possible error case above and not checking the return value below. You're missing an indent here anyway.
Quoted 12 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +3
-1+} +/* Check if the pdev needs to go through non-identity map and unmap process.*/ static int iommu_no_mapping (struct device *dev) {struct pci_dev *pdev; -int found; +int found, current_bus_master;if (unlikely (dev->bus != &pci_bus_type))return 1 ;
drivers/iommu/intel-iommu.c static int iommu_no_mapping(struct devic · line 2763
Quoted 7 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +1
-2@@ -2731,9 +2759,22 @@ static int iommu_no_mapping(struct devic* 32 bit DMA is removed from si_domain and fall back * to non-identity mapping. */ -domain_remove_one_dev_info (si_domain, pdev);printk (KERN_INFO "32bit %s uses non-identity mapping\n" , -pci_name (pdev)); +pci_name (pdev));
White space damage? Change this to a pr_info if you really want to touch it.
Quoted 4 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +4
-0+/* +* If a device gets this far we need to clear the Bus +* Master bit before we start moving devices from domain +* to domain. We will also reset the Bus Master bit
drivers/iommu/intel-iommu.c static int iommu_no_mapping(struct devic · line 2777
Quoted 10 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +10
-0+* after reprocessing the RMRR info. However, we only +* do both the clearing and setting if needed. +*/ + current_bus_master = pdev->is_busmaster; +if (current_bus_master) +pci_clear_master (pdev); +domain_remove_one_dev_info (si_domain, pdev); +reprocess_rmrr (dev); +if (current_bus_master) +pci_set_master (pdev);
I don't know any better way to halt DMA since we can't move the device to a new domain atomically, but what about the other cases where we switch domains? For instance, what if some unsuspecting user tries to assign the device to a guest? I think it's generally inadvisable to assign a devices with RMRRs to a guest, but if they do, they're going to run into the same problem. The RMRRs aren't reprocessed for the VM domain and again aren't reprocessed when returned to a standard device domain. Even more fun, if assigned to a VM domain, RMRRs should be added with the device and removed if the device is later detached from the domain.
The lazy solution might be to disallow devices with RMRRs from being attached via the IOMMU API interfaces (do we need more reasons not to use RMRRs?). Otherwise we need to be proactive about setting up a new domain with RMRR entries for every case and correctly tracking RMRRs for VM domains. Thanks,
Alex
Quoted 3 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +0
-0return 0 ; } }else {
A message that is not in the archive (sent off-list or not yet received): 1348778200.2320.241.camel-85EaTFmN5p//[email protected]
Alex,
Are you suggesting that a solution is to prevent devices with RMRRs from being placed in the SI Domain in the first place (when pt mode is used)?
Tom
-----Original Message-----
From: Alex Williamson [mailto:[email protected]]
Sent: Thursday, September 27, 2012 3:37 PM
To: Mingarelli, Thomas
Cc: iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org; Knippers, Linda; Khan, Shuah; Don Dutile; David Woodhouse
Subject: Re: [PATCH v2] Intel IOMMU patch to reprocess RMRR info
[adding David Woodhouse]
Quoted 28 lines · Tom Mingarelli
When a 32bit PCI device is removed from the SI Domain, the RMRR information for this device becomes invalid and needs to be reprocessed to avoid DMA Read errors. These errors are evidenced by the Present bit being cleared in the device's context entry. Fixing this problem with an enhancement to process the RMRR info when the device is assigned to another domain. The Bus Master bit is cleared during the move to another domain and during the reprocessing of the RMRR info so no DMA can take place at this time.
----
PATCH v1: https://lkml.org/lkml/2012/6/15/204
drivers/iommu/intel-iommu.c| 47 ++++++++++++++++++++++++++++++++++++++++--1 files changed, 44 insertions(+), 3 deletions(-)Signed-off-by Thomas Mingarelli <[email protected]>diff -up
./drivers/iommu/intel-iommu.c.ORIG./drivers/iommu/intel-iommu.cdrivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +6
-0@@ -2706,11 +2706,39 @@ static int iommu_dummy(struct pci_dev *preturn pdev->dev.archdata.iommu ==DUMMY_DEVICE_DOMAIN_INFO ; } +static int reprocess_rmrr (struct device *dev) +{ +struct dmar_rmrr_unit *rmrr; +struct pci_dev *pdev; +int i, ret; +
drivers/iommu/intel-iommu.c static int iommu_dummy(struct pci_dev *p · line 2724
Quoted 10 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +10
-0+ pdev =to_pci_dev (dev); + +for_each_rmrr_units (rmrr) { +for (i =0 ; i < rmrr->devices_cnt; i++) { +/* +* Here we are just concerned with +* finding the one device that was +* removed from the si_domain and +* re-evaluating its RMRR info. +*/
I'm still not sure why this comment is wrapped so tightly.
drivers/iommu/intel-iommu.c static int iommu_dummy(struct pci_dev *p · line 2731
Quoted 7 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +7
-0+if (rmrr->devices[i] != pdev) +continue ; +pr_info ("IOMMU: Reprocess RMRR information for device %s.\n" , +pci_name (pdev)); + ret =iommu_prepare_rmrr_dev (rmrr, pdev); +if (ret) +pr_err ("IOMMU: Reprocessing RMRR reserved region for device failed" );
This could be "if (iommu_prepare_rmrr...)" because...
drivers/iommu/intel-iommu.c static int iommu_dummy(struct pci_dev *p · line 2734
Quoted 3 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +3
-0+ } + } +return 0 ;
Why does return anything? Looks like it could be a void function since you're not returning the only possible error case above and not checking the return value below. You're missing an indent here anyway.
Quoted 12 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +3
-1+} +/* Check if the pdev needs to go through non-identity map and unmap process.*/ static int iommu_no_mapping (struct device *dev) {struct pci_dev *pdev; -int found; +int found, current_bus_master;if (unlikely (dev->bus != &pci_bus_type))return 1 ;
drivers/iommu/intel-iommu.c static int iommu_no_mapping(struct devic · line 2763
Quoted 7 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +1
-2@@ -2731,9 +2759,22 @@ static int iommu_no_mapping(struct devic* 32 bit DMA is removed from si_domain and fall back * to non-identity mapping. */ -domain_remove_one_dev_info (si_domain, pdev);printk (KERN_INFO "32bit %s uses non-identity mapping\n" , -pci_name (pdev)); +pci_name (pdev));
White space damage? Change this to a pr_info if you really want to touch it.
Quoted 4 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +4
-0+/* +* If a device gets this far we need to clear the Bus +* Master bit before we start moving devices from domain +* to domain. We will also reset the Bus Master bit
drivers/iommu/intel-iommu.c static int iommu_no_mapping(struct devic · line 2777
Quoted 10 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +10
-0+* after reprocessing the RMRR info. However, we only +* do both the clearing and setting if needed. +*/ + current_bus_master = pdev->is_busmaster; +if (current_bus_master) +pci_clear_master (pdev); +domain_remove_one_dev_info (si_domain, pdev); +reprocess_rmrr (dev); +if (current_bus_master) +pci_set_master (pdev);
I don't know any better way to halt DMA since we can't move the device to a new domain atomically, but what about the other cases where we switch domains? For instance, what if some unsuspecting user tries to assign the device to a guest? I think it's generally inadvisable to assign a devices with RMRRs to a guest, but if they do, they're going to run into the same problem. The RMRRs aren't reprocessed for the VM domain and again aren't reprocessed when returned to a standard device domain. Even more fun, if assigned to a VM domain, RMRRs should be added with the device and removed if the device is later detached from the domain.
The lazy solution might be to disallow devices with RMRRs from being attached via the IOMMU API interfaces (do we need more reasons not to use RMRRs?). Otherwise we need to be proactive about setting up a new domain with RMRR entries for every case and correctly tracking RMRRs for VM domains. Thanks,
Alex
Quoted 3 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +0
-0return 0 ; } }else {
A message that is not in the archive (sent off-list or not yet received): 9774516974AF5F4C8A2C3C69CD3412332338F452-KNyhpuZufFMSZAcGdq5asR6epYMZPwEe5NbjCUgZEJk@public.gmane.org
Quoted 5 lines · Thomas
Alex,
Are you suggesting that a solution is to prevent devices with RMRRs from being placed in the SI Domain in the first place (when pt mode is used)?
No, it seems like it's preferable that devices with RMRRs stay in the si domain if the device supports 64bit DMA. They're likely to cause less problems there and we can ignore the RMRRs in the si domain. The problem I'm trying to address is that the domain you're setting up with RMRRs is only used for dma_ops (ie. host driver use). If the device is attached to a guest, that domain gets discarded and the device is added to yet another domain, potentially with other devices. That domain is missing RMRRs, so now your device is going to generate VT-d faults. The device can then get removed from that domain, in which case the RMRRs should be removed, and again a new dma_ops domain needs to get created with RMRRs.
It really seems like RMRRs are incompatible with IOMMU API use though. If an RMRR is setup for a VM domain, that's bad because a) it gives the VM direct access to that range of host memory, and b) it interferes with the guest use of the address space. a) is also bad for isolating devices on the host, but the spec makes it available for abuse. For b), it's not hard to imagine an RMRR range on the host that overlaps with DMA'able space on the guest. Data is read or written to the host memory instead of the guest memory. So maybe the right answer is to make intel_iommu_attach_device return error if requested to act on a device with RMRR ranges. Thanks,
Alex
Quoted 132 lines
-----Original Message-----
From: Alex Williamson [mailto:[email protected]]
Sent: Thursday, September 27, 2012 3:37 PM
To: Mingarelli, Thomas
Cc: iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org; Knippers, Linda; Khan, Shuah; Don Dutile; David Woodhouse
Subject: Re: [PATCH v2] Intel IOMMU patch to reprocess RMRR info[adding David Woodhouse]
Quoted 28 lines · Tom Mingarelli
When a 32bit PCI device is removed from the SI Domain, the RMRR information for this device becomes invalid and needs to be reprocessed to avoid DMA Read errors. These errors are evidenced by the Present bit being cleared in the device's context entry. Fixing this problem with an enhancement to process the RMRR info when the device is assigned to another domain. The Bus Master bit is cleared during the move to another domain and during the reprocessing of the RMRR info so no DMA can take place at this time.
----
PATCH v1: https://lkml.org/lkml/2012/6/15/204
drivers/iommu/intel-iommu.c| 47 ++++++++++++++++++++++++++++++++++++++++--1 files changed, 44 insertions(+), 3 deletions(-)Signed-off-by Thomas Mingarelli <[email protected]>diff -up
./drivers/iommu/intel-iommu.c.ORIG./drivers/iommu/intel-iommu.cdrivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +6
-0@@ -2706,11 +2706,39 @@ static int iommu_dummy(struct pci_dev *preturn pdev->dev.archdata.iommu ==DUMMY_DEVICE_DOMAIN_INFO ; } +static int reprocess_rmrr (struct device *dev) +{ +struct dmar_rmrr_unit *rmrr; +struct pci_dev *pdev; +int i, ret; +drivers/iommu/intel-iommu.c static int iommu_dummy(struct pci_dev *p · line 2724
Quoted 10 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +10
-0+ pdev =to_pci_dev (dev); + +for_each_rmrr_units (rmrr) { +for (i =0 ; i < rmrr->devices_cnt; i++) { +/* +* Here we are just concerned with +* finding the one device that was +* removed from the si_domain and +* re-evaluating its RMRR info. +*/ I'm still not sure why this comment is wrapped so tightly.
drivers/iommu/intel-iommu.c static int iommu_dummy(struct pci_dev *p · line 2731
Quoted 7 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +7
-0+if (rmrr->devices[i] != pdev) +continue ; +pr_info ("IOMMU: Reprocess RMRR information for device %s.\n" , +pci_name (pdev)); + ret =iommu_prepare_rmrr_dev (rmrr, pdev); +if (ret) +pr_err ("IOMMU: Reprocessing RMRR reserved region for device failed" );This could be "if (
iommu_prepare_rmrr...)" because...drivers/iommu/intel-iommu.c static int iommu_dummy(struct pci_dev *p · line 2734
Quoted 3 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +3
-0+ } + } +return 0 ;Why does return anything? Looks like it could be a void function since you're not returning the only possible error case above and not checking the return value below. You're missing an indent here anyway.
Quoted 12 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +3
-1+} +/* Check if the pdev needs to go through non-identity map and unmap process.*/ static int iommu_no_mapping (struct device *dev) {struct pci_dev *pdev; -int found; +int found, current_bus_master;if (unlikely (dev->bus != &pci_bus_type))return 1 ;drivers/iommu/intel-iommu.c static int iommu_no_mapping(struct devic · line 2763
Quoted 7 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +1
-2@@ -2731,9 +2759,22 @@ static int iommu_no_mapping(struct devic* 32 bit DMA is removed from si_domain and fall back * to non-identity mapping. */ -domain_remove_one_dev_info (si_domain, pdev);printk (KERN_INFO "32bit %s uses non-identity mapping\n" , -pci_name (pdev)); +pci_name (pdev));White space damage? Change this to a
pr_infoif you really want to touch it.Quoted 4 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +4
-0+/* +* If a device gets this far we need to clear the Bus +* Master bit before we start moving devices from domain +* to domain. We will also reset the Bus Master bit drivers/iommu/intel-iommu.c static int iommu_no_mapping(struct devic · line 2777
Quoted 10 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +10
-0+* after reprocessing the RMRR info. However, we only +* do both the clearing and setting if needed. +*/ + current_bus_master = pdev->is_busmaster; +if (current_bus_master) +pci_clear_master (pdev); +domain_remove_one_dev_info (si_domain, pdev); +reprocess_rmrr (dev); +if (current_bus_master) +pci_set_master (pdev);I don't know any better way to halt DMA since we can't move the device to a new domain atomically, but what about the other cases where we switch domains? For instance, what if some unsuspecting user tries to assign the device to a guest? I think it's generally inadvisable to assign a devices with RMRRs to a guest, but if they do, they're going to run into the same problem. The RMRRs aren't reprocessed for the VM domain and again aren't reprocessed when returned to a standard device domain. Even more fun, if assigned to a VM domain, RMRRs should be added with the device and removed if the device is later detached from the domain.
The lazy solution might be to disallow devices with RMRRs from being attached via the IOMMU API interfaces (do we need more reasons not to use RMRRs?). Otherwise we need to be proactive about setting up a new domain with RMRR entries for every case and correctly tracking RMRRs for VM domains. Thanks,
Alex
Quoted 3 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +0
-0return 0 ; } }else {
A message that is not in the archive (sent off-list or not yet received): 1348781647.2320.264.camel-85EaTFmN5p//[email protected]
Quoted 19 lines · Alex Williamson
Quoted 5 lines · Thomas
Alex,
Are you suggesting that a solution is to prevent devices with RMRRs from being placed in the SI Domain in the first place (when pt mode is used)?
No, it seems like it's preferable that devices with RMRRs stay in the si domain if the device supports 64bit DMA. They're likely to cause less problems there and we can ignore the RMRRs in the si domain. The problem I'm trying to address is that the domain you're setting up with RMRRs is only used for
dma_ops(ie. host driver use). If the device is attached to a guest, that domain gets discarded and the device is added to yet another domain, potentially with other devices. That domain is missing RMRRs, so now your device is going to generate VT-d faults. The device can then get removed from that domain, in which case the RMRRs should be removed, and again a newdma_opsdomain needs to get created with RMRRs.
Quoted 10 lines
It really seems like RMRRs are incompatible with IOMMU API use though. If an RMRR is setup for a VM domain, that's bad because a) it gives the VM direct access to that range of host memory, and b) it interferes with the guest use of the address space. a) is also bad for isolating devices on the host, but the spec makes it available for abuse. For b), it's not hard to imagine an RMRR range on the host that overlaps with DMA'able space on the guest. Data is read or written to the host memory instead of the guest memory. So maybe the right answer is to make
intel_iommu_attach_devicereturn error if requested to act on a device with RMRR ranges.
That sounds like the right answer to me. Tom, can you make that
change when you address the rest of the comments? Or should it
be a separate patch?
-- ljk
Quoted 133 lines
Thanks,
Alex
Quoted 126 lines
-----Original Message-----
From: Alex Williamson [mailto:[email protected]]
Sent: Thursday, September 27, 2012 3:37 PM
To: Mingarelli, Thomas
Cc: iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org; Knippers, Linda; Khan, Shuah; Don Dutile; David Woodhouse
Subject: Re: [PATCH v2] Intel IOMMU patch to reprocess RMRR info[adding David Woodhouse]
Quoted 28 lines · Tom Mingarelli
When a 32bit PCI device is removed from the SI Domain, the RMRR information for this device becomes invalid and needs to be reprocessed to avoid DMA Read errors. These errors are evidenced by the Present bit being cleared in the device's context entry. Fixing this problem with an enhancement to process the RMRR info when the device is assigned to another domain. The Bus Master bit is cleared during the move to another domain and during the reprocessing of the RMRR info so no DMA can take place at this time.
----
PATCH v1: https://lkml.org/lkml/2012/6/15/204
drivers/iommu/intel-iommu.c| 47 ++++++++++++++++++++++++++++++++++++++++--1 files changed, 44 insertions(+), 3 deletions(-)Signed-off-by Thomas Mingarelli <[email protected]>diff -up
./drivers/iommu/intel-iommu.c.ORIG./drivers/iommu/intel-iommu.cdrivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +6
-0@@ -2706,11 +2706,39 @@ static int iommu_dummy(struct pci_dev *preturn pdev->dev.archdata.iommu ==DUMMY_DEVICE_DOMAIN_INFO ; } +static int reprocess_rmrr (struct device *dev) +{ +struct dmar_rmrr_unit *rmrr; +struct pci_dev *pdev; +int i, ret; +drivers/iommu/intel-iommu.c static int iommu_dummy(struct pci_dev *p · line 2724
Quoted 10 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +10
-0+ pdev =to_pci_dev (dev); + +for_each_rmrr_units (rmrr) { +for (i =0 ; i < rmrr->devices_cnt; i++) { +/* +* Here we are just concerned with +* finding the one device that was +* removed from the si_domain and +* re-evaluating its RMRR info. +*/ I'm still not sure why this comment is wrapped so tightly.
drivers/iommu/intel-iommu.c static int iommu_dummy(struct pci_dev *p · line 2731
Quoted 7 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +7
-0+if (rmrr->devices[i] != pdev) +continue ; +pr_info ("IOMMU: Reprocess RMRR information for device %s.\n" , +pci_name (pdev)); + ret =iommu_prepare_rmrr_dev (rmrr, pdev); +if (ret) +pr_err ("IOMMU: Reprocessing RMRR reserved region for device failed" );This could be "if (
iommu_prepare_rmrr...)" because...drivers/iommu/intel-iommu.c static int iommu_dummy(struct pci_dev *p · line 2734
Quoted 3 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +3
-0+ } + } +return 0 ;Why does return anything? Looks like it could be a void function since you're not returning the only possible error case above and not checking the return value below. You're missing an indent here anyway.
Quoted 12 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +3
-1+} +/* Check if the pdev needs to go through non-identity map and unmap process.*/ static int iommu_no_mapping (struct device *dev) {struct pci_dev *pdev; -int found; +int found, current_bus_master;if (unlikely (dev->bus != &pci_bus_type))return 1 ;drivers/iommu/intel-iommu.c static int iommu_no_mapping(struct devic · line 2763
Quoted 7 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +1
-2@@ -2731,9 +2759,22 @@ static int iommu_no_mapping(struct devic* 32 bit DMA is removed from si_domain and fall back * to non-identity mapping. */ -domain_remove_one_dev_info (si_domain, pdev);printk (KERN_INFO "32bit %s uses non-identity mapping\n" , -pci_name (pdev)); +pci_name (pdev));White space damage? Change this to a
pr_infoif you really want to touch it.Quoted 4 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +4
-0+/* +* If a device gets this far we need to clear the Bus +* Master bit before we start moving devices from domain +* to domain. We will also reset the Bus Master bit drivers/iommu/intel-iommu.c static int iommu_no_mapping(struct devic · line 2777
Quoted 10 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +10
-0+* after reprocessing the RMRR info. However, we only +* do both the clearing and setting if needed. +*/ + current_bus_master = pdev->is_busmaster; +if (current_bus_master) +pci_clear_master (pdev); +domain_remove_one_dev_info (si_domain, pdev); +reprocess_rmrr (dev); +if (current_bus_master) +pci_set_master (pdev);I don't know any better way to halt DMA since we can't move the device to a new domain atomically, but what about the other cases where we switch domains? For instance, what if some unsuspecting user tries to assign the device to a guest? I think it's generally inadvisable to assign a devices with RMRRs to a guest, but if they do, they're going to run into the same problem. The RMRRs aren't reprocessed for the VM domain and again aren't reprocessed when returned to a standard device domain. Even more fun, if assigned to a VM domain, RMRRs should be added with the device and removed if the device is later detached from the domain.
The lazy solution might be to disallow devices with RMRRs from being attached via the IOMMU API interfaces (do we need more reasons not to use RMRRs?). Otherwise we need to be proactive about setting up a new domain with RMRR entries for every case and correctly tracking RMRRs for VM domains. Thanks,
Alex
Quoted 3 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +0
-0return 0 ; } }else {
A message that is not in the archive (sent off-list or not yet received): [email protected]
I think this sounds like a separate patch.
Tom
-----Original Message-----
From: Knippers, Linda
Sent: Thursday, September 27, 2012 4:51 PM
To: Alex Williamson
Cc: Mingarelli, Thomas; iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org; Khan, Shuah; Don Dutile; David Woodhouse
Subject: Re: [PATCH v2] Intel IOMMU patch to reprocess RMRR info
Quoted 19 lines · Alex Williamson
Quoted 5 lines · Thomas
Alex,
Are you suggesting that a solution is to prevent devices with RMRRs from being placed in the SI Domain in the first place (when pt mode is used)?
No, it seems like it's preferable that devices with RMRRs stay in the si domain if the device supports 64bit DMA. They're likely to cause less problems there and we can ignore the RMRRs in the si domain. The problem I'm trying to address is that the domain you're setting up with RMRRs is only used for
dma_ops(ie. host driver use). If the device is attached to a guest, that domain gets discarded and the device is added to yet another domain, potentially with other devices. That domain is missing RMRRs, so now your device is going to generate VT-d faults. The device can then get removed from that domain, in which case the RMRRs should be removed, and again a newdma_opsdomain needs to get created with RMRRs.
Quoted 10 lines
It really seems like RMRRs are incompatible with IOMMU API use though. If an RMRR is setup for a VM domain, that's bad because a) it gives the VM direct access to that range of host memory, and b) it interferes with the guest use of the address space. a) is also bad for isolating devices on the host, but the spec makes it available for abuse. For b), it's not hard to imagine an RMRR range on the host that overlaps with DMA'able space on the guest. Data is read or written to the host memory instead of the guest memory. So maybe the right answer is to make
intel_iommu_attach_devicereturn error if requested to act on a device with RMRR ranges.
That sounds like the right answer to me. Tom, can you make that
change when you address the rest of the comments? Or should it
be a separate patch?
-- ljk
Quoted 133 lines
Thanks,
Alex
Quoted 126 lines
-----Original Message-----
From: Alex Williamson [mailto:[email protected]]
Sent: Thursday, September 27, 2012 3:37 PM
To: Mingarelli, Thomas
Cc: iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org; Knippers, Linda; Khan, Shuah; Don Dutile; David Woodhouse
Subject: Re: [PATCH v2] Intel IOMMU patch to reprocess RMRR info[adding David Woodhouse]
Quoted 28 lines · Tom Mingarelli
When a 32bit PCI device is removed from the SI Domain, the RMRR information for this device becomes invalid and needs to be reprocessed to avoid DMA Read errors. These errors are evidenced by the Present bit being cleared in the device's context entry. Fixing this problem with an enhancement to process the RMRR info when the device is assigned to another domain. The Bus Master bit is cleared during the move to another domain and during the reprocessing of the RMRR info so no DMA can take place at this time.
----
PATCH v1: https://lkml.org/lkml/2012/6/15/204
drivers/iommu/intel-iommu.c| 47 ++++++++++++++++++++++++++++++++++++++++--1 files changed, 44 insertions(+), 3 deletions(-)Signed-off-by Thomas Mingarelli <[email protected]>diff -up
./drivers/iommu/intel-iommu.c.ORIG./drivers/iommu/intel-iommu.cdrivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +6
-0@@ -2706,11 +2706,39 @@ static int iommu_dummy(struct pci_dev *preturn pdev->dev.archdata.iommu ==DUMMY_DEVICE_DOMAIN_INFO ; } +static int reprocess_rmrr (struct device *dev) +{ +struct dmar_rmrr_unit *rmrr; +struct pci_dev *pdev; +int i, ret; +drivers/iommu/intel-iommu.c static int iommu_dummy(struct pci_dev *p · line 2724
Quoted 10 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +10
-0+ pdev =to_pci_dev (dev); + +for_each_rmrr_units (rmrr) { +for (i =0 ; i < rmrr->devices_cnt; i++) { +/* +* Here we are just concerned with +* finding the one device that was +* removed from the si_domain and +* re-evaluating its RMRR info. +*/ I'm still not sure why this comment is wrapped so tightly.
drivers/iommu/intel-iommu.c static int iommu_dummy(struct pci_dev *p · line 2731
Quoted 7 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +7
-0+if (rmrr->devices[i] != pdev) +continue ; +pr_info ("IOMMU: Reprocess RMRR information for device %s.\n" , +pci_name (pdev)); + ret =iommu_prepare_rmrr_dev (rmrr, pdev); +if (ret) +pr_err ("IOMMU: Reprocessing RMRR reserved region for device failed" );This could be "if (
iommu_prepare_rmrr...)" because...drivers/iommu/intel-iommu.c static int iommu_dummy(struct pci_dev *p · line 2734
Quoted 3 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +3
-0+ } + } +return 0 ;Why does return anything? Looks like it could be a void function since you're not returning the only possible error case above and not checking the return value below. You're missing an indent here anyway.
Quoted 12 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +3
-1+} +/* Check if the pdev needs to go through non-identity map and unmap process.*/ static int iommu_no_mapping (struct device *dev) {struct pci_dev *pdev; -int found; +int found, current_bus_master;if (unlikely (dev->bus != &pci_bus_type))return 1 ;drivers/iommu/intel-iommu.c static int iommu_no_mapping(struct devic · line 2763
Quoted 7 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +1
-2@@ -2731,9 +2759,22 @@ static int iommu_no_mapping(struct devic* 32 bit DMA is removed from si_domain and fall back * to non-identity mapping. */ -domain_remove_one_dev_info (si_domain, pdev);printk (KERN_INFO "32bit %s uses non-identity mapping\n" , -pci_name (pdev)); +pci_name (pdev));White space damage? Change this to a
pr_infoif you really want to touch it.Quoted 4 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +4
-0+/* +* If a device gets this far we need to clear the Bus +* Master bit before we start moving devices from domain +* to domain. We will also reset the Bus Master bit drivers/iommu/intel-iommu.c static int iommu_no_mapping(struct devic · line 2777
Quoted 10 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +10
-0+* after reprocessing the RMRR info. However, we only +* do both the clearing and setting if needed. +*/ + current_bus_master = pdev->is_busmaster; +if (current_bus_master) +pci_clear_master (pdev); +domain_remove_one_dev_info (si_domain, pdev); +reprocess_rmrr (dev); +if (current_bus_master) +pci_set_master (pdev);I don't know any better way to halt DMA since we can't move the device to a new domain atomically, but what about the other cases where we switch domains? For instance, what if some unsuspecting user tries to assign the device to a guest? I think it's generally inadvisable to assign a devices with RMRRs to a guest, but if they do, they're going to run into the same problem. The RMRRs aren't reprocessed for the VM domain and again aren't reprocessed when returned to a standard device domain. Even more fun, if assigned to a VM domain, RMRRs should be added with the device and removed if the device is later detached from the domain.
The lazy solution might be to disallow devices with RMRRs from being attached via the IOMMU API interfaces (do we need more reasons not to use RMRRs?). Otherwise we need to be proactive about setting up a new domain with RMRR entries for every case and correctly tracking RMRRs for VM domains. Thanks,
Alex
Quoted 3 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +0
-0return 0 ; } }else {
Quoted 31 lines · Linda Knippers
Quoted 19 lines · Alex Williamson
Quoted 5 lines · Thomas
Alex,
Are you suggesting that a solution is to prevent devices with RMRRs from being placed in the SI Domain in the first place (when pt mode is used)?
No, it seems like it's preferable that devices with RMRRs stay in the si domain if the device supports 64bit DMA. They're likely to cause less problems there and we can ignore the RMRRs in the si domain. The problem I'm trying to address is that the domain you're setting up with RMRRs is only used for
dma_ops(ie. host driver use). If the device is attached to a guest, that domain gets discarded and the device is added to yet another domain, potentially with other devices. That domain is missing RMRRs, so now your device is going to generate VT-d faults. The device can then get removed from that domain, in which case the RMRRs should be removed, and again a newdma_opsdomain needs to get created with RMRRs.Quoted 10 lines
It really seems like RMRRs are incompatible with IOMMU API use though. If an RMRR is setup for a VM domain, that's bad because a) it gives the VM direct access to that range of host memory, and b) it interferes with the guest use of the address space. a) is also bad for isolating devices on the host, but the spec makes it available for abuse. For b), it's not hard to imagine an RMRR range on the host that overlaps with DMA'able space on the guest. Data is read or written to the host memory instead of the guest memory. So maybe the right answer is to make
intel_iommu_attach_devicereturn error if requested to act on a device with RMRR ranges.
Quoted 3 lines
That sounds like the right answer to me. Tom, can you make that
change when you address the rest of the comments? Or should it
be a separate patch?
Please include a pr_info when intel_iommu_attach_device takes this path or we're going to have a really hard time figuring out why a device won't attach to a guest. Thanks,
Alex
Quoted 130 lines
Quoted 129 lines
Quoted 126 lines
-----Original Message-----
From: Alex Williamson [mailto:[email protected]]
Sent: Thursday, September 27, 2012 3:37 PM
To: Mingarelli, Thomas
Cc: iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org; Knippers, Linda; Khan, Shuah; Don Dutile; David Woodhouse
Subject: Re: [PATCH v2] Intel IOMMU patch to reprocess RMRR info[adding David Woodhouse]
Quoted 28 lines · Tom Mingarelli
When a 32bit PCI device is removed from the SI Domain, the RMRR information for this device becomes invalid and needs to be reprocessed to avoid DMA Read errors. These errors are evidenced by the Present bit being cleared in the device's context entry. Fixing this problem with an enhancement to process the RMRR info when the device is assigned to another domain. The Bus Master bit is cleared during the move to another domain and during the reprocessing of the RMRR info so no DMA can take place at this time.
----
PATCH v1: https://lkml.org/lkml/2012/6/15/204
drivers/iommu/intel-iommu.c| 47 ++++++++++++++++++++++++++++++++++++++++--1 files changed, 44 insertions(+), 3 deletions(-)Signed-off-by Thomas Mingarelli <[email protected]>diff -up
./drivers/iommu/intel-iommu.c.ORIG./drivers/iommu/intel-iommu.cdrivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +6
-0@@ -2706,11 +2706,39 @@ static int iommu_dummy(struct pci_dev *preturn pdev->dev.archdata.iommu ==DUMMY_DEVICE_DOMAIN_INFO ; } +static int reprocess_rmrr (struct device *dev) +{ +struct dmar_rmrr_unit *rmrr; +struct pci_dev *pdev; +int i, ret; +drivers/iommu/intel-iommu.c static int iommu_dummy(struct pci_dev *p · line 2724
Quoted 10 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +10
-0+ pdev =to_pci_dev (dev); + +for_each_rmrr_units (rmrr) { +for (i =0 ; i < rmrr->devices_cnt; i++) { +/* +* Here we are just concerned with +* finding the one device that was +* removed from the si_domain and +* re-evaluating its RMRR info. +*/ I'm still not sure why this comment is wrapped so tightly.
drivers/iommu/intel-iommu.c static int iommu_dummy(struct pci_dev *p · line 2731
Quoted 7 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +7
-0+if (rmrr->devices[i] != pdev) +continue ; +pr_info ("IOMMU: Reprocess RMRR information for device %s.\n" , +pci_name (pdev)); + ret =iommu_prepare_rmrr_dev (rmrr, pdev); +if (ret) +pr_err ("IOMMU: Reprocessing RMRR reserved region for device failed" );This could be "if (
iommu_prepare_rmrr...)" because...drivers/iommu/intel-iommu.c static int iommu_dummy(struct pci_dev *p · line 2734
Quoted 3 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +3
-0+ } + } +return 0 ;Why does return anything? Looks like it could be a void function since you're not returning the only possible error case above and not checking the return value below. You're missing an indent here anyway.
Quoted 12 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +3
-1+} +/* Check if the pdev needs to go through non-identity map and unmap process.*/ static int iommu_no_mapping (struct device *dev) {struct pci_dev *pdev; -int found; +int found, current_bus_master;if (unlikely (dev->bus != &pci_bus_type))return 1 ;drivers/iommu/intel-iommu.c static int iommu_no_mapping(struct devic · line 2763
Quoted 7 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +1
-2@@ -2731,9 +2759,22 @@ static int iommu_no_mapping(struct devic* 32 bit DMA is removed from si_domain and fall back * to non-identity mapping. */ -domain_remove_one_dev_info (si_domain, pdev);printk (KERN_INFO "32bit %s uses non-identity mapping\n" , -pci_name (pdev)); +pci_name (pdev));White space damage? Change this to a
pr_infoif you really want to touch it.Quoted 4 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +4
-0+/* +* If a device gets this far we need to clear the Bus +* Master bit before we start moving devices from domain +* to domain. We will also reset the Bus Master bit drivers/iommu/intel-iommu.c static int iommu_no_mapping(struct devic · line 2777
Quoted 10 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +10
-0+* after reprocessing the RMRR info. However, we only +* do both the clearing and setting if needed. +*/ + current_bus_master = pdev->is_busmaster; +if (current_bus_master) +pci_clear_master (pdev); +domain_remove_one_dev_info (si_domain, pdev); +reprocess_rmrr (dev); +if (current_bus_master) +pci_set_master (pdev);I don't know any better way to halt DMA since we can't move the device to a new domain atomically, but what about the other cases where we switch domains? For instance, what if some unsuspecting user tries to assign the device to a guest? I think it's generally inadvisable to assign a devices with RMRRs to a guest, but if they do, they're going to run into the same problem. The RMRRs aren't reprocessed for the VM domain and again aren't reprocessed when returned to a standard device domain. Even more fun, if assigned to a VM domain, RMRRs should be added with the device and removed if the device is later detached from the domain.
The lazy solution might be to disallow devices with RMRRs from being attached via the IOMMU API interfaces (do we need more reasons not to use RMRRs?). Otherwise we need to be proactive about setting up a new domain with RMRR entries for every case and correctly tracking RMRRs for VM domains. Thanks,
Alex
Quoted 3 lines
drivers/iommu/intel-iommu.c renamed from drivers/iommu/intel-iommu.c.ORIG +0
-0return 0 ; } }else {
Quoted 10 lines · Alex Williamson
It really seems like RMRRs are incompatible with IOMMU API use though. If an RMRR is setup for a VM domain, that's bad because a) it gives the VM direct access to that range of host memory, and b) it interferes with the guest use of the address space. a) is also bad for isolating devices on the host, but the spec makes it available for abuse. For b), it's not hard to imagine an RMRR range on the host that overlaps with DMA'able space on the guest. Data is read or written to the host memory instead of the guest memory. So maybe the right answer is to make
intel_iommu_attach_devicereturn error if requested to act on a device with RMRR ranges.
That would include fairly much any USB host controller. The whole RMRR concept is completely broken and should never have been invented. The idea that firmware-controlled DMA should continue to happen *after* the operating system has been booted and taken control of the hardware is just insane.
The majority of RMRR use is for USB controllers, so that firmware can emulate a legacy keyboard for the benefit of pre-USB operating systems. But no operating system that old is ever going to support the IOMMU anyway, so that's just mad. But that's why we've managed to get away with setting up the RMRRs and then tearing them down when a native driver has actually taken control of the hardware — because at that point, it should have been reset and whatever the firmware had configured to do has been abandoned.
Perhaps we should have a special case for USB controllers, which are quite happy when you drop their RMRR regions when the OS takes over, and for any *other* device with RMRRs, have a TAINT_YOUR_FIRMWARE_IS_INSANE and just refuse to let a native driver do anything at all with them?
Signature · 1 line
dwmw2
A message that is not in the archive (sent off-list or not yet received): 1348782605.2036.117.camel-Fexsq3y4057IgHVZqg5X0TlWvGAXklZc@public.gmane.org
Quoted 6 lines · David Woodhouse
That would include fairly much any USB host controller. The whole RMRR concept is completely broken and should never have been invented. The idea that firmware-controlled DMA should continue to happen *after* the operating system has been booted and taken control of the hardware is just insane.
Quoted 8 lines
The majority of RMRR use is for USB controllers, so that firmware can emulate a legacy keyboard for the benefit of pre-USB operating systems. But no operating system that old is ever going to support the IOMMU anyway, so that's just mad. But that's why we've managed to get away with setting up the RMRRs and then tearing them down when a native driver has actually taken control of the hardware — because at that point, it should have been reset and whatever the firmware had configured to do has been abandoned.
Even on modern hardware with modern (IOMMU aware) kernels there is still this small time window when the OS has enabled the IOMMU and the USB driver is not initialized yet. In this time window the RMRR memory region is still necessary, no?
As I said already in another mail, I think it is safe to ignore any RMRR requirements when we start to use a device in the OS.
Joerg
Signature · 10 lines
AMD Operating System Research Center Advanced Micro Devices GmbH Einsteinring 24 85609 Dornach General Managers: Alberto Bozzo Registration: Dornach, Landkr. Muenchen; Registerger. Muenchen, HRB Nr. 43632 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 4 lines · Joerg Roedel
Even on modern hardware with modern (IOMMU aware) kernels there is still this small time window when the OS has enabled the IOMMU and the USB driver is not initialized yet. In this time window the RMRR memory region is still necessary, no?
Yes but there's no *reason* for that. It wouldn't be that hard to ask the firmware to quiesce all its own DMA *before* we enable the IOMMU.
Quoted 2 lines
As I said already in another mail, I think it is safe to ignore any RMRR requirements when we start to use a device in the OS.
I think the whole point in this patch is that there is some brain-dead hardware out there (vendor 'value subtract' I think) on which that common-sense observation isn't actually true.
I'm all for handling that broken hardware with quirks, giving clear messages to the user that the device(+firmware) in question is broken, and refusing to let either the kernel *or* VM guests do any DMA with it.
Signature · 1 line
dwmw2
A message that is not in the archive (sent off-list or not yet received): 1348827803.2036.121.camel-Fexsq3y4057IgHVZqg5X0TlWvGAXklZc@public.gmane.org
Quoted 8 lines · David Woodhouse
Quoted 4 lines · Joerg Roedel
Even on modern hardware with modern (IOMMU aware) kernels there is still this small time window when the OS has enabled the IOMMU and the USB driver is not initialized yet. In this time window the RMRR memory region is still necessary, no?
Yes but there's no *reason* for that. It wouldn't be that hard to ask the firmware to quiesce all its own DMA *before* we enable the IOMMU.
True. That would have been a better approach. Some kind of IOMMU handover from firmware to the OS.
Quoted 10 lines
Quoted 2 lines
As I said already in another mail, I think it is safe to ignore any RMRR requirements when we start to use a device in the OS.
I think the whole point in this patch is that there is some brain-dead hardware out there (vendor 'value subtract' I think) on which that common-sense observation isn't actually true.
I'm all for handling that broken hardware with quirks, giving clear messages to the user that the device(+firmware) in question is broken, and refusing to let either the kernel *or* VM guests do any DMA with it.
Well, it is probably mostly about southbridge devices and add-on USB controllers. Or is there any other way a given device can communicate RMRR requirements to the firmware?
Anyway, I am fine with completly blocking DMA for those devices too.
Joerg
Quoted 2 lines · Alex Williamson
It really seems like RMRRs are incompatible with IOMMU API use though.
I don't think so. The concept of RMRR is just not defined well enough (like the concept of unity mappings on the AMD side which is similar to
RMRR). The definition says, that any memory region must be mapped at
any time for the device. But that is not true (at least I have no counter-example yet). The right definition would be, that the RMRR regions are only necessary as long as the operating system does not control the particular device. And assigning a device to a guest also counts a 'taking control over the device'.
Joerg
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 5 lines · Joerg Roedel
Quoted 2 lines · Alex Williamson
It really seems like RMRRs are incompatible with IOMMU API use though.
Quoted 8 lines
I don't think so. The concept of RMRR is just not defined well enough (like the concept of unity mappings on the AMD side which is similar to
RMRR). The definition says, that any memory region must be mapped at
any time for the device. But that is not true (at least I have no counter-example yet). The right definition would be, that the RMRR regions are only necessary as long as the operating system does not control the particular device. And assigning a device to a guest also counts a 'taking control over the device'.
I think HP folks would be very unhappy with that definition. As David indicates, that's how things like USB use RMRR, but the actual definition in the spec leaves much more room for abuse. Thanks,
Alex
A message that is not in the archive (sent off-list or not yet received): 1348836008.2320.284.camel-85EaTFmN5p//[email protected]
Quoted 11 lines · Alex Williamson
Quoted 8 lines · Joerg Roedel
I don't think so. The concept of RMRR is just not defined well enough (like the concept of unity mappings on the AMD side which is similar to
RMRR). The definition says, that any memory region must be mapped at
any time for the device. But that is not true (at least I have no counter-example yet). The right definition would be, that the RMRR regions are only necessary as long as the operating system does not control the particular device. And assigning a device to a guest also counts a 'taking control over the device'.
Quoted 3 lines
I think HP folks would be very unhappy with that definition. As David indicates, that's how things like USB use RMRR, but the actual definition in the spec leaves much more room for abuse. Thanks,
To my experience, for a hardware designer, existing software overrides any Spec because it is much worse to break existing software than it is to break a Spec :) So, unless we break existing hardware/firmware, I still suggest that we use the assumption that OS controlled devices do not need RMRR/unity-mapped regions anymore.
Is HP doing anything in their firmware which would not work with that?
For the USB controlers, they only generate DMA to the RMRR/unity-mapped region until the OS takes over control from the firmware. After the USB driver is initialized the RMRR region should not be necessary anymore.
Joerg
Signature · 5 lines
AMD Operating System Research Center Advanced Micro Devices GmbH Einsteinring 24 85609 Dornach General Managers: Alberto Bozzo Registration: Dornach, Landkr. Muenchen; Registerger. Muenchen, HRB Nr. 43632
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 16 lines · Joerg Roedel
Quoted 11 lines · Alex Williamson
Quoted 8 lines · Joerg Roedel
I don't think so. The concept of RMRR is just not defined well enough (like the concept of unity mappings on the AMD side which is similar to
RMRR). The definition says, that any memory region must be mapped at
any time for the device. But that is not true (at least I have no counter-example yet). The right definition would be, that the RMRR regions are only necessary as long as the operating system does not control the particular device. And assigning a device to a guest also counts a 'taking control over the device'.Quoted 3 lines
I think HP folks would be very unhappy with that definition. As David indicates, that's how things like USB use RMRR, but the actual definition in the spec leaves much more room for abuse. Thanks,
Quoted 5 lines
To my experience, for a hardware designer, existing software overrides any Spec because it is much worse to break existing software than it is to break a Spec :) So, unless we break existing hardware/firmware, I still suggest that we use the assumption that OS controlled devices do not need RMRR/unity-mapped regions anymore.
HP has been shipping hardware that makes use of RMRRs for other purposes for a while.
Quoted 1 line
Is HP doing anything in their firmware which would not work with that?
Yes, I'll let them fill in the details.
Quoted 4 lines
For the USB controlers, they only generate DMA to the RMRR/unity-mapped region until the OS takes over control from the firmware. After the USB driver is initialized the RMRR region should not be necessary anymore.
I agree that was probably the intent, but vendors have found loopholes as their opportunity to innovate. Thanks,
Alex