Skip to contentSkip to conversation

Clean up VMD DMA Map Ops

RFC Superseded Jon Derrick 5 participants 21 messages Jan 1, 2020 – Jan 14, 2020

Subsystems (5)

v1 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

cover Clean up VMD DMA Map Ops

Inspired by Christoph's last set:
https://lkml.org/lkml/2019/8/28/667

VMD currently works with VT-d enabled by pointing DMA and IOMMU actions at the VMD endpoint. The problem with this approach is that the VMD endpoint's device-specific attributes, such as the dma mask, are used instead.

This set cleans up VMD by removing the override that redirects dma map operations to the VMD endpoint. Instead it introduces a new dma alias mechanism into the existing dma alias infrastructure.

Patch 1 and 2 are miscellaneous fixes discovered during development. Patch 1 is ready, but 2 likely doesn't go far enough for proper teardown on addition failure.

Jon Derrick (5):
iommu: Remove device link to group on failure iommu/vt-d: Unlink device if failed to add to group x86/PCI: Expose VMD's device in pci_sysdata
PCI: vmd: Stop overriding dma_map_ops x86/PCI: Remove unused X86_DEV_DMA_OPS

 arch/x86/Kconfig               |   3 -
 arch/x86/include/asm/device.h  |  10 ---
 arch/x86/include/asm/pci.h     |   4 +-
 arch/x86/pci/common.c          |  44 ++----------
 drivers/iommu/intel-iommu.c    |  26 ++++---
 drivers/iommu/iommu.c          |   1 +
 drivers/pci/controller/Kconfig |   1 -
 drivers/pci/controller/vmd.c   | 152 +----------------------------------------
 drivers/pci/pci.c              |   4 +-
 drivers/pci/search.c           |   6 ++
 include/linux/pci.h            |   1 +
 11 files changed, 37 insertions(+), 215 deletions(-)
Signature · 1 line
1.8.3.1
#1 lore raw

RFC 1/5 iommu: Remove device link to group on failure

This adds the missing teardown step that removes the device link from the group when the device addition fails.

---
 drivers/iommu/iommu.c | 1 +
 1 file changed, 1 insertion(+)
drivers/iommu/iommu.c +1 -0
index d5174f0..3e35284 100644
@@ -768,6 +768,7 @@ int iommu_group_add_device(struct iommu_group *group, struct device *dev)
 	mutex_unlock(&group->mutex);
 	dev->iommu_group = NULL;
 	kobject_put(group->devices_kobj);
+	sysfs_remove_link(group->devices_kobj, device->name);
 err_free_name:
 	kfree(device->name);
 err_remove_link:
Signature · 1 line
1.8.3.1
Reviewed-by on v1 1/5
#2 lore raw

Re: [RFC 1/5] iommu: Remove device link to group on failure

Hi,

Quoted 2 lines · Jon Derrick

This adds the missing teardown step that removes the device link from the group when the device addition fails.

This change looks good to me.

Best regards,
baolu

Quoted 19 lines
---
  drivers/iommu/iommu.c | 1 +
  1 file changed, 1 insertion(+)
drivers/iommu/iommu.c +1 -0
index d5174f0..3e35284 100644
@@ -768,6 +768,7 @@ int iommu_group_add_device(struct iommu_group *group, struct device *dev)
  	mutex_unlock(&group->mutex);
  	dev->iommu_group = NULL;
  	kobject_put(group->devices_kobj);
+	sysfs_remove_link(group->devices_kobj, device->name);
  err_free_name:
  	kfree(device->name);
  err_remove_link:
#3 lore raw

RFC 2/5 iommu/vt-d: Unlink device if failed to add to group

In reply to Jon Derrick

If the device fails to be added to the group, make sure to unlink the reference before returning.

---
 drivers/iommu/intel-iommu.c | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)
drivers/iommu/intel-iommu.c +10 -3
index b2526a4..978d502 100644
@@ -5625,8 +5625,10 @@ static int intel_iommu_add_device(struct device *dev)
 
 	group = iommu_group_get_for_dev(dev);
 
-	if (IS_ERR(group))
-		return PTR_ERR(group);
+	if (IS_ERR(group)) {
+		ret = PTR_ERR(group);
+		goto unlink;
+	}
 
 	iommu_group_put(group);
 
@@ -5652,7 +5654,8 @@ static int intel_iommu_add_device(struct device *dev)
 				if (!get_private_domain_for_dev(dev)) {
 					dev_warn(dev,
 						 "Failed to get a private domain.\n");
-					return -ENOMEM;
+					ret = -ENOMEM;
+					goto unlink;
 				}
 
 				dev_info(dev,
@@ -5667,6 +5670,10 @@ static int intel_iommu_add_device(struct device *dev)
 	}
 
 	return 0;
+
+unlink:
+	iommu_device_unlink(&iommu->iommu, dev);
+	return ret;
 }
 
 static void intel_iommu_remove_device(struct device *dev)
Signature · 1 line
1.8.3.1
Acked-by on v1 2/5
#4 lore raw

Re: [RFC 2/5] iommu/vt-d: Unlink device if failed to add to group

Quoted 4 lines · Jon Derrick

If the device fails to be added to the group, make sure to unlink the reference before returning.

This fix looks reasonable to me.

Best regards,
baolu

Quoted 43 lines
---
  drivers/iommu/intel-iommu.c | 13 ++++++++++---
  1 file changed, 10 insertions(+), 3 deletions(-)
drivers/iommu/intel-iommu.c +10 -3
index b2526a4..978d502 100644
@@ -5625,8 +5625,10 @@ static int intel_iommu_add_device(struct device *dev)
 
  	group = iommu_group_get_for_dev(dev);
 
-	if (IS_ERR(group))
-		return PTR_ERR(group);
+	if (IS_ERR(group)) {
+		ret = PTR_ERR(group);
+		goto unlink;
+	}
 
  	iommu_group_put(group);
 
@@ -5652,7 +5654,8 @@ static int intel_iommu_add_device(struct device *dev)
  				if (!get_private_domain_for_dev(dev)) {
  					dev_warn(dev,
  						 "Failed to get a private domain.\n");
-					return -ENOMEM;
+					ret = -ENOMEM;
+					goto unlink;
  				}
 
  				dev_info(dev,
@@ -5667,6 +5670,10 @@ static int intel_iommu_add_device(struct device *dev)
  	}
 
  	return 0;
+
+unlink:
+	iommu_device_unlink(&iommu->iommu, dev);
+	return ret;
  }
 
  static void intel_iommu_remove_device(struct device *dev)
maintainer
#6 lore raw

Re: [RFC 2/5] iommu/vt-d: Unlink device if failed to add to group

Quoted 7 lines · Lu Baolu
Quoted 4 lines · Jon Derrick

If the device fails to be added to the group, make sure to unlink the reference before returning.

Queued for v5.6.

No need to do so, I sent it upstream with the last pile of iommu fixes.

Thanks,

Joerg

#7 lore raw

Re: [RFC 2/5] iommu/vt-d: Unlink device if failed to add to group

Quoted 10 lines · Joerg Roedel
Quoted 7 lines · Lu Baolu
Quoted 4 lines · Jon Derrick

If the device fails to be added to the group, make sure to unlink the reference before returning.

Queued for v5.6.

No need to do so, I sent it upstream with the last pile of iommu fixes.

Got it. Thank you!

Best regards,
baolu

#8 lore raw

RFC 3/5 x86/PCI: Expose VMD's device in pci_sysdata

In reply to Jon Derrick

To be used by intel-iommu code to find the correct domain.

---
 arch/x86/include/asm/pci.h   | 4 ++--
 drivers/pci/controller/vmd.c | 2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)
arch/x86/include/asm/pci.h +2 -2
index 90d0731..7656807 100644
@@ -25,7 +25,7 @@ struct pci_sysdata {
 	void		*fwnode;	/* IRQ domain for MSI assignment */
 #endif
 #if IS_ENABLED(CONFIG_VMD)
-	bool vmd_domain;		/* True if in Intel VMD domain */
+	struct device *vmd_dev;		/* Non-null if in Intel VMD domain */
 #endif
 };
 
@@ -65,7 +65,7 @@ static inline bool is_vmd(struct pci_bus *bus)
 #if IS_ENABLED(CONFIG_VMD)
 	struct pci_sysdata *sd = bus->sysdata;
 
-	return sd->vmd_domain;
+	return !!sd->vmd_dev;
 #else
 	return false;
 #endif
drivers/pci/controller/vmd.c +1 -1
index 2128422..907b5bd 100644
@@ -679,7 +679,7 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)
 		.parent = res,
 	};
 
-	sd->vmd_domain = true;
+	sd->vmd_dev = &vmd->dev->dev;
 	sd->domain = vmd_find_free_domain();
 	if (sd->domain < 0)
 		return sd->domain;
Signature · 1 line
1.8.3.1
#9 lore raw

Re: [RFC 3/5] x86/PCI: Expose VMD's device in pci_sysdata

Quoted 1 line · Jon Derrick

To be used by intel-iommu code to find the correct domain.

Any reason to prefer this version over my patches 2 and 3 from the series in August?

#10 lore raw

Re: [RFC 3/5] x86/PCI: Expose VMD's device in pci_sysdata

Quoted 5 lines · Christoph Hellwig
Quoted 1 line · Jon Derrick

To be used by intel-iommu code to find the correct domain.

Any reason to prefer this version over my patches 2 and 3 from the series in August?

Mine uses the correct device's dma mask

#11 lore raw

Re: [RFC 3/5] x86/PCI: Expose VMD's device in pci_sysdata

In reply to Christoph Hellwig
Quoted 5 lines · Christoph Hellwig
Quoted 1 line · Jon Derrick

To be used by intel-iommu code to find the correct domain.

Any reason to prefer this version over my patches 2 and 3 from the series in August?

2 & 3 of your set is fine.

#12 lore raw

RFC 4/5 PCI: vmd: Stop overriding dma_map_ops

In reply to Jon Derrick

Devices on the VMD domain use the VMD endpoint's requester-id and have been relying on the VMD endpoint's dma operations. The downside of this was that VMD domain devices would use the VMD endpoint's attributes when doing DMA and IOMMU mapping. We can be smarter about this by only using the VMD endpoint when mapping and providing the correct child device's attributes during dma operations.

This patch adds a new dma alias mechanism by adding a hint to a pci_dev to point to a singular DMA requester's pci_dev. This integrates into the existing dma alias infrastructure to reduce the impact of the changes required to support this mode.

---
 arch/x86/pci/common.c          |   6 +-
 drivers/iommu/intel-iommu.c    |  13 ++--
 drivers/pci/controller/Kconfig |   1 -
 drivers/pci/controller/vmd.c   | 150 -----------------------------------------
 drivers/pci/pci.c              |   4 +-
 drivers/pci/search.c           |   6 ++
 include/linux/pci.h            |   1 +
 7 files changed, 23 insertions(+), 158 deletions(-)
arch/x86/pci/common.c +5 -1
index 1e59df0..4022609 100644
@@ -664,8 +664,12 @@ static void set_dma_domain_ops(struct pci_dev *pdev) {}
 
 static void set_dev_domain_options(struct pci_dev *pdev)
 {
-	if (is_vmd(pdev->bus))
+	if (is_vmd(pdev->bus)) {
+		struct pci_sysdata *sd = pdev->bus->sysdata;
+
+		pdev->dma_parent = to_pci_dev(sd->vmd_dev);
 		pdev->hotplug_user_indicators = 1;
+	}
 }
 
 int pcibios_add_device(struct pci_dev *dev)
drivers/iommu/intel-iommu.c +8 -5
index 978d502..5aee648 100644
@@ -776,11 +776,8 @@ static struct intel_iommu *device_to_iommu(struct device *dev, u8 *bus, u8 *devf
 
 		pdev = to_pci_dev(dev);
 
-#ifdef CONFIG_X86
-		/* VMD child devices currently cannot be handled individually */
-		if (is_vmd(pdev->bus))
-			return NULL;
-#endif
+		if (pdev->dma_parent)
+			pdev = pdev->dma_parent;
 
 		/* VFs aren't listed in scope tables; we need to look up
 		 * the PF instead to find the IOMMU. */
@@ -2428,6 +2425,12 @@ static struct dmar_domain *find_domain(struct device *dev)
 		     dev->archdata.iommu == DUMMY_DEVICE_DOMAIN_INFO))
 		return NULL;
 
+	if (dev_is_pci(dev)) {
+		struct pci_dev *pdev = to_pci_dev(dev);
+		if (pdev->dma_parent)
+			dev = &pdev->dma_parent->dev;
+	}
+
 	/* No lock here, assumes no domain exit in normal case */
 	info = dev->archdata.iommu;
 	if (likely(info))
drivers/pci/controller/Kconfig +0 -1
index c77069c..55671429 100644
@@ -239,7 +239,6 @@ config PCIE_TANGO_SMP8759
 
 config VMD
 	depends on PCI_MSI && X86_64 && SRCU
-	select X86_DEV_DMA_OPS
 	tristate "Intel Volume Management Device Driver"
 	---help---
 	  Adds support for the Intel Volume Management Device (VMD). VMD is a
drivers/pci/controller/vmd.c +0 -150

174 lines not on this page · raw message

drivers/pci/pci.c +3 -1
index ad746d9..ef7f219 100644
@@ -6034,7 +6034,9 @@ bool pci_devs_are_dma_aliases(struct pci_dev *dev1, struct pci_dev *dev2)
 	return (dev1->dma_alias_mask &&
 		test_bit(dev2->devfn, dev1->dma_alias_mask)) ||
 	       (dev2->dma_alias_mask &&
-		test_bit(dev1->devfn, dev2->dma_alias_mask));
+		test_bit(dev1->devfn, dev2->dma_alias_mask)) ||
+	       (dev1->dma_parent && dev1->dma_parent == dev2) ||
+	       (dev2->dma_parent && dev2->dma_parent == dev1);
 }
 
 bool pci_device_is_present(struct pci_dev *pdev)
drivers/pci/search.c +6 -0
index bade140..a1d4899 100644
@@ -32,6 +32,12 @@ int pci_for_each_dma_alias(struct pci_dev *pdev,
 	struct pci_bus *bus;
 	int ret;
 
+	if (unlikely(pdev->dma_parent)) {
+		pdev = pdev->dma_parent;
+		return fn(pdev, PCI_DEVID(pdev->bus->number, pdev->devfn),
+			  data);
+	}
+
 	ret = fn(pdev, pci_dev_id(pdev), data);
 	if (ret)
 		return ret;
include/linux/pci.h +1 -0
index c393dff..0a10d1e 100644
@@ -316,6 +316,7 @@ struct pci_dev {
 	u8		pin;		/* Interrupt pin this device uses */
 	u16		pcie_flags_reg;	/* Cached PCIe Capabilities Register */
 	unsigned long	*dma_alias_mask;/* Mask of enabled devfn aliases */
+	struct pci_dev	*dma_parent;	/* DMA requester on another bus */
 
 	struct pci_driver *driver;	/* Driver bound to this device */
 	u64		dma_mask;	/* Mask of the bits of bus address this
Signature · 1 line
1.8.3.1
#13 lore raw

Re: [RFC 4/5] PCI: vmd: Stop overriding dma_map_ops

[if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please…

Hi Jon,

[FYI, it's a private test report for your RFC patch.]
[auto build test ERROR on pci/next]
[also build test ERROR on iommu/next tip/x86/core v5.5-rc4 next-20191220] [if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please see https://stackoverflow.com/a/37406982]

url:    https://github.com/0day-ci/linux/commits/Jon-Derrick/Clean-up-VMD-DMA-Map-Ops/20200103-175834
base:   https://git.kernel.org/pub/scm/linux/kernel/git/helgaas/pci.git next
config: x86_64-lkp (attached as .config)
compiler: gcc-7 (Debian 7.5.0-3) 7.5.0
reproduce:
        # save the attached .config to linux build tree
        make ARCH=x86_64 

If you fix the issue, kindly add following tag

All error/warnings (new ones prefixed by >>):

In file included from arch/x86/include/asm/percpu.h:45:0,

                    from arch/x86/include/asm/current.h:6,
                    from include/linux/sched.h:12,
                    from arch/x86/pci/common.c:8:
   arch/x86/pci/common.c: In function 'set_dev_domain_options':
>> arch/x86/pci/common.c:670:35: error: 'struct pci_sysdata' has no member named 'vmd_dev'
      pdev->dma_parent = to_pci_dev(sd->vmd_dev);
                                      ^
   include/linux/kernel.h:995:26: note: in definition of macro 'container_of'
     void *__mptr = (void *)(ptr);     \
                             ^~~
>> arch/x86/pci/common.c:670:22: note: in expansion of macro 'to_pci_dev'
      pdev->dma_parent = to_pci_dev(sd->vmd_dev);
                         ^~~~~~~~~~
   In file included from arch/x86/include/asm/current.h:5:0,
                    from include/linux/sched.h:12,
                    from arch/x86/pci/common.c:8:
>> arch/x86/pci/common.c:670:35: error: 'struct pci_sysdata' has no member named 'vmd_dev'
      pdev->dma_parent = to_pci_dev(sd->vmd_dev);
                                      ^
   include/linux/compiler.h:330:9: note: in definition of macro '__compiletime_assert'
      if (!(condition))     \
            ^~~~~~~~~
   include/linux/compiler.h:350:2: note: in expansion of macro '_compiletime_assert'
     _compiletime_assert(condition, msg, __compiletime_assert_, __LINE__)
     ^~~~~~~~~~~~~~~~~~~
   include/linux/build_bug.h:39:37: note: in expansion of macro 'compiletime_assert'
    #define BUILD_BUG_ON_MSG(cond, msg) compiletime_assert(!(cond), msg)
                                        ^~~~~~~~~~~~~~~~~~
   include/linux/kernel.h:996:2: note: in expansion of macro 'BUILD_BUG_ON_MSG'
     BUILD_BUG_ON_MSG(!__same_type(*(ptr), ((type *)0)->member) && \
     ^~~~~~~~~~~~~~~~
   include/linux/kernel.h:996:20: note: in expansion of macro '__same_type'
     BUILD_BUG_ON_MSG(!__same_type(*(ptr), ((type *)0)->member) && \
                       ^~~~~~~~~~~
>> include/linux/pci.h:486:23: note: in expansion of macro 'container_of'
    #define to_pci_dev(n) container_of(n, struct pci_dev, dev)
                          ^~~~~~~~~~~~
>> arch/x86/pci/common.c:670:22: note: in expansion of macro 'to_pci_dev'
      pdev->dma_parent = to_pci_dev(sd->vmd_dev);
                         ^~~~~~~~~~
>> arch/x86/pci/common.c:670:35: error: 'struct pci_sysdata' has no member named 'vmd_dev'
      pdev->dma_parent = to_pci_dev(sd->vmd_dev);
                                      ^
   include/linux/compiler.h:330:9: note: in definition of macro '__compiletime_assert'
      if (!(condition))     \
            ^~~~~~~~~
   include/linux/compiler.h:350:2: note: in expansion of macro '_compiletime_assert'
     _compiletime_assert(condition, msg, __compiletime_assert_, __LINE__)
     ^~~~~~~~~~~~~~~~~~~
   include/linux/build_bug.h:39:37: note: in expansion of macro 'compiletime_assert'
    #define BUILD_BUG_ON_MSG(cond, msg) compiletime_assert(!(cond), msg)
                                        ^~~~~~~~~~~~~~~~~~
   include/linux/kernel.h:996:2: note: in expansion of macro 'BUILD_BUG_ON_MSG'
     BUILD_BUG_ON_MSG(!__same_type(*(ptr), ((type *)0)->member) && \
     ^~~~~~~~~~~~~~~~
   include/linux/kernel.h:997:6: note: in expansion of macro '__same_type'
        !__same_type(*(ptr), void),   \
         ^~~~~~~~~~~
>> include/linux/pci.h:486:23: note: in expansion of macro 'container_of'
    #define to_pci_dev(n) container_of(n, struct pci_dev, dev)
                          ^~~~~~~~~~~~
>> arch/x86/pci/common.c:670:22: note: in expansion of macro 'to_pci_dev'
      pdev->dma_parent = to_pci_dev(sd->vmd_dev);
                         ^~~~~~~~~~

vim +670 arch/x86/pci/common.c

664
665 static void set_dev_domain_options(struct pci_dev *pdev)
666 {
667 if (is_vmd(pdev->bus)) {
668 struct pci_sysdata *sd = pdev->bus->sysdata;
669

 > 670			pdev->dma_parent = to_pci_dev(sd->vmd_dev);

671 pdev->hotplug_user_indicators = 1;
672 }
673 }
674

---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org Intel Corporation

#14 lore raw

Re: [RFC 4/5] PCI: vmd: Stop overriding dma_map_ops

In reply to Jon Derrick
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please…

Hi Jon,

[FYI, it's a private test report for your RFC patch.]
[auto build test WARNING on pci/next]
[also build test WARNING on iommu/next tip/x86/core v5.5-rc4 next-20191220] [if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please see https://stackoverflow.com/a/37406982]

url:    https://github.com/0day-ci/linux/commits/Jon-Derrick/Clean-up-VMD-DMA-Map-Ops/20200103-175834
base:   https://git.kernel.org/pub/scm/linux/kernel/git/helgaas/pci.git next
config: x86_64-randconfig-f001-20200103 (attached as .config)
compiler: gcc-7 (Debian 7.5.0-3) 7.5.0
reproduce:
        # save the attached .config to linux build tree
        make ARCH=x86_64 

If you fix the issue, kindly add following tag

All warnings (new ones prefixed by >>):

In file included from arch/x86/include/asm/percpu.h:45:0,

                    from arch/x86/include/asm/current.h:6,
                    from include/linux/sched.h:12,
                    from arch/x86/pci/common.c:8:
   arch/x86/pci/common.c: In function 'set_dev_domain_options':
   arch/x86/pci/common.c:670:35: error: 'struct pci_sysdata' has no member named 'vmd_dev'
      pdev->dma_parent = to_pci_dev(sd->vmd_dev);
                                      ^
   include/linux/kernel.h:995:26: note: in definition of macro 'container_of'
     void *__mptr = (void *)(ptr);     \
                             ^~~
   arch/x86/pci/common.c:670:22: note: in expansion of macro 'to_pci_dev'
      pdev->dma_parent = to_pci_dev(sd->vmd_dev);
                         ^~~~~~~~~~
   In file included from arch/x86/include/asm/current.h:5:0,
                    from include/linux/sched.h:12,
                    from arch/x86/pci/common.c:8:
   arch/x86/pci/common.c:670:35: error: 'struct pci_sysdata' has no member named 'vmd_dev'
      pdev->dma_parent = to_pci_dev(sd->vmd_dev);
                                      ^
   include/linux/compiler.h:58:52: note: in definition of macro '__trace_if_var'
    #define __trace_if_var(cond) (__builtin_constant_p(cond) ? (cond) : __trace_if_value(cond))
                                                       ^~~~
>> include/linux/compiler.h:330:3: note: in expansion of macro 'if'
      if (!(condition))     \
      ^~
   include/linux/compiler.h:338:2: note: in expansion of macro '__compiletime_assert'
     __compiletime_assert(condition, msg, prefix, suffix)
     ^~~~~~~~~~~~~~~~~~~~
   include/linux/compiler.h:350:2: note: in expansion of macro '_compiletime_assert'
     _compiletime_assert(condition, msg, __compiletime_assert_, __LINE__)
     ^~~~~~~~~~~~~~~~~~~
   include/linux/build_bug.h:39:37: note: in expansion of macro 'compiletime_assert'
    #define BUILD_BUG_ON_MSG(cond, msg) compiletime_assert(!(cond), msg)
                                        ^~~~~~~~~~~~~~~~~~
   include/linux/kernel.h:996:2: note: in expansion of macro 'BUILD_BUG_ON_MSG'
     BUILD_BUG_ON_MSG(!__same_type(*(ptr), ((type *)0)->member) && \
     ^~~~~~~~~~~~~~~~
   include/linux/kernel.h:996:20: note: in expansion of macro '__same_type'
     BUILD_BUG_ON_MSG(!__same_type(*(ptr), ((type *)0)->member) && \
                       ^~~~~~~~~~~
   include/linux/pci.h:486:23: note: in expansion of macro 'container_of'
    #define to_pci_dev(n) container_of(n, struct pci_dev, dev)
                          ^~~~~~~~~~~~
   arch/x86/pci/common.c:670:22: note: in expansion of macro 'to_pci_dev'
      pdev->dma_parent = to_pci_dev(sd->vmd_dev);
                         ^~~~~~~~~~
   arch/x86/pci/common.c:670:35: error: 'struct pci_sysdata' has no member named 'vmd_dev'
      pdev->dma_parent = to_pci_dev(sd->vmd_dev);
                                      ^
   include/linux/compiler.h:58:52: note: in definition of macro '__trace_if_var'
    #define __trace_if_var(cond) (__builtin_constant_p(cond) ? (cond) : __trace_if_value(cond))
                                                       ^~~~
>> include/linux/compiler.h:330:3: note: in expansion of macro 'if'
      if (!(condition))     \
      ^~
   include/linux/compiler.h:338:2: note: in expansion of macro '__compiletime_assert'
     __compiletime_assert(condition, msg, prefix, suffix)
     ^~~~~~~~~~~~~~~~~~~~
   include/linux/compiler.h:350:2: note: in expansion of macro '_compiletime_assert'
     _compiletime_assert(condition, msg, __compiletime_assert_, __LINE__)
     ^~~~~~~~~~~~~~~~~~~
   include/linux/build_bug.h:39:37: note: in expansion of macro 'compiletime_assert'
    #define BUILD_BUG_ON_MSG(cond, msg) compiletime_assert(!(cond), msg)
                                        ^~~~~~~~~~~~~~~~~~
   include/linux/kernel.h:996:2: note: in expansion of macro 'BUILD_BUG_ON_MSG'
     BUILD_BUG_ON_MSG(!__same_type(*(ptr), ((type *)0)->member) && \
     ^~~~~~~~~~~~~~~~
   include/linux/kernel.h:997:6: note: in expansion of macro '__same_type'
        !__same_type(*(ptr), void),   \
         ^~~~~~~~~~~
   include/linux/pci.h:486:23: note: in expansion of macro 'container_of'
    #define to_pci_dev(n) container_of(n, struct pci_dev, dev)
                          ^~~~~~~~~~~~
   arch/x86/pci/common.c:670:22: note: in expansion of macro 'to_pci_dev'
      pdev->dma_parent = to_pci_dev(sd->vmd_dev);
                         ^~~~~~~~~~
   arch/x86/pci/common.c:670:35: error: 'struct pci_sysdata' has no member named 'vmd_dev'
      pdev->dma_parent = to_pci_dev(sd->vmd_dev);
                                      ^
   include/linux/compiler.h:58:61: note: in definition of macro '__trace_if_var'
    #define __trace_if_var(cond) (__builtin_constant_p(cond) ? (cond) : __trace_if_value(cond))
                                                                ^~~~
>> include/linux/compiler.h:330:3: note: in expansion of macro 'if'
      if (!(condition))     \
      ^~
   include/linux/compiler.h:338:2: note: in expansion of macro '__compiletime_assert'
     __compiletime_assert(condition, msg, prefix, suffix)
     ^~~~~~~~~~~~~~~~~~~~
   include/linux/compiler.h:350:2: note: in expansion of macro '_compiletime_assert'
     _compiletime_assert(condition, msg, __compiletime_assert_, __LINE__)
     ^~~~~~~~~~~~~~~~~~~
   include/linux/build_bug.h:39:37: note: in expansion of macro 'compiletime_assert'
    #define BUILD_BUG_ON_MSG(cond, msg) compiletime_assert(!(cond), msg)
                                        ^~~~~~~~~~~~~~~~~~
   include/linux/kernel.h:996:2: note: in expansion of macro 'BUILD_BUG_ON_MSG'
     BUILD_BUG_ON_MSG(!__same_type(*(ptr), ((type *)0)->member) && \
     ^~~~~~~~~~~~~~~~
   include/linux/kernel.h:996:20: note: in expansion of macro '__same_type'
     BUILD_BUG_ON_MSG(!__same_type(*(ptr), ((type *)0)->member) && \
                       ^~~~~~~~~~~
   include/linux/pci.h:486:23: note: in expansion of macro 'container_of'
    #define to_pci_dev(n) container_of(n, struct pci_dev, dev)
                          ^~~~~~~~~~~~
   arch/x86/pci/common.c:670:22: note: in expansion of macro 'to_pci_dev'
      pdev->dma_parent = to_pci_dev(sd->vmd_dev);
                         ^~~~~~~~~~
   arch/x86/pci/common.c:670:35: error: 'struct pci_sysdata' has no member named 'vmd_dev'
      pdev->dma_parent = to_pci_dev(sd->vmd_dev);
                                      ^
   include/linux/compiler.h:58:61: note: in definition of macro '__trace_if_var'
    #define __trace_if_var(cond) (__builtin_constant_p(cond) ? (cond) : __trace_if_value(cond))
                                                                ^~~~
>> include/linux/compiler.h:330:3: note: in expansion of macro 'if'
      if (!(condition))     \
      ^~
   include/linux/compiler.h:338:2: note: in expansion of macro '__compiletime_assert'
     __compiletime_assert(condition, msg, prefix, suffix)
     ^~~~~~~~~~~~~~~~~~~~
   include/linux/compiler.h:350:2: note: in expansion of macro '_compiletime_assert'
     _compiletime_assert(condition, msg, __compiletime_assert_, __LINE__)
     ^~~~~~~~~~~~~~~~~~~
   include/linux/build_bug.h:39:37: note: in expansion of macro 'compiletime_assert'
    #define BUILD_BUG_ON_MSG(cond, msg) compiletime_assert(!(cond), msg)
                                        ^~~~~~~~~~~~~~~~~~
   include/linux/kernel.h:996:2: note: in expansion of macro 'BUILD_BUG_ON_MSG'
     BUILD_BUG_ON_MSG(!__same_type(*(ptr), ((type *)0)->member) && \
     ^~~~~~~~~~~~~~~~
   include/linux/kernel.h:997:6: note: in expansion of macro '__same_type'
        !__same_type(*(ptr), void),   \
         ^~~~~~~~~~~
   include/linux/pci.h:486:23: note: in expansion of macro 'container_of'
    #define to_pci_dev(n) container_of(n, struct pci_dev, dev)
                          ^~~~~~~~~~~~
   arch/x86/pci/common.c:670:22: note: in expansion of macro 'to_pci_dev'
      pdev->dma_parent = to_pci_dev(sd->vmd_dev);
                         ^~~~~~~~~~
   arch/x86/pci/common.c:670:35: error: 'struct pci_sysdata' has no member named 'vmd_dev'
      pdev->dma_parent = to_pci_dev(sd->vmd_dev);
                                      ^
   include/linux/compiler.h:69:3: note: in definition of macro '__trace_if_value'
     (cond) ?     \
      ^~~~
   include/linux/compiler.h:56:28: note: in expansion of macro '__trace_if_var'
    #define if(cond, ...) if ( __trace_if_var( !!(cond , ## __VA_ARGS__) ) )
                               ^~~~~~~~~~~~~~
>> include/linux/compiler.h:330:3: note: in expansion of macro 'if'
      if (!(condition))     \
      ^~
   include/linux/compiler.h:338:2: note: in expansion of macro '__compiletime_assert'
     __compiletime_assert(condition, msg, prefix, suffix)
     ^~~~~~~~~~~~~~~~~~~~
   include/linux/compiler.h:350:2: note: in expansion of macro '_compiletime_assert'
     _compiletime_assert(condition, msg, __compiletime_assert_, __LINE__)
     ^~~~~~~~~~~~~~~~~~~
   include/linux/build_bug.h:39:37: note: in expansion of macro 'compiletime_assert'
    #define BUILD_BUG_ON_MSG(cond, msg) compiletime_assert(!(cond), msg)
                                        ^~~~~~~~~~~~~~~~~~
   include/linux/kernel.h:996:2: note: in expansion of macro 'BUILD_BUG_ON_MSG'
     BUILD_BUG_ON_MSG(!__same_type(*(ptr), ((type *)0)->member) && \
     ^~~~~~~~~~~~~~~~
   include/linux/kernel.h:996:20: note: in expansion of macro '__same_type'
     BUILD_BUG_ON_MSG(!__same_type(*(ptr), ((type *)0)->member) && \
                       ^~~~~~~~~~~
   include/linux/pci.h:486:23: note: in expansion of macro 'container_of'
    #define to_pci_dev(n) container_of(n, struct pci_dev, dev)
                          ^~~~~~~~~~~~
   arch/x86/pci/common.c:670:22: note: in expansion of macro 'to_pci_dev'
      pdev->dma_parent = to_pci_dev(sd->vmd_dev);
                         ^~~~~~~~~~
   arch/x86/pci/common.c:670:35: error: 'struct pci_sysdata' has no member named 'vmd_dev'
      pdev->dma_parent = to_pci_dev(sd->vmd_dev);
                                      ^
   include/linux/compiler.h:69:3: note: in definition of macro '__trace_if_value'
     (cond) ?     \
      ^~~~
   include/linux/compiler.h:56:28: note: in expansion of macro '__trace_if_var'
    #define if(cond, ...) if ( __trace_if_var( !!(cond , ## __VA_ARGS__) ) )
                               ^~~~~~~~~~~~~~
>> include/linux/compiler.h:330:3: note: in expansion of macro 'if'
      if (!(condition))     \
      ^~
   include/linux/compiler.h:338:2: note: in expansion of macro '__compiletime_assert'
     __compiletime_assert(condition, msg, prefix, suffix)
     ^~~~~~~~~~~~~~~~~~~~
   include/linux/compiler.h:350:2: note: in expansion of macro '_compiletime_assert'
     _compiletime_assert(condition, msg, __compiletime_assert_, __LINE__)
     ^~~~~~~~~~~~~~~~~~~
   include/linux/build_bug.h:39:37: note: in expansion of macro 'compiletime_assert'
    #define BUILD_BUG_ON_MSG(cond, msg) compiletime_assert(!(cond), msg)
                                        ^~~~~~~~~~~~~~~~~~
   include/linux/kernel.h:996:2: note: in expansion of macro 'BUILD_BUG_ON_MSG'
     BUILD_BUG_ON_MSG(!__same_type(*(ptr), ((type *)0)->member) && \
     ^~~~~~~~~~~~~~~~
   include/linux/kernel.h:997:6: note: in expansion of macro '__same_type'
        !__same_type(*(ptr), void),   \
         ^~~~~~~~~~~
   include/linux/pci.h:486:23: note: in expansion of macro 'container_of'
    #define to_pci_dev(n) container_of(n, struct pci_dev, dev)
                          ^~~~~~~~~~~~
   arch/x86/pci/common.c:670:22: note: in expansion of macro 'to_pci_dev'
      pdev->dma_parent = to_pci_dev(sd->vmd_dev);
                         ^~~~~~~~~~

vim +/if +330 include/linux/compiler.h

c361d3e54364d1 Daniel Santos   2013-02-21  325  
c03567a8e8d5cf Joe Stringer    2017-08-31  326  #ifdef __OPTIMIZE__
9a8ab1c39970a4 Daniel Santos   2013-02-21  327  # define __compiletime_assert(condition, msg, prefix, suffix)		\
9a8ab1c39970a4 Daniel Santos   2013-02-21  328  	do {								\
9a8ab1c39970a4 Daniel Santos   2013-02-21  329  		extern void prefix ## suffix(void) __compiletime_error(msg); \
81b45683487a51 Masahiro Yamada 2018-08-26 @330  		if (!(condition))					\
9a8ab1c39970a4 Daniel Santos   2013-02-21  331  			prefix ## suffix();				\
9a8ab1c39970a4 Daniel Santos   2013-02-21  332  	} while (0)
c03567a8e8d5cf Joe Stringer    2017-08-31  333  #else
c03567a8e8d5cf Joe Stringer    2017-08-31  334  # define __compiletime_assert(condition, msg, prefix, suffix) do { } while (0)
c03567a8e8d5cf Joe Stringer    2017-08-31  335  #endif

9a8ab1c39970a4 Daniel Santos 2013-02-21 336

:::::: The code at line 330 was first introduced by commit
:::::: 81b45683487a51b0f4d3b29d37f20d6d078544e4 compiler.h: give up __compiletime_assert_fallback()

:::::: TO: Masahiro Yamada <[email protected]> :::::: CC: Kees Cook <[email protected]>

---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org Intel Corporation

#15 lore raw

Re: [RFC 4/5] PCI: vmd: Stop overriding dma_map_ops

In reply to Jon Derrick
Quoted 11 lines · Jon Derrick

Devices on the VMD domain use the VMD endpoint's requester-id and have been relying on the VMD endpoint's dma operations. The downside of this was that VMD domain devices would use the VMD endpoint's attributes when doing DMA and IOMMU mapping. We can be smarter about this by only using the VMD endpoint when mapping and providing the correct child device's attributes during dma operations.

This patch adds a new dma alias mechanism by adding a hint to a pci_dev to point to a singular DMA requester's pci_dev. This integrates into the existing dma alias infrastructure to reduce the impact of the changes required to support this mode.

If we want to lift this check into common code I think it should go into struct device, as that is what DMA operates on normally. That being said given that this insane hack only exists for braindamage in Intel hardware I'd rather keep it as isolated as possible.

#16 lore raw

Re: [RFC 4/5] PCI: vmd: Stop overriding dma_map_ops

Quoted 8 lines · Christoph Hellwig
Quoted 7 lines · Jon Derrick

Devices on the VMD domain use the VMD endpoint's requester-id and have been relying on the VMD endpoint's dma operations. The downside of this was that VMD domain devices would use the VMD endpoint's attributes when doing DMA and IOMMU mapping. We can be smarter about this by only using the VMD endpoint when mapping and providing the correct child device's attributes during dma operations.

Quoted 7 lines
Quoted 4 lines

This patch adds a new dma alias mechanism by adding a hint to a pci_dev to point to a singular DMA requester's pci_dev. This integrates into the existing dma alias infrastructure to reduce the impact of the changes required to support this mode.

If we want to lift this check into common code I think it should go into struct device, as that is what DMA operates on normally.

I thought about that too, but the dma alias mechanism was in pci_dev. I can prepare a new version with struct device.

Quoted 3 lines

That
being said given that this insane hack only exists for braindamage in Intel hardware I'd rather keep it as isolated as possible.

jmho but the footprint of the new set is pretty minimal and removes a lot of dubious code in vmd.c.

#17 lore raw

RFC 5/5 x86/PCI: Remove unused X86_DEV_DMA_OPS

In reply to Jon Derrick

VMD was the only user of device dma operations. Now that the IOMMU has been made aware of direct DMA aliases, VMD domain devices can reference the VMD endpoint directly and the VMD device dma operations has been made obsolete.

---
 arch/x86/Kconfig              |  3 ---
 arch/x86/include/asm/device.h | 10 ----------
 arch/x86/pci/common.c         | 38 --------------------------------------
 3 files changed, 51 deletions(-)
arch/x86/Kconfig +0 -3
index 5e89499..77f9426 100644
@@ -2955,9 +2955,6 @@ config HAVE_ATOMIC_IOMAP
 	def_bool y
 	depends on X86_32
 
-config X86_DEV_DMA_OPS
-	bool
-
 source "drivers/firmware/Kconfig"
 
 source "arch/x86/kvm/Kconfig"
arch/x86/include/asm/device.h +0 -10
index 5e12c63..7e31f7f 100644
@@ -8,16 +8,6 @@ struct dev_archdata {
 #endif
 };
 
-#if defined(CONFIG_X86_DEV_DMA_OPS) && defined(CONFIG_PCI_DOMAINS)
-struct dma_domain {
-	struct list_head node;
-	const struct dma_map_ops *dma_ops;
-	int domain_nr;
-};
-void add_dma_domain(struct dma_domain *domain);
-void del_dma_domain(struct dma_domain *domain);
-#endif
-
 struct pdev_archdata {
 };
 
arch/x86/pci/common.c +0 -38
index 4022609..fcf03da 100644
@@ -625,43 +625,6 @@ unsigned int pcibios_assign_all_busses(void)
 	return (pci_probe & PCI_ASSIGN_ALL_BUSSES) ? 1 : 0;
 }
 
-#if defined(CONFIG_X86_DEV_DMA_OPS) && defined(CONFIG_PCI_DOMAINS)
-static LIST_HEAD(dma_domain_list);
-static DEFINE_SPINLOCK(dma_domain_list_lock);
-
-void add_dma_domain(struct dma_domain *domain)
-{
-	spin_lock(&dma_domain_list_lock);
-	list_add(&domain->node, &dma_domain_list);
-	spin_unlock(&dma_domain_list_lock);
-}
-EXPORT_SYMBOL_GPL(add_dma_domain);
-
-void del_dma_domain(struct dma_domain *domain)
-{
-	spin_lock(&dma_domain_list_lock);
-	list_del(&domain->node);
-	spin_unlock(&dma_domain_list_lock);
-}
-EXPORT_SYMBOL_GPL(del_dma_domain);
-
-static void set_dma_domain_ops(struct pci_dev *pdev)
-{
-	struct dma_domain *domain;
-
-	spin_lock(&dma_domain_list_lock);
-	list_for_each_entry(domain, &dma_domain_list, node) {
-		if (pci_domain_nr(pdev->bus) == domain->domain_nr) {
-			pdev->dev.dma_ops = domain->dma_ops;
-			break;
-		}
-	}
-	spin_unlock(&dma_domain_list_lock);
-}
-#else
-static void set_dma_domain_ops(struct pci_dev *pdev) {}
-#endif
-
 static void set_dev_domain_options(struct pci_dev *pdev)
 {
 	if (is_vmd(pdev->bus)) {
@@ -701,7 +664,6 @@ int pcibios_add_device(struct pci_dev *dev)
 		pa_data = data->next;
 		memunmap(data);
 	}
-	set_dma_domain_ops(dev);
 	set_dev_domain_options(dev);
 	return 0;
 }
Signature · 1 line
1.8.3.1
#18 lore raw

Re: [RFC 5/5] x86/PCI: Remove unused X86_DEV_DMA_OPS

Quoted 6 lines · Jon Derrick

VMD was the only user of device dma operations. Now that the IOMMU has been made aware of direct DMA aliases, VMD domain devices can reference the VMD endpoint directly and the VMD device dma operations has been made obsolete.

This seems to be a 1:1 copy of my patch from August?

#19 lore raw

Re: [RFC 5/5] x86/PCI: Remove unused X86_DEV_DMA_OPS

Quoted 9 lines · Christoph Hellwig
Quoted 6 lines · Jon Derrick

VMD was the only user of device dma operations. Now that the IOMMU has been made aware of direct DMA aliases, VMD domain devices can reference the VMD endpoint directly and the VMD device dma operations has been made obsolete.

This seems to be a 1:1 copy of my patch from August?

Sorry I didn't notice that. I'll give you attributions.

maintainer
#20 lore raw
In reply to Jon Derrick
Quoted 3 lines · Jon Derrick

Jon Derrick (5):
iommu: Remove device link to group on failure iommu/vt-d: Unlink device if failed to add to group

Added 'Fixes:' tags to these two and applied them for v5.5, thanks.