Skip to contentSkip to conversation

Fix coherence for VMbus and PCI pass-thru devices in Hyper-V VM

PATCH Superseded Michael Kelley 3 participants 9 messages Mar 23, 2022 – Mar 24, 2022

Subsystems (6)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

v2 cover Fix coherence for VMbus and PCI pass-thru devices in Hyper-V VM

Hyper-V VMs have VMbus synthetic devices and PCI pass-thru devices that are added dynamically via the VMbus protocol and are not represented in the ACPI DSDT. Only the top level VMbus node exists in the DSDT. As such, on ARM64 these devices don't pick up coherence information and default to not hardware coherent. This results in extra software coherence management overhead since the synthetic devices are always hardware coherent. PCI pass-thru devices are also hardware coherent in all current usage scenarios.

Fix this by propagating coherence information from the top level VMbus node in the DSDT to all VMbus synthetic devices and PCI pass-thru devices. While smaller granularity of control would be better, basing on the VMbus node in the DSDT gives as escape path if a future scenario arises with devices that are not hardware coherent.

Robin Murphy -- I'm not ignoring your feedback about pci_dma_configure(), but I wanted to try this alternate approach where pci_dma_configure() works as is. If reviewers prefer modifying pci_dma_configure() to handle the Hyper-V specifics, I can go back to that.

Changes since v1:
* Use device_get_dma_attr() instead of acpi_get_dma_attr(), eliminating the need to export acpi_get_dma_attr() [Robin Murphy]
* Use arch_setup_dma_ops() to set device coherence [Robin Murphy]
* Move handling of missing _CCA to vmbus_acpi_add() so it is only done once
* Rework handling of PCI devices so existing code in pci_dma_configure() just works

Michael Kelley (2):
Drivers: hv: vmbus: Propagate VMbus coherence to each VMbus device
PCI: hv: Propagate coherence from VMbus device to PCI device

 drivers/hv/hv_common.c              | 11 +++++++++++
 drivers/hv/vmbus_drv.c              | 23 +++++++++++++++++++++++
 drivers/pci/controller/pci-hyperv.c |  9 +++++++++
 include/asm-generic/mshyperv.h      |  1 +
 4 files changed, 44 insertions(+)
Signature · 1 line
1.8.3.1
#1 lore raw

PATCH v2 1/2 Drivers: hv: vmbus: Propagate VMbus coherence to each VMbus device

VMbus synthetic devices are not represented in the ACPI DSDT -- only the top level VMbus device is represented. As a result, on ARM64 coherence information in the _CCA method is not specified for synthetic devices, so they default to not hardware coherent. Drivers for some of these synthetic devices have been recently updated to use the standard DMA APIs, and they are incurring extra overhead of unneeded software coherence management.

Fix this by propagating coherence information from the VMbus node in ACPI to the individual synthetic devices. There's no effect on x86/x64 where devices are always hardware coherent.

---
 drivers/hv/hv_common.c         | 11 +++++++++++
 drivers/hv/vmbus_drv.c         | 23 +++++++++++++++++++++++
 include/asm-generic/mshyperv.h |  1 +
 3 files changed, 35 insertions(+)
drivers/hv/hv_common.c +11 -0
index 181d16b..820e814 100644
@@ -20,6 +20,7 @@
 #include <linux/panic_notifier.h>
 #include <linux/ptrace.h>
 #include <linux/slab.h>
+#include <linux/dma-map-ops.h>
 #include <asm/hyperv-tlfs.h>
 #include <asm/mshyperv.h>
 
@@ -216,6 +217,16 @@ bool hv_query_ext_cap(u64 cap_query)
 }
 EXPORT_SYMBOL_GPL(hv_query_ext_cap);
 
+void hv_setup_dma_ops(struct device *dev, bool coherent)
+{
+	/*
+	 * Hyper-V does not offer a vIOMMU in the guest
+	 * VM, so pass 0/NULL for the IOMMU settings
+	 */
+	arch_setup_dma_ops(dev, 0, 0, NULL, coherent);
+}
+EXPORT_SYMBOL_GPL(hv_setup_dma_ops);
+
 bool hv_is_hibernation_supported(void)
 {
 	return !hv_root_partition && acpi_sleep_state_supported(ACPI_STATE_S4);
drivers/hv/vmbus_drv.c +23 -0
index 12a2b37..2d2c54c 100644
@@ -905,6 +905,14 @@ static int vmbus_probe(struct device *child_device)
 	struct hv_device *dev = device_to_hv_device(child_device);
 	const struct hv_vmbus_device_id *dev_id;
 
+	/*
+	 * On ARM64, propagate the DMA coherence setting from the top level
+	 * VMbus ACPI device to the child VMbus device being added here.
+	 * On x86/x64 coherence is assumed and these calls have no effect.
+	 */
+	hv_setup_dma_ops(child_device,
+		device_get_dma_attr(&hv_acpi_dev->dev) == DEV_DMA_COHERENT);
+
 	dev_id = hv_vmbus_get_id(drv, dev);
 	if (drv->probe) {
 		ret = drv->probe(dev, dev_id);
@@ -2428,6 +2436,21 @@ static int vmbus_acpi_add(struct acpi_device *device)
 
 	hv_acpi_dev = device;
 
+	/*
+	 * Older versions of Hyper-V for ARM64 fail to include the _CCA
+	 * method on the top level VMbus device in the DSDT. But devices
+	 * are hardware coherent in all current Hyper-V use cases, so fix
+	 * up the ACPI device to behave as if _CCA is present and indicates
+	 * hardware coherence.
+	 */
+	ACPI_COMPANION_SET(&device->dev, device);
+	if (IS_ENABLED(CONFIG_ACPI_CCA_REQUIRED) &&
+	    device_get_dma_attr(&device->dev) == DEV_DMA_NOT_SUPPORTED) {
+		pr_info("No ACPI _CCA found; assuming coherent device I/O\n");
+		device->flags.cca_seen = true;
+		device->flags.coherent_dma = true;
+	}
+
 	result = acpi_walk_resources(device->handle, METHOD_NAME__CRS,
 					vmbus_walk_resources, NULL);
 
include/asm-generic/mshyperv.h +1 -0
index c08758b..c05d2ce 100644
@@ -269,6 +269,7 @@ static inline int cpumask_to_vpset_noself(struct hv_vpset *vpset,
 u64 hv_ghcb_hypercall(u64 control, void *input, void *output, u32 input_size);
 void hyperv_cleanup(void);
 bool hv_query_ext_cap(u64 cap_query);
+void hv_setup_dma_ops(struct device *dev, bool coherent);
 void *hv_map_memory(void *addr, unsigned long size);
 void hv_unmap_memory(void *addr);
 #else /* CONFIG_HYPERV */
Signature · 6 lines
1.8.3.1

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Acked-by on v2 1/2
#2 lore raw

Re: [PATCH v2 1/2] Drivers: hv: vmbus: Propagate VMbus coherence to each VMbus device

Quoted 53 lines · Michael Kelley

VMbus synthetic devices are not represented in the ACPI DSDT -- only the top level VMbus device is represented. As a result, on ARM64 coherence information in the _CCA method is not specified for synthetic devices, so they default to not hardware coherent. Drivers for some of these synthetic devices have been recently updated to use the standard DMA APIs, and they are incurring extra overhead of unneeded software coherence management.

Fix this by propagating coherence information from the VMbus node in ACPI to the individual synthetic devices. There's no effect on x86/x64 where devices are always hardware coherent.

---
  drivers/hv/hv_common.c         | 11 +++++++++++
  drivers/hv/vmbus_drv.c         | 23 +++++++++++++++++++++++
  include/asm-generic/mshyperv.h |  1 +
  3 files changed, 35 insertions(+)
drivers/hv/hv_common.c +11 -0
index 181d16b..820e814 100644
@@ -20,6 +20,7 @@
  #include <linux/panic_notifier.h>
  #include <linux/ptrace.h>
  #include <linux/slab.h>
+#include <linux/dma-map-ops.h>
  #include <asm/hyperv-tlfs.h>
  #include <asm/mshyperv.h>
 
@@ -216,6 +217,16 @@ bool hv_query_ext_cap(u64 cap_query)
  }
  EXPORT_SYMBOL_GPL(hv_query_ext_cap);
 
+void hv_setup_dma_ops(struct device *dev, bool coherent)
+{
+	/*
+	 * Hyper-V does not offer a vIOMMU in the guest
+	 * VM, so pass 0/NULL for the IOMMU settings
+	 */
+	arch_setup_dma_ops(dev, 0, 0, NULL, coherent);
+}
+EXPORT_SYMBOL_GPL(hv_setup_dma_ops);
+
  bool hv_is_hibernation_supported(void)
  {
  	return !hv_root_partition && acpi_sleep_state_supported(ACPI_STATE_S4);
drivers/hv/vmbus_drv.c +0 -0
index 12a2b37..2d2c54c 100644

drivers/hv/vmbus_drv.c static int vmbus_probe(struct device *child_device) · line 914

Quoted 10 lines
drivers/hv/vmbus_drv.c +7 -0
@@ -905,6 +905,14 @@ static int vmbus_probe(struct device *child_device)
  	struct hv_device *dev = device_to_hv_device(child_device);
  	const struct hv_vmbus_device_id *dev_id;
 
+	/*
+	 * On ARM64, propagate the DMA coherence setting from the top level
+	 * VMbus ACPI device to the child VMbus device being added here.
+	 * On x86/x64 coherence is assumed and these calls have no effect.
+	 */
+	hv_setup_dma_ops(child_device,
+		device_get_dma_attr(&hv_acpi_dev->dev) == DEV_DMA_COHERENT);

Would you mind hooking up the hv_bus.dma_configure method to do this? Feel free to fold hv_setup_dma_ops entirely into that if you're not likely to need to call it from anywhere else.

Quoted 12 lines
drivers/hv/vmbus_drv.c +5 -0
+
  	dev_id = hv_vmbus_get_id(drv, dev);
  	if (drv->probe) {
  		ret = drv->probe(dev, dev_id);
@@ -2428,6 +2436,21 @@ static int vmbus_acpi_add(struct acpi_device *device)
 
  	hv_acpi_dev = device;
 
+	/*
+	 * Older versions of Hyper-V for ARM64 fail to include the _CCA
+	 * method on the top level VMbus device in the DSDT. But devices
+	 * are hardware coherent in all current Hyper-V use cases, so fix

drivers/hv/vmbus_drv.c static int vmbus_acpi_add(struct acpi_device *device) · line 2452

Quoted 10 lines
drivers/hv/vmbus_drv.c +10 -0
+	 * up the ACPI device to behave as if _CCA is present and indicates
+	 * hardware coherence.
+	 */
+	ACPI_COMPANION_SET(&device->dev, device);
+	if (IS_ENABLED(CONFIG_ACPI_CCA_REQUIRED) &&
+	    device_get_dma_attr(&device->dev) == DEV_DMA_NOT_SUPPORTED) {
+		pr_info("No ACPI _CCA found; assuming coherent device I/O\n");
+		device->flags.cca_seen = true;
+		device->flags.coherent_dma = true;
+	}

I'm not the biggest fan of this, especially since I'm not convinced that there are any out-of-support deployments of ARM64 Hyper-V that can't be updated. However I suppose it's not "real" firmware, and one Hyper-V component is at liberty to hack another Hyper-V component's data if it really wants to...

If you can hook up .dma_configure, or clarify if it wouldn't work,

Cheers,
Robin.

Quoted 16 lines
drivers/hv/vmbus_drv.c +1 -0
+
  	result = acpi_walk_resources(device->handle, METHOD_NAME__CRS,
  					vmbus_walk_resources, NULL);
 
include/asm-generic/mshyperv.h +1 -0
index c08758b..c05d2ce 100644
@@ -269,6 +269,7 @@ static inline int cpumask_to_vpset_noself(struct hv_vpset *vpset,
  u64 hv_ghcb_hypercall(u64 control, void *input, void *output, u32 input_size);
  void hyperv_cleanup(void);
  bool hv_query_ext_cap(u64 cap_query);
+void hv_setup_dma_ops(struct device *dev, bool coherent);
  void *hv_map_memory(void *addr, unsigned long size);
  void hv_unmap_memory(void *addr);
  #else /* CONFIG_HYPERV */
#3 lore raw

RE: [PATCH v2 1/2] Drivers: hv: vmbus: Propagate VMbus coherence to each VMbus device

From: Robin Murphy <[email protected]> Sent: Thursday, March 24, 2022 4:59 AM

Quoted 58 lines · Michael Kelley
Quoted 56 lines

VMbus synthetic devices are not represented in the ACPI DSDT -- only the top level VMbus device is represented. As a result, on ARM64 coherence information in the _CCA method is not specified for synthetic devices, so they default to not hardware coherent. Drivers for some of these synthetic devices have been recently updated to use the standard DMA APIs, and they are incurring extra overhead of unneeded software coherence management.

Fix this by propagating coherence information from the VMbus node in ACPI to the individual synthetic devices. There's no effect on x86/x64 where devices are always hardware coherent.

---
  drivers/hv/hv_common.c         | 11 +++++++++++
  drivers/hv/vmbus_drv.c         | 23 +++++++++++++++++++++++
  include/asm-generic/mshyperv.h |  1 +
  3 files changed, 35 insertions(+)
drivers/hv/hv_common.c +11 -0
index 181d16b..820e814 100644
@@ -20,6 +20,7 @@
  #include <linux/panic_notifier.h>
  #include <linux/ptrace.h>
  #include <linux/slab.h>
+#include <linux/dma-map-ops.h>
  #include <asm/hyperv-tlfs.h>
  #include <asm/mshyperv.h>
 
@@ -216,6 +217,16 @@ bool hv_query_ext_cap(u64 cap_query)
  }
  EXPORT_SYMBOL_GPL(hv_query_ext_cap);
 
+void hv_setup_dma_ops(struct device *dev, bool coherent)
+{
+	/*
+	 * Hyper-V does not offer a vIOMMU in the guest
+	 * VM, so pass 0/NULL for the IOMMU settings
+	 */
+	arch_setup_dma_ops(dev, 0, 0, NULL, coherent);
+}
+EXPORT_SYMBOL_GPL(hv_setup_dma_ops);
+
  bool hv_is_hibernation_supported(void)
  {
  	return !hv_root_partition && acpi_sleep_state_supported(ACPI_STATE_S4);
drivers/hv/vmbus_drv.c +0 -0
index 12a2b37..2d2c54c 100644
@@ -905,6 +905,14 @@ static int vmbus_probe(struct device *child_device)
  	struct hv_device *dev = device_to_hv_device(child_device);
  	const struct hv_vmbus_device_id *dev_id;
 
Quoted 11 lines

drivers/hv/vmbus_drv.c static int vmbus_probe(struct device *child_device) · line 914

Quoted 7 lines
drivers/hv/vmbus_drv.c +7 -0
+	/*
+	 * On ARM64, propagate the DMA coherence setting from the top level
+	 * VMbus ACPI device to the child VMbus device being added here.
+	 * On x86/x64 coherence is assumed and these calls have no effect.
+	 */
+	hv_setup_dma_ops(child_device,
+		device_get_dma_attr(&hv_acpi_dev->dev) == DEV_DMA_COHERENT);

Would you mind hooking up the hv_bus.dma_configure method to do this? Feel free to fold hv_setup_dma_ops entirely into that if you're not likely to need to call it from anywhere else.

I'm pretty sure using hv_bus.dma_configure() is doable. A separate hv_setup_dma_ops() is still needed because arch_setup_dma_ops() isn't exported and this VMbus driver can be built as a module.

Quoted 21 lines
Quoted 21 lines
drivers/hv/vmbus_drv.c +13 -0
+
  	dev_id = hv_vmbus_get_id(drv, dev);
  	if (drv->probe) {
  		ret = drv->probe(dev, dev_id);
@@ -2428,6 +2436,21 @@ static int vmbus_acpi_add(struct acpi_device *device)
 
  	hv_acpi_dev = device;
 
+	/*
+	 * Older versions of Hyper-V for ARM64 fail to include the _CCA
+	 * method on the top level VMbus device in the DSDT. But devices
+	 * are hardware coherent in all current Hyper-V use cases, so fix
+	 * up the ACPI device to behave as if _CCA is present and indicates
+	 * hardware coherence.
+	 */
+	ACPI_COMPANION_SET(&device->dev, device);
+	if (IS_ENABLED(CONFIG_ACPI_CCA_REQUIRED) &&
+	    device_get_dma_attr(&device->dev) == DEV_DMA_NOT_SUPPORTED) {
+		pr_info("No ACPI _CCA found; assuming coherent device I/O\n");
+		device->flags.cca_seen = true;
Quoted 8 lines

drivers/hv/vmbus_drv.c static int vmbus_acpi_add(struct acpi_device *device) · line 2452

Quoted 2 lines
drivers/hv/vmbus_drv.c +2 -0
+		device->flags.coherent_dma = true;
+	}

I'm not the biggest fan of this, especially since I'm not convinced that there are any out-of-support deployments of ARM64 Hyper-V that can't be updated. However I suppose it's not "real" firmware, and one Hyper-V component is at liberty to hack another Hyper-V component's data if it really wants to...

Agreed, it's a hack. But Hyper-V instances are out there as part of Windows 10/11 on ARM64 PCs, and they run ARM64 VMs for the Windows Subsystem for Linux. Microsoft gets pilloried for breaking stuff, and this removes the potential for that happening if someone runs a new Linux kernel version in that VM.

Michael

Quoted 25 lines

If you can hook up .dma_configure, or clarify if it wouldn't work,

Cheers,
Robin.

include/asm-generic/mshyperv.h static inline int cpumask_to_vpset_noself(struct hv_vpset

Quoted 9 lines
drivers/hv/vmbus_drv.c +1 -0
+
  	result = acpi_walk_resources(device->handle, METHOD_NAME__CRS,
  					vmbus_walk_resources, NULL);
 
include/asm-generic/mshyperv.h +0 -0
index c08758b..c05d2ce 100644
@@ -269,6 +269,7 @@ static inline int cpumask_to_vpset_noself(struct hv_vpset

*vpset,

Quoted 7 lines
include/asm-generic/mshyperv.h +1 -0
  u64 hv_ghcb_hypercall(u64 control, void *input, void *output, u32 input_size);
  void hyperv_cleanup(void);
  bool hv_query_ext_cap(u64 cap_query);
+void hv_setup_dma_ops(struct device *dev, bool coherent);
  void *hv_map_memory(void *addr, unsigned long size);
  void hv_unmap_memory(void *addr);
  #else /* CONFIG_HYPERV */
#4 lore raw

Re: [PATCH v2 1/2] Drivers: hv: vmbus: Propagate VMbus coherence to each VMbus device

Quoted 71 lines · Michael Kelley (LINUX)

From: Robin Murphy <[email protected]> Sent: Thursday, March 24, 2022 4:59 AM

Quoted 58 lines · Michael Kelley
Quoted 56 lines

VMbus synthetic devices are not represented in the ACPI DSDT -- only the top level VMbus device is represented. As a result, on ARM64 coherence information in the _CCA method is not specified for synthetic devices, so they default to not hardware coherent. Drivers for some of these synthetic devices have been recently updated to use the standard DMA APIs, and they are incurring extra overhead of unneeded software coherence management.

Fix this by propagating coherence information from the VMbus node in ACPI to the individual synthetic devices. There's no effect on x86/x64 where devices are always hardware coherent.

---
   drivers/hv/hv_common.c         | 11 +++++++++++
   drivers/hv/vmbus_drv.c         | 23 +++++++++++++++++++++++
   include/asm-generic/mshyperv.h |  1 +
   3 files changed, 35 insertions(+)
drivers/hv/hv_common.c +11 -0
index 181d16b..820e814 100644
@@ -20,6 +20,7 @@
   #include <linux/panic_notifier.h>
   #include <linux/ptrace.h>
   #include <linux/slab.h>
+#include <linux/dma-map-ops.h>
   #include <asm/hyperv-tlfs.h>
   #include <asm/mshyperv.h>
 
@@ -216,6 +217,16 @@ bool hv_query_ext_cap(u64 cap_query)
   }
   EXPORT_SYMBOL_GPL(hv_query_ext_cap);
 
+void hv_setup_dma_ops(struct device *dev, bool coherent)
+{
+	/*
+	 * Hyper-V does not offer a vIOMMU in the guest
+	 * VM, so pass 0/NULL for the IOMMU settings
+	 */
+	arch_setup_dma_ops(dev, 0, 0, NULL, coherent);
+}
+EXPORT_SYMBOL_GPL(hv_setup_dma_ops);
+
   bool hv_is_hibernation_supported(void)
   {
   	return !hv_root_partition && acpi_sleep_state_supported(ACPI_STATE_S4);
drivers/hv/vmbus_drv.c +0 -0
index 12a2b37..2d2c54c 100644
@@ -905,6 +905,14 @@ static int vmbus_probe(struct device *child_device)
   	struct hv_device *dev = device_to_hv_device(child_device);
   	const struct hv_vmbus_device_id *dev_id;
 
Quoted 11 lines

drivers/hv/vmbus_drv.c static int vmbus_probe(struct device *child_device) · line 914

Quoted 7 lines
drivers/hv/vmbus_drv.c +7 -0
+	/*
+	 * On ARM64, propagate the DMA coherence setting from the top level
+	 * VMbus ACPI device to the child VMbus device being added here.
+	 * On x86/x64 coherence is assumed and these calls have no effect.
+	 */
+	hv_setup_dma_ops(child_device,
+		device_get_dma_attr(&hv_acpi_dev->dev) == DEV_DMA_COHERENT);

Would you mind hooking up the hv_bus.dma_configure method to do this? Feel free to fold hv_setup_dma_ops entirely into that if you're not likely to need to call it from anywhere else.

Quoted 3 lines

I'm pretty sure using hv_bus.dma_configure() is doable. A separate hv_setup_dma_ops() is still needed because arch_setup_dma_ops() isn't exported and this VMbus driver can be built as a module.

Ah, right you are, I keep forgetting that.

Quoted 29 lines
Quoted 20 lines
Quoted 20 lines
drivers/hv/vmbus_drv.c +13 -0
+
   	dev_id = hv_vmbus_get_id(drv, dev);
   	if (drv->probe) {
   		ret = drv->probe(dev, dev_id);
@@ -2428,6 +2436,21 @@ static int vmbus_acpi_add(struct acpi_device *device)
 
   	hv_acpi_dev = device;
 
+	/*
+	 * Older versions of Hyper-V for ARM64 fail to include the _CCA
+	 * method on the top level VMbus device in the DSDT. But devices
+	 * are hardware coherent in all current Hyper-V use cases, so fix
+	 * up the ACPI device to behave as if _CCA is present and indicates
+	 * hardware coherence.
+	 */
+	ACPI_COMPANION_SET(&device->dev, device);
+	if (IS_ENABLED(CONFIG_ACPI_CCA_REQUIRED) &&
+	    device_get_dma_attr(&device->dev) == DEV_DMA_NOT_SUPPORTED) {
+		pr_info("No ACPI _CCA found; assuming coherent device I/O\n");
+		device->flags.cca_seen = true;
Quoted 8 lines

drivers/hv/vmbus_drv.c static int vmbus_acpi_add(struct acpi_device *device) · line 2452

Quoted 2 lines
drivers/hv/vmbus_drv.c +2 -0
+		device->flags.coherent_dma = true;
+	}

I'm not the biggest fan of this, especially since I'm not convinced that there are any out-of-support deployments of ARM64 Hyper-V that can't be updated. However I suppose it's not "real" firmware, and one Hyper-V component is at liberty to hack another Hyper-V component's data if it really wants to...

Quoted 5 lines

Agreed, it's a hack. But Hyper-V instances are out there as part of Windows 10/11 on ARM64 PCs, and they run ARM64 VMs for the Windows Subsystem for Linux. Microsoft gets pilloried for breaking stuff, and this removes the potential for that happening if someone runs a new Linux kernel version in that VM.

And actually that one's on me as well - for some reason I was thinking that this had never worked, and therefore you could likely get a Hyper-V update pushed out long before users get this patch through distros, but of course it only becomes an issue now because previously there was no connection to any ACPI node at all. As I said, personally I'm happy to consider this a Hyper-V internal workaround, but if anyone else objects to poking at the ACPI flags, I suppose you've also got the fallback option of flipping it around and making the ACPI_COMPANION_SET() conditional, so that the behaviour for older versions remains entirely unchanged. If it happens, feel free to keep my ack for that approach too.

Cheers,
Robin.

Quoted 28 lines

Michael

Quoted 25 lines

If you can hook up .dma_configure, or clarify if it wouldn't work,

Cheers,
Robin.

include/asm-generic/mshyperv.h static inline int cpumask_to_vpset_noself(struct hv_vpset

Quoted 9 lines
drivers/hv/vmbus_drv.c +1 -0
+
   	result = acpi_walk_resources(device->handle, METHOD_NAME__CRS,
   					vmbus_walk_resources, NULL);
 
include/asm-generic/mshyperv.h +0 -0
index c08758b..c05d2ce 100644
@@ -269,6 +269,7 @@ static inline int cpumask_to_vpset_noself(struct hv_vpset

*vpset,

Quoted 7 lines
include/asm-generic/mshyperv.h +1 -0
   u64 hv_ghcb_hypercall(u64 control, void *input, void *output, u32 input_size);
   void hyperv_cleanup(void);
   bool hv_query_ext_cap(u64 cap_query);
+void hv_setup_dma_ops(struct device *dev, bool coherent);
   void *hv_map_memory(void *addr, unsigned long size);
   void hv_unmap_memory(void *addr);
   #else /* CONFIG_HYPERV */

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu

#5 lore raw

PATCH v2 2/2 PCI: hv: Propagate coherence from VMbus device to PCI device

In reply to Michael Kelley

PCI pass-thru devices in a Hyper-V VM are represented as a VMBus device and as a PCI device. The coherence of the VMbus device is set based on the VMbus node in ACPI, but the PCI device has no ACPI node and defaults to not hardware coherent. This results in extra software coherence management overhead on ARM64 when devices are hardware coherent.

Fix this by setting up the PCI host bus so that normal PCI mechanisms will propagate the coherence of the VMbus device to the PCI device. There's no effect on x86/x64 where devices are always hardware coherent.

---
 drivers/pci/controller/pci-hyperv.c | 9 +++++++++
 1 file changed, 9 insertions(+)
drivers/pci/controller/pci-hyperv.c +9 -0
index ae0bc2f..88b3b56 100644
@@ -3404,6 +3404,15 @@ static int hv_pci_probe(struct hv_device *hdev,
 	hbus->bridge->domain_nr = dom;
 #ifdef CONFIG_X86
 	hbus->sysdata.domain = dom;
+#elif defined(CONFIG_ARM64)
+	/*
+	 * Set the PCI bus parent to be the corresponding VMbus
+	 * device. Then the VMbus device will be assigned as the
+	 * ACPI companion in pcibios_root_bridge_prepare() and
+	 * pci_dma_configure() will propagate device coherence
+	 * information to devices created on the bus.
+	 */
+	hbus->sysdata.parent = hdev->device.parent;
 #endif
 
 	hbus->hdev = hdev;
Signature · 6 lines
1.8.3.1

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Acked-by on v2 2/2
#6 lore raw

Re: [PATCH v2 2/2] PCI: hv: Propagate coherence from VMbus device to PCI device

Quoted 7 lines · Michael Kelley

PCI pass-thru devices in a Hyper-V VM are represented as a VMBus device and as a PCI device. The coherence of the VMbus device is set based on the VMbus node in ACPI, but the PCI device has no ACPI node and defaults to not hardware coherent. This results in extra software coherence management overhead on ARM64 when devices are hardware coherent.

Quoted 6 lines

Fix this by setting up the PCI host bus so that normal PCI mechanisms will propagate the coherence of the VMbus device to the PCI device. There's no effect on x86/x64 where devices are always hardware coherent.

Regards,
Boqun

Quoted 28 lines
---
 drivers/pci/controller/pci-hyperv.c | 9 +++++++++
 1 file changed, 9 insertions(+)
drivers/pci/controller/pci-hyperv.c +9 -0
index ae0bc2f..88b3b56 100644
@@ -3404,6 +3404,15 @@ static int hv_pci_probe(struct hv_device *hdev,
 	hbus->bridge->domain_nr = dom;
 #ifdef CONFIG_X86
 	hbus->sysdata.domain = dom;
+#elif defined(CONFIG_ARM64)
+	/*
+	 * Set the PCI bus parent to be the corresponding VMbus
+	 * device. Then the VMbus device will be assigned as the
+	 * ACPI companion in pcibios_root_bridge_prepare() and
+	 * pci_dma_configure() will propagate device coherence
+	 * information to devices created on the bus.
+	 */
+	hbus->sysdata.parent = hdev->device.parent;
 #endif
 
 	hbus->hdev = hdev;
Signature · 1 line
1.8.3.1
Acked-by on v2 2/2
#7 lore raw

Re: [PATCH v2 2/2] PCI: hv: Propagate coherence from VMbus device to PCI device

In reply to Michael Kelley
Quoted 11 lines · Michael Kelley

PCI pass-thru devices in a Hyper-V VM are represented as a VMBus device and as a PCI device. The coherence of the VMbus device is set based on the VMbus node in ACPI, but the PCI device has no ACPI node and defaults to not hardware coherent. This results in extra software coherence management overhead on ARM64 when devices are hardware coherent.

Fix this by setting up the PCI host bus so that normal PCI mechanisms will propagate the coherence of the VMbus device to the PCI device. There's no effect on x86/x64 where devices are always hardware coherent.

Honestly, I don't hate this :)

It seems conceptually accurate, as far as I understand, and in functional terms I'm starting to think it might even be the most correct approach anyway. In the physical world we might be surprised to find the PCI side of a host bridge behind anything other than some platform/ACPI device representing the other side of a physical host bridge or root complex, but who's to say that a paravirtual world can't present a more abstract topology? Either way, a one-line way of tying in to the standard flow is hard to turn down.

Quoted 25 lines
---
  drivers/pci/controller/pci-hyperv.c | 9 +++++++++
  1 file changed, 9 insertions(+)
drivers/pci/controller/pci-hyperv.c +9 -0
index ae0bc2f..88b3b56 100644
@@ -3404,6 +3404,15 @@ static int hv_pci_probe(struct hv_device *hdev,
  	hbus->bridge->domain_nr = dom;
  #ifdef CONFIG_X86
  	hbus->sysdata.domain = dom;
+#elif defined(CONFIG_ARM64)
+	/*
+	 * Set the PCI bus parent to be the corresponding VMbus
+	 * device. Then the VMbus device will be assigned as the
+	 * ACPI companion in pcibios_root_bridge_prepare() and
+	 * pci_dma_configure() will propagate device coherence
+	 * information to devices created on the bus.
+	 */
+	hbus->sysdata.parent = hdev->device.parent;
  #endif
 
  	hbus->hdev = hdev;
#8 lore raw

Re: [PATCH v2 2/2] PCI: hv: Propagate coherence from VMbus device to PCI device

Quoted 8 lines · Robin Murphy
Quoted 7 lines · Michael Kelley

PCI pass-thru devices in a Hyper-V VM are represented as a VMBus device and as a PCI device.  The coherence of the VMbus device is set based on the VMbus node in ACPI, but the PCI device has no ACPI node and defaults to not hardware coherent.  This results in extra software coherence management overhead on ARM64 when devices are hardware coherent.

Quoted 8 lines
Quoted 4 lines

Fix this by setting up the PCI host bus so that normal PCI mechanisms will propagate the coherence of the VMbus device to the PCI device. There's no effect on x86/x64 where devices are always hardware coherent.

Honestly, I don't hate this :)

It seems conceptually accurate, as far as I understand, and in functional terms I'm starting to think it might even be the most correct approach anyway. In the physical world we might be surprised to find the PCI side of a host bridge

And of course by "the PCI side of a host bridge" I think I actually mean "a PCI root bus", because in my sloppy terminology I'm thinking about hardware bridging from PCI(e) to some SoC-internal protocol, which does not have to imply an actual PCI-visible Host Bridge device...

Robin.

Quoted 32 lines

behind anything other than some platform/ACPI device representing the other side of a physical host bridge or root complex, but who's to say that a paravirtual world can't present a more abstract topology? Either way, a one-line way of tying in to the standard flow is hard to turn down.

Quoted 14 lines

---
  drivers/pci/controller/pci-hyperv.c | 9 +++++++++

  1 file changed, 9 insertions(+)
drivers/pci/controller/pci-hyperv.c +1 -0
index ae0bc2f..88b3b56 100644
@@ -3404,6 +3404,15 @@ static int hv_pci_probe(struct hv_device *hdev,
       hbus->bridge->domain_nr = dom;
   #ifdef CONFIG_X86
       hbus->sysdata.domain = dom;
+#elif defined(CONFIG_ARM64)

drivers/pci/controller/pci-hyperv.c static int hv_pci_probe(struct hv_device *hdev, · line 3417

Quoted 10 lines
drivers/pci/controller/pci-hyperv.c +8 -0
+    /*
+     * Set the PCI bus parent to be the corresponding VMbus
+     * device. Then the VMbus device will be assigned as the
+     * ACPI companion in pcibios_root_bridge_prepare() and
+     * pci_dma_configure() will propagate device coherence
+     * information to devices created on the bus.
+     */
+    hbus->sysdata.parent = hdev->device.parent;
   #endif
       hbus->hdev = hdev;

_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu