PCI: hv: Hyper-V vPCI for arm64
Subsystems (7)
v4 cover PCI: hv: Hyper-V vPCI for arm64
From: Sunil Muthuswamy <[email protected]>
Current Hyper-V vPCI code only compiles and works for x86. There are some hardcoded assumptions about the architectural IRQ chip and other arch defines.
Add support for Hyper-V vPCI for arm64 by first breaking the current hard coded dependency using a set of new interfaces and implementing those for x86 first. That is in the first patch. The second patch adds support for Hyper-V vPCI for arm64 by implementing the above mentioned interfaces. That is done by introducing a Hyper-V vPCI specific MSI IRQ domain & chip for allocating SPI vectors.
changes in v1 -> v2:
- Moved the irqchip implementation to drivers/pci as suggested by Marc Zyngier
- Addressed Multi-MSI handling issues identified by Marc Zyngier
- Addressed lock/synchronization primitive as suggested by Marc Zyngier
- Addressed other code feedback from Marc Zyngier
changes in v2 -> v3:
- Addressed comments from Bjorn Helgaas about patch formatting and verbiage
- Using 'git send-email' to ensure that the patch series is correctly threaded. Feedback by Bjorn Helgaas
- Fixed Hyper-V vPCI build break for module build, reported by Boqun Feng
changes in v3 -> v4:
- Removed the separate file for IRQ chip that was there in previous iterations and moved the IRQ chip implementation to pci-hyperv.c. Feedback by Michael Kelley and Marc Zyngier.
- Addressed various comments from Marc Zyngier about structuring and layout.
- Addressed comment from Marc Zyngier about IRQ affinity and other miscellaneous comments.
Sunil Muthuswamy (2):
PCI: hv: Make the code arch neutral by adding arch specific interfaces
arm64: PCI: hv: Add support for Hyper-V vPCI
arch/arm64/include/asm/hyperv-tlfs.h | 9 + arch/x86/include/asm/hyperv-tlfs.h | 33 ++++ arch/x86/include/asm/mshyperv.h | 7 - drivers/pci/Kconfig | 2 +- drivers/pci/controller/Kconfig | 2 +- drivers/pci/controller/pci-hyperv.c | 284 ++++++++++++++++++++++++--- include/asm-generic/hyperv-tlfs.h | 33 ---- 7 files changed, 303 insertions(+), 67 deletions(-)
base-commit: e4e737bb5c170df6135a127739a9e6148ee3da82
Signature · 1 line
2.25.1
PATCH v4 1/2 PCI: hv: Make the code arch neutral by adding arch specific interfaces
From: Sunil Muthuswamy <[email protected]>
Encapsulate arch dependencies in Hyper-V vPCI through a set of arch-dependent interfaces. Adding these arch specific interfaces will allow for an implementation for other architectures, such as arm64.
There are no functional changes expected from this patch.
---
In v2, v3 & v4:
Changes are described in the cover letter.
arch/x86/include/asm/hyperv-tlfs.h | 33 ++++++++++++ arch/x86/include/asm/mshyperv.h | 7 --- drivers/pci/controller/pci-hyperv.c | 79 ++++++++++++++++++++--------- include/asm-generic/hyperv-tlfs.h | 33 ------------ 4 files changed, 87 insertions(+), 65 deletions(-)
arch/x86/include/asm/hyperv-tlfs.h +33 -0
@@ -585,6 +585,39 @@ enum hv_interrupt_type { HV_X64_INTERRUPT_TYPE_MAXIMUM = 0x000A, }; +union hv_msi_address_register { +u32 as_uint32; +struct { +u32 reserved1:2 ; +u32 destination_mode:1 ; +u32 redirection_hint:1 ; +u32 reserved2:8 ; +u32 destination_id:8 ; +u32 msi_base:12 ; + }; +} __packed; + +union hv_msi_data_register { +u32 as_uint32; +struct { +u32 vector:8 ; +u32 delivery_mode:3 ; +u32 reserved1:3 ; +u32 level_assert:1 ; +u32 trigger_mode:1 ; +u32 reserved2:16 ; + }; +} __packed; + +/* HvRetargetDeviceInterrupt hypercall */ +union hv_msi_entry { +u64 as_uint64; +struct { +union hv_msi_address_register address; +union hv_msi_data_register data; + } __packed; +}; +#include <asm-generic/hyperv-tlfs.h> #endif
arch/x86/include/asm/mshyperv.h +0 -7
@@ -176,13 +176,6 @@ bool hv_vcpu_is_preempted(int vcpu);static inline void hv_apic_init (void ) {}#endif -static inline void hv_set_msi_entry_from_desc (union hv_msi_entry *msi_entry, -struct msi_desc *msi_desc) -{ - msi_entry->address.as_uint32 = msi_desc->msg.address_lo; - msi_entry->data.as_uint32 = msi_desc->msg.data; -} -struct irq_domain *hv_create_pci_msi_domain (void ); int hv_map_ioapic_interrupt(int ioapic_id, bool level, int vcpu, int vector,
drivers/pci/controller/pci-hyperv.c +54 -25
181 lines not on this page · raw message
include/asm-generic/hyperv-tlfs.h +0 -33
@@ -539,39 +539,6 @@ enum hv_interrupt_source { HV_INTERRUPT_SOURCE_IOAPIC, }; -union hv_msi_address_register { -u32 as_uint32; -struct { -u32 reserved1:2 ; -u32 destination_mode:1 ; -u32 redirection_hint:1 ; -u32 reserved2:8 ; -u32 destination_id:8 ; -u32 msi_base:12 ; - }; -} __packed; - -union hv_msi_data_register { -u32 as_uint32; -struct { -u32 vector:8 ; -u32 delivery_mode:3 ; -u32 reserved1:3 ; -u32 level_assert:1 ; -u32 trigger_mode:1 ; -u32 reserved2:16 ; - }; -} __packed; - -/* HvRetargetDeviceInterrupt hypercall */ -union hv_msi_entry { -u64 as_uint64; -struct { -union hv_msi_address_register address; -union hv_msi_data_register data; - } __packed; -}; -union hv_ioapic_rte {u64 as_uint64;
Signature · 1 line
2.25.1
PATCH v4 2/2 arm64: PCI: hv: Add support for Hyper-V vPCI
In reply to Sunil MuthuswamyFrom: Sunil Muthuswamy <[email protected]>
Add support for Hyper-V vPCI for arm64 by implementing the arch specific interfaces. Introduce an IRQ domain and chip specific to Hyper-v vPCI that is based on SPIs. The IRQ domain parents itself to the arch GIC IRQ domain for basic vector management.
---
In v2, v3 & v4:
Changes are described in the cover letter.
arch/arm64/include/asm/hyperv-tlfs.h | 9 ++ drivers/pci/Kconfig | 2 +- drivers/pci/controller/Kconfig | 2 +- drivers/pci/controller/pci-hyperv.c | 207 ++++++++++++++++++++++++++- 4 files changed, 217 insertions(+), 3 deletions(-)
arch/arm64/include/asm/hyperv-tlfs.h +9 -0
@@ -64,6 +64,15 @@#define HV_REGISTER_STIMER0_CONFIG 0x000B0000#define HV_REGISTER_STIMER0_COUNT 0x000B0001 +union hv_msi_entry { +u64 as_uint64[2 ]; +struct { +u64 address; +u32 data; +u32 reserved; + } __packed; +}; +#include <asm-generic/hyperv-tlfs.h> #endif
drivers/pci/Kconfig +1 -1
@@ -184,7 +184,7 @@ config PCI_LABEL config PCI_HYPERV tristate "Hyper-V PCI Frontend" - depends on X86_64 && HYPERV && PCI_MSI && PCI_MSI_IRQ_DOMAIN && SYSFS + depends on ((X86 && X86_64) || ARM64) && HYPERV && PCI_MSI && PCI_MSI_IRQ_DOMAIN && SYSFS select PCI_HYPERV_INTERFACE help The PCI device frontend driver allows the kernel to import arbitrary
drivers/pci/controller/Kconfig +1 -1
@@ -280,7 +280,7 @@ config PCIE_BRCMSTB config PCI_HYPERV_INTERFACE tristate "Hyper-V PCI Interface" - depends on X86 && HYPERV && PCI_MSI && PCI_MSI_IRQ_DOMAIN && X86_64 + depends on ((X86 && X86_64) || ARM64) && HYPERV && PCI_MSI && PCI_MSI_IRQ_DOMAIN help The Hyper-V PCI Interface is a helper driver allows other drivers to have a common interface with the Hyper-V PCI frontend driver.
drivers/pci/controller/pci-hyperv.c +206 -1
237 lines not on this page · raw message
Signature · 1 line
2.25.1
Re: [PATCH v4 2/2] arm64: PCI: hv: Add support for Hyper-V vPCI
Quoted 18 lines · Sunil Muthuswamy
From: Sunil Muthuswamy <[email protected]>
Add support for Hyper-V vPCI for arm64 by implementing the arch specific interfaces. Introduce an IRQ domain and chip specific to Hyper-v vPCI that is based on SPIs. The IRQ domain parents itself to the arch GIC IRQ domain for basic vector management.
Signed-off-by Sunil Muthuswamy <[email protected]>---
In v2, v3 & v4:
Changes are described in the cover letter.arch/arm64/include/asm/hyperv-tlfs.h | 9 ++ drivers/pci/Kconfig | 2 +- drivers/pci/controller/Kconfig | 2 +- drivers/pci/controller/pci-hyperv.c | 207 ++++++++++++++++++++++++++- 4 files changed, 217 insertions(+), 3 deletions(-)
[...]
Quoted 5 lines
drivers/pci/controller/pci-hyperv.c +5
-0+static int hv_pci_vec_irq_domain_activate (struct irq_domain *domain, +struct irq_data *irqd,bool reserve) +{ +static int cpu; +
drivers/pci/controller/pci-hyperv.c static int hv_msi_prepare(struct irq_domain *domain, struct device *dev, · line 763
Quoted 10 lines
drivers/pci/controller/pci-hyperv.c +10
-0+/* +* Pick a cpu using round-robin as the irq affinity that can be +* temporarily used for composing MSI from the hypervisor. GIC +* will eventually set the right affinity for the irq and the +* 'unmask' will retarget the interrupt to that cpu. +*/ +if (cpu >=cpumask_last (cpu_online_mask)) + cpu =0 ; + cpu =cpumask_next (cpu, cpu_online_mask); +irq_data_update_effective_affinity (irqd,cpumask_of (cpu));
The mind boggles.
Let's imagine a single machine. cpu_online_mask only has bit 0 set, and nr_cpumask_bits is 1. This is the first run, and cpu is 1:
cpu = cpumask_next(cpu, cpu_online_mask);
cpu is now set to 1. Which is not a valid CPU number, but a valid return value indicating that there is no next CPU as it is equal to nr_cpumask_bits. cpumask_of(cpu) will then diligently return crap, which you carefully store into the irq descriptor. The IRQ subsystem thanks you.
The same reasoning applies to any number of CPUs, and you obviously never checked what any of this does :-(. As to what the behaviour is when multiple CPUs run this function in parallel, let's not even bother (locking is overrated).
Logic and concurrency issues aside, why do you even bother setting some round-robin affinity if all you need is to set *something* so that a hypervisor message can be composed? Why not use the first online CPU? At least it will be correct.
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
Re: [PATCH v4 2/2] arm64: PCI: hv: Add support for Hyper-V vPCI
Quoted 42 lines · Marc Zyngier
Quoted 18 lines · Sunil Muthuswamy
From: Sunil Muthuswamy <[email protected]>
Add support for Hyper-V vPCI for arm64 by implementing the arch specific interfaces. Introduce an IRQ domain and chip specific to Hyper-v vPCI that is based on SPIs. The IRQ domain parents itself to the arch GIC IRQ domain for basic vector management.
Signed-off-by Sunil Muthuswamy <[email protected]>---
In v2, v3 & v4:
Changes are described in the cover letter.arch/arm64/include/asm/hyperv-tlfs.h | 9 ++ drivers/pci/Kconfig | 2 +- drivers/pci/controller/Kconfig | 2 +- drivers/pci/controller/pci-hyperv.c | 207 ++++++++++++++++++++++++++- 4 files changed, 217 insertions(+), 3 deletions(-)[...]
Quoted 5 lines
drivers/pci/controller/pci-hyperv.c +5
-0+static int hv_pci_vec_irq_domain_activate (struct irq_domain *domain, +struct irq_data *irqd,bool reserve) +{ +static int cpu; +drivers/pci/controller/pci-hyperv.c static int hv_msi_prepare(struct irq_domain *domain, struct device *dev, · line 763
Quoted 10 lines
drivers/pci/controller/pci-hyperv.c +10
-0+/* +* Pick a cpu using round-robin as the irq affinity that can be +* temporarily used for composing MSI from the hypervisor. GIC +* will eventually set the right affinity for the irq and the +* 'unmask' will retarget the interrupt to that cpu. +*/ +if (cpu >=cpumask_last (cpu_online_mask)) + cpu =0 ; + cpu =cpumask_next (cpu, cpu_online_mask); +irq_data_update_effective_affinity (irqd,cpumask_of (cpu));The mind boggles.
Quoted 1 line
Let's imagine a single machine.
cpu_online_maskonly has bit 0 set,
single *CPU* machine
Quoted 1 line
and
nr_cpumask_bitsis 1. This is the first run, and cpu is 1:
cpu is *obviously* 0:
Quoted 14 lines
cpu = cpumask_next(cpu, cpu_online_mask);cpu is now set to 1. Which is not a valid CPU number, but a valid return value indicating that there is no next CPU as it is equal to
nr_cpumask_bits.cpumask_of(cpu)will then diligently return crap, which you carefully store into the irq descriptor. The IRQ subsystem thanks you.The same reasoning applies to any number of CPUs, and you obviously never checked what any of this does :-(. As to what the behaviour is when multiple CPUs run this function in parallel, let's not even bother (locking is overrated).
Quoted 4 lines
Logic and concurrency issues aside, why do you even bother setting some round-robin affinity if all you need is to set *something* so that a hypervisor message can be composed? Why not use the first online CPU? At least it will be correct.
Everything else holds.
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
RE: [EXTERNAL] Re: [PATCH v4 2/2] arm64: PCI: hv: Add support for Hyper-V vPCI
Quoted 63 lines · Marc Zyngier
Quoted 41 lines
Quoted 18 lines · Sunil Muthuswamy
From: Sunil Muthuswamy <[email protected]>
Add support for Hyper-V vPCI for arm64 by implementing the arch specific interfaces. Introduce an IRQ domain and chip specific to Hyper-v vPCI that is based on SPIs. The IRQ domain parents itself to the arch GIC IRQ domain for basic vector management.
Signed-off-by Sunil Muthuswamy <[email protected]>---
In v2, v3 & v4:
Changes are described in the cover letter.arch/arm64/include/asm/hyperv-tlfs.h | 9 ++ drivers/pci/Kconfig | 2 +- drivers/pci/controller/Kconfig | 2 +- drivers/pci/controller/pci-hyperv.c | 207 ++++++++++++++++++++++++++- 4 files changed, 217 insertions(+), 3 deletions(-)[...]
Quoted 5 lines
drivers/pci/controller/pci-hyperv.c +5
-0+static int hv_pci_vec_irq_domain_activate (struct irq_domain *domain, +struct irq_data *irqd,bool reserve) +{ +static int cpu; +drivers/pci/controller/pci-hyperv.c static int hv_msi_prepare(struct irq_domain *domain, struct device *dev, · line 763
Quoted 10 lines
drivers/pci/controller/pci-hyperv.c +10
-0+/* +* Pick a cpu using round-robin as the irq affinity that can be +* temporarily used for composing MSI from the hypervisor. GIC +* will eventually set the right affinity for the irq and the +* 'unmask' will retarget the interrupt to that cpu. +*/ +if (cpu >=cpumask_last (cpu_online_mask)) + cpu =0 ; + cpu =cpumask_next (cpu, cpu_online_mask); +irq_data_update_effective_affinity (irqd,cpumask_of (cpu));The mind boggles.
Quoted 1 line
Let's imagine a single machine.
cpu_online_maskonly has bit 0 set,single *CPU* machine
Quoted 1 line
and
nr_cpumask_bitsis 1. This is the first run, and cpu is 1:cpu is *obviously* 0:
Quoted 14 lines
cpu = cpumask_next(cpu, cpu_online_mask);cpu is now set to 1. Which is not a valid CPU number, but a valid return value indicating that there is no next CPU as it is equal to
nr_cpumask_bits.cpumask_of(cpu)will then diligently return crap, which you carefully store into the irq descriptor. The IRQ subsystem thanks you.The same reasoning applies to any number of CPUs, and you obviously never checked what any of this does :-(. As to what the behaviour is when multiple CPUs run this function in parallel, let's not even bother (locking is overrated).
Quoted 8 lines
Quoted 4 lines
Logic and concurrency issues aside, why do you even bother setting some round-robin affinity if all you need is to set *something* so that a hypervisor message can be composed? Why not use the first online CPU? At least it will be correct.
Everything else holds.
M.
Good call on not being able to pick cpu 0 and that being a problem for single cpu system. The cpu initialization should have been '-1' to be able to successfully pick cpu 0.
I don't see concurrency an issue because this was a best-case effort to randomize the interrupt distribution across cpu's. So, even if two irq's ended up with the same cpu, that will still work.
I also had thoughts of just using the first online cpu since this is just temporary. So, I will go with that as that will also simplify things. Thanks for your feedback.
- Sunil
RE: [EXTERNAL] Re: [PATCH v4 2/2] arm64: PCI: hv: Add support for Hyper-V vPCI
Quoted 80 lines
Quoted 63 lines
Quoted 41 lines
Quoted 18 lines · Sunil Muthuswamy
From: Sunil Muthuswamy <[email protected]>
Add support for Hyper-V vPCI for arm64 by implementing the arch specific interfaces. Introduce an IRQ domain and chip specific to Hyper-v vPCI that is based on SPIs. The IRQ domain parents itself to the arch GIC IRQ domain for basic vector management.
Signed-off-by Sunil Muthuswamy <[email protected]>---
In v2, v3 & v4:
Changes are described in the cover letter.arch/arm64/include/asm/hyperv-tlfs.h | 9 ++ drivers/pci/Kconfig | 2 +- drivers/pci/controller/Kconfig | 2 +- drivers/pci/controller/pci-hyperv.c | 207 ++++++++++++++++++++++++++- 4 files changed, 217 insertions(+), 3 deletions(-)[...]
Quoted 5 lines
drivers/pci/controller/pci-hyperv.c +5
-0+static int hv_pci_vec_irq_domain_activate (struct irq_domain *domain, +struct irq_data *irqd,bool reserve) +{ +static int cpu; +drivers/pci/controller/pci-hyperv.c static int hv_msi_prepare(struct irq_domain *domain, struct device *dev, · line 763
Quoted 10 lines
drivers/pci/controller/pci-hyperv.c +10
-0+/* +* Pick a cpu using round-robin as the irq affinity that can be +* temporarily used for composing MSI from the hypervisor. GIC +* will eventually set the right affinity for the irq and the +* 'unmask' will retarget the interrupt to that cpu. +*/ +if (cpu >=cpumask_last (cpu_online_mask)) + cpu =0 ; + cpu =cpumask_next (cpu, cpu_online_mask); +irq_data_update_effective_affinity (irqd,cpumask_of (cpu));The mind boggles.
Quoted 1 line
Let's imagine a single machine.
cpu_online_maskonly has bit 0 set,single *CPU* machine
Quoted 1 line
and
nr_cpumask_bitsis 1. This is the first run, and cpu is 1:cpu is *obviously* 0:
Quoted 14 lines
cpu = cpumask_next(cpu, cpu_online_mask);cpu is now set to 1. Which is not a valid CPU number, but a valid return value indicating that there is no next CPU as it is equal to
nr_cpumask_bits.cpumask_of(cpu)will then diligently return crap, which you carefully store into the irq descriptor. The IRQ subsystem thanks you.The same reasoning applies to any number of CPUs, and you obviously never checked what any of this does :-(. As to what the behaviour is when multiple CPUs run this function in parallel, let's not even bother (locking is overrated).
Quoted 8 lines
Quoted 4 lines
Logic and concurrency issues aside, why do you even bother setting some round-robin affinity if all you need is to set *something* so that a hypervisor message can be composed? Why not use the first online CPU? At least it will be correct.
Everything else holds.
M.Good call on not being able to pick cpu 0 and that being a problem for single cpu system. The cpu initialization should have been '-1' to be able to successfully pick cpu 0.
I don't see concurrency an issue because this was a best-case effort to randomize the interrupt distribution across cpu's. So, even if two irq's ended up with the same cpu, that will still work.
Quoted 5 lines
I also had thoughts of just using the first online cpu since this is just temporary. So, I will go with that as that will also simplify things. Thanks for your feedback.
- Sunil
But, yes, for the concurrency, I do see a possibility of a race condition with the last cpu check and 'cpumask_next' call where it could lead to a failure. v5 moves this to the first online cpu and that should fix this issue.
- Sunil