xen: msi support for Xen dom0
Subsystems (4)
cover xen: msi support for Xen dom0
This patch set adds support for msi in Xen dom0. It's based on the pci notifier patches of Weidong Han (on rebase/pci branch) and contains the following 3 patches.
[PATCH 1/3] xen: make pci notifier work with booting devices [PATCH 2/3] xen: add msi support for dom0
[PATCH 3/3] xen: re-enable msi (effectively revert bf89bc29)
One of the problem left is how to save/restore MSI across S3. Since pci_restore_msi_state() now doesn't have any arch specific hook, the code in arch/x86/ won't get a chance to run during S3 wakeup, so write_msi_msg() is called instead of xen specific functions. One of the possible solutions (and which I prefer) is to add something like arch_pci_restore_msi, but that involves slightly changing drivers/pci/msi.c, which probably needs more thinking and discussion.
An alternative is to trap and emulate any access to pci configuration space. In that case, nothing in dom0 needs changing, and write_msi_msg can be reused, but considerable logic may need to change in Xen hypervisor.
Thanks,
Qing
PATCH 1/3 xen: make pci notifier work with booting devices
change fs_initcall to arch_initcall so that pci notifier would handle booting pci devices
--- drivers/xen/pci.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
drivers/xen/pci.c +1 -1
@@ -113,4 +113,4 @@ static int __init register_xen_pci_notifier(void)return bus_register_notifier (&pci_bus_type, &device_nb); } -fs_initcall (register_xen_pci_notifier); +arch_initcall (register_xen_pci_notifier);
Signature · 1 line
1.6.0
PATCH 2/3 xen: add msi support for dom0
In reply to Qing HeThis patch adds msi support for dom0, based on arch_setup_msi_irqs hook, a xen_setup_msi_irqs is called if it's Xen domain. No interrupt remapping is handled since Xen domain isn't exposed with such feature at this time.
--- arch/x86/include/asm/xen/pci.h | 20 +++++++++ arch/x86/kernel/apic/io_apic.c | 9 +++- arch/x86/xen/pci.c | 24 ++++++++++ drivers/xen/events.c | 91 ++++++++++++++++++++++++++++++++++++++- include/xen/interface/physdev.h | 30 +++++++++++++ 5 files changed, 172 insertions(+), 2 deletions(-)
arch/x86/include/asm/xen/pci.h +20 -0
@@ -3,11 +3,31 @@#ifdef CONFIG_XEN_DOM0_PCI int xen_register_gsi (u32 gsi,int triggering,int polarity); +int xen_create_msi_irq (struct pci_dev *dev, +struct msi_desc *msidesc, +int type); +int xen_destroy_irq (int irq); +int xen_setup_msi_irqs (struct pci_dev *dev,int nvec,int type);#else static inline int xen_register_gsi (u32 gsi,int triggering,int polarity) {return -1 ; } + +static int xen_create_msi_irq (struct pci_dev *dev, +struct msi_desc *msidesc, +int type) +{ +return -1 ; +} +static int xen_destroy_irq (int irq) +{ +return -1 ; +} +static int xen_setup_msi_irqs (struct pci_dev *dev,int nvec,int type) +{ +return -1 ; +}#endif #endif /* _ASM_X86_XEN_PCI_H */
arch/x86/kernel/apic/io_apic.c +8 -1
@@ -66,6 +66,7 @@#include <asm/xen/hypervisor.h> #include <asm/apic.h> +#include <asm/xen/pci.h> #define __apicdebuginit (type) static type __init
@@ -3502,6 +3503,9 @@ int arch_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)if (type ==PCI_CAP_ID_MSI && nvec >1 )return 1 ; +if (xen_domain ()) +return xen_setup_msi_irqs (dev, nvec, type); + irq_want = nr_irqs_gsi; sub_handle =0 ;list_for_each_entry (msidesc, &dev->msi_list, list) {
@@ -3550,7 +3554,10 @@ error:void arch_teardown_msi_irq (unsigned int irq) { -destroy_irq (irq); +if (xen_domain ()) +xen_destroy_irq (irq); +else +destroy_irq (irq); }#if defined (CONFIG_DMAR ) || defined (CONFIG_INTR_REMAP )
arch/x86/xen/pci.c +24 -0
@@ -1,12 +1,14 @@#include <linux/kernel.h> #include <linux/acpi.h> #include <linux/pci.h> +#include <linux/msi.h> #include <asm/mpspec.h> #include <asm/io_apic.h> #include <asm/pci_x86.h> #include <asm/xen/hypervisor.h> +#include <asm/xen/pci.h> #include <xen/interface/xen.h> #include <xen/events.h>
@@ -84,3 +86,25 @@ void __init xen_setup_pirqs(void) polarity ? ACPI_ACTIVE_LOW : ACPI_ACTIVE_HIGH); } } + +int xen_setup_msi_irqs (struct pci_dev *dev,int nvec,int type) +{ +unsigned int irq; +int ret; +struct msi_desc *msidesc; + +list_for_each_entry (msidesc, &dev->msi_list, list) { + irq =xen_create_msi_irq (dev, msidesc, type); +if (irq ==0 ) +return -1 ; + + ret =set_irq_msi (irq, msidesc); +if (ret) + goto error; + } +return 0 ; + +error: +xen_destroy_irq (irq); +return ret; +}
drivers/xen/events.c +90 -1
116 lines not on this page · raw message
include/xen/interface/physdev.h +30 -0
@@ -106,6 +106,36 @@ struct physdev_irq {uint32_t vector; }; +#define MAP_PIRQ_TYPE_MSI 0x0 +#define MAP_PIRQ_TYPE_GSI 0x1 +#define MAP_PIRQ_TYPE_UNKNOWN 0x2 + +#define PHYSDEVOP_map_pirq 13 +struct physdev_map_pirq { +domid_t domid; +/* IN */ +int type; +/* IN */ +int index; +/* IN or OUT */ +int pirq; +/* IN */ +int bus; +/* IN */ +int devfn; +/* IN */ +int entry_nr; +/* IN */ +uint64_t table_base; +}; + +#define PHYSDEVOP_unmap_pirq 14 +struct physdev_unmap_pirq { +domid_t domid; +/* IN */ +int pirq; +}; +#define PHYSDEVOP_manage_pci_add 15#define PHYSDEVOP_manage_pci_remove 16struct physdev_manage_pci {
Signature · 1 line
1.6.0
PATCH 3/3 xen: re-enable msi
In reply to Qing Heremove pci_no_msi for Xen Dom 0, also move pci_no_msi out of public header
effectively revert
commit bf89bc290d429ce223c1018628130ddabc66614e xen: disable MSI
--- arch/x86/xen/apic.c | 3 --- drivers/pci/pci.h | 2 ++ include/linux/pci.h | 6 ------ 3 files changed, 2 insertions(+), 9 deletions(-)
arch/x86/xen/apic.c +0 -3
@@ -1,7 +1,6 @@#include <linux/kernel.h> #include <linux/threads.h> #include <linux/bitmap.h> -#include <linux/pci.h> #include <asm/io_apic.h> #include <asm/acpi.h>
@@ -48,8 +47,6 @@ void xen_init_apic(void)if (!xen_initial_domain ())return ; -pci_no_msi (); -#ifdef CONFIG_ACPI /* * Pretend ACPI found our lapic even though we've disabled it,
drivers/pci/pci.h +2 -0
@@ -111,8 +111,10 @@ extern struct rw_semaphore pci_bus_sem;extern unsigned int pci_pm_d3_delay;#ifdef CONFIG_PCI_MSI +void pci_no_msi (void );extern void pci_msi_init_pci_dev (struct pci_dev *dev);#else +static inline void pci_no_msi (void ) { }static inline void pci_msi_init_pci_dev (struct pci_dev *dev) { }#endif
include/linux/pci.h +0 -6
@@ -1256,11 +1256,5 @@ static inline irqreturn_t pci_sriov_migration(struct pci_dev *dev) }#endif -#ifdef CONFIG_PCI_MSI -void pci_no_msi (void ); -#else -static inline void pci_no_msi (void ) { } -#endif -#endif /* __KERNEL__ */ #endif /* LINUX_PCI_H */
Signature · 1 line
1.6.0
Quoted 8 lines · Qing He
This patch set adds support for msi in Xen dom0. It's based on the pci notifier patches of Weidong Han (on rebase/pci branch) and contains the following 3 patches.
[PATCH 1/3] xen: make pci notifier work with booting devices [PATCH 2/3] xen: add msi support for dom0
[PATCH 3/3] xen: re-enable msi (effectively revertbf89bc29)
Thanks, I've applied these to rebase/dom0/msi for now. I haven't
tested them (or really compiled them) yet, so please look at the branch and see that everything's OK.
Quoted 8 lines
One of the problem left is how to save/restore MSI across S3. Since
pci_restore_msi_state()now doesn't have any arch specific hook, the code inarch/x86/won't get a chance to run during S3 wakeup, sowrite_msi_msg()is called instead of xen specific functions. One of the possible solutions (and which I prefer) is to add something likearch_pci_restore_msi, but that involves slightly changingdrivers/pci/msi.c, which probably needs more thinking and discussion.
Quoted 5 lines
An alternative is to trap and emulate any access to pci configuration space. In that case, nothing in dom0 needs changing, and
write_msi_msgcan be reused, but considerable logic may need to change in Xen hypervisor.
The approach taken by 2/3 is not really going to fly upstream, and is broadly incompatible with my intended design for interrupt handling, which is to decouple the Xen/dom0 aspects of interrupt handling from the apic/ioapic code entirely. I don't know what impact this will have on MSI support. I'd appreciate it if you could look at the rebase/dom0/new-interrupt-routing branch and comment on it.
I'm not actually sure this approach is going to work; so far it just locks up the machine shortly after ACPI initialization. Trapping an emulating (IO-)APIC accesses may well turn out to be simpler (on the Linux side, at least) and more robust in the end...
J
Quoted 28 lines · Jeremy Fitzhardinge
Quoted 8 lines · Qing He
This patch set adds support for msi in Xen dom0. It's based on the pci notifier patches of Weidong Han (on rebase/pci branch) and contains the following 3 patches.
[PATCH 1/3] xen: make pci notifier work with booting devices [PATCH 2/3] xen: add msi support for dom0
[PATCH 3/3] xen: re-enable msi (effectively revertbf89bc29)Thanks, I've applied these to rebase/dom0/msi for now. I haven't
tested them (or really compiled them) yet, so please look at the branch and see that everything's OK.Quoted 8 lines
One of the problem left is how to save/restore MSI across S3. Since
pci_restore_msi_state()now doesn't have any arch specific hook, the code inarch/x86/won't get a chance to run during S3 wakeup, sowrite_msi_msg()is called instead of xen specific functions. One of the possible solutions (and which I prefer) is to add something likearch_pci_restore_msi, but that involves slightly changingdrivers/pci/msi.c, which probably needs more thinking and discussion.Quoted 5 lines
An alternative is to trap and emulate any access to pci configuration space. In that case, nothing in dom0 needs changing, and
write_msi_msgcan be reused, but considerable logic may need to change in Xen hypervisor.
Quoted 6 lines
The approach taken by 2/3 is not really going to fly upstream, and is broadly incompatible with my intended design for interrupt handling, which is to decouple the Xen/dom0 aspects of interrupt handling from the apic/ioapic code entirely. I don't know what impact this will have on MSI support. I'd appreciate it if you could look at the rebase/dom0/new-interrupt-routing branch and comment on it.
I'll have a look at it first.
Quoted 5 lines
I'm not actually sure this approach is going to work; so far it just locks up the machine shortly after ACPI initialization. Trapping an emulating (IO-)APIC accesses may well turn out to be simpler (on the Linux side, at least) and more robust in the end...
MSI by nature is vector based, but xen uses pirq as the interface to communicate with dom0, 2/3 is actually used to handle the pirq thing instead of solely vectors.
So trapping and emulating for msi will first need a decoupling of that, allowing dom0 to use only vector to index its msi. This requires changing of the current pirq based xen irq, which seems quite annoying at Xen side, especially if per-cpu vector is in.
Thanks,
Qing
Quoted 2 lines
J
Quoted 42 lines · Qing He
Quoted 30 lines · Jeremy Fitzhardinge
Quoted 9 lines · Qing He
This patch set adds support for msi in Xen dom0. It's based on the pci notifier patches of Weidong Han (on rebase/pci branch) and contains the following 3 patches.
[PATCH 1/3] xen: make pci notifier work with booting devices [PATCH 2/3] xen: add msi support for dom0
[PATCH 3/3] xen: re-enable msi (effectively revertbf89bc29)Thanks, I've applied these to rebase/dom0/msi for now. I haven't
tested them (or really compiled them) yet, so please look at the branch and see that everything's OK.Quoted 8 lines
One of the problem left is how to save/restore MSI across S3. Since
pci_restore_msi_state()now doesn't have any arch specific hook, the code inarch/x86/won't get a chance to run during S3 wakeup, sowrite_msi_msg()is called instead of xen specific functions. One of the possible solutions (and which I prefer) is to add something likearch_pci_restore_msi, but that involves slightly changingdrivers/pci/msi.c, which probably needs more thinking and discussion.Quoted 6 lines
An alternative is to trap and emulate any access to pci configuration space. In that case, nothing in dom0 needs changing, and
write_msi_msgcan be reused, but considerable logic may need to change in Xen hypervisor.Quoted 7 lines
The approach taken by 2/3 is not really going to fly upstream, and is broadly incompatible with my intended design for interrupt handling, which is to decouple the Xen/dom0 aspects of interrupt handling from the apic/ioapic code entirely. I don't know what impact this will have on MSI support. I'd appreciate it if you could look at the rebase/dom0/new-interrupt-routing branch and comment on it.
I'll have a look at it first.
Quoted 9 lines
Quoted 5 lines
I'm not actually sure this approach is going to work; so far it just locks up the machine shortly after ACPI initialization. Trapping an emulating (IO-)APIC accesses may well turn out to be simpler (on the Linux side, at least) and more robust in the end...
MSI by nature is vector based, but xen uses pirq as the interface to communicate with dom0, 2/3 is actually used to handle the pirq thing instead of solely vectors.
Yes, that's awkward. The new-interrupt-routing branch moves towards eliminating vectors from the kernel<->xen, replacing them entirely with pirqs. Part of the motivation for this is to insulate the kernel from Xen's decisions on how to allocate and route vectors (ie, the number of vectors and whether they're percpu or not).
Extending this to MSI would presumably require Xen to do the actual PCI (config space?) programming given a pirq for the device interrupt. Perhaps this could be best achieved by making the x86 arch_setup_msi_irqs call through a function pointer so we can direct it to a Xen-specific version. Would that be clean?
Quoted 5 lines
So trapping and emulating for msi will first need a decoupling of that, allowing dom0 to use only vector to index its msi. This requires changing of the current pirq based xen irq, which seems quite annoying at Xen side, especially if per-cpu vector is in.
What about making pirq == vector, so the "vector" seen by the kernel isn't a real vector at all, and Xen remaps from the pirq/vector to the real vector when emulating the write?
J
Quoted 16 lines · Jeremy Fitzhardinge
Quoted 9 lines
Quoted 5 lines
I'm not actually sure this approach is going to work; so far it just locks up the machine shortly after ACPI initialization. Trapping an emulating (IO-)APIC accesses may well turn out to be simpler (on the Linux side, at least) and more robust in the end...
MSI by nature is vector based, but xen uses pirq as the interface to communicate with dom0, 2/3 is actually used to handle the pirq thing instead of solely vectors.
Yes, that's awkward. The new-interrupt-routing branch moves towards eliminating vectors from the kernel<->xen, replacing them entirely with pirqs. Part of the motivation for this is to insulate the kernel from Xen's decisions on how to allocate and route vectors (ie, the number of vectors and whether they're percpu or not).
Quoted 5 lines
Extending this to MSI would presumably require Xen to do the actual PCI (config space?) programming given a pirq for the device interrupt. Perhaps this could be best achieved by making the x86
arch_setup_msi_irqscall through a function pointer so we can direct it to a Xen-specific version. Would that be clean?
I finished reading the new-interrupt-routing branch and had some revisit on MSI. Basically, I think this branch is OK, exposing Xen vectors to guest is quite weird (also includes index' field of map_pirq call), this vector' is now even renamed to `irq' because of per-cpu vector.
And my MSI patches don't conflict significantly with the new-interrupt-routing branch. We can consider the setting up of MSI as two aspects: one is programming the interrupt source (by writing to config space and MSIX mmio), the other is software: creating new irqs, and starting them. The method I use now is to add a branching in arch_setup_msi_irqs, to (1) set the irq_chip of these new irqs to pirq; (2) avoid programming the config space. This can be changed to funciton pointers with no additional pain.
Quoted 9 lines
Quoted 5 lines
So trapping and emulating for msi will first need a decoupling of that, allowing dom0 to use only vector to index its msi. This requires changing of the current pirq based xen irq, which seems quite annoying at Xen side, especially if per-cpu vector is in.
What about making pirq == vector, so the "vector" seen by the kernel isn't a real vector at all, and Xen remaps from the pirq/vector to the real vector when emulating the write?
But the vector is allocated by the kernel, right? How can Xen know which vector (and its CPU affinity) is related to a pirq?
For suspend/resume of MSI devices support, what the kernel does on bare metal is simply to reprogram the interrupt source, including writing vectors and set enable bit. On Xen, the reprogramming is done in the hypervisor, so we just need to bypass kernel the kernel reprogramming: If using T&E, just ignore address/data writing and if not, an arch specific hook is needed.
And what do you mean by I/O APIC T&E? Since dom0 is eventually notified by evtchn, a relation between vector and evtchn has to be established, which doesn't exist yet. I ever think of mapping evtchns for (vcpu, vector) pairs at vector allocation time, so that everything else can be keep untouched. But that seems to involve to much change.
Thanks,
Qing
Quoted 2 lines
J
Quoted 5 lines · Qing He
I finished reading the new-interrupt-routing branch and had some revisit on MSI. Basically, I think this branch is OK, exposing Xen vectors to guest is quite weird (also includes
index' field of map_pirq call), thisvector' is now even renamed to `irq' because of per-cpu vector.
OK, good.
Quoted 9 lines
And my MSI patches don't conflict significantly with the new-interrupt-routing branch. We can consider the setting up of MSI as two aspects: one is programming the interrupt source (by writing to config space and MSIX mmio), the other is software: creating new irqs, and starting them. The method I use now is to add a branching in
arch_setup_msi_irqs, to (1) set theirq_chipof these new irqs to pirq; (2) avoid programming the config space. This can be changed to funciton pointers with no additional pain.
Yep.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
So trapping and emulating for msi will first need a decoupling of that, allowing dom0 to use only vector to index its msi. This requires changing of the current pirq based xen irq, which seems quite annoying at Xen side, especially if per-cpu vector is in.
Quoted 7 lines
Quoted 4 lines
What about making pirq == vector, so the "vector" seen by the kernel isn't a real vector at all, and Xen remaps from the pirq/vector to the real vector when emulating the write?
But the vector is allocated by the kernel, right? How can Xen know which vector (and its CPU affinity) is related to a pirq?
In the current dom0 scheme, the kernel does a hypercall to ask Xen to allocate the vector for a given pirq.
Quoted 7 lines
For suspend/resume of MSI devices support, what the kernel does on bare metal is simply to reprogram the interrupt source, including writing vectors and set enable bit. On Xen, the reprogramming is done in the hypervisor, so we just need to bypass kernel the kernel reprogramming: If using T&E, just ignore address/data writing and if not, an arch specific hook is needed.
That's fairly straightforward, since the PCI config space is mapped anyway. We just need to have some way to create mappings that will trap (ie, we can't use the ptes the kernel establishes as-is, as they won't cause a pagefault).
Quoted 6 lines
And what do you mean by I/O APIC T&E? Since dom0 is eventually notified by evtchn, a relation between vector and evtchn has to be established, which doesn't exist yet. I ever think of mapping evtchns for (vcpu, vector) pairs at vector allocation time, so that everything else can be keep untouched. But that seems to involve to much change.
At the moment we need to paravirtualize IO APIC register writes via hypercall, specifically so that Xen can completely establish the apic+pin -> vector -> pirq -> evtchn mapping. It needs to be a hypercall because the IO APIC doesn't have an explicit mapping within the dom0 address space. If we created such a mapping and trapped reads and writes to it, it would be logically equivalent to the hypercall, but wouldn't require kernel changes.
However, this is moot in the new-interrupt-routing branch as we take over the whole interrupt path, so there's no problem making hypercalls (either to update the IO APIC via register writes, or the new hypercall to establish a gsi -> pirq mapping).
J