Generic PCIe host bridge INTx determination for INTx routing
v3 cover Generic PCIe host bridge INTx determination for INTx routing
This series implements INTx to gsi routing for ARM VIRT/Gpex. This is a respin of [1] which was lost in limbo.
ARM virt uses GPEX PCIe bridge. This latter does not implement INTx to GSI routing. PCIe/INTx assignment works but the consequence is irqfd is not used along with INTx interrupts and VFIO INTx handlers are executed on userspace leading to an important performance degradation.
This issue is witnessed by the following messages;
qemu-system-aarch64: -device vfio-pci,host=0006:90:00.0: PCI: Bug - unimplemented PCI INTx routing (gpex-pcihost)
qemu-system-aarch64: PCI: Bug - unimplemented PCI INTx routing (gpex-pcihost)
qemu-system-aarch64: PCI: Bug - unimplemented PCI INTx routing (gpex-pcihost)
So with this series, irqfd is set up for PCIe/INTx passthrough and we get the optimal performance. Also we get rid of the above messages.
This series can be found at:
https://github.com/eauger/qemu/tree/v2.9-gpex-intx-v3
References:
[1] Generic PCIe host bridge INTx determination for INTx routing
https://lists.gnu.org/archive/html/qemu-devel/2015-09/msg04975.html
Pranavkumar Sawargaonkar (3):
hw/pci-host/gpex: Set INTx index/gsi mapping
hw/arm/virt: Set INTx/gsi mapping
hw/pci-host/gpex: Implement PCI INTx routing
hw/arm/virt.c | 4 ++++ hw/pci-host/gpex.c | 22 ++++++++++++++++++++++ include/hw/pci-host/gpex.h | 3 +++ 3 files changed, 29 insertions(+)
Signature · 1 line
2.5.5
PATCH RESEND v3 1/3 hw/pci-host/gpex: Set INTx index/gsi mapping
From: Pranavkumar Sawargaonkar <[email protected]>
To implement INTx to gsi routing we need to pass the gpex host bridge the gsi associated to each INTx index. Let's introduce irq_num array and gpex_set_irq_num setter function.
--- hw/pci-host/gpex.c | 10 ++++++++++ include/hw/pci-host/gpex.h | 3 +++ 2 files changed, 13 insertions(+)
hw/pci-host/gpex.c +10 -0
@@ -43,6 +43,16 @@ static void gpex_set_irq(void *opaque, int irq_num, int level)qemu_set_irq (s->irq[irq_num], level); } +int gpex_set_irq_num (GPEXHost *s,int index,uint32_t gsi) +{ +if (index >=GPEX_NUM_IRQS ) { +return -EINVAL ; + } + + s->irq_num[index] = gsi; +return 0 ; +} +static void gpex_host_realize (DeviceState *dev,Error **errp) {PCIHostState *pci =PCI_HOST_BRIDGE (dev);
include/hw/pci-host/gpex.h +3 -0
@@ -51,6 +51,9 @@ typedef struct GPEXHost {MemoryRegion io_ioport;MemoryRegion io_mmio;qemu_irq irq[GPEX_NUM_IRQS ]; + uint32_t irq_num[GPEX_NUM_IRQS]; } GPEXHost; +int gpex_set_irq_num (GPEXHost *s,int index,uint32_t gsi); +#endif /* HW_GPEX_H */
Signature · 1 line
2.5.5
Re: [Qemu-arm] [Qemu-devel] [PATCH RESEND v3 1/3] hw/pci-host/gpex: Set INTx index/gsi mapping
Quoted 19 lines · Eric Auger
From: Pranavkumar Sawargaonkar <[email protected]>
To implement INTx to gsi routing we need to pass the gpex host bridge the gsi associated to each INTx index. Let's introduce
irq_numarray andgpex_set_irq_numsetter function.Signed-off-by Pranavkumar Sawargaonkar <[email protected]>Signed-off-by Tushar Jagad <[email protected]>Signed-off-by Eric Auger <[email protected]>--- hw/pci-host/gpex.c | 10 ++++++++++ include/hw/pci-host/gpex.h | 3 +++ 2 files changed, 13 insertions(+)hw/pci-host/gpex.c +0
-0
hw/pci-host/gpex.c static void gpex_set_irq(void *opaque, int irq_num, int level) · line 48
Quoted 6 lines
hw/pci-host/gpex.c +3
-0@@ -43,6 +43,16 @@ static void gpex_set_irq(void *opaque, int irq_num, int level)qemu_set_irq (s->irq[irq_num], level); } +int gpex_set_irq_num (GPEXHost *s,int index,uint32_t gsi) +{ +if (index >=GPEX_NUM_IRQS ) {
^extra space
Quoted 15 lines
hw/pci-host/gpex.c +7
-0+return -EINVAL ; + } + + s->irq_num[index] = gsi; +return 0 ; +} +static void gpex_host_realize (DeviceState *dev,Error **errp) {PCIHostState *pci =PCI_HOST_BRIDGE (dev);include/hw/pci-host/gpex.h +0
-0
include/hw/pci-host/gpex.h typedef struct GPEXHost { · line 54
Quoted 4 lines
include/hw/pci-host/gpex.h +1
-0@@ -51,6 +51,9 @@ typedef struct GPEXHost {MemoryRegion io_ioport;MemoryRegion io_mmio;qemu_irq irq[GPEX_NUM_IRQS ]; +uint32_t irq_num[GPEX_NUM_IRQS ];
This is an int when calling gpex_set_irq_num() and then used as an int later, requiring a cast. Why not just make it an int here and in the API?
Quoted 9 lines
include/hw/pci-host/gpex.h +2
-0} GPEXHost; +int gpex_set_irq_num (GPEXHost *s,int index,uint32_t gsi); +#endif /* HW_GPEX_H */ Signature · 1 line
2.5.5
PATCH RESEND v3 2/3 hw/arm/virt: Set INTx/gsi mapping
In reply to Eric AugerFrom: Pranavkumar Sawargaonkar <[email protected]>
Let's provide the GPEX host bridge with the INTx/gsi mapping. This is needed for INTx/gsi routing.
--- hw/arm/virt.c | 4 ++++ 1 file changed, 4 insertions(+)
hw/arm/virt.c +4 -0
@@ -1012,6 +1012,7 @@ static void create_pcie(const VirtMachineState *vms, qemu_irq *pic)char *nodename;int i;PCIHostState *pci; +GPEXHost *s; dev =qdev_create (NULL ,TYPE_GPEX_HOST );qdev_init_nofail (dev);
@@ -1047,8 +1048,11 @@ static void create_pcie(const VirtMachineState *vms, qemu_irq *pic)/* Map IO port space */ sysbus_mmio_map (SYS_BUS_DEVICE (dev),2 , base_pio); + s =GPEX_HOST (dev); +for (i =0 ; i <GPEX_NUM_IRQS ; i++) {sysbus_connect_irq (SYS_BUS_DEVICE (dev), i, pic[irq + i]); +gpex_set_irq_num (s, i, irq + i); } pci =PCI_HOST_BRIDGE (dev);
Signature · 1 line
2.5.5
Re: [Qemu-arm] [Qemu-devel] [PATCH RESEND v3 2/3] hw/arm/virt: Set INTx/gsi mapping
Quoted 25 lines · Eric Auger
From: Pranavkumar Sawargaonkar <[email protected]>
Let's provide the GPEX host bridge with the INTx/gsi mapping. This is needed for INTx/gsi routing.
Signed-off-by Pranavkumar Sawargaonkar <[email protected]>Signed-off-by Tushar Jagad <[email protected]>Signed-off-by Eric Auger <[email protected]>--- hw/arm/virt.c | 4 ++++ 1 file changed, 4 insertions(+)hw/arm/virt.c +1
-0@@ -1012,6 +1012,7 @@ static void create_pcie(const VirtMachineState *vms, qemu_irq *pic)char *nodename;int i;PCIHostState *pci; +GPEXHost *s; dev =qdev_create (NULL ,TYPE_GPEX_HOST );qdev_init_nofail (dev);
hw/arm/virt.c static void create_pcie(const VirtMachineState *vms, qemu_irq *pic) · line 1055
Quoted 8 lines
hw/arm/virt.c +3
-0@@ -1047,8 +1048,11 @@ static void create_pcie(const VirtMachineState *vms, qemu_irq *pic)/* Map IO port space */ sysbus_mmio_map (SYS_BUS_DEVICE (dev),2 , base_pio); + s =GPEX_HOST (dev); +for (i =0 ; i <GPEX_NUM_IRQS ; i++) {sysbus_connect_irq (SYS_BUS_DEVICE (dev), i, pic[irq + i]); +gpex_set_irq_num (s, i, irq + i);
nit: don't need the 's' variable for the one use, just do gpex_set_irq_num(GPEX_HOST(dev), ...)
Quoted 7 lines
hw/arm/virt.c +0
-0} pci =PCI_HOST_BRIDGE (dev);Signature · 1 line
2.5.5
Otherwise
PATCH RESEND v3 3/3 hw/pci-host/gpex: Implement PCI INTx routing
In reply to Eric AugerFrom: Pranavkumar Sawargaonkar <[email protected]>
Now we are able to retrieve the gsi from the INTx pin, let's enable intx_to_irq routing. From that point on, irqfd becomes usable along with INTx when assigning a PCIe device.
--- hw/pci-host/gpex.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)
hw/pci-host/gpex.c +12 -0
@@ -53,6 +53,17 @@ int gpex_set_irq_num(GPEXHost *s, int index, uint32_t gsi)return 0 ; } +static PCIINTxRoute gpex_route_intx_pin_to_irq (void *opaque,int pin) +{ +PCIINTxRoute route; +GPEXHost *s = opaque; + + route.mode =PCI_INTX_ENABLED ; + route.irq = (int )s->irq_num[pin]; + +return route; +} +static void gpex_host_realize (DeviceState *dev,Error **errp) {PCIHostState *pci =PCI_HOST_BRIDGE (dev);
@@ -77,6 +88,7 @@ static void gpex_host_realize(DeviceState *dev, Error **errp) &s->io_ioport, 0, 4, TYPE_PCIE_BUS);qdev_set_parent_bus (DEVICE (&s->gpex_root),BUS (pci->bus)); +pci_bus_set_route_irq_fn (pci->bus, gpex_route_intx_pin_to_irq);qdev_init_nofail (DEVICE (&s->gpex_root)); }
Signature · 1 line
2.5.5
Re: [Qemu-arm] [Qemu-devel] [PATCH RESEND v3 3/3] hw/pci-host/gpex: Implement PCI INTx routing
Quoted 47 lines · Eric Auger
From: Pranavkumar Sawargaonkar <[email protected]>
Now we are able to retrieve the gsi from the INTx pin, let's enable
intx_to_irqrouting. From that point on, irqfd becomes usable along with INTx when assigning a PCIe device.Signed-off-by Pranavkumar Sawargaonkar <[email protected]>Signed-off-by Tushar Jagad <[email protected]>Signed-off-by Eric Auger <[email protected]>--- hw/pci-host/gpex.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)hw/pci-host/gpex.c +12
-0@@ -53,6 +53,17 @@ int gpex_set_irq_num(GPEXHost *s, int index, uint32_t gsi)return 0 ; } +static PCIINTxRoute gpex_route_intx_pin_to_irq (void *opaque,int pin) +{ +PCIINTxRoute route; +GPEXHost *s = opaque; + + route.mode =PCI_INTX_ENABLED ; + route.irq = (int )s->irq_num[pin]; + +return route; +} +static void gpex_host_realize (DeviceState *dev,Error **errp) {PCIHostState *pci =PCI_HOST_BRIDGE (dev);@@ -77,6 +88,7 @@ static void gpex_host_realize(DeviceState *dev, Error **errp) &s->io_ioport, 0, 4, TYPE_PCIE_BUS);qdev_set_parent_bus (DEVICE (&s->gpex_root),BUS (pci->bus)); +pci_bus_set_route_irq_fn (pci->bus, gpex_route_intx_pin_to_irq);qdev_init_nofail (DEVICE (&s->gpex_root)); }Signature · 1 line
2.5.5
Wouldn't need the cast if irq_num was an array of ints, but otherwise
Hi,
Quoted 10 lines · Eric Auger
This series implements INTx to gsi routing for ARM VIRT/Gpex. This is a respin of [1] which was lost in limbo.
ARM virt uses GPEX PCIe bridge. This latter does not implement INTx to GSI routing. PCIe/INTx assignment works but the consequence is irqfd is not used along with INTx interrupts and VFIO INTx handlers are executed on userspace leading to an important performance degradation.
This issue is witnessed by the following messages;
Quoted 7 lines
qemu-system-aarch64: -device vfio-pci,host=0006:90:00.0: PCI: Bug - unimplemented PCI INTx routing (gpex-pcihost)
qemu-system-aarch64: PCI: Bug - unimplemented PCI INTx routing (gpex-pcihost)
qemu-system-aarch64: PCI: Bug - unimplemented PCI INTx routing (gpex-pcihost)So with this series, irqfd is set up for PCIe/INTx passthrough and we get the optimal performance. Also we get rid of the above messages.
If I haven't missed anything, I have not received any comment on this series despite ping and resend since mid June. Now most of the 2.10 rush is behind, please can anyone have a look?
The BUG message hits when your assign a PF and performance is largely decreased whenever INTx are used during passthrough.
Thanks
Eric
Quoted 19 lines
This series can be found at:
https://github.com/eauger/qemu/tree/v2.9-gpex-intx-v3References:
[1] Generic PCIe host bridge INTx determination for INTx routing
https://lists.gnu.org/archive/html/qemu-devel/2015-09/msg04975.htmlPranavkumar Sawargaonkar (3):
hw/pci-host/gpex: Set INTx index/gsi mapping
hw/arm/virt: Set INTx/gsi mapping
hw/pci-host/gpex: Implement PCI INTx routinghw/arm/virt.c | 4 ++++ hw/pci-host/gpex.c | 22 ++++++++++++++++++++++ include/hw/pci-host/gpex.h | 3 +++ 3 files changed, 29 insertions(+)
On Jul 9, 2017 2:04 PM, "Eric Auger" <[email protected]> wrote:
This series implements INTx to gsi routing for ARM VIRT/Gpex. This is a respin of [1] which was lost in limbo.
ARM virt uses GPEX PCIe bridge. This latter does not implement INTx to GSI routing. PCIe/INTx assignment works but the consequence is irqfd is not used along with INTx interrupts and VFIO INTx handlers are executed on userspace leading to an important performance degradation.
This issue is witnessed by the following messages;
qemu-system-aarch64: -device vfio-pci,host=0006:90:00.0: PCI: Bug - unimplemented PCI INTx routing (gpex-pcihost)
qemu-system-aarch64: PCI: Bug - unimplemented PCI INTx routing (gpex-pcihost)
qemu-system-aarch64: PCI: Bug - unimplemented PCI INTx routing (gpex-pcihost)
So with this series, irqfd is set up for PCIe/INTx passthrough and we get the optimal performance. Also we get rid of the above messages.
This series can be found at:
https://github.com/eauger/qemu/tree/v2.9-gpex-intx-v3
References:
[1] Generic PCIe host bridge INTx determination for INTx routing
https://lists.gnu.org/archive/html/qemu-devel/2015-09/msg04975.html
Pranavkumar Sawargaonkar (3):
hw/pci-host/gpex: Set INTx index/gsi mapping
hw/arm/virt: Set INTx/gsi mapping
hw/pci-host/gpex: Implement PCI INTx routing
hw/arm/virt.c | 4 ++++ hw/pci-host/gpex.c | 22 ++++++++++++++++++++++ include/hw/pci-host/gpex.h | 3 +++ 3 files changed, 29 insertions(+)
--
2.5.5
Works for APM XGene systems.
Thanks!