RFC: vfio-pci: Allow mmap of MSIX BAR
PATCH 1/1 RFC: vfio-pci: Allow mmap of MSIX BAR
This makes use of a new VFIO_REGION_INFO_CAP_MSIX_MAPPABLE capability which tells that a region with MSIX data can be mapped entirely, i.e. the VFIO PCI driver won't prevent MSIX vectors area from being mapped.
This adds a "msix-no-mmap" property to the vfio-pci device, it is "true" by default and "false" for pseries-2.12+ machines.
This requites kernel's "vfio-pci: Allow mapping MSIX BAR"
https://www.spinics.net/lists/kvm/msg160282.html
---
This is an RFC as it requires kernel headers update which is not there yet.
I'd like to make it "msix-mmap" (without "no") but could not find a way of enabling a device property for machine versions newer than some value.
I changed 2.11 machine just for the demonstration purpose.
--- hw/vfio/pci.h | 1 + include/hw/vfio/vfio-common.h | 1 + linux-headers/linux/vfio.h | 5 +++++ hw/ppc/spapr.c | 10 +++++++++- hw/vfio/common.c | 15 +++++++++++++++ hw/vfio/pci.c | 11 +++++++++++ 6 files changed, 42 insertions(+), 1 deletion(-)
hw/vfio/pci.h +1 -0
@@ -142,6 +142,7 @@ typedef struct VFIOPCIDevice {bool no_kvm_intx;bool no_kvm_msi;bool no_kvm_msix; +bool msix_no_mmap; } VFIOPCIDevice;uint32_t vfio_pci_read_config (PCIDevice *pdev,uint32_t addr,int len);
include/hw/vfio/vfio-common.h +1 -0
@@ -171,6 +171,7 @@ int vfio_get_region_info(VFIODevice *vbasedev, int index, struct vfio_region_info **info);int vfio_get_dev_region_info (VFIODevice *vbasedev,uint32_t type,uint32_t subtype,struct vfio_region_info **info); +bool vfio_is_cap_present (VFIODevice *vbasedev,uint16_t cap_type,int region);#endif extern const MemoryListener vfio_prereg_listener;
linux-headers/linux/vfio.h +5 -0
@@ -300,6 +300,11 @@ struct vfio_region_info_cap_type {#define VFIO_REGION_SUBTYPE_INTEL_IGD_HOST_CFG (2)#define VFIO_REGION_SUBTYPE_INTEL_IGD_LPC_CFG (3) +/* +* The MSIX mappable capability informs that MSIX data of a BAR can be mmapped. +*/ +#define VFIO_REGION_INFO_CAP_MSIX_MAPPABLE 3 +/** * VFIO_DEVICE_GET_IRQ_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 9, * struct vfio_irq_info)
hw/ppc/spapr.c +9 -1
@@ -3742,13 +3742,21 @@ static const TypeInfo spapr_machine_info = {/* * pseries-2.11 */ +#define SPAPR_COMPAT_2_11 \ + HW_COMPAT_2_10 \ + { \ + .driver = "vfio-pci", \ + .property = "msix-no-mmap", \ + .value = "on", \ + }, \ +static void spapr_machine_2_11_instance_options (MachineState *machine) { }static void spapr_machine_2_11_class_options (MachineClass *mc) { -/* Defaults for the latest behaviour inherited from the base class */ +SET_MACHINE_COMPAT (mc,SPAPR_COMPAT_2_11 ); } DEFINE_SPAPR_MACHINE(2_11, "2.11", true);
hw/vfio/common.c +15 -0
@@ -1408,6 +1408,21 @@ int vfio_get_dev_region_info(VFIODevice *vbasedev, uint32_t type,return -ENODEV ; } +bool vfio_is_cap_present (VFIODevice *vbasedev,uint16_t cap_type,int region) +{ +struct vfio_region_info *info =NULL ; +bool ret = false; + +if (!vfio_get_region_info (vbasedev, region, &info)) { +if (vfio_get_region_info_cap (info, cap_type)) { + ret = true; + } +g_free (info); + } + +return ret; +} +/* * Interfaces for IBM EEH (Enhanced Error Handling) */
hw/vfio/pci.c +11 -0
@@ -1289,6 +1289,12 @@ static void vfio_pci_fixup_msix_region(VFIOPCIDevice *vdev)off_t start, end;VFIORegion *region = &vdev->bars[vdev->msix->table_bar].region; +if (!vdev->msix_no_mmap && +vfio_is_cap_present (&vdev->vbasedev,VFIO_REGION_INFO_CAP_MSIX_MAPPABLE , + vdev->msix->table_bar)) { +return ; + } +/* * We expect to find a single mmap covering the whole BAR, anything else * means it's either unsupported or already setup.
@@ -1473,6 +1479,10 @@ static int vfio_msix_setup(VFIOPCIDevice *vdev, int pos, Error **errp)*/ memory_region_set_enabled (&vdev->pdev.msix_pba_mmio, false); +if (!vdev->msix_no_mmap) { +memory_region_set_enabled (&vdev->pdev.msix_table_mmio, false); + } +return 0 ; }
@@ -2986,6 +2996,7 @@ static Property vfio_pci_dev_properties[] = {DEFINE_PROP_BIT ("x-igd-opregion" , VFIOPCIDevice, features,VFIO_FEATURE_ENABLE_IGD_OPREGION_BIT , false),DEFINE_PROP_BOOL ("x-no-mmap" , VFIOPCIDevice, vbasedev.no_mmap, false), +DEFINE_PROP_BOOL ("msix-no-mmap" , VFIOPCIDevice, msix_no_mmap, true),DEFINE_PROP_BOOL ("x-no-kvm-intx" , VFIOPCIDevice, no_kvm_intx, false),DEFINE_PROP_BOOL ("x-no-kvm-msi" , VFIOPCIDevice, no_kvm_msi, false), DEFINE_PROP_BOOL("x-no-kvm-msix", VFIOPCIDevice, no_kvm_msix, false),
Signature · 1 line
2.11.0
Quoted 10 lines · Alexey Kardashevskiy
This makes use of a new
VFIO_REGION_INFO_CAP_MSIX_MAPPABLEcapability which tells that a region with MSIX data can be mapped entirely, i.e. the VFIO PCI driver won't prevent MSIX vectors area from being mapped.This adds a "msix-no-mmap" property to the vfio-pci device, it is "true" by default and "false" for pseries-2.12+ machines.
This requites kernel's "vfio-pci: Allow mapping MSIX BAR"
https://www.spinics.net/lists/kvm/msg160282.html
Quoted 7 lines
Signed-off-by Alexey Kardashevskiy <[email protected]>---
This is an RFC as it requires kernel headers update which is not there yet.
I'd like to make it "msix-mmap" (without "no") but could not find a way of enabling a device property for machine versions newer than some value.
Ah, this remark is wrong, making it "no" property does not help.
How do we enforce some property on some device depending on a machine type?
Quoted 143 lines
I changed 2.11 machine just for the demonstration purpose.
--- hw/vfio/pci.h | 1 + include/hw/vfio/vfio-common.h | 1 + linux-headers/linux/vfio.h | 5 +++++ hw/ppc/spapr.c | 10 +++++++++- hw/vfio/common.c | 15 +++++++++++++++ hw/vfio/pci.c | 11 +++++++++++ 6 files changed, 42 insertions(+), 1 deletion(-)hw/vfio/pci.h +1
-0@@ -142,6 +142,7 @@ typedef struct VFIOPCIDevice {bool no_kvm_intx;bool no_kvm_msi;bool no_kvm_msix; +bool msix_no_mmap; } VFIOPCIDevice;uint32_t vfio_pci_read_config (PCIDevice *pdev,uint32_t addr,int len);include/hw/vfio/vfio-common.h +1
-0@@ -171,6 +171,7 @@ int vfio_get_region_info(VFIODevice *vbasedev, int index, struct vfio_region_info **info);int vfio_get_dev_region_info (VFIODevice *vbasedev,uint32_t type,uint32_t subtype,struct vfio_region_info **info); +bool vfio_is_cap_present (VFIODevice *vbasedev,uint16_t cap_type,int region);#endif extern const MemoryListener vfio_prereg_listener;linux-headers/linux/vfio.h +5
-0@@ -300,6 +300,11 @@ struct vfio_region_info_cap_type {#define VFIO_REGION_SUBTYPE_INTEL_IGD_HOST_CFG (2)#define VFIO_REGION_SUBTYPE_INTEL_IGD_LPC_CFG (3) +/* +* The MSIX mappable capability informs that MSIX data of a BAR can be mmapped. +*/ +#define VFIO_REGION_INFO_CAP_MSIX_MAPPABLE 3 +/** * VFIO_DEVICE_GET_IRQ_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 9, * struct vfio_irq_info) hw/ppc/spapr.c +9
-1@@ -3742,13 +3742,21 @@ static const TypeInfo spapr_machine_info = {/* * pseries-2.11 */ +#define SPAPR_COMPAT_2_11 \ + HW_COMPAT_2_10 \ + { \ + .driver = "vfio-pci", \ + .property = "msix-no-mmap", \ + .value = "on", \ + }, \ +static void spapr_machine_2_11_instance_options (MachineState *machine) { }static void spapr_machine_2_11_class_options (MachineClass *mc) { -/* Defaults for the latest behaviour inherited from the base class */ +SET_MACHINE_COMPAT (mc,SPAPR_COMPAT_2_11 ); } DEFINE_SPAPR_MACHINE(2_11, "2.11", true);hw/vfio/common.c +15
-0@@ -1408,6 +1408,21 @@ int vfio_get_dev_region_info(VFIODevice *vbasedev, uint32_t type,return -ENODEV ; } +bool vfio_is_cap_present (VFIODevice *vbasedev,uint16_t cap_type,int region) +{ +struct vfio_region_info *info =NULL ; +bool ret = false; + +if (!vfio_get_region_info (vbasedev, region, &info)) { +if (vfio_get_region_info_cap (info, cap_type)) { + ret = true; + } +g_free (info); + } + +return ret; +} +/* * Interfaces for IBM EEH (Enhanced Error Handling) */ hw/vfio/pci.c +11
-0@@ -1289,6 +1289,12 @@ static void vfio_pci_fixup_msix_region(VFIOPCIDevice *vdev)off_t start, end;VFIORegion *region = &vdev->bars[vdev->msix->table_bar].region; +if (!vdev->msix_no_mmap && +vfio_is_cap_present (&vdev->vbasedev,VFIO_REGION_INFO_CAP_MSIX_MAPPABLE , + vdev->msix->table_bar)) { +return ; + } +/* * We expect to find a single mmap covering the whole BAR, anything else * means it's either unsupported or already setup. @@ -1473,6 +1479,10 @@ static int vfio_msix_setup(VFIOPCIDevice *vdev, int pos, Error **errp)*/ memory_region_set_enabled (&vdev->pdev.msix_pba_mmio, false); +if (!vdev->msix_no_mmap) { +memory_region_set_enabled (&vdev->pdev.msix_table_mmio, false); + } +return 0 ; }@@ -2986,6 +2996,7 @@ static Property vfio_pci_dev_properties[] = {DEFINE_PROP_BIT ("x-igd-opregion" , VFIOPCIDevice, features,VFIO_FEATURE_ENABLE_IGD_OPREGION_BIT , false),DEFINE_PROP_BOOL ("x-no-mmap" , VFIOPCIDevice, vbasedev.no_mmap, false), +DEFINE_PROP_BOOL ("msix-no-mmap" , VFIOPCIDevice, msix_no_mmap, true),DEFINE_PROP_BOOL ("x-no-kvm-intx" , VFIOPCIDevice, no_kvm_intx, false),DEFINE_PROP_BOOL ("x-no-kvm-msi" , VFIOPCIDevice, no_kvm_msi, false), DEFINE_PROP_BOOL("x-no-kvm-msix", VFIOPCIDevice, no_kvm_msix, false),
Signature · 1 line
Alexey
Quoted 141 lines · On Tue, 12 Dec 2017 16:21:31 +1100 Alexey Kardashevskiy <[email protected]> wrote:
This makes use of a new
VFIO_REGION_INFO_CAP_MSIX_MAPPABLEcapability which tells that a region with MSIX data can be mapped entirely, i.e. the VFIO PCI driver won't prevent MSIX vectors area from being mapped.This adds a "msix-no-mmap" property to the vfio-pci device, it is "true" by default and "false" for pseries-2.12+ machines.
This requites kernel's "vfio-pci: Allow mapping MSIX BAR"
https://www.spinics.net/lists/kvm/msg160282.htmlSigned-off-by Alexey Kardashevskiy <[email protected]>---
This is an RFC as it requires kernel headers update which is not there yet.
I'd like to make it "msix-mmap" (without "no") but could not find a way of enabling a device property for machine versions newer than some value.
I changed 2.11 machine just for the demonstration purpose.
--- hw/vfio/pci.h | 1 + include/hw/vfio/vfio-common.h | 1 + linux-headers/linux/vfio.h | 5 +++++ hw/ppc/spapr.c | 10 +++++++++- hw/vfio/common.c | 15 +++++++++++++++ hw/vfio/pci.c | 11 +++++++++++ 6 files changed, 42 insertions(+), 1 deletion(-)hw/vfio/pci.h +1
-0@@ -142,6 +142,7 @@ typedef struct VFIOPCIDevice {bool no_kvm_intx;bool no_kvm_msi;bool no_kvm_msix; +bool msix_no_mmap; } VFIOPCIDevice;uint32_t vfio_pci_read_config (PCIDevice *pdev,uint32_t addr,int len);include/hw/vfio/vfio-common.h +1
-0@@ -171,6 +171,7 @@ int vfio_get_region_info(VFIODevice *vbasedev, int index, struct vfio_region_info **info);int vfio_get_dev_region_info (VFIODevice *vbasedev,uint32_t type,uint32_t subtype,struct vfio_region_info **info); +bool vfio_is_cap_present (VFIODevice *vbasedev,uint16_t cap_type,int region);#endif extern const MemoryListener vfio_prereg_listener;linux-headers/linux/vfio.h +5
-0@@ -300,6 +300,11 @@ struct vfio_region_info_cap_type {#define VFIO_REGION_SUBTYPE_INTEL_IGD_HOST_CFG (2)#define VFIO_REGION_SUBTYPE_INTEL_IGD_LPC_CFG (3) +/* +* The MSIX mappable capability informs that MSIX data of a BAR can be mmapped. +*/ +#define VFIO_REGION_INFO_CAP_MSIX_MAPPABLE 3 +/** * VFIO_DEVICE_GET_IRQ_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 9, * struct vfio_irq_info) hw/ppc/spapr.c +9
-1@@ -3742,13 +3742,21 @@ static const TypeInfo spapr_machine_info = {/* * pseries-2.11 */ +#define SPAPR_COMPAT_2_11 \ + HW_COMPAT_2_10 \ + { \ + .driver = "vfio-pci", \ + .property = "msix-no-mmap", \ + .value = "on", \ + }, \ +static void spapr_machine_2_11_instance_options (MachineState *machine) { }static void spapr_machine_2_11_class_options (MachineClass *mc) { -/* Defaults for the latest behaviour inherited from the base class */ +SET_MACHINE_COMPAT (mc,SPAPR_COMPAT_2_11 ); } DEFINE_SPAPR_MACHINE(2_11, "2.11", true);hw/vfio/common.c +15
-0@@ -1408,6 +1408,21 @@ int vfio_get_dev_region_info(VFIODevice *vbasedev, uint32_t type,return -ENODEV ; } +bool vfio_is_cap_present (VFIODevice *vbasedev,uint16_t cap_type,int region) +{ +struct vfio_region_info *info =NULL ; +bool ret = false; + +if (!vfio_get_region_info (vbasedev, region, &info)) { +if (vfio_get_region_info_cap (info, cap_type)) { + ret = true; + } +g_free (info); + } + +return ret; +} +/* * Interfaces for IBM EEH (Enhanced Error Handling) */ hw/vfio/pci.c +6
-0@@ -1289,6 +1289,12 @@ static void vfio_pci_fixup_msix_region(VFIOPCIDevice *vdev)off_t start, end;VFIORegion *region = &vdev->bars[vdev->msix->table_bar].region; +if (!vdev->msix_no_mmap && +vfio_is_cap_present (&vdev->vbasedev,VFIO_REGION_INFO_CAP_MSIX_MAPPABLE , + vdev->msix->table_bar)) { +return ; + } +/* * We expect to find a single mmap covering the whole BAR, anything else * means it's either unsupported or already setup.
hw/vfio/pci.c static int vfio_msix_setup(VFIOPCIDevice *vdev, int pos, Error **errp) · line 1484
Quoted 6 lines
hw/vfio/pci.c +3
-0@@ -1473,6 +1479,10 @@ static int vfio_msix_setup(VFIOPCIDevice *vdev, int pos, Error **errp)*/ memory_region_set_enabled (&vdev->pdev.msix_pba_mmio, false); +if (!vdev->msix_no_mmap) { +memory_region_set_enabled (&vdev->pdev.msix_table_mmio, false); + }
No, you're conflating issues. There's (a) can we mmap over the MSI-X vector table and (b) do we require QEMU emulation of the MSI-X vector table. (a) does NOT imply (b). AFAICT, (a) should be enabled any time the kernel supports it, (b) should never be enabled on ppc, regardless of (a). Thanks,
Alex
Quoted 12 lines
hw/vfio/pci.c +2
-0+return 0 ; }@@ -2986,6 +2996,7 @@ static Property vfio_pci_dev_properties[] = {DEFINE_PROP_BIT ("x-igd-opregion" , VFIOPCIDevice, features,VFIO_FEATURE_ENABLE_IGD_OPREGION_BIT , false),DEFINE_PROP_BOOL ("x-no-mmap" , VFIOPCIDevice, vbasedev.no_mmap, false), +DEFINE_PROP_BOOL ("msix-no-mmap" , VFIOPCIDevice, msix_no_mmap, true),DEFINE_PROP_BOOL ("x-no-kvm-intx" , VFIOPCIDevice, no_kvm_intx, false),DEFINE_PROP_BOOL ("x-no-kvm-msi" , VFIOPCIDevice, no_kvm_msi, false), DEFINE_PROP_BOOL("x-no-kvm-msix", VFIOPCIDevice, no_kvm_msix, false),
Quoted 147 lines · Alex Williamson
Quoted 144 lines · On Tue, 12 Dec 2017 16:21:31 +1100 Alexey Kardashevskiy <[email protected]> wrote:
This makes use of a new
VFIO_REGION_INFO_CAP_MSIX_MAPPABLEcapability which tells that a region with MSIX data can be mapped entirely, i.e. the VFIO PCI driver won't prevent MSIX vectors area from being mapped.This adds a "msix-no-mmap" property to the vfio-pci device, it is "true" by default and "false" for pseries-2.12+ machines.
This requites kernel's "vfio-pci: Allow mapping MSIX BAR"
https://www.spinics.net/lists/kvm/msg160282.htmlSigned-off-by Alexey Kardashevskiy <[email protected]>---
This is an RFC as it requires kernel headers update which is not there yet.
I'd like to make it "msix-mmap" (without "no") but could not find a way of enabling a device property for machine versions newer than some value.
I changed 2.11 machine just for the demonstration purpose.
--- hw/vfio/pci.h | 1 + include/hw/vfio/vfio-common.h | 1 + linux-headers/linux/vfio.h | 5 +++++ hw/ppc/spapr.c | 10 +++++++++- hw/vfio/common.c | 15 +++++++++++++++ hw/vfio/pci.c | 11 +++++++++++ 6 files changed, 42 insertions(+), 1 deletion(-)hw/vfio/pci.h +1
-0@@ -142,6 +142,7 @@ typedef struct VFIOPCIDevice {bool no_kvm_intx;bool no_kvm_msi;bool no_kvm_msix; +bool msix_no_mmap; } VFIOPCIDevice;uint32_t vfio_pci_read_config (PCIDevice *pdev,uint32_t addr,int len);include/hw/vfio/vfio-common.h +1
-0@@ -171,6 +171,7 @@ int vfio_get_region_info(VFIODevice *vbasedev, int index, struct vfio_region_info **info);int vfio_get_dev_region_info (VFIODevice *vbasedev,uint32_t type,uint32_t subtype,struct vfio_region_info **info); +bool vfio_is_cap_present (VFIODevice *vbasedev,uint16_t cap_type,int region);#endif extern const MemoryListener vfio_prereg_listener;linux-headers/linux/vfio.h +5
-0@@ -300,6 +300,11 @@ struct vfio_region_info_cap_type {#define VFIO_REGION_SUBTYPE_INTEL_IGD_HOST_CFG (2)#define VFIO_REGION_SUBTYPE_INTEL_IGD_LPC_CFG (3) +/* +* The MSIX mappable capability informs that MSIX data of a BAR can be mmapped. +*/ +#define VFIO_REGION_INFO_CAP_MSIX_MAPPABLE 3 +/** * VFIO_DEVICE_GET_IRQ_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 9, * struct vfio_irq_info) hw/ppc/spapr.c +9
-1@@ -3742,13 +3742,21 @@ static const TypeInfo spapr_machine_info = {/* * pseries-2.11 */ +#define SPAPR_COMPAT_2_11 \ + HW_COMPAT_2_10 \ + { \ + .driver = "vfio-pci", \ + .property = "msix-no-mmap", \ + .value = "on", \ + }, \ +static void spapr_machine_2_11_instance_options (MachineState *machine) { }static void spapr_machine_2_11_class_options (MachineClass *mc) { -/* Defaults for the latest behaviour inherited from the base class */ +SET_MACHINE_COMPAT (mc,SPAPR_COMPAT_2_11 ); } DEFINE_SPAPR_MACHINE(2_11, "2.11", true);hw/vfio/common.c +15
-0@@ -1408,6 +1408,21 @@ int vfio_get_dev_region_info(VFIODevice *vbasedev, uint32_t type,return -ENODEV ; } +bool vfio_is_cap_present (VFIODevice *vbasedev,uint16_t cap_type,int region) +{ +struct vfio_region_info *info =NULL ; +bool ret = false; + +if (!vfio_get_region_info (vbasedev, region, &info)) { +if (vfio_get_region_info_cap (info, cap_type)) { + ret = true; + } +g_free (info); + } + +return ret; +} +/* * Interfaces for IBM EEH (Enhanced Error Handling) */ hw/vfio/pci.c +6
-0@@ -1289,6 +1289,12 @@ static void vfio_pci_fixup_msix_region(VFIOPCIDevice *vdev)off_t start, end;VFIORegion *region = &vdev->bars[vdev->msix->table_bar].region; +if (!vdev->msix_no_mmap && +vfio_is_cap_present (&vdev->vbasedev,VFIO_REGION_INFO_CAP_MSIX_MAPPABLE , + vdev->msix->table_bar)) { +return ; + } +/* * We expect to find a single mmap covering the whole BAR, anything else * means it's either unsupported or already setup. @@ -1473,6 +1479,10 @@ static int vfio_msix_setup(VFIOPCIDevice *vdev, int pos, Error **errp)*/ memory_region_set_enabled (&vdev->pdev.msix_pba_mmio, false);
Quoted 8 lines
hw/vfio/pci.c static int vfio_msix_setup(VFIOPCIDevice *vdev, int pos, Error **errp) · line 1484
Quoted 3 lines
hw/vfio/pci.c +3
-0+if (!vdev->msix_no_mmap) { +memory_region_set_enabled (&vdev->pdev.msix_table_mmio, false); + }No, you're conflating issues. There's (a) can we mmap over the MSI-X vector table and (b) do we require QEMU emulation of the MSI-X vector table. (a) does NOT imply (b). AFAICT, (a) should be enabled any time the kernel supports it,
This is the default setting, or you do not want it to be a property so the user cannot shoot himself in a foot?
Quoted 2 lines
(b) should never be enabled on ppc, regardless
of (a). Thanks,
The intention is to have a property - msix_no_mmap=true, except a single case - ppc-pseries, I just do not know how to enforce it for a specific machine type. It could also be a machine property, like "kernel_irqchip".
Quoted 16 lines
Alex
Quoted 12 lines
hw/vfio/pci.c +2
-0+return 0 ; }@@ -2986,6 +2996,7 @@ static Property vfio_pci_dev_properties[] = {DEFINE_PROP_BIT ("x-igd-opregion" , VFIOPCIDevice, features,VFIO_FEATURE_ENABLE_IGD_OPREGION_BIT , false),DEFINE_PROP_BOOL ("x-no-mmap" , VFIOPCIDevice, vbasedev.no_mmap, false), +DEFINE_PROP_BOOL ("msix-no-mmap" , VFIOPCIDevice, msix_no_mmap, true),DEFINE_PROP_BOOL ("x-no-kvm-intx" , VFIOPCIDevice, no_kvm_intx, false),DEFINE_PROP_BOOL ("x-no-kvm-msi" , VFIOPCIDevice, no_kvm_msi, false), DEFINE_PROP_BOOL("x-no-kvm-msix", VFIOPCIDevice, no_kvm_msix, false),
Signature · 1 line
Alexey
Quoted 160 lines · Alexey Kardashevskiy
Quoted 147 lines · Alex Williamson
Quoted 144 lines · On Tue, 12 Dec 2017 16:21:31 +1100 Alexey Kardashevskiy <[email protected]> wrote:
This makes use of a new
VFIO_REGION_INFO_CAP_MSIX_MAPPABLEcapability which tells that a region with MSIX data can be mapped entirely, i.e. the VFIO PCI driver won't prevent MSIX vectors area from being mapped.This adds a "msix-no-mmap" property to the vfio-pci device, it is "true" by default and "false" for pseries-2.12+ machines.
This requites kernel's "vfio-pci: Allow mapping MSIX BAR"
https://www.spinics.net/lists/kvm/msg160282.htmlSigned-off-by Alexey Kardashevskiy <[email protected]>---
This is an RFC as it requires kernel headers update which is not there yet.
I'd like to make it "msix-mmap" (without "no") but could not find a way of enabling a device property for machine versions newer than some value.
I changed 2.11 machine just for the demonstration purpose.
--- hw/vfio/pci.h | 1 + include/hw/vfio/vfio-common.h | 1 + linux-headers/linux/vfio.h | 5 +++++ hw/ppc/spapr.c | 10 +++++++++- hw/vfio/common.c | 15 +++++++++++++++ hw/vfio/pci.c | 11 +++++++++++ 6 files changed, 42 insertions(+), 1 deletion(-)hw/vfio/pci.h +1
-0@@ -142,6 +142,7 @@ typedef struct VFIOPCIDevice {bool no_kvm_intx;bool no_kvm_msi;bool no_kvm_msix; +bool msix_no_mmap; } VFIOPCIDevice;uint32_t vfio_pci_read_config (PCIDevice *pdev,uint32_t addr,int len);include/hw/vfio/vfio-common.h +1
-0@@ -171,6 +171,7 @@ int vfio_get_region_info(VFIODevice *vbasedev, int index, struct vfio_region_info **info);int vfio_get_dev_region_info (VFIODevice *vbasedev,uint32_t type,uint32_t subtype,struct vfio_region_info **info); +bool vfio_is_cap_present (VFIODevice *vbasedev,uint16_t cap_type,int region);#endif extern const MemoryListener vfio_prereg_listener;linux-headers/linux/vfio.h +5
-0@@ -300,6 +300,11 @@ struct vfio_region_info_cap_type {#define VFIO_REGION_SUBTYPE_INTEL_IGD_HOST_CFG (2)#define VFIO_REGION_SUBTYPE_INTEL_IGD_LPC_CFG (3) +/* +* The MSIX mappable capability informs that MSIX data of a BAR can be mmapped. +*/ +#define VFIO_REGION_INFO_CAP_MSIX_MAPPABLE 3 +/** * VFIO_DEVICE_GET_IRQ_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 9, * struct vfio_irq_info) hw/ppc/spapr.c +9
-1@@ -3742,13 +3742,21 @@ static const TypeInfo spapr_machine_info = {/* * pseries-2.11 */ +#define SPAPR_COMPAT_2_11 \ + HW_COMPAT_2_10 \ + { \ + .driver = "vfio-pci", \ + .property = "msix-no-mmap", \ + .value = "on", \ + }, \ +static void spapr_machine_2_11_instance_options (MachineState *machine) { }static void spapr_machine_2_11_class_options (MachineClass *mc) { -/* Defaults for the latest behaviour inherited from the base class */ +SET_MACHINE_COMPAT (mc,SPAPR_COMPAT_2_11 ); } DEFINE_SPAPR_MACHINE(2_11, "2.11", true);hw/vfio/common.c +15
-0@@ -1408,6 +1408,21 @@ int vfio_get_dev_region_info(VFIODevice *vbasedev, uint32_t type,return -ENODEV ; } +bool vfio_is_cap_present (VFIODevice *vbasedev,uint16_t cap_type,int region) +{ +struct vfio_region_info *info =NULL ; +bool ret = false; + +if (!vfio_get_region_info (vbasedev, region, &info)) { +if (vfio_get_region_info_cap (info, cap_type)) { + ret = true; + } +g_free (info); + } + +return ret; +} +/* * Interfaces for IBM EEH (Enhanced Error Handling) */ hw/vfio/pci.c +6
-0@@ -1289,6 +1289,12 @@ static void vfio_pci_fixup_msix_region(VFIOPCIDevice *vdev)off_t start, end;VFIORegion *region = &vdev->bars[vdev->msix->table_bar].region; +if (!vdev->msix_no_mmap && +vfio_is_cap_present (&vdev->vbasedev,VFIO_REGION_INFO_CAP_MSIX_MAPPABLE , + vdev->msix->table_bar)) { +return ; + } +/* * We expect to find a single mmap covering the whole BAR, anything else * means it's either unsupported or already setup. @@ -1473,6 +1479,10 @@ static int vfio_msix_setup(VFIOPCIDevice *vdev, int pos, Error **errp)*/ memory_region_set_enabled (&vdev->pdev.msix_pba_mmio, false);Quoted 8 lines
hw/vfio/pci.c static int vfio_msix_setup(VFIOPCIDevice *vdev, int pos, Error **errp) · line 1484
Quoted 3 lines
hw/vfio/pci.c +3
-0+if (!vdev->msix_no_mmap) { +memory_region_set_enabled (&vdev->pdev.msix_table_mmio, false); + }No, you're conflating issues. There's (a) can we mmap over the MSI-X vector table and (b) do we require QEMU emulation of the MSI-X vector table. (a) does NOT imply (b). AFAICT, (a) should be enabled any time the kernel supports it,
This is the default setting, or you do not want it to be a property so the user cannot shoot himself in a foot?
Quoted 7 lines
Quoted 2 lines
(b) should never be enabled on ppc, regardless
of (a). Thanks,The intention is to have a property - msix_no_mmap=true, except a single case - ppc-pseries, I just do not know how to enforce it for a specific machine type.
I do know now. For example,
static void spapr_machine_2_11_instance_options(MachineState *machine)
{
register_compat_prop("vfio-pci", "msix-no-mmap", "off");
}which is basically "-global vfio-pci.msix-no-mmap=off" but in the code.
Quoted 22 lines
It could also be a machine property, like "
kernel_irqchip".Quoted 16 lines
Alex
Quoted 12 lines
hw/vfio/pci.c +2
-0+return 0 ; }@@ -2986,6 +2996,7 @@ static Property vfio_pci_dev_properties[] = {DEFINE_PROP_BIT ("x-igd-opregion" , VFIOPCIDevice, features,VFIO_FEATURE_ENABLE_IGD_OPREGION_BIT , false),DEFINE_PROP_BOOL ("x-no-mmap" , VFIOPCIDevice, vbasedev.no_mmap, false), +DEFINE_PROP_BOOL ("msix-no-mmap" , VFIOPCIDevice, msix_no_mmap, true),DEFINE_PROP_BOOL ("x-no-kvm-intx" , VFIOPCIDevice, no_kvm_intx, false),DEFINE_PROP_BOOL ("x-no-kvm-msi" , VFIOPCIDevice, no_kvm_msi, false), DEFINE_PROP_BOOL("x-no-kvm-msix", VFIOPCIDevice, no_kvm_msix, false),
Signature · 1 line
Alexey
Quoted 158 lines · On Tue, 12 Dec 2017 18:01:40 +1100 Alexey Kardashevskiy <[email protected]> wrote:
Quoted 157 lines · Alexey Kardashevskiy
Quoted 155 lines · Alex Williamson
Quoted 141 lines · On Tue, 12 Dec 2017 16:21:31 +1100 Alexey Kardashevskiy <[email protected]> wrote:
This makes use of a new
VFIO_REGION_INFO_CAP_MSIX_MAPPABLEcapability which tells that a region with MSIX data can be mapped entirely, i.e. the VFIO PCI driver won't prevent MSIX vectors area from being mapped.This adds a "msix-no-mmap" property to the vfio-pci device, it is "true" by default and "false" for pseries-2.12+ machines.
This requites kernel's "vfio-pci: Allow mapping MSIX BAR"
https://www.spinics.net/lists/kvm/msg160282.htmlSigned-off-by Alexey Kardashevskiy <[email protected]>---
This is an RFC as it requires kernel headers update which is not there yet.
I'd like to make it "msix-mmap" (without "no") but could not find a way of enabling a device property for machine versions newer than some value.
I changed 2.11 machine just for the demonstration purpose.
--- hw/vfio/pci.h | 1 + include/hw/vfio/vfio-common.h | 1 + linux-headers/linux/vfio.h | 5 +++++ hw/ppc/spapr.c | 10 +++++++++- hw/vfio/common.c | 15 +++++++++++++++ hw/vfio/pci.c | 11 +++++++++++ 6 files changed, 42 insertions(+), 1 deletion(-)hw/vfio/pci.h +1
-0@@ -142,6 +142,7 @@ typedef struct VFIOPCIDevice {bool no_kvm_intx;bool no_kvm_msi;bool no_kvm_msix; +bool msix_no_mmap; } VFIOPCIDevice;uint32_t vfio_pci_read_config (PCIDevice *pdev,uint32_t addr,int len);include/hw/vfio/vfio-common.h +1
-0@@ -171,6 +171,7 @@ int vfio_get_region_info(VFIODevice *vbasedev, int index, struct vfio_region_info **info);int vfio_get_dev_region_info (VFIODevice *vbasedev,uint32_t type,uint32_t subtype,struct vfio_region_info **info); +bool vfio_is_cap_present (VFIODevice *vbasedev,uint16_t cap_type,int region);#endif extern const MemoryListener vfio_prereg_listener;linux-headers/linux/vfio.h +5
-0@@ -300,6 +300,11 @@ struct vfio_region_info_cap_type {#define VFIO_REGION_SUBTYPE_INTEL_IGD_HOST_CFG (2)#define VFIO_REGION_SUBTYPE_INTEL_IGD_LPC_CFG (3) +/* +* The MSIX mappable capability informs that MSIX data of a BAR can be mmapped. +*/ +#define VFIO_REGION_INFO_CAP_MSIX_MAPPABLE 3 +/** * VFIO_DEVICE_GET_IRQ_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 9, * struct vfio_irq_info) hw/ppc/spapr.c +9
-1@@ -3742,13 +3742,21 @@ static const TypeInfo spapr_machine_info = {/* * pseries-2.11 */ +#define SPAPR_COMPAT_2_11 \ + HW_COMPAT_2_10 \ + { \ + .driver = "vfio-pci", \ + .property = "msix-no-mmap", \ + .value = "on", \ + }, \ +static void spapr_machine_2_11_instance_options (MachineState *machine) { }static void spapr_machine_2_11_class_options (MachineClass *mc) { -/* Defaults for the latest behaviour inherited from the base class */ +SET_MACHINE_COMPAT (mc,SPAPR_COMPAT_2_11 ); } DEFINE_SPAPR_MACHINE(2_11, "2.11", true);hw/vfio/common.c +15
-0@@ -1408,6 +1408,21 @@ int vfio_get_dev_region_info(VFIODevice *vbasedev, uint32_t type,return -ENODEV ; } +bool vfio_is_cap_present (VFIODevice *vbasedev,uint16_t cap_type,int region) +{ +struct vfio_region_info *info =NULL ; +bool ret = false; + +if (!vfio_get_region_info (vbasedev, region, &info)) { +if (vfio_get_region_info_cap (info, cap_type)) { + ret = true; + } +g_free (info); + } + +return ret; +} +/* * Interfaces for IBM EEH (Enhanced Error Handling) */ hw/vfio/pci.c +6
-0@@ -1289,6 +1289,12 @@ static void vfio_pci_fixup_msix_region(VFIOPCIDevice *vdev)off_t start, end;VFIORegion *region = &vdev->bars[vdev->msix->table_bar].region; +if (!vdev->msix_no_mmap && +vfio_is_cap_present (&vdev->vbasedev,VFIO_REGION_INFO_CAP_MSIX_MAPPABLE , + vdev->msix->table_bar)) { +return ; + } +/* * We expect to find a single mmap covering the whole BAR, anything else * means it's either unsupported or already setup. hw/vfio/pci.c static int vfio_msix_setup(VFIOPCIDevice *vdev, int pos, Error **errp) · line 1484
Quoted 6 lines
hw/vfio/pci.c +3
-0@@ -1473,6 +1479,10 @@ static int vfio_msix_setup(VFIOPCIDevice *vdev, int pos, Error **errp)*/ memory_region_set_enabled (&vdev->pdev.msix_pba_mmio, false); +if (!vdev->msix_no_mmap) { +memory_region_set_enabled (&vdev->pdev.msix_table_mmio, false); + }No, you're conflating issues. There's (a) can we mmap over the MSI-X vector table and (b) do we require QEMU emulation of the MSI-X vector table. (a) does NOT imply (b). AFAICT, (a) should be enabled any time the kernel supports it,
Quoted 2 lines
Quoted 2 lines
This is the default setting, or you do not want it to be a property so the user cannot shoot himself in a foot?
If the kernel allows mmap, other than debugging, why would it ever need to be disabled?
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
(b) should never be enabled on ppc, regardless
of (a). Thanks,The intention is to have a property - msix_no_mmap=true, except a single case - ppc-pseries, I just do not know how to enforce it for a specific machine type.
The intention is wrong. (a) should be done any time the kernel allows it. (b) is an independent concept of disabling QEMU MSI-X emulation for platforms, ie. machine types, that do not require it. (b) has nothing to do with the mmap'ability of the msix table area. So far (b) includes only the ppc spapr machine and I don't see a reason to allow the user to control this. Thanks,
Alex
Quoted 176 lines · Alex Williamson
Quoted 158 lines · On Tue, 12 Dec 2017 18:01:40 +1100 Alexey Kardashevskiy <[email protected]> wrote:
Quoted 157 lines · Alexey Kardashevskiy
Quoted 155 lines · Alex Williamson
Quoted 141 lines · On Tue, 12 Dec 2017 16:21:31 +1100 Alexey Kardashevskiy <[email protected]> wrote:
This makes use of a new
VFIO_REGION_INFO_CAP_MSIX_MAPPABLEcapability which tells that a region with MSIX data can be mapped entirely, i.e. the VFIO PCI driver won't prevent MSIX vectors area from being mapped.This adds a "msix-no-mmap" property to the vfio-pci device, it is "true" by default and "false" for pseries-2.12+ machines.
This requites kernel's "vfio-pci: Allow mapping MSIX BAR"
https://www.spinics.net/lists/kvm/msg160282.htmlSigned-off-by Alexey Kardashevskiy <[email protected]>---
This is an RFC as it requires kernel headers update which is not there yet.
I'd like to make it "msix-mmap" (without "no") but could not find a way of enabling a device property for machine versions newer than some value.
I changed 2.11 machine just for the demonstration purpose.
--- hw/vfio/pci.h | 1 + include/hw/vfio/vfio-common.h | 1 + linux-headers/linux/vfio.h | 5 +++++ hw/ppc/spapr.c | 10 +++++++++- hw/vfio/common.c | 15 +++++++++++++++ hw/vfio/pci.c | 11 +++++++++++ 6 files changed, 42 insertions(+), 1 deletion(-)hw/vfio/pci.h +1
-0@@ -142,6 +142,7 @@ typedef struct VFIOPCIDevice {bool no_kvm_intx;bool no_kvm_msi;bool no_kvm_msix; +bool msix_no_mmap; } VFIOPCIDevice;uint32_t vfio_pci_read_config (PCIDevice *pdev,uint32_t addr,int len);include/hw/vfio/vfio-common.h +1
-0@@ -171,6 +171,7 @@ int vfio_get_region_info(VFIODevice *vbasedev, int index, struct vfio_region_info **info);int vfio_get_dev_region_info (VFIODevice *vbasedev,uint32_t type,uint32_t subtype,struct vfio_region_info **info); +bool vfio_is_cap_present (VFIODevice *vbasedev,uint16_t cap_type,int region);#endif extern const MemoryListener vfio_prereg_listener;linux-headers/linux/vfio.h +5
-0@@ -300,6 +300,11 @@ struct vfio_region_info_cap_type {#define VFIO_REGION_SUBTYPE_INTEL_IGD_HOST_CFG (2)#define VFIO_REGION_SUBTYPE_INTEL_IGD_LPC_CFG (3) +/* +* The MSIX mappable capability informs that MSIX data of a BAR can be mmapped. +*/ +#define VFIO_REGION_INFO_CAP_MSIX_MAPPABLE 3 +/** * VFIO_DEVICE_GET_IRQ_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 9, * struct vfio_irq_info) hw/ppc/spapr.c +9
-1@@ -3742,13 +3742,21 @@ static const TypeInfo spapr_machine_info = {/* * pseries-2.11 */ +#define SPAPR_COMPAT_2_11 \ + HW_COMPAT_2_10 \ + { \ + .driver = "vfio-pci", \ + .property = "msix-no-mmap", \ + .value = "on", \ + }, \ +static void spapr_machine_2_11_instance_options (MachineState *machine) { }static void spapr_machine_2_11_class_options (MachineClass *mc) { -/* Defaults for the latest behaviour inherited from the base class */ +SET_MACHINE_COMPAT (mc,SPAPR_COMPAT_2_11 ); } DEFINE_SPAPR_MACHINE(2_11, "2.11", true);hw/vfio/common.c +15
-0@@ -1408,6 +1408,21 @@ int vfio_get_dev_region_info(VFIODevice *vbasedev, uint32_t type,return -ENODEV ; } +bool vfio_is_cap_present (VFIODevice *vbasedev,uint16_t cap_type,int region) +{ +struct vfio_region_info *info =NULL ; +bool ret = false; + +if (!vfio_get_region_info (vbasedev, region, &info)) { +if (vfio_get_region_info_cap (info, cap_type)) { + ret = true; + } +g_free (info); + } + +return ret; +} +/* * Interfaces for IBM EEH (Enhanced Error Handling) */ hw/vfio/pci.c +6
-0@@ -1289,6 +1289,12 @@ static void vfio_pci_fixup_msix_region(VFIOPCIDevice *vdev)off_t start, end;VFIORegion *region = &vdev->bars[vdev->msix->table_bar].region; +if (!vdev->msix_no_mmap && +vfio_is_cap_present (&vdev->vbasedev,VFIO_REGION_INFO_CAP_MSIX_MAPPABLE , + vdev->msix->table_bar)) { +return ; + } +/* * We expect to find a single mmap covering the whole BAR, anything else * means it's either unsupported or already setup. hw/vfio/pci.c static int vfio_msix_setup(VFIOPCIDevice *vdev, int pos, Error **errp) · line 1484
Quoted 6 lines
hw/vfio/pci.c +3
-0@@ -1473,6 +1479,10 @@ static int vfio_msix_setup(VFIOPCIDevice *vdev, int pos, Error **errp)*/ memory_region_set_enabled (&vdev->pdev.msix_pba_mmio, false); +if (!vdev->msix_no_mmap) { +memory_region_set_enabled (&vdev->pdev.msix_table_mmio, false); + }No, you're conflating issues. There's (a) can we mmap over the MSI-X vector table and (b) do we require QEMU emulation of the MSI-X vector table. (a) does NOT imply (b). AFAICT, (a) should be enabled any time the kernel supports it,
Quoted 2 lines
Quoted 2 lines
This is the default setting, or you do not want it to be a property so the user cannot shoot himself in a foot?
If the kernel allows mmap, other than debugging, why would it ever need to be disabled?
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
(b) should never be enabled on ppc, regardless
of (a). Thanks,The intention is to have a property - msix_no_mmap=true, except a single case - ppc-pseries, I just do not know how to enforce it for a specific machine type.
Quoted 6 lines
The intention is wrong. (a) should be done any time the kernel allows it. (b) is an independent concept of disabling QEMU MSI-X emulation for platforms, ie. machine types, that do not require it. (b) has nothing to do with the mmap'ability of the msix table area. So far (b) includes only the ppc spapr machine and I don't see a reason to allow the user to control this. Thanks,
I don't either, but setting properties like this seems to be the more-or-less standard way for the machine type (and/or version) to affect operation of other devices.
Allowing the user to shoot themselves in the foot is a side effect of that, but seems to be one we're ok with.
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Quoted 11 lines · Alexey Kardashevskiy
This makes use of a new
VFIO_REGION_INFO_CAP_MSIX_MAPPABLEcapability which tells that a region with MSIX data can be mapped entirely, i.e. the VFIO PCI driver won't prevent MSIX vectors area from being mapped.This adds a "msix-no-mmap" property to the vfio-pci device, it is "true" by default and "false" for pseries-2.12+ machines.
This requites kernel's "vfio-pci: Allow mapping MSIX BAR"
https://www.spinics.net/lists/kvm/msg160282.htmlSigned-off-by Alexey Kardashevskiy <[email protected]>
Quoted 8 lines
---
This is an RFC as it requires kernel headers update which is not there yet.
I'd like to make it "msix-mmap" (without "no") but could not find a way of enabling a device property for machine versions newer than some value.
I changed 2.11 machine just for the demonstration purpose.
As Alex says, the mmap()ability of the MSI-X BAR isn't really the point. The point is whether we need to intercept guest MMIOs to the MSI-X region. Still, the logic's basically right, just rename your property to, say, "intercept_msix_mmio". It would be true by default, set to false by the pseries machine type.
I don't think you actually need to make it vary depending on the version of the pseries machine type: whether the BAR is mmap()ed or qemu emulated shouldn't be a guest visible change. No PAPR guest should have been directly poking the MSI-X region (ever), so we shouldn't need to intercept the region even for old versions.
Quoted 140 lines
--- hw/vfio/pci.h | 1 + include/hw/vfio/vfio-common.h | 1 + linux-headers/linux/vfio.h | 5 +++++ hw/ppc/spapr.c | 10 +++++++++- hw/vfio/common.c | 15 +++++++++++++++ hw/vfio/pci.c | 11 +++++++++++ 6 files changed, 42 insertions(+), 1 deletion(-)hw/vfio/pci.h +1
-0@@ -142,6 +142,7 @@ typedef struct VFIOPCIDevice {bool no_kvm_intx;bool no_kvm_msi;bool no_kvm_msix; +bool msix_no_mmap; } VFIOPCIDevice;uint32_t vfio_pci_read_config (PCIDevice *pdev,uint32_t addr,int len);include/hw/vfio/vfio-common.h +1
-0@@ -171,6 +171,7 @@ int vfio_get_region_info(VFIODevice *vbasedev, int index, struct vfio_region_info **info);int vfio_get_dev_region_info (VFIODevice *vbasedev,uint32_t type,uint32_t subtype,struct vfio_region_info **info); +bool vfio_is_cap_present (VFIODevice *vbasedev,uint16_t cap_type,int region);#endif extern const MemoryListener vfio_prereg_listener;linux-headers/linux/vfio.h +5
-0@@ -300,6 +300,11 @@ struct vfio_region_info_cap_type {#define VFIO_REGION_SUBTYPE_INTEL_IGD_HOST_CFG (2)#define VFIO_REGION_SUBTYPE_INTEL_IGD_LPC_CFG (3) +/* +* The MSIX mappable capability informs that MSIX data of a BAR can be mmapped. +*/ +#define VFIO_REGION_INFO_CAP_MSIX_MAPPABLE 3 +/** * VFIO_DEVICE_GET_IRQ_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 9, * struct vfio_irq_info) hw/ppc/spapr.c +9
-1@@ -3742,13 +3742,21 @@ static const TypeInfo spapr_machine_info = {/* * pseries-2.11 */ +#define SPAPR_COMPAT_2_11 \ + HW_COMPAT_2_10 \ + { \ + .driver = "vfio-pci", \ + .property = "msix-no-mmap", \ + .value = "on", \ + }, \ +static void spapr_machine_2_11_instance_options (MachineState *machine) { }static void spapr_machine_2_11_class_options (MachineClass *mc) { -/* Defaults for the latest behaviour inherited from the base class */ +SET_MACHINE_COMPAT (mc,SPAPR_COMPAT_2_11 ); } DEFINE_SPAPR_MACHINE(2_11, "2.11", true);hw/vfio/common.c +15
-0@@ -1408,6 +1408,21 @@ int vfio_get_dev_region_info(VFIODevice *vbasedev, uint32_t type,return -ENODEV ; } +bool vfio_is_cap_present (VFIODevice *vbasedev,uint16_t cap_type,int region) +{ +struct vfio_region_info *info =NULL ; +bool ret = false; + +if (!vfio_get_region_info (vbasedev, region, &info)) { +if (vfio_get_region_info_cap (info, cap_type)) { + ret = true; + } +g_free (info); + } + +return ret; +} +/* * Interfaces for IBM EEH (Enhanced Error Handling) */ hw/vfio/pci.c +11
-0@@ -1289,6 +1289,12 @@ static void vfio_pci_fixup_msix_region(VFIOPCIDevice *vdev)off_t start, end;VFIORegion *region = &vdev->bars[vdev->msix->table_bar].region; +if (!vdev->msix_no_mmap && +vfio_is_cap_present (&vdev->vbasedev,VFIO_REGION_INFO_CAP_MSIX_MAPPABLE , + vdev->msix->table_bar)) { +return ; + } +/* * We expect to find a single mmap covering the whole BAR, anything else * means it's either unsupported or already setup. @@ -1473,6 +1479,10 @@ static int vfio_msix_setup(VFIOPCIDevice *vdev, int pos, Error **errp)*/ memory_region_set_enabled (&vdev->pdev.msix_pba_mmio, false); +if (!vdev->msix_no_mmap) { +memory_region_set_enabled (&vdev->pdev.msix_table_mmio, false); + } +return 0 ; }@@ -2986,6 +2996,7 @@ static Property vfio_pci_dev_properties[] = {DEFINE_PROP_BIT ("x-igd-opregion" , VFIOPCIDevice, features,VFIO_FEATURE_ENABLE_IGD_OPREGION_BIT , false),DEFINE_PROP_BOOL ("x-no-mmap" , VFIOPCIDevice, vbasedev.no_mmap, false), +DEFINE_PROP_BOOL ("msix-no-mmap" , VFIOPCIDevice, msix_no_mmap, true),DEFINE_PROP_BOOL ("x-no-kvm-intx" , VFIOPCIDevice, no_kvm_intx, false),DEFINE_PROP_BOOL ("x-no-kvm-msi" , VFIOPCIDevice, no_kvm_msi, false), DEFINE_PROP_BOOL("x-no-kvm-msix", VFIOPCIDevice, no_kvm_msix, false),
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Quoted 27 lines · On Fri, 15 Dec 2017 15:07:31 +1100 David Gibson <[email protected]>…
Quoted 11 lines · Alexey Kardashevskiy
This makes use of a new
VFIO_REGION_INFO_CAP_MSIX_MAPPABLEcapability which tells that a region with MSIX data can be mapped entirely, i.e. the VFIO PCI driver won't prevent MSIX vectors area from being mapped.This adds a "msix-no-mmap" property to the vfio-pci device, it is "true" by default and "false" for pseries-2.12+ machines.
This requites kernel's "vfio-pci: Allow mapping MSIX BAR"
https://www.spinics.net/lists/kvm/msg160282.htmlSigned-off-by Alexey Kardashevskiy <[email protected]>Quoted 8 lines
---
This is an RFC as it requires kernel headers update which is not there yet.
I'd like to make it "msix-mmap" (without "no") but could not find a way of enabling a device property for machine versions newer than some value.
I changed 2.11 machine just for the demonstration purpose.
As Alex says, the mmap()ability of the MSI-X BAR isn't really the point. The point is whether we need to intercept guest MMIOs to the MSI-X region. Still, the logic's basically right, just rename your property to, say, "
intercept_msix_mmio". It would be true by default, set to false by the pseries machine type.
Quoted 5 lines
I don't think you actually need to make it vary depending on the version of the pseries machine type: whether the BAR is mmap()ed or qemu emulated shouldn't be a guest visible change. No PAPR guest should have been directly poking the MSI-X region (ever), so we shouldn't need to intercept the region even for old versions.
I have to ask, is the vfio-pci driver really the right point in the VM to be understanding whether the platform requires MSI-X MMIO emulation? vfio-pci is only unique here in that enabling that emulation harms performance, but AIUI it's unused on any device and there may eventually be other devices affected in the same way as vfio-pci. So should there be some post-realize platform code that disables MSI-X MemoryRegions or should the MSI-X code call out to some platform hook to determine whether to enable emulation? Seems like a case where the impact might be unique to vfio, but the root of the problem is not. Thanks,
Alex
Quoted 36 lines · Alex Williamson
Quoted 27 lines · On Fri, 15 Dec 2017 15:07:31 +1100 David Gibson <[email protected]>…
Quoted 11 lines · Alexey Kardashevskiy
This makes use of a new
VFIO_REGION_INFO_CAP_MSIX_MAPPABLEcapability which tells that a region with MSIX data can be mapped entirely, i.e. the VFIO PCI driver won't prevent MSIX vectors area from being mapped.This adds a "msix-no-mmap" property to the vfio-pci device, it is "true" by default and "false" for pseries-2.12+ machines.
This requites kernel's "vfio-pci: Allow mapping MSIX BAR"
https://www.spinics.net/lists/kvm/msg160282.htmlSigned-off-by Alexey Kardashevskiy <[email protected]>Quoted 8 lines
---
This is an RFC as it requires kernel headers update which is not there yet.
I'd like to make it "msix-mmap" (without "no") but could not find a way of enabling a device property for machine versions newer than some value.
I changed 2.11 machine just for the demonstration purpose.
As Alex says, the mmap()ability of the MSI-X BAR isn't really the point. The point is whether we need to intercept guest MMIOs to the MSI-X region. Still, the logic's basically right, just rename your property to, say, "
intercept_msix_mmio". It would be true by default, set to false by the pseries machine type.Quoted 5 lines
I don't think you actually need to make it vary depending on the version of the pseries machine type: whether the BAR is mmap()ed or qemu emulated shouldn't be a guest visible change. No PAPR guest should have been directly poking the MSI-X region (ever), so we shouldn't need to intercept the region even for old versions.
Quoted 10 lines
I have to ask, is the vfio-pci driver really the right point in the VM to be understanding whether the platform requires MSI-X MMIO emulation? vfio-pci is only unique here in that enabling that emulation harms performance, but AIUI it's unused on any device and there may eventually be other devices affected in the same way as vfio-pci. So should there be some post-realize platform code that disables MSI-X MemoryRegions or should the MSI-X code call out to some platform hook to determine whether to enable emulation? Seems like a case where the impact might be unique to vfio, but the root of the problem is not. Thanks,
That's a good point. If we can reasonably do it at the level of a generic PCI device, that would be preferable.
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson